noamaanMulla-03/transformers-issues huggingface dataset:
Mistral with FlashAttention2
Author: khalil-Hennara
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.1
- Accelerate version: 0.27.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16, attn_implementation="flash_attention_2")`
The code line has taken from the official website [Mistral](https://huggingface.co/docs/transformers/v4.37.2/model_doc/mistral#model-details)
TypeError: MistralForCausalLM.__init__() got an unexpected keyword argument 'attn_implementation'


when using `use_flash_attention_2=True` it's work fine 

### Expected behavior

The model should be loaded without error, using flash attention2 in the background.

Comments:
I think the problem related to @ArthurZucker and @stevhliu 

It looks like ‘attn_implementation’ is supported in version 4.36. Maybe you need to try it after upgrading the transfromers library version
![image](https://github.com/huggingface/transformers/assets/13672523/12703f91-a6b4-4688-9b67-50cd5896aa2e)


Yes, as @IYoreI mentions, feel free to upgrade the transformers version! 

Thanks @IYoreI , @ArthurZucker for your time 

Closing as it's resolved!
Prevent MLflow exception from disrupting training
Author: codiceSpaghetti
Pull request: yes

This PR prevents a training in progress from **being interrupted** due to a problem with the MLflow server (e.g., lack of connectivity) and will make the code more **faul-tolerant**, as also discussed in [this](https://github.com/mlflow/mlflow/issues/1550) MLflow issue.

This can be achieved by simply changing the `synchronous` parameter of `mlflow.log_metrics` to `False` (previously the default value was left, which is `True`)

In this way, as described in the [MLflow documentation](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_metrics), the function logs the metrics asynchronously and return a future representing the logging operation, instead of blocking until the log is successful.

Comments:
repeat tokenization
Author: calderma
Pull request: no

Hello, when using the example script run_clm.py in examples/pytorch/language-modeling/ with distributed training it seems to repeat the tokenization for each GPU. The messages “Running tokenizer on dataset” and “Grouping texts in chunks of {block_size}” repeat over and over. This does not happen when running on a single GPU. Also the tokenization takes significantly longer. My script call is:
deepspeed run_clm.py
–model_name_or_path EleutherAI/gpt-neo-125M
–train_file $TRAIN_FILE
–validation_file $VAL_FILE
–block_size $BLOCK_SIZE
–overwrite_output_dir
–fp16
–per_device_train_batch_size 3
–per_device_eval_batch_size 3
–do_train
–do_eval
–group_by_length
–gradient_accumulation_steps 4
–deepspeed “deepspeed_zero2_config.json”
–output_dir $OUTPUT_DIR
This is a huge issue when using a large body of text. Any help would be appreciated.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

i have the same issue, does anyone solve the problem
[`HFQuantizer`] Remove `check_packages_compatibility` logic
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the currently failing tests for AWQ: https://github.com/huggingface/transformers/actions/runs/7705429360/job/21003940543
I propose to remove the `check_package_compatiblity` logic in the `HfQuantizer` as:1

1- it is a duplicate of `validate_environment`
2- For some packages such as awq, `_is_package_available()` returns False because `importlib.util.find_spec(pkg_name) is not None` retruns correctly `True` but `importlib.metadata.version(pkg_name)` fails since autoawq is registered as `awq` module but the pypi package name is `autoawq`. 

As I expect to face similar behaviour in future quantization packages I propose to simply remove that logic and handle everything in `validate_environment`

cc @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28789). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Can you please provide the longformer version of the torch to tf file?
Author: Struggle-lsl
Pull request: no

### Feature request

Can you please provide the longformer version of the torch to tf file?

### Motivation

Can you please provide the longformer version of the torch to tf file?

### Your contribution

Can you please provide the longformer version of the torch to tf file?

Comments:
Hi @lsl200032, 

There isn't a specific file for Longformer to convert from torch to TF weights. The conversion happens when using `from_pt` in the `from_pretrained` call: 

```
from transformers import TFLongformerModel 

# Loads the pytorch weights and converts them to the equivalent TF format then loads into the TF model
model = TFLongformerModel.from_pretrained(checkpoint, from_pt=True)
```

I have a longformer weight file that has been retrained in a downstream task, but it is a .bin file. I want to load it under the tf framework. How to solve it?

> 你好@lsl200032,
> 
> Longformer 没有将火炬权重转换为 TF 权重的特定文件。`from_pt`在调用中使用时会发生转换`from_pretrained`：
> 
> ```
> from transformers import TFLongformerModel 
> 
> # Loads the pytorch weights and converts them to the equivalent TF format then loads into the TF model
> model = TFLongformerModel.from_pretrained(checkpoint, from_pt=True)
> ```

I have a longformer weight file that has been retrained in a downstream task, but it is a .bin file. I want to load it under the tf framework. How to solve it?

If the model was trained using Hugging Face's Longformer pytorch architecture, then you can use the code I mentioned above to load it as a TF model

```
from transformers import TFLongformerModel 

# Loads the pytorch weights and converts them to the equivalent TF format then loads into the TF model
model = TFLongformerModel.from_pretrained(path/to/model/folder, from_pt=True)
```

> If the model was trained using Hugging Face's Longformer pytorch architecture, then you can use the code I mentioned above to load it as a TF model
> 
> ```
> from transformers import TFLongformerModel 
> 
> # Loads the pytorch weights and converts them to the equivalent TF format then loads into the TF model
> model = TFLongformerModel.from_pretrained(path/to/model/folder, from_pt=True)
> ```

Thank you beautiful lady, your answer has given me confidence in my scientific research path. I hope to keep in touch with you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Implement Cross Attention in LLAMA Model
Author: eitamarSaraf
Pull request: no

### Feature request

The current implementation of the LLAMA model in the Hugging Face Transformers repository supports self-attention layers as per the standard design of transformer models. I propose the addition of an option to use several or all attention layers as cross-attention layers instead of self-attention layers. 

Cross-attention layers are crucial for tasks where the model needs to attend to different inputs other than its own output (e.g., encoder-decoder tasks in translation, image-captioning, etc.). The option to use cross-attention would enhance the LLAMA model's capabilities for a broader range of applications.


https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py

### Motivation

My motivation for this proposal stems from the need to apply the LLAMA model to tasks that inherently require cross-modal attention mechanisms. The current limitation of self-attention only restricts its applicability. While self-attention mechanisms are effective for a range of tasks, the flexibility of cross-attention layers could extend the model's utility, allowing researchers and developers to tackle a wider variety of problems.


### Your contribution

I am willing to assist in the implementation of this feature. While I am not an expert in decoder-only architecture, I believe with the right guidance, I can help.

I look forward to discussing this further with the maintainers of the repository.

Thank you for considering my proposal.

Comments:
Resolve DeepSpeed cannot resume training with PeftModel
Author: lh0x00
Pull request: yes

Hi all,

I found an issue about resume ft **PeftModel** with **DeepSpeed** while I using `accelerate launch` to follow [zephyr-7b-beta recipes](https://github.com/huggingface/alignment-handbook/tree/c74ed111710d57f563cfbf1806cfb8f07dd3dc67/recipes/zephyr-7b-beta) with **qlora**. 
In details, the process will be crashing on load resume checkpoint with **DeepSpeed** with **PeftModel**. I referred to https://github.com/ymcui/Chinese-LLaMA-Alpaca/issues/559#issuecomment-1585948697 and created a PR to resolve this issue. I was updated and it works accurately on my fork.

Thanks for review @pacman100 @muellerzr 


Comments:
hiii, any feedback are welcome, I waiting to a new version to use instead of a fork version or give me feedback if my pr having any issues. many thanks @pacman100 @muellerzr 

Nice! Assuming this works, really excited for it to get merged in, as I've had this issue with checkpoints.

Quick one - it looks like your local formatter may have different line length settings to the default for `transformers`, since the diff shows a bunch of changes that aren't relevant to your updates. Up to you if you want to revert those, but I'd hate for them to hold up the PR :)

(I don't expect it to be a problem since the style and quality checks passed, but it would make the PR look smaller and keep the file consistent)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28746). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @younesbelkada

yeah, @nathan-az  I was able to continue my fine tuning, and it worked well. You can use it through my branch while waiting for the PR to be merged.

waiting for this merge 

me too, waiting for this merge -.-

Hi everyone! Sure yes, please give us some time to review properly this PR and we'll make sure the fix is going to be landed! 

thanks for reviewed, @pacman100 @younesbelkada
Add InternLM1 & InternLM2 model
Author: PommesPeter
Pull request: no
Labels: name: New model

### Model description

Hey,
the recently released [InternLM](https://github.com/InternLM/InternLM) seems like it would be a nice addition to transformers.

Basically, the model has achieved performance that currently exceeds LLaMA2, Mistral, and other models on many Benchmarks. Adding the model to transformers makes it easier to use the model. It has two parameter sizes of 7B and 20B, for v1 and v2 versions. In addition, it also includes three types of models: base, sft, and chat.

Maybe there are already plans of integrating it @NielsRogge ?

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Project Page: https://internlm.intern-ai.org.cn/
GitHub Repo: https://github.com/InternLM/InternLM

Comments:
Hey! Thanks, seems like it's already supported: #26302 

> Hey! Thanks, seems like it's already supported: #26302

Alright. I will find another "good" model to implement. Closed.
Wrap Keras methods to support BatchEncoding
Author: Rocketknight1
Pull request: yes

One last Keras PR before I go back to chat templates - a recurring annoyance that I (and the forum users) have always had with our Keras models is that our tokenizers output `BatchEncoding` by default, which behaves like a mixed dict/list. Keras doesn't understand this at all and fails to handle it when passed to `fit()` or `predict()`. The result is that you have to manually remember to convert tokenizer outputs to a dict or you get a confusing error.

The right time to do this was about two and a half years ago, but late is better than never! This PR wraps the Keras methods to do that transparently, without changing other behaviour. Note that because we're changing the exact Keras we're importing, this PR shouldn't be merged before the `tf_keras` PR is in.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28734). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This has now been rebased following the tf-keras PR, will merge once the CI is green.
DeepSpeed: hardcode `torch.arange` dtype on `float` usage to avoid incorrect initialization
Author: gante
Pull request: yes

# What does this PR do?

Addresses #28685 -- check the issue (and related issues) for a full discussion.

TL;DR: some frameworks, such as DeepSpeed, may patch the initialization of a tensor. For instance, a `float32` tensor may be initialized as `bfloat16` instead. This is particularly problematic when `torch.arange` is used as a non-integer: its initialized value may be a source of problems if not in the right type. This PR casts `torch.arange` to `int64` at initialization time, preventing the frameworks' float type conversion, and subsequently casts the tensor to the desired type.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28760). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker I've added a test case that shows how it was failing before 👍  It only covers one model, though, as the test has to be tailored to each model.

The test will also be useful to show to users the importance of this ugly pattern :D
Add artifact name in jobs' step to maintain jobs and artifacts correspondence
Author: ydshieh
Pull request: yes

# What does this PR do?

When our (actual) CI workflow files are called using `workflow_call` event by other workflows files, the job names will be concatenated like `Nightly CI / Model Test (models/bert, single-gpu)`.

We currently have Nightly/Past/AMD CI using `workflow_call`. We will soon have to use it for daily CI too due to the 256 matrix jobs limit of GitHub Actions.

So the (model test) job names in daily CI will become something like `Part (0) / Model Test (models/bert, single-gpu)` and nightly CI will have `Nightly CI / Part (0) / Model Test (models/bert, single-gpu)`.

_This makes `utils/notification_service.py` more complex to handle the job names correctly to get job links_

**This PR implements a new approach to maintain the correspondence between jobs, links and artifacts, so `utils/notification_service.py` can have the necessary information more easily.**

Comments:
Before merge,  I will change all workflow files that use `notification_service.py`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28682). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A follow up PR based on  this one is #28773
Split daily CI using 2 level matrix
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR aims to bypass the 256 jobs limitation (in a matrix) on GitHub Actions, as we are approaching that limit soon.

The idea:

- move the model job logic into a new workflow file (it **still uses matrix**)
- call the new workflow file in the original workflow file, but pass some inputs to it
  - a (nested) list: each element is a list: a subset of model names 
  - a slice id

When the new workflow file is called with the inputs, it will use the slice id to get the corresponding subset of model names, and uses the matrix on that to generate the jobs to run.

In the original workflow file, we **generate the slice ids by a matrix**.

See example runs

**Full version**
https://github.com/huggingface/transformers/actions/runs/7701814182
<img width="512" alt="Screenshot 2024-01-30 111539" src="https://github.com/huggingface/transformers/assets/2521628/465554f6-ff92-4ac2-bf27-658353a982b3">



**Demo version**
https://github.com/huggingface/transformers/actions/runs/7702628211

<img width="512" alt="Screenshot 2024-01-30 110004" src="https://github.com/huggingface/transformers/assets/2521628/9262b176-4955-4215-a6c5-977fa124a4a8">




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28773). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
canonical repos moves
Author: julien-c
Pull request: yes

we are phasing out canonical models & datasets (as a reminder, "canonical" repos are those that were not under an org or user namespace) and moving them under ad hoc organization namespaces

Note that this move should be backward compatible i.e. old versions of transformers that do `AutoModel.from_pretrained("gpt2")` should still work. Download stats should also be backward-compatible.

Thanks for reading!

Comments:
Already done :)

too quick

test is flaky no @ArthurZucker ?

Yes merging without it

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks friends!
Add [VMamba] model
Author: dmus
Pull request: no
Labels: name: New model

### Model description

VMamba is a visual foundation model proposed in https://arxiv.org/pdf/2401.10166.pdf.

It is inspired by the recent advances in state stace models and in particular Mamba. The proposed architecture is computationally more efficient than vision transformer architectures because it scales linearly with growing resolution. It introduces a Cross-Scan Module (CSM) to have context from all directions (4 directions, starting in each corner and traversing in a horizontal or vertical direction). Evaluation on vision perception tasks shows promising capabilities.

Model weights will become available in a few days according to the repo of the authors.

1.  [x] (Optional) Understood theoretical aspects

2.  [x] Prepared transformers dev environment

3.  [x] Set up debugging environment of the original repository

4.  [x] Created script that successfully runs forward pass using
    original repository and checkpoint

5.  [x] Successfully opened a PR and added the model skeleton to Transformers

6.  [x] Successfully converted original checkpoint to Transformers
    checkpoint

7.  [x] Successfully ran forward pass in Transformers that gives
    identical output to original checkpoint

8.  [x] Finished model tests in Transformers

9.  [ ] ~~Successfully added Tokenizer in Transformers~~

10. [x] Run end-to-end integration tests

11. [x] Finished docs

12. [ ] Uploaded model weights to the hub

13. [x] Submitted the pull request for review

14. [ ] (Optional) Added a demo notebook


I am opening the issue to avoid duplicate work. My main motivation for porting this model is to learn a bit more about it (and about the internals of 🤗 Transformers). Some of you probably know this library much better than me, so feel free to write your own implementation if you can do it better or quicker. Otherwise, don’t hesitate to build on top of my fork.

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

- Original repo: https://github.com/MzeroMiko/VMamba
- Paper: https://arxiv.org/pdf/2401.10166.pdf
- implementation in progress: 
- youtube vmamba vs vision mamba: https://www.youtube.com/watch?v=RtHDu6kFPb8
- vision mamba paper (similar idea): https://arxiv.org/pdf/2401.09417.pdf

Comments:
Thank you for your attention. I am one of the authors of `VMamba`. We have just renewed the repo with code easier to transplanting. I hope this would helps you in your splendid work!
[docs] Correct the statement in the docstirng of compute_transition_scores in generation/utils.py
Author: Ki-Seki
Pull request: yes

# What does this PR do?

In the `compute_transition_scores` function's docstring, the table in Example 1 should refer to `log probability` instead of `logits`. This is because setting `normalize_logits=True` transforms the `logits` into `log probability`, according: to [generation/utils.py#L1012-L1015](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L1012-L1015)

Below is the table.

```text
        ...     # | token | token string | logits | probability
        ...     print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}")
        |   262 |  the     | -1.414 | 24.33%
        |  1110 |  day     | -2.609 | 7.36%
        |   618 |  when    | -2.010 | 13.40%
        |   356 |  we      | -1.859 | 15.58%
        |   460 |  can     | -2.508 | 8.14%
```

You can also easily check this. It's quite clear that the values in the third column are the logarithms of the probability values in the fourth column, i.e., $ln(\text{forth column}) = \text{third column}$.

This PR updates the term from `logits` to `log probability` in the table. Without this change, users could become confused when utilizing this feature without referring to the source code.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

Comments: