noamaanMulla-03/transformers-issues huggingface dataset:
Flash Attention on Finetuned Models
Author: RohitMidha23
Pull request: no

### System Info


- `transformers` version: 4.38.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.2 (gpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Finetuned a model as mentioned in the HuggingFace Blog.
2. Using InsanelyFastWhisper for predictions with the `translate` task.

```python
File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/transformers/models/whisper/modeling_whisper.py", line 432, in forward
raise ValueError("WhisperFlashAttention2 attention does not support output_attentions")
ValueError: WhisperFlashAttention2 attention does not support output_attentions
```

### Expected behavior

FlashAttention2 should work with finetuned models.

Comments:
Hey @RohitMidha23 - could you share a reproducible code snippet for your code? Without it, it's hard to say what the error is here, but it's likely fixed by ensuring `output_attentions=False`. This is because outputting the attentions is not compatible with Flash Attention 2.

Note for this, you will also need to disable word-level timestamps, since these implicitly set `output_attentions=True` in the generation code: https://github.com/huggingface/transformers/blob/536ea2aca234fb48c5c69769431d643b0d93b233/src/transformers/models/whisper/generation_whisper.py#L1013-L1016

You're right, it was due to word level timestamps. Thanks @sanchit-gandhi 

I need word level timestamps. How can I fix this? I know that it's possible because the insanely fast whisper replit hosting has it and it works there.

i have the exact same problem and i need the word level timestamps. how shoul i fix it?
where should i set `output_attentions=False`?

You should be able to set that during model initialization phase.

```python
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    STT_MODEL_PATH, 
    torch_dtype=torch_dtype, 
    low_cpu_mem_usage=True, 
    use_safetensors=True,
    attn_implementation="flash_attention_2" if self.device == "cuda:0" else "eager"
)
model.to(self.device)
model.generation_config.output_attentions = False
```
Update trainer for easier handling of accumulate, compile fixes, and proper reporting
Author: muellerzr
Pull request: yes

# What does this PR do?

Alternative to https://github.com/huggingface/transformers/pull/34442

TL;DR we just need to remove `lru_cache` and everything will work fine. (and adds a test)

This PR also takes the full lessons from my article and adds it to the `Trainer` for a simpler solution to the grad accum calculation (we shouldn't rely on `accelerator` from now on bc it can't handle the nuances with the grad accum fix at the highest level API, so we use a lower level version instead)

Fixes https://github.com/huggingface/transformers/issues/34402

**Would recommend a patch after this**


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Milad335t just warning you to stop spamming or we'll have to block you 😢
`SeamlessM4TForTextToSpeech.generate` not working if `generation_config` is passed
Author: ydshieh
Pull request: no
Labels: name: bug

For `SeamlessM4TForTextToSpeech` with checkpoint ` "facebook/hf-seamless-m4t-medium"`, 

> model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)

will fail with index error.

If not passing `generation_config` argument, it could run without error.

### Reproduction

```python
from transformers import AutoProcessor, SeamlessM4TForTextToSpeech

ckpt = "facebook/hf-seamless-m4t-medium"

processor = AutoProcessor.from_pretrained(ckpt)
model = SeamlessM4TForTextToSpeech.from_pretrained(ckpt)

model_inputs = processor(["This is a test"], return_tensors="pt")

# This works
outputs = model.generate(**model_inputs, tgt_lang="eng")
print(outputs)

# Index error
outputs2 = model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)
print(outputs2)
```

### Expected behavior

Should not fail if `generation_config` is passed.

Comments:
cc @ylacombe 

The traceback:

```bash
Traceback (most recent call last):
  File "/transformers/temp.py", line 16, in <module>
    outputs2 = model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/transformers/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py", line 3485, in generate
    waveform, waveform_lengths = self.vocoder(input_ids=unit_ids, spkr_id=spkr_id, lang_id=vocoder_tgt_lang_id)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/transformers/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py", line 2574, in forward
    hidden_states = self.unit_embedding(input_ids).transpose(1, 2)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

```

cc @eustlb as well!
[i18n-ar] Translated file : `docs/source/ar/perf_train_gpu_one.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/perf_train_gpu_one.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
`num_items_in_batch` should divided  by `gradient_accumulation_steps`
Author: Tobin-rgb
Pull request: yes

when calculating the gradient we divide by `gradient_accumulation_steps`

https://github.com/huggingface/accelerate/blob/b92fb4774f242ebdc1ecd86548857f07085be403/src/accelerate/accelerator.py#L2236-L2238




Comments:
How to load local transformers?
Author: yiyexy
Pull request: no

transformers==4.47.0.dev0
 
I want to use my local transformers. And I tried to set `sys.insert(0,'xxx/transformers/src')` and `PYTHONPATH=xxx/transformers/src`, but they doesn't work.

PLZ, tell me why.

Comments:
The easiest way to do this is to go to the directory of your local clone of the `transformers` repo and `pip install -e .`

This will install the pip package in editable mode, and any changes you make in that directory will be reflected when you `import transformers`.

@Rocketknight1  THX for your suggestion. I will try to do it!
Shape mismatch in RoPE embeddings gpt_neox model when rotary_ndims is odd
Author: mseeger
Pull request: no

This might be a bug, it certainly puzzles me.

It considers RoPE embeddings used in the gpt_neox model. In short, if `self.rotary_ndims` in `GPTNeoXAttention` is odd, then the `cos`, `sin` tensors in `position_embeddings` have a final dimension which is 1 too large, and so `apply_rotary_pos_emb` should fail in this case, unless maybe `rotary_ndims = 1`.

https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_rope_utils.py#L67:
`dim` is equal to `rotary_ndims`. If this is odd, the size of `inv_freq` is `ceil(dim / 2)`.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py#L554:
`self.inv_freq` has size `if_sz = ceil(rotary_ndims / 2)`. If `position_ids` has shape `(bs, seq_len)`, then `freqs` has shape `(bs, seq_len, if_sz)`, and `emb`, `cos`, `sin` have shape `(bs, seq_len, 2*if_sz)`. If `rotary_ndims` is odd, this is `(bs, seq_len, rotary_ndims + 1)`. The final dimension is 1 too large.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py#L585:
`apply_rotary_pos_emb` is called with `q`, `k` of shape `(bs, nh, seq_len, rnd)`, where `rnd = rotary_ndims`, see here:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py#L431.
But `cos`, `sin` have shape `(bs, seq_len, rnd+1)` if `rnd` is odd. So the computation of `q_embed` should fail. The only exception would be if `rnd == 1`, in which case broadcasting kicks in.

Maybe you never run this with `rotary_ndims` odd, but not equal to 1? But I think this should be fixed. Tell me if I am wrong.


Comments:
I can reproduce this, so I close this and file a bug report.
matrix multiplication
Author: LidaMsv
Pull request: no

Hello everyone.

Can someone please tell me where and in which files of the generation folder the matrix multiplication is happening? I'm having a hard time finding that. I need specific and detailed answers. Thank you all in advance

Comments:
The matrix multiplication? What matrix multiplication? There are many, many, many matrix multiplication happening.

> I need specific and detailed answers

We are first going to need specific and details questions if we are to help you

Sure. I apologize for not being more specific. My code uses the Generate's _beam_search function and it's confusing for me to find the matrix multiplication of the eos_tokens of my input data. I mean the calculated score for the eos data. Is it in the beam_search.py file? If yes, which part of it exactly?

I also want to know the format of the matrix multiplication cause I don't think it's using matmul from torch library. If you could just give me some insights about where and how are one of these matrix multiplications?


Thank you very much for your time.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding Mamba2ForTokenClassification to Mamba2
Author: ryan-minato
Pull request: no
Labels: name: Feature request

### Feature request

I’ve noticed that many newly added models do not include a `ForTokenClassification` implementation. Is this due to fundamental challenges in implementation (though I don’t perceive any major obstacles—perhaps I’ve overlooked something), or is it simply a matter of development priorities and time constraints?

### Motivation

I am currently testing a prototype based on the Mamba series models, which requires token classification outputs.

### Your contribution

If it’s merely a time issue preventing the implementation of `ForTokenClassification` in `transformers`, I’d be more than willing to contribute by adding this feature for Mamba/Mamba2. If time allows, I’d also be happy to extend the support to other models.

From my understanding, replicating the approach used in `LlamaForTokenClassification` should suffice to implement token classification model for most models. Any advice or guidance would be highly appreciated!

Comments:
Hi @ryan-minato, the reason for this is that `token-classification` is often challenging for models trained as causal language models. In the past, many popular models like BERT and RoBERTa were masked language models, but now almost all new models are causal.

With a masked language model like BERT, the model is trained with all tokens able to attend forwards and backwards through the sequence. This makes fine-tuning it as a `token-classification` model easy. With a causal language model like Llama, the model is trained with tokens only able to attend backwards in the sequence. This makes it difficult to fine-tune them as token classifiers, because it is very hard for tokens to guess their label when they cannot see the context on both sides. (For example - the token `New` needs to know the next token is `York` if it wants to classify itself correctly as a `Location`!)

As a result, we generally don't have `ForTokenClassification` implementations for causal language models. I'm aware that there is an implementation for `Llama`, but I'm not sure if anyone uses it, and I suspect performance is quite bad!

Thank you for your response and detailed explanation. I understand how the absence of bidirectional attention affects token classification. 
I’m currently working on a task that requires using Mamba2 to assign scores to individual tokens, and its linear vram consumption is a critical feature for my use case.

I’d like to ask: if I were to implement an interface similar to LlamaForTokenClassification, would it then be compatible with the Trainer for training purposes?

@ryan-minato Yes, it should be!
Breaking change due to `multiprocessing.Process` when loading `pytorch_model.bin`-based model 
Author: tomaarsen
Pull request: no

## Bug Overview
* Loading any `transformers` models fails if:
  * the model only has a `pytorch_model.bin`, and
  * you're not inside of `__name__ == "__main__"`

(Taken from https://github.com/huggingface/transformers/pull/34966#issuecomment-2538598145)

## Details
In short: `multiprocessing.Process` never works when not inside of `__name__ == "__main__"`. I recognize that most programs should be using that line, but I'd rather not force it on my users.

If one of my users loads any model that only has a `pytorch_model.bin`, then it'll fail, e.g.:
```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("embaas/sentence-transformers-gte-base")
```
or
```python
from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
```
which internally call
```python
from transformers import AutoModel

model = AutoModel.from_pretrained("embaas/sentence-transformers-gte-base")
```
or
```python
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("cross-encoder/ms-marco-MiniLM-L-6-v2")
```

All of these get:
```
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html
```

Edit: To prevent people experiencing errors, I've updated all [`cross-encoder` models](https://huggingface.co/cross-encoder) to safetensors. So you can't reproduce those anymore without specifying the `revision`.

- Tom Aarsen

Comments:
On it :-) Thanks for reporting
Change back to `Thread` for SF conversion
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #35228

On other platforms likke Mac / Windows, using `Process` (see #34966) is even much cost than `Thread` (on Linux, the cost is also higher but not that much).

In a CI environment, it may occurs many tests will eventually call `auto_conversion` (within `from_pretrained`), and the accumulation of using `Process` is too high and cause `PEFT (Mac / Windows) CI` running time increase a lot:

> Windows (~1h:22min) and MacOS (~52min) compared to Ubuntu (~22min)

This PR changes it back to using `Thread` and achieve the original goal of #34966 in another way.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35236). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added RAdamScheduleFree support | updated codecarbon to run test_trainer.py properly
Author: nhamanasu
Pull request: yes

# What does this PR do?

Integrates meta's https://github.com/facebookresearch/schedule_free for RAdam.
This PR greatly followed the previous PR to introduce schedulefree https://github.com/huggingface/transformers/pull/30079

## More details
- Added support for using RAdamScheduleFree through Trainer API
- Refined doc for trainer (clarified that we recommend to use constant LR scheduler with Schedule-Free family)
- Added test case for RAdamScheduleFree & confirmed the test passes
  - Environment: RTX 6000Ada (no distributed training, fp32)
  - 113 passed, 49 skipped, 52 warnings in 133.76s (0:02:13)

## Backgrounds
I'm the developer who recently released the RAdamScheduleFree optimizer.
It can be imported via schedulefree library since version 1.4.

- [The author of Schedule-Free paper's announcement post](https://x.com/aaron_defazio/status/1863722349138219459)
- [My announcement and release blog (in Japanese)](https://x.com/hamanasu_nagisa/status/1866413964235821492)

RAdamScheduleFree eliminates:
- Learning rate scheduler and terminal steps
- Warmup steps

## Bug Fixes
- Related to [this codecarbon PR](https://github.com/mlco2/codecarbon/pull/391), I found we need to update codecarbon to run test_trainer.py properly in Python 3.11 or above. In current main branch, `"codecarbon==1.2.0"` was set, so I rewrote this to `"codecarbon>=2.8.1"`. It properly worked.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr @younesbelkada @pacman100


Comments:
CircleCI shows "[tests_generate](https://app.circleci.com/pipelines/github/huggingface/transformers/113356/workflows/1e152e04-8f26-496a-9925-549ac7435e88/jobs/1514539?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary) - Failed", but I think it's not related to my commits...

I decided to split this PR into several smaller PRs to address each concern separately:
- [first PR] https://github.com/huggingface/transformers/pull/35243

I'll close this PR and submit a new one focusing on RAdamScheduleFree once all concerns are resolved.
[i18n-Chinese] Translating perf_infer_gpu_multi.md to Chinese
Author: HMJ0628
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->
I noticed that no one has translated docs/source/en/perf_infer_gpu_multi.md before. Could I translate this to Chinese?


Comments:
Yes, please feel free to! In the future, you can coordinate your translation efforts in #26803 instead of opening a new issue :)