noamaanMulla-03/transformers-issues huggingface dataset:
"cached cross_attention states don't have to be reordered -> they are always the same"
Author: YJYJLee
Pull request: no

### System Info

Sorry, this is not the bug report, but I couldn't find the proper category to ask this question

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

![Screenshot 2024-01-05 at 10 13 07 AM](https://github.com/huggingface/transformers/assets/28900943/d4abaec7-9dfd-4384-94dc-43a120964b19)

I have noticed that in many models, KV cache reordering is not happening for cross_attention states.

### Expected behavior

Could someone explain why we don't need KV cache reordering for cross_attention states?

To my knowledge, KV cache or cross_attention also needs to be reordered according to the selected beam indices during the beam search. But I'm not sure how "they are always the same" for cross_attentions.

Comments:
Cross attention only happends between the encoder and decoder, but the encoded states do not change, you always use the same, the decoder state change. Thus you don't re-compute / change the cross-attention states across the generate function

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Whisper Large-v3 has problems with language detection
Author: gau-nernst
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The audio file: https://drive.google.com/file/d/1EFWm7GpP79NUEmUO6rsLo444OugGRDHP/view?usp=sharing

```python
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3")
output = pipe("sample.flac")
print(output)
```

v3 output: `{'text': ' Mari kita perlahan-lahan dengan penyelidikan baru dan baru yang telah dilakukan tahun lepas.'}`
v2 output: `{'text': " Let's go slow with this new and novel legalization passed last year."}`
tiny.en output: `{'text': " Let's go slow with this new and novel legalization past last year."}`

### Expected behavior

Output should be like v2 and the tiny.en model. I suspect there is something very wrong with language detection. I couldn't run v3 in the original repo (https://github.com/openai/whisper) due to OOM so I'm not sure if this is a problem with the v3 model itself or inference code in HF.

Seems related: https://github.com/huggingface/transformers/issues/27368#issuecomment-1835564217

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Left it running as this is model specific rather than transformers bug IMO

Indeed - feel free to post on the OpenAI repo since it looks like a model regression from large-v2 -> v3: https://github.com/openai/whisper/discussions

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add dataloader prefetch factor in training args and trainer
Author: qmeeus
Pull request: yes

What does this PR do?
I added an option to the trainer to prefetch batches during data loading.

When training a model with heavy transformations and an iterable dataset, the dataloader might struggle to deliver fast enough for the GPU. I've found that prefetching batches helps to solve this issue.
The option is implemented in torch.utils.data.DataLoader but not in HF Trainer.


Comments:
cc @muellerzr 

Thanks for the suggestions @amyeroberts ! I included them in the PR and synced the branch to the latest commits

Have a nice day
Static Cache: load models with MQA or GQA
Author: gante
Pull request: yes

# What does this PR do?

Adds support to loading MQA or GQA models to the static cache (such as [this one](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0))

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28975). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Where size  5 comes from in LlamaModelforCasualLM??
Author: daehuikim
Pull request: no

```
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer
)

model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

input_text = "How are you?"
input_ids = tokenizer.encode(input_text, return_tensors="pt")
outputs = model(input_ids,output_hidden_states=True)

hidden_states = outputs.hidden_states
output_last = hidden_states[-1].detach()
print(output_last.shape)
```
Sorry for question here.
I am watching llama model codes for getting hidden states values from here(https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py)
When I try the code above, I can get the output like this
```
torch.Size([1, 5, 4096])
```
From here, I can understand ```1``` is batch size and ```4096```is size of max_position_embedding of this model.
Then What does ```5``` means from the hidden states of this model? 
(hidden states became logits once they go through ```lm_head``` module)
https://github.com/huggingface/transformers/blob/da20209dbc26a6a870a6e7be87faa657b571b7bc/src/transformers/models/llama/modeling_llama.py#L1155

Thanks for reading my question!

Comments:
Hi @daehuikim, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

The value 5 comes from the input sequence length (the number of input tokens) i.e. `len(input_ids[0])` 

@amyeroberts 
Thank you for your kindness!
[`Mixtral` & `Mistral`] Add support for sdpa
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Adds the SDPA attention for both classes cc @younesbelkada for visibility 😉 Will help for fast LLava

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks ! I don"t see why sliding window attention shouldn't be supported with SDPA because the only difference vs the eager attention implementation is on the attention mask. Passing arbitrary attention masks in SDPA should be supported without any problem IMO

I have the same problem here, why sdpa not support window attention? Is there any problems not been solved? @ArthurZucker 

@ehuaa the way the window attention is implemented in Mistral original code base is by changing the attention mask to a "more custom" attention mask to not attend to tokens that are before `sliding_windows`. Check out more by looking into the details of this method: https://github.com/huggingface/transformers/blob/d90acc16437e8c9e45e068fa1cc1a263b9a7208f/src/transformers/modeling_attn_mask_utils.py#L145
The point that I tried to convey is that passing that attention mask is supported I think in SDPA so you can implicitly get SDPA + sliding window attention by just passing that correct attention mask. Let me know if this makes sense to you!

> @ehuaa the way the window attention is implemented in Mistral original code base is by changing the attention mask to a "more custom" attention mask to not attend to tokens that are before `sliding_windows`. Check out more by looking into the details of this method:
> 
> https://github.com/huggingface/transformers/blob/d90acc16437e8c9e45e068fa1cc1a263b9a7208f/src/transformers/modeling_attn_mask_utils.py#L145
> 
> 
> The point that I tried to convey is that passing that attention mask is supported I think in SDPA so you can implicitly get SDPA + sliding window attention by just passing that correct attention mask. Let me know if this makes sense to you!

@younesbelkada Thank you for your quick reply! Your solution above can pass a custom mask to sdpa, and i think this way is the same as passing sliding_window param to this function.
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L1006-L1023
![image](https://github.com/huggingface/transformers/assets/5137359/60c63bfd-50c5-4eaf-aa24-fac8f31f7a0c)
Automatically add tokens when using model.generate()
Author: MikeDean2367
Pull request: no

### Feature request

When calling `model.generate()`, I hope that when a certain token is generated, a new token is automatically added after it. 

For example, when the model generates "The founder of Apple is Steve," I would like it to add a token "Jobs" at the end automatically. After that, the input of the next step is "The founder of Apple is Steve Jobs".

Below is a snippet of pseudocode to illustrate this concept:
```python
while eos_token:
  if current_last_token == 'Steve':
    # let the `rule` generate the next token
    current_token_seq.append('Jobs')
  else:
    # let the `model` generate the next token
    current_token_seq.append(generate_next_token())
```

### Motivation

In natural language processing, a single person's name can have multiple representations, including abbreviations, full names, and nicknames. In the context of my application, "Steve Jobs" is a frequently used term. Nonetheless, the model might produce variations such as "Steve jobs" with inconsistent capitalization or "Steven Paul Jobs," using his full name. These inconsistencies present challenges during the evaluation phase of my process.

### Your contribution

Do speculative decoding solve it?

Comments:
Hi @MikeDean2367, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

ok, i'll go to the forums. Thanks!
[Broken link] Link not working in LLaMA-2 pretraining_tp doc
Author: AdityaKane2001
Pull request: no

The `this document` link on [LLaMA2 doc page](https://huggingface.co/docs/transformers/model_doc/llama2#transformers.LlamaConfig.pretraining_tp) does not work.

Comments:
transferring you to the transformers repo instead!

@AdityaKane2001 Indeed! Thanks for flagging this. Would you like to open a PR to fix this? This way you get the github contribution. 

Sure
Update configuration_llama.py: fixed broken link
Author: AdityaKane2001
Pull request: yes

Placed the correct link in `pretraining_tp` docstring in configuration_llama.py

/auto Closes #28939 
/cc @amyeroberts

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28946). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts 
Committed change, thanks!


Thanks Amy!
[DO NOT MERGE] Remove big block of code in _from_pretrained()
Author: Rocketknight1
Pull request: yes

I can't figure out what this code is doing, and I suspect we don't need to run it all. I'm going to see what the CI says.

Comments:
> I'm going to see what the CI says.

🙃 

The whole block seems to be there just to throw a warning when you load a tokenizer with the wrong class!
Clean load keys
Author: sgugger
Pull request: yes

# What does this PR do?

This PR finishes the work done in and completely cleans up the `_keys_to_ignore_on_save`, `_keys_to_ignore_on_load_missing` and  `_keys_to_ignore_on_load_unexpected`. Those were used in three situations:

1. Not saving the tied weights. This came from the (wrong) assumption that torch would take twice the space for tied weights (which it doesn't) and also created bugs where non-tied weights were not saved (unless a hack was added like for RoBERTa models). This is not necessary since PyTorch doesn't take more space for tied weights and safetensors will properly remove them (with `_tied_weights_keys`)

2. Ignoring non-saved non-persistent buffers. This can be done automatically in the code of modeling_utils as non-persistent buffers are keys in the model named buffers not in the state dict, so easy to dectect

3. Ignoring known unexpected weights from another architecture (like the pooler). This isn't necessary anymore since we don't issue a warning in this case.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger with this change, a few of our trained models that used the older format no longer are able to properly load unless we set `strict=False` because they contain `embeddings.position_ids` key that no longer exists. I wonder if there is a way to land this change such that it would be backwards compat with older model files as well. I see a few different issues have popped up as a result of this change and a lot of just required loading and resaving the model files but that is sometimes difficult to do at scale.
Annotations not being transformed after padding on Deformable DETR preprocessing
Author: Tengoles
Pull request: no

### System Info

@amyeroberts 

Maybe I'm missing something but it seems like the annotations are not being transformed accordingly after applying pad to a batch of images:
https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/deformable_detr/image_processing_deformable_detr.py#L1330

Is this dealt with further down the train pipeline? when I render the output annotations of that method (encoded_inputs["labels"]) they are incorrect for the images of the batch that required to be padded.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr")

encoding = processor(images=imgs, annotations=targets, return_tensors="pt", 
                                  do_pad=True)



### Expected behavior

Annotations may require transformation just like they are transformed accordingly when applying resize and rescale on previous lines within the same method.

Comments:
/assign

Hi @Tengoles, thanks for raising this issue! 

Indeed, there's an issue with the transformations that are happening in deformable detr and other detr models if `do_pad=True` as the box coordinates aren't rescaled to account for the new image height / width cc @NielsRogge. 

I'll open a PR to update these.
ENH: Do not pass warning message in case `quantization_config` is in config but not passed as an arg
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently, transformers always warns users with a wrong message when loading a model that has a quantization_config without even passing quantization_config to from_pretrained. Indeed we should warn users only when `quantization_config_from_args` instead of all the time 

cc @amyeroberts 

```bash
/usr/local/lib/python3.10/dist-packages/transformers/quantizers/auto.py:151: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be prevail.
  warnings.warn(warning_msg)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28988). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ENH [`AutoQuantizer`]: enhance trainer + not supported quant methods
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently, if a quantization method do not support PEFT fine-tuning an old error with bistandbytes is raised:
```bash
 The model you want to train is loaded in 8-bit precision.  if you want to fine-tune an 8-bit
 model, please make sure that you have installed `bitsandbytes>=0.37.0`. 
```
Regardless of the quantization method. In fact, for example if one uses AWQ + Trainer (which is not supported yet, but soon with https://github.com/huggingface/transformers/pull/28987 / https://github.com/huggingface/peft/pull/1399), they'll get the old error which is very confusing.

Moreover, we should rely on the variable `hf_quantizer.is_trainable` instead of `_is_quantized_training_enabled`

We should instead be more precise and throw an error that states why this is not supported and how to request a fix.

cc @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28991). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AttributeError: ‘EvalPrediction’ object has no attribute ‘prediction’
Author: Lionel98
Pull request: no

Im trying to finetune Minilm via hugging face using the following 

Codes:
![image](https://github.com/huggingface/transformers/assets/92796786/945b681b-3eea-4b49-b710-8387f68227b1)

Error Message:
AttributeError: 'EvalPrediction' object has no attribute 'prediction'



### Who can help?
@sgugger @ArthurZucker @gan



### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)


### Reproduction

Custom Dataset was used and Mapping was done appropriately.

### Expected behavior
A fine tuned model

Comments:
Hey! Looking at your code, your `compute_metric` function is calling `pred.prediction` while it should be `pred.predictions`. 

Much appreciated  tyvm for your help.
[Flax] Freeze params when _do_init=True
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

As of version 0.7.1, Flax defaults to returning **regular dictionaries** with the methods `.init` and `.apply`, not frozen dictionaries as was the case before: https://github.com/google/flax/discussions/3191

This PR shows how we can update the Transformers modelling code to maintain the previous behaviour, where the `init_weights` method returns a frozen dict of params

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28367). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Doc`] Fix docbuilder - make `BackboneMixin` and `BackboneConfigMixin` importable from `utils`. 
Author: amyeroberts
Pull request: yes

# What does this PR do?

Several CIs have started having the doc builds failing e.g.:
* https://github.com/huggingface/transformers/actions/runs/7881443714/job/21529678872
* https://github.com/huggingface/transformers/actions/runs/7884708184/job/21530274589

On one case rerunning lead to a successful build: 
* Failed: https://github.com/huggingface/transformers/actions/runs/7881054565/attempts/1
* Passed: https://github.com/huggingface/transformers/actions/runs/7881054565/job/21518984718

However, trying this on other runs wasn't successful e.g.: 
* https://github.com/huggingface/transformers/actions/runs/7884708184/attempts/1
* https://github.com/huggingface/transformers/actions/runs/7884708184/attempts/2
* https://github.com/huggingface/transformers/actions/runs/7884708184

I haven't been able to identify why these tests have started failing. Obvious dependencies haven't been affected: 
* `backbones.md` was touched two weeks ago
* Latest update to doc-builder was two weeks ago
* `backbone_utils` was updated three weeks ago

This PR moves the offending backbone classes to be importable from `utils`. This was to match all other `~utils` references in the doc, which have the form `~utils.module.object`. 

Doing this appears to have resolved the issue: 
* Two runs on the same commit on github CI have passed - [run 1](https://github.com/huggingface/transformers/actions/runs/7889497104/attempts/1), [run 2](https://github.com/huggingface/transformers/actions/runs/7889497104)
* Another run on a different, empty commit passed - [here](https://github.com/huggingface/transformers/actions/runs/7890483169/job/21532741409?pr=29002)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29002). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ydshieh 

I'm merging as this is quite urgent for resolving failing tests on `main`