noamaanMulla-03/transformers-issues huggingface dataset:
fix get past_length
Author: jiqing-feng
Pull request: yes

Hi @gante . Multi-query models like gpt_bigcode have a different kv-cache layout; use .shape[-2] can handle it.

Comments:
Hi @gante . Have you been able to see it yet? I just found that this function is only used at the beginning of the generation, which means no pkv when calling it, so deleting it or just setting pkv=0 is also a choice. Would like to hear your opinion. Thx!

> @jiqing-feng hey! Multi-query gpt-bigcode is indeed a special case and it has `_get_initial_cache_position` overriden [here](https://github.com/huggingface/transformers/blob/9ef93fccade1f2b6349c2eec2188d0ea54329f84/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py#L1215-L1231).
> 
> We need this part of code in case some users pass in past key values from prev generations to continue generating. The `shape[-2]` gets past seq length to get correct cache-position for all models, and special cases like bigcode (afaik bigcode is the only one) are overriden in modeling. Also, "cache_position" currently is not used in relatively older models, which includes GPTBigCode, so there should be no failures 😄

Do you mean `shape[2]` can handle all models? If so, I could close this PR, WDYT?

@jiqing-feng yes, all cases should be handled by `shape[2]` and the PR can be closed
LlamaRMSNorm() Dtype Casting Error
Author: Ritz111
Pull request: no

### System Info

transformers==4.37.2

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

@ArthurZucker @younesbelkada 
Hi~ I found a bug in the **LlamaRMSNorm(nn.Module)** (lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py)


```
class LlamaRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        LlamaRMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        input_dtype = hidden_states.dtype
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
        return self.weight * hidden_states.to(input_dtype)
```


On the last line, if the **input_dtype** is bfloat16, the return tensor will still be float32 because the **self.weight** has been initialized as float32. Thus the last line should be modified to:

`
        return (self.weight * hidden_states).to(input_dtype)
`

### Expected behavior

see above and looking forward to your reply~ Thank you 

Comments:
Hi @Ritz111 
Thanks ! I think this is not a bug, see: https://github.com/huggingface/transformers/pull/23535 for more details

why should class LlamaRMSNorm do ”hidden_states = hidden_states.to(torch.float32)“ ，why not flow the type promotion rules of PyToch ops

self.weight is bf16，hidden_states is fp32
I found that the dtype of these two methods are different. 
method 1:
return  (self.weight * hidden_states).to(input_dtype)   # (bf16  *  fp32).to(input_dtype)
method 2:
return  self.weight * hidden_states.to(input_dtype)      # bf16  *  bf16


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support Clip QKV for MPT
Author: akakakakakaa
Pull request: yes

# What does this PR do?
- Support Clip QKV for MPT

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker and @younesbelkada


Comments:
@younesbelkada 
Is there any issue with merging this PR? Looking at the failed tests, it doesn't seem to be a problem with the PR code.

Yes this looks good for me, just waiting for final core maintainer review
cc @amyeroberts @ArthurZucker
Fix MusicGen SDPA
Author: ylacombe
Pull request: yes

Fix #31189 and #30020.

SDPA produces Nan when given a padding mask that attends to no position at all (see https://github.com/pytorch/pytorch/issues/103749#issuecomment-1597069280).
In the case of Musicgen, it can happen for two reasons:

1. when we use get_unconditional_inputs ([here](https://github.com/huggingface/transformers/blob/d475f76745c0c90bb3ffc85c0ee7511705a119ba/src/transformers/models/musicgen/modeling_musicgen.py#L2946))
2. when guidance_scale>1 (see [here](https://github.com/huggingface/transformers/blob/d475f76745c0c90bb3ffc85c0ee7511705a119ba/src/transformers/models/musicgen/modeling_musicgen.py#L2457-L2459))


There might be more elegant way to do this, WDYT @amyeroberts ?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31208). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MusicGen fails when being used with pipeline
Author: osanseviero
Pull request: no
Labels: name: Core: Pipeline

name: Audio

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.3 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sanchit-gandhi @ylacombe 



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This code gives an error

```
from transformers import pipeline

pipe = pipeline("text-to-audio", model="facebook/musicgen-small", device="cuda")
data = pipe("electric rock solo, very intense")
```

```
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```

Colab with different versions: https://colab.research.google.com/drive/1qYDk9_rbs7lC2EfqL4q-J9weMFyO7F7X#scrollTo=fzZBH7O4xjSQ

### Expected behavior

It should work just as it did in older versions (e.g. 4.39.2 works)

Seems related to #30020 

Comments:
Thanks for opening the issue @osanseviero, I've been digging this up a bit and I believe I finally got the reason why it and  #30020 happened.

The reason is that SDPA produces Nan when given a padding mask that attends to no position at all (see [this thread](https://github.com/pytorch/pytorch/issues/103749#issuecomment-1597069280)).
In the case of Musicgen, it can happen for two reasons:
1. when we use `get_unconditional_inputs`  ([here](https://github.com/huggingface/transformers/blob/d475f76745c0c90bb3ffc85c0ee7511705a119ba/src/transformers/models/musicgen/modeling_musicgen.py#L2946))
2. when `guidance_scale>1` (see [here](https://github.com/huggingface/transformers/blob/d475f76745c0c90bb3ffc85c0ee7511705a119ba/src/transformers/models/musicgen/modeling_musicgen.py#L2457-L2459))

Still have to think how to solve this but we'll probably switch back to eager attention when the padding mask falls into this edge case! (it happens only in the cross-attention layer btw)
Pin datasets<2.20.0 for examples
Author: amyeroberts
Pull request: yes

# What does this PR do?

Temporarily pin datasets upper version until PR merged:

https://github.com/huggingface/transformers/pull/31406

c.f. #31407 




Comments:
cc @albertvillanova @ydshieh 

Merging as this is blocking other PRs

@albertvillanova No worries - I forgot in my review!
SeamlessM4T: `test_retain_grad_hidden_states_attentions` is flaky
Author: gante
Pull request: no
Labels: name: Should Fix

name: bug

name: Audio

See the related PR which added the `is_flaky` decorator: https://github.com/huggingface/transformers/pull/28035

cc @ylacombe, to explore in case you have spare bandwidth :)

Comments:
@ylacombe Re-opening as this isn't resolved by #28060 (it just makes sure our CI doesn't break)

Alright, copying the reasons for flakiness from #28060 for trackability then!

> After investigating the reasons for the test_retain_grad_hidden_states_attentions flaky failure, I realized the speech encoder attentions can be None with a non-zero probability when training=True.


To circle back on this, let's figure out how to correctly fix this!

The feature surely works but the randomness of the skipping makes so that the first attention layer is sometimes skipped.

What I could do is to fix a seed with which the first layer is never skipped, WDYT @amyeroberts ?

Seed setting seems like a good idea!
Fix SpeechT5 `decoder_attention_mask` shape
Author: ylacombe
Pull request: yes

# What does this PR do?

 #26598 rightfully raised a warning when passing labels to `SpeechT5`. When it happens, a reduction factor is applied to the `labels` but not to the `decoder_attention_mask` that comes with it, resulting in a mismatch.


Fixes #26598

cc @amyeroberts @sanchit-gandhi 

Comments:
After running the slow tests, we have these two failing tests:
```
FAILED tests/models/speecht5/test_modeling_speecht5.py::SpeechT5ForTextToSpeechIntegrationTests::test_batch_generation - AssertionError: torch.Size([3, 264, 80]) != (3, 262, 80)
FAILED tests/models/speecht5/test_modeling_speecht5.py::SpeechT5ForTextToSpeechIntegrationTests::test_generation - AssertionError: torch.Size([226, 80]) != (230, 80)
```
This is most likely related to #25943, as the current PR only impacts output when `labels` are passed.

I'll deep dive into it, let's wait before merging


@ylacombe Thanks for checking and reporting back! 

Would be great to finish this one and close https://github.com/huggingface/transformers/issues/26598!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28071). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry for the long delay, I've added additional tests to make sure the forward pass works when passing `labels` and the attention mask !

The failed tests mentioned [here](https://github.com/huggingface/transformers/pull/28071#issuecomment-1858065107) now pass btw!

Could I have a final last check on the added tests ? Thanks @amyeroberts !
Question about LlavaProcessor
Author: lucasjinreal
Pull request: no

The LlavaPocessor provides a direclty input of images and text.
But when I just need text conversation with MLLMs, it throws error. because images if None, it will have error in returned BatchFeature.


Am curious about, why does it designed like this? Why we need couple images and text togather? Besides, since many customized LLava models using differecent ImageTokenID, there is no way to reuse LLavaProcessor. 

Say I have text like this `<image> descirbe this image`, how does LlavaProcessor treat `<image>` token??  

I am so confused by the design.



Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

I just got an solution.
Change potential `inputs_embeds` padding `logger.warning` to `logger.warning_once`
Author: naimenz
Pull request: yes

# What does this PR do?

This PR changes the warning for potential padding in embeddings from `logger.warning` to `logger.warning_once` 

Fixes #30871


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
@naimenz Could you rebase to include the recent changes on `main`? The datasets library had a recent release which is causing the failures on the CI runs for this PR. We pushed a commit #31417 which resolves this, but will need to be included on this branch to set up the right CI env

Oh sure, rebased now! Thanks @amyeroberts
Remove duplicate image processor in auto map
Author: amyeroberts
Pull request: yes

# What does this PR do?

As per title 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31383). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh I didn't! Credit goes to @SangbumChoi
[Docs] Fix bad doc: replace save with logging
Author: chenzizhao
Pull request: yes

# Fix doc typos in TrainingArguments.set_logging

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Remove empty create_and_test_config_common_properties tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

There's a bunch of tests which have `create_and_test_config_common_properties` implemented, but it's just an empty check. These should be removed as they falsely indicate config properties are being run in `test_config`, they clutter the code and this prevents them being copied again for new model implementations 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31359). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you!
Pure Python `PreTrainedTokenizer` is Broken
Author: daskol
Pull request: no

### System Info

transformers v4.40.2
tokenizers v0.19.1

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
>>> from transformers import GPT2Tokenizer
>>> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
...
TypeError: unhashable type: 'AddedToken'
```

### Expected behavior

No exception is raised.

Comments:
Sorry but I cannot reproduce, could you make sure you are running this on 4.40? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Image processors] Remove rescale
Author: NielsRogge
Pull request: yes

# What does this PR do?

Based on #29077 I noticed we can remove a bunch of `rescale` methods, as those are now automatically inherited from the BaseImageProcessor class.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30175). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@NielsRogge @amyeroberts Does this discussion has been resolved? I also notice that it could have slight side effect.

@SangbumChoi I wouldn't let it be a blocker for your RT-DETR PR. If your model is ready to merge before this PR, then this PR can just add RT-DETR as a model to update

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Can `LlamaTokenizerFast` support the argument `add_prefix_space = False`
Author: hnyls2002
Pull request: no

### System Info

With `transformers==4.36.2`

It seems the argument `add_prefix_space` is invalid here. 

### Who can help?

@ArthurZucker 

### Reproduction

```
>>> from transformers import LlamaTokenizerFast
>>> tokenizer = LlamaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer", add_prefix_space = False)
>>> tokenizer.tokenize("hello")
['▁hello']
>>> tokenizer.decode(tokenizer.encode("hello"))
'
 hello'
```

### Expected behavior

Is there a bug, or is it my wrong usage?

Comments:
Hey, #28010 will add support for this! 🤗 

Thanks, I am very excited about this.
Musicgen special tokens in tensors
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR adds the `prepare_special_tokens` method in musicgen's custom generation code, otherwise all generations will fail because special tokens are integers/lists.

Some musicgen generation test were failing, after this PR not anymore

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31420). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Using assistant in AutomaticSpeechRecognitionPipeline with different encoder size
Author: kamilakesbi
Pull request: yes
Labels: name: Good Second Issue

name: Audio

This PR aims at fixing issue #30611: 

- First: an error will be thrown if the assistant and main models encoders don't have the same size, and the assistant is loaded using `AutoModelForCausalLM`. 

- Second: This PR makes the pipeline work when using an assistant with a different encoder size (loaded with   `AutoModelForSpeechSeq2Seq`) than the main model: 

When using AutomaticSpeechRecognitionPipeline, If we use an assistant with a different encoder size than the main model , the pipeline is broken and we get the following error message: 

```
ValueError: Whisper expects the mel input features to be of length 3000, but found 1500. Make sure to pad the input mel features to 3000.
```

## Explanation of the solution

When doing short form generation with the pipeline, `input_features` aren't passed to the generate method, which instead takes the output of the main model's encoder. 

If the main model and the assistant don't share the same encoder, the encoder_output passed to generate cannot be used by the assistant for generation, and we get an error. 

The solution here is to also pass the `input_features` to the generate method to be used by the assistant. 

## Who can review?

@sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30637). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Is there a test that confirms correctness after the fix? There's likely a relevant slow pipeline test that was either failing, or was not rigorous enough

Please have a look at #30726 for an alternative fix -- IMO, the root source of problems is the ASR pipeline doing a redundant operation, and not in generate :) 

I hope you don't mind me crashing into the issue 🙌 (I only noticed this PR after opening #30726, when trying to link all related issues)

I think this PR is ready to be merged! 

cc @amyeroberts @gante if you want to have a look ;) 

Thanks @gante for the review :) 
@amyeroberts could you please merge this PR ? 

@kamilakesbi It still needs a final core maintainer review and approval before merge (+ resolution of conflicts) :). I'll review now 

hi @amyeroberts, I've added a slow test which pass :) I think quality checks fails are unrelated to this PR, and will be fixed by [#30932](https://github.com/huggingface/transformers/pull/30932). 

Quick checklist before merge:
- [ ] Resolve all comment threads that have been addressed
- [ ] Fix the merge conflict in `automatic_speech_recognition.py`
- [ ] Rebase onto main to get the style fixes from #30932 (once the PR is merged)
- [ ] Ping me to get this merged!

Nice work @kamilakesbi!