noamaanMulla-03/transformers-issues huggingface dataset:
Cache: standardize cache interface
Author: gante
Pull request: yes

# What does this PR do?

# What does this PR do?

In #27931, where the static cache was introduced, we noticed it had the following hard requirements:
1. The model instance holds the cache, as opposed to being a tensor passed around;
2. Each layer has its own cache, as opposed to a single cache for all layers.

This contrasts with previous implementations (e.g. `DynamicCache`). Given the hard requirements of the static cache, and the resulting benefits, this PR migrates the interface for all cache classes so as to match the static cache. As a result, the modeling code becomes slightly simpler 🤗  

Comments:
Closing this PR and other cache PRs, as we want to move in the opposite direction (static cache behaving like the other caches)

> Closing this PR and other cache PRs, as we want to move in the opposite direction (static cache behaving like the other caches)

is there any related PR for this design?

PR that standardized the interface: https://github.com/huggingface/transformers/pull/30476
(cc @liangan1 )
Update video-llava docs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR updates docs in Video-LLaVa by adding more examples and fixes the failing doctest


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30935). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix low cpu mem usage tests
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes the common tests about `low_cpu_mem_usage` (covered in accelerate_tests). The issue that we had was that we moved one of the model to cuda while the other was on cpu. Hence, we get an error while trying to compare those two models. After fixing the issue on the tests, the only failing model is UdopModel that I fixed (we need to tie the weights in `_tie_weights`. Otherwise, it won't work with `low_cpu_mem_usage`) 

See the run [here](https://github.com/huggingface/transformers/actions/runs/9116151177) with most of the tests passing

Furthermore, I'm running accelerate tests on every model to see if we still have failing tests + check if there are any failing tests for model parallelism, cpu_offload, disk_offload.
The other failing tests are fixed [here](https://github.com/huggingface/transformers/pull/30836). @ydshieh for viz 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30808). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Would be nice if an explanation about what happens. Like, if it's a change in accelerate caused this test starting to fail, and if this is the case, a link to a commit in accelerate could be provided.

In this PR, I fixed the common tests. The issue was that the tests were not designed correctly ( device placement issue ). As for the failing test on UdopModel, this is an issue on the modeling code. With my fix, we can load UdopModel with low_cpu_mem_usage=True. So it's more an issue on transformers.
[examples] update whisper fine-tuning
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #28814 and [distil-medium.en #14](https://huggingface.co/distil-whisper/distil-medium.en/discussions/14#6604a6e94b0ad9bc40a72d33) by setting the language and task arguments to the generation config during fine-tuning.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29938). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Request to add "keys_to_ignore_at_inference" in the configuration_gpt_neox.py to resolve the failure in downstream repo
Author: IT-Forrest
Pull request: no

Compared with the configuration file of the LLama model, https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/configuration_llama.py#L115 
![image](https://github.com/huggingface/transformers/assets/9160073/944fd098-afac-401d-87f7-72f2339deb23)

the `keys_to_ignore_at_inference = ["past_key_values"]` is missed in the configuration file of the gpt-neox, 
https://github.com/huggingface/transformers/blob/0ad770c3733f9478a8d9d0bc18cc6143877b47a2/src/transformers/models/gpt_neox/configuration_gpt_neox.py#L107

Due to missing the attribute `keys_to_ignore_at_inference `, the **logits and logits dtypes mismatch** will be hit in the downstream repo, e.g. optimum-habana. The logits dtypes mismatch behaviors similarly in the issue https://github.com/huggingface/optimum-habana/pull/457 

Question: could we add "keys_to_ignore_at_inference" in the configuration_gpt_neox.py just as that in configuration_llama.py?

Comments:
do you want to open a PR for a fix? 🤗 

Hi @ArthurZucker, yes, hope there will be a PR for this "keys_to_ignore_at_inference" issue.
Since we don't have the interface person with transformers, how about you file the PR for us?
Or if you want me to file the PR, please kindly guide me the routine. Thanks a lot

Hello, sorry to disturb you. Could you explain why `keys_to_ignore_at_inference` needs to add `past_key_values`? `past_key_value` is used to cache kv in auto-regressive inference, if we ignore it, how to do kv-cache?
🚨 [Idefics2] Update ignore index
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR proposes to set the ignore index of the loss function of Idefics2 like the value of any other LLM/multimodal in the Transformers library, which is -100 (the default ignore index of PyTorch's cross-entropy loss).

It's up to users to create the labels and set whatever they don't want to be included in the loss calculation to -100. In the case of Idefics2, this could look like so:

```
labels = batch["input_ids"].clone()
labels[labels == processor.tokenizer.pad_token_id] = -100
labels[labels == processor.image_processor.image_token_id] = -100
```
which means, it's up to the user to make sure padding and image tokens are ignored.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30898). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, btw @VictorSanh could you update the DocVQA notebook to account for this change?

> Thanks, btw @VictorSanh could you update the DocVQA notebook to account for this change?

good point! done!
Paligemma - fix slow tests, add bf16 and f16 slow tests
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

As per PR title.
cc @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

OOO but the models are gated @ydshieh , the CI bot needs access for testing, not sure how to do it! (the run-slow works apart from that :100: )

@molbap for that you can use `require_read_token` context manager: https://github.com/huggingface/transformers/blob/977ce58a78fc46f89151944da745188d2f21332e/tests/models/llama/test_modeling_llama.py#L475 :D !

cc @ydshieh should be good to go!

Apparently it comes from Llava code - I have no idea what this comment means! maybe @younesbelkada ?

In this case, we can merge, and once @younesbelkada has answered, we can update the comments in another PR (if needed)

If that comment is that confusing, let's just remove it in another PR (both here and in LLava testing file). WDTY?
[whisper] only trigger forced ids warning once
Author: sanchit-gandhi
Pull request: yes
Labels: name: Good First Issue

name: Audio

# What does this PR do?

Most warnings in `generation_whisper.py` that are related to `config`/`generation_config` parameters are triggered once, e.g.:
https://github.com/huggingface/transformers/blob/1518508467d96b3866fc4ebcb7a5b3a2e0df2aa4/src/transformers/models/whisper/generation_whisper.py#L1128-L1129

However, the warnings related to `forced_decoder_ids` are triggered at every call to `.generate`. This peppers the stdout and is quite irritating for users to see again and again:
```
You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, 50259], [2, 50359], [3, 50363]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.
You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, 50259], [2, 50359], [3, 50363]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.
You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, 50259], [2, 50359], [3, 50363]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.
etc.
```
This PR unifies the logger warnings, to only trigger the `forced_decoder_ids` warnings once!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30966). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🚨 out_indices always a list
Author: amyeroberts
Pull request: yes

# What does this PR do?

Recent updates to timm changed the type of the attribute `model.feature_info.out_indices`. Previously, `out_indices` would reflect the input type of `out_indices` on the `create_model` call i.e. either `tuple` or `list`. Now, this value is always a tuple. This causes the following failure on main:

```
FAILED tests/models/timm_backbone/test_modeling_timm_backbone.py::TimmBackboneModelTest::test_timm_transformer_backbone_equivalence - AssertionError: (1, 2, 3) != [1, 2, 3]
```
Example CI run: https://app.circleci.com/pipelines/github/huggingface/transformers/93542/workflows/38bfe697-908c-4d2c-a05c-7b99090fb084/jobs/1226834

As list are more useful and consistent for us -- we cannot save tuples in configs, they must be converted to lists first -- we instead choose to cast `out_indices` to always be a list. 

This has the possibility of being a slight breaking change if users are creating models and relying on `out_indices` on being a tuple. As this property only happens when a new model is created, and not if it's saved and reloaded (because of the config), then I think this has a low chance of having much of an impact. 

The following tests were run to make sure everything is still OK: 

```
pytest tests/utils/test_backbone_utils.py
pytest tests/models/timm_backbone/test_modeling_timm_backbone.py
pytest tests/models/swin/test_modeling_swin.py::SwinBackboneTest
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30941). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> If we need them to be mutable, cool to cast to list!

@ArthurZucker It's not that we need them to be mutable (in fact, we probably don't), it's to do with consistent typing when saving / loading configs. Previously, we just followed the out_indices from timm, which could be tuples or lists. However, for our models, because they're loaded through the configs, if a model is saved out with out_indices which are a tuple, it'll be cast to a list because of writing as a json. Previously, timm would respect whichever type was used to create the model, but now only uses tuples. This PR enforces a casting to list to make sure: 
* We're consistent across backbones: timm vs. transformers
* timm models have `out_indices` consistently set to lists across timm versions
Finally fix the missing new model failure CI report
Author: ydshieh
Pull request: yes

# What does this PR do?

#30313 and #30695 tried to fix the missing new model failure CI report, but we didn't get it after those 2 PRs. A final piece is fixed in this PR. Sorry ...

I made sure it is back by triggering a run.

See https://huggingface.slack.com/archives/C06LR9PQA00/p1716383854342599

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30968). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generation: get special tokens from model config
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixed #30892 . The error, described in the linked issue, was caused by passing a `GenerationConfig` type instead of a kwarg. If a user passes `GenerationConfig` we do not update it with model's generation config, thus cannot get special tokens used by the model. Two options were:
1. Update the user's `GenerationConfig` by that of models in the `_prepare_generation_config`
2. Try to get special tokens from the config, if not then from self.config 

I think the first option can mess-up with what the user wants when generating, for ex by adding a `do_sample=True` from model's generation config. So I went for option-2

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Are you sure this is the only argument that we need to get from the model's generation config for BC?

@ArthurZucker see my comment above :D 

@ArthurZucker @gante done, I brought back the `_get_decoder_start_token` from prev version and checked it works when user passes decoder_start_tokens, or when user does not pass anything.
transformers 4.41.0 breaks generate() for T5
Author: abdulfatir
Pull request: no
Labels: name: Should Fix

name: Generation

### System Info

- `transformers` version: 4.41.0
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

 @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following code breaks in `v4.41.0` (it works on earlier versions).

```py
import torch
from transformers import GenerationConfig
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained(
    "google/t5-efficient-tiny", device_map="cuda"
)
input_ids = torch.tensor([[4, 5, 6, 6, 7]], device="cuda")
model.generate(
    input_ids=input_ids,
    generation_config=GenerationConfig(do_sample=True),
)
```

Error: 
```
ValueError: `decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.
```

### Expected behavior

Expected generate to work like before without manually specifying `decoder_start_token_id` or `bos_token_id` in the `GenerationConfig`.

Comments:
Thanks @abdulfatir ! 
https://github.com/huggingface/transformers/pull/30899 from @zucchini-nlp should fix the issue ! :)
legacy to init the slow tokenizer when converting from slow was wrong
Author: ArthurZucker
Pull request: yes

# What does this PR do?
A small nit but the arg was not passed. cc @itazap one fix already 😉 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30972). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Whisper] Strip prompt before finding common subsequence
Author: sanchit-gandhi
Pull request: yes
Labels: name: Good Second Issue

name: Audio

# What does this PR do?

Fixes #27594, #27813. We should strip any Whisper prompt from the token ids before finding the longest common sub-sequence in our chunking algorithm. They should not be used to find the sub-sequence, only the generated token ids. Otherwise, we try to match the generated token ids in chunk N (on the right) with the prompted ids in chunk N+1 (on the left), which is inevitably going to give a mis-match.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27836). All of your documentation changes will be reflected on that endpoint.

Hi, I tested this PR, and I'm noticing an issue when a pipeline instance is re-run without re-instantiating the pipeline -- in my case, I load the pipeline, and then I iterate over audio samples in a loop.

The first loop works, but afterwards, subsequent runs of the pipeline include the previous prompt_ids, and when the prompt is changed, the previous ids are included.

I've also noticed that the subsequent runs of the pipeline produce less and less coherent responses.

EDIT: seems related to [this issue](https://github.com/huggingface/transformers/issues/27629), which was fixed by [this pr](https://github.com/huggingface/transformers/pull/27833#issue-2024117768)

I did some testing with this PR rebased onto main, and the issues with the prompts seem to be cleared up! 

In testing, I noticed some inconsistencies between the openai implementation and transformers, and I wanted to share these [results](https://colab.research.google.com/drive/1sGHYRBxOVvUjpgnB61fvD_K3ZdlCAg43?usp=sharing).  In this example I intentionally cut the audio about ~2 seconds before it actually ends to provoke hallucinations. The transformers implementation is more inconsistent between different prompts, sometimes omitting significant portions of the audio. This example uses large-v3, but I tested with tiny.en and medium.en, which produced lower quality results.

Do you know if this pull request is waiting on anything? 

Hi, Is it possible to merge this PR? I'm facing the same issue with prompts.

cc @sanchit-gandhi 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This is still waiting, right? 

Gently pinging @sanchit-gandhi might still be worth adding !

Another ping @sanchit-gandhi! 

This PR is all ready to go, sorry to keep you waiting @mmichelli @chemeris @benniekiss! Since the branch was quite out of date, I've rebased and forced pushed. Once the CI is green, I'll merge!

Thank you, this is great news.
[DOCS] - Wrong link to model#generative-models in Pipeline docs
Author: BrunoGomesCoelho
Pull request: no
Labels: name: Documentation

name: Good First Issue

### System Info

Latest docs on https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline



### Who can help?

@stevhliu and @MKhalusova

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Inside the documentation for [pipelines#transformers.TextGenerationPipeline.__call__](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline.__call__) the docs read:

> generate_kwargs — Additional keyword arguments to pass along to the generate method of the model (see the generate method corresponding to your framework [here](https://huggingface.co/docs/transformers/en/main_classes/model#generative-models)).

The header part of the link (#generative-models) does not exist in the [main model page](https://huggingface.co/docs/transformers/en/main_classes/model) though - I'm not sure any of the headers there are appropriate actually. 

The problem is repeated for two other tasks at least: Conversational, text2textgeneration.

For the text-generation case, the corresponding docs link is [pipelines/text_generation.py#L198](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_generation.py#L198)



### Expected behavior

The link sends me to documentations about common model-generation parameters - Maybe [this](https://huggingface.co/docs/transformers/generation_strategies)?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @BrunoGomesCoelho, thanks for raising this issue! 

Would you like to open a PR to fix these links? This way you get the github contribution

I think @coolyashas had attempted this in #28231 - It seems ready to go other than minor comments - Do you want to re-open and submit @coolyashas? 

> I think @coolyashas had attempted this in #28231 - It seems ready to go other than minor comments - Do you want to re-open and submit @coolyashas?

Yes, I would like to re-open and submit it! @amyeroberts 

Any update @coolyashas? 

@amyeroberts,  I have added a quick [PR](https://github.com/huggingface/transformers/pull/30562) for this issue. Kindly review it and share your feedback. Appreciate it!


@amyeroberts - I filed up [this PR](https://github.com/huggingface/transformers/pull/30948) to resolve it. Hopefully the third time is the lucky charm.
Fix link in Pipeline documentation
Author: junhl
Pull request: yes

# What does this PR do?

Fix link in few following Pipeline documentations, as originally reported by https://github.com/huggingface/transformers/issues/28224
- https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.Conversation
- https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline
- https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.Text2TextGenerationPipeline

Fixes https://github.com/huggingface/transformers/issues/28224
Consulted comments in previous PR attempts in https://github.com/huggingface/transformers/pull/28231 and https://github.com/huggingface/transformers/pull/30562


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova


Comments:
~~Currently checks seem to be failing from original `main`, not from changes in this PR. Will rebase once `main` is fixed.~~

Review can be done @stevhliu @gante 

EDIT: checks are now all green

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30948). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.