noamaanMulla-03/transformers-issues huggingface dataset:
Fix: avoid re-instantiating past_key_values(Cache Class)
Author: silencelamb
Pull request: yes

# What does this PR do?

Fix a bug when set `use_cache=True` and pass a `DynamicCache` class as the `past_key_values` 
## Bug Description

There is an issue when the `use_cache` option is enabled and `DynamicCache()` is passed as the `past_key_values` parameter. Example code is shown below. Specifically, the `past_key_values` are being utilized to reinstantiate a new `DynamicCache()` instance. This behavior is unexpected and results in incorrect outcomes.

```python
kv_cache = DynamicCache()
gen_length  = 0
max_gen_len = 30
while gen_length < max_gen_len:
    output = model(input_ids=prompt, past_key_values=kv_cache, use_cache=True)
    pred_token_idx = output.logits[:, -1, :].argmax(dim=-1)
    prompt = pred_token_idx
    gen_length +=1
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@gante 
@ArthurZucker 
@younesbelkada 

Comments:
cc @gante 

Hi  @gante, I appreciate the work you're doing and I am excited to contribute to its improvement. I understand you're likely very busy, but I was hoping you might have had a chance to review this. Your feedback would be invaluable, and I'm eager to make any necessary adjustments to help move things forward.



Hi @silencelamb 👋 Thank you for opening the PR!

My apologies for the slow reply, I catch up with my notification queue from oldest to newest, so your newest comment actually moved the PR down the queue 😅 I believe the issue is no longer present on `main`, as the current pattern is

```py
        return_legacy_cache = False
        if use_cache and not isinstance(past_key_values, Cache):  # kept for BC (non `Cache` `past_key_values` inputs)
            return_legacy_cache = True
            past_key_values = DynamicCache.from_legacy_cache(past_key_values)
```

In any case, thank you for being proactive and opening a PR 💪  I'm looking forward to your next contributions!

Thank you so much for taking the time to review and respond. Since the issue has been resolved, we can go ahead and close this pull request  💪 .
run_clip.py - use a processor instead of separate objects
Author: amyeroberts
Pull request: yes

# What does this PR do?

Uses a CLIPProcessor instead of saving separate tokenizers and image processors. This ensures the created objects after training can properly be loaded in as other CLIP objects in the library i.e. all of the following should work:

```py
model = CLIPModel.from_pretrained(saved_checkpoint)
tokenizer = CLIPTokenizer.from_pretrained(saved_checkpoint)
image_processor = CLIPImageProcessor.from_pretrained(saved_checkpoint)
processor = CLIPProcessor.from_pretrained(saved_checkpoint)
```


Fixes #30714


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30720). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this: the README.md in for run_clip.py specifies creating a model using the vision encoder-decoder model i.e. that any language and vision models can be composed together. Therefore, any tokenizer and image processor could be used. They can't then be bundled together in a CLIPProcessor
Token merging for LLM Inference
Author: samchaineau
Pull request: no
Labels: name: Feature request

### Feature request

I propose to create an Optimized Inference class for LLM where the input sequence is merged using SLERP to process smaller sequences. I did experiments on Mistral 7B Instruct and it turns out that you still have decent results while processing up to 2x smaller sequences.
The code is here https://github.com/samchaineau/llm_slerp_generation and I made it usable in a text-generation pipeline

![demo](https://github.com/huggingface/transformers/assets/47329252/1b10d269-c61d-4c7d-a499-dfbcf31b27bf)


### Motivation

While we are improving a lot on the training side, LLMs still generate in a brute force manner. From the experiments, processing the full sequence does not necessarily help to better predict the last token. Hence, we could speed up significantly the inference of the LLM by computing on a smaller sequence inside the attention modules. 

### Your contribution

I did experiments, proposed an OptimizedMistral class and a notebook to try it. I tested the merged inference model on alpaca eval and the results are encouraging as it still outperforms gemma 7B it for instance. I could help to build the wrapper class and see how to insure a dual usage : one for training and one for generation.

Comments:
cc @gante 

Hi @samchaineau -- thank you for opening this issue 🤗 

The technique seems interesting, and IMO it makes sense to me. 

I see two ways forward:
1. We support your work as an external library, where you're free to implement your wrapper at your own pace without going through our processes (CI, reviews, API requirements, docs, ...). I'm more than happy to amplify the results of your work as you obtain them, we want you to be successful 🔥 
2. We add your technique to `transformers`. However, we need much more data before we greenlight it -- adding features to `transformers` means we'll have to maintain them, so we need to be sure the technique is 100% worth it. On my end, to start the discussion in this direction, I would need some reproducible benchmarks showing that its benefits (e.g. increase in torkens/second, decrease in GPU memory consumption, better perplexity in long sequences) outweigh its drawbacks (worse perplexity), for multiple model architectures. 

Let us know how you'd like to proceed 🤗 (Note: the benchmarks requested for 2. would also be very useful for the community in case you decide to go with 1.)

Hi @gante ! Hi @amyeroberts ! 

To make sure I understood correctly the two possible ways

Option 1 : I develop the wrapper on my end as a dedicated library supported by HuggingFace (like PEFT for instance or am I getting wrong on this ?)

Option 2: We go deeper in the work and proposed a pull request on the transformers library (in the 'generate' part for instance)

Sorry if I don't fully understand the process 

Happy that it makes you enthusiast ! 

We can further discuss by email if needed 

@samchaineau correct!

To be added to `transformers`, we need empirical data about the technique's benefits :)
model_max_length arg has no effect when creating bert tokenizer
Author: galtay
Pull request: no
Labels: name: Should Fix

### System Info

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.37.2
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
new_tokenizer = AutoTokenizer.from_pretrained('google-bert/bert-base-uncased', model_max_length=8192)
print(new_tokenizer.model_max_length)
# 8192
old_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', model_max_length=8192)
print(old_tokenizer.model_max_length)
# 512
```

### Expected behavior

```python
print(old_tokenizer.model_max_length)
# 8192
```

Comments:
Hi @galtay, thanks for raising this issue! 

It looks related to #29050  

cc @LysandreJik 

```
In [7]: transformers.__version__
Out[7]: '4.39.0.dev0'

In [3]: nt = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased", model_max_length=8192)
In [4]: ot = AutoTokenizer.from_pretrained('bert-base-uncased', model_max_length=8192)

In [5]: nt.model_max_length
Out[5]: 512

In [6]: ot.model_max_length
Out[6]: 8192
```

Gentle ping @LysandreJik @ArthurZucker 

This is now fixed on `main`! It took a bit of time to go through the deprecation cycle, but it's live.

Thanks for the report @galtay!
KV cache is no longer a model attribute
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Addresses the [comment](https://github.com/huggingface/transformers/pull/30688#discussion_r1594448926) and removes lines where we got past kv cache from model attributes. Static cache is no longer a model attribute starting from #30476 

Slow tests on Llama passing locally for me.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30730). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: consistently handle special tokens as tensors
Author: gante
Pull request: yes

# What does this PR do?

(reopened from #29788, requirements were merged and made this PR simpler)

To enable `torch.compile` with `generate`, some special token-related operations have to be rewritten into torch operations. That requires special tokens to be tensors instead of integers or a list of integers. (See https://github.com/huggingface/transformers/pull/29374 for a working prototype)

This PR reworks special token usage in `generate` to consistently treat them as a tensor, as opposed to e.g. keeping track of `eos_token_id` in integer and in tensor form.

👉 Review suggestion: start by reading `_prepare_special_tokens` and how it fits in `generate`.

______________________________________________________________________
Tests ran locally:
- [x] logits processors doctests (`pytest --doctest-modules src/transformers/generation/logits_process.py -vv`)
- [x] generate doctests (`pytest --doctest-modules src/transformers/generation/utils.py -vv`)
- [x] generate integration tests (`RUN_SLOW=1 py.test tests/generation/ -vv`)
- [x] cache integration tests (`RUN_SLOW=1 py.test tests/test_cache_utils.py -vv`)
- [x] llama slow tests (`RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv`)
- [x] whisper slow tests (`RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv`) -- same failures as in `main`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30624). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @zucchini-nlp this one is the same as #29788, which you've already reviewed in its early state. Note that because we've merged other PRs first (e.g. removing the decoding functions from the public API), the diff is much smaller 💛 

@ArthurZucker addressed your comments :D 

(let's see this fast CI going brr)
Interpolate pos encode vitmae
Author: bhuvanmdev
Pull request: yes

# What does this PR do?
Adds interpolation capabilities for the tf and pytorch versions of vit_mae along with the test cases.

Addresses #30579 (vit_mae)


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested.

@amyeroberts


Comments:
I'm really Sorry, I got confused with one of my older branch and ended up polluting the PR. Will clean up the mess. Sorry for increasing your workload :(
Update CodeLlama references
Author: osanseviero
Pull request: yes

Update CodeLlama references

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30218). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I want to wait until https://github.com/huggingface/moon-landing/issues/9591 is ready before fixing/moving ahead with this
fix: link to HF repo/tree/revision when a file is missing
Author: mapmeld
Pull request: yes

When `cached_file` fails to download a file from a valid HF model, it reports `Checkout 'https://huggingface.co/{path_or_repo_id}/{revision}' for available files.` But [this gets a 404](https://huggingface.co/monsoon-nlp/tinyllama-mixpretrain-uniprottune/main). The correct link would include `/tree/{revision}`.

- It works for both branch names (e.g. `main`) and [commit IDs](https://huggingface.co/monsoon-nlp/hindi-bert/tree/c54eb831ba59dccff82d0bc7add9fd182606a9c7)
- Another error message already uses this URL https://github.com/huggingface/transformers/blob/main/src/transformers/utils/hub.py#L371
( edit: #29369 ) 

I had the error come up when trying to load a tokenizer from a LoRA repo instead of the base model repo: https://colab.research.google.com/drive/1pqdFMgo_6kqHfCGR5-qCIC6A_py5f84H?usp=sharing

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
Accept token in trainer.push_to_hub()
Author: mapmeld
Pull request: yes

# What does this PR do?

The trainer's HF token is set to `self.args.hub_token` when it is initialized.
I use Transformers and `trainer.train()` in CoLab notebooks.  If I start running with no token, a read-only token, or some other issue with Secrets/env vars, I can't correct the mistake later when calling `trainer.push_to_hub()`

With this change, I could try again with `trainer.push_to_hub(token="****")`

Alternatives: if overwriting `self.args.hub_token` is undesirable here, it could be passed to `create_repo` and `update_folder` in this file.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
Merging as the failing tests are unrelated and have been resolved on main
Added attention layers to wrap for fsdp
Author: alvations
Pull request: yes

# What does this PR do?

This PR defines the `fsdp_transformer_layer_cls_to_wrap` value in the Mistral config. This way user can easily load the config to figure out what values to use for FSDP, e.g. 

```
from transformers import AutoConfig
c = AutoConfig.from_pretrained("mistralai/Mistral-7B-v0.1")
c.fsdp_transformer_layer_cls_to_wrap
```

[out]:

```
MistralDecoderLayer
```

**Context:** Users has been asking when and which layer to wrap, there shouldn't be a need to load the model to figure it out by going through the state_dict of model summary,  

**Fixes:** https://discuss.huggingface.co/t/accelerate-fsdp-config-prompts/21262/3


## Who can review?

Models:

- text models: @ArthurZucker and @younesbelkada

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100

Documentation: @stevhliu and @MKhalusova




Comments:
What is the default multi-GPU training type?
Author: RonanKMcGovern
Pull request: no

### System Info

NA

### Who can help?

@ArthurZucker , @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When running training using the transformers trainer, and setting device_map to auto, what is the default distributed training type that is used when the model is too large to fit on one GPU?

(assume that I have not yet run `accelerate config`).

Does the model just run with naive model parallel with layers split between different GPUs and with DP (not DDP) on the data side? Are the full gradients and also the optimizer state copied onto each GPU?

It would be helpful if this could be described in the Trainer section of the docs and also in the Multi-GPU docs.

### Expected behavior

NA

Comments:
Hi @RonanKMcGovern 
Thanks for the issue !
If you did not run `accelerate config` it'll fall back to the default behaviour of `accelerator.prepare()`, which I think will return an unmodified model so it'll go for naive PP. Make sure to run the script with `python xxx.py` if that's what you intend for. 
In that case, each GPU will compute its own gradient and have the corresponding optimizer states.
Note the safest is to specify your intended behaviour through `accelerate config` and run your script with `accelerate run xxx.py`
cc @muellerzr @pacman100  in case I am missing something

Thanks @younesbelkada . To be clear, when you say naive PP (pipeline parallel), that means that the batch of data will pass through a first set of layers in a first GPU and then the rest of the layers in the next GPU (if there are just two GPUs).

So, this won't be DP (data parallel) then because there is no parallelisation of data.

yes this is correct and what i meant by naive PP and you are right in that case it wont be DP

Thanks. I'm not sure where in the docs exactly to put it, but possibly where multi-gpu and 'auto' are mentioned, it would make sense to explain that the default is PP.

And, btw, in the SFT docs, there's a statement that the default is DP (although I'm unsure now if that's correct and different to just the Trainer in transformers):
```
Multi-GPU Training

Trainer (and thus SFTTrainer) supports multi-GPU training. If you run your script with python script.py it will default to using DP as the strategy, which may be [slower than expected](https://github.com/huggingface/trl/issues/1303).
```

See [here](https://huggingface.co/docs/trl/main/en/sft_trainer#multi-gpu-training).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@younesbelkada closing this off
remove irrelevant japanese sentence from generation_strategies.md
Author: kumapo
Pull request: yes

# What does this PR do?

Remove the irrelevant Japanese sentence from generation_strategies.md. 
The sentence "希望されるテキストの翻訳がお手伝いできて嬉しいです！.." translates to "I'm happy to help you with the translation! ..". 
This seems like a typical phrase from ChatGPT :)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
PEFT / Trainer: Make use of `model.active_adapters()` instead of deprecated `model.active_adapter` whenever possible
Author: younesbelkada
Pull request: yes

As per title and as mentioned offline by @BenjaminBossan, `model.active_adapter` will be deprecated soon, let's make sure to slowly force Trainer to fallback into `model.active_adapters` instead

cc @BenjaminBossan @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30738). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.