noamaanMulla-03/transformers-issues huggingface dataset:
Extend Fx supported models with KV cache
Author: xuzifei-dmatrix
Pull request: no
Labels: name: Feature request

### Feature request

I noticed only llama and opt models are supported for FX tracing with KV Cache right now, can I check what is the plan to extend it to more models? Thanks!

### Motivation

I would like run fx traced graphmodules for generate(), which uses KV Cache. Right now it works for OPT and LLama, but I would like try on more models.

### Your contribution

If someone could point me to the general design pattern to make a model FX supported with KV cache or the lines of changes in modeling_opt.py or modeling_llama.py that made them work, I would be happy to submit PRs to make more models work.

Comments:
May be of interest to @michaelbenayoun!
Remove coupled type imports from integration_utils.py
Author: rasmi
Pull request: yes

In #30135, `TFPreTrainedModel` and `PreTrainedModel` were introduced as imports in `integration_utils.py` for the sole purpose of type checking (see diff [here](https://github.com/huggingface/transformers/pull/30135/files#diff-70576373928eee203a92800d5ea1f6270d45462a9bde494cde65338710a0a331)). This results in a fairly heavy set of imports that makes it difficult to separate out builds that are dependent on only TF or Torch (as any module importing `integration_utils` will now import both `modeling_utils.py` and `modeling_tf_utils.py`, which together require both TF and Torch). This logic removes the need for those coupled imports while still working as intended within this module.


Comments:
@amyeroberts your review would be appreciated!
Add SynthID (watermerking by Google DeepMind)
Author: gante
Pull request: yes

# What does this PR do?

Adds SynthID, a watermarking by DeepMind.

https://deepmind.google/technologies/synthid/

Applying watermarking and using a detector is added to `transfomers`. Training a detector is added as a research project.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34350). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

### Feature request

Enable Qwen to ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow


### Motivation

See details in #32253

### Your contribution

TBD

Comments:
Better pipeline defaults
Author: ArthurZucker
Pull request: yes

# What does this PR do?
```python
>>> from transformers import pipeline
>>> pipe = pipeline("text-generation", model="meta-llama/Llama-3.2-1B")
>>> pipe("Hey how are you doing my lord? Let's have fun! More and more tokens should be done. However long the input is")
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.
[{'generated_text': "Hey how are you doing my lord? Let's have fun! More and more tokens should be done. However long the input is."}]
```
this no longer fails saying that max_length is 20. We always generate 20 tokens more than the input. 
Also device should be inferred automatically

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34026). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added performance.mdx italian translation
Author: AhmedIssa11
Pull request: yes

# What does this PR do?

Italian translation of performance doc

Fixes #17459 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
CI: fix failures
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes failures caused by some previously merged PRs from our internal daily CI feedback. Some errors like `eager_matched_sdpa` were flaky, so no need to fix those. 

InstructBLIP was failing because earlier we dispatched in "eager" by default and now we dispatch on "sdpa" for LLM

@ydshieh the new feature for CI feedback is perfect, and super easy to navigate and locate when the tests started failing. Thanks!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34371). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add code generation to natural language processing section
Author: furtnerthomas
Pull request: yes

# What does this PR do?

add "code generation" to list of use cases in natural language processing section


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.




Comments:
Covert chemaleon weights to hf, ImportError
Author: ChinChyi
Pull request: no

### System Info

```shell
ImportError: cannot import name 'ChameleonForCausalLM' from 'transformers'  (/mnt/petrelfs/xxxx/miniconda3/envs/lumina_mgpt/lib/python3.10/site-packages/transformers/__init__.py)
```


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`python src/transformers/models/chameleon/convert_chameleon_weights_to_hf.py --input_dir /mnt/petrelfs/q-pth --model_size 7B --output_dir /mnt/petrelfs`

### Expected behavior

```shell
Hope this is fixed
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
Hey @ChinChyi, you should make sure that transformers is installed using `pip install -e .` 

> Hey @ChinChyi, you should make sure that transformers is installed using `pip install -e .`

@ArthurZucker I've installed transformers with this and I'm still getting this error


Ah you are completely right! The class is `ChameleonForConditionalGeneration` ! Would you like to open a PR to fix the class in the conversion script? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This was fixed: https://github.com/huggingface/transformers/blob/21beb57558f90131439bf03e1da672d117777901/src/transformers/models/chameleon/convert_chameleon_weights_to_hf.py#L375
Fix pil_torch_interpolation_mapping import in image_processing_detr_fast
Author: yonigozlan
Pull request: yes

# What does this PR do?

Fix an unprotected import in image_processing_detr_fast

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Who can review?

@ArthurZucker
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34375). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[PEFT] Add warning for missing key in LoRA adapter
Author: BenjaminBossan
Pull request: yes

# What does this PR do?

When loading a LoRA adapter, so far, there was only a warning when there were unexpected keys in the checkpoint. Now, there is also a warning when there are missing keys.

This change is consistent with
https://github.com/huggingface/peft/pull/2118 in PEFT and the planned PR https://github.com/huggingface/diffusers/pull/9622 in diffusers.

Apart from this change, the error message for unexpected keys was slightly altered for consistency (it should be more readable now). Also, besides adding a test for the missing keys warning, a test for unexpected keys warning was also added, as it was missing so far.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
@ArthurZucker Could you please review or suggest a reviewer?

Ping @ArthurZucker 

@SunMarc can you take a look as well if you have some bandwidth?

Thanks for the review.

> Thanks, I am wondering why we won't init the model with adapter, load with transformers, then just do the matching we usually have in transformers, but that's more for me!

I'm not sure what you mean there, but if there is something I can look into, LMK.

Is there anything missing to merge the PR? If not, as I don't have rights, could you merge please :bow: 

Don't worry it's for me ! 🤗
refactor: remove redundant if-condition and improve type correctness for `convert_tokens_to_ids`
Author: winstxnhdw
Pull request: yes

# What does this PR do?

Removes a redundant if-condition as discussed in #33644. I have kept the argument type change to `Iterable` as the function can correctly accept any `Iterable`, not just `List`.

Comments:
cc @ArthurZucker as you reviewed the other PR

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34030). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Bumping @ArthurZucker 

This seems to break our CI tho not 100% sure
Correct the new defaults
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

This corrects the new defaults set in https://github.com/huggingface/transformers/pull/34026. The previous PR had the effect that if `max_length` is explicitly set by the user in generate, it is automatically overriden to a new value which is extremely confusing and not wanted.  
This adds a check to avoid this scenario.


Comments:
cc @ArthurZucker 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34377). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[auto. ping] Avoid sending empty info + add more team members
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid having things like

```python
        "gpt2": {
            "single-gpu": []
        },
```
(see [for example](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/raw/ccbc621b1791f7713e2d8e977a481844a4a97140/2024-10-24/ci_results_run_models_gpu/new_model_failures_with_bad_commit_grouped_by_authors.json)) 

and add a few team members

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34383). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix glm 
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Fix duplicated variable coming from the recent changes and CI issues + fix duplicated import at the same time


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34388). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Memory usage: new dynamic cache for models supporting sliding window attention
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

This PR introduces `DynamicSlidingWindowCache`, a new kind of `DynamicCache` that will stop growing once its size is equal to the sliding window. This allows models using it to have a (dynamic) fix-sized cache, which is a big win for large inputs. 
The idea is that it becomes the default for models with sliding window attention when no cache arguments are given in `generate`. Let me know what you think @gante @ArthurZucker  

Here is a simple visual representation of the new cache for `mistralai/Mistral-7B-v0.1` (we stop growing after hitting the 4096 sliding window):
![old.pdf](https://github.com/user-attachments/files/17075923/old.pdf)
![new.pdf](https://github.com/user-attachments/files/17075929/new.pdf)

BTW: `SlidingWindowCache` (the static one) is completely broken atm, cannot even be instantiated. Will take a look.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33619). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> BTW: SlidingWindowCache (the static one) is completely broken atm, cannot even be instantiated. Will take a look.

@Cyrilvallez in a recent PR, extra `super().__init__()` lines slipped in. I started working on it on https://github.com/huggingface/transformers/pull/33297, where it is fixed and a test that checks all caches is added. That PR is actually about an often-requested feature, cache reuse. 

I don't think I will be able to continue it next week -- would you like to finish that PR for me? :p (the test needs to be finished, a few cache classes are not yet working according to the test)

@Cyrilvallez suggestion: you can paste images directly to a PR header/comment, which will render the image directly here. It's more convenient for the reader than downloading a file 🤗 

like this: screenshot -> drag image file into this text box
(old)
![Screenshot 2024-09-20 at 16 31 11](https://github.com/user-attachments/assets/8662059f-d0bc-46c5-a9c8-c483be536e7a)

(new)
![Screenshot 2024-09-20 at 16 31 43](https://github.com/user-attachments/assets/852f08e4-96a0-4494-a538-a5e77e476317)


It seems Qwen2 is not happy with these changes :)

> > BTW: SlidingWindowCache (the static one) is completely broken atm, cannot even be instantiated. Will take a look.
> 
> @Cyrilvallez in a recent PR, extra `super().__init__()` lines slipped in. I started working on it on #33297, where it is fixed and a test that checks all caches is added. That PR is actually about an often-requested feature, cache reuse.
> 
> I don't think I will be able to continue it next week -- would you like to finish that PR for me? :p (the test needs to be finished, a few cache classes are not yet working according to the test)

Sure, I'll have a look into it next week 🤗

Slow tests with Mistral were passing, but indeed Qwen2 started to complain, I'm investigating 

ping me once ready for review! 🤗
Fix right padding in LLaVA models
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33976 and fixes https://github.com/huggingface/transformers/issues/33662. As per title, fixes input merging when the text is right padded. 

I didn't add a test because we'll get rid of the old logic for merging in a few minor releases, but until then we should try to fix existing bugs if it's an easy fix.

Tested with 
```python
from transformers import AutoTokenizer, LlavaForConditionalGeneration
import torch

tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-1.5-7b-hf", use_fast=True)

model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf", device_map=0, torch_dtype=torch.float16)
model.eval()



prompt1 = "System prompt<image>Caption 1<pad><pad><pad><pad><pad>"
prompt2 = "System prompt<image>Caption 2<image>Caption 3"           # 12 tokens

print("`System Prompt` tokens: ", tokenizer.encode("System prompt", add_special_tokens=False, truncation=False))
print("`Caption 1` tokens: ", tokenizer.encode("Caption 1", add_special_tokens=False, truncation=False))

# Tokenize the prompt
input_ids = [
	tokenizer.encode(prompt1, return_tensors="pt", add_special_tokens=False, truncation=False).squeeze(0),
	tokenizer.encode(prompt2, return_tensors="pt", add_special_tokens=False, truncation=False).squeeze(0),
]
input_ids = torch.stack(input_ids)
assert isinstance(input_ids, torch.Tensor)

print(f"Input IDs: {input_ids.shape}")
print(f"Input IDs: {input_ids}")


with torch.no_grad():
	input_embeddings = model.get_input_embeddings()(input_ids.to('cuda'))

	embedded_images = torch.randn(3, 5, 4096, device='cuda', dtype=torch.float16)
	attention_mask = torch.ones(2, 12, device='cuda', dtype=torch.bool)
	result = model._merge_input_ids_with_image_features(embedded_images, input_embeddings, input_ids, attention_mask, None)[0]
	print(result.shape)

	print("`System Prompt` diff: ", (result[0, :2] - input_embeddings[0, :2]).abs().max())
	print("`image` diff: ", (result[0, 2:7] - embedded_images[0]).abs().max())
	print("`Caption 1` diff: ", (result[0, 7:11] - input_embeddings[0, 3:7]).abs().max())

	print("`System Prompt` diff: ", (result[1, :2] - input_embeddings[1, :2]).abs().max())
	print("`image` diff: ", (result[1, 2:7] - embedded_images[1]).abs().max())
	print("`Caption 2` diff: ", (result[1, 7:11] - input_embeddings[1, 3:7]).abs().max())
	print("`image` diff: ", (result[1, 11:16] - embedded_images[2]).abs().max())
	print("`Caption 3` diff: ", (result[1, 16:20] - input_embeddings[1, 8:12]).abs().max())
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34305). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `torch.fx` issue related to the new `loss_kwargs` keyword argument
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

As per title. This was introduced in #34283.

cc @ArthurZucker @muellerzr 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34380). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
no filter
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid surprise of missing tests!

See the effect on [this run](https://app.circleci.com/pipelines/github/huggingface/transformers/108673/workflows/668bb1e2-bd66-4308-8f5a-a16d5e688079/jobs/1444870/tests)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34391). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
SynthID: better example
Author: gante
Pull request: yes

# What does this PR do?

Better example for applying SynthID watermarking.

(the original example was missing `return_tensor='pt'`, so I took the chance to improve other aspects)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34372). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.