noamaanMulla-03/transformers-issues huggingface dataset:
Fix Bark saving
Author: ylacombe
Pull request: yes

# What does this PR do?
Fix #32224, slow tests pass locally!

cc @LysandreJik and @IlyasMoutawwakil ! Sorry for the delay!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33266). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`BarkModel` can't be saved anymore
Author: IlyasMoutawwakil
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.2
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModel


bark = AutoModel.from_pretrained("ylacombe/bark-small")
bark.save_pretrained("local_model")
```

fails with

```
/usr/local/lib/python3.10/dist-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
/usr/local/lib/python3.10/dist-packages/transformers/models/encodec/modeling_encodec.py:120: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.register_buffer("padding_total", torch.tensor(kernel_size - stride, dtype=torch.int64), persistent=False)
Traceback (most recent call last):
  File "/home/ilyas/optimum/sanity.py", line 5, in <module>
    bark.save_pretrained("local_model")
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2679, in save_pretrained
    raise RuntimeError(
RuntimeError: The weights trying to be saved contained shared tensors [{'fine_acoustics.input_embeds_layers.1.weight', 'fine_acoustics.lm_heads.0.weight'}, {'fine_acoustics.lm_heads.1.weight', 'fine_acoustics.input_embeds_layers.2.weight'}, {'fine_acoustics.input_embeds_layers.3.weight', 'fine_acoustics.lm_heads.2.weight'}, {'fine_acoustics.lm_heads.3.weight', 'fine_acoustics.input_embeds_layers.4.weight'}, {'fine_acoustics.input_embeds_layers.5.weight', 'fine_acoustics.lm_heads.4.weight'}, {'fine_acoustics.lm_heads.5.weight', 'fine_acoustics.input_embeds_layers.6.weight'}, {'fine_acoustics.lm_heads.6.weight', 'fine_acoustics.input_embeds_layers.7.weight'}] that are mismatching the transformers base configuration. Try saving using `safe_serialization=False` or remove this tensor sharing.
```



### Expected behavior

to save the model locally

Comments:
cc @ylacombe as well

Thanks for opening the issue @IlyasMoutawwakil, I've opened a PR to fix this!
Update chat template docs to remove Blenderbot
Author: Rocketknight1
Pull request: yes

The chat template docs reference Blenderbot, but this model is quite obsolete and doesn't actually have a chat template. The examples worked for a while because a default chat template was set, but these have now been deprecated.

This update rewrites the doc with more modern models.

Fixes #33246

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33254). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ready for review, cc @LysandreJik !
feat: Add data class for fsdp config and use it along with argument parser
Author: kmehant
Pull request: no
Labels: name: Feature request

### Feature request

Like the trainer arguments data class https://github.com/huggingface/transformers/blob/2a002d073a337051bdc3fbdc95ff1bc0399ae2bb/src/transformers/training_args.py#L167

Its good to have a data class for fsdp config as it allows for validation even before stepping into starting a training with ambiguous types used in the config. 

### Motivation

For instance, the fsdp config json file might have been written as

```json
{
...
"xla": "False" // or "false"
...
}
```

This check (https://github.com/huggingface/transformers/blob/745bbfe4bb2b61491dedd56e1e8ee4af8ef1a9ec/src/transformers/training_args.py#L1768) might pass through. This might be harder to debug though the problem is due a simple type bug. Having a fsdp arguments data class with parser would help identify such bugs early.

### Your contribution

I would be glad to raise a PR based on the discussion on this issue.

Comments:
FYI @muellerzr 

@ArthurZucker @muellerzr Any thoughts?

I suppose we can do so. i think it *might* be better to have users use the raw accelerate FSDP plugin here, let me think on this!

A dataclass similar to the `AcceleratorConfig` would be the way to go here. @kmehant would you like to take a stab at it? :) 

Sure @muellerzr !

https://github.com/huggingface/transformers/issues/29476#issuecomment-2304563418

@muellerzr just trying to understand, is there any reason for not simply importing the FSDP plugin data class (like you mentioned here - https://github.com/huggingface/transformers/issues/29476#issuecomment-2304553034) from accelerate and use it here? Creating a AcceleratorConfig similar dataclass would lead to duplication of work across transformers and accelerate libraries are we okay with that? Thanks

I suppose there actually wouldn't be a harm in just importing the FSDP one. My only fear would be how easy it is to pull up the docs/get to the source by adding yet another item that's in the accelerate lib users in transformers need to know about. But if we allow the fsdp config and keep its docs aligned with the accelerate ones (plus the few extra that are specific to trainer like the xla bits) we should be fine.

For *that* specific reason is why I suggested a different dataclass, since they are not 1:1

sure got it @muellerzr will be back with a PR thanks.

@muellerzr Have raised a PR for your review - https://github.com/huggingface/transformers/pull/33278

Have added code comments as well.
Qwen2-VL from_config broken?
Author: mobicham
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-5.4.0-176-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: 0.34.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.0.dev20240902+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```Python
import transformers, torch
config = transformers.AutoConfig.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")
with torch.device('meta'):
    model = transformers.AutoModel.from_config(config)
print(model)
##########################################################
Qwen2VLModel(
  (embed_tokens): Embedding(151936, 1536)
  (layers): ModuleList(
    (0-27): 28 x Qwen2VLDecoderLayer(
      (self_attn): Qwen2VLSdpaAttention(
        (q_proj): Linear(in_features=1536, out_features=1536, bias=True)
        (k_proj): Linear(in_features=1536, out_features=256, bias=True)
        (v_proj): Linear(in_features=1536, out_features=256, bias=True)
        (o_proj): Linear(in_features=1536, out_features=1536, bias=False)
        (rotary_emb): Qwen2RotaryEmbedding()
      )
      (mlp): Qwen2MLP(
        (gate_proj): Linear(in_features=1536, out_features=8960, bias=False)
        (up_proj): Linear(in_features=1536, out_features=8960, bias=False)
        (down_proj): Linear(in_features=8960, out_features=1536, bias=False)
        (act_fn): SiLU()
      )
      (input_layernorm): Qwen2RMSNorm((1536,), eps=1e-06)
      (post_attention_layernorm): Qwen2RMSNorm((1536,), eps=1e-06)
    )
  )
  (norm): Qwen2RMSNorm((1536,), eps=1e-06)
)
```

### Expected behavior

`from_config(config)` with Qwen2-VL model only loads the language model, the vision model is missing. 

Expected behavior:
`from_config` should load the whole model, in this case the language model + the vision tower. 


Comments:
Hey @mobicham, you're using `AutoModel` to load the model which automatically loads the `Qwen2VLModel` architecture; that's the text component of the model.

You seem to be looking for the model with the vision component, which is `Qwen2VLForConditionalGeneration`, used in the `AutoModelForVision2Seq` auto-class.

Could you try replacing `AutoModel` with `AutoModelForVision2Seq` in your example?

Works, thank you! Is there a way to automatically find the right AutoModel base class from a config ?
handle when from_pretrained_id is a list
Author: itazap
Pull request: yes

fix for #30045

Failing do to 2 reasons:

- [x] handle from_pretrained_id being a list, after #29473 (https://github.com/huggingface/transformers/pull/29473#pullrequestreview-1945687810)
- [ ] still fails when input starts with a space, slow adds space and fast does not in llama tests. working on this fix

Reviewer: @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31541). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix the parallel number of CI nodes when it is smaller than number of tests
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This fixes a bug where the `self.parallel` which defines the number of nodes to run the tests on circle-ci was bigger than the actual number of tests. 

This caused the `split tests` step to only give 1 test to 1 node, and all tests to other parallel nodes, thus failing them! 
This is related to the recent #30674. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33276). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Agents - Delineate sequences with easier tags
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Based on an idea by @plaggy.
This PR changes the tags that we use for now (successive headings like `Action:`, then `Observation:`) to more LLM-ready sequences (`<action>` and `<end_action>` tags) to make the different parts easier to read for the LLM.
The impact of this change on perf needs to be assessed though, I'm currently running tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33223). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Repo checks: check documented methods exist
Author: gante
Pull request: yes

# What does this PR do?

Adds a check to confirm that publicly documented methods exist. For instance, commenting out a publicly documented method like `DynamicCache.to_legacy_cache` triggers this check, which in essence is a reminder to be extra careful with methods tagged as documented.

In the process, also removed some old, unused logic.

Comments:
arf, this is getting docker image issues due to imports

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32320). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool tool @gante! LMK when you want me to take a look!

@LysandreJik the PR is now ready 🙌 (it was failing before because the check repository CI image didn't have all needed dependencies to check methods in objects)
Errors with nn.RMSNorm in DeepSpeed
Author: loadams
Pull request: no
Labels: name: bug

### System Info

Using the latest transformers from source (newer than the latest 4.44.2 release tag), the changes in [pytorch_utils ](https://github.com/huggingface/transformers/blame/5c1027bf09717f664b579e01cbb8ec3ef5aeb140/src/transformers/pytorch_utils.py#L27) from [this PR](https://github.com/huggingface/transformers/commit/c35d2ccf5a5ebcab24a0b9da3cb2527b3ae565a0) add nn.RMSNorm to the list of modules, but nn.RMSNorm isn't added to torch until the torch 2.4 release, causing CI failures when using DeepSpeed unless we either update torch or pin the transformers version.

### Who can help?

@muellerzr 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Clone latest DeepSpeed or run CI from hpu_gaudi2.yml workflow, failure [here](https://github.com/microsoft/DeepSpeed/actions/runs/10586654075/job/29335873815).

### Expected behavior

Error out when requiring a specific torch version if it doesn't exist, or similar.

Comments:
Fix at #33177 

Thanks @NielsRogge!

Fixed in linked PR, thanks!
pass module to Params4bit.from_prequantized to ensure quant_state
Author: winglian
Pull request: yes
Labels: name: Quantization

name: PyTorch FSDP

# What does this PR do?

see #32477 #32477 

Let's start with making sure pre-quantized models can be loaded properly for FSDP. This at least unlocks pre-quant models with qlora+fsdp where CPU RAM isn't a blocker.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @SunMarc 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32524). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MaskCLIP
Author: sushmanthreddy
Pull request: no
Labels: name: New model

### Model description


MaskCLIP represents a transformative step in the realm of open-vocabulary universal image segmentation. Built upon the robust foundation of pre-trained CLIP models, it negates the need for additional finetuning or distillation. The core of MaskCLIP is its innovative Transformer-based MaskCLIP Visual Encoder. This encoder is meticulously designed to integrate mask tokens with a pre-trained ViT CLIP model, making it adept at both semantic and instance segmentation, as well as class prediction. One of MaskCLIP's standout features is its ability to efficiently harness the power of pre-trained dense and local CLIP features within its Visual Encoder. This design choice not only streamlines the segmentation process but also sidesteps the traditionally lengthy student-teacher training phase. Demonstrating its prowess, MaskCLIP has consistently outperformed existing methods on renowned datasets like ADE20K and PASCAL, especially in tasks of semantic, instance, and panoptic segmentation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

project page  : https://maskclip.github.io
github : https://github.com/mlpc-ucsd/MaskCLIP


Comments:
I would like to work on this issue .

Hi @sushmanthreddy 
Any updates regarding integrating it?
fix bug in modeling_qwen2_vl
Author: zhshj0110
Pull request: yes

# What does this PR do?

See title :)

Fixes bug in modeling_qwen2_vl.
AttributeError  raise AttributeError("'{}' object has no attribute '{}'".format(^: ^'VisionAttention' object has no attribute 'head_dim' 

the [location](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L296) of this bug

Comments:
Hey @zhshj0110, thanks for your PR!

I believe this is already being handled by the following PR : https://github.com/huggingface/transformers/pull/33213

Thank you nonetheless for the prompt fix!
Fix: multigpu training
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Addresses issues in https://github.com/LLaVA-VL/LLaVA-NeXT/issues/79#issuecomment-2298261869 and https://github.com/huggingface/trl/issues/1785#issuecomment-2316330482. 

I could trace the issue to the line where we move the model to `args.device` which apparently moved the model from multigpu to a  single device. That caused device mismatch errors where the kwargs are moved to device from `hf-device-map` while the weights are all in `args.device`. I am not 100% sure this is the right fix, as I am not a Trainer master so pls let me know if there's a better way to check that

I guess @SunMarc for multigpu related issues

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33271). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Documentation of `SinkCache` has bug in example code
Author: robamler
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@zucchini-nlp, @gante

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run the following example code from the doc string of `SinkCache` ([link to doc string](https://github.com/huggingface/transformers/blob/87134662f73d5e89bb015531ddd1d4662371d317/src/transformers/cache_utils.py#L804-L819); it was added in commit 37c5ca5eb9):

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache

model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")

inputs = tokenizer(text="My name is GPT2", return_tensors="pt")

# Prepare a cache class and pass it to model's forward
past_key_values = SinkCache(window_length=256, num_sink_tokens=4)
outputs = model(**inputs, past_key_values=past_key_values, use_cache=True)
past_kv_length = outputs.past_key_values # access cache filled with key/values from generation
```

2. Obtain the following error message: `TypeError: 'SinkCache' object is not subscriptable`

It seems like `SinkCache` doesn't actually work for the GPT2 model (and so the model expects that `past_key_values` uses the legacy "list of tuples of tensors" format). An example of a model for which SinkCache works is [Locutusque/TinyMistral-248M](https://huggingface.co/Locutusque/TinyMistral-248M).

### Expected behavior

No error message, since it's an official example code.

Comments:
@robamler Thank you for opening this issue 🤗 

@zucchini-nlp all added examples are in fact broken, `gpt2` is not compatible with any of the `Cache` classes 😛 Could you open a PR to a) fix the examples b) make sure we run the examples as doctests in our daily CI?

Indeed, the model choice was not the best and seems like doctests are not run by CI before merging. WIl check those out
Config: unified logic to retrieve text config
Author: gante
Pull request: yes

# What does this PR do?

Expected in #32673, #33212

This PR unifies how we access the text config in composite models in a single function, instead of having to look for the right key. This avoids recurrent patterns of `foo if hasattr(...) else bar`.

[Previously discussed with @amyeroberts [here](https://github.com/huggingface/transformers/pull/32673#discussion_r1719792999)]

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33219). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cache docs: update
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Updates the docs with the feedback from community, makes some points more clear and fixes the docstring. I will run the doctest locally and see if we can trigger it by CI here

Fixes #32919

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Added a slow test for cache copying