noamaanMulla-03/transformers-issues huggingface dataset:
Add cosmos from Nvidia
Author: ArthurZucker
Pull request: no
Labels: name: New model

name: Good Difficult Issue

### Model description

https://www.nvidia.com/en-us/ai/cosmos/ on model is autoregressive, all open source

We might have a PR coming directly from NVIDIA! 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Is anyone currently working on PR to add to this Issue?

I would like to get some more inputs on this. Maybe, I could work on this issue after some help on ways to perform this implementation...

any progress? 

Nope. I am new to this so maybe a senior person should take up the issue...

I'll take this and make a draft PR soon

@zucchini-nlp are you working on it?

@Tanuj-rai yes, will submit a draft soon

@zucchini-nlp, I am also working on this issue. I was not aware that you were also working on it, so I made a PR. Please just go ahead and ignore my PR. Since I am new to this, can I work with you on this issue?
[don't merge] check required checks
Author: ydshieh
Pull request: yes

[don't merge] check required checks

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36452). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Should fix, testing
Author: muellerzr
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/36463 did not account for the fact that when the shapes mismatch, we just report that we're initing with new weights and continue forward. This PR keeps our old behavior, with the old warnings that were thrown.

Fixes broken main


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@qubvel @ArthurZucker 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36465). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Feature Request] We might need a function to change the sampler used in trainer dataloader
Author: dumpmemory
Pull request: no
Labels: name: Feature request

### Feature request

we need a function to custom the sampler used in trainer dataloader.

### Motivation

for sometimes, we might want to use https://github.com/imoneoi/multipack_sampler/blob/master/multipack_sampler.py#L94 for efficient reasoning

another example is DoReMi paper， we might also change the  sample distribution  for different task source

### Your contribution

I 

Comments:
Happy to look into that!

Hi, I am also interested in this feature, and as far as I know we can just override `_get_train_sampler` https://github.com/huggingface/transformers/blob/a22a4378d97d06b7a1d9abad6e0086d30fdea199/src/transformers/trainer.py#L969-L994  within the `Trainer`

One question that I have is that whether or not the `Trainer` will make sure that the sampler works in distributed mode. My understanding is that this happens under the hood https://discuss.huggingface.co/t/can-accelerator-handle-the-distributed-sampler/12943/2 but I would like to double check. Thanks! :)
How to change data
Author: kim90000
Pull request: no



https://huggingface.co/facebook/rag-token-nq

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)

input_dict = tokenizer.prepare_seq2seq_batch("who holds the record in 100m freestyle", return_tensors="pt") 

generated = model.generate(input_ids=input_dict["input_ids"]) 
print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0]) 

# should give michael phelps => sounds reasonable



My attempts
https://github.com/kim90000/Attempts-with-facebook-rag-token-nq/blob/main/README.md

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[generate] `torch.distributed`-compatible `DynamicCache`
Author: gante
Pull request: yes

# What does this PR do?

`DynamicCache` was not compatible with `torch.distributed` before, and #36212 exposed the issue. See the discussion starting [here](https://github.com/huggingface/transformers/pull/36212#issuecomment-2665247860) for more details. The added code contains comments explaining what's going on.

Fixes `Trainer` + DP when `use_cache=True`, such as in prefix tuning training runs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36373). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@S1ro1 it should be okay now 👌 `torch.distributed` is clever and builds the tensors themselves correctly, all we need is to place them in the right variables

@gante Comments look good to me, maybe small nit would be writing down the shapes? i.e. `(num_layers, kv, (...))`, else looks good to me, test as well.

(merging to unblock PEFT)
Fix loading models with mismatched sizes
Author: qubvel
Pull request: yes

# What does this PR do?

Fix loading model with mismatched sizes, the bug was introduced since:
 - https://github.com/huggingface/transformers/pull/36335

Reported in
 - #36261 

Fixes: #36464

Minimal reproducing example:

```python
from transformers import AutoModelForObjectDetection

model = AutoModelForObjectDetection.from_pretrained(
    "PekingU/rtdetr_r18vd",
    id2label={0: "person"},
    label2id={"person": 0},
    ignore_mismatched_sizes=True
)
```

cc @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36463). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AutoModelForObjectDetection isnt working due to wrong output size
Author: nikitabelooussovbtis
Pull request: no
Labels: name: bug

### System Info

new:
- `transformers` version: 4.50.0.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.29.1
- Safetensors version: 0.5.3
- Accelerate version: 1.4.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

old:
- `transformers` version: 4.50.0.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.29.0
- Safetensors version: 0.5.2
- Accelerate version: 1.4.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I was using the repisotry 2 days ago and it was working fine. With the new updates something has changed and now when I try to load a model it is telling me that the output size is incorrect: size mismatch for class_labels_classifier.weight: copying a param with shape torch.Size([92, 256]) from checkpoint, the shape in current model is torch.Size([11, 256]).  The ignore_mismatched_sizes has been set to true and I am using the facebook/detr-resnet-50 model specefically. I am pretty sure that it is due to a change in the repository since I copied the code and just reinstalled the repo in a new enviroment. In the old enviroment it works fine in the new one it is breaking. This is with the run_object_detection.py script.

### Expected behavior

that the model is able to adapt to the new model size and train

Comments:
The old comment id is: 7c5bd24ffac6510eb64a0cc2599d9372de809407
The new one is 222505c7e4d08da9095d12ddb72fb653f4b6da33







Hey @nikitabelooussovbtis! The fix is coming
 - #36463 

@qubvel Awesome thanks

It should be fixed now! Let me know if any other issues left 🤗
about siglip2
Author: zcswdt
Pull request: no

When I run this code, the following error occurs. I installed the latest version of transformers according to the official.

````
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("google/siglip2-base-patch16-224")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    image_features = model.get_image_features(**inputs)
    print('image_features',image_features)
````


The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'GemmaTokenizer'. 
The class this function is called from is 'SiglipTokenizer'.

transformers== 4.49.0



Comments:
Hi @zcswdt 

Please install transformers from main or from this stable branch: `pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2` to use SigLIP 2.

> Hi [@zcswdt](https://github.com/zcswdt)
> 
> Please install transformers from main or from this stable branch: `pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2` to use SigLIP 2.

thank you
[docs] fix bug in deepspeed config
Author: faaany
Pull request: yes

# What does this PR do?
While running the deepspeed example code, I got the following error, indicating the value of `stage3_prefetch_bucket_size` in the deepspeed config is not correct. The given value is 3774873.6, which is a float, but it should be an integer. So this PR fixes this. 

```bash
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sdp/fanli/doc_to_fix.py", line 112, in <module>
[rank0]:     model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
[rank0]:   File "/home/sdp/fanli/transformers/src/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
[rank0]:     return model_class.from_pretrained(
[rank0]:   File "/home/sdp/fanli/transformers/src/transformers/modeling_utils.py", line 261, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/sdp/fanli/transformers/src/transformers/modeling_utils.py", line 4144, in from_pretrained
[rank0]:     deepspeed.zero.Init(config_dict_or_path=deepspeed_config()),
[rank0]:   File "/home/sdp/miniforge3/envs/ipex-ww04/lib/python3.10/site-packages/deepspeed-0.16.3+66d3d3e94-py3.10.egg/deepspeed/runtime/zero/partition_parameters.py", line 949, in __init__
[rank0]:     _ds_config = deepspeed.runtime.config.DeepSpeedConfig(config_dict_or_path,
[rank0]:   File "/home/sdp/miniforge3/envs/ipex-ww04/lib/python3.10/site-packages/deepspeed-0.16.3+66d3d3e94-py3.10.egg/deepspeed/runtime/config.py", line 797, in __init__
[rank0]:     self._initialize_params(copy.copy(self._param_dict))
[rank0]:   File "/home/sdp/miniforge3/envs/ipex-ww04/lib/python3.10/site-packages/deepspeed-0.16.3+66d3d3e94-py3.10.egg/deepspeed/runtime/config.py", line 817, in _initialize_params
[rank0]:     self.zero_config = get_zero_config(param_dict)
[rank0]:   File "/home/sdp/miniforge3/envs/ipex-ww04/lib/python3.10/site-packages/deepspeed-0.16.3+66d3d3e94-py3.10.egg/deepspeed/runtime/zero/config.py", line 73, in get_zero_config
[rank0]:     return DeepSpeedZeroConfig(**zero_config_dict)
[rank0]:   File "/home/sdp/miniforge3/envs/ipex-ww04/lib/python3.10/site-packages/deepspeed-0.16.3+66d3d3e94-py3.10.egg/deepspeed/runtime/config_utils.py", line 57, in __init__
[rank0]:     super().__init__(**data)
[rank0]:   File "/home/sdp/.local/lib/python3.10/site-packages/pydantic/main.py", line 214, in __init__
[rank0]:     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
[rank0]: pydantic_core._pydantic_core.ValidationError: 1 validation error for DeepSpeedZeroConfig
[rank0]: stage3_prefetch_bucket_size
[rank0]:   Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=3774873.6, input_type=float]
[rank0]:     For further information visit https://errors.pydantic.dev/2.10/v/int_from_float
```

Documentation: @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36081). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @muellerz, could you pls take a look? Thx!
model.generate() produces different outputs with padding for flan-t5-small
Author: achartier
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-5.15.0-133-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.4.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0a0+ecf3bae40a.nv25.01 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: no
- GPU type: NVIDIA A30


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")

input_texts = [
    "Translate to French: Hello, how are you?",
    "Summarize: The quick brown fox jumps over the lazy dog."
]

inputs = tokenizer(input_texts, return_tensors="pt", padding=True, truncation=True)
outputs = model.generate(**inputs, max_new_tokens=50)

for i, output in enumerate(outputs):
    decoded_output = tokenizer.decode(output, skip_special_tokens=True)
    print(f"Input {i + 1}: {input_texts[i]}")
    print(f"Output {i + 1}: {decoded_output}\n")
```

Output with transformers 4.49.0:
```
Input 1: Translate to French: Hello, how are you?
Output 1:

Input 2: Summarize: The quick brown fox jumps over the lazy dog.
Output 2: The brown fox jumps over the dog.
```

I found references to similar issues (https://github.com/huggingface/transformers/issues/28385), but this one seems to be a regression between 4.48.3 and 4.49.0. I tried combinations of right and left padding with `use_cache`, but no luck.

Interestingly, `t5-small` model does not show this regression.

### Expected behavior

Output with transformers 4.48.3:
```
Input 1: Translate to French: Hello, how are you?
Output 1: Hello, c'est-ce que vous êtes?

Input 2: Summarize: The quick brown fox jumps over the lazy dog.
Output 2: The brown fox jumps over the dog.
```

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/36293. It was fixed on main branch, so you can upgrade `transformers`

Thanks! I can confirm it is indeed fixed in the main branch.
set empty list of suppress tokens to None
Author: Lewington-pitsos
Pull request: yes

# What does this PR do?


Fixes #36341 

pytest /tests/pipelines/ run locally and the same number of tests fail on main as on this branch

```
14 failed, 146 passed, 161 skipped, 25 warnings in 857.97s (0:14:17) 
```

this may not be the best way to solve the issue, but it looked like other aspects of the config were being overridden at this point. 

Arguably the line `prev_start_of_text = suppress_tokens[-2] if suppress_tokens is not None else None` which is causing the errors should be changed instead though it has been here for > 1 year so I assume the same kind of check exists in a lot of places in the codebase.

This new state of affairs is probably more in-line with existing documentation than before as in multiple places the docs state that this parameter should be a list and don't mention anything about overriding with None e.g. https://huggingface.co/docs/transformers/v4.49.0/en/internal/generation_utils#transformers.FlaxSuppressTokensLogitsProcessor 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),

@Rocketknight1 @ylacombe



Comments:
cc @gante for generation config - do you know if this usage of `suppress_tokens` is correct?

Hey @Lewington-pitsos 👋 

I believe we merged an equivalent fix yesterday: https://github.com/huggingface/transformers/pull/36336

Let us know whether the underlying issue is fixed with the PR linked above, or whether we need to merge your suggested fix as well 🤗 

[@Rocketknight1 yup, `supress_tokens` can be an empty list, and it should be equivalent to being `None`. ]

#36336 fixes the issue yes, good stuff guys!
remove to restriction for 4-bit model
Author: SunMarc
Pull request: yes
Labels: name: Quantization

# What does this PR do ? 

Since bnb 0.43.0, you freely move bnb models across devices. This PR removes the restriction we put in place. 
Needs to be tested. cc @matthewdouglas 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @SunMarc! I've tested moving between gpu->cpu->gpu, but not yet on multiple GPUs. We'll still see a warning from accelerate: 
> You shouldn't move a model that is dispatched using accelerate hooks.

Reference note: this should fix #24540 for 4bit.

For 8bit there is still a blocker: bitsandbytes-foundation/bitsandbytes#1332; once that's fixed & released on the bitsandbytes side we can do an additional PR.

Just to let you all know, this change breaks those using Quanto instead of BitsandBytes. Yeah, I know, Quanto seems to be the ugly duckling of quantizers. Before this update I could move my pipeline of of 'cuda' to the 'cpu'. When I changed my code to reflect this the 1st time I offload:
`pipe = pipe.to(dtype=torch.float32,device='cpu', silence_dtype_warnings=True)`

Then come back in to reuse my saved pipe:
`pipe = FluxPipeline.from_pipe(pipe)`
badabing badaboom!

I get this error:
![spxEEC9](https://github.com/user-attachments/assets/816e3239-3051-4df2-8eb6-3709f0b33741)
So it seems you don't want me offloading a quantized file for quanto?
convert_llama_weight_to_hf.py
Author: AyushSingh096
Pull request: no
Labels: name: bug

### System Info

@AshishMulupuri
I am getting the same error as you stated here, can you guide me how to resolve this

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.