noamaanMulla-03/transformers-issues huggingface dataset:
Mismatch Between txt img_token and Image Count in Multimodal Processor Causes Debugging
Author: jp1924
Pull request: no
Labels: name: Feature request

name: Multimodal

### Feature request

As with Qwen2-VL, if the number of img_tokens input in the multimodal processor does not match the number of images, 
a warning or error should be displayed.


### Motivation

## reproduction code
```python
import requests
from PIL import Image

from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor


model = Qwen2VLForConditionalGeneration.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
processor = Qwen2VLProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

prompts = [
    f"USER: {processor.image_token}{processor.image_token}\nWhat are the things I should be cautious about when I visit this place? What should I bring with me? ASSISTANT:",
]
image1 = Image.open(requests.get("https://llava-vl.github.io/static/images/view.jpg", stream=True).raw)

inputs = processor(images=[image1], text=prompts, return_tensors="pt", padding=True)
```
## env
```
- `transformers` version: 4.47.0.dev0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
```
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py", line 139, in __call__
    self.image_token, "
" * (image_grid_thw[index].prod() // merge_length), 1
IndexError: index 1 is out of bounds for dimension 0 with size 1
```

When running the code, an error like the one above occurs. 
The cause is that the number of img_tokens does not match the number of images.

However, the error is not very intuitive, so it took some time to find the cause. 
Therefore, I think it would be good to explicitly display a warning or error 
when the number of img_tokens and images do not match.

### Your contribution

It seems possible to add a statement that explicitly displays an error or warning 
when the number of img_tokens and images do not match in the multimodal processor.

Comments:
@jp1924 I think we already do similar checks for number of images and image tokens in some processors (e.g. Mllama), so it shouldn't hurt adding that in Qwen2-VL. Would like to submit a PR? 

@zucchini-nlp 
Oh, great. Then, how about adding this to the Llava series processor as well, since it is not applied there either?

@jp1924 feel free to add to other VLMs, though I would say that users must check their inputs before processing as we are adding too many checks at this moment imo
Adding Audio-MAE Model for Underwater Audio to Hugging Face Transformers Library
Author: Ahmed-Telili
Pull request: no
Labels: name: Feature request

### Feature request

Include the Audio-MAE model pretrained on underwater audio data in the Hugging Face Transformers library.

### Motivation

I have trained the Audio-MAE model on 30M underwater sound samples from the Orcasound dataset using 8 H100 GPUs for over a month. While exploring the Hugging Face Transformers library, I noticed that there is an implementation for Vision Transformer (ViT) MAE, but no implementation for Audio-MAE. Given the growing interest in underwater audio-based models, I believe that including a pretrained Audio-MAE model trained on 30 million underwater sound samples could be valuable for a wide range of audio-related applications, particularly in underwater acoustics.

### Your contribution

I would like to inquire whether a pretrained Audio-MAE model, with weights trained on 30 million underwater sound samples, would be considered for inclusion in the Transformers library. If needed, I am happy to contribute the model or further assist with its integration.

Comments:
Hi @Ahmed-Telili, thanks for the message, and the model sounds interesting!

We generally advise users that they don't really need our permission to upload models - you can create a "custom code" model and upload it to a Transformers repo even if the main library doesn't support your modeling code. See the guide [here](https://huggingface.co/docs/transformers/en/custom_models)
Another security patch for `self-comment-ci.yml`
Author: ydshieh
Pull request: yes

# What does this PR do?

For the jobs go after `get-sha` which will checkout to `refs/pull/$PR_NUMBER/merge`, we need to check the checked out commit sha (which we don't know if it is safe) is the same as the (already verified) one given by `${{ needs.get-sha.outputs.PR_MERGE_SHA }}`.

Comments:
Init cache on meta device
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33147.

Initializes cache on meta device until we get the first key/values and infer the device from that. Removed `layer_device_map` as not needed anymore in most cases, because initializing cache on meta device is the default value when no `device` is given. Offloaded Static cache still would require layer device map or one device, since it prefetches key/values in advance and we cannot infer the device as soon as we see the input. 

One thing to note is that Offloaded Static cache can never run on current `main` because `torch.cuda.stream()` is not fullgraph compile compatible. Found a related issue from torch team on that: https://github.com/pytorch/pytorch/issues/92804. On current `main` I can't run even with graph breaks, and it fails after 3rd layer on `torch.cuda.default_stream(self.device).wait_stream(self._prefetch_stream)`, but with this branch it runs if we allow graph breaks until compile cache limit is reached 

Since this is the only cache that is behaving different, maybe we can make it `not instance of StaticCache` at least until the fullgraph compile is working

----------------------
- [x] Slow tests on `test_cache_utils.py` and on Llama are green, when compared to `main` branch. Some slow llama tests are red on `main` and do not use static cache, so it wasn't caused by this PR

----------------------------------------------
Issue repro with 2 GPUs
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = 'google/gemma-2-2b-it'
tokenizer = AutoTokenizer.from_pretrained(model_id)

device_map = {"model.embed_tokens": 0, "model.norm": 1, "model.rotary_emb": 1, "lm_head": 0}
num_hidden_layers = 26
for i in range(num_hidden_layers):
    device_map[f"model.layers.{i}"] = 0 if i < 13 else 1

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="bfloat16",
    device_map=device_map,
)

inputs = tokenizer("Today is a beautiful day!", return_tensors='pt').to(0)
out = model(**inputs)
```


----------------------------------------------
Benchmark on llama + compile with `meta-llama/Llama-3.2-1B-Instruct`:

![image](https://github.com/user-attachments/assets/b1da987b-6278-4651-81cf-afef176d3deb)




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35164). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix : Nemotron tokenizer for GGUF format
Author: MekkCyber
Pull request: yes

# What does this PR do?
Adds the `PretrainedTokenizerFast` class as a tokenizer to `TOKENIZER_MAPPING_NAMES` to fix this : https://github.com/huggingface/transformers/actions/runs/12900045095/job/35970056210 
and fixes an expected output for the nemotron test

## Who can review ?
@SunMarc @Isotr0py 

Comments:
labels should not be same as input_ids for casual language model
Author: thomas010
Pull request: no

### System Info

in the example for casual language mode pretraing (examples/pytorch/language-modeling/run_clm.py: line 490),
labels in 'result' should not be the same as input_ids, but maybe as follows
````
input_ids = result["input_ids"]
  result["input_ids"] = input_ids[:, :-1].contiguous()
  result["labels"]=input_ids[:, :-1].contiguous()
````


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

examples/pytorch/language-modeling/run_clm.py: line 490

### Expected behavior

update

Comments:
@thomas010 did you mean to provide the same code for `result['input_ids']` and `result['labels']`?

I don't know if this helps you, but in standard causal language modeling, the input sequence and labels _should_ be the same.  Suppose the sequence is <token_A>, <token_B>, <token_C>.  In step 0, we want to predict <token_A>.  In Step 1, we want to predict <token_B> from <token_A>.  In step 2, we want to predict <token_C> from <token_A>, <token_B>.  `transformers` knows this is what we want when we provide the same sequence as `input_ids` and `labels`.

i got it, thank you

> [@thomas010](https://github.com/thomas010) did you mean to provide the same code for `result['input_ids']` and `result['labels']`?
> 
> I don't know if this helps you, but in standard causal language modeling, the input sequence and labels _should_ be the same. Suppose the sequence is <token_A>, <token_B>, <token_C>. In step 0, we want to predict <token_A>. In Step 1, we want to predict <token_B> from <token_A>. In step 2, we want to predict <token_C> from <token_A>, <token_B>. `transformers` knows this is what we want when we provide the same sequence as `input_ids` and `labels`.

Can it be considered that the `transformers` framework will automatically build `input_ids` based on the sequence `<token_A>, <token_B>, <token_C>`: `<start>, <token_A>, <token_B>, <token_C>`, and build `labels`: `<token_A>, <token_B>, <token_C>, <end>`
[Chat] Add Chat from TRL 🐈 
Author: gante
Pull request: yes

# What does this PR do?

Adds a `chat` command, allowing users to explore text models with a minimal effort. Except for the transformers-specific command boilerplate, this chat is a copy of the chat in TRL.

The purpose of this PR is to establish a baseline from something that already works well. In subsequent PRs, a few QoL upgrades will be added to this chat, including:
- Documentation in `transformers`
- Allowing any flag in `GenerationConfig` to be set
- Allowing a `generation_config.json` file to parameterize the chat, and allow users to export current chat settings into files
- Bring performance optimization tricks and settings presets from `local-gemma`, including compilation
- Allow files as input, for ASR models and VLMs

## Usage

1. checkout to this PR
2. `transformers-cli chat --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct`
3. for full list of commands: `transformers-cli chat --help` 
4. for help while chatting, type `help` 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35714). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker I've added docs, but the preview isn't showing them 🤔 I'm merging and then I'll check the docs on `main` to see if it needs some corrections
Fix `head_dim` in config extracted from Gemma2 GGUF model
Author: Isotr0py
Pull request: yes

When developing Gemma2 GGUF support for vLLM, it's reported that the `head_dim` from 27B model is incorrect. (https://github.com/vllm-project/vllm/pull/12186#issuecomment-2604677797)
- Fix incorrect `Gemma2Config`'s `head_dim` extracted from [gemma-2-27b-it-GGUF](https://huggingface.co/bartowski/gemma-2-27b-it-GGUF) models etc.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @MekkCyber

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
SG thanks for this fix !
Add PyTorch version check for FA backend on AMD GPUs
Author: mht-sharma
Pull request: yes

# What does this PR do?
Updates condition from PR https://github.com/huggingface/transformers/pull/30850
> Garbage values may occur during model generation with models like LLama, Mistral, and Mixtral, particularly when utilizing multi-gpu setup and device_map=auto alongside SDPA and FA.

The above issue is fixed in the newer PyTorch versions.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
cc @sunmarc @mekkcyber @muellerzr

Good catch ! LGTM, thanks for the fix !
Fix IsADirectoryError in notebook progress bar display methods. Fixes #34766
Author: noah-ing
Pull request: yes

## What does this PR do?
This PR fixes the IsADirectoryError that occurs when using the notebook progress bar in Azure ML Notebooks. The issue was caused by HTML content being incorrectly treated as a file path. Added error handling to gracefully fall back to basic display when HTML display fails.

## Changes
- Added try/except blocks to handle IsADirectoryError in NotebookProgressBar and NotebookTrainingTracker display methods
- Added graceful fallback to basic display when HTML display fails
- Added tests to verify the fix works correctly

## Before submitting
- [x] This PR fixes a bug reported in #34766
- [x] This was discussed/approved in issue #34766
- [x] Added tests to verify the fix
- [x] Updated relevant documentation regarding progress bar behavior

cc @RocketKnight1 who added the "Good First Issue" label and @LysandreJik who initially responded to the issue.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Not sure who owns the notebook code there - cc @muellerzr @sunmarc to review because it's Trainer-related, but feel free to ping someone else if you think they're better suited.
Fix compatibility issues when using auto_gptq with these older versions
Author: Alimu021
Pull request: yes

# What does this PR do?

The `convert_model` method in versions of optimum prior to 1.23.99 only accepts a single parameter `model: nn.Module`, latest version allows the acceptance of `model: nn.Module` and `**kwargs`. To resolve compatibility issues when using auto_gptq with these older versions, a condition needs to be added.

gptqmodel checks the version of optimum as well when verifying versions, so this issue does not exist. (https://github.com/huggingface/transformers/blob/a142f161313199bcfa67afe1990d1f0f39a973bb/src/transformers/quantizers/quantizer_gptq.py#L76-L80)

# Fixed this issue:

PKG Version:
`transformers`: latest source
`auto-gptq`: latest release
`optimum`: latest release

Traceback:
```python
self = <transformers.quantizers.quantizer_gptq.GptqHfQuantizer object at 0x7787215014d0>
model = OPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 768, padding_id...entwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=768, out_features=50272, bias=False)
)
kwargs = {'device_map': 'auto', 'keep_in_fp32_modules': []}

    def _process_model_before_weight_loading(self, model: "PreTrainedModel", **kwargs):
        if model.__class__.main_input_name != "input_ids":
            raise RuntimeError("We can only quantize pure text model.")
    
        if self.pre_quantized:
>           model = self.optimum_quantizer.convert_model(model, **kwargs)
E           TypeError: GPTQQuantizer.convert_model() got an unexpected keyword argument 'device_map'

../../../anaconda3/envs/peft-test/lib/python3.11/site-packages/transformers/quantizers/quantizer_gptq.py:104: TypeError
```

## Who can review?
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber @BenjaminBossan 


Comments:
LGTM @LRL-ModelCloud, thanks for the fix !
Fix NoneType type as it requires py>=3.10
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes an issue for older version of python (3.9) wrt `NoneType`. The `types.NoneType` was introduced in Python 3.10. So, we need to use type(None) instead. 
We have this [issue](https://github.com/huggingface/accelerate/actions/runs/12900277369/job/36002947315?pr=3350) in accelerate CI 

Comments:
cc @aymeric-roucher, merging this one instead of #35829
is it expected that the same token has two different token ids in T5TokenizerFast
Author: garychan22
Pull request: no

I am using the T5TokenizerFast from "black-forest-labs/FLUX.1-dev"
```
tokenizer_two = T5TokenizerFast.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    subfolder="tokenizer_2",
)

print(tokenizer_two.encode("person"))
Output:
[568, 1]

print(tokenizer_two.convert_tokens_to_ids("person"))
Output:
6075

print(tokenizer_two.decode([6075]))
Output:
person

print(tokenizer_two.decode([568]))
Output:
person

text_inputs = tokenizer_two(
    "A photo of a person.",
    padding="max_length",
    max_length=tokenizer_two.model_max_length,
    truncation=True,
    return_tensors="pt",
)
print(text_inputs.input_ids) 

Output:
tensor([  71, 1202,   13,    3,    9,  568,    5,    1,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0])

```

Since I am new to this tokenizer, is it expected that the token "person" has two different token id 568 and 6075? Thanks.

transformers version: 4.46.3

Comments:
Hi @garychan22, this is usually caused by one of the token variants including whitespace or a sentencepiece underline, which might be masked by the `decode()` method. Try finding those two tokens in `tokenizer.vocab`. It is not a bug!