noamaanMulla-03/transformers-issues huggingface dataset:
Nit about `model_accepts_loss_kwargs` for loss
Author: ArthurZucker
Pull request: yes

# What does this PR do?
There was a typo in #34915, as tests were passing, did not pay attention to it. Thanks to @techkang for reporting

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

```python
from transformers import Wav2Vec2Model
import inspect

sif = inspect.signature(Wav2Vec2Model.forward)
any(k.kind == inspect.Parameter.VAR_KEYWORD for k in sif.parameters.values())
```
Returns false. But when in the example torch, you have unwrapped_model.forward seems to have loss_kwargs
cc @muellerzr 

This PR still confuses me as I mentioned in previes PR.
There are two ways to use `num_items_in_batch` to fix GA loss issue.
1. Use `num_items_in_batch` in loss function defined by model. In this case, `model_accepts_loss_kwargs` is `True`.
2. The model doesn't have loss function or user has self-defined loss function, which is `compute_loss_func`.
However, this PR will make the second method invalid.

Hey! 
> Use num_items_in_batch in loss function defined by model. In this case, model_accepts_loss_kwargs is True.

No, `model_accepts_loss_kwargs` only depends on the `forward_pass` of the model, not the los function defined by the model. If the model accepts kwargs, we are gonna pass `num_items_in_batch`. Else we cannot. 

> The model doesn't have loss function or user has self-defined loss function, which is compute_loss_func.
However, this PR will make the second method invalid.

Even if user has a self defined loss function, if the `forward` pass does not support `num_items_in_batch`, we ought not to pass kwargs. I might be confusing with the trainer's  `compute_loss_func`, and in that case will leave @muellerzr have a look

@ArthurZucker I disagree with 2, it was explicitly designed that way to catch this in case the model doesn't have this figured out yet and the user can/should manually do it themselves instead
Fix GA loss bugs and add unit test
Author: techkang
Pull request: yes

# What does this PR do?
There are two ways to fix GA loss bugs:
1. Use `num_items_in_batch` in loss function defined by model. In this case, `model_accepts_loss_kwargs` is `True`.
2. The model doesn't have loss function or user has self-defined loss function, which is `compute_loss_func`.

However, previes unit test only test for the second condition. So I introduced a new unit test to cover the first condition and fix the bugs by the way.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@muellerzr @ArthurZucker 

Comments:
@ArthurZucker The Wav2Vec2Model bug is because SpeechEncoderDecoderModel takes variable argument as forward paramters:
https://github.com/huggingface/transformers/blob/c8c8dffbe45ebef0a8dba4a51024e5e5e498596b/src/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py#L457
But it dispatches the paramater to it's encoder and decode which doesn't accept variable argument: https://github.com/huggingface/transformers/blob/c8c8dffbe45ebef0a8dba4a51024e5e5e498596b/src/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py#L489-L493
I think the better solution is to modify it's decode to accept variable argument. I proposed a new commit and the test succeed.

Finally all unit test passed. Please check  again. @muellerzr @ArthurZucker
[I-JEPA] Update docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a follow-up of #33125 to add a figure and add a resource section.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35148). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Typo in agent system prompts
Author: jimgoo
Pull request: no

https://github.com/huggingface/transformers/blob/98e8062df3cf82b367e8a243963e4afb0d9d3407/src/transformers/agents/prompts.py#L132

System prompt for both the JSON and Code agents says: 

```
Above example were using notional tools that might not exist for you. You only have acces to those tools:
<<tool_descriptions>>
```

I suggest changing  it to refer to the tools below: `You only have access to these tools:`.

Comments:
I think langchain has a good prompt for this purpose that makes sense: https://python.langchain.com/docs/how_to/tool_calling/

cc @aymeric-roucher 

We built this system prompt with awareness of what's done in other frameworks like `LangChain` or `Autogen`.
I don't see the typo here? I think the clarification of "notional tools" is important and we should keep it.

It says `You only have acces to those tools:`, but "acces" is spelled wrong and "those" should be "these" since it's referring to the user provided tools below this and not the notional tools above.

Oh I see! Thanks a lot for pointing this out!
Fix typo in EETQ Tests
Author: MekkCyber
Pull request: yes

# What does this PR do?

Fix a typo in `eetq` tests, `modules_to_not_convert` instead of `linear_weights_not_to_quantize`

## Who can review ?

@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FileNotFoundError: Couldn't find a module script at /home/tooko/transformers-course/glue/glue.py.
Author: AmanoTooko7
Pull request: no
Labels: name: New model

### Model description

i run the example of hugging face NLP course(https://huggingface.co/learn/nlp-course/chapter3/4?fw=pt#the-training-loop)
my env at Vmware-ubuntu -vscode,evaluate Version is  0.4.3 . i always encounter this error:

import evaluate
metric = evaluate.load("glue", "mrpc") #此句依旧是出现FileNotFoundError错误
model.eval()

for epoch in range(num_epochs):
    for batch in train_dataloader:
        batch = {k: v.to(device) for k, v in batch.items()} #将batch中的数据转移到device上，batch是一个字典
        with torch.no_grad():
            outputs = model(**batch)
            
        logits = outputs.logits
        predictions = torch.argmax(logits, dim=-1)
        metric.add_batch(predictions=predictions, references=batch["labels"])

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 3
      1 import evaluate
----> 3 metric = evaluate.load("glue", "mrpc")
      4 model.eval()
      6 for epoch in range(num_epochs):

File ~/transformers-course/.env/lib/python3.11/site-packages/evaluate/loading.py:748, in load(path, config_name, module_type, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, **init_kwargs)
    703 """Load a [`~evaluate.EvaluationModule`].
    704 
    705 Args:
   (...)
    745     ```
    746 """
    747 download_mode = DownloadMode(download_mode or DownloadMode.REUSE_DATASET_IF_EXISTS)
--> 748 evaluation_module = evaluation_module_factory(
    749     path, module_type=module_type, revision=revision, download_config=download_config, download_mode=download_mode
    750 )
    751 evaluation_cls = import_main_class(evaluation_module.module_path)
    752 evaluation_instance = evaluation_cls(
    753     config_name=config_name,
    754     process_id=process_id,
   (...)
    760     **init_kwargs,
...
    684         ) from None
    685 else:
    686     raise FileNotFoundError(f"Couldn't find a module script at {relative_to_absolute_path(combined_path)}.")

FileNotFoundError: Couldn't find a module script at /home/tooko/transformers-course/glue/glue.py. Module 'glue' doesn't exist on the Hugging Face Hub either.

others opinion to solve this issue is to loging in hugging face account and not use proxy. B cos i am in China so i must use vpn to log in HF but after logged in no matter i used vpn or not the error was existing ,and i try to download glue/glue.py from Colab'example of HF provided but i didn't found. think you for your some advice for this problem (sorry for my bad english)

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
cc @merveenoyan for the course maybe? Feel free to ping someone else if you think they're more appropriate!
draft to test dduf compatibility with file explorer
Author: Wauplin
Pull request: yes

Related to https://github.com/huggingface/huggingface_hub/pull/2692, https://github.com/huggingface/diffusers/pull/10037 and https://github.com/huggingface/transformers/pull/35093.

And more precisely to https://github.com/huggingface/huggingface_hub/pull/2703 which aims at unifying the interface to handle a filesystem (e.g. a local directory) and a zip file (e.g. a DDUF file). Still a lot of changes to make if we go in this direction. I preferred to open the PR early for discussions.

### Reproduction steps:

#### Install
```sh
pip install git+https://github.com/huggingface/transformers.git@dduf-compatibility-with-file-explorer
pip install git+https://github.com/huggingface/huggingface_hub.git@draft-file-explorer
``` 

#### Run on test DDUF file
```py
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

dduf_path = hf_hub_download("DDUF/Flux.1-Dev-Tnf4-TE2NF4", "Flux.1-Dev-Tnf4-TE2NF4.dduf")

tokenizer = AutoTokenizer.from_pretrained(dduf_path, subfolder="tokenizer")
```

Comments:
It was fun for the science but let's go for a simple solution :smile: 

(discussed offline but changing stuff in transformers for the sake of diffusers' DDUF is meeh :man_shrugging: maintenance will be a huge burden, been able to load from zip affects many internal methods for low impact, it might bring breaking changes and many untested variations => let's drop it and let diffusers do the rest. Worth case we unzip the transformers's components before loading them. We want a v0 to ship, not a perfect integration aux petits oignons)
[CI] Fix bnb quantization tests with accelerate>=1.2.0
Author: matthewdouglas
Pull request: yes
Labels: name: Tests

name: Quantization

# What does this PR do?
Fixes [test failures ](https://github.com/huggingface/transformers/actions/runs/12227957829/job/34105744773#step:10:1172) for bitsandbytes that appear after `accelerate==1.2.0` was released. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. 
        [Slack thread](https://huggingface.slack.com/archives/C06ALV91VML/p1733487585960999)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35172). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Incorrect hardcoded consolidated.pth path for Llama 3.2 11B Vision+Instruct Model
Author: strangiato
Pull request: no

In the mllama convert_mllama_weights_to_hf.py script, when attempting to convert a model with the `--num_shards 1` flag, the script expects the file `consolidated.pth` in the following line:

https://github.com/huggingface/transformers/blob/f9c7e6021e9a9a9fd3fc8bb291da9451066aeb8d/src/transformers/models/mllama/convert_mllama_weights_to_hf.py#L341C1-L342C1

```
loaded = [torch.load(os.path.join(input_base_path, "consolidated.pth"), map_location="cpu", mmap=True)]
```

However, the Llama 3.2 11B vision instruct model is distributed as a single shard, but uses `consolidated.00.pth` as the filename instead:

```
ll ~/.llama/checkpoints/Llama3.2-11B-Vision-Instruct
total 41524824
drwxr-xr-x@ 6 user  group   192B Dec  2 14:05 ./
drwxr-xr-x@ 5 user  group   160B Dec  2 14:05 ../
-rw-r--r--@ 1 user  group   156B Dec  2 14:05 checklist.chk
-rw-r--r--@ 1 user  group    20G Dec  2 14:14 consolidated.00.pth
-rw-r--r--@ 1 user  group   321B Dec  2 14:05 params.json
-rw-r--r--@ 1 user  group   2.1M Dec  2 14:05 tokenizer.model
```

This results in a file not found error:

```
FileNotFoundError: [Errno 2] No such file or directory: 'path-to-model/consolidated.pth'
```


Comments:
A quick fix would be to simply update the file reference to `consolidated.00.pth` but I'm not sure if that would break compatibility with other models besides the 11B Vision Instruct model.

I'm happy to help contribute a fix if someone can weigh in on the possible compatibility issues.

Tagging @ArthurZucker since he seems to be the original author of this script.
Assisted decoding multi-gpu
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/35099, we need to manually move inputs to the correct device as assistant and target models can be initialized on different devices. We already move the input to assistant's device when `get_candidate_inputs` but don't do it in target models' forward

Btw, seems like we are now having more troubles with certain generation techniques or cache not working in multi-gpu setting. And we don't have much tests for multiple gpus, so I am not sure if that's intended or we can add more integration tests. Also if those tests will be run as part of pull request CI or not. cc @gante for this question

cc @jmamou 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35116). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp @ArthurZucker 
For AG with multi-GPU, it would be beneficial to include in the documentation that for optimal speedup, the assistant model should be placed on the default/first device of the target model (`target.device`). This setup avoids the overhead of transferring the candidate token IDs tensor from `assistant.device` to `target.device` after the speculative iteration and back to `assistant.device` with the validated token IDs following target validation

@jmamou yes, feel free to add it in the docs in subsequent open PRs if relevant
Add support for HuBERT batch norm instead of weight norm in pos_conv_emb
Author: gallilmaimon
Pull request: no
Labels: name: Feature request

### Feature request

- In the Hubert model convolutional positional encoding - add a support for batch norm instead of weight norm as added in Fairseq at - https://github.com/facebookresearch/fairseq/commit/4db264940f281a6f47558d17387b1455d4abd8d9
- This in turn will allow supporting mhubert-base-25hz which was introduced in - https://arxiv.org/abs/2305.13009 

### Motivation

- This will give the opportunity to support other textlesslib/fairseq models and specifically the newer, better HuBERT model - mhubert-base-25hz which uses this and is currently unsupported.
- This model is frequently used for training speech language models, and would benefit the community as well as myself in a project I am working on.

### Your contribution

I can create a PR to implement this, but would love some guidance @ylacombe

Comments:
Hi, would love this feature as well @ylacombe 

Hey @avishaiElmakies and @gallilmaimon , this would indeed be a great addition. 

Would you like to open a PR to correct this ?
You'd have to add the possibility to use batch norm in the `configuration_hubert.py`, propagate the change to the modeling file and the converting file, and finally add an integration test. How does it sound?

Thanks


@ylacombe Sounds good. I will work on something and let you know when the PR is ready
Did you mean masked_fill_ in modeling_deberta.py?
Author: xadupre
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: 1.0.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0.dev20241205+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4060 Laptop GPU

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

masked_fill is called but the result is not used: https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta/modeling_deberta.py#L293.

The same issue happens in modeling_deberta_v2.py.

### Expected behavior

Use masked_fill_ instead.

Comments:
This definitely looks like a bug! Would you be willing to open a PR to fix it?

I'll do that. Any recommandation for a unit test to add, any existing good one I could leverage?

@xadupre The ideal test here would load or initialize a small model, then create an attention mask, and then verify using `output_attentions` that attentions correctly follow the mask. Take a look at the tests in `tests/models/deberta/` and see if there's one you can copy/modify for your needs, and feel free to ping me for reviews or advice once the PR is made. And also, thank you for offering to do this!

I finally removed the lines. I give some explanations in the PR but using inplace masked_fill would set all values in attention_scores to zero. Maybe the code was refactored and these lines were forgotten.
Remove unnecessary masked_fill in deberta models
Author: xadupre
Pull request: yes

# What does this PR do?

Both models Deberta and DebertaV2 executes one instruction with no impact on the result. It is the last line among the 

```python
attention_scores = attention_scores.masked_fill(~(attention_mask), torch.finfo(query_layer.dtype).min)
# bsz x height x length x dimension
attention_probs = nn.functional.softmax(attention_scores, dim=-1)
attention_probs.masked_fill(attention_mask, 0)  # no impact on what follows
```

The line can be removed safely. If we replace it by one inplace modification ``attention_probs.masked_fill(attention_mask, 0)`` then the attention_probs will be null because:

* line 1: replace all False mask values by - infinity
* line 2: compute softmax, then the replaced values become 0
* line 3: if modified with inplace modification, it would replace all True mask value by 0

At the end, the output tensor would be null. No unit test was added because the numerical results are not impacted.

Fixes #35162.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35182). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Difference between vocab_size in model T5forConditionalGeneration “t5-small” and its corresponding Tokenizer “t5-small”
Author: ashutoshml
Pull request: no

## Environment info
- `transformers` version: 4.13.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @patil-suraj, @LysandreJik

## Information

Model I am using (Bert, XLNet ...): BART and T5

The problem arises when using:
* [Yes] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
from transformers import BartTokenizer, BartForConditionalGeneration

t5model = T5ForConditionalGeneration.from_pretrained('t5-small')
t5tokenizer = T5Tokenizer.from_pretrained('t5-small')

print("For T5:")
print("Tokenizer vocab_size: {}".format(t5tokenizer.vocab_size))
print("Model vocab size: {}\n".format(t5model.config.vocab_size))

bartmodel = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
barttokenizer = BartTokenizer.from_pretrained('facebook/bart-base')

print("For BART:")
print("Tokenizer vocab_size: {}".format(barttokenizer.vocab_size))
print("Model vocab size: {}".format(bartmodel.config.vocab_size))
```

Current Output

```
For T5:
Tokenizer vocab_size: 32100
Model vocab size: 32128

For BART:
Tokenizer vocab_size: 50265
Model vocab size: 50265
```

## Expected behavior

Both - the model and the corresponding tokenizer should ideally have the same `vocab_size`. Incase the additional `vocab_size` in the model is to incorporate prefix-tokens, why should they not be included in the corresponding Tokenizer.


Comments:
My main goal is to `add_tokens` to the T5Tokenizer, but not sure how the `vocab_size` of the model would get affected due to this, after resize_embeddings. It would help to get an idea of how to incorporate the above.

See #4875

Hi @ashutoshml, 

I need to add 3 special tokens to Google/long-t5-tglobal-large.

The tokenizer vocabulary is 32100 which will be increased by 3.
The Model vocab size/embedding tokens size is: 32128 (with 28 buffer tokens as per the T5 models)

My question is when I add the 3 special tokens I need to resize the embedding vocab size/embedding matrix from 32128 to 32131 or the additional 3 special tokens will be part of the buffer tokens so the embedding vocab size/embedding matrix will remain at 32128 (25 buffer tokens now).
Use modules_to_not_convert in AQLM
Author: MekkCyber
Pull request: yes

# What does this PR do?

Using `modules_to_not_convert` instead of `linear_weights_not_to_quantize` in `aqlm` to standardize it in all quantizers

## Who can review ?

@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35161). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.