noamaanMulla-03/transformers-issues huggingface dataset:
Llama: make slow tests green 🟢 
Author: gante
Pull request: yes

# What does this PR do?

Part 1 of #32685 -- update our tests to be sure we don't break things 🤗 

Makes slow `llama` tests happy on my local environment. Some tests are still failing on our slow CI, mostly due to hardware (e.g. out of memory), but they do not have an impact on #32685 [in other words, in checking the correctness of new changes].

The exception is `test_compile_static_cache`, which passes when run in isolation but fails if run with other tests due to `accelerate` + `torch.compile` incompatibilities (see our internal discussion [here](https://huggingface.slack.com/archives/C01Q6JPP6NA/p1724694563976419)).



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33138). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
redundant checkpointing in example scripts
Author: eminorhan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.4
- Platform: Linux-5.14.0-284.40.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.4
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In several of the example scripts, running with ``gradient_accumulation_steps > 1`` causes redundant checkpointing in step-based checkpointing. For example, [here](https://github.com/huggingface/transformers/blob/481e15604a0527fec3cbdcccd350f9374a9116b1/examples/pytorch/language-modeling/run_clm_no_trainer.py#L641):
```python 
if completed_steps % checkpointing_steps == 0:
```
This will keep needlessly checkpointing the accelerator state ``gradient_accumulation_steps`` times until the ``completed_steps`` is iterated again. This can be particularly problematic when ``gradient_accumulation_steps`` is large.

The same issue affects several of the example training scripts: _e.g._ ``run_clm_no_trainer``, ``run_fim_no_trainer``, ``run_mlm_no_trainer``, _etc._ (and probably more).

### Expected behavior

A quick fix is to add a clause to the if statement above to make sure checkpointing is done only once when ``gradient_accumulation_steps > 1``, _e.g._:
```python 
if completed_steps % checkpointing_steps == 0 and step % args.gradient_accumulation_steps == 0:
```
where ``step`` is the dataloader step. 

Comments:
hey! Would you like to open a pr with a fix? 🤗 

Sure, will do so shortly!

I submitted a PR [here](https://github.com/huggingface/transformers/pull/33131) that fixes the issue. I checked all training scripts in `transformers/examples` for places where this issue arises. In the end, I found 15 different training scripts affected by this issue and fixed them all.
ValueError: cannot reshape array of size <number> into shape
Author: TheSlackOne
Pull request: no
Labels: name: bug

### System Info

Python 3.11.2
Debian 12
ctransformers            0.2.27
transformers             4.42.4

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Following the example on Hugging Face: https://huggingface.co/docs/transformers/en/gguf

```python3
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

# Ensure the model is loaded on the GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"device set to: {device}")

# Load the model and tokenizer
model_name = "TheBloke/dolphin-2.0-mistral-7B-GGUF"
model_file = "dolphin-2.0-mistral-7b.Q4_K_M.gguf"
print(f"Loading tokenizer from {model_name}.")
tokenizer = AutoTokenizer.from_pretrained(model_name, gguf_file=model_file)
print(f"Loading model from repo {model_name}. Model file name {model_file}.")
model = AutoModelForCausalLM.from_pretrained(model_name, gguf_file=model_file)
```
I get
```
ValueError: cannot reshape array of size 36864000 into shape (222,72)
```

### Expected behavior

Model should be loaded.

Comments:
Problem solved for me, see [this commit](https://github.com/huggingface/transformers/commit/934fe1504e6d5e87e01d96305f4d97faa63cf4c1) and apply the changes or wait for the next update.

I think it was part of 4.43! 🤗 

Why do I still have this bug when I use the latest version?

- Python 3.10.14
- Windows 11
- ctransformers 0.2.27
- transformers 4.43.4
- gguf 0.9.1

![image](https://github.com/user-attachments/assets/86696611-0541-4670-b28b-732393c7dc03)


can you print `import transformers;transformers.__version__` ? 🤗 

I can confirm that it works on 4.44.0

I faced the same problem. Initially I was using Transformers 4.41 ig and upgraded to 4.44. Still faced the same problem. Looked at the code and had same commits. So, cleared cache and restarted the system, it worked.


Cool, closing as solved!

> Cool, closing as solved!

Hey @ArthurZucker, please, could you explain how it is solved?

> cleared cache and restarted the system, it worked. 
😉
`test_encode_decode_fast_slow_all_tokens` is failing
Author: ydshieh
Pull request: no
Labels: name: Core: Tokenization

name: WIP

This test starts to fail after #29473

See https://github.com/huggingface/transformers/pull/29473#pullrequestreview-1945687810

#30044 skips it for now, but let's fix it once we have the bandwidth.

Comments:
@itazap ! And @molbap this is the test that would have saved you

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
update torch req for 4-bit optimizer
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR updates the torch req in order to use torchao 4-bit optimizer. When I created the PR (using torchao nighlty), it was working with prior version of torch but now with torchao 0.4.0, you need to have torch nighlty to use it. cc @gau-nernst

Comments:
> When I created the PR (using torchao nighlty), it was working with prior version of torch

Do you remember which version of torch did it work with? Maybe we broke something when we added FSDP support 🌚 cc: @msaroufim

https://github.com/pytorch/ao/commits/v0.4.0/torchao/prototype/low_bit_optim/adam.py

Well, since I put the req for torch 2.3 before, I think it worked with that version.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33144). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Very small change to one of the function parameters
Author: alisalamatian1
Pull request: yes

# What does this PR do?
np.random.randint second parameter is not included in the possible options. Therefore, we want the upper range to be 2, so that we have some 1 labels in our classification as well. 

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
Anyone in the community is free to review the PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32548). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix redundant checkpointing in example training scripts
Author: eminorhan
Pull request: yes

# What does this PR do?

Fixes #32653.

Briefly, in several of the example training scripts, running with `gradient_accumulation_steps > 1` currently causes `gradient_accumulation_steps` times redundant checkpointing in the step-based checkpointing mode. This PR fixes the issue by adding a clause to the step-based checkpointing condition to make sure saving is done only once at the appropriate checkpointing step.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33131). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Microsoft CLAP model
Author: kamilakesbi
Pull request: yes
Labels: name: Audio

## What does this PR do ? 

This PR aims at adding the [Microsoft CLAP](https://github.com/microsoft/CLAP/tree/main) (MSClap)  model to Transformers. The architecture can be decomposed in two parts: 

The first part contains:
1. A GPT-2 Text Encoder, based on the Transformers GPT-2 model.
2. An Audio Encoder, based on the [HTS-AT architecture](https://arxiv.org/abs/2202.00874).

It can be used mostly for zero-shot audio classification or audio retrieval.

The second part adds:

 3.  A mapper model which maps the audio embeddings to a GPT2 input sequence.
 4.  A GPT-2 text decoder (also based on the Transformers model)
This second architecture can be used to perform audio captioning. 

For now, in this PR, we will only add the first part (text encoder + audio encoder) of the architecture. I will add the second part in a following PR. 

## What have been done for now

- [x]  Adapt the audio model using the current CLAP architecture in Transformers (laion clap).
- [x]  Integrate the Text encoder model. 
- [x] Successfully converted checkpoints and pushed them to the Hub. 
- [x] Added Config files, Feature Extractor and Preprocessor. 
- [x]  Made sure we get the same output from the FeatureExtractor / Processor and Models as we did from the original MS Clap. 
- [x] To Do: Add tests.  



Comments:
@ylacombe @sanchit-gandhi this should be ready for a first review :) 

@sanchit-gandhi common tests and integration tests are passing, this should be ready for second review :) 

This should be ready for final review @ylacombe @sanchit-gandhi 

@kamilakesbi Why did you delete my comment?
Is it possible to add L1/L2 regularization using the trainer class ?
Author: mayank64ce
Pull request: no
Labels: name: Feature request

### Feature request

I want to add L1/L2 regularization to the transformer training.

### Motivation

Adding L1/L2 reg can promote sparser models that can accelerate inference and reduce storage.

### Your contribution

Not sure.

Comments:
Hi,

If you want to add L1/L2 regularization to your loss function, you can overwrite the Trainer, as shown [here](https://discuss.huggingface.co/t/specify-loss-for-trainer-trainingarguments/10481/2?u=nielsr).

Thanks for the reply. Is it possible to pass additional parameters (like regularization weights) to the `compute_loss` method ?

No, I'm afraid adding those isn't possible, as we already support [114 flags](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).
[Whisper] Attention mask not detected in `Whisper.generate()`
Author: benniekiss
Pull request: no
Labels: name: Good Second Issue

name: bug

### System Info

- `transformers` version: 4.44.0.dev0
- Platform: Linux-6.9.7-201.fsync.fc40.x86_64-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.1
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3080

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import AutoProcessor, WhisperForConditionalGeneration
from datasets import load_dataset, Audio

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")
model.cuda()
# load audios > 30 seconds
ds = load_dataset("distil-whisper/meanwhile", "default")["test"]
# resample to 16kHz
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
# take first 8 audios and retrieve array
audio = ds[:8]["audio"]
audio = [x["array"] for x in audio]

# make sure to NOT truncate the input audio, to return the `attention_mask` and to pad to the longest audio
inputs = processor(audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)
inputs = inputs.to("cuda", torch.float32)

# transcribe audio to ids
generated_ids = model.generate(
    **inputs, 
)
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)
transcription[0]
```

### Expected behavior

When an `attention_mask` is passed to `generate()`, the following warning pops up indicating that an `attention_mask` was not set:
```
The attention mask is not set and cannot be inferred from input because pad token is same as eos token.As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
```

I think this is because `attention_mask` [is not actually passed down to `generate_with_fallback`](https://github.com/huggingface/transformers/blob/1392a6867f40a55dfabaf306745c67627598b1af/src/transformers/models/whisper/generation_whisper.py#L658-L676), so it doesn't get passed to the underlying `super().generate()` call

Comments:
Would you like to open a PR for a fix? cc @ylacombe as well! 

I would be happy to!
Add CRF layer after Transformer model
Author: yuyan-z
Pull request: no
Labels: name: wontfix

I've read a paper titled "Named Entity Recognition in Chinese Electronic Medical Records Using Transformer-CRF". 
It takes Transformer's output as CRF's input, as shown in the figure. 
Which function could I use to implement it? model.add() doesn't work.
![Screenshot_20200615_173206_cn wps moffice_eng](https://user-images.githubusercontent.com/64955334/84675939-7d782080-af5f-11ea-8d9d-9dc812b2229a.png)


Comments:
You can directly feed the output of a transformer's hidden states into a CRF implementation like this https://github.com/s14t284/TorchCRF

Hope it helps!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> I've read a paper titled "Named Entity Recognition in Chinese Electronic Medical Records Using Transformer-CRF".
> It takes Transformer's output as CRF's input, as shown in the figure.
> Which function could I use to implement it? model.add() doesn't work.
> ![Screenshot_20200615_173206_cn wps moffice_eng](https://user-images.githubusercontent.com/64955334/84675939-7d782080-af5f-11ea-8d9d-9dc812b2229a.png)

小哥哥 实现了没

This repository have showed how to add a CRF layer on transformers to get a better performance on token classification task.
https://github.com/shushanxingzhe/transformers_ner

I don't think that this implementation is good. 
First, it doesn't take into account the fact that WP get padding index (usually -100) which is not expected in torchcrf and also it go over all the tags, also the one you won't use like the not first WP of a token (token==space separated string)

Does anyone have a clean implementation of a BERTCRF? Preferably in a Jupyter notebook?

> You can directly feed the output of a transformer's hidden states into a CRF implementation like this https://github.com/s14t284/TorchCRF
> 
> Hope it helps!

@JetRunner  Is there a counterpart implement in Tensorflow?


> I don't think that this implementation is good. First, it doesn't take into account the fact that WP get padding index (usually -100) which is not expected in torchcrf and also it go over all the tags, also the one you won't use like the not first WP of a token (token==space separated string)

@shaked571 which implementation are you referring to? and what is WP?

@yuyan-z why the Decoder is necessary in the architecture illutrated in your graph?
Zero-shot pipelines: minor doc changes
Author: pcuenca
Pull request: yes

I was playing with the zero-shot image classification pipeline and did not immediately realize that the candidate labels would be formatted using `hypothesis_template`. Trying to make it more explicit and fix a typo (`cunjunction`).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

maybe @zucchini-nlp, @stevhliu, @Narsil 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33127). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Customise the separator used for splicing in DataCollatorWithFlattening
Author: beep-bebop
Pull request: yes

# What does this PR do?
#31629 added `DataCollatorWithFlattening`, which packs examples in a small batch into a long sequence and uses `-100` to splice the samples and returns `position ids` for attention calculation.
Since different models may use different token ids for splicing samples during training, for example, when using the Qwen model for post pre-training, short samples can be packed into long samples to speed up training and memory usage, and separated by `
`, which token id is `151643`. So allowing the user to customise the separator may be a more flexible implementation, allowing the user to use this DataCollator when building the pre-training dataset with different models.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
Models:
- text models: @ArthurZucker

Comments:
> LGTM, indeed this makes sense. can you just update the documentation of this datacolator please!

Updated!  Feel free to edit if needed:) @ArthurZucker 

Thanks 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix model name and copyright
Author: mayank31398
Pull request: yes

# What does this PR do?
This PR fixes model docs for Granite models added in this PR: https://github.com/huggingface/transformers/pull/31502

- text models: @ArthurZucker and @younesbelkada

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33152). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.