noamaanMulla-03/transformers-issues huggingface dataset:
Return attention mask in ASR pipeline to avoid warnings
Author: Rocketknight1
Pull request: yes

cc @sanchit-gandhi - this PR just sets `return_attention_mask=True` on the preprocessors in the `automatic_speech_recognition` pipeline to avoid warnings caused by missing attention masks. It works okay in my testing, but please let me know if you think it could cause any problems!

Fixes #33498

Comments:
cc: @ylacombe here

Hey @Rocketknight1, thanks for opening the PR!
In theory, I don't see any problems with this fix, have you been able to run every slow ASR pipeline tests here ? 

@ylacombe there are some slow tests that I can't get working on my local machine (even on `main`). However, of the tests that run, they all work with this PR as well!

Let me know if you want me to run them!

@ylacombe Sure! 

Okay, cool! cc @LysandreJik for core maintainer review.

Hi @Rocketknight1, 
I think `return_attention_mask` should also be added to `chunk_iter` function, so the warning will be removed even when we specify `chunk_length_s`.

> cc @sanchit-gandhi - this PR just sets `return_attention_mask=True` on the preprocessors in the `automatic_speech_recognition` pipeline to avoid warnings caused by missing attention masks. It works okay in my testing, but please let me know if you think it could cause any problems!
> 
> Fixes #33498

Hi, where can I set `return_attention_mask=True`?

I'm having the error fixed here using latest transformers==4.47.1:
```
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Whisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.
```

when running this code:
```
        pipe = pipeline(
            "automatic-speech-recognition",
            model=self.model,
            torch_dtype=torch.float16,
            chunk_length_s=30,
            batch_size=24,
            return_timestamps=True,
            device=self.device,
            tokenizer=processor.tokenizer,
            feature_extractor=processor.feature_extractor,
            model_kwargs={"use_flash_attention_2": True},
            generate_kwargs={
                "max_new_tokens": 128,
            },
        )
```
OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.
Author: kabyanil
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: Linux-6.8.0-49-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA GeForce RTX 3060


### Who can help?

@ylacombe @eustlb 

I am finetuning whisper-small on common corpus locally. When I run trainer.train(), I get the following error infinitely -

```
OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.
```

The exact same notebook works perfectly on Google Colab.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Please refer to [this page](https://huggingface.co/learn/audio-course/en/chapter5/fine-tuning) for the code.

### Expected behavior

The model should start fine-tuning.

Comments:
cc @eustlb, would you like to take a look at this issue?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Redundant Operations.
Author: haiduo
Pull request: no

``` 
sorted_logits, sorted_indices = torch.sort(scores_normalized, descending=True) 
min_thresh = sorted_logits[..., min_tokens_to_keep - 1] 
```
[This operation](https://github.com/huggingface/transformers/blob/6c3f168b36882f0beebaa9121eafa1928ba29633/src/transformers/generation/utils.py#L4736C5-L4737C60)

Please see the [issue](https://github.com/voidism/DoLa/issues/20).

Comments:
Hi @haiduo, if you think the code could be simplified or made faster, feel free to open a PR! cc @gante @zucchini-nlp for generation code

> Hi @haiduo, if you think the code could be simplified or made faster, feel free to open a PR! cc @gante @zucchini-nlp for generation code

Hi @Rocketknight1, I just submitted it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Idefics: fix docstring
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34628


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35079). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
for LlaVA output of image_hidden_states not possible?
Author: 7AtAri
Pull request: no
Labels: name: bug

name: Vision

### System Info

ubuntu, containerized development

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://stackoverflow.com/questions/78631255/how-to-extract-image-hidden-states-in-llavas-transformers-huggingface-impleme

apparently others than myself have already had this issue as well...

### Expected behavior

should output the image_hidden_states but returns a none type object without any content

Comments:
Hey @7AtAri ! Yes, unfortunately VLMs currently don't return `image_hidden_states`, and I had it in my TODO list. I might take longer to make a PR for that, so if you want to give it a try a PR is welcome :)

thanks! I used a forward hook meanwhile, but having a method is definitely much more convenient...

The model now returns hidden states for images after a forward pass (see linked PR 😉 ). But it will not work for generate as `generate()` usually doesn't return extra outputs from the model which are too model-specific

@7AtAri > forward hook

can you share your forward hook code? I WANT to get hidden states of llava with gradient.

I think the new hidden states feature should provide the hidden states with a gradient as well. @zucchini-nlp am I correct? It should also be more convenient to use.
        

@7AtAri Yes, it can provide hidden states with gradient as long as the vision tower is running the `forward` with gradients. The returned states are simply the outputs from vision tower + mm projector
Add explicit example for RAG chat templating
Author: A-Duss
Pull request: yes

# What does this PR do?

As discussed with @Rocketknight1 in #33421, this commit updates the `chat_templating` documentation to:
- Clarify that the documents input is not yet supported by most models.
- Provide an example with a model that do support it.
- Offer guidance on how to check if a model supports the `documents` feature.

Hoping to improve readers understanding of the feature.

Might be beneficial to update the Spanish #28936, Japanese #28301 , Korean #20179 , and Chinese #26803 corresponding docs, I did not risk myself as I do not speak those languages fluently enough. es, ja, ko, zh

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. 
@stevhliu @Rocketknight1


Comments:
Thanks @Rocketknight1 for the tweaking, seemed simpler and clearer to me, I committed your suggestion. 😄 
Should be ready to merge now. @stevhliu ? (not sure if I am supposed to @ you now, sorry if I am not)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, it's ready to merge, and don't worry, I can approve it, since it's just a doc change.

Thank you, and congratulations on your first PR!
import deepspeed error after update
Author: Murasame-sway
Pull request: no
Labels: name: bug

### System Info

transformers              4.46.1
deepspeed                 0.11.1

Ubuntu 20.04.6 LTS
python 3.10

### Who can help?

@muellerz 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers.deepspeed import is_deepspeed_zero3_enabled
```

error info: ModuleNotFoundError: No module named 'transformers.deepspeed' 

### Expected behavior

success import

Comments:
```
[rank0]:     trainer.train()
[rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 2122, in train
[rank0]:     return inner_training_loop(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
[rank0]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 3564, in training_step
[rank0]:     self.optimizer.train()
[rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/accelerate/optimizer.py", line 128, in train
[rank0]:     return self.optimizer.train()
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'DeepSpeedZeroOptimizer_Stage3' object has no attribute 'train'
```

confirm it

Directly import from `transformers.deepspeed` is deprocated. You can use `from transformers.integrations import deepspeed` instead.

from transformers.integrations.deepspeed import is_deepspeed_zero3_enabled

> ```
> [rank0]:     trainer.train()
> [rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 2122, in train
> [rank0]:     return inner_training_loop(
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
> [rank0]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
> [rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/transformers/trainer.py", line 3564, in training_step
> [rank0]:     self.optimizer.train()
> [rank0]:   File "/opt/anaconda3/envs/base/lib/python3.11/site-packages/accelerate/optimizer.py", line 128, in train
> [rank0]:     return self.optimizer.train()
> [rank0]:            ^^^^^^^^^^^^^^^^^^^^
> [rank0]: AttributeError: 'DeepSpeedZeroOptimizer_Stage3' object has no attribute 'train'
> ```
> 
> confirm it

did you figure out how to solve this ?
Add `position_ids` in `XLMRobertaXLForCausalLM.prepare_inputs_for_generation`
Author: ydshieh
Pull request: yes

# What does this PR do?

The model component `XLMRobertaXLEmbeddings` will perform

> position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)

if `position_ids` is not passed to the model (say `XLMRobertaXLForCausalLM`).

However, during the decoding, even if `past_key_values_length` is passed, the `create_position_ids_from_input_ids` will compute the wrong `position_ids` when the `input_ids` is only the single last token. This is because `create_position_ids_from_input_ids` relies the whole  `input_ids` in order to get the number of padding tokens:

>     mask = input_ids.ne(padding_idx).int()

(since `past_key_values_length` doesn't contain this information.)

In general, we need to prepare `position_ids` in `prepare_inputs_for_generation`, like what has been done in

> GenerationMixin.prepare_inputs_for_generation

This PR adds this for `XLMRobertaXLForCausalLM` (whose `prepare_inputs_for_generation` is already overwritten from the parent class)

---------------------------------------------------------------------------------------------------------------------------------

As a bonus:

> tests/models/xlm_roberta_xl/test_modeling_xlm_roberta_xl.py::XLMRobertaXLModelTest::test_assisted_decoding_matches_greedy_search_1_same

is also no longer flaky (running 10000 times) while it was failing 5% of the time before this PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35044). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: Handle BLIP-2 model output format
Author: thisisiron
Pull request: yes

# What does this PR do?

I have modified the code to return logits in addition to loss.

Instead of:

```
return {"loss": loss}
```

The code now returns:
```
return {
    "loss": loss,
    "logits": logits
}
```

Output:
```
Done!
First values of original logits: tensor([[ 1.9322,  1.9379,  7.4007],
        [-1.4741, -1.1189,  8.6593],
        [-1.4213, -1.2489,  6.1976]])
First values of HF logits: tensor([[ 1.9322,  1.9379,  7.4007],
        [-1.4741, -1.1189,  8.6593],
        [-1.4213, -1.2489,  6.1976]])
Looks ok!
Generating a caption...
Original generation: ['merlion']
HF generation: ['Question: what object is in this image? Answer: merlion']
```


Fixes # ([issue](https://github.com/huggingface/transformers/issues/34704#issue-2652956392))


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts, @qubvel


Comments:
In case you have bandwidth to review the conversion script fix @NielsRogge 

gentle ping @NielsRogge
Bug of self.accelerator.gather(num_items_in_batch) with enabling average_tokens_across_devices 
Author: Snowdar
Pull request: no

https://github.com/huggingface/transformers/blob/329f5dbf97a5cb2473914c88c05aa3dcb242e19a/src/transformers/trainer.py#L5146

The `num_items_in_batch` could be None for the `break` command
```
for _ in range(num_batches):
            try:
                batch_samples += [next(epoch_iterator)]
            except StopIteration:
                break
```
then it will raise this error in the self.accelerator.gather(num_items_in_batch):
```
TypeError: Unsupported types (<class 'NoneType'>) passed to `_gpu_gather_one`. Only nested list/tuple/dicts of objects that are valid for `is_torch_tensor` should be passed.
```

A condition `if self.args.average_tokens_across_devices and num_items_in_batch is not None:` should be added to process this problem. I don't intend to submit a separate PR for this issue. Can someone fix it along the way?

Comments:
cc @muellerzr @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add check for if num_items_in_batch is not None
Author: muellerzr
Pull request: yes

# What does this PR do?

Adds a check before calling `.gather()` to make sure that `num_items_in_batch` actually has a value

Fixes https://github.com/huggingface/transformers/issues/35076


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1 @SunMarc  We encountered the same issue. When will this MR be merged? Thank you！

I'll let @muellerzr decide either to merge this one or this PR : https://github.com/huggingface/transformers/pull/35399

Thanks! This one got swept under (not good)
`GPT2Attention()` class with `_attn()` method when `add_cross_attention=True` and therefore `is_cross_attention=True`.
Author: CHLEE-Leo
Pull request: no
Labels: name: Feature request

name: bug

### Feature request

### Model description

It seems like `GPT2Attention()` class allows`_attn()` method with `causal_mask` only when `is_cross_attention=False`, but not when `is_cross_attention=True`. 

It would be more productive if `GPT2Attention()` supports `_attn()` method with `causal_mask` even with `is_cross_attention=True`.

### Motivation

When developing `EncoderDecoderModel` where the encoder is `ViTModel` and the decoder is`GPT2Model`, the current `GPT2Model` class does not support `causal_mask` if `add_cross_attention=True` and therefore `is_cross_attention=True`.

### Your contribution

No contribution.

Comments:
cc @Cyrilvallez 

Hey @CHLEE-Leo! When using cross-attention, the mask in use is `encoder_attention_mask`. You can therefore pass whatever mask you wish with this argument (in your case a simple causal mask). I think this should solve your problem 🤗
Handle `None` before gathering num items in batch across devices
Author: chiragjn
Pull request: yes

# What does this PR do?

Fixes #35387


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr




Comments:
I am okay with either, please let's get a fix merged 😅 

Closing this as the other PR got merged
Fixed typo in Llama configuration docstring
Author: sudarshan-mukund
Pull request: yes

There is no `num_heads` parameter, only `num_attention_heads`

# What does this PR do?

Fixes docstring in llama configuration


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

- text models: @ArthurZucker 
- Documentation: @stevhliu 


Comments:
[Dinov2 with Registers] Some fixes
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes some things mentioned in #35348. cc @xenova 

To do:

- [x] not sure we can merge the update to interpolate_pos_encoding as the hidden states are not equivalent


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35411). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Remove duplicates in toctree
Author: jungnerd
Pull request: yes

# What does this PR do?
`ko/_toctree.yml` had duplicate entries for Quatization Methods, causing confusion during navigation. 
The redundant entries have been removed to ensure clarity and consistency.

Part of https://github.com/huggingface/transformers/issues/20179


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu, could you please review this PR when you have time? Thank you in advance! 😊



Comments: