noamaanMulla-03/transformers-issues huggingface dataset:
[pipeline] fix padding for 1-d tensors
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Currently on main, batched inference using the ASR pipeline fails:

```python
from transformers import pipeline, AutoProcessor, WhisperForConditionalGeneration
from transformers.utils import is_accelerate_available
from datasets import load_dataset

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", low_cpu_mem_usage=is_accelerate_available())

pipe = pipeline("automatic-speech-recognition", model=model, feature_extractor=processor.feature_extractor, tokenizer=processor.tokenizer)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[:2]["audio"]

pipe(sample, batch_size=2)
```

<details>

<summary> Traceback </summary>

```python
  File "/home/sanchit/transformers/src/transformers/pipelines/base.py", line 194, in inner                              
    padded[key] = _pad(items, key, _padding_value, padding_side)                                                        
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                        
  File "/home/sanchit/transformers/src/transformers/pipelines/base.py", line 100, in _pad                               
    max_length = max(item[key].shape[1] for item in items)                                                              
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                              
  File "/home/sanchit/transformers/src/transformers/pipelines/base.py", line 100, in <genexpr>                          
    max_length = max(item[key].shape[1] for item in items)                                                              
                     ~~~~~~~~~~~~~~~^^^                                                                                 
IndexError: tuple index out of range    
```

</details>

This is because the pipeline class attempts to pad the 1-d tensor of `num_frames`, which we added in https://github.com/huggingface/transformers/pull/30637 to correctly compute word-level timestamps:

https://github.com/huggingface/transformers/blob/048f599f3506e57e0a595b455d9d2834c8d45023/src/transformers/pipelines/automatic_speech_recognition.py#L452

The simple fix is to handle padding of 1-d tensors explicitly in the private `_pad` method, which we implement here. We also add a slow test to confirm batched generation works following the fix.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31776). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Gently pinging @kamilakesbi - if you could do a first review that would be grand!
Make fx traced model with the use of `past_key_values` pickable again?
Author: michaelbenayoun
Pull request: no
Labels: name: Compilation

With the support of the `transformers.cache_utils.Cache` classes when doing symbolic trace (#30259), it is not possible to pickle a traced model which uses `past_key_values`, should we work to add support for that?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @ArthurZucker @gante 

Created this issue to have the discussion but I do not know if we actually want to support that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@michaelbenayoun sorry, this was lost in the notification queue 😬 

I think the PT team is taking care of what you asked in this PR: #32253
Add stream messages from agent run for gradio chatbot
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Add a new method to stream messages from a transformers agent's `run()` method for the `gradio.Chatbot` component to use.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32142). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
use torch 2.4 in 2 CI jobs
Author: ydshieh
Pull request: yes

# What does this PR do?

use torch 2.4 in 2 CI jobs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs: fix GaLore optimizer code example
Author: giladturok
Pull request: yes

# What does this PR do?
Fix incorrect usage of GaLore optimizer in Developers Guide Trainer code example.

The GaLore optimizer uses low-rank gradient updates to reduce memory usage and is quite popular. GaLore is implemented by the authors in [https://github.com/jiaweizzhao/GaLore](https://github.com/jiaweizzhao/GaLore) and the ArXiv paper is [here]( https://arxiv.org/abs/2403.03507).

A few months ago GaLore was added to the HuggingFace Transformers library in https://github.com/huggingface/transformers/pull/29588. Documentation in the Developers Guide of the Trainer module includes a few code examples of how to use GaLore [here](https://huggingface.co/docs/transformers/main/en/trainer#galore). However, the `optim_targe_modules` argument to the `TrainingArguments` function is incorrect, as discussed in https://github.com/huggingface/transformers/pull/29588#issuecomment-2006289512.

This pull request simply fixes this issue by replacing all instances of  `optim_target_modules=["attn", "mlp"]`
with  `optim_target_modules=[r".*.attn.*", r".*.mlp.*"]`.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32249). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add size argument to GroundingDinoProcessor call (pass it to GroundingDinoImageProcessor)
Author: saskiabosma
Pull request: no
Labels: name: Feature request

### Feature request

When creating a GroundingDinoProcessor object, it is currently not possible to pass a size to which the image processor would resize the image before passing it on.  Since GroundingDinoProcessor passes the images to GroundingDinoImageProcessor, which itself allows a "do_resize" and "size" argument, it would be a simple change to allow custom resizing.


### Motivation

This feature is interesting to have when performing inferences on images that are small, since the current default resizing is putting the shortest edge at 800px or the longest edge at 1333px, and inference speed depends strongly on image size.  It also helps with GPU memory usage.

I saw a 60% speed difference for inference on one image when manually forcing sizes around 400px. I also went from GPU OOM errors when using a batch of 2 ~400px pictures to no issues for batches of > 20 pictures.

### Your contribution

I'm willing to do the PR if the maintainers think this is a good change! 

Comments:
Hey! If I understood correctly, you want to pass image processing related arguments when doing `processor(text, images)`. We are working on it and standardizing kwargs for processors. 

GroundingDINO PR is here (https://github.com/huggingface/transformers/pull/31964) to keep track of the progress :)

Yes, this PR would solve my issue!
Cast epochs_trained to int when resuming training
Author: teddy-f-47
Pull request: yes

# What does this PR do?

Ensures that `epochs_trained` is an integer when resuming training, so that the loop `for epoch in range(epochs_trained, num_train_epochs):` in `_inner_training_loop` can proceed correctly.

Fixes #27630


Comments:
 @muellerzr @SunMarc @LysandreJik

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32286). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Unfortunately I don't have write access to the repository, so one of you will have to merge the PR :)

:rocket:
feat(ci): set `fetch-depth: 0` in trufflehog checkout step
Author: McPatate
Pull request: yes

### What does this PR do?

Setting `fetch-depth: 0` in trufflehog CI, we were prematurely optimising I believe.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31663). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix M4T for ASR pipeline
Author: ylacombe
Pull request: yes

# What does this PR do?


#32277 rightfully highlighted an issue when using the M4T model with the ASR pipeline.

The M4T model is tested here in the ASR pipeline test, but it hasn't been caught before. I've managed to pinpoint the bug from the fact we're passing explicitely passing `inputs` in the `.generate` in the pipeline:
https://github.com/huggingface/transformers/blob/7f5d644e69068825bb5b6e84cdc56b3d3a9bd04f/src/transformers/pipelines/automatic_speech_recognition.py#L504-L507

This was introduced in the recent https://github.com/huggingface/transformers/pull/30637 but not caught before.


I've proposed a solution that only impacts the M4T modeling codes, but I'm not quite sure about it. It's a dirty easy fix and all tests pass in the M4T modeling tests and in the M4T pipeline test, but we should probably look at fixing tthis directly in the ASR pipeline code.

cc @LysandreJik @sanchit-gandhi and @kamilakesbi, what do you think?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32296). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs: formatting nits
Author: gante
Pull request: yes

# What does this PR do?

I noticed that we had poorly closed `type` blocks on some cache classes docstrings, so I quickly created a regex to catch all cases. Turns out that poor formatting was more common than I thought 😅 I've manually fixed all caught cases.

Regex for which we don't want matches: 
```
/(?!.*\*optional\*.*)(?!.*shape.*)(\(`.*[^`]\):)/g
```

In other words, we:
1. don't want unclosed docstring type code blocks, i.e. starting with (\` but not ending in \`):
2. EXCEPTION: doesn't apply if the docstring has \*optional\*, the identifier for optional arguments, as it may not end in \') -- e.g. (\`float\`, \*optional\*, defaults to 0.02):
3. EXCEPTION: doesn't apply for tensors where we specify the shapes, as we are not escaping the shapes atm -- e.g. (\`torch.tensor\` of shape (foo, bar)):

### Questions / notes for follow-up changes 🤔 

1. Question: We have a `check_docstrings.py` command as part of `make fixup`. Would it make sense to move some of the checks to `doc-builder`, so other libraries can re-use them? 🤔 
2. Note: the docstring check is only checking the class docstring, but not the documented methods of the class
3. NoteNeedless to say, undocumented classes/methods don't have any checks



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32247). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts @ArthurZucker as core maintainers

I spend a lot of time linking to individual parameters in the generation classes while answering issues, which led to this PR and also https://github.com/huggingface/doc-builder/pull/516 (regex is life)

Given that we are exploring the docs bot, I'm also assuming standardizing docstrings will make the bot more performant 🤗
Alternative agent plan
Author: plaggy
Pull request: yes

Proposes an alternative plan structure for agents. Loosely based on the [ReWOO paper](https://arxiv.org/abs/2305.18323#).
Improved a score on the GAIA for me from 40 to 41.8

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32295). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Just one comment to solve about renaming variables for clarity, then lgtm!
fix: Added missing raise keyword for few exceptions
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Fixes few exceptions that are not raised.


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@LysandreJik @amyeroberts 

Comments:
MLM the rest 10% of tokens: pytorch 0.5 vs. tf 0.1
Author: fkrasnov2
Pull request: no

Hey there,
I've been wondering about something for a while. Can you help me out? I'm confused about why there's a difference between 0.5 in one place and 0.1 in another. What's the deal with that?

##Lines 852-858
```
       # 10% of the time, we replace masked input tokens with random word
       indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
       random_words = torch.randint(len(self.tokenizer), labels.shape, dtype=torch.long)
       
       inputs[indices_random] = random_words[indices_random]
       # The rest of the time (10% of the time) we keep the masked input tokens unchanged
```

##Lines 753-760 
```
       # 10% of the time, we replace masked input tokens with random word
       indices_random = self.tf_bernoulli(input_shape, 0.1) & masked_indices & ~indices_replaced
       random_words = tf.random.uniform(input_shape, maxval=vocab_size, dtype=inputs.dtype)

       inputs = tf.where(indices_random, random_words, inputs)
       # The rest of the time (10% of the time) we keep the masked input tokens unchanged

```
https://github.com/huggingface/transformers/blob/084b5094eb490319719cc11cb05b751e0b419d49/src/transformers/data/data_collator.py#L754

https://github.com/huggingface/transformers/blob/084b5094eb490319719cc11cb05b751e0b419d49/src/transformers/data/data_collator.py#L852

Best, 
Ted

Comments:
Probably a mistake, cc @Rocketknight1 

Yes, this is a mistake - thank you for spotting it! The Torch code is correct - to get an average of 10% of the total, we want to take 50% of the remainder after we've already masked 80% with `[MASK]` in the previous step.

@fkrasnov2 Would you be willing to open a PR to fix the TF code?

@Rocketknight1 , Sure, I'll do a PR.

@fkrasnov2 merged - thank you for the PR!
fixes #32329 : The Torch code is correct - to get an average of 10% o…
Author: fkrasnov2
Pull request: yes

# What does this PR do?

Fixes # (32329)

https://github.com/huggingface/transformers/issues/32329

## Before submitting
- [*] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [*] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [*] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.

## Who can review?

@Rocketknight1 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32335). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Repo checks: skip docstring checks if not in the diff
Author: gante
Pull request: yes

# What does this PR do?

See title. A flag was added to optionally check all files.

Timed examples:
- No `src` files in the diff: ~30s -> ~3s
- Two `src` files in the diff: ~30s -> ~7s

Notes:
- After these changes, the majority of the time is spent on imports.
- ⚠️ Because we now exclusively check on the diff, complex patterns (e.g. a docstring built from an f-string using a variable from a different file) may deceive our checks. Added the ability to do a full run to `make quality`, so we can check on it once in a while.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32328). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLaVA-NeXT: fix anyres shapes
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31588, fixes #31529, fixes #31327. This PR switches height and width after getting anyres-shapes  of the image in llava-next models. It was pointed out by users several times that this is inconsistent with the original impl since the original uses a `width, height` shape in processing while we don't

Also verified by running the original implementation that the shapes were switched on out end

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32314). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks, I tried changing everywhere from (height, width) in the original implementation to (width, height) in our implementation since that's a convention we also adopt in https://github.com/huggingface/transformers/blob/main/src/transformers/image_utils.py.

I don't think this is correct. The convention in the library is to use `(height, width)` and this should be kept consistent
Gemma2 and flash-attention
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31953. As explained in the comments section, the issue with flash attn was static cache, as we were not preparing special causal masks with padding on both sides, for pad-token and for cache placeholders.

I tried to pass-in a 2D mask with pads for cache positions, yet it didn't work out because when unpadding for flash attention we fall into the path when query length is not same as key length, and remain with only the leftmost 1 query token in first forward.

So I think we can use a similar approach to the one discussed in one prev PR to cut out memory usage, cut out key/values to remove extra padding on the right

Plus, there were cache related issue on Gemma2, I will make another PR on that



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32188). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

It works, thanks for that, but as far as I can see the vram usage is the same, and the speed is a bit slower than without attention. There might still be something broken, in sense that it does not actually use the flash_attention_2 at all.

For phi 1.5 vram usage is drastically reduced for the same data by using flash attention 2.


    base_model = AutoModelForCausalLM.from_pretrained(
        base_model_id, 
        quantization_config=bnb_config, 
        device_map={"": 0},
        trust_remote_code=True,
        attn_implementation="flash_attention_2",
    )

     with torch.no_grad():
                outputs = ft_model.generate(**model_inputs, max_new_tokens=60, do_sample=True, top_k=100, top_p=1.0,
                        pad_token_id=tokenizer.pad_token_id,
                        use_cache=True,
                        )



@Oxi84 hmm, indeed Gemma2 doesn't benefit much from Flash-attention compared to other models. Seems like it's related to teh big vocab size it has (256000 tokens) which we have to store all in fp32 dtype for sampling. In pre-fill stage that is a tensor of shape `(bs, 256000, dim)` and needs a huge memory for long sequences. 

We can hugely benefit from https://github.com/huggingface/transformers/pull/31292  in Gemma2 after it's merged 

Will be merging today at night
update based on tokenizers release
Author: ArthurZucker
Pull request: yes

# What does this PR do?
A small improvement, but overall allows us to add tokens special and not special at the same time for fast tokenizers

Comments:
This needs a tokenizers update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30574). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Will prepare this again

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.