noamaanMulla-03/transformers-issues huggingface dataset:
Whisper: fix asr pipeline with seq2seq assistant model
Author: gante
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes #29869 
Fixes #30407
Fixes #30611 (related PR: #30637)

The ASR pipeline was preparing the encoder outputs before `generate` (and not passing `input_features`), but that's not needed: the exact same preparation is done inside `generate`, as it is a hard requirement to generate with encoder-decoder models.

However, by not passing `input_features`, it was blocking the proper use of encoder-decoder assistants that used a different `encoder` output shape (= decoder input shape), and thus needed the inputs to run their own encoding step.

Contrarily to #30637, this fix relies on lowering the complexity of our codebase 👼 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30726). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(diff included in #30637 )
Add AutoFeatureExtractor support to Wav2Vec2ProcessorWithLM
Author: ylacombe
Pull request: yes

# What does this PR do?

Using a N-GRAM based language-model on top of Wav2Vec2-based models is an easy way to get a performance boost. At the moment, Wav2Vec2ProcessorWithLM was only compatible with Wav2Vec2FeatureExtractor.

W2V2-Bert could also benefit from this boost, but need its feature extractor to also be compatible with Wav2Vec2ProcessorWithLM.

The easiest way to do it is to add AutoFeatureExtractor instead of Wav2Vec2FeatureExtractor in the code, since the processor only changes the tokenizer behaviour.


cc @sanchit-gandhi @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28706). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry for the long wait, I've added the final test you asked @amyeroberts ! Still good to merge (once all's green) ?
Add Wav2Vec2BertProcessorWithLM
Author: FredHaa
Pull request: no
Labels: name: Feature request

name: Audio

### Feature request

Wav2Vec2-Bert was open sourced and integrated with Transformers in the end of last year. However, it is missing an easy integration with pyctcdecode similar to Wav2Vec2ProcessorWithLM. This should be quite trivial to implement, since Wav2Vec2Processor is very similar to Wav2Vec2BertProcessor, the only difference being that they use different feature extractors.

### Motivation

Having a Wav2Vec2BertProcessorWithLM class would make it possible to use Wav2Vec2-Bert with a kenlm model in a Transformers ASR pipeline.

### Your contribution

I can submit a PR.

Comments:
cc @sanchit-gandhi @ylacombe 

Hey @FredHaa, #28706 should fix this, I'm reopening it! Note that you would have to use `Wav2Vec2ProcessorWithLM` and not `Wav2Vec2BertProcessorWithLM`!

#28706 has been merged, I'm closing the issue for now, feel free to ask questions
Add TokenClassification support for modern LLMs
Author: josephenguehard
Pull request: no
Labels: name: Feature request

### Feature request

Implement TokenClassification for Llama, Mistral and similar models.

### Motivation

A few of the recently developed LLMs (e.g. Llama, Mistral) do not support the TokenClassification task, which is used for instance for Named Entity Recognition (NER) and Part-of-Speech (PoS) tagging.

It would be beneficial for the `transformers` library to have this task implemented for these models.

### Your contribution

Implement the following classes:
- `LlamaForTokenClassification`
- `MistralForTokenClassification`
- Other models which are similar to Llama and Mistral, where the code can simply be copied from either of these models.

Comments:
Thank you @josephenguehard and I've already released two TokenClassification models, [Swallow-7b-plus-upos](https://huggingface.co/KoichiYasuoka/Swallow-7b-plus-upos) and [Swallow-MS-7b-upos](https://huggingface.co/KoichiYasuoka/Swallow-MS-7b-upos), in order to compare LLM POS-tagging methods between two papers:

-  https://aclanthology.org/2023.acl-long.367
-  https://arxiv.org/abs/2310.01208

One of these methods needs CausalLM and the other needs TokenClassification...

In [Swallow-7b-plus-upos](https://huggingface.co/KoichiYasuoka/Swallow-7b-plus-upos) I included Dropout layer in the decoder. So as [Swallow-MS-7b-upos](https://huggingface.co/KoichiYasuoka/Swallow-MS-7b-upos).

PR merged
`tokenizer_class = "AutoTokenizer"` Llava Family
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Propagate the changes to more Llava framework based processors

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Introduce configured_state arg for accelerator_config
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR starts to allow users to bring in their own `Accelerator`. To start, we are simply allowing users to define a `PartialState` or `AcceleratorState` outside the `TrainingArguments`, and then enable the user to use its state using a new `use_configured_state` arg.

For instance, a user can now do:

```python
from accelerate import PartialState
from transformers import TrainingArguments

state = PartialState()
args = TrainingArguments(accelerator_config={"use_configured_state":True})
```

And this will use the defined `state` already made.

These `states` are Singeltons, so defining it once and calling it will maintain the same state on subsequent calls. 

This *may* lead to issues with hyperparameter tuning, which requires the state to be reset each time, as noted in the doc description

Fixes related accelerate issue which occurs when defining an `Accelerator` before the `TrainingArguments`: https://github.com/huggingface/accelerate/issues/2564


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @pacman100 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29781). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts 

@amyeroberts requesting a rereview after having to gut much of it on further testing 😅 

@pacman100 Can you do a re-review?
Fix custom architectures
Author: not-lain
Pull request: no
Labels: name: Feature request

name: WIP

opening this issue for better visibility and to keep track to what needs to be fixed, any contributions are welcome.

| name | issue  | pull request | comment |
| ------------- | ------------- | ------------- | ------------- |
| dependency issue when working with a custom architecture in a repo that has a dot in its name| #28919  | #29175  |**fixed** ✅ |
| wrongly annotated configuration when saving a model that has a custom pipeline| #28907   | #29004  |  awaiting review |
| add `push_to_hub( )` method when working with pipelines| #28857   | #29172 | awaiting review |

Comments:
@Rocketknight1 this might help with keeping track to everything.
I'm going to start working on the 2nd issue above hope I fix it soon. 

@amyeroberts just for clarification. 
* the second issue is all about when a repo has a custom pipeline and then (let's say for example you finetune the model and push the new model to another repo) the custom pipeline breaks down when loaded from the final repo.
* the `push_to_hub` method to pipelines is self explanatory, I have reread [this](https://github.com/huggingface/transformers/pull/29172#issuecomment-2010055957) for some time now, and I think I am beginning to understand where they're coming from coz the pipelines library is built on top of a model, and again I agree that that should be put for another day since it is irrelevant for now, allowing people to easily push their pipelines takes priority for now.

apologies for failing to notice most of the stuff (i made some some wrong assumptions about the `{repo_id}--{module}` tag and it mislead both me and the maintainers at the beginning), so i wanted to give you a clarified update for what i came out with hope this helps you out


all done 🥳
Add support for torch.compile dynamic shapes
Author: warner-benjamin
Pull request: yes

This PR adds support for compiling models with dynamic shapes `dynamic=True` to almost all models with SDPAttention implementations which currently do not support dynamic shapes. #30442 added support for Llama, Gemma, OLMo, & Cohere. 

The only model not modified is DBRX, which needs the changes from both #30070 and #30442 to add support for SDPA's Flash Attention kernel and support for dynamic shapes, as it I believe it suffers from the same training memory issues detailed in #30010.

As mentioned in #30442, moving the `is_causal` dispatch logic from inline to an if statement is required to support both `fullgraph=True` and `dynamic=True`.

I kept the `qlen>1` comments but could remove them if we want to match Llama, which doesn't have it.

cc @ArthurZucker and @fxmarty

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30560). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fxmarty @ArthurZucker I didn't touch Llava. The current "Tensor-likes are not close" error shouldn't have anything to do with this PR. It should be ready to go from my end.

no worries, rebasing on main should most probably fix this!
separate kwargs in processor (similar to #30193)
Author: Eric2i
Pull request: yes

# What does this PR do?

Similar to #30193, this PR separates out the kwargs that are passed to the git processor into tokenizer and image processor kwargs.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30905). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**More Context**

When running the following piece of code: 

`processor = AutoProcessor.from_pretrained("microsoft/git-base")`
`encoding = processor(images=Image.open("Cat.jpg"), text="A cute cat.", padding="max_length")`

I observed a similar warning message ("unused or unrecognized kwargs: padding") in [issue#30106](https://github.com/huggingface/transformers/issues/30106), which is nicely fixed by [PR#30905](https://github.com/huggingface/transformers/pull/30193).

So, I tried to fix the same problem for this processing_git.py file and now it works without the warning message.

**Other Thought**

This looks like a common problem in multiple processors. Probably similar updates are also needed for other ones. 

Thanks!






@Eric2i Great thanks for the details! There's actually a big PR which will be merged in soon which should address this (and many other difficulties with processor kwargs!) #30511
How to get the data seen by the model during training?
Author: jaydeepborkar
Pull request: no

Hi! I haven't been able to find an answer to my question so opening an issue here. I'm fine-tuning the GPT-2 XL model using the trainer for 10 epochs and I'd like to save the data seen by the model during each epoch. More specifically, I want to save the data seen by the model every 242 steps. For instance, data seen from step 1 to step 242, step 243 to step 484, and so on until the end of the 10th epoch. I'm a bit confused about how to do this since the data is shuffled after each epoch. Is it possible to use `TrainerCallback` here?

These are my training args
` training_args = TrainingArguments(
        f"models/XL",
        evaluation_strategy = "steps",
        learning_rate=2e-5,
        weight_decay=0.01,
        push_to_hub=False,
        num_train_epochs=10,
        per_device_train_batch_size=8, 
        per_device_eval_batch_size=8, 
        save_strategy="epoch", 
        save_steps = 242, 
        fp16=True, 
        report_to="none", 
        logging_strategy="steps",
        logging_steps=100, 
    )`
    
 I'd appreciate any directions. Thanks :) 

Comments:
Hi @jaydeepborkar, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Sure, thanks!
FIX / TST: Fix expected results on Mistral slow test (A10)
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, see: https://github.com/huggingface/transformers/actions/runs/9155572295/job/25168224905#step:11:55 for more details

cc @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30909). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: LOMO support
Author: KaiLv69
Pull request: no
Labels: name: Feature request

### Feature request

Support for the LOMO training strategy, which allows full-weights training of 7B models on consumer 24GB cards like RTX 4090. 

ArXiv: https://arxiv.org/pdf/2306.09782.pdf https://arxiv.org/pdf/2310.10195.pdf
Implementation: https://github.com/OpenLMLab/LOMO (An installable package `lomo-optim` is also available here.)

### Motivation

This is another step towards democratizing LLM. And the variant AdaLomo performs on par with AdamW.

### Your contribution

I'm willing to submit a PR and modify the `lomo-optim` package if needed.
I may need some kind help for the integration on what I should do.

Comments:
cc @younesbelkada 

Nice and exciting work ! Happy to get to it after the work on https://github.com/huggingface/transformers/pull/29588 !
PaliGemma - fix processor with no input text
Author: hiyouga
Pull request: yes

# What does this PR do?

Fix the paligemma processor raises exception on empty text inputs

## Reproduce

```python
from PIL import Image
from transformers import AutoProcessor

image = Image.new("RGB", (100, 100), (255, 255, 255))
processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224")
processor(images=image)
```

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/python3.10/site-packages/transformers/models/paligemma/processing_paligemma.py", line 190, in __call__
    pass
TypeError: 'NoneType' object is not iterable
```

The following lines require `text` should be a **list**, we need to convert the Nonetype input to a string or list firstly.

https://github.com/huggingface/transformers/blob/92d1d97c05a01160d6e7fcf4198e93bf2cec0dfe/src/transformers/models/paligemma/processing_paligemma.py#L176-L179

https://github.com/huggingface/transformers/blob/92d1d97c05a01160d6e7fcf4198e93bf2cec0dfe/src/transformers/models/paligemma/processing_paligemma.py#L186-L198

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts 

Comments:
cc @molbap 

Hi @hiyouga , thanks for the fix! LGTM

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30916). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
VisEncoderDecoderModel generate text incomplete when predict image with long text label
Author: hideinhat
Pull request: no

Hi, I trained a VisionEncoderDecoderModel to regconize math expression LaTeX image.
when I predict image with long math expression label , (the length of label text is not over max_length), the result of model generation is incomplete.
label: 
`"= \left( \frac { 9 } { 4 } \right) ^ { \frac { 1 } { 2 } } - \left( \frac { 3 } { 4 } \right) ^ { 2 } + \left( \frac { 1 } { 6 } \right) ^ { 2 } \times \left( \frac { 2 7 } { 8 } \right) ^ { \frac { 2 } { 3 } }" ` (210 words include space)
model pred:
`"= { { \left( \frac { 9 } { 4 } \right) } ^ { \frac { 1 } { 2 } } } - { { \left( \frac { 3 } { 4 } \right) } ^ { 2 } } + { { \left( \frac"` (136 words include space)

this is a part of processing dataset labels
```
encoder_checkpoint = 'google/vit-base-patch16-224-in21k'
decoder_checkpoint = 'witiko/mathberta'

feature_extractor = ViTFeatureExtractor.from_pretrained(encoder_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(decoder_checkpoint)
processor = TrOCRProcessor(image_processor=feature_extractor,tokenizer=tokenizer)
. . 
self.max_target_length = 256
. . .
labels = self.processor.tokenizer(text,padding="max_length",max_length=self.max_target_length,truncation=True).input_ids
```
this is the paramaters of beam search
```
# set beam search paramaters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 256
model.config.early_stopping = True
model.config.no_repeat_ngram = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4
```

How can I fix this?




Comments:
Hi @hideinhat, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.
FEAT / Trainer: LOMO optimizer support
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/29649

As requested by the community, this PR integrates LOMO optimizers into HF Trainer

https://github.com/OpenLMLab/LOMO 

I am facing some issues with respect to AdaLOMO, which seems to have DS as a hard requirement as in the optimizers init: https://github.com/OpenLMLab/LOMO/blob/85d8105c48cbd676dbf6915ee755461cd241da9b/lomo_optim/adalomo.py#L85 - so leaving this as a draft for now until I try to figure out this issue

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30178). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @muellerzr would love to have a first review! 🙏 

thanks @muellerzr ! I offloaded most of the logic in https://github.com/huggingface/accelerate/pull/2695 - wdyt ? 🙏 

Thanks so much for the extensive review and help @amyeroberts @muellerzr !
CI:  AMD MI300 tests fix
Author: mht-sharma
Pull request: yes

# What does this PR do?

Fixes the failing tests on MI300.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ydshieh and @younesbelkada (as I think I remember you had handle something similar)

will check but likely not today

@mht-sharma Do you have permission to merge? If not, I can merge in for you

> @mht-sharma Do you have permission to merge? If not, I can merge in for you

@amyeroberts I do not have the permissions. Please merge. Thanks for the review 🤗
Enforce saving at end of training if saving option chosen
Author: muellerzr
Pull request: yes
Labels: name: trainer

# What does this PR do?

Currently we save at the end of training if an `epoch` strategy is chosen, but not if a `steps` strategy is chosen. This can be undesireable because the end of a users training might not be saved at all, sometimes upwards of 10% as seen here: https://github.com/huggingface/transformers/issues/28539

This PR mimics the behavior of `epoch`, by choosing to save the model at the end of training as well if `steps` is chosen.

I do not believe it makes sense to make this configurable, because a user should *always* have access to the end-resulting model when training!


Fixes https://github.com/huggingface/transformers/issues/28539


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @pacman100 

Comments:
We can certainly make this configurable as an arg, but I'm not sure it **makes sense** to do this

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts good for a rereview, no functional changes happened just switched out the test logic to be more obvious what's going on