noamaanMulla-03/transformers-issues huggingface dataset:
Add support for OmDet-Turbo multi-gpu inference with DataParallel.
Author: yonigozlan
Pull request: no
Labels: name: New model

name: Distributed Training / Models

name: Feature request

### Feature request

OmDet-Turbo will be added to Transformers soon, however it won't support using DataParallel for inference using multi-gpu, at least initially. 

### Motivation

If there is a large demand to support multi-gpu inference with DataParallel.

### Your contribution

A PR will be created if there is demand for it.

Comments:
support schedule sampling
Author: ezeli
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request

To solve the exposure bias problem, can schedule sampling be supported?
https://arxiv.org/abs/1906.07651
https://github.com/deep-spin/scheduled-sampling-transformers

### Motivation

For the same data, the training loss is very low but the inference result is not good.

### Your contribution

no

Comments:
Gemma2: fix config initialization (`cache_implementation`)
Author: gante
Pull request: yes

# What does this PR do?

Fixes failing tests like `test_greedy_generate_dict_outputs_use_cache`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix flash attention comment
Author: nbroad1881
Pull request: yes

@ArthurZucker, @gante 

I think this fix makes the comment correct, given the `view` calls in the following lines. If this is the case, I can copy these changes to all flash attention functions.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33394). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
VLM generate: tests can't generate image/video tokens
Author: gante
Pull request: yes

# What does this PR do?

Our VLM generate mixin tests, introduced in #33533, are flaky. Because they use randomly initialized models, nothing prevents the models from generating image/video tokens, which a) shouldn't happen b) crash the forward pass.

This PR ensure our generation tests don't generate those tokens.

Commands ran to ensure the issue is fixed:
✅ (the test is no longer flaky) `py.test tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_sample_generate_dict_output --flake-finder --flake-runs=1000`
✅ (we can run generation tests on all models after these changes) `py.test tests/models/ -k test_sample_generate_dict_output` 


cc @zucchini-nlp -- when you're back from holidays, plz review even if it is already merged, in case you want to change things 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh we do, `llava_onevision` fails without it :o (because the token in question is outside than the vocab size in the test settings)

@gante thanks so much for handling this in my absence ❤️ I saw your other comment, and yes, if we can make a default generation config for models, then we'll be able to block certain tokens. For that I think we should also allow passing/overwriting generation params when loading a config fsame way we can overwrite some ModelConfig params


> llava_onevision fails without it :o (because the token in question is outside than the vocab size in the test settings)

This is interesting because all llavas were/should be desinged in such a way that image tokens are part of embeddings' vocab size and we should be able to embed these image tokens. LMK check why llava-onevision fails, might need to fix tests if they are designed incorrectly

> thanks so much for handling this in my absence 

hehe no worries -- I would expect the reverse to be true if I was off 😉 

I'm starting to see this pattern many times, where it would help if we could define a generation config specific to the model in question. I'll bump it up in my inner mapping of priorities :)
Fix ByteLevel alphabet missing when Sequence pretokenizer is used
Author: umarbutler
Pull request: yes

When one attempts to train a tokenizer such as [`Xenova/gpt-4o`](https://huggingface.co/Xenova/gpt-4o) that uses a sequence of pretokenizers like so:
```json
    "pre_tokenizer": {
      "type": "Sequence",
      "pretokenizers": [
        {
          "type": "Split",
          "pattern": {
            "Regex": "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
          },
          "behavior": "Removed",
          "invert": true
        },
        {
          "type": "ByteLevel",
          "add_prefix_space": false,
          "trim_offsets": true,
          "use_regex": false
        }
      ]
    },
```

This has the effect that the initial ByteLevel alphabet is not added to the tokenizer's vocabulary because the current `PreTrainedTokenizerFast.train_new_from_iterator()` method only does a simple check for whether `pre_tokenizer` has the type `ByteLevel` and not whether it is of the type 'Sequence' but has a child pretokenizer of the type `ByteLevel`, which is how this PR fixes that bug.

Comments:
cc @xenova @ArthurZucker @itazap 

Thanks for raising! 💯 Agree with your change,

@ArthurZucker recently added features to `tokenizers` to support accessing rust objects and attributes

(To Arthur) would it be possible to also support updating params`vocab` and `merges`? Right now we can make some changes to check `type` and set attrs such as `end_of_word_suffix` for example, but wouldn't be able to fully factor out `tokenizer_json` without `vocab` and `merges`. wdyt? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33556). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@umarbutler some formatting tests failed, can you please run `ruff format <path_to>/tokenization_utils_fast.py` with `ruff==0.5.1`? 🤗
[DO NOT MERGE] Idefics3 - resolve data_format for idefics3 image processing 
Author: amyeroberts
Pull request: yes

Demo of changes for image processing to set `data_format` and `input_data_format` in idefics 3's image processing

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33599). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing as changes from diff have been included in the idefics 3 PR #32473
bump tokenizers, fix added tokens fast
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Since tokenizer is being released, we can now fix the way tokens are added

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yeah still a few bugs to fix tho :( getting this ready for next release!
[Pixtral] Improve docs, rename model
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a follow-up to #33449, some cosmetic updates, and renaming `PixtralModel` to `PixtralVisionModel` for consistency with models like `SiglipVisionModel`.


Comments:
cc @ArthurZucker feel free to takeover, not sure why CI is failing (`make fixup` is passing locally)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33491). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging I will fix failing in my end
Add SDPA support for M2M100
Author: isaac091
Pull request: yes
Labels: name: run-slow

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
This PR adds SDPA support for M2M100 models.

Part of #28005.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @fxmarty @amyeroberts 

Comments:
slow runs cc @amyeroberts 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33309). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Great - all looks good. @isaac091 could you resolve the conflicts? Once done we should be good to merge 

Resolved! Thanks @amyeroberts. 

Merging as the failing hub tests are unrelated (also on other PRs, shouldn't be impacted by this PR, looks like connection errors) and slow tests all pass (other than multi-gpu test which also failed for a connection error)
Support AdEMAMix optimizer
Author: fakerybakery
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request

Add support for the AdEMAMix optimizer

Paper: https://arxiv.org/abs/2409.03137
Implementation: https://github.com/nanowell/AdEMAMix-Optimizer-Pytorch

### Motivation

The AdEMAMix optimizer is a new optimizer from Apple that claims to have better performance and slows down forgetting.

### Your contribution

N/A

Comments:
Implemented here
https://github.com/bitsandbytes-foundation/bitsandbytes/releases/tag/v0.44.0
Flash Attention 2 support for BERT, DistilBERT, and T5
Author: AmenRa
Pull request: no

Hi and thanks for adding Flash Attention 2!

I was wondering if there's any plan to add support for Flash Attention 2 to BERT, DistilBERT, and T5 models.
Those models are still the go-to Transformer models in my research community (Information Retrieval).
Reducing training/inference time and memory usage of those models would be extremely helpful for many researchers and practitioners.

Thank you!

Elias

Comments:
Hey @AmenRa you might be interested in this issue: https://github.com/huggingface/transformers/issues/26350

cc @younesbelkada 

Thanks for your interest @AmenRa, I have just added these architectures to the list in #26350

@LysandreJik thanks for the fast reply.
@younesbelkada thanks for adding the models to your list.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

For anyone who is interested in optimized T5 version, I just finished my project on creating flash attention version with fused attention bias calculation. It allows to fix the major drawbacks of T5 and allow to run it on 100k sequences on single L4 GPU (22.5 GB). Check it  [here](https://github.com/Knowledgator/TurboT5). 

I am in a desperate need for FA on T5 to be officially supported in `transformers`.
Fix Megatron-LM tokenizer path
Author: hikari-kubota-fixstars
Pull request: yes

# What does this PR do?
This PR addresses an issue with the Megatron-LM checkpoint conversion script introduced in the [documentation](https://huggingface.co/docs/accelerate/usage_guides/megatron_lm#utility-for-checkpoint-reshaping-and-interoperability). The path to the Megatron-LM tokenizer was outdated and did not match the current path. I have updated the script to reflect the correct path to the tokenizer.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@muellerzr @ArthurZucker 


Comments:
@ArthurZucker 

Thank you for pointing that out. Yes, you're correct. 



In version `0.5.0` (tag: `core_v0.5.0`), Megatron-LM was using:

```
from megatron.tokenizer.tokenizer import _vocab_size_with_padding
```

However, starting with version `0.6.0` (tag: `core_v0.6.0`), this was changed to:

```
from megatron.training.tokenizer.tokenizer import _vocab_size_with_padding
```

@ArthurZucker 

 I’ve implemented the version checks for Megatron-LM and updated the code to handle different versions accordingly. If there are any additional changes or issues, please let me know!

@ArthurZucker 

I'd like to add that the commit where the change from `megatron/tokenizer/tokenizer.py` to`  megatron/training/tokenizer/tokenizer.py` happened is here :
https://github.com/NVIDIA/Megatron-LM/commit/38644dd756c46142787b1bd5dc08ed82ca0e6de1#diff-4d0a12b13825799ff228dad8740d28ea3a79dd0e83cd5fb72f796be6a6e5ce03

@ArthurZucker 
Thank you for providing the sample code for 'check the version.' I have modified the code to use `from packaging import version`. If you have any other suggestions, please let me know.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add sdpa for BioGpt
Author: OmarManzoor
Pull request: yes
Labels: name: run-slow

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Towards #28005

Adds sdpa for the Bio GPT model.


## Who can review?

@amyeroberts 
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33592). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add verification of cache position
Author: ManuelFay
Pull request: yes
Labels: name: Cache

# What does this PR do?

This PR checks if cache_position is not None before attempting to verify a statement at position 0.
In practice, this enables to use the `prepare_inputs_for_generation` without cache.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@zucchini-nlp 

Comments:
Hi! I think this is a duplicate of https://github.com/huggingface/transformers/pull/33654 
No issue merging either PR, just want to understand why cache_position isn't defined but we can add the check

Haha very slightly different code I see ! 

Personally, the reason is a bit "off the beaten path". Basically, there are use cases in which it's interesting to use the `prepare_inputs_for_generation` without actually running the generation. I may actually be wrong here, main thing here is we get the Rope deltas. Using the prepare_model_for_generation just enables to standardize this operation to make sure the inputs will look similar to what the generative model should see.

In my use cases, I guess best would be to get the rope deltas separately, but since I saw someone else getting the same cache issue, I thought I could just propose this seemingly harmless fix (even if it's to abuse the function later haha)

100% would understand if it's not a good enough reason to merge though !



fair enough - closing the PR
Add equivalence testing between transformers pipelines + huggingface.js tasks
Author: Rocketknight1
Pull request: yes

This PR sets up a basic testing framework for equivalence testing. Right now basically none of the pipelines are equivalent, but as the PR progresses we'll hopefully bring them into alignment and add them to the testing list.

UPDATE: This is now mostly ready!

TODO:
- [ ] FeatureExtractionPipeline not yet in sync
- [ ] TranslationPipeline not yet in sync
- [ ] TextGenerationPipeline not yet in sync
- [ ] ImageFeatureExtractionPipeline has no spec
- [ ] ZeroShotAudioClassificationPipeline has no spec
- [ ] MaskGenerationPipeline has no spec
- [ ] ImageToImagePipeline is totally divergent
- [ ] Find a proper home for the test that's not `test_modeling_gpt2.py`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33452). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cool, thanks @Rocketknight1! Would it be possible to split it into multiple PRs, and to introduce tests to ensure it stays true to the definition in `huggingface_hub`? This will make the review simpler and faster. Thanks!

@LysandreJik sure! I think I can refactor it like that
Uniformize kwargs for chameleon processor
Author: leloykun
Pull request: yes

# What does this PR do?

Uniformizes kwargs of Chameleon processors as discussed in https://github.com/huggingface/transformers/issues/31911

~~Currently a draft. Will set as ready for review once this PR gets merged: https://github.com/huggingface/transformers/pull/32013~~ The other PR will take longer to complete, but this can now be merged.

Fixes # (issue)

- https://github.com/huggingface/transformers/issues/31911

## Who can review?

@zucchini-nlp @molbap 

Comments:
@zucchini-nlp this should now also be ready for review

Thanks so much for your contribution @leloykun ! This PR was a bit outdated compared to main so I rebased it and modified some other nits, but otherwise it seems all good to me.
@molbap @amyeroberts this should be ready for review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32181). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

the changes look good to me

thanks for the help @yonigozlan!!
`processing_mllama.py` has a bug?
Author: Neo9061
Pull request: no
Labels: name: bug

### System Info

v4.45.0

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In https://github.com/huggingface/transformers/blob/f2c388e3f946862f657acc1e21b272ec946fc66c/src/transformers/models/mllama/processing_mllama.py#L305

should the code be following?

```
not all(batch_img_per_prompt == b for batch_img_per_prompt, b in zip(n_images_in_text, n_images_in_images))
```


### Expected behavior

The above code failed when you passed in a batch of examples

Comments:
It is expected to fail if you have a batch with no image in a sample, and images in an another one. Tho it should not fail for batch of 1 image in a sample and more in another one, as I suppose the batch would get padded and you can ignore them. Opening a PR in a bit! 🤗