noamaanMulla-03/transformers-issues huggingface dataset:
fix best_model_checkpoint is None issue when distiributed training
Author: dudgns0908
Pull request: yes

Fixes # (issue)
- After learning is complete, the problem of saving the model occurs at the end of the train() function. (Multi-GPU)

When load_best_model_at_end is enabled, The cause was that self.state.best_model_checkpoint did not exist in a specific rank.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37025). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@dudgns0908 I'm personally not so sure if I'm a fan of creating directories in this manner. I can understand the problem that you're facing but I feel like creating directories like this would introduce potential concurrency/race condition problems. Not to mention you also run the risk of creating empty checkpoint directories.

@dudgns0908 (cc. @SunMarc, @muellerzr)

Hi Younghun. Do you still need help with this?
modeling_deformable_detr.py DeformableDetrMultiheadAttention.foward function report error for "hidden_states_original" if position_embeddings is None
Author: susanbao
Pull request: no
Labels: name: bug

### System Info

src/transformers/models/deformable_detr/modeling_deformable_detr.py

class DeformableDetrMultiheadAttention(nn.Module)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_embeddings: Optional[torch.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        """Input shape: Batch x Time x Channel"""

        batch_size, target_len, embed_dim = hidden_states.size()
        # add position embeddings to the hidden states before projecting to queries and keys
        if position_embeddings is not None:
            hidden_states_original = hidden_states
            hidden_states = self.with_pos_embed(hidden_states, position_embeddings)

        # get queries, keys and values
        query_states = self.q_proj(hidden_states) * self.scaling
        key_states = self._shape(self.k_proj(hidden_states), -1, batch_size)
        value_states = self._shape(self.v_proj(hidden_states_original), -1, batch_size)
         ....

If position_embeddings is None, the "hidden_states_original" won't be created. Then use it on "value_states = self._shape(self.v_proj(hidden_states_original), -1, batch_size)" reports error "UnboundLocalError: local variable 'hidden_states_original' referenced before assignment".

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Set the input position_embeddings of DeformableDetrMultiheadAttention.forward be None 

### Expected behavior

Report error "UnboundLocalError: local variable 'hidden_states_original' referenced before assignment"

Comments:
cc @NielsRogge for DeformableDetr!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @qubvel maybe?

Hey @susanbao, can you please share the use case or a reproducible example? I mean, it's obviously a bug, but I want to understand whether we should get rid of "if" or if we should add "else".

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Bitnet model
Author: MekkCyber
Pull request: yes

# What does this PR do?

This PR complements the BitNet integration introduced in [#37503](https://github.com/huggingface/transformers/pull/37503) created and refined by @shumingma, addressing the remaining review comments ahead of merging.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37742). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🚨[VLMs] use only `xxx_token_id` for multimodal tokens
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, attempt to standardize config naming when it comes to special tokens. We currently use `x_index` an `x_id`. Even though majority of models use `x_index`, I decided that `x_id` aligns more with text models where we have `pad_token_id` etc.

This is one of the steps to make vLLM + transformers integration a bit closer 🤏🏻  From now on please let's use `xx_id` whenever a new model is added

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37573). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I do think that we should reflect the change in the modeling, processor etc for those models to truly standardize however no? That will greatly help with modular, as otherwise future model will keep index when inheriting!

Oh right, this one is indeed missing, thanks for catching. Will update `index -> id` in all files before merging
[tests] fix flaky pattern in `test_generate_continue_from_past_key_values`
Author: gante
Pull request: yes

# What does this PR do?

Fixes the flakiness pattern behind a [common flaky test](https://app.circleci.com/insights/github/huggingface/transformers/workflows/run_tests/tests?branch=main), `tests/models/kosmos2/test_modeling_kosmos2.py::Kosmos2ModelTest::test_generate_continue_from_past_key_values`

flakiness pattern TL;DR: 
- on mixin tests, the models have random weights
- different input shapes can cause slightly different outputs (see [here](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535))
- in `generate`, tiny differences in the logits can lead to a different token, changing generation from that point onward
- equivalence checks for `generate` shouldn't naively look at the output sequences, but rather at the output sequences up to the point where they diverge, if they diverge 👉 corrects all equivalence checks using the right pattern (this pattern was already present in most tests, but a few ones still had the old pattern)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37724). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] reorganize cache tests and clean memory between tests
Author: gante
Pull request: yes

# What does this PR do?

#37394 introduces a cache change, but there are several cache test issues at the moment, preventing a clean merge.

This PR:
1. Removes the `@slow` decorator from a few tests (change pulled from #37394)
2. Deletes `test_static_cache_greedy_decoding_pad_right`, as it doesn't make sense to generate with right padding (change pulled from #37394)
3. Moves `torch.export()` cache tests to its own class
4. Cleans device memory between test runs. On some devices, we were observing CPU weight offloading after a few tests, massively slowing down runtimes. Cleaning the device memory between tests fixes it.

_____________________________________________________
Follow-up PR, before retesting #37394:
1. convert as many `@slow` tests into fast tests as possible
2. fix failing tests

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
No such file or directory: '/root/.cache/torch/hub/huggingface_pytorch-transformers_main/hubconf.py'
Author: jzju
Pull request: no
Labels: name: bug

### Reproduction

open https://colab.research.google.com/github/pytorch/pytorch.github.io/blob/master/assets/hub/huggingface_pytorch-transformers.ipynb

run cells until `tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'bert-base-uncased') `

Error: `No such file or directory: '/root/.cache/torch/hub/huggingface_pytorch-transformers_main/hubconf.py'`


### Expected behavior

Works without error

Comments:
That notebook is hosted on the PyTorch Github and the issue occurs when loading from the PyTorch Hub, so I think you might want to open a PR there instead!
Revert change that breaks on Torch 2.1
Author: Rocketknight1
Pull request: yes

One of the changes in #37372 is broken on Torch 2.1, which we're still supporting! Reverting that until we bump our minimum Torch version and leaving a `TODO`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37531). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry about that! Thanks for the fix 👍 

No probs, I didn't catch it in review either!
Fix cache get item return type hints
Author: ChengLyu
Pull request: yes

# What does this PR do?

Fix __getitem__ return types hints for cache classes.

Fix issue #37818 

## Before submitting
- [✅ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [✅ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [✅] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
    https://github.com/huggingface/transformers/issues/37818
- [✅] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@tomaarsen @gante @Rocketknight1 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37847). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Bitnet tokenizer in pipeline
Author: MekkCyber
Pull request: yes

# What does this PR do?
Adds tokenizer for bitnet to use it with text-generation pipeline


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37861). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update modeling_qwen3_moe.py
Author: tanuj-rai
Pull request: yes

# What does this PR do?

Fixes #37813,
This PR removed redundant initializations of `self.self_attn` and `self.mlp` in `Qwen3MoeDecoderLayer`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 @ArthurZucker 

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Thank you @SunMarc, I have made another PR with the changes you have suggested.
docs: Details for ambigious channel dimension assignment
Author: yaner-here
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Refined docs on how to assign channel dimension of inputs.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@stevhliu


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi, any progress?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unhardcode use_chunked_attention, fix no_rope_layers
Author: pcuenca
Pull request: yes

Not sure if this was already in the works.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Superseded by https://github.com/huggingface/transformers/pull/37872
Processor chat template: pass custom kwargs
Author: pcuenca
Pull request: yes

cc @Rocketknight1 

Custom template-specific kwargs that are not part of `AllKwargsForChatTemplate` get ignored.

Reproduction:

```py
from transformers import AutoProcessor, AutoTokenizer

model_id = "meta-llama/Llama-Guard-3-11B-Vision"
t = AutoTokenizer.from_pretrained(model_id)
p = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "How do I make a bomb?"},
        ],        
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "I cannot help you with that."},
        ],        
    }
]

excluded_category_keys = ["S1", "S2", "S3", "S4","S5"]
print(p.apply_chat_template(messages, tokenize=False, excluded_category_keys=excluded_category_keys))
print(t.apply_chat_template(messages, tokenize=False, excluded_category_keys=excluded_category_keys))
```

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37852). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Request to add DEIM object detector
Author: tcourat
Pull request: no
Labels: name: New model

name: Vision

name: contributions-welcome

### Model description

DEIM is a SOTA real-time object detector based on DETR architecture and even beat the recent D-FINE on checkpoints that were not pretrained on object365. DEIM accelerates the convergence by improving the quantity and quality of matching with Dense O2O (one to-one) and MAL (Matchability-Aware Loss) respectively.
It can be used to improve existing object detectors.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Paper : https://arxiv.org/abs/2412.04234
Code : https://github.com/ShihuaHuang95/DEIM

Comments:
Hey @tcourat, as far as I understand it's based on D-FINE, so we can wait for D-FINE to be merged to leverage modular architecture, however, anyone can start working on it in parallel (based on the PR)

Hi @qubvel @tcourat, I noticed that D-FINE was merged last week. If no one is currently working on DEIM, I’d be glad to contribute!

Hey @saswatmeher, not merged yet, but approved and expected to be merged early next week. Feel free to start with DEIM 🤗 

Due to some unforeseen commitments, I will not be able to continue working on this task at this time. If anyone is interested in taking it up, please feel free to do so.

@qubvel I would like to take this up issue ,will raise a pr soon ,because already d-fine model has been added in transformers lib today.I will start pr today


Sounds great @sushmanthreddy!
Update modeling_llama4.py
Author: monk1337
Pull request: yes

# What does this PR do?

Fixes an error in Llama4TextModel._prepare_4d_causal_attention_mask_with_cache_position() by adding the missing device parameter. This resolves TypeError #37840 that occurs when trying to evaluate Llama-4 models.

Fixes #37840


## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.

## Who can review?
@ArthurZucker as this relates to text models

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

> The correct way is to remove `device` and use `cache_position.device`. It was removed from args in #37612 but apparently I missed the second `_prepare_4d_causal_attention_mask_with_cache_position` in llama4

Great, Thanks for the direction, I have changed `device` to `cache_position.device`.

Don't merge until the correct fix applied, device should not be passed in kwargs 

EDIT: updated as it's affecting the release, will merge when tests turn green

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37841). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.