noamaanMulla-03/transformers-issues huggingface dataset:
Fix decoder_bbox_embed bug and add test for GroundingDINO issue37333
Author: islemyakoubi
Pull request: yes

## What does this PR do?

- Fixes a shallow‐copy bug in `GroundingDinoMLPPredictionHead` when `config.decoder_bbox_embed_share=False`:
  - Previously all decoder layers shared the same head instance.
  - Now each layer builds its own `GroundingDinoMLPPredictionHead` and wraps them in a `nn.ModuleList`.
- Adds a new smoke test script at `tests/models/grounding_dino/test_grounding_dino.py` that:
  - Loads `IDEA-Research/grounding-dino-base` via the HF processor and model.
  - Runs a simple inference on the Lena image with prompt `"a face"`.
  - Prints out the shapes and a sample of `logits` and `pred_boxes` on GPU to verify end-to-end functionality.

## Why is this needed?

Without this change, all decoder layers share the same bbox‐head instance, which breaks fine-tuning and multi-layer refinements.

## Who to tag?

Vision models: @ArthurZucker @Rocketknight1,


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @ArthurZucker @Rocketknight1,
This PR fixes a bug in the GroundingDinoMLPPredictionHead where all decoder layers previously shared the same bbox_embed head. The fix ensures that each decoder layer gets its own prediction head, which is essential for proper fine-tuning and multi-layer behavior.

Additionally, I added a test script that runs inference on IDEA-Research/grounding-dino-base with a simple prompt. The test verifies shapes and prints outputs using GPU.

I'd be grateful if you could take a look.
Thanks in advance!



hi @islemyakoubi, right now this PR deletes half of the repo! Please try rebasing or opening a new PR so that only the relevant lines are changed. We can't even review it right now because it's impossible to see where the real changes are.

The most likely cause is that you ran some kind of code style tool that differs from ours, which reformatted most of the library. Try `pip install transformers[quality]` to ensure that you're using the same versions of tools as we use.


![image](https://github.com/user-attachments/assets/53058655-93df-4e2b-bd95-d2077f7ce17a)
Issue with ESM and DDP due to unused positional embeddings when rotary embeddings specified
Author: simonlevine
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.25.1
- Platform: Linux-5.4.0-136-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, 1 GPU
- Using distributed or parallel set-up in script?: Yes, DDP


### Who can help?

@ArthurZucker @Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using class `ESMModel` results in an unused parameter creation. In the DDP setting, this causes errors as some parameters don't receive grad. Within `EsmEmbeddings` , self.positional_embeddings should not be instantiated as an `nn.Embedding` if `config. position_embedding_type` is not `absolute`.

### Expected behavior

We would expect instantiaton of `EsmEmbeddings` to not create the unused `nn.Embedding` module as this can create cryptic errors later due to unused parameters.

Comments:
Note that you can avoid any error by passing `find_unused_parameters=True` when wrapping your model in DDP.

This is part of the model design that I believe is in the original repo as well, but if it's still causing issues with @sgugger's fix, we can look into making a PR to remove the base positional embeddings when the model is using rotary embeddings instead!

If it's working for you with that fix, though, feel free to close the issue.

Hi @Rocketknight1, it would fix the issue but due to other features of the particular model (ESM within a larger module) in use, the flag results in `RuntimeError: Expected to mark a variable ready only once`. At any rate, it turned out this portion of ESM was the only problematic portion, but feel free to close the issue since that find unused params fix will (probably) be sufficient for most users.

Alright, I'll do that for now - but if anyone comes across this issue and `find_unused_parameters=True` is not helping, feel free to reopen and comment here!

It would be nice if this could be fixed properly at some point. Setting `find_unused_parameters=True` comes with a performance hit due to an extra backward pass, and shouldn't really be necessary here.

We'd definitely welcome a PR if anyone wants to open one! 

[PR here](https://github.com/huggingface/transformers/pull/38089/files) @Rocketknight1  @max-niederman
Versions greater than 4.49 are not compatible with Ascend NPU
Author: 1737686924
Pull request: no
Labels: name: bug

### System Info

910B1,python==3.10,transformers==4.51.0

### Who can help?

![Image](https://github.com/user-attachments/assets/f08b4c44-1c47-41b0-87b2-5b85a5f6ad48)
When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

### Expected behavior

When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

Comments:
cc @ivarflakstad, with apologies!

Hey!
Just as a general note when you decide to use an older version of one library (torch in this instance) it is not completely unexpected that you would get issues with newer versions of another (transformers in this instance). Transformers supports a fairly wide range of `torch`, but when using `torch-npu` I'd recommend a smaller range as the entire ecosystem around npus is less mature. That's true for any software btw, not specific to transformers, torch, etc. 

Have you tried torch* > 2.4.0?


I discovered the problem. I upgraded torch*==2.5.1 and solved this problem
Continuous batching
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Draft continous batching for eager attention

TODO:
- test on gpu for the paged attention path
- better scheduling: sort request with keys being amount of blocks required, you only have to take the 1st sequence smaller than this
- better cache object: not having to create both the cache index and the fill index would earn some time
- profile + performance optimisation

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35727). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
update `require_read_token`
Author: ydshieh
Pull request: yes

# What does this PR do?

update `require_read_token`.

Currently, that decorator only works with function/method, but not for a class. The usage of it on classes will skip all the tests in those classes.

This PR fixes this issue.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38093). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added scores in the streamer classes based on generation flag
Author: LukasAfonso
Pull request: yes

This PR works on a personal issue I had when creating a JSON formatter.
I needed to know the tokens' scores as soon as they were ready; as such, I created 2 new classes that inherit BaseStreamer to support Immediate token output and forwarded the scores from each generate function into the streamer put method.

## Before submitting
- [ x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [  x ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ x ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ x ] Did you write any new necessary tests?

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @gante

Hey @LuisCarlos-104171 👋 Thank you for opening the PR with the feature 🤗 

We are in the (long) process of redesigning `generate` to be able to `yield` arbitrary outputs, as opposed to relying on dedicated streaming classes. As such, I'm not accepting more features on top of the streaming classes, to help us focus our energies on the project 🤞 

Oh ok, thanks por the quick reply!
There is an ETA on the time that will take to implement those changes?

If needed I may be able to help!

> There is an ETA 

Not really, its priority is below other short-term projects so it's hard to estimate :)
add timeout for downloading the `librispeech_asr` dataset
Author: faaany
Pull request: yes

## What does this PR do?
When running the `run_wav2vec2_pretraining_no_trainer.py` example, I got `fsspec.exceptions.FSTimeoutError`.

The reason is that the default timeout (5 * 60s) for the `aiohttp` library used internally by fsspec is too short for large files, e.g. 5.9G. 

I also found an issue discussed in the dataset library, which resolved the issue by the same approach: https://github.com/huggingface/datasets/issues/7164. 




Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @Rocketknight1 

This looks good, but the timeout should probably be even longer! Some connections may still struggle to fetch 6GB in 10 minutes. Maybe make it `60 * 60` to be safe?

> This looks good, but the timeout should probably be even longer! Some connections may still struggle to fetch 6GB in 10 minutes. Maybe make it `60 * 60` to be safe?

yes, good suggestion! code updated.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38073). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adding FP8 Quantization to transformers
Author: MekkCyber
Pull request: yes

# What does this PR do?
This PR adds the FP8 quantization to transformers to support Deepseek models https://github.com/huggingface/transformers/issues/35471, and allow native quantization in FP8.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36026). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the great work. Following the guidance in the doc, I found that there is one parameter called `kv_a_proj_with_mqa` with shape [7168, 576], where 576 can not be divided by 128. So there will be a problem.

cc @MekkCyber 

Hi @pprp, in the kernel used we use masks like
```
for k in range(0, tl.cdiv(K, BLOCK_SIZE_K)):
        a = tl.load(a_ptrs, mask=offs_k[None, :] < K - k * BLOCK_SIZE_K, other=0.0)
        b = tl.load(b_ptrs, mask=offs_k[:, None] < K - k * BLOCK_SIZE_K, other=0.0)
```
So I don't think the shape should be a problem
Don't drop dataset columns for custom collate functions
Author: yaswanth19
Pull request: yes

# What does this PR do?

The existing logic is we inspect model's forward and dataset's return dict keys() and drop unused keys. A simplest scenario where this logic will fail is if I return raw text and label from dataset class and then tokenize in collate_fn. Since model is not expecting the raw text as an input, we drop it but collate func code will break as there is no text to tokenize. 

So, don't drop unused cols if a custom collate func is passed as the user might have other intentions and we can safely assume the user can handle any errors caused then on.

Actually the existing logic won't break for the above case if a tuple is passed from dataset to the data collator :sweat_smile:  coz we don't know the columns names to drop in that case. So, better to standardize this when we also return dict

CC: @muellerzr for trainer and @ArthurZucker  for core stuff

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @sunmarc!

Yup MB, missed the If condition in `_get_collator_with_removed_columns`. Thank you !!
Disable report callbacks for certain training tests
Author: ivarflakstad
Pull request: yes

This fixes the `No module named 'pynvml'` error in the affected tests on non-cuda devices.

~It does not address `test_auto_batch_size_finder` however, as that test runs `run_glue.py` directly.~

I tried adding `--report_to none` as an argument to the `run_glue.py` call. I'll check if it actually works.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38088). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[smolvlm] skip the test
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, we should skip the test and fix later

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38099). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Oke, found the issue, we weren't passing the correct processor. Fixed it and I'll be cleaning up SmolVLM processor soon, to split out image from video processing
Fix `past_key_values` type hint in model output types
Author: ChengLyu
Pull request: yes

# What does this PR do?

This PR fixes the `past_key_values` type hint in model output types to adopt the new `Cache` type.


## Before submitting
- [✅ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten @sgugger @tanreinama 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

LGTM but cc @gante to double-check this is right! Feel free to approve + merge without additional review if you're happy

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37953). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix `check_bad commit.py` gives wrong results
Author: ydshieh
Pull request: yes

# What does this PR do?

During bisecting, it uses

> elif f"{target_test} FAILED" in result.stdout:

to determine a commit fails a test or not.

However, since #35912, the output is kind altered with `live log call`, and the results are wrong (it think every commit passes).

This PR fixes the issue by checking 

> elif f"FAILED {target_test}" in result.stdout:

(at `short test summary info`)

#### current output running on main

> tests/models/instructblipvideo/test_modeling_instructblipvideo.py::InstructBlipVideoModelIntegrationTest::test_expansion_in_processing 
-------------------------------- live log call ---------------------------------
WARNING  transformers.video_processing_utils:logging.py:[32](https://github.com/huggingface/transformers/actions/runs/14999267405/job/42141539393#step:5:33)8 You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
FAILED

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38107). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix InternVL interpolate_pos_encoding and add to video_processing_auto
Author: yonigozlan
Pull request: yes

# What does this PR do?
Fix small issue with InternVL interpolate_pos_encoding function, and add InternVL to video_processing_auto (currently all integration tests are crashing on main).

Cc @zucchini-nlp for video processor

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CSM] update test for t4 runners
Author: eustlb
Pull request: yes

This updates the tests for our CI T4 runners.
I obtained the values using the original code reproducer scripts (see test descriptions), which I ran on T4 runners.

cc @ydshieh 😊


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38110). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
proof of concept for using dataset of test cases for tokenizer tests
Author: itazap
Pull request: yes

To refactor comparing slow == fast, we need to 'freeze' current working behavior of tokenizers. We move the test strings to the hub (https://huggingface.co/datasets/hf-internal-testing/tokenization_test_data), with expected tokens, encoded_ids, and encoded_ids with special tokens. I got the test strings for all sentence piece based models and I ran all sp-based models on these test --> then uploaded to the dataset.

This allows us to

1. preserve testing expected behavior without relying on comparing fast == slow
2. reduce overwriting of tests per class
3. future tests can be added here!

*Note: will convert other slow == fast cases! this is a POC for `test_rust_and_python_full_tokenizers` to get some feedback :). then we can easily move tests like `test_tokenization_python_rust_equals` and many others.

**More ideas/ considerations**
- need to check if this slow downs running these tests


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37994). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @itazap 

> To refactor comparing slow == fast, we need to 'freeze' current working behavior of tokenizers.

Could you elaborate what refactorization you plan to achieve,  and explain why this approach is "needed" for it?

> preserve testing expected behavior without relying on comparing fast == slow

There is still `test_rust_and_python_full_tokenizers` in the common file. So IIRC, we don't  run it with slow tokenizer, but just fast with its  expected value?

> reduce overwriting of tests per class

This is very nice.

> future tests can be added here!

I am not in favor of putting the input and expected values on the hub repository like this. This makes working on test (write it and maintain it like debugging and update) more frictional. It also prevents the community to contribute directly, IIRC.

I am happy if these test values being kept in each test file, and the tests in `tests/test_tokenization_common.py` use them from there. I think this is still doable, and still achieve what you try to do.