noamaanMulla-03/transformers-issues huggingface dataset:
FIX FSDP plugin update for QLoRA
Author: BenjaminBossan
Pull request: yes

# What does this PR do?

Partially fixes the issue reported in https://github.com/huggingface/peft/issues/2407.

The `_fsdp_qlora_plugin_updates` checks for LoraConfig but other PEFT methods can also support quantized models, e.g. VeRA. Therefore, the isinstance check is now looking for PeftConfig in general.

Moreover, the fsdp_plugin variable may be undefined in the 2nd if condition, leading to an `UnboundLocalError` error. This is fixed by not assigning the variable at all.

I checked for tests that may need updating but only found [test_fsdp_config_transformers_auto_wrap](https://github.com/huggingface/transformers/blob/69bc848480d5f19a537a70ce14f09816b00cd80f/tests/fsdp/test_fsdp.py#L188) associated with this change (see #29587). AFAICT, this test does not cover the changed code, since the test does not start the training loop. Therefore, I haven't updated any tests. LMK if/how this fix should be tested.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36720). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr Could you please review or suggest a reviewer?

cc @SunMarc for secondary/final
Registering New Tokenizer + Possible Future Foundational Model Architecture Added to Transformers
Author: yuchenxie4645
Pull request: yes

This PR adds a new tokenizer, ArlowTokenizer, to the Hugging Face Transformers library. ArlowTokenizer is a BPE-based tokenizer designed for large language models, similar to GPT-2 and GPT-3. This architecture's got nothing to do with the 2 tokenizer versions I released on huggingface, I plan to train my V3 on 350 billion tokens once this PR gets approved. I basically rewrote it so that it is extremely similar to pretrainedfast tokenizer but made it so that it had the name ArlowTokenizer.

Changes Introduced

New directory src/transformers/models/arlow where it exists a init.py and tokenization_arlow.py (fast).
Edited init.py in models to include arlow in an import.
Updated src/transformers/init.py to append and define the model + the tokenizer for arlow.
Registers ArlowTokenizer for direct import (from transformers import ArlowTokenizer).

I verified that this indeed works with a dummy tokenizer. I had the test in transformers/testrun where it had the scripts and dummy tokenizer I used for testing.

A future PR may introduce an associated model architecture (ArlowForCausalLM), template already added.

Before submitting

[n] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[y] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
Pull Request section?
[n] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
to it if that's the case.
[y] Did you make sure to update the documentation with your changes? Here are the
[documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
[here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
[y?] Did you write any new necessary tests?

@ArthurZucker

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Ran make fix-copies, passed and worked.
Ran ruff check examples tests src utils, passed with no errors: "All checks passed!"
Don't know why check_code_quality and why repo consistency still gave me errors. 

Hi @yuchenxie4645 the cause is most likely an outdated version of `ruff`! Try `pip install -e .[quality]` in the `transformers` directory and then `make fixup` again.

@Rocketknight1 would there be any chance if I fix the code this would be merged? I really want to register my foundational model + tokenizer as an official hf transformers model.

@ArthurZucker I understand. But I just want to establish Arlow as an official huggingface transformer model. I most likely will also make a PR about the pretrained model & potentially a future VLM. I would really appreciate it if it was possible that it merges.

I can probably get the tokenizer trained this weekend.
Fix: Use config.use_sliding_window instead of config.sliding_window
Author: KarthikaRajagopal44
Pull request: yes

This PR fixes a bug where `config.sliding_window` (a number) is incorrectly treated as a boolean. 
The condition now checks `config.use_sliding_window` to avoid unintended warning messages.
Fixes: #36361





Comments:
I'm extremely sorry for the delay here! This PR slipped through the cracks.

cc @JustinLin610 since you made the original PR - can you check to make sure this is correct? If you don't have the time or bandwidth, let us know and we'll try to find another reviewer.
Update installation.md
Author: ariG23498
Pull request: yes

Fix virtual env activation command

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36826). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu apologies for not understanding it, but could you provide me with the link to the main readme?

Sorry I wasn't clear, I was referring to this [`README.md`](https://github.com/huggingface/transformers/blob/main/README.md#installation)!

@stevhliu I have made the update!
PixtralVisionModel does not support Flash Attention 2.0 yet
Author: xihuai18
Pull request: no
Labels: name: Feature request

### Feature request

Flash Attention 2.0 support for Mistral-small3.1

### Motivation

https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503

Mistral-small3.1 is a powerful small LM.

### Your contribution

No.

Comments:
PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
Author: gaceladri
Pull request: no
Labels: name: New model

name: Feature request

# 🚀 Feature request

PruneTrain. {...} By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, **PruneTrain achieves a reduction of 39% in the end-to-end training time of ResNet50 for ImageNet by reducing computation cost by 40% in FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the inter-accelerator communication by 55%.**

## Motivation

I'm pre-training some midsize language models from scratch. If you tell me that I can pretrain a network with 1% drop in performance while cutting down the energy demand of the training by up to 40% and speeding inference time at the same time, I will buy it.

## Your contribution

https://arxiv.org/abs/1901.09290. I can not understand why the authors did not open source the code, since it could reduce the global warming, speedup experimentation and reduce energy consumption. 

Comments:
[This repository](https://github.com/VictorSuciu/prunetrain) claims to be the source code, but I can't find anything that necessarily relates to the original authors. I'll try to walk through to code and paper together when I have some time to verify. It's a few years late but hopefully it helps someone coming across it.
fix Gemma3 Config
Author: eljandoubi
Pull request: yes

# What does this PR do?

There is a silent bug when trying to initialize Gemma3Config from SiglipVisionConfig. It gets initialized from the default vision config.

## Who can review
Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).
Support for SpatialLM series model
Author: nekomeowww
Pull request: no
Labels: name: New model

### Model description

[manycore-research](https://huggingface.co/manycore-research) just released the [`manycore-research/SpatialLM-Qwen-0.5B`](https://huggingface.co/manycore-research/SpatialLM-Qwen-0.5B) and [`https://huggingface.co/manycore-research/SpatialLM-Llama-1B`](https://huggingface.co/manycore-research/SpatialLM-Llama-1B) models. 

Currently it's not possible to ask `transformers` to inference the SpatialLM series model:

```
The checkpoint you are trying to load has model type `spatiallm_llama` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
```

I believe that the `model_type` here with value `spatiallm_llama` is causing the problem:

```
{
  "model_type": "spatiallm_llama",
}
```

> https://huggingface.co/manycore-research/SpatialLM-Llama-1B/blob/main/config.json#L20

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/datasets/manycore-research/SpatialLM-Testset

Comments:
Yes, this code is intended to be run only with their custom code, so it's not really a Transformers model at all. I think we could support a PR to add it, though? cc @qubvel @nielsrogge

The modeling code looks nice and is already based on the transformers code, so it should be relatively easy to add it natively to transformers. The only thing is point clouds processing, ImageProcessor is not really the place to do it, but we can probably make an exception to avoid adding a new processor class for a single model.

I would love to participate in implementing the support! I do know the codebase structure of `transformers.js`, but I haven't tried for `transformers`, may need some assistant on this.

@nekomeowww hey! You can refer to the below docs for adding a new model. For new models we also suggest to use `modular` which allows to easily re-use existing code from `transformers`. Also, it makes the review process faster and highlights differences between new model and any other existing model in the library

How to add a new model: https://huggingface.co/docs/transformers/add_new_model
How to use `modular` inheritance: https://huggingface.co/docs/transformers/en/modular_transformers
Update deprecated Jax calls
Author: rasmi
Pull request: yes
Labels: name: Flax

 jax v0.4.27 introduced the [deprecation](https://jax.readthedocs.io/en/latest/changelog.html#jax-0-4-27-may-7-2024:~:text=jax.numpy.clip()%20has%20a%20new%20argument%20signature%3A%20a%2C%20a_min%2C%20and%20a_max%20are%20deprecated%20in%20favor%20of%20x%20(positional%20only)%2C%20min%2C%20and%20max%20(%2320550).) of `jax.numpy.clip`'s `a_min` and `a_max` arguments (changing to just `min` and `max`, respectively). This PR updates [all uses of jnp.clip](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers+%22jnp.clip%22+%28%22a_max%22+OR+%22a_min%22%29&type=code) to use these new arguments. 

This PR was originally written by @jakevdp.

cc @sanchit-gandhi your review would be appreciated!



Comments:
@sanchit-gandhi -- the test environment seems to be using the [older version of Jax](https://app.circleci.com/pipelines/github/huggingface/transformers/116923/workflows/4cd223ef-a7e0-423b-ac20-6be703dd99d8/jobs/1571023/parallel-runs/0/steps/0-107?invite=true#step-107-766_11), despite `setup.py` and `dependency_versions_table.py` being updated. Is there a proper way to force the docker image to install the fresh set of requirements as it spins up the test environment?

@ArthurZucker @sanchit-gandhi  -- do you know who can help with this? There are various other upcoming jax deprecations that I'd like to fix, so being able to test them correctly would be helpful.

@Rocketknight1 @amyeroberts  -- please let me know how to proceed! I would love to get rid of some of these deprecated methods so that the library can be kept up-to-date.

Hi @ArthurZucker, I've rebased to main and all tests are passing!  Let me know if any additional action is needed.

cc @Rocketknight1 as well if you're able to review!

Hi @ArthurZucker, sorry this broke things! Do you know what went wrong? Is there a way to test this properly going forward? The CI does not seem to use an updated/custom Docker image if changes are made, so I'm not sure what the best approach is.

Yeah sorry, it was just `from transformers import *` or something like that 

We'll try to come back to this
tj-actions/changed-files action compromised
Author: dtrifiro
Pull request: no

As this is being widely reported ([stepsecurity.io blog](https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised), https://github.com/tj-actions/changed-files/issues/2464, https://github.com/tj-actions/changed-files/issues/2463), the `tj-actions/changed-files` action has been compromised.

This has already been fixed for `transformers` on March 15th (f263e88dcf), although no analysis or pull request is linked to this commit.

Can we expect an analysis/post-mortem of this (potential) security vulnerability for this project?

Thanks.

Comments:
Hi @dtrifiro, we don't think this has any security implications for downstream users. The compromised action was used for ~1 day in the CI for HuggingFace. This resulted in the potential leak of a Hugging Face Github token, which we have invalidated and rotated. We don't believe that token was used, and we don't think other tokens were exposed. If you want any additional info, let us know, and feel free to close the issue if not!

Thanks.
tests: fix asyncio.wait() usage for python>=3.11
Author: dvrogozh
Pull request: yes

Passing coroutings directly to `asyncio.wait()` is deprecated since python 3.8 and removed starting from python 3.11. Instead, it's required to explicitly wrap coroutine in the task with `asyncio.create_task()` which first appeared in python 3.7.

We step into this issue running the following Transformers tests on a systems with python 3.11 or later (for example, Ubuntu 24.04 has python 3.12). Note that multi-accelerator system is required to run these tests (multi-CUDA or multi-XPU):

* `tests/trainer/test_trainer_distributed.py`
* `tests/extended/test_trainer_ext.py`

The error will be:
```
src/transformers/testing_utils.py:2380: in execute_subprocess_async
    result = loop.run_until_complete(
/usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete
    return future.result()
src/transformers/testing_utils.py:2368: in _stream_subprocess
    await asyncio.wait(
...
E           TypeError: Passing coroutines is forbidden, use tasks explicitly.

```

See: https://docs.python.org/3.10/library/asyncio-task.html#asyncio.wait
See: https://docs.python.org/3.10/library/asyncio-task.html#asyncio.wait
See: https://docs.python.org/3.7/library/asyncio-task.html#asyncio.create_task

CC: @ydshieh, @SunMarc 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

merge as the failing tests are irrelevant to this PR
[chameleon] fix num image token check
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

As per title, reported by Harry as it affects vLLM internal tests. The error was never caught because apparently Chameleon tested only text inputs w/o images

Added tests for `Vision2Seq` and made sure all green

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36918). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Processor tests failing are all Mistral3, need a read-token which I just added. Not sure yet if the bot has access to Mistral3 though
Fix Compressed tensors to_dict_diff
Author: MekkCyber
Pull request: yes

# What does this PR do?
When creating the dict config, checks if the key exists in the defaut config before trying to access its value.
It was causing the error : 
```
File "./transformers/src/transformers/utils/quantization_config.py", line 1364, in to_diff_dict
    if value != default_config_dict[key]:
                ~~~~~~~~~~~~~~~~~~~^^^^^
KeyError: 'config_groups'
```



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36922). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[qwen2-audio] remove default template
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, blocked by PR on hub model repo, which adds chat template in `chat_template.json`


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36919). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, storing template in code was deprecated long ago but for some reason qwen2-audio didn't get its update on time. Not much blocking, just cleaning up so no-one copies bad patterns when adding new models
Adding FlexAttention Support for Qwen2 models
Author: MekkCyber
Pull request: yes

# What does this PR do?

Adding flex_attention for Qwen2 models following https://github.com/huggingface/transformers/issues/34809

## Who can review ?
@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35155). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @ArthurZucker, I fixed the issue related to `Qwen2Moe` copying from `Qwen2` attention classes, however I'm still running into some errors with `make fixup` and I don't see the relationship between the error related to `starcoder` and the `Qwen2` models : 
```
No differences found for src/transformers/models/llava_next_video/configuration_llava_next_video.py.
No differences found for src/transformers/models/llava_next_video/modeling_llava_next_video.py.
No differences found for src/transformers/models/ijepa/modeling_ijepa.py.
No differences found for src/transformers/models/aria/configuration_aria.py.
No differences found for src/transformers/models/aria/modeling_aria.py.
No differences found for src/transformers/models/aria/image_processing_aria.py.
No differences found for src/transformers/models/aria/processing_aria.py.
No differences found for src/transformers/models/olmo2/configuration_olmo2.py.
No differences found for src/transformers/models/olmo2/modeling_olmo2.py.
No differences found for src/transformers/models/glm/modeling_glm.py.
No differences found for src/transformers/models/gemma/configuration_gemma.py.
No differences found for src/transformers/models/gemma/tokenization_gemma.py.
No differences found for src/transformers/models/gemma/modeling_gemma.py.

Differences found between the generated code and src/transformers/models/starcoder2/modeling_starcoder2.py:

   1 --- src/transformers/models/starcoder2/modeling_starcoder2.py_generated
   2 +++ src/transformers/models/starcoder2/modeling_starcoder2.py
   3 @@ -646,7 +646,6 @@
   4      _supports_cache_class = True
   5      _supports_quantized_cache = True
   6      _supports_static_cache = True
   7 -    _supports_flex_attn = True
   8  
   9      def _init_weights(self, module):
  10          std = self.config.initializer_range
No differences found for src/transformers/models/gemma2/configuration_gemma2.py.
No differences found for src/transformers/models/gemma2/modeling_gemma2.py.
No differences found for src/transformers/models/instructblipvideo/configuration_instructblipvideo.py.
No differences found for src/transformers/models/instructblipvideo/modeling_instructblipvideo.py.
Traceback (most recent call last):
  File "/fsx/mohamed/dev/transformers/utils/check_modular_conversion.py", line 78, in <module>
    raise ValueError("Some diff and their modeling code did not match.")
ValueError: Some diff and their modeling code did not match.
make: *** [Makefile:39: repo-consistency] Error 1
```
[DON'T MERGE] test doc builder
Author: gante
Pull request: yes

# What does this PR do?

https://github.com/huggingface/doc-builder/pull/547/

We can see the runner being different
<img width="641" alt="Screenshot 2025-03-20 at 16 15 47" src="https://github.com/user-attachments/assets/8ad5364f-1b74-4cd1-a8fa-95b53eaa11d2" />


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

(closing: transformers found a solution that does not imply using groups, i.e. base GHA images are used)
Use another repo. for Mistral3 processor testing
Author: ydshieh
Pull request: yes

# What does this PR do?

Use another repo. for Mistral3 processor testing

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

slow model tests are running on our runners, where we have the bot token, but the tests needed to be decorated by `require_rad_token`.

I will update that, thank you for catching this.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36925). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: mistral3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mistral3']
quantizations: [] ...

The failing tests (slow) will be treated by

@Cyrilvallez (on slack)
> It's fine thanks! As I thought, they all blew the memory up :laughing: we'll use quantized version for the tests