noamaanMulla-03/transformers-issues huggingface dataset:
Remove hardcoded slow image processor class in processors supporting fast ones
Author: yonigozlan
Pull request: yes

# What does this PR do?

A lot of processors has their `image_processor_class` hardcoded to the slow version. This means that when instantiating the processor with `use_fast=True` will return a slow image processor even if a fast one is available.
This PR fixes this for all models, while still returning a slow version by default when `use_fast` is not specified.
Another solution would be to set `image_processor_class` to `"AutoImageProcessor" `, but this might not be as backward compatible as adding a hardcoded fast class.

I've also pushed back the version to switch to fast image processors by default to 4.50, as because of the issue solved in this PR, a lot of processors have not been able to start this deprecation cycle, and because fast image processors are not present for every models yet.

Happy to discuss this push back! @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36266). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's also ask for community help to add more fast processors, when you feel the general API is ready!
[quants] refactor logic for modules_to_not_convert
Author: SunMarc
Pull request: yes

# What does this do 

This PR simplifies a bit how we calculate modules that shouldn't be converted for the different quantization scheme.
Also, it fixes an issue with `keep_in_fp32_modules` as it is now `Optional` with default value `None`. This means that we should check beforehand with we want to use extend method.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

LGTM Thanks !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36672). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Appreciate the fixes!

:))
Refactor siglip2 fast image processor
Author: yonigozlan
Pull request: yes
Labels: name: Vision

name: Processing

# What does this PR do?
Refactor siglip2 fast image processor to be more consistent with others and reduce diffs.
Also introduce `unused_kwargs` in `BaseImageProcessorFast` for processors that don't support all the default kwargs. Here SigLip doesn't support the `size` kwarg for example, which might be surprising for users considering it is supported by (almost) all other image processors. adding it in `unused_kwargs` allows to show a warning instead of raising an error.
Cc @qubvel 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36406). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Can you please also check if we can make docs consistent for slow and fast image processors?

<img width="985" alt="Screenshot 2025-02-25 at 19 13 44" src="https://github.com/user-attachments/assets/8ca5acd8-b479-4b87-8519-093b4ff5159c" />
<img width="1070" alt="Screenshot 2025-02-25 at 19 13 23" src="https://github.com/user-attachments/assets/b1f37fd3-c78e-40d7-9bb3-ae21b8e51ba7" />




> Can you please also check if we can make docs consistent for slow and fast image processors?

Yes indeed, that's a problem with all fast processors really. I'm trying to strike a good balance between having meaningful and correct docs, and not having the same image processing kwargs docs duplicated in every processors.
Maybe we can have a decorator that allows to override the default description of a kwarg if we need to?
In any case, I'll try to come up with something better than the current situation, but I'll probably open another PR for that

Sounds good, decorator might be a good approach!

Also let's use as much modular as possible! (will easy all futur refactoring)
Saving Transformers model as binary instead of `safetensors` format?
Author: StatsGary
Pull request: no

I am trying to Torchserve (https://pytorch.org/serve/) a Transformers model to be deployed as a prediction API on GCP as a custom prediction service. 

However, torchserve requires that the model be in either `.bin` format or `.pt` format. Is there a way to deactivate safetensors and return the .bin PyTorch file?

I have tried to use torch.save(model.state_dict(), PATH) but have been unsuccessful thus far? Anyone encountered similar issues?


Comments:
➡️💡 SOLUTION ➡️💡 

I dig some digging of the parameters of the `save_pretrained` and `trainer` methods and you can actually 🔧 turn off the storing of the safetensors format models, by using the following example: 
```python
model.save_pretrained(<PATH_TO_SAVE_MODEL>, safe_serialization=False) #Replace path
```



Could luck trying to secure your servers from malicious actors ! :)



@StatsGary With the solution you gave, Post saving the model it creates three files,

1. config.json
2. generation_config.jso
3. pytorch_model.bin

1. Can you help me how you used these to import and deploy in GCP vertex model registry?
2. Have you use pre-build container like pytorch or went with custom container ?
- If prebuilt contained - How did you convert .bin file to .mar ?
- If custom container - Can you share your version or example image please ?
Refactor object-detection models
Author: qubvel
Pull request: yes
Labels: name: Vision

# What does this PR do?`

Improves code readability for object-detection models, should be merged after 
 - #36460 
 
**Note:**

> RT-DETR and RT-DETR-v2 are refactored, for other models just Freezed BatchNorm2d module updated.

**RT-DETR refactoring:** 

  - Make code closer to `transformers` standards (no one-letter vars, common var names, ...)
  - Remove some unused outputs (seems not necessary, but add overhead on implementation and docs)
  - Clean up signatures from unused arguments (where it's possible)
  - Better comments and shape comments
  - Add default docstring for object-detection
  - Unify RT-DETR and RT-DETRv2 implementations (simpler modular file for v2)

**In addition:**
 
 - Fix CoreML export for RT-DETR (avoid 6D tensors in deformable_attention)
 
Mostly, the changes are backward compatible, but some **internal** modules' signatures have changed. I suspect RT-DETR **internal** modules are used on their own somewhere outside the `transformers` repository. A quick search on GitHub shows only transformers forks, and I have never seen custom code samples on HF Hub for object detection. Therefore, I think we can change them to establish better standards for upcoming models such as D-Fine (a state-of-the-art real-time object detector) and RT-DETRv3, both of which are based on RT-DETR.

### Test changes

I tried to keep tests untouched, but there are a few modifications:
 - the number of outputs in ModelOuput is reduced

## Who can review?

cc @SangbumChoi @jadechoghari as model contributors, in case you want to have a look. Please let me know if anything is missing / incorrectly re-implemented

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update config.torch_dtype correctly
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes an issue with dtype when loading models like CLIPTextModelWithProjection that init the model using `_from_config`.  

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36679). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Users started noticing smth if wrong (https://github.com/huggingface/transformers/issues/36686) 😆
Fix slicing for 0-dim param
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes a slicing issue that we have with 0-dim tensors (both with param from bin file and safetensors pyslice). We can do [:] but [...] works instead. Thx [fxmarty-amd](https://github.com/fxmarty-amd) for finding this solution.

cc @DN6 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36580). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Changing the test model in Quanto kv cache
Author: MekkCyber
Pull request: yes

# What does this PR do?
Change the model used for testing the kv cache implementation in quanto, the previous model `llama2-7b-hf` is very old and is causing an error related to rotary_emb because they were moved to base model and are not part of attention anymore https://github.com/huggingface/transformers/pull/32135
## Who can review ?
@SunMarc 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36670). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix wandb hp search unable to resume from sweep_id
Author: bd793fcb
Pull request: yes

# What does this PR do?

When resuming an HP search, set environment variables for wandb's sweep agent to locate the existing records.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @LysandreJik given history in #14582

Comments:
@sunmarc @mekkcyber @muellerzr - this relates to our WandB integration, who's the right person to ping for it?

Not sure, maybe @ayulockin can have a look or ping another person 

Thanks for opening this PR. @bd793fcb can you share a wandb sweep page link?

cc: @parambharat can you do a quick review?

It looks good to me!

https://wandb.ai/d2n/wandb_test/sweeps/qzkss8wi
Upgrading torch version and cuda version in quantization docker
Author: MekkCyber
Pull request: yes

# What does this PR do?

Upgrading torch version to 2.6.0 and cuda version to 12.1. 
Switching to `gptqmodel` instead of `autogptq`
Some changes in the order of installation to avoid re-installing different torch versions by some libs

## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36264). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Feature] Support using FlashAttention2 and FlashVarlenAttention on Ascend NPU
Author: FightingZhen
Pull request: yes

# What does this PR do?
Package `flash-attn` is not supported on Ascend NPU, **even can not be installed**. In that situation, we can not use FlashAttention2 or FlashVarlenAttention with transformers natively 😞 

To solve this problem, we find from Ascend community [Ascend FlashAttentionScore Document](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0027.html) that Ascend `torch_npu` has provided FlashAttention2 api `torch_npu.npu_fusion_attention` on Ascend NPU, and has provided solutions for supporting FlashAttention2 and FlashVarlenAttention on Ascend NPU, which can play the same role as the apis in package `flash-attn`.

Therefore, this PR is committed for supporting using FlashAttention2 and FlashVarlenAttention on Ascend NPU. 

## Main Modification
1. Create a new file `src/transformers/utils/npu_flash_attention_utils.py` to organize necessary functions for using FlashAttention and FlashVarlenAttention on Ascend NPU. Part of codes are copied from [flash-attn/bert_padding.py](https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/bert_padding.py) due to package `flash-attn` unavailable on Ascend NPU.
2. patch `index_first_axis`, `pad_input`, `unpad_input`, `flash_attn_func` and `flash_attn_varlen_func` 5 functions online to replace corresponding functions in package `flash-attn` when detecting package `torch_npu` available.
3. Update models in `src/transformers/models` to activate FlashAttention2 or FlashVarlenAttention and set correct mask strategy when detecting package `torch_npu` available.

Fixes: 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).
Set evaluation and checkpointing defaults to 'epoch' and reduce loggi…
Author: chaymaeelaattabi
Pull request: yes

Hey everyone! 👋


I noticed that the default TrainingArguments settings don’t evaluate or save checkpoints unless explicitly set, which might not be ideal for most users. To improve the out-of-the-box experience, I’ve made a few tweaks:

Evaluation now runs at the end of each epoch (evaluation_strategy="epoch") instead of being disabled by default.
Checkpoints are saved at the end of each epoch (save_strategy="epoch") instead of every X steps.
Reduced logging steps from 500 to 100, so training logs appear more frequently and give better visibility.

Comments:
cc @sunmarc @muellerzr

save strategy default value is `steps`
`save_strategy (`str` or [`~trainer_utils.SaveStrategy`], *optional*, defaults to `"steps"`):`
Append best model checkpoint with active adapter when not default
Author: Thomas26948
Pull request: yes

# What does this PR do?

When a `PeftModel` is instantiated with an `adapter_name` different than `"default"`, the `Trainer` is unable to find the `best_model_checkpoint` because the path is set without the `adapter_name`.
This happens when `load_best_model_at_end=True`



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
cc @SunMarc @muellerzr
Change Qwen2_VL image processors to have init and call accept the same kwargs
Author: yonigozlan
Pull request: yes

# What does this PR do?

As the title says. This follows this PR https://github.com/huggingface/transformers/pull/36186, and fixes an issue mentioned in vLLM: https://github.com/vllm-project/vllm/issues/13143#issuecomment-2656958661

cc @ArthurZucker @hmellor 

<!--
## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36207). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix dtype for params without tp_plan
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Otherwise forward pass fail with errors such as `[rank2]: RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::BFloat16 != c10::Half`


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Do not merge: CI image for `pytest-rerunfailures`
Author: gante
Pull request: yes

# What does this PR do?

See title

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36427). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Image pushed for #36694
Add loading speed test
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. This should avoid regresions


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36671). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ok, spawned a t4 instance, and I can confirm that it works on it. Merging!
fix: fsdp sharded state dict wont work for save_only_model knob
Author: kmehant
Pull request: yes

# What does this PR do?

Fixes #36626


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- trainer: @muellerzr and @SunMarc
@ArthurZucker @Rocketknight1 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

@SunMarc @muellerzr requesting review on this nit fix. Thanks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36627). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
save_only_model with FSDP throws FileNotFoundError error
Author: kmehant
Pull request: no
Labels: name: bug

### System Info

* Transformers (4.50.0.dev0) main branch at commit [94ae1ba](https://github.com/huggingface/transformers/commit/94ae1ba5b55e79ba766582de8a199d8ccf24a021)
* (also tried) transformers==4.49
* python==3.12
* accelerate==1.0.1

### Who can help?

@muellerzr @SunMarc @ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run a simple FSDP training with state dict type `SHARDED_STATE_DICT` with `save_only_model` option.

```
  File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 2639, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval, start_time)
  File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 3092, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial)
  File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 3211, in _save_checkpoint
    self.state.save_to_json(os.path.join(output_dir, TRAINER_STATE_NAME))
  File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_callback.py", line 144, in save_to_json
    with open(json_path, "w", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './train_output/checkpoint-1/trainer_state.json'
```

### Expected behavior

Report the incompatibility early in the training lifecycle rather erroring out at the first checkpoint save event.

Comments:
Since the `save_only_model` is meant to be consumable with `from_pretrained` (like mentioned in the docs), it would be good to capture this incompatibility with FSDP sharded_state_dict at early stage of the training lifecycle. PR: https://github.com/huggingface/transformers/pull/36627