noamaanMulla-03/transformers-issues huggingface dataset:
Fix Noise Computation for NEFTune During Packed Training #34659
Author: sambhavnoobcoder
Pull request: yes

### Problem Statement
When training with NEFTune using packed sequences, the noise scaling was not being computed correctly for each individual sequence. Instead, a single scaling factor was incorrectly used across the entire batch.

Fixes : #34659 

### Investigation and Identification
1. It was noticed that in packed training, multiple sequences with different lengths exist in the same batch, yet the noise magnitude was being determined by the total batch length.  
2. This caused the smaller sequences to receive too much noise, and the longer sequences to receive too little noise.

### Proposed Solution
Modify the NEFTune hook to:
• Distinguish between single-sequence vs. packed-sequence inputs.  
• Read sequence lengths from the attention mask where needed.  
• Scale the noise magnitude separately for each sequence length to ensure correct per-sequence noise levels.

### Implementation
• I updated the NEFTune forward hook to compute noise magnitudes on a per-sequence basis instead of the entire batch length.  
• I introduced new tests that cover both single-sequence and packed-sequence cases to ensure correctness and consistency.

### Screenshots (Before/After)
• Each sequence now has noise scaled to its own length, and the tests confirm proper magnitude of noise for both short and long sequences in the same batch.
<img width="620" alt="Screenshot 2025-01-20 at 11 58 33 PM" src="https://github.com/user-attachments/assets/2e1a2308-68e8-4697-9037-8c46b0e8694b" />


### Tests
1. Single-Sequence Test: Ensures noise is consistently applied when there is only one sequence in a batch.  
2. Packed-Sequence Test: Verifies that noise magnitudes are different for each sequence according to their individual sequence lengths.  

These tests ensure that existing behavior remains unchanged for non-packed sequences (backwards compatibility) and that correct per-sequence scaling is applied for packed batches.  
Run `python -m pytest tests/utils/test_neftune.py -v` from root directory to run the test file and reproduce & verify the changes yourself .  

**cc :**  @SunMarc  , @muellerzr  , @Rocketknight1 

Comments:
Can you check @ccdv-ai if this solves your issue ?
Add timm_wrapper support to AutoFeatureExtractor
Author: Factral
Pull request: yes
Labels: name: Vision

# What does this PR do?

A few days ago, the PR that adds timm_wrapper was merged #34564 [blog post]( https://huggingface.co/blog/timm-transformers) , enabling the use of timm models directly with Hugging Face interfaces, especially the Auto* ones. However, currently the AutoFeatureExtractor interface doesn't work with these models. This PR addresses that gap.

This PR adds timm_wrapper compatibility to AutoFeatureExtractor.from_pretrained(), enabling it to work with fine-tuned/trained timm model checkpoints.

Currently, when using a checkpoint from a trained/fine-tuned timm model (e.g., using examples/pytorch/image-classification/run_image_classification.py), AutoFeatureExtractor.from_pretrained() fails because timm_wrapper is not included in the interface. 

While there's a warning about missing preprocessor_config.json in checkpoints, users can manually add it to their checkpoint following examples like https://huggingface.co/Factral/vit_large-model/blob/main/preprocessor_config.json. This PR ensures AutoFeatureExtractor works properly when this file is present.

## Changes
- Added timm_wrapper to AutoFeatureExtractor interface
- Enables compatibility with timm model checkpoints when preprocessor_config.json is present
- Added is_timm kwarg in from_dict function

## Before submitting
- [x] Read contributor guidelines
- [x] Updated documentation to reflect changes
- [x] Added necessary tests for timm_wrapper functionality

## Who can review?
@amyeroberts @qubvel - as this relates to vision models and timm integration

Comments:
```python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModel

checkpoint = "timm/resnet18.a1_in1k"

model = AutoModel.from_pretrained(checkpoint)
processor = AutoProcessor.from_pretrained(checkpoint)

# load your image here
image = Image.new("RGB", (224, 224), (255, 0, 0))

inputs = processor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

for k, v in outputs.items():
    print(k, v.shape)

# last_hidden_state torch.Size([1, 512, 7, 7])
# pooler_output torch.Size([1, 512])
```
Add LlavaImageProcessor
Author: NielsRogge
Pull request: yes
Labels: name: Vision

name: Multimodal

name: Processing

# What does this PR do?

Fixes #33175. It adds the option to pad an image before applying the same preprocessing as `CLIPImageProcessor` based on the [original implementation](https://github.com/haotian-liu/LLaVA/blob/c121f0432da27facab705978f83c4ada465e46fd/llava/mm_utils.py#L152). This allows people to match the logits with the original implementation.

For now I decided to set `do_pad` to `False` by default as otherwise it would be a breaking change.

To do:

- [x] update padding method to take numpy as input and produce numpy as output
- [x] add equivalence test
- [x] add tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33191). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Is there any update for this feature? It seems very important for aligning implementations between Hugging Face and the official ones.
Supporting Padding in llava processor
Author: BiEchi
Pull request: no
Labels: name: Feature request

### Feature request

https://github.com/haotian-liu/LLaVA uses padding for pre-processing the images by default. Current transformers implementation does not support that.

### Motivation

Request per @NielsRogge at (https://huggingface.co/llava-hf/llava-1.5-7b-hf/discussions/26#66cf46a5a523b74b5f90fa72).

### Your contribution

I successfully reproduced logits after conversion if we add padding in the Transformers library.

Comments:
Thanks for the request @BiEchi! cc @zucchini-nlp 

Where should I add `do_pad` to achieve the same implementation as original LLaVA.

cc @NielsRogge as the PR to add `do_pad` is not merged yet

The main reason why my PR wasn't merged yet was because
1) there's an open comment to more thoroughly test the `background_color` argument
2) I didn't have equivalence between numpy and the original Pillow implementation for all different values of `background_color` (only got equivalent results for some values).

So @MilchstraB feel free to work further on my PR

Thanks for your reply! @NielsRogge 

However, if I process images using the `expand2square()` function and pass the processed results into `LlavaProcessor` (`CLIPImageProcessor`), will I get the same result as the original LLaVA?
```python
def expand2square(pil_img, background_color):
    width, height = pil_img.size
    if width == height:
        return pil_img
    elif width > height:
        result = Image.new(pil_img.mode, (width, width), background_color)
        result.paste(pil_img, (0, (width - height) // 2))
        return result
    else:
        result = Image.new(pil_img.mode, (height, height), background_color)
        result.paste(pil_img, ((height - width) // 2, 0))
        return result
    
image = Image.open(<image_path>).convert('RGB')

# processor is a instance of LlavaProcessor
image = expand2square(image, tuple(int(x*255) for x in processor.image_mean))
# Assume there are text prompts
processed_features = processor(images=[image], text=prompt, add_special_tokens=False)
```

Yes, you should normally get the same results. 

The problem is that [all image transformations in Transformers](https://github.com/huggingface/transformers/blob/main/src/transformers/image_transforms.py) need to be implemented as functions which take numpy arrays as input and produce other numpy arrays as output, hence one needs to translate the `expand2square` function to a numpy input, numpy output function with equivalent results.

Thank you very much! I have been looking for some time to see if I can do some quantitative experiments to verify the difference between the two.
Mimic `adamw_torch_4bit` and have `adamw_torch_8bit`
Author: fzyzcjy
Pull request: no
Labels: name: Feature request

### Feature request

Hi thanks for the lib! Currently there is `adamw_torch_4bit`, but I hope to mimic it to have a `adamw_torch_8bit` that uses 8bit torchao adamw.

The reason is that, I would like to use deepspeed cpu offload for the optimizer, and also use 8bit adamw. However, the 8bit one in current hf transformers does not support cpu, so I need to use the torchao one.

### Motivation

-

### Your contribution

yes, willing to PR

Comments:
cc @muellerzr for deepspeed/accelerate!

A PR for this would be great 🤗 cc @SunMarc 

Thanks! I will do that later.

Feel free to add it ! Let me know if you need any help 

Thanks! I will firstly mimic the 4bit one and see whether it works.

PR created: https://github.com/huggingface/transformers/pull/34993
Add flex_attn to diffllama
Author: muellerzr
Pull request: yes

# What does this PR do?

Should solve failing consistency checks

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker 

Comments:
(Modeling file is not changed)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35601). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually I don't think DiffLlama supports flex, does it? 🤔 It's still in the old format with classes

mmmm yeah looks like it but the pretrained mode in modular is a bit wrong in that case
Fix : BLOOM tie_word_embeddings in GGUF
Author: MekkCyber
Pull request: yes

# What does this PR do?
Related to this PR : https://github.com/huggingface/transformers/pull/35715, this PR adds `Bloom` as an exception as well to fix issues here : https://github.com/huggingface/transformers/actions/runs/12879032529/job/35906087498

It modifies the expected output of falcon-q2_k to match the expected output

## Who can review ?
@SunMarc

Comments:
Cleanup: continue the init refactor
Author: LysandreJik
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35167). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @LysandreJik @ArthurZucker any pointer to explain this cleanup?

oups sorry 
yeah this simplifies a lot the way we need to define the inits and removes the need for dummy
Fixed typo in autoawq version number in an error message for IPEX backend requirements.
Author: InfroLab
Pull request: yes

# What does this PR do?

Basically what is said in the title.

Fixes # (issue)


## Before submitting
- [ Check] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [N/A] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [N/A ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [N/A] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [N/A] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
feat: DeepSeekMoE
Author: llllvvuu
Pull request: yes
Labels: name: New model

# What does this PR do?

Upstream custom code from https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/modeling_deepseek.py to huggingface/transformers. This is not DeepSeek V2. The newly released DeepSeek-Prover-V1.5 runs on this architecture for example (though without MoE layers, so it is actually just Llama).

https://huggingface.co/models?other=deepseek


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32862). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @ArthurZucker , should be ready for review now, thanks!

friendly ping

Any update?

> Any update?

Sorry, I got busy, will get back to this though

still nice to have! 🤗
tokenizers v0.20 not supported
Author: nimz
Pull request: no
Labels: name: Core: Tokenization

name: Feature request

### Feature request

https://github.com/huggingface/tokenizers/releases/tag/v0.20.0 has been out for over a month, but there still isn't a compatible tag of transformers. Is there any plan to add this compatibility soon?

### Motivation

It would be great to be able to use the latest tags of the huggingface repos together.

### Your contribution

N/A

Comments:
cc @ArthurZucker 

Hey! Will be shipped in #32535 in a bit! 

This was fixed in [v4.45.0](https://github.com/huggingface/transformers/releases/tag/v4.45.0)

Closing!
Optimized set_initialized_submodules.
Author: LagPixelLOL
Pull request: yes

Fixed an insanely slow function `set_initialized_submodules`, when loading DeepSeek V3, since it has so many experts, the amount of `state_dict_keys` are huge and also there are a lot of `named_modules`.
```py
loaded_keys = {k.replace(f"{module_name}.", "") for k in state_dict_keys if k.startswith(f"{module_name}.")}
```
This line, especially `k.startswith(f"{module_name}.")`, gets called at least 1 million times, which is insanely slow, it takes at least 10 minutes runtime.

My optimization reduces the runtime to less than 10 seconds by not iterating through `state_dict_keys` and deleting `module_name` from the start for every `named_module`, instead, it prepends the `module_name` before the `module.state_dict()` keys.

cc: @ArthurZucker

Comments:
[i18n-ar] Translated file: `docs/source/ar/tasks/masked_language_modeling.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/tasks/masked_language_modeling.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

### Who can review?
@stevhliu , @MKhalusova , @abodacs, may you please review this PR?



Comments:
Thank you @abodacs  and @stevhliu 

Merge PRs #35192 and  #35197 to this PR

Thank You @abodacs  and @stevhliu
Improve modular documentation
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title! Modular has changed quite a bit and is more and more important, so it is important to have a nice and thorough documentation about it so that people can understand it quickly and use it efficiently.
In the end, it makes all of our lives better in the review process! 🤪


Comments:
Super nice, thanks for this really detailed doc!

I recently reworked the [modular transformers](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31757/en/modular_transformers) doc a bit as a part of the larger docs [refactor](https://github.com/huggingface/transformers/pull/31757), and would love to include this in it. I'll give it a more detailed review on Tuesday! 🤗  

Sure! Let me know how you want to proceed, i.e. if I should merge this one before or not 😉
[Mimi] update test expected values for t4 runners
Author: eustlb
Pull request: yes

# What does this PR do?

This PR fixes Mimi CI test. 
The test passes on A100 but not on T4, so it is due to hardware-dependent floating-point behaviour.
Here are the updated values for T4 runners 🤗



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove old `benchmark` code
Author: gante
Pull request: yes

# What does this PR do?

`/src/transformers/benchmarks`, not to be confused with the new `/benchmarks`, was [deprecated in March 2022](https://github.com/huggingface/transformers/pull/15848). We can also observe in git history that the corresponding files haven't been actively maintained since that date.

This PR removes the corresponding code, with the goal of not only removing outdated code and docs bloating our library, but also to prevent confusion with the new benchmarking code 🤗 

Comments:
Remove pyav pin to allow python 3.11 to be used
Author: CalOmnie
Pull request: yes

# What does this PR do?
Remove the two year old pin on `av==9.2.0`, this pin was made due to issue with the audio stream in `av==10.0.0`, this seems to not be an issue anymore sinve `av` is not used to extract audio streams anywhere.

Fixes #35803


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@vanpelt


Comments:
(I think this should be fine, but I'll keep an eye on our nightly CI after this is merged, just in case we get any new issues as a result - cc @ydshieh)