noamaanMulla-03/transformers-issues huggingface dataset:
Disable inductor config setter by default
Author: HDCharles
Pull request: yes

This was requested in the huggingface x pytorch slack by @BenjaminBossan 

This setting for Torchao should be disabled by default to avoid side effects (https://github.com/pytorch/ao/blob/24c966cb8931507ef389715bc5f19a11c28b0484/torchao/quantization/utils.py#L574)

  torch._inductor.config.coordinate_descent_tuning = True
  torch._inductor.config.coordinate_descent_check_all_directions = True
  torch._inductor.config.force_fuse_int_mm_with_mul = True
  torch._inductor.config.fx_graph_cache = True
  torch._inductor.config.triton.unique_kernel_names = True
  torch.set_float32_matmul_precision("high")
  
  though the default performance will be worse, these settings can be set manually.
  
  @SunMarc   

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

If the recommended settings generally work better, would it make sense to document that somewhere and tell users how to enable them?

> If the recommended settings generally work better, would it make sense to document that somewhere and tell users how to enable them?

I've added a note to the .md file as well.

note: the failing test seems to be on trunk or at least unrelated, other PRs are also failing with teh same unrelated error: https://github.com/huggingface/transformers/pull/36610

looks like 2 HTTPS errors?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is ready to land @SunMarc @ArthurZucker
fix tiktoken convert to pass AddedToken to Tokenizer
Author: itazap
Pull request: yes

fix for https://github.com/huggingface/transformers/issues/36438

Reason for the issue: that a list of tokens is expected, not a dict of tokens / ids, which is what tiktoken's encoding_name_for_model provides. This converter function was designed for more 'behind the scenes' use I guess, but we can make this fix to support it better for direct use.

We don't explicitly test convert_slow_tokenizer, so we can test equivalence for gpt2 (since it's common between tiktoken and transformers)


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36566). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
num_items_in_batch unexpected in vision encoder decoder
Author: eljandoubi
Pull request: no
Labels: name: bug

### System Info

![Image](https://github.com/user-attachments/assets/31ce5f30-1a51-4782-8739-823b8a6e98c9)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

do forward on vision encoder decoder like donut.

### Expected behavior

model get trained

![Image](https://github.com/user-attachments/assets/19b67b65-8093-40da-909f-b26f46e50c47)

Comments:
Pass num_items_in_batch directly to loss computation
Author: eljandoubi
Pull request: yes

# What does this PR do?

Fixes #36744

Models:

- vision models: @amyeroberts, @qubvel



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Thanks @eljandoubi
Fix fp16 ONNX export for RT-DETR and RT-DETRv2
Author: qubvel
Pull request: yes
Labels: name: Vision

# What does this PR do?

Fix fp16 ONNX export for RT-DETR and RT-DETRv2, related to
 - https://github.com/huggingface/optimum/pull/2201

Comments:
run-slow: rt_detr, rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36460). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: rt_detr, rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...

run-slow: rt_detr, rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...

cc @xenova if you have bandwidth
(ugly) Use  `parallelism=4` for `check_repository_consistency`
Author: ydshieh
Pull request: yes

# What does this PR do?

I tried to make it more flexible to have `continue` with a new dynamically created config file, but `CircleCI` doesn't allow multiple `continue` at the same time.

The current workaround is ugly and not scalable, but it works anyway. 

Let me know if you would accept such solution.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36197). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ok, i will make a final check . thanks for the review
Fix Batch Size Mismatch When Using `crops_n_layers` in `mask-generation` Pipeline #35530
Author: sambhavnoobcoder
Pull request: yes
Labels: name: bug

name: Vision

## Overview
This pull request addresses a batch size mismatch issue that arises when users set the parameter `crops_n_layers > 0` in the Segment Anything Model (SAM) powered `mask-generation` pipeline. Previously, attempting to run the pipeline caused a runtime error, as the model would receive mismatched batch dimensions for image embeddings and input points. fixes issue #35530  . @Rocketknight1 kindly review this at the earliest . 

Fixes #35530 

## The Problem
When segmenting images in "segment everything" mode with cropping enabled (`crops_n_layers=1` or more), the pipeline splits the image into multiple crops. It generates a grid of points for each crop to compute masks in parallel. In some scenarios, particularly where the input is not batched before the cropping procedure, the pipeline would produce additional crops as separate batch entries. This led to:
• A mismatch in the batch dimension between the generated image embeddings and the corresponding point embeddings.  
• A ValueError reporting different batch sizes, preventing the pipeline from completing the segmentation process.

## Proposed Solution
We introduced internal improvements that unify the handling of image crops and points so they remain synchronized:  
• Ensured that all crops from a single original image preserve the same batch index.  
• Correctly merged the batched embeddings back together, with consistent shapes, so each batch item has the correct number of crops and associated point embeddings.

## Approach
1. Revisited the internal crop generation logic to ensure that each cropped region is tracked as part of the same batch, instead of being treated as separate entries.  
2. Adjusted how point embeddings are grouped per crop, clarifying that they should share the original batch dimension.  
3. Reworked the post-processing step to handle the final combination of crops so the final output remains consistent with the user’s single input image.

## Screenshots
### Before

<img width="1128" alt="Screenshot 2025-01-11 at 9 12 15 PM" src="https://github.com/user-attachments/assets/4eee358f-b61a-41cc-bfd9-a4209774570f" />

### After 

<img width="1127" alt="Screenshot 2025-01-11 at 9 13 13 PM" src="https://github.com/user-attachments/assets/aa60d792-1d8f-4b56-ba42-bf5c86228b7d" />

Prior to this fix, the `Before` screenshot would visibly show an error stack trace indicating the batch mismatch. The `After` screenshot demonstrates a smooth run, producing valid masks across cropped regions.


Comments:
cc @qubvel can you take a first look at this? You can ping me if there's anything in the pipeline code you need me to check!

@qubvel i have addressed and resolved all the comments , as well as added new tests in accordance to your suggestions . Please do tell me if there is something else this PR needs before it can get merged , i'll get it done . 

gentle ping @qubvel!
[qwen2 audio] remove redundant code and update docs
Author: gante
Pull request: yes

# What does this PR do?

Carved from #36238: Removes redundant code and updates outdated docs in `qwen2_audio`.

In a nutshell, the attention layers were copied from `whisper`. However, in `qwen2_audio`, the attention layers are exclusively used in the encoder, and thus never use cache. Removing cache-related code there prevents us from having to refactor every time we revisit caches, and results in more readable code.

The following bits were also updated in this PR:
- Missing/outdated docs
- Redundant overwrites

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36282). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[modular] Sort modular skips
Author: gante
Pull request: yes

# What does this PR do?

Easier visual inspection of the skipped checks: the model names are now sorted (previously, they were in topological order)

Example:
![Screenshot 2025-02-20 at 15 50 29](https://github.com/user-attachments/assets/fcb36f91-b143-4349-ae55-ef52782a8e4b)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36304). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate] clarify docstrings: when to inherit `GenerationMixin`
Author: gante
Pull request: yes

# What does this PR do?

From the discussion in #36570: clarify docstrings regarding when should a model inherit `GenerationMixin`. This information was not present anywhere.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update min safetensors bis
Author: SunMarc
Pull request: yes

# What does this PR do?

ci is behaving strangely for the [PR](https://github.com/huggingface/transformers/pull/36767)  so i opened this to see if it fixes the issue. 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update min version of safetensors to allow integer indexing
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR update the minimum version of safetensors to 0.4.3 in order to have support for integer indexing which is required in our TP integration. 
PR : https://github.com/huggingface/safetensors/pull/440 
release : https://github.com/huggingface/safetensors/releases/tag/v0.4.3



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36767). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing as I was able to merge the pr above
Fix import for torch 2.0, 2.1 - guard typehint for "device_mesh" 
Author: qubvel
Pull request: yes

# What does this PR do?

Device mesh was added since torch 2.2 and not available in earlier versions, while `transformers` support torch >= 2.0

Reported in #36754

## Who can review?

cc @Cyrilvallez 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36768). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
GPT2 repetition of words in output
Author: vpandya-quic
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-5.15.0-122-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NO

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
@pytest.mark.parametrize("dtype", [torch.float16, torch.float32])
def test_gpt2_cpu_inductor(dtype):
    tokenizer = AutoTokenizer.from_pretrained("gpt2")
    model = AutoModelForCausalLM.from_pretrained("gpt2").to(dtype)
    prompt1 = "GPT2 is model developed by OpenAI"

    # run on CPU
    input = tokenizer(prompt1, return_tensors="pt")
    input_ids1 = input.input_ids
    attention_mask = input.attention_mask
    gen_tokens1 = model.generate(
        input_ids1,
        attention_mask = attention_mask,
        max_new_tokens=30,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )
    gen_text1 = tokenizer.batch_decode(gen_tokens1)[0]
    print(gen_text1)

    import torch._inductor.config as inductor_config
    inductor_config.inplace_buffers = False

    model.transformer.wte.forward = torch.compile(
        model.transformer.wte.forward, backend="inductor", fullgraph=False
    )

    gen_tokens_cpu1 = model.generate(
        input_ids1,
        attention_mask = attention_mask,
        max_new_tokens=30,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

    gen_text_cpu1 = tokenizer.batch_decode(gen_tokens_cpu1)[0]

    assert gen_text1 == gen_text_cpu1
```


### Expected behavior

For above test I see output as 
`GPT2 is model developed by OpenAI and is based on the OpenAI-based OpenAI-based OpenAI-based OpenAI-based OpenAI-based OpenAI-based Open`
is this expected behavior? 
@ArthurZucker could you please explain why the output is like this?

Comments:
Hi @vpandya-quic, GPT-2 is a very old model at this point, and `gpt-2` is the smallest checkpoint. Its text generations were often repetitive or garbage. This is not a bug! 

If you want higher quality output, I highly recommend a newer model. If you want a tiny model like `gpt2`, you can use [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), but there are also much larger and smarter models available, such as the very recent Gemma3: https://huggingface.co/google/gemma-3-27b-it . Try following the instructions in their model cards!
Fix ONNX export for sequence classification head 
Author: echarlaix
Pull request: yes

Since https://github.com/huggingface/transformers/pull/35911, some models with a sequence classification head will be exported to ONNX with an ArgMax operator in int64, making the resulting model incompatible with ORT. Here we set `token_indices` to int32, fixing :

```
NOT_IMPLEMENTED : Could not find an implementation for ArgMax(13) node with name '/ArgMax'
``` 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36332). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker @Rocketknight1  let me know if that works for you 

(If you don't want to do that, though, we can just merge this PR as-is. The change I suggested is just future-proofing rather than strictly necessary!)

Yeah, let's just merge this as-is! If the int64/int32 inputs thing becomes a problem when exporting later we can fix it then.
FEAT / Trainer: Add adamw 4bit optimizer
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR adds the 4-bit optimizer from torchao library into HF Trainer. For now, it requires the main branch of `torchao` and torch >=2.3 (maybe we can wait a bit before merging). For those who wants to try, you can pass `optim="adamw_torch_4bit"` in `TrainingArguments`. 

Since we already have the 8-bit optimizer from bnb that works well, i'm not adding it. 

Related thread : https://x.com/marksaroufim/status/1809398186198593566

cc @muellerzr as you might be interested

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31865). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

There's also an `AdamWFp8` btw and it's the fastest one we've found when the HW supports it https://github.com/pytorch/ao/tree/main/torchao/prototype/low_bit_optim#benchmarks

Also cc @gau-nernst this is very exciting!

> There's also an AdamWFp8 btw and it's the fastest one we've found when the HW supports it https://github.com/pytorch/ao/tree/main/torchao/prototype/low_bit_optim#benchmarks

Nice ! I'll add it in a separate PR ! 

Heads up @SunMarc we just released torchao 0.4! https://github.com/pytorch/ao/releases/tag/v0.4.0

Nice ! I'll merge it as soon as we merge the torchao quantization PR in transformers as there is some overlap ! 

Hi, is it OK if I PR to add the 8bit counterpart? (see https://github.com/huggingface/transformers/issues/34893 for details) Thanks!
add DeepSpeed tensor parallel initialization.
Author: inkcherry
Pull request: yes

Initialization for DeepSpeed tensor parallel+zero optimizer, which needs to be done before the optimizer initializes the model weights. The related APIs are already supported in DeepSpeed.  https://github.com/deepspeedai/DeepSpeed/pull/6922  @muellerzr

cc DeepSpeed team: @tjruwase
move to https://github.com/huggingface/transformers/pull/36825


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks for the review!  @muellerzr 
accelerate includes modifications related to the ```dataloader``` and ```saving``` and works together with this feature.
https://github.com/huggingface/accelerate/pull/3390

thanks for the review. @muellerzr!  The related PR has already been merged: [huggingface/accelerate#3390](https://github.com/huggingface/accelerate/pull/3390). Is this PR ready for merge? Thanks!









sorry to bother you again and I hope this doesn't disturb you : )  the doc https://github.com/deepspeedai/DeepSpeed/pull/7151  is ready but pending with this pr,  I would appreciate it if this PR can be merged.  Thank you very much! 
FYI  @muellerzr  @ArthurZucker @shethaadit @tjruwase  


> LGTM ! Is the CI stalling @muellerzr ?

Not quite sure, it's been hanging on Checking for the ability to merge automatically... for several hours.
![image](https://github.com/user-attachments/assets/ceca0d47-f6bb-404d-a65b-8d2373becf5d)
 It's strange. Not sure if it's due to the CI or some GitHub service issue (https://www.githubstatus.com/incidents/lg4s05t6ttxb). Force pushing to refresh also results in it hanging ):
 
 move to https://github.com/huggingface/transformers/pull/36825 fix this hang issue.

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

@inkcherry I think we can close this one in favor of https://github.com/huggingface/transformers/pull/36825 yes?

> @inkcherry I think we can close this one in favor of #36825 yes?

yes, still hang now, let me close this one.
Support loading Quark quantized models in Transformers
Author: fxmarty-amd
Pull request: yes

This PR adds the ability to load Quark models through `PreTrainedModel.from_pretrained` by integrating Quark library through a quantizer in Transformers.

Example:

```python
model_id = "EmbeddedLLM/Llama-3.1-8B-Instruct-w_fp8_per_channel_sym"
model = AutoModelForCausalLM.from_pretrained(model_id)
model = model.to("cuda")
```

AMD is developing its in-house quantizer (https://quark.docs.amd.com/latest/) released under MIT license, which supports a broad range of quantization pre-processing, algorithms, dtypes and target hardware.

This PR can be tested with the release version of Quark [on PyPI](https://pypi.org/project/amd-quark/):
```bash
pip install amd-quark
```

## Why integrating in Transformers?

Quark is helpful in exploring a wide range of quantization strategies, and integrating in Transformers would mean easy binding with the broader OSS ecosystem, e.g. including `lm-evaluation-harness` or Transformers as a vLLM backend.

In the future, we plan to broaden the support of data types and algorithms in Quark to support current and future hardware, and usable through Transformers.

## To who goes the maintenance burden?

@fxmarty-amd @kewang-xlnx @BowenBao are committed to support Quark support development and maintenance in Transformers.

## Alternative

https://github.com/huggingface/transformers/pull/35915 -> could be an option, but does not really allow integrating with external OSS libraries, including native Transformers (without pulling our own code registering quant config / quantizer).

Comments:
cc @SunMarc @MekkCyber for quantization!

@SunMarc let me fix the conflicts that emerged. Happy to share more context if needed!

Yeah, I'll review it after the conflits are fixed. Sorry for the delay ;)

Thanks for this integration @fxmarty-amd ! Only two very small nits

Hi @fxmarty-amd! Can you resolve the conflicts please, otherwise good to go from my side

@MekkCyber thanks a lot, will resolve the conflicts once https://github.com/huggingface/transformers/pull/36580 is merged.

I'll be away for a bit, so @kewang-xlnx will take over this PR. cc @kewang-xlnx

Thanks @fxmarty-amd, merged with main and resolved conflicts.

@MekkCyber thanks for reviewing! Please take a look.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36372). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @BowenBao ! I just left some small comments and questions
DeepSpeed tensor parallel+ZeRO
Author: inkcherry
Pull request: yes

move from  https://github.com/huggingface/transformers/pull/36114 , Initialization for DeepSpeed tensor parallel+ZeRO optimizer for 2d training,  cc @muellerzr

The original PR workflow seems to have encountered some issues and cannot be resolved by refreshing.









Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

LGTM but more for @muellerzr