noamaanMulla-03/transformers-issues huggingface dataset:
Crash when running Llama4 on transformers-4.54.1
Author: IKACE
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.54.1
- Platform: Linux-6.8.0-1032-aws-x86_64-with-glibc2.35
- Python version: 3.10.0
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.5.3
- Accelerate version: 1.8.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.6.0+cu124 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: distributed
- Using GPU in script?: yes
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I encountered process crash when running Llama-4 with TP=8 on newest `transformers` library 4.54.1. It either gives direct `SIGTERM` from any rank of the GPUs or gives error `runtimeerror: aten.mul.tensor: got mixed torch.tensor and dtensor, need to convert all torch.tensor to dtensor before calling distributed operators!`. I have verified that fallback to `transformers` library version 4.53.3 works with no errors.

Please see the attached script I am running. You can run with command `torchrun --standalone --nproc-per-node 8 run_llama4.py`.


```
# run_llama4.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# enable tensor parallelism
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-4-Scout-17B-16E-Instruct",
    torch_dtype=torch.bfloat16,
    tp_plan="auto"
)

# prepare input tokens
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-4-Scout-17B-16E-Instruct")
prompt = "The University of Washington is"
inputs = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

max_new_tokens = 50
generated = inputs
past_key_values = None
with torch.no_grad():
    for step in range(max_new_tokens):
        if step > 0:
            inputs = next_token
        outputs = model(input_ids=inputs, past_key_values=past_key_values, use_cache=True)
        past_key_values = outputs.past_key_values  # cache past key values for next iteration
        logits = outputs.logits  # shape: [batch, seq_len, vocab]
        next_token_logits = logits[:, -1, :]  # only use last token's logits
        next_token = torch.argmax(next_token_logits, dim=-1).unsqueeze(-1)  # greedy decoding
        generated = torch.cat([generated, next_token], dim=-1)

decoded = tokenizer.batch_decode(generated)

print("Generated text:", decoded[0])

```

### Expected behavior

No runtime error is encountered during execution.

Comments:
Absolutely! a quick fix is to remove the `layer-norm` form the `tp_plan`! 

cc @matej-svejda and @3outeille 

@IKACE fixed with mentionned PR
Fix fsdp for generic-task models
Author: Cyrilvallez
Pull request: yes
Labels: name: for patch

# What does this PR do?

As per the title. Fixes https://github.com/huggingface/transformers/issues/39977
Inheriting from `ABC` would result in this kind of errors when using fsdp: `TypeError: __class__ assignment: 'FSDPQwen2ForTokenClassification' object layout differs from 'Qwen2ForTokenClassification'`.

As having them inherit from `ABC` was sugar to indicate they should only be used through proper inheritance, it's safe to remove it

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40191). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add repr to EncoderDecoderCache
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Without it, it would inherit the one from Cache, and fail because `self.layers` does not exist in `EncoderDecoderCache`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40195). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typos
Author: cyyever
Pull request: yes

# What does this PR do?

Fix some typos in comments.
 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: efficientloftr, mask2former, maskformer, oneformer, qwen2_vl

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40175). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix various Pylint warnings
Author: cyyever
Pull request: yes

# What does this PR do?

Fix various Pylint warnings about unused comments, bad indentation and imports.

Comments:
(This looks like the CI issues are back - try rerunning or rebasing and ping me once it's ready)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: convnext, convnextv2, cvt, deepseek_vl, deepseek_vl_hybrid, encoder_decoder, janus, led, longformer, mobilevit, patchtsmixer, patchtst, perception_lm, reformer, regnet

@Rocketknight1 CI is green.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40107). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Running jobs with SkyPilot
Author: alex000kim
Pull request: no
Labels: name: Feature request

### Feature request

There are SLURM examples in your docs e.g. https://github.com/huggingface/transformers/blob/31b6e6e1dac0d32f74ec5cd6b3c1868534ccd7b5/docs/source/en/deepspeed.md 

but I am a [SkyPilot](https://docs.skypilot.co/en/latest/docs/index.html) user and would like to have those as well. 


### Motivation

SkyPilot is used by/integrated with most major hyperscalers and neoclouds and it'd be good to have SkyPilot examples in your docs. 

- https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemorun/guides/execution.html
- https://aws.amazon.com/blogs/machine-learning/streamline-machine-learning-workflows-with-skypilot-on-amazon-sagemaker-hyperpod/
- https://gke-ai-labs.dev/docs/tutorials/workflow-orchestration/skypilot/cross-region-capacity-chasing/
- https://cloud.google.com/blog/topics/hpc/salk-institute-brain-mapping-on-google-cloud-with-skypilot
- https://nebius.com/blog/posts/nebius-ai-cloud-skypilot-integration
- https://a16z.com/announcing-our-latest-open-source-ai-grants/


### Your contribution


If you want to can pick up the task and create a PR. LMK, thanks!  

Comments:
@alex000kim Ok I will take this.Thanks!

thanks, lmk if I can help.
Remove _prepare_flash_attention_from_position_ids
Author: cyyever
Pull request: yes

# What does this PR do?

This function is deprecated and none of its use can be found.

Comments:
@Rocketknight1 Done

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40069). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1 Ping

Merging, and sorry for the delay!
Fix GPT-OSS `swiglu_limit` not passed in for MXFP4
Author: danielhanchen
Pull request: yes
Labels: name: for patch

`swiglu_limit = 7.0` was not passed in when doing MXFP4 inference when using OpenAI's triton_kernels library.

Comments:
cc @arthurzucker or possibly @MekkCyber 

run-slow: mxfp4, gpt_oss

This comment contains run-slow, running the specified jobs: 

models: ['models/gpt_oss']
quantizations: ['quantization/mxfp4'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40197). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Well, the testing suit is fucked up cc @MekkCyber or @SunMarc for when you come back, logits will be different but merging!
make model doc device agnostic
Author: yao-matrix
Pull request: yes

As `torch` and `transformers` already support multiple devices("cuda", "xpu" etc) as their built-in devices, maybe it's a good time to extend model doc examples to a device-agnostic one rather than binding to "cuda". I am add a `infer_device` API in utils to automatically detect available device in runtime.

@stevhliu , pls help review and comment, thx very much.

Comments:
@stevhliu , done, thx very much.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Feat/add gpt oss sequence classification
Author: robin-ede
Pull request: yes

# What does this PR do?

This PR implements `GptOssForSequenceClassification` for text classification tasks.

**Key Changes:**
- ✅ **New Model Class**: Added `GptOssForSequenceClassification` inheriting from `GenericForSequenceClassification` and `GptOssPreTrainedModel`
- ✅ **Consistent Implementation**: Implemented in both `modeling_gpt_oss.py` (generated) and `modular_gpt_oss.py` (source) files
- ✅ **Auto-Model Integration**: Added to `modeling_auto.py` registry for automatic model discovery
- ✅ **Test Infrastructure**: Added `sequence_classification_class` to test configuration and enabled all sequence classification tests
- ✅ **Pipeline Support**: Configured for `text-classification` and `zero-shot` pipelines

**Implementation Details:**
- Follows HuggingFace's standard pattern used by other MoE models (Qwen2MoE, Mixtral)
- Uses `GenericForSequenceClassification` mixin for automatic classification head and forward pass logic
- Maintains compatibility with existing GptOss architecture and weight initialization
- Supports both single-label and multi-label classification scenarios

**Testing:**
- All sequence classification tests pass (3/3): basic, single-label, and multi-label variants
- Model instantiation and forward pass work correctly with proper logits shape `[batch_size, num_labels]`
- Integration with HuggingFace pipelines confirmed

Fixes #40018

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
      **Link:** https://github.com/huggingface/transformers/issues/40018
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
      **Yes:** Added `GptOssForSequenceClassification` to `docs/source/en/model_doc/gpt_oss.md` for auto-generated documentation
- [x] Did you write any new necessary tests?
      **Yes:** Added `sequence_classification_class = GptOssForSequenceClassification` to test configuration, enabling all sequence classification tests

## Who can review?

@ArthurZucker - This adds a new text model class following standard patterns for sequence classification

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gpt_oss

@ArthurZucker all done!

Hey sorry by the time I got back I merged another PR that did the same! Super sorry!

No worries!
Implemented fast image processor for VitPose
Author: msminhas93
Pull request: yes

# What does this PR do?

This PR implements a fast PyTorch image processor for VitPose (`VitPoseImageProcessorFast`) that extends the existing `BaseImageProcessorFast` class. The new processor provides optimized image processing capabilities specifically designed for pose estimation tasks using the VitPose model.

## Key Features

- **Fast PyTorch-native implementation**: Leverages PyTorch operations for efficient image processing
- **Affine transformations**: Supports image cropping and resizing based on bounding boxes
- **Channel dimension handling**: Automatic detection and conversion between different channel formats (CHW vs HWC)
- **Keypoint processing**: Includes methods for heatmap-based keypoint extraction and refinement
- **Post-processing utilities**: Dark unbiased data processing and coordinate transformation functions
- **Optimized normalization**: Handles variable channel counts (1, 3, 4 channels) with proper mean/std scaling

## Before submitting
- [x] This PR implements a new fast image processor for VitPose
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. https://github.com/huggingface/transformers/issues/36978
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@yonigozlan

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, vitpose

Hi @yonigozlan! Could you please take a look. Thank you.
docs: clarify decoder_input_ids vs decoder_inputs_embeds usage (#39542)
Author: Wricha
Pull request: yes

This PR adds a note under PreTrainedModel clarifying that seq2seq models 
such as MarianMT, BART, and T5 accept either decoder_input_ids or 
decoder_inputs_embeds, but not both. Passing both arguments raises a 
ValueError. 

This addresses issue #39542.


Comments:
Adding api key to `transformers serve`
Author: xhluca
Pull request: no
Labels: name: Feature request

### Feature request

Enabling setting an api-key when serving in a way that's compatible with the openai api.

### Motivation

This would be a great feature to make our endpoint accessible over the internet.

### Your contribution

I'm not sure where to start to make a PR for this. Any pointer?

Comments:
Not 100% sure but it's actively being worked on atm :D see #39338
cc @gante 

Hi @xhluca 👋 

Do you mean adding some form of authentication through e.g. an`OPENAI_API_KEY ` sent with the request?

Yeah, like how one would use `OpenAi(..., api_key=xxxxxx)` to access their API resource via the python api. Without the API key the requests would fail.

is there any progress on this? 

@luvwinnie i've been back to using vllm since it has api key support (I don't want random internet users to send request to my server), will check out transformers serve again once api keys will be added. sorry i am not familiar enough with the codebase to propose a pr.
Revert "Pin torch to 2.7.1 on CircleCI for now" + Final fix for `too long with no output`
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#40174 + Final fix for `too long with no output`

This will use a custom pytest where we replace 

> raise SystemExit(pytest.console_main())

to 

> os._exit(pytest.console_main())

to force exit immediately, without calling exit (cleanup) handlers and flush the buffers.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40201). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen3 MOE models w/non-empty `mlp_only_layers` fail when `output_router_logits=True`
Author: ntenenz
Pull request: no
Labels: name: bug

### System Info

MLPs insert a `None` into the router logit list. Therefore, when enabled in the model, they need to be filtered out either before or within the aux_loss function.

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("Qwen/Qwen3-30B-A3B")  # or any qwen3_moe model
config.update({"mlp_only_layers": [0]})  # or any non-empty list
model = AutoModelForCausalLM.from_config(config)

_ = model(INPUT_TOKENS, output_router_logits=True)  # raises an error in the aux_loss function
```

### Expected behavior

Model should output router logits and not raise an exception.

Comments:
I’d like to work on this, can you confirm filtering out None router logits is the right approach?

Hi all,
This issue is now resolved by [[PR #39206](https://github.com/huggingface/transformers/pull/39206)](https://github.com/huggingface/transformers/pull/39206).

* The fix ensures that Qwen3 MoE models with non-empty `mlp_only_layers` no longer include `None` values in the router logits when `output_router_logits=True`.
* The router logits are now filtered so only valid tensors from MoE layers are included, matching the Qwen2 MoE implementation pattern.
* Auxiliary loss computation now works correctly in all configurations, including edge cases where all layers are MLP-only.

I’ve run comprehensive tests on all combinations of `mlp_only_layers` and confirmed the error no longer occurs.

**Reference:**
PR #39206: [[fix: filter None router logits in Qwen3 MoE and handle empty router logits (#39203)](https://github.com/huggingface/transformers/pull/39206)](https://github.com/huggingface/transformers/pull/39206)

cc @ArthurZucker @ntenenz

Having a look! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
CPMANT Model Fails to Run Following Official Tutorial
Author: iugoood
Pull request: no
Labels: name: bug

### System Info

Operating System: Linux
Python Version: 3.10.16
Transformers Library Version: 4.51.0
Hardware Configuration: GPU A100




### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to Reproduce：

import torch
from transformers import AutoTokenizer, CpmAntForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/cpm-ant-10b")
model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs, labels=inputs["input_ids"])
loss = outputs.loss
logits = outputs.logits

### Expected behavior

The model should load and start training without any errors.

Comments:
Hi @iugoood, can you link the official tutorial referring to and paste the error you're getting?

Traceback (most recent call last):
  File "/mnt/data3/mindone/ymc/cpmant/test_cpmant.py", line 8, in <module>
    outputs = model.generate(**input_ids)
  File "/nvme1n1p1/mindone/miniconda3/envs/ymc/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/nvme1n1p1/mindone/miniconda3/envs/ymc/lib/python3.10/site-packages/transformers/generation/utils.py", line 2482, in generate
    result = self._beam_search(
  File "/nvme1n1p1/mindone/miniconda3/envs/ymc/lib/python3.10/site-packages/transformers/generation/utils.py", line 3977, in _beam_search
    log_probs = torch.reshape(log_probs, (batch_size, num_beams * vocab_size))
RuntimeError: shape '[1, 92160]' is invalid for input of size 95232

referring to:https://huggingface.co/docs/transformers/v4.50.0/en/model_doc/cpmant#transformers.CpmAntForCausalLM


Hi @iugoood, I tried running the script but the repo seems to be quite old and misconfigured. In particular, the tokenizer and architecture names don't align with the library, so I wasn't able to load the tokenizer correctly. There's not a lot I can do if they're not accepting PRs to update the repo!

Maybe just use one of the more modern models from the same lab? https://huggingface.co/openbmb/MiniCPM4-8B

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Use correct `model_input_names` for PixtralImageProcessor
Author: rohitrango
Pull request: yes

# What does this PR do?

Returned tensors from the PixtralProcessor does not match the list in `model_input_names` (useful for inferring expected tensor keys).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: pixtral

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40226). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.