noamaanMulla-03/transformers-issues huggingface dataset:
Fix DeepSpeed mixed precision precedence over Accelerate defaults
Author: notkisk
Pull request: yes

## Summary

Fixes issue [[#39849](https://github.com/huggingface/transformers/issues/39849)](https://github.com/huggingface/transformers/issues/39849) where Accelerate would default to `bf16` mixed precision even when a DeepSpeed config specifies `fp16`, causing the following error:

> `ValueError: --mixed_precision arg cannot be set to bf16 when fp16 is set in the DeepSpeed config file.`

This PR ensures that DeepSpeed configuration takes precedence over `TrainingArguments` defaults while preserving explicit user settings.

---

## Root Cause

The issue was caused by the initialization order in `TrainingArguments.__post_init__()`. The `ACCELERATE_MIXED_PRECISION` environment variable was being set **before** the DeepSpeed config was processed, preventing it from overriding Accelerate’s defaults.

---

## Changes Made

### 1. Added DeepSpeed Config Override Logic

* Added `override_training_args_from_deepspeed()` method to `HfTrainerDeepSpeedConfig` class.
* This method checks DeepSpeed config for `fp16`/`bf16` settings and overrides `TrainingArguments` defaults accordingly.
* Explicit user choices are preserved, but DeepSpeed config can override defaults if no user input is provided.

### 2. Fixed Initialization Order

* Moved the mixed precision environment variable setting in `TrainingArguments.__post_init__()` to occur **after** DeepSpeed config processing.
* Ensures DeepSpeed config overrides are applied **before** environment variables are set.

---

## Behavior

The fix enforces the following **precedence hierarchy**:

1. **Explicit user settings** – Highest priority
   E.g., `fp16=True` or `bf16=True` passed by user.
2. **DeepSpeed config** – Medium priority
   E.g., `"fp16": {"enabled": true}` or `"bf16": {"enabled": true}` in config file.
3. **TrainingArguments defaults** – Lowest priority

---

## Test Plan

* ✅ Verified the original reproduction case no longer fails.
* ✅ Tested that DeepSpeed `fp16` config overrides default correctly.
* ✅ Tested that DeepSpeed `bf16` config overrides default correctly.
* ✅ Confirmed explicit user settings take precedence over DeepSpeed config.
* ✅ Ensured environment variables are set correctly in all scenarios.
* ✅ Ran existing DeepSpeed test suite to check for regressions.
* ✅ Rebased on latest `main` and verified fix still works.

---

## Files Modified

* `src/transformers/integrations/deepspeed.py` – Added override logic and method call.
* `src/transformers/training_args.py` – Reordered mixed precision env var setup.

---

## Branch Info

* **PR Branch:** `fix-deepspeed-mixed-precision-precedence` (rebased on latest `main`)
* **Base Branch:** `main`


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v, qwen2_5_vl, qwen2_vl
remove dtensors, not explicit
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Removed dtensor redistribute for many reasons: 
- 2x slower for the add of the bias
- not explicit enough
- does not allow us to run deepspeed 
- we were already not using tensors for MoE because of how cumbersome doing a .view was

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add multimodal executorch support
Author: mergennachin
Pull request: yes

New Class: TorchExportableModuleForImageTextLM

  Dedicated wrapper for image-text language models:
  - Purpose: Handles multimodal models that need inputs_embeds instead of input_ids
  - Architecture: Automatically chooses HybridCache vs StaticCache based on model config
  - Usage: Takes embeddings from vision encoder + text tokenizer as input

New Class: ImageEncoderExportableModule

  Wrapper for vision encoder components:
  - Purpose: Exports the vision processing pipeline (vision_tower → multi_modal_projector)
  - Function: Converts images to language-compatible embeddings
  - Integration: Works with TorchExportableModuleForImageTextLM for complete multimodal export

```
  Multimodal Model Export:

  # Vision encoder export
  vision_encoder = ImageEncoderExportableModule(model)
  exported_vision = vision_encoder.export()

  # Text decoder export  
  text_decoder = TorchExportableModuleForImageTextLM(model.language_model)
  exported_text = text_decoder.export()

  Runtime Usage:

  # Process image → embeddings
  image_embeddings = exported_vision.module()(pixel_values)

  # Process text → embeddings
  text_embeddings = model.embed_tokens(text_ids)

  # Combined inference
  inputs_embeds = torch.cat([image_embeddings, text_embeddings], dim=1)
  logits = exported_text.module()(inputs_embeds=inputs_embeds, cache_position=cache_position)
```

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: moshi

Hey @zucchini-nlp 

Thanks a lot for the thoughtful reviews. 

@jackzhxng will take this over the finish line in https://github.com/huggingface/transformers/pull/39836

I'm gonna close this PR for the time being but hope @jackzhxng can incorporate some of your suggestions and recommendations in the PR.
'Mistral3Model' object has no attribute 'prepare_inputs_for_generation'
Author: timpal0l
Pull request: no
Labels: name: bug

### System Info

```bash
transformers==4.53.0.dev0
peft==0.15.2
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Cannot perform Lora finetuning on the new mistral 3.2 model yet.

```python
from transformers import AutoModel
from peft import LoraConfig, get_peft_model

model = AutoModel.from_pretrained("mistralai/Mistral-Small-3.2-24B-Instruct-2506")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj","k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config) # <--- crash
```
```
'Mistral3Model' object has no attribute 'prepare_inputs_for_generation'
```

### Expected behavior

It should work just as mistral <= 3.1

Comments:
Hi @timpal0l, the problem here is probably that you're using the base model with no causal LM head, but setting `task_type="CAUSAL_LM"` in the LoRA config. Try loading with `AutoModelForCausalLM` instead. I'm not sure why it worked before!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TypeError: couldn't find storage object Float8_e4m3fnStorage - which version is needed for this?
Author: FurkanGozukara
Pull request: no
Labels: name: bug

Tested so many versions but can't find a version that won't give this error


```
!pip install bitsandbytes==0.45.0 --upgrade
!pip install insightface --upgrade
!pip install huggingface_hub==0.25.1 hf_transfer diffusers==0.31.0 transformers==4.36.0
!pip uninstall xformers triton --yes
!pip install torch==2.2.0+cu121 torchvision --index-url https://download.pytorch.org/whl/cu121
!pip install xformers==0.0.24 --index-url https://download.pytorch.org/whl/cu121
```

```

 File "/kaggle/temp/InstantID/gradio_demo/web-ui-multicontrolnet.py", line 975, in generate_image
    reload_pipe(model_input, model_dropdown, scheduler, adapter_strength_ratio, enable_LCM, depth_type, lora_model_dropdown, lora_scale,test_all_loras,single_lora)
  File "/kaggle/temp/InstantID/gradio_demo/web-ui-multicontrolnet.py", line 654, in reload_pipe
    pipe = load_model(_pretrained_model_folder, model_to_load)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/temp/InstantID/gradio_demo/web-ui-multicontrolnet.py", line 528, in load_model
    pipeline = StableDiffusionPipeline.from_pretrained(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/diffusers/pipelines/pipeline_utils.py", line 896, in from_pretrained
    loaded_sub_model = load_sub_model(
                       ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/diffusers/pipelines/pipeline_loading_utils.py", line 704, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 4027, in from_pretrained
    dtype_orig = cls._set_default_torch_dtype(torch_dtype)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 1584, in _set_default_torch_dtype
    torch.set_default_dtype(dtype)
  File "/usr/local/lib/python3.11/dist-packages/torch/__init__.py", line 1009, in set_default_dtype
    _C._set_default_dtype(d)
TypeError: couldn't find storage object Float8_e4m3fnStorage
```



Comments:
hi @FurkanGozukara, it looks like you're storing 8-bit floats, so I'd guess that you want a recent version of PyTorch. I notice you're trying to install 2.2, which seems quite old - are you maybe stuck on an Intel mac that doesn't support the latest version, 2.7.1?
Fix pylint warnings
Author: cyyever
Pull request: yes

# What does this PR do?

A general clean-up to fix various pylint warnings, before it's possible to enable most pylint ruff rules.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark, bert, bert_japanese, clip, convbert, cvt, distilbert, electra, emu3, falcon_h1, funnel, herbert, imagegpt, layoutlm, layoutlmv2, lxmert

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39477). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
GLM-4 Update
Author: zRzRzRzRzRzRzR
Pull request: yes

@ArthurZucker  Can view the full vision of this model

Comments:
I’ve updated to the latest version of the modular code, removing most of the unnecessary parameters while ensuring the model still works properly. The remaining parts haven’t been removed because modifying them would lead to duplicated code—one such bug appeared in Glm4MoeTopkRouter, where two e_score_correction_bias buffers were generated.
Please take a look at the latest modular file. @Cyrilvallez 

Also, note that there are some tests related to this model which are failing and should be fixed, or skipped if needed (but only for good reasons, and explaining the exact reason of the skip)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, glm4_moe

> Also, note that there are some tests related to this model which are failing and should be fixed, or skipped if needed (but only for good reasons, and explaining the exact reason of the skip)

I reducing n_routed_experts to 8 and setting num_experts_per_tok=8 ensures 100% expert utilization rate, which guarantees all experts receive gradients during training and fixes the gradient checkpointing test failures without requiring code modifications to the MoE implementation. Now it will pass



Hi @zRzRzRzRzRzRzR 

Do you know what is the correct repo_id for `Glm4MoeI`

> THUDM/GLM-4.5 

doesn't exist.



```
    "models_glm4_moe": {
            {
                "line": "tests/models/glm4_moe/test_modeling_glm4_moe.py::Glm4MoeIntegrationTest::test_compile_static_cache",
                "trace": "(line 508)  OSError: THUDM/GLM-4.5 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'"
            }
        ]
    },
```

This model has not been released yet, and I don't know the specific model name yet. We will submit a new PR to improve it once the model is released.

Tensor parallelism?

<class 'transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeModel'> does not support tensor parallel yet!
pytorch version 1.8.1 compatibility
Author: Dominiq24
Pull request: no

which transformers version is compatible with pytorch 1.8.1 and also peft 0.4.0

Comments:
Upgrade your PyTorch to at least 1.10 (preferably 1.12 or later). This will allow you to use transformers >=4.30.0 and peft 0.4.0.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
model.generate custom encoder and decoder outputs/inputs
Author: agu18dec
Pull request: no
Labels: name: Feature request

### Feature request

For encoder decoder models, I would like to have the following feature where I can generate using:

1. encoded representations of my context tokens passed in while
2. putting tokens of ONLY my question related to the context in decoder_input_ids 



### Motivation

I think this is broadly useful to understand how encoder decoder models operate also how cross attention works between the decoder token generation and the relevant encoded context (for e.g. what tokens are retrieved etc.).

I dont want to put the entirety of my question + context in my input_ids parameter and I don't know if there exists an easy way to do this right now. Would be super useful. Thank you!

### Your contribution

N/A

Comments:
Improve `is_wandb_available` function to verify WandB installation
Author: qgallouedec
Pull request: yes

To reproduce:

- Install WandB
- Run an exp
- Uninstall
- check what returns `is_wandb_avaialble()` (returns `True`)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39875). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Refactor label name handling for PEFT models in Trainer class
Author: qgallouedec
Pull request: yes

# What does this PR do?

Fix https://github.com/huggingface/trl/issues/3694

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


@SunMarc



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39265). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use comment to build doc on PRs
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline with @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39846). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> If I got it right, comments with build-doc will trigger the full doc build, correct?

Yes :-)

[An example run](https://github.com/huggingface/transformers/actions/runs/16717979114)
Tensor parallelism for GLM-4.5
Author: fernandaspets
Pull request: no
Labels: name: Feature request

### Feature request

I get this error when trying to load in vllm with -tp 2

transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeModel'> does not support tensor parallel yet!

### Motivation

I can't fit the model in one 96gb gpu

### Your contribution

Happy to test etc. :)

Comments:
Hey @fernandaspets ! Are you running on vLLM with `model_impl="transformers"`? vLLM has its own implementation of the model and if you aren't running on tranformers backend, this issue is more probably belong to vLLM repo
Allow video objects (np array etc.) in apply_chat_template (not just paths or urls)
Author: FredrikNoren
Pull request: no
Labels: name: Feature request

name: Chat Template

name: VLM

### Feature request

Currently it seems like it only supports `{ "type": "video", "path/url": ".."}`.

The code seems to be made to allow numpy arrays, torch tensors and other things as values: https://github.com/huggingface/transformers/blob/main/src/transformers/image_utils.py#L851
But the `video.startswith` will always crash if anything other than a string is passed in

### Motivation

I'd like to pass in videos from memory instead of writing them to disk first.

### Your contribution

I can update the code to make it support video objects

Comments:
Hey @FredrikNoren !

Yes, that was supposed to be supported and prob I missed it when testing. Feel free to open a PR, and add some test in `test_processing_common`. Also, would be better to do same for images, enabling passing `PIL/array` images

Hi @FredrikNoren , @zucchini-nlp , 
Is the issue still open? I would love to take this if the issue is still there.

Sure, feel free, I don't think there was a PR for that

@akibjawad Yeah I never had time to do a PR, feel free

@zucchini-nlp is this still free if then, i would like to contribute


@devesssi  I have developed the feature and currently testing it. My PR is in WIP it should be completed by early next week.

Thanks for clarifying @akibjawad . Let me know if there's anything I could do
ValueError: Max cache length is not consistent across layers
Author: TheTharz
Pull request: no
Labels: name: bug

### System Info

transformers==4.54.0

torch==2.6.0+cu124

peft==0.16.0

unsloth==2025.8.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb

Used above notebook provided in the unsloth documentation in finetuning the Gemma3_(1B)-GPRO.
That notebook also gives the error in the training step.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/unsloth/models/vision.py](https://localhost:8080/#) in unsloth_base_fast_generate(self, *args, **kwargs)
    232         with torch.inference_mode(), autocaster:
--> 233             output = self._old_generate(*args, **kwargs)
    234     except:

20 frames
ValueError: Max cache length is not consistent across layers: [512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512]

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/transformers/cache_utils.py](https://localhost:8080/#) in max_cache_len(self)
   1255         values = [layer.max_cache_len for layer in self.layers]
   1256         if len(set(values)) > 1:
-> 1257             raise ValueError(f"Max cache length is not consistent across layers: {values}")
   1258         return values[0]
   1259 

ValueError: Max cache length is not consistent across layers: [512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512, 512, 512, 512, 868, 512, 512]

### Expected behavior

should handle or pre-validate inconsistent max_cache_len values without crashing

Comments:
cc @manueldeprada 

Hi @TheTharz thanks for reporting! This is fixed in [v4.54.1](https://github.com/huggingface/transformers/releases/tag/4.54.1), sorry for the inconvenience
Fix attn_implementation setter for models with `backbone_config`
Author: qubvel
Pull request: yes
Labels: name: Core: Modeling

name: bug

# What does this PR do?

Fix attn_implementation setter for models with `backbone_config`

#### On main:

```python
from transformers import VitPoseForPoseEstimation
model = VitPoseForPoseEstimation.from_pretrained(
    "usyd-community/vitpose-base-simple", attn_implementation="eager"
)

print(model.config._attn_implementation)
# "eager"
print(model.config.backbone_config._attn_implementation)
# "sdpa"
```

#### On branch:

```python
from transformers import VitPoseForPoseEstimation
model = VitPoseForPoseEstimation.from_pretrained(
    "usyd-community/vitpose-base-simple", attn_implementation="eager"
)

print(model.config._attn_implementation)
# "eager"
print(model.config.backbone_config._attn_implementation)
# "eager"
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39855). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr, d_fine, dab_detr, deformable_detr, depth_anything, detr, fsmt, grounding_dino, mask2former, maskformer, mm_grounding_dino, omdet_turbo, oneformer, pix2struct, prompt_depth_anything, rt_detr
Fix quant docker for fp-quant 
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes an issue with the quantization [docker](https://github.com/huggingface/transformers/actions/runs/16484958005/job/46652740114). fp-quant requires to have [py3.11](https://github.com/IST-DASLab/FP-Quant/blob/b7f7462b54c8d572a63bb7a8668da22a4e93079a/inference_lib/setup.py#L19) but our ci uses 3.9. Hence we don't install for now. Thanks @gante for spotting this. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39641). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /style

Style bot fixed some files and pushed the changes.
Rework add-new-model-like with modular and make test filenames coherent
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title! Also fixes the image processing auto mapping the same way as we do with tokenizers for slow/fast ones -> it was not coherent

I basically rewrote everything as it was very complicated and out-dated.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39612). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, colpali, colqwen2