noamaanMulla-03/transformers-issues huggingface dataset:
mulitmodal forward pass for ministral 3 family
Author: rishavranaut
Pull request: no
Labels: name: bug

### System Info

https://github.com/huggingface/transformers/blob/main/src/transformers/models/ministral3/modeling_ministral3.py#L505

seems like here we are using generic class which takes only the input ids as input ignoring the pixel values. when can we expect this implemented ?



### Who can help?

@Cyrilvallez 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

please implement https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma3/modeling_gemma3.py#L1174 for ministral family as well with multimodal capabilities

### Expected behavior

need multimodal capabilities using ministral for finetuning ministral for sequence classification like gemma 3 4b
https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma3/modeling_gemma3.py#L1174

Comments:
cc @zucchini-nlp

Looks like it is supported via `Mistral3ForConditionalGeneration`, see usage example in https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512#transformers-bf16 :)

there is difference how generation vs classification works although fundamentally it is same but still it doesnt work that way

Ah you requested for Sequence Classification head? In that case feel free to submit a PR adding a task head on top of Mistral3 in the same way as in Gemma3
Remove tied weights from internal attribute if they are not tied
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sam3_video

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42871). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42871&sha=ecbeed
Fix: Improve AutoTokenizer error message for Voxtral model
Author: tamiraji-bot
Pull request: yes
Labels: name: Code agent slop

Fix: Improve AutoTokenizer error message for Voxtral.
Fixes #41553

This PR enhances the error message raised by AutoTokenizer when attempting to load
the `Qwen/Qwen2.5-VL-Voxtral` model, which does not provide a tokenizer.

The updated message clearly informs users that:
- The Voxtral model does not include a tokenizer,
- They must use a compatible tokenizer or provide one manually,
- Instructions can be found in the model card.

This resolves issue #41553 and improves user experience by preventing misleading
"invalid model identifier" errors.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42849&sha=58c633
Fix: dtype-safe weight initialization for quantized models (skip non-floating tensors)
Author: ladder2sky
Pull request: yes
Labels: name: Code agent slop

Summary
Fixes #39366. Prevents crashes when loading quantized (e.g., W8A8) models by making weight initialization dtype-safe: only initialize floating-point or complex tensors, and skip non-floating dtypes (like int8).

Motivation
Quantized weights (e.g., int8) can appear in various model flows (compression, W8A8 loading, conversions). The generic initialization routines call operations like nn.init.normal_ which require floating/complex dtypes. This raises runtime errors (e.g., "expected a floating-point or complex dtype, but got dtype=torch.int8" or "normal_kernel_cpu not implemented for 'Char'"), blocking model load.

Reproduction
- Minimal script: scripts/repro_issue39366_min.py:1–50
- Trigger: create a Linear layer, replace weight with int8, call the library’s generic init; prior behavior crashes on normal_.
- Real flows: quantized model loading or conversion workflows.

Solution
- Introduce a local guard in weight initialization: a helper _can_init that returns true only for floating-point or complex tensors.
- Apply the guard around init.normal_, init.zeros_, init.ones_ calls in modeling_utils._init_weights to skip non-floating dtypes safely.
- This preserves existing behavior for standard FP models, while preventing erroneous init calls for quantized weights.

Additional Changes
- Add AutoModelForCausalLM mapping for Qwen2_5_VL so Auto APIs correctly resolve the conditional generation model.
- Minor RoPE validation consistency (no behavior change in default flows).

Impact
- Backward compatible for all existing floating-point models.
- No performance regressions (guards are trivial checks).
- Safer initialization for quantized and mixed dtype scenarios.

Tests & Verification
- Minimal repro passes (no crash) and prints "dtype-safe init: OK".
- Local runs with quantized flows avoid dtype-related init errors.
- Code references:
  - src/transformers/modeling_utils.py:2125 (define _can_init)
  - src/transformers/modeling_utils.py:2138–2144 (guard normal_/zeros_ for linear/embedding)
  - src/transformers/models/auto/modeling_auto.py:974, 1033 (Qwen2_5_VL mapping)
  - scripts/repro_issue39366_min.py:1–50 (minimal repro)

Checklist
- [x] Code follows library patterns and keeps defaults intact
- [x] Quantized dtype paths are now safe
- [x] Auto mapping updated for Qwen2_5_VL
- [x] Minimal repro included for reviewers


Comments:
[CB] Easy optimizations for continuous batching
Author: remi-or
Pull request: yes

## Summary
This PR adds a few optimizations for continuous batching:
- removes non-needed `torch.cuda.synchronize` 
- add sorting of the inputs to maximize prefix caching hits
- sampling is done on the GPU
- removed an extranuous axis from the output_ids

## Performance 
| Attention         | Version       | Generated tokens | Duration (s) | Throughput (tok/s) |
|------------------------|----------------|--------------------------|------------------|--------------------------|
| Flash attention 3 | This PR         | 112599 | 16.73       | 6729.27           |
| Flash attention 3 | Main branch  | 111823 | 25.67       |  4355.68          |
| Flash attention 2 |  This PR        | 112822 | 24.61       | 4584.74          |
| Flash attention 2 | Main branch  | 112126 | 33.12       | 3385.46          |
| SDPA                  | This PR         | 113254  | 82.49       | 1373.00         |
| SDPA                  | Main branch  | 113725  | 170.84    | 665.67            |
* the performances on `main` were lowered by #42699 when compared to #42516 but this PR fixes this.

## Tests
No new failures for the CB tests. The only two failing tests are the same as in #42699 because of compile not working with `gemma2` which seems out of scope and acceptable.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42839). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi sir just wanted to say this seems like a nice PR thanks
Enforce call to `post_init` and fix all of them
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. It is high-time to enforce a check in repo consistency for it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42873). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart, bigbird_pegasus, blenderbot, blenderbot_small, blip, blt, bridgetower, chameleon, clipseg, decision_transformer, dia, evolla, flaubert, gemma3n, got_ocr2, gpt2
Remove null values from fast image processors dict
Author: yonigozlan
Pull request: yes

# What does this PR do?

As discussed internally, Cc @ArthurZucker, @zucchini-nlp 

This is a bit more involved than I first thought, as we need to keep explicitly passed null values if the default of the class is not None, otherwise we would be loading the processor with the default when it should be None.

Before (Devstral):

```json
{
  "image_break_token": "[IMG_BREAK]",
  "image_end_token": "[IMG_END]",
  "image_processor": {
    "crop_size": null,
    "data_format": "channels_first",
    "device": null,
    "disable_grouping": null,
    "do_center_crop": null,
    "do_convert_rgb": true,
    "do_normalize": true,
    "do_pad": null,
    "do_rescale": true,
    "do_resize": true,
    "image_mean": [
      0.48145466,
      0.4578275,
      0.40821073
    ],
    "image_processor_type": "PixtralImageProcessorFast",
    "image_seq_length": null,
    "image_std": [
      0.26862954,
      0.26130258,
      0.27577711
    ],
    "input_data_format": null,
    "pad_size": null,
    "patch_size": 14,
    "processor_class": "PixtralProcessor",
    "resample": 3,
    "rescale_factor": 0.00392156862745098,
    "return_tensors": null,
    "size": {
      "longest_edge": 1540
    }
  },
  "image_token": "[IMG]",
  "patch_size": 14,
  "processor_class": "PixtralProcessor",
  "spatial_merge_size": 2
}
```
After:
```json
{
  "image_break_token": "[IMG_BREAK]",
  "image_end_token": "[IMG_END]",
  "image_processor": {
    "data_format": "channels_first",
    "do_convert_rgb": true,
    "do_normalize": true,
    "do_rescale": true,
    "do_resize": true,
    "image_mean": [
      0.48145466,
      0.4578275,
      0.40821073
    ],
    "image_processor_type": "PixtralImageProcessorFast",
    "image_std": [
      0.26862954,
      0.26130258,
      0.27577711
    ],
    "patch_size": 14,
    "processor_class": "PixtralProcessor",
    "resample": 3,
    "rescale_factor": 0.00392156862745098,
    "size": {
      "longest_edge": 1540
    }
  },
  "image_token": "[IMG]",
  "patch_size": 14,
  "processor_class": "PixtralProcessor",
  "spatial_merge_size": 2
}
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42780). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: flava, mask2former
Raise explicit error when FP8 is requested via from_config
Author: dikshyantacharya
Pull request: yes

### Summary
`from_config()` currently ignores FP8 intent specified via `quantization_config`, silently falling back to FP32.
This PR makes the behavior explicit by raising a clear error instead.

### What this PR does
- Raises `NotImplementedError` when FP8 is requested via `from_config()`
- Adds a regression test to lock in the behavior

### Why
- Prevents silent misconfiguration
- Improves API correctness
- Establishes a safe foundation for future FP8 support

### Scope
This PR does **not** enable FP8 kernels or backend support.
It focuses solely on correctness and explicit behavior.


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: config

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42865&sha=417544
Simplify using custom resolution for sam3 and sam3_video inference
Author: yonigozlan
Pull request: yes

# What does this PR do?

It seems like there's demand for supporting this feature (see https://github.com/huggingface/transformers/issues/42331), and while in the current state, it is possible to make it work, it's not intuitive.
This simplifies it a lot.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42787). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! I see the use indeed, but in terms of interface, maybe we could use a `setter` as well as a `property` for `image_size`? With the setter doing exactly what your `set_image_size` is doing now, and the property just returning the value. That way `config.image_size = 560` will propagate automatically and set the values needed downstream, WDYT?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sam3, sam3_tracker_video, sam3_video
[docs] optimizations quickstart
Author: stevhliu
Pull request: yes

Adds an overview/quickstart of available Transformers optimization techniques to provide a clear and centralized place where they're all documented and help users select the right optimization to increase speed or reduce memory footprint

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
doc(kernels): update kernels integration documentation
Author: mfuntowicz
Pull request: yes

Add some more content to the kernels integration in Transformers.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42277). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Feel free to ping me again for another review whenever you're ready!
[Quantization] Allow loading of transform configs
Author: kylesayrs
Pull request: yes

## Purpose ##
* Support loading models with online transforms applied via Compressed Tensors (LLM Compressor)

## Prerequisites ##
* https://github.com/neuralmagic/compressed-tensors/pull/445

## Changes ##
* Require a minimum compressed tensors version of `0.11.0` (to support transform features)
* Load transform configs (if available), and apply them to the model before weight loading
* (misc) Refactor compressed tensors tests to check for perplexity, rather than exact output matches
* (misc) Remove `update_dtype` in order to reduce complexity and give users more control/predictability of model data types

## Testing ##
* Regression tested using `CompressedTensorsTest`, added an online quip-style transformed model for testing
  * Perplexity results match expectations

Comments:
cc @mekkcyber

Putting in draft for now, need to do some more testing

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: compressed_tensors_integration

https://github.com/huggingface/transformers/pull/42887
sdpa_paged: How does it handle paged cache without padding?
Author: jiqing-feng
Pull request: no

Hi @ArthurZucker ,

I was analyzing the [sdpa_paged](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/sdpa_paged.py#L18) implementation and found the approach quite fascinating. I have a question regarding how the input shapes are handled.

If I have a batch of 4 sequences with lengths **32, 32, 64, and 128**, a standard SDPA call usually expects a shape of `[4, 128]` (Batch Size, Max Seq Len), where the shorter sequences are padded to 128.

However, in this implementation, it appears that the input to SDPA is a flattened tensor with shape **`[1, 256]`** (the sum of all lengths: $32+32+64+128$), implying that no padding is used and the sequences are concatenated.

Could you explain how standard SDPA produces the correct result in this case? Specifically, how does it differentiate between the sequences to prevent cross-sequence attention within this single packed batch?

Thanks for your time!


related PR: #38085

Comments:
The CPU can also use paged cache with sdpa or eager attention. Please review the PR #42869 . Before that, I really want to figure out how sdpa handles paged cache without padding. It would be great if you could give me some explanations. Thanks!

cc @McPatate @gante 

cc @remi-or 

Hi @jiqing-feng ! Good question, this is at the heart of continuous batching. Long story short: we use the attention mask. By setting all the cross-sequence coefficient in the attention mask to `False` (or `-inf` for attention bias) we can make sure there is no interaction between cross-sequence tokens. This is illustrated below for two sequences (called prompts 0 and 1 in the diagram): 

<img width="1855" height="1496" alt="Image" src="https://github.com/user-attachments/assets/a20b1249-51b9-4ad4-9745-5651f45d6ea8" />

We have written a whole blog post about continuous batching [here](https://huggingface.co/blog/continuous_batching) explaining how it works. To clarify: continuous batching is the only context where paged SDPA (or any paged attention for that matter) is used. Hope that answers your question!


Amazing! Thanks for your explanation.
Add ability to push Trackio logs
Author: abidlabs
Pull request: yes


Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42751&sha=f65f97

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Blt Model can't pass training_ci
Author: 3outeille
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0.dev0
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.12.9
- Huggingface_hub version: 1.1.5
- Safetensors version: 0.6.2
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: 0.18.2
- PyTorch version (accelerator?): 2.9.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

As part of the `training_ci` (https://github.com/huggingface/transformers/pull/42597), we noticed that BLT model can't overfit a single sentence (and thus generated the same sentence back). There must be a bug somewhere. Raising issue to flag it

`pytest tests/models/blt/test_modeling_blt.py::BltModelTest::test_training_overfit -s`

<img width="1144" height="483" alt="Image" src="https://github.com/user-attachments/assets/d2afb856-0a9f-4856-888a-d063f577393f" />


### Expected behavior

matching generation and `loss` + `grad_norm` reduction should be at least 90%

Comments:
Hi, I’d like to investigate this issue and work on a fix.

Sure ! Just wait for https://github.com/huggingface/transformers/pull/42597 to be merged so that you can run the specific test !

Note the test will be skipped for BLT so you'll have to undo it https://github.com/huggingface/transformers/pull/42597/files#diff-954e1fdb97b37cf254922cc5d32f5d83e1800e2ecae3dcf4c611bebd04422b43R180 for you to fix

@preetam1407 PR is merged !

Hi, I’ve opened a PR for this: https://github.com/huggingface/transformers/pull/42685

Summary:
- Tiny BLT config does overfit the fixed batch (loss ~3.46 → 0.18, ~95%), but grad norm only drops ~81% with clipping, so the generic 0.9 threshold is too strict.
- Generation over the fixed batch only matches the training pattern when `use_cache=False`.

PR changes:
- Add a `use_cache` flag (default `False`) to `BltConfig` and forward it to `PreTrainedConfig`, so BLT defaults to `use_cache=False` but still respects explicit settings.
- Override `training_grad_norm_reduction_threshold = 0.8` in `BltModelTest` (loss threshold stays at `0.9`).

Locally, `pytest tests/models/blt/test_modeling_blt.py::BltModelTest::test_training_overfit -s -vv` now passes and the generated sequence exactly matches the training sequence.
Fix name mix-up in tie weights logic
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title! This one slipped through!
The name cannot be updated before we append it to the targets! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42893). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42893&sha=973835
Add missing ModelOutput subclass return type hints
Author: tomaarsen
Pull request: yes

# What does this PR do?

This PR adds missing ModelOutput subclass return type hints. I'm running an analysis on the Model classes to see what kind of outputs are used for the various modalities, and I noticed a handful of architectures were missing them.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41219). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: dac, deepseek_vl, deepseek_vl_hybrid, flava, grounding_dino, janus, maskformer, mm_grounding_dino, tvp, udop

I fixed the merge conflicts. I totally forgot about this PR, but it should still be valuable to improve the typings somewhat.

- Tom Aarsen

@vasqu The script I wrote to check the type hints indeed only checked the base models, also because Sentence Transformers will in the future rely on the type hints of the base models. Extending it to the non-base classes would be nice, but I don't have the bandwidth for that now, as I'm aiming to get #42564 ready before the full v5.

- Tom Aarsen

Gotcha, makes sense then. Let's merge, having more typing is better than nothing in the first place
[Devstral] Make sure FP8 conversion works correctly
Author: patrickvonplaten
Pull request: yes

Improve script for FP8 conversion.
Tested on all relevant checkpoints (both Devstral 2 & Ministral 3B Reasoning & 14B - FP8)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ministral3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42715). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[modular] Fix a weird renaming edge-case
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Needed for https://github.com/huggingface/transformers/pull/36895 (see https://github.com/huggingface/transformers/pull/36895#issuecomment-3635144592)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42844). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Works perfectly !