noamaanMulla-03/transformers-issues huggingface dataset:
Remove image-to-text pipeline
Author: Rocketknight1
Pull request: yes

Follow-up to #43256, this PR removes the `image-to-text` pipeline as well, as all modern uses of that pipeline should really be replaced by VLMs

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43306&sha=ddd648

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43306). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @cyrilvallez this should be ready except for unrelated test failures

Merging as the CI is hard to circumvemt these days!
Replace config.get() with getattr(config)
Author: Rocketknight1
Pull request: yes

A buggy line was introduced in #42894 that uses `config.get()`, but `config` classes are not dicts and don't have item access like that. Replaced it with `getattr()`!

Fixes #43272, cc @arthurzucker for review

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43321). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43321&sha=781ff5
Getting closer
Author: BenjaminBossan
Pull request: yes

It was necessary to flatten the LoRA weights for 3d MoE, as LoRA always expected 2d weights (being nn.Linear).

Therefore, I added new conversion ops: `PermuteDims` and `FlattenDims`. I also added a convenience function `_block_diag_3d`, it could probably be optimized.

Furthermore, I now have separate branches for `down_proj` and `gate_up_proj` as they are different enough that treating them in one branch is too confusing.

Comments:
Current error:

```
MixtralForCausalLM LOAD REPORT from: peft-internal-testing/mixtral-pre-v5-lora
Key                                                              | Status   | Details                                                                                  
-----------------------------------------------------------------+----------+------------------------------------------------------------------------------------------
model.layers.{0, 1}.mlp.experts.lora_A.default.weight            | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([64, 3584]) vs model:torch.Size([64, 1024]) 
model.layers.{0, 1}.mlp.experts.lora_B.default.weight            | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([64, 1024]) vs model:torch.Size([3584, 64]) 
model.layers.{0, 1}.mlp.experts.base_layer.lora_A.default.weight | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([128, 1024]) vs model:torch.Size([64, 7168])
model.layers.{0, 1}.mlp.experts.base_layer.lora_B.default.weight | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([128, 7168]) vs model:torch.Size([1024, 64])
```

So there isn't much missing to align the shapes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43327). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43327&sha=ba6769
Fix post_process_semantic_segmentation removing valid class in Conditional DETR
Author: vasanthrpjan1-boop
Pull request: yes

Description:
Fixes #42679
What does this PR do?
Conditional DETR does not have a null class unlike DETR (its classifier outputs num_labels classes, not num_labels + 1). However, the post_process_semantic_segmentation method was incorrectly applying [..., :-1] to remove what it assumed was a null class, which actually removed the last valid class.
This PR removes the incorrect slice in both:
image_processing_conditional_detr.py
image_processing_conditional_detr_fast.py

Comments:
I'm mainly curious here why Conditional DETR has a `post_process_semantic_segmentation` method, cause the only [pre-trained checkpoints](https://huggingface.co/microsoft/conditional-detr-resnet-50) shared are for object detection. Otherwise it looks good to me.

yes I also checked in the docs we only have randomly initialized model + in the paper there's none either, maybe it's for random head initialization with a backbone

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan - I noticed few checks are failed, Any action to be taken from my side  

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr
[`Docs`] Fix other lang deprecations of a few pipelines
Author: vasqu
Pull request: yes

As per title, follow up to #43256 and #43306 - removing some traces in other lang docs

Comments:
build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/21004453499)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43292). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/21068917722)
Make TF32 tests hardware-aware for PyTorch 2.9+
Author: Shraman123
Pull request: yes

Fixes #42371

TF32 tests assumed fp32_precision == "tf32" after enabling, which is not true on
CPU-only or unsupported hardware. PyTorch reports "none" in those cases.

This change:
- Makes TF32 tests hardware-aware via is_torch_tf32_available
- Supports PyTorch >= 2.9 fp32_precision API
- Preserves legacy allow_tf32 behavior
- Restores backend state after tests


Comments:
@Shraman123 I dont think we can write torch tests, cause CI does not have multiple version of torch. 

@Shraman123 looking into this further, seems like pytorch issue, can you try filing but and link the issue from Huggingface. As I just checked another comment on the issue and in traceback I see pytorch inductor still using old API. thanks.

Thanks for pointing this out.

After removing the TF32 test, there are no additional functional changes left in this PR. This is intentional — the behavior was determined to be PyTorch-owned rather than a Transformers bug.

The purpose of this PR is therefore limited to removing a fragile test that relies on PyTorch internal precision semantics, which cannot be reliably validated in CI.


Please let me know if you’d prefer closing this PR instead, or if removing the test alone is acceptable.
[GLM-4.7] GLM-4.7-Flash Model Supoort
Author: zRzRzRzRzRzRzR
Pull request: yes

GLM-Lite is the class updated for GLM-4.7-Flash, which adopts a different model architecture from GLM-4.7. The intermediate state was named GLM-Lite and has now been renamed to GLM-4.7-Flash. At the same time, a PR has been submitted to improve the documentation.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43031). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, glm4_moe, glm4_moe_lite
fix: Updated BridgeTower Image processor
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Updated BridgeTower Image processor.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments:
@Sai-Suraj-27 This affects the image processing tests for bridgetower, which will need to be updated to reflect this fix 

> @Sai-Suraj-27 This affects the image processing tests for bridgetower, which will need to be updated to reflect this fix

Thanks for the review. Yes, I am looking into it. Will try to fix and update the PR 👍🏻
**Edit:** I updated the tests and now they are all passing. But I am not 100% sure, if I did it the correct way. Let me know if there is any problem.

Hi @Sai-Suraj-27, thanks for iterating on this! 

Looking at the changes in the test, I realized that this fix (which is addressing a bug in the code) would be introducing a change in the default behaviour of the image processor, and therefore would produce different outputs for the user. Previously, because `do_center_crop` was never defaulting to `self.do_center_crop` if unset, we were never cropping the image, and the flag in the preprocessor config had no effect. 

What we should do is: 
* remove the `do_center_crop` flag from the image processor
* remove `crop_size` logic from the image processor
* Add a deprecation warning in the crop method to say that it's going to be removed in two releases time 

Thanks for the review and explanation @amyeroberts. I've made the required changes. Please look into it and let me know if I missed anything.



cc @qubvel can you take a look at this while Amy is out?
when finetune deepseek v3.1 with lora，an ValueError was reported: QuantizationMethod.FP8 do not support training
Author: Francisapzii
Pull request: no
Labels: name: bug

### System Info

The error was as below. Please help.
Traceback (most recent call last):
  File "/hail/cjs/finetune/ft0/lora.py", line 231, in <module>
    model, tokenizer = finetune_with_lora(model_name, output_dir)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hail/cjs/finetune/ft0/lora.py", line 156, in finetune_with_lora
    trainer = Trainer(
              ^^^^^^^^
  File "/hail/cjs/anaconda3/envs/lora1/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/hail/cjs/anaconda3/envs/lora1/lib/python3.11/site-packages/transformers/trainer.py", line 572, in __init__
    raise ValueError(
ValueError: The model you are trying to fine-tune is quantized with QuantizationMethod.FP8 but that quantization method do not support training. Please open an issue on GitHub: https://github.com/huggingface/transformers to request the support for training support for QuantizationMethod.FP8

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetune "deepseek-ai/DeepSeek-V3.1-Terminus" with lora, this error will happen.

### Expected behavior

correct this error, please.

Comments:
cc @mekkcyber @sunmarc

came across this blog post today, think it might help:
https://huggingface.co/docs/accelerate/en/usage_guides/low_precision_training#what-training-on-fp8-means

```python
import torch
print(torch.cuda.get_device_name(0))
print(torch.cuda.get_device_capability(0))
``` 
@Francisapzii Can you elaborate your system specs too, because apparently, Only certain gpus have the capability to train using FP8?
if you can gimme the output of this,I can help

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

您好  请问解决了吗


> 您好 请问解决了吗

Its still not solved tho
Why the use of masked_fill
Author: tanmay2004
Pull request: no

Hi, I was recently going through the implementation of the deepseek_v3 model and was confused by the use of `masked_fill` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deepseek_v3/modeling_deepseek_v3.py#L228). Can't this line be replaced by `scores_for_choice = router_logits_for_choice * score_mask` and do the same thing? It feels like the simpler way to apply the mask on the logits. I'm just curious about the decision to use `masked_fill` instead, and I was wondering if you would agree about replacing it with the simpler way I wrote above.

Comments:
hi，maybe：
masked_fill is preferred:

Ensures masked logits are sufficiently small (-inf) for strict zero probability in softmax
Avoids numerical precision issues
Makes the code intention more explicit

I expect both of those lines are equivalent, and neither is a big part of the runtime! My intuition is that `masked_fill` should be faster than multiplication because it doesn't need to actually do any arithmetic, but maybe the conversion to bool + negation on the mask array cancels that out.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Inconsistencies in Rotary Embedding Implementations between Mistral and LLama Models
Author: Kacogo
Pull request: no

I've noticed some inconsistencies in the source code between the Mistral and LLama models, specifically in the implementation of rotary embeddings and auxiliary functions. While earlier versions seemed to have 1:1 copies of these functions, recent versions show growing disparities. This is evident in the rotary embedding classes and apply_rotary_embedding functions in both models:

https://github.com/huggingface/transformers/blob/517a3e670d8fc11374895e870dd0dd041467c7fe/src/transformers/models/mistral/modeling_mistral.py#L93-L126

https://github.com/huggingface/transformers/blob/517a3e670d8fc11374895e870dd0dd041467c7fe/src/transformers/models/llama/modeling_llama.py#L96-L145

https://github.com/huggingface/transformers/blob/517a3e670d8fc11374895e870dd0dd041467c7fe/src/transformers/models/mistral/modeling_mistral.py#L139-L164

https://github.com/huggingface/transformers/blob/517a3e670d8fc11374895e870dd0dd041467c7fe/src/transformers/models/llama/modeling_llama.py#L184-L208

These disparities are hindering the possibility of using the same function for custom attention computation across different models. I'm interested in understanding the reason behind these differences. Also, I would appreciate any guidance on how to alter the code for better compatibility or if there are plans to standardize these implementations.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix typo: interupted -> interrupted
Author: NAM-MAN
Pull request: yes

just a typo fix in core_model_loading.py:
interupted -> interrupted

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43333&sha=37df01

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43333). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[BUG] ESM config validation error with incorrect divisibility check
Author: harshaljanjani
Pull request: no
Labels: name: bug

### System Info

* `transformers` version: `5.0.0.dev0`
* Platform: `Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39`
* Python version: `3.12.3`
* `huggingface_hub` version: `1.3.2`
* `safetensors` version: `0.7.0`
* `accelerate` version: `1.12.0`
* Accelerate config: `not installed`
* DeepSpeed version: `not installed`
* PyTorch version (accelerator?): `2.9.1+cu128 (CUDA)`
* GPU type: `NVIDIA GeForce RTX 4060 Laptop GPU`

### Who can help?

@ArthurZucker @sayakpaul

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers.models.esm.configuration_esm import TrunkConfig
config = TrunkConfig(
    sequence_state_dim=100,
    sequence_head_width=30,
    pairwise_state_dim=128,
    pairwise_head_width=32,
)
```

In [configuration_esm.py lines 104–113](https://github.com/huggingface/transformers/blob/main/src/transformers/models/esm/configuration_esm.py#L104-L113), the validation checks `self.sequence_state_dim % self.sequence_state_dim != 0` instead of `self.sequence_state_dim % self.sequence_head_width != 0`, making this validation dead code that never triggers. The error is eventually caught by secondary validation at lines [118-127](https://github.com/huggingface/transformers/blob/main/src/transformers/models/esm/configuration_esm.py#L118-L127), but with a less clear error message.

### Expected behavior

**WITHOUT FIX:**
```
ValueError: `sequence_state_dim` should be equal to `sequence_num_heads * sequence_head_width, got 100 != 3 * 30
```
**WITH FIX:**
```
ValueError: `sequence_state_dim` should be a round multiple of `sequence_head_width`, got 100 and 30

Comments:
fix: Correct ESM TrunkConfig self-modulo in validation
Author: harshaljanjani
Pull request: yes

### What does this PR do?

`TrunkConfig` validation contained a self-modulo check that would never trigger (dead code). This PR corrects the divisibility check to use the head width instead.

Fixes #43347.

### Before submitting

* [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
* [x] Did you read the [[contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
  Pull Request section?
* [x] Was this discussed/approved via a GitHub issue or the [[forum](https://discuss.huggingface.co/)](https://discuss.huggingface.co/)? Please add a link
  to it if that's the case.
* [x] Did you make sure to update the documentation with your changes? Here are the
  [[documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs)](https://github.com/huggingface/transformers/tree/main/docs), and
  [[here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation)](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).

cc: @sayakpaul @Rocketknight1

Comments:
Thanks for the review, @Rocketknight1 -- that sounds about right. I've made the change.
Also I was wondering if I may ask if [this PR](https://github.com/huggingface/transformers/pull/43336) may be reviewed when time permits as well!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: esm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
GLM-Doc Update
Author: zRzRzRzRzRzRzR
Pull request: yes

Update documentation for GLM-4.5, GLM-4.6, and GLM-4.7, as some models indeed lack official documentation and technical reports.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4_moe_lite

Thanks for bearing with me :hugs: merging 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43338). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CB] PrefillFirstScheduler: prioritize waiting requests before decoding 
Author: dstekanov
Pull request: yes
Labels: name: Code agent slop

# What does this PR do?

This PR fixes the TODO comment: `prioritize adding from waiting reqs before scheduling RequestStatus.DECODING when cache space allows it` in PrefillFirstScheduler.

The change also adds the first unit tests for the scheduler module.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@remi-or @ArthurZucker @McPatate

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43337&sha=c9909e

@Rocketknight1  Oh, I know I don't need these tests, but what about "src/transformers/generation/continuous_batching/scheduler.py" ? I believe it correctly addresses the FIXME comment. 

The fix reorders candidates in PrefillFirstScheduler:

**Before:** `candidates = priority_states + [DECODING, waiting]`  
**After:** `candidates = priority_states + waiting_states + decoding_states`

The existing `_process_candidates` logic handles "when cache space allows it":
 
1. Iterates through candidates in order
2. Calls `_allocate_blocks_if_needed()` for each request
3. If allocation fails -> `continue` to next candidate
4. If allocation succeeds -> schedule the request
 
as a result when cache available the `waiting` scheduled before decoding and when cache is insufficient for `waiting` it will be skipped and decoding proceeds