noamaanMulla-03/transformers-issues huggingface dataset:
Fix: AutoVideoProcessor returns incorrect shape for batched tensor inputs (#43450)
Author: Aakash0440
Pull request: yes

When passing a batched tensor (B x T x C x H x W) directly to AutoVideoProcessor, the processor incorrectly adds extra singleton dimensions, returning a tensor of shape:
1 x 1 x B x T x C x H x W
instead of the expected:
B x T x C x H x W
This occurs because the processor assumes any tensor input is a single video and wraps it in a list, which adds extra batch/sample dimensions.

Fix Implemented:
- Added a check for already-batched tensor inputs (ndim == 5) in the video processor.
- Batched tensor inputs are now processed as-is, without adding extra dimensions.
- Compatibility maintained for:
    - Single video input
    - List of videos
    - Batched tensor input

Test Added:
- test_video_processor_batched_tensor() ensures that passing a batched tensor returns the correct shape (B x T x C x H x W) without extra dimensions.

Issue Reference:
Fixes #43450

Checklist:
- [x] Tested locally with pytest
- [x] Added a minimal reproducible test case
- [x] Code is clean and follows project conventions

This PR ensures that AutoVideoProcessor behaves correctly when handling batched video tensors, making batched processing simpler and avoiding unnecessary reshaping.

Comments:
There's an internal PR for this at #43486, so that one will probably take priority! Thank you for the contribution though, I hope this doesn't discourage you from contributing in future
refactor: use BatchLinear in Qwen2Moe to enable PEFT/LoRA support
Author: karthikthota-03
Pull request: yes
Labels: name: Code agent slop

**Related Issue** Fixes #43472

**Overview**  This PR is to refactor the expert implementation in Qwen2Moe by introducing a standardized BatchLinear utility.

The current implementation of Qwen2MoeExperts relies on a custom expert registry and manual looping. While this works for standard inference, it creates a major bottleneck for Parameter-Efficient Fine-Tuning (PEFT). Specifically, because the expert weights aren't stored as standard nn.Parameter tensors in a way that maps to linear logic, tools like LoRA cannot easily target the experts.

**Changes**

src/transformers/pytorch_utils.py: Added a BatchLinear class. This module stores expert weights in a single 3D tensor [num_experts, out_features, in_features] and utilizes torch.bmm for the forward pass. This makes the weights discoverable as standard parameters for PEFT and quantization.

src/transformers/models/qwen2_moe/modeling_qwen2_moe.py: Refactored Qwen2MoeExperts to use the new BatchLinear layers for gate_up_proj and down_proj. I've also removed the @use_experts_implementation dependency for this model to simplify the logic.

src/transformers/__init__.py: Added BatchLinear to the lazy-loading structure.

**Scope and Future Work** I have started with Qwen2-Moe to demonstrate the effectiveness of this architecture. By moving to a 3D tensor format, we unlock the ability for LoRA to target these layers directly.

If the maintainers prefer, I am happy to include similar refactors for all other MoE models (such as Mixtral, DeepSeek, etc.) directly within this PR. This would ensure a unified implementation across the library and provide library-wide support for MoE-LoRA immediately.

**Testing** I've verified the implementation locally:

Confirmed the forward pass produces the correct output shapes.

Verified that weight parameters are now correctly registered and accessible via model.named_parameters().


Looking forward to your feedback on whether I should proceed with the other models here!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2_moe

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43478&sha=ff7879

Sorry, we generally don't review "drive-by" code agent PRs from new contributors anymore! They usually have a lot of issues and it's very annoying to take the time to review/comment on them.
Phi4MultimodalConfig incorrectly initializes default vision/audio configs when passed as None
Author: charlieJ107
Pull request: no
Labels: name: bug

### System Info

> Note: This bug report has been refined by AI, I have review the report and taken the full responsibility for this issue. 

## 🐛 Bug description

In `Phi4MultimodalConfig.__init__`, there are two issues related to default initialization of multimodal sub-configs:

1. When `vision_config` is `None`, a default `Phi4MultimodalVisionConfig()` is instantiated, but **not assigned back** to the `vision_config` variable.
2. When checking whether to initialize `audio_config`, the code mistakenly checks `vision_config is None` instead of `audio_config is None`.

As a result, when users construct `Phi4MultimodalConfig` without explicitly passing `vision_config` or `audio_config`, the resulting config may contain `None` instead of valid default sub-configs.

## 🔬 Affected code

Current implementation (simplified):

```python
if isinstance(vision_config, dict):
    vision_config = Phi4MultimodalVisionConfig(**vision_config)
elif vision_config is None:
    Phi4MultimodalVisionConfig()
self.vision_config = vision_config

if isinstance(audio_config, dict):
    audio_config = Phi4MultimodalAudioConfig(**audio_config)
elif vision_config is None:
    audio_config = Phi4MultimodalAudioConfig()
self.audio_config = audio_config
```

# 🧩 Proposed fix

I have prepared a PR #43480 that:

- Correctly assigns default config objects when inputs are `None`
    
- Fixes the incorrect conditional check for `audio_config`
    

Happy to submit it if this behavior is confirmed as intended.

## 📦 Environment

- transformers version: `5.0.0`
    
- model: Phi-4 Multimodal
    
- Python: `3.13 Linux amd64`

### Who can help?

@Cyrilvallez 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Initialize `Phi4MultimodalConfig` with no parameters.
```python
>>> config = Phi4MultimodalConfig()
```
2. Check the vision config
```python
>>> print(config.vision_config)
None
```


### Expected behavior

The vision_config in` Phi4MultimodalConfig` when initialized with no parameters should have default values in `Phi4MultimodalVisionConfig` initialized. 

Comments:
cli: Improve serve missing dependency error
Author: antznette1
Pull request: yes
Labels: name: Code agent slop

## Summary
Improve the UX of `transformers serve` when serving dependencies are not installed by raising an actionable `ImportError` that lists the missing packages and provides copy/pastable installation commands.

## What changed
- Updated src/transformers/cli/serve.py to:
   - detect which of `pydantic`, `fastapi`, `uvicorn`, `openai` are missing
   - include a missing-deps list in the error message
   - include install instructions:
      - `pip install transformers[serving]`
      - or `pip install pydantic fastapi uvicorn openai`
- Added a unit test in tests/cli/test_serve.py to assert the new error message content when deps are missing.

Comments:
Remove stale TODO comments in UDOP tied weights
Author: raimbekovm
Pull request: yes

## Summary
Remove outdated TODO comments claiming patch embedding weight tying is "not working".

## Details
Testing confirms the tying mechanism works correctly:
- `patch_embed.proj.weight` and `encoder.embed_patches.proj.weight` share the same tensor
- Only source weights are saved to state_dict
- Weight tying is correctly re-established when loading

The TODO was added during #41580 refactor but the mechanism works identically to token embedding tying.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: udop

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43477). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lol yeah that happens
argparser: Allow optional bool flags without values
Author: antznette1
Pull request: yes

## Summary
Improve HfArgumentParser boolean UX so `Optional[bool]` arguments can be passed as flags without an explicit value.

## What changed
- Updated HfArgumentParser to treat `typing.Optional[bool]` `(Union[bool, None])` the same as `bool | None`:
   - keeps default `None`
   - allows `--flag` (no value) to parse as `True` via `nargs="?"` + `const=True`
- Updated unit tests in tests/utils/test_hf_argparser.py to reflect the new argparse configuration and verify `--opt` works without an explicit value.

Comments:
Is there an issue or reproducer to show the bug that this is fixing..?

> Is there an issue or reproducer to show the bug that this is fixing..?

This is fixing a CLI UX bug in HfArgumentParser for `Optional[bool]`.
Before this PR, an `Optional[bool]` flag behaves like a value-taking option, so passing the flag without a value errors (argparse expects an argument), even though most CLIs treat `--flag` as “enable”.

Reproducer:
>> from dataclasses import dataclass
from typing import Optional
from transformers.hf_argparser import HfArgumentParser
@dataclass
class Args:
    opt: Optional[bool] = None
parser = HfArgumentParser(Args)
parser.parse_args_into_dataclasses(["--opt"])  # previously errors, now sets opt=True

This PR makes `Optional[bool]` behave like a standard boolean flag: `--opt` sets `True`, while still allowing explicit values (`--opt false`) and preserving the default `None`.
Fix GLM-Image integration tests to use correct model class
Author: engmohamedsalah
Pull request: yes

## What does this PR do?

Fixes #43344 

The integration tests for GLM-Image were failing because they were attempting to load the `zai-org/GLM-4.5V` checkpoint (a GLM-4V MoE model) using `GlmImageForConditionalGeneration` classes, causing architecture mismatches.

## Issue Details

The tests were failing with the following mismatches:
1. **lm_head vocab size mismatch**: checkpoint has 151,552 tokens (text vocab) but GlmImage expects 16,512 (vision vocab for image generation)
2. **patch_embed shape mismatch**: checkpoint uses Conv3d `[1536, 3, 2, 14, 14]` but GlmImage uses Conv2d `[1536, 3, 14, 14]`

## Root Cause

GLM-4V and GLM-Image are fundamentally different model architectures:
- **GLM-4V**: Vision-to-text model (similar to LLaVA) using Conv3d for video support
- **GLM-Image**: Text-to-image generation model using Conv2d for images only

## Solution

Updated the integration tests to use `Glm4vMoeForConditionalGeneration` instead of `GlmImageForConditionalGeneration`, which correctly matches the `zai-org/GLM-4.5V` checkpoint architecture.

## Changes Made

- Added import for `Glm4vMoeForConditionalGeneration`
- Updated `get_model()` method to use `Glm4vMoeForConditionalGeneration`
- Updated `test_small_model_integration_test_batch_flashatt2()` to use `Glm4vMoeForConditionalGeneration`

## Testing

The tests were previously failing with size mismatch errors. With this fix, the model loads correctly with the matching architecture.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm_image

cc @zucchini-nlp do we want these tests in this file, then, or should they move to the MoE class instead?
invalid test cases for glm_image model
Author: kaixuanliu
Pull request: no
Labels: name: bug

### System Info
A100 w/ 80GB memory
For test cases:
```
tests/models/glm_image/test_modeling_glm_image.py::GlmImageIntegrationTest::test_small_model_integration_test_batch_flashatt2
tests/models/glm_image/test_modeling_glm_image.py::GlmImageIntegrationTest::test_small_model_integration_test_batch
```
it will throw error when loading model using `from_pretrained`, as there are mismatched sizes between the checkpoint and what the model architecture expects. Even after I set `ignore_mismatched_sizes: bool = False,
`, the test cases still fail w/ error:
```
lm_head.weight                                                           | MISMATCH   | Reinit due to size mismatch ckpt: torch.Size([151552, 4096]) vs model:torch.Size([16512, 4096])
model.visual.patch_embed.proj.weight                                     | MISMATCH   | Reinit due to size mismatch ckpt: torch.Size([1536, 3, 2, 14, 14]) vs model:torch.Size([1536, 3, 14, 14])
```


### Reproduction

git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
export RUN_SLOW=1
pytest -rA tests/models/glm_image/test_modeling_glm_image.py::GlmImageIntegrationTest::test_small_model_integration_test_batch_flashatt2

### Expected behavior

cases pass

Comments:
Closing as code agents seem to be attracted to any open issue. It will be resolved in https://github.com/huggingface/transformers/pull/43342  :)
Skip glm_image integration tests (wrong checkpoint)
Author: raimbekovm
Pull request: yes

# What does this PR do?

The integration tests for `GlmImageForConditionalGeneration` were failing because they load `zai-org/GLM-4.5V` which is a `glm4v_moe` model, not `glm_image`. This causes shape mismatches:

```
lm_head.weight: [151552, 4096] vs [16512, 4096]  
patch_embed.proj.weight: [1536, 3, 2, 14, 14] vs [1536, 3, 14, 14]
```

The correct checkpoint is `zai-org/GLM-Image` with `subfolder='vision_language_encoder'`.

There's also a deeper issue — tests use VLM prompts ("What kind of dog is this?") but GlmImage is for image generation, not visual Q&A. So the test logic itself needs rewriting.

This PR skips the tests until they can be properly fixed.

Fixes #43344

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #43344
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

## Who can review?

@zucchini-nlp @ydshieh

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm_image

For context, here's how the checkpoint should be loaded according to the docs:

```python
model = GlmImageForConditionalGeneration.from_pretrained(
    "zai-org/GLM-Image",
    subfolder="vision_language_encoder",
    ...
)
processor = AutoProcessor.from_pretrained(
    "zai-org/GLM-Image",
    subfolder="processor",
    ...
)
```

The model outputs vision tokens for a diffusion decoder, not text answers. So the test prompts and expected outputs would need to match that behavior.

Looks like the tests were added in #43100 and probably written before the checkpoint was fully available on the hub.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43354&sha=9f995a

cc @ydshieh as well

Actually, I think slow tests will be added as part of https://github.com/huggingface/transformers/pull/43342 by the authors :)

Glad I could help identify the issue — feel free to ping me if you need any help! рахмет:)

Let's close the PR here, thanks for your help!
Add missing GPURawMetrics.from_dict() method in benchmark_v2
Author: Abdennacer-Badaoui
Pull request: yes

  Adds the missing `from_dict()` classmethod to the `GPURawMetrics` dataclass in `benchmark_v2/framework/hardware_metrics.py`.      
  
 (`BenchmarkResult.from_dict()` is used at line 144 of `data_classes.py`)
                                                                                                                                                              

Comments:
Fix SAM-HQ weight loading bug causing flaky tests
Author: engmohamedsalah
Pull request: yes

## What does this PR do?

Fixes #42890 by addressing the **root cause**: a weight loading bug introduced in commit `0b369802cf`.

## Root Cause

Commit `0b369802cf` ("fix sam family!") removed `_tied_weights_keys` because positional embeddings are buffers (not parameters) and cannot be tied. However, it left `_keys_to_ignore_on_load_missing` in place, causing:

```python
# BUGGY CODE (before this PR):
_keys_to_ignore_on_load_missing = ["prompt_encoder.shared_embedding.positional_embedding"]
```

This caused `prompt_encoder.shared_embedding.positional_embedding` to be **randomly initialized on every model load** instead of being loaded from the checkpoint.

The flaky integration tests were actually **exposing this weight loading bug** - outputs varied because one embedding was randomly initialized each time with `torch.randn()`.

## The Fix

Remove `_keys_to_ignore_on_load_missing` entirely so both positional embeddings are properly loaded from checkpoint:

```python
# FIXED CODE (this PR):
class SamHQModel(SamModel):
    # (removed _keys_to_ignore_on_load_missing)
    def __init__(self, config):
        ...
```

## Changes

- **src/transformers/models/sam_hq/modular_sam_hq.py**: Remove `_keys_to_ignore_on_load_missing`  
- **src/transformers/models/sam_hq/modeling_sam_hq.py**: Regenerated from modular file
- **tests/models/sam_hq/test_modeling_sam_hq.py**: Remove temporary `set_seed()` workaround from previous PR

## Testing

The integration tests should now pass consistently without requiring `set_seed()` because the model loads the correct weights from the checkpoint.

## Credit

Special thanks to @Rocketknight1 for identifying that this was a weight loading bug rather than just a randomness issue!

---

**Note**: This PR supersedes #43211 which attempted to fix the symptom (randomness) rather than the root cause (weight loading bug).

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sam_hq

cc @nielsrogge @yonigozlan @molbap I think this makes sense and I think you're the right maintainers for SAM, can you review?

Thanks for raising this! However this is already fixed on main by https://github.com/huggingface/transformers/pull/43420, so closing this PR.
Add pytest-random-order for reproducible test randomization
Author: tarekziade
Pull request: yes

# What does this PR do?

Uses bucket=module to randomize tests within files, compatible with --dist=loadfile. CI uses CIRCLE_BUILD_NUM as seed for reproducibility across parallel containers. Local runs use random seed to catch order dependencies over time. Enables debugging by reproducing exact test order from failed CI builds.
<!-- Remove if not applicable -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: vivit

💔 This comment contains `run-slow`, but unknown error occurred and [the workflow run](https://github.com/huggingface/transformers/actions/runs/21352196657) aborted!

run-slow: vivit

This comment contains `run-slow`, running the specified jobs: 

models: ["models/vivit"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/21352407697)

✅ No failing test specific to this PR 🎉 !


Nice to have this  🚀 ! LGTM just a few nit comments. Let you to make the final call if to update.
Fix failing `markuplm` integration tests
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?

Fixes failing [MarkupLMModelIntegrationTest](https://github.com/huggingface/transformers/actions/runs/21307980813/job/61339620056#step:14:1451).



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 @vasqu

Comments:
run-slow: markuplm, perception_lm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/markuplm", "models/perception_lm"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/21359987306)

✅ No failing test specific to this PR 🎉 !


> Seems like the perception lm fixes don't really work, not sure if it's a GPU diff tbh since it varies a lot
> 
> Imo we can make this focused on markuplm first and merge

Yeah, the intial error is fixed, but model is generating something totally weird. Reverted `perception lm` changes. Thanks for the review!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: markuplm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43464). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enhance repo consistence bot
Author: ydshieh
Pull request: yes

# What does this PR do?

Allow it to do a subset of simpler fixes instead a full set.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[don't merge] bad format to check repo bot
Author: ydshieh
Pull request: yes

# What does this PR do?

check repo bot, don't merge

Comments:
@bot /repo

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43488). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /style

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: vit

@bot /repo

Repo. Consistency bot fixed some files and pushed the changes.