noamaanMulla-03/transformers-issues huggingface dataset:
[Core] [Offloading] Enable saving offloaded models with multiple shared tensor groups
Author: kylesayrs
Pull request: yes

## Purpose ##
* Fixes https://github.com/huggingface/transformers/issues/39097
* Enables models with multiple groups of shared tensors to be saved
  * For example, the token_embeds and lm_head weights are one group of shared tensors
  * Right now, only one group can be shared at a time, as indicated by `tied_names = tied_params[0]`
  * This enablement is useful for registering additional modules with shared tensors, such as quip/spinquant-like transforms

## Changes ##
* Instead of building a separate pathway for calculating shared tensors when offloaded, instead update `ptrs` calculation logic to be capable of handling meta tensors
  * This is equivalent to the `find_tied_parameters` method, since in the meta device case, both check for equality of the python object itself
* Misc, update tied tensors runtime error to suggest using `_dynamic_tied_weights_keys`, which is also a valid way of saving tied tensors

## Testing ##
```python3
import torch
from transformers import AutoModelForCausalLM

device_map = {
    "model.embed_tokens": "cuda:0",
    "model.layers": "cpu",
    "model.norm": "cpu",
    "model.rotary_emb": "cpu",
    "lm_head": "cuda:0"
}
model = AutoModelForCausalLM.from_pretrained("nm-testing/llama2.c-stories15M", device_map=device_map)

transform_a = torch.nn.Linear(1, 1, bias=False)
transform_a._dynamic_tied_weights_keys = ["weight"]
transform_b = torch.nn.Linear(1, 1, bias=False)
transform_b._dynamic_tied_weights_keys = ["weight"]

model.model.layers[0].self_attn.q_proj.register_module("transform", transform_a)
model.model.layers[1].self_attn.q_proj.register_module("transform", transform_a)
model.model.layers[2].self_attn.q_proj.register_module("transform", transform_b)
model.model.layers[3].self_attn.q_proj.register_module("transform", transform_b)

model.save_pretrained("tmp")
```

* Added test fails on main but succeeds with these changes

Comments:
I've validated that my added test passes. The CI failure seems to be unrelated
[Core] [Offloading] Fix saving offloaded submodules
Author: kylesayrs
Pull request: yes

## Purpose ##
* Fixes issue with saving offloaded modules with offloaded submodules whose parent is also offloaded
  * Sometimes `get_state_dict_from_offload` will update the state dict with meta tensors, since `align_module_device` only aligns direct parameters, not submodule parameters. If these submodule parameters are offloaded, they will be returned as meta tensors
* This fix is useful for models with finer-grained offloading as well as registering offloadable submodules such as transforms

## Changes ##
* This change attempts to load the shard state dict if either the entry is empty ("") or if the entry is a meta tensor (returned by a parent which does not place submodules)

## Testing ##
* Added test fails on main but succeeds with these changes

Comments:
Can you solve the conflicts! 

@ArthurZucker Done!

Hm, looks like a type error in accelerate. I wasn't able to recreate this locally, so perhaps this is a torch version issue
```python3
>                   new_value = param_cls(new_value, requires_grad=old_value.requires_grad).to(device)
E                   TypeError: new() received an invalid combination of arguments - got (Tensor, requires_grad=bool), but expected one of:
E                    * (*, torch.device device = None)
E                         didn't match because some of the keywords were incorrect: requires_grad
E                    * (torch.Storage storage)
E                    * (Tensor other)
E                    * (tuple of ints size, *, torch.device device = None)
E                    * (object data, *, torch.device device = None)

/usr/local/lib/python3.9/site-packages/accelerate/utils/modeling.py:363: TypeError
```

EDIT: the merge ended up removing the `accelerate` test decorators

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39280). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Responses API in `transformers serve`
Author: LysandreJik
Pull request: yes

This PR offers an initial scaffolding of `transformers serve` so that it has both `/v1/chat/completions` and `/v1/responses` endpoints.

So far, it only implements a streaming version of both endpoints.

To implement the responses endpoint, I'm basing myself off of the "Streaming" tab in the OpenAPI API ref for Responses available here: https://platform.openai.com/docs/api-reference/responses/create

I validate that the implementation works as expected by using the `openai` package directly:

```py
import sys
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct", instructions="You are a helpful assistant.", input="Hello!", stream=True
)

for event in response:
    print(event)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39155). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

☝️ oops, misclicked :D

(resolved conflicts vs #39230, accepting all changes from this PR. #39338 includes all the fixes needed for a correct rebase, and its diff is a clean diff vs [66af54e](https://github.com/huggingface/transformers/pull/39155/commits/66af54e6c196ce50ce684fd758405425a9da6a13))

Thanks @gante, LGTM after your changes!

Yay, congrats! :tada:
Better typing for model.config
Author: qubvel
Pull request: yes
Labels: name: typing

# What does this PR do?

Makes `model.config` type resolved to specific config

### On `main` (before)

`PretrainedConfig`, no model-specific attributes
 
![config-typings-before](https://github.com/user-attachments/assets/e22b1b51-4652-4444-a2ff-f92d5aac1cea)

### On branch (after)

`LlamaConfig` with specific attributes for Llama model

![config-typings-after](https://github.com/user-attachments/assets/da0c9540-238c-462f-832b-f24ff4bb4979)



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39132). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After some experiments, I found that using `Generic[Type]` can get a bit messy when working with multimodal models that have different config types. So, I ended up simply adding `config: <Type>` as an additional class attribute for each model that already defines a `config_class`.

This approach is much easier to follow, even for those not familiar with type annotations, since `Generic[Type]` is a fairly uncommon pattern.

@ArthurZucker, regarding reusing `config_class`, I didn't find a way to reuse it for typing purpose, let me know if you have anything in mind!

@ArthurZucker, it's ready for the next review! Thanks for the hint. I'm still not sure if it's the best option to mix static type annotations and runtime attributes, but it's not the only instance of this in the repository, so it might be acceptable. Also, it's a bit more complex in terms of backward compatibility with remote code models. But finally, it seems to work fine.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, albert, align, altclip, arcee, aria, audio_spectrogram_transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert_generation, big_bird
SigLip2 text pooler output selection
Author: Jason3900
Pull request: no
Labels: name: bug

name: Vision

### System Info

transformers==4.51.3
python==3.10.16
torch==2.7.0+cu128

### Who can help?

@amyeroberts @qubvel 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hey. I'm finetuning a siglip2 model and find some issues. 
From the original code from [google](https://github.com/google-research/big_vision/blob/0127fb6b337ee2a27bf4e54dea79cff176527356/big_vision/models/proj/image_text/text_transformer.py#L83), the text ids' last position is assumed to be eos token. While if I use transformers preprocessor following the doc, text whose token length is less than max_length will pad to max_length. In that case, the last token is not eos but pad. Since the [code](https://github.com/huggingface/transformers/blob/17b3c96c00cd8421bff85282aec32422bdfebd31/src/transformers/models/siglip2/modeling_siglip2.py#L692) always takes the last position to feed into the pool layer, I think there might be some mismatches?  
```
        # Assuming "sticky" EOS tokenization, last token is always EOS.
        pooled_output = last_hidden_state[:, -1, :]
        pooled_output = self.head(pooled_output)
```
And another question is the preprocessor's output doesn't contain attention_mask, should I manually build it according to padding positions or the original model doesn't use attention_mask for padding at all?

### Expected behavior

I want some explanation.

Comments:
Hey @Jason3900, thanks for opening the issue. SIglip and Siglip2 models were initially trained **without** an attention mask to mask text padding tokens, that's why the processor doesn't create it.

Regarding

```
        # Assuming "sticky" EOS tokenization, last token is always EOS.
        pooled_output = last_hidden_state[:, -1, :]
        pooled_output = self.head(pooled_output)
```

As far as I recall, I paid attention to it while converting the model, and the implementation should be OK, because we benchmarked converted Siglip2 on Imagenet and got the same score as the original implementation (as well as getting the same logits on dummy examples). It might be just a misleading comment

@qubvel Thanks a lot! To be clear, actually it should use the last token's hidden states to feed into the head layer regardless whether it's a pad or eos, right?

> To be clear, actually it should use the last token's hidden states to feed into the head layer regardless whether it's a pad or eos, right?

Yes.

Feel free to submit a PR to remove the comment if you have bandwidth 🤗 

Fixed in https://github.com/huggingface/transformers/pull/39378
[Bugfix] [Quantization] Remove unused init arg
Author: kylesayrs
Pull request: yes

## Purpose ##
* Remove argument which is not used by the CompressedTensors QuantizationConfig
  * https://github.com/neuralmagic/compressed-tensors/blob/main/src/compressed_tensors/quantization/quant_config.py#L138-L144

## Changes ##
* Remove `run_compressed` from list of QuantizationConfig init kwargs

Comments:
Yeah, that's checking `transformers.CompressedTensorsConfig.run_compressed`, not `compressed_tensors.QuantizationConfig.run_compressed`
Fix processor tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fix a failing test reported by @vasqu 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2_5_omni

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39450). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

It was failing just because we had template code overwritten, and it was always wrapping conversations in a list. The test compares string to list and fails
Remove something that should have never been there
Author: ArthurZucker
Pull request: yes

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38254). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bamba, granitemoehybrid

run-slow: bamba, granitemoehybrid

This comment contains run-slow, running the specified jobs: 

models: ['models/bamba', 'models/granitemoehybrid']
quantizations: [] ...
Fixes #39204: add fallback if get_base_model missing
Author: sebastianvlad1
Pull request: yes

This PR adds a fallback mechanism to extract the base model from a PEFT-wrapped model in case the get_base_model() method is missing. Specifically, it attempts to retrieve model.base_model.model if get_base_model is not available. This improves compatibility with different PEFT wrappers and custom model classes.

The logic has been extracted into a dedicated try_get_base_model utility function to improve code reuse and robustness.

Fixes #39204.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://github.com/huggingface/transformers/issues/39204)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
@ArthurZucker I've inlined the function as suggested. Thank you for your comment!

@ArthurZucker please see if the above can be approved. thank you.

@bot /style

Style bot fixed some files and pushed the changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39226). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Unrelated failures, failures, merging
[`CI`] Fix partially red CI
Author: vasqu
Pull request: yes

- #37358 modified a test which fails
- Make `fix-copies` revealed some strange behavior with the new modernbert docstrings -> "Args:..." is not necessary

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: git, modernbert_decoder

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39448). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Defaults to adamw_torch_fused for  Pytorch>=2.8
Author: cyyever
Pull request: yes

# What does this PR do?
 
According to the comment, transformers should default to adamw_torch_fused after PT 2.1. However, the condition is lift to PT 2.8 to ensure that adamw_torch_fused is mature enough.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@cyyever are there known bugs in fused_adamw in torch 2.1 - 2.5 ?

@Rocketknight1 For example , https://github.com/pytorch/pytorch/issues/101449

May be it's better to limit that to 2.7

Hmm - given that it's still unstable up to 2.7, maybe we should leave the default unfused for now until we're sure that 2.7 is okay? It's a very recent release.

I agree.

@Rocketknight1 PT2.8 is ready to release.

Can you make sure CI goes green? 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: git

@ArthurZucker A test failure is fixed, although it is unrelated 

Thanks
Vectorize deepseek moe
Author: eljandoubi
Pull request: yes

# What does this PR do?

Vectorize deepseek moe

## Who can review?

Models:

- text models: @ArthurZucker


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@ArthurZucker @Rocketknight1 any feedback back? 

Let's also align notations to what we have in `Llama4` this way we can leverage kernels

The `model.safetensors.index.json` of `deepseek-v3` has each expert separated; otherwise, I would have opted for that option from the beginning.

Is it possible to merge sperated weights from https://huggingface.co/deepseek-ai/DeepSeek-V3 of the experts into big parameters like `Llama4` ?

Sorry I don't undertand the question

DeepSeek V3 experts are implemented as a `ModuleList` of small MLPs, whereas LLaMA 4 uses a fused, large `nn.Parameter` for its experts. If we want to follow the LLaMA 4 approach, the stored weights would need to have the same fused structure.



That is something we want to do as well, but it needs a bit more refactoring as we need to: 
1. do it on the fly: for example if you look at Llama4 the register kernel does some hot swap + caching 
2. define an API to allow loading arbitrary state dict, and sharding

You must understand that there are various cases to take into account:
- tensor parallel
- device_map auto 
Both have a different weight format expectation. 

Now, If you are motivated I am down to review a PR to would introduce something like that, I might even work on it at some point for such cases! 

I'll add expert parallel for these model (a naïve implementation at least) this week!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v3, dots1
Updated Megatron conversion script for gpt2 checkpoints 
Author: LckyLke
Pull request: yes

This PR updates the convert_megatron_gpt2_checkpoint script to the newest Megatron version!

Comments:
Hi @LckyLke, do you have an example of a checkpoint that fails with the old script but passes with this one? Just so we can verify that it's working as expected.

Also, code quality failures can be resolved with `pip install -e .[quality]` followed by `make fixup`

Hi @Rocketknight1, a Megatron-LM training setup like here: 
```bash
srun --export=ALL torchrun ${DISTRIBUTED_ARGS[@]} ../Megatron-LM/pretrain_gpt.py \
    --ckpt-format torch \
    --use-legacy-models \
    --attention-softmax-in-fp32 \
    ${GPT_MODEL_ARGS[@]} \
    ${TRAINING_ARGS[@]} \
    ${MODEL_PARALLEL_ARGS[@]} \
    ${DATA_ARGS[@]} \
    ${EVAL_AND_LOGGING_ARGS[@]}
```
 and than converting using `PYTHONPATH=Megatron-LM python ../transformers/src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py checkpoints/.../model_optim_rng.pt `

will not convert to a transformers model. Importantly, I used the legacy model format and a single torch checkpoint (not distributed, as i think the script is made for a single .pt file), but the mcore model format would not convert either and fail even earlier. That why assumed this scripted was made for the legacy format, but as mentioned it would not either work. 

Also #32486 had similar problems, but I think their fix is outdated as well. 
 


Linking #39048 which is also reporting issues with the conversion script

Hi, sorry for the delay! I checked over this issue again and I think `torch.load(weights_only=False)` is actually fine, so I'm sorry about the difficulties I caused by flagging that. The reason is that conversion scripts are generally not intended to be secure, because they often have to load potentially-malicious files in external formats, and as a result we remove them from release versions.

@LckyLke are there any other changes at this point or are you happy with the PR?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38969). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1 this script currently only works for Megatron models trained in their 'legacy' format - ig this should be mentioned somewhere? 
When i have more time next week I can implement another one handling their newer 'mcore' format but for the legacy format the script is fine as is :) 
-> and i think it makes more sense to handle mcore in a separate script for the time being 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: megatron_gpt2

@LckyLke if you're adding a new script to handle the modern format, we can add the notice then! Until then, I'm happy to merge this one in its current state if you are.

@Rocketknight1 ok yes, sounds good 👍🏼 

Merging, and thanks for the PR!
Fix indentation bug in SmolVLM image processor causing KeyError
Author: Krish0909
Pull request: yes

# What does this PR do?

This PR fixes an indentation bug in `SmolVLMImageProcessorFast` that causes a `KeyError` when `do_image_splitting=False`.

The issue was in line 434 where `split_images_grouped[shape] = stacked_images` was incorrectly indented outside the loop, causing only the last shape to be stored in the dictionary. When `reorder_images()` tries to access other shapes, it fails with a KeyError.

**Fix:** Move the assignment inside the loop with proper indentation (add 4 spaces).

**Before:**
```python
for shape, stacked_images in grouped_images.items():
    # ... resize code ...
split_images_grouped[shape] = stacked_images  # Outside loop - bug!
```
**After:**
```python
for shape, stacked_images in grouped_images.items():
    # ... resize code ...
    split_images_grouped[shape] = stacked_images  # Inside loop - fixed!
```

Fixes #39442
Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
 Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
 Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
 Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
 Did you write any new necessary tests?

Who can review?
@yonigozlan @amyeroberts @qubvel

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: idefics3, smolvlm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39452). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing 4 spaces in SmolVLMImageProcessorFast
Author: gnobitab
Pull request: no
Labels: name: bug

### System Info

transformers==4.53.0 the bug still exists in the latest github version

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Set do_image_splitting=False

### Expected behavior

It reports: KeyError: torch.Size([590, 2048])

I checked the code and I believe four spaces are missing in this line:
https://github.com/huggingface/transformers/blob/0dc2df5ddafe3cb5824ad24e85beba13e0aa6726/src/transformers/models/smolvlm/image_processing_smolvlm_fast.py#L434

I fixed the spacing issue and it runs good on my local machine. Could you have a look and fix this in the next version? Thx. @yonigozlan 



Comments: