noamaanMulla-03/transformers-issues huggingface dataset:
[CI] doc builder with `custom_runner`
Author: gante
Pull request: yes

# What does this PR do?

WIP, don't merge, testing https://github.com/huggingface/doc-builder/pull/547

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).
Add model visual debugger
Author: molbap
Pull request: yes

# What does this PR do?

This PR introduces a tool to trace ALL model inputs and outputs to a json nested format. Main values, shapes, dtypes are all outputted.

# Usage

It's a simple context manager to add before calling a forward on your inputs. To note, `inference_mode` is enforced here.

```python
import torch
from PIL import Image
from transformers import LlavaProcessor, LlavaForConditionalGeneration
from transformers.model_debugging_utils import  model_addition_debugger_context

# load pretrained model and processor
model_id = "llava-hf/llava-1.5-7b-hf"
processor = LlavaProcessor.from_pretrained(model_id)
model = LlavaForConditionalGeneration.from_pretrained(model_id, low_cpu_mem_usage=True)

# create random image input
torch.random.manual_seed(673)
random_image = Image.fromarray(torch.randint(0, 256, (224, 224, 3), dtype=torch.uint8).numpy())

# prompt
prompt = "<image>Describe this image."

# process inputs
inputs = processor(text=prompt, images=random_image, return_tensors="pt")

with model_addition_debugger_context(model=model, debug_path="model_debug"):
    output = model.forward(**inputs)

```

# Why??

Because when porting models to `transformers`, even from python to python, model adders often have to do a lot of manual operations, involving saving and loading tensors, comparing dtypes, etc. This small tool can hopefully shave off some time.



## Example output

Here, you can see two jsons of the same model, where the sole difference is the epsilon of a layer normalization. I added the decorator, and we can see with string matching that the outputs start to differ. I'll include a snippet with `difflib` too I think to make it simpler. 

![Screenshot from 2025-03-18 16-44-16](https://github.com/user-attachments/assets/7e171eea-a6f7-40f4-8e5d-aee314f35aa4)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36798). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[torchao] revert to get_apply_tensor_subclass
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR reverts a method name in torchaoconfig back to `get_apply_tensor_subclass` as this might be breaking for other libraries such as peft. 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).
Gemma3: fix test
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, got a ping from our CI bot about it


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

run-slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

run-slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

The failing test -> 'tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_model_parallel_beam_search'

is not related to the PR and seems was broken always. I don't know if static cache + multi GPU ever worked with VLMs. In Gemma3 we can't turn off static cache, so will check later. Not big issue for now I hope

Will merge to unblock others :)
[CI] fix update metadata job
Author: gante
Pull request: yes

# What does this PR do?

The `Update Transformers metadata / build_and_package` job is failing on main. This PR fixes underlying issues (see comments on the line changes)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

@ArthurZucker yes, it seems to be another separate issue, having a look!
Refactor Aya Vision with modular
Author: yonigozlan
Pull request: yes

# What does this PR do?

Inherit from Llava with modular to remove most of the custom code.

I ran the integration tests with both 8B and 32B version and everything seems to work as expected.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Hey @Cyrilvallez !
I have made a little modifications in `modular_model_converter`, the idea is to be able to provide a custom `MODEL_NAME_START/INPUT_DOCSTRING`.  I used an ugly workaround to also keep the current behavior, that is to copy the `MODEL_NAME_START/INPUT_DOCSTRING` of the inherited model when it is set to `None`:
https://github.com/huggingface/transformers/blob/0ba87849c78f2f2c15fd6fcf5b4aeebf65e3357a/utils/modular_model_converter.py#L861-L865

This new behavior is also extended to all constants in `ASSIGNMENTS_REGEX_TO_KEEP`. It doesn't crash with any of the current modular files, but it does seem like in this state it could crash at any moment 😅. Maybe there is a better way to check if the value of a constant is None with `libcst`?


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36688). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @yonigozlan! I agree that we should allow something like that. It is a bit magic, and it's not ideal, but for now it will make our lives easier with the docstrings (while we wait on better docstring support). What you did is quite close to what is proposed in https://github.com/huggingface/transformers/pull/35902/files#diff-dcfd09afc0f209bd5e8403add12d0cc6707dca021dcb639bc0b9a95c41843c7e.

Actually, I'd like a mix of both approaches: yours checks for all other patterns None value, and the one proposed by @nikosanto13 is a bit opaque, being directly hard-coded in the `merge_assignment` function. What I believe is the best approach right now would be to have a new list `ASSIGNMENTS_REGEX_TO_KEEP_IF_NOT_NONE` or something along those lines, and perform the None check only for assignment matching those patterns (which would be only `r""DOCSTRING"` as of now I think?)

Let me know what you think. It's a bit annoying to have the `None` magic, but I don't really see any better solution for those big blocks of docstrings 🥲🥲

@Cyrilvallez sounds good to me! Maybe it's best that I open a separate PR to make those changes then :)

Ready for final review @ArthurZucker ;)
Batched pipeline
Author: berryweinst
Pull request: no
Labels: name: wontfix

Hi, 

Is there a way to run batches with QuestionAnsweringPipeline rather than just one example?

Thanks.

Comments:
Yes, building off of the model and example [here](https://huggingface.co/twmkn9/albert-base-v2-squad2): 

```
from transformers.pipelines import pipeline

model_name = "twmkn9/albert-base-v2-squad2"
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
QA_input = {
    'question': 'Why is model conversion important?',
    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'}, {
    'question': 'What is the name of the repository ?',
    'context': 'Pipeline have been included in the huggingface/transformers repository. '
    }
res = nlp(QA_input, handle_impossible_answer=True)
print(res)
# [{'score': 0.2479676753282547, 'start': 59, 'end': 132, 'answer': 'gives freedom to the user and let people easily switch between frameworks.'}, {'score': 0.5168691277503967, 'start': 35, 'end': 71, 'answer': 'huggingface/transformers repository.'}]


```



Hi. 

I used your example for testing. It seems like even though I put multiple question-context pairs in as input, it really is just doing a one-by-one prediction on them in the background. 
So for 1 example the inference time is: 0.56 sec
For 2 examples the inference time is: 1.05 sec
For 16 examples it is: 8.4 sec., etc..

Is there a way to do batch inference with the model to save some time ? (I use 12 GB gpu, transformers 2.4.0 or 3.2.0)
 
Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Possible duplicate of #3007

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

still an open point. highly required, any information on the progress?

This is implemented in recent versions: https://huggingface.co/docs/transformers/master/en/main_classes/pipelines#pipeline-batching

cc @Narsil 

For the sake of completion: 
```python
pipe = pipeline('question-answering', model=model_name, tokenizer=model_name)
questions = [{"question": "Who am I ?", "context": "There is something about me"}, .... ]
for answer in pipe(questions, batch_size=16):
    print(answer)
````
Mllama: raise better error
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Passing images as a flat list do not give same logits in mllama, as passing the images in nested batch format. To get the model working correctly, one should pass as many images per batch, as there are image tokens.

See below reproducer

```python
import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = ["

What do you see here?", "

What do you see here but longer?"]

repo_id = "mv11/11"
processor = AutoProcessor.from_pretrained(repo_id)
model = MllamaForConditionalGeneration.from_pretrained(repo_id, device_map='auto')

batch = processor(text=texts, images=[image, image], return_tensors="pt", padding=True) # .to(model.device)
with torch.no_grad():
    model_output = model(
        input_ids = batch['input_ids'],
        attention_mask = batch['attention_mask'], 
        pixel_values = batch['pixel_values'], 
        aspect_ratio_ids = batch['aspect_ratio_ids'], 
        aspect_ratio_mask = batch['aspect_ratio_mask'], 
        cross_attention_mask = batch['cross_attention_mask'],
        )


batch = processor(text=texts, images=[[image], [image]], return_tensors="pt", padding=True) # .to(model.device)
with torch.no_grad():
    model_output_2 = model(
        input_ids = batch['input_ids'],
        attention_mask = batch['attention_mask'], 
        pixel_values = batch['pixel_values'], 
        aspect_ratio_ids = batch['aspect_ratio_ids'], 
        aspect_ratio_mask = batch['aspect_ratio_mask'], 
        cross_attention_mask = batch['cross_attention_mask'],
        )

print(torch.allclose(model_output_2.logits, model_output.logits))
>>> False
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35934). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker yeah, updated the test and added check for this condition. We might need to update idefics models also, but I will do it later in another PR
[WIP] Add EVEv2 model
Author: sbucaille
Pull request: yes
Labels: name: New model

name: Vision

name: Multimodal

# What does this PR do?

Add EVEv2 model

Fixes #36379

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @qubvel 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

cc @zucchini-nlp for VLMs as well!

Hi @sbucaille! Great model! 

I would recommend to upload the model to the hub as custom code using [this guide](https://huggingface.co/docs/transformers/en/custom_models). This will let you share the model immediately, and it'll work exactly the same as a library model (except that users will need to set `trust_remote_code=True`)

In general we accept PRs to add new architectures to the core transformers library when one of these are true:

- There's a pretrained model with a lot of interested users
- There's a paper on the architecture with SOTA results or lots of interest/citations
- The model comes from a company or research group whose past models have gotten a lot of usage (because this means the new model will probably get a lot of users too)
The reason for this is that once a model is actually in transformers itself, then the team at Hugging Face takes full responsibility for maintaining the code, testing it and making sure it stays compatible with new versions of transformers. We can't do this for every model architecture!

Remember that just because a model is a custom code model, doesn't make it less important. A lot of extremely popular and high-performance models are custom code models and don't have Transformers PRs, for example [Phi-3.5-vision](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) with >250k downloads per month. Starting with a custom code model is definitely the right approach for most authors! If the model gets high usage and many user requests with custom code, we can add it to the core library 🤗 

Hi @zucchini-nlp ,

I hear you, I'll close this PR then and make a custom model to see how it goes in the future with this model :)
[CI] doc builder without custom image
Author: gante
Pull request: yes

# What does this PR do?

Using a custom image is now resulting in out of disk space issues. This PR removes the use of a custom image, and relies in the original job setup phase.

This PR also:
- Fixes docs issues from other PRs merged on 2025-03-20 (the doc builder CI was down)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36862). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
:red_circle: :red_circle: :red_circle: supersede paligemma forward to shift pos id indexing
Author: molbap
Pull request: yes

# What does this PR do?

Should fix #36856 

This is because, in the modular file, `Gemma3ForConditionalGeneration` inherits from `PaliGemmaForConditionalGeneration` where this specific fix happens. But it is specific to PaliGemma input ordering and should not have been propagated to Gemma3. 

Comments:
run slow: gemma3

run_slow: gemma3

run_slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

@molbap BTW, this is not the only place where this shift occurs, it also happens in `prepare_inputs_for_generation`, so you probably need to fix that too.

Nice catch, thanks a lot @oceanusxiv :hugs: 
I repushed the slow tests, seems there's no change. To note, `beyond_sliding_window` tests seem broken though.

run_slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

run_slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36859). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

don't merge yet- the 4b crops test with pan_and_scan is broken with this change. Unsure yet if related to #36864 , checking

Ok, I think unrelated as well, https://huggingface.co/google/gemma-3-4b-pt/commit/2166bd1fba050eab3ed4fdd0b591ad1e11c96802 added another eos_token which causes the generation to stop earlier. Confirming, fixing in #36882 then we can merge

Not sure what is causing the difference, but it's not big AFAIK and other integration tests pass. I'll merge now.
Gemma 3 tests expect greedy decoding
Author: molbap
Pull request: yes

# What does this PR do?

Slow tests in gemma 3 expect greedy decoding. Recently hub config has changed https://huggingface.co/google/gemma-3-4b-pt/commit/2166bd1fba050eab3ed4fdd0b591ad1e11c96802 so it is preferred to enforce greedy decoding.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36882). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: dtype cannot be str
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36258


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36262). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bug introduced in `from_pretrained` `v4.48.3`..`v4.49.0`
Author: hlky
Pull request: no
Labels: name: bug

Hi 🤗

Diffusers 🧨 noticed some failing tests starting with `v4.49.0` in `Kolors`, one of our models that uses a custom text encoder.

## Reproduction

This is working on `v4.48.3`.

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b", trust_remote_code=True)
```

On `v4.49.0`:

```python
TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=str, device=str), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
```

The issue seems to be that the config in the [test model](https://huggingface.co/hf-internal-testing/tiny-random-chatglm3-6b/blob/main/config.json) and checkpoints like [`Kwai-Kolors/Kolors-diffusers`](https://huggingface.co/Kwai-Kolors/Kolors-diffusers/blob/main/text_encoder/config.json) contain `torch_dtype` as a string.

On Diffusers end explicitly setting `torch_dtype` when using `ChatGLMModel` and setting a default `torch_dtype` for `from_pretrained` paths is working https://github.com/huggingface/diffusers/pull/10816 and it's mainly internal effects as `torch_dtype` wasn't passed for some tests, should be ok for end users as they would generally pass `torch_dtype`.

Comments:
Hey @hlky ! Thanks for reporting that. Seems to have been broken after the recent addition of `dtype` for composite configs. I will submit a PR to fix it

same problem ， how to fix? thanks

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Minor Gemma 3 fixes 
Author: molbap
Pull request: yes

# What does this PR do?

In the attention block, Gemma3 enforces dtype/device matching of the attention mask and query states, which fails if there is no attention mask provided. This fixes it.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Simplify keep_in_fp32_modules logic
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. It's easier to use a regex from the beginning compared to start with a list, then switch to regex near the end of the loading logic. 
Also, make sure that only full layer names can be matched with the regex. Otherwise some old and bad layer names such as `wo` in T5 could be match against layers such as `word_embedding`, which contains `wo`.

Also, use the flag only when loading in fp16 or using a quantizer expecting it, as `_keep_in_fp32_modules` was introduced only to avoid issues when casting bf16 -> fp16. See https://github.com/huggingface/transformers/issues/20287 for details. That is, the layers should not always be kept in fp32, despite the name of the flag... I added a detailed comment about that because it is not clear at all otherwise


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36722). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I just modified blip2 so that no model in the library can have the flag set to a non-existing layer (😮‍💨), so I don't think we actually need a check at init time - IMO it would clutter the code for something that would never happen. It can add it to be super super safe though!