noamaanMulla-03/transformers-issues huggingface dataset:
Fix the eval_use_gather_object flag usage
Author: ducha-aiki
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36213

After the first eval, the 2nd eval is crashing, while trying to concat batches with different shapes, as if the flag` eval_use_gather_object` stopped working. In evaluation_loop, the `self.gather_function` is reset, and the `eval_use_gather_object` is not used

This PR fixes that, using the same code line, which is used at accelerator creation.

@muellerzr and @SunMarc


Comments:
The test failure seems to be unrelated:

```
=================================== FAILURES ===================================
______________ LlavaForConditionalGenerationModelTest.test_config ______________
[gw3] linux -- Python 3.9.21 /usr/local/bin/python3

self = <tests.models.llava.test_modeling_llava.LlavaForConditionalGenerationModelTest testMethod=test_config>

    def test_config(self):
>       self.config_tester.run_common_tests()

tests/models/llava/test_modeling_llava.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_configuration_common.py:205: in run_common_tests
    self.check_config_can_be_init_without_params()
tests/test_configuration_common.py:169: in check_config_can_be_init_without_params
    config = self.config_class()
E   AssertionError: ValueError not raised

```

LMK if you have time to add the msall test @ducha-aiki, otherwise, I will just merge it as is. Thanks again !
Fix casting dtype for qunatization
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes an issue with quantized models as we changed recently how to cast the dtype of each parameters. We rely now on the model architecture and we check the dtype associated to each param. However, this won't work for model that we are going to quantize (quantization on the fly) as the architecture changed.
This impacts eetq quantization and maybe some fp8 methods also


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Is it possible to add a test?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36799). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Is it possible to add a test?

[eetq ci](https://github.com/huggingface/transformers/actions/runs/13914217685/job/38934582403) is indeed failing but not sure why we didn't manage to get the notification on slack cc @MekkCyber
Support custom dosctrings in modular
Author: yonigozlan
Pull request: yes

# What does this PR do?

As discussed in #36688 and #35902. Override `_DOCSTRING` constants defined at the top level in modular if they are not None, otherwise copy the inherited ones.

Cc @Cyrilvallez 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36726). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Done @Cyrilvallez :)
@ArthurZucker this should be ready for final review!
Expectations test utils
Author: ivarflakstad
Pull request: yes

This PR adds the concept of `Expectations`, which can be used to simplify handling different expected results depending on the environment the test is running in.
```python
result = fn_being_tested()

expectations = Expectations({
    (None, None): "1",  // Default
    ("cuda, 8): "2"),
    ("cuda, 7): "3"),
    ("rocm, None): "4"),
)
expected = expectations.get_expectation()  # Automatically gets the correct expected result from detected environment.
assert result == expected
```
Addresses #36561

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36569). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

My first look has the impression that this is a bit too much. I have to think it a bit more and see what we can reduce but still get the improvements we want to have. 

> Agree a bit with @ydshieh (who is the owner 👀 ) a bit too bloated. Happy to have a simple and explicit function to help with these issue + an updated test to show proper usage

Let me see if I can simplify it a bit. I'll fix a test with it as proper usage as well 👍 

@ArthurZucker @ydshieh 
I've simplified quite a bit. Added usage of it as well [here](https://github.com/huggingface/transformers/pull/36569/files#diff-c31ced352e596366ee5bda58e354540a22f72f21306c8a265d5c6b45bae0b19eL505) ☺️ 

Hi @ivarflakstad I am wondering what's your thoughts about:

- Remove `Expectation`
- Make `Expectations` a subclass of `dict`
  - no need to have `_inner` and `_default`
  - `key` being `(type, major)`, `value` being `result`  
  - move `_get_device_properties` out to a place (as functioin) that could be reused
  - A new method named `get_closed` (`get_best` or whatever) with one argument named `key`
    -  merge all the logic that needs to get the closet expectation inside its body and return it
  - to use it, just `expection = my_expectation(_get_device_properties())` 

Overall, my suggestion is just to make it less abstract: just a simple class inherited a familar class (`dict`) with a new method `get_best` .

WDYK?
Traning loss not showing with trainer
Author: mwnthainarzary
Pull request: no
Labels: name: bug

### System Info

Python 3.11.11
transformers 4.48.2

### Who can help?

@muellerzr 
@SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

# Define the TrainingArguments for fine-tuning
training_args = TrainingArguments(
    output_dir='/content/drive/MyDrive/Legal_Dataset/BartlargeFineTuned/',
    num_train_epochs=10,
    per_device_train_batch_size=10,
    gradient_accumulation_steps=8,
    evaluation_strategy="epoch",
    save_total_limit=1,
    save_steps=1000,
    learning_rate=1e-3,
    do_train=True,
    do_eval=True,
    remove_unused_columns=False,
    push_to_hub=False,
    report_to='tensorboard',
    load_best_model_at_end=False,
    lr_scheduler_type="cosine_with_restarts",
    warmup_steps=100,
    weight_decay=0.01,
    logging_dir='/content/drive/MyDrive/Legal_Dataset/BartlargeFineTuned/',
    logging_steps=200,

)


# Create a data collator for sequence-to-sequence tasks
data_collator = MyDataCollatorForSeq2Seq(
    tokenizer=tokenizer,
    model=model,
    padding=False,
    max_length=80,
    label_pad_token_id=tokenizer.pad_token_id,
)


# Create Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=validation_dataset,
    optimizers=(custom_optimizer, None),
)

trainer.train()



### Expected behavior

I trained the model for 10 epochs but in every epochs I saw the validation loss only not training loss. Please help

![Image](https://github.com/user-attachments/assets/e535a5f3-4d6f-4c96-b92a-2e1e2723fb46)

Comments:
In your `TrainingArguments` perhaps try

- examine output in your `logging_dir` to confirm 
- switching `evaluation_strategy` to `eval_strategy` (it looks like `evaluation_strategy` [is depreciated](https://github.com/huggingface/transformers/blob/080e14b24c8923e4bc18fcd54010fc7396c67bc0/src/transformers/training_args.py#L1366))
- lowering your `logging_steps` - at its current value of `200` may too large given the size of your dataset (you might try evaluating every X steps instead of epoch to triangulate the right value)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Subtle difference with Pytorch AdamW?
Author: kyleliang919
Pull request: no

https://github.com/huggingface/transformers/blob/e5fd865ebae062b7cf03a81b8c6affeb39f30bec/src/transformers/optimization.py#L648
It does correction after epsilon is added, whereas pytorch：
https://pytorch.org/docs/stable/_modules/torch/optim/adamw.html#AdamW
```python
step = _get_value(step_t)

bias_correction1 = 1 - beta1**step
bias_correction2 = 1 - beta2**step

step_size = lr / bias_correction1

bias_correction2_sqrt = bias_correction2**0.5

if amsgrad:
    # Maintains the maximum of all 2nd moment running avg. till now
    torch.maximum(max_exp_avg_sqs[i], exp_avg_sq, out=max_exp_avg_sqs[i])

    # Use the max. for normalizing running avg. of gradient
    denom = (max_exp_avg_sqs[i].sqrt() / bias_correction2_sqrt).add_(eps)
else:
    denom = (exp_avg_sq.sqrt() / bias_correction2_sqrt).add_(eps)

param.addcdiv_(exp_avg, denom, value=-step_size)
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @arthurzucker - is there a reason for us to maintain our own `AdamW`? In future, should we just wrap Torch's optimizer for this class?

Yeah no reason anymore AFAIK! 

Opened a PR at #36177 to do that!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove `dist": "loadfile"` for `pytest` for CircleCI jobs
Author: ydshieh
Pull request: yes

# What does this PR do?

This pytest option forces all tests  from a test file to be run in the same CPU process when using `-n X` (from `pytest-xdist` pluging) to distribute tests across multiple CPU processes.

However, this leads some worker(s) to run much longer if a test file have tests that are slower. Removing `dist": "loadfile"` allows those tests are also being distributed to multiple processes.

Results: 

tests_tokenization: `main` | 1 run on PR

- the longest runtime among 8 paralle CircleCI runners: 8m30s | 5m
- total runtime from all 8 paralle CircleCI runners: 43m | 27m
- total idle time from 8 parallel CircleCI runners: 25m40s | 12m

tests_torch: `main` | 2 runs on PR

- the longest runtime among 8 paralle CircleCI runners: 6m40s | (6m | 5m10s)
- total runtime from all 8 paralle CircleCI runners: 31m15s | (30m | 28m)
- total idle time from 8 parallel CircleCI runners: 22m 10s | (18m | 13m15s)

tests_non_model: `main` | 1 run on PR

- the longest runtime among 8 paralle CircleCI runners: 6m | 2m50
- total runtime from all 8 paralle CircleCI runners: 21m20 | 14m50
- total idle time from 8 parallel CircleCI runners: 26m 50s | 7m 55s

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36811). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Device map for bitsandbytes tests
Author: MekkCyber
Pull request: yes

# What does this PR do?
Fixing the device map for bitsandbytes tests for smaller unbalanced models


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36800). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix dtype when loading to meta model
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Before https://github.com/huggingface/transformers/pull/36335, when loading the weights to meta model we kept the `dtype` of the module params. This PR fixes it by setting `assign=True`. I am not sure though, why `strict=False`, should I change it to `True`?

Before (no `dtype` is passed to accelerate):
`set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)`

After:
`module.load_state_dict({param_type: param[:].to(param_device)}, False, True)`

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36447). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fixed by recent refactor already
Clean-up: `get_patch_output_size` can be removed
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, removing redundant helpers :)

The tests for the models are passing and I verified that both helpers are identical, except for one ceils up and the other rounds down. Not much of a difference imo, and I opted for `ceil` because that's what we do in modeling code when postprocessing llava image features

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36274). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Lets close it, at first I thought it is not a big change since but we don't want to break things :)
Add attention visualization tool 
Author: ArthurZucker
Pull request: yes

# What does this PR do?

TODOS
- [ ] Add some tests
- [ ] make it work on all models

cc @yonigozlan and @zucchini-nlp @molbap 

This will be available as: 
```python

from transformers.utils.attention_visualizer import AttentionMaskVisualizer
visualizer = AttentionMaskVisualizer("meta-llama/Llama-3.2-3B-Instruct")
visualizer("A normal attention mask")

visualizer = AttentionMaskVisualizer("mistralai/Mistral-Small-24B-Instruct-2501")
visualizer("A normal attention mask with a long text to see how it is displayed, and if it is displayed correctly")

visualizer = AttentionMaskVisualizer("google/paligemma2-3b-mix-224")
visualizer("<img> You are an assistant.", suffix = "What is on the image?")

visualizer = AttentionMaskVisualizer("google/gemma-2b")
visualizer("You are an assistant. Make sure you print me") # we should have slidiing on non sliding side by side

visualizer = AttentionMaskVisualizer("google/gemma-3-27b-it")
visualizer("<img>You are an assistant. Make sure you print me") # we should have slidiing on non sliding side by side
```
![image](https://github.com/user-attachments/assets/d10f63f2-ad5e-4550-a30f-7181be554a67)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36630). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi, I was trying to visualize the attention mask for Gemma 3 but I get 

`AttributeError: 'Gemma3ForConditionalGeneration' object has no attribute 'visualize_attention_mask'`


For reference, sharing relevant excerpts of the code:

```
from transformers import AutoProcessor, Gemma3ForConditionalGeneration, AutoTokenizer
from PIL import Image
import json
import torch

model_id = "google/gemma-3-27b-it"

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
).eval()

processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
.
.
.
inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt"
    ).to(model.device, dtype=torch.bfloat16)

model.visualize_attention_mask(tokenizer, inputs)
```

Hey ! Will work a bit to make it more easy to use! THanks 🤗 

Doing a first review, so happy about this
GPT2 always has largest attention on first token?
Author: jzhoubu
Pull request: no

I am trying to print attentions from GPT2 model, and find something strange.

```
import torch
from transformers import *
device = torch.device("cuda")

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
config = GPT2Config.from_pretrained("gpt2", output_attentions=True)
model = GPT2LMHeadModel.from_pretrained("gpt2", config=config, cache_dir="./cached").to(device)

input_text = "hello, this is Tom ."
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)
logits, _, attns = model(input_ids)
last_layer_attns = attns[-1].squeeze(0) 
last_layer_attns_per_head = last_layer_attns.mean(dim=0)  # (sequence_length, sequence_length)
print(last_layer_attns_per_head[-1])
```
Output:
> tensor([0.6496, 0.0758, 0.0439, 0.0328, 0.0853, 0.1125], device='cuda:0', grad_fn=<SelectBackward>)

I have also tried different sentences but the distributions of attn are the same -- the first token always has the largest attention. Is there anything wrong in my code? Or can somebody explain why this the first token has the largest attn?



Comments:
Hi @sysu-zjw, this is indeed a very interesting observation that you made! 
One thing to notice first is that GPT2 uses casual masking. So when looking at the attention weights ( corresponding to your variable `last_layer_attns_per_head` which I think should actually be called `last_layer_attns_avg_over_heads` ;-) ):

```
[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
[0.8310, 0.1690, 0.0000, 0.0000, 0.0000, 0.0000],
[0.7752, 0.1165, 0.1083, 0.0000, 0.0000, 0.0000],
[0.6962, 0.1208, 0.1039, 0.0790, 0.0000, 0.0000],
[0.8273, 0.0428, 0.0410, 0.0356, 0.0533, 0.0000],
[0.6496, 0.0758, 0.0439, 0.0328, 0.0853, 0.1125]
```
It is obvious that the attention weights of the first token for example can only attend to the first token and so on.
But as you noticed the more interesting part is that also the last token seems to always focus most of its attention on the first token (for the last layer).

I played around with different inputs and also noticed that pretty much the latter half of the transformer layers focus by far most of "its attention" on the first token. After googling a bit, I found that your observations were already put in a paper (check it out [here](https://www.aclweb.org/anthology/W19-4808.pdf) - especially Section 4.2). Looking at Figure 2, you should recognize your observations ;-) 

From my point of view, there is nothing wrong with your code. I think it's a pattern that has been observed but its reason is not well understood.
If you find a good explanation for GPT2's behavior in this case (might also be very similar for other transformer architectures), let me know!



@patrickvonplaten Thanks much for your detailed comment :) I will post here if I find any good explanation.

seems that similar observation is used in streamingllm for speeding up inference

did you find the reason please?
Model Card to include key information (e.g. max_sequence_length, etc.)
Author: ai-nikolai
Pull request: no
Labels: name: Feature request

### Feature request

It would be great to add a (non-optional) part in the model card page & when authors upload their models. Where all the main parameters of the model are highlighted (i.e. so that it can be used with model.generate() easily. & recommended parameters are indicated as well [optional]).

### Motivation

The reason for this feature, is that often models fail silently and at a later stage - when using them, because some key parameter is not set during model loading or model generation.

### Your contribution

Not sure, I would love to, but I think this is a request that only HF can fulfill. 

Comments:
I'm afraid we can't really enforce model card content! The model repos are under the authors' control, and we don't want to block uploads of non-compliant model cards.

In general, recommended generation parameters can be set as defaults in the `config` or `generation_config`. These will automatically be read by `generate()` if the user doesn't pass a value. 

Do you have examples of model cards that are missing critical information? If this is a common issue, we might consider updating our docs or suggestions, but we probably can't make it mandatory!
Support for Ovis2 models
Author: abhiaagarwal
Pull request: no
Labels: name: New model

### Model description

Hi there,

Alibaba released the Ovis2 series a few months ago. They are a mix of Qwen2VL + Apple's AIMv2 VIT.

While their transformers-compatible code works, it would great for it to be upstreamed as to avoid `trust_remote_code` and to be adapted to transformer's processors for use in VLLM.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Transformers-compatible implementation here: https://huggingface.co/AIDC-AI/Ovis2-2B/tree/main

Comments:
When what needs to be loaded is in the cache directory, there is no need to make a request to the remote
Author: JinFish
Pull request: no
Labels: name: Feature request

### Feature request

When what needs to be loaded is in the cache directory, there is no need to make a request to the remote.



### Motivation

I noticed that when `AutoTokenizer` loads a file using `from_pretrained`, it first tries to load it from a cached directory when `pretrained_model_name_or_path` is a model_id (such as gpt2).

However, `commit_hash` is `None` by default, e.g. `AutoTokenizer` will call `get_tokenizer_config` to load the configuration file, where the code to get `commit_hash` is: `commit_hash = kwargs.get("_commit_ hash”, None)`. 

Since it is None, the `cached_file` method doesn't know where the corresponding file is actually stored, so it uses the `hf_hub_download` method to request the corresponding `commit_hash` first. 
Although this request is very simple and infrequent, **in offline environments (e.g., a company or school intranet that does not allow access to the extranet), it will report an error.**

I know I can copy files from the cache to my project directory, but the host is usually used by multiple people, which means it may have to be copied many times, which defeats the purpose of using a cached directory in the first place.

### Your contribution

**I suggest changing `commit_hash = kwargs.get(“_commit_hash”, None)` to `commit_hash = kwargs.get(“_commit_hash”, “main”)`**.

Comments:
I knew that providing `_commit_hash=“main”` would work, but I didn't know that until I read the source code. For people who haven't read the source code (like me before), it's a huge headache.

cc @cyrilvallez who's doing a rework of the `from_pretrained` code, WDYT about defaulting to `main` here? Would this cause an issue if older repos are using `master`?

Hey! No, we actually don't want to do that, as using `main` as default would prevent people from getting the last updated files from the hub repo automatically, if the files changed! It would grab the existing cached files, without pinging the remote for potential new changes 😉
We provide the optional keyword argument `local_files_only=True` in all `from_pretrained` methods for use-cases such as yours, when you don't want to make any request to the outside world!