noamaanMulla-03/transformers-issues huggingface dataset:
fix: prevent model access error during Optuna hyperparameter tuning
Author: emapco
Pull request: yes

# What does this PR do?

The `transformers.integrations.integration_utils.run_hp_search_optuna` function releases model memory and sets trainer.model to None after each trial. This causes an AttributeError when  subsequent Trainer.train calls attempt to access the model before reinitialization. This is only an issue when `fp16_full_eval` or `bf16_full_eval` flags are enabled.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@SunMarc

Comments:
@SunMarc My bad for closing the previous issue. I included a test that targets the specific conditions that trigger the issue.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36395). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Chat template: return vectorized output in processors
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Part of https://github.com/huggingface/transformers/issues/33948. This PR adds support for `return_tensors="pt"` when calling chat templates for processors. That way users can obtain inputs in tensor format and pass it directly to the model, instead of having to call processor with a formatted prompt + visuals.

For images we use the existing functionality `load_images` and for videos I added a few functions. We usually use `av` in all video related model docs since `decord` had problems with CUDA in the past. Apart from that we can use `opencv` or `torchvision` for video loading. I did a small benchmark run to load and sample uniformly 32 frames from around ~100 videos and `av` was the slowest of all them, while `decord` was the fastest. Therefore I decided to add helper with all possible backends and let users switch whenever they want to. By default we use `opencv` as it is a more common CV framework than any others provided here.

In the future we might start using `torchvision` when we add `VideoProcessor` class and support `VideoProcessorFast` (see https://github.com/huggingface/transformers/issues/33504).


These are the results of small benchmarking with ~100 videos:
```
# Time taken for decord: 475.2979 sec
# Time taken for opencv: 614.6062 sec
# Time taken for av: 1067.0860 sec
# Time taken for torchvision: 1924.0433 sec
```

Review from @Rocketknight1 for templates and @qubvel for general CV related modifications. 



Comments:
> the idea is that if you pass a chat to apply_chat_template and some of the content fields contain images or videos, and tokenize=True, then images and videos are loaded and processed, so that the output is ready to pass to the model?

Yep, exactly! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34275). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

welcome back @qubvel ! Okey, I'll add more typehints and better docs. The backend should be selectable by the user, but we default to the one that works in all cases and has no weird cuda related failures. Prob we should document this somewhere, but I didn't yet find a good place for it

Huh, I don't know why it requested review from some many people, feel free to unsubscribe, sorry

Hmm, good point about the security. We actually already have a few processors that open links for you, e.g. Idefics and Pixtral. Haven't seen anyone flag it as a security breach so maybe it's not a big deal?
Fix compressed tensors config
Author: MekkCyber
Pull request: yes

# What does this PR do?
Verifies if `compressed-tensors` available before import in the config
## Review
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36421). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Flash Attention support to Marian model
Author: mrunsung7
Pull request: yes

# What does this PR do?  
This PR adds support for Flash Attention to the Marian model, enabling faster and more memory-efficient attention computation. Flash Attention significantly boosts performance, especially for large sequence lengths, by reducing memory usage and improving training and inference speed.  

<!-- Remove if not applicable -->  
Fixes #36169  
Partially addresses performance optimization discussions for the Marian model.  

## Motivation and Context  
Marian models benefit from efficient attention mechanisms due to their common usage in translation tasks with long sequences. Flash Attention makes them more scalable and faster without sacrificing accuracy.  

## Changes Introduced  
- Integrated Flash Attention into the `MarianAttention` class in `modeling_marian.py`.  
- Refactored the attention mechanism to use optimized operations while preserving backward compatibility.  
- Removed redundant operations and improved memory efficiency.  

## Before submitting  
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).  
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?  
- [x] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. [Discussion here](https://github.com/huggingface/transformers/issues/36169#issuecomment-2656983754)  
- [ ] Did you make sure to update the documentation with your changes? ([Documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs))  
- [ ] Did you write any new necessary tests?  

## Who can review?  
@ArthurZucker  
@Rocketknight1  

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.  


Comments:
@ArthurZucker
@Rocketknight1
Add LayoutLMProcessor
Author: gau-nernst
Pull request: no
Labels: name: Feature request

### Feature request

Add processor for LayoutLM. I'm not sure why v2 and v3 have their respective processors, but the original v1 doesn't. It should be almost identical their v2 and v3 counterparts (apply tesseract OCR + call the tokenizer appropriately), without returning the resized image (`pixel_values`), since LayoutLMv1 is text-only.

This would also simplify `document-question-answering` pipeline, since right now the pipeline repeats the above logic for LayoutLM.

### Motivation

Make LayoutLM feature-parity with its v2 and v3.

### Your contribution

I can submit a PR to add LayoutLMProcessor. It should be almost identical to v2 and v3, so the task should be straight-forward.

Updating `document-question-answering` pipeline to use the new processor would be too complex since I'm not familiar with the codebase.

Comments:
cc @amyeroberts and @NielsRogge if LayoutLM is just not as good we should use newest models

There are several advantages in using LayoutLMv1:
- It's text-only, so it can be much more lightweight. Not depening on detectron2 is also a plus (there are no pre-built detectron2 for latest versions of PyTorch/CUDA)
- From what I know, v2 and v3 don't permit commercial use, while v1 does.
- impira/layoutlm-document-qa is very good. I haven't found a good fine-tuned v2 and v3 on DocVQA.

Alright then! Feel free to open a PR if you have time 


Thanks @gau-nernst for opening this issue, indeed we only started defining processors for v2 and v3 but we could define one for v1 as well. Your PR already looks in a great state, let me know if you need any help.

Would love if someone can make a processor for layoutlmv1.
Fix Expected output for compressed-tensors tests
Author: MekkCyber
Pull request: yes

# What does this PR do?
As the title suggests.

## Review
@SunMarc 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36425). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] Pixtral Text Config
Author: kylesayrs
Pull request: yes

## Purpose ##
* This is a potential solution to the issue of conflicting Pixtral and Mistral configs, as detailed [here](https://github.com/huggingface/transformers/pull/36077)
* The downside to this approach is that (afaict) transformers makes an assumption that all configs have a one to one mapping with models. In this case, the PixtralTextConfig would map to the MistralModel, which is already associated with the MistralConfig. Additional logic and testing would be require to handle this case.

Comments:
cc @ArthurZucker - I think this is something that's usually handled with `modular` instead of directly importing the config, no?

@ArthurZucker This was fixed by https://github.com/huggingface/transformers/pull/36077, however that solution is technically a hack as it saves more attributes than are necessary and expands the use of the `is_composition` beyond its original intended use.

I believe that this is a good solution for now, this PR can be reopened/forked if a better solution is desired.
Load models much faster on accelerator devices!!
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Cut model loading time by a large margin (7x factor for an 8B model, 6x for a 32B 🚀🚀)
CudaMalloc was in fact the bottleneck in our current loading pipeline. We can pre-allocate tensor of the expected size to warm-up the caching allocator, which makes much fewer calls to CudaMalloc and cuts loading time accordingly.

The following snippet:

```py
import time
t_ini = time.time()
import torch
from transformers import AutoModelForCausalLM
print(f"Time for the imports: {time.time() - t_ini:.2f} s")

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
# model_id = "codellama/CodeLlama-34b-Instruct-hf"
device = torch.device(f"cuda:2")


t0 = time.time()
torch.cuda.synchronize(device)
print(f"Time for cuda warmup before loading model: {time.time() - t0:.2f} s")
t0 = time.time()
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map=device)
torch.cuda.synchronize(device)
dt = time.time() - t0
print(f"time to load the model: {dt:.2f}")

max_mem = torch.cuda.max_memory_allocated(device) / 1024**3
current_mem = torch.cuda.memory_allocated(device) / 1024**3
print(f"Max: {max_mem:.2f} GiB")
print(f"Current: {current_mem:.2f} GiB")
print(f"Full time: {time.time() - t_ini:.2f} s")
```

returns:

```
Time for the imports: 4.38 s
Time for cuda warmup before loading model: 16.18 s
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00,  1.21s/it]
time to load the model: 5.92
Max: 14.96 GiB
Current: 14.96 GiB
Full time: 26.49 s
```

and before:

```
Time for the imports: 4.54 s
Time for cuda warmup before loading model: 15.23 s
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:42<00:00, 10.51s/it]
time to load the model: 42.54
Max: 14.96 GiB
Current: 14.96 GiB
Full time: 62.32 s
```

SO we basically cut loading time by 4x for an 8B model.

Note that in both cases, there is basically 15s overhead for cuda before loading the model.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36380). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Ascend NPU support SDPA
Author: zheliuyu
Pull request: yes

# What does this PR do?
add features: Ascend NPU supports SDPA.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## notes
 Ascend NPU requires torch>=2.1.0 to use SDPA in Transformers.


Comments:
cc @ArthurZucker @younesbelkada 

Sorry for the delay, do you want to get this merged? 

> Sorry for the delay, do you want to get this merged?

Thanks for your reply, this feature has been merged through this [PR](https://github.com/huggingface/transformers/pull/35165).
Support sliding_window for sdpa in qwen2
Author: cyr0930
Pull request: no
Labels: name: Feature request

### Feature request

Can we implement sliding_window support in here (https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/models/qwen2/modeling_qwen2.py#L237) like this (https://github.com/huggingface/transformers/blob/v4.41.2/src/transformers/models/mistral/modeling_mistral.py#L985)?

Or is this responsible for torch? (https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)

### Motivation

Sliding window for sdpa is supported with mistral but qwen2

### Your contribution

maybe I can submit PR?

Comments:
hi @cyr0930, in general, there are three main attention implementations supported by models in `transformers`:

`eager`: Attention algorithm written by hand in the model code using basic linear algebra operations
`sdpa`: uses `scaled_dot_product_attention()` from `torch`
`flash_attention_2`: uses the `flashattn` library

Qwen2 already supports sliding window attention with `flash_attention_2` but I believe it should be possible to add sliding window support to Qwen2 with either `SDPA` or `eager`, based on the implementations in other models

As qwen2 & mistral have the same logic for sliding window, maybe we can just remove warning safely.
(https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/models/mistral/modeling_mistral.py#L725)
(https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/models/qwen2/modeling_qwen2.py#L738)

But I'm not sure, because of this revert. (https://github.com/huggingface/transformers/commit/96f01a36ac60c57be279cb7f027eac7e37fa90e8)

@ArthurZucker might be able to confirm this.
Added Cosmos model files 
Author: tanuj-rai
Pull request: yes

# What does this PR do?
 This PR adds up Cosmos (Nvidia) model files to the transformers repository. 

Fixes #35565


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker and anyone in the community is free to review the PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Apply dualpipe from deepseek-v3 to a trainer or model
Author: jp1924
Pull request: no
Labels: name: Feature request

### Feature request

Applying deepseek's dual-pipe to transformers or accelrate

### Motivation

Recently, deepseek released the proposed [dual-pipe code](https://github.com/deepseek-ai/DualPipe) in the [DeepSeek-V3](https://arxiv.org/html/2412.19437v1) Technical Report.
Looking at the code structure, it is 100% python, and it seems easy to apply.

### Your contribution

We can modify the code of dual-pipe and apply it to transformer.

Comments:
restrict cache allocator to non quantized model
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR restrict cache allocator to non quantized for now. Also we should apply it to `model_to_load` and not `model`. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36428). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Change PR to draft when it is (re)opened
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR is a first step: it simply change a PR to draft when it is (re)opened. It doesn't modify yet the logic regarding how CI should be triggered.

See [this example](https://github.com/ydshieh/transformers/pull/8).

It also sends a comment, see

![Screenshot 2025-02-26 151945](https://github.com/user-attachments/assets/c1f93a32-a00f-4f4f-8c90-11ebbe0585bd)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36417). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes we can check of course. I will update. Also  disable tests  but keep checks on circleCI
Add `contents: write`
Author: ydshieh
Pull request: yes

GitHub doc is really bad ....

I hope this is the last commit that fixes everything

https://github.com/cli/cli/discussions/6924#discussioncomment-4817576

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36445). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: dtype might change during resize
Author: LarsHaalck
Pull request: yes

# What does this PR do?

I noticed a small bug, where an np.ndarray changes dtype when calling resize:

```python
import numpy as np
from transformers.image_transforms import resize

x = np.zeros((3, 100, 100), dtype=np.float32)
print(x.dtype) # prints np.float32
x = resize(x, size=(50, 50))
print(x.dtype) # prints np.uint8`
```

When using natural numbers in a float array, the image is not rescaled and converted into a `PIL.Image` as-is but when returning back to a numpy array, the type stays `uint8`.

This PR, stores the original dtype and casts the array back to this type before returning. I also added tests to check for this behavior in the future. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

I think @amyeroberts might be best suited since she wrote lots of surrounding code. 


Comments:
안녕하세요, 모두 감사합니다.

4D 환경을 얘기할때 그날 저는 머리에 두통을 너무나 느꼈고, 귀신얼굴이 보였습니다. 너무 무서웠어요.

I'm not sure if the changed dtype is the intended behaviour - cc @qubvel @zucchini-nlp

> I'm not sure if the changed dtype is the intended behaviour

Any update on this? I think a resize function should not change the datatype of the passed image but if this is indented behavior than the caller just has to make sure to convert back to the desired type.
Lazy import libraries in `src/transformers/image_utils.py`
Author: hmellor
Pull request: yes

Lazy import the imports added by #34275.

Enables https://github.com/vllm-project/vllm/pull/13905 (which strictly enforces that `cv2` is lazily imported).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36435). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `hub_retry`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `hub_retry`

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36449). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.