noamaanMulla-03/transformers-issues huggingface dataset:
Update executorch.md
Author: jackzhxng
Pull request: yes

# What does this PR do?

Update ExecuTorch docs, keep things simple and just redirect to Optimum repo.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @stevhliu 


Comments:
Hmm, I think it may be better to keep at least some of the examples in the ExecuTorch docs or make it a bit more minimal (can just copy what is already in the optimum-executorch quickstart)? This way users don't have to jump from one page to another and can stay in the Transformers docs.

@stevhliu added an example, let's merge?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41582). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Backbone API fine-tuning tutorial
Author: merveenoyan
Pull request: yes

We have recently merged this PR to revive the backbone API using DINOv3 as opportunity to do so https://github.com/huggingface/transformers/pull/40651
So I have written documentation. I need to add resulting image before merger. For this I want to train the model a bit more since bboxes are slightly shifted and it annoyed me 😄 the model can be better. 
In the meantime @stevhliu can you please review wording etc? 💗

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41590). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add fuyu Fast Image Processor
Author: Aravind-11
Pull request: yes

# What does this PR do?

This PR introduces `FuyuImageProcessorFast`, providing a faster alternative to the original `FuyuImageProcessor` by leveraging `torchvision` for image transformations.

Key changes include:
- Implementation of `FuyuImageProcessorFast` inheriting from `BaseImageProcessorFast`.
- Adaptation of Fuyu-specific resize and padding logic using `torchvision.transforms.functional`.
- Updates to `tests/models/fuyu/test_image_processing_fuyu.py` to include the fast processor, override save/load tests, and adjust tolerances for equivalence checks. All Fuyu image processing tests now pass.
- Addition of documentation for `FuyuImageProcessorFast` and update of the main usage example to recommend `AutoProcessor` for loading.

Fixes #36978

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/36978
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@yonigozlan

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @yonigozlan , could you please check? Thank you. 

> Hi @Aravind-11 , thanks a lot for the huge work! This is quite a difficult image processor to adapt, as it is a bit different to others. However you should be able to simplify it and use method from the base fast image processor a lot more. This is important for readability and maintainability, which is a key aspect of the refactoring of fast image processors

Got it. I will make the changes. Thank you!

Closing this PR to give opportunities to others to contribute, as it appears to be stale.

Hi, can I work on this ? What is the status on the fuyu processor ?
torch can not be detected by transformers on aarch
Author: HuangChiEn
Pull request: no
Labels: name: bug

### System Info

transformers              4.55.0
python 3.10.12
plateform aarch

### Who can help?

@ivarflakstad

i know it's different between AMD and arm, unfortunately i can not image who can help me in this issue.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

ngc torch container : `nvcr.io/nvidia/pytorch:24.01-py3`
`pip install transformers==4.55.0`
`python -c "from transformers import AutoModel"`

error : 
<img width="1282" height="777" alt="Image" src="https://github.com/user-attachments/assets/ff70e166-afe3-49a9-b8b1-bf41123aa496" />

i had search for this error, someone said it's because you didn't install torch. However, i had checked the NGC container does contain torch v2.2.0 as follows.

<img width="457" height="157" alt="Image" src="https://github.com/user-attachments/assets/32f1b3e5-ae57-4af3-9c64-d4134148db88" />


### Expected behavior

successfully import AutoModel.

Comments:
jetson orin?
download torch from: https://pypi.jetson-ai-lab.io/jp6/cu126

> jetson orin? download torch from: https://pypi.jetson-ai-lab.io/jp6/cu126

wow, thanks for your quick reply, i'll try to use it while i re-encounter this issue again.
Actually, i'm prepare the running environment for **server**, the compute node apply aarch. 
I try to downgrade the `transformers==4.46.0` and went well, unfortunately i need to test if the codebase accept this version (debugging...).

So i leave this issue open here, since the specific transformers version still raised this issue.

i think i just closed the issue, since you can downgrade transformers version slightly (not change version too much)

`transformers==4.46.0` is fine, a bit larger version may also be fine
Add aux loss for GLM-4.5V
Author: zRzRzRzRzRzRzR
Pull request: yes

# What does this PR do?

Add Aux Loss for GLM-4.5V

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /style

Style bot fixed some files and pushed the changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v_moe
Allow passing `tp_plan` in `from_pretrained` directly
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Also fixes the tp_plan internal for non-based models.

I checked with the following, and it works nicely:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from torch.distributed.tensor.debug import visualize_sharding
from torch.distributed.tensor import DTensor
import os

rank = int(os.environ["RANK"])

model_id = "meta-llama/Llama-3.1-8B-Instruct"
# only shard attentions, but not mlps
tp_plan = {
    "model.layers.*.self_attn.q_proj": "colwise",
    "model.layers.*.self_attn.k_proj": "colwise",
    "model.layers.*.self_attn.v_proj": "colwise",
    "model.layers.*.self_attn.o_proj": "rowwise",
}

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    tp_plan=tp_plan,
)

inputs = tokenizer(["How are you?"], return_tensors="pt").to(rank)
out = model.generate(**inputs, max_new_tokens=10, do_sample=False)
text = tokenizer.batch_decode(out[:,inputs.input_ids.shape[-1]:])

if rank == 0:
    print(text)
    print(model.tp_plan)
    print("SHARDING OF Q_PROJ")
    visualize_sharding(model.model.layers[0].self_attn.q_proj.weight)
    print("SHARDING OF UP_PROJ")
    if isinstance(model.model.layers[0].mlp.up_proj.weight, DTensor):
        visualize_sharding(model.model.layers[0].mlp.up_proj.weight)
    else:
        print("NOT SHARDED")

torch.distributed.destroy_process_group()
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41435). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Fast Image Processors] Add `call_only_args` attribute + test
Author: yonigozlan
Pull request: yes

# What does this PR do?

As the title says, this seems to be more and more relevant for downstream libraries, namely vLLM Cc @hmellor

It's another attribute to think about when contributing a new fast image processor, but as this one is tested for, no choice but to add it correctly 😉. Maybe we should add this kind of tests for other models/processor attributes that are not always respected?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40172). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: beit, conditional_detr, deformable_detr, detr, dpt, eomt, grounding_dino, mask2former, maskformer, mobilenet_v2, mobilevit, oneformer, qwen2_vl, rt_detr, sam, sam2

What is it used for? I'm missing some context here haha

> What is it used for? I'm missing some context here haha

The original context for this was:
- vLLM using factory classes to get and instantiate image processors
- different image processors supporting differnt kwargs
- some kwargs only being supported at `__init__` and others only supported at `__call__`

Related vLLM issues:
- https://github.com/vllm-project/vllm/issues/21995
- https://github.com/vllm-project/vllm/pull/22005

For me, this is clearly a sign that something is wrong in the kwarg handling of the Processors, and something we should refactor on our end instead of adding more and more attributes and complexity all the time. This is also something I experienced, kwargs flow is extremely hard to understand and follow
Remove randomly added script
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

cc @zucchini-nlp you added it in https://github.com/huggingface/transformers/pull/41474. Please be very careful about adding commits after we approve a PR, and always review the diffs of the PR before merging to avoid such issues!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41650). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry about that
[`Ernie 4.5 Moe`] Fix Moe and offloading
Author: vasqu
Pull request: yes

The model is broken
- Moe was initialized with wrong dims (loading the models already failed)
- Gate needs the autocast decorator as well (we need to guarantee fp32)
- Not sure why but using the forward with the moe statics works with offloading while retrieving the parameter etc does not
- The router seems like an artefact

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: ernie4_5_moe

This comment contains run-slow, running the specified jobs: 

models: ['models/ernie4_5_moe']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_moe
[CI] Build translated docs
Author: stevhliu
Pull request: yes

Actually, this might be the reason why the translated docs aren't building when commenting `build-doc`. The `tr` and `te` language codes don't exist 😅 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41632). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/18554884817)
Fix fp32_ln for various models
Author: remi-or
Pull request: yes

This PR fixes the test `test_flash_attn_2_fp32_ln` for several models:
- `bark` was failing the test because it call `_flash_attention_forward` directly without checking the `queries` dtype, and so the test could fail if the dtype was `torch.float32`. To fix this we re-factored out a code block into a function  `get_target_dtype`  that takes care of infering whether to cast the fp32 tesnor to fp16 or bf16, and added a called to it before the call to FA
- same for `stablelm`
- `mllama` was failing the test because `MllamaTextSelfAttention` lacks the `is_causal`attribute, which was added and set to True (it's a text attention so it's causal, as discussed in #39182)
- same for `kosmos2` but the test still fails for many many other reasons

The list of fixed test is here:
```
FAILED tests/models/bark/test_modeling_bark.py::BarkSemanticModelTest::test_flash_attn_2_fp32_ln - RuntimeError: FlashAttention only support fp16 and bf16 data type
FAILED tests/models/bark/test_modeling_bark.py::BarkCoarseModelTest::test_flash_attn_2_fp32_ln - RuntimeError: FlashAttention only support fp16 and bf16 data type
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForCausalLMModelTest::test_flash_attn_2_fp32_ln - AttributeError: 'MllamaTextSelfAttention' object has no attribute 'is_causal'
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForConditionalGenerationModelTest::test_flash_attn_2_fp32_ln - AttributeError: 'MllamaTextSelfAttention' object has no attribute 'is_causal'
FAILED tests/models/stablelm/test_modeling_stablelm.py::StableLmModelTest::test_flash_attn_2_fp32_ln - RuntimeError: FlashAttention only support fp16 and bf16 data type
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark, blt, kosmos2, mllama, stablelm
Adjust device logging level and add minor fixes
Author: mario-koddenbrock
Pull request: yes

This commit addresses a noisy warning and improves the robustness of the base pipeline implementation.

- The device placement message in the pipeline base class has been changed from a `warning` to a `debug` log. This reduces log noise for users who are aware of their device setup, while still providing the information for debugging purposes.

- Additionally, potential `UnboundLocalError` exceptions in the `_pad` and `check_model_type` functions have been prevented by initializing variables before their conditional assignment.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41636). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix EncoderDecoder cache
Author: remi-or
Pull request: yes

In #41569 we restored thr `__iter__` method to `DynamicCache` but I missed the fact that it was also removed from `EncoderDecoderCache`. This PR fixes that and modifies the `__init__` of `EncoderDecoderCache` in the case of DDP, so it is compatible with the new system.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41612). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@remi-or I tested this branch on PEFT (`pytest tests/test_decoder_models.py tests/test_encoder_decoder_models.py -k "prefix or prompt"`) and the tests are passing, thanks! 

Nice! I think the failing tests have nothing to do with the PR, so this is ready to be merged imo :)

run-slow: vit

This comment contains run-slow, running the specified jobs: 

models: ['models/vit']
quantizations: [] ...

run-slow: vit

This comment contains run-slow, running the specified jobs: 

models: ['models/vit']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: rag, whisper
Fix typos in documentation
Author: cyyever
Pull request: yes

# What does this PR do?
As the title says.
 

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41641). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Format MarkDown documentation and tiny fixes
Author: cyyever
Pull request: yes

# What does this PR do?

This PR fixes MarkDown document issues detected by markdownlint-cli2. Some duplicated `autodoc` entries are also removed. 

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41638). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add check for kernel config
Author: MekkCyber
Pull request: yes

# What does this PR do?

Erroring when `kernel_config` is provided without specifically setting `use_kernels` to `True`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41467). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.