noamaanMulla-03/transformers-issues huggingface dataset:
🌐 [i18n-KO] Translated `perf_infer_gpu_multi.md` to Korean
Author: nevertmr
Pull request: yes

# What does this PR do?

Translated the `perf_infer_gpu_multi.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

May you please review this PR?
@4N3MONE, @yijun-lee, @jungnerd , @harheem

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

@stevhliu May you please review this PR?

Comments:
cc @stevhliu 

@4N3MONE Thanks for the detailed feedback — I’ve updated the phrasing and terminology accordingly.

@harheem Appreciate your suggestions — incorporated them into the recent updates.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39441). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ModuleNotFoundError: No module named 'habana_frameworks.torch'
Author: orkungedik
Pull request: no
Labels: name: bug

### System Info

Hi,

I finetuned unsloth/Meta-Llama-3.1-8B-Instruct model and tried to run code below on CPU;

```
model_name = "orkungedik/ege-8b-1.0"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name,torch_dtype="auto", device_map="auto")
```

And getting error below;

> File /usr/local/lib/python3.10/dist-packages/neural_compressor/torch/utils/environ.py:67
     65 if is_package_available("habana_frameworks"):
     66     _hpex_available = True
---> 67     import habana_frameworks.torch.hpex  # pylint: disable=E0401
     68 else:
     69     _hpex_available = False

ModuleNotFoundError: No module named 'habana_frameworks.torch'

To solve this problem executed commands below;

pip install --upgrade neural-compressor
pip install habana-tensorflow

But nothing changed.

What may cause this problem?

Thank you

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Execute code on CPU based system and run provided source code

### Expected behavior

Loading model without a problem

Comments:
Sorry my bad 😕

> Sorry my bad 😕

what was your issue?
fix ndim check of device_mesh for TP
Author: winglian
Pull request: yes

# What does this PR do?

This check is inverted. When I passed in a TP device mesh that's a single ndim, I hit this assertion. The TODO implies that multi-dim isn't supported yet, as well as the error message indicates that the check should be the inverse of the current check.

@SunMarc @S1ro1 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Ernie 4.5`] Add ernie text models
Author: vasqu
Pull request: yes
Labels: name: New model

Adding the Ernie 4.5 suite of models. 

Progress:
- [x] Ernie 4.5 pure text model (0.3B)
- [x] MoE Ernie
  - [x] Loading check with untied weights (tested on a dummy model)
  - [x] TP tests 
    - [x] Failing with tied weights, needs to be fixed then it's done 
  - [x] Correction bias clarification
    - Following the paddle code instead of remote - added a note so subject to change  
  - [x] Update configs on hub
  - [ ] (MTP support in training)
- [x] Integration test ^ (needs slow runs to cross check)
- [x] Check whether the MoE also need a rotation conversion (0.3B modeling files differ to the other ones regarding RoPE)
  - Yes, they do - turns out they use a similar trick as I did in 393c2c772aebeac0c6816bd2086c600254161ea5
  - Adapted from GLM as they do the same RoPE style as well
- [x] Fixup tokenization
  - [x] ~[Conversion](https://gist.github.com/vasqu/7828357fd24929f4d7a51202b0801c3e)~ see `convert...tokenizer`
  - [x] Update on the hub
- [x] Docs (might need updates based on the tokenizer ^)
- [x] Update original hub on baidu side --> tokenizer + configs

New/Followup PR:
- [ ] MoE Ernie VL
  - [ ] MoE is different (not allowing for the original MoE formula (Mixtral-based)?)
    - [ ] It can have different capacities
    - [ ] Different gating :eyes: 
  - [ ] 3D RoPE in image and text (with different RoPE formulation ~GLM style, even/odd instead of half/half)
  - [ ] Miscellaneous, as in the other remote code
    - [ ] Attention 
    - [ ] RMS norm
    - [ ] Residual
    - [ ] Proper padding support etc.

Comments:
run-slow: ernie4_5

This comment contains run-slow, running the specified jobs: 

models: ['models/ernie4_5']
quantizations: [] ...

Let's go! 🚀 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39228). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

`MoE Ernie VL` use `modality-isolation` routing (which is different from mainstream approaches)

should determin token-type in advance. (e.g. use `token_type_ids = input_ids >100295`)

then apply standart moe in two type of token：

```python
text_hidden =  hidden[token_type_ids==0]
text_router_logits = self.text_gate(text_hidden)
text_hidden = MoE(text_router_logits, text_hidden, self.text_experts)


visual_hidden =  hidden[token_type_ids==0]
visual_router_logits = self.text_gate(text_hidden)
visual_hidden = MoE(visual_router_logits, visual_hidden, self.visual_experts)


layer_output_hidden = torch.empty_like(hidden)
layer_output_hidden[token_type_ids==0] =  text_hidden
layer_output_hidden[token_type_ids==1] =  visual_hidden
```



Thanks, that clarifies a few things :) 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, ernie4_5, ernie4_5_moe
[Fast image processors] Improve handling of image-like inputs other than images (segmentation_maps)
Author: yonigozlan
Pull request: yes

# What does this PR do?

As the title says, **unbloats** (😉) a lot of the fast processing code for models needing to processed segmentation maps, trimaps, depth_maps etc.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39489). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker Should be 100% BC 🤗

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: beit, dpt, eomt, idefics2, idefics3, llava_next, llava_onevision, mobilenet_v2, mobilevit, qwen2_vl, sam, smolvlm, vitmatte
[`CI`] Fix post merge ernie 4.5
Author: vasqu
Pull request: yes

#39339 was merged right before the Ernie 4.5 models and I didn't check with main again :/ fixes the repo consistency based on that

cc @molbap @ArthurZucker 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5, ernie4_5_moe

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39561). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-KO] Translated `<auto_docstring>.md` to Korean
Author: chelsseeey
Pull request: yes

# What does this PR do?

Translated the `<auto_docstring>.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

May you please review this PR?
@4N3MONE, @yijun-lee, @jungnerd , @harheem 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
<!-- @stevhliu May you please review this PR? -->

Comments:
[gemma3] fix bidirectional image mask
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/39389. There were two issues. First, using `getattr` on dict will never look for a dict key thus we were skipping on `token_type_ids_mask`. 

Second, `kv_idx` can go beyond `token_type_ids` length because it includes the new tokens to be generated. Using a simple check on `ids.shape` fails in vmap due to dynamic control flow. One option would be to create dummy padded token type ids using `kv_idx` but I think a simple try-except is also fine

Comments:
run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39396). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3']
quantizations: [] ...

Just realized the `is_same_image` doesn't work correctly and unmasks bidirectional mask to future images. Working on it, don't review yet


run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3']
quantizations: [] ...

Can be reviewed now, slow tests are failing and I'll fix them tomorrow. No idea how to get `read_token` from inside runners, will ask Yih-Dar

Edit: yeah, we need to login with our own access tokens after ssh-ing

run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3']
quantizations: [] ...

Actually the slow tests should fail whenever it breaks, prob we didn't notice it because slow tests were failing for another reason already. I will try to make a fast test to check attentions

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3
Gemma3 bidirectional mask for image tokens isn't reaching attention forward
Author: brb-nv
Pull request: no
Labels: name: bug

### System Info

Transformers version: 4.53.2
Ubuntu x86, Nvidia H100

- I've noticed previous [issue](https://github.com/huggingface/transformers/issues/38053) about bidirectional mask for image tokens and MRs fixing it merged[[1](https://github.com/huggingface/transformers/pull/38080)][[2](https://github.com/huggingface/transformers/pull/38295)].
- I was looking at the attention mask being passed into attention forward at prefill for a single image and noticed that the mask is purely causal i.e. no bidirectional mask for image tokens.
- I'm afraid there might be a plumbing issue. Can someone please take a look?

### Who can help?

@zucchini-nlp @ArthurZucker @Cyrilvallez 

### Reproduction

Script to reproduce:
```
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

model_dir = "path/to/gemma-3-4b-it/"
model = Gemma3ForConditionalGeneration.from_pretrained(model_dir)
processor = AutoProcessor.from_pretrained(model_dir)
messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are a helpful assistant."}
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."},
        ]
    },
]

inputs = processor.apply_chat_template(messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True)
generate_ids = model.generate(**inputs, do_sample=False, num_beams=1, top_p=None, top_k=None, max_new_tokens=2)
outputs = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print(outputs)
```

### Expected behavior

Mask should be bidirectional for images as shown in this MR description:
Reference 1: https://github.com/huggingface/transformers/issues/38053
Reference 2: https://github.com/huggingface/transformers/pull/38295

Request to also validate the multi-image case.

Comments:
Will submit a PR, indeed the latest mask refactor broke inference
Add EfficientLoFTR model
Author: sbucaille
Pull request: no
Labels: name: New model

name: Vision

### Model description

EfficientLoFTR is an image matching model performing dense matching (contrary to SuperPoint + SuperGlue).
It is a variant to LoFTR that runs in realtime (at most 40ms for inference).
The base model performances are ok, but with the soon release of [MatchAnything](https://github.com/zju3dv/MatchAnything) version of EfficientLoFTR, we should have an even better model. 

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

https://github.com/zju3dv/EfficientLoFTR

Comments:
Bump AMD container for 2.7.1 PyTorch
Author: ahadnagy
Pull request: yes

# What does this PR do?

This PR bumps the base image for the AMD GPU container to get PyTorch version 2.7.1. Since a wheel package is not available yet, we need to pin to this specific container.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39458). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ydshieh since this is the docker CI images
object detection : matchin outputs.last_hidden_state with results
Author: fenaux
Pull request: no
Labels: name: Feature request

### Feature request

it seems to me that would be possible with a little modification in the function post_process_object_detection

with
```
``for score, label, box, index in zip(scores, labels, boxes, indexes):
            results.append(
                {
                    "scores": score[score > threshold],
                    "labels": label[score > threshold],
                    "boxes": box[score > threshold],
                    "indexes": index[score > threshold],
                }
            )``
```
 and then 
`outputs.last_hidden_state[0][results[0]['indexes']] `
gives me the desired vector features

Am I right or is there a better way to obtain this matching ? 

Thanks for your help

### Motivation

I would like to use outputs.last_hidden_state as features for auxiliary tasks. So I need to know the label and the bounding box associated to one given vector of outputs.last_hidden_state

### Your contribution

I am not a top coder and do not know how to submit a PR

Comments:
Hey @fenaux, `outputs.logits` and `outputs.pred_boxes` are used to obtain the results in `post_process_object_detection`

@qubvel, sorry but your comment doesn't explain to match the results to the embeddings. Let's say if i want to select embeddings of one given class as obtained after post_process_object_detection

Hi @fenaux, sorry, maybe I did not get what you mean, you can explore the modeling file to get a better understanding, e.g. `src/transformers/detr/modeling_detr.py`

https://github.com/huggingface/transformers/blob/main/src/transformers/models/detr/modeling_detr.py

Hi @qubvel 
In order to apply a tracker (for instance https://github.com/mikel-brostrom/boxmot/tree/master) the common format is to have detection 
- where detection is (bounding box, confidence, class). This is given by results.
- and the appearance of this detection being given by an embedding vector (very often the last hidden state of Clip or SigLip or a detection model)

Of course the appearance vector as to be associated with the right detection. So my question / proposal aims to find a way to make this matching without having to recompute every thing that has already been done in post_process_object_detection

I hope this help clarify

In that case, I would recommend writing a custom post-processing step based on `post_process_object_detection` to preserve the order of the features and detections

I suppose, the simplest case would be for DETR model

https://github.com/huggingface/transformers/blob/a44dcbe513e3e073271e0b8e369b75aca51affae/src/transformers/models/detr/image_processing_detr.py#L1775-L1826