noamaanMulla-03/transformers-issues huggingface dataset:
feat(utils): add vision utils for embedding images and getting the hidden size
Author: AmitMY
Pull request: yes

# What does this PR do?

Adds generic utilities to understand the hidden size of an image encoder (currently need to write complex logic), and generically encode images using an image encoder.

Fixes #40457


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@qubvel @zucchini-nlp @ArkVex

Comments:
cc @zucchini-nlp @molbap @yonigozlan 

@molbap this is mainly a utility **for** VLMs
When building a VLM, you might want to map the dimensions of the images to the hidden dimension of the transformer - so you need to know how to "encode" and what is the "hidden_size"

How do you see it "extended" for VLMs?

As it is it appeared to be mostly for the `AutoModelForImageClassification` mapped models, right? We would need  to have an accompanying documentation then on how to use these utils, with several types of VLMs, also with vision backbones - without the classification head, it's a different class in that case.

And what I meant is that you might want to take an existing VLM and probe for the embedding images/hidden sizes, the documentation should explain how to do that.
Cannot load models in latest transformers version
Author: ri938
Pull request: no
Labels: name: bug

### System Info

master branch: transformers @ git+https://github.com/huggingface/transformers.git@1f0b490a2c42eb129dccc69031ccb537058689c4

its not possible to load models in latest transformers version. Because adaptor_config gets a token added  to it which results in duplicate value being passed later on.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code sample
```
model = "Qwen/Qwen3-30B-A3B"
token = '<redacted>'

config = AutoConfig.from_pretrained(
  model,
  token=token,
  trust_remote_code=True
)

args = dict(
    token=token,
    config=config,
    torch_dtype='bfloat16',
    trust_remote_code=True,
    force_download=False,
    attn_implementation='flash_attention_2'
)

AutoModelForSequenceClassification.from_pretrained(model, **args)
```

error message
```
[rank5]: Traceback (most recent call last):
[rank5]:   File "/code/reward_model/train.py", line 640, in <module>
[rank5]:     trainer = get_trainer(training_args)
[rank5]:   File "/code/reward_model/train.py", line 538, in get_trainer
[rank5]:     model = load_model(training_args)
[rank5]:   File "/code/reward_model/train.py", line 184, in load_model
[rank5]:     model = AutoModelForSequenceClassification.from_pretrained(
[rank5]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 372, in from_pretrained
[rank5]:     return model_class.from_pretrained(
[rank5]:   File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 270, in _wrapper
[rank5]:     return func(*args, **kwargs)
[rank5]:   File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4356, in from_pretrained
[rank5]:     _adapter_model_path, pretrained_model_name_or_path, adapter_kwargs = maybe_load_adapters(
[rank5]:   File "/usr/local/lib/python3.10/dist-packages/transformers/integrations/peft.py", line 655, in maybe_load_adapters
[rank5]:     _adapter_model_path = find_adapter_config_file(
[rank5]: TypeError: transformers.utils.peft_utils.find_adapter_config_file() got multiple values for keyword argument 'token'
```

### Expected behavior

It should not pass duplicate tokens.

Comments:
Seems to be a quick fix cc @ArthurZucker
make lfm2_moe integration test pass on XPU
Author: yao-matrix
Pull request: yes

@ydshieh , pls help review, thx very much

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: lfm2_moe

our CI runners have some issues and I'm waiting some fix from the infra team. Let me merge the PR as it is so not to block XPU progress. Thank you again.
🌐 [i18n-KO] Translated `perf_train_gaudi.md` to Korean
Author: D15M4S
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `perf_train_gaudi.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 KREW 팀원들에게 리뷰를 요청하는 아래 주석을 노출해주세요!-->
May you please review this PR?
@jungnerd, @luckyvickyricky, @chelsseeey, @skwh54, @amo33, @maximizemaxwell
@harheem
@4N3MONE
@yijun-lee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
<!-- @stevhliu May you please review this PR? -->


Comments:
🌐 [i18n-KO] Updated `text_generation.md`
Author: skwh54
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?
 
Updated the `text_generation.md` file in the documentation with a Korean translation.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179
Update of #[34590](https://github.com/huggingface/transformers/pull/33777)

## Before reviewing
- [X] Check for missing / redundant translations (번역 누락/중복 검사)
- [X] Grammar Check (맞춤법 검사)
- [X] Review or Add new terms to glossary (용어 확인 및 추가)
- [X] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [X] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 KREW 팀원들에게 리뷰를 요청하는 아래 주석을 노출해주세요!-->
May you please review this PR?
@jungnerd, @luckyvickyricky, @chelsseeey, @maximizemaxwell, @D15M4S

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
<!-- @stevhliu May you please review this PR? -->

Comments:
speed up loading checkpoints for zero stage 3
Author: ri938
Pull request: yes


Loading checkpoints for model Qwen/Qwen3-Next-80B-A3B-Instruct was very slow

this change brought down checkpoint loading times from 10mins+ to about 1.5mins

the load method is called over 4 million times (123,799 per layer and there are 41 layers). Looping through the state dict (size ~1500) was therefore very slow. This change speed the loading method by avoiding looping through the state dict.

Flagged for review
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- Big Model Inference: @SunMarc

Comments:
taking a look 

Removed accidenal addition of some other temporary code. Merge request should be good to go now.

lgtm, need approval from core maintainers @ArthurZucker @Cyrilvallez 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41850). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Allow parse_response to accept token IDs
Author: Rocketknight1
Pull request: yes

While I was working on `parse_response` I noticed that the UX was annoying in places because I had to keep decoding and then calling `parse_response`, which added a lot of boilerplate. Making `parse_response` optionally able to handle token IDs directly cut down on that a lot, so I figured I'd fix it now while there's still time before release!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41849). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update some workflow files
Author: ydshieh
Pull request: yes

# What does this PR do?

Mostly:

- Make `docker/transformers-all-latest-gpu/Dockerfile` more readable and clean as we now need to handle `torchcodec` (using `cpu`) along with `torch` (`cuda`)

- Remove `push-ci` stuff. We are not paying any attention to it. We have something running on a very small subset now.
- Separate CI workflows and their docker images: with `flash-attn` and without it



Comments:
I will try to check torch 2.9 + torchcodec 0.8 next week. At this moment, `FA2` and `detectorn2` fail to install with `torch 2.9`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merge as discussed offline on slack
fix some ut failures on XPU w/ torch 2.9
Author: yao-matrix
Pull request: yes

cases are below, all passed. @ydshieh , pls help review, thx very much.

> tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only
> tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_forward
> tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image
> tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_whisper_longform
> tests/test_pipeline_mixin.py::AutomaticSpeechRecognitionPipelineTests::test_whisper_longform
> tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_generation_no_images

> tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_bf16
> tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_crops
> tests/models/glm4v/test_modeling_glm4v.py::Glm4vIntegrationTest::test_small_model_integration_test_expand
> tests/models/mistral3/test_modeling_mistral3.py::Mistral3IntegrationTest::test_mistral3_integration_generate
> tests/models/mllama/test_modeling_mllama.py::MllamaForConditionalGenerationIntegrationTest::test_11b_model_integration_generate_text_only

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, aya_vision, gemma3, glm4v, mistral3, mllama
fix llama issue
Author: yao-matrix
Pull request: yes

similar like this PR https://github.com/huggingface/transformers/pull/39646, fix the same issue found while enabling llama lora finetuning across multiple card. 


@SunMarc , pls help review, thx very much. 

Comments:
@bot /style 

Style fix runs successfully without any file modified.

You need to propagate the changes to other repos also using make fix-copies. Btw since we are touching llama model, can you share more details to the issue you had ? 

> You need to propagate the changes to other repos also using make fix-copies. Btw since we are touching llama model, can you share more details to the issue you had ?

@SunMarc when I enable this multi-adapter lora inference in peft https://github.com/huggingface/peft/pull/2711 on a 2-card env, the issue raised, it's because this line https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L66 is not a Module, so not hooked, same issue as before we discussed.

> https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L66 is not a Module, so not hooked, same issue as before we discussed.

it is a nn.Module no ? `class LlamaRMSNorm(nn.Module):`. Can you share the traceback ? 


> > https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L66 is not a Module, so not hooked, same issue as before we discussed.
> 
> it is a nn.Module no ? `class LlamaRMSNorm(nn.Module):`. Can you share the traceback ?

I mean `return self.weight * hidden_states.to(input_dtype)` here `*` is not a `nn.Module`, so cannot hooked to place to same device automatically by `accelerate`, I'm trying to reproduce and get the log

seems i lost the env to reproduce it, let's put it on hold here, and once i reproduced it, i'll put the log back

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: arcee, aria, bitnet, cohere, cohere2, deepseek_v2, deepseek_v3, diffllama, doge, dots1, ernie4_5, gemma, gemma2, glm, glm4, glm4_moe
Fix confusing warning in EncoderDecoderModel when creating decoder_input_ids from labels
Author: st81
Pull request: yes

# What does this PR do?

Fixes #40168

As mentioned in the issue, it's confusing that the warning `"The decoder_input_ids are now created based on the labels, no need to pass them yourself anymore"` is shown even when the user doesn't pass `decoder_input_ids`. Fixed to show this warning only if `decoder_input_ids` is actually passed.

Note that the original warning message contains two parts: one about the `decoder_input_ids` issue, and another about the v4.12.0 change. I kept the latter warning message so it's still always shown, because it's a slight breaking change worth notifying users about (as discussed in [here](https://github.com/huggingface/transformers/pull/14139#pullrequestreview-789728506)).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

- @NielsRogge (since you refactored the EncoderDecoderModel to create decoder_input_ids from labels in #14139)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: encoder_decoder

cc @ArthurZucker is this warning still necessary given v5 is on the roadmap?

@ArthurZucker cc @NielsRogge 

This PR has been open for about a week now. Would you mind taking a look when you get a chance? Thanks!
evaluate>=0.4.6 is needed
Author: stas00
Pull request: yes

Some HF Transformers tests/examples fail with main and `evaluate < 0.4.6`

Fixing:
```
stderr: [rank0]: Traceback (most recent call last):
stderr: [rank0]:   File "/code/users/stas/github/transformers-alst-integration/examples/pytorch/question-answering/run_qa.py", line 692, in <module>
stderr: [rank0]:     main()
stderr: [rank0]:   File "/code/users/stas/github/transformers-alst-integration/examples/pytorch/question-answering/run_qa.py", line 608, in main
stderr: [rank0]:     metric = evaluate.load(
stderr: [rank0]:              ^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/loading.py", line 748, in load
stderr: [rank0]:     evaluation_module = evaluation_module_factory(
stderr: [rank0]:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/loading.py", line 680, in evaluation_module_factory
stderr: [rank0]:     raise e1 from None
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/loading.py", line 639, in evaluation_module_factory
stderr: [rank0]:     ).get_module()
stderr: [rank0]:       ^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/loading.py", line 479, in get_module
stderr: [rank0]:     local_path = self.download_loading_script(revision)
stderr: [rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/loading.py", line 469, in download_loading_script
stderr: [rank0]:     return cached_path(file_path, download_config=download_config)
stderr: [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/utils/file_utils.py", line 175, in cached_path
stderr: [rank0]:     output_path = get_from_cache(
stderr: [rank0]:                   ^^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/utils/file_utils.py", line 448, in get_from_cache
stderr: [rank0]:     headers = get_authentication_headers_for_url(url, token=token)
stderr: [rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
stderr: [rank0]:   File "/home/yak/miniconda3/envs/dev/lib/python3.12/site-packages/evaluate/utils/file_utils.py", line 236, in get_authentication_headers_for_url
stderr: [rank0]:     token = hf_api.HfFolder.get_token()
stderr: [rank0]:             ^^^^^^^^^^^^^^^
stderr: [rank0]: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'HfFolder'
```

`evaluate>=0.4.6` is needed to fix this.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41920). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you, Matt
No use `no_sync` context manager when using gradient accumulation w/ deepspeed's zero stage 2 or 3 via `accelerate`
Author: nzw0301
Pull request: no
Labels: name: Feature request

### Feature request

`trainer.train` with deepspeed stage 2 or 3 via `accelerate` and gradient accumulation does not work as I expected. I suspect this is because deepspeed 0.16.0 introduces `no_sync` context manager https://github.com/microsoft/DeepSpeed/pull/6675.

For example, the error looks like 

```
Traceback (most recent call last):
  File "main.py", line 70, in <module>
    main()
  File "main.py", line 90, in main
    trainer.train()
  File "/usr/local/lib/python3.11/dist-packages/transformers/trainer.py", line 2123, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/trainer.py", line 2480, in _inner_training_loop
    with context():
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/accelerate/accelerator.py", line 973, in no_sync
    with context():
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/deepspeed/runtime/engine.py", line 1995, in no_sync
    assert not self.zero_optimization_partition_gradients(), \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: no_sync context manager is incompatible with gradient partitioning logic of ZeRO stage 3
```


### Motivation

In my understanding, `trainer.train()` with deepspeed via HF `accelerate` and gradient accumulation uses own forward implementation rather than `accelerate.accumulate` defined at 

https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L2474C75-L2482

So `no_sync` is always used. Even we give

```python
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': {'sync_each_batch': True}, 'use_configured_state': False}
``` 
to `TrainingArguments` to use a feature introduced by https://github.com/huggingface/transformers/issues/29425.


### Your contribution

As suggested in https://github.com/huggingface/transformers/issues/29425#issuecomment-2505015569, adding some warning to config args on docs might be helpful, but ideally addressing this feature is much nicer but I do not know the right approach.

Comments:
cc @SunMarc @muellerzr

I'm not sure but https://github.com/huggingface/transformers/pull/35157 might fix the issue?

Let us know if this indeed fixes your issue ! 

Nope, 4.48.0 still have the exact same problem.

By updating transformers and deepspeed, the stage 3 works as I expected now.
Zero GPU quota doesn't reset
Author: EvgenyZaretskiy
Pull request: no
Labels: name: bug

### System Info

Thats second part for [#3362](https://github.com/huggingface/huggingface_hub/issues/3362).

Still want to know 1) Why somedays quota doesn't reset? (check for 18.09), 2) Why other days it does reset at different upredictable time of day?

I use the script to control status of my account page to retrieve quota. Check the file attached.
[hf-quota.txt](https://github.com/user-attachments/files/22457577/hf-quota.txt)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Control account page during the day and log seconds quota left.

### Expected behavior

Quota resets one a day at predictable time of day.

Comments:
Tagging @Michellehbn 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.