noamaanMulla-03/transformers-issues huggingface dataset:
Simplify Idefics2, Idefics3, SmolVLM images handling
Author: yonigozlan
Pull request: yes

Simplify the handling of images in both processing and modeling.

Now the images/patches are flattened before being processed and passed to the models. This means that the image processing is simplified (no need for padding in the number of images/patches dimension), along with the modeling code ( No more padding images/patches containing only 0/False needing to be removed).

I tested thoroughly for each models with multiple images, batched images etc. and found no differences.

Cc @andimarafioti @orrzohar 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37291). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp Hello! Pinging you here as smolvlm also handles video inputs, and I'm wondering what you think about having flattened pixel_values by default when processing videos, instead of grouping them by frames or video instance. Also since most image (and maybe video?) processors for vlm using some kind of patching/splitting flatten the patches when preprocessing, we might want to update the base processing tests to account for that? Or at least make them parameterized

Just for info, there's a bug in video processing with smolvlm2 where the list of frames is malformed after calling `processor.apply_chat_template`, which makes generation not work properly. This started happening in transformers `4.52.1` (or maybe `4.52.0`, which was yanked). This PR works fine.

I saw it reported [here](https://github.com/Blaizzy/mlx-vlm/issues/388), but it's a transformers issue, not MLX's.
More robust tied weight test
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Models without tied weights should not need to override it to skip it -> let's do it inside directly


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: csm, dbrx, mamba2, musicgen, musicgen_melody, pix2struct, timm_backbone, xlstm
Update OLMoE model card
Author: nlhmnlhmnlhm
Pull request: yes

# What does this PR do?

Update OLMoE card
Unfortunately my computer do not have enough power to run the code and Colab too, making it a draft until I try a smaller OLMoE. 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case : https://github.com/huggingface/transformers/issues/36979.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
Hi @stevhliu, unfortunately I get to run only some of the code, not the pipeline and the quantization because of the reason above, i.e. too big checkpoints, I think that's the best I can do... 

Hi, sorry for the delay, concerning [this](https://github.com/huggingface/transformers/pull/39344#discussion_r2219967314), I changed it already unless I made a mistake. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skipping `initialize_weights` when model is quantized
Author: dwarez
Pull request: yes

# What does this PR do?
Avoids to perform weights initialization when loading a quantized model. The previous implementation was checking the `is_quantized` condition only when trying to initialize weights with deepspeed. 

Fixes #39366


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @zach-huggingface


Comments:
cc @mekkcyber as well!

@SunMarc any feedback on this?
Allow `device_mesh` have multiple dim 
Author: S1ro1
Pull request: yes

Experimental PR for me to just make stuff work on the accelerate side, will follow up with this after.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38949). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The wrong config parameter found in src/transformers/models/qwen2_5_vl/configuration_qwen2_5_vl.py.
Author: Bytes-Lin
Pull request: no
Labels: name: bug

### System Info

According to Qwen2.5-VL Technical Report, the channel of ViT and the in channel of VL-merger could be 1280. 

![Image](https://github.com/user-attachments/assets/4428c2ff-0888-4cde-99f6-a3341317eb43)

And I try to load the pretrained Qwen2.5-VL-7B model, clearly, the intput hidden size of merger is 1280. After operation, the in channel of linear is 5120. It's correct.

![Image](https://github.com/user-attachments/assets/297e9a4b-5604-4ae8-acb1-d62f46d557dc)

![Image](https://github.com/user-attachments/assets/ed9f6f92-e5fa-46e0-9b99-ea039ed99c86)

But in the qwen2_5_vl/configuration_qwen2_5_vl.py, we can see the hidden size in VisionConfig is set to 3584, so the in channel of merger becomes 3854 instead of 1280. I have some confusion with it, is it a wrong parameter?

![Image](https://github.com/user-attachments/assets/9b1bf3bb-af69-46e2-b556-0e43ca0659af)

![Image](https://github.com/user-attachments/assets/5e95c65d-36f2-48b8-8b35-5b07512d38d0)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

[        hidden_size=3584,](https://github.com/huggingface/transformers/blob/9cd7570f34fdb833ed874b2eba4d4ea3ae9ccb03/src/transformers/models/qwen2_5_vl/configuration_qwen2_5_vl.py#L37)

### Expected behavior

Please make sure the parameter is correct.

Comments:
Hey @Bytes-Lin !

The parameter values are defined by the Qwen authors and I am quite sure this was intended. The team tested the model for equivalence with the original implementation when shipping. Can you address this question to the authors under model-repo on the hub (https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)?

They will have more context on why these specific hidden dimensions were chosen

Thanks. @zucchini-nlp 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Not installable on arm64 due to jaxlib upper bound
Author: paveldikov
Pull request: no
Labels: name: Flax

`jaxlib` does not ship source distributions on PyPI, i.e. is only available as a wheel.

Newer versions of jaxlib do provide `aarch64` wheels, but `transformers` constrains `jaxlib` to `<=0.4.13`

Is it possible to relax this upper bound constraint? Or is there some very specific API-level breakage that was being guarded against? (I tried to infer by trawling through the `git blame` but nothing obvious jumped out at me -- looked like general defensiveness more than anything specific)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still an issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Bumping again.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Bumping yet again.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @paveldikov, sorry for the delay! JAX support in transformers has been deprecated and is likely to be removed entirely at some point, so we aren't really updating support for it right now.

Actually deprecation/removal sounds great -- thank you @Rocketknight1!

Since it's deprecated, perhaps this additional dependency can be moved off to a `transformers[jax]` extra to keep the dependency closure nice & tight for the base case? This should resolve the issue.

Hi @paveldikov, it should already be moved into `transformers[flax]`! The `deps_list` in `setup.py` lists versions for all deps, even optional ones.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[i18n-<bn>] Translating docs to <Bengali>
Author: ankitdutta428
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the Bengali-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [x] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
I would like to translate the index.md, quicktour.md and installation.md. @stevhliu can you kindly assign me the above. Thank you.
Fix checkpoint saving after interrupted training
Author: ved1beta
Pull request: yes

# What does this PR do?



Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
model.parameters() return [Parameter containing: tensor([], device='cuda:0', dtype=torch.bfloat16, requires_grad=True)] when using zero3
Author: fanfanffff1
Pull request: no
Labels: name: bug

### System Info

transformers             4.44.2
accelerate               1.2.1
deepspeed                0.12.2
torch                    2.2.2
torchaudio               2.2.2
torchvision              0.17.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Try to print **model.parameters()**  in transfomers trainer(), but get **Parameter containing: tensor([], device='cuda:0', dtype=torch.bfloat16, requires_grad=True)** for all layers
In fact, I am trying to return the correct **model.parameters()** in DeepSpeed Zero-3 mode and use the EMA model. Could you suggest any ways to solve the above issue, or any other methods to use the EMA model under Zero-3?

### Expected behavior

expect to see the gathered parameters

Comments:
Hi @fanfanffff1, I think we'll need some reproducer code for this! cc @sunmarc @muellerzr for Trainer either way.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Same issue
🌐 [i18n-KO] Translated `main_classes/processors.md` to Korean
Author: nevertmr
Pull request: yes

# What does this PR do?

Translated the `main_classes/processors.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
May you please review this PR?
@jungnerd
@harheem 
@4N3MONE 
@yijun-lee 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
<!-- @stevhliu May you please review this PR? -->

Comments:
[39519](https://github.com/huggingface/transformers/pull/39519) 와 중복 PR입니다.
Fix SigLIP2 documentation model/processor mismatch
Author: killerdevildog
Pull request: yes

- Change processor from siglip2-base-patch16-224 to siglip2-large-patch16-512 to match model
- Ensures quantization example works correctly
- Fixes issue #39692

# What does this PR do?

Documentation Update

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Documentation: @stevhliu


 -->


Comments:
Delete bad rebasing functions
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/39339 added back some functions that were not supposed to be here anymore!

cc @molbap for viz!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39672). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ah, missed that! Wrong rebase indeed, thanks for fixing!