noamaanMulla-03/transformers-issues huggingface dataset:
[tests] make `test_from_pretrained_low_cpu_mem_usage_equal` less flaky
Author: gante
Pull request: yes

# What does this PR do?

`test_from_pretrained_low_cpu_mem_usage_equal` has been failing frequently:
- [list of flaky tests](https://app.circleci.com/insights/github/huggingface/transformers/workflows/run_tests/tests?branch=main)
- [example](https://app.circleci.com/pipelines/github/huggingface/transformers/119017/workflows/2226c588-cf6d-409b-9b5b-dc39ace4af59/jobs/1606813)

Increasing the tolerance a little bit should do it 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36255). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @gante I was also looking at this, but interrupted by the lunch . It looks like it fails in the first run (where there is still no local cache for this model), and I am going to push something and let's see.

For the record: Still get 3 failures in a 100 runs.

@SunMarc the test indeed becomes slow (~10 secs) -- added the `@slow` decorator 🤗
Add BidirectionalWordpieceTokenizer for improved Korean/Japanese tokenization
Author: springkim
Pull request: yes

This PR adds a new BidirectionalWordpieceTokenizer class that supports bidirectional tokenization while maintaining compatibility with the existing WordpieceTokenizer interface. This implementation is based on the technique introduced in KR-BERT: A Small-Scale Korean-Specific Language Model (https://arxiv.org/abs/2008.03979), which has shown improved performance particularly for Korean and Japanese text processing.
Key changes:

Add BidirectionalWordpieceTokenizer class
Implement bidirectional tokenization with forward and backward vocabularies
Choose tokenization based on token length
Add test cases to verify functionality

The implementation has been tested and passes all core tokenization tests.
The bidirectional approach has demonstrated better performance specifically for Korean and Japanese languages, making it a valuable addition to our tokenization capabilities.


Comments:
Hi @springkim, I appreciate the PR but we **generally don't add new tokenizers to existing models!** Our goal with the tokenizer classes is to reproduce the tokenization that the model was trained with.

Instead, we recommend uploading KR-BERT as a [custom code repo](https://huggingface.co/docs/transformers/en/custom_models) with your new tokenizer code. This will let you share the tokenizer and the model weights together and let people easily download and use KR-BERT!
Add Example for Custom quantization
Author: MekkCyber
Pull request: yes

# What does this PR do?
Adds a working example for custom quantization 

## Who can review 
@SunMarc

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36286). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix callback handler reference
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR fixes a regression from https://github.com/huggingface/transformers/pull/35567. We updated the state instead of the callback handler. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36250). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[bugfix] Update modeling_llama.py so it skips keys correctly
Author: HDCharles
Pull request: yes

the llama model was using past_key_value and past_key_values interchangeably which caused issues because only one of those was actually skipped in _skip_keys_device_placement when both needed to be.

This PR changes it so only past_key_values is used.

without this fix, llama + torch.compile was having issues as in the surfaced issue below. Note, though this was surfaced due to a TorchAO recipe though the bug was unrelated to torchao

Fixes # (issue)

https://github.com/pytorch/ao/issues/1705

Comments:
ci/v4.49-release: tests collection fail with "No module named 'transformers.models.marian.convert_marian_to_pytorch'" on v4.48/v4.49 release branches
Author: dvrogozh
Pull request: no

Tests collection fails with the following error in https://github.com/huggingface/transformers/tree/v4.49-release branch (and for v4.48-release branch too).

```
python3 -m pytest tests/
===================================================== test session starts =====================================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/dvrogozh/git/huggingface/transformers
configfile: pyproject.toml
plugins: anyio-4.8.0, rich-0.2.0, subtests-0.14.1, xdist-3.6.1, asyncio-0.23.8, timeout-2.3.1, hypothesis-6.122.3, reportlog-0.4.0, dash-2.18.2, cov-6.0.0, typeguard-4.3.0
asyncio: mode=strict
collected 84774 items / 1 error

=========================================================== ERRORS ============================================================
________________________________ ERROR collecting tests/models/marian/test_modeling_marian.py _________________________________
ImportError while importing test module '/home/dvrogozh/git/huggingface/transformers/tests/models/marian/test_modeling_marian.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/models/marian/test_modeling_marian.py:50: in <module>
    from transformers.models.marian.convert_marian_to_pytorch import (
E   ModuleNotFoundError: No module named 'transformers.models.marian.convert_marian_to_pytorch'
====================================================== warnings summary =======================================================
src/transformers/optimization.py:640
  /home/dvrogozh/git/huggingface/transformers/src/transformers/optimization.py:640: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================== short test summary info ===================================================
ERROR tests/models/marian/test_modeling_marian.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================ 1 warning, 1 error in 17.54s =================================================
```

This happens due to some files are being pruned on release creation commit, see https://github.com/huggingface/transformers/commit/a22a4378d97d06b7a1d9abad6e0086d30fdea199. **Can we resolve this issue to be able to trigger all the tests on release branches?**

CC: @ArthurZucker @ydshieh 


Comments:
Hi @dvrogozh 

Thank you for reporting!

We no longer include the conversion files in the release. We are happy to adjust the test files to avoid the failure.
Fix broken CI due to missing conversion files 
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #36267.

Marian have a few tests regarding its conversion file, but in the release branch, we remove those files and tests are failing.
I don't have clear context if we should keep these (3) tests, and `marian` seems to be the only model where have tests involving conversion files.

In order to be run CI with [v4.49-release](https://github.com/huggingface/transformers/tree/v4.49-release), we will need a patch however.

cc @Rocketknight1 for visibility.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36275). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We are removing conversion scripts in the release as those cause some security reports and prevent users from enterprises from using `transformers`. Let's see what the other maintainers say 🤗  

I think we can just remove the tests for conversion files entirely!

Merge it. It's only about conversion files and `marian` is the only model that has such tests.
Ignore conversion files in test fetcher
Author: ydshieh
Pull request: yes

# What does this PR do?

Ignore conversion files in test fetcher

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36251). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
SmolVLM2
Author: orrzohar
Pull request: yes

# What does this PR do?


SmolVLM2 support

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [yes] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [no] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Comments:
cc @ArthurZucker  fyi this is needed for release otherwise it will be too much work on user's side 🥲

@molbap @zucchini-nlp 
Overall comments:
1. I have committed a revision that uses modular transformers, as suggested by @molbap. This required a minor edit to modeling_idefics3 to be compatible with the modular workflow.
2. I have refactored the inputs to the processor to be closer to the standard
3. I have removed our overwrite of apply_chat_template in favor of a new function that converts the video tokens into the expected sequence of text and image tokens, so we can use the original apply_chat_template
4. I have updated load_video to include `frame_indicies` to allow users to pass what frame idx's they are interested in loading. I also created a `get_video_details`, which fetches video metadata (fps, duration, frame count). We now use load_video rather than having custom video handling logic in smolvlm.


I believe all the comments on PR have been addressed. Let me know if I missed anything/if you have any new comments

Pushed a fix for the test_training being unhappy + an attempt at vectorizing the merger that seems to work, would need another eye on it :) 

@molbap thanks! I see you also added back the auto-map. The model was removed from `MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES` intentionally. We should aim to use only the ImageTextToText mapping for VLMs, instead of duplicating over two auto-classes

Yeah, it's just that if the model is not on the correct mappings, `test_training` fails unless we add manually some `labels` (else, they are not added by the auto class mapper). + Idefics3 had this double mapping AFAIK?
+1 to removing it though as long as tests pass, esp. training tests!

@molbap I think we can add the `ImageTextToText` mapping in `_prepare_for_class` to make tests happy, along with CausalMap etc. Idefics3 indeed has both, and all other VLMs too, but it doesn't really make sense to use Vision2Seq for new models. Especially after we released the pipeline and have been promoting the `image-text-to-text` tag on the hub

alright let me do that! then should be good

Thanks all!
[smolvlm] make CI green
Author: gante
Pull request: yes

# What does this PR do?

- adds smolvlm to doc toctree
- adds `num2words` requirement

Comments:
We're also getting lots of CI failures because all of the processor tests for SmolVLM depend on `num2words`!

Sorry, didn't notice it would fail. Actually the package is used only when chatting with video, so we might raise error only when `num2words` is actually called. And guard video chat template tests with `num2-word_available`. WDYT?

@zucchini-nlp at the end of the day, would fail too: either we have no tests for the feature, or we need to push the new requirement :D 

I'm leaving that part of the logic changes to you 🤗 And I agree, the check should only exist where it is strictly needed

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36306). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yep, we can later raise error only when video code is called. But making `num2words` a requirements means users won't have a problem with the import error at all now :)

Note: CI images being built [here](https://github.com/huggingface/transformers/pull/36308)

CI images pushed in the `push-ci-branch` branch ✅
add RAdamScheduleFree optimizer
Author: nhamanasu
Pull request: yes

# What does this PR do?

Integrates the new RAdamScheduleFree optimizer in meta's [schedule_free](https://github.com/facebookresearch/schedule_free).
This PR has greatly followed the previous PR to introduce schedule_free https://github.com/huggingface/transformers/pull/30079

## More details
- Added support for using RAdamScheduleFree through Trainer API
- Refined doc for trainer (clarified that we recommend to use constant LR scheduler with Schedule-Free family)
- Added test case for RAdamScheduleFree in `test_trainer.py` & confirmed the test passes

## Backgrounds
I recently made the RAdamScheduleFree optimizer, and it's been merged in schedule_free library since version 1.4.

- [The author of Schedule-Free paper's announcement post](https://x.com/aaron_defazio/status/1863722349138219459)
- [My announcement and release blog (in Japanese)](https://x.com/hamanasu_nagisa/status/1866413964235821492)

RAdamScheduleFree eliminates:
- Learning rate scheduler and terminal steps
- Warmup steps


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @ydshieh  @muellerzr


Comments:
@SunMarc

Thank you for the valuable feedback!
I've reflected your suggestions. I'd be grateful if you could review the changes.

Hi guys, is anyone working on this?

I think I did everything I could do for merging this PR, and just waiting for the approval from @ArthurZucker (sorry for mentioning...)

Could you just fix the conflits ? I will merge this right after ! 

Thanks for the fix, merging as the failing test is unrelated!

Sorry for the delayed follow-up, and thank you for taking care of the final fix to get this PR merged.
I really appreciate everyone’s help! 🎉
past_key_values cat out of model generate, output appear disorder
Author: lzlwakeup
Pull request: no
Labels: name: bug

name: Generation

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA 4090-24G

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


When I use past_key_values for generate, I have a problem. I want to cat prompt kv cache with current chat kv cache for limit gpu mem, but its fasle, this can be output, but not a right order sentence. 
 
code be like:

#prompt_kv_cache can be used
past_key_values = None
output = model.generate(input_ids,  past_key_values=past_key_values)
past_key_values = output.past_key_values

...some chat count 

last_past = []
for layer_past in past_key_values:
        past_keys, past_values = layer_past
        trimmed_past_keys =  past_keys[:, :, -len_kv:, :]
        trimmed_past_values =  past_values[:, :, -len_kv:, :]
        last_past.append((trimmed_past_keys, trimmed_past_values))
_past_all = []
    for p1_layer, p2_layer in zip(prompt_kv_cache, tuple(last_past)):     #prompt_kv_cache can be used for init, this true
        _past_all.append(tuple(torch.cat((p1, p2), dim=-2) for p1, p2 in zip(p1_layer, p2_layer)))
output = model.generate(input_ids,  past_key_values=tuple(_past_all))

Now, output.text is no-order.
For example, last input is "i want to eat"
output text maybe generate like "you \n\n need need some some some \n\n food food\n" or "you \n\n food \n need need some some some \n\n food food\n"

what wrong with past_key_values cat?


### Expected behavior

I want to know, how to cat a early kv cache with current kv cache

Comments:
Hey @lzlwakeup !

Not sure if I understood you correctly. can you share a small reproducible code with the prompts + model you are using? I think this might be related to the attention mask being constructed incorrectly or some padding issues when concatenating

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FA2 support for Aria
Author: molbap
Pull request: no
Labels: name: Vision

name: Flash Attention

name: Multimodal

Currently, Aria does not support FA2 in its modeling code: https://github.com/huggingface/transformers/blob/2fa876d2d824123b80ced9d689f75a153731769b/src/transformers/models/aria/modeling_aria.py#L656 
However the reference checkpoint for the Aria model does set `attn_implementation` to FA2: https://huggingface.co/rhymes-ai/Aria/blob/104c6548d7da08fc3a30c4232d35cc9fb3239942/config.json#L36 

So the modeling code on transformers side should be updated to support it. It currently causes errors on the CI and I think it's due to this mismatch. 

cc @aymeric-roucher @zucchini-nlp 

Comments:
I don't have write access to the repo, but I guess there should be a colab in slack to ping the authors

I have the write access to Aria repo. I will take a look at it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support for Multi-Modality Models (DeepSeek Janus-Pro-7B)
Author: smiling621
Pull request: no
Labels: name: Feature request

### Feature request

I’m requesting support for multi_modality models in transformers, specifically for models like DeepSeek Janus-Pro-7B.

Currently, when attempting to load this model using AutoModel.from_pretrained(), I received the following error:
KeyError: 'multi_modality'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/transformers/models/auto/configuration_auto.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
   1092                 config_class = CONFIG_MAPPING[config_dict["model_type"]]
   1093             except KeyError:
-> 1094                 raise ValueError(
   1095                     f"The checkpoint you are trying to load has model type `{config_dict['model_type']}` "
   1096                     "but Transformers does not recognize this architecture. This could be because of an "

ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

### Motivation

I’d like to use DeepSeek Janus-Pro-7B with transformers, but since it’s labeled as multi_modality, it cannot be loaded.

Is there an ETA for support?
Are there any workarounds to load the model until official support is added?

### Your contribution

At this time, I am unable to submit a PR, but I am happy to help with testing once support for multi_modality models is added. If there are any workarounds or steps to try, I’d be glad to assist in debugging and verifying compatibility.

Comments:
The same error! But I see someone use Janus-Pro-7B with transformers successfully. I'm wondering whether there was a release vision of transformers which can meet the need!

> The same error! But I see someone use Janus-Pro-7B with transformers successfully. I'm wondering whether there was a release vision of transformers which can meet the need!

And I notice that #35928 has mentioned the problem.

Hey, this is a duplicate of https://github.com/huggingface/transformers/issues/35928 and there's already a draft PR for that. Will close this as duplicate

The solution to your error can be found in https://github.com/huggingface/transformers/issues/36294, because the model needs to use Janus repo to load correctly 🤗
VLMs: even more clean-up
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Cleans up some code in VLMs, as we don't really need to override it. It will save us energy of copying a tiny change in `prepare_inputs_for_generation` across all models 

We'll need to prioritize using the `super()` whenever possible, imo, for future models! Less maintaining cost and less bugs

Ran generation tests + slow test for changed models, all green (except for those failing on `main`)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36249). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.