noamaanMulla-03/transformers-issues huggingface dataset:
🌐 [i18n-KO] Translated `conversations.md` to Korean
Author: newfull5
Pull request: yes

# What does this PR do?

Translated the `conversations.md` file of the documentation to Korean.
Thank you in advance for your review. 

OSSCA 팀원은 아니지만 우연히 발견하고 참여했습니다.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@jungnerd @heuristicwave 
<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
<!-- Team OSSCA, may you please review this PR? @bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @Sunmin0520, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
@stevhliu May you please review this PR?
<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? @sgugger, @ArthurZucker, @stevhliu -->

Comments:
안녕하세요~, @newfull5님 기여 감사합니다! 리뷰를 받기 전 Draft 상태를 권장드리고 있어요!
Open > Draft로 변경해주시면, 리뷰어들의 제안을 반영하기 더욱 수월하답니다!
live-preview 도 아직은 볼 수 없는 단계라, 체크리스트 해제를 부탁드려요!
리뷰가 끝나면, Open과 변경함과 동시에 최종 리뷰어를 호출해주시면 됩니다!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32468). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Training on Teacher model logits
Author: Theodotus1243
Pull request: no
Labels: name: Feature request

### Feature request

Is there a possibility to add training on bigger model logits
It's a question of training on logits instead of one-hot vectors from dataset text

### Motivation

DistillKit slows and Nvidia-Minitron slow good result of such technique

### Your contribution

DistillKit https://github.com/arcee-ai/DistillKit
Minitron https://arxiv.org/pdf/2408.11796

Dublicate of https://github.com/huggingface/trl/issues/1975

Comments:
update push CI workflow files for security
Author: ydshieh
Pull request: yes

# What does this PR do?

See [internal discussion](https://huggingface.slack.com/archives/C01NE71C4F7/p1724750779061389?thread_ts=1724157181.387969&cid=C01NE71C4F7)



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33142). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Granite language models
Author: mayank31398
Pull request: yes
Labels: name: New model

name: run-slow

# What does this PR do?
This PR adds support for IBM's upcoming LLMs 3B and 8B.

- text models: @ArthurZucker and @younesbelkada


Comments:
@ArthurZucker @younesbelkada 

hey @younesbelkada 
This is for our upcoming open model releases.
3B and 8B language models (lots of tokens :D)

lets just leave this PR for now.
I will get back to this in a few days.

Hey! Do you need another review here? 

yeah, @ArthurZucker
I need to fix the model on HF to make it work with the renamed params: https://huggingface.co/mayank-mishra/granite-3b-mup
but you can give a review, that will be helpful

@ArthurZucker dropping this breaks a unit tests
https://github.com/huggingface/transformers/pull/31502#discussion_r1675633090

Which test? 

FAILED tests/models/granite/test_modeling_granite.py::GraniteModelTest::test_model_outputs_equivalence - RuntimeError: The size of tensor a (7) must match the size of tensor b (32) at non-singleton dimension 3

dug a little and found that since `past_key_values` is None its being removed when outputting a tupple from the model.
this results in the tuple and dict outputs to have different number of outputs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31502). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker quick question do you think that this commit is a reasonable change?
https://github.com/huggingface/transformers/pull/31502/commits/b64c16d05decc13e23314e192807ccdbefdd5b85
It moves rope out of the attention modules since with the padding free integration, we already pass rope to each attention layer?

Yep absolutely perfect


Cool! Ping me once this needs a merge ~= when checkpoints are released! 

@ArthurZucker this is ready for merge

I have addressed the changes

Thanks for bearing with me 🤗 

passed docs 🥳 

thanks Arthur :)

Thank you as well! 🤗 

hello!

module 'torch.nn' has no attribute 'RMSNorm'

The version of torch < 2.4.0 will report an error.
Error when converting "state-spaces/mamba2-130m" weights to huggingface-compatible format
Author: learning-chip
Pull request: no
Labels: name: Good Second Issue

name: bug

### System Info

- Transformers version: 4.40.0

### Who can help?

@molbap @ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I tried to load https://huggingface.co/state-spaces/mamba2-130m into HF-compatible Mamba-2 (https://github.com/huggingface/transformers/pull/32080), using the [convert_mamba2_ssm_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/v4.44.0/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py) script. But the script assumes model weights to be in safetensors format:

https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py#L32-L35

but the weight file is [is in torch bin format](https://huggingface.co/state-spaces/mamba2-130m/blob/main/pytorch_model.bin) and cannot be opened in this way.

Also, the script requires a tokenizer path:

https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py#L55-L61

but `state-spaces/mamba2-130m` [reuses](https://github.com/state-spaces/mamba/blob/v2.2.2/benchmarks/benchmark_generation_mamba_simple.py#L37) `EleutherAI/gpt-neox-20b` tokenizer instead of having its own.

### Expected behavior

`convert_mamba2_ssm_checkpoint_to_pytorch.py` should be able to convert those Mamba-2 weights:

- https://huggingface.co/state-spaces/mamba2-130m
- https://huggingface.co/state-spaces/mamba2-370m
- https://huggingface.co/state-spaces/mamba2-780m
- https://huggingface.co/state-spaces/mamba2-1.3b
- https://huggingface.co/state-spaces/mamba2-2.7b

Comments:
Thanks for the issue! Yes, the current conversion script is made for the mistral/codestral mamba 2 release, which uses safetensors + their own tokenizer - if you want to work on a modification to the conversion script on a new PR, feel free to do so and we can help! Else, I'll take a look at that later :)
Static KV cache with CPU offloading
Author: gerbenvv
Pull request: no
Labels: name: Feature request

name: Cache

### Feature request

I'd like to contribute a static offloading KV cache to the existing proposal of the dynamic offloading KV cache proposed here: #30704

### Motivation

Nice to have an option that is static as well, since it allows for much improvement performance (comparable to `StaticCache` vs `DynamicCache`). Also, I believe some models only work with a static cache.

### Your contribution

I have created this gist:

https://gist.github.com/gerbenvv/282ed3c981a63ad71a301cdc1a705ef1

as the proposal. Also, see discussion at #30704

Comments:
cc @gante @ArthurZucker 

I'm testing my implementation but I think the argument `num_return_sequences` is broken with the static caches. It isn't taken into account when creating the maximum batch size for the cache.

Tested it and unfortunately had to remove the non-blocking transfers to the CPU. It was just too flaky since it didn't reliably work on batch sizes bigger than 1. Anyway, tested it with:

- L40S (48 GB)
- Llama 3 70B (4 bit quantized)
- sdpa attention
- batch size of 1000
- sequence size of 25 (no prompt, fully generative)

and I am getting a throughput of about 535 tokens/s (OOM static).

Also with:

- batch size of 1
- sequence size of 100 (no prompt, fully generative)

which gets 10.6 tokens/s (12.8 tokens/s static).

And with:

- batch size of 1
- sequence size of 100 (fully prompt)

which does 98.8 tokens/s (106.8 tokens/s static)

And with:

- batch size of 50
- sequence size of 100 (fully prompt)

which does 939.5 tokens/s (995.6 tokens/s static)

This offloading really shines for high batch sizes since the model scales super well in batch size. Expect even higher tokens/s when doing long prompts, since then it can also parallelize over the sequence dimension. Unfortunately I cannot test this since I am limited to the L40S for the moment.

Cool stuff! What kind of throughput does StaticCache without offloading achieve?

For Llama 7B (llama2) you have the same as GPT-Fast sor 110 token / second (for batch 1 I think), have some old gists. 
This is nice and I can test for you as well if needed

Num return sequences reported here: #31965 is indeed broken for static AFAIK

Added static numbers for comparison. Looks like overhead is lower the more the model can parallelize in batch/sequence dimensions. And since that is now decoupled from GPU memory (for the most part), it will be interesting to see this being applied for inference at scale.

Now all we need is to offload the quantized (4 bit) parameters as well, quantize the KV cache and add tensor parallelization to be able to run Llama 3.1 405B on a quad 4090 setup. By calculation, that should give realtime performance for a machine that should cost 12.000 euros. :grin: 

Yeah tensor parallelism is .... I don't want to say planned but I'd actually like to do it 👁️ 

The performance numbers are 🔥 

_____________________________

> Yeah tensor parallelism is .... I don't want to say planned but I'd actually like to do it 👁️

👆 ➕
PR #31502 bumps up PyTorch requirement to >=2.4
Author: gheinrich
Pull request: no
Labels: name: bug

### System Info

Hello, unless I am mistaken, PR #31502 bumps up the PyTorch version requirement to >=2.4. This requirement comes from [pytorch_utils.py](https://github.com/huggingface/transformers/blob/main/src/transformers/pytorch_utils.py#L27):

```
ALL_LAYERNORM_LAYERS = [nn.LayerNorm, nn.RMSNorm]
```

I think `torch.nn.RMSNorm` was introduced in PyTorch 2.4.

However the main [README](https://github.com/huggingface/transformers/blob/main/README.md) still states that:

> This repository is tested on Python 3.8+, Flax 0.4.1+, PyTorch 1.11+, and TensorFlow 2.6+.

cc @ArthurZucker 



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

With PyTorch<2.4:

```
python -c "from transformers import pytorch_utils"
Traceback (most recent call last):
  File "/opt/conda/envs/vila/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1659, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/opt/conda/envs/vila/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/conda/envs/vila/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 27, in <module>
    ALL_LAYERNORM_LAYERS = [nn.LayerNorm, nn.RMSNorm]
AttributeError: module 'torch.nn' has no attribute 'RMSNorm'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/opt/conda/envs/vila/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1647, in __getattr__
    value = self._get_module(name)
  File "/opt/conda/envs/vila/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1661, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.pytorch_utils because of the following error (look up to see its traceback):
module 'torch.nn' has no attribute 'RMSNorm'
```

### Expected behavior

The documentation should be updated, or we should not have to use PyTorch>=2.4.

Comments:
 #33177 fixes this, it will be reverted to the non-torch version of RMSNorm so that older versions are compatible. 

I see, thank you.
[WIP]Add ProPainter
Author: shauray8
Pull request: yes

# What does this PR do?

Adds ProPainter to transformers.
Author - https://github.com/sczhou/ProPainter
hub - https://huggingface.co/shauray/ProPainter-hf/

Fixes https://github.com/huggingface/transformers/issues/26360
*issue contains all the other relevant links*

## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@rafaelpadilla @LysandreJik 


Comments:
Hi @shauray8 :)

Please, let me know when it is ready for review 

I'll ping you before the EOD.

*Update - got caught up with something, Writing tests!*

@shauray8 Any update on adding this model?

@amyeroberts it's almost done, I'm stuck with tests rn and I might be able to push it for 1st review tomorrow.

@shauray8 Great! Let us know if you need any help with the tests.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

What happens with this PR? :-)

@royvelich This PR became inactive and was then closed automatically. If you or anyone else in the community would like to open another PR to add this model we'd be happy to review! 

@amyeroberts Can I take this on?! As I'm already looking into it. :)

@RUFFY-369 Sure!
[i18n-ur] Translating docs to Urdu (اردو)
Author: akkefa
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the Urdu-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

@akkefa 

## Get Started section

- [ ] index.md ( Work in progress )
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
@stevhliu Please review my pull request.
Add ProPainter to transformers
Author: shauray8
Pull request: no
Labels: name: New model

### Model description

ProPainter can be an excellent addition to Transformers given its impressive inpainting capabilities. 

*Relevant Links*
Paper - https://arxiv.org/abs/2309.03897
Project Page - https://shangchenzhou.com/projects/ProPainter/
Original Code - https://github.com/sczhou/ProPainter
Weights - https://github.com/sczhou/ProPainter/releases/tag/v0.1.0 (Needs to be merged into a unified one)

Author - @sczhou 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

@amyeroberts @ArthurZucker
If you think this is a valuable addition I'm more than happy to work on it. 

Comments:
LGTM, I would also like to be a part this contrib... @shauray8 

@mahimairaja of course, let's wait for approval from a core maintainer.

Looks good :)

What do you think @rafaelpadilla ?

Great suggestion @shauray8! 
The original repo has 1.4k stars in 3 weeks. I thiink it is a great model to add! :) 

Okay then, working on it!

Hi @shauray8, I would like to connect with you...
regard this issue!

@mahimairaja what is your preferred mode of conversation?

Yeah, shall we connect on discord? my username - mahimairaja

I find this model and idea awesome
I would like to work on this issue, can you please assign it to me?

Glad to see that you have been working on ProPainter together :) 
When it's ready, please, ping me for the first review. 

Is anyone still working on this?

@royvelich The previous PRs were closed because of inactivity. If you or anyone else in the community would like to open a PR to add the model we'd be very happy to review! 

@amyeroberts  I would be happy to complete it :)

@RUFFY-369 Great! 

@RUFFY-369 @amyeroberts 
I already have a semi-productized adaptation for ProPainer in my own private project, so if you want to collaborate, count me in.
Fix: fix paligemma generation
Author: hiyouga
Pull request: yes

# What does this PR do?

In the latest transformers, Llama and Gemma models introduce `num_logits_to_keep`, whereas the Llava and PaliGemma VLMs do not accept this argument. The program raises exceptions:

https://github.com/huggingface/transformers/blob/c409cd81777fb27aadc043ed3d8339dbc020fb3b/src/transformers/models/gemma/modeling_gemma.py#L1184-L1193

```
TypeError: PaliGemmaForConditionalGeneration.forward() got an unexpected keyword argument 'num_logits_to_keep'
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts 

Comments:
Close because duplicate of https://github.com/huggingface/transformers/pull/33168
Fix jetmoe model
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Fixes #31266
cc @ArthurZucker @amyeroberts 

I am sorry I did not notice that `JetMoePreTrainedModel` got `_supports_cache_class = True` support while working on #30536.

To avoid the same bug in the future, any model getting `_supports_cache_class = True` in the future should change the line:

```
if inputs_embeds is not None and past_key_values is None:
            model_inputs = {"inputs_embeds": inputs_embeds}
``` 

to 

```
if inputs_embeds is not None and past_length == 0:
            model_inputs = {"inputs_embeds": inputs_embeds}
``` 

in `prepare_inputs_for_generation()` because checking `past_key_values is None` is no longer correct as empty but initialized `DynamicCache` can be passed.

Also to minimize code complexity, the if-else that checks `if isinstance(past_key_values, Cache):` (still in `prepare_inputs_for_generation()`) can be safely removed in favor of the case when this is True when `_supports_cache_class = True` as `generate()` will only use proper `Cache` classes in this case.
Some similar if-else could also be removed in upstream model architectures classes as well.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31279). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.