noamaanMulla-03/transformers-issues huggingface dataset:
Incorrect repr string for tokenizer objects
Author: gpetho
Pull request: no
Labels: name: Core: Tokenization

name: Good First Issue

name: bug

### System Info

transformers 4.46.0
Any OS and python version

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct') 
print(tokenizer)
```

### Expected behavior

The repr of tokenizer objects is incorrectly formatted due to this part of the code:
https://github.com/huggingface/transformers/blob/1d063793318b20654ebb850f48f43e0a247ab7bb/src/transformers/tokenization_utils_base.py#L1684C1-L1692C10

The repr of a Tokenizer object looks like this: `Tokenizer(...), added_tokens_decoder={...}`
Whereas is should look like this: `Tokenizer(..., added_tokens_decoder={...})`
The dict that is the value of the `added_tokens_decoder` attribute should be listed within the parentheses along with the other attributes, not after the closing parenthesis.

The current representation is problematic because having the `added_tokens_decoder` outside the main parenthesized structure breaks the expected flow of representing object attributes, and it's confusing. It suggests that the relationship between the tokenizer parameters and the added tokens decoder is different from what it actually is. Someone reading the string representation could assume it's a separate entity instead of an attribute belonging to the tokenizer.

Lines 1690-1691 should be corrected like this:
```
            f" special_tokens={self.special_tokens_map}, clean_up_tokenization_spaces={self.clean_up_tokenization_spaces}, "
            " added_tokens_decoder={\n\t" + added_tokens_decoder_rep + "\n})"
```

Comments:
Hey there is a small typo indeed:
```diff
(
            f"{self.__class__.__name__}(name_or_path='{self.name_or_path}',"
            f" vocab_size={self.vocab_size}, model_max_length={self.model_max_length}, is_fast={self.is_fast},"
            f" padding_side='{self.padding_side}', truncation_side='{self.truncation_side}',"
-            f" special_tokens={self.special_tokens_map}, clean_up_tokenization_spaces={self.clean_up_tokenization_spaces})"
+            f" special_tokens={self.special_tokens_map}, clean_up_tokenization_spaces={self.clean_up_tokenization_spaces},"
-            " added_tokens_decoder={\n\t" + added_tokens_decoder_rep + "\n}"
+            " added_tokens_decoder={\n\t" + added_tokens_decoder_rep + "\n}\n)"
        )
```
is probably what you want!

do you want to open a PR? 🤗 

Hi there, thanks for catching that typo and providing the fix! I appreciate you taking the time to point it out.
If @gpetho is busy at the moment, I’d be happy to quickly submit a small PR with the correction. 

@ArthurZucker  Shouldn't this issue be closed now?
🌐 [i18n-KO] Translated `llava_next.md` to Korean
Author: jeongiin
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `<your_file>.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조-->
<!-- May you please review this PR? --> 
@harheem, @jeongiin, @1kmmk1, @Seungahson, @win2dvp21
<!-- @junejae, @Jwaminju, @010kim, @4N3MONE, @boyunJang --> 
<!-- @jungnerd, @cjfghk5697, @yijun-lee, @mreraser --> 
<!-- @heuristicwave, @nuatmochoi, @chhaewxn, @jun048098, @ahnjj, @fabxoe -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33568). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MPS: `isin_mps_friendly` can support 0D tensors
Author: gante
Pull request: yes

# What does this PR do?

Fixes #33786 

Applies the fix in `isin_mps_friendly` as suggested by @Swastik-Swarup-Dash [here](https://github.com/huggingface/transformers/issues/33786#issuecomment-2382363240)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Do we need a patch for this? 🤗

I don't think so, it's quite niche 🤔 If we make a patch, yes, let's include it :D 

Thanks @gante for using my approach.
🌐 [i18n-KO] Translated perf_train_special.md to Korean
Author: maximizemaxwell
Pull request: yes

# What does this PR do?
Translated the `perf_train_special.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
-  [x] Check for missing / redundant translations (번역 누락/중복 검사)
 - [x] Grammar Check (맞춤법 검사)
 - [x] Review or Add new terms to glossary (용어 확인 및 추가)
 - [x] Check Inline TOC (e.g. [[lowercased-header]])
 - [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)
## Who can review? (Initial)
@junejae, @Jwaminju, @010kim, @4N3MONE, @boyunJang

## Before submitting
 - [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
 - [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
[documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
[here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
 - [ ] Did you write any new necessary tests?
## Who can review? (Final)
@stevhliu May you please review this PR?

Comments:
LGTM! Thanks for the nice translation 👍🏻

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34590). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Whisper CI
Author: ydshieh
Pull request: yes

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34541). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks, this fixes quite some tests no? Can we make sure with a tiny pipeline that by default we generate more than 20 tokens?

Do you mean a new test case that will `generate more than 20 tokens`? Any motivation for such a new test?

merge it as it will unblock whisper CI (170 -> 20 failures) + (quite) some pipeline tests
allow head_dim for llama like gemma or mistral
Author: bzantium
Pull request: yes

# What does this PR do?
- allow head_dim not to depend on hidden_size for llama like gemma or mistral
- it enables more flexible model hyper parameters without disturbing original models
- refer to: #32050
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #32846

## Who can review?

@ArthurZucker

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
make `LlamaDynamicNTKScalingRotaryEmbedding` work correctly
Author: bzantium
Pull request: yes

Fixes #27226 
to: @ArthurZucker, @younesbelkada 

Comments:
Great PR @bzantium!
I also tested RoPE scaling and realized the cache related code of it was wrong before.
this version. looks much more better.

cc @gante 

Hi @bzantium @hyunwoongko 👋 

There has been more than one issue raised regarding DynamicNTK as implemented by the authors and here in `transformers`:
1 - When `use_cache=True`, `past_key_values` is not using the right rotation for sequence lengths larger than the original model length -- https://github.com/huggingface/transformers/issues/25104
2 - [The one this PR addresses] When increasing then decreasing the sequence length, the base is not scaled back -- https://github.com/huggingface/transformers/pull/27033

There are solutions to both problems, but in both cases, they massively decrease the inference speed, which makes it a net negative change. I suspect this PR suffers from the same slowdowns as in https://github.com/huggingface/transformers/pull/27033 -- feel free to use the same test script as in there, taking [this comment](https://github.com/huggingface/transformers/pull/27033#issuecomment-1782738869) into consideration.

If the results are positive, I'd be happy to merge.

Thanks for the comment @gante!
Even though this PR would be slower than current code, the results are very different. I tested with `scaling_type=dynamic, scaling_factor=1` without truncation on the [longchat](https://github.com/DachengLi1/LongChat). The results are following:

lines | 50 | 100 | 200 | 300 | 400 | 500 | 600 | 680
-- | -- | -- | -- | -- | -- | -- | -- | --
original | 42 | 6 | 14 | 10 | 4 | 0 | 0 | 0
This PR | 98 | 100 | 38 | 12 | 6 | 4 | 2 | 0

Since this PR code resulted in far better performance, I think it should be fixed.

Hey @bzantium 

As I've written above, I will only consider merging after showing conclusive results that the change is a net positive -- with perplexity plots and execution time measurements. So far, you've only shown results in line with https://github.com/huggingface/transformers/pull/27033
Wrong ValueError in modeling_videomae.py?
Author: HadiHammoud44
Pull request: no
Labels: name: bug

name: Vision

### System Info

Hello, 

In **line 928** of modeling_videomae.py, I was wondering if the condition and `ValueError `raised is _wrong_ as **line 883** already checks if `num_channels != 3` and if so, does not _unnormalize_. Whilst the `else` block of line 928 is concerned with 'not normalizing' the pixels and not 'unnormalizing' in the sense of reversing the preprocessing steps (**line 887**). Even though I am setting `norm_pix_loss = False` when using `num_channels = 1`, I am still getting a `ValueError: Can't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.` 

Thank you in advance.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Not needed

### Expected behavior

N/A

Comments:
cc @NielsRogge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
deepspeed zero3 and bitsandbytes are not working
Author: eljandoubi
Pull request: no
Labels: name: bug

### System Info

docker image: 
nvcr.io/nvidia/pytorch:24.07-py3

requirements:

transformers==4.45.2
accelerate==1.0.1
bitsandbytes==0.44.1
sentencepiece==0.2.0
deepspeed==0.15.3



### Who can help?

@muellerz @SunMarc @MekkCyber 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

accelerate config:
compute_environment: LOCAL_MACHINE                                                                                                                                           
debug: false
deepspeed_config:
  deepspeed_multinode_launcher: standard
  gradient_accumulation_steps: auto
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_process_ip: 0.0.0.0
main_process_port: 0
main_training_function: main
mixed_precision: bf16
num_machines: 3
num_processes: 24
rdzv_backend: etcd-v2
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

================================================

I launch the code using 

```
accelerate launch --config_file config.yml code.py
```

```
from transformers import AutoModelForVision2Seq, BitsAndBytesConfig

model_kwgs = {
     "pretrained_model_name_or_path": "local_folder/contains/paligemma-3b-pt-896",
     "trust_remote_code":True,
     "attn_implementation": "eager",
     "quantization_config": BitsAndBytesConfig(load_in_8bit=True),
     }

model = AutoModelForVision2Seq.from_pretrained(**model_kwgs)
```
I got NotImplementedError: Cannot copy out of meta tensor; no data! Or a related error.
![image](https://github.com/user-attachments/assets/a1b1d51c-a86f-4679-bb1e-e74c3ca7d359)


### Expected behavior

Quantized model to train using deepsped

Comments:
FIX: Broken repr of TorchAoConfig
Author: BenjaminBossan
Pull request: yes

# What does this PR do?

The `__repr__` method references a non-existent `self.kwargs`. This is now fixed.

There does not appear to be a uniform way of defining `__repr__` for quantization configs. I copied the method as implemented for HQQ:

https://github.com/huggingface/transformers/blob/e2ac16b28a0b8b900e136750309ca40c49d975c5/src/transformers/utils/quantization_config.py#L285-L287

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

-

Comments:
@SunMarc Could you please review or suggest a reviewer?

The failing test is a timeout error and unrelated to this PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34560). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I'm not actually a core maintainer, but as this is obviously a bug and the fix is clean and without side effects elsewhere, I'm happy to merge it!

Not sure what the policy is, I think it's a very straightforward change and might not require a core maintainer to approve.

@BenjaminBossan, yes I'm happy for you to merge it once you can rerun/rebase/etc. and get the CI green!

> yes I'm happy for you to merge it once you can rerun/rebase/etc. and get the CI green!

Not sure if I should merge into transformers :D Let's wait if one of the cores chimes in, it's not critical.

Yes LGTM ! Thanks for the fix @BenjaminBossan
Add in free_memory passthrough
Author: muellerzr
Pull request: yes

# What does this PR do?

Per request in https://github.com/huggingface/accelerate/pull/2716, this PR introduces a passthrough `Trainer.free_memory` function. 

Now, since most of the refs are built inside the `Trainer` the user really only needs to include/pass through a `model` rather than needing *everything else*

Secondary fix to https://github.com/huggingface/transformers/issues/28178

(Duplicate of https://github.com/huggingface/transformers/pull/30549 but it never got merged 😅 )


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34519). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skip DeepSpeed ZeRO Stage 3 model initialization when bnb
Author: eljandoubi
Pull request: yes

# What does this PR do?

Skip DeepSpeed ZeRO Stage 3 model initialization when it is intended to be quantized.

Fixes #34378

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Comments:
@SunMarc @muellerzr What do you think of this solution?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34395). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Load sub-configs from composite configs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR removed overwritten `from_pretrained` in many configs that are part of a bigger config. Instead it moved the loading logic to the base config class through `base_config_key`. Additionally it adds a few more flags to the config, in a similar fashion to processors where each sub-config's class name is recorded. Currently those are needed only for tests to load the sub-config directly and compare with `base_config's` value

Weird thing is that `config._is_composition` is not an indicator of whether the config is composite or not. Several configs were missing the flag and operating good, because the flag is used only to separate out configs that can be init without params and those that require params. Those that require are flagged as `is_composition`, see more in https://github.com/huggingface/transformers/pull/25237 description and in https://github.com/huggingface/transformers/pull/7943 which introduced the flag.

I personally see only a few models that require input params (other configs) which are indeed composite, and all others don't need the flag so I removed it. But still I think the naming is quite misleading so maybe we should rename that in long-term or even remove that and find a better way to save configs when default params are overwritten.



Comments:
Modular is trying to overwrite InstructBlip and remove `logger`, even though we do `log.warning` in configuration code, so I have no idea why. Same thing is happening in my other PR for porting Emu3

Other failing tests have no relation to the current PR

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34410). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.