noamaanMulla-03/transformers-issues huggingface dataset:
Error loading Gemma-2-9b-it tokenizer
Author: murthyrudra
Pull request: no
Labels: name: bug

### System Info

```
- `transformers` version: 4.43.2
- Platform: Linux-4.18.0-513.11.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.9
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB
```

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
My tokeizers version is `0.19.1`
When I load Gemma-2-9B model, I don't face any issues

```
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b")
```

However, when I try to load the instruction-tuned variant I am getting the following error
```
>>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/conda/lm_eval_eleuther/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 833, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/conda/lm_eval_eleuther/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 687, in get_tokenizer_config
    result = json.load(reader)
             ^^^^^^^^^^^^^^^^^
  File "/user/conda/lm_eval_eleuther/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/user/conda/lm_eval_eleuther/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/conda/lm_eval_eleuther/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1758 column 1 (char 39618)
```

### Expected behavior

No error and the tokenizer loads successfully

Comments:
Hey! I can't reproduce this, maybe a download issue? 

Hi, sorry seems to be the case. I tried it on a different machine and it's working. Thanks for the reply :)
Llama 3.1: Fix incorrect `inv_freq` assignment
Author: gante
Pull request: yes

# What does this PR do?

Fixes an incorrect `inv_freq` assignment, which caused the RoPE frequencies to be incorrect. Thank you @lostkevin for [raising the issue](https://github.com/huggingface/transformers/pull/32244#issuecomment-2257880431) 🤗 

Perhaps more importantly, adds a hard integration test to ensure we don't have numerical regressions.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32330). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Idefics2] - Fix FA2 call for Perceiver layer
Author: amyeroberts
Pull request: yes
Labels: name: run-slow

# What does this PR do?

The generation for IDEFICS2 was broken by https://github.com/huggingface/transformers/pull/31446

The previous logic controlled the sliding window using a boolean flag, whereas the refactored `flash_attn_forward` assumes `None` is for no `sliding_window`, and passing in `False` will propagate `flash_kwargs` of `{"window_size": (False, False)}` 

Adds a test so this will be caught in future

Fixes #32271

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32275). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Are there othermodel that are using `False` ❔ 

> Are there othermodel that are using False ❔

@ArthurZucker No, I don't think so from searching the repo. Technically it could happen if `sliding_window` was set to `False` in some model configs, but as that goes against the typing in the config docstrings I don't think it's likely 


I think some mistral models on the hub https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/blob/main/config.json have it to None so should be good!
Gemma 2: support assisted generation
Author: gante
Pull request: yes

# What does this PR do?

1. Makes Gemma 2 compatible with `DynamicCache`
2. Enabling `DynamicCache` makes Gemma 2 compatible with assisted gen. We might get some result drift after the size of the sliding window, though. I left a WIP to enable assisted gen with fixed-sized caches.

Results sanity checked, I can confirm that the output is UNCHANGED with an assistant model on relevant checkpoints

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32357). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker yes: e.g. [this test](https://github.com/huggingface/transformers/blob/5f1fcc299cb00c1edce5eb1efb8bacdde2365690/tests/generation/test_utils.py#L1708)
Fix error when streaming to gradio with non-string tool arguments
Author: aymeric-roucher
Pull request: yes

# What does this PR do?

Follow-up on [this question on Discord](https://discord.com/channels/879548962464493619/1267867376272150652/1267867376272150652).

Fixes an error raised in gradio when some non-string tool arguments were passed as the `content` attribute of a `gradio.ChatMessage`.

cc @freddyaboulton 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32360). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: warmup_steps check for training_args
Author: Ricardo-L-C
Pull request: yes

# What does this PR do?

A check is added at #30764 for `warmup_setps is either 0 or >= 1. Otherwise, raise an error.`

However, the checker in the code is 
```python
if not ... or 0 < self.warmup_steps <= 1:
    raise ValueError
```
that is, when `self.warmup_steps == 1` will raise this `ValueError`, and this is not as expected.

I discovered this issue because, in some of my scenarios, the "warmup_steps" was always being set to `1` by default with previous `transformers` versions.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @muellerzr 

Comments:
Thanks for your PR @Ricardo-L-C! @muellerzr should get back to you shortly.
LLaVa: add cache class attribute
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #32234. After removing support for tuple in major LLMs, llava models are not able to do beam search anymore. This PR fixes it by adding a `supports_cache=True` on all LLaVAs and on Paligemma.

Two things to note here:
- Yes, we could do smth like `self.lm._supports_cache_class`, but it was found recently that making it property doesn't work and we are thinking on how to make it work for SDPA flags in https://github.com/huggingface/transformers/pull/32238#issuecomment-2252949013. Since all models used by llava are now on cache class, imo we can set attr to `True`
- We cannot set other cache-related attr because LLaVa will not be able to support it before merging https://github.com/huggingface/transformers/pull/30962 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32278). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

If LLM doesn't support cache class, we'll raise an error but it won't be very informative from user's perspective. Currently informative errors are raised only checking these flags, which we set to `True`

But as I've said we now support cache class in all LLMs, expect for a few old-style like OPT or GPT2. I will work on those in the coming weeks
fix: Fixed `staticmethods` with self as first argument
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
The methods are not staticmethods. So, I removed @staticmethod decorator


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@LysandreJik @amyeroberts 

Comments:
> Thanks for looking into this @Sai-Suraj-27!
> 
> In this case, I think the decorators are correct, what we should update is the naming of `self` to `ctx` c.f. [the torch docs](https://pytorch.org/docs/stable/autograd.html#function)

Thanks for the quick review @amyeroberts. I made the changes please check now.
Serialization error when tokenizer_config key matches function name in PreTrainedTokenizerBase
Author: avnermay
Pull request: no
Labels: name: Core: Tokenization

name: Good Second Issue

https://github.com/huggingface/transformers/blob/37bba2a32d2742a10216ffd925bb8f145a732ce1/src/transformers/tokenization_utils_base.py#L2449-L2451

When one of the keys in `self.init_kwargs` matches the name of a function in PreTrainedTokenizerBase (e.g., `add_special_tokens`), this for loops replaces the value for that key in `tokenizer_config` with the function object, which is not serializable, thus causing an error during `save_pretrained`.

To solve this issue, one option is to add an assert in the `__init__` function that throws an error if one of the keys matches an existing attribute/function on the `PreTrainedTokenizerBase`:
https://github.com/huggingface/transformers/blob/37bba2a32d2742a10216ffd925bb8f145a732ce1/src/transformers/tokenization_utils_base.py#L1569

This error was also raised in the Stack Overflow issue below:
https://stackoverflow.com/questions/78062739/huggingface-transformers-error-when-saving-model-typeerror-object-of-type-meth

Comments:
cc @ArthurZucker 

Yep, this is known. I remember saying that I'd rather have a failure than duplicate attribute / functions. 
Do you want to open a PR to add some kind of check? 
I am fine with doing this in the __init__ as long as it does not slow it down too much
Make a new way to cusomize is_flash_attn_2_available 
Author: lucasjinreal
Pull request: no

Currrently, we mostly using same docker image for both V100 and A100, but for many models out there, they just didn't respect `attn_implementation` args at all in their code. They just brutly judge by is_flash_attn_2_available 

But this will always return true if it installed, regardless if it actually worked on V100.

And then,  error goes: 

rng_state = flash_attn_cuda.varlen_fwd(
RuntimeError: FlashAttention only supports Ampere GPUs or newer.


I know it might be reasonable, but **we have to uninstall flashattn maually** which is annoying.

An elegant way to do this, is to at least give a interface to customize `is_flash_attn_2_available ` in AutoModel.from_pretrained

Comments:
> but for many models out there, they just didn't respect attn_implementation args at all in their code

Hi @lucasjinreal, could you provide some examples of this, ideally with reproducible code snippets? 

I narrow down my issue to a new one, same request.

I don't know a elegant way to make my code didn't using flash-attn without chaning the model pretrained files. Please help a littl.
Fixed Hybrid Cache Shape Initialization.
Author: OsamaS99
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #31965 


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Narsil @gante 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32163). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama: RoPE refactor
Author: gante
Pull request: yes

# What does this PR do?

Same as #31999, but with llama being the only changed model.

_______________________________________

Confirmed: slow tests are "passing" (same failures as `main`)
👉 `RUN_SLOW=1 py.test -vv tests/models/llama/test_modeling_llama.py`
👉 `RUN_SLOW=1 py.test -vv tests/utils/test_cache_utils.py`
👉 `RUN_SLOW=1 py.test -vv tests/utils/test_modeling_rope_utils.py` (new tests)

_______________________________________

Throughput benchmarks: No changes vs previous `main` 💔 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merged the yarn PR (percursor), now merging this one as soon as CI goes green

Yarn PR is failing code quality checks on main. Could you make sure to rebase and then run make fix-copies etc here before merge?
pin gguf 
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR pins the gguf packages since it is incompatible with transformers integration of gguf file for now. I will unpin it when it's fixed ! 

cc @Titus-von-Koeller 

Comments:
@ydshieh Am I right in thinking we have to explicitly build a new docker file, rather than change here? Or is this sufficient c.f. this internal doc: https://www.notion.so/huggingface2/Circle-Ci-Docker-images-3bedc8bc602e4ca88c34c6c194accd74

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32290). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts 

No, for **daily** quantization CI job, it's this docker file to be modified. At midnight, a new image will be built automatically.

(I don't know much regarding if we run quantization tests on CircleCI for pull requests however)

As far as I know, we don't run the quantization tests for pull request ! 

Finally, I think we don't need to merge this PR !  https://github.com/huggingface/transformers/pull/32298

Thanks a lot @SunMarc, really appreciated that you coordinated this ❤️
AutoModel how to modify config?
Author: lucasjinreal
Pull request: no

```
config = AutoConfig.from_pretrained(
                    **self.params, trust_remote_code=True
                )
                config.vision_config.use_flash_attn = False
                print(config.vision_config)
                self.model = AutoModel.from_pretrained(
                    **self.params, trust_remote_code=True, config=config
                ).eval()
```

I need disable `use_flash_attn ` to False forcely when loading a model from pretrained. But looks like the config set didn't have any effect.

Why and how

Comments:
Hi @lucasjinreal, 

`use_flash_attn` isn't a standard config parameter used in transformers, so setting it won't have any effect. 

You can specify the attention mechanism you'd like to be used by specifying `attn_implementation` in the model's `from_pretrained` call


```py
self.model = AutoModel.from_pretrained(
    **self.params, trust_remote_code=True, attn_implementation="eager"
)
```

@amyeroberts  Sadly the model I using is InternVL, which didn't repect this config in their code. They just judge use flashattn only when it is installed, regardless the GPU support it or not.

I need:

- Don't uninstall flashattn since I don't want change the docker container;
- Make the judge fail even though the flashattn installed.

I have resolved this issue.
Tokenizers: Character encoding inconsistencies between __call__ and .convert_tokens_to_ids
Author: FrancescoManfredi
Pull request: no

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-6.1.0-21-amd64-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA")
original_tokens = ["à", "è", "ì", "ò", "ù"]

# converting tokens to ids with __call__ gives a wrong encoding
ids_direct_call = tokenizer(original_tokens)['input_ids']
print(ids_direct_call)
# > [[6496], [4558], [24439], [22980], [15273]]

ids_direct_call = [i[0] for i in ids_direct_call]
print(ids_direct_call)
# > [6496, 4558, 24439, 22980, 15273]

tokens_direct_call = tokenizer.convert_ids_to_tokens(ids_direct_call)
print(tokens_direct_call)
# > ['Ãł', 'Ã¨', 'Ã¬', 'Ã²', 'Ã¹']

# Converting the original tokens with .convert_tokens_to_ids gives the
# correct encoding
ids_explicit_convert = tokenizer.convert_tokens_to_ids(original_tokens)
print(ids_explicit_convert)
# > [156, 164, 168, 174, 181]

tokens_explicit_convert = tokenizer.convert_ids_to_tokens(ids_explicit_convert)
print(tokens_explicit_convert)
# > ['à', 'è', 'ì', 'ò', 'ù']

```

### Expected behavior

I'd expect both methods to behave in the same way and maybe some straightforward way to explicitly suggest the correct encoding to use.  
  
Also I'd like to know if I'm missing something and this is actually an expected behavior.

Comments:
Hey! I am not sure I understand why you would expect both methods to behave the same way? 
One is the full `encode`, which applies the `pre_tokenizer` , the `normalizer` while the other one, only converts the id to the corresponding token in the vocab. 

These methods are inherently different. 

Right. I don't know what made me expect these two to perform the same action but now I understand the whole process better. Thank you for your answer and your work!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.