noamaanMulla-03/transformers-issues huggingface dataset:
SeamlessM4TFeatureExtractor fails with pad_to_multiple_of not being a multiple of stride
Author: TechInterMezzo
Pull request: no
Labels: name: Feature extractors

name: Audio

https://github.com/huggingface/transformers/blob/e314395277d784a34ee99526f48155d4d62cff3d/src/transformers/models/seamless_m4t/feature_extraction_seamless_m4t.py#L285-L290

I tried to study and understand the code for the SeamlessM4TFeatureExtractor.
The code that I referenced above does not seem to be complete.
num_frames is already the size for the second dimension, so the last two lines would not be doing anything.
My guess is, that remainder should be subtracted from num_frames to get a size that is a multiple of stride.
And because of the padding this was never a problem.
Or is my thought process wrong?

Comments:
cc @ylacombe 

Hey @TechInterMezzo, thanks for catching this, the code should indeed substract `remainder` from `num_frames`!

In the default parameters, this wasn't caught because `stride == pad_to_multiple_of`, but there could be some cases in which it's not the case. Would you mind opening a PR to correct this? 
You could add a test [here](https://github.com/huggingface/transformers/blob/e314395277d784a34ee99526f48155d4d62cff3d/tests/models/seamless_m4t/test_feature_extraction_seamless_m4t.py#L112C1-L113C1) to make sure there's no error when  `stride != pad_to_multiple_of` and `remainder!=0`.

Let me know if this is something you're willing to do !


Hi @ylacombe!
I would like to give it a try at the end of the week.
As this would be my first contribution to the transformers repo I will need to study the [contribution guide](https://huggingface.co/docs/transformers/contributing) first.

Great @TechInterMezzo, let us know if you run into any problems after reading the contribution guide!

@ylacombe I created a pull request after following the contribution guide. I hope I did it right. The test I made is based on the existing call test with numpy inputs. It failed before the fix and worked after it.

Thanks @TechInterMezzo for fixing this issue.
@ylacombe Can we please get this merged? Would be helpful for me as well.

> @ylacombe I created a pull request after following the contribution guide. I hope I did it right. The test I made is based on the existing call test with numpy inputs. It failed before the fix and worked after it.

You did it right, thanks ;)
Phi3 tests: fix typing for Python 3.8
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, fixes typing to be compatible with older python versions


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32388). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
#32184 save total_vocab_size
Author: itazap
Pull request: yes

fixes #32184 save total_vocab_size to speed up operation

- updates` len(tokenizer)` to `len(self.get_vocab())` from `len(set(self.get_vocab().keys()))` as it is equivalent for Python 2.x and 3.x due to dict key uniqueness.

- stores an internal variable called total_vocab_size to avoid bottleneck in `len(self.get_vocab()`.

TESTS:

- existing tests check `len(tokenizer)` and pass! (see: `test_tokenization_common` --> `test_add_tokens`

@ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32240). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Leaving the review to @ArthurZucker, tokenizer mastermind :)

added a small test in `test_tokenization_utils.py` to explicitly test len(tokenizer). Other tests also implicitly test the length behaviour, such as `test_tokenization_common.TokenizerTesterMixin.test_add_tokens_tokenizer`
Avoid decoding the entire token cache for every new token during text streaming
Author: prith189
Pull request: yes

# What does this PR do?
In the current implementation of the put() function in the TextStreamer class, the entire token cache is decoded for every new token. This is redundant, and expensive especially for very long context generations. In the case of long context generations, the total time it takes to complete the generation with streaming is higher compared to the case without streaming,

In this change, instead of maintaining token cache and decoding the entire cache every time, we maintain the decoded text and append to it each time a new token is generated.

I have tested a mini repro on M2 Pro, where the following example is faster by about 1 second in generating the long sequence. But in a real long context generation case, the difference is larger.

```
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
inputs = tok(["An increasing sequence from one to thousand: one,"], return_tensors="pt")
streamer = TextStreamer(tok)

# Despite returning the usual output, the streamer will also print the generated text to stdout.
model.generate(**inputs, streamer=streamer, max_new_tokens=1023)
```

Before change:
Time taken = 21.9 seconds (Averaged over 100 iterations)

After change:
Time taken = 20.2 seconds (Averaged over 100 iterations)

Verified that the tests pass.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@gante 

Comments:
cc @gante for the text streamer!
Added HHCache class implementing H2O Cache
Author: belericant
Pull request: yes
Labels: name: Cache

# What does this PR do?
This PR adds the feature requested in #30758. The HHCache class is almost directly taken from the original H2O paper's authors code found [here](https://github.com/FMInference/H2O). Currently the PR only adds the changes required to Llama model class. As of now I have taken @gante 's suggestion of adding `Cache.post_process()` and calling it within `LlamaAttention.forward`.

# To-Do
1. I'm not sure if the logic for RoPE rerotation is 100% correct. I think the recent tokens are correct, but not the hh tokens after eviction. Would love to have another set of eyes on that.
2. Write tests to ensure that this HHCache class has the same behavior compared to the original code by paper authors.
3. Benchmarking(?)

Feedback and/or help would be appreciated. Thanks!

Comments:
cc @gante @ArthurZucker 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

RE-opened as we were waiting for the todos, @belericant should I close it?
add values for neftune
Author: nbroad1881
Pull request: yes

I always forget what typical values are, and I have to look at the paper every time. This will be a helpful reminder 

@muellerzr, @SunMarc

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32399). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
compute_metric(eval_pred) in trainer is not mini-batch
Author: SamYuen101234
Pull request: no

I am trying to implement a custom compute metric for trainer. The logits and labels are numpy array of the full evaluation data, however, my evaluation data input has the size (1000, 43, 50257). The computation can't be done in a 24GB L4 GPU on colab. Any way to load the data in mini batch like using dataloader instead of given a full numpy array.

`# eval_pred is all the valid data not only the mini-batch
def compute_metrics(eval_pred):
    accuracy_metric = load_metric("accuracy")
    logits, labels = eval_pred

    # Get predictions (next word prediction)
    predictions = np.argmax(logits, axis=-1)

    # Shift labels to the left
    labels_shifted = labels[:, 1:].flatten()
    predictions_shifted = predictions[:, :-1].flatten()

    # Create an attention mask based on labels (assuming -100 is padding)
    attention_mask_shifted = (labels[:, 1:] != -100).flatten()

    # Remove padding tokens using attention mask
    predictions_shifted = predictions_shifted[attention_mask_shifted]
    labels_shifted = labels_shifted[attention_mask_shifted]

    # Compute accuracy
    if len(labels_shifted) == 0:
        return {"accuracy": 0.0}
    accuracy = accuracy_metric.compute(predictions=predictions_shifted, references=labels_shifted)

    return {"accuracy": accuracy["accuracy"]}`

Comments:
Hi @SamYuen101234, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Bug] Modifying normalizer for pretrained tokenizers don't consistently work
Author: alvations
Pull request: no
Labels: name: Core: Tokenization

### System Info

transformers==4.41.2

### Who can help?

 @ArthurZucker

### Reproduction

From https://github.com/huggingface/tokenizers/issues/1552#issue-2348487489


```python
from transformers import AutoTokenizer
from tokenizers.normalizers import Sequence, Replace, Prepend

tokenizer_name = "mistralai/Mistral-7B-v0.1"
old_tok = AutoTokenizer.from_pretrained(tokenizer_name)

assert old_tok.backend_tokenizer.normalizer != None

new_normalizer = Sequence(
    [Prepend('▁'), Replace('▁', ' '), Replace("foo", "bar"), Replace('<br>', '\n')]
)

old_tok.backend_tokenizer.normalizer = new_normalizer
new_tokenizdr_name = f"new_tokenizer-{tokenizer_name}"
old_tok.save_pretrained(new_tokenizdr_name)


old_tok = AutoTokenizer.from_pretrained(tokenizer_name)
new_tok = AutoTokenizer.from_pretrained(new_tokenizdr_name)
```

[out]:

```
>>> print(' '.join(old_tok.batch_decode(old_tok("I foo you<br>hello world")['input_ids'])))

 I foo you < br > hello world

>>> print(' '.join(new_tok.batch_decode(new_tok("I foo you<br>hello world")['input_ids'])))

  I  bar  you 
 hello  world
```

The same process above won't work for `"mistralai/Mistral-7B-v0.3"`.

But if we reinitialize with `__class__` after the `.from_pretrained` it loads the tokenizer config correctly with the extended normalizer. https://stackoverflow.com/questions/78612251/how-do-we-add-modify-the-normalizer-in-a-pretrained-huggingface-tokenizer/78624238#78624238

### Expected behavior

The same `.from_pretrained` should work for other model's tokenizers after changes to the normalizer.

Comments:
Hey! This is not because you can't change it, but because the v3 does not have a normalizer at all.
<img width="493" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/579d7a75-d48c-499c-957f-9464871e9667">
This is the `"legacy=False"` version of the tokenizer. This should be fixed soon btw, the mistralv01 should end up without a normalizer

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
flash attention support for chatglm3-6b
Author: elimsjxr
Pull request: no

Is there anyone working on a FlashAttention support for chatglm3-6b?

Comments:
Hi @elimsjxr, thanks for opening this issue! 

As the modeling code for [ChatGLM3-6B](https://huggingface.co/THUDM/chatglm3-6b/blob/main/modeling_chatglm.py) is hosted on the hub, the best place to request this feature is by opening a [discussion on the community tab](https://huggingface.co/THUDM/chatglm3-6b/discussions), directly on the checkpoint's page on the hub.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Rewrite all modeling_whisper.py ,,, is there any reason at all why it's written like that?
Author: sin2pie
Pull request: no
Labels: name: Feature request

### Feature request
I've been trying to make transformers work by editing the code with my model architecture.. Feature request.. more plasticity when it comes to unique architecture.. I want to be able to use hf and hf side hustles with my whisper-like transformer mashups.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!
Added french version of preprocessing data and corrected English version
Author: jadechoghari
Pull request: yes

### What does this PR do?
This pull request makes the following changes:
- Adds a French translation of `preparation des donnees.md` from English to French.
- Clarifies the English version of `preparation des donnees.md` for improved readability.

### Changes made
- Translated `preparation des donnees.md` from English to French.
- Revised the English version of `preparation des donnees.md` to improve clarity.

### Why is this change important?
Providing documentation in multiple languages helps make the information accessible to a broader audience and supports users who are more comfortable reading in French. 🇫🇷

### Review
Tagging:
@ArthurZucker (for the review 🇫🇷)

Comments:
ok super!

Let's run make style and should be good!
🌐 [i18n-KO] Translated `deepspeed.md` to Korean
Author: 4N3MONE
Pull request: yes

# What does this PR do?

Translated the `<your_file>.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

Comments:
> 좋은 번역 감사합니다. 문서가 길어서 번역하는데 힘드셨을 것 같아요 :) 몇 가지 문장을 다듬고, glossary 바탕으로 용어를 통일했습니다. 그리고 Files changed 목록에 toctree.yml 파일이 없어서 추가하셔야 할 것 같습니다!

@SeungAhSon 꼼꼼한 리뷰 감사합니다!! toctree파일같은 경우에는 main 브랜치와 sync change하는 과정에서 최신 파일로 적용되어서(디스코드 버전과 같음) 파일이 없는 것 같은데 혹시 착오가 있다면 알려주시면 감사하겠습니다!
fix: Updated `test_embeded_special_tokens` for luke and mluke models
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Here, the second argument is missing for `assertEqual`
https://github.com/huggingface/transformers/blob/621fb3c0edddf98f3272f3b197e772af4fa30b6c/tests/models/luke/test_tokenization_luke.py#L153-L155


Here, we are checking the same thing twice.
https://github.com/huggingface/transformers/blob/621fb3c0edddf98f3272f3b197e772af4fa30b6c/tests/models/luke/test_tokenization_luke.py#L146-L150

The tests are still passing because, here `self.tokenizers_list` is empty = [ ] for these models.
https://github.com/huggingface/transformers/blob/621fb3c0edddf98f3272f3b197e772af4fa30b6c/tests/models/luke/test_tokenization_luke.py#L138

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @LysandreJik 

Comments:
kwargs pop "attn_implement" twice in modeling_utils.py and configuration_utils.py when use AutoConfig/AutoModel
Author: Hz188
Pull request: no
Labels: name: bug

### System Info
Pop “attn_implement” twice from modeling_utils.py and configuration_utils.py causes a bug in which attn_cls selection is affected.

<img width="1524" alt="image" src="https://github.com/user-attachments/assets/75bf7b84-e64a-4d0f-908f-d8e87ee5b1d7">



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

    config = AutoConfig.from_pretrained(
        "deepseek-ai/deepseek-moe-16b-base",
        hidden_size=32,
        intermediate_size=32,
        num_hidden_layers=2,
        num_attention_heads=8,
        # vocab_size=2000,
        first_k_dense_replace=1,
        attn_implementation="flash_attention_2",
        torch_dtype="float16",
        n_routed_experts=8,
        trust_remote_code=True,
    )
    model = transformers.AutoModel.from_config(config, trust_remote_code=True)


### Expected behavior
Set the implementation of attention to flash attention instead of sdpa


Comments:
Hi @Hz188, thanks for raising this! 

Looking into it 🕵️
Respect the config's attn_implementation if set
Author: amyeroberts
Pull request: yes

# What does this PR do?

At the moment - if a user sets the attn_implementatation within a config, and then creates a model using `from_config`, the attention implementation isn't respected. 

For example, in the following, SDPA would be selected for the model: 

```py
from transformers import AutoConfig, AutoModel

checkpoint = "bert-base-uncased"
config = AutoConfig.from_pretrained(checkpoint, attn_implementation="eager")
model = AutoModel.from_config(config)
print(model.config._attn_implementation)
```

This PR:
* Ensures `attn_implementation` is respected if set in the config
* Ensures `attn_implementation` can be overridden in the `from_config` call
* Ensures previous `attn_implementation` setting in `from_config` remains if not set in config intially
* Adds test for the above

We have to use `_attn_implementation_internal` as this tells us if this has actually been set by the user. Otherwise `_attn_implementation` will return `"eager"` both if this has been specified by the user and if nothing is set

Fixes #32082


Comments:
cc @zucchini-nlp as this might relate to your current work on attn_implemenation and the configs 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32383). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Interesting, thanks for letting know! One more case to take care of :)

> If someone saved the attn_implementation in the config.json I suppose it would be propagated (config.form_pretrained(), model.from_config) but it's logical IMO!

@ArthurZucker True! I think this only happens if someone manually edits the config file though, as we don't save a config's attn_implementation out automatically. In this case, as it's pretty intentional by the user, I think it should be OK

```py
In [1]: from transformers import AutoConfig

In [2]: config = AutoConfig.from_pretrained('bert-base-uncased')

In [3]: config = AutoConfig.from_pretrained('bert-base-uncased', attn_implementation="flash_attention_2")

In [4]: config._attn_implementation
Out[4]: 'flash_attention_2'

In [5]: config.save_pretrained("/Users/amyroberts/Desktop/test_attn_config")

In [6]: config = AutoConfig.from_pretrained("/Users/amyroberts/Desktop/test_attn_config")

In [7]: config._attn_implementation
Out[7]: 'eager'
```