noamaanMulla-03/transformers-issues huggingface dataset:
Fix Trainer MLU Availability Import For Accelerate < 0.29
Author: alex-jw-brooks
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/32461

I checked Accelerate, and the utility for checking if MLU is available was added in the `0.29.0` release, so the accelerate import in the trainer breaks versions `0.21.0 >= x <0.29.0` if accelerate is available at import time (since the minimum version of accelerate transformers is happy with is 0.21.0 [in the deps table](https://github.com/huggingface/transformers/blob/main/src/transformers/dependency_versions_table.py#L6)).

It appears that the `is_mlu_available` check in the Accelerate import utils ([here](https://github.com/huggingface/accelerate/blob/v0.33.0/src/accelerate/utils/imports.py#L349)) is also only used one time and is redundant with the `is_torch_mlu_available ` check in transformers import utils [here](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/import_utils.py#L630), which is used in other parts of the trainer. 

This PR removes the accelerate util that is breaking the import, and just uses the transformers one in its place.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr and @SunMarc

Comments:
This is fixed by the changes in https://github.com/huggingface/transformers/pull/32292
`is_torchdynamo_compiling` -- cast a wide exception net
Author: gante
Pull request: yes

# What does this PR do?

Fixes `torch._dynamo.exc.InternalTorchDynamoError: module 'torch.compiler' has no attribute 'is_compiling'` with a wide exception catch.

Bonus: fix broken `make fixup` on `main` :)

Comments:
Let's fix ci and good to go 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32476). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated `mask_generation.md` to Korean
Author: jeongiin
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file_name>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `mask_generation.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조-->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
@010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32257). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated `idefics.md` to Korean
Author: boyunJang
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `idefics.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
May you please review this PR?
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
@010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32258). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: add flash_attn 2 to bert
Author: chiennv2000
Pull request: yes

Feat: Add flash attention option for BERT
Usage:
model = BertModel.from_pretrained('bert-base-uncased', torch_dtype=torch.bfloat16, use_flash_attention_2=True)

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@ArthurZucker  and @younesbelkada 


Comments:
Thanks a lot for your review and your suggestions @younesbelkada.
But I don't really familiar with ```make fix-copies``` command. Can you guide me on how to do that?


2) I appreciate your feedback. I'm happy to receive your assistance in implementing these changes. 
If you could help me with other architectures, that would be fantastic. Additionally, I'm open to collaborating on extending this to the Roberta and XLMR model. @younesbelkada 

Perfect thanks! 
As a first step, can you simply run `make fix-copies` and push the changes here? Then we'll take it over from there !

Thanks @younesbelkada , I did it

cc @younesbelkada 

Didn't had time to properly look into it, will do it asap!

Any updates on getting this PR merged?

Hello there! 

I'm working on integrating scaled_dot_product_attention to BERT #28802, and there might be some merge conflicts with this change. Mostly, if my changes go through, then we can get rid of most of the downstream dependencies from fix-copies. 

Let me know if you have any questions. Happy to discuss and/or chat on the best way forward if necessary.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This would be a lifesaver for me, I hope merging this is prioritized! cc @younesbelkada 
@chiennv2000 what kind of speedups are you observing with this?

> This would be a lifesaver for me, I hope merging this is prioritized! cc @younesbelkada @chiennv2000 what kind of speedups are you observing with this?

@loswald - you can see a quick estimate of the speedups in #28802. The pytorch SDPA implementation uses FA2 under the hood (if your hardware supports it). The PR is ready but we're just waiting on the HG team to merge it.
[feat] Apply rope_scaling for general use in phi3, llama
Author: SeongBeomLEE
Pull request: yes

# What does this PR do?

Can use 'su', 'yarn', 'longrope' in llama.

Can use "linear", "dynamic" in phi3.

However, it is not available because of the _rope_scaling_validation of each model.

So, it is combined _rope_scaling_validation of each model.

And I added position embeddings implementation.

Thanks.

## Who can review?

@ArthurZucker


Comments:
@SeongBeomLEE for reference, this is the PR that Arthur mentioned 🤗 #31999
Cache: create docs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR creates docs for cache classes and adds example usage in each classes docstring. Given that we often get issue that cache doesn't work, this docs page should help us to decrease our notifications. Partially addresses #31795, but I'll make another PR for that issue changing docs of each model

I made a quick docs, so let me know if anything was missed or needs a modification 
cc @gante @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32150). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sure, I will add a subsection for Encoder-decoder cache below with more details on how to use it and when. Ughf, how many cache classes we have added 🙃 

hehe the price of a useful feature 😎 

Okey, this is ready now. Did some final updates for the formatting and links, also addressed comments from @gante  and added a small paragraph about `cache_position` in the hidden section

I think we can add more on dialogue generation with `compile` and static cache later, prob it should go in the `llm-optims.md` file. I already asked Lewis (don't know his GH tag), who experimented with that, if there are any tricky parts we could shape with users. I can work on it in another PR

@zucchini-nlp Lewis -> @lewtun :)
Flash-Attn: fix generation when no attention mask or no pading
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes #32237. Briefly, the issue is that the PR at https://github.com/huggingface/transformers/pull/31629 added a functionality to use flash attention with position ids, which was intended for training efficiency afaik. But the condition check was not verifying that we're in train/pre-fill stage and the added code block was triggered in decoding stage. That causes shape mismatch errors in flash-attn.

This PR fixes by checking that query is not `length=1` and adds a test for that. Apparently it wasn't catched before, because our tests do not check generation with `bs=1`. 

All new tests are passing on my end. For old tests some were failing even before this current fix, but other are all green on my end

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32241). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

actually the commit was not in the release so all good


I just observed this error (similar to https://github.com/huggingface/transformers/issues/32237) with flash_attn 2.6.1 and 2.6.3 using transformers=4.44.dev0 with commit c1aa0edb48217f416f4bbe6e3a9db1500284513b

transformers-4.42.4 worked OK.
transformers-4.43.3 worked OK too

@ArthurZucker 

@poedator hey! Would be nice to get a reproducer to see if it's related to the issue from current PR :)

> Would be nice to get a reproducer to see if it's related to the issue from current PR :)

Hi, @zucchini-nlp !

Here it is: https://gist.github.com/poedator/602a881353b8105f3618902f2f827fb2

The error happens only in subsequent model call with reusing cache.
No problems with SDPA or without cache or with release versions up to 4.43.3 [CORRECTION: 4.43.4 OK too]

`4.43.4` should be alright as well as it only includes #32214

@poedator thanks! Seems like there was an extra `not` while checking for position ids. Will open a PR to fix it :)
Gemma2: add cache warning
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31981 by adding an informative warning and falling back to DynamicCache if one calls `forward()` directly. Actually DynamicCache will work same as Hybrid, as long as the text sequence stays below sliding window length (i guess 4k). So it should be enough for basic use-cases as in the issue, when a user wants cache for one call only.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32279). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> we have to initialize any cache as long as the person is simply running a forward

So we have to raise error if `use_cache=True` asking to initialize `HybridCache` or set caching to `False`? Can do that, and better with a link to docs. From issue seems like users don't know how to init these new cache classes. 

I made it this way following general trend in transformers, that cache is init by us if user doesn't set it. Is it okay we're being a bit inconsistent with caches?

Oke, raising error when use_cache=True but no Cache object is passed is a bad idea, that will break most of forward calls because people do not always set `cache=False` and `self.training` sometimes is not set to False. If we don't want to magically set default cache, we can raise warning and set `use_cache=False`. WDYT?
Agents use grammar
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Add grammars to transformers agents, in order to remove parsing errors, thus drastically improving agent behavior.

Roadblocks: 
- [x] Integrating a grammar handling in TGI, solved [here](https://github.com/huggingface/text-generation-inference/pull/2046) by @drbh 
- [x] Integrating this parameter in `InferenceClient`, still ongoing. (cc @Wauplin)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31735). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Integrating this parameter in InferenceClient, still ongoing. (cc @Wauplin)

Officially supported in `huggingface_hub` 0.24.0: 
- [`grammar` in `text_generation`](https://huggingface.co/docs/huggingface_hub/v0.24.0/en/package_reference/inference_client#huggingface_hub.InferenceClient.text_generation)
- [`response_format` in `chat_completion`](https://huggingface.co/docs/huggingface_hub/v0.24.0/en/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion)
fix broken link in docs
Author: jorahn
Pull request: yes

# What does this PR do?

fix broken link in docs:

in
`https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline.__call__`
under
`generate_kwargs (dict, optional) — Additional keyword arguments to pass along to the generate method of the model (see the generate method corresponding to your framework here).`

link [here](https://huggingface.co/docs/transformers/en/main_classes/main_classes/text_generation) doesnt work, specifies a directory twice

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- text models: @ArthurZucker
- pipelines: @Narsil
- documentation: @stevhliu


Comments:
🌐 [i18n-KO] Translated `gptq.md` to Korean
Author: 1kmmk1
Pull request: yes

<!-- PR의 제목은 "" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `gptq.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
@win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR? 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32293). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated `quantization/quanto.md` to Korean
Author: fabxoe
Pull request: yes

# What does this PR do?

Translated the `quantization/quanto.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
@win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32281). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.