noamaanMulla-03/transformers-issues huggingface dataset:
when using FSDP, Trainer initializes the optimizer in a wrong order that results in huge waste of GPU memory
Author: yundai424
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.111.1-rolling-lts-linkedin-x86_64-with-glibc2.17
- Python version: 3.10.2
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0a0+gitf998869 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: 8 A100 GPUs
- Using distributed or parallel set-up in script?: Nah just `torchrun`


### Who can help?

trainer: @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

**Summary**:
When using FSDP (non-XLA), it'll first create the optimizer using the original full model (without being wrapped into FSDP) [here](https://github.com/huggingface/transformers/blob/v4.31.0/src/transformers/trainer.py#L1647), and then it wraps the model w/ FSDP [here](https://github.com/huggingface/transformers/blob/v4.31.0/src/transformers/trainer.py#L1656)... This results in a **huge waste of GPU memory as there'll be a ton of dangling non-useful optimizer states associated with the model params that don't belong to this shard**; While it should instead first create model and wrap it with FSDP, and initialize optimizer from the FSDP-wrapped model.

I noticed this issue when finetuning llama2-7b model using alpaca dataset. Here is the colab that has the code example for reproduction (The colab can't be executed directly out of the box; Just a way to put all the code into a sharable link): https://colab.research.google.com/drive/1Rj9jDuOjmnZUiEUZwEmLHoL03pGLqv7V#scrollTo=eFfSsG756poB

Another indication is that you'll see a warning thrown by `accelerate`: 
>FSDP Warning: When using FSDP, it is efficient and recommended to call prepare for the model before creating the optimizer

And `accelerate` also notes down this caveat and suggests the same thing https://huggingface.co/docs/accelerate/usage_guides/fsdp

### Expected behavior

It should instead first create model, and then wrap it with FSDP, and finally initialize optimizer from the FSDP-wrapped model. I modified the source code of `transformers` a bit to make it work as expected (don't initialize optimizer until model is wrapped  by `accelerate` with FSDP), I notice a reduction in GPU memory usage by **25Gi** even before any training step begins.

Comments:
CC @pacman100 

This has already been fixed. Please use the main branch of Transformers.

thanks @pacman100 i found the PR that fixed this. Closing this issue

I'm curious which PR fixed this and how was it fixed?
T5 Attention fix and tests
Author: gardberg
Pull request: yes

# What does this PR do?

Fixes a bug which crashes the forward pass of T5Attention when not using the KV Cache or the `cache_position` parameter, and adds tests to verify the fix.

- [x] Add test
- [ ] Fix bug in T5Attention
- [ ] Linting & code quality

Fixes # (issue)
https://github.com/huggingface/transformers/issues/34448

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@zucchini-nlp 


Comments:
Enable Gradient Accumulation fix across all models + trainer fully in forward()
Author: muellerzr
Pull request: yes
Labels: name: Core: Modeling

name: trainer

# What does this PR do?

Since most users still want OOTB, this trickles the loss kwargs to the rest of the models so that causal loss can be calculated properly 

Fixes # (issue)

Fully fixes https://github.com/huggingface/transformers/issues/34263 / finishes https://github.com/huggingface/transformers/pull/34191 & https://github.com/huggingface/transformers/pull/34198 & fixes https://github.com/huggingface/transformers/issues/34242


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 


Comments:
@ArthurZucker `LlamaForCausalLM` isn't there bc you did it in your PR 😉

Awesome work by all of you on this. Insane dev speed over the past few days 🙏 🔥 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Rebasing should probably fix quality! 

I wonder whether encoder models need to be fixed as well. @ArthurZucker  @muellerzr 

Yep! Fill free to open a PR 🤗
Fix  #34494 assistant tokens when truncated
Author: yonigottesman
Pull request: yes
Labels: name: bug

name: Chat Template

This pr fixes a bug that caused  #34494.
when using `apply_chat_template` with `return_assistant_tokens_mask` and token truncation, the assistant mask was not correct.

@Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34531). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

rebased and ready for merge...

Merged!
fix(dvclive): pass fake dataset to avoid exception in trainer init
Author: shcheklein
Pull request: yes

Fixes these failing tests https://github.com/iterative/dvclive/actions/runs/11545815176/job/32134865977?pr=846 downstream.

The cause is this change: https://github.com/huggingface/transformers/pull/33743 which made `trainer` ctr a bit more strict.

I see that `WandbCallback` is also using the same technique (initializing fake trainer). @parambharat could you check if we need to add a similar fix there.

Comments:
cc @SunMarc @jackyjinjing for the original PR!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34455). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc done, rebased
update clean_up_tokenization_spaces warning
Author: itazap
Pull request: yes

ref #31884 

start of deprecating `clean_up_tokenization_spaces`. Right now it defaults to True, will be updated in the future to default to False.

future pr: https://github.com/huggingface/transformers/pull/31938

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32371). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

clean_up_tokenization_spaces
🌐 [i18n-KO] Translated `timesformer.md` to Korean
Author: mreraser
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `timesformer.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
May you please review this PR?
@yijun-lee, @mreraser, @jungnerd, @cjfghk5697
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

Hello @stevhliu !  May you please review this PR? Thank you! 
Have a nice day :)

Comments:
Hello! @yijun-lee, @jungnerd, @cjfghk5697
May I kindly ask for your review? Thank you for your consistent reviews 🤗 


LGTM!

cc @stehvliu!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33972). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ReactAgent's run method calls a non-existing step method
Author: sarodz
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.12.1
- Huggingface_hub version: 0.26.0
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.0+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker @gante

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I was following this post from the [website](https://huggingface.co/blog/agents)
```python
from transformers.agents import HfApiEngine, ReactAgent

llm_engine = HfApiEngine("meta-llama/Meta-Llama-3-70B-Instruct")
agent = ReactAgent(
    tools=[],
    llm_engine=llm_engine
)
agent.run("Some Question")
```

### Expected behavior

The code should run without any issues, it currently throws the following message:
```
AttributeError: 'ReactAgent' object has no attribute 'step'
```

Comments:
Need to call `ReactCodeAgent` or `ReactJsonAgent` rather than `ReactAgent`
SPLIT PR: eos bos tokens
Author: itazap
Pull request: yes

Fix for 2 issues:

1. `add_bos_token` & `add_eos_token` flags ignored for `PreTrainedTokenizerFast`: issue discussed [here](https://huggingface.co/meta-llama/Meta-Llama-3-8B/discussions/140) and [here](https://github.com/huggingface/transformers/issues/30947#issuecomment-2128057992)
2. `add_special_tokens` does not update `bos_token` or `eos_token` - ex `.add_special_tokens({'bos_token': '<new_bos>'})`

TASKS:

- [x] added an `update_post_processor` function in `PreTrainedTokenizerFast` based on llamatokenizer, allows reading of bos / eos token flag

**SUPPORTS FAST ONLY 
slow required updating kwargs to be passed into `sp_model` , so that bos/eos tokens can be added accordingly..

Reviewer: @ArthurZucker 

NOTE: hub token seems to not have access to llama 3, should pass after addressed

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31316). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated bert.md to Korean 
Author: maximizemaxwell
Pull request: yes

# What does this PR do?

Translated the bert.md file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179
## Before reviewing

  - [x]  Check for missing / redundant translations (번역 누락/중복 검사)
  - [x] Grammar Check (맞춤법 검사)
  - [x] Review or Add new terms to glossary (용어 확인 및 추가)
  - [x]  Check Inline TOC (e.g. [[lowercased-header]])
  - [ ]  Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

@junejae, @Jwaminju, @010kim, @4N3MONE, @boyunJang
## Before submitting

   - [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
   - [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
    Pull Request section?
   - [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
    to it if that's the case.
   - [x]  Did you make sure to update the documentation with your changes? Here are the
    [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
    [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
   - [ ]  Did you write any new necessary tests?

## Who can review? (Final)

@stevhliu May you please review this PR?

Comments:
사소한 수정사항 제안 남겨두었습니다.

번역 고생하셨습니다 😊

Thanks for suggestions!!
Those are updated now.
Any other things to change?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34627). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add EXAONE
Author: lgai-exaone
Pull request: no
Labels: name: New model

### Model description

EXAONE is a large language model developed by LG AI Research. We released [EXAONE 3.0](https://github.com/LG-AI-EXAONE/EXAONE-3.0) in August, but we've since updated the model code and are working on integrating our implementation with the Huggingface transformers library.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

- Model URL : https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct
- GitHub repo for EXAONE : https://github.com/LG-AI-EXAONE/EXAONE-3.0

Comments:
PR link : https://github.com/huggingface/transformers/pull/34652
Update utils.py
Author: andrewqianpublic
Pull request: yes

Check for low memory availability and clearing unused memory to make more space available.

# What does this PR do?

<!--
I have made a small change to the generate function generation/utils.py. It is meant to check for low memory on the users GPU devices. If the available memory is below 1 GB, unused cache memory will then be cleared. This causes a little bit of an overhead, but given that this will only be called at the end of generation and only when memory is low, this shouldn't impact performance. 
-->


Fixes # (issue)
- Memory out of error due to unused allocated cache space taking up GPU memory. 


## Who can review?
@gante


Comments:
Reorganizing
Clear unused allocated memory in GPU 
Author: andrewqianpublic
Pull request: yes

# What does this PR do?

<!--
This update checks if the user has 1 gigabyte or less of vram in their gpu. Then it clears the cache for unused memory at the end of the generation. That is, this change basically checks for critically low memory available after generation is done, and clears the memory to avoid out of memory error. This was solution that worked for my personal chatbot implementation using the transformers library. 
-->

<!-- Remove if not applicable -->

Fixes # (issue)

- Frees up unused memory in gpu.


## Who can review?
@gante 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @


Comments:
[whisper] transcription is different from hf & openai
Author: ghost
Pull request: no
Labels: name: bug

name: Audio

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

[silence-middle.wav.zip](https://github.com/user-attachments/files/16676870/silence-middle.wav.zip)
the following wav file produces 
- with hf whisper repo (tiny model):
Split infinity, and a time when less is more. Where too much is never 
- with openai whisper repo (tiny model)
Split infinity, and a time when less is more.

### Expected behavior

I would expect the result to be 
Split infinity, and a time when less is more. Where too much is never 

Comments:
you are missing the re encoding mechanism happening when eos is reached within a 30s segment

cc @ylacombe @sanchit-gandhi 

Hey @jsoto-gladia, many thanks for opening this issue!

This looks like an interesting finding, do you think you could provide code snippets (both in transformers and in the whisper repo) to allow us to reproduce it ?


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @jsoto-gladia, I believe this PR addresses your problem: https://github.com/huggingface/transformers/pull/33917

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[i18n-KO] Translated backbones.md to Korean
Author: jun048098
Pull request: yes

<!— PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 —>
# What does this PR do?

Translated the `backbones.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? --> 
<!-- @harheem, @jeongiin, @1kmmk1, @Seungahson, @win2dvp21 --> 
<!-- @junejae, @Jwaminju, @010kim, @4N3MONE, @boyunJang --> 
<!-- @jungnerd, @cjfghk5697, @yijun-lee, @mreraser --> 
@heuristicwave, @nuatmochoi, @chhaewxn, @jun048098, @ahnjj, @fabxoe

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
<!-- @stevhliu May you please review this PR? -->

Comments:
Auto Classes docs _toctree
Author: jun048098
Pull request: no

Is the path to [Auto Classes](https://huggingface.co/docs/transformers/v4.46.2/en/model_doc/auto) docs appropriate?

It is written as ```model_doc/auto``` in [line 275 of _toctree.yml](https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/docs/source/en/_toctree.yml#L275). 

<https://huggingface.co/docs/transformers/v4.46.2/en/model_doc/auto>

Because of this, the end of the url is ```model_doc/auto```. However, in the left list of [docs](https://huggingface.co/docs/transformers/v4.46.2/en/model_doc/auto), it belongs to ```MAIN CLASSES```.

Should it belong to ```main_classes``` like other docs?

Or is there a special reason why the path to _toctree is ```model_doc/auto```?

If it's an error, can I change that line in _toctree to ```main_classes/auto```?

Please check @stevhliu

Thank You!

Comments:
Yeah, this is correct. When the docs were reorganized, the `AutoClass` was moved under the Main classes section from the Models section because it included other objects (configuration, processors, tokenizers, etc.) that weren't just models. To avoid breaking links in the docs, we just moved `model_doc/auto` under Main classes instead of renaming it.

I understand. I will close this issue without making any changes. Thank you for checking the issue!