noamaanMulla-03/transformers-issues huggingface dataset:
Rename torch.run to torchrun
Author: steven-safeai
Pull request: yes

torch.run does not exist anywhere as far as I can tell.

# What does this PR do?

Updates documentation.

Fixes # (issue)


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30405). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix use_cache for xla fsdp
Author: alanwaketan
Pull request: yes

# What does this PR do?

use_cache cannot be used with gradient checkpointing. In PyTorch/XLA, we have to rely on our own gradient checkpointing function instead of the upstream one. Somehow, transformers regress and couldn't recognize our gradient checkpointing anymore. This PR fixes it.

Fixes #30155


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @amyeroberts 


Comments:
I don't know how to fix the issue with the use_cache parameter in the modeling code. Is that widely used?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30353). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, @amyeroberts. In most of the modeling code, I saw this use_cache is passed as a parameter:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L949

And then there is this check happened in the modeling code:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L967-L971

Since we currently still cannot directly use the upstream grad ckpt, we cannot relay on the gradient_checkpointing flag and then reuse this logic. And I also don't want to add a new flag and modify all modeling codes. lol

Thanks, @muellerzr. Can someone help me merge this?

I can merge. Thanks again for fixing this! 

Thanks, @amyeroberts
Add beam search visualizer to the doc
Author: aymeric-roucher
Pull request: yes

# What does this PR do?

Add this [visualization tool for beam-search](https://huggingface.co/spaces/m-ric/beam_search_visualizer) to the documentation.

## Who can review?
@gante 


Comments:
@gante I've added a reference were you pointed, and a shorter one in the intro where beam-search was described. If you prefer we can remove the shortest.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29876). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks a lot @ArthurZucker ! PR to add an image to the doc dataset open here: https://huggingface.co/datasets/huggingface/documentation-images/discussions/305

@gante I just amended the commit to include the image!

It's ok to merge on my side @gante !
Make EosTokenCriteria compatible with mps
Author: pcuenca
Pull request: yes

# What does this PR do?

EOS termination was moved to stopped criteria in #29459. It uses `torch.isin()`, which is not compatible with the `mps` device: https://github.com/pytorch/pytorch/issues/77764#issuecomment-2067838075. This PR uses the old method to detect whether any of the EOS tokens were found in the generated sequences.

Fixes https://github.com/huggingface/transformers/pull/29459#issuecomment-2066324318


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30376). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, good point, I thought about that too. I opted to keep both because the new method is much more readable and intuitive, so it doesn't really add too much noise in my opinion. It also documents why we are using the long approach, and provides a hint (via the link to the github issue) to remove that case when we have full support of `isin()` for `mps` at some point in the future. But sure, we can keep just one if that's preferred :) 

I'm pro the if/else: it serves as a reminder to revisit the code in the future, when `mps` supports this more readable code path 

I'm not sure we should be doing this - the lack of support is really a torch issue, and not a transformers one. There's many operations which are (unfortunately) unsupported by the mps backend, and I wouldn't want to see a tonne of if/else statements across the library to support this. At the moment, `PYTORCH_ENABLE_MPS_FALLBACK=1` is the standard approach to try and use mps, which I believe resolves this issue  

I don't really mind this as it should be mostly for `Llama3`, but only if that produces the correct outputs. If not, then we should just recommend to use mlx? 
@pcuenca does this produce gibberish or same as on cuda? 

Thanks so much for the quick fix. Could this get a release?

Yes, will do this asap, just waiting to make sure we don't have other failures.
[`LlamaTokenizerFast`] Refactor default llama
Author: ArthurZucker
Pull request: yes

# What does this PR do?
```python
from transformers import LlamaTokenizerFast, AddedToken
tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", legacy=False, from_slow=True)
tokenizer.add_tokens([AddedToken("<REPR_END>", rstrip=True, lstrip=True)], special_tokens=False)
tokenizer.tokenize("<REPR_END>inform
. Hey.       .")
['<REPR_END>', 'in', 'form', '
', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']
tokenizer.tokenize("inform
. Hey.       .")
['in', 'form', '
', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']
```

This requires https://github.com/huggingface/tokenizers/pull/1476 to be merged. 
Finally FIXES ALL THE remaining tokenizer issues!

Fixes #29617 as well (will update Gemma one)
fixes #28577
fixes #29617
fixes #29626
fixes #29694
fixes #29868
fixes #29872
fixes #30416
enabled by https://github.com/huggingface/tokenizers/pull/1476

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28881). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Coming next release!
Inconsistent behavior between tokenizer and fast tokenizer
Author: xuzhenqi
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-4.18.0-193.6.3.el8_2.v1.4.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

``` python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf", trust_remote_code=True, use_fast=False)
fast_tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf", trust_remote_code=True, use_fast=True)
prompt = "▁<PRE>//"
inputs = tokenizer(prompt, return_tensors="pt")
print(f"tokenizer ids: {inputs.input_ids}")
inputs = fast_tokenizer(prompt, return_tensors="pt")
print(f"fast tokenizer ids: {inputs.input_ids}")
```

This scripts will output:
```
tokenizer ids: tensor([[    1, 32007,   458]])
fast tokenizer ids: tensor([[    1, 32007,   849]])
```

In the `tokenizer.json` from the model folder, we can see:
```
"//": 458,
"▁//": 849,
```

Fast tokenizer probably ignores the `<PRE>` token, is it a correct behavior?

### Expected behavior

Fast tokenizer should be consistent with normal tokenizer.

Comments:
Thanks for reporting! This is pretty much a known bug but will be fixed by the likes of #26678 (when propagated to Llama)

#28881 will fix this issue!
Inconsistent tokenization between fast and slow tokenizers for sentencepiece user-defined tokens.
Author: TissueC
Pull request: no

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0a0+fe05266 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Fast tokenizers cannot view the added token as a whole but split it as many small parts.
I found the similar issue in the huggingface discussion (but with no solution) : https://discuss.huggingface.co/t/sentencepiece-user-defined-symbols-and-fast-tokenizers/52208
Here is a script to reproduce this issue:

```python
from transformers import AutoTokenizer
import transformers.utils.sentencepiece_model_pb2 as sp_model
import json

proto = sp_model.ModelProto()

llama_orgin_tokenizer_path = 'Llama-2-7b-hf' # this a local path
llama_tokenizer_with_added_token_path = 'Llama-2-7b-hf-with-added-token'

with open(f'{llama_orgin_tokenizer_path}/tokenizer.model', 'rb') as fin:
    proto_str = fin.read()
    proto.ParseFromString(proto_str)

# append a user-defined token
adding_token = 'aaabbbcc'
proto.pieces.append(sp_model.ModelProto.SentencePiece(piece=adding_token, score=0.0, type=4)) #type 4 means user defined
with open(f'{llama_tokenizer_with_added_token_path}/tokenizer.model', 'wb') as fout:
    fout.write(proto.SerializeToString())


with open(f'{llama_tokenizer_with_added_token_path}/tokenizer_config.json','w', encoding='utf-8') as fout:
    tokenizer_config = {
        "model_max_length":99999999999999,
        "tokenizer_class": "LlamaTokenizer"
    }
    json.dump(tokenizer_config, fout)



tokenizer_fast = AutoTokenizer.from_pretrained(llama_tokenizer_with_added_token_path, use_fast=True)
tokenizer_slow = AutoTokenizer.from_pretrained(llama_tokenizer_with_added_token_path, use_fast=False)

text = 'I love aaabbbcc.'
print('fast:', tokenizer_fast.tokenize(text))
print('slow:', tokenizer_slow.tokenize(text))

# The outputs:
# fast: ['▁I', '▁love', '▁aa', 'ab', 'bb', 'cc', '.']
# slow: ['▁I', '▁love', '▁', 'aaabbbcc', '.']
```

### Expected behavior


It should be consistent tokenization results between fast and slow tokenizers for **those added tokens.**

Comments:
Hey this is a duplicate of https #28577 and will be fixed by #28881
Gemma's Tokenizer fails to split on spaces 
Author: vasqu
Pull request: no
Labels: name: Core: Tokenization

### Preface

This is related to #29617, similar issue as described in there so this is kind of a placeholder. Should be fixed by #28881.

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('google/gemma-7b', use_auth=True)
sentence = 'Sampel to demonstrate the issue'

input_encoding = tokenizer(sentence, add_special_tokens=True)
tokens = input_encoding.tokens()
word_ids = input_encoding.word_ids()

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')
```

### Expected behavior

Correct split(s) assigned.

Comments:
Remove task guides auto-update in favor of links towards task pages
Author: LysandreJik
Pull request: yes

This aims to further reduce the number of files that should be changed on each model contribution.

cc @ArthurZucker @amyeroberts

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30429). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @amyeroberts!
Remove add-new-model in favor of add-new-model-like
Author: LysandreJik
Pull request: yes

This PR removes the very outdated `add_new_model` script (that wasn't working correctly anymore) in favor of the `add_new_model_like` script that is favored.

@amyeroberts @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30424). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks both!
Remove mentions of models in the READMEs and link to the documentation page in which they are featured.
Author: LysandreJik
Pull request: yes

This aims to reduce the number of files that should be changed on each model contribution.

cc @ArthurZucker @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30420). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

will remove the tasks in another PR as it's something different; but thanks for the heads-up! 

cc @stevhliu FYI

Thanks @amyeroberts!
Badam support
Author: Theodotus1243
Pull request: no
Labels: name: Feature request

name: optimization

### Feature request

https://arxiv.org/pdf/2404.02827.pdf

Memory efficient optimizer like galore but with less hyperparameters 

### Motivation

https://github.com/hiyouga/LLaMA-Factory/pull/3287

LlamaFactory supports it, but having it in transformers is much more convenient

### Your contribution

-

Comments:
cc @younesbelkada 

@Theodotus1243 @amyeroberts @younesbelkada 

Hi, thanks so much for acknowledging our BAdam optimizer (https://github.com/Ledzy/BAdam). Here's a brief overview of its features:

1. **_Memory Efficiency:_** It is memory efficient **full parameter** finetuning method. We tune **Llama 2-8B and Llama 3-8B using a single RTX3090** with BAdam; see our Github page https://github.com/Ledzy/BAdam for detailed performance metrics.

2. **_Simplicity in Hyperparameters:_** BAdam introduces **only one** additional hyperparameter, and it can be **adaptively set**; see the "Hyperparameter Suggestion" section in our Github page: https://github.com/Ledzy/BAdam#hyperparameter-suggestion.

3. **_Time Efficiency:_** Compared to LoRA and traditional Adam, BAdam **cuts the actual backward time by half** after the same number of epochs, thanks to the Chain rule property.   

4. **_Rapid Convergence:_** The algorithm converges very fast, and we observe that only epoch is often enough for instruction tuning (e.g., tuning llama 3-8B using Alpaca-GPT4 dataset).

So far, [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/extras/badam/sft.sh) has integrated our method. We believe BAdam offers significant advancements for the LLMs community, and we are eager to support its integration into the Hugging Face Transformers library. Should you find these features compelling, we would be delighted to assist in its implementation.



Hey @Theodotus1243, sorry for the very slow reply, we're catching up on a big backlog.

BAdam is a neat method (nice NeurIPS'24 paper), but we're going to pass on a native integration for now: this is the only issue requesting it, and after two years its adoption hasn't really spread beyond LLaMA-Factory. It's also not a drop-in optimizer like GaLore/APOLLO: BAdam's block-coordinate scheme toggles requires_grad per block inside the training loop and has known FSDP/DeepSpeed-ZeRO3 caveats, so it doesn't fit our optimizer plug-in path cleanly, and every new `--optim` we add is a permanent backward-compat commitment.

The good news is you don't need us to use it today: the standalone `badam` package (`from badam import BlockOptimizer`) works with any HF model, and LLaMA-Factory already wires it into the Trainer.
New model PR needs green (slow tests) CI
Author: ydshieh
Pull request: yes

# What does this PR do?

For a new model PR, let's trigger and run the slow tests of that model.

Question: if we want to trigger the slow tests for each commit in a PR, or we prefer to trigger it toward the end of the PR (i.e. almost ready to be merged to main)? In the later case, we can use the commit message (i.e. having a prefix like `[run_slow]`). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30341). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

~~Convert to draft to check if I can using some github action events to control the CI trigger more precisely.~~

Hi @amyeroberts @ArthurZucker 

I think we can use [issue_comment](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#issue_comment) and `github.event.comment.body` to trigger the CI with a comment on the pull request.

However, `issue_comment` event (this contain the pull request comment, and can be configured to pull-only comment) will only trigger a workflow run if the workflow file is on the default branch, so I am not able to test it while this PR is still not merged to `main` yet.

I will have to test it in my own repo. first.
[`_unmask_unattended`] Refactor
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Follow up on #29318, if we keep the `AttentionMaskConverter` then let's put the checks in the `_unmask_unattended` since this function is bound to disappear once the bug is fixed in torch

@fxmarty let's make this easier to refactor later on, and more readable!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Llava`] + CIs fix red cis and llava integration tests
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Now fast and slow tokenizers really match 🥳 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30440). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I ran the slow tests for affected models, let me try with T5 which should be the only one left that has "legacy" parameter. t5 all good as well
Add messages to agents (#5)
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Convert agents to chat format.

Thank you @cyril-k!

cc @LysandreJik @Jofthomas 

Comments:
New version in https://github.com/huggingface/transformers/pull/30385

Let's merge this one in Agents
Add message passing format
Author: LysandreJik
Pull request: yes
Labels: name: Agents

Updated & rebased PR of https://github.com/huggingface/transformers/pull/30337

Comments:
@LysandreJik this PR also needs to take [this commit](https://github.com/huggingface/transformers/pull/30337/commits/881614b5bd04157e76e3f7d2dd1d9f5f4fbaddc1).

closed in favor of https://github.com/huggingface/transformers/pull/30337
[tests] make test device-agnostic
Author: faaany
Pull request: yes

## What does this PR do?
Make `test_from_pretrained_disk_offload_derived_to_base_model` and `test_from_pretrained_disk_offload_task_model` tests work not only on GPU device, but also other accelerator devices. 

Results on my XPU device: 
```bash
========================================================== PASSES ==========================================================
__________________________ ModelUtilsTest.test_from_pretrained_disk_offload_derived_to_base_model __________________________
---------------------------------------------------- Captured log call -----------------------------------------------------
WARNING  root:big_modeling.py:435 Some parameters are on the meta device device because they were offloaded to the disk and cpu.
WARNING  root:big_modeling.py:435 Some parameters are on the meta device device because they were offloaded to the disk and cpu.
================================================= short test summary info ==================================================
PASSED tests/test_modeling_utils.py::ModelUtilsTest::test_from_pretrained_disk_offload_derived_to_base_model
============================================== 1 passed, 3 warnings in 1.46s ===============================================
```

@amyeroberts and @ArthurZucker 


Comments: