noamaanMulla-03/transformers-issues huggingface dataset:
Deprecate legacy cache + use cache position
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR deprecates legacy cache in all models that currently support Cache class. Also, these models now rely on cache position and update-causal-mask to get 4d attention mask. Tests are passing on my end, will activate slow tests with commit msg on PR later



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31491). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I think in the futur we will want to use generation_config.cache_config.ma_length maybe instead of calling get_max_length() 

Yes, sounds good if we can start adopting cache config for all cache related arguments
FEAT: Add Badam optimizer
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/30308

This PR adds Badam optimizer to transformers Trainer API 

## TODOs:

- [ ] Add ratio optimizer
- [x] Add layer optimizer
- [ ] Add docs

cc @amyeroberts @muellerzr 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30692). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thank you for your efforts in integrating our work! 

BAdam supports model parallel offered by Deepspeed ZeRO-3 now. Integrating the latest feature only requires 
* set `ds_zero3_enabled=True` when initializing the `BlockOptimizer`
* add a customized callback: `BAdamCallback`

Please see our implementation [here](https://github.com/Ledzy/BAdam). It enable doing full parameter finetune on a 70B model with only 8 RTX-3090 GPUs or 3 A100 GPUs. In terms of time and computational efficiency, BAdam saves half backward FLOPs in average and thereby requires less training time compared with some existing PEFT approaches, e.g. LoRA.

The current ratio-based implementation is not well optimized and requires computing the full gradient of the model for each backward (it sparsifies the gradient on the fly). We left the implementation-level optimization as a future work, and suggest not including the ratio-based implementation for now.

Should you encounter any issue when integrating BAdam, please feel free to let us know. We are more than happy to assist in the integration, and are willing to change the source code if needed. @younesbelkada @amyeroberts @muellerzr
Getting Loss : nan while fine-tuning blip2-opt-2.7b
Author: tan7vir
Pull request: no

I am using this code to fine-tuning model but get nan loss value and the model give strange output. 

**My Fine-tuning code.**
![image](https://github.com/huggingface/transformers/assets/117768126/d1020d75-6f51-4241-abd9-0d927aee3b56)
And getting output like this:
![image](https://github.com/huggingface/transformers/assets/117768126/0c8d6ad7-5f03-425d-b775-3f6ecb0e8c1c)


Comments:
Hi @tan7vir, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

@amyeroberts  Thank You.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
SPLIT PR:  add user defined symbols and control symbols
Author: itazap
Pull request: yes

Fixes portion of #30824 -> 

- [x] adds user_defined_symbols and control_symbols from proto so that they are not split during encoding / decoding. 
- [x] tests gemma
- [x] removed same logic from GemmaConverter as it is handled in general SpmConverter class
- [x] updates common test (!!!) bc fast != slow since fast an read from SPM converter and get `user_defined_symbols` and `control_symbols`. @ArthurZucker thoughts on this?
- [x] copied test from common to camember and rembert tests 
- [x] updated deberta v2 tests to not use `'.' ` in test cases' texts because it is a user added token for fast tokenizers, so the spacing around it differs from slow.
- [x] `add_special_tokens` and `add_tokens` has same effect for now - but fix is already merged in `Tokenizers` by @ArthurZucker  https://github.com/huggingface/tokenizers/pull/1521#:~:text=ArthurZucker%20merged%20commit-,f2ec3b2,-into

TODO in new PR:
- [ ] other PR for prefix space (mentioned in #30824) -> PR open: #31315 
- [ ] open issue to inspect other attrs of proto to see if they can be added in conversion 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31305). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[roberta] fix lm_head.decoder.weight ignore_key handling
Author: stas00
Pull request: yes

This PR fixes https://github.com/huggingface/transformers/issues/12426 where `RobertaForMaskedLM` and `RobertaForCausalLM` do:

```
lm_head.decoder.weight = embeddings.word_embeddings.weight
```
and thus `lm_head.decoder.weight` shouldn't be saved or expected to be loaded unless `config.tie_word_embeddings` is `False`.

So this PR:
- adds `_keys_to_ignore_on_save = [r"lm_head.decoder.weight", r"lm_head.decoder.bias"]` - note also added `lm_head.decoder.bias`
- adds `lm_head.decoder.weight` to `_keys_to_ignore_on_load_missing`
- adds a test. For now "private" but may become common down the road if many other models need this feature.
- tweaks `test_save_load_keys_to_ignore_on_save` to be more debug-friendly and not dump the whole `state_dict` - ouch.

Fixes: https://github.com/huggingface/transformers/issues/12426

As discussed in the Issue other models may have the same issue. So let's follow this pattern of making a model "model" and then ask the community to help identify and fix other models.

@LysandreJik 


Comments:
When I tried to resume training from a checkpoint, it failed due to missing key(s) in state_dict: `"lm_head.decoder.weight", "lm_head.decoder.bias"`. I suppose it is because of this? Do you guys know how to fix it? I'm running `run_mlm_no_trainer.py`

@binhna Could you open a new issue, and include a reproducible code snippet, information about the running environment ( run `transformers-cli env` in the terminal and copy-paste the output) and all relevant information about the error including expected and observed behaviour and full error traceback? This helps us keep track of issues and their resolutions
Removed torch.cuda.empty_cache from train loop.
Author: FoamoftheSea
Pull request: yes

# What does this PR do?

Removes the addition of torch.cuda.empty_cache from the training loop (introduced in https://github.com/huggingface/transformers/pull/28769).

This line caused training slowdowns observed in issue https://github.com/huggingface/transformers/issues/31372

While [this thread](https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/27) in the PyTorch forums recommends not to use this function because it is slow, it appears many in the comments there still find it necessary to save them from OOMs on their training jobs, so it might be nice to have the option, but users can just add it on their own if they're in a jam.

Fixes [# 31372](https://github.com/huggingface/transformers/issues/31372)

@muellerz @SunMarc 


Comments:
Does it make sense if its added as a `TrainingArgument` and default to `False`, but with a tip to turn it on if vram usage is near the limit? It is useful because many OOMs only happen after X unpredictable steps and many don't watch them all the way before going off

This may also cause some behaviour changes where hyperparams/models working previously OOMs after the change

@aliencaocao I'm not sure we necessarily want to actively monitor the memory and trigger a tip (I suspect this is more fiddly and flaky than expected as you have to balance catching in time vs not spamming, making sure values are correct etc.). 

A flag which we can configure for clearing after every n-steps seems reasonable. Would you like to open a PR with a proposal and we can iterate from there? 

sure i can do it
Failed to Download GPT2-large Model from Hub
Author: daskol
Pull request: no

### System Info

py312, transformers 4.40.2, huggingface-hub 0.23.0.

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained('gpt2-large')
...
OSError: gpt2-large does not appear to have a file named config.json. Checkout 'https://huggingface.co/gpt2-large/tree/None' for available files.
```

It seems that `tranformers` failed to resolve branch and tried to find `config.json` in non-existing branch `None`.

### Expected behavior

The script above has to download and load model.

Comments:
Hi @daskol 
Thanks for the issue ! Can you try with `openai-community/gpt2-large` instead ? We recently migrated the canonical models into dedicated organisations. It is surprising as automatic redirection should work, I just ran your snippet in my local env and it worked fine

Yes, `openai-community/gpt2-large` works for me as well. I have noticed that `gpt2*` repos were moved and populate them manually from `openai-community/gpt2*`. I think the issue exists in case if there is nothing cached.

Thanks @daskol ! 
Hmmm, that's strange, I tried 

```bash
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("gpt2-large", force_download=True)
```

It seems the model is being downloaded correctly, `force_download=True` makes sure it re-downloads the model even if it is present in the cache

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add time-based save, log, eval
Author: BramVanroy
Pull request: yes

As discussed in https://github.com/huggingface/transformers/issues/29984, this feature PR adds the ability to add an additional "security" for logging, evaluating, and saving. (The last two being the most useful.) It is motivated by HPC environments where a job has a given time (e.g. 72 hours), so just to be sure you may want to save your progress at the 70hr mark. This interval is _in addition_ to the other strategies for saving, logging, and evaluating, and does not replace them! It's sort-of a failsafe.

Details:

- Adds three arguments to `TrainingArguments`: `save_every_minutes`, `log_every_minutes`, and `eval_every_minutes`. Their default is None
- Add `time_checkpoints` as a field to `TrainerState`, which is a dictionary of `[log, save, eval] => datetime object` to track for each event type when its previous event occurred
- Updated `TrainerState.save_to_json` and `TrainerState.load_from_json` to account for the newly added `time_checkpoints`. This is necessary because datetime objects are not serializable. To accommoodate them, we serialize datetime object as an ISO string and deserialize them in the same way. This also ensures that the Trainer callback tests pass
- Updated `DefaultFlowCallback`:
  - Add `on_train_begin` which will set the aforementioned dictionary keys to the `now` timing so the we can start keep tracking at every step whether the interval has been reached
  - Update `on_step_end` to also check our intervals for every log, eval, save event and set `should_X` accordingly. If this is the case, we also update `time_checkpoints[event_type]` with the current time


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30188). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is on my list, so un-staling.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: Error 503: {'error': 'Service Unavailable'}
Author: jeremierostan
Pull request: no
Labels: name: Agents

### System Info

I am using this colab: https://colab.research.google.com/drive/1c7MHD-T1forUPGcC_jlwsIptOzpG3hSj
Which is linked here: https://huggingface.co/docs/transformers/en/transformers_agents

### Who can help?

@stevhliu

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1c7MHD-T1forUPGcC_jlwsIptOzpG3hSj

ValueError                                Traceback (most recent call last)
[<ipython-input-19-efe57595a5cd>](https://localhost:8080/#) in <cell line: 1>()
----> 1 boat = agent.run("Generate an image of a boat")
      2 boat

1 frames
/usr/local/lib/python3.10/dist-packages/transformers/tools/agents.py in generate_one(self, prompt, stop)

ValueError: Error 503: {'error': 'Service Unavailable'}

### Expected behavior

boat = agent.run("Generate an image of a boat")

Comments:
cc @aymeric-roucher
Update mask_generation.md
Author: nicholicaron
Pull request: yes

# What does this PR do?
<!--
Minor bug fixes -- rearrange import & add missing parentheses 
-->

## Before submitting
- [ x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
When running `ruff format src/transformers`, some files needs to be reformatted
Author: aliencaocao
Pull request: no
Labels: name: Feature request

### Feature request

Should a PR be made to fix those?
Here is a list as of https://github.com/huggingface/transformers/commit/74a207404e8d4524d1fdc4aa23789694f9eef347
![image](https://github.com/huggingface/transformers/assets/20109683/93adbfd5-e7a5-4270-931e-e1d297104101)


### Motivation

Simply workflows so that one can run `ruff format` and expect all changes made by it to be relevant to uncommitted work.

### Your contribution

I can make a PR

Comments:
Hi @aliencaocao, thanks for opening this issue! 

If you run `ruff format` directly, you might see many files to be updated. When we run `make fixup` it only runs ruff on the files which are being modified in the PR. This is OK, we don't necessarily want to have many diff which just update formats whenever ruff or our settings are updated. 

Alright,  thanks
First token optimization in beam search
Author: jiqing-feng
Pull request: no
Labels: name: Feature request

### Feature request

The beam search in the generation will repeat the `input_ids` for `num_beams` times. I understand it's for beam scorer, but it will introduce extra computation.



### Motivation

We cannot control users' behavior if users give a large input length (like 1024 or even more) and `num_beams` (like 256 or even more).

### Your contribution

Actually, we can repeat the outputs after the first token instead of repeating the first token inputs. The outputs are identical because the inputs are the same in each beam.


Comments:
Update convert_llama_weights_to_hf.py
Author: SWivid
Pull request: yes

This script is regarding to convert original llama weights to hf weights

Change: set safe_serialization default to **_False_**
otherwise it's always True 
no ways to getting .bin format output which is needed for other following-up repos

logically,
adding --safe_serialization and turn on safetensor format output

Comments:
Hi @SWivid, 

The conversion scripts in the repo are to convert from the original format to a format compatible for models in transformers. As HF now uses safetensors by default, I'm not sure it makes sense to commit this change into the repo. You can of course adapt the script to your own needs! 

if there's checkpoints which already exists in safetensors format, which you'd like in pytorch's `.bin` format, you can obtain this directly by doing:

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
model.save_pretrained(new_checkpoint_name, safe_serialization=False)
```
Correct @is_flaky test decoration
Author: qubvel
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Some tests are marked with `@is_flaky` decorator without brackets, which leads to always passing the test even if an exception is always raised in the test.

For example:
```python
@is_flaky  # <------ should be @is_flaky()
def test_flash_attn_2_equivalence(self):
    ...
```

P.S. Fixing this may lead to failing flaky tests, but now these tests are just broken.

## Who can review?

@ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31480). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's trigger a CI that runs slow tests for a few models involved.

I have put a label `run_slow`, could you push an empty commit with `[run_slow] gemma, xxx, yyy`, where `xxx`, `yyy` are some model names that you can think of relevant.



@ydshieh I triggered slow tests, but it seems like flash attention is not installed in CI (according to pip freeze output) -> tests were skipped:
<img width="984" alt="Screenshot 2024-06-19 at 10 23 11" src="https://github.com/huggingface/transformers/assets/31920396/b423d882-23d7-43aa-a4ec-afc08f3085ac">


oh, right. Those flash attn tests will only run with `.github/workflows/push-important-models.yml` which is triggered when a PR being merged to `main`.

One last thing could try is to use

https://github.com/huggingface/transformers/actions/workflows/ssh-runner.yml

Internal notion doc: https://www.notion.so/huggingface2/SSH-into-our-runners-d9c24de77d0843b2a308f28171106001

It's more a good chance to learn how to use this. Would you like to give it a try even it's somehow manual?

For docker image, you will have to fill `huggingface/transformers-all-latest-gpu`


You will have to do (once inside the runner)
```
cd transformers
```

and
```
          apt install sudo
          sudo -H pip install --upgrade pip
          sudo -H pip uninstall -y transformers 
          sudo -H pip install -U -e ".[testing]" 
          MAX_JOBS=4 pip install flash-attn --no-build-isolation
```

will merge after an approval from @amyeroberts  (as a manual run inside the runner is done .)

Gemma FA2 tests 
<img width="892" alt="Screenshot 2024-06-19 at 15 10 25" src="https://github.com/huggingface/transformers/assets/31920396/9b2a88fc-4137-4eba-9afd-a1d6693dd5b1">
Original Llama-3 tokenizer behaves differently from `transformers` version
Author: chawins
Pull request: no

### System Info

`transformers==4.41.2`
`tiktoken==0.7.0` and `0.4.0`

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The issue can be produced with the following snippet.

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
ids = tokenizer("! ! !").input_ids
print(ids)  # >> [128000, 0, 758, 758]
print(tokenizer.decode(tokenizer("! ! !").input_ids))  # >> 
!!! (this is wrong?)

# Llama-3 from https://github.com/meta-llama/llama3
# Download tokenizer.model from https://llama.meta.com/llama-downloads/
from llama import Tokenizer
tokenizer = Tokenizer("path/to/tokenizer.model")
ids = tokenizer.encode("! ! !", bos=True, eos=False)
print(ids)  # >> [128000, 0, 758, 758]
print(tokenizer.decode(ids))  # 
! ! ! (this is expected)
```

### Expected behavior

The string after encoding and decoding back should be the same. The original tokenizer has this behavior, but not the `transformers` version which throws out the whitespace. Is this expected?

I know that original Llama-3's tokenizer is based on `tiktoken` now. Is that the reason we see this difference?

Comments:
Hey! This is because of a default in `transformers`:
```print(tokenizer.decode(tokenizer("! ! !").input_ids, clean_up_tokenization_spaces=False) )``` 
should do the trick. 
Let's set it to default `False` and deprecated it cc @itazap ! 

Awesome! I confirmed that `clean_up_tokenization_spaces=False` fixed the issue. Thanks a lot for pointing it out.