noamaanMulla-03/transformers-issues huggingface dataset:
Allow disabling of deletion of leading SPIECE_UNDERLINE during llama decoding (tokenizer).
Author: JoshC8C7
Pull request: no

### Feature request

The `LlamaTokenizer`'s `convert_tokens_to_string` method used during decoding [has the statement](https://github.com/huggingface/transformers/blob/6f7d5db58c7c149c75642b5a4647b5cbc6c55643/src/transformers/models/llama/tokenization_llama.py#L286):
```
if tokens[0].startswith(SPIECE_UNDERLINE):
    tokens[0] = tokens[0][1:]
```
which deletes a space if it falls at the start of the first token being decoded. There are cases where this is undesirable - namely in building streaming applications where an output is decoded in chunks and so a decoded sequence may begin with a space, which is unhelpfully deleted here. AFAIK there is then no way of knowing if the space was deleted without looking up the first token of each chunk in the vocabulary, and thus no way to faithfully recombine the chunks into a complete output.

**Ideally this deletion should be parameterised so it can be turned off in cases like these.**

My current workaround is to prefix a 'fake' token to the start of every sequence, before deleting it from the outputted text. I believe TGI [have a similar workaround](https://github.com/huggingface/text-generation-inference/blob/2d56f106a60c7b698705494e7539f8a7e4c85dd9/server/text_generation_server/models/model.py#L86).

### Motivation

When writing streaming applications with llama/codellama, you may want to decode in chunks. Where this chunk boundary falls between two words (i.e. the last token of the previous chunk is a word, and the first token of the next chunk is a word), then when decoding the second chunk the output string does not have a preceding space (even if its mapping in `tokenizer.json` has a SPIECE_UNDERLINE at the start). This information loss means the outputted chunks cannot faithfully be joined up, as it isn't known whether a space is deleted.

### Your contribution

I could submit a PR - this requires changing individual tokenizer files so I would have to read up on the procedure for that.

Comments:
I believe this is somewhat related to #28010 

Yes, will be fixed by #28010 ! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Pr merged and released. Closing!
Fix llama + gemma accelete tests
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR fixes the accelerate test for llama and gemma (`test_cpu_offload`).
The problem was that the `causal_mask` buffer was big compared to the small testing model. This resulted in edge cases where the model wasn't split since the `model_size` was too big to start with and `auto_infer_device_map` only allocates the parameters and not the buffers (supposed to be small). 

Related [PR](https://github.com/huggingface/accelerate/pull/2511) to fix these tests. 

Tests passing: 
```
pytest -m accelerate_tests tests/models/gemma/test_modeling_gemma.py::GemmaModelTest -s -vvvv
pytest -m accelerate_tests tests/models/llama/test_modeling_llama.py::LlamaModelTest -s -vvvv
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29380). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix deprecated arg issue
Author: muellerzr
Pull request: yes

# What does this PR do?

Dataclasses don't support item assignment 😓 

(And added tests)

Fixes https://github.com/huggingface/transformers/issues/29183


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29372). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ImportError: cannot import name '_expand_mask' from 'transformers.models.bloom.modeling_bloom'
Author: dipta007
Pull request: no

### System Info

```
- `transformers` version: 4.38.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

text models: @ArthurZucker and @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1tIgcJVofkzTic9ChhG8rNAQmFjrE1HYd?usp=sharing

### Expected behavior

It should run, but now its giving following error:

```
ImportError: cannot import name '_expand_mask' from 'transformers.models.bloom.modeling_bloom' (/usr/local/lib/python3.10/dist-packages/transformers/models/bloom/modeling_bloom.py)
```

Comments:
This is a duplicate of #28305, I am going to close this as we had a proper deprecation cycle for this. You should open the issue on the `open_flamingo` github repo 😉
How to set stopping criteria in model.generate() when a certain word appear 
Author: pradeepdev-1995
Pull request: no

### Feature request

stopping criteria in model.generate() when a certain word appear 

The word I need to stop the generation when found is : [/SENTENCE]
But the model doesn't generate the word itself, instead, it generates the subwords
 [ [/,SEN,TE,NC,E] ] 
like this . 

corresponding ids from the tokenizer are, 
( Id and subword word)
28792 => [
28748 => /
28759 => SEN
2654 => TE
1197 => NC
28793 => E]

so how can i put the condition in **StoppingCriteriaList** that i should stop the generation when the [/SENTENCE] found. 

### Motivation

stopping criteria in model.generate() when a certain word appear 

The word I need to stop the generation when found is : [/SENTENCE]
But the model doesn't generate the word itself, instead, it generates the subwords
 [ [/,SEN,TE,NC,E] ] 
like this . 

corresponding ids from the tokenizer are, 
( Id and subword word)
28792 => [
28748 => /
28759 => SEN
2654 => TE
1197 => NC
28793 => E]

so how can i put the condition in **StoppingCriteriaList** that i should stop the generation when the [/SENTENCE] found. 

### Your contribution

stopping criteria in model.generate() when a certain word appear 

The word I need to stop the generation when found is : [/SENTENCE]
But the model doesn't generate the word itself, instead, it generates the subwords
 [ [/,SEN,TE,NC,E] ] 
like this . 

corresponding ids from the tokenizer are, 
( Id and subword word)
28792 => [
28748 => /
28759 => SEN
2654 => TE
1197 => NC
28793 => E]

so how can i put the condition in **StoppingCriteriaList** that i should stop the generation when the [/SENTENCE] found. 

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ESM Rotary Embedding implementation is not TorchScript safe
Author: ChenchaoZhao
Pull request: no

### System Info

This issue is independent of the env. It’s purely about the PyTorch implementation of ESM position embedding

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Load an ESM model from pretrained 
2. To cuda
3. Use ‘torch.jit.trace’

### Expected behavior

It may crash when you trace the model.
Even if it doesn't crash at trace time, when save the model and move the model to a different device and perform inference it will crash with a device error.
The reason is the cached sin and cos are NOT registered as buffers which means pytorch will not know how to move the python attribute when using the ‘to’ method. I would suggest copy and use the llama ROPE implementation.

Comments:
cc @Rocketknight1 

Hi @ChenchaoZhao, I think you're correct - would you be willing to open a PR to fix the issue?

Hi @Rocketknight1, sure. Anything else other than the contribution guide I should be aware before making the PR?https://github.com/huggingface/transformers/blob/03cc17775b961d16cc4d0d7ab0c8487120d0b708/CONTRIBUTING.md

Hi @ChenchaoZhao, that guide should cover everything. Just open the PR to transformers and tag me in it + link this issue. And thank you - it's an important fix so it's great that you're willing to handle it!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Verify if output has logits or prediction logits in fill-mask pipeline
Author: pedrogengo
Pull request: yes

# What does this PR do?

It checks if the output has the key "logits" or "prediction_logits" to avoid breaking the fill-mask pipeline for some models like BertForPreTraining that returns:

```
return BertForPreTrainingOutput(
            loss=total_loss,
            prediction_logits=prediction_scores,
            seq_relationship_logits=seq_relationship_score,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )
```

Error without this change:

<img width="1122" alt="image" src="https://github.com/huggingface/transformers/assets/27240528/a7491281-db09-4b23-a795-5f102ec9d911">

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker @younesbelkada

Comments:
Maybe we should have a different error message for these models?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
GPT2 after few finetune epochs starts to generate sequence of only EOS tokens
Author: tempdeltavalue
Pull request: no

### System Info

I get output like this:










### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Check this ipynb
https://github.com/tempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb

### Expected behavior

I expect the model should returns something

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

@amyeroberts 
Ok, thank you 
(sorry)

![vvvvvvvvv](https://github.com/huggingface/transformers/assets/36921178/9553ed9c-6be0-4238-b7df-11f0a57ff009)

probably it's a some kind of bug (I can silence this left padding warning but it doesn't change anything some inputs just generates eos eos eos ...)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
refactor replace_linear for ease of additions and add unit tests
Author: Titus-von-Koeller
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28279). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Cache: dynamic cache with cross attention and UMT5 `Cache` support
Author: gante
Pull request: yes

# What does this PR do?

#28065 was becoming messy due to all Bart "copied from" dependencies, so this PR is a tiny version of it.

This PR:
1. Introduces `DynamicCacheWithCrossAttention`, which expands `DynamicCache` [cache object equivalent to the previous `past_key_values` input/output] with the ability to hold a cross-attention cache. This design was intentional: most LLMs (and now even multimodel models) tend to be decoder-only, so this separation will keep the cache class for decoder-only models simpler. It also enables us to be more strict -- in #28065 I've caught an unintended cache deletion in Whisper thanks to the increased specificity!
2. Adds `Cache` support to `modeling_umt5.py`, which is a form to test whether `DynamicCacheWithCrossAttention` is equivalent to the previous cache. These changes are the equivalent of the modeling changes in #26681, but for encoder-decoder models.

______________________________________

Local tests run:
1. `RUN_SLOW=1 py.test tests/models/umt5/test_modeling_umt5.py -vv` [Note: adds a test to ensure we keep the same results as in `main`]

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28185). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Flash Attention 2 to Persimmon
Author: jeromeku
Pull request: yes

# What does this PR do?

Integrates FA2 to Persimmon per #26350, #27052 (former branch was messed up after trying to rebase, so PR'ing a new branch).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@younesbelkada @ArthurZucker 

## Notes

- Fixed comments per #27052.  Requesting new PR as former branch was messed up after trying to `rebase` on `main`.
- Tried making changes as suggested in #27661 for `generate_padding_right` test.  However, `Persimmon` tokenizer configs do not have either `eos` or `pad` tokens (both are set to `null` see [here](https://huggingface.co/adept/persimmon-8b-chat/blob/main/tokenizer_config.json)), so simply copying the `LlamaModelTest` `generate_padding_right` test override does not work.
  - Also tried running `dummy inputs` on the full pretrained model for the `generate_padding_right` test, no luck either -- this is left as the current implementation in `test_persimmon_modeling.py`.
  - Ran some additional experiments on the `generate_padding_test` for other models for FA2 -- see [comments](https://github.com/huggingface/transformers/pull/27052#issuecomment-1820156930).
  - Marking `generate_padding_right` test as `skip` for now.
- Files other than those related to `persimmon` were changed in this PR due to fixes from running `make {quality, style, fixup}`
    

Comments:
@younesbelkada 

Re-based, installed `ruff==0.1.5`, and re-ran `make style`, still getting test failure for `PhiModelTest.test_pipeline_text_generation`.

cc @molbap as younes is offline

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Consistency] Automatically set decoder_input_ids for TFEncoderDecoderModel
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

Currently, the `EncoderDecoderModel` class in PyTorch automatically creates the `decoder_input_ids` based on the `labels` provided by the user (similar to how this is done for T5/BART). This should also be implemented for `TFEncoderDecoderModel`, because currently users should manually provide `decoder_input_ids` to the model.

One can take a look at the TF implementation of BART for example to see how to shift the labels in order to automatically create the `decoder_input_ids`, namely [here](https://github.com/huggingface/transformers/blob/29dfb2dbb10cdba6327ff287db56b182c1db29b1/src/transformers/models/bart/modeling_tf_bart.py#L1120).

One should then also update the docstring correspondingly. 



Comments:
Hi, I would like to work on this. Is there anyone already working on it?

No, you can work on it if you want!

Hi, Is it still open? I can work on this issue

There's a PR open (#14469), however it's not finished yet.

Hi @NielsRogge Can I finish up this issue with the closed PR as reference? I'd like to make a PR soon.

I believe this has already been resolved in #15175.

It appears that way. I am new to this codebase (and looking for a good first contribution), so take any of my opinion with a grain of salt, but this block appears to implement this functionality as part of the PR: https://github.com/huggingface/transformers/blob/831bc25d8fdb85768402f772cf65cc3d7872b211/src/transformers/models/vision_encoder_decoder/modeling_tf_vision_encoder_decoder.py#L572-L575

If this is true, would it make sense to close this issue?

Closing this one as it has ben resolved by #15175. Thanks for reporting
Add mlx support to BatchEncoding.convert_to_tensors
Author: tidely
Pull request: yes

# What does this PR do?

Adds support for the [mlx array framework by Apple](https://github.com/ml-explore/mlx) to BatchEncoding.convert_to_tensors

Fixes #29405 



Comments:
Should be ready for review @ArthurZucker 

Failing test is unrelated
Add mlx support for BatchEncoding.convert_to_tensors
Author: tidely
Pull request: no

### Feature request

Support for the [mlx array framework](https://github.com/ml-explore/mlx) in BatchEncoding.convert_to_tensors

### Motivation

Currently when using mlx in combination with transformers, during batch encoding we need convert back and forth a lot from numpy to mlx. By giving it official support in transformers, it would remove that overhead.

### Your contribution

I'll be submitting a PR shortly.

Comments:
update path to hub files in the error message
Author: poedator
Pull request: yes

need to add `tree/` to path to files at HF hub.
see example path:
`https://huggingface.co/meta-llama/Llama-2-7b-hf/tree/main`

Just makes the warning message point to the correct location.

cc @younesbelkada @ArthurZucker 

Comments: