noamaanMulla-03/transformers-issues huggingface dataset:
Specify dtype=torch.bool to avoid xla error
Author: ysulsky
Pull request: yes

The StoppingCriteriaList allocates is_done without specifying dtype=torch.bool. On XLA this allocates a float tensor and causes a failure on the following line:

    is_done = is_done | criteria(input_ids, scores, **kwargs)

by attempting to OR a float tensor with bool tensor.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@Narsil @zucchini-nlp 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31191). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

You're very welcome, and thanks for the quick reviews!
Early labels validation
Author: amyeroberts
Pull request: yes

# What does this PR do?

Moves all of the labels validation to the start of the forward method, to prevent failing after an expensive forward pass.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31240). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
doc: add info about wav2vec2 bert in older wav2vec2 models.
Author: Vaibhavs10
Pull request: yes

# What does this PR do?

doc: add info about wav2vec2 bert in older wav2vec2 models.

An attempt to improve the usage of Wav2Vec2 BERT model checkpoints.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

cc: @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31120). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks again @Vaibhavs10!
unpin uv
Author: ydshieh
Pull request: yes

# What does this PR do?

But added 

> ENV UV_PYTHON=/usr/local/bin/python

See https://github.com/astral-sh/uv/issues/3784#issuecomment-2129554631

Workflow run look good

https://app.circleci.com/pipelines/github/huggingface/transformers/94031/workflows/de2db3f1-c5fc-4c62-97d3-3d6b2e66081c

Comments:
CI is green. Just have to push the images and remove `:dev` and we are ready to merge.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31055). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add missing Flaubert tokenizer tests
Author: bastrob
Pull request: yes
Labels: name: Core: Tokenization

# What does this PR do?

This PR implements tests for Flaubert Tokenization part. It contributes to the issue referred in [#16627 ](https://github.com/huggingface/transformers/issues/16627)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 

Comments:
Hey! Pinging @itazap for a first review here! 😉 

thanks @bastrob 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30492). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add condition to `benchmark` job in `push-important-models.yml`
Author: ydshieh
Pull request: yes

# What does this PR do?

For now, we want to run this benchmark only when some important models are modified in the merged into `main` event.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add a new model for RNA sequence modeling
Author: yangheng95
Pull request: no
Labels: name: New model

### Model description

I want to add a new model for RNA sequence modeling in next few weeks, like DNABERT. I have drafted a branch [here](https://github.com/yangheng95/transformers/tree/add_omnigenome) to add the model and prepared some [examples](https://github.com/yangheng95/transformers/tree/add_omnigenome/examples/pytorch/genomic-sequence-modeling) for RNA modeling. There is a manuscript for the model: [OmniGenome - Aligning RNA Sequences with Secondary Structures in Genomic Foundation Models.pdf](https://github.com/user-attachments/files/15568804/OmniGenome.-.Aligning.RNA.Sequences.with.Secondary.Structures.in.Genomic.Foundation.Models.pdf)


I have never contributed any model to Transformers before, can anyone let me know what should I do to sort it out?


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The URL of [implementation](https://github.com/yangheng95/transformers/tree/add_omnigenome/src/transformers/models/omnigenome)

The URL of [model weights](https://huggingface.co/anonymous8/OmniGenome-186M)

Comments:
cc @Rocketknight1 

Hi @yangheng95, this looks really cool! I think the right approach here is to upload the model with the custom code in the model repo itself! In general, most new architectures start this way. Code in the `transformers` library itself is maintained by the team at Hugging Face, so we generally only port architectures into the main library after they get a lot of users.

Please see the guide [here](https://huggingface.co/docs/transformers/en/custom_models). It seems like you've already done most of the work by writing the `configuration` and `modeling` files, so all you'd need to do is initialize your model, register the auto classes and `push_to_hub()`.

Let me know if you encounter any difficulties!
`generate` do not stop after generating `eos_token` with batch process
Author: notoookay
Pull request: no
Labels: name: Generation

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @gante 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
prompts = example["prompt"] # (batch_size,)
inputs = tokenizer(prompts,
                  return_tensors="pt",
                  padding=True,
                  truncation=True).to(model.device)
input_ids = inputs["input_ids"]
outputs = model.generate(input_ids,
                             max_new_tokens=max_new_tokens,
                             do_sample=False,
                             eos_token_id=tokenizer.eos_token_id,)
```

### Expected behavior

I found when generating with only one input, model (I use llama2 to test) acts normal, when generating with a batch of inputs, model still generating `<unk>` after `eos_token` has been generated, like this:

`### Instruction:\nAnswer the following question.\n\n### Input:\nWhat is the capital of France?\n\n### Response:\nThe capital of France is Paris.
<unk><unk><unk><unk><unk>`

Is this the reason: **The `generate` function is designed to generate sequences of a consistent length for all sequences in a batch. When generating sequences in a batch, the `generate` function continues until the longest sequence in the batch reaches the specified `max_length`, or until all sequences have generated an EOS token, whichever comes first. This means that if one sequence in the batch generates an EOS token before the others, the `generate` function will continue generating tokens for that sequence (likely just padding tokens) until the other sequences have finished.**

If it is, will it be fixed in the future? If not, please give me some suggestion.

Thank you.

Comments:
Hey! This is expected I think. Unless it's actually generating them, I suppose it's just the padding we add to make the batch fit

Got it, thank you.
Why cast to float32 in this line?
Author: brynhayder
Pull request: no

https://github.com/huggingface/transformers/blob/edf0935dca9189db599ac6c3f3ef714160acbbd8/src/transformers/models/llama/modeling_llama.py#L87

Is there a numerical reason for doing this? For memory reasons, one might want to keep the the dtype that the user has chosen.

Comments:
Maybe this is the reason...? https://github.com/google/flax/issues/612

Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

I'll let @ArthurZucker confirm, but yes, I suspect your intuition is correct. The models are ported from their original repo. To ensure our models are as close as possible to the original implementation, we need to reflect design decisions like this and so would do the same as in the flax model (in which case the numerical stability reason applies)

Softmax on float32 works better (more precision) than on bfloat16 or float16. It's now standard 😉 

Thanks.

Out of interest, is this something that has been explore rigorously (i.e. in a paper), or is it more anecdotal/folk lore?

There is [this](https://github.com/huggingface/transformers/pull/17437#issuecomment-1139723683) (from the OPT paper that is related) , [this](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#training_tensorflow) for training, and #24519 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Softmax on float32 works better (more precision) than on bfloat16 or float16. It's now standard 😉

How to use fp32 when calculating softmax in flash_attention? Matrix multiplication still wants to be done with half precision, do you know any implementations?
Attention mask is None when 0.0 is not in attention_mask
Author: mxjmtxrm
Pull request: no

### System Info

- `transformers` version: 4.41.0.dev0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.0a0+ebedce2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker and @younesbelkada


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In modeling_llama.py,  attention mask is None when 0.0 is not in attention_mask  with flash_attention:
```
        if self.config._attn_implementation == "flash_attention_2":
            if attention_mask is not None and 0.0 in attention_mask:
                return attention_mask
            return None
```
Why this? If I can modify `if attention_mask is not None and 0.0 in attention_mask:` to `if attention_mask is not None:`?

### Expected behavior

--

Comments:
Hey! The reason is that we want to automatically dispatch to the optimised kernel of FA2 . When the attention mask is full of 1s, it means that there is no attention mask. Thus we don't send it to FA2

Got it. Thanks
Skip failing JetMOE generation tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

Two JetMOE generation tests started failing after the merge of #30536 - skipping them for now to make `main` happy again and avoid blocking PRs.  

```
FAILED tests/models/jetmoe/test_modeling_jetmoe.py::JetMoeModelTest::test_beam_sample_generate - RuntimeError: cannot reshape tensor of 0 elements into shape [4, 0, 2, -1] because the unspecified dimension size -1 can be any value and is ambiguous
FAILED tests/models/jetmoe/test_modeling_jetmoe.py::JetMoeModelTest::test_generate_from_inputs_embeds_decoder_only - AssertionError: AssertionError not raised
```

Example run: https://app.circleci.com/pipelines/github/huggingface/transformers/94919/workflows/7b36e120-b4ce-45d4-ba80-15a681557e87/jobs/1247043

cc @ArthurZucker 

Comments:
Merging as flax equivalence tests for BigBird is unrelated 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31266). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, FYI @gante and @zucchini-nlp . Pinging @Cyrilvallez in case he has time to work on a fix 😉 We should have checked if `Jamba` tests also pass since the arch is also a bit new / hybrid!

Hi! Sorry about that, looking into it right now! Thanks @amyeroberts for noticing!
Mamba: `use_cache` is not passed through in `prepare_inputs_for_generation` 
Author: uwu-420
Pull request: no

Hi :)

I think that [use_cache](https://huggingface.co/docs/transformers/main/en/model_doc/mamba#transformers.MambaForCausalLM.forward.use_cache) is supposed to be passed through here as well:
https://github.com/huggingface/transformers/blob/9fd606dbdb392e76e66a4e4e852948ebf814dbb6/src/transformers/models/mamba/modeling_mamba.py#L634-L647

I noticed when I wanted to get the cache when using `model.generate`, but it was not there although I set `use_cache=True`.

Edit: Just saw that `GenerateDecoderOnlyOutput` would have to be adjusted as well. It would need to contain `cache_params` similarly to `past_key_values`. I don't know if it's okay for you to bloat that even more.
https://github.com/huggingface/transformers/blob/9fd606dbdb392e76e66a4e4e852948ebf814dbb6/src/transformers/generation/utils.py#L2244-L2251

Cheers and thanks for the great work!

Comments:
cc @gante 

@zucchini-nlp could you have a look at this issue? 🤗
Flash attention implementation with BERT base model
Author: ghost
Pull request: no
Labels: name: New model

### Model description

hello and thanks community.
I am trying to replace standard attention by flash attention in the BERT base Model. Anyone please help not able to find any tutorial or any discussions.
or just give some directions how to do that ..I have got the idea of making attention prob drop prob = 0 . it makes sense but not sure how it is going to work.

@tridao
@arthur
@jamaliki
@sorenmc
 @LysandreJik @ArthurZucker  




### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Not that expert but I suggest you can try bettertransformer for extreme speed up. （ In my knowledge that flash-attn is mainly focused on kv cache which is not exist on Bert-like model in most cases. )

> Not that expert but I suggest you can try bettertransformer for extreme speed up. （ In my knowledge that flash-attn is mainly focused on kv cache which is not exist on Bert-like model in most cases. )

mosaic bert is based on flash attention. 

Hi @rahul-k01, 

Please make sure to only tag a limited set of relevant people when opening an issue - everyone is very busy and if this was done on all issues we wouldn't be able to meaningfully address our notifications. 

FlashAttention isn't implemented yet for BERT. There's an open issue where you can track which models have it added and on-going work: #26350. At the moment, it seems @filippo82 is working on the addition for BERT. 

> Hi @rahul-k01,
> 
> Please make sure to only tag a limited set of relevant people when opening an issue - everyone is very busy and if this was done on all issues we wouldn't be able to meaningfully address our notifications.
> 
> FlashAttention isn't implemented yet for BERT. There's an open issue where you can track which models have it added and on-going work: #26350. At the moment, it seems @filippo82 is working on the addition for BERT.

Thanks for your response


@ghost Note also that https://github.com/huggingface/transformers/pull/28802 is on the way to be merged, and would allow to use FA2 through SDPA in case no custom `attention_mask` is used.

There was a PR (https://github.com/huggingface/transformers/pull/27478) opened to add flash_attn 2 support to BERT a while ago, but it looks like the reviewers never got around to getting it merged for some reason and it was automatically closed as stale.

I've been using this implementation while official support is still pending.

Adding `sdpa` first (should be merged today / tomorrow) then I think the fa2 will be picked up 

mark
Truncation Error is inconsistent between fast and standard (non-fast) tokenizers
Author: dkajtoch
Pull request: no

## Environment info
- `transformers` version: 4.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help
Library:
- Tokenizers: @SaulLu

## Information
When truncation cannot be performed due to length issue Fast tokenizer raises Exception whereas standard (non-fast) only logs error and do not perform truncation at all. I only tested with AutoTokenizer (bert-base-uncased) but believe this is general on the level of `PreTrainedTokenizer` and `PreTrainedTokenizerFast`.

## To reproduce

```python
from transformers import AutoTokenizer
from transformers.tokenization_utils_base import (
    BatchEncoding,
    PaddingStrategy,
    TruncationStrategy,
)

fast_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False)

text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."
text_pair = "Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."

kwargs = {
    "text": text,
    "text_pair": text_pair,
    "max_length": 16,
    "truncation": TruncationStrategy.ONLY_FIRST
}

# logs: We need to remove 69 to truncate the input but the first sequence has a length 44. Please select another truncation strategy than TruncationStrategy.ONLY_FIRST, for instance 'longest_first' or 'only_second'.
print("Tokenizer encode_plus")
batch = tokenizer.encode_plus(**kwargs)
print(len(batch.input_ids) <= kwargs["max_length"])

# raises Exception: Truncation error: Sequence to truncate too short to respect the provided max_length
print("Fast Tokenizer encode_plus")
fast_tokenizer.encode_plus(**kwargs)
```

## Expected behavior

I expect consistent behavior among tokenizers. Either fast and non-fast just log error and do not truncate or both of them raise exception that can be easily identified (ideally custom not just general `Exception` with string description).


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Is this fixed? I'm encountering the same error.

Hi @hyusterr, this issue is quite old, so it's hard to know if the same problem is being encountered here. Could you share some details, specifically the environment (run `transformers-cli env` in the terminal and copy-paste the output) and a reproducible code snippet which generates the error?
Generation: fix handling of special tokens
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31251. For backwards compatibility we have to check all special tokens firstly in kwargs, and then in self.config. Earlier we fixed that for `decoder_start_token_id`, this PR extends it for all special tokens.

Verified that it works with the provided script and the script from last fix on `decoder_start_token_id`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31254). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I had to fix the test by passing special token in config because previously all special tokens were updated inside generation strategy code blocks. I will see how to make it BC, not ready for merge yet

I think it's better to just raise warning and ask users to provide attention masks, which I already commited. Another option would be to prepare special tokens twice, once in generate and the second time juts before pre-fill. 

I prefer the first options as it's less redundant code and more intuitive, lmk if we want full BC even with repeating code