noamaanMulla-03/transformers-issues huggingface dataset:
FlashAttention2 issue with Mistral/Mixtral related to max length and RotaryEmbedding
Author: psinger
Pull request: no

I have been seeing very weird behavior when training and running Mistral or Mixtral with samples being exactly the length of `max_position_embeddings`. The strange behavior manifested itself with complete broken outputs that interestingly resolved itself after reloading the model and running samples with shorter length through.

So the following combination always broke:
Model with `max_position_embeddings=8192` and using FA2 and using some samples with size `max_length=8192`.
It was resolved by either disabling FA2, or actually using samples with `max_length=8191`.

After a lot of debugging, I figured out that this issue only happens with Flash Attention 2 and not with SDPA or vanilla attention.

I am suspecting that this issue stems from the following line:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mixtral/modeling_mixtral.py#L447

```
rotary_seq_len = max(kv_seq_len, position_ids[:, -1].max().item()) + 1
cos, sin = self.rotary_emb(value_states, seq_len=rotary_seq_len)
```

If we have a batch with a sequence length of let's say `8192`, which could be the same as `max_position_embeddings`, then `kv_seq_len` will be  `8192` which is the max here, but then we are adding `1`, which will lead to `8193` and then we are calling `rotary_emb` with it.

There, we then call:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mixtral/modeling_mixtral.py#L214
and thus re-init the cache with a longer than supported max sequence length.

I think it can be already solved by changing it to:
`rotary_seq_len = max(kv_seq_len, position_ids[:, -1].max().item() + 1)`

I actually noticed that this code has been changed very recently for Mistral to not take the max length and reset it anylonger:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L108

This was done in PR https://github.com/huggingface/transformers/pull/30642

I think this might have been just a side effect and does not fix Mixtral behavior.

Comments:
cc @ArthurZucker @younesbelkada 

I think this is heavily related to #29496, and should simply need an update in the ROPE precision! Would you like to open a PR to update this and see if that solved your issues ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Whisper: move to tensor cpu before converting to np array at decode time
Author: gante
Pull request: yes

# What does this PR do?

Follow up to #27818 

`pytest --doctest-modules src/transformers/models/whisper/generation_whisper.py -vv` started failing on `main` due to the PR above. 

In a nutshell, if Whisper was running on GPU, the generated tensors would also be on GPU. The new decoding code called `token_ids.numpy()`, which failed if the `token_ids` tensor was on GPU. This PR moves it to the CPU before the numpy conversion :)

cc @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31954). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add torch.compile for Whisper  
Author: zhenglongjiepheonix
Pull request: yes

this PR adds torch.compile support for whisper model which is encoder-decoder architecture

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30949). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add torch compile for mixtral
Author: zhenglongjiepheonix
Pull request: yes

This PR is working in progress and it tries to add torch compile support for Mixtral, it currently also contains changes from  #30642  because there are some common ground shared between these two models, and there are several issues regarding Mixtral:

1. we have to set the following flag to True in order to capture full graph with MOE
```
torch._dynamo.config.capture_dynamic_output_shape_ops = True
```
I believe it's inevitable because `MistralSparseMoeBlock` uses `torch.where` to extract tokens that each expert cares about, and the number and indexes of tokens that each expert attends to are variable, even if we do make a static shape(which means we zero out the non-care tokens for each expert), we are adding extra computation cost because zero-out values still get to take participate in computation, and each expert will have to run full tokens in terms of computation, which makes the whole point of computation-saving of MOE invalid.

2. The logits tests on main branch are currently failing on my dev machine
```
=========================================== short test summary info ===========================================
FAILED tests/models/mixtral/test_modeling_mixtral.py::MixtralModelTest::test_custom_4d_attention_mask - AssertionError: assert False
FAILED tests/models/mixtral/test_modeling_mixtral.py::MixtralIntegrationTest::test_small_model_logits - AssertionError: Tensor-likes are not close!
FAILED tests/models/mixtral/test_modeling_mixtral.py::MixtralIntegrationTest::test_small_model_logits_batched - AssertionError: Tensor-likes are not close!
=========================== 3 failed, 112 passed, 35 skipped, 47 warnings in 34.78s ===========================
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30793). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ArthurZucker 

> thanks a lot for working on this. Seems like it would be too breaking to merge as is 😢 But I'll ping you if we have a new MoE model to use this as default... A related PR / implementation is #31173! Does this version support compile

Yes, it does, with fullgraph and cudagraphs enabled



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided []
Author: gtanya89
Pull request: no
Labels: name: trainer

### System Info

Using Trainer with PyTorch DDP on single node multiple GPU. torch.dist.init_process_group() is setup ok. Seems like Trainer [_get_train_sampler()](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L839) does not use DistributedSampler but rather RandomSampler? Or could this be another issue I am missing? Any inputs appreciated! Thanks!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Error originates in data collator. Same code works with Single GPU.

### Expected behavior

Expect to train in a distributed fashion on multiple GPUs using Trainer API.

Comments:
cc @pacman100 @muellerzr 

I'm also curious why removing DistributedSampler from [_get_train_sampler()](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L839) as I remember older version has it implemented for training with multi-gpu case.



It uses `Accelerate`'s sampler for the data now @yuyemin since the trainer has a complete integration. Can you post the error @gtanya89 with the full trace and a reproducer?

Here is the error:

<img width="770" alt="Screenshot 2024-05-14 at 3 43 52 PM" src="https://github.com/huggingface/transformers/assets/4960465/f2ce2069-47a3-4f5f-afb4-e1320e9dc75c">

I am running it with this command single node multi-GPU training:
torchrun --nnodes 1 --nproc_per_node 2 --rdzv-endpoint=localhost:port bert_dist.py

Pytorch: 2.x
CUDA: 12.2

@gtanya89 any chance you have a small subset of data we can look at? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix: Removed a wrong key-word argument in `sigmoid_focal_loss()` function call
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
The `reduction` argument passed in the following function call is wrong. We can see from the function definition of `sigmoid_focal_loss` there is no parameter named `reduction`:
https://github.com/huggingface/transformers/blob/7f79a97399bb52aad8460e1da2f36577d5dccfed/src/transformers/models/rt_detr/modeling_rt_detr.py#L2166
https://github.com/huggingface/transformers/blob/7f79a97399bb52aad8460e1da2f36577d5dccfed/src/transformers/models/rt_detr/modeling_rt_detr.py#L1933-L1950

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @amyeroberts 

Comments:
Integrating R3M Models into Transformers
Author: suraj-nair-1
Pull request: yes

# What does this PR do?

This PR integrates the new R3M visual encoder model into transformers. See this [issue](https://github.com/huggingface/transformers/issues/16403).

In its current state, the new model/config can be loaded, trained, and pre-trained models located [here](https://huggingface.co/surajnair) can be loaded using `AutoModel.from_pretrained` and behave as expected.

However, the model is simply a visual encoder on a ResNet backbone, taking a batch of images passing them through the pre-trained ResNet18,34,50, and returning a batch of embeddings. Therefore most of the testing infrastructure fails (as this is not an NLP model). There are also probably boilerplate versions of the model (ex: CausalLM) which are not applicable.

<!-- Remove if not applicable -->

Fixes # 16403


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? - still a WIP.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@edbeeching @LysandreJik  have been helping me so far.


Comments:
Thanks a lot for your contribution!

@NielsRogge, could you do a first pass on this model?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@edbeeching @LysandreJik Is this issue still open to work on?!

@RUFFY-369 yes feel free to work on it. However we now recommend this guide: https://huggingface.co/docs/transformers/custom_models, which makes models compatible and usable through the Transformers library using the `trust_remote_code=True` flag. This means that the code lives on the hub.
Requires for torch.tensor before casting
Author: echarlaix
Pull request: yes

Fixes ONNX export for swin, swin-donut and clap models

https://github.com/huggingface/transformers/blob/82486e5995ed0a65520b10ce1ea938214a199231/src/transformers/models/swin/modeling_swin.py#L643

coming from : 

```
self.shift_size = torch_int(0)
```

introduced in https://github.com/huggingface/transformers/pull/31311

as `torch_int` is expecting a `torch.Tensor` : https://github.com/huggingface/transformers/blob/82486e5995ed0a65520b10ce1ea938214a199231/src/transformers/utils/generic.py#L765


also I think we should be able to have here 
```
self.shift_size = 0
```

cc @merveenoyan @xenova 





Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31755). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts do you think this fix could be included in a patch release ? 

@echarlaix I'm happy to include it in a patch release towards the end of the week

> @echarlaix I'm happy to include it in a patch release towards the end of the week

thanks a lot!
Notify new docker images built for circleci
Author: ydshieh
Pull request: yes

# What does this PR do?

Notify new docker images built for circleci

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31701). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I change to a new channel `#transformers-ci-circleci-images` as suggested
@ArthurZucker Our original goal is to avoid surprise of new images pushed and used by everyone while we are not aware of it.

I will merge this PR as it is. If in the future we see the real need of notifying of dev ci images too, we can iterate.
Avoid race condition
Author: ydshieh
Pull request: yes

# What does this PR do?

We have some hub tests failing from time to time.

https://app.circleci.com/pipelines/github/huggingface/transformers/97781/workflows/d3365fb8-a6b4-489a-912e-6d142409f848/jobs/1295791

This is due to the race condition of deleting the repo. 

This PR avoid it by using `try ... except ...` just as in some `def tearDownClass(cls):`.

(Might be better to catch specific error code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @LysandreJik as we have  talked a bit on slack few weeks back.

Instead of using the same repo name for all the CI runs and a try/except catch could we instead us a randomly generate string at the end of each name? Or even the CI run ID? At the moment, we still will have the issue that concurrent tests will push to the same repo, which might obscure failures introduced by a PR

@amyeroberts  Yes, I am also thinking that! Would it be ok if we merge as it is to see how it goes for a few days (I am curious)? 

I will definitely work as what is suggested, but it would be nice to avoid the failures on main as soon as we can :-)
Masking: remove flakiness from test
Author: gante
Pull request: yes

# What does this PR do?

`test_custom_4d_attention_mask` was trying to compare the argmax of two sets of logits. The logits should be the same in theory -- one is created through batched inference, the other through properly masked and stacked inputs.

We know from experience that batching introduces tiny fluctuations -- see [this comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535) explaining why.

As such, the argmax of these two tensors can have different values, resulting in a flaky test. This PR removes the flaky part of the test, the equality check of the argmax. We still check that the two tensors are very similar.

Example of a failing test due to flakiness: https://app.circleci.com/pipelines/github/huggingface/transformers/97709/workflows/f407565a-dd9f-423f-9af2-86f0925cf109/jobs/1294534

Comments:
Do we know how different they are? Could we check with a tolerance instead? I'm find removing the hard check, but the two outputs being similar seems like a good thing to make sure we're keeping 

@amyeroberts that check already exists 🙌 (expand the diff)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: doc nits
Author: gante
Pull request: yes

# What does this PR do?

Fixes doc nits like this one: 
- the automatically generated link to `force_words_ids` is "https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/text_generation#transformers.GenerationConfig.force_words_ids(List[List[int]]" (ends in `(List[List[int]]` 🤮 ) because a space is missing


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31982). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How to manually stop the LLM output？
Author: invokerbyxv
Pull request: no

I'm using `TextIteratorStreamer` for streaming output.

Since LLM may repeat its output indefinitely, I would like to be able to have LLM stop generating when it receives a request to cancel.

Is there any way to accomplish this?

model: glm-4-9b-chat

```python
async def predict(messages, model_id: str, raw_request: Request, gen_kwargs: Dict):
    global model, tokenizer
    choice_data = ChatCompletionResponseStreamChoice(index=0, delta=DeltaMessage(role='assistant'), finish_reason=None)
    chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object='chat.completion.chunk')
    yield '{}'.format(_dump_json(chunk, exclude_unset=True))

    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors='pt')
    inputs = inputs.to(model.device)
    streamer = TextIteratorStreamer(tokenizer=tokenizer, skip_prompt=True, timeout=60.0, skip_special_tokens=True)
    generation_kwargs = dict(input_ids=inputs, streamer=streamer)
    generation_kwargs.update(gen_kwargs)
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()
    for new_text in streamer:
        print(new_text)
        if raw_request is not None and await raw_request.is_disconnected():
            print("disconnected")
            # todo stop generate
        choice_data = ChatCompletionResponseStreamChoice(index=0, delta=DeltaMessage(content=new_text), finish_reason=None)
        chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object='chat.completion.chunk')
        yield '{}'.format(_dump_json(chunk, exclude_unset=True))

    choice_data = ChatCompletionResponseStreamChoice(index=0, delta=DeltaMessage(content=''), finish_reason='stop')
    chunk = ChatCompletionResponse(model=model_id, choices=[choice_data], object='chat.completion.chunk')
    yield '{}'.format(_dump_json(chunk, exclude_unset=True))
    yield '[DONE]'
```

Comments:
cc @gante 

Hi @invokerbyxv  👋

 Following our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) or our [discord](https://discord.com/invite/hugging-face-879548962464493619) 🤗
 
Since this is your first issue with us, I'm going to answer your question :) 

Stopping text based on what we see in our stream is not possible. However, we can encourage our LLM to avoid the behavior we dislike! In this case, repetitions can be tamed with these two `generate` flags (which you can also pass to a pipeline):
1. `repetition_penalty=x`, which lowers the odds of the model repeating tokens if `x>1.0` (the higher the value, the bigger the impact)
2. `no_repeat_ngram_size=n`, which forbids the model to repeat ngrams of size `n`

(docs [here](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/text_generation#transformers.GenerationConfig), click on `Expand parameters`)
Add Yuan2M32
Author: wafers3
Pull request: yes

This PR adds the support of codes for Yuan2.0 M32 models.

Comments:
cc @ArthurZucker 

Hey could you open an issue linking to this PR with the details about this model, link to original weights, code and paper? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.