noamaanMulla-03/transformers-issues huggingface dataset:
Model load Regression due to Auto Safetensor conversion merged in #29390
Author: Qubitium
Pull request: no

### System Info

Ubuntu 22.04
Torch 2.2.1
Latest git/head Transformers

### Who can help?

@LysandreJik @ArthurZucker 

### Reproduction

Since https://github.com/huggingface/transformers/pull/29390, our model loading will auto start safetensor conversion from non-safetensor model and attempt to submit a PR...and crashes. Huh?

Shouldn't there be a toggle for this feature and default to False instead of auto-enabled for all?

```
Exception in thread Thread-autoconversion:
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/root/miniconda3/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/safetensors_conversion.py", line 89, in auto_conversion
    sha = get_conversion_pr_reference(api, pretrained_model_name_or_path, **cached_file_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/safetensors_conversion.py", line 82, in get_conversion_pr_reference
    sha = f"refs/pr/{pr.num}"
                     ^^^^^^
AttributeError: 'NoneType' object has no attribute 'num'
```






### Expected behavior

1. Not crash and burn
2. Not auto start safetensor conversion

Comments:
Hey! This should not crash your runtime. I'll quickly revert and work on a fix. 

Thanks for reporting the issue!

It was reverted, feel free to update your clone. This call should only happen in the background with no effect to your runtime, the conversion happens server side. 

This is to continue pushing `safetensors` forward as an alternative to raw `.bin` checkpoints as we see more bad actors with pure `.bin` checkpoints. Thanks again for your report!

Fix confirmed.
Typo fix in error message
Author: clefourrier
Pull request: yes

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@MKhalusova, @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
StableLM: Fix dropout argument type error
Author: liangjs
Pull request: yes

# What does this PR do?

This resolves the following issues:
1. In StableLM, the `dropout_rate` passed to `_flash_attention_forward` should be a float instead of a Dropout object.
2. In almost all copies of the `_flash_attention_forward` method's documentation, the dropout argument is wrongly typed.

Comments:
Could you rebase on main to make sure you have the latest version of the code? 


> Could you rebase on main to make sure you have the latest version of the code?

Thanks for reviewing. I rebased it on the main branch.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29236). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Thread-autoconversion error
Author: geoffvdr
Pull request: no

Hello,

I am trying to load the `facebook/mms-1b` speech model and I am getting a 'conversion error' that I have never encountered before.

The code:
```python
from transformers import (
    AutoConfig,
    AutoModelForCTC
)
config = AutoConfig.from_pretrained("facebook/mms-1b")
model = AutoModelForCTC.from_pretrained(
         "facebook/mms-1b",
        config=config,
    )
``` 

The error:
```
Error during conversion: ValueError('Queue is full! Please try again.')
Exception in thread Thread-autoconversion:
Traceback (most recent call last):
  File "[/opt/conda/lib/python3.11/threading.py", line 1045](https://jupyterhub.ilabt.imec.be/user/gvdrreyd@ilabt.imec.be/lab/tree/project_antwerp/huggingface/exp/opt/conda/lib/python3.11/threading.py#line=1044), in _bootstrap_inner
    self.run()
  File "[/opt/conda/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 761](https://jupyterhub.ilabt.imec.be/user/gvdrreyd@ilabt.imec.be/lab/tree/project_antwerp/huggingface/exp/opt/conda/lib/python3.11/site-packages/ipykernel/ipkernel.py#line=760), in run_closure
    _threading_Thread_run(self)
  File "[/opt/conda/lib/python3.11/threading.py", line 982](https://jupyterhub.ilabt.imec.be/user/gvdrreyd@ilabt.imec.be/lab/tree/project_antwerp/huggingface/exp/opt/conda/lib/python3.11/threading.py#line=981), in run
    self._target(*self._args, **self._kwargs)
  File "[/opt/conda/lib/python3.11/site-packages/transformers/safetensors_conversion.py", line 89](https://jupyterhub.ilabt.imec.be/user/gvdrreyd@ilabt.imec.be/lab/tree/project_antwerp/huggingface/exp/opt/conda/lib/python3.11/site-packages/transformers/safetensors_conversion.py#line=88), in auto_conversion
    sha = get_conversion_pr_reference(api, pretrained_model_name_or_path, **cached_file_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/opt/conda/lib/python3.11/site-packages/transformers/safetensors_conversion.py", line 82](https://jupyterhub.ilabt.imec.be/user/gvdrreyd@ilabt.imec.be/lab/tree/project_antwerp/huggingface/exp/opt/conda/lib/python3.11/site-packages/transformers/safetensors_conversion.py#line=81), in get_conversion_pr_reference
    sha = f"refs/pr/{pr.num}"
                     ^^^^^^
AttributeError: 'NoneType' object has no attribute 'num'
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/mms-1b and are newly initialized: ['lm_head.bias', 'lm_head.weight', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original1']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
``` 

I am using `transformers 4.39.0.dev0` Any idea?

Thank you.

Comments:
Hey @geoffvdr, this is fixed on `main`; if you update your clone you shouldn't have this error anymore. Thanks!

thank you @LysandreJik !

Awesome! Let me close this as fixed.
Sliding window inconsistency between PyTorch and Flax
Author: jonatanklosko
Pull request: no

### System Info

transformers main (ae49b218c), Python 3.10.8

### Who can help?

@ArthurZucker, @sanchit-gandhi

### Reproduction

The attention `sliding_window` has different interpretation for PyTorch and Flax. Here's are matching examples:

**PyTorch**

```python
from transformers import MistralModel
import torch

model = MistralModel.from_pretrained("hf-internal-testing/tiny-random-MistralModel", sliding_window=2)

inputs = {
  "input_ids": torch.tensor([[10, 20, 30, 40, 50, 60, 70, 80, 0, 0]]),
  "attention_mask": torch.tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])
}

outputs = model(**inputs)

print(outputs.last_hidden_state[:, 1:4, 1:4])
```

**Flax**

```python
from transformers import FlaxMistralModel
import jax.numpy as jnp

model = FlaxMistralModel.from_pretrained("hf-internal-testing/tiny-random-MistralModel", sliding_window=2, from_pt=True)

inputs = {
  "input_ids": jnp.array([[10, 20, 30, 40, 50, 60, 70, 80, 0, 0]]),
  "attention_mask": jnp.array([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])
}

outputs = model(**inputs)

print(outputs.last_hidden_state[:, 1:4, 1:4])
```

Both snippets return different results, however, if we use `sliding_window=3` in the PyTorch version, the results are the same.

In the Flax implementation, `sliding_window=2` means that a position will attend to self, and two previous position inclusive (which intuitively seems correct to me). It looks like in the PyTorch version it is not inclusive. Which behaviour is expected?

### Expected behavior

The `sliding_window` meaning to be consistent.

Comments:
Hey!  Pretty sure `MistralSdpaAttention` does not support sliding window yet!  Are you using `attn_implementation="flash_attention_2"`? 

@ArthurZucker I'm using the default implementation on the CPU, I've just checked to make sure and it's "eager". Initially I thought the issues may be in flash_attn, but you made me realise it's obviously not used on the CPU, so I tracked it down and I think there is an off by one error here:

https://github.com/huggingface/transformers/blob/3f60d11a8750992287cd0d1f3dbc9df6ffc34288/src/transformers/modeling_attn_mask_utils.py#L169



I think one way to fix it would be this:

```python
if sliding_window is not None:
    diagonal = past_key_values_length - sliding_window - 1

    context_mask = torch.tril(torch.ones_like(mask, dtype=torch.bool), diagonal=diagonal)
    mask.masked_fill_(context_mask, torch.finfo(dtype).min)
```

By a quick glance, the initial implementation ([ref](https://github.com/huggingface/transformers/blame/f09a081d2765c6535256b0e2d65bf54fc03f7fee/src/transformers/models/mistral/modeling_mistral.py#L81-L88)) was actually inclusive, and it got lost during refactoring.

[flash_attn](https://github.com/Dao-AILab/flash-attention) also says "inclusive" in function docs, so I think that's the expected behaviour (and does make more sense).

would you like to open a PR? 🤗 

@ArthurZucker sure, opened #29519!
fix typos in FSDP config parsing logic in `TrainingArguments`
Author: yundai424
Pull request: yes

# What does this PR do?

There're 2 typos in how `TrainingArguments` is parsing the provided FSDP config.
1. We should not expect a leading `fsdp_` in user-provided config as 1) this is not how the docstring suggests, 2) other fields don't expect this
2. `prefect` should be `prefetch`


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
- trainer: @muellerzr and @pacman100

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Might need to add a test!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29189). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`object of type 'NoneType' has no len()` when trying to use `WhisperNoSpeechDetection`
Author: cifkao
Pull request: no

### System Info

- `transformers` version: 4.38.0.dev0 (5b6fa23 – after merging #28687)
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@patrickvonplaten @sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# %%
from transformers import WhisperForConditionalGeneration, AutoProcessor
import torch
import numpy as np

# %%
processor = AutoProcessor.from_pretrained("openai/whisper-base")
model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-base", torch_dtype=torch.float16
)
model.to("mps")

# %%
# 60 seconds of silence
raw_audio = np.zeros((1, 16_000 * 60))
inputs = processor(
    raw_audio,
    return_tensors="pt",
    truncation=False,
    padding="longest",
    return_attention_mask=True,
    sampling_rate=16_000,
)
inputs = inputs.to(model.device, torch.float16)

# %%
result = model.generate(
    **inputs,
    # chosen here to make it trigger the filter:
    no_speech_threshold=0.2,
    logprob_threshold=-0.2,
    # no_speech_threshold requires logprob_threshold, which in turn assumes temperature to be a list/tuple
    temperature=(0.0,),
)
decoded = processor.batch_decode(
    result, skip_special_tokens=False, decode_with_timestamps=True
)
print(decoded)
```
Result:
```pytb
Traceback (most recent call last):
  File "/Users/ondra/test-hf/test.py", line 27, in <module>
    result = model.generate(
  File "/Users/ondra/mambaforge/envs/transformers/lib/python3.10/site-packages/transformers/models/whisper/generation_whisper.py", line 735, in generate
    sequences = _pad_to_max_length(final_segments, generation_config.pad_token_id, padding="right")
  File "/Users/ondra/mambaforge/envs/transformers/lib/python3.10/site-packages/transformers/models/whisper/generation_whisper.py", line 148, in _pad_to_max_length
    pad_length = max_total_length - len(sequences[i])
TypeError: object of type 'NoneType' has no len()
```

### Expected behavior

Should not error out, should output `['']`.

Comments:
cc @sanchit-gandhi @ylacombe 

Thanks for opening the issue! Opened a PR to fix this!
Fix WhisperNoSpeechDetection when input is full silence 
Author: ylacombe
Pull request: yes

# What does this PR do?

@cifkao found an edge case that happens when the input of Whisper.generate is a full silence. This is a simple tentative PR.

cc @sanchit-gandhi 
<!-- Remove if not applicable -->

Fixes #29036



Comments:
I want to add a test, but I realized most of the slow tests of Whisper were already failing, independently of this PR. 
cc @sanchit-gandhi 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29065). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Seems to work for me!

Do you need a review @ylacombe ? 

I do, thanks !
Make sliding window size inclusive in eager attention
Author: jonatanklosko
Pull request: yes

Closes #29176. See the issue for details.

Comments:
Hi @jonatanklosko, thanks for opening this PR! 

It looks like this breaks some tests, specifically: 

```
FAILED tests/test_modeling_utils.py::AttentionMaskTester::test_2d_to_4d_causal_sliding - assert 3 == 6
FAILED tests/test_modeling_utils.py::AttentionMaskTester::test_causal_mask_sliding - assert 9 == 12
```

which will need to be addressed before merge 

@amyeroberts fixed!

@jonatanklosko Great! Last things to do before merging: 

* Rebase on main. We recently merged in a patch which should resolve the currently failing test
* Run the slow model tests for some of the main models this might affect, namely Mistral, Mixtral, LLama, Whisper, Bart, Falcon, Idefics, stablelm, starcoder2 `RUN_SLOW=1 pytest tests/models/mistral tests/models/mixtral tests/models/llama tests/models/whisper tests/models/bart tests/models/falcon tests/models/idefics tests/models/starcoder2 tests/models/stablelm tests/models/qwen2 tests/models/gpt_bigcode`  

Rebased!

As for the slow tests, would someone who already has these models cached be kind enough to run the tests? Currently, I don't think I can reasonably download all these models with my current internet connection (and I don't have a GPU, not sure whether that's relevant).

@jonatanklosko I can run them on this branch

@jonatanklosko OK, I've run on my side. All the tests looks OK (some fail when running on my local, but I get the same failures on main). Thanks again for working on this! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29519). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts fantastic, thank you!
Fix eval thread fork bomb
Author: muellerzr
Pull request: yes

# What does this PR do?

Currently we're creating a fork bomb due to recreating the eval dataloader again and again when `persistent_workers` is set to `True`. 

Since the dataloader for training should be recreated, but eval should *not*, this PR fixes this by making the eval dataloader persistent.

Proved it works by fork-bombing myself 😅 

Fixes https://github.com/huggingface/transformers/issues/28469


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: use `warning_advice` for tensorflow warning
Author: winstxnhdw
Pull request: yes

# What does this PR do?

Hey @amyeroberts, as discussed in #27214, we are now using `warning_advice` instead of `warning`. This will allow us to disable this warning via `TRANSFORMERS_NO_ADVISORY_WARNINGS=1`.

Ref #27214

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
[TrOCR] Dealing with occasional multi-line images
Author: aureliusnoble
Pull request: no

Hi, I am using TrOCR to transcribe historical (18th century) handwritten French data.

I am feeding in text-line images which are automatically segmented. However, due to the nature of the documents, sometimes this segmentation is not perfect, and the image contains multiple lines of text. These have been transcribed with the \n token.

I know that ideally I would not pass in multi-line images, however these will be occasionally present in the data I want to run inference on. What are your thoughts on dealing with these when fine-tuning TrOCR? About 5-10% of lines include a "\n" character, about 1% contain multiple "\n" characters. Currently I just convert these to whitespaces.

cc: @NielsRogge . Any advice would be greatly appreciated.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ReBASED
Author: kabachuha
Pull request: no
Labels: name: New model

### Model description

Mirroring https://github.com/huggingface/transformers/issues/29466, the newer model utilizing LinearAttention adds RMS norm to the attention forwarding and contracts the Taylor expansion to include only the third term, showing better performance and like BASED outperforms Mamba and stuff

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

The repo https://github.com/corl-team/rebased

Comments:
If there are no model weights, the chances of merging it are very low 😢 

@ArthurZucker Their team now provided the weights and generation code https://github.com/corl-team/rebased/issues/2

From my perspective, there is a problem with Based/ReBased models in Huggingface Transformers: if someone wants to train/finetune a model, they need to utilize IO-Aware Triton kernels because the vanilla Torch implementation has a significant memory footprint. There is an implementation of several models [here](https://github.com/sustcsonglin/flash-linear-attention), and theoretically, it is possible to include flash_linear_attention as an optional dependency, similar to flash_attention. Additionally, there is still an option to use model for inference without any kernels.