noamaanMulla-03/transformers-issues huggingface dataset:
AttributeError: 'NoneType' object has no attribute 'from_pretrained'
Author: louisabraham
Pull request: no

This code was working yesterday but doesn't work today:

```py
from transformers import AutoTokenizer
AutoTokenizer("Helsinki-NLP/opus-mt-en-fr")
```

Comments:
Same here a couple of hours ago

1. Hi, could you please provide the information related to your environment? 

2. When you say it was working yesterday but was working before, do you mean to say you've upgraded to version v4.0.0 released yesterday? If this is so, you may be obtaining the following error message: `AttributeError: 'NoneType' object has no attribute 'from_pretrained'`. This would be because you do not have `sentencepiece` installed.

3. Are you sure this worked previously? This should never have worked, as `AutoTokenizer` cannot be initialized like this, but has to be instantiated from the `from_pretrained` method:

```py
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
```

which works on v4.0.0 and on `master`, as long as you have SentencePiece installed.

Putting a better error message in #8881.

Right, I was using
```py
AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
```

Thanks, `pip install sentencepiece` fixed the issue!

It looks that previously the tokenizer outputted torch tensors and now lists. Is this intended? It breaks existing code.

Yes, this was a bug. Tokenizers are framework-agnostic and should not output a specific framework's tensor. The implementation of the Marian tokenizer was not respecting the API in that regard.

Tokenizers can still handle torch tensors, you need to specify that you want them though:

```py
tokenizer(xxx, return_tensors="pt")
```

I guess in your situation it has to do with the `prepare_seq2seq_batch`:

```py
tokenizer.prepare_seq2seq_batch(xxx, return_tensors="pt")
```

Thanks!

For me just upgrading the transformers package worked

`pip install --upgrade transformers`
[whisper] fix multilingual fine-tuning
Author: sanchit-gandhi
Pull request: yes
Labels: name: Audio

# What does this PR do?

Fixes #30654.

<details>

<summary> The script previously failed with the following args, but passes after this fix: </summary>

```sh
python run_speech_recognition_seq2seq.py \
	--model_name_or_path="openai/whisper-small" \
	--dataset_name="mozilla-foundation/common_voice_11_0" \
	--dataset_config_name="hi" \
	--language="hindi" \
	--task="transcribe" \
	--train_split_name="train+validation" \
	--eval_split_name="test" \
	--max_steps="5" \
	--output_dir="./whisper-small-hi" \
	--per_device_train_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--per_device_eval_batch_size="16" \
	--logging_steps="25" \
	--learning_rate="1e-5" \
	--warmup_steps="2" \
	--eval_strategy="steps" \
	--eval_steps="5" \
	--save_strategy="steps" \
	--save_steps="5" \
	--generation_max_length="20" \
	--preprocessing_num_workers="16" \
	--max_duration_in_seconds="30" \
	--text_column_name="sentence" \
	--freeze_feature_encoder="False" \
	--gradient_checkpointing \
	--fp16 \
	--overwrite_output_dir \
	--do_train \
	--do_eval \
	--predict_with_generate \
	--use_auth_token
```

</details>

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30865). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @sanchit-gandhi  for fixing the issue and @amyeroberts for approving the changes!
My latest evaluation tests are positive. Well done!
Add missing whitespaces in RuntimeError message
Author: ftnext
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #16905

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

---

Add whitespace in the message

```python
>>> from transformers import pipeline
>>> pipeline(task=None, model=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../venv/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 495, in pipeline
    raise RuntimeError(
RuntimeError: Impossible to instantiate a pipeline without either a task or a model being specified. Please provide a task class or a model
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._
DPO Trainer Crashes on multi-gpu setup!
Author: August-murr
Pull request: no

### System Info

Kaggle Notebook With 2X T4 GPUs.
[link to Kaggle notebook:] (https://www.kaggle.com/code/augustmurr/dpo-issue-recreationl)
The issue does not occur when loading the model on one GPU (for example "cuda:0"), but the trainer will only use 1 GPU which is very inefficient. 

### Who can help?

@muellerzr
@pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

link to Kaggle notebook: https://www.kaggle.com/code/augustmurr/dpo-issue-recreation


### Expected behavior

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Comments:
This looks to be an issue with how the weights are loaded with `device_map=auto` rather than with trainer. Possibly to do with `_no_split_modules`

cc @younesbelkada @ArthurZucker 

Hi @August-murr !
I agree with what @amyeroberts said, I think that you are loading the model with `device_map="auto"`. In order to correctly perform multi-gpu training, please refer to this comment: https://github.com/huggingface/accelerate/issues/1840#issuecomment-1683105994 - let us know how it goes!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Rewriting usage of `torch.bucketize` with more elementary functions
Author: EricLBuehler
Pull request: no

Hello everyone,

Thank you for your excellent work here. We are implementing the Idefics 2 model into `mistral.rs` which is built on Candle.

Candle has a Pytorch-like API, however, it does not have some functions. In particular, we noticed that Candle is missing the `torch.bucketize` function. I have opened an issue on Candle but thought it would also be a good idea to check here how to implement `torch.bucketize` using more elementary functions or generally rewriting that part of the code to emulate the behavior.

This is the part of the code I am referencing:
> https://github.com/huggingface/transformers/blob/4b3eb19fa7f359d25f62ca9108479f71de912ebc/src/transformers/models/idefics2/modeling_idefics2.py#L174-L185

Is there a way to rewrite this such that I do not need to call `torch.bucketize`? Thank you for any help, I would really appreciate it!

Comments:
FIX / Quantization: Fix Dockerfile build
Author: younesbelkada
Pull request: yes

# What does this PR do?

Quantization docker file build is failing, leading to the quant CI failing

https://github.com/huggingface/transformers/actions/runs/9135816678/job/25123768850

cc @ydshieh


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30890). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLaVa-Next: Update docs with batched inference
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

PR adds an example of batched inference for Llava-next in the docs section. Also as discussed with @NielsRogge , I verified that we can support multi-turn conversation format by feeding multiple images in one prompt. Both usage cases are added to the example code snippet.

Comments:
I will see check how the fine-tuning works and will make a short colab on that 😄 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30857). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The implementation of ROPE seems a little off for k
Author: wzhcz8902
Pull request: no

### System Info

main branch

### Who can help?

@ArthurZucker @younesbelkada @bojone

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/481a95781404e48b1c80940be17e8279dec82fe8/src/transformers/models/llama/modeling_llama.py#L160-L184

The rotary embedding for q and k is done exactly the same, but should they be different so that when they do multiply together, we end up with a relative position as the difference between them. The current implementation results in a relative position as a sum of the positions of q and k, so I think we should put a negative sign in front of k's angle, thus a correct implementation for k may look like
`k_embed = (k * cos) + (rotate_half(k) * (-sin))`

### Expected behavior

The angle between q and k should differ by a sign

Comments:
Hey, inviting you to read https://blog.eleuther.ai/rotary-embeddings/ on which the current implementation of ROPE is based. 

A quick test you can do is directly change this and check the outputs of the Llama 3 or 2 model 😉
DeformableDETR two stage support bfloat16 
Author: DonggeunYu
Pull request: yes

# What does this PR do?
bfloat16 does not work in DeformableDETR two stage.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/30906


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- vision models: @amyeroberts

Comments:
add return_token_timestamps to WhisperProcessor
Author: kamilakesbi
Pull request: yes
Labels: name: Good Second Issue

name: Audio

# What this PR do ? 

This PR fixes #30433 by making sure we can compute timestamps with both `WhisperForConditionalGeneration` and `AutomaticSpeechRecognitionPipeline`. 

We add a `return_timestamps` hyperparameter to `WhisperProcessor.feature_extractor` to be used when we want to compute timestamps.  When True, the processor will return a `num_frames` parameter containing the number of frames of the input audios. `num_frames` is then passed to `generate` and used to compute timestamps. 

Prior to that, timestamps were broken for whisper-large-v3 when used with WhisperForConditionalGeneration. 

# Who can review ? 

cc @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30812). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts @sanchit-gandhi Could you please merge this PR as I don't have the rights to do so? 
Thanks!
Timestamps are broken for whisper large with WhisperForConditionalGeneration
Author: kamilakesbi
Pull request: no
Labels: name: Good Second Issue

name: Audio

### System Info

**System Info**

- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

**Who can help**

@kamilakesbi @sanchit-gandhi 

**Reproduction**

Timestamps are broken for whisper-large-v3 when used with  `WhisperForConditionalGeneration`: 

Note: This issue is related to #30224 

### Reproduction

```
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch 
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
model.to(device)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
speech_samples = ds.sort("id").select(range(4))[:4]["audio"]

input_speech = [x["array"] for x in speech_samples]
features = processor.feature_extractor(raw_speech=input_speech, return_tensors="pt")

input_features = features.input_features.to(device)
generate_kwargs = {}

generate_outputs = model.generate(
    input_features, return_timestamps=True, return_token_timestamps=True, **generate_kwargs
)
print(generate_outputs.token_timestamps)
```
We get: 

```
tensor([[ 0.0000,  0.0000, 29.3000, 29.3000, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800],
        [ 0.0000,  0.0000, 29.3000, 29.3000, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800, 29.9800,
         29.9800],...
```
indicating that the timestamps are broken...

Comments:
This notebook should be useful: https://github.com/sanchit-gandhi/codesnippets/blob/main/whisper-word-level.ipynb

While we fix this issue, we can also consider how to make the API simpler for users, since it currently requires some post-processing outside the model + processor API

@sanchit-gandhi is this issue open for taking?
(Willing to PR) Make `tokenizer.padding_side` an argument instead of only being a field
Author: fzyzcjy
Pull request: no
Labels: name: Core: Tokenization

name: Feature request

### Feature request

Hi thanks for the library! When using tokenizer, for example, for batch-generation with GPT2 (in https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517), it seems that currently I have to do something like:

```
tokenizer.padding_side = 'left'
data = tokenizer(['sentence one', 'another'])
tokenizer.padding_side = 'right'
```

Therefore, it would be great to have:

```
data = tokenizer(['sentence one', 'another'], padding_side = 'left')
```

just like what we do today for many options like `padding_strategy` etc.

### Motivation

(see above)

### Your contribution

Yes, I am willing to PR

Comments:
cc @ArthurZucker 

Sure, feel free to open a PR and ping @itazap 🤗
TypeError: export() got an unexpected keyword argument 'use_external_data_format'
Author: ZLKong
Pull request: no

Hi, 

I tried to run `convert_graph_to_onnx.py` using
`convert(framework="pt", model="bert-base-uncased", output="onnx/bert-base-uncased.onnx", opset=11)`

But I get error in the 
```
        export(
            nlp.model,
            model_args,
            f=output,
            input_names=ordered_input_names,
            output_names=output_names,
            dynamic_axes=dynamic_axes,
            do_constant_folding=True,
            use_external_data_format=use_external_format,
            enable_onnx_checker=True,
            opset_version=opset,
        )
```

the error is :
```
TypeError: export() got an unexpected keyword argument 'use_external_data_format'
TypeError: export() got an unexpected keyword argument 'enable_onnx_checker'
```

When I deleted these two lines, it does not report errors. Is it OK to remove these two lines?

Thanks,

ZLK

Comments:
I just did this too and it seems to work :S 

I looked into this and I believe the issue is that use_external_data_format is a recent [addition to PyTorch from the onnx team](https://github.com/pytorch/pytorch/commit/96989a2a114de9b77e7dd9495d62c4a8a549b40d). If you upgrade to torch>=1.5.0 it should work. Also I added a PR #5687 to make this issue more straightforward.

i have torch==1.13.1+cu116 installed and got this error, but i'm using in conjunction with this: https://github.com/quic/aimet

I have 2.1.0+cu121, still got this error

@theoctopusride @ecilay The recommended and best supported way to export HF models to onnx is now through the optimum library: https://huggingface.co/docs/optimum/en/exporters/onnx/usage_guides/export_a_model
SPDA/FA2 Attention for the Wav2Vec2 Family of Models
Author: sanchit-gandhi
Pull request: no
Labels: name: Feature request

name: Audio

### Feature request

Addition of [PyTorch SDPA](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) and [Flash Attention 2](https://github.com/Dao-AILab/flash-attention) to the Wav2Vec2 modelling code.

### Motivation

Wav2Vec2 and its derived models remain some of the most popular speech recognition and audio classification models in the library. However, only one [attention implementation](https://github.com/huggingface/transformers/blob/9b5a6450d481b0f02834684ffd8b3ba4cbbd6fe0/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L487) is available to users: the slowest and most memory consuming "eager" mode. We should update the modelling code to provide two newer attention implementations: SDPA and FA2, both of which are faster and more memory efficient.

Since Wav2Vec2 copies its attention from BART, and SDPA & FA2 were added for BART in [this PR](https://github.com/huggingface/transformers/pull/27203), this should be quite a straightforward PR, mostly copying out the logic from the BART PR and pasting it into Wav2Vec2. We should then be sure to add two fast tests (one for each of SDPA and FA2), e.g. in the style of the test [here](https://github.com/huggingface/transformers/blob/9b5a6450d481b0f02834684ffd8b3ba4cbbd6fe0/tests/models/whisper/test_modeling_whisper.py#L891), and two slow integration tests, e.g. in the style of the tests [here](https://github.com/huggingface/transformers/blob/9b5a6450d481b0f02834684ffd8b3ba4cbbd6fe0/tests/models/gemma/test_modeling_gemma.py#L657-L659).

### Your contribution

Want to take this one @kamilakesbi?

Comments:
Hi @sanchit-gandhi, @kamilakesbi!

I would love to be a part of this work, is it possible?

I have been planning to pretrain with Wav2Vec2 with flash-attn added.

And I've checked that it works fine when I pretrain with the current one applied.

here-wandb & [code](https://github.com/jp1924/wav2vec2/blob/main/models/wav2vec2/modeling_wav2vec2.py#L987-L991)
[flash-attn](https://wandb.ai/jp_42maru/Wav2Vec2/runs/gtkup657?nw=nwuserjp_42maru), [eager](https://wandb.ai/jp_42maru/Wav2Vec2/runs/t3smejv7?nw=nwuserjp_42maru), [sdpa](https://wandb.ai/jp_42maru/Wav2Vec2/runs/jalknj08?nw=nwuserjp_42maru)


So I was wondering if we could work together on this if you don't mind?


Hi @jp1924 ! 

Thanks for the great work! it seems like you have already done most of the modelling code here :) 
We can definitely work together on this and I've added you as a co-author to [this PR](https://github.com/huggingface/transformers/pull/30121). 
Feel free to participate ;)  



Oh, thank you for taking me up on my offer!