noamaanMulla-03/transformers-issues huggingface dataset:
Load Phi 3 small on Nvidia Tesla V100 - Flash Attention
Author: BigDataMLexplorer
Pull request: no
Labels: name: bug

### System Info

Hi,
I would like to inquire about the possibility of uploading and fine tuning a Phi 3 8k small. When I load the model, I get an error about missing Flash attention. If I want to install the given package, I get this error : 

```
RuntimeError: FlashAttention is only supported on CUDA 11.6 and above.  Note: make sure nvcc has a supported version by running nvcc -V.


      torch.__version__  = 2.3.1+cu121
```


But I have the required version of pytorch and CUDA (torch 2.3.1 and cuda 12.1)
Is it because I am using a Tesla V100 graphics card? Is there any way to load the model also with this graphics card?
I found this in the documentation for the Phi 3 mini on Huggingface:

> If you want to run the model on:
> 
> NVIDIA V100 or earlier generation GPUs: call AutoModelForCausalLM.from_pretrained() with attn_implementation="eager"

Does this also apply to the Phi3 Small 8k?? Beacause when I try to load it, the error occurs
```
model = AutoModelForSequenceClassification.from_pretrained("path", num_labels=num_labels ,attn_implementation="eager" )

AssertionError: Flash Attention is not available, but is needed for dense attention
```

Or should I try the ONNX version or it is just for inference?
Thank you.

### Who can help?

@Arthu @BenjaminBossan 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
Hi @BigDataMLexplorer, 

Are you assuming this [model](https://huggingface.co/microsoft/Phi-3-small-8k-instruct)? If so, the modeling code is hosted directly on the hub and not maintained by the `transformers` (see modeling file [here](https://huggingface.co/microsoft/Phi-3-small-8k-instruct/blob/main/modeling_phi3_small.py)). 

I see in the modeling code that Flash attention is required to run the model, but unfortunately, Flash attention is not supported for V100 [cards](https://github.com/Dao-AILab/flash-attention/issues/148).
So, it would be better to open a discussion on the Hub to ask authors if there is a way to run a model without flash attention.

Hope that helps 🤗 


@qubvel 
Thank you for your response.
I tried to find out anywhere if it is possible to load the model without flash attention, but it didn't find anything. Do you have any idea who can help me? I asked on the huggingface forum, but no one answered.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback): module 'tensorflow' has no attribute 'data'
Author: Rizwanali324
Pull request: no

im using following 
librosa
streamlit
pandas
soundfile
numpy
scipy
sounddevice
pydub
pyaudio
matplotlib
streamlit==1.36.0
transformers==4.41.2
torch==2.1
accelerate>=0.21.0
tensorflow==2.13.0
keras==2.13.1
deep-utils
noisereduce
datasets
evaluate
huggingface_hub
librosa
scikit-learn


and try to deploy my streamlit app but facing same in local and cloud as well

Comments:
Hi @Rizwanali324, thanks for opening an issue! 

Please make sure to follow the issue template and provide all the requested information such that someone can help you. This includes:
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* A minimal code reproducer 
* Information about the expected and observed behaviour 
* All relevant error information including the full error traceback 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Inconsistent special_token addition in EncoderDecoderModel forward pass
Author: emergenz
Pull request: no

### System Info

- `transformers` version: 4.40.0
- Platform: Linux-4.18.0-513.11.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.9
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1) Add a breakpoint at: https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L625

2) Run the example script of EncoderDecoderModel
https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L562-L577

3) Examine the content of `decoder_input_ids`. You will see that the bos_token has been added twice.

### Expected behavior

The bos_token should only be prepended once to the input to the decoder.

The reason for this bug occurring is that `BertTokenizer` already wraps the text in bos_token and eos_token. Then, during the `EncoderDecoderModel` forward pass (specifically during construction of the `decoder_input_ids` from the `labels`), another bos_token is added.

This means that in order to use EncoderDecoderModel correctly, one has to tokenize the decoder inputs in a way such that an eos_token is added at the end of the text, but a bos_token is NOT added at the beginning. This is funky and clearly not wanted behavior.

The decoder part of an EncoderDecoderModel is rarely initialized with an encoder-only Transformer. Since the tokenizer of decoder-only Transformers do not add bos_tokens nor eos_tokens, I think the correct way to change the behavior of the EncoderDecoderModel would be to add both the eos_token as well as the bos_token in the model forward pass. To clear up confusion, we should change the official example to constructing an EncoderDecoderModel from an encoder-only Transformer and a decoder-only Transformer.

Comments:
Hey! Not sure this was reported before, but if you want to propose a general fix, feel free to open a PR! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
meta-llama/Llama-2-7b-chat-hf tokenizer `model_max_length` attribute needs to be fixed.
Author: rohitdwivedula
Pull request: no

### System Info

transformers==4.42.3, python3.9.19

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python3
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
print(tokenizer.model_max_length)
```

This outputs `1000000000000000019884624838656`

### Expected behavior

Expected output should be `4096` since that is the max sequence length of this specific Llama model.

Comments:
Hey @rohitdwivedula, sorry for the overdue response! The best thing to do here would be to open a PR on this repository on the Hub changing that attribute. Would you mind doing this if you have the bandwidth?

@LysandreJik okay, will open a PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
GemmaTokenizerFast word_ids() returns only zeros
Author: Alienmaster
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.41.2
- Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The method `word_ids()` does only return a list of zeros instead of the correct word_ids.

```
sentence = "I love my cat"
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("google/Gemma-7b") #-version a0eac5b
encoded = tokenizer(sentence, return_tensors="pt")
print(encoded.word_ids())
# [None, 0, 0, 0, 0]
```

I tried several variations of configurations stated in the linked issues in #28881 , but for Gemma it doesn't change the result. The llama3 tokenizer outputs the correct values with this code.

### Expected behavior

The output of `word_ids` should look like
`[None, 0, 1, 2, 3]`

Comments:
Hey! Will have a look thanks for reporting

It seems that we need this:
```python
tokenizer._tokenizer.pre_tokenizer = Sequence([Split("▁","merged_with_next")])
encoded = tokenizer(sentence, return_tensors="pt")
print(encoded.word_ids())
[None, 0, 1, 2, 3]
```


#32191 should fix this !

Not included in https://github.com/huggingface/transformers/pull/32191 since the proposed fix breaks encoding. Will need to do in a follow-up PR :)

Small update:
```python3
pre_tokenzier = pre_tokenizers.Split(Regex('(?<!▁)▁'), "merged_with_next")
pre_tokenzier.pre_tokenize_str(sentence.replace(" ", "▁"))
Out[41]: [('I', (0, 1)), ('▁love', (1, 6)), ('▁my', (6, 9)), ('▁cat', (9, 13))]
tokenizer(sentence).tokens()
Out[48]: ['<bos>', 'I', '▁love', '▁my', '▁cat']
tokenizer(sentence).word_ids()
Out[49]: [None, 0, 1, 2, 3]
```
Gives somewhat acceptable results, 

But I don't recommend word ids, separations are "brittle" and this changes the output of the tokenization 

Is there a better way to link the tokens to words than `word_ids`?

Offsets! 

The offset mapping give you the exact place where the token corresponds to in the original string


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[flax_llama] Why is the return value of the `create_sinusoidal_positions` truncated by `num_pos`?
Author: mantle2048
Pull request: no
Labels: name: jax

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.35
- Python version: 3.12.2
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (gpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I noticed that in the implementation of [flax-llama](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_flax_llama.py), the return value of [create_sinusoidal_positions](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_flax_llama.py#L136C33-L136C40) was truncated by `num_pos`. 

``` python
def create_sinusoidal_positions(num_pos, dim):
    inv_freq = 1.0 / (10000 ** (np.arange(0, dim, 2) / dim))
    freqs = np.einsum("i , j -> i j", np.arange(num_pos), inv_freq).astype("float32")

    emb = np.concatenate((freqs, freqs), axis=-1)
    out = np.concatenate((np.sin(emb)[:, None, :], np.cos(emb)[:, None, :]), axis=-1)
    return jnp.array(out[:, :, :num_pos]) # Why?
```

Why is this? Is it to prevent the sequence length from being too short?

### Expected behavior

See above.

Comments:
cc @sanchit-gandhi 

Gentle ping @sanchit-gandhi 

Another ping @sanchit-gandhi 


Another another ping @sanchit-gandhi. Could you nominate someone to take over this whilst you're away?



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Return image hidden states
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33329 and there was another issue which I lost by now. We never returned image features in VLMs except for Idefics despite having it in the docs, which caused confusion among users.

In this PR we'll be returning the features after all manipulations are done, including feature-layer-selection and projection, which I believe is expected. Idefics2 for ex returns image last hidden state after perceiver, so should be consistent across models

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enable `padding_side` as call time kwargs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR adds `padding_side` as a valid kwargs when calling tokenizers, so that the users can set padding side when tokenizing. Is a follow-up from https://github.com/huggingface/transformers/pull/32858#discussion_r1743272231, where we found that most processors accept `padding_side` but don't use it when tokenizing. 

Added test for that and ran it for all models

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The failing test is not related to this PR, seems to be related to training datasets and tasks. PR is ready for review!

Looks good to me thanks @zucchini-nlp, this should unblock a lot of the kwargs uniformization PRs :). Anything blocking before this gets merged? Is it because of the tests failing?

I wanted to get @ArthurZucker 's review as well as this is tokenizers related. For tests I can rerun them and hope they won't fail this time


Thanks, can someone merge pls as the tests won't pass after rerunning?

@zucchini-nlp Yep - I can merge
Process list of list of images
Author: amyeroberts
Pull request: yes

# What does this PR do?

Processes a list of list of images e.g. to run the following script:

```py
import requests
from PIL import Image

from transformers import PixtralProcessor, PixtralImageProcessor, AutoTokenizer

url_0 = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_0 = Image.open(requests.get(url_0, stream=True).raw)

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_1 = Image.open(requests.get(url_1, stream=True).raw)

url_2 = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_2 = Image.open(requests.get(url_2, stream=True).raw)

image_processor = PixtralImageProcessor()
tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-1.5-7b-hf")
processor = PixtralProcessor(tokenizer=tokenizer, image_processor=image_processor, patch_size=16)

# single image processing
image_inputs = image_0
prompt = "USER: [IMG]\nWhat's the content of the image? ASSISTANT:"
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)


# single list of images
prompt = ["USER: [IMG][IMG]\nWhat's the difference between these two images? ASSISTANT:"]
image_inputs = [image_0, image_1]
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)

# batched list of images
prompt = ["USER: [IMG][IMG]\nWhat's the difference between these two images? ASSISTANT:", "USER: [IMG]\nWhat's the content of the image? ASSISTANT:"]
image_inputs = [[image_0, image_1], [image_2]]
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33465). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ArthurZucker
release plan for 4.45 ?
Author: jinzhen-lin
Pull request: no
Labels: name: Usage

Hi, is there any release plan for 4.45? We want to use qwen2-vl which is added two weeks ago, and it is difficult to install transformers from github in our internal network. 

Comments:
Hey @jinzhen-lin, likely today :)

> Hey @jinzhen-lin, likely today :)

Great! Thank you!
Broken links in `Agents and tools` documentation
Author: sergiopaniego
Pull request: no

In [Agents and tools](https://huggingface.co/docs/transformers/en/agents) docs, there are some broken link in 

```
Large Language Models (LLMs) trained to perform [causal language modeling](https://huggingface.co/docs/transformers/en/tasks/language_modeling.) can tackle a wide range of tasks
```

The final dot in `https://huggingface.co/docs/transformers/en/tasks/language_modeling.` is causing the issue.

Same in:

```
method which accepts a list of [messages](./chat_templating.)
```

, and

```
it follows the [messages format](./chat_templating.md)
```


Comments:
Clarify what kwargs can be accepted by "AutoModelForCausalLM.from_pretrained()" 
Author: bzr1
Pull request: no
Labels: name: Feature request

### Feature request

i saw in the doc saying:If a configuration is not provided, kwargs will be first passed to the configuration class initialization function ([from_pretrained()](https://huggingface.co/docs/transformers/main/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained)). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

but I can't seem to find the __init__(I found 3 lines with __init__, but no useful info). 

I saw some one added "use_cache:Ture" but don't know how this got handled and set,maybe it's discarded. 

### Motivation

i checked the doc and source code but still confused. 

### Your contribution

I'll keep looking

Comments:
max_length calculation for padding the generation outputs in the Seq2SeqTrainer prediction_step function 
Author: JackCai1206
Pull request: no

I think the calculation for the max_length is wrong when only `max_new_tokens` is provided. 

https://github.com/huggingface/transformers/blob/fe008d6ebea1f5770b740991daeefd9322fa434a/src/transformers/trainer_seq2seq.py#L318-L324

`generated_tokens.shape[-1]` is the sum of the lengths of the input and the generated tokens, so when you have `max_new_tokens`, the actual max length to pad to should be `generation_inputs['input_ids'].shape[-1] + gen_config.max_new_tokens`

Also, since `max_length` and `max_new_tokens` can come from either `gen_config` in the model and `gen_kwargs` provided as kwargs, we might want to handle both cases. 

Proposed change: 
```
        # Retrieves GenerationConfig from model.generation_config
        gen_config = self.model.generation_config
        # in case the batch is shorter than max length, the output should be padded
        max_length = max(
            max(gen_config.max_length or 0, gen_kwargs.get("max_length", 0)),
            generation_inputs['input_ids'].shape[-1] + max(gen_config.max_new_tokens or 0, gen_kwargs.get('max_new_tokens', 0))
        )
        if generated_tokens.shape[-1] < max_length:
            generated_tokens = self._pad_tensors_to_max_len(generated_tokens, max_length)
```

Comments:
cc @gante @zucchini-nlp 

@JackCai1206 Hey!

In this code snippet we're looking at Seq2Seq trainer so the `generated_tokens.shape[-1]` is always shape of the generated tokens. Your assumption would be correct for a Trainer class, but we currently do not support `predict_with_generate` there, it's work in progress.

> Also, since max_length and max_new_tokens can come from either gen_config in the model and gen_kwargs provided as kwargs, we might want to handle both cases.

Right, we should check both `kwargs` and `config`. But from the codebase I see that Trainer internally doesn't pass in any kwargs, so I guess this is the case only if you call `trainer.predict(*args, **kwargs)` on test set after training. We can have a fix for that by checking if `kwargs` has any max length set. In case we get max length set by `kwargs` and `config`, we should prob give priority to the `kwargs` and raise a warning. 

Would you like to open a PR for this? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support Pixtral
Author: fakerybakery
Pull request: no
Labels: name: New model

name: Feature request

### Feature request

Hi,
It would be great to have support for Pixtral!
https://huggingface.co/mistral-community/pixtral-12b-240910
thanks!

### Motivation

Pixtral is a new model released by Mistral AI

### Your contribution

N/A

Comments:
FYI, it was implemented in vllm by Mistral with the PRs https://github.com/vllm-project/vllm/pull/8377 & https://github.com/vllm-project/vllm/pull/8399

Thanks for the request, the pull request will be opened shortly!

You can follow the development here: https://github.com/huggingface/transformers/pull/33449