noamaanMulla-03/transformers-issues huggingface dataset:
Indicies element out of bounds from inclusive range
Author: nogifeet
Pull request: no

### System Info

Hello, we are using the TR-OCR model exported to Onnx. We notice a problem with the large checkpoints for both printed and handwritten; when we run inference using the onnxruntime java library. 

Dataset: IAM handwritten (Lines)

Different behaviours are observed on CPU and GPU:

CPU: (we might get an error like below)

          Status Message: Non-zero status code returned while running the Gather node. Name:'Gather_346' Status Message: 
          indices element out of data bounds, idx=514 must be within the inclusive range [-514,513]
	            at ai.onnxruntime.OrtSession.run(Native Method)
	            at ai.onnxruntime.OrtSession.run(OrtSession.java:301)
	            at ai.onnxruntime.OrtSession.run(OrtSession.java:242)

GPU: We notice that the end token is not generated and the decoder keeps repeating the tokens after a point. 

This is the main problem, usually, the Gather_346 and Gather_320 operators fail and throw data bounds error.

We have also noticed different behaviour when we turn caching on/off.  Note we don't face this problem on the base or small checkpoints but only on the "large" checkpoints. Looking to understand whether this is an onnxruntime issue or hf, please let me know. 

A similar issue was raised in the onnxruntime page: https://github.com/microsoft/onnxruntime/issues/2080

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Export the large checkpoints of TR-OCR.
2. Run a simple example from the IAM dataset image attached.
3. Use this image ![e04-083-00](https://github.com/huggingface/transformers/assets/72322393/f4210a4b-bc28-42e2-a3a7-ff05a9f022ab)
4. Don't use any max_length limit and you will notice that the end token is not generated and the tokens are repeated.
5. Current Output: The edges of the transoms should be bevelled to be edges to the edges of the

### Expected behavior

Current Output: The edges of the transoms should be bevelled to be edges to the edges of the
Expected Output: The edges of the transoms should be bevelled to 

Comments:
Hey! THanks for opening an issue, but without an isolated reproducible snippet + a traceback that shows this is an issue from transfromers I am not sure how we can help! 🤗 

@nogifeet  hi, did you have solve this problem? i meet similar problem when run trocr_onnx with onnxruntime(c++, cpu), can you share your solution for reference? Thanks.

@linglongxian  
https://github.com/huggingface/transformers/issues/28550
Forbid `PretrainedConfig` from saving `generate` parameters; Update deprecations in `generate`-related code 🧹 
Author: gante
Pull request: yes

# What does this PR do?

See title :) I did a quick scan and update over things that fulfilled the following criteria:
- had deprecation messages
- were related to what I've been working on

______________________________________________

Key changes: 
1. Saving a model config with parameters meant to be in the generation config is no longer allowed
2. However, at model saving time (which saves related configs), move those parameters to the generation config before potentially complaining about it. This is to prevent fine-tuning jobs from failing after significant compute $ is spent.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32659). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(@amyeroberts don't re-review yet, I'm going through failed tests -- mostly cases where we are explicitly setting `generate` arguments in `model.config`, precisely what we don't want users to do)

@gante I'm going to unsubscribe to avoid notifications for each push. Just ping / @ me to let me know when it's ready for review again! 

@amyeroberts Ready 🙌 

Notes:
1. saving a config with custom generate parameters now creates an exception -> all testers that relied on setting custom generation parameters to work around edge cases failed [because we pass the tester args to the model config] -> this PR fixes edge cases/updates tests 💪 
2. touching many files triggers a lot of test runs. A few unrelated broken tests were found (and fixed) in the process
3. I've added GH comments on details that are both not useful for the future codebase AND not obvious why the change was made
4. an unrelated pipeline test is failing due to timeout, triggering a new run is not making CI green. I'll need you to merge when you're happy with the PR :D 

CI was failing, rebased with main to check if that was the issue

EDIT: root cause identified, it's unrelated to this PR and being fixed (the CI image needs an update)
DeviceGuard added to use Deformable Attention more safely on multi-GPU
Author: DonggeunYu
Pull request: yes

# What does this PR do?

I got zero results from deformable attention when multi-GPU inference.
However, this issue happens only in my environment.

The DeviceGuard has been added to ensure that calculations are performed only on the value device. Thus, my problem has been resolved.

Fixes [# (issue)](https://github.com/huggingface/transformers/issues/31399)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:

- vision models: @amyeroberts

Comments:
@DonggeunYu We had an issue with our test runners. Could you push an empty commit to trigger a re-run and make everything green? 

> @DonggeunYu We had an issue with our test runners. Could you push an empty commit to trigger a re-run and make everything green?

It has been completed!
Update Jinja docs with new functions and general cleanup
Author: Rocketknight1
Pull request: yes

This PR documents the new `strftime_now()` function, and the `raise_exception()` function, which wasn't documented before.

It also generally cleans up the template writing docs a bit, now that we have something approaching a recommended 'style'!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33097). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix bad links in quantization overview table
Author: rbrugaro
Pull request: yes

Doc fix: fix bad links in quantization overview table
@stevhliu

## Before submitting
- [ X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33084). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu that's weird! main branch links don't work for me and when i tested with .md in my fork then the issue is fixed. The last two rows (FBGEMM_FP8, torchao) of the table had the .md before I added the other rows and those were the only 2 links working. 
![Screenshot 2024-08-23 at 9 39 55 AM](https://github.com/user-attachments/assets/53340240-1a14-4229-b579-8e5731ad49b2)


Yeah I recommend viewing the docs on the docs [site](https://huggingface.co/docs/transformers/main/en/quantization/overview) instead of on GitHub!

You're right that adding `.md` will correctly link on GitHub, but it won't work on our docs site (as you can see when you check out the doc build preview [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33084/en/quantization/overview)).
updates_readmedocs
Author: Swastik-Swarup-Dash
Pull request: yes

# This PR is done for the update of Quick_tour under readme and added some piece of code for Enhanced Output Formatting
ie 
```python
from transformers import pipeline
# Allocate a pipeline for sentiment-analysis
classifier = pipeline('sentiment-analysis')
result = classifier('We are very happy to introduce pipeline to the transformers repository.')
# Enhanced output formatting
label = result[0]['label']
score = result[0]['score']
print(f"Sentiment: {label}\nScore: {score:.4f}")
```
 


Comments:
Create the FSDP Plugin in the trainer explicitly rather wait for it to be setup by accelerate
Author: winglian
Pull request: yes
Labels: name: WIP

# What does this PR do?

See the corresponding PR here https://github.com/huggingface/accelerate/pull/3033. Ultimately I want to see device_mesh support, but this brings the fsdp_plugin to parity with the deepspeed plugin in training args.

@ArthurZucker @muellerzr 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
This is the corresponding PR to https://github.com/huggingface/accelerate/pull/3033 that will enable passing a device_mesh
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33085). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🐛 Add device parameter to model variable
Author: kadirnar
Pull request: yes

# What does this PR do?
Fixes  https://github.com/huggingface/transformers/issues/31987#issuecomment-2231450210

Error Message:
```
skipping cudagraphs due to skipping cudagraphs due to cpu device. Found from : 
```
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sanchit-gandhi

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
the attention output from llama2 generate differs from other llama models
Author: manyuanbin
Pull request: no

### System Info

transformers: 4.41.2
os: ubuntu
python: 3.10.14

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "path/to/vicuna/model"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Ensure the model returns attentions
model.config.output_attentions = True
model.config.return_dict = True


input_text = "Once upon a time"
input_ids = tokenizer(input_text, return_tensors='pt').input_ids


outputs = model.generate(input_ids, max_length=50, output_attentions=True, return_dict_in_generate=True)

generated_ids = outputs.sequences
attention_weights = outputs.attentions


generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(generated_text)

# Print the shape of the attention weights for each layer
for layer_idx, layer_attention in enumerate(attention_weights):
    print(f"Layer {layer_idx} attention shape: {layer_attention.shape}")

### Expected behavior

<img width="815" alt="image" src="https://github.com/user-attachments/assets/7bcb760b-68cb-43d7-8092-93be4cb51e69">


Comments:
model_name = 'lmsys/vicuna-7b-v1.1'

expected:

attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

hey! Can you update to a latest version of transformers and provide a proper reproduction script? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`attention_mask` must be in the same device as model?
Author: fahadh4ilyas
Pull request: no

### System Info

Transformer version: 4.42.4
Platform: Ubuntu
Python version: 3.10.14

### Who can help?

@ArthurZucker @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Here is an example code:

```python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B', device_map='auto', attn_implementation='flash_attention_2', token='you-hf-token')

with torch.inference_mode():
    model.generate(input_ids=torch.tensor([[1,2]], dtype=torch.long, device='cuda:0'), attention_mask=torch.tensor([[1,1]], dtype=torch.long))
```

### Expected behavior

Model generate without issue. I never have this issue when I'm using old transformers. But, now this error shows up

```python
    114 device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
    115 with torch.autocast(device_type=device_type, enabled=False):
--> 116     freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
    117     emb = torch.cat((freqs, freqs), dim=-1)
    118     cos = emb.cos()

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)
```

Comments:
Hi @fahadh4ilyas 👋 

Pytorch doesn't automatically move tensors across devices -- the model and all its inputs must be manually moved to the desired device. In your example, `attention_mask` doesn't have the `device` argument.

See the basic generation example [here](https://huggingface.co/docs/transformers/llm_tutorial#generate-text) 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Make special image tokens attribute of tokenizer
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR adds three most commonly used special tokens in VLMs as attributes to the tokenizer, following on https://github.com/huggingface/transformers/pull/31534#discussion_r1672697899. Now we don't have to add as kwargs to init/hardcode the special tokens in `processing` code. We can call simply `self.tokenizer.image_token`.

The PR only adds an attribute in tokenizer's code, all the changes for specific VLMs will go on another PR. I believe it's only Idefics models that uses special tokens within processing and it will require BC.

Other models will benefit from this feature after VLM processor refactor is merged.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31967). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for opening this PR @zucchini-nlp! 

I'm not sure this is something we want to add as a property for all tokenizers: they predominantly work with text. In this regard, it makes sense to talk about "end of sentence" as pretty much all tokenizers work with text in sentence structures.  However image tokens aren't a general properly of text data (what does it mean for a bert tokenizer to have an image token?). 

I would instead try to restructure such that these are only added to certain tokenizers e.g. bundle as a mixin which we can selectively add. 

I also suggest adding an `image_token_ids` attribute that contains all the VQVAE image tokens (excluding the start & end marker tokens)

@amyeroberts I see what you mean, but going your way will require some indicator that the tokenizer is loaded from VLM because we load the same tokenizer classes as LMs. I can give it a try and add an indicator through config file, and depending on that add new special token mixin. Will ping you when it's ready

@leloykun in case it goes to its own VLMSpecialTokensMixin, I can add `image_token_ids`, sure

@zucchini-nlp OK - I'm open to other solutions - a mixin might not be a good idea! I just want to avoid leakiness in our abstractions. 

Thanks for this @zucchini-nlp btw!

Currently I have to do something this:
![image](https://github.com/user-attachments/assets/3088b137-7c8b-4672-842f-c76205ed38b9)
which is a PITA

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Load llama-2-70b model need too much CPU memory
Author: JuiceLemonLemon
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <yes>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download Alpaca code.
https://github.com/tatsu-lab/stanford_alpaca

2. Run the command to load Llama2-70b model
torchrun --nproc_per_node=8 --master_port=29505 train.py --model_name_or_path ../models/Llama-2-70b-hf/ --data_path ./alpaca_data.json --bf16 True --output_dir ./output --num_train_epochs 1 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 1 --evaluation_strategy "no" --save_strategy "steps" --save_steps 2000 --save_total_limit 1 --learning_rate 1e-5 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --fsdp "full_shard auto_wrap" --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' --tf32 True --report_to none

3. The model cannot be loaded successfully, because the CPU memory has been used > 1T, then the server hangs up.

### Expected behavior

It used too much CPU memory when loading Llama-2-70b model on 8 GPUs.
How to fix this issue?

Comments:
Hi @JuiceLemonLemon, thanks for opening this issue! 

Without knowing the GPUs your running on, it'd be hard to say what's reasonable in terms of CPU offloading utilization. Have you inspected with tools like `nvidia-smi` and `top` to see the memory usage and ensuring the model is loading as expected? 

As the command comes from the https://github.com/tatsu-lab/stanford_alpaca repo, I'd suggest opening an issue on this repo, and they'll have more knowledge and experience with the expected behaviour and possible gotchas

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.