AcceleratorError: CUDA error: device-side assert triggered
Author: monochandan
Pull request: no
Labels: name: bug

### System Info

Python 3.11.13
torch==2.6.0+cu124
transformers==4.55.2
bitsandbytes==0.47.0
peft==0.17.0
accelerate==1.10.0
numpy==1.26.4
scipy==1.14.1

GPU:  NVIDIA A100-SXM4-40GB 
CUDA Version: 12.4

# Dataset
Train
{'text': Value('string'), 'label': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8'))}
Validation
{'text': Value('string'), 'label': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8'))}
Test
{'text': Value('string'), 'label': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8'))}


### Who can help?

# Code

```
model = "FacebookAI/xlm-roberta-base"
original_model_xlmroberta = AutoModelForSequenceClassification.from_pretrained(
    model,
    num_labels = 2,
    )
tokenizer_xlmroberta = AutoTokenizer.from_pretrained(model_multilingual_1)
if tokenizer_xlmroberta.pad_token is None:
    tokenizer_xlmroberta.pad_token = tokenizer_xlmroberta.eos_token
tokenizer_xlmroberta.padding_side = "right"

def convert_label(example):
    example["label"] = 1 if example["label"] == "Yes" else 0
    return example

train_dataset = train_dataset.map(convert_label) 
dev_dataset = dev_dataset.map(convert_label)
test_dataset = test_dataset.map(convert_label)
test_dataset_d = test_dataset_d.map(convert_label)

train_dataset = preprocess_dataset(tokenizer_xlmroberta, max_length, train_dataset, seed)
dev_dataset = preprocess_dataset(tokenizer_xlmroberta, max_length, dev_dataset, seed)
test_dataset =  preprocess_dataset(tokenizer_xlmroberta, max_length, test_dataset, seed)# WL
test_dataset_d = preprocess_dataset(tokenizer_xlmroberta, max_length, test_dataset_d, seed)

def compute_metrics(eval_pred):
  logits, labels = eval_pred
  predictions = np.argmax(logits, axis = -1)
  f1 = f1_score(labels, predictions, average = "binary")
  return {"f1": f1}


output_dir = f'/content/drive/MyDrive/'

args = TrainingArguments(
    output_dir = output_dir,
    weight_decay=0.22511642804764023,
    warmup_ratio=0.12890328790683203,
    adam_beta1=0.9348819720458172,
    adam_beta2=0.9285998615546803,
    adam_epsilon=1.9972958061508847e-07,
    max_grad_norm=4.222172817940239,
    gradient_accumulation_steps=2,
    # max_steps=712,
    do_train = True,
    do_eval = True,
    lr_scheduler_type='polynomial',
    warmup_steps=488,
    # metric_for_best_model = "eval_f1",
    optim='paged_adamw_32bit',
    learning_rate = 2.1106713456200193e-05,
    num_train_epochs = 10,
    logging_dir = "./logs/",
    logging_strategy = "epoch",
    eval_strategy = "epoch",
    save_strategy = "epoch",
    label_names = ["label"],
    # load_best_model_at_end = True,
    save_total_limit = 2,
    # use_cpu = False
    per_device_train_batch_size = 8,
    per_device_eval_batch_size = 8
)

original_model_xlmroberta.config.use_cache = False

trainer = Trainer(
    model = original_model_xlmroberta,
    train_dataset = train_dataset,
    eval_dataset = dev_dataset,
    args = args,
    compute_metrics = compute_metrics,
    data_collator = DataCollatorWithPadding(tokenizer = tokenizer_xlmroberta, padding=True)
)

trainer.train()


```




### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

# Some time Error (after initializing the Trainer):

<img width="1080" height="960" alt="Image" src="https://github.com/user-attachments/assets/35cf7ab5-c986-4cd1-99e7-324038cc9530" />

# Sometimes, the Same Error Occurs after Starting Training on the Trainer:

<img width="1080" height="960" alt="Image" src="https://github.com/user-attachments/assets/be5748f9-e3ce-4fb1-bf34-ce237ceffc15" />

### Expected behavior

Train the model.

Comments:
This error is often caused by passing an index to an embedding that exceeds the size of the embedding layer. For example, passing a token with index `1025` to a model with a vocabulary size of `1024`. Alternatively, the same thing can happen with `position_ids`, if you pass a sequence longer than the model position IDs tensor.

I notice in your code sample that you're tokenizing with a multilingual model, but passing the output to the original model. Mixing and matching tokenizers and models generally doesn't work, and is very likely to generate out-of-range token IDs like this.

Closing because I doubt there's actually a `transformers` bug here!

I encountered the same problem. Is there a solution?