How to fine tune facebook/esm2_t33_650M_UR50D
Author: Admire7494
Pull request: no

### System Info

How to fine tune facebook/esm2_t33_650M_UR50D？It's too big and the model.half() couldn't work. Besids, i always met the error : CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc). Is it possible that the model in the huggingface is wrong?
The following is the script:
from os.path import join
import os
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data
import transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer
from datasets import Dataset,load_metric
from sklearn.model_selection import train_test_split
#os.environ['CUDA_VISIBLE_DEVICES'] = '1'
CURRENT_DIR = os.getcwd()
check_point = join(CURRENT_DIR,"esm1b_t33_650M_UR50S")

#Data processing
def process_tsv(file):
    sequences = list()
    labels = list()
    df = pd.read_csv(file,sep="\t")
    for ind in df.index:
        sequences.append(df["sequence"][ind])
        labels.append(df["label"][ind])

    return sequences,labels


def tokenize_add_label(sequences, labels, tokenizer):
    """This function takes sequences and labels creates a Dataset containing tokenized sequences and add labels to it

       args:
           sequences (str): a list of sequences
           labels (int): a list of labels
           tokenizer : a pre-trained tokenizer

       return:
            Dataset: tokenized sequences and associated labels)"""
    sequences_tokenized = tokenizer(sequences, padding=True, truncation=True)
    sequences_tokenized = torch.float16(sequences_tokenized)
    labels = torch.tensor(labels)
    labels = labels.long()
    sequences_dataset = Dataset.from_dict(sequences_tokenized)
    sequences_dataset = sequences_dataset.add_column("labels", labels)

    return sequences_dataset

sequences,labels = process_tsv(join(CURRENT_DIR,"example.tsv"))
tokenizer = AutoTokenizer.from_pretrained(check_point)
sequences_dataset = tokenize_add_label(sequences,labels,tokenizer)
num_labels = max(labels)+1
model = AutoModelForSequenceClassification.from_pretrained(check_point,num_labels=num_labels)
#device = "cuda" if torch.cuda.is_available() else "cpu"
#model.to(device)
model.cuda()


#model = model.half()

#model.enable_input_require_grads()
model_name = check_point.split("/")[-1]
trainer_dir = f"{model_name}-finetuned-model_esm-1b_on_7beta"
if not os.path.exists(trainer_dir):
    os.mkdir(trainer_dir)

batch_size = 1
training_args = transformers.TrainingArguments(
    output_dir=trainer_dir,          # output directory
    overwrite_output_dir=True,
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=batch_size,  # batch size per device during training
    per_device_eval_batch_size=batch_size,   # batch size for evaluation
    learning_rate=2e-5,
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir=trainer_dir,            # directory for storing logs
    logging_steps=10,
    load_best_model_at_end=True,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=1,
    metric_for_best_model="accuracy",
    greater_is_better=True,
    disable_tqdm=True,
    gradient_accumulation_steps = 2,
    gradient_checkpointing=True

    )

metric = load_metric(join(CURRENT_DIR,"metrics","accuracy/accuracy.py"))

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    print("logits",logits)
    print("labels",labels)
    predictions = np.argmax(logits, axis=-1)
    print("predictions",predictions)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model = model,
    args = training_args,
    train_dataset=sequences_dataset,
    eval_dataset=sequences_dataset,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,

)

model.config.problem_type
trainer.train()
trainer.state.log_history

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
Some weights of EsmForSequenceClassification were not initialized from the model checkpoint at /home/wangmuqiang/fine_tune_esm2/esm1b_t33_650M_UR50S and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.bias', 'classifier.out_proj.weight', 'classifier.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/home/wangmuqiang/fine_tune_esm2/fine_tune_esm1b_7beta.py:87: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 馃 Evaluate: https://huggingface.co/docs/evaluate
  metric = load_metric(join(CURRENT_DIR,"metrics","accuracy/accuracy.py"))
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [102,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [78,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1699449181081/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [62,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/home/wangmuqiang/fine_tune_esm2/fine_tune_esm1b_7beta.py", line 108, in <module>
    trainer.train()
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/transformers/trainer.py", line 1854, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/transformers/trainer.py", line 2737, in training_step
    self.accelerator.backward(loss)
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/accelerate/accelerator.py", line 1905, in backward
    loss.backward(**kwargs)
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/autograd/function.py", line 288, in apply
    return user_fn(self, *args)
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 288, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/home/wangmuqiang/.conda/envs/esm/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`


### Expected behavior

the script that successfully ran in RTX 3090

Comments:
Hi @Admire7494, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.