RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32742
Author: nafiul-araf
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.10.133+-x86_64-with-glibc2.27
- Python version: 3.8.16
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu116 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is my code sample


```python
X_train=list(train['clean_review'])
X_test=list(test['clean_review'])
y_train=list(train['rating'])
y_test=list(test['rating'])

model_name="distilbert-base-uncased"
tokenizer=DistilBertTokenizer.from_pretrained(model_name)
model=DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=2)

X_train_tokenized=tokenizer(X_train, padding=True, truncation=True, max_length=512)
X_test_tokenized=tokenizer(X_test, padding=True, truncation=True, max_length=512)

class Dataset(torch.utils.data.Dataset):    
  def __init__(self, encodings, labels=None):          
    self.encodings=encodings        
    self.labels=labels
     
  def __getitem__(self, idx):
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    if self.labels:
        item["labels"]=torch.tensor(self.labels[idx])
    return item
  def __len__(self):
    return len(self.encodings["input_ids"])


train_dataset=Dataset(X_train_tokenized, y_train)
test_dataset=Dataset(X_test_tokenized, y_test)

args=TrainingArguments(
    output_dir="output",
    evaluation_strategy="steps",
    eval_steps=500,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    seed=0,
    load_best_model_at_end=True,)

trainer=Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    compute_metrics=compute_metrics,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],)
 
trainer.train()
```

### Expected behavior

I'm trying to create a classifier to classify the drug review. The outcome labels are 2: Positive or Negative. For this, I'm using transformer ```DistilBert-base-uncased```. However, when the ```Trainer``` function is run, it produces the following error: 

```RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32742```.

I've searched into this a lot, but I haven't been able in finding a solution. Please help........

Comments:
We won't really be able to help you since your reproducer does not contain any way to get the dataset you're using.
We have a whole chapter around debugging the training pipeline in [our online course](https://huggingface.co/course/chapter8/4?fw=pt).

I had the same error in a similar setting. I installed CUDA 12.0 and now I get this error:

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

What can I do?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.