RuntimeError: CUDA driver error: invalid argument
Author: murthyrudra
Pull request: no

### System Info

- `transformers` version: 4.29.1
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.13
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@gante

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2', device_map = 'auto')

tokenizer.padding_side = "left"
tokenizer.truncation_side = "left"

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.pad_token_id = tokenizer.eos_token_id

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

if torch.cuda.is_available():
    model = model.to(device=device)

prompt = "What is the capital of India?"
input_ids = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True, add_special_tokens=False).input_ids.to(
    dtype=torch.long, device=device
)

max_new_tokens = 10

model.eval()
with torch.no_grad():
    generated_ids = model.generate(
        input_ids,
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.eos_token_id,
    )
    preds = [
        tokenizer.decode(
            g, skip_special_tokens=True, clean_up_tokenization_spaces=True
        ).strip()
        for g in generated_ids
    ]
```

Running the above script produces the following error

```
Using pad_token, but it is not set yet.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/scripts/temp.py:28 in <module>                                                             │
│                                                                                                  │
│   25                                                                                             │
│   26 model.eval()                                                                                │
│   27 with torch.no_grad():                                                                       │
│ ❱ 28 │   generated_ids = model.generate(                                                         │
│   29 │   │   input_ids,                                                                          │
│   30 │   │   max_new_tokens=max_new_tokens,                                                      │
│   31 │   │   pad_token_id=tokenizer.eos_token_id,                                                │
│                                                                                                  │
│ /home/rudra/miniconda3/envs/FM/lib/python3.9/site-packages/torch/utils/_c                        │
│ ontextlib.py:115 in decorate_context                                                             │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /home/rudra//miniconda3/envs/FM/lib/python3.9/site-packages/transformers/g                       │
│ eneration/utils.py:1515 in generate                                                              │
│                                                                                                  │
│   1512 │   │   │   │   )                                                                         │
│   1513 │   │   │                                                                                 │
│   1514 │   │   │   # 11. run greedy search                                                       │
│ ❱ 1515 │   │   │   return self.greedy_search(                                                    │
│   1516 │   │   │   │   input_ids,                                                                │
│   1517 │   │   │   │   logits_processor=logits_processor,                                        │
│   1518 │   │   │   │   stopping_criteria=stopping_criteria,                                      │
│                                                                                                  │
│ /home/rudra//miniconda3/envs/FM/lib/python3.9/site-packages/transformers/g                       │
│ eneration/utils.py:2385 in greedy_search                                                         │
│                                                                                                  │
│   2382 │   │   │   # if eos_token was found in one sentence, set sentence to finished            │
│   2383 │   │   │   if eos_token_id_tensor is not None:                                           │
│   2384 │   │   │   │   unfinished_sequences = unfinished_sequences.mul(                          │
│ ❱ 2385 │   │   │   │   │   next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_te  │
│   2386 │   │   │   │   )                                                                         │
│   2387 │   │   │   │                                                                             │
│   2388 │   │   │   │   # stop when each sentence is finished                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: CUDA driver error: invalid argument

```

### Expected behavior

No error and some text generated by the model. Works perfectly on CPU.

Comments:
I can reproduce this with the much simpler:

```
import transformers
gpt2_generator = transformers.pipeline('text-generation', model='gpt2', device=1)
sentences = gpt2_generator("To be honest, neural networks", do_sample=True, top_k=50, temperature=0.6, max_length=128, num_return_sequences=3)
for sentence in sentences:
  print(sentence["generated_text"])
  print("="*50)
```

This is example code from the transformers docs and should "just work". It feels like an environment issue, but the error is super indistinct. `CUDA_LAUNCH_BLOCKING=1` does not produce a clearer error. I've tried in a fresh environment, with different versions of transformers, torch, and cuda to no avail. 

Hey @blucz @murthyrudra 👋 

I am unable to reproduce the issue on my end, which seems to indicate this is an environment issue 🤔 

This is my current env:
```
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.10.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (gpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.