CUBLAS_STATUS_INVALID_VALUE when generating with OPT models
Author: jchwenger
Pull request: no

### System Info

Hi,

I'm encountering an error when trying to do text generation with the OPT models. Here are the specs and the error, and below the steps to reproduce.

Ubuntu 18.04

Conda env:
torch                     1.13.1                   pypi_0    pypi
transformers              4.26.1                   pypi_0    pypi
Python 3.9.7

The error:

```python
---------------------------------------------------------------------------                                                                                                                                                          
RuntimeError                              Traceback (most recent call last)                                                                                                                                                          
<ipython-input-1-45f0d829e48e> in <module>                                                                                                                                                                                           
      6 prompt = "Hello, I am conscious and"                                                                                                                                                                                         
      7 input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()                                                                                                                                                          
----> 8 generated_ids = model.generate(input_ids)                                                                                                                                                                                    
      9 text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)                                                                                                                                                       
     10 print(text)                                                                                                                                                                                                                  
                                                                                                                                                                                                                                     
~/anaconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)                                                                                                                  
     25         def decorate_context(*args, **kwargs):                                                                                                                                                                               
     26             with self.clone():                                                                                                                                                                                               
---> 27                 return func(*args, **kwargs)                                                                                                                                                                                 
     28         return cast(F, decorate_context)                                                                                                                                                                                     
     29                                                                                                                                                                                                                              
                                                                                                                                                                                                                                     
~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/generation/utils.py in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, **kwargs)               
   1389                                                                                                                                                                                                                              
   1390             # 11. run greedy search                                                                                                                                                                                          
-> 1391             return self.greedy_search(                                                                                                                                                                                       
   1392                 input_ids,                                                                                                                                                                                                   
   1393                 logits_processor=logits_processor,                                                                                                                                                                           
                                                                                                                                                                                                                                     
~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/generation/utils.py in greedy_search(self, input_ids, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_s
tates, output_scores, return_dict_in_generate, synced_gpus, **model_kwargs)                                                                                                                                                          
   2177                                                                                                                                                                                                                              
   2178             # forward pass to get next token                                                                                                                                                                                 
-> 2179             outputs = self(                                                                                                                                                                                                  
   2180                 **model_inputs,                                                                                                                                                                                              
   2181                 return_dict=True,                                                                                                                                                                                            
                                                                                                                                                                                                                                     
~/anaconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)                                                                                                                  
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks                                                                                                             
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):                                                                                                                                                      
-> 1194             return forward_call(*input, **kwargs)                                                                                                                                                                            
   1195         # Do not call functions when jit is used                                                                                                                                                                             
   1196         full_backward_hooks, non_full_backward_hooks = [], []                                                                                                                                                                

~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py in forward(self, input_ids, attention_mask, head_mask, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)                                    
    930 
    931         # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
--> 932         outputs = self.model.decoder(
    933             input_ids=input_ids,
    934             attention_mask=attention_mask,

~/anaconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py in forward(self, input_ids, attention_mask, head_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states,
 return_dict)                                                                                                     
    695             else:                                
    696                                                                                                                                                                                                                              
--> 697                 layer_outputs = decoder_layer(                                                                                                                                                                               
    698                     hidden_states,                                                                                                                                                                                           
    699                     attention_mask=attention_mask,                                                                                                                                                                           
                                                                                                                  
~/anaconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):                                                                                                                                                      
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used                                                                                                                                                                             
   1196         full_backward_hooks, non_full_backward_hooks = [], []                
                                                         
~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py in forward(self, hidden_states, attention_mask, layer_head_mask, output_attentions, use_cache, past_key_value)
    324                                                                                                           
    325         # Self Attention                                                                                                                                                                                                     
--> 326         hidden_states, self_attn_weights, present_key_value = self.self_attn(                                                                                                                                                
    327             hidden_states=hidden_states,                                                                                                                                                                                         328             past_key_value=past_key_value,
                                                                                                                                                                                                                                     ~/anaconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):                                   
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used                                                                                                                                                                             
   1196         full_backward_hooks, non_full_backward_hooks = [], []
                                                                                                                                                                                                                                     ~/anaconda3/envs/torch/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py in forward(self, hidden_states, key_value_states, past_key_value, attention_mask, layer_head_mask, output_attentions)
    206                                                                                                           
    207         src_len = key_states.size(1)
--> 208         attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))                                
    209                                                                                                           
    210         if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):                                                                                                                                                                             
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmStridedBatchedExFix( handle, opa, opb, m, n, k, (void*)(&falpha), a, CUDA_R_16F, lda, stridea, b, CUDA_R_16F, ldb, strideb, (void*)(&fbeta), c, CUDA_R_
16F, ldc, stridec, num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)`   
```

### Who can help?

@ArthurZucker, @younesbelkada, @sgugger, @sgugger, @muellerzr, @gante

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In an environment with PyTorch and Transformers, open an IPython console and paste the example [from the documentation](https://huggingface.co/facebook/opt-66b) (here with a 1.3b model, but I first tried the 6.7b):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b", torch_dtype=torch.float16).cuda() # I also tried without half-precision
# the fast tokenizer currently does not work correctly
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b", use_fast=False)
prompt = "Hello, I am conscious and"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
generated_ids = model.generate(input_ids)
text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print(text)
```

### Expected behavior

The model should do inference and generate text out of the box.

Any help would be greatly appreciated, thanks for reading!

Comments:
Hey @jchwenger 👋 

I was able to run the script you shared without bugs on my end. Looking at other threads online, it may be due to an incorrect environment on your end -- check this thread from the comment I link [here](https://discuss.pytorch.org/t/runtimeerror-cuda-error-cublas-status-invalid-value-when-calling-cublassgemm-handle-opa-opb-m-n-k-alpha-a-lda-b-ldb-beta-c-ldc/124544/18) 🤗 

Hi @gante,

Oh, I see, thanks a lot for this reference, I'll investigate.

Hi again @gante, thanks for the help, I had a PyTorch/Cuda mismatch, after reinstall using the command from the website l it works!