CUDA error at 'cublasSgemm' when using the pretrained BERT
Author: hahmyg
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [v] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [v] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

My code:
```
torch.cuda.set_device(0)
        
        sequence_output, pooled_output = self.bert(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
        sequence_output = self.dropout(sequence_output)
        pooled_output = self.dropout(pooled_output)
        
        sense_logits = self.sense_classifier(pooled_output)
        arg_logits = self.arg_classifier(sequence_output) 
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 16.04
* Python version: 3.6.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? yes (1080)
* Distributed of parallel setup ? (no)
* Any other relevant information:

## Additional context

My error message is here:
When my model is on the training process, the error message occurs within some hours. 

I'm sure my problem is exactly same with this link: https://github.com/huggingface/transformers/issues/1760

however, update pytorch and transformers doesn't help.

```
  File "training_finetuning.py", line 144, in train
    token_type_ids=b_token_type_ids, attention_mask=b_input_masks)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "../kaiser/src/modeling.py", line 53, in forward
    sequence_output, pooled_output = self.bert(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 738, in forward
    encoder_attention_mask=encoder_extended_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 384, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 355, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 309, in forward
    self_outputs = self.self(hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 230, in forward
    mixed_value_layer = self.value(hidden_states)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
```


<!-- Add any other context about the problem here. -->


Comments:
In [this](https://github.com/pytorch/pytorch/issues/24018) thread on PyTorch's GitHub, they said that this bug has been fixed. In more details, _"this bug was solved in cublas 10.2.0.186. The latest public version of cublas is 10.2.1.243 that was released with CUDA 10.1 Update 2."_

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.