'RuntimeError: CUDA error' is occured when encoding text with pre-trained model on cuda
Author: jcw521
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): I am trying to use Bert encoder

Language I am using the model on (English, Chinese....): English

There are two problems when I use BERT for encoding text to vectors
Here is Error log

First,
```
File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
    embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
    head_mask=head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
    attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1374, in linear
    output += bias
RuntimeError: CUDA error: device-side assert triggered
```


Second,
```
File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
    embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
    head_mask=head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
    attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
```


These two errors are coming from BERT encode :(
here is my code
```
class Text2Vec(object):
     def __init__(self, args):
          self.max_lens = args.max_lens
          self.tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
          self.encoder = self.get_encoder('bert-base-multilingual-cased', args.num_layers)
          self.encoder.to('cuda')
          self.pad_token = self.tokenizer.convert_tokens_to_ids(self.tokenizer.pad_token)

     def get_encoder(self, encoder_type, num_layers):
          model = AutoModel.from_pretrained(encoder_type)
          model.eval()
          model.encoder.layer = torch.nn.ModuleList([layer for layer in model.encoder.layer[:num_layers]])
          return model

     def padding(self, arr, pad_token):
          lens = torch.LongTensor([len(a) for a in arr])
          padded = torch.ones(len(arr), self.max_lens) * pad_token
          mask = torch.zeros(len(arr), self.max_lens)
          for i, a in enumerate(arr):
               padded[i, :lens[i]] = torch.tensor(a)
               mask[i, :lens[i]] = 1
          return padded, mask

     def encode(self, text):
          text = sent_tokenize(text)
          tokens = [self.tokenizer.encode(a, add_special_tokens=True) for a in text]
          padded_tokens, masks = self.padding(tokens, self.pad_token)

          padded_tokens = padded_tokens.to('cuda')
          masks = masks.to('cuda')

          self.encoder.eval()
          with torch.no_grad():
               embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]

          embed = []
          masks = list(torch.split(masks, 1, dim=0))
          embeddings = list(torch.split(embeddings, 1, dim=0))
          for embedding, mask in zip(embeddings, masks):
               masked_embed = embedding * mask.squeeze()[:, None]
               embed.append(masked_embed)

          return torch.cat(embed, dim=0)
```

What am I missing? please advise to me!!

## Environment
* OS: ubuntu 16.04
* Python version: Python 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? titan-xp
* Distributed of parallel setup ? not yet
* Any other relevant information:


Comments:
> ##  Bug
> Model I am using (Bert, XLNet....): I am trying to use Bert encoder
> 
> Language I am using the model on (English, Chinese....): English
> 
> There are two problems when I use BERT for encoding text to vectors
> Here is Error log
> 
> First,
> 
> ```
> File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
>     embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
>     head_mask=head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
>     layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
>     attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
>     self_outputs = self.self(input_tensor, attention_mask, head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
>     mixed_query_layer = self.query(hidden_states)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
>     return F.linear(input, self.weight, self.bias)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1374, in linear
>     output += bias
> RuntimeError: CUDA error: device-side assert triggered
> ```
> 
> Second,
> 
> ```
> File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
>     embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
>     head_mask=head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
>     layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
>     attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
>     self_outputs = self.self(input_tensor, attention_mask, head_mask)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
>     mixed_query_layer = self.query(hidden_states)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
>     return F.linear(input, self.weight, self.bias)
>   File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear
>     output = input.matmul(weight.t())
> RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
> ```
> 
> These two errors are coming from BERT encode :(
> here is my code
> 
> ```
> class Text2Vec(object):
>      def __init__(self, args):
>           self.max_lens = args.max_lens
>           self.tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
>           self.encoder = self.get_encoder('bert-base-multilingual-cased', args.num_layers)
>           self.encoder.to('cuda')
>           self.pad_token = self.tokenizer.convert_tokens_to_ids(self.tokenizer.pad_token)
> 
>      def get_encoder(self, encoder_type, num_layers):
>           model = AutoModel.from_pretrained(encoder_type)
>           model.eval()
>           model.encoder.layer = torch.nn.ModuleList([layer for layer in model.encoder.layer[:num_layers]])
>           return model
> 
>      def padding(self, arr, pad_token):
>           lens = torch.LongTensor([len(a) for a in arr])
>           padded = torch.ones(len(arr), self.max_lens) * pad_token
>           mask = torch.zeros(len(arr), self.max_lens)
>           for i, a in enumerate(arr):
>                padded[i, :lens[i]] = torch.tensor(a)
>                mask[i, :lens[i]] = 1
>           return padded, mask
> 
>      def encode(self, text):
>           text = sent_tokenize(text)
>           tokens = [self.tokenizer.encode(a, add_special_tokens=True) for a in text]
>           padded_tokens, masks = self.padding(tokens, self.pad_token)
> 
>           padded_tokens = padded_tokens.to('cuda')
>           masks = masks.to('cuda')
> 
>           self.encoder.eval()
>           with torch.no_grad():
>                embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
> 
>           embed = []
>           masks = list(torch.split(masks, 1, dim=0))
>           embeddings = list(torch.split(embeddings, 1, dim=0))
>           for embedding, mask in zip(embeddings, masks):
>                masked_embed = embedding * mask.squeeze()[:, None]
>                embed.append(masked_embed)
> 
>           return torch.cat(embed, dim=0)
> ```
> 
> What am I missing? please advise to me!!
> 
> ## Environment
> * OS: ubuntu 16.04
> * Python version: Python 3.7.4
> * PyTorch version: 1.3.0
> * PyTorch Transformers version (or branch): how do I check PyTorch Transformers version?
> * Using GPU ? titan-xp
> * Distributed of parallel setup ? not yet
> * Any other relevant information:

In order to check the Transformers version you can insert this line in your code:

```
import transformers
print(transformers.__version__)
```

Hi! There was a few things to change on my side to get your code to run.

1 - when doing `torch.ones(...)`, it initializes a tensor of dtype `float` while it should be of dtype `long`, i had to change the following line in Text2Vec.padding():

```py
padded = torch.ones(len(arr), self.max_lens, dtype=torch.int64) * pad_token
```

2 - it crashed on my side at first because I was on the pypi release which has a bug related to the initialization of `token_type_ids` when they're not given to the model (they're not put on GPU). I had to install from master with the following command:

```
pip install git+https://github.com/huggingface/transformers
```

Once you've done that it should work fine!