Bert forward reports error on GPU; but runs fine on CPU
Author: RayLei
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I would like to encode articles using Bert. My code runs fine on CPU, but failed on GPU. The GPU is a on a remote server.

```
def assign_gpu(token):
    token_tensor = token['input_ids'].to('cuda')
    token_typeid = token['token_type_ids'].to('cuda')
    attention_mask = token['attention_mask'].to('cuda')
    output = {'input_ids': token_tensor,
              'token_type_ids': token_typeid,
              'attention_mask': attention_mask}
    return output

bs = 16
data_dl = DataLoader(PatentDataset(df), batch_size=bs, shuffle=False)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased/')
model = BertModel.from_pretrained('bert-base-uncased/')
inputs_list = []
for a in data_dl:
     inputs_list.append(tokenizer.batch_encode_plus(a, pad_to_max_length=True, return_tensors='pt'))

# GPU part 
model.cuda()
model.eval()
out_list = []
with torch.no_grad():
    for i, inputs in enumerate(inputs_list):
        inputs = assign_gpu(inputs)
        output = model(**inputs)[0][:, 0, :]
        out_list.append(output)
```
The error message is 

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f6979d077f60> in <module>
      6     for i, inputs in enumerate(inputs_list):
      7         inputs = assign_gpu(inputs)
----> 8         output = model(**inputs)[0][:, 0, :]
      9         out_list.append(output)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
    751 
    752         embedding_output = self.embeddings(
--> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    754         )
    755         encoder_outputs = self.encoder(

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181 
--> 182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    183         embeddings = self.LayerNorm(embeddings)
    184         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered
```


<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/63026701/bert-forward-reports-error-on-gpu-but-runs-fine-on-cpu

Comments:
CUDA errors are always very cryptic 😕.

Do you mind giving us an example that we can reproduce (e.g. with an article you're trying to encode that fails), so that we can see what's going on?

Thank you for the comment. I split my code into 2 script files: one is tokenize; the other is transform. Somehow it works. Probably it was caused by some issue in the remote server, or my custom data class. I will close the topic. Thank you for your time.