Errors when using Apex
Author: ibeltagy
Pull request: no
Labels: name: Help wanted

name: Need more information

name: wontfix

Using `pytorch-pretrained-BERT` with `apex` installed breaks with the errors below. I am using it through allennlp, and my environment settings are: 
```
ubuntu 16.04
nvidia driver 410.48
4 Titan V gpus
python 3.6.8
cuda 9
pytorch 1.0.1.post2
pytorch-pretrained-bert 0.6.1 
```
I also tried python 3.7, cuda 10 and nvidia driver 390 with the same errors.

At training time, the crashes from the first batch with the following error:
```
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 110, in forward
    token_vectors = embedder(*tensors)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/allennlp/modules/token_embedders/bert_token_embedder.py", line 91, in forward
    attention_mask=util.combine_initial_dims(input_mask))
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 711, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 263, in forward
    embeddings = self.dropout(embeddings)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 749, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1549636813070/work/aten/src/THC/THCGeneral.cpp:405
```

At prediction time, it crashes with the following error after exactly 13 batches (even when I shuffle the data or change the batch size) 
```  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 110, in forward
    token_vectors = embedder(*tensors)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/allennlp/modules/token_embedders/bert_token_embedder.py", line 91, in forward
    attention_mask=util.combine_initial_dims(input_mask))
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 714, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 396, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 381, in forward
    attention_output = self.attention(hidden_states, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 339, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 290, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/functional.py", line 1354, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : an internal operation failed at /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCBlas.cu:258
```

Any thoughts what might be wrong, or how I can debug this ? 

Comments:
Can you try to run it with `CUDA_LAUNCH_BLOCKING=1` so we can see which exact CUDA call fails?
Also, do you have a simple way for me to try to reproduce this error?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.