run_classifier on CoLA fails with illegal memory access
Author: prematurelyoptimized
Pull request: no
Labels: name: wontfix

I am trying to run the run_classifier.py script against the CoLA task as a smoke test to make sure I have everything installed correctly.  However, when I run 
`CUDA_LAUNCH_BLOCKING=1 python run_classifier.py --task_name CoLA --do_train --do_eval --do_lower_case --data_dir /workspace/glue/CoLA/ --bert_model bert-base-uncased --max_seq_length128 --train_batch_size 8 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir /tmp/CoLA`
I get the following stack trace: 
`THCudaCheck FAIL file=/tmp/pip-req-build-k527kqpu/aten/src/ATen/native/cuda/Embedding.cu line=340 error=700 : an illegal memory access was encountered

Traceback (most recent call last):
  File "run_classifier.py", line 669, in <module>
    main()
  File "run_classifier.py", line 581, in main
    loss.backward()
  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 106, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /tmp/pip-req-build-k527kqpu/aten/src/ATen/native/cuda/Embedding.cu:340`
before even a single batch finishes.

I'm running on a Tesla P4 out of NVidia's official pytorch container.  I installed pytorch-pretrained-BERT using pip.  The only thing that I have changed from the example on the doc page is reducing the batch size from 32 to 8 on account of the P4's lower memory.  I wasn't able to check if it works in CPU mode due to [this](https://github.com/huggingface/pytorch-pretrained-BERT/issues/150).

If there is any other useful information that I can provide, let me know.

Comments:
My guess is that the inputs are either larger than the maximum input size of the model (512) or outside the vocabulary (larger than the vocabulary size).

Do you think you can try to check this?

@ananyahjha93 did the implementation of the additional GLUE tasks. Maybe he has some additional insights on this.

@prematurelyoptimized I have not been able to re-create this error with a batch size of 8 on CoLA. I will post an update if I find something on this. In the meantime, is it possible for you to check this code on a different GPU? Also, can you post the version of PyTorch you are using? 

My apologies for the delay, I have been out of town.

@ananyahjha93 Unfortunately, I only have the one GPU for testing.  print(torch.__version__) returns 1.1.0a0+be364ac

@thomwolf I didn't change the tokenizer at all, so it shouldn't be an out-of-vocabulary problem and I didn't change either convert_examples_to_features or _truncate_seq_pair, so it shouldn't be a problem with the input being too long.  I can, however, dig in and verify those assumptions.

Unfortunately, my IT group installed a new NVidia license and now torch reports cuda being unavailable.  I should still be able to check the OOV and input overflow problems, but any testing on the GPU I will have to put on hold until we get that issue sorted out.

Ok, I am back up and running.  Neither I nor my IT folks know what was causing the issue with cuda being unavailable (It was an error 35, indicating a newer runtime than the driver, but it was the same runtime and driver that were working before).  In any case, upgrading to CUDA 10.1 and driver version 418.40.04 resolved the unavailability issue after rebuilding the container for the new environment.  So now I can start addressing the illegal memory access issue again (sadly, it was not resolved with the driver update)

I added a validation check before training to ensure that the tokenizer could convert the ids back into tokens and the length of the training samples didn't exceed the maximum length.  All the checks passed, so that should rule out OOV and input overflow.

I also noticed that the illegal memory access in non-deterministic.  One example stack is the following:

Traceback (most recent call last):                                                                | 0/1069 [00:00<?, ?it/s]
  File "run_classifier2.py", line 677, in <module>
    main()
  File "run_classifier2.py", line 580, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 970, in forward
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 711, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 261, in forward
    embeddings = words_embeddings + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: an illegal memory access was encountered

However, running the same command immediately afterward gives

Traceback (most recent call last):                                                                | 0/1069 [00:00<?, ?it/s]
  File "run_classifier2.py", line 677, in <module>
    main()
  File "run_classifier2.py", line 580, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 970, in forward
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 714, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 396, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 382, in forward
    intermediate_output = self.intermediate(attention_output)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 354, in forward
    hidden_states = self.dense(hidden_states)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1401, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-k527kqpu/aten/src/THC/THCBlas.cu:259

And then again,

Traceback (most recent call last):
  File "run_classifier2.py", line 677, in <module>
    main()
  File "run_classifier2.py", line 580, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 970, in forward
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 714, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 396, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 381, in forward
    attention_output = self.attention(hidden_states, attention_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 339, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 305, in forward
    attention_probs = nn.Softmax(dim=-1)(attention_scores)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 877, in forward
    return F.softmax(input, self.dim, _stacklevel=5)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1256, in softmax
    ret = input.softmax(dim)
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /tmp/pip-req-build-k527kqpu/aten/src/ATen/native/cuda/SoftMax.cu:545

This feels like an OOM error, but watching nvidia-smi, the memory usage peaks at only about 800MiB.  This doesn't change whether my batch size is 2, 4, 8, or 32, so it looks like the batch size is not the culprit either.  The non-determinism could be caused by the kernel running asynchronously, but that shouldn't happen since I'm running with CUDA_LAUNCH_BLOCKING=1.

Looking at the pytorch source at the lines in the stack traces hasn't been illuminating, so are there some recommended debugging steps at this point?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.