GPT2 crashing at loss.backward()
Author: vibhavagarwal5
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.5.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

@LysandreJik 

## Information

Trying to finetune GPT2 model but the GPU is crashing after `loss.backward()`. I thought it might be just my code but I ran some different code involving finetuning GPT2 and that as well crashed in the same manner. 

Getting this warning as well. 
```
WARNING - transformers.modeling_utils -   Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
``` 

A week or 2 back, everything was working fine but now the same code is crashing on `loss.backward()`. 



Comments:
Hi @vibhavagarwal5 , you can safely ignore this warning, that issue is resolved in this PR #5922 .

Do you think you can post the stack-trace after the crash, and also the version and memory of the GPU used

**GPU DETAILS:** NVIDIA 2080 TI (12GB)

NVIDIA-SMI 440.95.01
Driver Version: 440.95.01
CUDA Version: 10.2

**STACK TRACE:**
```
Epoch:   0%|                                                                                                                                          | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                                                                                 | 0/87472 [00:00<?, ?it/s]
  File "finetune_lm.py", line 553, in <module>
    main()
  File "finetune_lm.py", line 507, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "finetune_lm.py", line 157, in train
    loss.backward()
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` (createCublasHandle at /opt/conda/conda-bld/pytorch_1587428266983/work/aten/src/ATen/cuda/CublasHandlePool.cpp:8)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x7f6cca012b5e in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xdba405 (0x7f6ccaff9405 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #2: at::cuda::getCurrentCUDABlasHandle() + 0x94c (0x7f6ccaffa1ec in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdafb01 (0x7f6ccafeeb01 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x1263db7 (0x7f6ccb4a2db7 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #5: THCudaTensor_addmm + 0x5c (0x7f6ccb4a84ac in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xea5f28 (0x7f6ccb0e4f28 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc92e8 (0x7f6ccb0082e8 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #8: <unknown function> + 0xe224d0 (0x7f6cf5c264d0 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x29f9d0e (0x7f6cf77fdd0e in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0xe224d0 (0x7f6cf5c264d0 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #11: at::Tensor::mm(at::Tensor const&) const + 0xf0 (0x7f6cf57ea180 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x264517c (0x7f6cf744917c in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #13: torch::autograd::generated::MmBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x151 (0x7f6cf7449f81 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2ae8215 (0x7f6cf78ec215 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #15: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x16f3 (0x7f6cf78e9513 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7f6cf78ea2f2 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #17: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f6cf78e2969 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f6cfac29558 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0xc819d (0x7f6d1200b19d in /mnt/c7cfa338-89cd-4d15-b0b9-f1befc9a2c68/vibhav/anaconda3/envs/vesnli/bin/../lib/libstdc++.so.6)
frame #20: <unknown function> + 0x76db (0x7f6d269a16db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #21: clone + 0x3f (0x7f6d266caa3f in /lib/x86_64-linux-gnu/libc.so.6)
```

Seems like memory error, you can try running with batch size 1 and see if it still crashes.

Nope not a memory error. Still crashing with batch size 1

Which GPT-2 model are you using ? 

Tried both 'gpt2' and 'gpt2-medium'. Same issue

Could you run it in on CPU, erros will be more readable.

```
raceback (most recent call last):                                                                                                                | 0/174944 [00:00<?, ?it/s]
  File "finetune_lm.py", line 553, in <module>
    main()
  File "finetune_lm.py", line 507, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "finetune_lm.py", line 144, in train
    outputs = model(inputs, labels=labels)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 601, in forward
    output_hidden_states=output_hidden_states,
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 469, in forward
    inputs_embeds = self.wte(input_ids)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

So looks like error is form the embedding layer, what's the shape of your `inputs`

bs x seq_len (4x63 or 4*52 ... anything)

could you be more specific, what is your seq_len ? 

Input:
```
torch.Size([4, 47])
Premise: Children smiling and waving at camera Hypothesis: There are children present [EXP] The children must be present to see them smiling and waving. [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS] [EOS]
```
label:
```
torch.Size([4, 47])
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, 'The', 'Ġchildren', 'Ġmust', 'Ġbe', 'Ġpresent', 'Ġto', 'Ġsee', 'Ġthem', 'Ġsmiling', 'Ġand', 'Ġwaving', '.', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]', '[EOS]']
```

Did you add any new tokens to the tokenizer ?

get the shape of embeddings using `model.transformer.wte.weight.shape` the first dim of shape and len of tokenizer should match. See if this asserts is True
```python3
assert modle.transformer.wte.weight.shape[0] == len(tokenizer)
```

if not then that means, your vocab size and embed input size are not matching. If you added new tokens to the vocab, you'll need to resize the token embeddings of the model. You can resize it using

```python3
model.resize_token_embeddings(len(tokenizer))
```



I'm doing this already..

```python
tokenizer.add_special_tokens(SPECIAL_TOKENS_DICT)
model.resize_token_embeddings(len(tokenizer))
```

I was able to reprouce the bug only when embedding size and vocab len didn't match. 

`assert modle.transformer.wte.weight.shape[0] == len(tokenizer)` did this assert result in `False` ?

No, its True because I did the model.resize so this should have anyways asserted True.

For whatever reason, `transformers v2.3` is working and the latest `3.x`

Hi @vibhavagarwal5, could you provide a sample script so that we may reproduce on our side? Something with a sample text that makes it crash would be wonderful, if you have one.

I figured it out, it was due to the change in ignore_index=-100 instead of -1 in the cross entropy loss which was causing the issue. I'll close this. 

Glad you could find the source of the issue!