Error in AlbertForMaskedLM with add_tokens and model.resize_token_embeddings
Author: mckunkel
Pull request: no
Labels: name: wontfix

## 🐛 Bug


Model I am using: Albert & Bert

Language I am using the model on English

The problem arise when using:
* [X] the official example scripts: run_lm_finetuning.py

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: mlm

## To Reproduce

Steps to reproduce the behavior:

1. add following lines after line 244 in run_lm_finetuning
`tokenizer.add_tokens(['mewhomp', 'skype', 'kiltrim'])
model.resize_token_embeddings(len(tokenizer)) 
`

Error 

> RuntimeError: The size of tensor a (30003) must match the size of tensor b (30000) at non-singleton dimension 2

## Expected behavior

Add the 3 additional tokens and train

## Environment

* OS: Ubuntu 16.04
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): All branches with Albert. All version of pip install that supports Albert
* Using GPU ? yes
* Distributed or parallel setup ? Yes
* Any other relevant information:

## Additional context

This is similar to issues 
[2373](https://github.com/huggingface/transformers/issues/2373)
[2468](https://github.com/huggingface/transformers/issues/2468)
[2480](https://github.com/huggingface/transformers/issues/2480)


Comments:
Hi, I've pushed a fix that was just merged in `master`. Could you please try and install from source:
```py
pip install git+https://github.com/huggingface/transformers
```
and tell me if you face the same error?

Greetings, 
Thanks for the reply. 
I do not get the same error anymore, I get a different error.

> RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/ATen/native/cuda/Normalization.cuh:581

Here is a full stack trace if it helps.

> Epoch:   0%|                                                                                                                      | 0/10 [00:00<?, ?it/s/
code/src/pretrain_roberta.py:93: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return torch.tensor(self.examples[item])
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/ATen/native/cuda/Normalization.cuh line=581 error=710 : device-side assert triggered
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
...
...
...
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py", line 659, in forward
    prediction_scores = self.predictions(sequence_outputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py", line 588, in forward
    hidden_states = self.LayerNorm(hidden_states)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/normalization.py", line 153, in forward
    input, self.normalized_shape, self.weight, self.bias, self.eps)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1696, in layer_norm
    torch.backends.cudnn.enabled)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/ATen/native/cuda/Normalization.cuh:581

Intuitively I would say it has to do with a Cross Entropy having it's `ignore index` set to -1. We have recently updated all our Cross Entropy methods to be set to a default of -100 like the official PyTorch default.

Would you mind checking if you don't have a something similar in your code? If you're using one of our hosted scripts, you can simply take the updated version of the script which is updated as we update source code.

@LysandreJik   Thank you ! i changed the -100 to -1 ,then the program works.

Hi @LysandreJik 

After fixing the modeling_bert.py file, now I can successfully add new tokens and train rm_lm_finetuning file using one gpu. However, when I try to allocate 2 gpus, an error came out below, any thoughts?

```
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 723, in <module>
    main()
  File "run_lm_finetuning.py", line 673, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 317, in train
    loss.backward()
  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 32, in backward
    return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 43, in forward
    return comm.reduce_add_coalesced(grads, destination)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 121, in reduce_add_coalesced
    flat_result = reduce_add(flat_tensors, destination)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 77, in reduce_add
    nccl.reduce(inputs, outputs, root=nccl_root)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/nccl.py", line 51, in reduce
    torch._C._nccl_reduce(inputs, outputs, root, op, streams, comms)
RuntimeError: NCCL Error 2: unhandled system error
```

Thank you in advance for your help.


Hi @jasonwu0731, do you mind opening a new issue with your problem, detailing your environment (python, pytorch, transformers versions) following the `bug` issue template?

Thank you 

@LysandreJik is it the same issue as #2373 ? Is the issue resolved?
Thank you for your help. 


Greetings, 
I have had issues getting the fix to work, however I think the issue is on my end and I have been slowly investigating it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.