DistilRoberta Model fine tuning on Squad dataset
Author: graviraja
Pull request: no
Labels: name: PyTorch

name: Distillation

# 🐛 Bug

I am trying to train the **distilroberta-base** model on **SQuAD** dataset using distillation code. I have trained the Roberta model on SQuAD 2.0 dataset, so that I can use it as a teacher model. I am using the **distilroberta-base** model as student.

## Information

Model I am using : Roberta

Language I am using the model on: English

The problem arises when using:

* [x] my own modified scripts: (give details below)
I am using the run_squad_w_distillation.py code, with the following modification
- Added the imports relevant to roberta
- Removed token_type_ids from inputs while sending to student model

```python
MODEL_CLASSES = {
    "bert": (BertConfig, BertForQuestionAnswering, BertTokenizer),
    "xlnet": (XLNetConfig, XLNetForQuestionAnswering, XLNetTokenizer),
    "xlm": (XLMConfig, XLMForQuestionAnswering, XLMTokenizer),
    "distilbert": (DistilBertConfig, DistilBertForQuestionAnswering, DistilBertTokenizer),
    "roberta": (RobertaConfig, RobertaForQuestionAnswering, RobertaTokenizer)
}

```

```python
if args.model_type in ["roberta"]:
     del inputs["token_type_ids"]
```
The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD 2.0

## To reproduce

Steps to reproduce the behavior:

1. python run_squad_w_distillation.py --model_type roberta --model_name_or_path distilroberta-base --output_dir ./distil_roberta --teacher_type roberta --teacher_name_or_path $ROBERTA_MODEL --train_file $SQUAD_DIR/train-v2.0.json --predict_file $SQUAD_DIR/dev-v2.0.json --version_2_with_negative --do_train --do_eval --do_lower_case --save_steps 5000 --logging_steps 5000

```python
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [206,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "run_squad_w_distillation.py", line 871, in <module>
    main()
  File "run_squad_w_distillation.py", line 813, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, teacher=teacher)
  File "run_squad_w_distillation.py", line 207, in train
    "input_ids": batch[0],
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: transform: failed to synchronize: cudaErrorAssert: device-side assert triggered
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

To train properly and evaluate on predict file dataset with f1_score near roberta model f1_score(81.5)

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.3.0
- Platform: "CentOS Linux 7" 
- Python version: 3.6.9
- PyTorch version (GPU?): Yes
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes, CUDA Version: 10.2, Driver Version: 440.33.01
- Using distributed or parallel set-up in script?: No


Comments:
Hello @graviraja 
You don't need to remove the `token_type_ids` for RoBERTa models. There is one matrix of token type and it has only one type (a matrix of 0).
You can remove the `do_lower_case` flag for RoBERTa models. The vocabulary is case sensitive.
Have you tried WITHOUT GPU? (`CUDA_VISIBLE_DEVICES="")

Hi @VictorSanh 
Running the code on cpu throws the following error
```python
Traceback (most recent call last):
  File "run_squad_w_distillation.py", line 871, in <module>
    main()
  File "run_squad_w_distillation.py", line 813, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, teacher=teacher)
  File "run_squad_w_distillation.py", line 207, in train
    "input_ids": batch[0],
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu
```
I have mentioned `--no_cuda flag` in the command and removed `--do_lower_case`.

Thank you for your help!

I was not able to reproduce your bug @graviraja.

I pushed an update to `run_squad_w_distillation.py` on master to include RoBERTa, let me know if that works.

As I suggested, to test without GPU, you should use the `CUDA_VISIBLE_DEVICES=""` as I believe there is an inconsistency between the `--no_cuda` flag and the `args.n_gpu = torch.cuda.device_count()`. I'll correct it.

Hi @VictorSanh still it is not working for me after pulling the latest code. I have tried with `CUDA_VISIBLE_DEVICES=""` without `--no_cuda` flag and with `--no_cuda` flag also. I am getting the below error.

```python
03/03/2020 13:26:59 - INFO - __main__ -     Num examples = 135302
03/03/2020 13:26:59 - INFO - __main__ -     Num Epochs = 3
03/03/2020 13:26:59 - INFO - __main__ -     Instantaneous batch size per GPU = 8
03/03/2020 13:26:59 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 8
03/03/2020 13:26:59 - INFO - __main__ -     Gradient Accumulation steps = 1
03/03/2020 13:26:59 - INFO - __main__ -     Total optimization steps = 50739
Epoch:   0%|          | 0/3 [00:00<?, ?it/s]        Traceback (most recent call last):
  File "run_squad_w_distillation.py", line 868, in <module>
    main()
  File "run_squad_w_distillation.py", line 810, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, teacher=teacher)
  File "run_squad_w_distillation.py", line 217, in train
    outputs = model(**inputs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 708, in forward
    head_mask=head_mask,
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 801, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 64, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 190, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
With gpu, by setting `CUDA_VISIBLE_DEVICES=1`, I am getting the following error:

```python
  File "run_squad_w_distillation.py", line 868, in <module>
    main()
  File "run_squad_w_distillation.py", line 810, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, teacher=teacher)
  File "run_squad_w_distillation.py", line 217, in train
    outputs = model(**inputs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 708, in forward
    head_mask=head_mask,
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 808, in forward
    encoder_attention_mask=encoder_extended_attention_mask,
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 422, in forward
    hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 383, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 329, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/transformers/modeling_bert.py", line 231, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : library not initialized at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCGeneral.cpp:216
```

I have trained the model `roberta` on `squad 2.0` dataset using GPU. Does this cause an issue?


Reading the error when it's running on CPU, It looks like that you have a tokenization problem (it tries to access an index that is out of range). Are you on master? Could you make sure you tokenize the dataset and run the inference with the same version? --> Add a `--overwrite_cache` to retokenize.

@VictorSanh  same issue is happening with `--overwrite_cache`. May I know the command you are using for training the distilroberta model?

@VictorSanh any update on this ?

Updating the transformers version fixes it.