Language model fine tuning using scibert as the base model
Author: graviraja
Pull request: no
Labels: name: Usage

# 🐛 Bug

## Information
I am trying to finetune the Scibert model on Covid dataset. Features for the train data is not being calculated properly. 
Model I am using (Bert, XLNet ...): allenai/scibert_scivocab_uncased

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Language Modelling
* [ ] my own task or dataset: my own dataset in text files

## To reproduce

Steps to reproduce the behavior:

1. python examples/run_language_modeling.py --output_dir=./lm_finetune --model_name_or_path=allenai/scibert_scivocab_uncased --do_train --train_data_file=lm_data/train.txt --do_eval --eval_data_file=lm_data/dev.txt --mlm --tokenizer_name allenai/scibert_scivocab_uncased  --model_type  bert


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Finetuned model and perplexity score on evaluation data
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.6
- Platform: "CentOS Linux 7"
- Python version: 3.6.9
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

What should I provide the `model_type` as?


Comments:
SciBert is a BERT model.

Please be more descriptive. Saying that features are not calculated correctly is not very helpful. Please describe the problem in full.

Hi @BramVanroy , following is the error stack
```code
04/01/2020 09:10:02 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1000000000000, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='/media/data1/ravi/covid-challenge/lm_data/dev.txt', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='allenai/scibert_scivocab_uncased', model_type='bert', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='./lm_finetune', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name='allenai/scibert_scivocab_uncased', train_data_file='/media/data1/ravi/covid-challenge/lm_data/train.txt', warmup_steps=0, weight_decay=0.0)
04/01/2020 09:10:02 - INFO - __main__ -   Creating features from dataset file at /media/data1/ravi/covid-challenge/lm_data
04/01/2020 09:28:10 - INFO - __main__ -   Saving features into cached file /media/data1/ravi/covid-challenge/lm_data/bert_cached_lm_999999999998_train.txt
Traceback (most recent call last):
  File "examples/run_language_modeling.py", line 781, in <module>
    main()
  File "examples/run_language_modeling.py", line 731, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "examples/run_language_modeling.py", line 224, in train
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
  File "/media/data2/anaconda/envs/covid/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 94, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
```

In the `bert_cached_lm_999999999998_train.txt` file there is only one line. `train.txt` file is of size **345MB**.

Thank you for your help!!

This seems to indicate a problem with the dataset. Can you post the contents of bert_cached_lm_999999999998_train.txt? Maybe you chose a block size that is larger than your data size.

This is probably the same error as https://github.com/huggingface/transformers/issues/3443#issuecomment-607422291

@graviraja  Had the same issue which is most likely related to 
`model = torch.nn.DataParallel(model)` in [trainer.py](https://github.com/huggingface/transformers/blob/8e093e5981e573a0b591dc57e8d52cc3efe82230/src/transformers/trainer.py#L250)

Uncommenting this line or using only one GPU 
`export CUDA_VISIBLE_DEVICES=1`
works in my case:


> CUDA_VISIBLE_DEVICES

Note that `CUDA_VISIBLE_DEVICES=1` does not mean to use "just one GPU", but it means to specifically use GPU with ID#1. However, device are zero-indexed, so the first GPU on your system will typically be #0: `CUDA_VISIBLE_DEVICES=0`

Thanks, @BramVanroy , makes perfect sense.

@graviraja I've set up a [notebook](https://github.com/Nikoschenk/language_model_finetuning/blob/master/scibert_fine_tuner.ipynb)  with the required functionality. Previous comments regarding `block_size` were in fact crucial.



Thank you @Nikoschenk  @BramVanroy  for the support.