TypeError: an integer is required (got type NoneType) while using  run_language_modeling.py
Author: cabhijith
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below) Yes
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below) Own task

## To reproduce

Steps to reproduce the behavior:
Train the tokenizer: 
```python
from tokenizers import    BertWordPieceTokenizer

paths = 'Clean_merged.txt'

# Initialize a tokenizer
tokenizer =    BertWordPieceTokenizer()

# Customize training
tokenizer.train(files=paths, vocab_size=32_000, min_frequency=4)
tokenizer.save_model('./') 
```

```
%env TRAIN_FILE= Clean_merged.txt

!python transformers/examples/language-modeling/run_language_modeling.py \
    --output_dir=output_from_scratch \
    --model_type=bert \
    --do_train \
    --tokenizer_name save_tokenizer \
    --save_steps 2000 \
    --per_gpu_train_batch_size 8 \
    --train_data_file=$TRAIN_FILE \
    --mlm \
    --block_size 510
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
To train a BERT model from scratch, on an mlm task.

### Stack trace 
```
2020-07-14 07:52:19.364573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
07/14/2020 07:52:21 - INFO - transformers.training_args -   PyTorch: setting up devices
07/14/2020 07:52:21 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
07/14/2020 07:52:21 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='output_from_scratch', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=8, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Jul14_07-52-21_cf8085fe2205', logging_first_step=False, logging_steps=500, save_steps=2000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)
07/14/2020 07:52:21 - WARNING - __main__ -   You are instantiating a new config instance from scratch.
07/14/2020 07:52:21 - INFO - transformers.configuration_utils -   loading configuration file save_tokenizer/config.json
07/14/2020 07:52:21 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 514,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 32000
}

07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   Model name 'save_tokenizer' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). Assuming 'save_tokenizer' is a path, a model identifier, or url to a directory containing tokenizer files.
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   Didn't find file save_tokenizer/added_tokens.json. We won't load it.
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   Didn't find file save_tokenizer/special_tokens_map.json. We won't load it.
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   Didn't find file save_tokenizer/tokenizer_config.json. We won't load it.
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   Didn't find file save_tokenizer/tokenizer.json. We won't load it.
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   loading file save_tokenizer/vocab.txt
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   loading file None
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   loading file None
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   loading file None
07/14/2020 07:52:21 - INFO - transformers.tokenization_utils_base -   loading file None
07/14/2020 07:52:21 - INFO - __main__ -   Training new model from scratch
/usr/local/lib/python3.6/dist-packages/transformers/modeling_auto.py:709: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
07/14/2020 07:52:26 - INFO - filelock -   Lock 140270211965616 acquired on cached_lm_BertTokenizer_508_Clean_merged.txt.lock
07/14/2020 07:52:31 - INFO - transformers.data.datasets.language_modeling -   Loading features from cached file cached_lm_BertTokenizer_508_Clean_merged.txt [took 3.803 s]
07/14/2020 07:52:31 - INFO - filelock -   Lock 140270211965616 released on cached_lm_BertTokenizer_508_Clean_merged.txt.lock
07/14/2020 07:52:33 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
07/14/2020 07:52:33 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
07/14/2020 07:52:33 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
07/14/2020 07:52:33 - INFO - transformers.trainer -   ***** Running training *****
07/14/2020 07:52:33 - INFO - transformers.trainer -     Num examples = 101259
07/14/2020 07:52:33 - INFO - transformers.trainer -     Num Epochs = 3
07/14/2020 07:52:33 - INFO - transformers.trainer -     Instantaneous batch size per device = 8
07/14/2020 07:52:33 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 8
07/14/2020 07:52:33 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
07/14/2020 07:52:33 - INFO - transformers.trainer -     Total optimization steps = 37974
Epoch:   0% 0/3 [00:00<?, ?it/s]
Iteration:   0% 0/12658 [00:00<?, ?it/s]Traceback (most recent call last):
  File "transformers/examples/language-modeling/run_language_modeling.py", line 296, in <module>
    main()
  File "transformers/examples/language-modeling/run_language_modeling.py", line 260, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1104, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/transformers/data/datasets/language_modeling.py", line 75, in __getitem__
    return torch.tensor(self.examples[i], dtype=torch.long)
TypeError: an integer is required (got type NoneType)

Epoch:   0% 0/3 [00:00<?, ?it/s]
Iteration:   0% 0/12658 [00:00<?, ?it/s]
```


## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: <No

Comments:
The error was caused because I did not specify the type of dataset (```LinebyLine``` in my case). After doing that, it worked. 

I'm facing the same error and below is my run command. Any pointers, @cabhijith ?
`cmd = "python run_language_modeling.py \
    --output_dir ./bertout \
    --model_type bert \
    --do_train \
    --do_eval \
    --train_data_file ./test.txt \
    --eval_data_file ./test.txt \
    --mlm \
    --line_by_line \
    --learning_rate 1e-4 \
    --num_train_epochs 5 \
    --save_total_limit 2 \
    --save_steps 2000 \
    --per_gpu_train_batch_size 16 \
    --evaluate_during_training \
    --warmup_steps=10000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --seed 666 \
    --block_size=512 \
    --tokenizer_name ./bert2 \
    --config_name ./bert2"`

I still got this error after I use `LinebyLine` dataset, anyone can help me ?

> I still got this error after I use `LinebyLine` dataset, anyone can help me ?

It works after a close a file is opening...