Reading files takes for ever in language modeling
Author: abdallah197
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (GPT2):

Language I am using the model on (Music Midifiles tokens):

The problem arises when using:
* [ ] the official example scripts: (give details below)


The tasks I am working on is:
* language modeling, running the script run_langauge_modling.py

## To reproduce

Steps to reproduce the behavior:

```
python -m torch.distributed.launch \
--nproc_per_node 4 run_language_modeling.py \
 --train_data_file /nethome/abashir/data/train.txt \
--output_dir /data/users/abashir/model \
--model_type gpt2 --tokenizer_name /nethome/abashir/data/PianoAI \
  --do_train --line_by_line --learning_rate 1e-4 --num_train_epochs 5 \
 --save_total_limit 2 --save_steps 1000 --per_gpu_train_batch_size 8 \
 --seed 42 --overwrite_cache --block_size 128 
```


The output freeze at this stage for more than a day. train file size is less than 1 GB:
`03/23/2020 19:43:11 - INFO - __main__ -   Creating features from dataset file at /nethome/abashir/data/train.txt`


## Expected behavior

Start the training right away after adding --line_by_line
## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-4.4.0-45-generic-x86_64-with-debian-jessie-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes



Comments:
the same issue happens with me when running on 30 GB English Text, with the same parameters given to the script

@julien-c any recommendations on this issue?

Maybe launch on a single node and in a debugger, to see what's happening?

I have run the debugging both with pre-trained tokenizer using --model_name_or_path flag and with a trained tokenizer and config
In both cases, this line is where the conde hangs
`self.examples = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size)["input_ids"]`


One suggestion is that you could create the cached dataset file once locally and then copy it over to wherever (cluster etc..) you're gonna be using it to train.

I have the same issue. I was training XML-Roberta and the training gets stuck in the step of creating features. Does anyone have a solution? Thanks!

@Genius1237 the cluster that I am using have better specs than my local machine. but also, I have tried this before and ended up in the same problem

@abdallah197 Can you do something like this (https://github.com/Microsoft/ptvsd/issues/1354#issuecomment-487289774). It basically allows you to debug installed packages. This way, you can debug into the `batch_encode_plus` function and see if it's failing for one particular example or just slow in general.

Visual Studio Code also has a pretty neat and easy-to-use debugger that you can even run on a remote machine.

Let us know if you find root causes for your issue.

I am facing the same issue, may I ask if anyone got a solution? Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.