Type of Training file needed for finetuning
Author: ankush20m
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
What kind of **training text** file needed for `run_lm_finetuning.py` script. I have created the **text file** in which I have put **line by line sentences**.
Is this format correct for finetuning?
Because I want to finetune the **BERT** with **unlabeled data** i.e. unsupervised training.

In may **_train.txt_** file I have **total 32 sentences**, but while running the script, it shows:
**Num examples = 5**

What's going wrong here?

Below is my command:
```
!python transformers/examples/run_lm_finetuning.py \
    --output_dir=./output \
    --model_type=bert \
    --model_name_or_path=bert-base-uncased \
    --config_name=./custom \
    --do_train \
    --train_data_file=./train.txt \
    --do_eval \
    --eval_data_file=./test.txt \
    --do_lower_case \
    --learning_rate=5e-5 \
    --num_train_epochs=10 \
    --warmup_steps=0 \
    --overwrite_output_dir \
    --per_gpu_train_batch_size=1 \
    --per_gpu_eval_batch_size=1 \
    --mlm
```

Below is the information came while running script on which I need clarification:

```
01/14/2020 10:54:53 - INFO - __main__ -   Loading features from cached file ./bert-base-uncased_cached_lm_510_train.txt
01/14/2020 10:54:53 - INFO - __main__ -   ***** Running training *****
01/14/2020 10:54:53 - INFO - __main__ -     Num examples = 5
01/14/2020 10:54:53 - INFO - __main__ -     Num Epochs = 10
01/14/2020 10:54:53 - INFO - __main__ -     Instantaneous batch size per GPU = 1
01/14/2020 10:54:53 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 1
01/14/2020 10:54:53 - INFO - __main__ -     Gradient Accumulation steps = 1
01/14/2020 10:54:53 - INFO - __main__ -     Total optimization steps = 50
```

Comments:
Hi, currently the `run_lm_finetuning` script does not take into account the line returns to split the data. It splits the data according to the maximum length the model will allow (which is 512 tokens for BERT), as it is generally used to fine-tune a model on a lengthy text corpus.

If you want to do a line by line split, you could modify the `TextDataset` so that it constructs a dataset like the one you want (creating examples from line returns).

> Hi, currently the `run_lm_finetuning` script does not take into account the line returns to split the data. It splits the data according to the maximum length the model will allow (which is 512 tokens for BERT), as it is generally used to fine-tune a model on a lengthy text corpus.
> 
> If you want to do a line by line split, you could modify the `TextDataset` so that it constructs a dataset like the one you want (creating examples from line returns).

@LysandreJik Thanks a lot. Could you suggest any other way how can I finetune the BERT model with unlabeled data i.e. only with a text file containing sentences?

Re. a `LineByLineTextDataset`, you could take a look at the implementation in https://github.com/huggingface/transformers/pull/2570 (should be merged to master soon).

However, a 32 sentences dataset is very, very small, even for finetuning.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.