What is the data format of transformers language modeling run_clm.py fine-tuning?
Author: gongshaojie12
Pull request: no

I now use run_clm.py to fine-tune gpt2, the command is as follows:

```
python run_clm.py \\
    --model_name_or_path gpt2 \\
    --train_file train1.txt \\
    --validation_file validation1.txt \\
    --do_train \\
    --do_eval \\
    --output_dir /tmp/test-clm
```

The training data is as follows:
[train1.txt](https://github.com/huggingface/transformers/files/6847229/train1.txt)
[validation1.txt](https://github.com/huggingface/transformers/files/6847234/validation1.txt)


The following error always appears:

```
[INFO|modeling_utils.py:1354] 2021-07-20 17:37:01,399 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
Running tokenizer on dataset: 100%|██████████| 1/1 [00:00<00:00, 90.89ba/s]
Running tokenizer on dataset: 100%|██████████| 1/1 [00:00<00:00, 333.09ba/s]
Grouping texts in chunks of 1024:   0%|          | 0/1 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "D:/NLU/tanka-reminder-suggestion/language_modeling/run_clm.py", line 492, in <module>
    main()
  File "D:/NLU/tanka-reminder-suggestion/language_modeling/run_clm.py", line 407, in main
    desc=f"Grouping texts in chunks of {block_size}",
  File "D:\lib\site-packages\datasets\dataset_dict.py", line 489, in map
    for k, dataset in self.items()
  File "D:\lib\site-packages\datasets\dataset_dict.py", line 489, in <dictcomp>
    for k, dataset in self.items()
  File "D:\lib\site-packages\datasets\arrow_dataset.py", line 1673, in map
    desc=desc,
  File "D:\lib\site-packages\datasets\arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "D:\lib\site-packages\datasets\fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "D:\lib\site-packages\datasets\arrow_dataset.py", line 2024, in _map_single
    writer.write_batch(batch)
  File "D:\lib\site-packages\datasets\arrow_writer.py", line 388, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow\table.pxi", line 1631, in pyarrow.lib.Table.from_pydict
  File "pyarrow\array.pxi", line 332, in pyarrow.lib.asarray
  File "pyarrow\array.pxi", line 223, in pyarrow.lib.array
  File "pyarrow\array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "D:\lib\site-packages\datasets\arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow\array.pxi", line 1076, in pyarrow.lib.Array.__getitem__
  File "pyarrow\array.pxi", line 551, in pyarrow.lib._normalize_index
IndexError: index out of bounds
```

Is the format of my training data incorrect? Please help me thanks！

Comments:
The format seems correct. From the [language modeling page](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling): 

> If your dataset is organized with one sample per line, you can use the --line_by_line flag (otherwise the script concatenates all texts and then splits them in blocks of the same length).

So as you're not specifying that flag, it concatenates the text. Perhaps your dataset is just to small to group it into chunks?

Hi @NielsRogge Thanks for your reply. This `run_clm.py` does not have a `--line_by_line` flag, my `transformers` version is 4.8.2  . 

I want to use a function similar to `--line_by_line` in `run_clm.py`, what should I do? Because the length of my training data is not consistent, I don’t want a piece of training data to be divided into two different blocks. Thanks!

Cc @sgugger: can we add a `line_by_line` option to `run_clm.py`?

That is the behavior of the script, there is no need to add it. What the script does not support is`line_by_line=False` since it is never used to train causal language models.

Oh sorry, I just realized I was mistaken: the behavior of the `run_clm` script is to have `line_by_line=False` which is the way masked language models are pretrained. We did not implement the `--line_by_line` option in the run_clm script because GPT-like models are pretrained by concatenating all examples together and then makes blocks of contiguous texts of a given size.

To go back to the initial problem, are you sure you are using the last version of the run_clm script form the master branch? This looks like a bug that was fixed a few days ago for smaller datasets.

Hi @sgugger Thanks for your reply. My `transformers` version is `4.8.2`.  The `run_clm.py` on the `master` branch does not have this bug, but the version on the master has not been officially released yet. The latest version I installed through `pip install transformers` is 4.8.2. By the way, when will the latest version of transformers be released?

v4.9.0 was released today.

Thanks a lot!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.