SortedDL for contiguous LM
Author: simonschoe
Pull request: no

Hi there,
I am currently implementing LM re-training of a RoBERTa model using the `Trainer` API. Since I have a huge training corpus, I was wondering if there is a functionality in the `Trainer` or the corresponding `DataCollatorForLanguageModeling` that allows for sorted batching as in `fastai`? 

More precisely, I would like to feed in all my training data as a contiguous text stream and let the respective functions handle sorted batching irrespective of the sequence length of the individual sequences.

Best,
Simon

Comments:
This issue is more suited for the forum, but maybe @sgugger has some hints to share!

I'm not sure what you would want to sort a single text stream. The `Trainer` supports `--group_by_length` but that's when you have multiple texts.

Note that `DataCollatorForLanguageModeling` only performs random masking on your prepared data, nothing more.

Sorry, maybe I was imprecise here (still trying to wrap my head around a lot of these new concepts). Essentially, what I was wondering if batches for language modeling are constructed in a way that is similar to the approach that you described in [Chapter 10: 
NLP Deep Dive](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) (section 'Putting Our Texts into Batches for a Language Model') of the book you co-authored. If I got it correctly, that would allow me to train the transformer over variable length input sequences without worrying about sequences being truncated due to the constraints imposed by `tokenizer.max_len_single_sentence` (overflowing parts would simply end up at the appropriate position in the nex mini-batch).

I think you may be referring to the LM DataLoader. This kind of preprocessing is done using the `datasets` library on our side. Take a look at the [run_clm](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) or [run_mlm](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) examples (in run_mlm the part that is not in the block "line_by_line") or the [language modeling notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) to see how.

Thanks a lot! Indeed, the `group_text()` function was exactly what I was looking for. 

Hey @sgugger,

sorry for reopening this one. I am still not 100% sure if I conveyed my issue properly the first time. Hence, let me briefly restate it using an example:

Let's say I chunk my text stream of length 100 (let's say including 3 sentences) into blocks of `block_size=5`, resulting in 20 blocks. Now, I'd like to feed them into my model using a `batch_size=5`, resulting in 4 batches à 5 text blocks.

I am still not 100% sure how they are fed into the model using the `DataCollatorForLanguageModeling` and `Trainer` API:
```python
# Variant A

batch_1 = [block1, block2, block3, block4, block5]
# ...
batch_4 = [block16, block17, block18, block19, block20]
``` 

```python
# Variant B

batch_1 = [block1, block5, block9, block13, block17]
batch_2 = [block2, block6, block10, block14, block18]
# ...
batch_4 = [block4, block8, block12, block16, block20]
``` 

If I got it correctly, the method presented in the book referred to in the previous comment relates to *variant B*.

You will need to write your own data collator for that as this is not in the Transformers library: contrary to LSTMs, Transformers do not have a state so we don't care about the ordering across batches for those models.

This makes entirely sense, thanks for lifting this barrier in my head!