[FLAX] Potential bug in CLM script when using text files
Author: stefan-it
Pull request: no

Hi,

I've seen the following bug when using the CLM script with FLAX in combination with text files, pretty much the same as reported on StackOverflow:

https://stackoverflow.com/questions/65145526/why-new-lines-arent-generated-with-my-fine-tuned-distilgpt2-model

The underlying problem is, that newlines are removed and then the output of a fully trained model looks like:

```text
'Der Sinn des Lebens ist es, sich in ein und derselben Welt selbst niederzulassen und den anderen zum Leben auf der Grundlage dieses Modells zu berufen.Denn auch sie sind nur möglich, weil sie mit dem Göttlichen und mit dem göttlichen Willen in Verbindung stehen, die sich aus einer Welt der Liebe und des Friedens füreinander ergeben.Wir müssen die Freiheit der menschlichen und physischen Existenz verteidigen.Denn die Freiheit geht davon aus, dass Menschen nur existieren, weil sie in einem Zustand von Freiheit, Würde, Harmonie und'
```

So no newlines are generated. I've modified:

https://github.com/huggingface/transformers/blob/a4340d3b85fa8a902857d26d7870c53f82a4f666/examples/flax/language-modeling/run_clm_flax.py#L376

to

```python
output = tokenizer([example + "\n" for example in examples[text_column_name]])
```

and the output of the model now is:

```text
Mein Name ist Alexey.\nIch bin...\nOh, verdammt.\n- Was?\n- Ich kann mit dir nicht gut befreundet sein.\n- Sollen wir nicht?\n- Es ist ein bisschen komplizierter.\nUnd du solltest dir die Zähne putzen.\n- Hier war es noch nie.\n- Ich weiß.\nJetzt mal raus, bitte.\n- Das ist ja großartig.\n- Das ist es ja.\n- Wirklich?\n- Es muss nicht nur ein Spaß sein, das weiß ich doch.\nDas ist wirklich gut.\nJa.\nDie Leute mögen es, wenn du hier bist.\nGenau wie ich's tue.\n- Warum?\n- Ich wollte das Gefühl haben.\nEs ist nicht leicht, füreinander zu sorgen.\nAber das ist, was ich wollte.\nIch wollte mich bedanken, dass ich alles getan habe, was du auf die Beine gebracht hast.\nDu hast alles tun, was du wolltest.\nAber das
```

This is only a temporarily workaround, but maybe the best option is to use the `keep_linebreaks` in the dataset loader (when using text files), but I haven't tested it yet. This option was introduced in https://github.com/huggingface/datasets/pull/1913.

/cc @patrickvonplaten 

Comments:
If I understand correctly this also applies to PyTorch's `run_clm.py` script. I am happy to add the `keep_linebreaks` parameter to both `run_clm.py` and `run_flax_clm.py` if `load_dataset("text")` is used. 

@sgugger @lhoestq - what do you think? It seems like multiple people had this problem.

Also @lhoestq - maybe it's a good idea to add some documentation about `keep_linebreaks` as it can't be find anywhere in the docs. Maybe here: https://huggingface.co/docs/datasets/loading_datasets.html#text-files ? 

This also applies to the TensorFlow script as well. I have no problem adding the `keep_linebreaks` parameter there.

Ok great! @stefan-it - would you maybe be interested in opening a PR to change the following files:

- https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py
- https://github.com/huggingface/transformers/blob/master/examples/tensorflow/language-modeling/run_clm.py
- https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py

Also, we should probs open a PR add docs to https://huggingface.co/docs/datasets/loading_datasets.html#text-files

And don't forget https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm_no_trainer.py

Hi,

I tested it with the `keep_linebreaks` parameter and output of the model is then correct :hugs: 

Yeah, I would like to open a PR for these changes, should I wait until  #13024 is merged, @patrickvonplaten :thinking: 

Actually, I think #13024 doesn't actually lead to a speed-up :D Feel free to open as soon as you want :-) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.