Having problem Pre-training GPT models
Author: mosh98
Pull request: no

## Environment info


- `transformers` version: 4.9.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): EleutherAI/gpt-neo-2.7B 

The problem arises when using:
* [X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce
I used a csv file with each line have a sample of example

Steps to reproduce the behavior:

My input: `!python /content/transformers/examples/pytorch/language-modeling/run_clm.py --model_name_or_path EleutherAI/gpt-neo-2.7B  --train_file /content/df.csv --output_dir /tmp/test-clm` 

i also tried using the no trainer version but still doesn't work.
What am i doing wrong?

What i got back:
```

Traceback (most recent call last):
  File "/content/transformers/examples/pytorch/language-modeling/run_clm.py", line 520, in <module>
    main()
  File "/content/transformers/examples/pytorch/language-modeling/run_clm.py", line 291, in main
    cache_dir=model_args.cache_dir,
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 830, in load_dataset
    **config_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 710, in load_dataset_builder
    **config_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 271, in __init__
    **config_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 370, in _create_builder_config
    builder_config = self.BUILDER_CONFIG_CLASS(**config_kwargs)
TypeError: __init__() got an unexpected keyword argument 'keep_linebreaks'
```
## Expected behavior

Just want to further train the GPT model

notebook: https://colab.research.google.com/drive/1bk8teH0Egu-gAmBC_zlvUifMHS7y_SyM?usp=sharing

Any help is much appreciated

Comments:
Probably caused by this PR: #13150 

cc @stefan-it 

I'm looking into it right now :)

Oh no, this is only happening when using CSV files as input :thinking: 

@mosh98 As a very quick workaround, could you try to "convert" your csv file into a normal text file (file extension .txt) and then re-run the training :thinking: 

The `keep_linebreaks` argument is only implemented for text files in :hugs: Datasets:

https://github.com/huggingface/datasets/blob/67574a8d74796bc065a8b9b49ec02f7b1200c172/src/datasets/packaged_modules/text/text.py#L19

For CSV it is not available:

https://github.com/huggingface/datasets/blob/67574a8d74796bc065a8b9b49ec02f7b1200c172/src/datasets/packaged_modules/csv/csv.py

I'm working on a fix now (so that `keep_linebreaks` is only used when file extension is `.txt`)

Sure i can try that, i do have aquestion tho,

 when i convert my csv into a text file how will i organize it so that it uses each line as a sample  and also what command do i have to put when i run the script? 

At the moment i have each row is the csv file as an individual sample

You can use the same structure (one individual sample per line) for the text file. 

Command would be pretty much the same, but you need to use the file ending `.txt`, so that the training script will infer the correct extension for the `load_dataset` argument :)

Thank you @stefan-it  the script works now, running out of cuda memeory tho but i think it's irrelevant to the actual script and more to do with my device. 

Thanks Again!