Bugs when fine tuning the gpt2
Author: yanan1116
Pull request: no

Transformers Version: 4.8.2
Torch Version: 1.8.0

I am using the official script to fine tune the gpt2 on the csv files. 
the script: 
https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm_no_trainer.py

train and validation file makeup:
```
df_train_ft_aug.rename(columns={'content': 'text'}).sample(frac=1).to_csv(train_file, index=False)
df_train_ft_aug.rename(columns={'content': 'text'}).sample(frac=0.2).to_csv(validation_file, index=False)

```
My shell command:
```
python -u ./run_clm_no_trainer.py \
                --num_train_epochs 7 \
                --train_file './fintune_csvs/stsa_train_finetune.csv' \
                --validation_file './fintune_csvs/stsa_test_finetune.csv'  \
                --model_name_or_path gpt2 \
                --per_device_train_batch_size 16 \
                --per_device_eval_batch_size 16 \
                --output_dir "./finetune_gpt2_stsa" \
                --preprocessing_num_workers 16 \
                --block_size 256 --overwrite_cache True
```
where ths csv files contain a column, named 'text' for fine tuning the model.

However, there are always errors below, suggesting the lengths of the dataloader 

>  File "./run_clm_no_trainer.py", line 503, in <module>
>     main()exts in chunks of 256 #12:   0%|                              | 0/1 [00:00<?, ?ba/s]
>   File "./run_clm_no_trainer.py", line 480, in main
>     for step, batch in enumerate(eval_dataloader):
>   File "/usr/local/lib/python3.6/dist-packages/accelerate/data_loader.py", line 289, in __iter__
>     for batch in super().__iter__():
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
>     data = self._next_data()
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
>     return self.collate_fn(data)
>   File "/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py", line 80, in default_data_collator
>     batch[k] = torch.tensor([f[k] for f in features])
> ValueError: expected sequence of length 256 at dim 1 (got 52)

Next time I run it, it returns the similar error:

> ValueError: expected sequence of length 168 at dim 1 (got 136)


Then I modified the input params of tokenizer:
```
tokenizer.pad_token = tokenizer.eos_token 
def tokenize_function(examples):
    return tokenizer(examples[text_column_name],) , padding=True, truncation=True )

```

This seems fix the problem. However, the generated texts are quite short after this change.
Any suggestions?


 





Comments:
Pinging @sgugger

It's hard to investigate more without having the data. Adding padding when fine-tuning GPT-2 is a very bad idea when fine-tuning GPT-2, which does not have a padding token, and it shouldn't be necessary. Could you provide us with a reproducer that includes the data?

> It's hard to investigate more without having the data. Adding padding when fine-tuning GPT-2 is a very bad idea when fine-tuning GPT-2, which does not have a padding token, and it shouldn't be necessary. Could you provide us with a reproducer that includes the data? 

Thanks for your suggestion. I will check my data to meet the default setting of fine-tuning. 
By the way, should the eos_token, <endoftext>, be append to the end of each sample ? (the text column in the csv files )
@sgugger 


If it's not done by the tokenizer, yes it should.

> some people do deserve'right to be forgotten'– but law's power shouldn't rest...

> cyrus bus burns on way to no ; she surprises cat's meow crowd

> eu commission approves uk's carphone, dixons merger

> miley cyrus fan arrested

> rdio, crackle, vudu add chromecast support

> being a cynic linked to tripled risk of developing dementia, finland study suggests

> australia, japan strike trade deal

> record low teen birth rate not low enough, says cdc

> legendary house music dj frankie knuckles dies aged 59

> nhtsa closes tesla investigations : reuters

> brad pitt speaks out on premiere punching

> twitter's users are in asia, but its revenue is in the us

> new report questions effectiveness of flu drug tamiflu

> hilary duff talks " really difficult " split from mike comrie

> the top 10 reasons why'guardians of the galaxy'is awesome

> we had a blast at the planes : fire and rescue red carpet premiere!

> fcc extends neutrality comment deadline after site crashes

> olivia munn lives in a haunted house

> uk agency invests in vfx house to create virtual reality content

> of mice and men must die

> death toll in w. african ebola outbreak rises to 518

> cheaper gas, food push down producer prices

> tesla opens up patent portfolio to promote innovation in electronic car...

> useful android tips that you should know

> autism diagnoses on the rise

> u. s. stock futures rising ahead of testimony from fed chair

> blackberry z3 review

> update 1 - buffett's berkshire hathaway buys stake in verizon, adds to wal - mart

> st. luke's improves, but easton hospital falters in safety ratings

> drowsy driving is more common than you think

> republicans nab approval for '. gop'internet domain

> apple says sold well over 800 million mobile devices

> the dot view case for the one m8 is in htc's store for $ 50, not available for...

> physicians push for extension of medicaid reimbursement increase

> mobile fix : chinese ipos, first party data and iphone 6

> ranking the country's best and worst jobs

> nerdnews : marvel comics picks a woman to be the next thor

> men with eating disorders slow to get help, study shows

> apple eyeing beats electronics for $ 3. 2 bln

> measles update for the united states

> former'scandal'star arrested

> us economy shrank at steep 2. 9 percent rate

> white house : medicaid expansion would have covered 120k wisconsinites

> samsung galaxy k zoom goes official with 20. 7mp camera, 10x optical zoom

> asian stocks tumble on weak china, japan data

> killer virus boosts bacon prices

> e - cig industry awaits federal regs

> what would you do to get your cell phone back?

> dc circuit brings back rule limiting bank fees

> texas nuke site increases monitoring of containers

> 10 worst cities for spring allergies

> taxi drivers in europe protest over uber cab service

> taco bell fires second shot at mcdonald's

> a brand - new meteor shower could be spectacular tonight — here's how to...

> argentina debt default 101 : what's at stake? ( + video )

> wikipedia medical entries 90 % inaccurate

> selweski : april 15 may have marked the last tax day

> no real progress on child obesity, latest report says

> skin cancer rate increases in north east

> ambassador drives into history : hm kills india's oldest car

> super moon to brighten summer sky

> google inc ( nasdaq : goog ) beats apple inc. ( nasdaq : aapl ) in introducing...

> samsung galaxy s5 zoom gets fcc certification

> overdose death rates drop in states with medical marijuana laws

> japanese automakers recall 3 mn vehicles for airbag defect

> the white house has released the definitive report on climate change, and...

> bitcoin value and price in silk road auction : us marshals receive offers from...

> see christian hendricks, elisabeth moss & others before they were on " mad...

> bnp paribas nears up to usd9bn settlement with us authorities - source

> browns owner jimmy haslam won't be punished by nfl, per report

> kristin cavallari defends her choice not to vaccinate her child

> us manufacturing gaining on china, brazil and rest of world, study finds

> emma stone addresses weight criticisms in ( typically awesome ) fashion

> billions wasted on flu drug : researchers

> spacecraft crashes on moon to end mission

> chinese manufacturing reaches six - month high, official figures show

> sports day at greatham primary

> pluto's moon may have had an underground ocean

> starbucks'oprah - branded tea ; nyc's macaron day

> microsoft has unveiled the new nokia x2

> caught on tape : emt driver voguing

> ' deliver us from evil'is a genre hopping & highly entertaining piece of cinema

> mobile county : 12 new hiv cases reported in may alone, free testing offered

> roche, exelixis skin cancer drug delays tumor progression

> ntsb faults pilot'mismanagment'in asiana flight - ktbs. com - shreveport, la...

> new skype translator offers nearly real - time audio translation

> the grand budapest hotel is both a sly crime caper and a charming ode to old...

> driverless cars will be on uk roads by january 2015

> space giants join forces to battle spacex : this is how cheap space travel begins

> weekend report :'captain america'wins close fight with'rio 2 '

> sc business notebook, may 24

> 21st century fox confirms rejected bid for time warner

> usher bounces his head on nicki minaj's butt at the 2014 mtv vmas : gif

> apple opens os x beta testing to all users with new seed program

> anthrax discovered in beef in hungary

> iowa farmer chris soules is abc's next'bachelor'| the republic

> murdoch names son lachlan as vice president of media empire

> cdc reports first chikungunya case acquired in the united states ; disease...

> shailene woodley on being cut from amazing spider - man 2 : " was i awful? "

> justina pelletier heads home after judge ends state custody

> singer chris brown's dc assault trial is delayed for months ; judge says singer to...

> android wear : 5 things developers need to know

> micro machine macro funding

> fcc forced to push back comment deadline on net neutrality rules

> hgtv slammed for excluding anti - gay christian consumers from america's...

> ' mom mobiles'a shrinking category for automakers

> malaysia airlines considers re - branding itself

> review : 50 cent's " animal ambition "

> hump day unusual moment : little roger & the goosebumps “ stairway to...

> women happier at work than home, study finds

> awfully good : sharknado 2

> annie leibovitz axed kim and kanye west wedding gig at last minute

> former astrazeneca chief executive attacks pfizer deal

> private funeral for mick jagger's longtime girlfriend, l'wren scott, held in los...

> government allots p6. 8m for aquino's trip to myanmar

> ( click the phrases to see a list )

> the - dream arrested for felony assault on pregnant ex - girlfriend

> kanye west gives 20 - minute speech, says the kardashians are'the most...

> team clones stem cells from 75 - year - old's skin

> sober smartphone app aids boozers

> spread of polio is now a world health emergency, u. n. says

> ' true blood'recap : [ spoiler ] is killed off — shocking death

> how game - changing was game of thrones'big reveal?

> alcohol costs us $ 224bn a year

> bmw investing $ 1 billion in mexican assembly plant

> report finds st. johns county florida's healthiest county

> giant of the skies was like'a dragon '

> beyonce named as world's most powerful celebrity


@sgugger  Hello, I try to reproduce this error. The texts above is the samples for finetuning for GPT2. It is the column of `text`.

```
train_file = './fintune_csvs/{}_train_finetune_32_{}.csv'.format(args.dsn, seed)
validation_file = './fintune_csvs/{}_test_finetune_32_{}.csv'.format(args.dsn, seed)


ds.df_train['text'] = ds.df_train['content'] + tokenizer_gpt2.eos_token
ds.df_test['text'] = ds.df_test['content'] + tokenizer_gpt2.eos_token

ds.df_train[['text']].sample(frac=1).to_csv(train_file, index=False)
ds.df_test[['text']].sample(frac=1).to_csv(validation_file, index=False)


model_output_path = "./finetune_gpt2/{}_32_{}".format(args.dsn, seed) 
os.system(
"CUDA_VISIBLE_DEVICES=1 python -u ./run_clm_no_trainer.py \
        --num_train_epochs {} \
        --train_file {} \
        --validation_file {} \
        --model_name_or_path gpt2 \
        --per_device_train_batch_size 16 \
        --per_device_eval_batch_size 16 \
        --output_dir {} \
        --preprocessing_num_workers 16 --overwrite_cache True \
        --block_size 256".format(args.ft_epochs, train_file, validation_file, model_output_path) ) 

```
`run_clm_no_trainer.py` is the official script from transformers repo.

When I use another dataset, which have longer sentences than this dataset, there is no error and the finetuning process is OK.



I also tried sentiment analysis dataset, which also consists of relatively short sentences. The error came out too.

> Grouping texts in chunks of 256 #11: 100%|█████████████████████████████| 1/1 [00:00<00:00, 25.61ba/s]
> Grouping texts in chunks of 256 #12: 100%|█████████████████████████████| 1/1 [00:00<00:00, 28.63ba/s]
> Grouping texts in chunks of 256 #13: 100%|█████████████████████████████| 1/1 [00:00<00:00, 25.03ba/s]
> Grouping texts in chunks of 256 #14: 100%|█████████████████████████████| 1/1 [00:00<00:00, 23.64ba/s]
> Grouping texts in chunks of 256 #15: 100%|█████████████████████████████| 1/1 [00:00<00:00, 30.86ba/s]
> 08/20/2021 03:43:32 - INFO - __main__ - ***** Running training *****
> 08/20/2021 03:43:32 - INFO - __main__ -   Num examples = 16
> 08/20/2021 03:43:32 - INFO - __main__ -   Num Epochs = 1                       | 0/1 [00:00<?, ?ba/s]
> 08/20/2021 03:43:32 - INFO - __main__ -   Instantaneous batch size per device = 16
> 08/20/2021 03:43:32 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 16
> 08/20/2021 03:43:32 - INFO - __main__ -   Gradient Accumulation steps = 1      | 0/1 [00:00<?, ?ba/s]
> 08/20/2021 03:43:32 - INFO - __main__ -   Total optimization steps = 1
>   0%|                                                                          | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
>   File "./run_clm_no_trainer.py", line 503, in <module>                        | 0/1 [00:00<?, ?ba/s]
>     main()
>   File "./run_clm_no_trainer.py", line 463, in main                            | 0/1 [00:00<?, ?ba/s]
>     for step, batch in enumerate(train_dataloader):
>   File "/usr/local/lib/python3.6/dist-packages/accelerate/data_loader.py", line 289, in __iter__
>     for batch in super().__iter__():
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
>     data = self._next_data()
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
>     return self.collate_fn(data)
>   File "/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py", line 80, in default_data_collator
>     batch[k] = torch.tensor([f[k] for f in features])
> ValueError: expected sequence of length 135 at dim 1 (got 112)

I try another manner to organise the training corpus, as txt file:
```
with open (train_file, 'w') as f:
    f.write(" {} ".format(tokenizer_gpt2.eos_token).join(ds.df_train['content'].tolist()))

with open (validation_file, 'w') as f:
    f.write(" {} ".format(tokenizer_gpt2.eos_token).join(ds.df_test['content'].tolist()))
```

The error comes the same.

> 33%|███▎      | 1/3 [00:00<00:01,  1.33it/s]Traceback (most recent call last):
>   File "./run_clm_no_trainer.py", line 483, in <module>
>     main()
>   File "./run_clm_no_trainer.py", line 460, in main
>     for step, batch in enumerate(eval_dataloader):
>   File "/usr/local/lib/python3.6/dist-packages/accelerate/data_loader.py", line 289, in __iter__
>     for batch in super().__iter__():
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
>     data = self._next_data()
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
>   File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
>     return self.collate_fn(data)
>   File "/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py", line 80, in default_data_collator
>     batch[k] = torch.tensor([f[k] for f in features])
> ValueError: expected sequence of length 256 at dim 1 (got 117)

Yes, this all points out to your corpus being too short to form a full batch. You should use a lower batch size or a lower block size.