run_clm with gpt2 and wiki103 throws ValueError: expected sequence of length 1024 at dim 1 (got 1012) during training.
Author: TrentBrick
Pull request: no
Labels: name: bug

### System Info

```shell
transformers              4.20.0.dev0               dev_0
python                    3.7.11               h12debd9_0
linux os (docker container).
```


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I run the following code using `run_clm.py`

`python run_clm.py --train_name testing-gpt2 --output_dir ../data/training_outputs/ --model_type gpt2 --dataset_name wikitext --dataset_config_name wikitext-103-v1 --tokenizer_name gpt2 --preprocessing_num_workers 30 --do_train --overwrite_output_dir --save_steps 100000000 --save_total_limit 2 --num_train_epochs 175 --config_overrides n_layer=3`


The error is: 

``` 
{'loss': 7.0678, 'learning_rate': 4.998998477686083e-05, 'epoch': 0.04}                                                      
  0%|                                                                             | 1000/2496200 [03:17<138:25:20,  5.01it/s]{'loss': 6.2992, 'learning_rate': 4.997996955372166e-05, 'epoch': 0.07}                                | 0/1 [00:00<?, ?ba/s]
  0%|                                                                             | 1500/2496200 [04:57<138:30:20,  5.00it/s]{'loss': 6.0153, 'learning_rate': 4.9969954330582484e-05, 'epoch': 0.11}
  0%|                                                                             | 2000/2496200 [06:35<137:22:45,  5.04it/s]{'loss': 5.8504, 'learning_rate': 4.995993910744331e-05, 'epoch': 0.14}                                | 0/1 [00:00<?, ?ba/s]
  0%|                                                                             | 2500/2496200 [08:14<136:41:10,  5.07it/s]{'loss': 5.6997, 'learning_rate': 4.994992388430414e-05, 'epoch': 0.18}
{'loss': 5.5797, 'learning_rate': 4.993990866116497e-05, 'epoch': 0.21}                                                      
  0%|                                                                             | 3500/2496200 [11:32<135:57:45,  5.09it/s]{'loss': 5.4796, 'learning_rate': 4.99298934380258e-05, 'epoch': 0.25}
{'loss': 5.3864, 'learning_rate': 4.991987821488663e-05, 'epoch': 0.28}                                                      
  0%|▏                                                                            | 4500/2496200 [14:50<137:03:49,  5.05it/s]{'loss': 5.3007, 'learning_rate': 4.990986299174746e-05, 'epoch': 0.32}
  0%|▏                                                                            | 5000/2496200 [16:29<139:10:06,  4.97it/s]{'loss': 5.2367, 'learning_rate': 4.989984776860829e-05, 'epoch': 0.35}
  0%|▏                                                                            | 5500/2496200 [18:08<137:22:34,  5.04it/s]{'loss': 5.16, 'learning_rate': 4.988983254546912e-05, 'epoch': 0.39}
  0%|▏                                                                            | 6000/2496200 [19:47<138:31:23,  4.99it/s]{'loss': 5.109, 'learning_rate': 4.987981732232994e-05, 'epoch': 0.42}
{'loss': 5.0511, 'learning_rate': 4.986980209919077e-05, 'epoch': 0.46}                                                      
  0%|▏                                                                            | 7000/2496200 [23:06<138:01:44,  5.01it/s]{'loss': 5.0114, 'learning_rate': 4.98597868760516e-05, 'epoch': 0.49}
  0%|▏                                                                            | 7500/2496200 [24:45<136:05:20,  5.08it/s]{'loss': 4.957, 'learning_rate': 4.984977165291243e-05, 'epoch': 0.53}
  0%|▏                                                                            | 8000/2496200 [26:24<137:43:44,  5.02it/s]{'loss': 4.8985, 'learning_rate': 4.983975642977326e-05, 'epoch': 0.56}
  0%|▎                                                                            | 8500/2496200 [28:04<136:25:23,  5.07it/s]{'loss': 4.8615, 'learning_rate': 4.982974120663408e-05, 'epoch': 0.6}
  0%|▎                                                                            | 9000/2496200 [29:43<137:32:41,  5.02it/s]{'loss': 4.8158, 'learning_rate': 4.981972598349491e-05, 'epoch': 0.63}
  0%|▎                                                                            | 9259/2496200 [30:35<137:00:18,  5.04it/s]Traceback (most recent call last):
  File "run_clm.py", line 649, in <module>
    main()
  File "run_clm.py", line 597, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/workspace/transformers/src/transformers/trainer.py", line 1327, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/workspace/transformers/src/transformers/trainer.py", line 1539, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/workspace/transformers/src/transformers/data/data_collator.py", line 67, in default_data_collator
    return torch_default_data_collator(features)
  File "/workspace/transformers/src/transformers/data/data_collator.py", line 131, in torch_default_data_collator
    batch[k] = torch.tensor([f[k] for f in features])
ValueError: expected sequence of length 1024 at dim 1 (got 1012)
```

Where every time I am getting this same error. I tried following https://stackoverflow.com/questions/71166789/huggingface-valueerror-expected-sequence-of-length-165-at-dim-1-got-128 by adding `padding='max_length'` to my tokenizer but then I get an error where this tokenizer does not have a padding token. 

NB. I think this only happens when `--preprocessing_num_workers 30` is not to a power of 2? Using 8, 16 or 128 avoids the problem. 

### Expected behavior

```shell
Model should keep training and not throw an error at training step: 9259
```


Comments:
Hi @TrentBrick ,

I think you can assign a custom padding token to GPT2, or discard the sequences that are too short (i.e. having < 1024 tokens)

@ydshieh is this common practice for GPT2? 

And I think the bigger issue is that an error is only thrown if my number of workers is not to the power of 2? 

Hi, I don't know if this is a common practice, but it is a reasonable approach. The important thing is to make sure the attention masks for those (meant to be padded) tokens to have mask value `0` when doing training.

Otherwise, you can always discard the short sequences (if it is the rare case).

I don't think the issue is coming from the number of workers, it is more about the sequence length.

See previous discussion
https://github.com/huggingface/transformers/issues/12594
https://github.com/huggingface/transformers/issues/2630

I'm telling you that empirically when I use `--preprocessing_num_workers n` where n is the power of 2 there is no error that gets thrown. It is only when it is not to a power of 2 that this problem appears. 

@TrentBrick 

I still believe it is not about `preprocessing_num_workers` being the power of 2 or not. However, this value might indeed have some effect. This method in `clm.py`
https://github.com/huggingface/transformers/blob/b424f0b4a301abcbf3c282114159371ee44c3e01/examples/pytorch/language-modeling/run_clm.py#L440

tries to group texts and split them into `block_size` (1024 here). So there should be no shorter sequence.

However, there is a condition
https://github.com/huggingface/transformers/blob/b424f0b4a301abcbf3c282114159371ee44c3e01/examples/pytorch/language-modeling/run_clm.py#L446

In some cases, it might happen a batched examples has very few number of examples, and `total_length < block_size`, so it is not thrown away. This indeed depends on `preprocessing_num_workers`. I think you can set a breakpoint around this place and use try /except to verify the situation.

I could talk to my colleague about this though - probably we can improve this condition here.

Hi @TrentBrick Could you try if the following change will work? Thanks.

Change
https://github.com/huggingface/transformers/blob/b424f0b4a301abcbf3c282114159371ee44c3e01/examples/pytorch/language-modeling/run_clm.py#L446-L448
to 
```python

if total_length >= block_size: 
     total_length = (total_length // block_size) * block_size 
else:
    total_length = 0
 # Split by chunks of max_len. 
```

Hi @TrentBrick, if you get the chance to verify it works, don't hesitate to open a PR if you would like to :-). Otherwise, I will open a PR later. Thank you for finding this issue!

As mentioned, it actually depends on the batches of examples received in the preprocessing function, and I don't try to run with 30 processes.

Close this issue - see [this comment](https://github.com/huggingface/transformers/pull/18304#pullrequestreview-1051141051)