fine-tunes RoBERTa on WikiText-2
Author: skye95git
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.9.12
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@LysandreJik   @sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The example in `transformers/examples/pytorch/language-modeling/`: https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling

```
python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-mlm
```

### Expected behavior


When I run
```shell
wikitext_dataset = load_dataset('wikitext', 'wikitext-2-v1')
print(wikitext_dataset)
```
The log is as follow:
```
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 4358
    })
    train: Dataset({
        features: ['text'],
        num_rows: 36718
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 3760
    })
})
```
But when I run he previous fine-tune code, The log is as follow:
```
[INFO|trainer.py:1469] 2022-06-11 20:59:08,340 >> ***** Running training *****
[INFO|trainer.py:1470] 2022-06-11 20:59:08,340 >>   Num examples = 4798
[INFO|trainer.py:1471] 2022-06-11 20:59:08,340 >>   Num Epochs = 1
[INFO|trainer.py:1472] 2022-06-11 20:59:08,340 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:1473] 2022-06-11 20:59:08,340 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:1474] 2022-06-11 20:59:08,340 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1475] 2022-06-11 20:59:08,340 >>   Total optimization steps = 1800
```
1. Why is the examples number in training(**4798**) different from the examples number(**36718**) in the dataset?
2. 4789 doesn't seem to match the Wikitext data
![image](https://user-images.githubusercontent.com/41561936/173191149-6155a669-b9d7-4392-a56e-3b9b93d9f28f.png)

4. The epoch seems to be 2 every time I finetune it. How to modify the epoch when fine-tuning?



Comments:
Please use the [forums](https://discuss.huggingface.co/) to ask questions like this and help debug your code as we keep issues for feature requests and bugs only.

The texts are all concatenated and then split in blocks of the block size you pass to the script, so your dataset is not composed of documents of wikitext-2, but parts of the concatenation of all documents of wikitext-2.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.