Unable to pre-train Roberta from scratch using example/run_mlm.py script
Author: sarthusarth
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.12
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: No

### Who can help?

I trained a custom tokenizer by and tried pre-training Roberta using the official script https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py

Using the parameters:

python run_mlm.py \
    --model_type roberta \
    --tokenizer_name new_gcs/cnn_final/ \
    --dataset_name new_gcs/cnn_final/ \
    --max_seq_length 512 \
    --line_by_line true \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train true\
    --do_eval true \
    --output_dir ./test-mlm

But I get this error and I checked model.vocab is same as len(token) and max_token in sample is 512 as well. 
transformers versions is: '4.27.0.dev0'



File "run_mlm.py", line 632, in <module>
    main()
  File "run_mlm.py", line 581, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1635, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1898, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 2640, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 2672, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 1109, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 850, in forward
    past_key_values_length=past_key_values_length,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 128, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 2199, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
  0%|                                                                                                                      | 0/8883 [00:00<?, ?it/s]
@sgugger @ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running the official scipt: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py

### Expected behavior

Should triain the model

Comments:
roberta is coded in a hacky way which requires you to set its `max_position_embeddings` to the len of the tokenizer +2 (for instance its 514 for `roberta-base`).

How can I set that in the script?

By changing the line creating the config for instance.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.