Train robertatokenizer failed due to pad token not found
Author: pjuangph
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Windows
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7?  3080 RTX
- Tensorflow version (GPU?):
- Using GPU in script?: 
- Using distributed or parallel set-up in script?:

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Roberta

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
My first step is to download some of the esperberto data from the sites mentioned in this tutorial https://huggingface.co/blog/how-to-train 

Few issues
1. Regarding the tutorial, they make you train a ByteLevelBPETokenizer but this is never used in the training code. The training code isn't even in the tutorial 👎 
2. I came across this https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
It looks good except whatever ByteLevelBPETokenizer is never used in the training process so I tried to find a way to use it. I tried two approaches both result in the same outcome. I tried using the BPE and not ByteLevelBPETokenizer. I have no clue what is the best practice or why neither of them are working.

This is my code to do the tokenizer. You can uncomment whatever

```
#! pip install tokenizers
#%% Import Statements

from pathlib import Path
from transformers import RobertaTokenizer

from tokenizers import Tokenizer
from tokenizers.trainers import BpeTrainer
from tokenizers.models import BPE 
from tokenizers.pre_tokenizers import Whitespace

# from tokenizers import ByteLevelBPETokenizer
# from tokenizers.implementations import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing
import os.path as osp

#%% Train Tokenizer
if (not osp.exists('models/BPEtokenizer.json')):
    paths = [str(x) for x in Path("./eo_data/").glob("**/*.txt")]

    # Initialize a tokenizer
    # tokenizer = ByteLevelBPETokenizer()

    # # Customize training
    # tokenizer.train(files=paths, vocab_size=52000, min_frequency=3, special_tokens=[
    #     "
",
    #     "<pad>",
    #     "
",
    #     "<unk>",
    #     "<mask>"
    # ])

    tokenizer = Tokenizer(BPE())
    trainer = BpeTrainer(vocab_size=52000,min_frequency=3, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
    tokenizer.pre_tokenizer = Whitespace()
    tokenizer.train(trainer, paths)
    
    # Save files to disk    
    tokenizer.save('models/BPEtokenizer.json')

#%% Tokenize
tokenizer = Tokenizer.from_file('models/BPEtokenizer.json')
# tokenizer._tokenizer.post_processor = BertProcessing(
#     ("
", tokenizer.token_to_id("
")),
#     ("
", tokenizer.token_to_id("
")),
# )
# tokenizer.enable_truncation(max_length=512)
output = tokenizer.encode("Mi estas Julien.😁")
print(output.tokens)

print(output.ids)
# Encoding(num_tokens=7, ...)
# tokens: ['
', 'Mi', 'Ġestas', 'ĠJuli', 'en', '.', '
']
```


This is my code to do training 

```
import torch
from transformers import RobertaConfig
from transformers import RobertaTokenizerFast
from transformers import RobertaForMaskedLM
from transformers import LineByLineTextDataset
from transformers import DataCollatorForLanguageModeling
from pathlib import Path
from transformers import DataCollatorForLanguageModeling
from tokenizers import ByteLevelBPETokenizer
from transformers import PreTrainedTokenizerFast
from tokenizers import Tokenizer
# Tutorial from https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=BzMqR-dzF4Ro 

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# tokenizer = RobertaTokenizerFast.from_pretrained("./EsperBERTo", max_len=512)
# tokenizer = ByteLevelBPETokenizer("models/esperberto-vocab.json","models/esperberto-merges.txt") # ? This actually doesn't work. You will get an error saying tokenizer is not callable. 

tokenizer = PreTrainedTokenizerFast(tokenizer_file='models/BPEtokenizer.json')
# tokenizer = Tokenizer.from_file('models/BPEtokenizer.json')

mlm=False

config = RobertaConfig(
    vocab_size=52_000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

# Training from scratch 
model = RobertaForMaskedLM(config=config)
model.num_parameters()

paths = [str(x) for x in Path("eo_data/").glob("**/*.txt")]
# Build the dataset 

dataset = LineByLineTextDataset(tokenizer=tokenizer, file_path="eo_data/shuff-orig/eo/eo.txt",block_size=128)

# mlm = mask modeling language 
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=mlm, mlm_probability=0.15
    )



from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="models/EsperBERTo-small",
    overwrite_output_dir=True,
    num_train_epochs=1000,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset)

trainer.train()
```

I keep getting the error `Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.`

Also I couldn't set mlm=True either. Do you have any good tutorials on how to train your own set of data using Roberta? 

If anyone wants to pull my files you can grab them and the dataset here 
https://1drv.ms/u/s!Apa0_j-AivqTpqNz7r0M3NNhCm2W_A?e=BMLvqv

If you guys resolve this then I'll update and post a public google colab 





Comments:
Roberta was train on a causal language model objective, therefore the `LineByLineDataset` is not adapted to train it: it considers one line for one text when the roberta objective is to have several lines concatenated and separated by the sep token until it reaches the block size, to avoid padding. 

You need to use a different dataset for this. You should also check the new [`run_mlm` script](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) that offers both options.

Hi, I had the same issue, here are the workarounds I used
Platform: Ubuntu 18
Python version: 3.7.9
PyTorch version (GPU): 1.7.1 
cuda11

- save your tokenizer with `save_model()`, instead of `save()`, will save a `merges.json` and a` vocab.json`. 

`  tokenizer.save_model('models/BPEtokenizer')`

- you'll need `config.json`, a `tokenizer_config.json`  and a `special_tokens_map.json` files in your tokenizer repo, you can get them from the base model you want to use your tokenizer with, i.e. just quickly run the `run_mlm` script with 2 batches to get them and add them in your tokenizer repo.

I'm not sure the config.json is actually loaded, as it is the model config and not the tokenizer's, but the script wants is to accept your tokenizer path.
tokenizer repo should contain:
```
|__config.json
|__merges.txt
|__special_tokens_map.json
|__tokenizer_config.json
|__vocab.json
```

- in `tokenizer_config.json`, change the ` name_or_path:"roberta-base"` to `model_type: "roberta"`

- then train your model running the mlm script with your options and

`-- tokenizer_name ./models/BPEtokenizer`


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.