Need clarity on training Albert from scratch
Author: 008karan
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
Using transformers 2.9.1
I am following EsperBERTo tutorial for Training Albert LM from scratch. I am able to start training for roberta but for Albert getting tokenizer issue:
```
05/15/2020 16:23:29 - INFO - transformers.training_args -   PyTorch: setting up devices
05/15/2020 16:23:29 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: False
05/15/2020 16:23:29 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='./EsperBERTo-small-v1', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=0.0001, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=2000, save_total_limit=2, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False)
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Model name './EsperBERTo' not found in model shortcut name list (albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2). Assuming './EsperBERTo' is a path, a model identifier, or url to a directory containing tokenizer files.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/spiece.model. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/added_tokens.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/special_tokens_map.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file ./EsperBERTo/tokenizer_config.json
Traceback (most recent call last):
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 289, in <module>
    main()
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 188, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 203, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 902, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1055, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_albert.py", line 155, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 275, in _sentencepiece_processor_load
    return self._Load_native(model_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 75, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
TypeError: not a string
CPU times: user 13.6 ms, sys: 27.4 ms, total: 41.1 ms
Wall time: 1.12 s
```
I guess it was looking for spiece model. So I added spice.model in EsperBERTo folder and it started training.
Now I confuse as vocab it is using now is generated using Huggingface tokenizer and taking spice.model also... can any one clarify whats going on here? 


Comments:
The Albert original model has a SentencePiece tokenizer, so the `AlbertTokenizer` can only handle SentencePiece vocabularies right now. The `tokenizers` library doesn't handle these vocabularies as of now.

We're still thinking of how we should proceed when training a model from scratch with a tokenizer different to the original, as right now the model-tokenizer pairs have a fixed tokenizer backend. cc @julien-c @thomwolf 

@LysandreJik But  when I added spiece.model it started training. So is it taking vocab created by tokenizer library along with spiece.model?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Does this issue have been solved? When can we train the lm from scratch?

You can already train an LM from scratch, there are examples in `examples/language-modeling`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Are there any updates on training Albert from scratch? Perhaps via the Trainer?

Hi, a `DataCollatorForSOP` has been added to the library, which you can indeed use with the `Trainer` object. You can see it's implementation here:

https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/data_collator.py#L201-L206

> Hi, a `DataCollatorForSOP` has been added to the library, which you can indeed use with the `Trainer` object. You can see it's implementation here:
> 
> https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/data_collator.py#L201-L206

When i try to use this `DataCollatorForSOP` it says that it is deprecated and that i should use `DataCollatorForLanguageModeling`. It is however not clear whether this data collator also produces the objective sentence order prediction as in the original ALBERT paper. Can anyone verify whether it does this or not?