MBART/MBART-50 PreTraining ? 
Author: yash-srivastava19
Pull request: no

I am trying to pre-train a  MBART-50 model from scratch(using the IndicCorpus dataset), similar to the one given in this blog post(https://huggingface.co/blog/how-to-train). There seems to be no issue when I do for RoBERTa and BART, but for MBART and MBART-50 architecture, the model loading fails, and I get the following error message : 

`OSError: Can't load tokenizer for '/path/to/mbart-model. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './assamese_MBART' is the correct path to a directory containing all relevant files for a MBartTokenizerFast tokenizer.`

I wanted to ask, whether it is an issue with the way I have implemented the tokenizer(similar to the one given the blog post(without adding language codes and special tokens)) ;  or the way the data is there(Line by Line in the language) or it is something else altogether.

If there is any other clarification from my side, I'm more than happy to justify. @patrickvonplaten   

Comments:
cc @ArthurZucker 

Could you share a minimal reproducing script to isolate the bug that you are getting? 
Seems like the path that is trying to be reached is `./assamese_MBART`, check that the local file has a tokenizer, maybe use `use_local` only.

Hi !

Similar to the one given in the blog post, I started with something like : 
```
from tokenizers import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing

tokenizer = ByteLevelBPETokenizer(
    "/content/assamese_BART/vocab.json",
    "/content/assamese_BART/merges.txt",)

tokenizer._tokenizer.post_processor = BertProcessing(
    ("
", tokenizer.token_to_id("
")),
    ("
", tokenizer.token_to_id("
")),
)

tokenizer.enable_truncation(max_length=512)
```
This seems to run fine, but here : 

```
config = MBartConfig( vocab_size=50000,max_position_embeddings=32,num_attention_heads=2,
    num_hidden_layers=1,type_vocab_size=1,
)

tokenizer = MBart50TokenizerFast.from_pretrained("./assamese_BART", max_len=64)

model = AutoModelForMaskedLM.from_config(config=config)
print(model.num_parameters())
```
In the `MBart50TokenizerFast.from_pretrained` line, it is showing that it doesn't recognize the path, but I used the same path to train two other models(RoBERTa and BART), and this exception was not raised.  

Also, the local file has the tokenizer(`vocab.json` and `merges.txt`) in the same folder `./assamese_MBART`

I'll try to use the `use_local` argument and let you know shortly..

The`use_local` argument didn't work either. I reckon the issue is due to the `BertProcessing`, which worked for BART and RoBERTa, but not for MBart50.

I look forward to your answer. 

The blogpost is 3 years old, so very much outdated. If you look at the script you gave me, you are never saving the tokenizer to the `./assamese_BART` repo.  

I agree that the blog post is a little to old, but I couldn't find any relevant articles anywhere.  

> The blogpost is 3 years old, so very much outdated. If you look at the script you gave me, you are never saving the tokenizer to the `./assamese_BART` repo.

I did save it, just that I forgot to include it in the code. The only problem is that I can't figure out why is it working for RoBERTa and BART, but not MBart50, as I need to implement that only 😿. 

Also, the log produced : 
` ... Otherwise, make sure './assamese_BART' is the correct path to a directory containing all relevant files for a MBart50TokenizerFast tokenizer`

Apart from `vocab.json` and `merges.txt` does MBart50Tokenizer needs some other file due to which it is not reading it?

If you could point me/tell me how I can do it would be really really helpful 🙏🏻     

No worries. Could you just give me a full reproducing script of what your are doing with MBart50,that way I can check if we indeed have aproblem with the blogpost and it might have to be update! 

I somehow managed to make it work by using `SentencePiece` tokenizer instead of the one given in the script. It now reads the spm file and works when it reads the `spm.model` file. This is what I did : 

```
import sentencepiece as spm 

spm.SentencePieceTrainer.Train(
    input='/content/as_mod.txt',
    model_prefix='spm',
    vocab_size=1000,
    pad_piece='<pad>',
    bos_piece='
',
    eos_piece='
',
    user_defined_symbols='<mask>',
    model_type='unigram'
)

sp = spm.SentencePieceProcessor()
sp.Load('spm.model')
```

Now, instead of the `from_pretrained` method, I directly use the `spm.model` file. Code for that : 
```
config = MBartConfig(
    vocab_size=1000,
    max_position_embeddings=32,
    num_attention_heads=2,
    num_hidden_layers=1,
    type_vocab_size=1,
)

# TODO: I believe, due to the BERTPreProcessor, the MBART doesn't seem to recognizes it.
tokenizer = MBartTokenizerFast(vocab_file='/content/spm.model')

model = AutoModelForMaskedLM.from_config(config=config)
```
This seems to work, and it can read the model. However, now when I try to train the model, it gives an error which I haven't even seen in this context. First, the `Trainer` and `TrainerArguments` code :

```
training_args = TrainingArguments(
    output_dir='/content/assamese_BART',
    num_train_epochs=1,
    per_device_train_batch_size=32,
    save_steps=5000,
    save_total_limit=1,
    prediction_loss_only=True
)

# Set the trainer. 
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset
)
``` 
And on running `trainer.train()`, I come across this scary error : 

```
/usr/local/lib/python3.10/dist-packages/transformers/models/mbart/modeling_mbart.py in shift_tokens_right(input_ids, pad_token_id)
     73 
     74     index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1)
---> 75     decoder_start_tokens = prev_output_tokens.gather(1, index_of_eos).squeeze()
     76     prev_output_tokens[:, 1:] = prev_output_tokens[:, :-1].clone()
     77     prev_output_tokens[:, 0] = decoder_start_tokens

RuntimeError: index -1 is out of bounds for dimension 1 with size 16
```

I can't seem to find anything on this anywhere. Can I get some help in this regard? This is when I didn't use GPU from colab environment, but when I used GPU the error was : 

```
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect
```

This just means that the tokens outputed by the tokenizer are not in the correct format. I recommend using `tokenizers` and the `tokenizers` from tranformers, otherwise your are mixing two libraries, which are not made to work together.

Ok. Will do 👍🏻 

> No worries. Could you just give me a full reproducing script of what your are doing with MBart50,that way I can check if we indeed have a problem with the blogpost and it might have to be update!

I'll link the Colab notebook here for your reference, if there is anything missing just tell : 
https://colab.research.google.com/drive/1cNfQn9nPITpwCS4i8P8loeN2SjLON1Tx?usp=sharing

Thanks for the collab. I can't help you if you keep using the spm model raw. I can help you regarding the original issue, that is if the tokenizer you are using is from `tokenizers` or `transformers`! 😉 

Yeah, my bad. I accidentally gave gave you the different version of the same repo. Just wait a minute ...

> Thanks for the collab. I can't help you if you keep using the spm model raw. I can help you regarding the original issue, that is if the tokenizer you are using is from `tokenizers` or `transformers`! wink

https://colab.research.google.com/drive/1cNfQn9nPITpwCS4i8P8loeN2SjLON1Tx?usp=sharing

Check(I think it's the same link, but the changes are made)

Notebook doesn't work for me! Could you check? (ALso I think you gave me writing rights 😅 so I might have changed things, dont't give them to me I'll copy your notebook)

Sorry for the late reply...

> Notebook doesn't work for me! Could you check? (ALso I think you gave me writing rights sweat_smile so I might have changed things, don't give them to me I'll copy your notebook)

What exactly doesn't work?  Can you access the data ?

Changed the settings to view only.
https://colab.research.google.com/drive/1cNfQn9nPITpwCS4i8P8loeN2SjLON1Tx?usp=sharing

I think it should work just fine...

Again, the notebook does not work out of the box. Try to open a private tab and run it without the cached inputs. Anyway after debugging, it's just that you are trying to convert a BertTokenizer (which is a `simple` basic tokenizer) to a MBart50 tokenizer, which is based on `sentencepiece`. This is impossible and this is also the reason why it is failing : the `.json` file is ignored because it is looking for a `.bpe` file. 

Ok , I think I understand the problem. Thanks a lot for being patient with me, as this was my first issue. I tried to do my best here.