mBART 50
Author: Bachstelze
Pull request: no
Labels: name: New model

# mBART 50

## Additional languages to the mBART 25 model

The model is described in [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/pdf/2008.00401.pdf)

## Open source status

* [x] the model implementation is available: [mBART](https://github.com/pytorch/fairseq/tree/master/examples/mbart)
* [ ] the model weights are available: They should be released with the benchmark dataset, but I can't find them.
* [ ] who are the authors: @tangyuq @huihuifan @xianxl ...


Comments:
Thank you for your interest. We are doing a few clean-up and will release them shortly. 

Waiting for the model too :)

Thanks

@tangyuq Is the benchmark included in [Beyond English-Centric Multilingual Machine Translation](https://github.com/pytorch/fairseq/tree/master/examples/m2m_100)?

We'll be adding a full evaluation on the ML50 benchmark in an updated version. The main reason why I haven't added it yet is that we want to update ML50's evaluation pairs done on WMT20 to the WMT20 test sets. When ML50 was originally released, the WMT20 test sets were not released yet. So we'll update both works together. 

A portion of ML50 evaluation, from the original mBART paper, is available in Beyond English Centric MMT work already. 

Hi, thanks for the great work and releasing the models to public.
Do we have any update on mBART50 model release? 

The [mBART50 model](https://github.com/pytorch/fairseq/tree/master/examples/multilingual#mbart50-models) is now available on the fairseq repository.

@jaspock Has someone been able to load these mBART50 models with huggingface mBART version?
I was able to convert it (and the weight appear to be right). However, when generating translations, it just repeats the same word over and over again.


Maybe @patil-suraj has some input?

Hi @jaspock 

I'm also facing the same issue, I've been able to convert the weights, but as you said the generation is not working, I'm investigating this currently.

mBART-50 is now available on master!

https://huggingface.co/transformers/master/model_doc/mbart.html#overview-of-mbart-50

https://huggingface.co/models?filter=mbart-50

Hi, I want to finetune the multilingual model for some languages.
I cannot find the exact script , number of updates used to finetune mbart50.
The paper suggested 500k updates for training the model that was trained from scratch. Paper [Link](https://arxiv.org/pdf/2008.00401.pdf)
The multilingual fine-tuning script shows 40k updates in the training [script](https://github.com/pytorch/fairseq/tree/master/examples/multilingual#mbart50-models) (Which was also used for Bilingual model in mBart paper)

But I cannot find specifications for MT-FT (Multilingual Fine Tuning) for 50 languages.
I suspect 40k won't be enough if combined data for all languages is high.
Can anyone guide me if I am missing something


> Hi, I want to finetune the multilingual model for some languages.
> I cannot find the exact script , number of updates used to finetune mbart50.
> The paper suggested 500k updates for training the model that was trained from scratch. Paper [Link](https://arxiv.org/pdf/2008.00401.pdf)
> The multilingual fine-tuning script shows 40k updates in the training [script](https://github.com/pytorch/fairseq/tree/master/examples/multilingual#mbart50-models) (Which was also used for Bilingual model in mBart paper)
> 
> But I cannot find specifications for MT-FT (Multilingual Fine Tuning) for 50 languages.
> I suspect 40k won't be enough if combined data for all languages is high.
> Can anyone guide me if I am missing something

Hi @kaivu1999 Did you figure out how to fine-tune? I am also exploring for finetuning the multilingual model but haven't found a proper specification yet.

Hi @kaivu1999, @pdakwal 

> I cannot find the exact script 

There is a `run_translation.py` script [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/translation), which can be used for fine-tuning mBART-50 but it does not support multi-lingual fine-tuning. But it should be easy to adapt to do that.

For multi-lingual fine-tuning, what we need to do here is, say we are fine-tuning on two language pairs, in that case, we need to concatenate the two datasets or in case the two language pairs don't have the same number of examples then add some sort of sampler which will sample the example from the datasets depending on the number of examples in each one. And when processing each language pair, set the appropriate `src_lang` and `tgt_lang` tokens. The processing part is explained in the [docs](https://huggingface.co/transformers/model_doc/mbart.html#overview-of-mbart-50).

Hi @kaivu1999, @pdakwal  were u able to finetune mbart50 on custom data? Please help.