[ported model] FSMT (FairSeq MachineTranslation)
Author: stas00
Pull request: yes
Labels: name: model card

This PR implements the spec specified at https://github.com/huggingface/transformers/issues/5419

The new model is FSMT (aka FairSeqMachineTranslation): `FSMTForConditionalGeneration` which comes with 4 models:

 *   "facebook/wmt19-ru-en"
 *   "facebook/wmt19-en-ru"
 *   "facebook/wmt19-de-en"
 *   "facebook/wmt19-en-de"

This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) which includes 3 languages and 4 pairs.

For more details of the original, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).

**Huge, huge thanks to @sshleifer, who has been incredibly supportive of this very difficult, yet, fun learning experience! Thank you, Sam!**

**And many thanks to all those who wrote all the existing transformers code, so that I just needed to tweak a few things here and there, rather than write from scratch. And, last, but not least, to the fairseq developers, who have done the heavy lifting with the initial training and finetuning, and coding.**

The tokenizer is a tweaked XLM tokenizer, the model is a tweaked Bart model. There were too many differences that I couldn't just subclass either of these 2, having 2 unmerged dictionaries of different sized being the main cause. But there were quite a few other nuances, please see the porting notes in the code.

There are a few more things to complete, in particular we currently don't have support for model ensemble, which is used by fairseq - they run eval on an ensemble of 4 model checkpoints. This implementation currently uses only the first checkpoint.
And then more work on matching fairseq outputs is needed - no beam is perfect, and with beam search there are some small differences - I was encouraged to release the model and continue working on improving it. 

I'm still a few points behind on the BLEU score - most likely due to having the ensemble, but since I am not able to reproduce fairseq reported scores, I'm not sure how to evaluate against a single model. See the [issue](https://github.com/pytorch/fairseq/issues/2544). I added the current and the expected scores in the model cards. If one of you has already started working on ensemble support please let me know.

You will find 'Porting Notes' in `modeling_fsmt.py` and `tokenization_fsmt.py` with what has been done, nuances and what still needs to be done.

The 4 models are up on s3 and can be used already.

Usage:

```python
from transformers.tokenization_fsmt import FSMTTokenizer
from transformers.modeling_fsmt import FSMTForConditionalGeneration
mname = "facebook/wmt19-en-ru"
tokenizer = FSMTTokenizer.from_pretrained(mname)
model = FSMTForConditionalGeneration.from_pretrained(mname)

input = "Machine learning is great, isn't it?
input_ids = tokenizer.encode(input, return_tensors="pt")
outputs = model.generate(input_ids)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # Машинное обучение - это здорово, не так ли?
```

**edit**: we have 5 more wmt models en/de from https://github.com/jungokasai/deep-shallow/ ready to be added as well, once this is merged.

@sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=h1) Report
> Merging [#6940](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/90cde2e938638e64a8696a12b79ee5f52364b162?el=desc) will **increase** coverage by `2.47%`.
> The diff coverage is `94.60%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6940/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6940      +/-   ##
==========================================
+ Coverage   79.62%   82.10%   +2.47%     
==========================================
  Files         168      171       +3     
  Lines       32284    33044     +760     
==========================================
+ Hits        25706    27130    +1424     
+ Misses       6578     5914     -664     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_fsmt.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19mc210LnB5) | `93.58% <93.58%> (ø)` | |
| [src/transformers/tokenization\_fsmt.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZnNtdC5weQ==) | `95.23% <95.23%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.35% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `96.15% <100.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/configuration\_fsmt.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2ZzbXQucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `82.38% <100.00%> (+0.08%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `91.93% <100.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.55% <0.00%> (-34.28%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.84% <0.00%> (-23.17%)` | :arrow_down: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `70.19% <0.00%> (-23.08%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/huggingface/transformers/pull/6940/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=footer). Last update [90cde2e...1be40e3](https://codecov.io/gh/huggingface/transformers/pull/6940?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Here is a little paraphrase script to amuse you:

```python
from transformers.tokenization_fsmt import FSMTTokenizer
from transformers.modeling_fsmt import FSMTForConditionalGeneration

text = "Every morning when I wake up, I experience an exquisite joy - the joy of being Salvador Dalí - and I ask myself in rapture: What wonderful things is this Salvador Dalí going to accomplish today?"

def translate(src_lang, tgt_lang, text):
    mname = f"facebook/wmt19-{src_lang}-{tgt_lang}"
    tokenizer = FSMTTokenizer.from_pretrained(mname)
    model = FSMTForConditionalGeneration.from_pretrained(mname)

    input_ids = tokenizer.encode(text, return_tensors='pt')
    outputs = model.generate(input_ids, num_beams=5, early_stopping=True)
    decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return decoded

def paraphrase(src_lang, tgt_lang, text):
    return translate(tgt_lang, src_lang, translate(src_lang, tgt_lang, text))

print(f"original:\n{text}")
print(f"paraphrased en-ru-en:\n{paraphrase('en', 'ru', text)}")
print(f"paraphrased en-de-en:\n{paraphrase('en', 'de', text)}")
```

* original: 

   Every morning when I wake up, I experience an exquisite joy - the joy of being Salvador Dalí - and I ask myself in rapture: What wonderful things is this Salvador Dalí going to accomplish today?

* paraphrased en-ru-en: 

   Every morning when I wake up, I have an amazing joy - the joy of being Salvador Dali - and I ask myself in awe: what wonderful things is this Salvador Dali going to do today?

* paraphrased en-de-en:

   Every morning when I wake up, I experience an exquisite joy - the joy of being Salvador Dalí - and I ask myself in ecstasy: what wonderful things will this Salvador Dalí do today?

Dali would have been proud! :)

Hi, @stas00 
Can the models be torchscripted or quantized?
I understand they are from fairseq and are pre-trained. What about optimizations in training a seq2seq model in transfomers?

Also the integration test fails in my torch 1.5.1 environment: https://gist.github.com/sshleifer/4ba0386e06d2b348c809f80c19f283fd


Super excited about this!

> Hi, @stas00
> Can the models be torchscripted or quantized?
> I understand they are from fairseq and are pre-trained. What about optimizations in training a seq2seq model in transfomers?

The first step is just to make things work and have a similar BLEU performance. At a later stage we can work on more goals. The plan is to polish this PR, have it merged and then I plan to post to the forums and then you guys can experiment, report problems, ask for things, etc. How does that sound?


> Have not read modeling.py yet, but left some other nitpicks.

Thank you very much, @sshleifer  - I will address those later today.

> More importantly, I couldn't replicate `run_eval.py` results from this branch.

I know why. I uploaded an experimental version of the models last night, thought I forced the caching off, as the models were re-downloaded, but just now while re-running run_eval I got suddenly a re-download and blue is 0.1. So the experimental model didn't work. :(

So I still need to sort out the caching issue: https://github.com/huggingface/transformers/issues/6916

I'm reverting the models - takes a while to upload 5GB. I will update once this is complete and then you can re-eval.

----

I'm also thinking - needing an actual run_eval quality test, which can be run as a part of the test suite. perhaps on a small sample, maybe 100 instead of 2000 and a smallish beam size? then it can be slow, but not too slow? 

----

Also, as I mentioned earlier there is no way to override `num_beans` in run_eval so one has to manually change it in configuration_fsmt.py. 

So you were running it with `num_beans=8`.

Here are the results that I get for `PAIR=en-ru`:
```
#  15:
# {'bleu': 31.2512, 'n_obs': 1997, 'runtime': 521, 'seconds_per_sample': 0.2609}
#  50:
# {'bleu': 31.2695, 'n_obs': 1997, 'runtime': 1692, 'seconds_per_sample': 0.8473}
```

I will rebase, once this is merged https://github.com/huggingface/transformers/pull/6948 - thank you!


**edit**: CDN has been updated so you're good to go to eval the model.

So models have been updated, but I can't figure out how to bypass caching, so still getting the old versions - might have to wait 24h :( See this issue: https://github.com/huggingface/transformers/issues/6916#issuecomment-687321087

So until this caching issue is sorted out (or 24h have passed) please don't waste your time on trying to eval this model. It won't work.

I wrote a bash script that `run_eval.py`s each of 4 checkpoints separately for each pair. So let's see which is the winner and use that one for now (could be different for different models):

```

export BS=8
# set to 5 for a quick test run, set to 2000 to eval all available records
export OBJS=2000
# at the end we want NUM_BEAMS=50 (as that's what fairseq used in their eval)
export NUM_BEAMS=50

pairs=(ru-en en-ru en-de de-en)
for pair in "${pairs[@]}"
do
    export PAIR=$pair
    export DATA_DIR=data/$PAIR
    export SAVE_DIR=data/$PAIR
    mkdir -p $DATA_DIR
    sacrebleu -t wmt19 -l $PAIR --echo src | head -$OBJS > $DATA_DIR/val.source
    sacrebleu -t wmt19 -l $PAIR --echo ref | head -$OBJS > $DATA_DIR/val.target

    if [[ $pair =~ "ru" ]]
    then
        subdir=ensemble # ru folders
    else
        subdir=joined-dict.ensemble # de data folders are different
    fi

    END=4;
    for i in $(seq 1 $END);
    do
        model=model$i.pt;
        CHKPT=$model PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19.$PAIR.$subdir --pytorch_dump_folder_path data/fsmt-wmt19-$PAIR > log.$PAIR-$model 2>&1
        echo "###" $PAIR $model num_beams=$NUM_BEAMS objs=$OBJS
        PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py /code/huggingface/transformers-fair-wmt/data/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation 2> /dev/null
    done

    echo
    echo
done
```
If someone decides to run this, you have to modify `convert_fsmt_original_pytorch_checkpoint_to_pytorch.py` to get the checkpoint name from `os.getenv("CHKPT")`

Results:

```
### ru-en model1.pt num_beams=50 objs=2000
{'bleu': 38.8222, 'n_obs': 2000, 'runtime': 233, 'seconds_per_sample': 0.1165}
### ru-en model2.pt num_beams=50 objs=2000
{'bleu': 38.4053, 'n_obs': 2000, 'runtime': 233, 'seconds_per_sample': 0.1165}
### ru-en model3.pt num_beams=50 objs=2000
{'bleu': 38.8222, 'n_obs': 2000, 'runtime': 234, 'seconds_per_sample': 0.117}
### ru-en model4.pt num_beams=50 objs=2000
{'bleu': 38.933, 'n_obs': 2000, 'runtime': 236, 'seconds_per_sample': 0.118}


### en-ru model1.pt num_beams=50 objs=2000
{'bleu': 31.2898, 'n_obs': 1997, 'runtime': 295, 'seconds_per_sample': 0.1477}
### en-ru model2.pt num_beams=50 objs=2000
{'bleu': 31.4669, 'n_obs': 1997, 'runtime': 293, 'seconds_per_sample': 0.1467}
### en-ru model3.pt num_beams=50 objs=2000
{'bleu': 33.4736, 'n_obs': 1997, 'runtime': 289, 'seconds_per_sample': 0.1447}
### en-ru model4.pt num_beams=50 objs=2000
{'bleu': 33.2873, 'n_obs': 1997, 'runtime': 296, 'seconds_per_sample': 0.1482}


### en-de model1.pt num_beams=50 objs=2000
{'bleu': 40.7906, 'n_obs': 1997, 'runtime': 304, 'seconds_per_sample': 0.1522}
### en-de model2.pt num_beams=50 objs=2000
{'bleu': 40.7677, 'n_obs': 1997, 'runtime': 307, 'seconds_per_sample': 0.1537}
### en-de model3.pt num_beams=50 objs=2000
{'bleu': 40.7677, 'n_obs': 1997, 'runtime': 306, 'seconds_per_sample': 0.1532}
### en-de model4.pt num_beams=50 objs=2000
{'bleu': 42.7892, 'n_obs': 1997, 'runtime': 305, 'seconds_per_sample': 0.1527}


### de-en model1.pt num_beams=50 objs=2000
{'bleu': 39.4096, 'n_obs': 2000, 'runtime': 238, 'seconds_per_sample': 0.119}
### de-en model2.pt num_beams=50 objs=2000
{'bleu': 39.3009, 'n_obs': 2000, 'runtime': 238, 'seconds_per_sample': 0.119}
### de-en model3.pt num_beams=50 objs=2000
{'bleu': 38.9375, 'n_obs': 2000, 'runtime': 238, 'seconds_per_sample': 0.119}
### de-en model4.pt num_beams=50 objs=2000
{'bleu': 41.1808, 'n_obs': 2000, 'runtime': 237, 'seconds_per_sample': 0.1185}
```

So the differences between checkpoints are quite significant, clearly the 4th checkpoint outperforms them all for each pair.

Here is where we are at right now BLEU score-wise: (w/ `num_beams=50`) (after switching to using the 4th checkpoint file):

pair | fairseq | transformers
-------|----|----------
"en-ru"|36.4| 33.29
"ru-en"|41.3| 38.93
"de-en"|42.3| 41.18
"en-de"|43.1| 42.79

We are very close with de/en/de, but 2-3 points below on ru/en/ru

So wrt/ model ensemble - as it was suggested transformers currently won't support that mechanism - so do we just stop here and release this ported version with a slight handicap? 

I'm going to work on the remaining divergence in beam search and may be score a little bit more. But I doubt we will get to the same level w/o ensemble.

p.s. I'm uploading new models, so in about 11 hours the CDN cache should update, if you want to validate these numbers.

1) We can definitely merge this PR without ensemble, smart move checking each .pt file.
2) Would be good to figure out how to eval 1 model.pt file with `fairseq-generate` so that we can figure out whether the discrepancy is from anything besides ensembling.



> Would be good to figure out how to eval 1 model.pt file with `fairseq-generate` so that we can figure out whether the discrepancy is from anything besides ensembling.

I agree. That would help a lot! But no word yet from @edunov: https://github.com/pytorch/fairseq/issues/2544



wrt to shrinking it and reusing more from bart: probably it would take modifying bart to work with two vocabs and fall back on one, e.g. `if tgt_vocab is None: tgt_vocab_size = src_vocab_size`? That's the major reason for the "fork".

Then we can definitely fold most of it back with a few extra flags.

Here is an update on fairseq bleu scores validation. Got a [reply with great instructions](https://github.com/pytorch/fairseq/issues/2544#issuecomment-688054859) from @edunov and as a result I was able to get 35.7 with 4 models and 36.0 with model4 (en-ru pair). Sergei suggests that one more step is needed to re-rank the scores to reach the reported in the paper 36.4 score. Our best score at the moment is 33.29 for this pair. (other pairs are much closer to the goal than this one).

So now I know we are comparing apples to apples and I have more figuring out to do.

@sshleifer, I added a new test `test_bleu_scores` that evals bleu score on a small batch, which I think is very useful for regression testing - as it's now built into the test suite. it's the same speed as other integration tests (model loading still takes much longer). Surely, it gives about 2/3rd of the best score, but it's enough to detect a regression in the model.

I added caching so now it should be almost as fast to have many more integration tests.

Question: currently I had to hack `sys.path` to get to the code in `examples/seq2seq`.

```
# XXX: make calculate_bleu accessible to integration tests?
examples_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "examples"))
sys.path.insert(0, examples_dir)
from seq2seq.utils import calculate_bleu  # noqa
```

Is it time we make this and similar functions available to our "normal" integration tests? Thoughts?

**edit**: I ended up just copying the function as it's just 1 line.

but then of course CI doesn't have `sacrebleu` installed.

Where is the information on how/where slow tests are run - i.e. not on CI? I understand they are being run, just don't know where and how to see the status. Then need to add `sacrebleu` to the requirements file of that special CI.

Tried to answer your CI Q: https://discuss.huggingface.co/t/circleci-github-actions-which-tests-run-where-and-when/1042

In terms of moving the scorers, you will encounter some resistance because of the need to add dependencies.

You could just put your test in `examples/seq2seq/test_fsmt_bleu_score.py`. 

`sacrebleu` is in `examples/requirements.txt` so the self-scheduled nightly CI will run it (hopefully).

OK, the model has been decoupled from Bart and a lot of the unneeded code removed. 

Please let me know if anything else is needed. Thank you.

Finally, before merging this, let's discuss the model naming with thinking into the future.

What we give user is not fairseq, but a wmtXX-based/trained model, so perhaps fairseq shouldn't be anywhere in the name. 

fairseq have done other wmt datasets in the past. And so very likely to do wmt20 and others. If such future versions of the model are not much different perhaps those too could be folded into this model, therefore it shouldn't be hardwired to wmt19.

They call their line of "products" related to wmt: `transformer.wmt\d\d.\w\w-\w\w` (`transformer.wmt19.en-ru`, `transformer.wmt20.de-en`, etc.

Perhaps therefore `TWMT` is most fitting then as the base name? As in `TransformerWMT`, but shorter? So we end up with:

* `TWMTTokenizer`
* `TWMTForConditionalGeneration`

Thoughts?

p.s. for better context, the loading code for fairseq wmt is:

```
model = torch.hub.load('pytorch/fairseq', "transformer.wmt19.en-ru", 
checkpoint_file='model1.pt:model2.pt:model3.pt:model4.pt', tokenizer='moses', bpe='fastbpe')
```

- transformer is unhelpful -- all models in this lib are transformers.
- `FairseqMTModel` works for me -- its analogous to `MarianMTModel`.

@thomwolf: this is good from my side. You may have opinions on naming/nonstandard use of `DecoderConfig` 


> transformer is unhelpful -- all models in this lib are transformers.

agreed!

> FairseqMTModel works for me -- its analogous to MarianMTModel.

Too big of a scope? this one is wmt-specific.

Perhaps `FWMT`? as in FairseqWMT

and lowercased `fairseqmt` where it's needed sucks readability-wise, full abbreviation `fmt` works much better - i guess that's why it's `modeling_marian.py` and not `modeling_marianmt.py`.

Or just `WMT` from `wmtxx` series?

>   nonstandard use of `DecoderConfig`

this is a non-standard model with 2 vocabs of different sizes, that if I'm correct is the first one in the family, so it calls for a non-standard solution in lieu of changing the core functions to support such models. 

There are at least 3 other hacks I had to add in the tokenizer and the model/config to fit into the current world of "same size src/tgt vocab". And there is at least one core function (resize) that will most likely break on this model, since it resizes to the same size, but we haven't had a need for it so far.

p.s. Oddly enough their en-de/de--en models are of the same size merged vocabs, but ru-en/en-ru are not.


I wrote a script that translates with fairseq/model4 and this model based on model4 side by side and comparing the outputs.
I fed it all of sacrebleu eval text, so out of 8000 sentences there were ~10 mismatches - the rest matches up perfectly on the top ranking beam choice (beams=5). Excellent!

Yet, we are still behind on the bleu scores. 

We don't have (1) the model ensemble, and also (2) the re-ranking algorithm that they use, which is responsible for the extra points.

Bringing some of the insights from porting allen nlp models at https://github.com/huggingface/transformers/issues/7049, I tried to re-run eval with  `len_penalty=0.6` (until now we used the default  `len_penalty=1.0`). 

And yes, for 3 out of 4 models we get a significant improvement.

| pair    | fairseq  +rerank | fairseq -rerank |  transformers |
| ------- |        --------- |         --------- | ------------- |
| "ru-en" |             41.3 |             38.55 |         38.14/39.05 |
| "en-ru" |             36.4 |             31.26 |         32.76 |
| "en-de" |             43.1 |             40.88 |         42.23 |
| "de-en" |             42.3 |             39.38 |         40.71 |

We score higher on ru-en with `len_penalty=1.0` 38.8524, vs `38.13` with `len_penalty=0.6`. Rerunning with `len_penalty=1.1`, I get `39.0498` - almost a point higher!

I'm not sure how to get the  `len_penalty` used by fairseq - this data is not being shared, other than the paper alluding that they found the best fit by searching the space.

I suppose we could search too, but how are we to know that finding the length penalty that leads to the highest bleu score on 2000 items is generic enough to lead to the best translation quality for any other input? 

What do you think?

And we now beat fairseq's results on a single model with no re-ranking.

Using the new `run_eval_search.py` script https://github.com/huggingface/transformers/pull/7109 I run an extensive search last night and got some extra score!

```
PAIR=en-de
--search="num_beams=5:8:11:15 length_penalty=0.6:0.7:0.8:0.9:1.0:1.1 early_stopping=true:false"
```
Here is just the top results.
```
bleu  | num_beams | length_penalty | early_stopping
----- | --------- | -------------- | --------------
42.83 |        15 |            1.0 |              0
42.79 |         8 |            1.0 |              0
42.79 |        15 |            0.9 |              0
42.79 |        15 |            1.1 |              1
42.77 |         5 |            1.0 |              0
42.76 |         8 |            0.8 |              0
```
I think I will run it for all others and use the best reasonable hparam set as the default. Here it'd be: `5 | 1 | False` - the user can of course override these during `generate`.

> This is great, impressive work @stas00!

Thank you for the kind words, @thomwolf  and doing the review!

> Regarding naming I like both `FSMTModel` and `FairseqMTModel` with a preference for the later more explicit naming option.

The only issue I see with `FairseqMTModel` is that when we have to use the lowercased version of it in the code: `fairseqmt` it doesn't lend to readability `fsmt` on the other hand reads easily.  

Also when typing it out often I couldn't remember whether to use FairSeq or Fairseq. This was my initial name, and it was quite painful working with it. Once I switched to FSMT I experienced much more flow.

So based on these 2 points my vote goes for `FSMTModel`

Sounds like `FSMTModel` wins!

> Sounds like `FSMTModel` wins!

Excellent! So once @LysandreJik and @sgugger get a chance to review we can finally merge it!

Hmm, since we removed the `fsmt-` prefix in model names, it is no longer possible to identify all models for this arch:

https://huggingface.co/models?search=wmt

gives models from other arch as well.

@sshleifer - do you have any Ideas how to solve this? 

Restore the `fsmt-` prefix?

Your yaml front matter allows filters!
Is this page correct: https://huggingface.co/models?filter=fsmt ?

> Your yaml front matter allows filters!
> Is this page correct: https://huggingface.co/models?filter=fsmt ?

Oh, fantastic! All is good then! Thank you, @sshleifer 

FYI, I moved all the data-prep-convert/eval/card writing scripts into their own place: https://github.com/huggingface/transformers/pull/7155 so the convert script got much shorter.

I think we freaked out github, it stopped reporting checks.

@LysandreJik - this is good to go - thanks a lot for your feedback.

> I think you can do `Add suggestion to batch` to prevent the issue where you can't find the comments anymore from happening! 

Oh, that was a super-helpful hint. I wish I knew about it 2 days ago. Thanks a lot!

> Otherwise, feel free to just to the modifications yourself, I don't need to be co-author!

It's a team work ;) Thank you for your contribution, @LysandreJik!