Contributing trained Greek<->English NMT models implemented with fairseq
Author: lighteternal
Pull request: no
Labels: name: New model

Hi there, quick question that I couldn't answer by searching the docs:

I trained an EL-EN (Greek to English) and an EN-EL machine translation model using the fairseq implementation of the `transformer_iwslt_de_en `architecture on ~6GB of parallel corpora. Given that the models report a better BLEU score compared to the existing SotA, I would like to share them somehow. I thought that fairseq might offer a huggingface-like way to upload trained models but I couldn't find any, so I would appreciate any guidance.

If there's a straightforward way to convert and upload these as huggingface models it would be great!

Many thanks!

Comments:
We'd love to help you share your models! Both @sshleifer and @stas00 worked on MT models and used Fairseq recently so might be able to help.

If I'm not mistaken the only difference between [wmt19](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) and iwslt is the configuration of the layers. In which case it should be trivial to port it to `transformers` via [FSMT](https://huggingface.co/transformers/model_doc/fsmt.html). FSMT = FairSeqMachineTranslation.

You can try it yourself using the [conversion script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py) and if you get stuck please ask for help, pasting the code of what you have tried. You can see how it is used [here](https://github.com/huggingface/transformers/blob/master/scripts/fsmt/convert-facebook-wmt19.sh) and 2 more [here](https://github.com/huggingface/transformers/tree/master/scripts/fsmt).

The only thing the script can't automate at the moment is hyper param presetting, since they are not part of the model dump, we probably need to add clargs to optionally set those. Until now I embedded them in the script itself but that's not the best way to move forward. But let's handle that when everything else is working for you, the converted model will just use the default hparam settings.

Many thanks for the prompt response. I will try the script and update on the progress. 
Apart from the model weights themselves, I assume I will need to take care of the preprocessing (Moses tokenization and fastBPE) as well,  in order to load the model and perform inference without issues. 

FSMT already does moses+bpe. No pre- or post-processing is required.

That's great, thx! I also just read it on the FSMT doc. ^_^

Edit: Updated with proper code block formatting. 

Sorry for the delay @stas00! After updating to the latest transformers and fairseq versions, I had some progress. 

OK so I followed the steps and it seems that the conversion starts succesfully using this command: 

```
PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/trans_elen/checkpoint_best.pt --pytorch_dump_folder_path data/wmt16-el-en-dist
```

But after a few seconds, it returns an error:
```
(base) earendil@C3PO$~/Desktop/conversion/transformers PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/trans_elen/checkpoint_best.pt --pytorch_dump_folder_path data/wmt16-el-en-dist
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/earendil/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Writing results to data/wmt16-el-en-dist
using checkpoint checkpoint_best.pt
/home/earendil/anaconda3/lib/python3.6/site-packages/hydra/_internal/hydra.py:71: UserWarning: 
@hydra.main(strict) flag is deprecated and will removed in the next version.
See https://hydra.cc/docs/next/upgrades/0.11_to_1.0/strict_mode_flag_deprecated
  warnings.warn(message=msg, category=UserWarning)
Traceback (most recent call last):
  File "src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py", line 271, in <module>
    convert_fsmt_checkpoint_to_pytorch(args.fsmt_checkpoint_path, args.pytorch_dump_folder_path)
  File "src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py", line 118, in convert_fsmt_checkpoint_to_pytorch
    src_lang = args["source_lang"]
KeyError: 'source_lang'
```

Which I cannot debug since I don't recall inputting any argument regarding src and tgt languages. Aren't these arguments acquired from the model checkpoint?

Could you please re-edit you comment and use proper code block formatting? it's impossible to figure out what it says since there are warnings mixed in - the new lines are needed to be able to parse it.

Please use the menu bar (`<>` button) or start/end with three backticks if you do it manually.

It should appear like so (I pasted a totally random error just as a demo):

```
        """
        tens_ops = (input, weight)
        if not torch.jit.is_scripting():
            if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
                return handle_torch_function(linear, tens_ops, input, weight, bias=bias)
        if input.dim() == 2 and bias is not None:
            # fused op is marginally faster
            ret = torch.addmm(bias, input, weight.t())
        else:
>           output = input.matmul(weight.t())
E           RuntimeError: CUDA out of memory. Tried to allocate 100.00 MiB (GPU 0; 23.70 GiB total capacity; 21.83 GiB already allocated; 19.69 MiB free; 22.08 GiB reserved in total by PyTorch)
```


Sorry, just fixed it.

Ah, much better - thank you!

So your model is different from wmt19's series, it fails here:

```
    src_lang = args["source_lang"]
    tgt_lang = args["target_lang"]
```
which comes from the checkpoint we are trying to convert.

Before it fails do:
```
print(args.keys())
```
and see what you have in there.  

Most likely you're converting a different architecture, in which case this script won't work as is.

If you can't figure it out please send me the info at how to get the checkpoint and all the vocab/config files it comes with and I will have a look.


The output of `print(args.keys())` is :


```
dict_keys(['_metadata', '_parent', '_content'])
```

OK, so this is a totally different arch then. In wmt19 the args contain a large set of model configuration, see: a few paragraphs into this section https://huggingface.co/blog/porting-fsmt#porting-weights-and-configuration

So where does it store the model configuration? Or does it not and there is just a fixed config - in which case what is it? how does one derive this from the checkpoint? Is it possible that you forgot to save it in the checkpoint? OR the code you were using for some reason wasn't saving it? 

In addition to answering the above, please send me the download info (checkpoint file, and dict, config files) and I will see whether the FSMT arch can somehow be re-used.

I am not sure where the model configuration is saved, tbh. In my implementation I was just following the steps from this guide: 
https://github.com/pytorch/fairseq/tree/master/examples/translation#training-a-new-model
but using my own data of course. If you check the following script: 
```
CUDA_VISIBLE_DEVICES=0 fairseq-train \
    data-bin/iwslt14.tokenized.de-en \
    --arch transformer_iwslt_de_en --share-decoder-input-output-embed \
    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
    --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \
    --dropout 0.3 --weight-decay 0.0001 \
    --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
    --max-tokens 4096 \
    --eval-bleu \
    --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \
    --eval-bleu-detok moses \
    --eval-bleu-remove-bpe \
    --eval-bleu-print-samples \
    --best-checkpoint-metric bleu --maximize-best-checkpoint-metric
```
it seems that the `--arch transformer_iwslt_de_en` is enough for the trainer to understand the architecture (according to this [post](https://github.com/pytorch/fairseq/issues/1301), the key difference is in the ffn hidden dim ,iwslt_de_en is 1024 and transformer is 2048) . 

I am uploading the files to a GDrive folder (it will take a while for the checkpoint) and will email you with the link if that's ok (mail found on your website).

Thank you for that info, @lighteternal. 

I will have a look at the data you sent to me (thank you) and will get back to you.



Let's continue over https://github.com/huggingface/transformers/pull/8374

Closing this, as it's solved by by @stas00 in #8374