Update `convert_bart` script to allow loading nonstandard model architectures and custom pretrained Fairseq models.
Author: tomsherborne
Pull request: no

**edit 30/10/20** - this approach doesn't work anymore given the differences in interface from when I submitted it it now (mostly because of Hydra configs in fairseq). Closed. 


# 🚀 Feature request

Allow the `convert_bart` script to load custom BART pre-trained models from fairseq. 

## Motivation

The current BART conversion script to migrate from Fairseq to Huggingface assumes that you are using predefined  shape/parameter values from a BART model. There might be a usecase where you want to experiment with your own Seq2Seq pretraining and train a smaller BART model (i.e. cutting dimensions by 1/3rd) because of small datasets/pruning/some other reason. There presently isn't a dedicated Fairseq tutorial on training your own BART models but it is possible using the "denoising" task and the appropriate parameters (e.g. [preprocess](https://gist.github.com/tomsherborne/ab1a5a28f9d843cf633d6f7843e96a63) and [train](https://gist.github.com/tomsherborne/ae3529375b7a538a1b03a53f34850234)). This is what I have been trying with a small dataset and a smaller BART model.

Then if you want to use this model within `transformers` - the conversion script doesn't work because (a) `load_xsum_checkpoint` assumes your model should be the shape of `bart.large.cnn` and rejects the incorrect tensor sizes and (b) an additional weight `decoder.output_projection.weight` needs to be ignored. 

To convert correctly to `transformers` and pass all the tests - I found you can switch from `torch.hub.load` to `fairseq.models.bart.BARTModel.from_pretrained` which calls `torch.hub` internally and gives you the same output model. This means you can convert local, custom BART models into `transformers` models and use them in downstream tasks or upload them to the models archive. 

## Your contribution

I made my own version of `convert_bart` [here](https://gist.github.com/tomsherborne/e7b629ee9cf0618febb211683a410ce5) which passes the scripts own tests. This could be improved/refactored if this feature is still useful. There could be a better way to do all this that I have missed. There's a hack because I call `torch.load` and `from_pretrained()` on the same weights and I'm not sure if they are both needed. 

The execution is bit ugly because I set `hf_config` as a path to a JSON file created from `BartConfig.from_pretrained()` and then manually adjusting to the new model size.
```
SCRIPT_PATH="${HOME}/ed/sp/transformers/src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py"
TEMPLATE_ARCHIVE=""/Users/tom/ed/sp/pretrain/runs/fairseq/bart_base_enqa_small/bart-mini.tar.gz""
CHECKPOINT_PATH="${HOME}/ed/sp/pretrain/runs/fairseq/bart_base_enqa_small/ckpt/checkpoint_best.pt"
DUMP_PATH="${HOME}/ed/sp/pretrain/runs/hf_fromfairseq/bart-mini"
CONFIG_PATH="${HOME}/ed/sp/pretrain/config/hf_bart_configs/bart-mini/config.json"

source ~/miniconda3/bin/activate $ENV_NAME

python $SCRIPT_PATH --hf_config $CONFIG_PATH --model_template $TEMPLATE_ARCHIVE $CHECKPOINT_PATH $DUMP_PATH 

```

Comments: