[WIP] FSMT bart-like refactor
Author: patil-suraj
Pull request: yes
Labels: name: WIP

# What does this PR do?

This PR refactors `FSMT` to align it with other (bart-like) seq-2-seq models in the lib.

This PR refactors `FSMT` similar to `Bart` in that it moves the time dimension to be always at the 2nd place and the batch dimensions always in the first place. Also, the cache is refactored to consists of `tuples` instead of a `dict`. 

This refactor is very similar to  #10501.

I have verified that all slow-tets are passing and that all metrics (BLEU score) can be reproduced.  I ran the evaluation of the following four models and the results are similar to those reported in the [model cards](https://huggingface.co/facebook/wmt19-en-ru).
- en-ru: 33.42
- ru-en: 39.20
- en-de: 42.83
- de-en: 41.39

### Benchmarking

This PR however introduces some speed and memory regression, which I'm currently investigating.

On this PR:
```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
     facebook/wmt19-en-ru            4               8             0.009     
     facebook/wmt19-en-ru            4               32             0.01     
     facebook/wmt19-en-ru            4              128            0.026     
     facebook/wmt19-en-ru            4              512            0.109     
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
     facebook/wmt19-en-ru            4               8              2172     
     facebook/wmt19-en-ru            4               32             2200     
     facebook/wmt19-en-ru            4              128             2306     
     facebook/wmt19-en-ru            4              512             2792     
--------------------------------------------------------------------------------
```

On master:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
     facebook/wmt19-en-ru            4               8             0.007     
     facebook/wmt19-en-ru            4               32            0.007     
     facebook/wmt19-en-ru            4              128            0.013     
     facebook/wmt19-en-ru            4              512            0.046     
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
     facebook/wmt19-en-ru            4               8              2170     
     facebook/wmt19-en-ru            4               32             2176     
     facebook/wmt19-en-ru            4              128             2204     
     facebook/wmt19-en-ru            4              512             2356     
--------------------------------------------------------------------------------


```


Comments:
Thanks Stas! I'm not sure what exactly introduced this memory/speed regression, so I'm going to investigate it and won't merge this PR before that.

> Thank you for doing this refactoring, @patil-suraj!
> 
> It's a bit hard to review since all the code is moved around, so no easy diff to follow - so while I skimmed through it - I trust your expertise and the tests on the correctness.
> 
> With regards to memory/performance regression - (thank you for running this important check!) could it be that it was introduced in the initial Bart refactor? i.e. perhaps running the same check on Bart pre and post PR that did the main refactoring (when all the Barts were split up)? And if so then the issue is bigger and needs to be looked in that PR that introduced it.

I remember that I checked that the [Bart refactor](https://github.com/huggingface/transformers/pull/8900) didn't show any regression both on the forward pass and `generate()`. I might however have overlooked something. Would definitely be a good idea to verify this first with the exact same testing params (batch_size=4, ...)!

@patil-suraj, what needs to be done to complete this? 

Last we talked there was a performance regression and the suggestion was to test Bart's performance pre and post its original refactoring.

@patil-suraj, FYI, recently I made a few fixes to the model to make it work with Deepspeed:
https://github.com/huggingface/transformers/pull/12477/files#diff-564f6d9b78eec17b410c924f868840770a9ad9649032bcf3754827317b9eaba3

Are we still planning to merge this PR? As we said earlier if there is a regression it'll be on the whole Bart family, so perhaps it might be easier to just merge this? Otherwise a lot of time gets waste getting back to it again and again and not getting anywhere.

Thank you.

Closing as this PR is super old