Generating text with MBart Large 50 on GPU with Tensorflow is significantly slower than with Pytorch
Author: xhluca
Pull request: no

This applies to the [MBart Large MMT 50-language model](https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt). Takes about 1m20s to process 10 batches of size 16 x 171 on pytorch, but 8min for tensorflow. Both are running on P100 through Kaggle.

- [Tensorflow notebook](https://www.kaggle.com/xhlulu/tf-mbartforconditionalgeneration-speed-test)
- [Pytorch Notebook](https://www.kaggle.com/xhlulu/mbartforconditionalgeneration-speed-test)


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.4.89+-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


## Code for pytorch

```python
from tqdm.auto import tqdm
import torch
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"

for i in tqdm(range(10)):
    encoded_hi = tokenizer([article_hi*10]*16, return_tensors="pt")

    generated_tokens = model.generate(
        **encoded_hi,
        forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"]
    )
    out = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
    # => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire dans la Syrie."

```

## Code for Tensorflow

```python
from tqdm.auto import tqdm
import tensorflow as tf
from transformers import TFMBartForConditionalGeneration, MBart50TokenizerFast

strategy = tf.strategy.get_strategy()

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"

with strategy.scope():
    model = TFMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt", from_pt=True)

tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"

with strategy.scope():
    for i in tqdm(range(10)):
        encoded_hi = tokenizer([article_hi*10]*16, return_tensors="tf")

        generated_tokens = model.generate(
            **encoded_hi,
            forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"]
        )
        out = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
        # => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire dans la Syrie."

```


Comments:
It seems that the generation function is handled by the [`TFGenerationMixin`](https://github.com/huggingface/transformers/blob/696e8a43655a63b7312e036616f4abd2106e179e/src/transformers/generation_tf_utils.py#L48-L72) whereas in torch it is handled by [`GenerationMixin`](https://github.com/huggingface/transformers/blob/d4d4447d536e5cf8c78518b8b3359168346a4134/src/transformers/generation_utils.py#L665-L699); quickly glancing over the code I notice that the implementation is different. Could there be a discrepancy in implementation that would affect the generation speed?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.