Example of TFMarianMTModel is not working very well
Author: Zhenzi-Weng
Pull request: no

Hello, I copied the TFMarianMTModel example code from [https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/marian#transformers.TFMarianMTModel](url) and compiled in Tensorflow 2.4.1. But it turned out the output length of "model.generate(**batch)" is always 512.

```python
from transformers import MarianTokenizer, TFMarianMTModel
from typing import List

src = "fr"  # source language
trg = "en"  # target language
sample_text = "où est l'arrêt de bus ?"
model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}"

model = TFMarianMTModel.from_pretrained(model_name)
tokenizer = MarianTokenizer.from_pretrained(model_name)
batch = tokenizer([sample_text], return_tensors="tf")
gen = model.generate(**batch)
tokenizer.batch_decode(gen, skip_special_tokens=True)
```

Could you help with it?

@patrickvonplaten 

Comments:
Hey @Zhenzi-Weng, 

When I execute the code above, I'm getting an error. I'm assuming you want to execute the following:

```python
from transformers import MarianTokenizer, TFMarianMTModel
from typing import List

src = "fr"  # source language
trg = "en"  # target language
sample_text = "où est l'arrêt de bus ?"
model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}"

model = TFMarianMTModel.from_pretrained(model_name, from_pt=True)
tokenizer = MarianTokenizer.from_pretrained(model_name)
batch = tokenizer([sample_text], return_tensors="tf")
gen = model.generate(**batch)
tokenizer.batch_decode(gen, skip_special_tokens=True)
```


When executing the above code, I'm getting a pretty reasonable answer: 

```
["Where's the bus stop?"]   
```

=> What is the problem here exactly?


> When executing the above code, I'm getting a pretty reasonable answer:
> 
> ```
> ["Where's the bus stop?"]   
> ```
> 
> => What is the problem here exactly?

Hello @patrickvonplaten , thanks for your reply. I tried your code and it did output a reasonable translation, but I believe your code is for Pytorch. I‘m using Tensorflow and the problem is the output will contain many invalid characters. Would you mind to try the following code again? Many thanks. My Tensorflow version is 2.4.1 and I installed transformers v4.19.0.
`from transformers import MarianTokenizer, TFMarianMTModel

src = "en"  # source language
trg = "zh"  # target language

model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}"
model = TFMarianMTModel.from_pretrained(model_name)
tokenizer = MarianTokenizer.from_pretrained(model_name)

sample_text = "My name is Wolfgang and I live in Berlin"
batch = tokenizer([sample_text], return_tensors="tf")

generated_ids = model.generate(**batch)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)`

Hey @Zhenzi-Weng,

Note that my codesnippet above is for Tensorflow, the model is loaded into a `TFMarianMTModel` class. We just need to have PyTorch installed to convert the PT weights to TF before 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @patrickvonplaten, related to this question, why is it that the TF version of MarianMTModel much slower than the PT version? Maybe it has something to do with the length of generating output?

TF is generally much slower in eager mode than PyTorch - you could try to compile the generate method in Tensorflow which should speed things up :-) 

cc @gante  

@jamie0725 correct, TF (eager execution) is much slower than PyTorch. However, with XLA, it should be much faster. Have a look at [this blog post](https://huggingface.co/blog/tf-xla-generate)!