New API for TensorFlow saved models not compatible with T5 and MarianMT
Author: maziyarpanahi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.2 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@jplu @patrickvonplaten 

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 ray/raytune: @richardliaw @amogkam
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): MarianMT and T5

I was trying to use the new Serving feature introduced in `4.2.0` (https://github.com/huggingface/transformers/pull/9419)

## To reproduce

Steps to reproduce the behavior:

1. `pip install tensorflow transformers sentencepiece`
2. Try to use `.save_pretrained("./serve_tf", saved_model=True)` with T5 or MarianMT

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Code to reproduce the error with T5:

```python
import tensorflow as tf
from transformers import TFT5ForConditionalGeneration

TFT5ForConditionalGeneration.from_pretrained('t5-small')\
.save_pretrained("./serve_tf_t5", saved_model=True)
```

The error:

```
All the layers of TFT5ForConditionalGeneration were initialized from the model checkpoint at t5-small.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-c3e4b575f41e> in <module>()
      4 from transformers import T5Tokenizer, TFT5ForConditionalGeneration
      5 
----> 6 TFT5ForConditionalGeneration.from_pretrained('t5-small').save_pretrained("./serve_tf_t5", saved_model=True)

14 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_serialization.py:126 signature_wrapper  *
        structured_outputs, signature_function.name, signature_key)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_serialization.py:174 _normalize_outputs  **
        .format(value, key, compat.as_str_any(function_name)))

    ValueError: Got a dictionary containing non-Tensor value (<tf.Tensor 'StatefulPartitionedCall:2' shape=(1, 6, 4, None, 8, None, 64) dtype=float32>,) for key past_key_values in the output of the function __inference_serving_25314 used to generate a SavedModel signature. Dictionaries outputs for functions used as signatures should have one Tensor output per string key.
```

Code to reproduce the error with MarianMT:

```python

import tensorflow as tf
from transformers import TFMarianModel, TFMarianMTModel

# It seems `opus-mt-mt-en`, `opus-mt-en-zh` and `opus-mt-en-ROMANCE` are the only
# TF compatible models
TFMarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-mt-en')\
.save_pretrained("./serve_tf_marian", saved_model=True)
```

The error:

```
All the layers of TFMarianMTModel were initialized from the model checkpoint at Helsinki-NLP/opus-mt-mt-en.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFMarianMTModel for predictions without further training.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-0fe7915512e9> in <module>()
      5 from transformers import TFMarianModel, TFMarianMTModel, MarianMTModel
      6 
----> 7 TFMarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-mt-en', use_cache=False).save_pretrained("./serve_tf_marian", saved_model=True)

14 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_serialization.py:126 signature_wrapper  *
        structured_outputs, signature_function.name, signature_key)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_serialization.py:174 _normalize_outputs  **
        .format(value, key, compat.as_str_any(function_name)))

    ValueError: Got a dictionary containing non-Tensor value (None,) for key past_key_values in the output of the function __inference_serving_109486 used to generate a SavedModel signature. Dictionaries outputs for functions used as signatures should have one Tensor output per string key.
```

I have tried to see the `use_cache` to `False` hoping `past_key_values` won't be used as output layer but it didn't help.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Like other models such as BERT or ALBERT, these two should be saved to be served later via the TensorFlow Serving environment.

Comments:
Pinging @jplu 

Hello!!

Yes this is a known issue for the Seq2Seq models, that is already fixed in master.

Hi @jplu 

I'll close the issue as the fix will be in the next release for sure. 

Thanks again.