Fix saved model creation
Author: jplu
Pull request: yes

Fix a bug when a saved model is created the parameters `output_hidden_states` and `output_attentions` was ignored.

Reproducibility with TF 2.2:

```python
import tensorflow as tf
from transformers import TFBertModel, BertTokenizer, BertConfig
config = BertConfig.from_pretrained("bert-base-multilingual-uncased", output_hidden_states=True)
model = TFBertModel.from_pretrained('bert-base-multilingual-uncased', config=config)
tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-uncased", use_fast=False)
features = tokenizer.encode_plus("Hello world.", add_special_tokens=True, max_length=48, pad_to_max_length=True, return_tensors="tf", truncation=True)
model._saved_model_inputs_spec = None
model._set_save_spec(dict(features))
tf.saved_model.save(model, "save/model")
```

Then run the serving CLI with:

```
saved_model_cli show --dir save/model/ --tag_set serve --signature_def serving_default
```

There will be only 2 outputs instead of 3.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=h1) Report
> Merging [#5468](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5a0dac53bfd6e69ae64fb3119d607445e1a308d8&el=desc) will **increase** coverage by `0.34%`.
> The diff coverage is `93.30%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5468/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5468      +/-   ##
==========================================
+ Coverage   79.33%   79.67%   +0.34%     
==========================================
  Files         146      146              
  Lines       26611    26582      -29     
==========================================
+ Hits        21111    21180      +69     
+ Misses       5500     5402      -98     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `12.50% <0.00%> (-0.60%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.22% <16.66%> (-63.98%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.97% <71.42%> (-1.85%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.87% <88.23%> (+0.22%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `76.73% <89.28%> (+0.26%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG5ldC5weQ==) | `91.34% <95.83%> (-0.05%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.75% <96.15%> (-0.03%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.93% <100.00%> (-0.05%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9jdHJsLnB5) | `97.84% <100.00%> (-0.01%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.78% <100.00%> (+34.60%)` | :arrow_up: |
| ... and [20 more](https://codecov.io/gh/huggingface/transformers/pull/5468/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=footer). Last update [5a0dac5...69ea0de](https://codecov.io/gh/huggingface/transformers/pull/5468?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Sure! Will try to find a proper test for that.

Awesome! Fearing many merge conflicts with https://github.com/huggingface/transformers/pull/5395#pullrequestreview-442374095 :D 

I fear the same! We should wait to merge #5395 before to merge this one.

CirlceCi gives the following message error `Too long with no output (exceeded 10m0s): context deadline exceeded` does it means that the tests takes too long now?

oof yeah, that's what it means! Do you have any idea of how long the added tests take in your local environment, compared to the full test suite?

I get around 1.5 min per new test / per model => 3min per model => ~33min but this is on my laptop which is really cheap

That's a slow test :) We can mark them as slow for now (using the `@slow` decorator) and monitor how long they take. If they take too long, we'll have to think of a different way to test those.

Ok good to me

Still have to do some bugfix and once all the models pass the tests I will put the `@slow` decorator.

Ok, now all the models can be saved in TF saved model format. I put some tests to be sure of that, but they have the `@slow` decorator.

This is good to merge to me. Nevertheless, I have done several changes in the input of several layers, @LysandreJik can you check if you are ok with that?

Basically, saved models cannot be run with `list / tuple` inputs, because this is very Python specific and cannot be translated into gRPC.

@jplu Thanks for your work first! The transformers-based model now can be served by Tensorflow Serving. But I still have one question about max_seg_length. 
In order to make inference faster, is it possible to set max_length to be None? In tf 1.x, I can use the following codes to make serving accept dynamic max_seq_length.
```python
estimator = ...

def serving_input_receiver_fn(max_seq_len):
    input_ids = tf.compat.v1.placeholder(shape=[None, max_seq_len], dtype=tf.int32, name='input_ids')
    input_mask = tf.compat.v1.placeholder(shape=[None, max_seq_len], dtype=tf.int32, name='input_mask')
    segment_ids = tf.compat.v1.placeholder(shape=[None, max_seq_len], dtype=tf.int32, name='segment_ids')
    features = {'input_ids': input_ids, 'input_mask': input_mask, 'segment_ids': segment_ids}
    return tf.estimator.export.build_raw_serving_input_receiver_fn(features)

estimator.export_savedmodel(model_output_dir, serving_input_receiver_fn(None))
```
I found following codes work in TF2.x. Just ignore this message.
```python
input_feature = {
    'input_ids': tf.TensorSpec(shape=(None, None), dtype=tf.int32, name='input_ids'),
    'token_type_ids': tf.TensorSpec(shape=(None, None), dtype=tf.int32, name='token_type_ids'),
    'attention_mask': tf.TensorSpec(shape=(None, None), dtype=tf.int32, name='attention_mask')
}
model._set_save_spec(input_feature)
```


Hello! Thanks for you suggestion. The saved model creation is postponed, and will be for a next PR. This one is here for bugfix only.

Your code will work fine, but not for all the models and tasks. For example token classification uses a different shape, and DistilBert doesn't have a token_type_ids. Unfortunately, it is a bit more complicated than just putting this piece of code somewhere, it has to be task and model independant.

@jplu What are the issues in deleting `cast_bool_to_primitive` altogether?

T5 will not work anymore because the number of output depends on the use_cache parameter. And for now we still want to keep a variable length output.

We are currently reworking the outputs approach of all the models the output dictionaries instead of tuples. I will come back on this issue of boolean tensor once this new type of output will be available.

Indeed, I think this is fine, mostly because I expect people to hack around the hidden layers mostly for the PyTorch implementations.

Good for me!

This is a valuable point indeed @LysandreJik! Nevertheless, unpacking data is not compliant with TensorFlow Autograph as far as I know, basically we loose this usage as it was before.

Alright, sounds good! Could you resolve the merge conflict and then we merge?

Fixed!