Update serving code to enable `saved_model=True`
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes and adds any missing`serving` and `serving_output` code to our TF models to enable 
`model.save_pretrained(path, saved_model=True)`

I've added comments throughout the code to explain any areas where the TF refactor might no be obvious. 

I'm aware the diff of this PR is quite big, but most of it repetitive changes to enable passing of tests, so I hope acceptable. 

## Specifically: 
**1. Adds missing serving logic to: ResNet, Swin, TAPAS**

**2. Adds missing `serving_output` logic to models** 
Some vision models didn't have `serving_output` implemented - `serving` returned the model outputs directly. This was to enable testing (see 4.) and to keep consistent with the rest of the library, 

**3. Update or add `input_signature` decorator for models**
**4. Adds a test to check `serving_output` is implemented and return types are as expected**

We can't test `model.serving` directly i.e. this is not possible: 
```
model = model_class(config)
inputs = self._prepare_for_class(inputs_dict, model_class)
serving_outputs = model.serving(inputs)
```

Running this on vision models raises the following: 
```
E                           ValueError: The channel dimension of the inputs should be defined. The input_shape received is (None, None, None, None), where axis -1 (0-based) is the channel dimension, which found to be `None`.
```

This is because the input signature defined in the `tf.function` decorator for the `serving` method has all of the input dimensions defined as `None`:
```
 @tf.function(input_signature=[{
     "pixel_values": tf.TensorSpec((None, None, None, None), tf.float32, name="pixel_values"),
            }]
    )
    def serving(self, inputs):
        ....
```
We can't hard code the channel dimensions as e.g. `3` as we want to support both RGB and greyscale images (although testing this locally does work). 


**5. Moves `test_saved_model_creation` back into `test_modeling_tf_common` and add explicit skips**

There were quite a few models that couldn't be saved with `model.save_pretrained(path, saved_model=True)` and quite a few whose input signature or return types from `serving_output` were broken or inconsistent with the model. 

I think this is in part because the relevant test was moved to only be applied to certain core models, and those models didn't explicitly skip. See [#14415](https://github.com/huggingface/transformers/pull/14415), [#9478](https://github.com/huggingface/transformers/pull/9478)

I've:
* added it back to common so that it's added to models by default. CI currently running and passing, 
* added `unittest.skip` decorator so it's counted as a skipped rather than passed test on all models that were previously skipping.

**6. Update logic in models such that their graph can be created and saved.** 

Adds serving logic to enable saving of models and ensures their outputs are transformed in line with the rest of the library

## Fixes 
https://github.com/huggingface/transformers/issues/18179
https://github.com/huggingface/transformers/issues/18164
https://github.com/huggingface/transformers/issues/17233
https://github.com/huggingface/transformers/issues/17285
https://discuss.huggingface.co/t/tfresnetforimageclassification-fails-with-save-pretrained-when-saved-model-is-true/20404

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts thank you for digging into this one. An important one albeit. 

> We can't hard code the channel dimensions as e.g. 3 as we want to support both RGB and greyscale images (although testing this locally does work).

Could you shed more details on the locally testing you're referring to here?

> Could you shed more details on the locally testing you're referring to here?

@sayakpaul Sure :) In the test `test_prepare_serving_output`, if serving outputs are got by calling `serving` directly i.e. 
```
            inputs = self._prepare_for_class(inputs_dict, model_class)
            serving_outputs = model.serving(inputs)
```
This will fail for vision models with the channel dim error I posted above. If I changed the input signature such that channel dimension is hard coded then it runs e.g. 

```
    @tf.function(
        input_signature=[
            {
                "pixel_values": tf.TensorSpec((None, 3, None, None), tf.float32, name="pixel_values"),
            }
        ]
    )
```

I run the test with 
```pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output```

Thanks. A follow-up question. So if you run with (`pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output`) how does it get the hardcoded value for channels? Or do you first hard-code it and then run it? 

> Thanks. A follow-up question. So if you run with (`pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output`) how does it get the hardcoded value for channels? Or do you first hard-code it and then run it?

@sayakpaul Yes, I hardcode then run the tests. In this case, they pass. 

Before merge, could you measure the timing for the tests `test_saved_model_creation` on **CPU**? You can run like

```python
python -m pytest -v tests -k "test_saved_model_creation" --durations=0 --make-reports=tests_timing
```
and copy-paste the results from `reports/tests_timing/durations.txt `

@ydshieh We don't run the slow tests on CPU, only on GPU/multi-GPU.

> @ydshieh We don't run the slow tests on CPU, only on GPU/multi-GPU.

I should double check the latest version. My memory was in a previoius commit `Add in tests (505cb774b1b7eb5c9a6c8e2bc63f12061824b8bd)` while I asked the question 😢 

@amyeroberts The `attention_mask` and `token_type_ids` in TFHubert / TFWav2Vec2 should be `int32` I believe. I think we don't put this clearly in their docstrings in TF models, but we have this information in their PyTorch model files.

@ydshieh I know from @sgugger's comment, we don't run on CPU, but I ran the tests for reference (Macbook Pro 2021 M1 Max)
Based on this I disabled the test for Swin. The slowest tests  - `test_saved_model_creation_extended` - are independent of this PR. 

```
slowest durations
242.92s call     tests/models/convbert/test_modeling_tf_convbert.py::TFConvBertModelTest::test_saved_model_creation_extended
202.38s call     tests/models/bert/test_modeling_tf_bert.py::TFBertModelTest::test_saved_model_creation_extended
177.82s call     tests/models/gptj/test_modeling_tf_gptj.py::TFGPTJModelTest::test_saved_model_creation_extended
150.56s call     tests/models/bart/test_modeling_tf_bart.py::TFBartModelTest::test_saved_model_creation_extended
82.65s call     tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_saved_model_creation_extended
61.12s call     tests/models/lxmert/test_modeling_tf_lxmert.py::TFLxmertModelTest::test_saved_model_creation_extended
26.88s call     tests/models/swin/test_modeling_tf_swin.py::TFSwinModelTest::test_saved_model_creation
20.14s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPTextModelTest::test_saved_model_creation_extended
19.51s call     tests/models/convbert/test_modeling_tf_convbert.py::TFConvBertModelTest::test_saved_model_creation
19.40s call     tests/models/gptj/test_modeling_tf_gptj.py::TFGPTJModelTest::test_saved_model_creation
18.54s call     tests/models/deberta_v2/test_modeling_tf_deberta_v2.py::TFDebertaModelTest::test_saved_model_creation
16.77s call     tests/models/speech_to_text/test_modeling_tf_speech_to_text.py::TFSpeech2TextModelTest::test_saved_model_creation
16.72s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPVisionModelTest::test_saved_model_creation_extended
14.51s call     tests/models/roformer/test_modeling_tf_roformer.py::TFRoFormerModelTest::test_saved_model_creation
14.37s call     tests/models/deberta/test_modeling_tf_deberta.py::TFDebertaModelTest::test_saved_model_creation
13.15s call     tests/models/wav2vec2/test_modeling_tf_wav2vec2.py::TFWav2Vec2ModelTest::test_saved_model_creation
13.12s call     tests/models/hubert/test_modeling_tf_hubert.py::TFHubertModelTest::test_saved_model_creation
12.79s call     tests/models/mpnet/test_modeling_tf_mpnet.py::TFMPNetModelTest::test_saved_model_creation
12.47s call     tests/models/tapas/test_modeling_tf_tapas.py::TFTapasModelTest::test_saved_model_creation
12.16s call     tests/models/layoutlm/test_modeling_tf_layoutlm.py::TFLayoutLMModelTest::test_saved_model_creation
12.16s call     tests/models/electra/test_modeling_tf_electra.py::TFElectraModelTest::test_saved_model_creation
12.12s call     tests/models/xlm/test_modeling_tf_xlm.py::TFXLMModelTest::test_saved_model_creation
11.92s call     tests/models/flaubert/test_modeling_tf_flaubert.py::TFFlaubertModelTest::test_saved_model_creation
11.54s call     tests/models/transfo_xl/test_modeling_tf_transfo_xl.py::TFTransfoXLModelTest::test_saved_model_creation
11.42s call     tests/models/bert/test_modeling_tf_bert.py::TFBertModelTest::test_saved_model_creation
11.36s call     tests/models/rembert/test_modeling_tf_rembert.py::TFRemBertModelTest::test_saved_model_creation
11.12s call     tests/models/distilbert/test_modeling_tf_distilbert.py::TFDistilBertModelTest::test_saved_model_creation
10.85s call     tests/models/ctrl/test_modeling_tf_ctrl.py::TFCTRLModelTest::test_saved_model_creation
10.77s call     tests/models/roberta/test_modeling_tf_roberta.py::TFRobertaModelTest::test_saved_model_creation
10.51s call     tests/models/wav2vec2/test_modeling_tf_wav2vec2.py::TFWav2Vec2RobustModelTest::test_saved_model_creation
10.44s call     tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_saved_model_creation
10.30s call     tests/models/hubert/test_modeling_tf_hubert.py::TFHubertRobustModelTest::test_saved_model_creation
10.23s call     tests/models/deit/test_modeling_tf_deit.py::TFDeiTModelTest::test_saved_model_creation
10.20s call     tests/models/t5/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
10.14s call     tests/models/albert/test_modeling_tf_albert.py::TFAlbertModelTest::test_saved_model_creation
10.14s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPTextModelTest::test_saved_model_creation
9.53s call     tests/models/openai/test_modeling_tf_openai.py::TFOpenAIGPTModelTest::test_saved_model_creation
9.11s call     tests/models/vit_mae/test_modeling_tf_vit_mae.py::TFViTMAEModelTest::test_saved_model_creation
8.71s call     tests/models/vit/test_modeling_tf_vit.py::TFViTModelTest::test_saved_model_creation
8.68s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPVisionModelTest::test_saved_model_creation
8.44s call     tests/models/dpr/test_modeling_tf_dpr.py::TFDPRModelTest::test_saved_model_creation
8.14s call     tests/models/xlnet/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_saved_model_creation
7.54s call     tests/models/data2vec/test_modeling_tf_data2vec_vision.py::TFData2VecVisionModelTest::test_saved_model_creation
6.70s call     tests/models/convnext/test_modeling_tf_convnext.py::TFConvNextModelTest::test_saved_model_creation
6.21s call     tests/models/regnet/test_modeling_tf_regnet.py::TFRegNetModelTest::test_saved_model_creation
4.62s call     tests/models/resnet/test_modeling_tf_resnet.py::ResNetModelTest::test_saved_model_creation
```

I noticed a cool thing -- assuming the models with `@tooslow` also pass the tests (which I'm assuming they do, from [this](https://github.com/huggingface/transformers/pull/18153#issuecomment-1191706099) comment), this PR fixes:
1. https://github.com/huggingface/transformers/issues/17233, as the problematic line was rewritten with TF code in this PR 
2. https://github.com/huggingface/transformers/issues/17285, as we know we can create a `SavedModel` (which contains a graph) for all models. 

@amyeroberts can you add these issues to your `Fixes` list above? :D

@gante Nice spot :D Yep - I just double checked, and all the models with the @tooslow decorator can be saved with `saved_model=True` - and so their graph can be built. Added the fixes. 

The only model we can't save at the moment is CLIP, due to the nested dict of outputs. 



@ydshieh @Rocketknight1 Am I OK to merge?