Overhaul TF serving signatures + dummy inputs
Author: Rocketknight1
Pull request: yes

Right now, our default TF serving signature is only really appropriate for BERT-like models, which means it needs to be overridden in most cases. This PR does inspection of `self.call` to figure out what to actually use, but can still be overridden if required. It also moves the definition of the serving signature to the `__init__()`, which allows it to use values from the `config` to set parts of the shape (e.g. `num_channels`)

I might also explore doing something similar with `dummy_inputs` in this PR and build models via the serving signature, without needing to explicitly define `dummy_inputs`. Ideally, we could eliminate a lot of that boilerplate, which would make it much easier for users to contribute models and reduce the amount of work needed to turn a LLM translation from PyTorch into a working TF model.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This should be ready to review now - it shouldn't affect any existing models, as all our existing models override `serving`, `serving_output` and `dummy_inputs`. However, it should hopefully be a default that "just works" for a lot of future models, and means we can stop specifying the same information in three different places.

@gante Agreed! I can make that cleanup part of this PR, WDYT?

@Rocketknight1 Sounds good!

@Rocketknight1 plz ping when this PR is ready for a re-review 🔥 

@gante @amyeroberts This should be ready for re-review now, but no rush because it's late on a Friday! The core idea is that I collapsed all of the redundant information we had down to a single source of truth. That source of truth is a new property on our models: `input_signature`. **All** serving methods and decorators have been removed from our models - serving on all models is now set in the `__init__` with 
`self.serving = tf.function(self.eager_serving, input_signature=[self.input_signature]`.

This fixes a major problem we had: As well as being a huge source of repetitive boilerplate, the serving signatures were incorrect in several places, and because they were compiled with a decorator, the decorator could not access `self.config`, which meant the serving signature could not include shape constraints that are defined in the config (such as `config.image_size`). This meant we just used `None` dimensions for dimensions that were actually not variable!

Additionally, `dummy_inputs` is now inferred from `self.input_signature` as well. Specifically, the `dummy_inputs` property fills in `None` dimensions in the `input_signature` with `2` and then just generates tensors with that shape and dtype, then builds the model with those.

`dummy_inputs` can still be overridden, and this is used in a few models when they need particular dummy inputs to build without errors. The vast majority of `dummy_inputs` have been removed, though. `serving` can in theory be overridden too, but there was no need to do this in any of our models.

Finally, the new base `serving_output` code covers most cases, and I'd estimate about 75% of `serving_outputs` in the codebase are gone. I expect there are going to be a few issues, but I'll keep an eye on the tests and make sure it's all okay!

It seems like there's a few issues caused by the default dummy input values triggering assertions or issues - I'll add dummy_inputs overrides to those models.

@gante @amyeroberts I think everything should pass now - ready for final review!

This PR reduces the size of our TF codebase (files matching `*_tf_*.py`) by a little under 5%, lol

I believe my question is related to this PR, but let me know if I should redirect this elsewhere! I'm trying to convert a pretrained model to a TFLite model but am running into issues due to its incorrect input_signature. If I understand correctly, this PR may have fixed that issue, but maybe only for future models? I'm using https://huggingface.co/microsoft/layoutlm-base-cased which was uploaded over 2 years ago.

My question - is there a way to override the pre-existing (incorrect) input_signature of a pretrained model that was presumably trained before this fix went in? I already tried reassigning but it errors due to it being a read-only property.

Reproducible code below:
Note that `model.input_signature` includes `input_ids`, `attention_mask`, and `token_type_ids` only and that it's missing `bbox` (call definition [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlm/modeling_tf_layoutlm.py#L1433))
I'm on transformers == 4.36.2
```
import tensorflow as tf
from transformers import AutoTokenizer, TFLayoutLMForTokenClassification

def get_sample_input():

	tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased")

	words = ["Hello", "world"]
	normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782]

	token_boxes = []
	for word, box in zip(words, normalized_word_boxes):
	    word_tokens = tokenizer.tokenize(word)
	    token_boxes.extend([box] * len(word_tokens))

	# add bounding boxes of cls + sep tokens
	token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]]

	encoding = tokenizer(" ".join(words), return_tensors="tf")
	input_ids = encoding["input_ids"]
	bbox = tf.convert_to_tensor([token_boxes])

	return input_ids, bbox


model = TFLayoutLMForTokenClassification.from_pretrained("microsoft/layoutlm-base-uncased")

# has only input_ids, attention_mask, and token_type_ids
print(model.input_signature)

input_ids, bbox = get_sample_input()

# note that the below works
outputs = model(input_ids=input_ids, bbox=bbox)

# now, convert to TFLite model
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

my_signature = interpreter.get_signature_runner()

# below errors with "ValueError: Invalid number of inputs provided for running a SignatureDef, expected 3 vs provided 2"
# because of the incorrect input_signature I believe
my_signature(input_ids=input_ids, bbox=bbox)
```



Hi @echan5, the issue here is actually that our input signature for LayoutLM is missing `bbox`! This doesn't affect the standard model, but it does mean the export signature for TFLite is incorrect. I'll open a PR to fix it.

If you want to get your code working in the meantime, though, here's some background info:

When you use `tf.lite.TFLiteConverter.from_keras_model`, what TFLite does internally is it first saves the model as a SavedModel export, then uses that as the base for the TFLite conversion (you can verify this in TF's [source code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/lite.py#L1474))

When a model is saved as SavedModel, you have to specify the **signatures** it will be saved with. Because a SavedModel is essentially a compiled version of the model, it doesn't have the same flexibility as the original - you have to specify the inputs it's going to receive and their dtypes.

By default, our models use the input names, shapes and dtypes specified in the model's `input_signature`. However, you can overrule that by passing your own `signatures`. You can see how we do this in the source code for `save_pretrained` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_tf_utils.py#L2404-L2423).

So, if you want to convert a model to TFLite with a custom signature, what I'd do is save the model as SavedModel with a custom signature, and then do the TFLite conversion from the saved model so you have control of that attribute.

If that seems a bit complex, though, hold tight - I'll try to have the PR in soon!



@echan5 quick update - the newer LayoutLMv3 has the correct signature already. I'll still fix the issue for LayoutLM v1, but you should be able to switch to `microsoft/layoutlmv3-base` to fix your issue immediately and get improved performance too!

PR is open at #28640

PR is merged - @echan5 if you want to continue using LayoutLMv1 instead of switching to v3, you can get the updated signature by installing from main with `pip install --upgrade git+https://github.com/huggingface/transformers.git`

@Rocketknight1 - thank you so much for the quick fix/PR! I do have to use v1 for now, unfortunately, so this has been helpful. 

A note in case this helps others looking at my original code snippet `my_signature(input_ids=input_ids, bbox=bbox)` will still error with "ValueError: Invalid number of inputs provided for running a SignatureDef, expected 4 vs provided 2"  because all 4 args are expected (no default values; `model(input_ids=input_ids, bbox=bbox)` does handle default values).

However, the the below will make it work (I'm essentially copy and pasting the code for setting default values for `attention_mask` and `token_type_ids` from the original layoutlm `call` definition [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlm/modeling_tf_layoutlm.py#L840-L853)):
```
from transformers.tf_utils import shape_list
input_shape = shape_list(input_ids)

# set default values
attention_mask = tf.fill(dims=input_shape, value=1)
token_type_ids = tf.fill(dims=input_shape, value=0)

# below now works
tflite_outputs = my_signature(input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, token_type_ids=token_type_ids)
```

I'm not sure if the team might want to consider handling default values too, or if that's a more complicated implementation, but this works for me for now - thank you for your help! (Also thank you for pointing out how to override the signatures in save_pretrained, as I had missed that)

edit: link was pointing to the incorrect lines

Hi @echan5 - we can add default values in the Python code for the model, but not to the serving signature, unfortunately! We can exclude an input entirely, but in general we prefer to leave important inputs available in the default signature.

You can, however, do what you did there and just pass dummy values, or alternatively you can save the model as SavedModel with a custom signature that excludes those inputs, with something like this:

```python
signature = model.input_signature
del signature['attention_mask']  # Remove/edit keys that you don't want
serving_fn = model.serving.get_concrete_function(signature)
model.save('my_save_dir', include_optimizer=False, signatures=serving_fn)

tflite_model = tf.lite.TFLiteConverter.from_saved_model('my_save_dir')
```

That's a much cleaner approach than the dummy values - thank you!