Repository: tensorflow/serving

Issue #1720: TF 2.3 Serving | Experimental features not supported
Created: 2020-08-15T19:10:10Z
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
TensorFlow Serving installed from (source or binary): Source
TensorFlow Serving version: 2.3.0

Describe the problem
While serving a TF 2.3 trained saved model created with tf.keras.layers.experimental.preprocessing features, TMS(Tensorflow model server ) gives error on prediction. However predict method on the model runs absolutely fine without any error, seems to be something specific to TMS. Does TMS currently not support the new experimental pre-processing features that were added in TF 2.3.
I also noticed that saved_model_cli also throws error that Op type not registered 'DenseBincount' in binary but that could be related to saved_model_cli not yet updated in Colab with TF 2.3 support.
Exact Steps to Reproduce

Run GIST - https://colab.research.google.com/gist/rafiqhasan/6f00aecf1feafd83ba9dfefef8907ee8/dl-e2e-taxi-dataset-tf2-keras.ipynb
Error on cell - https://colab.research.google.com/gist/rafiqhasan/6f00aecf1feafd83ba9dfefef8907ee8/dl-e2e-taxi-dataset-tf2-keras.ipynb#scrollTo=4uJvRYMmHy4N&line=15&uniqifier=1

Source code / logs
import json
import requests

#Create payload
data_py = {"instances":[{'dropoff_latitude': [41.920452],
                         'dropoff_longitude': [-87.679955],
                         'pickup_latitude': [41.952823],
                         'pickup_longitude': [-87.653244],
                         'trip_start_day': ["1"],
                         'trip_start_hour': ["5"],
                         'trip_start_month': ["6"]}]}

data = json.dumps(data_py)
print("payload: ", data)

#Run request on TMS
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8507/v1/models/model:predict', data=data, headers=headers)
json_response.text

Error message:
{
    "error": "Missing 0-th output from {{node functional_3/category_encoding_13/bincount/DenseBincount}}"
}

Duplicate issue #1812: TF 2.4.1 to 2.3.0 Serving broken for experimental.preprocessing features
Created: 2021-02-16T06:12:33Z
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
TensorFlow Serving installed from (source or binary): Binary
TensorFlow Serving version: 2.4.1( >2.3.0 )

Describe the problem:
A bug was already opened for same issue in TFMS 2.3 nearly 6 months back and still unresolved even in TFMS 2.4.1. There has been no resolution for that bug and I am not sure why TF team is not taking this one seriously.
While serving a TF 2.4.1( any model >TF2.3 ) trained saved model created with tf.keras.layers.experimental.preprocessing features, TMS(Tensorflow model server ) gives error on inference. However inferencing on the model directly( model( ) ) runs absolutely fine without any error, this seems to be something specific to TMS / saved_model_cli
saved_model_cli also throws an error that Op type not registered 'DenseBincount' in binary. Make sure the Op and Kernel are registered in the binary running in this process but the saved model can still be used without any issues for inferencing using model( ) . If this is not a TMS issue, then can we please route it to correct team for the resolution. This is an overall TF issue and its impossible to serve TF models on GCP AI platform / AWS until this is resolved.
Exact Steps to Reproduce

Run GIST - Link
Error on cell - Link

Source code / logs
import json
import requests

#Create payload
data_py = {"instances":[{'WERKS': ["ML01"],
                         'DIFGRIRD': [-80.0],
                         'SCENARIO': ["3"],
                         'TOTIRQTY': [80.0],
                         'VSTATU': ["1"],
                         'EKGRP': ["A"],
                         'TOTGRQTY': [0.0],
                         'VPATD': [30.0],
                         'EKORG': ["1"],
                         'NODLGR': [0.0],
                         'DIFGRIRV': [-38100.0],
                         'NODLIR': [90.0],
                         'KTOKK': ["1"]}]}

data = json.dumps(data_py)
print("payload: ", data)

#Run request on TMS
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8511/v1/models/model:predict', data=data, headers=headers)
json_response.text

Error message:
payload:  {"instances": [{"WERKS": ["ML01"], "DIFGRIRD": [-80.0], "SCENARIO": ["3"], "TOTIRQTY": [80.0], "VSTATU": ["1"], "EKGRP": ["A"], "TOTGRQTY": [0.0], "VPATD": [30.0], "EKORG": ["1"], "NODLGR": [0.0], "DIFGRIRV": [-38100.0], "NODLIR": [90.0], "KTOKK": ["1"]}]}
{\n    "error": "Shape must be at most rank 2 but is rank 3\\n\\t [[{{node model/category_encoding_2/bincount/DenseBincount}}]]"\n}

CC: @christisg

Closing comment (2021-02-17T07:16:12Z): @rafiqhasan , as per my understanding this issue is a duplicate of #1720 .
Please confirm on this so that we can close this and  track it in the above link.
Thanks.