Repository: kserve/kserve

Issue:
custom predictor request body content-type is limited
/kind feature

**Describe the solution you'd like**
[A clear and concise description of what you want to happen.]
The content-type of the request body in the kserve custom predictor seems to be limited to json
If I make a request in a non-Json format, there is an error, but I think it is because of the code below
kserve/handlers/predict.py
```python
  async def post(self, name: str):
        is_cloudevent = False
        is_binary_cloudevent = False

        if has_binary_headers(self.request.headers):
            is_cloudevent = True
            is_binary_cloudevent = True
            body = self.get_binary_cloudevent()
        else:
            try:
                body = json.loads(self.request.body)
            except json.decoder.JSONDecodeError as e:
                raise tornado.web.HTTPError(
                    status_code=HTTPStatus.BAD_REQUEST,
                    reason="Unrecognized request format: %s" % e
                )
                
```

If base64 encoding is performed to make the same torch sensor into json, the size will be increased as shown below
  - before base64 encoding
      - 350K
  - after base64 encoding
      - 466K

I would like to ask if there is a way to request a custom predictor in a format other than json
After base64 encoding, the request body size is increased, and that part is a bottleneck


**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]



Discussion:
I'm having the same problem

When sending an input in base64 encoding or list format, a botleneck occurs for an input with a large size.

- input smaple : torch.rand(1,3,299,299)
- total server Running time : 170ms
- inferenctime : 30ms
- bottleneck : 140ms
```
 [I 220829 11:23:43 web:2271] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 163.88ms 
 0.026619672775268555 
 [I 220829 11:23:43 web:2271] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 160.11ms 
 0.030611753463745117 
 [I 220829 11:23:44 web:2271] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 187.27ms 
 0.026568174362182617 
 [I 220829 11:23:44 web:2271] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 181.59ms 
 0.0265047550201416 
```

However, if I modify part of the kserve module code and dump the same input to pickle, it will be processed as follows.
```
[I 220831 14:58:11 web:2239] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 16.02ms
[I 220831 14:58:11 web:2239] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 19.87ms
[I 220831 14:58:11 web:2239] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 16.12ms
[I 220831 14:58:11 web:2239] 200 POST /v1/models/receipt-cls-v1:predict (127.0.0.1) 15.83ms
```

The running time difference is very large. 
Also, if the input size is large, It is difficult to provide service at the moment. 

I think it would be good if Kserve accepts byte input like pickle.
The custom predictor also supports `CloudEvent`, so you should be able to send in  `CloudEvent` format with binary data and this is an example for [avro input data](https://github.com/kserve/kserve/blob/master/python/kserve/test/test_server.py#L500). We are also adding [gRPC support](https://github.com/kserve/kserve/issues/2107) for custom predictor which should be the way going forward.
@yuzisun 

Thank you for your avro example.

However, avro needs a schema definition of json type and appears to serialize the json format.
According to the avro official document below, if the byte type is not a string after all, it seems that base64 encoding is needed to be included in the avro input.
https://dmg.org/pfa/docs/avro_types/#bytes-type

Therefore, I think there will be no benefit in reducing the input size. Even if I use avro, I think I should include it in base64 encoding or list format.
According to the current design of kserve, I understood that serialization formats such as pickle cannot be used as input, is that right?


If the two lines below are modified, I think we can use the serialize format input such as Pickle.
https://github.com/kserve/kserve/blob/b89e7fe2a32298117b10b95fb13998df1b216aa4/python/kserve/kserve/handlers/predict.py#L57
https://github.com/kserve/kserve/blob/b89e7fe2a32298117b10b95fb13998df1b216aa4/python/kserve/kserve/model.py#L116

For example, in kserve internal logic, I think we can hand it over to custom predictor without json de-serialization.
```
in kserve/python/kserve/kserve/handlers/predict.py

body = self.request.body
```

```
in custom predictor

class RcModel(kserve.Model):

                       ...

    def preprocess(self, request):
        return json.loads(request) or pickle.loads(request)

                       ...
```

In this case, wouldn't side effect occur or fit the kserve culture?

Resolving patch:
diff --git a/python/kserve/kserve/protocol/dataplane.py b/python/kserve/kserve/protocol/dataplane.py
--- a/python/kserve/kserve/protocol/dataplane.py
+++ b/python/kserve/kserve/protocol/dataplane.py
@@ -32,6 +32,8 @@
 import time
 import logging
 
+JSON_HEADERS = ["application/json", "application/cloudevents+json", "application/ld+json"]
+
 
 class DataPlane:
     """KServe DataPlane
@@ -207,12 +209,21 @@ def model_ready(self, model_name: str) -> bool:
 
     def decode(self, body, headers) -> Tuple[Union[Dict, InferRequest, CloudEvent], Dict]:
         t1 = time.time()
-        decoded_body = body
         attributes = {}
         if isinstance(body, InferRequest):
-            return decoded_body, attributes
-        if headers and has_binary_headers(headers):
-            body = self.get_binary_cloudevent(body, headers)
+            return body, attributes
+        if headers:
+            if has_binary_headers(headers):
+                # returns CloudEvent
+                body = self.get_binary_cloudevent(body, headers)
+            elif "content-type" in headers and headers["content-type"] not in JSON_HEADERS:
+                return body, attributes
+            else:
+                if type(body) is bytes:
+                    try:
+                        body = orjson.loads(body)
+                    except orjson.JSONDecodeError as e:
+                        raise InvalidInput(f"Unrecognized request format: {e}")
         elif type(body) is bytes:
             try:
                 body = orjson.loads(body)


Test patch: