Repository: kserve/kserve

Issue:
Readiness probe failed when huge tensor file send to predictor in triton
/kind bug

**What steps did you take and what happened:**
I have deployed the onnx model to the server using protocol v2 (trition). 

My model is supposed to get an  ```[1, 3, 640, 640]``` shaped FP32 tensor.
However, when writing the corresponding tensor as json, it is 16MB.

When the predictor receives the corresponding tensor, it stops responding, and a timeout occurs during the readiness probe process and is restarted.

It is the same even if it is transmitted using a base64 image (only 120KB) using a transformer and transmitted to the predictor through transformation inside kubernetes.

**What did you expect to happen:**
The model  will be referenced.

**Anything else you would like to add:**
There seems to be no special log in the predictor.
```
=============================
== Triton Inference Server ==
=============================

NVIDIA Release 20.10 (build <unknown>)

Copyright (c) 2018-2020, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for the inference server.  NVIDIA recommends the use of the following flags:
   nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...

I1213 05:21:21.489391 1 metrics.cc:184] found 1 GPUs supporting NVML metrics
I1213 05:21:21.496163 1 metrics.cc:193]   GPU 0: NVIDIA GeForce RTX 3090
I1213 05:21:21.759404 1 pinned_memory_manager.cc:195] Pinned memory pool is created at '0x7fb504000000' with size 268435456
I1213 05:21:21.760695 1 cuda_memory_manager.cc:98] CUDA memory pool is created on device 0 with size 67108864
I1213 05:21:21.764328 1 model_repository_manager.cc:714] loading: insightface:1
I1213 05:21:21.812585 1 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I1213 05:21:21.812622 1 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I1213 05:21:21.812627 1 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I1213 05:21:21.821293 1 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: insightface (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I1213 05:21:21.823365 1 onnxruntime.cc:1814] TRITONBACKEND_ModelInstanceInitialize: insightface_0 (device 0)
I1213 05:21:25.939947 1 model_repository_manager.cc:896] successfully loaded 'insightface' version 1
I1213 05:21:25.940210 1 server.cc:141]
+-------------+-----------------------------------------------------------------+------+
| Backend     | Config                                                          | Path |
+-------------+-----------------------------------------------------------------+------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {}   |
+-------------+-----------------------------------------------------------------+------+

I1213 05:21:25.940282 1 server.cc:184]
+-------------+---------+--------+
| Model       | Version | Status |
+-------------+---------+--------+
| insightface | 1       | READY  |
+-------------+---------+--------+

I1213 05:21:25.940436 1 tritonserver.cc:1621]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                              |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                             |
| server_version                   | 2.4.0                                                                                                                                              |
| server_extensions                | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0]         | /mnt/models                                                                                                                                        |
| model_control_mode               | MODE_NONE                                                                                                                                          |
| strict_model_config              | 1                                                                                                                                                  |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                                          |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                           |
| min_supported_compute_capability | 6.0                                                                                                                                                |
| strict_readiness                 | 1                                                                                                                                                  |
| exit_timeout                     | 30                                                                                                                                                 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+

I1213 05:21:25.944552 1 grpc_server.cc:3914] Started GRPCInferenceService at 0.0.0.0:9000
I1213 05:21:25.944811 1 http_server.cc:2717] Started HTTPService at 0.0.0.0:8080
I1213 05:21:25.987316 1 http_server.cc:2736] Started Metrics Service at 0.0.0.0:8002
```
I thought it might be a problem with shm_size, so I tested it on a server without shm_size, but it works well. In my opinion, it may be a problem in the front interface that receives data.

**Environment:**

- Istio Version: 1.10
- Knative Version: 0.22.3
- KFServing Version: 0.7.0
- Kubeflow version: X
- Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] X
- Minikube version: X
- Kubernetes version: (use `kubectl version`): 1.18.5
- OS (e.g. from `/etc/os-release`): ubuntu 20.04.2



Discussion:
I am ignorant of the web.

After several tests, a file of ~4 MB is acceptable. Files of 8 MB are not acceptable.

If the --header "Content-Type: application/json" is appended on curl command, even 16MB files are transmitted successfully.

It seems that the server works differently depending on the content-type.

Anyway my problem was solved.
Oh, but when transformer is used, the same error occurs whether the corresponding header is not attached when the transformer passes it to the predictor. I think we still need to open an issue on this part.
I think it would be better to add a content-type header to that line.
https://github.com/kserve/kserve/blob/07e4d5d3ea54164026bbf7d87de0b06c6d34c014/python/kserve/kserve/kfmodel.py#L164-L169
Or please comment.

Resolving patch:
diff --git a/python/kserve/kserve/kfmodel.py b/python/kserve/kserve/kfmodel.py
--- a/python/kserve/kserve/kfmodel.py
+++ b/python/kserve/kserve/kfmodel.py
@@ -161,10 +161,12 @@ async def _http_predict(self, request: Dict) -> Dict:
         predict_url = PREDICTOR_URL_FORMAT.format(self.predictor_host, self.name)
         if self.protocol == PredictorProtocol.REST_V2.value:
             predict_url = PREDICTOR_V2_URL_FORMAT.format(self.predictor_host, self.name)
+        json_header = {'Content-Type': 'application/json'}
         response = await self._http_client.fetch(
             predict_url,
             method='POST',
             request_timeout=self.timeout,
+            headers=json_header,
             body=json.dumps(request)
         )
         if response.code != 200:


Test patch: