Repository: kserve/kserve

Issue:
XGBoost Server – Performance degradation after upgrade from 0.9.0 to 0.10.1
/kind bug

**What steps did you take and what happened:**
After upgrading our kserve deployment from **`v0.9.0`** to **`v0.10.1`**, I noticed a significant performance degradation in latencies, which affects only xgboost models served by `kserve-xgbserver` ClusterServingRuntime. No changes to the model, model configuration or ClusterSertvingRuntime configuration were applied during the upgrade, so the only difference is the version of `kserve/xgbserver` image, that is used for serving models:
* `kserve/xgbserver:v0.9.0` – is used in Kserve release 0.9.0
* `kserve/xgbserver:v0.10.1` – is used in Kserve release 0.10.1

In order to isolate the problem and eliminate other moving parts from the suspicion list, I run a simple standalone load test of the `kserve/xgbserver` itself with a sample XGB model. To do so, I ran `kserve/xgbserver` on my local Docker runtime as this:
```bash
docker run --rm -it \
    -p 8080:8080 \ 
    -v ~/xgboost/model/:/mnt/models \
   kserve/xgbserver:v0.9.0 --model_name=my_model --model_dir=/mnt/models --http_port=8080 --nthread=1
```

Then, I executed `vegeta` load test to compare serving performance between v0.9.0 and v0.10.1:
```bash
vegeta attack -duration=1m --targets=./target.txt | tee results.bin | vegeta report
```

`target.txt` Content:
```
POST http://localhost:8080/v1/models/my_model:predict
@./request.json
```

The comparison of test results between 0.9.0 and 0.10.1:
* _**`kserve/xgbserver:v0.9.0:`**_
```
Requests      [total, rate, throughput]         3000, 50.02, 50.00
Duration      [total, attack, wait]             59.995s, 59.979s, 15.412ms
Latencies     [min, mean, 50, 90, 95, 99, max]  14.628ms, 16.535ms, 15.82ms, 18.181ms, 18.966ms, 22.101ms, 87.353ms
Bytes In      [total, mean]                     3618000, 1206.00
Bytes Out     [total, mean]                     36078000, 12026.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:3000
Error Set:
```

* _**`kserve/xgbserver:v0.10.1:`**_
```
Requests      [total, rate, throughput]         3000, 50.02, 49.97
Duration      [total, attack, wait]             1m0s, 59.979s, 58.345ms
Latencies     [min, mean, 50, 90, 95, 99, max]  16.078ms, 56.207ms, 57.51ms, 67.96ms, 69.085ms, 80.107ms, 122.035ms
Bytes In      [total, mean]                     3459000, 1153.00
Bytes Out     [total, mean]                     36078000, 12026.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:3000
Error Set:
```

Such a gap in observed latencies between 0.9.0 and 0.10.1 is persistent and reproducible over multiple test runs, different warm-up and test durations.

**What did you expect to happen:**
It doesn't seem like version of `xgboost` library has changed between Kserve release 0.9.0 and 0.10.1, so no performance degradation of the `xgbserver` is expected.





Discussion:
@romanwozniak Can you share what is the xgboost model you are using for testing?
> @romanwozniak Can you share what is the xgboost model you are using for testing?

Sure, @yuzisun, thanks for looking into this. I performed the same testing scenario for multiple XGB models and the gap in latency between `0.9.0` and `0.10.1` is nearly constant for each model.  You can use this sample model to reproduce the same test scenario I described: https://github.com/caraml-dev/merlin/blob/main/python/sdk/test/xgboost-model/model.bst

Sample inference request that can be used with this model:
```json
{"instances": [[2.8, 1.0, 6.8, 0.4], [3.1, 1.4, 4.5, 1.6]]}
```

@romanwozniak can you help attach the logs of the pods as there should be latency numbers logged for each request which can help debug the performance issue? 
This is the result I got with v0.10.1, p50 is 7ms. I think it might be the case that your local box is overloaded which affects your testing? 
```
Requests      [total, rate, throughput]         3000, 50.02, 50.01
Duration      [total, attack, wait]             59.989s, 59.981s, 8.306ms
Latencies     [min, mean, 50, 90, 95, 99, max]  5.109ms, 19.25ms, 7.421ms, 47.868ms, 48.609ms, 50.843ms, 61.963ms
Bytes In      [total, mean]                     75000, 25.00
Bytes Out     [total, mean]                     180000, 60.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:3000  
Error Set:
```
> This is the result I got with v0.10.1, p50 is 7ms. I think it might be the case that your local box is overloaded which affects your testing?
> 
> ```
> Requests      [total, rate, throughput]         3000, 50.02, 50.01
> Duration      [total, attack, wait]             59.989s, 59.981s, 8.306ms
> Latencies     [min, mean, 50, 90, 95, 99, max]  5.109ms, 19.25ms, 7.421ms, 47.868ms, 48.609ms, 50.843ms, 61.963ms
> Bytes In      [total, mean]                     75000, 25.00
> Bytes Out     [total, mean]                     180000, 60.00
> Success       [ratio]                           100.00%
> Status Codes  [code:count]                      200:3000  
> Error Set:
> ```

I'm using my local machine for load testing only after I observed exactly the same performance degradation in our production GKE cluster. The same issue was persistent across all of the replicas of XGB model servers running on the different nodes in the cluster. We had to temporarily patch `ClusterServingRuntime` with `kserve/xgbserver:v0.9.0` image to fix the increased latency. This has nothing to do with my local machine, it's just that it's easier to describe and replicate using a local container with `xgbserver` directly.

> This is the result I got with v0.10.1, p50 is 7ms. 

Your p50 looks indeed better than what I described in my test scenario, however, p50 is not sufficient to draw a conclusion, because p90, p95 and p99 are twice as high as compared to p90, p95 and p99 respectively in my test results with `v0.9.0`. Can you also run the same test using `kserve/xgbserver:v0.9.0` on your setup to compare the two versions? 
> @romanwozniak can you help attach the logs of the pods as there should be latency numbers logged for each request which can help debug the performance issue?

Sure. From the log I can see that prediction time is comparable in both cases (it's even better most of the time with `v0.10.1`), however the end-to-end latency is significantly worse with 0.10.1. Could it be that the serving layer performs much worse in the new version of xgbserver? 

[kserve_0.9.0.log](https://github.com/kserve/kserve/files/11769907/kserve_0.9.0.log)
[kserve_0.10.1.log](https://github.com/kserve/kserve/files/11769908/kserve_0.10.1.log)

I did a quick diff of `pip freeze` between the two versions of xgbserver and it seems like `tornado` server in 0.9.0 was replaced with `uvicorn` in 0.10.1? Could that change contribute to increased e2e latency? 

Resolving patch:
diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py
--- a/python/kserve/kserve/model_server.py
+++ b/python/kserve/kserve/model_server.py
@@ -151,29 +151,28 @@ def start(self, models: Union[List[Model], Dict[str, Deployment]]) -> None:
             concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))
 
         async def serve():
-            serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-            serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-            serversocket.bind(('0.0.0.0', self.http_port))
-            serversocket.listen(5)
-
             logger.info(f"Starting uvicorn with {self.workers} workers")
             loop = asyncio.get_event_loop()
             for sig in [signal.SIGINT, signal.SIGTERM, signal.SIGQUIT]:
                 loop.add_signal_handler(
                     sig, lambda s=sig: asyncio.create_task(self.stop(sig=s))
                 )
-            self._rest_server = UvicornServer(self.http_port, [serversocket],
-                                              self.dataplane, self.model_repository_extension,
-                                              self.enable_docs_url,
-                                              log_config=self.log_config,
-                                              access_log_format=self.access_log_format)
             if self.workers == 1:
+                self._rest_server = UvicornServer(self.http_port, [],
+                                                  self.dataplane, self.model_repository_extension,
+                                                  self.enable_docs_url,
+                                                  log_config=self.log_config,
+                                                  access_log_format=self.access_log_format)
                 await self._rest_server.run()
             else:
                 # Since py38 MacOS/Windows defaults to use spawn for starting multiprocessing.
                 # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
                 # Spawn does not work with FastAPI/uvicorn in multiprocessing mode, use fork for multiprocessing
                 # https://github.com/tiangolo/fastapi/issues/1586
+                serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+                serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+                serversocket.bind(('0.0.0.0', self.http_port))
+                serversocket.listen(5)
                 multiprocessing.set_start_method('fork')
                 server = UvicornServer(self.http_port, [serversocket],
                                        self.dataplane, self.model_repository_extension,
diff --git a/python/kserve/kserve/protocol/rest/server.py b/python/kserve/kserve/protocol/rest/server.py
--- a/python/kserve/kserve/protocol/rest/server.py
+++ b/python/kserve/kserve/protocol/rest/server.py
@@ -172,7 +172,7 @@ def run_sync(self):
         asyncio.run(server.serve(sockets=self.sockets))
 
     async def run(self):
-        await self.server.serve(sockets=self.sockets)
+        await self.server.serve()
 
     async def stop(self, sig: Optional[int] = None):
         if self.server:


Test patch: