Repository: kserve/kserve

Issue:
XGBoost Model Perfomance Issue
/kind bug
XGboost model server perform poorly after deployed to k8s.

**What steps did you take and what happened:**
I did performance test for xgbserver using provided sample model.

`load.cfg`:
```
POST http://<target>/v1/models/default:predict
@request.json
```

`req.json`:
```
{"instances": [[1,2,3,4]]}
```
Running locally (on Mac OS, 8 core machine) with vegeta
```
cat load.cfg | vegeta attack -duration 1m -rate 1000 | vegeta report -type=text
```

I am able to hit 1000 RPS with good latencies result.

```
Requests      [total, rate, throughput]    60000, 1000.02, 1000.00
Duration      [total, attack, wait]        1m0.000012024s, 59.999069988s, 942.036µs
Latencies     [mean, 50, 90, 95, 99, max]  1.070524ms, 946.443µs, 1.414653ms, 1.520775ms, 2.401196ms, 17.650717ms
Bytes In      [total, mean]                1320000, 22.00
Bytes Out     [total, mean]                1560000, 26.00
Success       [ratio]                      100.00%
Status Codes  [code:count]                 200:60000  
```

But after deployed to K8S (with 4 core as CPU request and limit)

The performance deteriorates at 100 RPS
```
cat load-1.cfg | vegeta attack -rate 100 -duration 60s | vegeta report
```

```
Requests      [total, rate, throughput]    6000, 100.02, 89.47
Duration      [total, attack, wait]        1m7.064629048s, 59.989775627s, 7.074853421s
Latencies     [mean, 50, 90, 95, 99, max]  1.524605151s, 3.683733ms, 5.575147148s, 7.007148666s, 7.789309456s, 8.606190814s
Bytes In      [total, mean]                132000, 22.00
Bytes Out     [total, mean]                192000, 32.00
Success       [ratio]                      100.00%
Status Codes  [code:count]                 200:6000
```

**What did you expect to happen:**

The performance should not be affected too much after deployed to K8S.

**Anything else you would like to add:**
I suspected the issue come from `nthread` in xgbserver are left to default (i.e. 0). 
In: 
https://github.com/kubeflow/kfserving/blob/c46a0da8c0cf168dccfcee3b79f306fea135faf4/python/xgbserver/xgbserver/model.py#L36
and
https://github.com/kubeflow/kfserving/blob/c46a0da8c0cf168dccfcee3b79f306fea135faf4/python/xgbserver/xgbserver/model.py#L42

Which will leave the decision of number of thread to use to xgboost by detecting number of cpu. However, possibly due to this https://bugs.python.org/issue36054, the `nthread` is incorrect and causing the performance issues.

I did a test with setting `nthread=1` :
```
class XGBoostModel(kfserving.KFModel):
    def __init__(self, name: str, model_dir: str, booster: XGBModel = None):
        super().__init__(name)
        self.name = name
        self.model_dir = model_dir
        if not booster is None:
            self._booster = booster
            self.ready = True

    def load(self):
        model_file = os.path.join(
            kfserving.Storage.download(self.model_dir), BOOSTER_FILE)
        self._booster = xgb.Booster(params={"nthread" = 1},model_file=model_file)
        self.ready = True

    def predict(self, request: Dict) -> Dict:
        try:
            # Use of list as input is deprecated see https://github.com/dmlc/xgboost/pull/3970
            dmatrix = xgb.DMatrix(request["instances"], nthread=1)
            result: xgb.DMatrix = self._booster.predict(dmatrix)
            return { "predictions": result.tolist() }
        except Exception as e:
            raise Exception("Failed to predict %s" % e)
```

```
cat load.cfg | vegeta attack -rate 1000 -duration 60s | vegeta report
```

and it performed much better:

```
Requests      [total, rate, throughput]    60000, 1000.02, 999.99
Duration      [total, attack, wait]        1m0.00064503s, 59.9988685s, 1.77653ms
Latencies     [mean, 50, 90, 95, 99, max]  1.927362ms, 1.73989ms, 1.941139ms, 2.490536ms, 5.660895ms, 56.707145ms
Bytes In      [total, mean]                1320000, 22.00
Bytes Out     [total, mean]                1920000, 32.00
Success       [ratio]                      100.00%
Status Codes  [code:count]                 200:60000
```

The question here is:
1. Is setting `nthread=1` a good idea for other kind of workload?
2. Should this be configurable?

**Environment:**

- Istio Version: 1.1.7
- Knative Version: 0.8.1
- KFServing Version: 0.2.1
- Kubeflow version: NA
- Minikube version: NA
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/xgbserver/xgbserver/__main__.py b/python/xgbserver/xgbserver/__main__.py
--- a/python/xgbserver/xgbserver/__main__.py
+++ b/python/xgbserver/xgbserver/__main__.py
@@ -19,15 +19,18 @@
 
 DEFAULT_MODEL_NAME = "default"
 DEFAULT_LOCAL_MODEL_DIR = "/tmp/model"
+DEFAULT_NTHREAD = 1
 
 parser = argparse.ArgumentParser(parents=[kfserving.kfserver.parser]) #pylint:disable=c-extension-no-member
 parser.add_argument('--model_dir', required=True,
                     help='A URI pointer to the model directory')
 parser.add_argument('--model_name', default=DEFAULT_MODEL_NAME,
                     help='The name that the model is served under.')
+parser.add_argument('--nthread', default=DEFAULT_NTHREAD,
+                    help='Number of threads to use by XGBoost.')
 args, _ = parser.parse_known_args()
 
 if __name__ == "__main__":
-    model = XGBoostModel(args.model_name, args.model_dir)
+    model = XGBoostModel(args.model_name, args.model_dir, args.nthread)
     model.load()
     kfserving.KFServer().start([model]) #pylint:disable=c-extension-no-member
diff --git a/python/xgbserver/xgbserver/model.py b/python/xgbserver/xgbserver/model.py
--- a/python/xgbserver/xgbserver/model.py
+++ b/python/xgbserver/xgbserver/model.py
@@ -22,10 +22,12 @@
 BOOSTER_FILE = "model.bst"
 
 class XGBoostModel(kfserving.KFModel):
-    def __init__(self, name: str, model_dir: str, booster: XGBModel = None):
+    def __init__(self, name: str, model_dir: str, nthread: int, booster: \
+        XGBModel = None):
         super().__init__(name)
         self.name = name
         self.model_dir = model_dir
+        self.nthread = nthread
         if not booster is None:
             self._booster = booster
             self.ready = True
@@ -33,13 +35,14 @@ def __init__(self, name: str, model_dir: str, booster: XGBModel = None):
     def load(self):
         model_file = os.path.join(
             kfserving.Storage.download(self.model_dir), BOOSTER_FILE)
-        self._booster = xgb.Booster(model_file=model_file)
+        self._booster = xgb.Booster(params={"nthread" : self.nthread},
+                                    model_file=model_file)
         self.ready = True
 
     def predict(self, request: Dict) -> Dict:
         try:
             # Use of list as input is deprecated see https://github.com/dmlc/xgboost/pull/3970
-            dmatrix = xgb.DMatrix(request["instances"])
+            dmatrix = xgb.DMatrix(request["instances"], nthread=self.nthread)
             result: xgb.DMatrix = self._booster.predict(dmatrix)
             return { "predictions": result.tolist() }
         except Exception as e:


Test patch:
diff --git a/pkg/apis/serving/v1alpha2/framework_xgboost_test.go b/pkg/apis/serving/v1alpha2/framework_xgboost_test.go
--- a/pkg/apis/serving/v1alpha2/framework_xgboost_test.go
+++ b/pkg/apis/serving/v1alpha2/framework_xgboost_test.go
@@ -51,17 +51,12 @@ func TestFrameworkXgBoost(t *testing.T) {
 }
 
 func TestCreateXGBoostContainer(t *testing.T) {
-
 	var requestedResource = v1.ResourceRequirements{
 		Limits: v1.ResourceList{
-			"cpu": resource.Quantity{
-				Format: "100",
-			},
+			"cpu": resource.MustParse("100m"),
 		},
 		Requests: v1.ResourceList{
-			"cpu": resource.Quantity{
-				Format: "90",
-			},
+			"cpu": resource.MustParse("90m"),
 		},
 	}
 	var config = InferenceServicesConfig{
@@ -87,6 +82,7 @@ func TestCreateXGBoostContainer(t *testing.T) {
 			"--model_name=someName",
 			"--model_dir=/mnt/models",
 			"--http_port=8080",
+			"--nthread=0",
 		},
 	}
 
@@ -94,3 +90,58 @@ func TestCreateXGBoostContainer(t *testing.T) {
 	container := spec.GetContainer("someName", false, &config)
 	g.Expect(container).To(gomega.Equal(expectedContainer))
 }
+
+func TestCreateXGBoostContainerWithNThread(t *testing.T) {
+	var config = InferenceServicesConfig{
+		Predictors: &PredictorsConfig{
+			Xgboost: PredictorConfig{
+				ContainerImage:      "someOtherImage",
+				DefaultImageVersion: "0.1.0",
+			},
+		},
+	}
+	g := gomega.NewGomegaWithT(t)
+
+	scenarios := map[string]struct {
+		nthread     int
+		resourceReq v1.ResourceRequirements
+		expArgs     []string
+	}{
+		"TestNThread": {
+			nthread: 4,
+			resourceReq: v1.ResourceRequirements{
+				Limits: v1.ResourceList{
+					"cpu": resource.MustParse("1100m"),
+				},
+				Requests: v1.ResourceList{
+					"cpu": resource.MustParse("1100m"),
+				},
+			},
+			expArgs: []string{
+				"--model_name=someName",
+				"--model_dir=/mnt/models",
+				"--http_port=8080",
+				"--nthread=4",
+			},
+		},
+	}
+
+	// Test Create with config
+	for name, scenario := range scenarios {
+		spec := XGBoostSpec{
+			StorageURI:     "gs://someUri",
+			Resources:      scenario.resourceReq,
+			RuntimeVersion: "0.1.0",
+			NThread:        scenario.nthread,
+		}
+		container := spec.GetContainer("someName", false, &config)
+
+		expContainer := &v1.Container{
+			Image:     "someOtherImage:0.1.0",
+			Name:      constants.InferenceServiceContainerName,
+			Resources: scenario.resourceReq,
+			Args:      scenario.expArgs,
+		}
+		g.Expect(container).To(gomega.Equal(expContainer), fmt.Sprintf("Testing %s", name))
+	}
+}
diff --git a/pkg/apis/serving/v1alpha2/inferenceservice_defaults_test.go b/pkg/apis/serving/v1alpha2/inferenceservice_defaults_test.go
--- a/pkg/apis/serving/v1alpha2/inferenceservice_defaults_test.go
+++ b/pkg/apis/serving/v1alpha2/inferenceservice_defaults_test.go
@@ -168,6 +168,9 @@ func TestXGBoostDefaults(t *testing.T) {
 	g.Expect(isvc.Spec.Default.Predictor.XGBoost.Resources.Requests[v1.ResourceMemory]).To(gomega.Equal(defaultResource[v1.ResourceMemory]))
 	g.Expect(isvc.Spec.Default.Predictor.XGBoost.Resources.Limits[v1.ResourceCPU]).To(gomega.Equal(defaultResource[v1.ResourceCPU]))
 	g.Expect(isvc.Spec.Default.Predictor.XGBoost.Resources.Limits[v1.ResourceMemory]).To(gomega.Equal(defaultResource[v1.ResourceMemory]))
+
+	defaultCpu := defaultResource[v1.ResourceCPU]
+	g.Expect(isvc.Spec.Default.Predictor.XGBoost.NThread).To(gomega.Equal(int(defaultCpu.Value())))
 	g.Expect(isvc.Spec.Canary.Predictor.XGBoost.RuntimeVersion).To(gomega.Equal(constants.KFServingDefaultVersion))
 	g.Expect(isvc.Spec.Canary.Predictor.XGBoost.Resources.Requests[v1.ResourceCPU]).To(gomega.Equal(defaultResource[v1.ResourceCPU]))
 	g.Expect(isvc.Spec.Canary.Predictor.XGBoost.Resources.Requests[v1.ResourceMemory]).To(gomega.Equal(resource.MustParse("3Gi")))
diff --git a/pkg/controller/inferenceservice/resources/knative/service_test.go b/pkg/controller/inferenceservice/resources/knative/service_test.go
--- a/pkg/controller/inferenceservice/resources/knative/service_test.go
+++ b/pkg/controller/inferenceservice/resources/knative/service_test.go
@@ -332,6 +332,7 @@ func TestInferenceServiceToKnativeService(t *testing.T) {
 													"--model_name=xgboost",
 													"--model_dir=" + constants.DefaultModelLocalMountPath,
 													"--http_port=8080",
+													"--nthread=0",
 												},
 											},
 										},
@@ -389,6 +390,7 @@ func TestInferenceServiceToKnativeService(t *testing.T) {
 													"--model_name=xgboost",
 													"--model_dir=" + constants.DefaultModelLocalMountPath,
 													"--http_port=8080",
+													"--nthread=0",
 												},
 											},
 										},
diff --git a/python/xgbserver/xgbserver/test_model.py b/python/xgbserver/xgbserver/test_model.py
--- a/python/xgbserver/xgbserver/test_model.py
+++ b/python/xgbserver/xgbserver/test_model.py
@@ -19,6 +19,7 @@
 
 model_dir = model_dir = os.path.join(os.path.dirname(__file__), "example_model")
 BST_FILE = "model.bst"
+NTHREAD = 1
 
 def test_model():
     iris = load_iris()
@@ -35,7 +36,7 @@ def test_model():
     xgb_model = xgb.train(params=param, dtrain=dtrain)
     model_file = os.path.join((model_dir), BST_FILE)
     xgb_model.save_model(model_file)
-    model = XGBoostModel("xgbmodel", model_dir)
+    model = XGBoostModel("xgbmodel", model_dir, NTHREAD)
     model.load()
     request = [X[0].tolist()]
     response = model.predict({"instances": request})