Repository: kserve/kserve

Issue:
[Test] Fix lightgbm test in new test-infra
During the process I helped kfserving migrate to new test-infra, noticed that lightgbm failed every time, and to mitigate the issue, I disabled it in the pytest.

I'm working on the lightgbm test cases, and see if I can reproduce locally. 

Need to spend time and dig into it 

Detailed can be found https://github.com/kubeflow/kfserving/pull/1329#issuecomment-770325947

/cc @yuzisun 
/assign 


Discussion:
@lynnmatrix not sure if there is some dependency change, but seems now lightgbmserver is broken and I can replicate the issue with `make deploy-dev-lgb`. In the log I can see

```
[E 210202 07:01:10 web:1793] Uncaught exception POST /v1/models/lightgbm-iris:predict (127.0.0.1)
    HTTPServerRequest(protocol='http', host='lightgbm-iris-predictor-default.default.svc.cluster.local', method='POST', uri='/v1/models/lightgbm-iris:predict', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/lgbserver/lgbserver/model.py", line 51, in predict
        result = self._booster.predict(inputs)
      File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 2415, in predict
        data_has_header, is_reshape)
      File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 504, in predict
        data = _data_from_pandas(data, None, None, self.pandas_categorical)[0]
      File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 318, in _data_from_pandas
        cat_cols = list(data.select_dtypes(include=['category']).columns)
      File "/usr/local/lib/python3.7/site-packages/pandas/core/frame.py", line 3442, in select_dtypes
        include_these = Series(not bool(include), index=self.columns)
      File "/usr/local/lib/python3.7/site-packages/pandas/core/series.py", line 314, in __init__
        data = sanitize_array(data, index, dtype, copy, raise_cast_failure=True)
      File "/usr/local/lib/python3.7/site-packages/pandas/core/internals/construction.py", line 712, in sanitize_array
        subarr = construct_1d_arraylike_from_scalar(value, len(index), dtype)
      File "/usr/local/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 1233, in construct_1d_arraylike_from_scalar
        subarr = np.empty(length, dtype=dtype)
    TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/kfserving/kfserving/handlers/http.py", line 59, in post
        response = (await model.predict(request)) if inspect.iscoroutinefunction(model.predict) else model.predict(request)
      File "/lgbserver/lgbserver/model.py", line 54, in predict
        raise Exception("Failed to predict %s" % e)
    Exception: Failed to predict Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type
[E 210202 07:01:10 web:2243] 500 POST /v1/models/lightgbm-iris:predict (127.0.0.1) 15.44ms
```




there is new numpy release v1.20.0 which potentially breaks this?

Resolving patch:
diff --git a/python/aixexplainer/setup.py b/python/aixexplainer/setup.py
--- a/python/aixexplainer/setup.py
+++ b/python/aixexplainer/setup.py
@@ -35,7 +35,8 @@
         "argparse >= 1.4.0",
         "aix360 >= 0.1.0",
         "lime >= 0.1.1.37",
-        "nest_asyncio>=1.4.0"
+        "nest_asyncio>=1.4.0",
+        "cvxpy == 1.1.7"
     ],
     tests_require=tests_require,
     extras_require={'test': tests_require}
diff --git a/python/lgbserver/setup.py b/python/lgbserver/setup.py
--- a/python/lgbserver/setup.py
+++ b/python/lgbserver/setup.py
@@ -36,7 +36,8 @@
         "kfserving>=0.4.0",
         "lightgbm == 2.3.1",
         "pandas == 0.25.3",
-        "argparse >= 1.4.0"
+        "argparse >= 1.4.0",
+        "numpy == 1.19.5",
     ],
     tests_require=tests_require,
     extras_require={'test': tests_require}


Test patch:
diff --git a/test/scripts/run-e2e-tests.sh b/test/scripts/run-e2e-tests.sh
--- a/test/scripts/run-e2e-tests.sh
+++ b/test/scripts/run-e2e-tests.sh
@@ -170,5 +170,5 @@ popd
 
 echo "Starting E2E functional tests ..."
 pushd test/e2e >/dev/null
-  pytest -n 3 --ignore=credentials/test_set_creds.py --ignore=predictor/test_lightgbm.py
+  pytest -n 3 --ignore=credentials/test_set_creds.py
 popd