Repository: kserve/kserve

Issue:
Setting workers more than 1
If I set `workers=2` when starting KFServer, it always kills the 2nd worker with the following error message

```
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 261, in _add_reader
    key = self._selector.get_key(fd)
  File "/usr/lib/python3.8/selectors.py", line 192, in get_key
    raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '6 is not registered'

[I 210811 10:50:57 kfserver:151] Registering model: test
[I 210811 10:50:57 kfserver:121] Setting asyncio max_workers as 12
[I 210811 10:50:57 kfserver:128] Listening on port 8080
[I 210811 10:50:57 kfserver:130] Will fork 2 workers
[I 210811 10:50:57 process:123] Starting 2 processes
[E 210811 10:50:57 __main__:51] fail to load model test from dir /home/swapnesh/test/test_files/. exception type <class 'FileExistsError'>, exception msg: [Errno 17] File exists
[I 210811 10:50:58 process:163] child 1 (pid 5829) exited normally
```

Why can't I set more than 1 worker when max is 12?


Discussion:
@swapkh91 12 is actually the async io threads not the workers. Are you running a custom model server? Check out the [tornado issue](https://github.com/tornadoweb/tornado/issues/2426#issuecomment-400895086), AsyncIO is not multi processing friendly. If you want to run multiprocessing we suggest trying out the kfserving's [parallel inference](https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1beta1/custom/custom_model#parallel-inference). 
@yuzisun oh I didn't know about that issue, thanks.

I'm running an xgbserver, with a custom image build using the SDK. 

So is `config={"num_replicas": 2}` different from `minReplicas` which we set in the inference service yaml? 

If yes, then would every ray replica consume the same resource as I've specified in the yaml?
@swapkh91 `minReplica` specifies the number of replicas for a given inference service, while `config={"num_replicas": 2}` specifies the worker replicas within a pod so they share the container resource you set on the yaml.

Resolving patch:
diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py
--- a/python/kserve/kserve/model_server.py
+++ b/python/kserve/kserve/model_server.py
@@ -117,10 +117,6 @@ def start(self, models: Union[List[Model], Dict[str, Deployment]], nest_asyncio:
             # formula as suggest in https://bugs.python.org/issue35279
             self.max_asyncio_workers = min(32, utils.cpu_count()+4)
 
-        logging.info(f"Setting asyncio max_workers as {self.max_asyncio_workers}")
-        asyncio.get_event_loop().set_default_executor(
-            concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))
-
         self._http_server = tornado.httpserver.HTTPServer(
             self.create_application(), max_buffer_size=self.max_buffer_size)
 
@@ -129,6 +125,10 @@ def start(self, models: Union[List[Model], Dict[str, Deployment]], nest_asyncio:
         logging.info("Will fork %d workers", self.workers)
         self._http_server.start(self.workers)
 
+        logging.info(f"Setting max asyncio worker threads as {self.max_asyncio_workers}")
+        asyncio.get_event_loop().set_default_executor(
+            concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))
+
         # Need to start the IOLoop after workers have been started
         # https://github.com/tornadoweb/tornado/issues/2426
         # The nest_asyncio package needs to be installed by the downstream module


Test patch: