Repository: SeldonIO/MLServer

Issue:
No stdout output for default parallel_workers=1
I'm new to seldon. When I added some debug statements (using `print()` and also the `logging` module) to the model I found that it will work in `load()` but not `predict()`. I spend multiple hours to understand that `multiprocessing` is used by default and it is responsible for swallowing the output. 

I have a few questions and suggestions:
1) AFAI understand seldon uses the same amount of CPU (1) when parallel_workers is 0 or 1 (the default). So why was 1 chosen as the default?
1) Is it possible to see the log messages from predict calls when the parallel pool is used? How?
   1) If it is possible to get the output: Can it be made the default? Or at least log a warning that further messages from predict-calls will not be shown.
   2) If it is not possible to get the output: Can parallel_workers default to 0? Or at least log a warning that further messages from predict-calls will not be shown. 

I'm using 1.1.0


Discussion:
Hey @rfalke ,

This is currently a bug in the parallel worker implementation. Main reason seems to be that the logger is not initialised properly on the workers themselves. It should be a quick fix though, which will be addressed in time for the next release of MLServer.

As a temporary workaround (until the above is fixed), `print()` statements should show up, even when the model is executed in the parallel worker pool. 

Regarding your other point, when parallel workers are set to `1` (i.e. the default), there will be a separate worker process where inference will be executed. This gives some "space" to the main process to handle open connections, adaptive batching, manage requests from the admin API (e.g. load / unload models), etc (which otherwise would get blocked due to the GIL). Therefore, it's generally recommended to always run with at least one worker. 
Sadly I don't see any messages from `print()` when parallel_workers is 1.

Thanks for clearing up on the other point.
Hey @rfalke ,

We are unable to replicate the point where `print()` messages don't show up. We can do so with `logger.info()` though, so we'll focus on solving the latter. 

Resolving patch:
diff --git a/benchmarking/testserver/models.py b/benchmarking/testserver/models/sum-model/models.py
similarity index 100%
rename from benchmarking/testserver/models.py
rename to benchmarking/testserver/models/sum-model/models.py
diff --git a/mlserver/parallel/pool.py b/mlserver/parallel/pool.py
--- a/mlserver/parallel/pool.py
+++ b/mlserver/parallel/pool.py
@@ -44,7 +44,7 @@ def __init__(self, settings: Settings):
         for idx in range(self._settings.parallel_workers):
             # TODO: Set callback to restart worker if it goes down (would
             # `worker.join` help with that?)
-            worker = Worker(responses)
+            worker = Worker(settings, responses)
             worker.start()
             self._workers[worker.pid] = worker  # type: ignore
 
diff --git a/mlserver/parallel/worker.py b/mlserver/parallel/worker.py
--- a/mlserver/parallel/worker.py
+++ b/mlserver/parallel/worker.py
@@ -8,6 +8,8 @@
 
 from ..registry import MultiModelRegistry
 from ..utils import install_uvloop_event_loop, schedule_with_callback
+from ..logging import configure_logger
+from ..settings import Settings
 
 from .messages import (
     ModelRequestMessage,
@@ -26,8 +28,9 @@ def _noop():
 
 
 class Worker(Process):
-    def __init__(self, responses: Queue):
+    def __init__(self, settings: Settings, responses: Queue):
         super().__init__()
+        self._settings = settings
         self._responses = responses
         self._requests: Queue[ModelRequestMessage] = Queue()
         self._model_updates: JoinableQueue[ModelUpdateMessage] = JoinableQueue()
@@ -48,6 +51,7 @@ def _executor(self):
 
     def run(self):
         install_uvloop_event_loop()
+        configure_logger(self._settings)
         self._ignore_signals()
         asyncio.run(self.coro_run())
 


Test patch:
diff --git a/tests/parallel/conftest.py b/tests/parallel/conftest.py
--- a/tests/parallel/conftest.py
+++ b/tests/parallel/conftest.py
@@ -46,11 +46,12 @@ async def responses() -> Queue:
 
 @pytest.fixture
 async def worker(
+    settings,
     event_loop,
     responses: Queue,
     load_message: ModelUpdateMessage,
 ) -> Worker:
-    worker = Worker(responses)
+    worker = Worker(settings, responses)
 
     # Simulate the worker running on a different process, but keep it to a
     # thread to simplify debugging.