Repository: SeldonIO/MLServer

Issue:
500 Error on MacOS
I'm running MLServer on MacOS (arm). I was following [Serving XGBoost models](https://mlserver.readthedocs.io/en/latest/examples/xgboost/README.html) example. I receive a 500 error when requesting a model with url `http://localhost:8080/v2/models/mushroom-xgboost/versions/v0.1.0/infer`.

MLServer throws an error

```
...
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/queues.py", line 126, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError
```

Developers of `queues.py` left a comment that this is broken on MacOS 😬 

```python
    def qsize(self):
        # Raises NotImplementedError on Mac OSX because of broken sem_getvalue()
        return self._maxsize - self._sem._semlock._get_value()
```

Is it possible to get around this when running models with MLServer?
Thanks!


Discussion:
I may suggest a solution based on [vterron/lemon](https://github.com/vterron/lemon/blob/d60576bec2ad5d1d5043bcb3111dff1fcb58a8d6/methods.py#L536-L573) that uses synchronized counter. 

In `mlserver/parallel/worker.py`

```python
# Few imports added / changed
from multiprocessing import Process, Value, get_context
from multiprocessing.queues import Queue


# New classes
class SharedCounter:

    def __init__(self, n=0):
        self.count = Value('i', n)

    def increment(self, n=1):
        """ Increment the counter by n (default = 1) """
        with self.count.get_lock():
            self.count.value += n

    @property
    def value(self):
        """ Return the value of the counter """
        return self.count.value


class SharedCounterQueue(Queue):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs, ctx=get_context())
        self._size = SharedCounter(0)

    def put(self, *args, **kwargs):
        self._size.increment(1)
        super().put(*args, **kwargs)

    def get(self, *args, **kwargs):
        if hasattr(self, "_size"):
            self._size.increment(-1)
        return super().get(*args, **kwargs)

    def qsize(self):
        """ Reliable implementation of multiprocessing.Queue.qsize() """
        return self._size.value

    def empty(self):
        """ Reliable implementation of multiprocessing.Queue.empty() """
        return not self.qsize()


# The actual update in Worker constructor
class Worker(Process):
    def __init__(
        self, settings: Settings, responses: Queue, env: Optional[Environment] = None
    ):
        super().__init__()
        self._settings = settings
        self._responses = responses

        if sys.platform == "darwin":
            self._requests = SharedCounterQueue()
            self._model_updates = SharedCounterQueue()
        else:
            self._requests: Queue[ModelRequestMessage] = Queue()
            self._model_updates: Queue[ModelUpdateMessage] = Queue()
        self._env = env

        self.__executor = None
```


I'm sure this code is far from production-ready but allows us to test mlserver running models on MacOS.
Same issue here. Running mlserver inside a Docker container works.
Hey @MikhailKravets @JustasUrbonas ,

Thanks for raising this one.

From this [SO thread](https://stackoverflow.com/a/66940259/5015573), I think the best solution may be to just use the `Queue` returned by the `Manager()` object? That one seems to have a `qsize()` implementation working in macOS. The [vterron/lemon](https://github.com/vterron/lemon/blob/d60576bec2ad5d1d5043bcb3111dff1fcb58a8d6/methods.py#L536-L573) implementation is (unfortunately) GPL, so we can't use that one. 
Hi @adriangonz,

Unfortunately, this approach will not work. `multiprocessing.Manager()` creates [SyncManager](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager). So, when you run

```python
manager = multiprocessing.Manager()
queue = manager.Queue()
```

The `queue` is [Proxy Object](https://docs.python.org/3/library/multiprocessing.html#proxy-objects) to a real Queue. This object doesn't have all attributes of the original Queue which leads to further errors, for example, `coro_run` method on line 137 of `worker.py` uses the `_reader` [Connection](https://docs.python.org/3/library/multiprocessing.html#connection-objects) which isn't present in Proxy.

Anyway, I noticed that `qsize()` method of the Queue is used for some kind of monitoring (?). So I've slightly updated `_workers_queue_monitor ` method in `mlserver/parallel/dispatcher.py:Dispatcher` line 113

```python
  def _workers_queue_monitor(self, worker: Worker, worker_pid: int):
      """Get metrics from every worker request queue"""

      if sys.platform == "darwin":
          logger.warning(
              "Worker metrics are not available on MacOS. "
              "qsize method from multiprocessing:queues.py:126 Raises NotImplementedError "
              "on Mac OSX because of broken sem_getvalue()"
          )
          return
      queue_size = worker._requests.qsize()

      self.parallel_request_queue_size.labels(workerpid=str(worker_pid)).observe(
          float(queue_size)
      )
```

I can't say what it breaks so far but it makes model endpoints work 🎉 

Resolving patch:
diff --git a/mlserver/parallel/dispatcher.py b/mlserver/parallel/dispatcher.py
--- a/mlserver/parallel/dispatcher.py
+++ b/mlserver/parallel/dispatcher.py
@@ -41,7 +41,6 @@ def _get_or_create_metric(self) -> Histogram:
         return Histogram(
             QUEUE_METRIC_NAME,
             "counter of request queue size for workers",
-            ["workerpid"],
             registry=REGISTRY,
         )
 
@@ -96,7 +95,6 @@ async def dispatch_request(
         self, request_message: ModelRequestMessage
     ) -> ModelResponseMessage:
         worker, wpid = self._get_worker()
-        self._workers_queue_monitor(worker, wpid)
         worker.send_request(request_message)
 
         return await self._dispatch(request_message)
@@ -109,14 +107,6 @@ def _get_worker(self) -> Tuple[Worker, int]:
         worker_pid = next(self._workers_round_robin)
         return self._workers[worker_pid], worker_pid
 
-    def _workers_queue_monitor(self, worker: Worker, worker_pid: int):
-        """Get metrics from every worker request queue"""
-        queue_size = worker._requests.qsize()
-
-        self.parallel_request_queue_size.labels(workerpid=str(worker_pid)).observe(
-            float(queue_size)
-        )
-
     async def dispatch_update(
         self, model_update: ModelUpdateMessage
     ) -> List[ModelResponseMessage]:
@@ -143,6 +133,8 @@ async def _dispatch(self, message: Message) -> ModelResponseMessage:
         internal_id = message.id
         self._async_responses[internal_id] = async_response
 
+        # Monitor current in-flight requests
+        self.parallel_request_queue_size.observe(len(self._async_responses))
         return await self._wait_response(internal_id)
 
     async def _wait_response(self, internal_id: str) -> ModelResponseMessage:


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -25,9 +25,10 @@ jobs:
           make lint
 
   tests:
-    runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
+        os: [ubuntu-latest, macos-latest]
         python-version: ["3.8", "3.9", "3.10"]
         tox-environment:
           - mlserver
@@ -38,7 +39,18 @@ jobs:
           - huggingface
           - alibi-explain
           - alibi-detect
+    runs-on: ${{ matrix.os }}
     steps:
+      - name: Setup docker (missing on MacOS)
+        if: runner.os == 'macos'
+        run: |
+          # From https://github.com/actions/runner-images/issues/17#issuecomment-1537238473
+          # From https://github.com/abiosoft/colima/discussions/273#discussioncomment-4959736
+          brew install docker docker-buildx
+          mkdir -p $HOME/.docker/cli-plugins
+          ln -sfn $(which docker-buildx) $HOME/.docker/cli-plugins/docker-buildx
+          colima start --memory 4
+          sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
@@ -56,12 +68,24 @@ jobs:
           tox -e ${{ matrix.tox-environment }}
 
   all-runtimes:
-    runs-on: ubuntu-latest
     if: github.event_name == 'push'
     strategy:
+      fail-fast: false
       matrix:
+        os: [ubuntu-latest, macos-latest]
         python-version: ["3.8", "3.9", "3.10"]
+    runs-on: ${{ matrix.os }}
     steps:
+      - name: Setup docker (missing on MacOS)
+        if: runner.os == 'macos'
+        run: |
+          # From https://github.com/actions/runner-images/issues/17#issuecomment-1537238473
+          # From https://github.com/abiosoft/colima/discussions/273#discussioncomment-4959736
+          brew install docker docker-buildx
+          mkdir -p $HOME/.docker/cli-plugins
+          ln -sfn $(which docker-buildx) $HOME/.docker/cli-plugins/docker-buildx
+          colima start --memory 4
+          sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
diff --git a/tests/utils.py b/tests/utils.py
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -120,7 +120,12 @@ async def _retry_get(self, endpoint: str):
             attempts=10,
             start_timeout=0.5,
             statuses=[400],
-            exceptions={ClientConnectorError, ClientOSError, ServerDisconnectedError},
+            exceptions={
+                ClientConnectorError,
+                ClientOSError,
+                ServerDisconnectedError,
+                ConnectionRefusedError,
+            },
         )
         retry_client = RetryClient(raise_for_status=True, retry_options=retry_options)