Repository: bentoml/BentoML

Issue:
bug: fail to request readyz interface if the model is still loading
### Describe the bug

### background

When the model loading time is very long, about seven minutes, I try to find a time point available for the runner to provide external services

### resolution

I found that the readyz interface can effectively solve this issue.  In development mode,  it's works. But the following error occurred in production mode

When the runner fails to load the model completely, requesting the readyz interface will result in the following error

```python
2023-07-05T08:48:25+0000 [ERROR] [api_server:stable_diffusion_cg_fp16:2] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http/traffic.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http/traffic.py", line 52, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http/instruments.py", line 176, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/usr/local/lib/python3.8/dist-packages/opentelemetry/instrumentation/asgi/__init__.py", line 579, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http/access.py", line 126, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http_app.py", line 286, in readyz
    runners_ready = all(await asyncio.gather(*runner_statuses))
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/runner/runner.py", line 156, in runner_handle_is_ready
    return await self._runner_handle.is_ready(timeout)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/runner/runner_handle/remote.py", line 304, in is_ready
    async with self._client.get(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client_reqrep.py", line 914, in start
    self._continue = None
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
```

### To reproduce

_No response_

### Expected behavior

_No response_

### Environment

bentoml: 1.0.22
python: 3.8.0


### configure

```yaml
version: 1
runners: 
  traffic: 
    timeout: 900
  resources:
    cpu: 8
    nvidia.com/gpu: 1
  workers_per_resource: 2
api_server:
  traffic:
    timeout: 900
    max_concurrency: 2
  workers: 2
  http:
    port: 5000
```

### Start command

`BENTOML_CONFIG=/src/config/bentoml_configuration.yaml bentoml serve service:svc --production --host 0.0.0.0`



Discussion:
It should be a bug that the timeout exception is not caught. We will try to fix it.

But I thinks your case can still works. You can wait until the /readyz response 200.

Resolving patch:
diff --git a/src/bentoml/_internal/runner/runner_handle/remote.py b/src/bentoml/_internal/runner/runner_handle/remote.py
--- a/src/bentoml/_internal/runner/runner_handle/remote.py
+++ b/src/bentoml/_internal/runner/runner_handle/remote.py
@@ -297,18 +297,22 @@ async def is_ready(self, timeout: int) -> bool:
         # default kubernetes probe timeout is also 1s; see
         # https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
         aio_timeout = aiohttp.ClientTimeout(total=timeout)
-        async with self._client.get(
-            f"{self._addr}/readyz",
-            headers={
-                "Bento-Name": component_context.bento_name,
-                "Bento-Version": component_context.bento_version,
-                "Runner-Name": self._runner.name,
-                "Yatai-Bento-Deployment-Name": component_context.yatai_bento_deployment_name,
-                "Yatai-Bento-Deployment-Namespace": component_context.yatai_bento_deployment_namespace,
-            },
-            timeout=aio_timeout,
-        ) as resp:
-            return resp.status == 200
+        try:
+            async with self._client.get(
+                f"{self._addr}/readyz",
+                headers={
+                    "Bento-Name": component_context.bento_name,
+                    "Bento-Version": component_context.bento_version,
+                    "Runner-Name": self._runner.name,
+                    "Yatai-Bento-Deployment-Name": component_context.yatai_bento_deployment_name,
+                    "Yatai-Bento-Deployment-Namespace": component_context.yatai_bento_deployment_namespace,
+                },
+                timeout=aio_timeout,
+            ) as resp:
+                return resp.status == 200
+        except asyncio.TimeoutError:
+            logger.warn("Timed out waiting for runner to be ready")
+            return False
 
     def __del__(self) -> None:
         self._close_conn()


Test patch: