Repository: kserve/kserve

Issue:
Add exception handler for model server
/kind bug
Model server shutdown is not properly handled when an Exception occurs.
```
Traceback (most recent call last):
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/sklearnserver/sklearnserver/__main__.py", line 41, in <module>
    kserve.ModelServer(registered_models=SKLearnModelRepository(args.model_dir), workers=6).start(
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 229, in start
    asyncio.run(servers_task())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 227, in servers_task
    await asyncio.gather(*servers)
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 214, in serve
    raise RuntimeError("Some Exception")
RuntimeError: Some Exception
Exception ignored in: <function Server.__del__ at 0x7f90add16170>
Traceback (most recent call last):
  File "/home/ubuntu/go/src/github.com/kserve/kserve/.venv/lib/python3.10/site-packages/grpc/aio/_server.py", line 185, in __del__
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 118, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 110, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "/usr/lib/python3.10/asyncio/base_events.py", line 436, in create_task
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'AioServer.shutdown' was never awaited
```
**Solution**:
Add an exception handler to the event loop to handle the shutdown process.


**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]


**What did you expect to happen:**
The model server should be properly shutdown when an exception occurs.

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version: 0.10 - 0.12.0-rc1
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:

Resolving patch:
diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py
--- a/python/kserve/kserve/model_server.py
+++ b/python/kserve/kserve/model_server.py
@@ -20,7 +20,7 @@
 import signal
 import socket
 from multiprocessing import Process
-from typing import Dict, List, Optional, Union
+from typing import Dict, List, Optional, Union, Callable, Any
 
 from ray import serve as rayserve
 from ray.serve.api import Deployment
@@ -140,6 +140,7 @@ def __init__(self, http_port: int = args.http_port,
             self.log_config = None
 
         self.access_log_format = access_log_format
+        self._custom_exception_handler = None
 
     def start(self, models: Union[List[Model], Dict[str, Deployment]]) -> None:
         """ Start the model server with a set of registered models.
@@ -187,6 +188,10 @@ async def serve():
                 loop.add_signal_handler(
                     sig, lambda s=sig: asyncio.create_task(self.stop(sig=s))
                 )
+            if self._custom_exception_handler is None:
+                loop.set_exception_handler(self.default_exception_handler)
+            else:
+                loop.set_exception_handler(self._custom_exception_handler)
             if self.workers == 1:
                 self._rest_server = UvicornServer(self.http_port, [],
                                                   self.dataplane, self.model_repository_extension,
@@ -231,6 +236,31 @@ async def stop(self, sig: Optional[int] = None):
             logger.info("Stopping the grpc server")
             await self._grpc_server.stop(sig)
 
+    def register_exception_handler(self, handler: Callable[[asyncio.events.AbstractEventLoop, Dict[str, Any]], None]):
+        """
+        Add a custom handler as the event loop exception handler.
+
+        If a handler is not provided, the default exception handler will be set.
+
+        handler should be a callable object, it should have a signature matching '(loop, context)', where 'loop'
+        will be a reference to the active event loop, 'context' will be a dict object (see `call_exception_handler()`
+        documentation for details about context).
+        """
+        self._custom_exception_handler = handler
+
+    def default_exception_handler(self, loop: asyncio.events.AbstractEventLoop, context: Dict[str, Any]):
+        """
+        Default exception handler for event loop.
+
+        This is called when an exception occurs and no exception handler is set.
+        By default, this will shut down the server gracefully.
+
+        This can be called by a custom exception handler that wants to defer to the default handler behavior.
+        """
+        # gracefully shutdown the server
+        loop.run_until_complete(self.stop())
+        loop.default_exception_handler(context)
+
     def register_model_handle(self, name: str, model_handle: RayServeHandle):
         self.registered_models.update_handle(name, model_handle)
         logger.info("Registering model handle: %s", name)


Test patch: