Repository: kserve/kserve

Issue:
ModelServer not stopped correctly when using more than 1 worker
/kind bug

**What steps did you take and what happened:**
I created a custom model and served it with ModelServer with workers > 1
I trigger the autoscale making multiple requests and wait to pod to be deleted

**What did you expect to happen:**
All stopped pods will be deleted

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]
```
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    meta.helm.sh/release-name: my-model
    meta.helm.sh/release-namespace: my-model-services
  creationTimestamp: "2023-06-12T12:57:01Z"
  finalizers:
  - inferenceservice.finalizers
  generation: 62
  labels:
    app.kubernetes.io/managed-by: Helm
  name: my-model
  namespace: my-model-services
  resourceVersion: "125402392"
  uid: 26dfba87-cb4c-4be8-aefc-13c98ff34264
spec:
  predictor:
    batcher:
      maxBatchSize: 32
      maxLatency: 60000
    containerConcurrency: 5
    containers:
    - env:
      - name: KF_SERVER_WORKERS
        value: "1"
      - ...other model specific env variables...
      image: <INTERNAL_REPOSITORY_MODEL_IMAGE_URI>
      livenessProbe:
        failureThreshold: 5
        httpGet:
          path: /
          port: 8080
        initialDelaySeconds: 60
        periodSeconds: 10
      name: kserve-container
      readinessProbe:
        failureThreshold: 30
        httpGet:
          path: /v1/models/my-model
          port: 8080
        initialDelaySeconds: 60
        periodSeconds: 10
      resources:
        limits:
          cpu: "3"
          memory: 8Gi
        requests:
          cpu: "3"
          memory: 8Gi
    maxReplicas: 5
    minReplicas: 1
    serviceAccountName: my-model-sa
    timeout: 600
status:
  address:
    url: http://my-model.my-models-services.svc.cluster.local/v1/models/sim-dev:predict
  components:
    predictor:
      address:
        url: http://my-model-predictor-default.my-models-services.svc.cluster.local
      latestCreatedRevision: my-model-predictor-default-00056
      latestReadyRevision: my-model-predictor-default-00056
      latestRolledoutRevision: my-model-predictor-default-00056
      previousRolledoutRevision: my-model-predictor-default-00055
      traffic:
      - latestRevision: true
        percent: 100
        revisionName: my-model-predictor-default-00056
      url: http://my-model-predictor-default.my-models-services.example.com
  conditions:
  - lastTransitionTime: "2023-09-01T13:04:10Z"
    status: "True"
    type: IngressReady
  - lastTransitionTime: "2023-09-01T13:04:09Z"
    severity: Info
    status: "True"
    type: PredictorConfigurationReady
  - lastTransitionTime: "2023-09-01T13:04:10Z"
    status: "True"
    type: PredictorReady
  - lastTransitionTime: "2023-09-01T13:04:10Z"
    severity: Info
    status: "True"
    type: PredictorRouteReady
  - lastTransitionTime: "2023-09-01T13:04:10Z"
    status: "True"
    type: Ready
  modelStatus:
    copies:
      failedCopies: 0
      totalCopies: 3
    states:
      activeModelState: Loaded
      targetModelState: Loaded
    transitionStatus: UpToDate
  observedGeneration: 56
  url: http://my-model.my-models-services.example.com
```

**Anything else you would like to add:**
This is the code my model use to start Model Server:
```
if __name__ == "__main__":
    model = MyModel("my-model")
    model.load() # load the model before starting to serve
    ModelServer(workers=int(os.getenv('KF_SERVER_WORKERS', 1))).start([model])
```
I think that the main problem is [here](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L201): 
if workers is bigger than one, that attribute is never set and python raise an AttributeError exception


**Environment:**

- Istio Version: 1.15
- Knative Version: 1.7.0
- KServe Version: 0.10
- Kubeflow version:
- Cloud Environment: aws
- Minikube/Kind version: /
- Kubernetes version: (use `kubectl version`): 1.25
- OS (e.g. from `/etc/os-release`):


**EDIT 1**: Improved formatting of YAML
**EDIT 2**: Added the code I use to start the server


Discussion:
I created a fork with a possible fix: https://github.com/DanielWhite95/kserve/tree/fix-shutdown-multiple-workers

Resolving patch:
diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py
--- a/python/kserve/kserve/model_server.py
+++ b/python/kserve/kserve/model_server.py
@@ -108,6 +108,7 @@ def __init__(self, http_port: int = args.http_port,
         self.model_repository_extension = ModelRepositoryExtension(
             model_registry=self.registered_models)
         self._grpc_server = None
+        self._rest_server = None
         if self.enable_grpc:
             self._grpc_server = GRPCServer(grpc_port, self.dataplane,
                                            self.model_repository_extension)
@@ -180,12 +181,12 @@ async def serve():
                 serversocket.bind(('0.0.0.0', self.http_port))
                 serversocket.listen(5)
                 multiprocessing.set_start_method('fork')
-                server = UvicornServer(self.http_port, [serversocket],
-                                       self.dataplane, self.model_repository_extension,
-                                       self.enable_docs_url, log_config=self.log_config,
-                                       access_log_format=self.access_log_format)
+                self._rest_server = UvicornServer(self.http_port, [serversocket],
+                                                  self.dataplane, self.model_repository_extension,
+                                                  self.enable_docs_url, log_config=self.log_config,
+                                                  access_log_format=self.access_log_format)
                 for _ in range(self.workers):
-                    p = Process(target=server.run_sync)
+                    p = Process(target=self._rest_server.run_sync)
                     p.start()
 
         async def servers_task():


Test patch: