Repository: kserve/kserve

Issue:
kubernetes probes using HealthHandler endpoint succeed when models are not ready
/kind bug

**What steps did you take and what happened:**
Used the [HealthHandler endpoint](https://github.com/kubeflow/kfserving/blob/92c5f3c9df662c853beef86bddf7f30ab5d8b996/python/kfserving/kfserving/kfserver.py#L62-L63) to set up a Kubernetes readiness probe. The probe succeeded even though the model was not ready. 

**What did you expect to happen:**
Readiness probe failure when model is not ready. 

**Anything else you would like to add:**
[Kubernetes probe documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) says `Any code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure`, which seems to imply that the [response contents](https://github.com/kubeflow/kfserving/blob/92c5f3c9df662c853beef86bddf7f30ab5d8b996/python/kfserving/kfserving/kfserver.py#L108-L111) (which include the model's `.ready` property) are ignored. 


**Environment:**

- Istio Version:
- Knative Version:
- KFServing Version:
- Kubeflow version:
- Minikube version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):



Discussion:
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| bug | 0.99 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
/good-first-issue
@yuzisun: 
	This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed [here](https://git.k8s.io/community/contributors/guide/help-wanted.md#good-first-issue).

If this request no longer meets these requirements, the label can be removed
by commenting with the `/remove-good-first-issue` command.


<details>

In response to [this](https://github.com/kubeflow/kfserving/issues/799):

>/good-first-issue


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
I can make a PR for this. should the error code when the model is not ready be 404 or a custom one ?
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| area/engprod | 0.76 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
Wait we're returning 200 before the model is loaded? 400 or 503 (service unavailable) is fine to start. Happy to hear suggestions if you have other ideas.
i was thinking 503, but i don't feel too strongly about the particular code that's used. 

Resolving patch:
diff --git a/python/kfserving/kfserving/kfserver.py b/python/kfserving/kfserving/kfserver.py
--- a/python/kfserving/kfserving/kfserver.py
+++ b/python/kfserving/kfserving/kfserver.py
@@ -105,6 +105,12 @@ def get(self, name: str):
             )
 
         model = self.models[name]
+        if not model.ready:
+            raise tornado.web.HTTPError(
+                status_code=503,
+                reason="Model with name %s is not ready." % name
+            )
+
         self.write(json.dumps({
             "name": model.name,
             "ready": model.ready


Test patch:
diff --git a/python/kfserving/test/test_server.py b/python/kfserving/test/test_server.py
--- a/python/kfserving/test/test_server.py
+++ b/python/kfserving/test/test_server.py
@@ -14,6 +14,7 @@
 
 import pytest
 import kfserving
+from tornado.httpclient import HTTPClientError
 
 
 class DummyModel(kfserving.KFModel):
@@ -65,3 +66,18 @@ async def test_explain(self, http_server_client):
         assert resp.code == 200
         assert resp.body == b'{"predictions": [[1, 2]]}'
         assert resp.headers['content-type'] == "application/json; charset=UTF-8"
+
+
+class TestTFHttpServerModelNotLoaded():
+
+    @pytest.fixture(scope="class")
+    def app(self):  # pylint: disable=no-self-use
+        model = DummyModel("TestModel")
+        server = kfserving.KFServer()
+        server.register_model(model)
+        return server.create_application()
+
+    async def test_model_not_ready_error(self, http_server_client):
+        with pytest.raises(HTTPClientError) as excinfo:
+            _ = await http_server_client.fetch('/v1/models/TestModel')
+        assert excinfo.value.code == 503