Repository: kserve/kserve

Issue:
Unexpected result for SDK replace method with watch=True
/kind bug

**What steps did you take and what happened:**
Used Python SDK to:
* Execute SDK `create` method to create isvc and wait until it's available
* Execute SDK `replace` method (with `watch=True`) to create new revision

**What did you expect to happen:**
In SDK `replace` method with `watch=True` I'd expect to wait for the new revision to become available. Instead watch immediately returns with `READY` equal to `True` because isvc was already available from the previous revision.

The wording in SDK documentation for any non-`create` method that accepts `watch=True` is a bit misleading. What would be the case for using `watch=True` in `replace`, `patch`, `promote` etc.?


**Anything else you would like to add:**
I'd like to understand how in general one can (with SDK) `create` isvc and then `replace` it while checking if and when new revision becomes available. 

In other words, how can one use SDK to create pinned or canary rollout and observe new revision's progress.


Discussion:
I've noticed that after calling `replace` isvc is going to `READY` == `False` (while previous revision is still available for predictions). It's just not happening fast enough for watch logic to catch it.

Quick solution to the issue would be to wait for a few seconds and then call [isvc_watch](https://github.com/kubeflow/kfserving/blob/1381f86e9b6ad831416008f5d25f0ed693f70871/python/kfserving/kfserving/api/kf_serving_client.py#L258). Seems a bit flaky though.

Resolving patch:
diff --git a/python/kserve/kserve/api/kserve_client.py b/python/kserve/kserve/api/kserve_client.py
--- a/python/kserve/kserve/api/kserve_client.py
+++ b/python/kserve/kserve/api/kserve_client.py
@@ -251,7 +251,8 @@ def replace(self, name, inferenceservice, namespace=None, watch=False,
             isvc_watch(
                 name=outputs['metadata']['name'],
                 namespace=namespace,
-                timeout_seconds=timeout_seconds)
+                timeout_seconds=timeout_seconds,
+                generation=outputs['metadata']['generation'])
         else:
             return outputs
 
diff --git a/python/kserve/kserve/api/watch.py b/python/kserve/kserve/api/watch.py
--- a/python/kserve/kserve/api/watch.py
+++ b/python/kserve/kserve/api/watch.py
@@ -21,7 +21,7 @@
 from ..utils import utils
 
 
-def isvc_watch(name=None, namespace=None, timeout_seconds=600):
+def isvc_watch(name=None, namespace=None, timeout_seconds=600, generation=0):
     """Watch the created or patched InferenceService in the specified namespace"""
 
     if namespace is None:
@@ -46,24 +46,29 @@ def isvc_watch(name=None, namespace=None, timeout_seconds=600):
         if name and name != isvc_name:
             continue
         else:
+            status = 'Unknown'
             if isvc.get('status', ''):
                 url = isvc['status'].get('url', '')
                 traffic = isvc['status'].get('components', {}).get(
                     'predictor', {}).get('traffic', [])
                 traffic_percent = 100
-                for t in traffic:
-                    if t["latestRevision"]:
-                        traffic_percent = t["percent"]
-                status = 'Unknown'
-                for condition in isvc['status'].get('conditions', {}):
-                    if condition.get('type', '') == 'Ready':
-                        status = condition.get('status', 'Unknown')
-                tbl(isvc_name, status, 100-traffic_percent, traffic_percent, url)
+                if constants.OBSERVED_GENERATION in isvc['status']:
+                    observed_generation = isvc['status'][constants.OBSERVED_GENERATION]
+                    for t in traffic:
+                        if t["latestRevision"]:
+                            traffic_percent = t["percent"]
+
+                    if generation != 0 and observed_generation != generation:
+                        continue
+                    for condition in isvc['status'].get('conditions', {}):
+                        if condition.get('type', '') == 'Ready':
+                            status = condition.get('status', 'Unknown')
+                    tbl(isvc_name, status, 100-traffic_percent, traffic_percent, url)
+                    if status == 'True':
+                        break
+
             else:
-                tbl(isvc_name, 'Unknown', '', '', '')
+                tbl(isvc_name, status, '', '', '')
                 # Sleep 2 to avoid status section is not generated within a very short time.
                 time.sleep(2)
                 continue
-
-            if name == isvc_name and status == 'True':
-                break
diff --git a/python/kserve/kserve/constants/constants.py b/python/kserve/kserve/constants/constants.py
--- a/python/kserve/kserve/constants/constants.py
+++ b/python/kserve/kserve/constants/constants.py
@@ -66,3 +66,8 @@
     "FP64": "fp64_contents",
     "BYTES": "bytes_contents"
 }
+# K8S status key constants
+OBSERVED_GENERATION = 'observedGeneration'
+
+# K8S metadata key constants
+GENERATION = 'generation'


Test patch: