Repository: kserve/kserve

Issue:
SDK can't create and watch inferenceservice at the same time
/kind bug

**What steps did you take and what happened:**
Running the below python SDK code to create and watch the inferenceservice occasionally have race condition on getting the status.
```python
KFServing.create(isvc, watch=True, timeout_seconds=120)
```

```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-06b315a74aac> in <module>
      1 KFServing = KFServingClient()
----> 2 KFServing.create(isvc, watch=True, timeout_seconds=120)

~/.local/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py in create(self, inferenceservice, namespace, watch, timeout_seconds)
    114                 name=outputs['metadata']['name'],
    115                 namespace=namespace,
--> 116                 timeout_seconds=timeout_seconds)
    117         else:
    118             return outputs

~/.local/lib/python3.6/site-packages/kfserving/api/kf_serving_watch.py in watch(name, namespace, timeout_seconds)
     46             continue
     47         else:
---> 48             url = isvc['status'].get('url', '')
     49             default_traffic = isvc['status'].get('traffic', '')
     50             canary_traffic = isvc['status'].get('canaryTraffic', '')

KeyError: 'status'
```
**What did you expect to happen:**
it should watch the inferenceservice without error out on race condition.


**Anything else you would like to add:**
This error happens way more often on kfserving SDK 0.3.0+


**Environment:**

- Istio Version: 1.1.6
- Knative Version: 1.11.2
- KFServing Version: 0.3.0
- Kubeflow version: 1.0.2
- Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube version:
- Kubernetes version: (use `kubectl version`): 1.15
- OS (e.g. from `/etc/os-release`):



Discussion:
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| area/inference | 0.83 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
Issue-Label Bot is automatically applying the labels:

| Label  | Probability |
| ------------- | ------------- |
| area/inference | 0.83 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 
Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://github.com/marketplace/issue-label-botdata/kubeflow/kfserving) and [code](https://github.com/hamelsmu/MLapp) for this bot.
/cc @jinchihe 
/assign

I will take a look asap. Thanks.
I did not reproduce the problem. But I think this is caused by env issue such as machine is slow so that the `status` of isvc is not generated. We can enhance here to ensure the status section is generated and then go on. Thanks.

Resolving patch:
diff --git a/python/kfserving/kfserving/api/kf_serving_watch.py b/python/kfserving/kfserving/api/kf_serving_watch.py
--- a/python/kfserving/kfserving/api/kf_serving_watch.py
+++ b/python/kfserving/kfserving/api/kf_serving_watch.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import time
 from kubernetes import client
 from kubernetes import watch as k8s_watch
 from table_logger import TableLogger
@@ -45,14 +46,20 @@ def watch(name=None, namespace=None, timeout_seconds=600):
         if name and name != isvc_name:
             continue
         else:
-            url = isvc['status'].get('url', '')
-            default_traffic = isvc['status'].get('traffic', '')
-            canary_traffic = isvc['status'].get('canaryTraffic', '')
-            status = 'Unknown'
-            for condition in isvc['status'].get('conditions', {}):
-                if condition.get('type', '') == 'Ready':
-                    status = condition.get('status', 'Unknown')
-            tbl(isvc_name, status, default_traffic, canary_traffic, url)
+            if isvc.get('status', ''):
+                url = isvc['status'].get('url', '')
+                default_traffic = isvc['status'].get('traffic', '')
+                canary_traffic = isvc['status'].get('canaryTraffic', '')
+                status = 'Unknown'
+                for condition in isvc['status'].get('conditions', {}):
+                    if condition.get('type', '') == 'Ready':
+                        status = condition.get('status', 'Unknown')
+                tbl(isvc_name, status, default_traffic, canary_traffic, url)
+            else:
+                tbl(isvc_name, 'Unknown', '', '', '')
+                # Sleep 2 to avoid status section is not generated within a very short time.
+                time.sleep(2)
+                continue
 
             if name == isvc_name and status == 'True':
                 break


Test patch: