Repository: kserve/kserve

Issue:
[Help wanted] Add e2e test for canary rollout
/kind feature

**Describe the solution you'd like**
[A clear and concise description of what you want to happen.]


**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]



Discussion:
/area testing
/priority p2
@yuzisun 
I can help to add e2e test for this issue (as well as https://github.com/kubeflow/kfserving/issues/539) 

May I know what e2e test environment is? What is setup? I don't get much  info from the following site:
https://github.com/kubeflow/kfserving/tree/master/test
_To run the e2e tests, you need to have a running environment that meets the e2e test environment requirements. (@TODO)_
@janeman98 sorry just saw this, thanks for taking this issue!  Currently e2e tests run in our CI after PR is created or merged to the master, we will need to update the doc there.
- This is the e2e script which runs in the CI https://github.com/kubeflow/kfserving/blob/master/test/scripts/run-e2e-tests.sh
- All the e2e tests(https://github.com/kubeflow/kfserving/tree/master/test/e2e) are written with pytest and KFServing python sdk, you should be able to run that locally as well if you have the KFServing installation.

Let me know if you have further question! Really appreciate your help!
/assign janeman98

Resolving patch:
diff --git a/python/kfserving/kfserving/constants/constants.py b/python/kfserving/kfserving/constants/constants.py
--- a/python/kfserving/kfserving/constants/constants.py
+++ b/python/kfserving/kfserving/constants/constants.py
@@ -19,6 +19,7 @@
 KFSERVING_KIND = 'InferenceService'
 KFSERVING_PLURAL = 'inferenceservices'
 KFSERVING_VERSION = os.environ.get('KFSERVING_VERSION', 'v1alpha2')
+KFSERVING_API_VERSION = KFSERVING_GROUP + '/' + KFSERVING_VERSION
 
 KFSERVING_LOGLEVEL = os.environ.get('KFSERVING_LOGLEVEL', 'INFO').upper()
 


Test patch:
diff --git a/test/e2e/predictor/test_canary.py b/test/e2e/predictor/test_canary.py
new file mode 100644
--- /dev/null
+++ b/test/e2e/predictor/test_canary.py
@@ -0,0 +1,73 @@
+# Copyright 2019 kubeflow.org.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+from kubernetes import client
+
+from kfserving import KFServingClient
+from kfserving import constants
+from kfserving import V1alpha2EndpointSpec
+from kfserving import V1alpha2PredictorSpec
+from kfserving import V1alpha2TensorflowSpec
+from kfserving import V1alpha2InferenceServiceSpec
+from kfserving import V1alpha2InferenceService
+from kubernetes.client import V1ResourceRequirements
+
+from ..common.utils import predict
+from ..common.utils import KFSERVING_TEST_NAMESPACE
+
+
+# Setting config_file is required since SDK is running in a different cluster than KFServing
+KFServing = KFServingClient(config_file="~/.kube/config")
+
+
+def test_canary_rollout():
+    service_name = 'isvc-canary'
+    default_endpoint_spec = V1alpha2EndpointSpec(
+        predictor=V1alpha2PredictorSpec(
+            min_replicas=1,
+            tensorflow=V1alpha2TensorflowSpec(
+                storage_uri='gs://kfserving-samples/models/tensorflow/flowers',
+                resources=V1ResourceRequirements(
+                    requests={'cpu': '100m', 'memory': '256Mi'},
+                    limits={'cpu': '100m', 'memory': '256Mi'}))))
+
+    isvc = V1alpha2InferenceService(api_version=constants.KFSERVING_API_VERSION,
+                                    kind=constants.KFSERVING_KIND,
+                                    metadata=client.V1ObjectMeta(
+                                        name=service_name, namespace=KFSERVING_TEST_NAMESPACE),
+                                    spec=V1alpha2InferenceServiceSpec(default=default_endpoint_spec))
+
+    KFServing.create(isvc)
+    KFServing.wait_isvc_ready(service_name, namespace=KFSERVING_TEST_NAMESPACE)
+
+    # define canary endpoint spec, and then rollout 10% traffic to the canary version
+    canary_endpoint_spec = V1alpha2EndpointSpec(
+        predictor=V1alpha2PredictorSpec(
+            tensorflow=V1alpha2TensorflowSpec(
+                storage_uri='gs://kfserving-samples/models/tensorflow/flowers-2',
+                resources=V1ResourceRequirements(
+                    requests={'cpu':'100m','memory':'256Mi'},
+                    limits={'cpu':'100m', 'memory':'256Mi'}))))
+
+    KFServing.rollout_canary(service_name, canary=canary_endpoint_spec, percent=10,
+       namespace=KFSERVING_TEST_NAMESPACE, watch=True, timeout_seconds=120)
+    KFServing.wait_isvc_ready(service_name, namespace=KFSERVING_TEST_NAMESPACE)
+
+    # Promote Canary to Default
+    KFServing.promote(service_name, namespace=KFSERVING_TEST_NAMESPACE, watch=True, timeout_seconds=120)
+    KFServing.wait_isvc_ready(service_name, namespace=KFSERVING_TEST_NAMESPACE)
+
+    # Delete the InferenceService
+    KFServing.delete(service_name, namespace=KFSERVING_TEST_NAMESPACE)