Repository: kubeflow/kfp-tekton (language: python)

Issue:
kfp-tekton 0.8.0 Pypi install may fail
/kind bug

**What steps did you take and what happened:**
When a user install kfp-tekton sdk with `pip install kfp-tekton --upgrade` without building KFP 1.5.0 from source, it will show the below errors because KFP skipped the 1.5.0 SDK release.
```
tommyli$ pip install kfp-tekton --upgrade
Looking in indexes: https://pypi.org/simple/, https://tommy.chaoping.li%40ibm.com:****@na.artifactory.swg-devops.com/artifactory/api/pypi/dbg-aiworkspace-team-pypi-local/simple
Requirement already satisfied: kfp-tekton in ./opt/anaconda3/envs/py3/lib/python3.8/site-packages (0.7.0)
Collecting kfp-tekton
  Using cached kfp-tekton-0.8.0.tar.gz (46 kB)
ERROR: Packages installed from PyPI cannot depend on packages which are not also hosted on PyPI.
kfp-tekton depends on kfp@ git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python
```

**What did you expect to happen:**

pip install from pypi should work without errors.

**Additional information:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

* Python Version (use `python --version`):
* SDK Version: 
* Tekton Version (use `tkn version`):
* Kubernetes Version (use `kubectl version`):
* OS (e.g. from `/etc/os-release`):



Discussion / hints:
/assign @ckadner 
@ckadner are you planning just update the kfp sdk version to 1.6.x to work around this? I think some other users are starting to hit this issue.
kfp 1.6 sdk release is more focus on the v2 design, so no significant impact to the actual yaml other than few bug fixes.
Sorry, dropped the ball on this one. I am trying [kfp==1.6.3](https://pypi.org/project/kfp/1.6.3/) right now

Fix patch:
diff --git a/sdk/python/README.md b/sdk/python/README.md
index a8d5332f6e..6224709f30 100644
--- a/sdk/python/README.md
+++ b/sdk/python/README.md
@@ -108,7 +108,8 @@ the Tekton YAML instead of Argo YAML. Since the KFP SDK was not designed and imp
 _monkey-patching_ was used to replace non-class methods and functions at runtime.
 
 In order for the _monkey patch_ to work properly, the `kfp-tekton` compiler source code has to be aligned with a
-specific version of the `kfp` SDK compiler. As of now that version is [`1.5.0`](https://github.com/kubeflow/pipelines/releases/tag/1.5.0).
+specific version of the `kfp` SDK compiler. As of now the `kfp-tekton` SDK version is `0.8.1` which is aligned with KFP 
+SDK version [`1.6.3`](https://pypi.org/project/kfp/1.6.3/).
 
 
 ## Adding New Code
@@ -223,7 +224,7 @@ access secrets:
 
 You can also run the dynamically generated end-to-end test suite which takes all of the "golden" YAML files from the
 compiler `testdata` directory and runs them on a Kubernetes cluster, prerequisite that the environment variable
-`KUBECONFIG` is set and the K8s cluster has both Kubeflow Pipelines as well as Tekton Pipelines installed:
+`KUBECONFIG` is set and the K8s cluster has both Kubeflow Pipelines and Tekton Pipelines installed:
 
     make e2e_test
 
@@ -246,7 +247,7 @@ To update the ["Compiler Status Report"](/sdk/python/tests/README.md) use the ou
 ## License Headers
 
 All source files should have the following license header. Adjust the year accordingly to reflect the year the file was
-added and the last year it was modified:
+added, and the last year it was modified:
 
     # Copyright 2019-2021 kubeflow.org
     #
@@ -329,6 +330,3 @@ if __name__ == '__main__':
     from kfp_tekton.compiler import TektonCompiler
     TektonCompiler().compile(pipeline_func, __file__.replace('.py', '.yaml'))
 ```
-
-
-
diff --git a/sdk/python/kfp_tekton/__init__.py b/sdk/python/kfp_tekton/__init__.py
index 843687e209..9511bfeefe 100644
--- a/sdk/python/kfp_tekton/__init__.py
+++ b/sdk/python/kfp_tekton/__init__.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = '0.8.0'
+__version__ = '0.8.1'
 
 from ._client import TektonClient  # noqa F401
 from .k8s_client_helper import env_from_secret  # noqa F401
diff --git a/sdk/python/requirements.in b/sdk/python/requirements.in
index 9743334aeb..8a1727e2e2 100644
--- a/sdk/python/requirements.in
+++ b/sdk/python/requirements.in
@@ -1,3 +1,1 @@
-# TODO
-#kfp==1.5.0
-git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python
+kfp==1.6.3
diff --git a/sdk/python/requirements.txt b/sdk/python/requirements.txt
index cfd4ee3dbe..b9c8b98ff3 100644
--- a/sdk/python/requirements.txt
+++ b/sdk/python/requirements.txt
@@ -30,10 +30,18 @@ docstring-parser==0.7.3
 fire==0.4.0
     # via kfp
 google-api-core==1.26.0
-    # via google-cloud-core
+    # via
+    #   google-api-python-client
+    #   google-cloud-core
+google-api-python-client==1.12.8
+    # via kfp
+google-auth-httplib2==0.1.0
+    # via google-api-python-client
 google-auth==1.27.0
     # via
     #   google-api-core
+    #   google-api-python-client
+    #   google-auth-httplib2
     #   google-cloud-core
     #   google-cloud-storage
     #   kfp
@@ -48,17 +56,19 @@ google-resumable-media==1.2.0
     # via google-cloud-storage
 googleapis-common-protos==1.52.0
     # via google-api-core
+httplib2==0.19.1
+    # via
+    #   google-api-python-client
+    #   google-auth-httplib2
 idna==2.10
     # via requests
 jsonschema==3.2.0
     # via kfp
-kfp-pipeline-spec==0.1.6
+kfp-pipeline-spec==0.1.8
     # via kfp
 kfp-server-api==1.3.0
     # via kfp
-# kfp==1.5.0
-    # TODO
-git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python
+kfp==1.6.3
     # via -r sdk/python/requirements.in
 kubernetes==12.0.1
     # via kfp
@@ -70,6 +80,7 @@ protobuf==3.15.2
     # via
     #   google-api-core
     #   googleapis-common-protos
+    #   kfp
     #   kfp-pipeline-spec
 pyasn1-modules==0.2.8
     # via google-auth
@@ -80,7 +91,9 @@ pyasn1==0.4.8
 pycparser==2.20
     # via cffi
 pyparsing==2.4.7
-    # via packaging
+    # via
+    #   httplib2
+    #   packaging
 pyrsistent==0.17.3
     # via jsonschema
 python-dateutil==2.8.1
@@ -111,7 +124,9 @@ six==1.15.0
     #   absl-py
     #   fire
     #   google-api-core
+    #   google-api-python-client
     #   google-auth
+    #   google-auth-httplib2
     #   google-cloud-core
     #   google-resumable-media
     #   jsonschema
@@ -126,6 +141,8 @@ tabulate==0.8.9
     # via kfp
 termcolor==1.1.0
     # via fire
+uritemplate==3.0.1
+    # via google-api-python-client
 urllib3==1.26.3
     # via
     #   kfp-server-api
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
index 993529a457..8298c9e863 100644
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -20,14 +20,14 @@
 #
 # To create a distribution for PyPi run:
 #
-#    $ export KFP_TEKTON_VERSION=0.8.0-rc1
+#    $ export KFP_TEKTON_VERSION=0.8.1-rc1
 #    $ python3 setup.py sdist
 #    $ twine check dist/kfp-tekton-${KFP_TEKTON_VERSION/-rc/rc}.tar.gz
 #    $ twine upload --repository pypi dist/kfp-tekton-${KFP_TEKTON_VERSION/-rc/rc}.tar.gz
 #
 #   ... or:
 #
-#    $ make distribution KFP_TEKTON_VERSION=0.8.0-rc1
+#    $ make distribution KFP_TEKTON_VERSION=0.8.1-rc1
 #
 # =============================================================================
 
@@ -54,7 +54,7 @@
 # NOTICE, after any updates to the following, ./requirements.in should be updated
 # accordingly.
 REQUIRES = [
-    "kfp @ git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python",
+    "kfp==1.6.3",
 ]
 
 TESTS_REQUIRE = [


Test patch:
diff --git a/sdk/python/tests/README.md b/sdk/python/tests/README.md
index 692d256947..a66f2c9c9a 100644
--- a/sdk/python/tests/README.md
+++ b/sdk/python/tests/README.md
@@ -27,7 +27,7 @@ You should see an output similar to the one below, outlining which test scripts
 which are failing:
 
 ```YAML
-KFP version: 1.5.0
+KFP SDK version: 1.6.3
 
 SUCCESS: add_pod_env.py
 SUCCESS: artifact_passing_using_volume.py
@@ -140,5 +140,5 @@ Occurences of other Errors:
 
 ## Disclaimer
 
-**Note:** The reports above were created for the pipeline scripts found in KFP version `1.5.0` since
-the `kfp_tekton` compiler code is currently based on the `kfp` SDK compiler version `1.5.0`.
+**Note:** The reports above were created for the pipeline scripts found in KFP SDK version `1.6.3` since
+the `kfp_tekton` `0.8.1` compiler code is based on the `kfp` SDK compiler version `1.6.3`.
diff --git a/sdk/python/tests/compiler/testdata/resourceop_basic.yaml b/sdk/python/tests/compiler/testdata/resourceop_basic.yaml
index ba76a81ded..5f1fad3de6 100644
--- a/sdk/python/tests/compiler/testdata/resourceop_basic.yaml
+++ b/sdk/python/tests/compiler/testdata/resourceop_basic.yaml
@@ -67,7 +67,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
diff --git a/sdk/python/tests/compiler/testdata/volume_op.yaml b/sdk/python/tests/compiler/testdata/volume_op.yaml
index 892b9c47b2..065cec06d4 100644
--- a/sdk/python/tests/compiler/testdata/volume_op.yaml
+++ b/sdk/python/tests/compiler/testdata/volume_op.yaml
@@ -78,7 +78,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
diff --git a/sdk/python/tests/compiler/testdata/volume_snapshot_op.yaml b/sdk/python/tests/compiler/testdata/volume_snapshot_op.yaml
index 3128da9c1b..42842147a0 100644
--- a/sdk/python/tests/compiler/testdata/volume_snapshot_op.yaml
+++ b/sdk/python/tests/compiler/testdata/volume_snapshot_op.yaml
@@ -84,7 +84,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -197,7 +197,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -310,7 +310,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -423,7 +423,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
diff --git a/sdk/python/tests/test_kfp_samples.sh b/sdk/python/tests/test_kfp_samples.sh
index 5af67d19cf..817526b008 100755
--- a/sdk/python/tests/test_kfp_samples.sh
+++ b/sdk/python/tests/test_kfp_samples.sh
@@ -52,9 +52,9 @@ function help {
 # process command line parameters
 while (( $# > 0 )); do
   case "$1" in
-    -v|--kfp-version)          KFP_VERSION="$2";            shift 2 ;;  # KFP SDK version, default: 1.0.0
+    -v|--kfp-version)          KFP_VERSION="$2";            shift 2 ;;  # KFP SDK version, default: 1.5.0
     -a|--include-all-samples)  ALL_SAMPLES="TRUE";          shift 1 ;;  # Compile all DSL scripts in KFP repo
-    -s|--dont-list-files)       SKIP_FILES="TRUE";           shift 1 ;;  # Suppress compile status for each DSL file
+    -s|--dont-list-files)      SKIP_FILES="TRUE";           shift 1 ;;  # Suppress compile status for each DSL file
     -e|--print-error-details)  PRINT_ERRORS="TRUE";         shift 1 ;;  # Print summary of compilation errors
     -h|--help)                 help;                        exit 0  ;;  # Show this help message
     -*)                        echo "Unknown option '$1'";  exit 1  ;;
@@ -63,7 +63,8 @@ while (( $# > 0 )); do
 done
 
 # define global variables
-KFP_VERSION=${KFP_VERSION:-1.5.0}
+KFP_GIT_VERSION=${KFP_VERSION:-1.5.0}
+KFP_SDK_VERSION=${KFP_VERSION:-1.6.3}
 KFP_REPO_URL="https://github.com/kubeflow/pipelines.git"
 SCRIPT_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd)"
 PROJECT_DIR="${TRAVIS_BUILD_DIR:-$(cd "${SCRIPT_DIR%/sdk/python/tests}"; pwd)}"
@@ -76,6 +77,10 @@ COMPILER_OUTPUTS_FILE="${TEMP_DIR}/test_kfp_samples_output.txt"
 CONFIG_FILE="${PROJECT_DIR}/sdk/python/tests/config.yaml"
 REPLACE_EXCEPTIONS="FALSE" # "TRUE" | "FALSE"
 
+# print KFP SDK and GIT versions (might be different)
+#echo "KFP_GIT_VERSION=${KFP_GIT_VERSION}"
+#echo "KFP_SDK_VERSION=${KFP_SDK_VERSION}"
+
 # show value of the _DIR variables, use set in POSIX mode in a sub-shell to supress functions definitions (https://stackoverflow.com/a/1305273/5601796)
 #(set -o posix ; set) | grep "_DIR" | sort
 
@@ -99,14 +104,14 @@ cp "${COMPILE_REPORT_FILE}" "${COMPILE_REPORT_FILE_OLD}"
 
 # clone kubeflow/pipeline repo to get the testdata DSL scripts
 if [ ! -d "${KFP_CLONE_DIR}" ]; then
-  git -c advice.detachedHead=false clone -b "${KFP_VERSION}" "${KFP_REPO_URL}" "${KFP_CLONE_DIR}" -q
+  git -c advice.detachedHead=false clone -b "${KFP_GIT_VERSION}" "${KFP_REPO_URL}" "${KFP_CLONE_DIR}" -q
 else
   cd "${KFP_CLONE_DIR}"
   git fetch --all -q
-  git -c advice.detachedHead=false checkout "${KFP_VERSION}" -f -q
+  git -c advice.detachedHead=false checkout "${KFP_GIT_VERSION}" -f -q
   cd - &> /dev/null
 fi
-echo "KFP version: $(git --git-dir "${KFP_CLONE_DIR}"/.git tag --points-at HEAD)"
+echo "KFP clone version: $(git --git-dir "${KFP_CLONE_DIR}"/.git tag --points-at HEAD)"
 
 # check if we are running in a Python virtual environment, if not create one
 if [ ! -d "${VENV_DIR}" ]; then
@@ -117,10 +122,10 @@ if [ ! -d "${VENV_DIR}" ]; then
 fi
 source "${VENV_DIR}/bin/activate"
 
-# install KFP with the desired KFP_VERSION (unless already installed)
-if ! (pip show "kfp" | grep Version | grep -q "${KFP_VERSION}"); then
-  echo "Installing KFP ${KFP_VERSION} ..."
-  pip install -q -e "${KFP_CLONE_DIR}/sdk/python"
+# install KFP with the desired KFP SDK version (unless already installed)
+if ! (pip show "kfp" | grep Version | grep -q "${KFP_SDK_VERSION}"); then
+  echo "Installing KFP SDK ${KFP_SDK_VERSION} ..."
+  pip install -q kfp==${KFP_SDK_VERSION}
 fi
 
 # install KFP-Tekton compiler, unless already installed
@@ -138,9 +143,13 @@ if [[ "${ALL_SAMPLES}" == "TRUE" ]]; then
 
   # reinstall KFP with the desired version to get all of its dependencies with their respective desired versions
   # pip uninstall -q -y kfp
-  pip install -q -e "${KFP_CLONE_DIR}/sdk/python"
+  pip install -q kfp==${KFP_SDK_VERSION} --force-reinstall
 fi
 
+# confirm installed kfp version(s)
+echo "KFP Python SDK version(s):"
+pip list | grep kfp
+
 echo  # just adding some separation for console output
 
 # replace NotImplementedError with simple print out