Repository: kubeflow/pipelines

Issue:
Kubeflow / AI Platform Pipelines runtime context missing when output is taken from cache
### What steps did you take:

1. deploy pipeline with one component
2. run pipeline with one component (👍 works)
3. add another component
4. run the pipeline (this time the output is taken from cache) (👎 fails)

### What happened:
The pipeline runs into the following error 
```
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main
    execution_info = launcher.launch()
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 198, in launch
    self._exec_properties)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 167, in _run_driver
    component_info=self._component_info)
  File "/opt/conda/lib/python3.7/site-packages/tfx/dsl/components/base/base_driver.py", line 270, in pre_execution
    driver_args, pipeline_info)
  File "/opt/conda/lib/python3.7/site-packages/tfx/dsl/components/base/base_driver.py", line 158, in resolve_input_artifacts
    producer_component_id=input_channel.producer_component_id)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/metadata.py", line 948, in search_artifacts
    pipeline_info)
RuntimeError: Pipeline run context for PipelineInfo(pipeline_name: sentiment4, pipeline_root: gs://sascha-playground-doit-kubeflowpipelines-default/sentiment4, run_id: sentiment4-qnknl) does not exist
```
Assume the second component doesn't find the cached data because the component did not exist in the first run.

First run:
<img width="1216" alt="1run" src="https://user-images.githubusercontent.com/1991664/111158263-1e9c1200-8598-11eb-969f-370aec9c4be2.png">


Second run with additional component
<img width="1216" alt="2run" src="https://user-images.githubusercontent.com/1991664/111158281-22c82f80-8598-11eb-8e2a-ec3b070163f0.png">


### What did you expect to happen:
Pipeline run completes without errors

### Environment:
AI Platform Pipelines

How did you deploy Kubeflow Pipelines (KFP)?
AI Platform Pipelines

KFP version: https://github.com/kubeflow/pipelines/commit/d79071c0bef19442483abc101769a0d893e72f42

KFP SDK version: no pip in AI Platform Pipelines


### Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
<!--
// /area frontend
// /area backend
// /area sdk
// /area testing
// /area engprod
-->



Discussion:
Hi @SaschaHeyer, thank you for the report!

What is your TFX version? KFP cache shouldn't run on TFX pipelines. This is a bug.
Hi @Bobgy 
issue can be reproduced with 0.27.0 and 0.28.0
@SaschaHeyer can you provide a minimum reproducible TFX pipeline with a requirements.txt for all dependencies versions?
I tried TFX 0.28.0 with KFP 1.5.0-rc.0 (which should be very similar to 1.4.1 you tested on), and cannot reproduce this problem.
Hi @Bobgy 
here is the example you can use https://github.com/SaschaHeyer/Sentiment-Analysis-TFX
Thank you! I will try it out today
Hi @SaschaHeyer, sorry I do not know how to compile and run your pipeline.
Can you document that?
An important thing to verify is whether the following annotation exists on your pods:
![image](https://user-images.githubusercontent.com/4957653/115177640-379d6300-a102-11eb-8d61-b756b431b098.png)

`pipelines.kubeflow.org/pipeline-sdk-type: tfx`
@Bobgy I have the same issue, and I have the annotation `pipelines.kubeflow.org/pipeline-sdk-type: tfx` on my pods.
Could you explain how this annotation affect if it exists?
@judyliou sorry for the late reply, the annotation is used to distinguish tfx pipelines from KFP pipelines.
For TFX pipelines, they should not be cached by KFP cache controller, because TFX come with their first party caching feature. This bug is caused by KFP caching TFX pipelines.

Can you confirm your KFP backend is upgraded to 1.5.0+? The fix was released in 1.5.0, see https://github.com/kubeflow/pipelines/commit/76cad52d3b405f44b987e9e3a9d27a40d2b0c0d5.
Ohh, so that explains this original issue, KFP backend is 1.4.1 and the fix was only released in 1.5.0
Google Marketplace does not offer 1.5.0 yet, only 1.4.1. Is there a workaround to bring back visualisations (as described in [this ticket](https://github.com/kubeflow/pipelines/issues/5302), which leads here)?
@axeltidemann one workaround is to upgrade cache server after install 

```
# 1. connect to cluster kubectl

kubectl edit deployment cache-server -n namespace

# edit the image to gcr.io/ml-pipeline/cache-server:1.5.0
```
@Bobgy That would not work. Google Marketplace does not have any container images with a version 1.5.0 tag, you can check that [here](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google-cloud-ai-platform/kubeflow-pipelines/cacheserver?gcrImageListsize=30). 
@aneeshc-c you are right, I fixed the instruction to use oss image repo instead
That did not bring back visualisations, @Bobgy . Any other suggestions?
@axeltidemann can you report your ML Metadata tab screenshot after the fix? (And it should be a new run too)
Sure, @Bobgy. You can tell that the Outputs are wrong, it should be Type **ExampleStatistics**.
![Screenshot 2021-06-01 at 08 58 45](https://user-images.githubusercontent.com/6352118/120280202-aea84880-c2b7-11eb-8c46-e79eb9aa6ac5.png)

@axeltidemann thanks for the extra info, that reminded me https://github.com/kubeflow/pipelines/pull/5364 fix is also required. (the metadata you showed are logged from KFP metadata-writer, but we should show metadata of native TFX pipelines).

Refer to https://www.kubeflow.org/docs/components/pipelines/installation/overview/#choosing-an-installation-option, I'd suggest deploying Kubeflow Pipelines standalone 1.6.0 to get all the fixes, before GCP AI Platform Pipelines has an update. Feature diff between KFP Standalone and GCP AI Platform Pipelines is very minimal, you only get extra features from KFP standalone, because of the flexibility.
Thank you for that suggestion, @Bobgy. We are a bit hesitant to not use the Google Marketplace one. But thanks for letting us know we should wait for 1.6.0.
I'm curious why, but I'm working on blockers to release the latest version. Rough ETA would be 2 weeks
@Bobgy roughly 2 weeks to get 1.6.0 on Google Marketplace? That is great!
@Bobgy We also have the standalone pipeline installation through Google Cloud Marketplace, now I'm wondering what's the proper upgrade strategy? We have had 3 clusters so far, `0.4`, `1.0` and `1.4`, and everytime `upgrade` for us has been:
* Go through the market place installation
* Create a new cluster and install the newer version 
* Try to deprecate the older one and migrate pipelines into the newer one

1. Is there anyway to `upgrade` instead of `fresh re-install` from the Marketplace package?
2. Considering there's no way to `upgrade` from Market place, do you recommend us follow the manual upgrade docs [here](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#upgrading-kubeflow-pipelines) for gcp market place installations or the same `install and migrate` strategy?

@AlirezaSadeghi did you read the docs: https://www.kubeflow.org/docs/distributions/gke/pipelines/upgrade/?
Regarding this issue, I also just learned that tfx templates may override to use `tfx-template` as label value: https://github.com/tensorflow/tfx/blob/985105d58d04292163fd1e25d51a319d32448b9a/tfx/experimental/templates/penguin/kubeflow_runner.py#L66.

So KFP should match loosely, any label value with the word tfx should be ignored by KFP cacher and metadata-writer.

Resolving patch:
diff --git a/backend/metadata_writer/src/metadata_writer.py b/backend/metadata_writer/src/metadata_writer.py
--- a/backend/metadata_writer/src/metadata_writer.py
+++ b/backend/metadata_writer/src/metadata_writer.py
@@ -114,8 +114,10 @@ def argo_artifact_to_uri(artifact: dict) -> str:
 
 
 def is_tfx_pod(pod) -> bool:
-    # Later versions of TFX pods do not match the pattern in command line, but now they have this label.
-    if pod.metadata.labels.get(KFP_SDK_TYPE_LABEL_KEY) == TFX_SDK_TYPE_VALUE:
+    # The label defaults to 'tfx', but is overridable.
+    # Official tfx templates override the value to 'tfx-template', so
+    # we loosely match the word 'tfx'.
+    if TFX_SDK_TYPE_VALUE in pod.metadata.labels.get(KFP_SDK_TYPE_LABEL_KEY, ''):
         return True
     main_containers = [container for container in pod.spec.containers if container.name == 'main']
     if len(main_containers) != 1:


Test patch:
diff --git a/backend/src/cache/server/mutation_test.go b/backend/src/cache/server/mutation_test.go
--- a/backend/src/cache/server/mutation_test.go
+++ b/backend/src/cache/server/mutation_test.go
@@ -139,6 +139,12 @@ func TestMutatePodIfCachedWithTFXPod2(t *testing.T) {
 	patchOperation, err := MutatePodIfCached(GetFakeRequestFromPod(&tfxPod), fakeClientManager)
 	assert.Nil(t, patchOperation)
 	assert.Nil(t, err)
+	// test variation 2
+	tfxPod = *fakePod.DeepCopy()
+	tfxPod.Labels["pipelines.kubeflow.org/pipeline-sdk-type"] = "tfx-template"
+	patchOperation, err = MutatePodIfCached(GetFakeRequestFromPod(&tfxPod), fakeClientManager)
+	assert.Nil(t, patchOperation)
+	assert.Nil(t, err)
 }
 
 func TestMutatePodIfCachedWithKfpV2Pod(t *testing.T) {