Repository: kubeflow/pipelines

Issue:
[testing] postsubmit test failure 2021-08-12
https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-standalone-component-test/1425587097510612992#1:build-log.txt%3A5394

For standalone components test, it fails when building sample test image:


> sample-test-rb7fv-2769807467: error: google-api-core 1.26.0 is installed but google-api-core<3.0dev,>=1.29.0 is required by {'google-cloud-storage'}
sample-test-rb7fv-2769807467: The command '/bin/sh -c cd /sdk/python && python3 setup.py install' returned a non-zero code: 1



Discussion:
before this error shows up, there was a different error that we should also fix:
https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-integration-test/1425263307043901440#1:build-log.txt%3A9072

The error seems like a bug in argo, docker executor also do not support optional output artifacts:
> executor error: path /tmp/outputs/MLPipeline_UI_metadata/data does not exist in archive /tmp/argo/outputs/artifacts/launch-python-MLPipeline-UI-metadata.tgz\ngithub.com/argoproj/argo-workflows/v3/errors.New\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:49\ngithub.com/argoproj/argo-workflows/v3/errors.Errorf\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:55\

EDIT: I'm not sure now, because I found a passing postsubmit test after 1.7.0-rc.3 release, that proves current argo version can successfully run postsubmit tests.
![image](https://user-images.githubusercontent.com/4957653/129191372-bedf2542-f0ed-4eb7-8942-276828e5e518.png)

EDIT2: I think I'm getting closer to the root cause, here's argo outputs annotation for the failed step in dataflow pipeline sample:

> workflows.argoproj.io/outputs: '{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/tmp/outputs/MLPipeline_UI_metadata/data","optional":true},{"name":"launch-python-MLPipeline-UI-metadata","path":"/tmp/outputs/MLPipeline_UI_metadata/data"},{"name":"launch-python-job_id","path":"/tmp/outputs/job_id/data"},{"name":"main-logs","s3":{"key":"artifacts/dataflow-launch-python-pipeline-8hdph/2021/08/11/dataflow-launch-python-pipeline-8hdph-2670089030/main.log"}}]}'

Note that "launch-python-MLPipeline-UI-metadata" artifact should have "optional": "true", but it didn't. That caused the task to fail, so I guess this is a side effect bug caused by one of the recent KFP v1 compiler changes.

reference: pod info for the failed pod in dataflow sample pipeline: https://storage.googleapis.com/oss-prow/logs/kubeflow-pipeline-postsubmit-integration-test/1425515476460507136/artifacts/pods_info/dataflow-launch-python-pipeline-8hdph-2670089030.txt
Thanks @Bobgy. Can you point me to the pipeline sample that fails? 
/assign @neuromage 

Resolving patch:
diff --git a/sdk/python/kfp/dsl/_container_op.py b/sdk/python/kfp/dsl/_container_op.py
--- a/sdk/python/kfp/dsl/_container_op.py
+++ b/sdk/python/kfp/dsl/_container_op.py
@@ -14,7 +14,7 @@
 
 import re
 import warnings
-from typing import Any, Dict, Iterable, List, TypeVar, Union, Callable, Optional, Sequence
+from typing import Any, Callable, Dict, List, Optional, Sequence, TypeVar, Tuple, Union
 
 from kubernetes.client import V1Toleration, V1Affinity
 from kubernetes.client.models import (V1Container, V1EnvVar, V1EnvFromSource,
@@ -1062,6 +1062,15 @@ def __init__(self, argument, input=None, path=None):
     self.path = path
 
 
+def _is_legacy_output_name(output_name: str) -> Tuple[bool, str]:
+    normalized_output_name = re.sub('[^a-zA-Z0-9]', '-', output_name.lower())
+    if normalized_output_name in [
+        'mlpipeline-ui-metadata',
+        'mlpipeline-metrics',
+    ]:
+        return True, normalized_output_name
+    return False, output_name
+
 class ContainerOp(BaseOp):
   """Represents an op implemented by a container image.
 
@@ -1257,11 +1266,9 @@ def _decorated(*args, **kwargs):
       # outputs that should always be saved as artifacts
       # TODO: Remove when outputs are always saved as artifacts
       for output_name, path in dict(file_outputs).items():
-        normalized_output_name = re.sub('[^a-zA-Z0-9]', '-', output_name.lower())
-        if normalized_output_name in [
-            'mlpipeline-ui-metadata', 'mlpipeline-metrics'
-        ]:
-          output_artifact_paths[normalized_output_name] = path
+        is_legacy_name, normalized_name = _is_legacy_output_name(output_name)
+        if is_legacy_name:
+          output_artifact_paths[normalized_name] = path
           del file_outputs[output_name]
 
     # attributes specific to `ContainerOp`
@@ -1382,6 +1389,14 @@ def _set_metadata(self, metadata, arguments: Optional[Dict[str, Any]] = None):
           output_filename = _components._generate_output_file_name(output.name)
           self.file_outputs[output.name] = output_filename
 
+      if not kfp.COMPILING_FOR_V2:
+        for output_name, path in dict(self.file_outputs).items():
+          is_legacy_name, normalized_name = _is_legacy_output_name(output_name)
+          if is_legacy_name:
+            self.output_artifact_paths[normalized_name] = path
+            del self.file_outputs[output_name]
+            del self.outputs[output_name]
+
     if arguments is not None:
       for input_name, value in arguments.items():
         self.artifact_arguments[input_name] = str(value)


Test patch: