Repository: kubeflow/pipelines

Issue:
Can't loop over output from lightweight function
**What happened:**

When calling ParallelFor on the output of a lightweight python component, I get an error saying that the task is not defined.

```py
@dsl.pipeline(
    name="Parallel Pipeline",
)
def parallel_pipeline(
        project_id,
        region,
        pipeline_root,
        job_config=''
):
    get_areas_op = func_to_container_op(get_active_areas)
    test_op = func_to_container_op(test)

    areas_task = get_areas_op()

    with dsl.ParallelFor(areas_task.output) as item:
        test_op(item)
```

Pipeline returns the error

```
invalid spec: templates.parallel-pipeline.tasks.for-loop-for-loop-c9def31b-1.dependencies[1] dependency 'Get active areas' not defined
```

This occurs when triggering a run through `create_run_from_pipeline_func`. If the pipeline is compiled and uploaded using the browser, attempting to view the pipeline leads to a blank screen with the following errors in the console.

```
react-dom.production.min.js:179 TypeError: Cannot read property 'y' of undefined
    at Graph.tsx:161
    at Array.forEach (<anonymous>)
    at t.render (Graph.tsx:136)
    at so (react-dom.production.min.js:163)
    at ao (react-dom.production.min.js:162)
    at po (react-dom.production.min.js:169)
    at Ko (react-dom.production.min.js:204)
    at Xo (react-dom.production.min.js:205)
    at Da (react-dom.production.min.js:220)
    at Ma (react-dom.production.min.js:219)
bo @ react-dom.production.min.js:179
To.n.callback @ react-dom.production.min.js:190
ci @ react-dom.production.min.js:128
ui @ react-dom.production.min.js:128
Pa @ react-dom.production.min.js:229
Da @ react-dom.production.min.js:220
Ma @ react-dom.production.min.js:219
Sa @ react-dom.production.min.js:216
ea @ react-dom.production.min.js:214
enqueueSetState @ react-dom.production.min.js:134
x.setState @ react.production.min.js:13
t.setStateSafe @ Page.tsx:80
(anonymous) @ PipelineDetails.tsx:341
(anonymous) @ PipelineDetails.tsx:1
(anonymous) @ PipelineDetails.tsx:1
a @ PipelineDetails.tsx:1
Promise.then (async)
l @ PipelineDetails.tsx:1
a @ PipelineDetails.tsx:1
Promise.then (async)
l @ PipelineDetails.tsx:1
(anonymous) @ PipelineDetails.tsx:1
F @ PipelineDetails.tsx:1
t.load @ PipelineDetails.tsx:243
(anonymous) @ PipelineDetails.tsx:240
(anonymous) @ PipelineDetails.tsx:1
(anonymous) @ PipelineDetails.tsx:1
(anonymous) @ PipelineDetails.tsx:1
F @ PipelineDetails.tsx:1
t.componentDidMount @ PipelineDetails.tsx:239
Pa @ react-dom.production.min.js:229
Da @ react-dom.production.min.js:220
Ma @ react-dom.production.min.js:219
La @ react-dom.production.min.js:231
Tn @ react-dom.production.min.js:85
Graph.tsx:161 Uncaught (in promise) TypeError: Cannot read property 'y' of undefined
    at Graph.tsx:161
    at Array.forEach (<anonymous>)
    at t.render (Graph.tsx:136)
    at so (react-dom.production.min.js:163)
    at ao (react-dom.production.min.js:162)
    at po (react-dom.production.min.js:169)
    at Ko (react-dom.production.min.js:204)
    at Xo (react-dom.production.min.js:205)
    at Da (react-dom.production.min.js:220)
    at Ma (react-dom.production.min.js:219)
```

**What did you expect to happen:**

ParallelFor to run a task for each entry in the list returned by get_active_areas

**Anything else you would like to add:**

Pipeline compiled with 0.1.36, run on kubeflow 0.7.0




Discussion:
Another error occurs if the python function compiled with `func_to_container_op` and passed into the ParallelFor has a single word name. e.g. `def areas() -> list:`

```python
...
  File "/Users/colincampbell/.virtualenvs/venvs/recommendations/lib/python3.7/site-packages/kfp/_client.py", line 342, in create_run_from_pipeline_func
    compiler.Compiler().compile(pipeline_func, pipeline_package_path, pipeline_conf=pipeline_conf)
  File "/Users/colincampbell/.virtualenvs/venvs/recommendations/lib/python3.7/site-packages/kfp/compiler/compiler.py", line 846, in compile
    package_path=package_path)
  File "/Users/colincampbell/.virtualenvs/venvs/recommendations/lib/python3.7/site-packages/kfp/compiler/compiler.py", line 908, in _create_and_write_workflow
    pipeline_conf)
  File "/Users/colincampbell/.virtualenvs/venvs/recommendations/lib/python3.7/site-packages/kfp/compiler/compiler.py", line 790, in _create_workflow
    workflow = fix_big_data_passing(workflow)
  File "/Users/colincampbell/.virtualenvs/venvs/recommendations/lib/python3.7/site-packages/kfp/compiler/_data_passing_rewriter.py", line 155, in fix_big_data_passing
    upstream_template_name = task_name_to_template_name[upstream_task_name]
KeyError: 'Active'
```
There seems to be two bugs:
1) PipelineParam.op_name is not getting sanitized for when added as dependency by `ParallelFor`. @numerology 
2) UX handling of invalid pipelines. @Bobgy 

This also highlights the lack of validation in Backend's create_run and upload_pipeline methods. @IronPan @jingzhang36 @rmgogogo 
@ColinCampbell-GBI do you mind sharing your `get_active_areas` and `test` functions so that we can easily verify the fix? Thanks!
I can reproduce the issue:

```python
import kfp

produce_op = kfp.components.load_component_from_text('''\
name: Produce list
outputs:
- name: data_list
implementation:
  container:
    image: busybox
    command:
    - sh
    - -c
    - echo "[1, 2, 3]" > "$0"
    - outputPath: data_list
''')

consume_op = kfp.components.load_component_from_text('''\
name: Consume data
inputs:
- name: data
implementation:
  container:
    image: busybox
    command:
    - echo
    - inputValue: data
''')

def pip1():
    source_task = produce_op()
    with kfp.dsl.ParallelFor(source_task.output) as item:
        consume_op(item)

kfp.compiler.Compiler()._compile(pip1)
```

```
...
   {'dag': {'tasks': [{'arguments': {'parameters': [{'name': 'produce-list-data_list',
          'value': '{{item}}'}]},
       'dependencies': ['produce-list', 'Produce list'],
       'name': 'for-loop-for-loop-75795c46-1',
       'template': 'for-loop-for-loop-75795c46-1',
       'withParam': '{{tasks.Produce list.outputs.parameters.Produce list-data_list}}'},
      {'name': 'produce-list', 'template': 'produce-list'}]},
...
```

As you see, the task name is not sanitized here (has spaces).
Thanks Alexey! I will fix UI handling
> @ColinCampbell-GBI do you mind sharing your `get_active_areas` and `test` functions so that we can easily verify the fix? Thanks!

```
def get_active_areas() -> list:
    return [
        {"a": 1},
        {"b": 2},
    ]


def test(areas: dict) -> str:
    print(areas)
    return "test"
```

It seems like we need to sanitize the output names eagerly during compilation. Will put together a fix shortly.

Resolving patch:
diff --git a/sdk/python/kfp/compiler/compiler.py b/sdk/python/kfp/compiler/compiler.py
--- a/sdk/python/kfp/compiler/compiler.py
+++ b/sdk/python/kfp/compiler/compiler.py
@@ -477,14 +477,19 @@ def _group_to_dag_template(self, group, inputs, outputs, dependencies):
           if pipeline_param.op_name is None:
             withparam_value = '{{workflow.parameters.%s}}' % pipeline_param.name
           else:
-            param_name = '%s-%s' % (pipeline_param.op_name, pipeline_param.name)
-            withparam_value = '{{tasks.%s.outputs.parameters.%s}}' % (pipeline_param.op_name, param_name)
+            param_name = '%s-%s' % (
+              sanitize_k8s_name(pipeline_param.op_name), pipeline_param.name)
+            withparam_value = '{{tasks.%s.outputs.parameters.%s}}' % (
+                sanitize_k8s_name(pipeline_param.op_name),
+                param_name)
 
             # these loop args are the output of another task
             if 'dependencies' not in task or task['dependencies'] is None:
               task['dependencies'] = []
-            if pipeline_param.op_name not in task['dependencies']:
-              task['dependencies'].append(pipeline_param.op_name)
+            if sanitize_k8s_name(
+                pipeline_param.op_name) not in task['dependencies']:
+              task['dependencies'].append(
+                  sanitize_k8s_name(pipeline_param.op_name))
 
           task['withParam'] = withparam_value
         else:


Test patch:
diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
--- a/sdk/python/tests/compiler/compiler_tests.py
+++ b/sdk/python/tests/compiler/compiler_tests.py
@@ -780,6 +780,9 @@ def test_withparam_output(self):
   def test_withparam_output_dict(self):
     self._test_py_compile_yaml('withparam_output_dict')
 
+  def test_withparam_lightweight_out(self):
+    self._test_py_compile_yaml('loop_over_lightweight_output')
+
   def test_py_input_artifact_raw_value(self):
     """Test pipeline input_artifact_raw_value."""
     self._test_py_compile_yaml('input_artifact_raw_value')
diff --git a/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.py b/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.py
new file mode 100644
--- /dev/null
+++ b/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.py
@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import kfp
+from kfp import dsl
+from kfp.dsl import _for_loop
+
+class Coder:
+  def __init__(self, ):
+    self._code_id = 0
+
+  def get_code(self, ):
+    self._code_id += 1
+    return '{code:0{num_chars:}d}'.format(code=self._code_id, num_chars=_for_loop.LoopArguments.NUM_CODE_CHARS)
+
+
+dsl.ParallelFor._get_unique_id_code = Coder().get_code
+
+produce_op = kfp.components.load_component_from_text('''\
+name: Produce list
+outputs:
+- name: data_list
+implementation:
+  container:
+    image: busybox
+    command:
+    - sh
+    - -c
+    - echo "[1, 2, 3]" > "$0"
+    - outputPath: data_list
+''')
+
+consume_op = kfp.components.load_component_from_text('''\
+name: Consume data
+inputs:
+- name: data
+implementation:
+  container:
+    image: busybox
+    command:
+    - echo
+    - inputValue: data
+''')
+
+@dsl.pipeline(
+    name='Loop over lightweight output',
+    description='Test pipeline to verify functions of par loop.'
+)
+def pipeline():
+  source_task = produce_op()
+  with dsl.ParallelFor(source_task.output) as item:
+    consume_op(item)
\ No newline at end of file
diff --git a/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.yaml b/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.yaml
new file mode 100644
--- /dev/null
+++ b/sdk/python/tests/compiler/testdata/loop_over_lightweight_output.yaml
@@ -0,0 +1,123 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"apiVersion": |-
+  argoproj.io/v1alpha1
+"kind": |-
+  Workflow
+"metadata":
+  "annotations":
+    "pipelines.kubeflow.org/pipeline_spec": |-
+      {"description": "Test pipeline to verify functions of par loop.", "name": "Loop over lightweight output"}
+  "generateName": |-
+    loop-over-lightweight-output-
+"spec":
+  "arguments":
+    "parameters": []
+  "entrypoint": |-
+    loop-over-lightweight-output
+  "serviceAccountName": |-
+    pipeline-runner
+  "templates":
+    - "container":
+        "args": []
+        "command":
+          - |-
+            echo
+          - |-
+            {{inputs.parameters.produce-list-data_list}}
+        "image": |-
+          busybox
+      "inputs":
+        "parameters":
+          - "name": |-
+              produce-list-data_list
+      "metadata":
+        "annotations":
+          "pipelines.kubeflow.org/component_spec": |-
+            {"inputs": [{"name": "data"}], "name": "Consume data"}
+      "name": |-
+        consume-data
+    - "dag":
+        "tasks":
+          - "arguments":
+              "parameters":
+                - "name": |-
+                    produce-list-data_list
+                  "value": |-
+                    {{inputs.parameters.produce-list-data_list}}
+            "name": |-
+              consume-data
+            "template": |-
+              consume-data
+      "inputs":
+        "parameters":
+          - "name": |-
+              produce-list-data_list
+      "name": |-
+        for-loop-for-loop-00000001-1
+    - "dag":
+        "tasks":
+          - "arguments":
+              "parameters":
+                - "name": |-
+                    produce-list-data_list
+                  "value": |-
+                    {{item}}
+            "dependencies":
+              - |-
+                produce-list
+            "name": |-
+              for-loop-for-loop-00000001-1
+            "template": |-
+              for-loop-for-loop-00000001-1
+            "withParam": |-
+              {{tasks.produce-list.outputs.parameters.produce-list-data_list}}
+          - "name": |-
+              produce-list
+            "template": |-
+              produce-list
+      "name": |-
+        loop-over-lightweight-output
+    - "container":
+        "args": []
+        "command":
+          - |-
+            sh
+          - |-
+            -c
+          - |-
+            echo "[1, 2, 3]" > "$0"
+          - |-
+            /tmp/outputs/data_list/data
+        "image": |-
+          busybox
+      "metadata":
+        "annotations":
+          "pipelines.kubeflow.org/component_spec": |-
+            {"name": "Produce list", "outputs": [{"name": "data_list"}]}
+      "name": |-
+        produce-list
+      "outputs":
+        "artifacts":
+          - "name": |-
+              produce-list-data_list
+            "path": |-
+              /tmp/outputs/data_list/data
+        "parameters":
+          - "name": |-
+              produce-list-data_list
+            "valueFrom":
+              "path": |-
+                /tmp/outputs/data_list/data