Repository: kubeflow/pipelines

Issue:
ParallelFor parameter parsing does not retain user input format

When running a pipeline with a ParallelFor condition over a list of dictionaries, camelCase and snake_case inputs are altered.

This won't work:

    with kfp.dsl.ParallelFor([{"helloWorld": "blah"}]) as item:
        my_op(item.helloWorld)

    with kfp.dsl.ParallelFor([{"hello_world": "blah"}]) as item:
        my_op(item.hello_world)


If I am reading this correctly, then the `my_op` is parsing `item.hello_world` as `item.hello-world` (and `helloWorld` as `helloworld`. However, the ParallelFor operation isn't  performing the same parsing. (I presume it is delivering `item.hello_world`, exactly as written in the dictionary, but I cannot confirm this).

The following works correctly:

    with kfp.dsl.ParallelFor([{"helloworld": "blah"}]) as item:
        my_op(item.helloWorld)

    with kfp.dsl.ParallelFor([{"hello-world": "blah"}]) as item:
        my_op(item.hello_world)




Discussion:
Thanks for reporting. It seems like the sanitization of var name is not consistent b/w the declaration of the loop var and its usage. We'll fix it quickly.
/kind bug
/area sdk/dsl

Resolving patch:
diff --git a/sdk/python/kfp/compiler/compiler.py b/sdk/python/kfp/compiler/compiler.py
--- a/sdk/python/kfp/compiler/compiler.py
+++ b/sdk/python/kfp/compiler/compiler.py
@@ -487,7 +487,18 @@ def _group_to_dag_template(self, group, inputs, outputs, dependencies):
 
           task['withParam'] = withparam_value
         else:
-          task['withItems'] = sub_group.loop_args.to_list_for_task_yaml()
+          # Need to sanitize the dict keys for consistency.
+          loop_tasks = sub_group.loop_args.to_list_for_task_yaml()
+          sanitized_tasks = []
+          if isinstance(loop_tasks[0], dict):
+            for argument_set in loop_tasks:
+              c_dict = {}
+              for k, v in argument_set.items():
+                c_dict[sanitize_k8s_name(k)] = v
+              sanitized_tasks.append(c_dict)
+          else:
+            sanitized_tasks = loop_tasks
+          task['withItems'] = sanitized_tasks
 
       tasks.append(task)
     tasks.sort(key=lambda x: x['name'])
diff --git a/sdk/python/kfp/dsl/_for_loop.py b/sdk/python/kfp/dsl/_for_loop.py
--- a/sdk/python/kfp/dsl/_for_loop.py
+++ b/sdk/python/kfp/dsl/_for_loop.py
@@ -20,14 +20,14 @@ def _subvar_name_is_legal(cls, proposed_variable_name: Text):
         return re.match(cls.LEGAL_SUBVAR_NAME_REGEX, proposed_variable_name) is not None
 
     def __init__(self, items: Union[ItemList, dsl.PipelineParam], code: Text, name_override: Optional[Text]=None, op_name: Optional[Text]=None, *args, **kwargs):
-        """_LoopArguments represent the set of items to loop over in a ParallelFor loop.  This class shoudn't be
+        """LoopArguments represent the set of items to loop over in a ParallelFor loop.  This class shouldn't be
         instantiated by the user but rather is created by _ops_group.ParallelFor.
 
         Args:
             items: List of items to loop over.  If a list of dicts then, all dicts must have the same keys and every
                 key must be a legal Python variable name.
             code: A unique code used to identify these loop arguments.  Should match the code for the ParallelFor
-                ops_group which created these _LoopArguments.  This prevents parameter name collissions.
+                ops_group which created these _LoopArguments.  This prevents parameter name collisions.
         """
         if name_override is None:
             super().__init__(name=self._make_name(code), *args, **kwargs)


Test patch: