Repository: kubeflow/pipelines

Issue:
Cloudpickle fails to deserialize a function
Deserializing a python function during runtime fails in case the function has a default parameter, which uses an argo variable.

Here is a simple example:
```
def _test_func(
    path: str,
    local_path: str,
    remote_artifact_name: str,
    minio_bucket: str = 'mlpipeline',
    minio_key: str = 'runs/{{workflow.uid}}/{{pod.name}}',
    minio_ext: str = '.tgz',
):
    remote_path = f'minio://{minio_bucket}/{minio_key}/{remote_artifact_name}{minio_ext}'
    print(remote_path)
```

This can be executed fine when compiled with 0.1.19 (latest version in PyPI), but fails with the latest master (which uses cloudpickle). The error message is one of the following:
- in case only `{{workflow.uid}}` is used:
```
Traceback (most recent call last):
  File "<string>", line 9, in <module>
AttributeError: 'dict' object has no attribute 'append'
```
- in case `{{pod.name}}` is used:
```
Traceback (most recent call last):
  File "<string>", line 9, in <module>
_pickle.UnpicklingError: invalid load key, '5'.
```

Using a default value without brackets resolves the issue.


Discussion:
Unfortunately this is probably "Won't fix" for now.
Argo's templating language is text-based and it brings all sorts of problems in many scenarios. 
For example this command-line fails in Argo:
```
command: [docker inspect --format="{{index .RepoDigests 0}}" org/image]
```

You probably imaging what's happening in your case: Argo is replacing some text snipped inside binary data which breaks the binary format.

In your case the workaround is easy and better for future-proofing:
1. Don't use Argo templates inside the program code
2. Don't use Argo templates in `component.yaml`

You should only pass the templated strings as arguments when you call the component.

Argo templates are implementation detail. You should not expect them to always work.
Artifacts in Minio is an implementation detail. You should not expect them to always be there (e.g. in future we can decide to store artifacts in mounted volumes).

P.S. Why do you want to access the artifacts that would be written by your own task after it ends? They'll never be there until your code finishes...

Resolving patch:
diff --git a/sdk/python/kfp/components/_python_op.py b/sdk/python/kfp/components/_python_op.py
--- a/sdk/python/kfp/components/_python_op.py
+++ b/sdk/python/kfp/components/_python_op.py
@@ -46,6 +46,7 @@ def _python_function_name_to_component_name(name):
 
 
 def _capture_function_code_using_cloudpickle(func, modules_to_capture: List[str] = None) -> str:
+    import base64
     import sys
     import cloudpickle
     import pickle
@@ -59,10 +60,13 @@ def _capture_function_code_using_cloudpickle(func, modules_to_capture: List[str]
         for module_name in modules_to_capture:
             if module_name in sys.modules:
                 old_modules[module_name] = sys.modules.pop(module_name)
-        func_pickle = cloudpickle.dumps(func, pickle.DEFAULT_PROTOCOL)
+        func_pickle = base64.b64encode(cloudpickle.dumps(func, pickle.DEFAULT_PROTOCOL))
     finally:
         sys.modules.update(old_modules)
-    func_code = '{func_name} = pickle.loads({func_pickle})'.format(func_name=func.__name__, func_pickle=repr(func_pickle))
+    func_code = '{func_name} = pickle.loads(base64.b64decode({func_pickle}))'.format(
+        func_name=func.__name__,
+        func_pickle=repr(func_pickle)
+    )
 
     code_lines = [
         'try:',
@@ -72,6 +76,7 @@ def _capture_function_code_using_cloudpickle(func, modules_to_capture: List[str]
         '    import sys',
         '    subprocess.call([sys.executable, "-m", "pip", "install", "cloudpickle==1.1.1", "--quiet"])'
         '',
+        'import base64',
         'import pickle',
         '',
         func_code,
@@ -103,7 +108,7 @@ def _func_to_component_spec(func, extra_code='', base_image=_default_base_image,
     import inspect
     import re
     from collections import OrderedDict
-    
+
     single_output_name_const = 'Output'
     single_output_pythonic_name_const = 'output'
 
@@ -263,7 +268,7 @@ def add_multiply_two_numbers(a: float, b: float) -> NamedTuple('DummyName', [('s
                     Note: The image can also be specified by decorating the function with the @python_component decorator. If different base images are explicitly specified in both places, an error is raised.
         extra_code: Optional. Extra code to add before the function code. Can be used as workaround to define types used in function signature.
         modules_to_capture: Optional. List of module names that will be captured (instead of just referencing) during the dependency scan. By default the func.__module__ is captured. The actual algorithm: Starting with the initial function, start traversing dependencies. If the dependecy.__module__ is in the modules_to_capture list then it's captured and it's dependencies are traversed. Otherwise the dependency is only referenced instead of capturing and its dependencies are not traversed.
-    
+
     Returns:
         Textual representation of a component definition
     '''
@@ -294,7 +299,7 @@ def add_multiply_two_numbers(a: float, b: float) -> NamedTuple('DummyName', [('s
     '''
 
     component_yaml = func_to_component_text(func, extra_code, base_image, modules_to_capture)
-    
+
     Path(output_component_file).write_text(component_yaml)
 
 


Test patch: