Repository: kubeflow/pipelines

Issue:
[v2compat] wrong param passed when param contains JSON
### Environment

*  KFP SDK version: master branch

### Steps to reproduce

```python
import kfp
from kfp import dsl
from kfp.components import create_component_from_func


@create_component_from_func
def test(json_param: str):
    print(json_param)


@dsl.pipeline(name='bug-json-param')
def my_pipeline(
    json_param: str = '{"a":"b"}',
    value: str = 'c',
):
    test(json_param).set_display_name('test1')
    test('{"a":"' + f'{value}' + '"}').set_display_name('test2')


if __name__ == '__main__':
    kfp.Client().create_run_from_pipeline_func(
        my_pipeline,
        arguments={},
        mode=dsl.PipelineExecutionMode.V2_COMPATIBLE,
    )
```

I got the following log for `test2` task (`task1` is expected).
```
{\"a\":\"c\"}
```

### Expected result

<!-- What should the correct behavior be? -->
```
{"a":"c"}
```

### Materials and Reference

<!-- Help us debug this issue by providing resources such as: sample code, background context, or links to references. -->

---

<!-- Don't delete message below to encourage users to support your issue! -->
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.



Discussion:
/assign @chensun 
this is obviously a corner case we didn't think of for the "BEGIN-KFP-PARAM[...]END-KFP-PARAM" hack
There is a corner case where we pass empty string to a component function: https://github.com/kubeflow/pipelines/pull/5938#discussion_r660498207.

In the example above, I passed `empty string` to `gcsPath` parameter, but I got `BEGIN-KFP-PARAM[]END-KFP-PARAM` within the component function. This behavior is unexpected to user, so recording this here.
Currently FE implements a hack to workaround `MLPipeline UI Metadata` name conversion, we need to make sure to update frontend to remove the hack, once SDK has made corresponding change:

https://github.com/kubeflow/pipelines/pull/5961#discussion_r662785344
> Currently FE implements a hack to workaround `MLPipeline UI Metadata` name conversion, we need to make sure to update frontend to remove the hack, once SDK has made corresponding change:
> 
> [#5961 (comment)](https://github.com/kubeflow/pipelines/pull/5961#discussion_r662785344)

@zijianjoy we might consider keeping the sanitization in UI permanently, because artifact naming convention changed when going to KFP v2. In v2, we will no longer sanitize artifact names automatically in SDK.
After some investigation, I think I understood the root cause.

1. Compiler `json.dumps` when encoding runtime info
https://github.com/kubeflow/pipelines/blob/07ac566fcbaf12387e5e2bef89a029e5ef183ee6/sdk/python/kfp/compiler/v2_compat.py#L139-L141
   At this point, constant text will be encoded once, but text passed from parameters are written as "{{inputs.parameters.<name>}}", therefore they won't be JSON encoded. The difference is the root cause of this bug.
2. Launcher tries to JSON encode for the second time, because string value passed from parameters are not JSON encoded: https://github.com/kubeflow/pipelines/blob/bd6350dc2730514b53733be5890426b315d0eda9/v2/component/runtime_info.go#L83-L85
3. Launcher tries to JSON decode, but because of the difference in 1, some part of the parameters are JSON encoded for 1 extra time.
Therefore, I can think of two ways to fix:
1. BEGIN-KFP-PARAM token should only be injected wrapping parameter placeholders, but I'm not sure whether further corner cases may exist.
2. (preferred) Design a new way to pass raw data, so that we do not need to play around with JSON encoding.

I'd like to choose option 2, because I prefer a solid solution.
## Proposal

op.command + op.args will have the following sections:
1. args passed to kfp-launcher
2. a separator "--"
3. input parameters in format "key1=value1", "key2=value2", ... (we need to assume keys cannot contain "=" character, is it a fair assumption?)
4. a separator "--" as end of arguments passed to launcher
5. (start of op.args) arguments of the original user program command + args

this will make it possible to pass parameter values without JSON encoding

Resolving patch:
diff --git a/sdk/python/kfp/compiler/v2_compat.py b/sdk/python/kfp/compiler/v2_compat.py
--- a/sdk/python/kfp/compiler/v2_compat.py
+++ b/sdk/python/kfp/compiler/v2_compat.py
@@ -52,6 +52,40 @@ def update_op(op: dsl.ContainerOp,
   op.add_volume_mount(
       k8s_client.V1VolumeMount(name='kfp-launcher', mount_path='/kfp-launcher'))
 
+  # op.command + op.args will have the following sections:
+  # 1. args passed to kfp-launcher
+  # 2. a separator "--"
+  # 3. parameters in format "key1=value1", "key2=value2", ...
+  # 4. a separator "--" as end of arguments passed to launcher
+  # 5. (start of op.args) arguments of the original user program command + args
+  #
+  # example:
+  # - command:
+  # - /kfp-launcher/launch
+  # - '--mlmd_server_address'
+  # - $(METADATA_GRPC_SERVICE_HOST)
+  # - '--mlmd_server_port'
+  # - $(METADATA_GRPC_SERVICE_PORT)
+  # - ... # more launcher params
+  # - '--pipeline_task_id'
+  # - $(KFP_POD_NAME)
+  # - '--pipeline_root'
+  # - ''
+  # - '--' # start of parameter values
+  # - first=first
+  # - second=second
+  # - '--' # start of user command and args
+  # args:
+  # - sh
+  # - '-ec'
+  # - |
+  #     program_path=$(mktemp)
+  #     printf "%s" "$0" > "$program_path"
+  #     python3 -u "$program_path" "$@"
+  # - >
+  #     import json
+  #     import xxx
+  #     ...
   op.command = [
       "/kfp-launcher/launch",
       "--mlmd_server_address",
@@ -93,17 +127,17 @@ def update_op(op: dsl.ContainerOp,
       "outputArtifacts": collections.OrderedDict(),
   }
 
+  op.command += ["--"]
   component_spec = op.component_spec
   for parameter, spec in sorted(
       component_spec.input_definitions.parameters.items()):
     parameter_info = {
         "type":
             pipeline_spec_pb2.PrimitiveType.PrimitiveTypeEnum.Name(spec.type),
-        "value":
-            "BEGIN-KFP-PARAM[{}]END-KFP-PARAM".format(
-                op._parameter_arguments[parameter])
     }
+    op.command += [f"{parameter}={op._parameter_arguments[parameter]}"]
     runtime_info["inputParameters"][parameter] = parameter_info
+  op.command += ["--"]
 
   for artifact_name, spec in sorted(
       component_spec.input_definitions.artifacts.items()):


Test patch:
diff --git a/samples/core/visualization/tensorboard_minio_test.py b/samples/core/visualization/tensorboard_minio_test.py
--- a/samples/core/visualization/tensorboard_minio_test.py
+++ b/samples/core/visualization/tensorboard_minio_test.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import unittest
+import json
 from pprint import pprint
 import kfp
 import kfp_server_api
@@ -43,12 +44,52 @@ def verify(
     storage_client = storage.Client()
     blob = storage.Blob.from_string(mlpipeline_ui_metadata.uri, storage_client)
     data = blob.download_as_text(storage_client)
-    print('=== artifact content begin ===')
-    print(data)
-    print('=== artifact content end ===')
-    # TODO(#5830): fix the JSON encoding issue, then update the following to assert parsed JSON data.
-    # https://github.com/kubeflow/pipelines/issues/5830
-    t.assertTrue('"type": "tensorboard"' in data)
+    data = json.loads(data)
+    t.assertTrue(data["outputs"][0]["source"])
+    # source is a URI that is generated differently each run
+    data["outputs"][0]["source"] = "<redacted>"
+    t.assertEqual({
+        "outputs": [{
+            "type": "tensorboard",
+            "source": "<redacted>",
+            "image": "gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest",
+            "pod_template_spec": {
+                "spec": {
+                    "containers": [{
+                        "env": [{
+                            "name": "AWS_ACCESS_KEY_ID",
+                            "valueFrom": {
+                                "secretKeyRef": {
+                                    "name": "mlpipeline-minio-artifact",
+                                    "key": "accesskey"
+                                }
+                            }
+                        }, {
+                            "name": "AWS_SECRET_ACCESS_KEY",
+                            "valueFrom": {
+                                "secretKeyRef": {
+                                    "name": "mlpipeline-minio-artifact",
+                                    "key": "secretkey"
+                                }
+                            }
+                        }, {
+                            "name": "AWS_REGION",
+                            "value": "minio"
+                        }, {
+                            "name": "S3_ENDPOINT",
+                            "value": "minio-service:9000"
+                        }, {
+                            "name": "S3_USE_HTTPS",
+                            "value": "0"
+                        }, {
+                            "name": "S3_VERIFY_SSL",
+                            "value": "0"
+                        }]
+                    }]
+                }
+            }
+        }]
+    }, data)
 
 
 run_pipeline_func([
diff --git a/samples/test/config.yaml b/samples/test/config.yaml
--- a/samples/test/config.yaml
+++ b/samples/test/config.yaml
@@ -76,7 +76,6 @@
   path: samples.test.placeholder_concat_test
 - name: metrics_visualization_v2
   path: samples.test.metrics_visualization_v2_test
-
 # TODO(Bobgy): enable this test when https://github.com/kubeflow/pipelines/pull/5424
 # is deployed to test infra.
 # - name: fail_parameter_value_missing
diff --git a/samples/test/lightweight_python_functions_v2_pipeline.py b/samples/test/lightweight_python_functions_v2_pipeline.py
--- a/samples/test/lightweight_python_functions_v2_pipeline.py
+++ b/samples/test/lightweight_python_functions_v2_pipeline.py
@@ -39,6 +39,8 @@ def preprocess(
     output_dict_parameter_path: OutputPath(Dict[str, int]),
     # A locally accessible filepath for an output parameter of type list.
     output_list_parameter_path: OutputPath(List[str]),
+    # An input message that defaults to empty.
+    empty_message: str = "",
 ):
   """Dummy preprocessing step"""
 
diff --git a/samples/test/lightweight_python_functions_v2_pipeline_test.py b/samples/test/lightweight_python_functions_v2_pipeline_test.py
--- a/samples/test/lightweight_python_functions_v2_pipeline_test.py
+++ b/samples/test/lightweight_python_functions_v2_pipeline_test.py
@@ -43,7 +43,8 @@ def verify(
             'inputs': {
                 'artifacts': [],
                 'parameters': {
-                    'message': 'message'
+                    'message': 'message',
+                    'empty_message': '',
                 }
             },
             'name': 'preprocess',
diff --git a/samples/test/util.py b/samples/test/util.py
--- a/samples/test/util.py
+++ b/samples/test/util.py
@@ -414,11 +414,11 @@ def _parse_parameters(execution: metadata_store_pb2.Execution) -> dict:
     for item in custom_properties.items():
         (name, value) = item
         raw_value = None
-        if value.string_value:
+        if value.HasField('string_value'):
             raw_value = value.string_value
-        if value.int_value:
+        if value.HasField('int_value'):
             raw_value = value.int_value
-        if value.double_value:
+        if value.HasField('double_value'):
             raw_value = value.double_value
         if name.startswith('input:'):
             parameters['inputs'][name[len('input:'):]] = raw_value
diff --git a/v2/component/runtime_info_test.go b/v2/component/runtime_info_test.go
--- a/v2/component/runtime_info_test.go
+++ b/v2/component/runtime_info_test.go
@@ -40,16 +40,14 @@ func Test_parseRuntimeInfo(t *testing.T) {
 			jsonEncoded: `{
 				"inputParameters": {
 					"my_param": {
-						"type": "INT",
-						"value": "BEGIN-KFP-PARAM[123]END-KFP-PARAM"
+						"type": "INT"
 					}
 				}
 			}`,
 			want: &runtimeInfo{
 				InputParameters: map[string]*inputParameter{
 					"my_param": {
-						Type:  "INT",
-						Value: "123",
+						Type: "INT",
 					},
 				},
 			},
@@ -60,16 +58,14 @@ func Test_parseRuntimeInfo(t *testing.T) {
 			jsonEncoded: `{
 				"inputParameters": {
 					"my_param": {
-						"type": "STRING",
-						"value": "BEGIN-KFP-PARAM[some random "value" 'with' "quotes"]END-KFP-PARAM"
+						"type": "STRING"
 					}
 				}
 			}`,
 			want: &runtimeInfo{
 				InputParameters: map[string]*inputParameter{
 					"my_param": {
-						Type:  "STRING",
-						Value: "some random \"value\" 'with' \"quotes\"",
+						Type: "STRING",
 					},
 				},
 			},
@@ -80,16 +76,14 @@ func Test_parseRuntimeInfo(t *testing.T) {
 			jsonEncoded: `{
 				"inputParameters": {
 					"my_param": {
-						"type": "STRING",
-						"value": "BEGIN-KFP-PARAM[{"A": "a-value", "B": 123}]END-KFP-PARAM"
+						"type": "STRING"
 					}
 				}
 			}`,
 			want: &runtimeInfo{
 				InputParameters: map[string]*inputParameter{
 					"my_param": {
-						Type:  "STRING",
-						Value: "{\"A\": \"a-value\", \"B\": 123}",
+						Type: "STRING",
 					},
 				},
 			},
@@ -180,20 +174,27 @@ func TestExecutorInputGeneration(t *testing.T) {
 	tests := []struct {
 		name        string
 		jsonEncoded string
+		args        []string
 		want        *pipeline_spec.ExecutorInput
 		wantErr     bool
 	}{
 		{
 			name: "Parses InputParameters Correctly",
+			args: []string{
+				"message=Some string value with { \"special\": \"chars\" }",
+				"num_steps=5",
+				"--",
+				"sh",
+				"-c",
+				"user cmd args",
+			},
 			jsonEncoded: fmt.Sprintf(`{
 				"inputParameters": {
 					"message": {
-						"type": "STRING",
-						"value": "Some string value"
+						"type": "STRING"
 					},
 					"num_steps": {
-						"type": "INT",
-						"value": "5"
+						"type": "INT"
 					}
 				},
 				"inputArtifacts": {
@@ -234,7 +235,7 @@ func TestExecutorInputGeneration(t *testing.T) {
 			want: &pipeline_spec.ExecutorInput{
 				Inputs: &pipeline_spec.ExecutorInput_Inputs{
 					Parameters: map[string]*pipeline_spec.Value{
-						"message":   {Value: &pipeline_spec.Value_StringValue{StringValue: "Some string value"}},
+						"message":   {Value: &pipeline_spec.Value_StringValue{StringValue: "Some string value with { \"special\": \"chars\" }"}},
 						"num_steps": {Value: &pipeline_spec.Value_IntValue{IntValue: 5}},
 					},
 					Artifacts: map[string]*pipeline_spec.ArtifactList{
@@ -299,6 +300,10 @@ func TestExecutorInputGeneration(t *testing.T) {
 				t.Errorf("parseRuntimeInfo() error = %v", err)
 				return
 			}
+			_, err = parseArgs(test.args, rti)
+			if (err != nil) != test.wantErr {
+				t.Errorf("parseArgs() error = %v", err)
+			}
 
 			got, err := rti.generateExecutorInput(generateOutputUri, outputMetadataFilepath)
 			if (err != nil) != test.wantErr {