Repository: kubeflow/pipelines

Issue:
[bug] TFX sample fails after upgrading to 1.4
### What steps did you take

<!-- A clear and concise description of what the bug is.-->
* MLMD & TFX upgrade: https://github.com/kubeflow/pipelines/pull/6910


### What happened:

* Check test: https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-integration-test/1464225919869652992

### What did you expect to happen:

We need to upgrade the TFX sample to 1.4.0

### Anything else you would like to add:
<!-- Miscellaneous information that will assist in solving the issue.-->


### Labels
<!-- Please include labels below by uncommenting them to help us better triage issues -->

<!-- /area frontend -->
<!-- /area backend -->
<!-- /area sdk -->
<!-- /area testing -->
/area samples
<!-- /area components -->


---

<!-- Don't delete message below to encourage users to support your issue! -->
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.



Discussion:
/assign @zijianjoy @jiyongjung0 
@Bobgy: GitHub didn't allow me to assign the following users: jiyongjung0.

Note that only [kubeflow members](https://github.com/orgs/kubeflow/people), repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see [the contributor guide](https://git.k8s.io/community/contributors/guide/first-contribution.md#issue-assignment-in-github)

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/6974#issuecomment-981279275):

>/assign @zijianjoy @jiyongjung0 


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
Failure error message coming from upgrading TFX from 1.2 to 1.4:

```
sample-test-7c4kw-3475281298: =========Argo Workflow Log=========
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: time="2021-11-26T14:58:03.147Z" level=info msg="capturing logs" argo=true
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: 2021-11-26 14:58:05.872653: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: usage: container_entrypoint.py [-h] --pipeline_root PIPELINE_ROOT
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462:                                --kubeflow_metadata_config
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462:                                KUBEFLOW_METADATA_CONFIG --serialized_component
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462:                                SERIALIZED_COMPONENT --tfx_ir TFX_IR --node_id
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462:                                NODE_ID [--runtime_parameter RUNTIME_PARAMETER]
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: container_entrypoint.py: error: the following arguments are required: --serialized_component
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: time="2021-11-26T14:58:15.441Z" level=error msg="cannot save artifact /mlpipeline-ui-metadata.json" argo=true error="stat /mlpipeline-ui-metadata.json: no such file or directory"
sample-test-7c4kw-3475281298: parameterized-tfx-oss-c6n2c-1861378462: Error: exit status 2
```
Thank you Yuan for finding this issue!

Will it be related to the tfx version in the https://github.com/kubeflow/pipelines/blob/master/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py#L167?

Maybe at least we should upgrade tfx version to 1.4.0 in the parameterized_tfx_oss sample first, then investigate further for the issue?

TFX release changelog: https://github.com/tensorflow/tfx/blob/master/RELEASE.md
Thanks! I agree that we need to upgrade the code @zijianjoy pointed.
By the way, how about using the [default value](https://github.com/tensorflow/tfx/blob/d00b80c92792ec202564e2e90d081b361835e314/tfx/orchestration/kubeflow/kubeflow_dag_runner.py#L54) to avoid this kind of failure? It uses the image in the docker hub, then we don't need to update the version every time. (Or we can use tfx.version.__version__ in the parameterized_tfx_oss.py.)
> Thanks! I agree that we need to upgrade the code @zijianjoy pointed.
> By the way, how about using the [default value](https://github.com/tensorflow/tfx/blob/d00b80c92792ec202564e2e90d081b361835e314/tfx/orchestration/kubeflow/kubeflow_dag_runner.py#L54) to avoid this kind of failure? It uses the image in the docker hub, then we don't need to update the version every time. (Or we can use tfx.version.__version__ in the parameterized_tfx_oss.py.)

Using version.__version__ sounds like a good idea, because we don't want new image releases to break the sample.

Resolving patch:
diff --git a/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py b/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
--- a/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
+++ b/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
@@ -164,7 +164,7 @@ def _create_pipeline(
   config = tfx.orchestration.experimental.KubeflowDagRunnerConfig(
       kubeflow_metadata_config=tfx.orchestration.experimental.
       get_default_kubeflow_metadata_config(),
-      tfx_image='gcr.io/tfx-oss-public/tfx:1.2.0',
+      tfx_image='gcr.io/tfx-oss-public/tfx:%s' % tfx.__version__,
   )
   kfp_runner = tfx.orchestration.experimental.KubeflowDagRunner(
       output_filename=__file__ + '.yaml', config=config


Test patch: