Repository: kubeflow/pipelines

Issue:
allow output artifact store configuration (vs hard coded)
it seems like the output artifacts are always stored in a specific minio service, port, namespace, bucket, secrets, etc (`minio-service.kubeflow:9000`). 

see: https://github.com/kubeflow/pipelines/blob/f40a22a3f4a8e06d20cf3e3f425b5058d5c87e0b/sdk/python/kfp/compiler/_op_to_template.py#L148

it would be great to make it flexible, e.g. allow using S3, or change namespace or bucket names.
i suggest making it configurable, i can do such PR if we agree its needed. 
flexible pipeline service (host) path in client SDK 
when creating an SDK `Client()` the path to `ml-pipeline` API service is loaded from a hard coded value (`ml-pipeline.kubeflow.svc.cluster.local:8888`) which indicate a specific k8s namespace. it can be valuable to load that default value from an env variable, i.e. changing the line in `_client.py` from:

`config.host = host if host else Client.IN_CLUSTER_DNS_NAME`

to:

`config.host = host or os.environ.get('ML_PIPELINE_DNS_NAME',Client.IN_CLUSTER_DNS_NAME)`

also note that when a user provide the `host` parameter, the ipython output points to the API server and not to the UI service (see the logic in `_get_url_prefix()`), it seems like a potential bug

if its acceptable i can submit a PR for the line change above
 


Discussion:
There is already a PR for this. But we'd like to solve this in a different way - by having an in-cluster configMap with the storage configuration. Argo already supports that, but we're hesitating to use that mechanism since KF Pipelines might not be the exclusive user of the Argo controller.

Fortunately, the problem will be solved in Argo 2.4 (per-pipeline configMap reference)
Also https://github.com/kubeflow/pipelines/issues/363 https://github.com/kubeflow/pipelines/issues/1195 https://github.com/kubeflow/pipelines/issues/362
@Ark-kun a mechanism like config map sounds better, it will also make the YAMLs cleaner, currently u get the same 27 lines added to every container, see below, would be nice to hide it:

```yaml
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
```
Fixed by https://github.com/kubeflow/pipelines/pull/1297
>it can be valuable to load that default value from an env variable

I'm not sure about the user scenarios, but I guess it might be useful for some cases.

>ML_PIPELINE_DNS_NAME

Maybe, `KF_PIPELINES_ENDPOINT`?

>it seems like a potential bug

It probably is. The Client class can definitely use some improvements, especially for out-of-cluster usage.

> if its acceptable i can submit a PR

We always appreciate PRs.
Fixed in https://github.com/kubeflow/pipelines/pull/1293

Resolving patch:
diff --git a/samples/core/iris/iris.py b/samples/core/iris/iris.py
--- a/samples/core/iris/iris.py
+++ b/samples/core/iris/iris.py
@@ -198,11 +198,11 @@ def _create_pipeline(
 if __name__ == '__main__':
   absl.logging.set_verbosity(absl.logging.INFO)
   # Make sure the version of TFX image used is consistent with the version of
-  # TFX SDK. Here we use tfx:0.21.2 image.
+  # TFX SDK. Here we use tfx:0.22.0 image.
   config = kubeflow_dag_runner.KubeflowDagRunnerConfig(
       kubeflow_metadata_config=kubeflow_dag_runner.
       get_default_kubeflow_metadata_config(),
-      tfx_image='gcr.io/tfx-oss-public/tfx:0.21.2',
+      tfx_image='gcr.io/tfx-oss-public/tfx:0.22.0',
   )
   kfp_runner = kubeflow_dag_runner.KubeflowDagRunner(
       output_filename=__file__ + '.yaml', config=config
diff --git a/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py b/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
--- a/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
+++ b/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py
@@ -178,7 +178,7 @@ def _create_pipeline(
   config = kubeflow_dag_runner.KubeflowDagRunnerConfig(
       kubeflow_metadata_config=kubeflow_dag_runner.
       get_default_kubeflow_metadata_config(),
-      tfx_image='gcr.io/tfx-oss-public/tfx:0.21.2',
+      tfx_image='gcr.io/tfx-oss-public/tfx:0.22.0',
   )
   kfp_runner = kubeflow_dag_runner.KubeflowDagRunner(
       output_filename=__file__ + '.yaml', config=config


Test patch:
diff --git a/test/sample-test/requirements.in b/test/sample-test/requirements.in
--- a/test/sample-test/requirements.in
+++ b/test/sample-test/requirements.in
@@ -3,7 +3,7 @@ minio
 papermill
 fire
 yamale
-tfx==0.21.2
+tfx==0.22.0
 
 # Avoiding conflicts:
 # There are incompatible versions in the resolved dependencies:
diff --git a/test/sample-test/requirements.txt b/test/sample-test/requirements.txt
--- a/test/sample-test/requirements.txt
+++ b/test/sample-test/requirements.txt
@@ -6,157 +6,163 @@
 #
 absl-py==0.8.1            # via ml-metadata, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx, tfx-bsl
 ansiwrap==0.8.4           # via papermill
-apache-beam[gcp]==2.17.0  # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx, tfx-bsl
-appdirs==1.4.3            # via black
-argo-models==2.2.1a       # via -r -
-astor==0.8.1              # via tensorflow
+apache-beam[gcp]==2.22.0  # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx, tfx-bsl
+appdirs==1.4.4            # via black
+astunparse==1.6.3         # via tensorflow
 async-generator==1.10     # via nbclient
 attrs==19.3.0             # via black, jsonschema
 avro-python3==1.9.2.1     # via apache-beam
-backcall==0.1.0           # via ipython
+backcall==0.2.0           # via ipython
 black==19.10b0            # via papermill
-bleach==3.1.4             # via nbconvert
+bleach==3.1.5             # via nbconvert
 cachetools==3.1.1         # via apache-beam, google-auth
-certifi==2020.4.5.1       # via kfp-server-api, kubernetes, minio, requests
+certifi==2020.6.20        # via kubernetes, minio, requests
 chardet==3.0.4            # via requests
-click==7.1.1              # via -r -, black, papermill, tfx
-cloudpickle==1.3.0        # via -r -
+click==7.1.2              # via black, papermill, tfx
+colorama==0.4.3           # via keras-tuner
 configparser==5.0.0       # via minio
 crcmod==1.7               # via apache-beam
 decorator==4.4.2          # via ipython, traitlets
 defusedxml==0.6.0         # via nbconvert
-deprecated==1.2.9         # via -r -
-dill==0.3.0               # via apache-beam
-docker==4.2.0             # via tfx
+dill==0.3.1.1             # via apache-beam
+docker==4.2.2             # via tfx
 docopt==0.6.2             # via hdfs
 entrypoints==0.3          # via nbconvert, papermill
-fastavro==0.21.24         # via apache-beam
+fastavro==0.23.5          # via apache-beam
 fasteners==0.15           # via google-apitools
 fire==0.3.1               # via -r -
-future==0.18.2            # via apache-beam
-gast==0.2.2               # via tensorflow
-google-api-core[grpc]==1.17.0  # via google-api-python-client, google-cloud-bigtable, google-cloud-core, google-cloud-datastore, google-cloud-pubsub
-google-api-python-client==1.8.0  # via tfx, tfx-bsl
-google-apitools==0.5.28   # via apache-beam
+future==0.18.2            # via apache-beam, keras-tuner
+gast==0.3.3               # via tensorflow
+google-api-core[grpc,grpcgcp]==1.21.0  # via google-api-python-client, google-cloud-bigtable, google-cloud-core, google-cloud-datastore, google-cloud-dlp, google-cloud-language, google-cloud-pubsub, google-cloud-spanner, google-cloud-videointelligence, google-cloud-vision
+google-api-python-client==1.9.3  # via tfx, tfx-bsl
+google-apitools==0.5.31   # via apache-beam
 google-auth-httplib2==0.0.3  # via google-api-python-client
 google-auth-oauthlib==0.4.1  # via tensorboard
-google-auth==1.14.0       # via -r -, google-api-core, google-api-python-client, google-auth-httplib2, google-auth-oauthlib, google-cloud-storage, kubernetes, tensorboard
+google-auth==1.18.0       # via google-api-core, google-api-python-client, google-auth-httplib2, google-auth-oauthlib, google-cloud-storage, kubernetes, tensorboard
 google-cloud-bigquery==1.17.0  # via -r -, apache-beam
 google-cloud-bigtable==1.0.0  # via apache-beam
-google-cloud-core==1.3.0  # via apache-beam, google-cloud-bigquery, google-cloud-bigtable, google-cloud-datastore, google-cloud-storage
+google-cloud-core==1.3.0  # via apache-beam, google-cloud-bigquery, google-cloud-bigtable, google-cloud-datastore, google-cloud-spanner, google-cloud-storage
 google-cloud-datastore==1.7.4  # via apache-beam
+google-cloud-dlp==0.13.0  # via apache-beam
+google-cloud-language==1.3.0  # via apache-beam
 google-cloud-pubsub==1.0.2  # via apache-beam
+google-cloud-spanner==1.13.0  # via apache-beam
 google-cloud-storage==1.17.0  # via -r -
+google-cloud-videointelligence==1.13.0  # via apache-beam
+google-cloud-vision==0.42.0  # via apache-beam
 google-pasta==0.2.0       # via tensorflow
-google-resumable-media==0.5.0  # via google-cloud-bigquery, google-cloud-storage
-googleapis-common-protos[grpc]==1.51.0  # via google-api-core, grpc-google-iam-v1, tensorflow-metadata
-grpc-google-iam-v1==0.12.3  # via google-cloud-bigtable, google-cloud-pubsub
-grpcio==1.28.1            # via apache-beam, google-api-core, googleapis-common-protos, grpc-google-iam-v1, tensorboard, tensorflow, tensorflow-serving-api, tfx
-h5py==2.10.0              # via keras-applications
+google-resumable-media==0.5.1  # via google-cloud-bigquery, google-cloud-storage
+googleapis-common-protos[grpc]==1.52.0  # via google-api-core, grpc-google-iam-v1, tensorflow-metadata
+grpc-google-iam-v1==0.12.3  # via google-cloud-bigtable, google-cloud-pubsub, google-cloud-spanner
+grpcio-gcp==0.2.2         # via apache-beam, google-api-core
+grpcio==1.30.0            # via apache-beam, google-api-core, googleapis-common-protos, grpc-google-iam-v1, grpcio-gcp, tensorboard, tensorflow, tensorflow-serving-api, tfx
+h5py==2.10.0              # via tensorflow
 hdfs==2.5.8               # via apache-beam
-httplib2==0.12.0          # via apache-beam, google-api-python-client, google-apitools, google-auth-httplib2, oauth2client
-idna==2.9                 # via requests
-importlib-metadata==1.6.0  # via jsonschema
-ipykernel==5.2.1          # via ipywidgets, jupyter, jupyter-console, notebook, qtconsole
+httplib2==0.17.4          # via apache-beam, google-api-python-client, google-apitools, google-auth-httplib2, oauth2client
+idna==2.10                # via requests
+importlib-metadata==1.7.0  # via jsonschema, markdown
+ipykernel==5.3.0          # via ipywidgets, jupyter, jupyter-console, notebook, qtconsole
 ipython-genutils==0.2.0   # via nbformat, notebook, qtconsole, traitlets
-ipython==7.13.0           # via ipykernel, ipywidgets, jupyter-console, tensorflow-data-validation
+ipython==7.16.1           # via ipykernel, ipywidgets, jupyter-console
 ipywidgets==7.5.1         # via jupyter, tensorflow-model-analysis
-jedi==0.17.0              # via ipython
+jedi==0.17.1              # via ipython
 jinja2==2.11.2            # via nbconvert, notebook, tfx
 joblib==0.14.1            # via scikit-learn, tensorflow-data-validation
-jsonschema==3.2.0         # via -r -, nbformat
+jsonschema==3.2.0         # via nbformat
 junit-xml==1.9            # via -r -
-jupyter-client==6.1.3     # via ipykernel, jupyter-console, nbclient, notebook, papermill, qtconsole
+jupyter-client==6.1.5     # via ipykernel, jupyter-console, nbclient, notebook, papermill, qtconsole
 jupyter-console==6.1.0    # via jupyter
 jupyter-core==4.6.3       # via jupyter-client, nbconvert, nbformat, notebook, qtconsole
 jupyter==1.0.0            # via tensorflow-model-analysis
-keras-applications==1.0.8  # via tensorflow
-keras-preprocessing==1.1.0  # via tensorflow
-kfp-server-api==0.3.0     # via -r -
-kubernetes==11.0.0        # via -r -, argo-models
-markdown==3.2.1           # via tensorboard
+keras-preprocessing==1.1.2  # via tensorflow
+keras-tuner==1.0.1        # via tfx
+kubernetes==11.0.0        # via tfx
+markdown==3.2.2           # via tensorboard
 markupsafe==1.1.1         # via jinja2
 minio==5.0.10             # via -r -
 mistune==0.8.4            # via nbconvert
-ml-metadata==0.21.2       # via tfx
+ml-metadata==0.22.1       # via tfx
 mock==2.0.0               # via apache-beam
 monotonic==1.5            # via fasteners
-nbclient==0.2.0           # via papermill
+nbclient==0.4.0           # via papermill
 nbconvert==5.6.1          # via jupyter, notebook
-nbformat==5.0.5           # via ipywidgets, nbclient, nbconvert, notebook, papermill
-nest-asyncio==1.3.2       # via nbclient
+nbformat==5.0.7           # via ipywidgets, nbclient, nbconvert, notebook, papermill
+nest-asyncio==1.3.3       # via nbclient
 notebook==6.0.3           # via jupyter, widgetsnbextension
-numpy==1.18.2             # via h5py, keras-applications, keras-preprocessing, opt-einsum, pandas, pyarrow, scikit-learn, scipy, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx-bsl
+numpy==1.19.0             # via apache-beam, h5py, keras-preprocessing, keras-tuner, opt-einsum, pandas, pyarrow, scikit-learn, scipy, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx-bsl
 oauth2client==3.0.0       # via apache-beam, google-apitools
 oauthlib==3.1.0           # via requests-oauthlib
 opt-einsum==3.2.1         # via tensorflow
-pandas==0.25.3            # via tensorflow-data-validation, tensorflow-model-analysis
+packaging==20.4           # via bleach
+pandas==1.0.5             # via tensorflow-data-validation, tensorflow-model-analysis, tfx-bsl
 pandocfilters==1.4.2      # via nbconvert
-papermill==2.1.0          # via -r -
+papermill==2.1.2          # via -r -
 parso==0.7.0              # via jedi
 pathspec==0.8.0           # via black
 pbr==5.4.5                # via mock
 pexpect==4.8.0            # via ipython
 pickleshare==0.7.5        # via ipython
-prometheus-client==0.7.1  # via notebook
+prometheus-client==0.8.0  # via notebook
 prompt-toolkit==3.0.5     # via ipython, jupyter-console
-protobuf==3.11.3          # via apache-beam, google-api-core, google-cloud-bigquery, googleapis-common-protos, ml-metadata, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-metadata, tensorflow-model-analysis, tensorflow-serving-api, tensorflow-transform, tfx, tfx-bsl
+protobuf==3.12.2          # via apache-beam, google-api-core, google-cloud-bigquery, googleapis-common-protos, ml-metadata, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-metadata, tensorflow-model-analysis, tensorflow-serving-api, tensorflow-transform, tfx, tfx-bsl
 ptyprocess==0.6.0         # via pexpect, terminado
-pyarrow==0.15.1           # via apache-beam, tensorflow-data-validation, tensorflow-model-analysis, tfx, tfx-bsl
+pyarrow==0.16.0           # via apache-beam, tensorflow-data-validation, tensorflow-model-analysis, tfx, tfx-bsl
 pyasn1-modules==0.2.8     # via google-auth, oauth2client
 pyasn1==0.4.8             # via oauth2client, pyasn1-modules, rsa
 pydot==1.4.1              # via apache-beam, tensorflow-transform
 pygments==2.6.1           # via ipython, jupyter-console, nbconvert, qtconsole
 pymongo==3.10.1           # via apache-beam
-pyparsing==2.4.7          # via pydot
+pyparsing==2.4.7          # via packaging, pydot
 pyrsistent==0.16.0        # via jsonschema
-python-dateutil==2.8.1    # via apache-beam, jupyter-client, kfp-server-api, kubernetes, minio, pandas
-pytz==2019.3              # via apache-beam, google-api-core, minio, pandas
-pyyaml==5.3.1             # via -r -, kubernetes, papermill, tfx, yamale
-pyzmq==19.0.0             # via jupyter-client, notebook, qtconsole
-qtconsole==4.7.3          # via jupyter
+python-dateutil==2.8.1    # via apache-beam, jupyter-client, kubernetes, minio, pandas
+pytz==2020.1              # via apache-beam, fastavro, google-api-core, minio, pandas
+pyyaml==5.3.1             # via kubernetes, papermill, tfx, yamale
+pyzmq==19.0.1             # via jupyter-client, notebook, qtconsole
+qtconsole==4.7.5          # via jupyter
 qtpy==1.9.0               # via qtconsole
-regex==2020.4.4           # via black
+regex==2020.6.8           # via black
 requests-oauthlib==1.3.0  # via google-auth-oauthlib, kubernetes
-requests-toolbelt==0.9.1  # via -r -
-requests==2.23.0          # via docker, google-api-core, hdfs, kubernetes, papermill, requests-oauthlib, requests-toolbelt, tensorboard
-rsa==4.0                  # via google-auth, oauth2client
-scikit-learn==0.21.3      # via tensorflow-data-validation
-scipy==1.4.1              # via scikit-learn, tensorflow, tensorflow-model-analysis
+requests==2.24.0          # via docker, google-api-core, hdfs, keras-tuner, kubernetes, papermill, requests-oauthlib, tensorboard
+rsa==4.6                  # via google-auth, oauth2client
+scikit-learn==0.23.1      # via keras-tuner
+scipy==1.4.1              # via keras-tuner, scikit-learn, tensorflow, tensorflow-model-analysis
 send2trash==1.5.0         # via notebook
-six==1.14.0               # via absl-py, bleach, docker, fasteners, fire, google-api-core, google-api-python-client, google-apitools, google-auth, google-pasta, google-resumable-media, grpcio, h5py, hdfs, jsonschema, junit-xml, keras-preprocessing, kfp-server-api, kubernetes, ml-metadata, mock, oauth2client, protobuf, pyarrow, pyrsistent, python-dateutil, tenacity, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx, tfx-bsl, traitlets, websocket-client
-strip-hints==0.1.8        # via -r -
-tabulate==0.8.7           # via -r -
-tenacity==6.1.0           # via papermill
-tensorboard==2.1.1        # via tensorflow
-tensorflow-data-validation==0.21.5  # via tfx
-tensorflow-estimator==2.1.0  # via tensorflow
-tensorflow-metadata==0.21.2  # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx-bsl
-tensorflow-model-analysis==0.21.6  # via tfx
-tensorflow-serving-api==2.1.0  # via tfx, tfx-bsl
-tensorflow-transform==0.21.2  # via tensorflow-data-validation, tfx
-tensorflow==2.1.0         # via ml-metadata, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-serving-api, tensorflow-transform, tfx, tfx-bsl
+six==1.15.0               # via absl-py, astunparse, bleach, docker, fasteners, fire, google-api-core, google-api-python-client, google-apitools, google-auth, google-pasta, google-resumable-media, grpcio, h5py, hdfs, jsonschema, junit-xml, keras-preprocessing, kubernetes, ml-metadata, mock, oauth2client, packaging, protobuf, pyarrow, pyrsistent, python-dateutil, tenacity, tensorboard, tensorflow, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx, tfx-bsl, traitlets, websocket-client
+tabulate==0.8.7           # via keras-tuner
+tenacity==6.2.0           # via papermill
+tensorboard-plugin-wit==1.7.0  # via tensorboard
+tensorboard==2.2.2        # via tensorflow
+tensorflow-data-validation==0.22.2  # via tfx
+tensorflow-estimator==2.2.0  # via tensorflow
+tensorflow-metadata==0.22.2  # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx-bsl
+tensorflow-model-analysis==0.22.2  # via tfx
+tensorflow-serving-api==2.2.0  # via tfx, tfx-bsl
+tensorflow-transform==0.22.0  # via tensorflow-data-validation, tfx
+tensorflow==2.2.0         # via ml-metadata, tensorflow-data-validation, tensorflow-model-analysis, tensorflow-serving-api, tensorflow-transform, tfx, tfx-bsl
 termcolor==1.1.0          # via fire, tensorflow
 terminado==0.8.3          # via notebook
+terminaltables==3.1.0     # via keras-tuner
 testpath==0.4.4           # via nbconvert
 textwrap3==0.9.2          # via ansiwrap
-tfx-bsl==0.21.4           # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx
-tfx==0.21.2               # via -r -
-toml==0.10.0              # via black
+tfx-bsl==0.22.1           # via tensorflow-data-validation, tensorflow-model-analysis, tensorflow-transform, tfx
+tfx==0.22.0               # via -r -
+threadpoolctl==2.1.0      # via scikit-learn
+toml==0.10.1              # via black
 tornado==6.0.4            # via ipykernel, jupyter-client, notebook, terminado
-tqdm==4.45.0              # via papermill
+tqdm==4.47.0              # via keras-tuner, papermill
 traitlets==4.3.3          # via ipykernel, ipython, ipywidgets, jupyter-client, jupyter-core, nbclient, nbconvert, nbformat, notebook, qtconsole
 typed-ast==1.4.1          # via black
+typing-extensions==3.7.4.2  # via apache-beam
 uritemplate==3.0.1        # via google-api-python-client
-urllib3==1.25.9           # via kfp-server-api, kubernetes, minio, requests
-wcwidth==0.1.9            # via prompt-toolkit
+urllib3==1.25.9           # via kubernetes, minio, requests
+wcwidth==0.2.5            # via prompt-toolkit
 webencodings==0.5.1       # via bleach
 websocket-client==0.57.0  # via docker, kubernetes
 werkzeug==1.0.1           # via tensorboard
-wheel==0.34.2             # via strip-hints, tensorboard, tensorflow
+wheel==0.34.2             # via astunparse, tensorboard, tensorflow
 widgetsnbextension==3.5.1  # via ipywidgets
-wrapt==1.12.1             # via deprecated, tensorflow
-yamale==2.0.1             # via -r -
+wrapt==1.12.1             # via tensorflow
+yamale==2.2.0             # via -r -
 zipp==3.1.0               # via importlib-metadata
 
 # The following packages are considered to be unsafe in a requirements file: