Repository: kubeflow/pipelines

Issue:
allow output artifact store configuration (vs hard coded)
it seems like the output artifacts are always stored in a specific minio service, port, namespace, bucket, secrets, etc (`minio-service.kubeflow:9000`). 

see: https://github.com/kubeflow/pipelines/blob/f40a22a3f4a8e06d20cf3e3f425b5058d5c87e0b/sdk/python/kfp/compiler/_op_to_template.py#L148

it would be great to make it flexible, e.g. allow using S3, or change namespace or bucket names.
i suggest making it configurable, i can do such PR if we agree its needed. 
flexible pipeline service (host) path in client SDK 
when creating an SDK `Client()` the path to `ml-pipeline` API service is loaded from a hard coded value (`ml-pipeline.kubeflow.svc.cluster.local:8888`) which indicate a specific k8s namespace. it can be valuable to load that default value from an env variable, i.e. changing the line in `_client.py` from:

`config.host = host if host else Client.IN_CLUSTER_DNS_NAME`

to:

`config.host = host or os.environ.get('ML_PIPELINE_DNS_NAME',Client.IN_CLUSTER_DNS_NAME)`

also note that when a user provide the `host` parameter, the ipython output points to the API server and not to the UI service (see the logic in `_get_url_prefix()`), it seems like a potential bug

if its acceptable i can submit a PR for the line change above
 


Discussion:
There is already a PR for this. But we'd like to solve this in a different way - by having an in-cluster configMap with the storage configuration. Argo already supports that, but we're hesitating to use that mechanism since KF Pipelines might not be the exclusive user of the Argo controller.

Fortunately, the problem will be solved in Argo 2.4 (per-pipeline configMap reference)
Also https://github.com/kubeflow/pipelines/issues/363 https://github.com/kubeflow/pipelines/issues/1195 https://github.com/kubeflow/pipelines/issues/362
@Ark-kun a mechanism like config map sounds better, it will also make the YAMLs cleaner, currently u get the same 27 lines added to every container, see below, would be nice to hide it:

```yaml
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
```
Fixed by https://github.com/kubeflow/pipelines/pull/1297
>it can be valuable to load that default value from an env variable

I'm not sure about the user scenarios, but I guess it might be useful for some cases.

>ML_PIPELINE_DNS_NAME

Maybe, `KF_PIPELINES_ENDPOINT`?

>it seems like a potential bug

It probably is. The Client class can definitely use some improvements, especially for out-of-cluster usage.

> if its acceptable i can submit a PR

We always appreciate PRs.
Fixed in https://github.com/kubeflow/pipelines/pull/1293

Resolving patch:
diff --git a/sdk/python/kfp/aws.py b/sdk/python/kfp/aws.py
--- a/sdk/python/kfp/aws.py
+++ b/sdk/python/kfp/aws.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-def use_aws_secret(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_KEY_ID', aws_secret_access_key_name='AWS_SECRET_ACCESS_KEY'):
+def use_aws_secret(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_KEY_ID', aws_secret_access_key_name='AWS_SECRET_ACCESS_KEY', aws_region=None):
     """An operator that configures the container to use AWS credentials.
 
     AWS doesn't create secret along with kubeflow deployment and it requires users
@@ -32,31 +32,38 @@ def use_aws_secret(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_
 
     def _use_aws_secret(task):
         from kubernetes import client as k8s_client
-        (
-            task.container
-                .add_env_variable(
-                    k8s_client.V1EnvVar(
-                        name='AWS_ACCESS_KEY_ID',
-                        value_from=k8s_client.V1EnvVarSource(
-                            secret_key_ref=k8s_client.V1SecretKeySelector(
-                                name=secret_name,
-                                key=aws_access_key_id_name
-                            )
+        task.container \
+            .add_env_variable(
+                k8s_client.V1EnvVar(
+                    name='AWS_ACCESS_KEY_ID',
+                    value_from=k8s_client.V1EnvVarSource(
+                        secret_key_ref=k8s_client.V1SecretKeySelector(
+                            name=secret_name,
+                            key=aws_access_key_id_name
                         )
                     )
                 )
+            ) \
+            .add_env_variable(
+                k8s_client.V1EnvVar(
+                    name='AWS_SECRET_ACCESS_KEY',
+                    value_from=k8s_client.V1EnvVarSource(
+                        secret_key_ref=k8s_client.V1SecretKeySelector(
+                            name=secret_name,
+                            key=aws_secret_access_key_name
+                        )
+                    )
+                )
+            )
+
+        if aws_region:
+            task.container \
                 .add_env_variable(
                     k8s_client.V1EnvVar(
-                        name='AWS_SECRET_ACCESS_KEY',
-                        value_from=k8s_client.V1EnvVarSource(
-                            secret_key_ref=k8s_client.V1SecretKeySelector(
-                                name=secret_name,
-                                key=aws_secret_access_key_name
-                            )
-                        )
+                        name='AWS_REGION',
+                        value=aws_region
                     )
                 )
-        )
         return task
 
     return _use_aws_secret


Test patch:
diff --git a/sdk/python/tests/dsl/aws_extensions_tests.py b/sdk/python/tests/dsl/aws_extensions_tests.py
--- a/sdk/python/tests/dsl/aws_extensions_tests.py
+++ b/sdk/python/tests/dsl/aws_extensions_tests.py
@@ -21,10 +21,11 @@
 class TestAwsExtension(unittest.TestCase):
   def test_default_aws_secret_name(self):
     spec = inspect.getfullargspec(use_aws_secret)
-    assert len(spec.defaults) == 3
+    assert len(spec.defaults) == 4
     assert spec.defaults[0] == 'aws-secret'
     assert spec.defaults[1] == 'AWS_ACCESS_KEY_ID'
     assert spec.defaults[2] == 'AWS_SECRET_ACCESS_KEY'
+    assert spec.defaults[3] == None
 
   def test_use_aws_secret(self):
       op1 = ContainerOp(name='op1', image='image')
@@ -37,3 +38,20 @@ def test_use_aws_secret(self):
           assert op1.container.env[index].value_from.secret_key_ref.name == 'myaws-secret'
           assert op1.container.env[index].value_from.secret_key_ref.key == expected_key
           index += 1
+
+  def test_use_aws_secret_with_region(self):
+      op1 = ContainerOp(name='op1', image='image')
+      aws_region = 'us-west-2'
+      op1 = op1.apply(use_aws_secret('myaws-secret', 'key_id', 'access_key', aws_region))
+      assert len(op1.container.env) == 3
+
+      index = 0
+      for expected_name, expected_key in [('AWS_ACCESS_KEY_ID', 'key_id'), ('AWS_SECRET_ACCESS_KEY', 'access_key')]:
+          assert op1.container.env[index].name == expected_name
+          assert op1.container.env[index].value_from.secret_key_ref.name == 'myaws-secret'
+          assert op1.container.env[index].value_from.secret_key_ref.key == expected_key
+          index += 1
+
+      for expected_name, expected_key in [('AWS_REGION', aws_region)]:
+          assert op1.container.env[index].name == expected_name
+          assert op1.container.env[index].value == expected_key