internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: kubeflow/pipelines

Issue:
Bug: Dataset not found in US
I'm getting an error while running the BigQuery kfp component to export some data from BQ to GCS. It says: `google.api_core.exceptions.NotFound: 404 Not found: xxx was not found in location US` which is correct, because the dataset is in EU, but I couldn't find any way to change the default behaviour of looking for datasets in the US.

On our regular codebase we specify the location on the BigQuery client initialisation, but the code here just uses the default: https://github.com/danicat/pipelines/blob/master/component_sdk/python/kfp_component/google/bigquery/_query.py

It is a one line change in the code, but I don't know how to rebuild the ml-pipeline container. The developer guide doesn't mention how to do it (but it does explain how to build containers for other components of the solution).

Please help! :)


Discussion:
>the code here just uses the default: 

Hmm. I see that there is the `dataset_location` parameter. 
And that parameter is exposed in the component:
https://github.com/danicat/pipelines/blob/b29fbb5041433649335b2ed376940bbb300e3eef/components/gcp/bigquery/query/component.yaml#L40

>It is a one line change in the code

We happily accept Pull Requests.

>I don't know how to rebuild the ml-pipeline container.

Counter-intuitively, the Dockerfile and the build_image script are here: https://github.com/danicat/pipelines/tree/master/components/gcp/container


@danicat, have you tried the dataset_location parameter of the component? Please let us know your client code if it still fails. Thanks.
Assuming that this is resolved. Please re-open if not. 
Definitely not solved, just didn’t have time to look into this.
@hongye-sun that parameter isn’t used for client instantiation. It’s documented as a dataset *output* location.

But the existence of that parameter is why I say it would be a trivial change.
I see. I didn't notice that there is a default location setting in the Client. I will set it by reusing the dataset_location. 

Resolving patch:
diff --git a/component_sdk/python/kfp_component/google/bigquery/_query.py b/component_sdk/python/kfp_component/google/bigquery/_query.py
--- a/component_sdk/python/kfp_component/google/bigquery/_query.py
+++ b/component_sdk/python/kfp_component/google/bigquery/_query.py
@@ -43,7 +43,7 @@ def query(query, project_id, dataset_id=None, table_id=None,
     Returns:
         The API representation of the completed query job.
     """
-    client = bigquery.Client(project=project_id)
+    client = bigquery.Client(project=project_id, location=dataset_location)
     if not job_config:
         job_config = bigquery.QueryJobConfig()
         job_config.create_disposition = bigquery.job.CreateDisposition.CREATE_IF_NEEDED


Test patch:
Repository: kubeflow/pipelines

Issue:
cmle training component should output job_dir.txt
It should output the file regardless user provides jodDir or not, otherwise argo will fail.


Discussion:
The workaround is to provide a jobDir.

Resolving patch:
diff --git a/component_sdk/python/kfp_component/google/ml_engine/_create_job.py b/component_sdk/python/kfp_component/google/ml_engine/_create_job.py
--- a/component_sdk/python/kfp_component/google/ml_engine/_create_job.py
+++ b/component_sdk/python/kfp_component/google/ml_engine/_create_job.py
@@ -133,6 +133,8 @@ def _dump_job(self, job):
         logging.info('Dumping job: {}'.format(job))
         gcp_common.dump_file('/tmp/kfp/output/ml_engine/job.json', json.dumps(job))
         gcp_common.dump_file('/tmp/kfp/output/ml_engine/job_id.txt', job['jobId'])
+        job_dir = ''
         if 'trainingInput' in job and 'jobDir' in job['trainingInput']:
-            gcp_common.dump_file('/tmp/kfp/output/ml_engine/job_dir.txt', 
-                job['trainingInput']['jobDir'])
+            job_dir = job['trainingInput']['jobDir']
+        gcp_common.dump_file('/tmp/kfp/output/ml_engine/job_dir.txt', 
+            job_dir)


Test patch:
Repository: kubeflow/pipelines

Issue:
Crate VolumeOP from  k8s_resource error
**What happened:**
when I run
```
 dsl.VolumeOP(k8s_resource=my_vpc)
```

```
 Raises:
            ValueError: if k8s_resource is provided along with other arguments
```


I think the reason is :
``` 
    if "k8s_resource" in kwargs:
            if resource_name or size or storage_class or modes or annotations:
                raise ValueError("You cannot provide k8s_resource along with "
                                 "other arguments.")
```
```
    def __init__(self,
                 resource_name: str = None,
                 size: str = None,
                 storage_class: str = None,
                 modes: List[str] = VOLUME_MODE_RWM,
                 annotations: Dict[str, str] = None,
                 data_source=None,
                 **kwargs):
```

but the mode has a default value
sdk/python/kfp/dsl/_volume_op.py
**What did you expect to happen:**
I think, I should only put k8s_resource and name in it.
**What steps did you take:**
[A clear and concise description of what the bug is.]

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]




Discussion:

Resolving patch:
diff --git a/sdk/python/kfp/dsl/_volume_op.py b/sdk/python/kfp/dsl/_volume_op.py
--- a/sdk/python/kfp/dsl/_volume_op.py
+++ b/sdk/python/kfp/dsl/_volume_op.py
@@ -41,7 +41,7 @@ def __init__(self,
                  resource_name: str = None,
                  size: str = None,
                  storage_class: str = None,
-                 modes: List[str] = VOLUME_MODE_RWM,
+                 modes: List[str] = None,
                  annotations: Dict[str, str] = None,
                  data_source=None,
                  **kwargs):
@@ -112,7 +112,7 @@ def __init__(self,
             requests={"storage": size}
         )
         pvc_spec = V1PersistentVolumeClaimSpec(
-            access_modes=modes,
+            access_modes=modes or VOLUME_MODE_RWM,
             resources=requested_resources,
             storage_class_name=storage_class,
             data_source=data_source


Test patch:
Repository: kubeflow/pipelines

Issue:
`pip install kfp` does not install CLI
**What happened:**

```
$ virtualenv .venv
...
$ pip install kfp==0.1.23
...
$ kfp
Traceback (most recent call last):
  File "/private/tmp/.venv/bin/kfp", line 6, in <module>
    from kfp.__main__ import main
  File "/private/tmp/.venv/lib/python3.7/site-packages/kfp/__main__.py", line 15, in <module>
    from .cli.cli import main
ModuleNotFoundError: No module named 'kfp.cli'
```

**What did you expect to happen:**
To run the CLI.

**Anything else you would like to add:**
I could be confused about what is expected to be available after installing the kfp package from pip - setup.py mentions an entrypoint named kfp in 

https://github.com/kubeflow/pipelines/blob/812ca7f8836c47039c3b1f3daf23e68fbcee1a92/sdk/python/setup.py#L74

but main.py imports a `kfp.cli` package https://github.com/kubeflow/pipelines/blob/812ca7f8836c47039c3b1f3daf23e68fbcee1a92/sdk/python/kfp/__main__.py#L15

which is not included in the distribution https://github.com/kubeflow/pipelines/blob/812ca7f8836c47039c3b1f3daf23e68fbcee1a92/sdk/python/setup.py#L46-L54



Discussion:

Resolving patch:
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -45,6 +45,7 @@
     install_requires=REQUIRES,
     packages=[
         'kfp',
+        'kfp.cli',
         'kfp.compiler',
         'kfp.components',
         'kfp.components.structures',


Test patch:
Repository: kubeflow/pipelines

Issue:
[Component] GCP dataproc create_cluster component cannot correctly specify image_version.
Issue:
When specifying not-null image version, create_cluster component raises:
`<HttpError 400 when requesting 
https://dataproc.googleapis.com/v1/projects/ml-pipeline-dogfood/regions/us-central1/clusters?alt=json&requestId=7c933fdacb068cd6811fb40b8334a3d4
 returned "Invalid JSON payload received. Unknown name "softwareConfig" at 'cluster': Cannot find field.">`

Initial investigation shows that in [here](https://github.com/kubeflow/pipelines/blob/7dab30085e2edda6fb4ecb61a61c9f37664009a1/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_create_cluster.py#L70), `softwareConfig` was specified as a top-layer member of cluster payload, but actually it should be a member of cluster['config']. See [this reference](https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#Cluster).

Will work out a fix shortly.


Discussion:
Blocking #1913 
cc @gaoning777 

Resolving patch:
diff --git a/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_create_cluster.py b/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_create_cluster.py
--- a/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_create_cluster.py
+++ b/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_create_cluster.py
@@ -66,9 +66,9 @@ def create_cluster(project_id, region, name=None, name_prefix=None,
     if config_bucket:
         cluster['config']['configBucket'] = config_bucket
     if image_version:
-        if 'softwareConfig' not in cluster:
-            cluster['softwareConfig'] = {}
-        cluster['softwareConfig']['imageVersion'] = image_version
+        if 'softwareConfig' not in cluster['config']:
+            cluster['config']['softwareConfig'] = {}
+        cluster['config']['softwareConfig']['imageVersion'] = image_version
 
     return _create_cluster_internal(project_id, region, cluster, name_prefix,
         wait_interval)


Test patch:
diff --git a/components/gcp/container/component_sdk/python/tests/google/dataproc/test__create_cluster.py b/components/gcp/container/component_sdk/python/tests/google/dataproc/test__create_cluster.py
--- a/components/gcp/container/component_sdk/python/tests/google/dataproc/test__create_cluster.py
+++ b/components/gcp/container/component_sdk/python/tests/google/dataproc/test__create_cluster.py
@@ -60,10 +60,10 @@ def test_create_cluster_with_specs_succeed(self, mock_dataproc_client,
                         'executableFile': 'gs://action/2'
                     }
                 ],
-                'configBucket': 'gs://config/bucket'
-            },
-            'softwareConfig': {
-                'imageVersion': '1.10'
+                'configBucket': 'gs://config/bucket',
+                'softwareConfig': {
+                    'imageVersion': '1.10'
+                },
             },
             'labels': {
                 'label-1': 'value-1'
Repository: kubeflow/pipelines

Issue:
kfp 0.1.35 tar.gz in pypi.org is missing diagnose_me directory
**What happened:**
The 0.1.35 release of kfp available on pypi.org (i.e. what is installed via `pip3 install kfp`) seems to be missing the `kfp/cli/diagnose_me` directory containing the diagnose_me modules required by the cli. The release hosted on github contains these files.

This is the tar.gz file hosted on pypi: https://files.pythonhosted.org/packages/e8/02/51dbeae211ddf1c931b2d1613db90856b7d94a53c1d9f704593dfa6253ae/kfp-0.1.35.tar.gz

If you try to install and run kfp 0.1.35 via pip it causes an error:
```
Traceback (most recent call last):
  File "/Users/shenderson/venvs/kubeflow/bin/kfp", line 5, in <module>
    from kfp.__main__ import main
  File "/Users/shenderson/venvs/kubeflow/lib/python3.7/site-packages/kfp/__main__.py", line 15, in <module>
    from .cli.cli import main
  File "/Users/shenderson/venvs/kubeflow/lib/python3.7/site-packages/kfp/cli/cli.py", line 21, in <module>
    from .diagnose_me_cli import diagnose_me
  File "/Users/shenderson/venvs/kubeflow/lib/python3.7/site-packages/kfp/cli/diagnose_me_cli.py", line 6, in <module>
    from .diagnose_me import dev_env
ModuleNotFoundError: No module named 'kfp.cli.diagnose_me'
```

**What did you expect to happen:**
All kfp modules including the diagnose_me package to be installed.

**What steps did you take:**
* Run `pip3 install --upgrade --force --no-cache-dir kfp`
* Run `kfp`





Discussion:
Looks like the `kfp.cli.diagnose_me` package is missing from setup.py:
https://github.com/kubeflow/pipelines/blob/master/sdk/python/setup.py#L64-L73

Resolving patch:
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -64,6 +64,7 @@ def find_version(*file_path_parts):
     packages=[
         'kfp',
         'kfp.cli',
+        'kfp.cli.diagnose_me',
         'kfp.compiler',
         'kfp.components',
         'kfp.components.structures',


Test patch:
Repository: kubeflow/pipelines

Issue:
KFP 0.2.1 SDK client broken when inverse proxy host is used
This bug is introduced in #2896 

Under the hood `subprocess.check_output()` return bytes instead of string. However in `kfp_server_api/configuration.py` it was assumed to be string, which caused error like the following:

```
~/.local/lib/python3.5/site-packages/kfp_server_api/configuration.py in get_api_key_with_prefix(self, identifier)
    203         if (self.api_key.get(identifier) and
    204                 self.api_key_prefix.get(identifier)):
--> 205             return self.api_key_prefix[identifier] + ' ' + self.api_key[identifier]  # noqa: E501
    206         elif self.api_key.get(identifier):
    207             return self.api_key[identifier]

TypeError: Can't convert 'bytes' object to str implicitly
```
This impacts some CUJs involving TFX CLI and CAIP notebook experiences. Will send out a fix today.




Discussion:
/priority p0
/kind bug

Resolving patch:
diff --git a/sdk/python/kfp/_auth.py b/sdk/python/kfp/_auth.py
--- a/sdk/python/kfp/_auth.py
+++ b/sdk/python/kfp/_auth.py
@@ -37,7 +37,8 @@ def get_gcp_access_token():
     https://cloud.google.com/sdk/gcloud/reference/auth/application-default/print-access-token
     """
     args = ['gcloud', 'auth', 'print-access-token']
-    return subprocess.check_output(args).rstrip()
+    # Casting to string to accommodate API server request schema.
+    return subprocess.check_output(args).rstrip().decode("utf-8")
 
 def get_auth_token(client_id, other_client_id, other_client_secret):
     """Gets auth token from default service account or user account."""


Test patch:
Repository: kubeflow/pipelines

Issue:
SDK - Support placeholders in task display names
Fixes https://github.com/kubeflow/pipelines/issues/4145


Discussion:

Resolving patch:
diff --git a/sdk/python/kfp/compiler/_op_to_template.py b/sdk/python/kfp/compiler/_op_to_template.py
--- a/sdk/python/kfp/compiler/_op_to_template.py
+++ b/sdk/python/kfp/compiler/_op_to_template.py
@@ -176,6 +176,10 @@ def _outputs_to_json(op: BaseOp,
 def _op_to_template(op: BaseOp):
     """Generate template given an operator inherited from BaseOp."""
 
+    # Display name
+    if op.display_name:
+        op.add_pod_annotation('pipelines.kubeflow.org/task_display_name', op.display_name)
+
     # NOTE in-place update to BaseOp
     # replace all PipelineParams with template var strings
     processed_op = _process_base_ops(op)
@@ -270,10 +274,6 @@ def _op_to_template(op: BaseOp):
         template['volumes'] = [convert_k8s_obj_to_json(volume) for volume in processed_op.volumes]
         template['volumes'].sort(key=lambda x: x['name'])
 
-    # Display name
-    if processed_op.display_name:
-        template.setdefault('metadata', {}).setdefault('annotations', {})['pipelines.kubeflow.org/task_display_name'] = processed_op.display_name
-
     if isinstance(op, dsl.ContainerOp) and op._metadata:
         template.setdefault('metadata', {}).setdefault('annotations', {})['pipelines.kubeflow.org/component_spec'] = json.dumps(op._metadata.to_dict(), sort_keys=True)
 


Test patch:
diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
--- a/sdk/python/tests/compiler/compiler_tests.py
+++ b/sdk/python/tests/compiler/compiler_tests.py
@@ -624,6 +624,16 @@ def some_pipeline():
     template = workflow_dict['spec']['templates'][0]
     self.assertEqual(template['metadata']['annotations']['pipelines.kubeflow.org/task_display_name'], 'Custom name')
 
+  def test_set_dynamic_display_name(self):
+    """Test a pipeline with a customized task names."""
+
+    def some_pipeline(custom_name):
+      some_op().set_display_name(custom_name)
+
+    workflow_dict = kfp.compiler.Compiler()._compile(some_pipeline)
+    template = [template for template in workflow_dict['spec']['templates'] if 'container' in template][0]
+    self.assertNotIn('pipelineparam', template['metadata']['annotations']['pipelines.kubeflow.org/task_display_name'])
+
   def test_set_parallelism(self):
     """Test a pipeline with parallelism limits."""
     def some_op():
Repository: kubeflow/pipelines

Issue:
Client helper get_experiment() raises irrelevant error when there are no experiments
### What steps did you take:
Use `client.get_experiment(experiment_name="test")` when there are **zero** experiments.

### What happened:
`client.get_experiment()` raises error `'NoneType' object is not iterable`
```
>>> client.get_experiment(experiment_name="test")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 333, in get_experiment
    for experiment in list_experiments_response.experiments:
TypeError: 'NoneType' object is not iterable
```

### What did you expect to happen:
`client.get_experiment()` to raise the error `ValueError('No experiment is found with name {}.'.format(experiment_name))`
flexible pipeline service (host) path in client SDK 
when creating an SDK `Client()` the path to `ml-pipeline` API service is loaded from a hard coded value (`ml-pipeline.kubeflow.svc.cluster.local:8888`) which indicate a specific k8s namespace. it can be valuable to load that default value from an env variable, i.e. changing the line in `_client.py` from:

`config.host = host if host else Client.IN_CLUSTER_DNS_NAME`

to:

`config.host = host or os.environ.get('ML_PIPELINE_DNS_NAME',Client.IN_CLUSTER_DNS_NAME)`

also note that when a user provide the `host` parameter, the ipython output points to the API server and not to the UI service (see the logic in `_get_url_prefix()`), it seems like a potential bug

if its acceptable i can submit a PR for the line change above
 


Discussion:

>it can be valuable to load that default value from an env variable

I'm not sure about the user scenarios, but I guess it might be useful for some cases.

>ML_PIPELINE_DNS_NAME

Maybe, `KF_PIPELINES_ENDPOINT`?

>it seems like a potential bug

It probably is. The Client class can definitely use some improvements, especially for out-of-cluster usage.

> if its acceptable i can submit a PR

We always appreciate PRs.
Fixed in https://github.com/kubeflow/pipelines/pull/1293

Resolving patch:
diff --git a/sdk/python/kfp/_client.py b/sdk/python/kfp/_client.py
--- a/sdk/python/kfp/_client.py
+++ b/sdk/python/kfp/_client.py
@@ -379,7 +379,7 @@ def get_experiment(self, experiment_id=None, experiment_name=None, namespace=Non
     while next_page_token is not None:
       list_experiments_response = self.list_experiments(page_size=100, page_token=next_page_token, namespace=namespace)
       next_page_token = list_experiments_response.next_page_token
-      for experiment in list_experiments_response.experiments:
+      for experiment in list_experiments_response.experiments or []:
         if experiment.name == experiment_name:
           return self._experiment_api.get_experiment(id=experiment.id)
     raise ValueError('No experiment is found with name {}.'.format(experiment_name))


Test patch:
Repository: kubeflow/pipelines

Issue:
[sdk] KFP SDK fail to use argo CLI >=2.12 to lint workflows
### What happened:
After https://github.com/kubeflow/pipelines/pull/5266, I noticed in tests, the warning

> Cannot validate the compiled workflow. Found the argo program in PATH, but it's not usable. argo v2.4.3 should work.

### What did you expect to happen:

KFP SDK can use latest argo CLI to validate workflows locally.

### Environment:
<!-- Please fill in those that seem relevant. -->

KFP SDK version: 1.4.0

### Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
/area sdk

<!--
// /area frontend
// /area backend
// /area testing
// /area engprod
-->

The root cause seems to be that, KFP SDK tries to lint empty input using argo CLI to verify if argo CLI works.

However, the default behavior of Argo CLI changed, it gives an error when validating empty string.

We should update to validate an empty workflow instead.

See https://github.com/kubeflow/pipelines/blob/21e3ef8311308c748c0b1656a5f8a06df0ea3045/sdk/python/kfp/compiler/compiler.py#L1094


Discussion:

Resolving patch:
diff --git a/sdk/python/kfp/compiler/compiler.py b/sdk/python/kfp/compiler/compiler.py
--- a/sdk/python/kfp/compiler/compiler.py
+++ b/sdk/python/kfp/compiler/compiler.py
@@ -1091,7 +1091,17 @@ def _validate_workflow(workflow: dict):
   if argo_path:
     has_working_argo_lint = False
     try:
-      has_working_argo_lint = _run_argo_lint('')
+      has_working_argo_lint = _run_argo_lint("""
+        apiVersion: argoproj.io/v1alpha1 
+        kind: Workflow 
+        metadata: 
+          generateName: hello-world- 
+        spec: 
+          entrypoint: whalesay 
+          templates: 
+          - name: whalesay 
+            container: 
+              image: docker/whalesay:latest""")
     except:
       warnings.warn("Cannot validate the compiled workflow. Found the argo program in PATH, but it's not usable. argo v2.4.3 should work.")
 


Test patch:
Repository: kubeflow/pipelines

Issue:
Add clear instructions for authorization code / token
I was following [this official example](https://colab.research.google.com/github/kubeflow/website/blob/master/content/en/docs/pipelines/sdk/python-function-components.ipynb) to submit a pipeline via Jupyter or Python script

I filled in the client id and credentials [following these instructions](https://www.kubeflow.org/docs/gke/pipelines/authentication-sdk/#connecting-to-kubeflow-pipelines-in-a-full-kubeflow-deployment), but when I try to [compile the pipeline](https://www.kubeflow.org/docs/pipelines/sdk/build-component/#compile-the-pipeline) with
```
dsl-compile --py [path/to/python/file] --output [path/to/output/tar.gz]
```
The resulting prompt doesn't make it evident that the url can be copy pasted into the browser, to get the Authorization code, and I got stuck on it for a while. Looking back it's kind of an Occam's razor thing, but it could have saved a lot of time with even simple instructions "copy paste this url"

![Screen Shot 2021-01-12 at 5 46 15 PM copy](https://user-images.githubusercontent.com/23287722/104479213-2982f800-5591-11eb-81ca-e2cf4df92b4f.png)




Discussion:

Resolving patch:
diff --git a/sdk/python/kfp/_auth.py b/sdk/python/kfp/_auth.py
--- a/sdk/python/kfp/_auth.py
+++ b/sdk/python/kfp/_auth.py
@@ -178,7 +178,7 @@ def get_auth_code(client_id):
     auth_url = "https://accounts.google.com/o/oauth2/v2/auth?client_id=%s&response_type=code&scope=openid%%20email&access_type=offline&redirect_uri=urn:ietf:wg:oauth:2.0:oob"%client_id
     print(auth_url)
     open_new_tab(auth_url)
-    return input("Authorization code: ")
+    return input("If there's no browser window prompt, please direct to the URL above, then copy and paste the authorization code here: ")
 
 def get_refresh_token_from_code(auth_code, client_id, client_secret):
     payload = {"code": auth_code, "client_id": client_id, "client_secret": client_secret,


Test patch: