Repository: kubeflow/pipelines

Issue:
[sdk] Invalid OAuth Scope error when uploading pipeline to Artifact Registry using Service Account Credentials
I need to implement an automatic update flow of Artifact Registry's KFP pipelines. To push to a KFP registry, I need to authenticate with Google Cloud API using a Service Account (SA). The `kfp` SDK (v2 beta 12) does not infer default credentials with correct scope to do so.

### Environment
*  Python Version: 3.10.6
*  PIP Version: 22.3.1 (with Pipenv 2022.12.19)
*  KFP version: kfp 2.0.0b12
<!-- For more information, see an overview of KFP installation options: https://www.kubeflow.org/docs/pipelines/installation/overview/. -->
*  KFP SDK version: kfp 2.0.0b12
<!-- Specify the version of Kubeflow Pipelines that you are using. The version number appears in the left side navigation of user interface.
To find the version number, See version number shows on bottom of KFP UI left sidenav. -->
*  All dependencies version:
<!-- Specify the output of the following shell command: $pip list | grep kfp -->
```
kfp                              2.0.0b12
kfp-pipeline-spec                0.2.0
kfp-server-api                   2.0.0a6
```

### Steps to reproduce

1. Generate a Google SA, with Artifact Registry Push capability and download JSON credentials;
2. Create a KFP repository on Artifact Registry (e.g. `https://us-central1-kfp.pkg.dev/test-project/kfp-repository `)
3. Define `GOOGLE_APPLICATION_CREDENTIALS` environment variable as `/path/to/sa/credentials.json`
4. Upload compiled Pipeline using `RegistryClient`:
```python
from kfp.registry import RegistryClient 
client = RegistryClient(host="https://us-central1-kfp.pkg.dev/test-project/kfp-repository") 
template_name, version = client.upload_pipeline("pipeline.yaml", tags=["0.1", "latest"])
```
The given pipeline is not pushed to Artifact Registry, and the program raises the error (some paths were omitted):
```
  Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<omitted>/__main__.py", line 127, in <module>
    upload_pipeline(args.filename, args.tags)
  File "<omitted>/__main__.py", line 104, in upload_pipeline
    templateName, versionName = kfp_client.upload_pipeline(
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/kfp/registry/registry_client.py", line 335, in upload_pipeline
    self._refresh_creds()
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/kfp/registry/registry_client.py", line 317, in _refresh_creds
    self._auth.refresh(google.auth.transport.requests.Request())
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/google/oauth2/service_account.py", line 429, in refresh
    access_token, expiry, _ = _client.jwt_grant(
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/google/oauth2/_client.py", line 294, in jwt_grant
    response_data = _token_endpoint_request(
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/google/oauth2/_client.py", line 265, in _token_endpoint_request
    _handle_error_response(response_data, retryable_error)
  File "<omitted>/.local/share/virtualenvs/ml-ops-pipelines-uL9EmYuZ/lib/python3.10/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response
    raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', {'error': 'invalid_scope', 'error_description': 'Invalid OAuth scope or ID token audience provided.'})
```

### Expected result

<!-- What should the correct behavior be? -->
The expected behavior is to push the pipeline to Artifact Registry, just like when using default user credentials. 

### Materials and Reference

The default behavior of `RegistryClient` when registry is hosted by `Artifact Registry` is to infer the credentials from environment [without any scope](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/registry/registry_client.py#L190):
```python
elif self._is_ar_host():
        auth, _ = google.auth.default()
        return auth
```
Defining a default scope (using [Google Auth library](https://github.com/googleapis/google-auth-library-python/blob/main/google/auth/credentials.py#L327)), as some Google SDK do, shoud fix this:
```python
elif self._is_ar_host():
        auth, _ = google.auth.default()
        return with_scopes_if_required(auth, self.GOOGLE_DEFAULT_AR_SCOPES)
```
With:
```python
class RegistryClient:
    ...
    GOOGLE_DEFAULT_AR_SCOPES = ('https://www.googleapis.com/auth/cloud-platform', )
```

<!-- Help us debug this issue by providing resources such as: sample code, background context, or links to references. -->

---

<!-- Don't delete message below to encourage users to support your issue! -->
Impacted by this bug? Give it a 👍. 


Discussion:
Thanks for raising this @rodrigocaus . I was fighting this exact same issue earlier and managed to fix it by adding the scopes attribute to default() 🙌.
I was actually initially trying using federated workload identity, which isn't working either (failing with invalid argument error). My next attend was via SA key which got me investigating the issue you are fixing here.

Awesome timing and I keep digging to eventually get federated workload identity working too (feels great not having to share SA keys around)
Hi @rodrigocaus, @tuliodesouza !

Thank you for reporting this. If you have a proposal, please submit a PR and request @connor-mccarthy and @chensun to review it.
For anyone here with the exact same issue and use case.  You can work around this by getting the credentials directly and feeding them into the client as an auth argument (though if you do this it wont work locally unless you set the env var)

```
from google.oauth2 import service_account
from kfp.registry import RegistryClient 
import os

credentials = service_account.Credentials.from_service_account_file(
    os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
    scopes=['https://www.googleapis.com/auth/cloud-platform']
)
client = RegistryClient(host=KFP_REGISTRY_URL, auth=credentials)
template_name, version_name = client.upload_pipeline(
  file_name="pipeline.yaml",
  tags=["latest"],
)

```

I'm not sure if the scopes is needed though
We faced same issue when using the stack Github Actions + Vertex AI + KFP v2. We were able to solve it by using Oauth2 token for authentication.

In github workflow we get access token:
```yaml
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v1'
        with:
          token_format: access_token
          workload_identity_provider: ....
          service_account: ....
```

We pass it to the step running the python code using by referring to `${{ steps.auth.outputs.access_token }}`.

The actual python code then is:
```python
from kfp.registry import RegistryClient, ApiAuth

client = RegistryClient(host=KFP_REGISTRY_URL, auth=ApiAuth(access_token))
template_name, version_name = client.upload_pipeline(
  file_name="pipeline.yaml",
  tags=["latest"],
)
```

In case of using WIF and service accounts to provide Github actions access to GCP, it seems RegistryClient cannot use default Google credentials from current environment (unlike Vertex AI SDK). OAuth2 token needs to be provided explicitly. However, when one is authenticated using personal GCP-account and runs the Python script from localhost it seems it works without OAuth2 token.
> We faced same issue when using the stack Github Actions + Vertex AI + KFP v2. We were able to solve it by using Oauth2 token for authentication.
> 
> In github workflow we get access token:
> 
> ```yaml
>       - id: 'auth'
>         name: 'Authenticate to Google Cloud'
>         uses: 'google-github-actions/auth@v1'
>         with:
>           token_format: access_token
>           workload_identity_provider: ....
>           service_account: ....
> ```
> 
> We pass it to the step running the python code using by referring to `${{ steps.auth.outputs.access_token }}`.
> 
> The actual python code then is:
> 
> ```python
> from kfp.registry import RegistryClient, ApiAuth
> 
> client = RegistryClient(host=KFP_REGISTRY_URL, auth=ApiAuth(access_token))
> template_name, version_name = client.upload_pipeline(
>   file_name="pipeline.yaml",
>   tags=["latest"],
> )
> ```
> 
> In case of using WIF and service accounts to provide Github actions access to GCP, it seems RegistryClient cannot use default Google credentials from current environment (unlike Vertex AI SDK). OAuth2 token needs to be provided explicitly. However, when one is authenticated using personal GCP-account and runs the Python script from localhost it seems it works without OAuth2 token.

Thank you so much for this. We are facing same and your experience has helped us
I'm facing the same problem. 
Wednesday was running without any problem.
Since yesterday I'm facing it, as decribed below.

I'm receiving the errors 
- `HTTPError: 400 Client Error: Bad Request for url` : when I try running without auth.
- `HTTPError: 401 Client Error: Unauthorized for url` : when I run with auth, using personal GCP account and with SA.

I tried changing kfp version to the latest, but the problem continued.

What should I do?
I would also like to bump this issue. 
Ideally, to run KFP in enterprise setting, it is important to support authentication with artifact registry for respective cloud provider.

In the case of google cloud, if KFP could authenticate with Google Artifact registry using KFP @dsl . It will be cleanest solution because during auth, it can check for the service account and it's permissions.

Otherwise, the only other option left is to build your own custom docker container and use it as the base image for KFP
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Closing this issue, there is a workaround documented and it's platform specific.

/close
@rimolive: Closing this issue.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/8878#issuecomment-2035005950):

>Closing this issue, there is a workaround documented and it's platform specific.
>
>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

Resolving patch:
diff --git a/sdk/python/kfp/registry/registry_client.py b/sdk/python/kfp/registry/registry_client.py
--- a/sdk/python/kfp/registry/registry_client.py
+++ b/sdk/python/kfp/registry/registry_client.py
@@ -189,7 +189,9 @@ def _load_auth(
             return auth
         elif self._is_ar_host():
             auth, _ = google.auth.default()
-            return auth
+            auth_scopes = self._config.get('auth_scopes')
+            auth_scopes = auth_scopes.split(',') if auth_scopes else None
+            return credentials.with_scopes_if_required(auth, auth_scopes)
         elif auth_file:
             if os.path.exists(auth_file):
                 # Fetch auth token using the locally stored credentials.


Test patch:
diff --git a/sdk/python/kfp/registry/registry_client_test.py b/sdk/python/kfp/registry/registry_client_test.py
--- a/sdk/python/kfp/registry/registry_client_test.py
+++ b/sdk/python/kfp/registry/registry_client_test.py
@@ -55,14 +55,10 @@ def test_load_config(self):
         host = _DEFAULT_HOST
         client = RegistryClient(host=host, auth=ApiAuth(''))
         expected_config = {
-            'host':
-                host,
-            'upload_url':
-                host,
-            'download_version_url':
-                f'{host}/{{package_name}}/{{version}}',
-            'download_tag_url':
-                f'{host}/{{package_name}}/{{tag}}',
+            'host': host,
+            'upload_url': host,
+            'download_version_url': f'{host}/{{package_name}}/{{version}}',
+            'download_tag_url': f'{host}/{{package_name}}/{{tag}}',
             'get_package_url':
                 ('https://artifactregistry.googleapis.com/v1/projects/'
                  'proj/locations/us-central1/repositories'
@@ -114,7 +110,8 @@ def test_load_config(self):
                            '/repo/packages/{package_name}/tags/{tag}'),
             'version_format':
                 ('projects/proj/locations/us-central1/repositories'
-                 '/repo/packages/{package_name}/versions/{version}')
+                 '/repo/packages/{package_name}/versions/{version}'),
+            'auth_scopes': 'https://www.googleapis.com/auth/cloud-platform'
         }
         self.assertEqual(self._mock_open.call_args_list[0][0],
                          (_KFP_CONFIG_FILE, 'r'))