Repository: skypilot-org/skypilot
Pull request: [UX] Make sky check more fine-grained for GCP
<!-- Describe the changes in this PR -->

Fixes #2184 

This PR makes the `sky check` provides information based on the status of the user's current setup.

Missing dependency:
```
$ sky check
Checking credentials to enable clouds for SkyPilot.
  AWS: enabled          
  Azure: enabled          
  GCP: disabled          
    Reason: GCP tools are not installed. Run the following commands:
      $ pip install google-api-python-client
      $ conda install -c conda-forge google-cloud-sdk -y
    Credentials may also need to be set. Run the following commands:
      $ gcloud init
      $ gcloud auth application-default login
    For more info: https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#google-cloud-platform-gcp
    Details: [builtins.ModuleNotFoundError] No module named 'googleapiclient'
  Lambda: enabled          
  IBM: enabled          
  SCP: enabled          
  OCI: enabled          
  Cloudflare (for R2 object store): enabled          

SkyPilot will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check.
```

Without `gcloud init`:
```
$ sky check                   
Checking credentials to enable clouds for SkyPilot.
  AWS: enabled          
  Azure: enabled          
  GCP: disabled          
    Reason: Credentails are not set. Run the following commands:
      $ gcloud init
      $ gcloud auth application-default login
    For more info: https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#google-cloud-platform-gcp
    Details: [builtins.FileNotFoundError] ~/.config/gcloud/access_tokens.db
  Lambda: enabled          
  IBM: enabled          
  SCP: enabled          
  OCI: enabled          
  Cloudflare (for R2 object store): enabled          

SkyPilot will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check.
If any problems remain, please file an issue at https://github.com/skypilot-org/skypilot/issues/new
```

Without `gcloud auth application-default login`:
```
$ sky check
Checking credentials to enable clouds for SkyPilot.
  AWS: enabled          
  Azure: enabled          
  GCP: disabled          
    Reason: Application credentials are not set. Run the following commands:
      $ gcloud auth application-default login
    Or set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of your service account key file.
    For more info: https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#google-cloud-platform-gcp
    Details: [builtins.FileNotFoundError] ~/.config/gcloud/application_default_credentials.json
  Lambda: enabled          
  IBM: enabled          
  SCP: enabled          
  OCI: enabled          
  Cloudflare (for R2 object store): enabled          

SkyPilot will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check.
If any problems remain, please file an issue at https://github.com/skypilot-org/skypilot/issues/new
```





<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
[UX] `sky check` should include fine grained error messages
From a user:

> .. when I ran `sky check` it said gcp was disabled--even though I have `gcloud` installed locally and connected to Google Cloud.  I ran a few of the commands, but when I got to `gcloud init` I didn't want to wipe my current settings.
> 
> After looking at the docs, what it really wanted me to do was run `gcloud auth application-default login`.  It would be nice to have more granular feedback about the checks that are run and what remains to be done, rather than installation instructions--especially incomplete ones.

Hints from issue discussion:
This is excellent feedback! Important for self-setup.

Commit message: Make sky check more fine-grained for GCP

Patch to review:
diff --git a/sky/clouds/gcp.py b/sky/clouds/gcp.py
index c9d44f54cf8..82422a437a0 100644
--- a/sky/clouds/gcp.py
+++ b/sky/clouds/gcp.py
@@ -114,6 +114,33 @@ class GCP(clouds.Cloud):
     # lower limit.
     _MAX_CLUSTER_NAME_LEN_LIMIT = 35
 
+    _INDENT_PREFIX = '    '
+    _DEPENDENCY_HINT = (
+        'GCP tools are not installed. Run the following commands:\n'
+        # Install the Google Cloud SDK:
+        f'{_INDENT_PREFIX}  $ pip install google-api-python-client\n'
+        f'{_INDENT_PREFIX}  $ conda install -c conda-forge '
+        'google-cloud-sdk -y')
+
+    _CREDENTIAL_HINT = (
+        'Run the following commands:\n'
+        # This authenticates the CLI to make `gsutil` work:
+        f'{_INDENT_PREFIX}  $ gcloud init\n'
+        # This will generate
+        # ~/.config/gcloud/application_default_credentials.json.
+        f'{_INDENT_PREFIX}  $ gcloud auth application-default login\n'
+        f'{_INDENT_PREFIX}For more info: '
+        'https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#google-cloud-platform-gcp'  # pylint: disable=line-too-long
+    )
+    _APPLICATION_CREDENTIAL_HINT = (
+        'Run the following commands:\n'
+        f'{_INDENT_PREFIX}  $ gcloud auth application-default login\n'
+        f'{_INDENT_PREFIX}Or set the environment variable GOOGLE_APPLICATION_CREDENTIALS '
+        'to the path of your service account key file.\n'
+        f'{_INDENT_PREFIX}For more info: '
+        'https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#google-cloud-platform-gcp'  # pylint: disable=line-too-long
+    )
+
     @classmethod
     def _cloud_unsupported_features(
             cls) -> Dict[clouds.CloudImplementationFeatures, str]:
@@ -503,6 +530,17 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
             # Check google-api-python-client installation.
             import googleapiclient
 
+            # Check the installation of google-cloud-sdk.
+            _run_output('gcloud --version')
+        except (ImportError, subprocess.CalledProcessError) as e:
+            return False, (
+                f'{cls._DEPENDENCY_HINT}\n'
+                f'{cls._INDENT_PREFIX}Credentials may also need to be set. '
+                f'{cls._CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
+
+        try:
             # These files are required because they will be synced to remote
             # VMs for `gsutil` to access private storage buckets.
             # `auth.default()` does not guarantee these files exist.
@@ -512,37 +550,36 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
             ]:
                 if not os.path.isfile(os.path.expanduser(file)):
                     raise FileNotFoundError(file)
+        except FileNotFoundError as e:
+            return False, (
+                f'Credentails are not set. '
+                f'{cls._CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
 
+        try:
             cls._find_application_key_path()
+        except FileNotFoundError as e:
+            return False, (
+                f'Application credentials are not set. '
+                f'{cls._APPLICATION_CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
 
-            # Check the installation of google-cloud-sdk.
-            _run_output('gcloud --version')
-
+        try:
             # Check if application default credentials are set.
             project_id = cls.get_project_id()
 
             # Check if the user is activated.
             identity = cls.get_current_user_identity()
         except (auth.exceptions.DefaultCredentialsError,
-                subprocess.CalledProcessError,
-                exceptions.CloudUserIdentityError, FileNotFoundError,
-                ImportError) as e:
+                exceptions.CloudUserIdentityError) as e:
             # See also: https://stackoverflow.com/a/53307505/1165051
             return False, (
-                'GCP tools are not installed or credentials are not set. '
-                'Run the following commands:\n    '
-                # Install the Google Cloud SDK:
-                '  $ pip install google-api-python-client\n    '
-                '  $ conda install -c conda-forge google-cloud-sdk -y\n    '
-                # This authenticates the CLI to make `gsutil` work:
-                '  $ gcloud init\n    '
-                # This will generate
-                # ~/.config/gcloud/application_default_credentials.json.
-                '  $ gcloud auth application-default login\n    '
-                'For more info: '
-                'https://skypilot.readthedocs.io/en/latest/getting-started/installation.html'  # pylint: disable=line-too-long
-                f'\nDetails: {common_utils.format_exception(e, use_bracket=True)}'
-            )
+                'Credentials are not correctly set. '
+                f'{cls._CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
 
         # Check APIs.
         apis = (


Code review comments:
diff_hunk: @@ -512,37 +550,36 @@ def check_credentials(cls) -> Tuple[bool, Optional[str]]:
             ]:
                 if not os.path.isfile(os.path.expanduser(file)):
                     raise FileNotFoundError(file)
+        except FileNotFoundError as e:
+            return False, (
+                f'Credentails are not set. '
+                f'{cls._CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
 
+        try:
             cls._find_application_key_path()
+        except FileNotFoundError as e:
+            return False, (
+                f'Application credentials are not set. '
+                f'{cls._APPLICATION_CREDENTIAL_HINT}\n'
+                f'{cls._INDENT_PREFIX}Details: '
+                f'{common_utils.format_exception(e, use_bracket=True)}')
 
-            # Check the installation of google-cloud-sdk.
-            _run_output('gcloud --version')
-
+        try:
             # Check if application default credentials are set.
             project_id = cls.get_project_id()
 
             # Check if the user is activated.
             identity = cls.get_current_user_identity()
         except (auth.exceptions.DefaultCredentialsError,
-                subprocess.CalledProcessError,
-                exceptions.CloudUserIdentityError, FileNotFoundError,
-                ImportError) as e:
+                exceptions.CloudUserIdentityError) as e:
             # See also: https://stackoverflow.com/a/53307505/1165051
             return False, (
-                'GCP tools are not installed or credentials are not set. '
-                'Run the following commands:\n    '
-                # Install the Google Cloud SDK:
-                '  $ pip install google-api-python-client\n    '
-                '  $ conda install -c conda-forge google-cloud-sdk -y\n    '
-                # This authenticates the CLI to make `gsutil` work:
-                '  $ gcloud init\n    '
-                # This will generate
-                # ~/.config/gcloud/application_default_credentials.json.
-                '  $ gcloud auth application-default login\n    '
-                'For more info: '
-                'https://skypilot.readthedocs.io/en/latest/getting-started/installation.html'  # pylint: disable=line-too-long
-                f'\nDetails: {common_utils.format_exception(e, use_bracket=True)}'
-            )
+                'Credentials are not correctly set. '
path: sky/clouds/gcp.py
text: @user1:
Can we also hint the CLI commands that are equivalent to L571 & 574? Something like "Getting project ID / current user failed. You can debug with "cmd1" and "cmd2". {_CREDENTIAL_HINT}"

@author:
Good point! Added. There is no exactly equivalent command for `auth.default()`, but this exception is a very rare case, if the previous checks pass. We only add the `gcloud auth list` command here, and can wait for more feedbacks if user encounter this issue. : )