Repository: skypilot-org/skypilot
Pull request: [core] optimizer/provisioner fails over for cloud with expired credentials
<!-- Describe the changes in this PR -->
Fix #4373
- For Kubernetes with expired credentials, auto-exclude the contexts
- For invalid credentials of aws, azure, and gcp after enabled in cache, failover to the other clouds for provisioning 

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
  - Details are posted in separate comments 
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [x] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[Core] Expired credentials causes unexpected failure of `sky launch`
<!-- Describe the bug report / feature request here -->

Many cloud credentials have expiration, e.g. AWS SSO, or kubeconfig with gcloud authentication. Expiration of these credentials means, although the cloud is enabled in the cache, `sky launch` will go to those cloud and try to launch the cluster on those cloud causing unexpected failure and not automatically failover, e.g.:
```
Exception: ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7f9b7f8a-d526-44fc-ace8-5c12093e441f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 15 Nov 2024 20:40:38 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
```


<!-- If relevant, fill in versioning info to help us troubleshoot -->
_Version & Commit info:_
* `sky -v`: PLEASE_FILL_IN
* `sky -c`: PLEASE_FILL_IN

Hints from issue discussion:
Related [issue](https://github.com/skypilot-org/skypilot/issues/4360)
Root cause:
The enabled clouds list will be cached after `sky check`, while for the following operations, they only refresh the list if no clouds are enabled in the cache or refresh the ones not enabled in the cache.

Solution:
To fix this issue, we need to refresh the enabled clouds (in https://github.com/skypilot-org/skypilot/blob/master/sky/optimizer.py#L1229) in optimizer before getting the best plan.

@Michaelvll Is my understanding correct? 

We want to avoid refreshing the enabled clouds, as it's a pretty slow operation. It should be okay for us to proceed with the optimizer even if our information is outdated.

However, once we fail to launch the instance due to credential expiration, we should continue to fail over to the next cloud, instead of crashing completely or trying other regions in the same cloud.
> We want to avoid refreshing the enabled clouds, as it's a pretty slow operation. It should be okay for us to proceed with the optimizer even if our information is outdated.
> 
> However, once we fail to launch the instance due to credential expiration, we should continue to fail over to the next cloud, instead of crashing completely or trying other regions in the same cloud.

@cg505 OK, thanks for the info! I will fix with this solution.
Hi @Michaelvll @cg505, what about Kubernetes, take EKS for example, if there is no permission to access the EKS, the launch will fail at optimizer, I think we should fail in this case, right?

Here are some test cases and results:
```
1. Change API server endpoint in kubeconfig so that k8s cannot be accessed - failover to other clouds
(.venv) ➜  skypilot git:(4373) ✗ sky launch --cpus 1+ --dryrun

D 03-22 15:04:43 utils.py:161] Kubernetes API call get_kubernetes_nodes failed with HTTPSConnectionPool(host='127.0.0.1', port=5823): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x11b6d0220>: Failed to establish a new connection: [Errno 61] Connection refused')). Retrying in 1.0s...
D 03-22 15:04:44 utils.py:161] Kubernetes API call get_kubernetes_nodes failed with HTTPSConnectionPool(host='127.0.0.1', port=5823): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x11b6d07f0>: Failed to establish a new connection: [Errno 61] Connection refused')). Retrying in 1.8s...
I 03-22 15:04:46 kubernetes.py:203] Excluding Kubernetes context kind-kind: Timed out when trying to get node info from Kubernetes cluster. Please check if the cluster is healthy and retry. To debug, run: kubectl get nodes --context kind-kind
W 03-22 15:04:46 kubernetes.py:209] All Kubernetes contexts are unreachable. Retry if it is a transient error, or run sky check to refresh Kubernetes availability if permanent.
D 03-22 15:04:46 utils.py:161] Kubernetes API call get_kubernetes_nodes failed with HTTPSConnectionPool(host='127.0.0.1', port=5823): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x11b6e7550>: Failed to establish a new connection: [Errno 61] Connection refused')). Retrying in 0.9s...
D 03-22 15:04:47 utils.py:161] Kubernetes API call get_kubernetes_nodes failed with HTTPSConnectionPool(host='127.0.0.1', port=5823): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x11f88b790>: Failed to establish a new connection: [Errno 61] Connection refused')). Retrying in 0.9s...
D 03-22 15:04:55 optimizer.py:302] #### Task<name=sky-cmd>(run=<empty>)
D 03-22 15:04:55 optimizer.py:302]   resources: <Cloud>(cpus=1+) ####
D 03-22 15:04:55 optimizer.py:317] Defaulting the task's estimated time to 1 hour.
D 03-22 15:04:55 optimizer.py:339] resources: AWS(m6i.large)
...
I 03-22 15:04:55 optimizer.py:954] ----------------------------------------------------------------------------------------------
I 03-22 15:04:55 optimizer.py:954]  CLOUD   INSTANCE          vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN
I 03-22 15:04:55 optimizer.py:954] ----------------------------------------------------------------------------------------------
I 03-22 15:04:55 optimizer.py:954]  AWS     m6i.large         2       8         -              us-east-1     0.10          ✔
I 03-22 15:04:55 optimizer.py:954]  Azure   Standard_D2s_v5   2       8         -              eastus        0.10
I 03-22 15:04:55 optimizer.py:954] ----------------------------------------------------------------------------------------------
Launching a new cluster 'sky-dce6-danz'. Proceed? [Y/n]: n

2. Create an EKS first, then update the aws config with expired credentials - launch fails at the optimizer

(.venv) ➜  skypilot git:(4373) ✗ sky launch --cpus 1+ --dryrun

D 03-22 15:45:53 sdk.py:1449] Got request with error: sky.optimize
E 03-22 15:45:53 sdk.py:1461] === Traceback on SkyPilot API Server ===
E 03-22 15:45:53 sdk.py:1461] kubernetes error (ApiException): (401)
E 03-22 15:45:53 sdk.py:1461] Reason: Unauthorized
E 03-22 15:45:53 sdk.py:1461] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f27a3280-6c2c-4203-abc8-b73c0ac5bb05', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 21 Mar 2025 07:45:53 GMT', 'Content-Length': '129'})
E 03-22 15:45:53 sdk.py:1461] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
E 03-22 15:45:53 sdk.py:1461]
E 03-22 15:45:53 sdk.py:1461]
D 03-22 15:45:53 sdk.py:77] To stream request logs: sky api logs 748237b1-aafd-42ed-8c39-2fcf1c60d595
sky.exceptions.CloudError: kubernetes error (ApiException): (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f27a3280-6c2c-4203-abc8-b73c0ac5bb05', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 21 Mar 2025 07:45:53 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

3. Launch with aws VM with expired credentials - launch fails at provisioner and no failover

E 03-22 15:55:38 utils.py:64] Failed to fetch IAM instance profile data for skypilot-v1 from AWS.
E 03-22 15:55:38 utils.py:64] Error code: ExpiredToken
C 03-22 15:55:38 utils.py:67] Your AWS session has expired.
C 03-22 15:55:38 utils.py:67] You can request a new one using aws sts get-session-token --serial-number arn:aws:iam::ROOT_ACCOUNT_ID:mfa/AWS_USERNAME --token-code TWO_FACTOR_AUTH_CODE then expose it to SkyPilot by setting export AWS_SECRET_ACCESS_KEY = REPLACE_ME # found at Credentials.SecretAccessKey export AWS_SESSION_TOKEN = REPLACE_ME # found at Credentials.SessionToken export AWS_ACCESS_KEY_ID = REPLACE_ME # found at Credentials.AccessKeyId
C 03-22 15:55:38 utils.py:67] You can find a script that automates this at:https://gist.github.com/maximsmol/a0284e1d97b25d417bd9ae02e5f450cf
```

So case 1 works as expected, case 3 should be fixed according to the above discussion, what about case 2? Should we keep the same behavior as case 1?
FYI, #5015 keeps the same behavior for both case 1 and case 2.

Commit message: address comments

Patch to review:
diff --git a/sky/adaptors/gcp.py b/sky/adaptors/gcp.py
index a12a41d07c7..6ed2ee5e56a 100644
--- a/sky/adaptors/gcp.py
+++ b/sky/adaptors/gcp.py
@@ -68,6 +68,13 @@ def credential_error_exception():
     return exceptions.DefaultCredentialsError
 
 
+@common.load_lazy_modules(_LAZY_MODULES)
+def auth_error_exception():
+    """GoogleAuthError exception."""
+    from google.auth import exceptions
+    return exceptions.GoogleAuthError
+
+
 @common.load_lazy_modules(_LAZY_MODULES)
 def gcp_auth_refresh_error_exception():
     """GCP auth refresh error exception."""
diff --git a/sky/authentication.py b/sky/authentication.py
index 6c852778a56..bda53a50127 100644
--- a/sky/authentication.py
+++ b/sky/authentication.py
@@ -33,6 +33,7 @@
 import filelock
 
 from sky import clouds
+from sky import exceptions
 from sky import sky_logging
 from sky import skypilot_config
 from sky.adaptors import common as adaptors_common
@@ -214,6 +215,11 @@ def setup_gcp_authentication(config: Dict[str, Any]) -> Dict[str, Any]:
             sys.exit(1)
         else:
             raise
+    except gcp.auth_error_exception() as e:
+        logger.error(
+            f'Error getting GCP project: {common_utils.format_exception(e)}')
+        raise exceptions.InvalidCloudCredentials(
+            f'{common_utils.format_exception(e)}')
     except socket.timeout:
         logger.error('Socket timed out when trying to get the GCP project. '
                      'Please check your network connection.')
diff --git a/sky/backends/backend_utils.py b/sky/backends/backend_utils.py
index 1e642a35fd3..efce8f2a331 100644
--- a/sky/backends/backend_utils.py
+++ b/sky/backends/backend_utils.py
@@ -815,7 +815,11 @@ def write_cluster_config(
             logger.warning(f'Failed to calculate config_hash: {e}')
             logger.debug('Full exception:', exc_info=e)
         return config_dict
-    _add_auth_to_cluster_config(cloud, tmp_yaml_path)
+
+    try:
+        _add_auth_to_cluster_config(cloud, tmp_yaml_path)
+    except exceptions.InvalidCloudCredentials as e:
+        raise e
 
     # Restore the old yaml content for backward compatibility.
     if os.path.exists(yaml_path) and keep_launch_fields_in_existing_config:
diff --git a/sky/backends/cloud_vm_ray_backend.py b/sky/backends/cloud_vm_ray_backend.py
index bde455175cb..c682d086e21 100644
--- a/sky/backends/cloud_vm_ray_backend.py
+++ b/sky/backends/cloud_vm_ray_backend.py
@@ -1089,6 +1089,22 @@ def _lambda_handler(blocked_resources: Set['resources_lib.Resources'],
             FailoverCloudErrorHandlerV2._default_handler(
                 blocked_resources, launchable_resources, region, zones, error)
 
+    @staticmethod
+    def _aws_handler(blocked_resources: Set['resources_lib.Resources'],
+                     launchable_resources: 'resources_lib.Resources',
+                     region: 'clouds.Region',
+                     zones: Optional[List['clouds.Zone']],
+                     error: Exception) -> None:
+        output = str(error)
+        logger.info(f'AWS handler error: {output}')
+        # Block AWS if the credential has expired.
+        if output.find('InvalidCloudCredentials') != -1:
+            _add_to_blocked_resources(
+                blocked_resources, resources_lib.Resources(cloud=clouds.AWS()))
+        else:
+            FailoverCloudErrorHandlerV2._default_handler(
+                blocked_resources, launchable_resources, region, zones, error)
+
     @staticmethod
     def _default_handler(blocked_resources: Set['resources_lib.Resources'],
                          launchable_resources: 'resources_lib.Resources',
@@ -1419,6 +1435,17 @@ def _retry_zones(
                 # does not have nodes labeled with GPU types.
                 logger.info(f'{e}')
                 continue
+            except exceptions.InvalidCloudCredentials as e:
+                # Failed due to invalid cloud credentials.
+                logger.warning(f'{common_utils.format_exception(e)}')
+                # We should block the entire cloud for invalid cloud credentials
+                _add_to_blocked_resources(
+                    self._blocked_resources,
+                    to_provision.copy(region=None, zone=None))
+                raise exceptions.ResourcesUnavailableError(
+                    f'Failed to provision on cloud {to_provision.cloud} due to '
+                    f'invalid cloud credentials: '
+                    f'{common_utils.format_exception(e)}')
             except exceptions.InvalidCloudConfigs as e:
                 # Failed due to invalid user configs in ~/.sky/config.yaml.
                 logger.warning(f'{common_utils.format_exception(e)}')
diff --git a/sky/exceptions.py b/sky/exceptions.py
index 99aa11ff972..5e21b5a5dae 100644
--- a/sky/exceptions.py
+++ b/sky/exceptions.py
@@ -180,6 +180,11 @@ class InvalidCloudConfigs(Exception):
     pass
 
 
+class InvalidCloudCredentials(Exception):
+    """Raised when the cloud credentials are invalid."""
+    pass
+
+
 class ProvisionPrechecksError(Exception):
     """Raised when a managed job fails prechecks before provision.
 
diff --git a/sky/provision/aws/utils.py b/sky/provision/aws/utils.py
index 2a299933ca3..d8e84e7fad2 100644
--- a/sky/provision/aws/utils.py
+++ b/sky/provision/aws/utils.py
@@ -1,6 +1,7 @@
 """Utils for AWS provisioner."""
 import colorama
 
+from sky import exceptions
 from sky import sky_logging
 
 logger = sky_logging.init_logger(__name__)
@@ -73,9 +74,10 @@ def handle_boto_error(exc: Exception, msg: str) -> None:
             f'{colorama.Style.RESET_ALL}\n'
             f'You can find a script that automates this at:'
             f'{aws_session_script_url}')
-        # Do not re-raise the exception here because it looks awful
-        # and we already print all the info in verbose
-        raise SystemExit(1)
+        # Raise the InvalidCloudCredentials exception so that
+        # the provisioner can failover to other clouds
+        raise exceptions.InvalidCloudCredentials(
+            f'InvalidCloudCredentials: {generic_message}') from exc
 
     # todo: any other errors that we should catch separately?
 
diff --git a/sky/provision/azure/config.py b/sky/provision/azure/config.py
index e7ab59daa33..b00a532f1ae 100644
--- a/sky/provision/azure/config.py
+++ b/sky/provision/azure/config.py
@@ -120,7 +120,7 @@ def bootstrap_instances(
             except azure.exceptions().ClientAuthenticationError as e:
                 message = (
                     'Failed to authenticate with Azure. Please check your '
-                    'Azure credentials. Error: '
+                    'Azure credentials. ClientAuthenticationError: '
                     f'{common_utils.format_exception(e)}').replace('\n', ' ')
                 logger.error(message)
                 raise exceptions.NoClusterLaunchedError(message) from e
diff --git a/sky/provision/kubernetes/utils.py b/sky/provision/kubernetes/utils.py
index 33784ca15f5..82f4468df3e 100644
--- a/sky/provision/kubernetes/utils.py
+++ b/sky/provision/kubernetes/utils.py
@@ -159,7 +159,10 @@ def wrapper(*args, **kwargs):
                     # or 403 (Forbidden)
                     if (isinstance(e, kubernetes.api_exception()) and
                             e.status in (401, 403)):
-                        raise
+                        # Raise KubeAPIUnreachableError exception so that the
+                        # optimizer/provisioner can failover to other clouds.
+                        raise exceptions.KubeAPIUnreachableError(
+                            f'Kubernetes API error: {str(e)}') from e
                     if attempt < max_retries - 1:
                         sleep_time = backoff.current_backoff()
                         logger.debug(f'Kubernetes API call {func.__name__} '
diff --git a/sky/provision/provisioner.py b/sky/provision/provisioner.py
index d094d4537c0..b88c6931327 100644
--- a/sky/provision/provisioner.py
+++ b/sky/provision/provisioner.py
@@ -145,6 +145,10 @@ def bulk_provision(
         except exceptions.NoClusterLaunchedError:
             # Skip the teardown if the cluster was never launched.
             raise
+        except exceptions.InvalidCloudCredentials:
+            # Skip the teardown if the cloud config is expired and
+            # the provisioner should failover to other clouds.
+            raise
         except Exception:  # pylint: disable=broad-except
             zone_str = 'all zones'
             if zones:
diff --git a/tests/unit_tests/kubernetes/test_kubernetes_utils.py b/tests/unit_tests/kubernetes/test_kubernetes_utils.py
new file mode 100644
index 00000000000..606f3b1ba3a
--- /dev/null
+++ b/tests/unit_tests/kubernetes/test_kubernetes_utils.py
@@ -0,0 +1,21 @@
+"""Tests for Kubernetes utils.
+
+"""
+
+from unittest.mock import patch
+
+import kubernetes
+import pytest
+
+from sky import exceptions
+from sky.provision.kubernetes import utils
+
+
+# Test for exception on permanent errors like 401 (Unauthorized)
+def test_get_kubernetes_nodes():
+    with patch('sky.provision.kubernetes.utils.kubernetes.core_api'
+              ) as mock_core_api:
+        mock_core_api.return_value.list_node.side_effect = kubernetes.client.rest.ApiException(
+            status=401)
+        with pytest.raises(exceptions.KubeAPIUnreachableError):
+            utils.get_kubernetes_nodes(context='test')
diff --git a/tests/unit_tests/test_authentication.py b/tests/unit_tests/test_authentication.py
new file mode 100644
index 00000000000..193eee8fb3c
--- /dev/null
+++ b/tests/unit_tests/test_authentication.py
@@ -0,0 +1,32 @@
+"""Tests the functions in authentication.py.
+
+"""
+
+from unittest.mock import patch
+
+from google.auth import exceptions as google_exceptions
+import pytest
+
+from sky import authentication as auth
+from sky import exceptions
+
+
+def test_setup_gcp_authentication():
+    # Create a mock config with required fields
+    mock_config = {
+        'provider': {
+            'project_id': 'test-project'
+        },
+        'gcp_credentials': {
+            'type': 'service_account',
+            'credentials': '{"type": "service_account", "project_id": "test-project"}'
+        }
+    }
+
+    with patch('sky.adaptors.gcp.build') as mock_build:
+        # Mock the compute API response to raise RefreshError
+        mock_build.return_value.projects.return_value.get.return_value.execute.side_effect = (
+            google_exceptions.RefreshError('test'))
+
+        with pytest.raises(exceptions.InvalidCloudCredentials):
+            auth.setup_gcp_authentication(mock_config)
diff --git a/tests/unit_tests/test_aws_utils.py b/tests/unit_tests/test_aws_utils.py
new file mode 100644
index 00000000000..861cf92ac97
--- /dev/null
+++ b/tests/unit_tests/test_aws_utils.py
@@ -0,0 +1,29 @@
+"""Tests for AWS utils.
+
+"""
+
+import pytest
+
+from sky import exceptions
+from sky.adaptors import aws
+from sky.provision.aws import utils
+
+
+def test_handle_boto_error():
+    # Create a mock error response
+    error_response = {
+        'Error': {
+            'Code': 'ExpiredToken',
+            'Message': 'Token expired',
+            'Type': 'Sender'
+        },
+        'ResponseMetadata': {
+            'RequestId': '123456-7890',
+            'HTTPStatusCode': 400
+        }
+    }
+
+    with pytest.raises(exceptions.InvalidCloudCredentials):
+        utils.handle_boto_error(
+            aws.botocore_exceptions().ClientError(error_response, 'test'),
+            'test')


Code review comments:
diff_hunk: @@ -1089,6 +1089,22 @@ def _lambda_handler(blocked_resources: Set['resources_lib.Resources'],
             FailoverCloudErrorHandlerV2._default_handler(
                 blocked_resources, launchable_resources, region, zones, error)
 
+    @staticmethod
+    def _aws_handler(blocked_resources: Set['resources_lib.Resources'],
+                     launchable_resources: 'resources_lib.Resources',
+                     region: 'clouds.Region',
+                     zones: Optional[List['clouds.Zone']],
+                     error: Exception) -> None:
+        output = str(error)
+        logger.info(f'AWS handler error: {output}')
+        # Block AWS if the credential has expired.
+        if output.find('InvalidCloudCredentials') != -1:
path: sky/backends/cloud_vm_ray_backend.py
text: @user1:
```suggestion
        logger.info(f'AWS handler error: {error}')
        # Block AWS if the credential has expired.
        if isinstance(error, exceptions.InvalidCloudCredentials):
```
But I'm still not sure we need this handler at all.

@author:
Updated and replied here for why we need this handler https://github.com/skypilot-org/skypilot/pull/5015#discussion_r2013289389

diff_hunk: @@ -815,7 +815,11 @@ def write_cluster_config(
             logger.warning(f'Failed to calculate config_hash: {e}')
             logger.debug('Full exception:', exc_info=e)
         return config_dict
-    _add_auth_to_cluster_config(cloud, tmp_yaml_path)
+
+    try:
+        _add_auth_to_cluster_config(cloud, tmp_yaml_path)
+    except exceptions.InvalidCloudCredentials as e:
+        raise e
path: sky/backends/backend_utils.py
text: @user1:
I think we can revert this change entirely now, since we don't catch the other error type.

@author:
Updated

diff_hunk: @@ -1089,6 +1089,22 @@ def _lambda_handler(blocked_resources: Set['resources_lib.Resources'],
             FailoverCloudErrorHandlerV2._default_handler(
                 blocked_resources, launchable_resources, region, zones, error)
 
+    @staticmethod
+    def _aws_handler(blocked_resources: Set['resources_lib.Resources'],
+                     launchable_resources: 'resources_lib.Resources',
+                     region: 'clouds.Region',
+                     zones: Optional[List['clouds.Zone']],
+                     error: Exception) -> None:
+        output = str(error)
+        logger.info(f'AWS handler error: {output}')
+        # Block AWS if the credential has expired.
+        if output.find('InvalidCloudCredentials') != -1:
path: sky/backends/cloud_vm_ray_backend.py
text: @user1:
```suggestion
        if output.find('InvalidCloudCredentials') != -1:
        if 'InvalidCloudCredentials' in output:
```
Maybe more narrative

@author:
Updated