Repository: skypilot-org/skypilot
Pull request: Fail early when spot cluster name too long occurs on GCP
Fixes #1178 

Tested:
- [x] `sky spot launch -n i-am-a-very-long-name-too-make-it-fail --cloud gcp 'sleep 10'`
- [x] `sky spot launch -n i-am-a-very-long-name-too-make-it-fail --cloud gcp 'sleep 10' --retry-until-up`

Problem statement:
spot vm provision keep retry when name is too long
Hi Sky team,

I specified a long name (which I should prevent in future) for spot VM, and the provision keeps retry. Obviously, this is not necessary as the criteria will keep failing.

Thanks!

```
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:25 storage.py:536] Storage type StoreType.GCS already exists.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:27 storage.py:536] Storage type StoreType.GCS already exists.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:29 storage.py:536] Storage type StoreType.GCS already exists.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:31 storage.py:536] Storage type StoreType.GCS already exists.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 controller.py:49] Started monitoring spot task worker-220929-motif-enrichment-test-0 (id: 49)
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 spot_state.py:108] Launching the spot cluster...
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:605] == Optimizer ==
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:628] Estimated cost: $0.4 / hour
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:628] 
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:685] Considered resources (1 node):
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713] ---------------------------------------------------------------------------
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713]  CLOUD   INSTANCE               vCPUs   ACCELERATORS   COST ($)   CHOSEN   
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713] ---------------------------------------------------------------------------
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713]  GCP     n2d-highmem-64[Spot]   64      -              0.37          ✔     
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713] ---------------------------------------------------------------------------
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:32 optimizer.py:713] 
(worker-220929-motif-enrichment-test-0 pid=3925) NAME                            LAUNCHED   RESOURCES                      STATUS  AUTOSTOP  COMMAND                       
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-4-38   1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-11-33  1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-14-36  1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-1-23   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-3-25   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-2-24   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-13-35  3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-15-37  3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-8-30   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-9-31   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-7-29   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-6-28   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-5-27   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-0-22   4 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) 
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:33 recovery_strategy.py:165] Failed to launch the spot cluster with error: Cluster name 'worker-220929-motif-enrichment-test-0-49' has 40 chars; maximum length is 35 chars.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:06:33 recovery_strategy.py:250] Retrying to launch the spot cluster in 81.7 seconds.
(worker-220929-motif-enrichment-test-0 pid=3925) NAME                            LAUNCHED   RESOURCES                      STATUS  AUTOSTOP  COMMAND                       
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-4-38   1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-11-33  1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-14-36  1 hr ago   1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-1-23   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-3-25   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-2-24   2 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-13-35  3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-15-37  3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-8-30   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-9-31   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-7-29   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-6-28   3 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-5-27   4 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) worker-220929-m3c-coolds-0-22   4 hrs ago  1x GCP(n2d-standard-48[Spot])  UP      -         /opt/conda/lib/python3.7/...  
(worker-220929-motif-enrichment-test-0 pid=3925) 
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:07:56 recovery_strategy.py:165] Failed to launch the spot cluster with error: Cluster name 'worker-220929-motif-enrichment-test-0-49' has 40 chars; maximum length is 35 chars.
(worker-220929-motif-enrichment-test-0 pid=3925) I 09-29 23:07:56 recovery_strategy.py:250] Retrying to launch the spot cluster in 179.7 seconds.
```

Hints from issue discussion:

Commit message: format

Patch to review:
diff --git a/sky/backends/backend_utils.py b/sky/backends/backend_utils.py
index 902d1f0f5d3..5eaf9edc9d7 100644
--- a/sky/backends/backend_utils.py
+++ b/sky/backends/backend_utils.py
@@ -1831,9 +1831,9 @@ def check_cluster_name_is_valid(cluster_name: str,
         if len(cluster_name) > _MAX_CLUSTER_NAME_LEN_FOR_GCP:
             with ux_utils.print_exception_no_traceback():
                 raise ValueError(
-                    f'Cluster name {cluster_name!r} has {len(cluster_name)}'
-                    f' chars; maximum length is {_MAX_CLUSTER_NAME_LEN_FOR_GCP}'
-                    ' chars.')
+                    f'ClusterNameTooLong: Cluster name {cluster_name!r} has '
+                    f'{len(cluster_name)} chars; maximum length is '
+                    f'{_MAX_CLUSTER_NAME_LEN_FOR_GCP} chars.')
 
 
 def check_cluster_name_not_reserved(
diff --git a/sky/spot/recovery_strategy.py b/sky/spot/recovery_strategy.py
index e9ad327f6e9..9c627db8412 100644
--- a/sky/spot/recovery_strategy.py
+++ b/sky/spot/recovery_strategy.py
@@ -161,8 +161,12 @@ def _launch(self, max_retry=3, raise_on_failure=True) -> Optional[float]:
             except Exception as e:  # pylint: disable=broad-except
                 # If the launch fails, it will be recovered by the following
                 # code.
-                logger.info(
-                    f'Failed to launch the spot cluster with error: {e}')
+                if (isinstance(e, ValueError) and
+                        'ClusterNameTooLong: ' in str(e)):
+                    # The cluster name is too long.
+                    raise exceptions.ResourcesUnavailableError(str(e)) from e
+                logger.info('Failed to launch the spot cluster with error: '
+                            f'{type(e)}: {e}')
                 retry_launch = True
                 exception = e
 


Code review comments:
diff_hunk: @@ -1831,9 +1831,9 @@ def check_cluster_name_is_valid(cluster_name: str,
         if len(cluster_name) > _MAX_CLUSTER_NAME_LEN_FOR_GCP:
             with ux_utils.print_exception_no_traceback():
                 raise ValueError(
-                    f'Cluster name {cluster_name!r} has {len(cluster_name)}'
-                    f' chars; maximum length is {_MAX_CLUSTER_NAME_LEN_FOR_GCP}'
-                    ' chars.')
+                    f'ClusterNameTooLong: Cluster name {cluster_name!r} has '
+                    f'{len(cluster_name)} chars; maximum length is '
path: sky/backends/backend_utils.py
text: @user1:
nit, not necessary to fix: I prefer creating a new exception class (e.g. `ClusterNameError`) when the error will be surfaced to the user and/or is meant to be caught elsewhere in the code. That way we don't have to use hardcoded strings to catch this specific exception and looks nicer to the user too :) 

@author:
Good point! I renamed the exceptions to `InvalideClusterNameError`.

Tested:
- [x] `sky spot launch -n i-am-a-very-long-name-too-make-it-fail --cloud gcp 'sleep 10'`
- [x] `sky spot launch -n i-am-a-very-long-name-too-make-it-fail --cloud gcp 'sleep 10' --retry-until-up`
- [x] `tests/run_smoke_tests.sh` (except Azure and TPU VM due to no resources available)