Repository: skypilot-org/skypilot
Pull request: Change added `USER_HASH` in models name to full digits (8) instead of half (4) to prevent potential hash conflict
<!-- Describe the changes in this PR -->

Resolves #4143

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [x] Backward compatibility tests: `conda deactivate; bash -i tests/backward_compatibility_tests.sh`

![Uploading image.png…]()



Problem statement:
[BUG][AWS] Serving a new controller can result in terminating previously served models.
When these conditions are true, a new controller may terminate existing resources being served by another controller.
* User id hash matches last 4 characters, but has a difference otherwise.
* Service name is equivalent


This may results in a model node's `cluster_name` with the same name as an existing model node depending on version.
e.g. 

-----

Existing controller --
USER_ID_HASH=12345678
Controller node: `sky-sky-serve-controller-12345678-5678-head`
model node:       `sky-<SERVICE_NAME>-<VERSION>-5678-head`

-----

New Controller --
USER_ID_HASH=11115678
Controller node: `sky-sky-serve-controller-11115678-5678-head`
model node:       `sky-<SERVICE_NAME>-<VERSION>-5678-head`

-----

In this case, if the `<VERSION>` matches, the existing may get terminated.

I believe this results from the filter in terminate only looking for the Name as opposed:
```
return [{
    'Name': f'tag:{constants.TAG_RAY_CLUSTER_NAME}',
    'Values': [cluster_name_on_cloud],
}]
```

I'm not sure if this widespread across other deployment platforms.

In AWS, this could be resolved by including a new TAG in addition the the name that specifies the correct controller/cluster association and using that in the filter in as well.

Possible tags:
* USER_ID_HASH (easy fix)
* UUID for every controller / model associated with that controller (needs to be stored in the db)

Hints from issue discussion:
Hi @JGSweets ! Thanks for reporting this error. Just want to make sure, this is for multiple users (in multiple laptops) running SkyPilot in a shared AWS project?
> Hi @JGSweets ! Thanks for reporting this error. Just want to make sure, this is for multiple users (in multiple laptops) running SkyPilot in a shared AWS project?

@cblmemo  Actually, this case would be for a single compute resource where `SKYPILOT_USER_ID` is set via environment variables. I would be inclined to believe that multiple laptops would have a similar effect.
Just want to bump this issue. 

Could we put a controller specific uuid tag on the resource that associates it with the proper controller? 
This would allow us to filter which resources are being controlled by it.

Or is that not feasible for each cloud provider?

Commit message: style: format and type hints

Patch to review:
diff --git a/sky/backends/backend_utils.py b/sky/backends/backend_utils.py
index 7385c7cb9fc..28e6d1ce0f0 100644
--- a/sky/backends/backend_utils.py
+++ b/sky/backends/backend_utils.py
@@ -997,7 +997,7 @@ def write_cluster_config(
 
     # Read the cluster name from the tmp yaml file, to take the backward
     # compatbility restortion above into account.
-    # TODO: remove this after 2 minor releases, 0.8.0.
+    # TODO: remove this after 2 minor releases, 0.10.0.
     yaml_config = common_utils.read_yaml(tmp_yaml_path)
     config_dict['cluster_name_on_cloud'] = yaml_config['cluster_name']
 
diff --git a/sky/jobs/dashboard/dashboard.py b/sky/jobs/dashboard/dashboard.py
index b5b6ff39613..085a6632b0c 100644
--- a/sky/jobs/dashboard/dashboard.py
+++ b/sky/jobs/dashboard/dashboard.py
@@ -94,7 +94,7 @@ class JobTableColumns(enum.IntEnum):
 
 def _extract_launch_history(log_content: str) -> str:
     """Extract launch history from log content.
-    
+
     Args:
         log_content: Content of the log file.
     Returns:
diff --git a/sky/utils/common_utils.py b/sky/utils/common_utils.py
index ee8f5cf7bec..71d0fac1603 100644
--- a/sky/utils/common_utils.py
+++ b/sky/utils/common_utils.py
@@ -28,7 +28,6 @@
 
 _USER_HASH_FILE = os.path.expanduser('~/.sky/user_hash')
 USER_HASH_LENGTH = 8
-USER_HASH_LENGTH_IN_CLUSTER_NAME = 4
 
 # We are using base36 to reduce the length of the hash. 2 chars -> 36^2 = 1296
 # possibilities. considering the final cluster name contains the prefix as well,
@@ -98,10 +97,12 @@ def _is_valid_user_hash(user_hash: Optional[str]) -> bool:
             return user_hash
 
     hash_str = user_and_hostname_hash()
-    user_hash = hashlib.md5(hash_str.encode()).hexdigest()[:USER_HASH_LENGTH]
-    if not _is_valid_user_hash(user_hash):
+    hex_hash = hashlib.md5(hash_str.encode()).hexdigest()[:USER_HASH_LENGTH]
+    if not _is_valid_user_hash(hex_hash):
         # A fallback in case the hash is invalid.
-        user_hash = uuid.uuid4().hex[:USER_HASH_LENGTH]
+        hex_hash = uuid.uuid4().hex[:USER_HASH_LENGTH]
+    user_hash = base36_encode(hex_hash)
+
     os.makedirs(os.path.dirname(_USER_HASH_FILE), exist_ok=True)
     if not force_fresh_hash:
         # Do not cache to file if force_fresh_hash is True since the file may
@@ -182,7 +183,7 @@ def make_cluster_name_on_cloud(display_name: str,
             f'on the cloud, we convert it to {cluster_name_on_cloud}.')
     user_hash = ''
     if add_user_hash:
-        user_hash = get_user_hash()[:USER_HASH_LENGTH_IN_CLUSTER_NAME]
+        user_hash = get_user_hash()
         user_hash = f'-{user_hash}'
     user_hash_length = len(user_hash)
 
diff --git a/tests/smoke_tests/smoke_tests_utils.py b/tests/smoke_tests/smoke_tests_utils.py
index 9b9b7584ef7..2469a193b35 100644
--- a/tests/smoke_tests/smoke_tests_utils.py
+++ b/tests/smoke_tests/smoke_tests_utils.py
@@ -4,7 +4,7 @@
 import subprocess
 import sys
 import tempfile
-from typing import Dict, List, NamedTuple, Optional, Tuple
+from typing import Dict, List, NamedTuple, Optional, Sequence
 import uuid
 
 import colorama
@@ -32,7 +32,7 @@
 
 STORAGE_SETUP_COMMANDS = [
     'touch ~/tmpfile', 'mkdir -p ~/tmp-workdir',
-    'touch ~/tmp-workdir/tmp\ file', 'touch ~/tmp-workdir/tmp\ file2',
+    r'touch ~/tmp-workdir/tmp\ file', r'touch ~/tmp-workdir/tmp\ file2',
     'touch ~/tmp-workdir/foo',
     '[ ! -e ~/tmp-workdir/circle-link ] && ln -s ~/tmp-workdir/ ~/tmp-workdir/circle-link || true',
     'touch ~/.ssh/id_rsa.pub'
@@ -55,7 +55,7 @@
     [status.value for status in sky.ManagedJobStatus])
 
 
-def _statuses_to_str(statuses: List[enum.Enum]):
+def _statuses_to_str(statuses: Sequence[enum.Enum]):
     """Convert a list of enums to a string with all the values separated by |."""
     assert len(statuses) > 0, 'statuses must not be empty'
     if len(statuses) > 1:
@@ -74,8 +74,8 @@ def _statuses_to_str(statuses: List[enum.Enum]):
     'fi; '
     'current_status=$(sky status {cluster_name} --refresh | '
     'awk "/^{cluster_name}/ '
-    '{{for (i=1; i<=NF; i++) if (\$i ~ /^(' + _ALL_CLUSTER_STATUSES +
-    ')$/) print \$i}}"); '
+    r'{{for (i=1; i<=NF; i++) if (\$i ~ /^(' + _ALL_CLUSTER_STATUSES +
+    r')$/) print \$i}}"); '
     'if [[ "$current_status" =~ {cluster_status} ]]; '
     'then echo "Target cluster status {cluster_status} reached."; break; fi; '
     'echo "Waiting for cluster status to become {cluster_status}, current status: $current_status"; '
@@ -136,8 +136,8 @@ def get_cmd_wait_until_cluster_is_not_found(cluster_name: str, timeout: int):
     'fi; '
     'current_status=$(sky queue {cluster_name} | '
     'awk "\\$1 == \\"{job_id}\\" '
-    '{{for (i=1; i<=NF; i++) if (\$i ~ /^(' + _ALL_JOB_STATUSES +
-    ')$/) print \$i}}"); '
+    r'{{for (i=1; i<=NF; i++) if (\$i ~ /^(' + _ALL_JOB_STATUSES +
+    r')$/) print \$i}}"); '
     'found=0; '  # Initialize found variable outside the loop
     'while read -r line; do '  # Read line by line
     '  if [[ "$line" =~ {job_status} ]]; then '  # Check each line
@@ -196,7 +196,8 @@ def get_cmd_wait_until_job_status_contains_matching_job_name(
 
 
 def get_cmd_wait_until_managed_job_status_contains_matching_job_name(
-        job_name: str, job_status: List[sky.JobStatus], timeout: int):
+        job_name: str, job_status: Sequence[sky.ManagedJobStatus],
+        timeout: int):
     return _WAIT_UNTIL_MANAGED_JOB_STATUS_CONTAINS_MATCHING_JOB_NAME.format(
         job_name=job_name,
         job_status=_statuses_to_str(job_status),
@@ -215,7 +216,7 @@ class Test(NamedTuple):
     # Timeout for each command in seconds.
     timeout: int = DEFAULT_CMD_TIMEOUT
     # Environment variables to set for each command.
-    env: Dict[str, str] = None
+    env: Optional[Dict[str, str]] = None
 
     def echo(self, message: str):
         # pytest's xdist plugin captures stdout; print to stderr so that the
@@ -254,7 +255,7 @@ def terminate_gcp_replica(name: str, zone: str, replica_id: int) -> str:
             f' --quiet $({query_cmd})')
 
 
-def run_one_test(test: Test) -> Tuple[int, str, str]:
+def run_one_test(test: Test) -> None:
     # Fail fast if `sky` CLI somehow errors out.
     subprocess.run(['sky', 'status'], stdout=subprocess.DEVNULL, check=True)
     log_to_stdout = os.environ.get('LOG_TO_STDOUT', None)
diff --git a/tests/smoke_tests/test_managed_job.py b/tests/smoke_tests/test_managed_job.py
index 7c6a94f88cf..435cee49c70 100644
--- a/tests/smoke_tests/test_managed_job.py
+++ b/tests/smoke_tests/test_managed_job.py
@@ -77,7 +77,7 @@ def test_managed_jobs_basic(generic_cloud: str):
             # Test the functionality for logging.
             f's=$(sky jobs logs -n {name}-2 --no-follow); echo "$s"; echo "$s" | grep "start counting"',
             f's=$(sky jobs logs --controller -n {name}-2 --no-follow); echo "$s"; echo "$s" | grep "Cluster launched:"',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep {name}-2 | head -n1 | grep "RUNNING\|SUCCEEDED"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep {name}-2 | head -n1 | grep "RUNNING\|SUCCEEDED"',
         ],
         # TODO(zhwu): Change to f'sky jobs cancel -y -n {name}-1 -n {name}-2' when
         # canceling multiple job names is supported.
@@ -105,19 +105,19 @@ def test_job_pipeline(generic_cloud: str):
         [
             f'sky jobs launch -n {name} tests/test_yamls/pipeline.yaml --cloud {generic_cloud} -y -d',
             'sleep 5',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep {name} | head -n1 | grep "STARTING\|RUNNING"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep {name} | head -n1 | grep "STARTING\|RUNNING"',
             # `grep -A 4 {name}` finds the job with {name} and the 4 lines
             # after it, i.e. the 4 tasks within the job.
             # `sed -n 2p` gets the second line of the 4 lines, i.e. the first
             # task within the job.
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 2p | grep "STARTING\|RUNNING"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 2p | grep "STARTING\|RUNNING"',
             f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 3p | grep "PENDING"',
             f'sky jobs cancel -y -n {name}',
             'sleep 5',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 2p | grep "CANCELLING\|CANCELLED"',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 3p | grep "CANCELLING\|CANCELLED"',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 4p | grep "CANCELLING\|CANCELLED"',
-            f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 5p | grep "CANCELLING\|CANCELLED"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 2p | grep "CANCELLING\|CANCELLED"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 3p | grep "CANCELLING\|CANCELLED"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 4p | grep "CANCELLING\|CANCELLED"',
+            rf'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 5p | grep "CANCELLING\|CANCELLED"',
             'sleep 200',
             f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 2p | grep "CANCELLED"',
             f'{smoke_tests_utils.GET_JOB_QUEUE} | grep -A 4 {name}| sed -n 3p | grep "CANCELLED"',
@@ -212,7 +212,7 @@ def test_managed_jobs_recovery_aws(aws_config_region):
     test = smoke_tests_utils.Test(
         'managed_jobs_recovery_aws',
         [
-            f'sky jobs launch --cloud aws --region {region} --use-spot -n {name} "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800"  -y -d',
+            rf'sky jobs launch --cloud aws --region {region} --use-spot -n {name} "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -258,7 +258,7 @@ def test_managed_jobs_recovery_gcp():
     test = smoke_tests_utils.Test(
         'managed_jobs_recovery_gcp',
         [
-            f'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot --cpus 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800"  -y -d',
+            rf'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot --cpus 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -288,14 +288,13 @@ def test_managed_jobs_pipeline_recovery_aws(aws_config_region):
     """Test managed job recovery for a pipeline."""
     name = smoke_tests_utils.get_cluster_name()
     user_hash = common_utils.get_user_hash()
-    user_hash = user_hash[:common_utils.USER_HASH_LENGTH_IN_CLUSTER_NAME]
     region = aws_config_region
     if region != 'us-east-2':
         pytest.skip('Only run spot pipeline recovery test in us-east-2')
     test = smoke_tests_utils.Test(
         'managed_jobs_pipeline_recovery_aws',
         [
-            f'sky jobs launch -n {name} tests/test_yamls/pipeline_aws.yaml  -y -d',
+            f'sky jobs launch -n {name} tests/test_yamls/pipeline_aws.yaml -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -342,7 +341,6 @@ def test_managed_jobs_pipeline_recovery_gcp():
     name = smoke_tests_utils.get_cluster_name()
     zone = 'us-east4-b'
     user_hash = common_utils.get_user_hash()
-    user_hash = user_hash[:common_utils.USER_HASH_LENGTH_IN_CLUSTER_NAME]
     query_cmd = (
         'gcloud compute instances list --filter='
         f'"(labels.ray-cluster-name:*-${{MANAGED_JOB_ID}}-{user_hash})" '
@@ -352,7 +350,7 @@ def test_managed_jobs_pipeline_recovery_gcp():
     test = smoke_tests_utils.Test(
         'managed_jobs_pipeline_recovery_gcp',
         [
-            f'sky jobs launch -n {name} tests/test_yamls/pipeline_gcp.yaml  -y -d',
+            f'sky jobs launch -n {name} tests/test_yamls/pipeline_gcp.yaml -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -426,7 +424,7 @@ def test_managed_jobs_recovery_multi_node_aws(aws_config_region):
     test = smoke_tests_utils.Test(
         'managed_jobs_recovery_multi_node_aws',
         [
-            f'sky jobs launch --cloud aws --region {region} -n {name} --use-spot --num-nodes 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800"  -y -d',
+            rf'sky jobs launch --cloud aws --region {region} -n {name} --use-spot --num-nodes 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -473,7 +471,7 @@ def test_managed_jobs_recovery_multi_node_gcp():
     test = smoke_tests_utils.Test(
         'managed_jobs_recovery_multi_node_gcp',
         [
-            f'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot --num-nodes 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800"  -y -d',
+            rf'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot --num-nodes 2 "echo SKYPILOT_TASK_ID: \$SKYPILOT_TASK_ID; sleep 1800" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -512,7 +510,7 @@ def test_managed_jobs_cancellation_aws(aws_config_region):
         'managed_jobs_cancellation_aws',
         [
             # Test cancellation during spot cluster being launched.
-            f'sky jobs launch --cloud aws --region {region} -n {name} --use-spot "sleep 1000"  -y -d',
+            f'sky jobs launch --cloud aws --region {region} -n {name} --use-spot "sleep 1000" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -532,7 +530,7 @@ def test_managed_jobs_cancellation_aws(aws_config_region):
              '--output text) && echo "$s" && echo; [[ -z "$s" ]] || [[ "$s" = "terminated" ]] || [[ "$s" = "shutting-down" ]]'
             ),
             # Test cancelling the spot cluster during spot job being setup.
-            f'sky jobs launch --cloud aws --region {region} -n {name}-2 --use-spot tests/test_yamls/test_long_setup.yaml  -y -d',
+            f'sky jobs launch --cloud aws --region {region} -n {name}-2 --use-spot tests/test_yamls/test_long_setup.yaml -y -d',
             # The job is set up in the cluster, will shown as RUNNING.
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
@@ -551,7 +549,7 @@ def test_managed_jobs_cancellation_aws(aws_config_region):
              '--output text) && echo "$s" && echo; [[ -z "$s" ]] || [[ "$s" = "terminated" ]] || [[ "$s" = "shutting-down" ]]'
             ),
             # Test cancellation during spot job is recovering.
-            f'sky jobs launch --cloud aws --region {region} -n {name}-3 --use-spot "sleep 1000"  -y -d',
+            f'sky jobs launch --cloud aws --region {region} -n {name}-3 --use-spot "sleep 1000" -y -d',
             # The job is running in the cluster, will shown as RUNNING.
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
@@ -605,7 +603,7 @@ def test_managed_jobs_cancellation_gcp():
         'managed_jobs_cancellation_gcp',
         [
             # Test cancellation during spot cluster being launched.
-            f'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot "sleep 1000"  -y -d',
+            f'sky jobs launch --cloud gcp --zone {zone} -n {name} --use-spot "sleep 1000" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
@@ -618,7 +616,7 @@ def test_managed_jobs_cancellation_gcp():
                 job_status=[sky.ManagedJobStatus.CANCELLED],
                 timeout=155),
             # Test cancelling the spot cluster during spot job being setup.
-            f'sky jobs launch --cloud gcp --zone {zone} -n {name}-2 --use-spot tests/test_yamls/test_long_setup.yaml  -y -d',
+            f'sky jobs launch --cloud gcp --zone {zone} -n {name}-2 --use-spot tests/test_yamls/test_long_setup.yaml -y -d',
             # The job is set up in the cluster, will shown as RUNNING.
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
@@ -632,7 +630,7 @@ def test_managed_jobs_cancellation_gcp():
                 job_status=[sky.ManagedJobStatus.CANCELLED],
                 timeout=155),
             # Test cancellation during spot job is recovering.
-            f'sky jobs launch --cloud gcp --zone {zone} -n {name}-3 --use-spot "sleep 1000"  -y -d',
+            f'sky jobs launch --cloud gcp --zone {zone} -n {name}-3 --use-spot "sleep 1000" -y -d',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=f'{name}-3',
@@ -885,7 +883,7 @@ def test_managed_jobs_inline_env(generic_cloud: str):
     test = smoke_tests_utils.Test(
         'test-managed-jobs-inline-env',
         [
-            f'sky jobs launch -n {name} -y --cloud {generic_cloud} --env TEST_ENV="hello world" -- "echo "\\$TEST_ENV"; ([[ ! -z \\"\$TEST_ENV\\" ]] && [[ ! -z \\"\${constants.SKYPILOT_NODE_IPS}\\" ]] && [[ ! -z \\"\${constants.SKYPILOT_NODE_RANK}\\" ]] && [[ ! -z \\"\${constants.SKYPILOT_NUM_NODES}\\" ]]) || exit 1"',
+            rf'sky jobs launch -n {name} -y --cloud {generic_cloud} --env TEST_ENV="hello world" -- "echo "\$TEST_ENV"; ([[ ! -z \"\$TEST_ENV\" ]] && [[ ! -z \"\${constants.SKYPILOT_NODE_IPS}\" ]] && [[ ! -z \"\${constants.SKYPILOT_NODE_RANK}\" ]] && [[ ! -z \"\${constants.SKYPILOT_NUM_NODES}\" ]]) || exit 1"',
             smoke_tests_utils.
             get_cmd_wait_until_managed_job_status_contains_matching_job_name(
                 job_name=name,
diff --git a/tests/unit_tests/test_common_utils.py b/tests/unit_tests/test_common_utils.py
index 38c31263baa..d174c401c21 100644
--- a/tests/unit_tests/test_common_utils.py
+++ b/tests/unit_tests/test_common_utils.py
@@ -36,16 +36,17 @@ class TestMakeClusterNameOnCloud:
     @mock.patch('sky.utils.common_utils.get_user_hash')
     def test_make(self, mock_get_user_hash):
         mock_get_user_hash.return_value = MOCKED_USER_HASH
-        assert "lora-ab12" == common_utils.make_cluster_name_on_cloud("lora")
+        assert "lora-ab12cd34" == common_utils.make_cluster_name_on_cloud(
+            "lora")
 
     @mock.patch('sky.utils.common_utils.get_user_hash')
     def test_make_with_hyphen(self, mock_get_user_hash):
         mock_get_user_hash.return_value = MOCKED_USER_HASH
-        assert "seed-1-ab12" == common_utils.make_cluster_name_on_cloud(
+        assert "seed-1-ab12cd34" == common_utils.make_cluster_name_on_cloud(
             "seed-1")
 
     @mock.patch('sky.utils.common_utils.get_user_hash')
     def test_make_with_characters_to_transform(self, mock_get_user_hash):
         mock_get_user_hash.return_value = MOCKED_USER_HASH
-        assert "cuda-11-8-ab12" == common_utils.make_cluster_name_on_cloud(
+        assert "cud-73-ab12cd34" == common_utils.make_cluster_name_on_cloud(
             "Cuda_11.8")


Code review comments:
diff_hunk: @@ -98,10 +97,12 @@ def _is_valid_user_hash(user_hash: Optional[str]) -> bool:
             return user_hash
 
     hash_str = user_and_hostname_hash()
-    user_hash = hashlib.md5(hash_str.encode()).hexdigest()[:USER_HASH_LENGTH]
-    if not _is_valid_user_hash(user_hash):
+    hex_hash = hashlib.md5(hash_str.encode()).hexdigest()[:USER_HASH_LENGTH]
+    if not _is_valid_user_hash(hex_hash):
         # A fallback in case the hash is invalid.
-        user_hash = uuid.uuid4().hex[:USER_HASH_LENGTH]
+        hex_hash = uuid.uuid4().hex[:USER_HASH_LENGTH]
+    user_hash = base36_encode(hex_hash)
path: sky/utils/common_utils.py
text: @user1:
We should not take the shorter version before the base36 encoding? Otherwise, it will be shorter than `USER_HASH_LENGTH`, right?

@user1:
```python
    hex_hash = hashlib.md5(hash_str.encode()).hexdigest()
    if not _is_valid_user_hash(hex_hash):
        # A fallback in case the hash is invalid.
        hex_hash = uuid.uuid4().hex
    user_hash = base36_encode(hex_hash)[:USER_HASH_LENGTH]
```

@user1:
Actually, since this may inflate the user hash, if a user accidentally delete the user hash file, let's keep it in the original way: 16^8 > 4B space, which is large enough for our users.

@author:
That's a good point! I reverted in 1bc79d0. The original approach already provides enough hash space, so keeping it makes sense.