Repository: skypilot-org/skypilot
Pull request: [Core][RunPod] Show error for RunPod multi-node
<!-- Describe the changes in this PR -->
Fixes #5344 
<img width="976" alt="dimmed" src="https://github.com/user-attachments/assets/184b669d-2bfa-4e5c-ad4a-076e17f49f07" />

This is a general fix to these classes of issues. First, we propagate resource infeasibility hints due to unsupported features to the optimizer, letting users know why specific resources were not considered. We also gather all hints in the process and display as a final error message to the user. This fix will take care of all `CloudImplementationFeatures`-related exceptions that were not displayed before during the optimization process. 


<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [X] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [X] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [ ] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[Core] Multi-node support error message is not shown
```
(base) ➜  ~ sky launch --cloud runpod --num-nodes 2
No resource satisfying 2x RunPod() on RunPod.
sky.exceptions.ResourcesUnavailableError: Catalog does not contain any instances satisfying the request: 2x RunPod().
To fix: relax or change the resource requirements.

Hint: sky show-gpus to list available accelerators.
      sky check to check the enabled clouds.
```

Error should be better; should check `CloudImplementationFeatures.MULTI_NODE` feature during optimization. 

User ran into this.

Hints from issue discussion:
Will look into this!

Commit message: update test for changed return param count

Patch to review:
diff --git a/sky/clouds/cloud.py b/sky/clouds/cloud.py
index a5777f58a86..d4f4163ebf4 100644
--- a/sky/clouds/cloud.py
+++ b/sky/clouds/cloud.py
@@ -415,13 +415,16 @@ def get_feasible_launchable_resources(
         try:
             self.check_features_are_supported(resources,
                                               resources_required_features)
-        except exceptions.NotSupportedError:
+        except exceptions.NotSupportedError as e:
             # TODO(zhwu): The resources are now silently filtered out. We
             # should have some logging telling the user why the resources
             # are not considered.
+            # UPDATE(kyuds): passing in NotSupportedError reason string
+            # to hint for issue #5344. Did not remove above comment as
+            # reason is not displayed when other resources are valid.
             return resources_utils.FeasibleResources(resources_list=[],
                                                      fuzzy_candidate_list=[],
-                                                     hint=None)
+                                                     hint=str(e))
         return self._get_feasible_launchable_resources(resources)
 
     def _get_feasible_launchable_resources(
diff --git a/sky/optimizer.py b/sky/optimizer.py
index ea8d50e464e..aac9b5820a2 100644
--- a/sky/optimizer.py
+++ b/sky/optimizer.py
@@ -290,11 +290,11 @@ def get_reservations_available_resources(
             fuzzy_candidates: List[str] = []
             if node_i < len(topo_order) - 1:
                 # Convert partial resource labels to launchable resources.
-                launchable_resources, cloud_candidates, fuzzy_candidates = (
-                    _fill_in_launchable_resources(
-                        task=node,
-                        blocked_resources=blocked_resources,
-                        quiet=quiet))
+                (launchable_resources, cloud_candidates, fuzzy_candidates,
+                 resource_hints) = (_fill_in_launchable_resources(
+                     task=node,
+                     blocked_resources=blocked_resources,
+                     quiet=quiet))
                 node_to_candidate_map[node] = cloud_candidates
                 # Has to call the printing after the launchable resources are
                 # computed, because the missing fields of the resources are
@@ -390,16 +390,27 @@ def get_reservations_available_resources(
                 node_resources_reprs = ', '.join(f'{node.num_nodes}x ' +
                                                  r.repr_with_region_zone
                                                  for r in node.resources)
+                hints_concat = '\n'.join([
+                    f'{bold}Resource: {repr(resource)}{reset}\n' +
+                    '\n'.join(hint_list)
+                    for resource, hint_list in resource_hints.items()
+                    if hint_list
+                ])
+                hints_formatted = '\n'.join(
+                    map(lambda r: f'        {r}', hints_concat.split('\n')))
+                resource_hints_string = (
+                    f'Hint 2: Check Per Resource Hint\n{hints_formatted}'
+                    if hints_formatted else '')
                 error_msg = (
                     f'{source_hint.capitalize()} does not contain any '
                     f'instances satisfying the request: '
                     f'{node_resources_reprs}.'
                     f'\nTo fix: relax or change the '
                     f'resource requirements.{fuzzy_candidates_str}\n\n'
-                    f'Hint: {bold}sky show-gpus{reset} '
+                    f'Hint 1: {bold}sky show-gpus{reset} '
                     'to list available accelerators.\n'
-                    f'      {bold}sky check{reset} to check the enabled '
-                    'clouds.')
+                    f'        {bold}sky check{reset} to check the enabled '
+                    f'clouds.\n{resource_hints_string}')
                 with ux_utils.print_exception_no_traceback():
                     raise exceptions.ResourcesUnavailableError(error_msg)
         return node_to_cost_map, node_to_candidate_map
@@ -1047,7 +1058,7 @@ def ordinal_number(n):
                 for resources in task.resources:
                     # Check if there exists launchable resources
                     local_task.set_resources(resources)
-                    launchable_resources_map, _, _ = (
+                    launchable_resources_map, _, _, _ = (
                         _fill_in_launchable_resources(
                             task=local_task,
                             blocked_resources=blocked_resources,
@@ -1213,7 +1224,8 @@ def _fill_in_launchable_resources(
     blocked_resources: Optional[Iterable[resources_lib.Resources]],
     quiet: bool = False
 ) -> Tuple[Dict[resources_lib.Resources, List[resources_lib.Resources]],
-           _PerCloudCandidates, List[str]]:
+           _PerCloudCandidates, List[str], Dict[resources_lib.Resources,
+                                                List[str]]]:
     """Fills in the launchable resources for the task.
 
     Returns:
@@ -1235,6 +1247,8 @@ def _fill_in_launchable_resources(
     all_fuzzy_candidates = set()
     cloud_candidates: _PerCloudCandidates = collections.defaultdict(
         List[resources_lib.Resources])
+    resource_hints: Dict[resources_lib.Resources,
+                         List[str]] = collections.defaultdict(list)
     if blocked_resources is None:
         blocked_resources = []
     for resources in task.resources:
@@ -1259,6 +1273,7 @@ def _fill_in_launchable_resources(
         for cloud, feasible_resources in feasible_list:
             if feasible_resources.hint is not None:
                 hints[cloud] = feasible_resources.hint
+                resource_hints[resources].append(feasible_resources.hint)
             if feasible_resources.resources_list:
                 # Assume feasible_resources is sorted by prices. Guaranteed by
                 # the implementation of get_feasible_launchable_resources and
@@ -1301,8 +1316,11 @@ def _fill_in_launchable_resources(
                                     'to allow for larger instances.'
                                     f'{colorama.Style.RESET_ALL}')
                 for cloud, hint in hints.items():
-                    logger.info(f'{repr(cloud)}: {hint}')
+                    logger.info(f'{colorama.Fore.LIGHTBLACK_EX}'
+                                f'{repr(cloud)}: {hint}'
+                                f'{colorama.Style.RESET_ALL}')
 
         launchable[resources] = _filter_out_blocked_launchable_resources(
             launchable[resources], blocked_resources)
-    return launchable, cloud_candidates, list(sorted(all_fuzzy_candidates))
+    return launchable, cloud_candidates, list(
+        sorted(all_fuzzy_candidates)), resource_hints
diff --git a/tests/test_optimizer_dryruns.py b/tests/test_optimizer_dryruns.py
index f3f40352d01..2de21695bd9 100644
--- a/tests/test_optimizer_dryruns.py
+++ b/tests/test_optimizer_dryruns.py
@@ -702,7 +702,7 @@ def test_optimize_disk_tier(enable_all_clouds):
     def _get_all_candidate_cloud(r: sky.Resources) -> Set[clouds.Cloud]:
         task = sky.Task()
         task.set_resources(r)
-        _, per_cloud_candidates, _ = optimizer._fill_in_launchable_resources(
+        _, per_cloud_candidates, _, _ = optimizer._fill_in_launchable_resources(
             task, blocked_resources=None)
         return set(per_cloud_candidates.keys())
 


Code review comments:
diff_hunk: @@ -390,16 +390,27 @@ def get_reservations_available_resources(
                 node_resources_reprs = ', '.join(f'{node.num_nodes}x ' +
                                                  r.repr_with_region_zone
                                                  for r in node.resources)
+                hints_concat = '\n'.join([
+                    f'{bold}Resource: {repr(resource)}{reset}\n' +
+                    '\n'.join(hint_list)
+                    for resource, hint_list in resource_hints.items()
+                    if hint_list
+                ])
+                hints_formatted = '\n'.join(
+                    map(lambda r: f'        {r}', hints_concat.split('\n')))
+                resource_hints_string = (
+                    f'Hint 2: Check Per Resource Hint\n{hints_formatted}'
+                    if hints_formatted else '')
                 error_msg = (
                     f'{source_hint.capitalize()} does not contain any '
                     f'instances satisfying the request: '
                     f'{node_resources_reprs}.'
                     f'\nTo fix: relax or change the '
                     f'resource requirements.{fuzzy_candidates_str}\n\n'
-                    f'Hint: {bold}sky show-gpus{reset} '
+                    f'Hint 1: {bold}sky show-gpus{reset} '
path: sky/optimizer.py
text: @user1:
I don't think numbering hints is necessary, just having multiple unnumbered hints should be fine

diff_hunk: @@ -390,16 +390,27 @@ def get_reservations_available_resources(
                 node_resources_reprs = ', '.join(f'{node.num_nodes}x ' +
                                                  r.repr_with_region_zone
                                                  for r in node.resources)
+                hints_concat = '\n'.join([
+                    f'{bold}Resource: {repr(resource)}{reset}\n' +
+                    '\n'.join(hint_list)
+                    for resource, hint_list in resource_hints.items()
+                    if hint_list
+                ])
+                hints_formatted = '\n'.join(
+                    map(lambda r: f'        {r}', hints_concat.split('\n')))
path: sky/optimizer.py
text: @user1:
I'm not a fan for the hardcoded spaces here, we should try to make it more obvious why the number of spaces are the way there are. What do you think about something like
```suggestion
                indent_prefix = ' '*len('Hint: ')
                hints_formatted = '\n'.join(
                    map(lambda r: f'{indent_prefix}{r}', hints_concat.split('\n')))
```