Repository: skypilot-org/skypilot
Pull request: [UX] Improve error message when one node fails
<!-- Describe the changes in this PR -->

Fixes #4232. Prints:
> ERROR: Job X failed with return code list: [137, 1, 137 137] (A worker failed with return code 1, SkyPilot cleaned up the processes on other nodes with return code 137)

<!-- Describe the tests ran -->

Manually ran with a yaml file that runs `exit 1`
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [ ] Any manual or new tests for this PR (please specify below)
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [ ] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[Core/UX] Improve the display of returncode for multi-node
<!-- Describe the bug report / feature request here -->

When a user's job is running on multiple nodes and one node fails with a return code, e.g. 1, SkyPilot will kill the processes on the other nodes, with a return code 137. It is confusing to users to see a list of return code like the following: `ERROR: Job 1 failed with return code list: [1, 137, 137]`

Instead, we should show message like the following:
```
ERROR: Job 1 failed with returncode: 1 on one node worker-2, SkyPilot cleaned the processes on other nodes with returncode 137
```

<!-- If relevant, fill in versioning info to help us troubleshoot -->
_Version & Commit info:_
* `sky -v`: PLEASE_FILL_IN
* `sky -c`: PLEASE_FILL_IN

Hints from issue discussion:

Commit message: Removed rank number

Patch to review:
diff --git a/sky/backends/cloud_vm_ray_backend.py b/sky/backends/cloud_vm_ray_backend.py
index 147a08e63d5..cfae58e65a3 100644
--- a/sky/backends/cloud_vm_ray_backend.py
+++ b/sky/backends/cloud_vm_ray_backend.py
@@ -699,6 +699,11 @@ def add_epilogue(self) -> None:
                 # 139 is the return code of SIGSEGV, i.e. Segmentation Fault.
                 if any(r == 139 for r in returncodes):
                     reason = '(likely due to Segmentation Fault)'
+                if any(r == 137 for r in returncodes):
+                    # Find the first non-137 return code and its index
+                    non_137 = next((i, r) for i, r in enumerate(returncodes) if r != 137)
+                    # +1 because the worker rank is 0-based, but I think the worker number is 1-based
+                    reason = f'(A Worker failed with return code {{non_137[1]}}, SkyPilot cleaned up the processes on other nodes with return code 137)'
                 print('ERROR: {colorama.Fore.RED}Job {self.job_id} failed with '
                       'return code list:{colorama.Style.RESET_ALL}',
                       returncodes,


Code review comments:
diff_hunk: @@ -699,6 +699,11 @@ def add_epilogue(self) -> None:
                 # 139 is the return code of SIGSEGV, i.e. Segmentation Fault.
                 if any(r == 139 for r in returncodes):
                     reason = '(likely due to Segmentation Fault)'
+                if any(r == 137 for r in returncodes):
+                    # Find the first non-137 return code and its index
+                    non_137 = next((i, r) for i, r in enumerate(returncodes) if r != 137)
+                    # +1 because the worker rank is 0-based, but I think the worker number is 1-based
path: sky/backends/cloud_vm_ray_backend.py
text: @user1:
We can remove the index stuff now. (The `enumerate` call and the comments.)