Repository: ray-project/ray
Pull request: [core][autoscaler] make cluster constraints clear in the autoscaler status report
<!-- Thank you for your contribution! Please review https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?

As requested in the https://github.com/ray-project/ray/issues/37959. Make the cluster constraints clear in the autoscaler status report by not showing them with normal demands together but showing them in an explicit section.

## Before
```sh
Resources
---------------------------------------------------------------
Usage:
 0B/8.21GiB memory
 0B/2.00GiB object_store_memory

Demands:
 {'CPU': 3.0}: 3+ pending tasks/actors
 {'CPU': 2.0}: 3+ pending tasks/actors
 {'CPU': 1}: 1000+ from request_resources()
```

## After
```sh
Resources
---------------------------------------------------------------
Usage:
 0B/8.21GiB memory
 0B/2.00GiB object_store_memory

Constraints:
 {'CPU': 1}: 1000 from request_resources()
Demands:
 {'CPU': 3.0}: 3+ pending tasks/actors
 {'CPU': 2.0}: 3+ pending tasks/actors
```

Note that this change is also reflected in the dashboard:
<img width="552" alt="image" src="https://github.com/user-attachments/assets/478218e8-73e7-4ecd-a5a7-fe3df6e5da76" />


## Related issue number

Closes https://github.com/ray-project/ray/issues/37959

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
    - [ ] I've added any new APIs to the API Reference. For example, if I added a
           method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(


Problem statement:
[core][autoscaler] Better observability for request resources 
### What happened + What you expected to happen

Cluster constraint from requests resources should not be shown together with other delta demands (e.g. pending placement groups, pending tasks/acotrs) 

### Versions / Dependencies

master

### Reproduction script

NA

### Issue Severity

None

Hints from issue discussion:
Feel free to assign this issue to me @kevin85421 
Adding some more context in the original issue. 

I think this is more of a nit in letting users what's the current cluster resource constrain through `autoscaler_sdk.request_resources()`. For example in the `ray status [-v]` output, we should probably explicitly show what the cluster resource constraint is. 

However, in terms of priority, this is probably a good to have, like a P1.5. 
+1 for this. In addition to the `autoscaler_sdk.request_resources()`, I also want those reported backlogs to be explicitly visible from the dashboard. That will be very helpful when debugging issues like https://github.com/ray-project/ray/issues/48950 and https://github.com/ray-project/ray/issues/47531
WDYT @kevin85421 and @jjyao?
If that makes sense to you, I would like to take over this work.
@rueian I fully agree that the autoscaler on the dashboard would be very helpful. However, its scope is a bit broad and it is not the top priority for KubeRay v1.4.0—our top priority being stability—and the dashboard is currently undergoing significant refactoring by @MortalHappiness. I would recommend improving ray status for now and opening an issue to track progress on adding the autoscaler to the dashboard. Does it make sense to you?
No problem. Actually, the `ray status` and the dashboard share the same underlying information on this. So, we just need to augment that shared information, and there’s no need for any additional code changes on the dashboard to display the new information.
> Cluster constraint from requests resources should not be shown together with other delta demands (e.g. pending placement groups, pending tasks/acotrs)
> For example in the ray status [-v] output, we should probably explicitly show what the cluster resource constraint is.

To make cluster constraints clearer to users, I'd like to move them from the current `Demands` section to a new `Constraints` section in the output of `ray status`.

> I also want those reported backlogs to be explicitly visible from the dashboard. That will be very helpful when debugging issues like https://github.com/ray-project/ray/issues/48950 and https://github.com/ray-project/ray/issues/47531

I also like to introduce 3 counters, `num_ready_requests_queued`, `num_infeasible_requests_queued`, and `backlog_size` to the `Demands` section. I believe that will help us debug the leaking problems mentioned earlier.

So, I am going to make the result output look like this, for example:

```sh
Resources
---------------------------------------------------------------
Usage:
 0B/8.21GiB memory
 0B/2.00GiB object_store_memory

Constraints:
 {'CPU': 1}: 1000+ from request_resources()
Demands:
 {'CPU': 3.0}: 3+ pending tasks/actors
 {'CPU': 2.0}: 3+ pending tasks/actors
 2+ ready requests queued
 3+ infeasible requests queued
 1+ total backlog queued
```
@kevin85421, @jjyao does this look good to you?
Or this could be better:
```sh
Resources
---------------------------------------------------------------
Usage:
 0B/8.21GiB memory
 0B/2.00GiB object_store_memory

Constraints:
 {'CPU': 1}: 1000+ from request_resources()
Demands:
 {'CPU': 3.0}: 3+ pending tasks/actors
 {'CPU': 2.0}: 3+ pending tasks/actors
Demand Counts:
 2+ ready requests queued
 3+ infeasible requests queued
 1+ total backlog queued
```

ping @kevin85421, @jjyao. Does the change in the above new `ray status` look good to you?

Commit message: [core][autoscaler] make cluster constraints in the autoscaler status report clear

Signed-off-by: Rueian <rueiancsie@gmail.com>

Patch to review:
diff --git a/python/ray/autoscaler/_private/util.py b/python/ray/autoscaler/_private/util.py
index fde2e2cd7fac..2b36a9706cf0 100644
--- a/python/ray/autoscaler/_private/util.py
+++ b/python/ray/autoscaler/_private/util.py
@@ -724,6 +724,18 @@ def filter_placement_group_from_bundle(bundle: ResourceBundle):
     return demand_lines
 
 
+def get_constraints_report(lm_summary: LoadMetricsSummary):
+    constraint_lines = []
+    for bundle, count in lm_summary.request_demand:
+        line = f" {bundle}: {count}+ from request_resources()"
+        constraint_lines.append(line)
+    if len(constraint_lines) > 0:
+        constraints_report = "\n".join(constraint_lines)
+    else:
+        constraints_report = " (no cluster constraints)"
+    return constraints_report
+
+
 def get_demand_report(lm_summary: LoadMetricsSummary):
     demand_lines = []
     if lm_summary.resource_demand:
@@ -733,9 +745,6 @@ def get_demand_report(lm_summary: LoadMetricsSummary):
         pg_str = format_pg(pg)
         line = f" {pg_str}: {count}+ pending placement groups"
         demand_lines.append(line)
-    for bundle, count in lm_summary.request_demand:
-        line = f" {bundle}: {count}+ from request_resources()"
-        demand_lines.append(line)
     if len(demand_lines) > 0:
         demand_report = "\n".join(demand_lines)
     else:
@@ -893,6 +902,7 @@ def format_info_string(
         failure_report += " (no failures)"
 
     usage_report = get_usage_report(lm_summary, verbose)
+    constraints_report = get_constraints_report(lm_summary)
     demand_report = get_demand_report(lm_summary)
     formatted_output = f"""{header}
 Node status
@@ -914,6 +924,8 @@ def format_info_string(
 {separator}
 {"Total " if verbose else ""}Usage:
 {usage_report}
+{"Total " if verbose else ""}Constraints:
+{constraints_report}
 {"Total " if verbose else ""}Demands:
 {demand_report}"""
 
diff --git a/python/ray/autoscaler/v2/tests/test_utils.py b/python/ray/autoscaler/v2/tests/test_utils.py
index 72ecfca2c986..53d3908967a3 100644
--- a/python/ray/autoscaler/v2/tests/test_utils.py
+++ b/python/ray/autoscaler/v2/tests/test_utils.py
@@ -567,11 +567,12 @@ def test_cluster_status_formatter():
  0.0/4.0 GPU
  5.42KiB/10.04KiB object_store_memory
 
+Total Constraints:
+ {'GPU': 2, 'CPU': 100}: 2+ from request_resources()
 Total Demands:
  {'CPU': 1, 'GPU': 1}: 11+ pending tasks/actors
  {'CPU': 1, 'GPU': 1} * 1 (STRICT_SPREAD): 1+ pending placement groups
  {'GPU': 2} * 1 (STRICT_PACK): 2+ pending placement groups
- {'GPU': 2, 'CPU': 100}: 2+ from request_resources()
 
 Node: instance1 (head_node)
  Id: fffffffffffffffffffffffffffffffffffffffffffffffffff00001
diff --git a/python/ray/autoscaler/v2/utils.py b/python/ray/autoscaler/v2/utils.py
index d8a986f6d7fd..fedef13978a2 100644
--- a/python/ray/autoscaler/v2/utils.py
+++ b/python/ray/autoscaler/v2/utils.py
@@ -319,6 +319,7 @@ def format(cls, data: ClusterStatus, verbose: bool = False) -> str:
         pending_report = cls._pending_node_report(data)
         failure_report = cls._failed_node_report(data, verbose)
         cluster_usage_report = cls._cluster_usage_report(data, verbose)
+        constraints_report = cls._constraints_report(data)
         demand_report = cls._demand_report(data)
         node_usage_report = (
             ""
@@ -343,6 +344,8 @@ def format(cls, data: ClusterStatus, verbose: bool = False) -> str:
             separator,
             f"{'Total ' if verbose else ''}Usage:",
             cluster_usage_report,
+            f"{'Total ' if verbose else ''}Constraints:",
+            constraints_report,
             f"{'Total ' if verbose else ''}Demands:",
             demand_report,
             node_usage_report,
@@ -539,6 +542,21 @@ def _pending_node_report(data: ClusterStatus) -> str:
             return "\n".join(pending_lines)
         return " (no pending nodes)"
 
+    @staticmethod
+    def _constraints_report(data: ClusterStatus) -> str:
+        # Process cluster constraint demand
+        constraint_lines = []
+        request_demand = [
+            (bc.bundle, bc.count)
+            for constraint_demand in data.resource_demands.cluster_constraint_demand
+            for bc in constraint_demand.bundles_by_count
+        ]
+        for bundle, count in request_demand:
+            constraint_lines.append(f" {bundle}: {count}+ from request_resources()")
+        if constraint_lines:
+            return "\n".join(constraint_lines)
+        return " (no cluster constraints)"
+
     @staticmethod
     def _demand_report(data: ClusterStatus) -> str:
         # Process resource demands
@@ -580,15 +598,6 @@ def _demand_report(data: ClusterStatus) -> str:
             pg_str = format_pg(pg)
             demand_lines.append(f" {pg_str}: {count}+ pending placement groups")
 
-        # Process cluster constraint demand
-        request_demand = [
-            (bc.bundle, bc.count)
-            for constraint_demand in data.resource_demands.cluster_constraint_demand
-            for bc in constraint_demand.bundles_by_count
-        ]
-        for bundle, count in request_demand:
-            demand_lines.append(f" {bundle}: {count}+ from request_resources()")
-
         # Generate demand report
         if demand_lines:
             return "\n".join(demand_lines)
diff --git a/python/ray/tests/test_cli_patterns/test_ray_status.txt b/python/ray/tests/test_cli_patterns/test_ray_status.txt
index 7e9a98b4c66d..f429ba4b7f1f 100644
--- a/python/ray/tests/test_cli_patterns/test_ray_status.txt
+++ b/python/ray/tests/test_cli_patterns/test_ray_status.txt
@@ -17,5 +17,7 @@ Usage:
  0.+
  0.+
 
+Constraints:
+ \(no cluster constraints\)
 Demands:
  \(no resource demands\)
diff --git a/python/ray/tests/test_cli_patterns/test_ray_status_multinode.txt b/python/ray/tests/test_cli_patterns/test_ray_status_multinode.txt
index 8e9d05cd56b8..6c4215b10086 100644
--- a/python/ray/tests/test_cli_patterns/test_ray_status_multinode.txt
+++ b/python/ray/tests/test_cli_patterns/test_ray_status_multinode.txt
@@ -20,5 +20,7 @@ Usage:
  0.+
  0.+
 
+Constraints:
+ \(no cluster constraints\)
 Demands:
  \(no resource demands\)
diff --git a/python/ray/tests/test_cli_patterns/test_ray_status_multinode_v1.txt b/python/ray/tests/test_cli_patterns/test_ray_status_multinode_v1.txt
index d4c717f209a5..39188ba5f2aa 100644
--- a/python/ray/tests/test_cli_patterns/test_ray_status_multinode_v1.txt
+++ b/python/ray/tests/test_cli_patterns/test_ray_status_multinode_v1.txt
@@ -18,5 +18,7 @@ Usage:
  0.+
  0.+
 
+Constraints:
+ \(no cluster constraints\)
 Demands:
  \(no resource demands\)
diff --git a/python/ray/tests/test_cli_patterns/test_ray_status_v1.txt b/python/ray/tests/test_cli_patterns/test_ray_status_v1.txt
index 3f4319d0b1b4..ca88b93034d0 100644
--- a/python/ray/tests/test_cli_patterns/test_ray_status_v1.txt
+++ b/python/ray/tests/test_cli_patterns/test_ray_status_v1.txt
@@ -15,5 +15,7 @@ Usage:
  0.+
  0.+
 
+Constraints:
+ \(no cluster constraints\)
 Demands:
  \(no resource demands\)
diff --git a/python/ray/tests/test_resource_demand_scheduler.py b/python/ray/tests/test_resource_demand_scheduler.py
index 81f296da84f1..adc0a9ab1bfb 100644
--- a/python/ray/tests/test_resource_demand_scheduler.py
+++ b/python/ray/tests/test_resource_demand_scheduler.py
@@ -3166,10 +3166,11 @@ def test_info_string():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 """.strip()
     actual = format_info_string(
         lm_summary,
@@ -3256,10 +3257,11 @@ def test_info_string_verbose():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Total Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Total Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 
 Node: 192.168.1.1
  Usage:
@@ -3370,10 +3372,11 @@ def test_info_string_verbose_node_types():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Total Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Total Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 
 Node: 192.168.1.1 (head-node)
  Usage:
@@ -3461,10 +3464,11 @@ def test_info_string_verbose_no_breakdown():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Total Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Total Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 """.strip()
     actual = format_info_string(
         lm_summary,
@@ -3555,10 +3559,11 @@ def test_info_string_with_launch_failures():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 """.strip()
     actual = format_info_string(
         lm_summary,
@@ -3647,10 +3652,11 @@ def test_info_string_with_launch_failures_verbose():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Total Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Total Demands:
  {'CPU': 1}: 150+ pending tasks/actors
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 """.strip()
     actual = format_info_string(
         lm_summary,
@@ -3735,11 +3741,12 @@ def test_info_string_failed_node_cap():
  2.00GiB/8.00GiB memory
  3.14GiB/16.00GiB object_store_memory
 
+Constraints:
+ {'CPU': 16}: 100+ from request_resources()
 Demands:
  {'CPU': 2.0}: 153+ pending tasks/actors (3+ using placement groups)
  {'GPU': 0.5}: 100+ pending tasks/actors (100+ using placement groups)
  {'CPU': 4} * 5 (PACK): 420+ pending placement groups
- {'CPU': 16}: 100+ from request_resources()
 """
 
     actual = format_info_string(


Code review comments:
diff_hunk: @@ -724,6 +724,18 @@ def filter_placement_group_from_bundle(bundle: ResourceBundle):
     return demand_lines
 
 
+def get_constraints_report(lm_summary: LoadMetricsSummary):
+    constraint_lines = []
+    for bundle, count in lm_summary.request_demand:
+        line = f" {bundle}: {count}+ from request_resources()"
+        constraint_lines.append(line)
+    if len(constraint_lines) > 0:
+        constraints_report = "\n".join(constraint_lines)
+    else:
+        constraints_report = " (no cluster constraints)"
path: python/ray/autoscaler/_private/util.py
text: @user1:
request_resources() is a DeveloperAPI, most users don't know it and I doubt they know what `cluster constraints` is.

@author:
It is indeed hard to link `request_resources` with `cluster constraints`. How about `(no request_resources() constraints)`?

diff_hunk: @@ -724,6 +724,18 @@ def filter_placement_group_from_bundle(bundle: ResourceBundle):
     return demand_lines
 
 
+def get_constraints_report(lm_summary: LoadMetricsSummary):
+    constraint_lines = []
+    for bundle, count in lm_summary.request_demand:
+        line = f" {bundle}: {count}+ from request_resources()"
path: python/ray/autoscaler/_private/util.py
text: @user1:
actually what does the `+` sign after count mean? We should know the exact count

@author:
I guess it was originally trying to align with the style of other demand texts. We indeed have the exact count. I will remove the `+`.