Repository: ray-project/ray

Issue:
[Serve] create_backend() hangs on adding too many replicas
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?
When adding certain numbers of replicas using create_backend(), it doesn't realize there aren't enough resources available, so create_backend() hangs forever waiting for another CPU to become available.  The repro script below makes it seem like it's an off-by-one error, but I'm not sure that's the case; I ran into this issue frequently on a multi-node cluster (5 nodes, 2 CPUs each) when adding several backends with ~4 replicas each.  Maybe the HTTPproxyactor isn't being counted or something?

*Ray version and other system information (Python version, TensorFlow version, OS):*
Ray 1.0.0rc2, Python 3.8, Mac OS
### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

```python
import ray
from ray import serve
ray.init(address="auto") # 1 node, 16 cores
client = serve.start() 

def f(_):
    return "hi"

# As expected: "RayServeException: Cannot scale backend hi to 20 replicas. 
# Ray Serve tried to add 20 replicas but the resources only allows 16 to be added."
client.create_backend("hi", f, config=serve.BackendConfig(num_replicas=20))

# hangs forever: "WARNING worker.py:1073 -- The actor or task with ID 
# ffffffffffffffff591ac74701000000 is pending and cannot currently be 
# scheduled. It requires {CPU: 1.000000} ..."
client.create_backend("hi", f, config=serve.BackendConfig(num_replicas=16))

# With num_replicas=15 it would have succeeded.
```
If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
🤦 ok i found the bug. we were checking *total resource* but we should check *available resource*
We can fix it in Serve once #11014 is merged

Resolving patch:
diff --git a/python/ray/serve/controller.py b/python/ray/serve/controller.py
--- a/python/ray/serve/controller.py
+++ b/python/ray/serve/controller.py
@@ -536,12 +536,11 @@ def _scale_replicas(self, backend_tag: str, num_replicas: int) -> None:
 
         backend_info = self.backends[backend_tag]
         if delta_num_replicas > 0:
-            can_schedule = try_schedule_resources_on_nodes(
-                requirements=[
-                    backend_info.replica_config.resource_dict
-                    for _ in range(delta_num_replicas)
-                ],
-                ray_nodes=ray.nodes())
+            can_schedule = try_schedule_resources_on_nodes(requirements=[
+                backend_info.replica_config.resource_dict
+                for _ in range(delta_num_replicas)
+            ])
+
             if _RESOURCE_CHECK_ENABLED and not all(can_schedule):
                 num_possible = sum(can_schedule)
                 raise RayServeException(
diff --git a/python/ray/serve/utils.py b/python/ray/serve/utils.py
--- a/python/ray/serve/utils.py
+++ b/python/ray/serve/utils.py
@@ -6,7 +6,7 @@
 import random
 import string
 import time
-from typing import List
+from typing import List, Dict
 import io
 import os
 from ray.serve.exceptions import RayServeException
@@ -237,27 +237,23 @@ def unwrap_callback(fut: asyncio.Future):
 
 def try_schedule_resources_on_nodes(
         requirements: List[dict],
-        ray_nodes: List = None,
+        ray_resource: Dict[str, Dict] = None,
 ) -> List[bool]:
     """Test given resource requirements can be scheduled on ray nodes.
 
     Args:
         requirements(List[dict]): The list of resource requirements.
-        ray_nodes(Optional[List]): The list of nodes. By default it reads from
-            ``ray.nodes()``.
+        ray_nodes(Optional[Dict[str, Dict]]): The resource dictionary keyed by
+            node id. By default it reads from
+            ``ray.state.state._available_resources_per_node()``.
     Returns:
         successfully_scheduled(List[bool]): A list with the same length as
             requirements. Each element indicates whether or not the requirement
             can be satisied.
     """
 
-    if ray_nodes is None:
-        ray_nodes = ray.nodes()
-
-    node_to_resources = {
-        node["NodeID"]: node["Resources"]
-        for node in ray_nodes if node["Alive"]
-    }
+    if ray_resource is None:
+        ray_resource = ray.state.state._available_resources_per_node()
 
     successfully_scheduled = []
 
@@ -265,7 +261,7 @@ def try_schedule_resources_on_nodes(
         # Filter out zero value
         resource_dict = {k: v for k, v in resource_dict.items() if v > 0}
 
-        for node_id, node_resource in node_to_resources.items():
+        for node_id, node_resource in ray_resource.items():
             # Check if we can schedule on this node
             feasible = True
             for key, count in resource_dict.items():
@@ -274,7 +270,6 @@ def try_schedule_resources_on_nodes(
 
             # If we can, schedule it on this node
             if feasible:
-                node_resource = node_to_resources[node_id]
                 for key, count in resource_dict.items():
                     node_resource[key] -= count
 


Test patch:
diff --git a/python/ray/serve/tests/test_util.py b/python/ray/serve/tests/test_util.py
--- a/python/ray/serve/tests/test_util.py
+++ b/python/ray/serve/tests/test_util.py
@@ -69,20 +69,15 @@ def make():
 
 
 def test_mock_scheduler():
-    ray_nodes = [{
-        "NodeID": "AAA",
-        "Alive": True,
-        "Resources": {
+    ray_nodes = {
+        "AAA": {
             "CPU": 2.0,
             "GPU": 2.0
-        }
-    }, {
-        "NodeID": "BBB",
-        "Alive": True,
-        "Resources": {
+        },
+        "BBB": {
             "CPU": 4.0,
         }
-    }]
+    }
 
     assert try_schedule_resources_on_nodes(
         [