Repository: ray-project/ray

Issue:
[autoscaler] Inconsistent reporting of worker node types
On master, it seems I have a cluster with a cpu_4_ondemand head node and cpu_64_ondemand worker (68 CPUs in total), but somehow the debug string is reporting 2 workers of cpu_64_ondemand (which is wrong, since there is only 1 worker node, and the head node is cpu_4_ondemand).

YAML: https://gist.github.com/ericl/f9c32b8bbbe4e1aed43899832730f70d (switching the Ray version to latest wheel)

```
 - ResourceUsage: 0.0/68.0 CPU, 0.0/1.0 Custom1, 0.0 GiB/198.65 GiB memory, 0.0
GiB/59.19 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0
Worker node types:
 - cpu_64_ondemand: 2

```

cc @wuisawesome 


Discussion:
this is a bug in the debug string right? or are you noticing the wrong number of instances?

```
2020-09-17 23:33:35,234	INFO resource_demand_scheduler.py:68 -- Cluster resources: [{'node:172.30.2.59': 1.0, 'CPU': 4.0, 'memory': 187.0, 'object_store_memory': 64.0}, {'CPU': 64.0, 'memory': 3598.0, 'node:172.30.2.114': 1.0, 'object_store_memory': 1064.0}]
2020-09-17 23:33:35,234	INFO resource_demand_scheduler.py:69 -- Node counts: defaultdict(<class 'int'>, {'cpu_4_ondemand': 2, 'cpu_64_ondemand': 2})
2020-09-17 23:33:35,235	INFO resource_demand_scheduler.py:72 -- Resource demands: [{'accelerator_type:V100': 0.001, 'CPU': 1.0}]
2020-09-17 23:33:35,235	INFO resource_demand_scheduler.py:73 -- Unfulfilled demands: [{'accelerator_type:V100': 0.001, 'CPU': 1.0}]
2020-09-17 23:33:35,235	INFO resource_demand_scheduler.py:79 -- Node requests: []
```

Here's some more inconsistency, with the node counts dict reporting 4 nodes instead of 2.

I'm not sure if this is impacting autoscaling, but it seems likely if the node counts dict is reused for scheduling.
ok i can reproduce the logging issue, doesn't seem to impact autoscaling behavior 
IIUC now the conclusion is it does impact the behavior, right?
Yeah this will impact behavior if we need to instances of a type that is already up 

Resolving patch:
diff --git a/python/ray/autoscaler/resource_demand_scheduler.py b/python/ray/autoscaler/resource_demand_scheduler.py
--- a/python/ray/autoscaler/resource_demand_scheduler.py
+++ b/python/ray/autoscaler/resource_demand_scheduler.py
@@ -56,6 +56,7 @@ def get_nodes_to_launch(self, nodes: List[NodeID],
             nodes: List of existing nodes in the cluster.
             pending_nodes: Summary of node types currently being launched.
             resource_demands: Vector of resource demands from the scheduler.
+            usage_by_ip: Mapping from ip to available resources.
         """
 
         if resource_demands is None:
@@ -113,7 +114,6 @@ def add_node(node_type, available_resources=None):
             tags = self.provider.node_tags(node_id)
             if TAG_RAY_USER_NODE_TYPE in tags:
                 node_type = tags[TAG_RAY_USER_NODE_TYPE]
-                node_type_counts[node_type] += 1
                 ip = self.provider.internal_ip(node_id)
                 available_resources = usage_by_ip.get(ip)
                 add_node(node_type, available_resources)


Test patch:
diff --git a/python/ray/tests/test_resource_demand_scheduler.py b/python/ray/tests/test_resource_demand_scheduler.py
--- a/python/ray/tests/test_resource_demand_scheduler.py
+++ b/python/ray/tests/test_resource_demand_scheduler.py
@@ -180,6 +180,27 @@ def test_get_nodes_to_launch_limits():
     assert to_launch == []
 
 
+def test_calculate_node_resources():
+    provider = MockProvider()
+    scheduler = ResourceDemandScheduler(provider, TYPES_A, 10)
+
+    provider.create_node({}, {TAG_RAY_USER_NODE_TYPE: "p2.8xlarge"}, 2)
+
+    nodes = provider.non_terminated_nodes({})
+
+    ips = provider.non_terminated_node_ips({})
+    # 2 free p2.8xls
+    utilizations = {ip: {"GPU": 8} for ip in ips}
+    # 1 more on the way
+    pending_nodes = {"p2.8xlarge": 1}
+    # requires 4 p2.8xls (only 3 are in cluster/pending)
+    demands = [{"GPU": 8}] * (len(utilizations) + 2)
+    to_launch = scheduler.get_nodes_to_launch(nodes, pending_nodes, demands,
+                                              utilizations)
+
+    assert to_launch == [("p2.8xlarge", 1)]
+
+
 class LoadMetricsTest(unittest.TestCase):
     def testResourceDemandVector(self):
         lm = LoadMetrics()