Repository: ray-project/ray

Issue:
[autoscaler] AWS node provider broken due to external ip never getting updated
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Ray installed from (source or binary)**: Source @ https://github.com/ray-project/ray/commit/002531b199aaeb40d267b4ac50dbc7ae7689da1b
- **Ray version**: 0.6.2 @ 002531b199aaeb40d267b4ac50dbc7ae7689da1b
- **Python version**: 3.6
- **Exact command to reproduce**: `ray teardown -y ${RAY_PATH}/python/ray/autoscaler/aws/example-minimal.yaml && ray exec ${RAY_PATH}/python/ray/autoscaler/aws/example-minimal.yaml "echo test" --start`

### Describe the problem
When running the command above, the autoscaler correctly creates a head node but then hangs waiting for its ip. This happens because when the node is first created, we insert the node (without ip) to the node provider's node cache, and then when waiting for the ip we never refresh the node data, thus leading to just infinite loop [here](https://github.com/ray-project/ray/blob/002531b199aaeb40d267b4ac50dbc7ae7689da1b/python/ray/autoscaler/updater.py#L85) until `NODE_START_WAIT_S` breaks it.

This didn't happen before because we used an external ip cache, which would not have a key for a node until the external ip actually existed. The behavior was changed here: https://github.com/ray-project/ray/commit/315edab08508e7b4ca07ce22467d76dad4031a89#diff-6c1c6ac4425d69b4155cb41c15eeb277L89

### Source code / logs
```
(softlearning) ➜ kristian@jensen2: ~/github/hartikainen/ray on master ✗
$ ray teardown -y ~/github/hartikainen/ray/python/ray/autoscaler/aws/example-minimal.yaml && ray exec ~/github/hartikainen/ray/python/ray/autoscaler/aws/example-minimal.yaml "echo test" --start
2019-02-02 16:12:11,822 INFO commands.py:108 -- teardown_cluster: Terminating 1 nodes...
2019-02-02 16:12:13,112 INFO log_timer.py:21 -- teardown_cluster: Termination done. [LogTimer=1291ms]
2019-02-02 16:12:14,149 INFO commands.py:172 -- get_or_create_head_node: Launching new head node...
2019-02-02 16:12:15,510 INFO commands.py:185 -- get_or_create_head_node: Updating files on head node...
2019-02-02 16:12:15,512 INFO updater.py:126 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Updating to 7d28d7c48d6357f567a59188f2e558932a05ac3f
2019-02-02 16:12:15,512 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:15,633 INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=waiting-for-ssh on ['i-0a4c4fcfc2ee6a75b'] [LogTimer=120ms]
2019-02-02 16:12:25,520 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:35,529 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:45,534 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:55,541 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:05,549 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:15,556 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:25,566 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:35,576 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:45,584 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:55,594 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:05,605 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:15,615 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:25,624 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:35,632 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:45,642 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:55,653 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:05,660 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:15,668 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:25,676 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:35,684 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:45,692 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:55,700 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:05,708 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:15,718 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:25,728 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:35,736 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:45,744 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:55,752 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:17:05,761 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:17:15,764 INFO log_timer.py:21 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Got IP [LogTimer=300251ms]
2019-02-02 16:17:15,764 INFO log_timer.py:21 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Applied config 7d28d7c48d6357f567a59188f2e558932a05ac3f [LogTimer=300251ms]
2019-02-02 16:17:15,764 ERROR updater.py:138 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Error updating Unable to find IP of node
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/kristian/conda/envs/softlearning/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 141, in run
    raise e
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 130, in run
    self.do_update()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 183, in do_update
    self.set_ssh_ip_if_required()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 105, in set_ssh_ip_if_required
    assert ip is not None, "Unable to find IP of node"
AssertionError: Unable to find IP of node
```
[autoscaler] AWS node provider broken due to external ip never getting updated
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Ray installed from (source or binary)**: Source @ https://github.com/ray-project/ray/commit/002531b199aaeb40d267b4ac50dbc7ae7689da1b
- **Ray version**: 0.6.2 @ 002531b199aaeb40d267b4ac50dbc7ae7689da1b
- **Python version**: 3.6
- **Exact command to reproduce**: `ray teardown -y ${RAY_PATH}/python/ray/autoscaler/aws/example-minimal.yaml && ray exec ${RAY_PATH}/python/ray/autoscaler/aws/example-minimal.yaml "echo test" --start`

### Describe the problem
When running the command above, the autoscaler correctly creates a head node but then hangs waiting for its ip. This happens because when the node is first created, we insert the node (without ip) to the node provider's node cache, and then when waiting for the ip we never refresh the node data, thus leading to just infinite loop [here](https://github.com/ray-project/ray/blob/002531b199aaeb40d267b4ac50dbc7ae7689da1b/python/ray/autoscaler/updater.py#L85) until `NODE_START_WAIT_S` breaks it.

This didn't happen before because we used an external ip cache, which would not have a key for a node until the external ip actually existed. The behavior was changed here: https://github.com/ray-project/ray/commit/315edab08508e7b4ca07ce22467d76dad4031a89#diff-6c1c6ac4425d69b4155cb41c15eeb277L89

### Source code / logs
```
(softlearning) ➜ kristian@jensen2: ~/github/hartikainen/ray on master ✗
$ ray teardown -y ~/github/hartikainen/ray/python/ray/autoscaler/aws/example-minimal.yaml && ray exec ~/github/hartikainen/ray/python/ray/autoscaler/aws/example-minimal.yaml "echo test" --start
2019-02-02 16:12:11,822 INFO commands.py:108 -- teardown_cluster: Terminating 1 nodes...
2019-02-02 16:12:13,112 INFO log_timer.py:21 -- teardown_cluster: Termination done. [LogTimer=1291ms]
2019-02-02 16:12:14,149 INFO commands.py:172 -- get_or_create_head_node: Launching new head node...
2019-02-02 16:12:15,510 INFO commands.py:185 -- get_or_create_head_node: Updating files on head node...
2019-02-02 16:12:15,512 INFO updater.py:126 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Updating to 7d28d7c48d6357f567a59188f2e558932a05ac3f
2019-02-02 16:12:15,512 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:15,633 INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=waiting-for-ssh on ['i-0a4c4fcfc2ee6a75b'] [LogTimer=120ms]
2019-02-02 16:12:25,520 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:35,529 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:45,534 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:12:55,541 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:05,549 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:15,556 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:25,566 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:35,576 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:45,584 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:13:55,594 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:05,605 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:15,615 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:25,624 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:35,632 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:45,642 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:14:55,653 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:05,660 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:15,668 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:25,676 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:35,684 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:45,692 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:15:55,700 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:05,708 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:15,718 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:25,728 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:35,736 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:45,744 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:16:55,752 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:17:05,761 INFO updater.py:88 -- NodeUpdater: Waiting for IP of i-0a4c4fcfc2ee6a75b...
2019-02-02 16:17:15,764 INFO log_timer.py:21 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Got IP [LogTimer=300251ms]
2019-02-02 16:17:15,764 INFO log_timer.py:21 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Applied config 7d28d7c48d6357f567a59188f2e558932a05ac3f [LogTimer=300251ms]
2019-02-02 16:17:15,764 ERROR updater.py:138 -- NodeUpdater: i-0a4c4fcfc2ee6a75b: Error updating Unable to find IP of node
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/kristian/conda/envs/softlearning/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 141, in run
    raise e
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 130, in run
    self.do_update()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 183, in do_update
    self.set_ssh_ip_if_required()
  File "/home/kristian/github/hartikainen/ray/python/ray/autoscaler/updater.py", line 105, in set_ssh_ip_if_required
    assert ip is not None, "Unable to find IP of node"
AssertionError: Unable to find IP of node
```


Discussion:

Resolving patch:
diff --git a/python/ray/autoscaler/aws/node_provider.py b/python/ray/autoscaler/aws/node_provider.py
--- a/python/ray/autoscaler/aws/node_provider.py
+++ b/python/ray/autoscaler/aws/node_provider.py
@@ -127,11 +127,11 @@ def nodes(self, tag_filters):
         return [node.id for node in nodes]
 
     def is_running(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         return node.state["Name"] == "running"
 
     def is_terminated(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         state = node.state["Name"]
         return state not in ["running", "pending"]
 
@@ -142,10 +142,20 @@ def node_tags(self, node_id):
             return dict(d1, **d2)
 
     def external_ip(self, node_id):
-        return self._node(node_id).public_ip_address
+        node = self._get_cached_node(node_id)
+
+        if node.public_ip_address is None:
+            node = self._get_node(node_id)
+
+        return node.public_ip_address
 
     def internal_ip(self, node_id):
-        return self._node(node_id).private_ip_address
+        node = self._get_cached_node(node_id)
+
+        if node.private_ip_address is None:
+            node = self._get_node(node_id)
+
+        return node.private_ip_address
 
     def set_node_tags(self, node_id, tags):
         with self.tag_cache_lock:
@@ -205,7 +215,7 @@ def create_node(self, node_config, tags, count):
         self.ec2.create_instances(**conf)
 
     def terminate_node(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         node.terminate()
 
         self.tag_cache.pop(node_id, None)
@@ -218,14 +228,22 @@ def terminate_nodes(self, node_ids):
             self.tag_cache.pop(node_id, None)
             self.tag_cache_pending.pop(node_id, None)
 
-    def _node(self, node_id):
-        if node_id not in self.cached_nodes:
-            self.nodes({})  # Side effect: should cache it.
+    def _get_node(self, node_id):
+        """Refresh and get info for this node, updating the cache."""
+        self.nodes({})  # Side effect: fetches and caches the node.
 
         assert node_id in self.cached_nodes, "Invalid instance id {}".format(
             node_id)
+
         return self.cached_nodes[node_id]
 
+    def _get_cached_node(self, node_id):
+        """Return node info from cache if possible, otherwise fetches it."""
+        if node_id in self.cached_nodes:
+            return self.cached_nodes[node_id]
+
+        return self._get_node(node_id)
+
     def cleanup(self):
         self.tag_cache_update_event.set()
         self.tag_cache_kill_event.set()
diff --git a/python/ray/autoscaler/gcp/node_provider.py b/python/ray/autoscaler/gcp/node_provider.py
--- a/python/ray/autoscaler/gcp/node_provider.py
+++ b/python/ray/autoscaler/gcp/node_provider.py
@@ -51,10 +51,6 @@ def __init__(self, provider_config, cluster_name):
         # excessive DescribeInstances requests.
         self.cached_nodes = {}
 
-        # Cache of ip lookups. We assume IPs never change once assigned.
-        self.internal_ip_cache = {}
-        self.external_ip_cache = {}
-
     def nodes(self, tag_filters):
         if tag_filters:
             label_filter_expr = "(" + " AND ".join([
@@ -97,15 +93,15 @@ def nodes(self, tag_filters):
         return [i["name"] for i in instances]
 
     def is_running(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         return node["status"] == "RUNNING"
 
     def is_terminated(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         return node["status"] not in {"PROVISIONING", "STAGING", "RUNNING"}
 
     def node_tags(self, node_id):
-        node = self._node(node_id)
+        node = self._get_cached_node(node_id)
         labels = node.get("labels", {})
         return labels
 
@@ -114,7 +110,7 @@ def set_node_tags(self, node_id, tags):
         project_id = self.provider_config["project_id"]
         availability_zone = self.provider_config["availability_zone"]
 
-        node = self._node(node_id)
+        node = self._get_node(node_id)
         operation = self.compute.instances().setLabels(
             project=project_id,
             zone=availability_zone,
@@ -130,23 +126,30 @@ def set_node_tags(self, node_id, tags):
         return result
 
     def external_ip(self, node_id):
-        if node_id in self.external_ip_cache:
-            return self.external_ip_cache[node_id]
-        node = self._node(node_id)
-        # TODO: Is there a better and more reliable way to do this?
-        ip = (node.get("networkInterfaces", [{}])[0].get(
-            "accessConfigs", [{}])[0].get("natIP", None))
-        if ip:
-            self.external_ip_cache[node_id] = ip
+        node = self._get_cached_node(node_id)
+
+        def get_external_ip(node):
+            return node.get("networkInterfaces", [{}])[0].get(
+                "accessConfigs", [{}])[0].get("natIP", None)
+
+        ip = get_external_ip(node)
+        if ip is None:
+            node = self._get_node(node_id)
+            ip = get_external_ip(node)
+
         return ip
 
     def internal_ip(self, node_id):
-        if node_id in self.internal_ip_cache:
-            return self.internal_ip_cache[node_id]
-        node = self._node(node_id)
-        ip = node.get("networkInterfaces", [{}])[0].get("networkIP")
-        if ip:
-            self.internal_ip_cache[node_id] = ip
+        node = self._get_cached_node(node_id)
+
+        def get_internal_ip(node):
+            return node.get("networkInterfaces", [{}])[0].get("networkIP")
+
+        ip = get_internal_ip(node)
+        if ip is None:
+            node = self._get_node(node_id)
+            ip = get_internal_ip(node)
+
         return ip
 
     def create_node(self, base_config, tags, count):
@@ -206,14 +209,16 @@ def terminate_node(self, node_id):
 
         return result
 
-    def _node(self, node_id):
+    def _get_node(self, node_id):
+        self.nodes({})  # Side effect: fetches and caches the node.
+
+        assert node_id in self.cached_nodes, "Invalid instance id {}".format(
+            node_id)
+
+        return self.cached_nodes[node_id]
+
+    def _get_cached_node(self, node_id):
         if node_id in self.cached_nodes:
             return self.cached_nodes[node_id]
 
-        instance = self.compute.instances().get(
-            project=self.provider_config["project_id"],
-            zone=self.provider_config["availability_zone"],
-            instance=node_id,
-        ).execute()
-
-        return instance
+        return self._get_node(node_id)


Test patch: