Repository: ray-project/ray

Issue:
[autoscaler] monitor.py has a memory/thread leak
### What is the problem?

With this cluster yaml, you can watch monitor.py via `htop` quickly eat up all of the available memory.

It also seems to be creating hundreds of NodeUpdater threads.

*Ray version and other system information (Python version, TensorFlow version, OS):* `master` latest wheels

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

```
cluster_name: memoryleak
min_workers: 2
max_workers: 2
initial_workers: 2
autoscaling_mode: default
target_utilization_fraction: 0.8
idle_timeout_minutes: 60
provider:
  type: aws
  region: us-east-1
  cache_stopped_nodes: false
auth:
  ssh_user: ubuntu
head_node:
  BlockDeviceMappings:
  - DeviceName: /dev/sda1
    Ebs:
      VolumeSize: 300
      DeleteOnTermination: true
      VolumeType: gp2
  ImageId: latest_dlami
  InstanceType: t3.xlarge
worker_nodes:
  BlockDeviceMappings:
  - DeviceName: /dev/sda1
    Ebs:
      VolumeSize: 300
      DeleteOnTermination: true
      VolumeType: gp2
  ImageId: latest_dlami
  InstanceType: t3.xlarge
file_mounts: {}
cluster_synced_files:
- ~/my-cluster-synced-file
file_mounts_sync_continuously: true
initialization_commands: []
setup_commands:
- pip install --user -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
head_setup_commands:
- tmux new-session -d "python3 -m http.server"
- pip install --user -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
- sleep 5
- curl localhost:8000 -o ~/my-cluster-synced-file
worker_setup_commands: []
```
cc @ericl @wuisawesome 



Discussion:
@wuisawesome could this be related to your new reloading code that creates a new node provider on config update?
Just did some digging - we cannot create providers constantly, as is currently done in `autoscaler.reset()`.

```
==> /tmp/ray/session_2020-09-10_02-30-21_292916_30660/logs/monitor.err <==
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 368, in <module>
    monitor.run()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 120, in update
    self.reset(errors_fatal=False)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 302, in reset
    self.config["cluster_name"])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/node_provider.py", line 148, in get_node_provider
    return provider_cls(provider_config, cluster_name)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/aws/node_provider.py", line 55, in __init__
    traceback.print_stack()

==> /tmp/ray/session_2020-09-10_02-30-21_292916_30660/logs/monitor.out <==
Creating a new NODE PROVIDER!

==> /tmp/ray/session_2020-09-10_02-30-21_292916_30660/logs/monitor.err <==
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 368, in <module>
    monitor.run()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 120, in update
    self.reset(errors_fatal=False)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 302, in reset
    self.config["cluster_name"])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/node_provider.py", line 148, in get_node_provider
    return provider_cls(provider_config, cluster_name)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/aws/node_provider.py", line 55, in __init__
    traceback.print_stack()
```

Seems like this is a necessary fix (or at least you need to clean up the existing provider):
```
@@ -295,9 +297,9 @@ class StandardAutoscaler:
             self.config = new_config
             self.runtime_hash = new_runtime_hash
             self.file_mounts_contents_hash = new_file_mounts_contents_hash
-
-            self.provider = get_node_provider(self.config["provider"],
-                                              self.config["cluster_name"])
+            if not self.provider:
+                self.provider = get_node_provider(self.config["provider"],
+                                                  self.config["cluster_name"])
             # Check whether we can enable the resource demand scheduler.
             if "available_node_types" in self.config:
                 self.available_node_types = self.config["available_node_types"]

Resolving patch:
diff --git a/python/ray/autoscaler/autoscaler.py b/python/ray/autoscaler/autoscaler.py
--- a/python/ray/autoscaler/autoscaler.py
+++ b/python/ray/autoscaler/autoscaler.py
@@ -64,6 +64,9 @@ def __init__(self,
                  process_runner=subprocess,
                  update_interval_s=AUTOSCALER_UPDATE_INTERVAL_S):
         self.config_path = config_path
+        # Keep this before self.reset (self.provider needs to be created
+        # exactly once).
+        self.provider = None
         self.reset(errors_fatal=True)
         self.load_metrics = load_metrics
 
@@ -250,6 +253,7 @@ def _update(self):
                 self.should_update(node_id) for node_id in nodes):
             if node_id is not None:
                 resources = self._node_resources(node_id)
+                logger.debug(f"{node_id}: Starting new thread runner.")
                 T.append(
                     threading.Thread(
                         target=self.spawn_updater,
@@ -295,9 +299,9 @@ def reset(self, errors_fatal=False):
             self.config = new_config
             self.runtime_hash = new_runtime_hash
             self.file_mounts_contents_hash = new_file_mounts_contents_hash
-
-            self.provider = get_node_provider(self.config["provider"],
-                                              self.config["cluster_name"])
+            if not self.provider:
+                self.provider = get_node_provider(self.config["provider"],
+                                                  self.config["cluster_name"])
             # Check whether we can enable the resource demand scheduler.
             if "available_node_types" in self.config:
                 self.available_node_types = self.config["available_node_types"]
@@ -462,6 +466,8 @@ def should_update(self, node_id):
 
     def spawn_updater(self, node_id, init_commands, ray_start_commands,
                       node_resources, docker_config):
+        logger.info(f"Creating new (spawn_updater) updater thread for node"
+                    f" {node_id}.")
         updater = NodeUpdaterThread(
             node_id=node_id,
             provider_config=self.config["provider"],
@@ -492,6 +498,8 @@ def can_update(self, node_id):
             return False
         if self.num_failed_updates.get(node_id, 0) > 0:  # TODO(ekl) retry?
             return False
+        logger.debug(f"{node_id} is not being updated and "
+                     "passes config check (can_update=True).")
         return True
 
     def launch_new_node(self, count: int, node_type: Optional[str]) -> None:
diff --git a/python/ray/autoscaler/commands.py b/python/ray/autoscaler/commands.py
--- a/python/ray/autoscaler/commands.py
+++ b/python/ray/autoscaler/commands.py
@@ -455,7 +455,7 @@ def kill_node(config_file, yes, hard, override_cluster_name):
 
 def monitor_cluster(cluster_config_file, num_lines, override_cluster_name):
     """Tails the autoscaler logs of a Ray cluster."""
-    cmd = "tail -n {} -f /tmp/ray/session_*/logs/monitor*".format(num_lines)
+    cmd = f"tail -n {num_lines} -f /tmp/ray/session_latest/logs/monitor*"
     exec_cluster(
         cluster_config_file,
         cmd=cmd,
@@ -717,7 +717,7 @@ def get_or_create_head_node(config,
                 logger, "get_or_create_head_node: "
                 "Head node up-to-date, IP address is: {}", head_node_ip)
 
-        monitor_str = "tail -n 100 -f /tmp/ray/session_*/logs/monitor*"
+        monitor_str = "tail -n 100 -f /tmp/ray/session_latest/logs/monitor*"
         if override_cluster_name:
             modifiers = " --cluster-name={}".format(
                 quote(override_cluster_name))


Test patch: