Repository: ray-project/ray

Issue:
[AWS Autoscaler] create_node hangs forever when insufficient AWS Capacity
In aws/node_provider.py:create_node, the call to ```self.ec2.create_instances``` can block for over half an hour, stopping all other Monitor activity. This has happened to me for spot requests when AWS has no capacity, but it could plausibly happen in other situations too.

A quick fix might be to reduce the number of retries that boto makes. They support a [global parameter](http://botocore.readthedocs.io/en/latest/reference/config.html), but it doesn't seem like it can be set per request.

A more robust approach would be to use request_spot_instances, reserving create_instance only for on-demand instances; however, this would complicate the code significantly, as I think we'd need to poll for when the instances are created.

Alternately, we could switch to using {request,modify}_spot_fleet, which supports both on-demand and spot types. I think this is closest to how Amazon intends the API to be used, but it is a bit of a mismatch with the Ray autoscaling API. The main problem is in scaling down: I think we'd need to follow [this procedure](https://stackoverflow.com/questions/45870613/terminate-specific-instance-in-aws-spot-fleet) to terminate a specific node. We'd also still need to poll to discover new nodes.

## Logs

```
StandardAutoscaler: Launching 2 new nodes
StandardAutoscaler: Error during autoscaling: {} Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/ray/autoscaler/autoscaler.py", line 246, in update
    self._update()
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/ray/autoscaler/autoscaler.py", line 308, in _update
    min(self.max_concurrent_launches, target_num - len(nodes)))
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/ray/autoscaler/autoscaler.py", line 455, in launch_new_node
    count)
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/ray/autoscaler/aws/node_provider.py", line 113, in create_node
    self.ec2.create_instances(**conf)
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/anaconda3/envs/pirl/lib/python3.6/site-packages/botocore/client.py", line 622, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 12): There is no Spot capacity available that matches your request.

StandardAutoscaler [2018-05-25 19:16:35.413360]: 0/2 target nodes
 - NodeIdleSeconds: Min=0 Mean=0 Max=0
 - NumNodesConnected: 1
 - NumNodesUsed: 1.0
 - ResourceUsage: 4.0/4.0 CPU, 4.0/4.0 GPU
StandardAutoscaler: Launching 2 new nodes
StandardAutoscaler [2018-05-25 19:17:15.009775]: 2/2 target nodes
```


Discussion:
I looked into this a little more. Changing the number of retries in boto seems quite messy and I'd prefer not to go down that route. I think the easiest change to make would be to use the SpotInstanceRequest, and then block for some configurable period (e.g. a minute) until we know if the request is satisfied; if not, then cancel the request. This would require relaxing the assertion in autoscaler.py:launch_new_node that checks new nodes have in fact been launched, but I can't see any harm coming from this.

I'd be happy to implement this, however I don't feel it's a great solution, so would appreciate some feedback before starting.
Will this happen any time that I try to create more instances than my limits allow?

If you are able to help out with fixing this that'd be awesome! @ericl is the person to talk to.
Another approach would be to do the request in another thread, so we could still use CreateInstances. This would require some additional tracking logic for outstanding requests, but would avoid blocking the main loop.

Let me know what you think. Btw, the assertion that the number of nodes increases has already been removed, due to the possibility of a race with node removal.
@robertnishihara I think if you hit instance limits EC2 returns an error immediately rather than blocking (at least I did not encounter this problem and I have hit instance limits before). The issue can be reproduced fairly reliably by trying to create spot instances with limited capacity, e.g. a p2.16xlarge request in us-west-2b will almost certainly not be fulfilled.

@ericl I'm happy with a threading solution, I was reluctant to suggest it as I hadn't seen it used elsewhere in Ray, but a search throws up some other use cases. 

The main challenge I see with introducing concurrency here is that the autoscaler keeps trying to add nodes so long as the current number of nodes is less than the target. If the requests blocks in a separate thread (but is eventually satisfied), we could end up launching a lot more nodes than was originally intended. Options I see are to either change the API and have the autoscaler expose the target number of nodes to the node provider, or to have the AWS node provider only process one instance launch request at a time (silently dropping the others). Thoughts?
@AdamGleave originally I was thinking of something like the following:

1. Keep a "pending nodes" counter in the autoscaler which tracks outstanding instance creation requests.
2. When checking whether to create new nodes, compare num_running + num_pending to the target.
3. When checking whether to remove nodes, compare against just num_running.

The idea to just avoid launching more than 1 instance creation request at a time could also work though, and might be simpler, since it only involves only adding a single boolean to the autoscaler state. I would do this in the autoscaler logic itself instead of the node provider so that it will work with other providers (perhaps create a new node provider copy for the instance launcher thread, to avoid potential thread safety issues).
@ericl That approach sounds good. The only problem I can foresee is we won't cancel requests when we no longer need them (only terminating the node once it starts); this problem seems minor, however.

An alternative that avoids this problem (and does not require threading) would be to have create_node make a [RequestSpotInstance](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RequestSpotInstances.html) call, which should always return immediately. nodes() would be changed to check both the running instances and unfulfilled spot instance requests. (There's potentially a race here, but it seems like it would happen very rarely, and would not have major consequences.) {internal,external}_ip would return None while the instance request is pending.

The only change I can see this requiring to the autoscaler would having _update preferentially terminate nodes without a recorded last_used time. However, I'm not familiar with this part of the code base so I may have overlooked some other implications.

Any thoughts on pros/cons of this compared to alternative approaches? I'll try and carve out some time later this week to implement whatever we decide on.
I think the main advantage of RequestSpotInstance would be the built-in cancellation. However, I think this is something we can add separately, for example, by cancelling the underlying HTTP request, or running the request in another process and killing the process to kill all its outstanding requests.

Another thing is that RequestSpotInstance is AWS-specific, and would not address the more general problem of instance creation requests hanging for other reasons. Whereas, a threading / process based solution would work for GCE as well and we can make it tolerant of hangs in the general case.
Could you elaborate on what other hangs in ```create_node``` we might expect to encounter, this might help me understand the design trade-offs better? Note we make (blocking) API calls in many other places, e.g. each call to nodes() retrieves all the currently running EC2 instances.

If there are some other cases where this is likely to arise then I'd favor the threading solution, but otherwise I feel a single-threaded approach is cleaner. I'm also not sure Amazon's API is clever enough to interpret closing the HTTP session as corresponding to cancelling the spot request. 

@hartikainen do you know if the API call to start an instance in GCE ever blocks for a significant amount of time? The [documentation](https://cloud.google.com/compute/docs/instances/create-start-instance) doesn't make this clear. If so this is definitely an argument in favor of threading. If not then this problem feels like an AWS-specific quirk that we should fix in the AWS node provider.


I believe that even a normal instance request can take a while if there are insufficient instances available (e.g., I've seen this when requesting large numbers of ondemand GPU instances).

My preference here is to avoid additional AWS dependencies such as createSpotInstances, for the sake of longer-term maintainability, I'm willing to increase architectural complexity a bit to avoid these special cases.
@AdamGleave Good question. I don't know for sure how long GCE instance creation might take, but I do know that the blocking doesn't happen on the library level. [instances.insert](https://cloud.google.com/compute/docs/reference/rest/v1/instances/insert) method returns an [operation](https://cloud.google.com/deployment-manager/docs/reference/latest/operations), which we then [explicitly wait to complete](https://github.com/ray-project/ray/blob/940c1b1fd1d5b168cdbab3e63f501d94eea23b4d/python/ray/autoscaler/gcp/config.py#L51).

Resolving patch:
diff --git a/python/ray/autoscaler/autoscaler.py b/python/ray/autoscaler/autoscaler.py
--- a/python/ray/autoscaler/autoscaler.py
+++ b/python/ray/autoscaler/autoscaler.py
@@ -2,10 +2,14 @@
 from __future__ import division
 from __future__ import print_function
 
+import copy
 import json
 import hashlib
+import math
 import os
+import queue
 import subprocess
+import threading
 import time
 import traceback
 
@@ -16,8 +20,8 @@
 import yaml
 
 from ray.ray_constants import AUTOSCALER_MAX_NUM_FAILURES, \
-    AUTOSCALER_MAX_CONCURRENT_LAUNCHES, AUTOSCALER_UPDATE_INTERVAL_S, \
-    AUTOSCALER_HEARTBEAT_TIMEOUT_S
+    AUTOSCALER_MAX_LAUNCH_BATCH, AUTOSCALER_MAX_CONCURRENT_LAUNCHES,\
+    AUTOSCALER_UPDATE_INTERVAL_S, AUTOSCALER_HEARTBEAT_TIMEOUT_S
 from ray.autoscaler.node_provider import get_node_provider, \
     get_default_config
 from ray.autoscaler.updater import NodeUpdaterProcess
@@ -199,6 +203,64 @@ def _info(self):
         }
 
 
+class NodeLauncher(threading.Thread):
+    def __init__(self, queue, pending, *args, **kwargs):
+        self.queue = queue
+        self.pending = pending
+        self.provider = None
+        super(NodeLauncher, self).__init__(*args, **kwargs)
+
+    def _launch_node(self, config, count):
+        if self.provider is None:
+            self.provider = get_node_provider(config["provider"],
+                                              config["cluster_name"])
+
+        tag_filters = {TAG_RAY_NODE_TYPE: "worker"}
+        before = self.provider.nodes(tag_filters=tag_filters)
+        launch_hash = hash_launch_conf(config["worker_nodes"], config["auth"])
+        self.provider.create_node(
+            config["worker_nodes"], {
+                TAG_RAY_NODE_NAME: "ray-{}-worker".format(
+                    config["cluster_name"]),
+                TAG_RAY_NODE_TYPE: "worker",
+                TAG_RAY_NODE_STATUS: "uninitialized",
+                TAG_RAY_LAUNCH_CONFIG: launch_hash,
+            }, count)
+        after = self.provider.nodes(tag_filters=tag_filters)
+        if set(after).issubset(before):
+            print("Warning: No new nodes reported after node creation")
+
+    def run(self):
+        while True:
+            config, count = self.queue.get()
+            try:
+                self._launch_node(config, count)
+            finally:
+                self.pending.dec(count)
+
+
+class ConcurrentCounter():
+    def __init__(self):
+        self._value = 0
+        self._lock = threading.Lock()
+
+    def inc(self, count):
+        with self._lock:
+            self._value += count
+            return self._value
+
+    def dec(self, count):
+        with self._lock:
+            assert self._value >= count, "counter cannot go negative"
+            self._value -= count
+            return self._value
+
+    @property
+    def value(self):
+        with self._lock:
+            return self._value
+
+
 class StandardAutoscaler(object):
     """The autoscaling control loop for a Ray cluster.
 
@@ -220,6 +282,7 @@ class StandardAutoscaler(object):
     def __init__(self,
                  config_path,
                  load_metrics,
+                 max_launch_batch=AUTOSCALER_MAX_LAUNCH_BATCH,
                  max_concurrent_launches=AUTOSCALER_MAX_CONCURRENT_LAUNCHES,
                  max_failures=AUTOSCALER_MAX_NUM_FAILURES,
                  process_runner=subprocess,
@@ -233,6 +296,7 @@ def __init__(self,
                                           self.config["cluster_name"])
 
         self.max_failures = max_failures
+        self.max_launch_batch = max_launch_batch
         self.max_concurrent_launches = max_concurrent_launches
         self.verbose_updates = verbose_updates
         self.process_runner = process_runner
@@ -246,6 +310,17 @@ def __init__(self,
         self.last_update_time = 0.0
         self.update_interval_s = update_interval_s
 
+        # Node launchers
+        self.launch_queue = queue.Queue()
+        self.num_launches_pending = ConcurrentCounter()
+        max_batches = math.ceil(
+            max_concurrent_launches / float(max_launch_batch))
+        for i in range(int(max_batches)):
+            node_launcher = NodeLauncher(
+                queue=self.launch_queue, pending=self.num_launches_pending)
+            node_launcher.daemon = True
+            node_launcher.start()
+
         # Expand local file_mounts to allow ~ in the paths. This can't be done
         # earlier when the config is written since we might be on different
         # platform and the expansion would result in wrong path.
@@ -278,6 +353,7 @@ def _update(self):
             return
 
         self.last_update_time = time.time()
+        num_pending = self.num_launches_pending.value
         nodes = self.workers()
         print(self.debug_string(nodes))
         self.load_metrics.prune_active_ips(
@@ -318,9 +394,11 @@ def _update(self):
 
         # Launch new nodes if needed
         target_num = self.target_num_workers()
-        if len(nodes) < target_num:
-            self.launch_new_node(
-                min(self.max_concurrent_launches, target_num - len(nodes)))
+        num_nodes = len(nodes) + num_pending
+        if num_nodes < target_num:
+            max_allowed = min(self.max_launch_batch,
+                              self.max_concurrent_launches - num_pending)
+            self.launch_new_node(min(max_allowed, target_num - num_nodes))
             print(self.debug_string())
 
         # Process any completed updates
@@ -453,27 +531,19 @@ def can_update(self, node_id):
 
     def launch_new_node(self, count):
         print("StandardAutoscaler: Launching {} new nodes".format(count))
-        num_before = len(self.workers())
-        self.provider.create_node(
-            self.config["worker_nodes"], {
-                TAG_RAY_NODE_NAME: "ray-{}-worker".format(
-                    self.config["cluster_name"]),
-                TAG_RAY_NODE_TYPE: "worker",
-                TAG_RAY_NODE_STATUS: "uninitialized",
-                TAG_RAY_LAUNCH_CONFIG: self.launch_hash,
-            }, count)
-        if len(self.workers()) <= num_before:
-            print("Warning: Num nodes failed to increase after node creation")
+        self.num_launches_pending.inc(count)
+        config = copy.deepcopy(self.config)
+        self.launch_queue.put((config, count))
 
     def workers(self):
-        return self.provider.nodes(tag_filters={
-            TAG_RAY_NODE_TYPE: "worker",
-        })
+        return self.provider.nodes(tag_filters={TAG_RAY_NODE_TYPE: "worker"})
 
     def debug_string(self, nodes=None):
         if nodes is None:
             nodes = self.workers()
         suffix = ""
+        if self.num_launches_pending:
+            suffix += " ({} pending)".format(self.num_launches_pending.value)
         if self.updaters:
             suffix += " ({} updating)".format(len(self.updaters))
         if self.num_failed_updates:
diff --git a/python/ray/ray_constants.py b/python/ray/ray_constants.py
--- a/python/ray/ray_constants.py
+++ b/python/ray/ray_constants.py
@@ -16,6 +16,11 @@ def env_integer(key, default):
 # is a safety feature to prevent e.g. runaway node launches.
 AUTOSCALER_MAX_NUM_FAILURES = env_integer("AUTOSCALER_MAX_NUM_FAILURES", 5)
 
+# The maximum number of nodes to launch in a single request.
+# Multiple requests may be made for this batch size, up to
+# the limit of AUTOSCALER_MAX_CONCURRENT_LAUNCHES.
+AUTOSCALER_MAX_LAUNCH_BATCH = env_integer("AUTOSCALER_MAX_LAUNCH_BATCH", 5)
+
 # Max number of nodes to launch at a time.
 AUTOSCALER_MAX_CONCURRENT_LAUNCHES = env_integer(
     "AUTOSCALER_MAX_CONCURRENT_LAUNCHES", 10)


Test patch:
diff --git a/test/autoscaler_test.py b/test/autoscaler_test.py
--- a/test/autoscaler_test.py
+++ b/test/autoscaler_test.py
@@ -4,6 +4,7 @@
 
 import shutil
 import tempfile
+import threading
 import time
 import unittest
 import yaml
@@ -50,6 +51,8 @@ def __init__(self):
         self.next_id = 0
         self.throw = False
         self.fail_creates = False
+        self.ready_to_create = threading.Event()
+        self.ready_to_create.set()
 
     def nodes(self, tag_filters):
         if self.throw:
@@ -75,6 +78,7 @@ def external_ip(self, node_id):
         return self.mock_nodes[node_id].external_ip
 
     def create_node(self, node_config, tags, count):
+        self.ready_to_create.wait()
         if self.fail_creates:
             return
         for _ in range(count):
@@ -182,6 +186,20 @@ def waitFor(self, condition):
             time.sleep(.1)
         raise Exception("Timed out waiting for {}".format(condition))
 
+    def waitForNodes(self, expected, comparison=None, tag_filters={}):
+        MAX_ITER = 50
+        for i in range(MAX_ITER):
+            n = len(self.provider.nodes(tag_filters))
+            if comparison is None:
+                comparison = self.assertEqual
+            try:
+                comparison(n, expected)
+                return
+            except Exception:
+                if i == MAX_ITER - 1:
+                    raise
+            time.sleep(.1)
+
     def create_provider(self, config, cluster_name):
         assert self.provider
         return self.provider
@@ -241,9 +259,9 @@ def testScaleUp(self):
             config_path, LoadMetrics(), max_failures=0, update_interval_s=0)
         self.assertEqual(len(self.provider.nodes({})), 0)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
 
     def testTerminateOutdatedNodesGracefully(self):
         config = SMALL_CLUSTER.copy()
@@ -254,16 +272,16 @@ def testTerminateOutdatedNodesGracefully(self):
         self.provider.create_node({}, {TAG_RAY_NODE_TYPE: "worker"}, 10)
         autoscaler = StandardAutoscaler(
             config_path, LoadMetrics(), max_failures=0, update_interval_s=0)
-        self.assertEqual(len(self.provider.nodes({})), 10)
+        self.waitForNodes(10)
 
         # Gradually scales down to meet target size, never going too low
         for _ in range(10):
             autoscaler.update()
-            self.assertLessEqual(len(self.provider.nodes({})), 5)
-            self.assertGreaterEqual(len(self.provider.nodes({})), 4)
+            self.waitForNodes(5, comparison=self.assertLessEqual)
+            self.waitForNodes(4, comparison=self.assertGreaterEqual)
 
         # Eventually reaches steady state
-        self.assertEqual(len(self.provider.nodes({})), 5)
+        self.waitForNodes(5)
 
     def testDynamicScaling(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -271,28 +289,115 @@ def testDynamicScaling(self):
         autoscaler = StandardAutoscaler(
             config_path,
             LoadMetrics(),
+            max_launch_batch=5,
             max_concurrent_launches=5,
             max_failures=0,
             update_interval_s=0)
-        self.assertEqual(len(self.provider.nodes({})), 0)
+        self.waitForNodes(0)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
 
         # Update the config to reduce the cluster size
         new_config = SMALL_CLUSTER.copy()
         new_config["max_workers"] = 1
         self.write_config(new_config)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 1)
+        self.waitForNodes(1)
 
         # Update the config to reduce the cluster size
         new_config["min_workers"] = 10
         new_config["max_workers"] = 10
         self.write_config(new_config)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 6)
+        self.waitForNodes(6)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 10)
+        self.waitForNodes(10)
+
+    def testDelayedLaunch(self):
+        config_path = self.write_config(SMALL_CLUSTER)
+        self.provider = MockProvider()
+        autoscaler = StandardAutoscaler(
+            config_path,
+            LoadMetrics(),
+            max_launch_batch=5,
+            max_concurrent_launches=5,
+            max_failures=0,
+            update_interval_s=0)
+        self.assertEqual(len(self.provider.nodes({})), 0)
+
+        # Update will try to create, but will block until we set the flag
+        self.provider.ready_to_create.clear()
+        autoscaler.update()
+        self.assertEqual(autoscaler.num_launches_pending.value, 2)
+        self.assertEqual(len(self.provider.nodes({})), 0)
+
+        # Set the flag, check it updates
+        self.provider.ready_to_create.set()
+        self.waitForNodes(2)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
+
+        # Update the config to reduce the cluster size
+        new_config = SMALL_CLUSTER.copy()
+        new_config["max_workers"] = 1
+        self.write_config(new_config)
+        autoscaler.update()
+        self.assertEqual(len(self.provider.nodes({})), 1)
+
+    def testDelayedLaunchWithFailure(self):
+        config = SMALL_CLUSTER.copy()
+        config["min_workers"] = 10
+        config["max_workers"] = 10
+        config_path = self.write_config(config)
+        self.provider = MockProvider()
+        autoscaler = StandardAutoscaler(
+            config_path,
+            LoadMetrics(),
+            max_launch_batch=5,
+            max_concurrent_launches=8,
+            max_failures=0,
+            update_interval_s=0)
+        self.assertEqual(len(self.provider.nodes({})), 0)
+
+        # update() should launch a wave of 5 nodes (max_launch_batch)
+        # Force this first wave to block.
+        rtc1 = self.provider.ready_to_create
+        rtc1.clear()
+        autoscaler.update()
+        # Synchronization: wait for launchy thread to be blocked on rtc1
+        if hasattr(rtc1, '_cond'):  # Python 3.5
+            waiters = rtc1._cond._waiters
+        else:  # Python 2.7
+            waiters = rtc1._Event__cond._Condition__waiters
+        self.waitFor(lambda: len(waiters) == 1)
+        self.assertEqual(autoscaler.num_launches_pending.value, 5)
+        self.assertEqual(len(self.provider.nodes({})), 0)
+
+        # Call update() to launch a second wave of 3 nodes,
+        # as 5 + 3 = 8 = max_concurrent_launches.
+        # Make this wave complete immediately.
+        rtc2 = threading.Event()
+        self.provider.ready_to_create = rtc2
+        rtc2.set()
+        autoscaler.update()
+        self.waitForNodes(3)
+        self.assertEqual(autoscaler.num_launches_pending.value, 5)
+
+        # The first wave of 5 will now tragically fail
+        self.provider.fail_creates = True
+        rtc1.set()
+        self.waitFor(lambda: autoscaler.num_launches_pending.value == 0)
+        self.assertEqual(len(self.provider.nodes({})), 3)
+
+        # Retry the first wave, allowing it to succeed this time
+        self.provider.fail_creates = False
+        autoscaler.update()
+        self.waitForNodes(8)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
+
+        # Final wave of 2 nodes
+        autoscaler.update()
+        self.waitForNodes(10)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
 
     def testUpdateThrottling(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -300,16 +405,22 @@ def testUpdateThrottling(self):
         autoscaler = StandardAutoscaler(
             config_path,
             LoadMetrics(),
+            max_launch_batch=5,
             max_concurrent_launches=5,
             max_failures=0,
             update_interval_s=10)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         new_config = SMALL_CLUSTER.copy()
         new_config["max_workers"] = 1
         self.write_config(new_config)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)  # not updated yet
+        # not updated yet
+        # note that node termination happens in the main thread, so
+        # we do not need to add any delay here before checking
+        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
 
     def testLaunchConfigChange(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -317,18 +428,18 @@ def testLaunchConfigChange(self):
         autoscaler = StandardAutoscaler(
             config_path, LoadMetrics(), max_failures=0, update_interval_s=0)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
 
         # Update the config to change the node type
         new_config = SMALL_CLUSTER.copy()
         new_config["worker_nodes"]["InstanceType"] = "updated"
         self.write_config(new_config)
-        existing_nodes = set(self.provider.nodes({}))
+        self.provider.ready_to_create.clear()
         for _ in range(5):
             autoscaler.update()
-        new_nodes = set(self.provider.nodes({}))
-        self.assertEqual(len(new_nodes), 2)
-        self.assertEqual(len(new_nodes.intersection(existing_nodes)), 0)
+        self.waitForNodes(0)
+        self.provider.ready_to_create.set()
+        self.waitForNodes(2)
 
     def testIgnoresCorruptedConfig(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -336,15 +447,19 @@ def testIgnoresCorruptedConfig(self):
         autoscaler = StandardAutoscaler(
             config_path,
             LoadMetrics(),
+            max_launch_batch=10,
             max_concurrent_launches=10,
             max_failures=0,
             update_interval_s=0)
         autoscaler.update()
+        self.waitForNodes(2)
 
         # Write a corrupted config
         self.write_config("asdf")
         for _ in range(10):
             autoscaler.update()
+        time.sleep(0.1)
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         self.assertEqual(len(self.provider.nodes({})), 2)
 
         # New a good config again
@@ -353,7 +468,7 @@ def testIgnoresCorruptedConfig(self):
         new_config["max_workers"] = 10
         self.write_config(new_config)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 10)
+        self.waitForNodes(10)
 
     def testMaxFailures(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -372,12 +487,12 @@ def testLaunchNewNodeOnOutOfBandTerminate(self):
             config_path, LoadMetrics(), max_failures=0, update_interval_s=0)
         autoscaler.update()
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         for node in self.provider.mock_nodes.values():
             node.state = "terminated"
         self.assertEqual(len(self.provider.nodes({})), 0)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
 
     def testConfiguresNewNodes(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -393,7 +508,7 @@ def testConfiguresNewNodes(self):
             update_interval_s=0)
         autoscaler.update()
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         for node in self.provider.mock_nodes.values():
             node.state = "running"
         assert len(
@@ -401,9 +516,7 @@ def testConfiguresNewNodes(self):
                 TAG_RAY_NODE_STATUS: "uninitialized"
             })) == 2
         autoscaler.update()
-        self.waitFor(
-            lambda: len(self.provider.nodes(
-                {TAG_RAY_NODE_STATUS: "up-to-date"})) == 2)
+        self.waitForNodes(2, tag_filters={TAG_RAY_NODE_STATUS: "up-to-date"})
 
     def testReportsConfigFailures(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -419,7 +532,7 @@ def testReportsConfigFailures(self):
             update_interval_s=0)
         autoscaler.update()
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         for node in self.provider.mock_nodes.values():
             node.state = "running"
         assert len(
@@ -427,9 +540,8 @@ def testReportsConfigFailures(self):
                 TAG_RAY_NODE_STATUS: "uninitialized"
             })) == 2
         autoscaler.update()
-        self.waitFor(
-            lambda: len(self.provider.nodes(
-                {TAG_RAY_NODE_STATUS: "update-failed"})) == 2)
+        self.waitForNodes(
+            2, tag_filters={TAG_RAY_NODE_STATUS: "update-failed"})
 
     def testConfiguresOutdatedNodes(self):
         config_path = self.write_config(SMALL_CLUSTER)
@@ -445,13 +557,11 @@ def testConfiguresOutdatedNodes(self):
             update_interval_s=0)
         autoscaler.update()
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         for node in self.provider.mock_nodes.values():
             node.state = "running"
         autoscaler.update()
-        self.waitFor(
-            lambda: len(self.provider.nodes(
-                {TAG_RAY_NODE_STATUS: "up-to-date"})) == 2)
+        self.waitForNodes(2, tag_filters={TAG_RAY_NODE_STATUS: "up-to-date"})
         runner.calls = []
         new_config = SMALL_CLUSTER.copy()
         new_config["worker_setup_commands"] = ["cmdX", "cmdY"]
@@ -472,33 +582,37 @@ def testScaleUpBasedOnLoad(self):
             config_path, lm, max_failures=0, update_interval_s=0)
         self.assertEqual(len(self.provider.nodes({})), 0)
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 2)
+        self.waitForNodes(2)
         autoscaler.update()
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         self.assertEqual(len(self.provider.nodes({})), 2)
 
         # Scales up as nodes are reported as used
         lm.update("172.0.0.0", {"CPU": 2}, {"CPU": 0})
         lm.update("172.0.0.1", {"CPU": 2}, {"CPU": 0})
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 4)
+        self.waitForNodes(4)
         lm.update("172.0.0.2", {"CPU": 2}, {"CPU": 0})
         autoscaler.update()
-        self.assertEqual(len(self.provider.nodes({})), 6)
+        self.waitForNodes(6)
 
         # Holds steady when load is removed
         lm.update("172.0.0.0", {"CPU": 2}, {"CPU": 2})
         lm.update("172.0.0.1", {"CPU": 2}, {"CPU": 2})
         autoscaler.update()
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         self.assertEqual(len(self.provider.nodes({})), 6)
 
         # Scales down as nodes become unused
         lm.last_used_time_by_ip["172.0.0.0"] = 0
         lm.last_used_time_by_ip["172.0.0.1"] = 0
         autoscaler.update()
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         self.assertEqual(len(self.provider.nodes({})), 4)
         lm.last_used_time_by_ip["172.0.0.2"] = 0
         lm.last_used_time_by_ip["172.0.0.3"] = 0
         autoscaler.update()
+        self.assertEqual(autoscaler.num_launches_pending.value, 0)
         self.assertEqual(len(self.provider.nodes({})), 2)
 
     def testRecoverUnhealthyWorkers(self):
@@ -515,12 +629,11 @@ def testRecoverUnhealthyWorkers(self):
             node_updater_cls=NodeUpdaterThread,
             update_interval_s=0)
         autoscaler.update()
+        self.waitForNodes(2)
         for node in self.provider.mock_nodes.values():
             node.state = "running"
         autoscaler.update()
-        self.waitFor(
-            lambda: len(self.provider.nodes(
-                {TAG_RAY_NODE_STATUS: "up-to-date"})) == 2)
+        self.waitForNodes(2, tag_filters={TAG_RAY_NODE_STATUS: "up-to-date"})
 
         # Mark a node as unhealthy
         lm.last_heartbeat_time_by_ip["172.0.0.0"] = 0