Repository: DataBiosphere/toil

Issue:
Autoscaler can try to launch zero nodes
I caught the autoscaler trying to launch 0 nodes, getting rebuked by Amazon, and then having an error in the scaler thread.

```
2018-06-21 20:51:08,554 - toil.provisioners.clusterScaler - DEBUG - Nodes considered to terminate: i-05f6149a065ead637 at 34.219.47.64
INFO:toil.provisioners.clusterScaler:Terminating 1 instance(s).
2018-06-21 20:51:08,555 - toil.provisioners.clusterScaler - INFO - Terminating 1 instance(s).
INFO:toil.provisioners.aws.awsProvisioner:Terminating instance(s): [u'i-05f6149a065ead637']
2018-06-21 20:51:08,555 - toil.provisioners.aws.awsProvisioner - INFO - Terminating instance(s): [u'i-05f6149a065ead637']
INFO:toil.provisioners.aws.awsProvisioner:Instance(s) terminated.
2018-06-21 20:51:08,688 - toil.provisioners.aws.awsProvisioner - INFO - Instance(s) terminated.
INFO:toil.provisioners.clusterScaler:Cluster contains 1 instances
2018-06-21 20:51:08,758 - toil.provisioners.clusterScaler - INFO - Cluster contains 1 instances
INFO:toil.provisioners.clusterScaler:Cluster contains 1 instances of type i3.8xlarge (1 ignored and draining jobs until they can be saf
ely terminated)
2018-06-21 20:51:08,758 - toil.provisioners.clusterScaler - INFO - Cluster contains 1 instances of type i3.8xlarge (1 ignored and drain
ing jobs until they can be safely terminated)
INFO:toil.provisioners.clusterScaler:Unignoring 1 nodes because we want to scale back up again.
2018-06-21 20:51:08,758 - toil.provisioners.clusterScaler - INFO - Unignoring 1 nodes because we want to scale back up again.
INFO:toil.provisioners.clusterScaler:Adding 0 non-preemptable nodes to get to desired cluster size of 1.
2018-06-21 20:51:08,758 - toil.provisioners.clusterScaler - INFO - Adding 0 non-preemptable nodes to get to desired cluster size of 1.
INFO:toil:Overriding docker registry of quay.io/ucsc_cgl with quay.io/adamnovak from TOIL_DOCKER_REGISTRY.
2018-06-21 20:51:09,438 - toil - INFO - Overriding docker registry of quay.io/ucsc_cgl with quay.io/adamnovak from TOIL_DOCKER_REGISTRY
.
INFO:toil:Using default docker name of toil as TOIL_DOCKER_NAME is not set.
2018-06-21 20:51:09,438 - toil - INFO - Using default docker name of toil as TOIL_DOCKER_NAME is not set.
INFO:toil:Overriding docker appliance of quay.io/adamnovak/toil:3.17.0a1-79c241c0eb273a3af6952202e38bd8d0c4bfe05c with quay.io/adamnova
k/toil:3.17.0a1-79c241c0eb273a3af6952202e38bd8d0c4bfe05c from TOIL_APPLIANCE_SELF.
2018-06-21 20:51:09,438 - toil - INFO - Overriding docker appliance of quay.io/adamnovak/toil:3.17.0a1-79c241c0eb273a3af6952202e38bd8d0
c4bfe05c with quay.io/adamnovak/toil:3.17.0a1-79c241c0eb273a3af6952202e38bd8d0c4bfe05c from TOIL_APPLIANCE_SELF.
INFO:toil.provisioners.aws.awsProvisioner:Launching 0 non-preemptable nodes
2018-06-21 20:51:09,693 - toil.provisioners.aws.awsProvisioner - INFO - Launching 0 non-preemptable nodes
INFO:toil.lib.ec2:Creating i3.8xlarge instance(s) ... 
2018-06-21 20:51:09,693 - toil.lib.ec2 - INFO - Creating i3.8xlarge instance(s) ... 
ERROR:toil.provisioners.clusterScaler:Exception encountered in scaler thread. Making a best-effort attempt to keep going, but things ma
y go wrong from now on.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 857, in tryRun
    self.scaler.updateClusterSize(estimatedNodeCounts)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 526, in updateClusterSize
    newNodeCount = self.setNodeCount(nodeType=nodeType, numNodes=estimatedNodeCount, preemptable=nodeShape.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 596, in setNodeCount
    preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 612, in _addNodes
    return self.provisioner.addNodes(nodeType=nodeType, numNodes=numNodes, preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 274, in addNodes
    spec=kwargs, num_instances=numNodes)
  File "/usr/local/lib/python2.7/dist-packages/toil/lib/ec2.py", line 365, in create_ondemand_instances
    **spec).instances
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Minimum instance count must be greater than zero</Message></Error></Errors><RequestID>77a28cd2-eb9d-492c-b36c-6934abe89a0d</RequestID></Response>
```

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-285)
┆Issue Number: TOIL-285



Discussion:
Hahaha, whoops. Hopefully non-fatal?

It should be an easy fix. This is probably my bad so self-assigning.

Resolving patch:
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -579,8 +579,7 @@ def setNodeCount(self, nodeType, numNodes, preemptable=False, force=False):
                     delta = numNodes - (numCurrentNodes - numIgnoredNodes)
                 else:
                     delta = numNodes - numCurrentNodes
-                if delta > 0:
-                    if numIgnoredNodes > 0:
+                if delta > 0 and numIgnoredNodes > 0:
                         # We can un-ignore a few nodes to compensate for the additional nodes we want.
                         numNodesToUnignore = min(delta, numIgnoredNodes)
                         logger.info('Unignoring %i nodes because we want to scale back up again.' % numNodesToUnignore)
@@ -588,6 +587,7 @@ def setNodeCount(self, nodeType, numNodes, preemptable=False, force=False):
                         for node in ignoredNodes[:numNodesToUnignore]:
                             self.ignoredNodes.remove(node.privateIP)
                             self.leader.batchSystem.unignoreNode(node.privateIP)
+                if delta > 0:
                     logger.info('Adding %i %s nodes to get to desired cluster size of %i.',
                                 delta,
                                 'preemptable' if preemptable else 'non-preemptable',


Test patch:
diff --git a/src/toil/test/provisioners/clusterScalerTest.py b/src/toil/test/provisioners/clusterScalerTest.py
--- a/src/toil/test/provisioners/clusterScalerTest.py
+++ b/src/toil/test/provisioners/clusterScalerTest.py
@@ -383,6 +383,30 @@ def testPreemptableDeficitIsSet(self):
         scaler.updateClusterSize(estimatedNodeCounts)
         self.assertEqual(scaler.preemptableNodeDeficit['c4.8xlarge'], 0)
 
+    def testNoLaunchingIfDeltaAlreadyMet(self):
+        """
+        Check that the scaler doesn't try to launch "0" more instances if
+        the delta was able to be met by unignoring nodes.
+        """
+        # We have only one node type for simplicity
+        self.provisioner.nodeTypes = ['c4.8xlarge']
+        self.provisioner.nodeShapes = [c4_8xlarge]
+        scaler = ClusterScaler(self.provisioner, self.leader, self.config)
+        # Pretend there is one ignored worker in the cluster
+        self.provisioner.getProvisionedWorkers = MagicMock(
+            return_value=[Node('127.0.0.1', '127.0.0.1', 'testNode',
+                               datetime.datetime.now().isoformat(),
+                               nodeType='c4.8xlarge', preemptable=True)])
+        scaler.ignoredNodes.add('127.0.0.1')
+        # Exercise the updateClusterSize logic
+        self.provisioner.addNodes = MagicMock()
+        scaler.updateClusterSize({c4_8xlarge: 1})
+        self.assertFalse(self.provisioner.addNodes.called,
+                         "addNodes was called when no new nodes were needed")
+        self.assertEqual(len(scaler.ignoredNodes), 0,
+                         "The scaler didn't unignore an ignored node when "
+                         "scaling up")
+
     def testBetaInertia(self):
         # This is really high, but makes things easy to calculate.
         self.config.betaInertia = 0.5