Repository: DataBiosphere/toil

Issue:
Autoscaler crash when downscaling
Looks like this got introduced sometime in the past few months. Not sure why the tests didn't catch it:
```
Removing 2 preemptable nodes to get to desired cluster size of 6.
All nodes in cluster: [Instance:i-0d3ac67054db38221, Instance:i-03a4d1eecf3c8141b, Instance:i-0773c589ab7098f44, Instance:i-061ecaade928b2954, Instance:i-0588c4858cf74b9ab, Instance:i-0001994efad1708c3, Instance:i-0aeaab212e71c7ceb, Instance:i-0803c1872fb7aed1a, Instance:i-0b193afd
42a64543e, Instance:i-0d00830782fcbab7c]
All workers found in cluster: [Instance:i-03a4d1eecf3c8141b, Instance:i-0773c589ab7098f44, Instance:i-061ecaade928b2954, Instance:i-0588c4858cf74b9ab, Instance:i-0001994efad1708c3, Instance:i-0aeaab212e71c7ceb, Instance:i-0803c1872fb7aed1a, Instance:i-0b193afd42a64543e, Instance:i-
0d00830782fcbab7c]
non-preemptable workers found in cluster: [Instance:i-0b193afd42a64543e]
Cluster already at desired size of 1. Nothing to do.
All nodes in cluster: [Instance:i-0b193afd42a64543e, Instance:i-0d3ac67054db38221, Instance:i-0588c4858cf74b9ab, Instance:i-03a4d1eecf3c8141b, Instance:i-061ecaade928b2954, Instance:i-0001994efad1708c3, Instance:i-0d00830782fcbab7c, Instance:i-0803c1872fb7aed1a, Instance:i-0aeaab21
2e71c7ceb, Instance:i-0773c589ab7098f44]
All workers found in cluster: [Instance:i-0b193afd42a64543e, Instance:i-0588c4858cf74b9ab, Instance:i-03a4d1eecf3c8141b, Instance:i-061ecaade928b2954, Instance:i-0001994efad1708c3, Instance:i-0d00830782fcbab7c, Instance:i-0803c1872fb7aed1a, Instance:i-0aeaab212e71c7ceb, Instance:i-
0773c589ab7098f44]
preemptable workers found in cluster: [Instance:i-0588c4858cf74b9ab, Instance:i-03a4d1eecf3c8141b, Instance:i-061ecaade928b2954, Instance:i-0001994efad1708c3, Instance:i-0d00830782fcbab7c, Instance:i-0803c1872fb7aed1a, Instance:i-0aeaab212e71c7ceb, Instance:i-0773c589ab7098f44]
Consolidating state between mesos and provisioner
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Nodes considered to terminate: i-0001994efad1708c3 at 54.214.201.144 i-0803c1872fb7aed1a at 54.191.57.56 i-03a4d1eecf3c8141b at 54.244.16.175 i-0aeaab212e71c7ceb at 54.201.40.245 i-0d00830782fcbab7c at 54.244.64.198 i-0773c589ab7098f44 at 54.202.143.18 i-0588c4858cf74b9ab at 54.200.5.84 i-061ecaade928b2954 at 54.218.38.119
Did not find 172.31.33.179 in preemptable static nodes
Did not find 172.31.41.161 in preemptable static nodes
Did not find 172.31.37.253 in preemptable static nodes
Did not find 172.31.46.93 in preemptable static nodes
Did not find 172.31.46.35 in preemptable static nodes
Not terminating instances i-0d00830782fcbab7c at 54.244.64.198. Node info: <toil.batchSystems.abstractBatchSystem.NodeInfo object at 0x7f9a6cd96f50>
Did not find 172.31.44.186 in preemptable static nodes
Did not find 172.31.42.92 in preemptable static nodes
Did not find 172.31.32.40 in preemptable static nodes
Exception in thread preemptable-scaler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 478, in tryRun
    self.totalNodes = self.setNodeCount(numNodes=estimatedNodes, preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 533, in setNodeCount
    force=force)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 557, in _removeNodes
    nodesToTerminate = self.chooseNodes(nodeToNodeInfo, force, preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 608, in chooseNodes
    self.scaler.provisioner.remainingBillingInterval(node) <= 0.15]
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 230, in remainingBillingInterval
    return awsRemainingBillingInterval(node)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/__init__.py", line 44, in awsRemainingBillingInterval
    return 1.0 - partialBillingInterval(instance)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/__init__.py", line 39, in partialBillingInterval
    launch_time = parse_iso_utc(instance.launchTime)
AttributeError: 'tuple' object has no attribute 'launchTime'
```

The line numbers may be off because this is using slightly different code, but [here's](https://github.com/BD2KGenomics/toil/blob/a272a4bed2692886e123c3ff56047267b025aca9/src/toil/provisioners/clusterScaler.py#L556) the line with the bug. It didn't get updated when the nodesToTerminate variable got changed to a list of tuples.

Self-assigning this and will make a PR shortly, but first I want to make sure other parts of the autoscaler are still working as well.
"Worker node at %s is not reporting executor information"
The debug log has a ton of messages like this:
```
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
Worker node at %s is not reporting executor information
```

Obviously something is supposed to go in `%s`, but not sure what yet. I'll fix this at the same time as #1728.


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -553,9 +553,9 @@ def chooseNodes(self, nodeToNodeInfo, force=False, preemptable=False):
                               )
         if not force:
             # don't terminate nodes that still have > 15% left in their allocated (prepaid) time
-            nodesToTerminate = [node for node in nodesToTerminate if
+            nodesToTerminate = [(node, i) for node, i in nodesToTerminate if
                                 self.scaler.provisioner.remainingBillingInterval(node) <= 0.15]
-        return [node for node,_ in nodesToTerminate]
+        return [node for node, _ in nodesToTerminate]
 
     def getNodes(self, preemptable):
         """
@@ -616,7 +616,7 @@ def _getInfo(allMesosNodes, ip):
         for node, ip in ((node, node.privateIP) for node in provisionerNodes):
             info = None
             if ip not in recentMesosNodes:
-                logger.debug("Worker node at %s is not reporting executor information")
+                logger.debug("Worker node at %s is not reporting executor information", ip)
                 # we don't have up to date information about the node
                 info = _getInfo(allMesosNodes, ip)
             else:


Test patch: