Repository: DataBiosphere/toil

Issue:
AWSProvisioner requests non-integer number of instances, killing workflow
Looks like this is probably caused by the "run-time length correction" functionality. I might get around to fixing this tonight--if not, probably early tomorrow, as it looks like this may be a blocker for us, so assigning this to myself for the time being.

```
Running bin packing for node shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360) and 25 job(s).
Done running bin packing for node shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360) and 25 job(s) resulting in 1 node reservations.
Historical avg. runtime (60134.5817099) is less than current avg. runtime (60302.0889423) and cluster is being well utilised (3 running jobs), increasing cluster requirement by: 1.00278553916
Adding 0 preemptable nodes to compensate for a deficit of 0 non-preemptable ones.
Estimating that cluster needs 1.00278553916 non-preemptable nodes of shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360), from current size of 1, given a queue size of 2, the number of jobs per node estimated to be 25.0, an alpha parameter of 0.8 and a run
-time length correction of 1.00278553916.
Changing the number of non-preemptable nodes from 1 to 1.00278553916.
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
All nodes in cluster [Instance:i-0312d75c8689a023e, Instance:i-0cf3c579845fb423d, Instance:i-07b8c0284a13d69b5]
Workers found in cluster [Instance:i-0312d75c8689a023e]
Adding 0 non-preemptable nodes to get to desired cluster size of 1.
Device mapping: {'/dev/xvdb': <boto.ec2.blockdevicemapping.BlockDeviceType object at 0x7f77942a78d0>, '/dev/xvdc': <boto.ec2.blockdevicemapping.BlockDeviceType object at 0x7f77942a76d0>}
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
There are no queued tasks. Declining Mesos offers.
Declining offer a6c0ccb3-7aff-4c92-84dc-a709b1a68f59-O63757.
Using default docker registry of quay.io/joelarmstrong as TOIL_DOCKER_REGISTRY is not set.
Using default docker name of toil as TOIL_DOCKER_NAME is not set.
Overriding docker appliance of quay.io/joelarmstrong/toil:3.7.0a1-1335235114eabcac96e52dd015ce40984ac9572e-dirty with quay.io/joelarmstrong/toil:latest from TOIL_APPLIANCE_SELF.
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
Launching 0.0027855391634 non-preemptable nodes
Creating r3.8xlarge instance(s) ... 
Exception in thread scaler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 406, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/abstractProvisioner.py", line 177, in setNodeCount
    preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 566, in _addNodes
    spec=kwargs, num_instances=numNodes)
  File "/usr/local/lib/python2.7/dist-packages/cgcloud/lib/ec2.py", line 414, in create_ondemand_instances
    **spec ).instances
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid integer value 0.0027855391634</Message></Error></Errors><RequestID>faa542ae-86e4-4917-97e1-26cf60a50785</RequestID></Response>
```
AWSProvisioner requests non-integer number of instances, killing workflow
Looks like this is probably caused by the "run-time length correction" functionality. I might get around to fixing this tonight--if not, probably early tomorrow, as it looks like this may be a blocker for us, so assigning this to myself for the time being.

```
Running bin packing for node shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360) and 25 job(s).
Done running bin packing for node shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360) and 25 job(s) resulting in 1 node reservations.
Historical avg. runtime (60134.5817099) is less than current avg. runtime (60302.0889423) and cluster is being well utilised (3 running jobs), increasing cluster requirement by: 1.00278553916
Adding 0 preemptable nodes to compensate for a deficit of 0 non-preemptable ones.
Estimating that cluster needs 1.00278553916 non-preemptable nodes of shape _Shape(wallTime=3600, memory=261993005056, cores=32, disk=687194767360), from current size of 1, given a queue size of 2, the number of jobs per node estimated to be 25.0, an alpha parameter of 0.8 and a run
-time length correction of 1.00278553916.
Changing the number of non-preemptable nodes from 1 to 1.00278553916.
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
All nodes in cluster [Instance:i-0312d75c8689a023e, Instance:i-0cf3c579845fb423d, Instance:i-07b8c0284a13d69b5]
Workers found in cluster [Instance:i-0312d75c8689a023e]
Adding 0 non-preemptable nodes to get to desired cluster size of 1.
Device mapping: {'/dev/xvdb': <boto.ec2.blockdevicemapping.BlockDeviceType object at 0x7f77942a78d0>, '/dev/xvdc': <boto.ec2.blockdevicemapping.BlockDeviceType object at 0x7f77942a76d0>}
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
There are no queued tasks. Declining Mesos offers.
Declining offer a6c0ccb3-7aff-4c92-84dc-a709b1a68f59-O63757.
Using default docker registry of quay.io/joelarmstrong as TOIL_DOCKER_REGISTRY is not set.
Using default docker name of toil as TOIL_DOCKER_NAME is not set.
Overriding docker appliance of quay.io/joelarmstrong/toil:3.7.0a1-1335235114eabcac96e52dd015ce40984ac9572e-dirty with quay.io/joelarmstrong/toil:latest from TOIL_APPLIANCE_SELF.
Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
Cached credentials exist and are still fresh.
Launching 0.0027855391634 non-preemptable nodes
Creating r3.8xlarge instance(s) ... 
Exception in thread scaler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/bd2k/util/threading.py", line 51, in run
    self.tryRun( )
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 406, in tryRun
    preemptable=self.preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/abstractProvisioner.py", line 177, in setNodeCount
    preemptable=preemptable)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/aws/awsProvisioner.py", line 566, in _addNodes
    spec=kwargs, num_instances=numNodes)
  File "/usr/local/lib/python2.7/dist-packages/cgcloud/lib/ec2.py", line 414, in create_ondemand_instances
    **spec ).instances
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid integer value 0.0027855391634</Message></Error></Errors><RequestID>faa542ae-86e4-4917-97e1-26cf60a50785</RequestID></Response>
```
"beta" scaler parameter has never done anything
This line:

```python
# Use inertia parameter to stop small fluctuations
if estimatedNodes <= self.totalNodes * self.scaler.config.betaInertia <= estimatedNodes:
```

is obviously not what is intended (it reduces to `estimatedNodes == self.totalNodes * self.scaler.config.betaInertia`), but it's had this wrong behavior ever since @benedictpaten drafted it. It even got refactored, preserving the wrong behavior perfectly but with more Pythonic style!

What's intended is definitely something like:

```
delta = self.totalNodes * self.scaler.config.betaInertia
if self.totalNodes - delta <= estimatedNodes <= self.totalNodes + delta:
```


Discussion:


Oops...

Resolving patch:
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -347,8 +347,8 @@ def tryRun(self):
                 numberOfRunningJobs, currentAvgRuntime  = self.scaler.leader.getNumberAndAvgRuntimeOfCurrentlyRunningJobs()
                 
                 # Average runtime of recently completed jobs
-                historicalAvgRuntime = sum(map(lambda jS : jS.wallTime, recentJobShapes))
-            
+                historicalAvgRuntime = sum(map(lambda jS : jS.wallTime, recentJobShapes))/len(recentJobShapes)
+
                 # Ratio of avg. runtime of currently running and completed jobs
                 runtimeCorrection = float(currentAvgRuntime)/historicalAvgRuntime if currentAvgRuntime > historicalAvgRuntime and numberOfRunningJobs >= estimatedNodes else 1.0
                 
@@ -358,7 +358,7 @@ def tryRun(self):
                     logger.warn("Historical avg. runtime (%s) is less than current avg. runtime (%s) and cluster"
                                 " is being well utilised (%s running jobs), increasing cluster requirement by: %s" % 
                                 (historicalAvgRuntime, currentAvgRuntime, numberOfRunningJobs, runtimeCorrection))
-                    estimatedNodes *= runtimeCorrection
+                    estimatedNodes = int(round(estimatedNodes * runtimeCorrection))
 
                 # If we're the non-preemptable scaler, we need to see if we have a deficit of
                 # preemptable nodes that we should compensate for.
@@ -373,17 +373,18 @@ def tryRun(self):
                                 'non-preemptable ones.', compensationNodes, _preemptableNodeDeficit)
                     estimatedNodes += compensationNodes
 
-                fix_my_name = (0 if nodesToRunRecentJobs <= 0
+                jobsPerNode = (0 if nodesToRunRecentJobs <= 0
                                else len(recentJobShapes) / float(nodesToRunRecentJobs))
                 logger.info('Estimating that cluster needs %s %s of shape %s, from current '
                              'size of %s, given a queue size of %s, the number of jobs per node '
                              'estimated to be %s, an alpha parameter of %s and a run-time length correction of %s.',
                              estimatedNodes, self.nodeTypeString, self.nodeShape, 
-                             self.totalNodes, queueSize, fix_my_name,
+                             self.totalNodes, queueSize, jobsPerNode,
                              self.scaler.config.alphaPacking, runtimeCorrection)
 
                 # Use inertia parameter to stop small fluctuations
-                if estimatedNodes <= self.totalNodes * self.scaler.config.betaInertia <= estimatedNodes:
+                delta = self.totalNodes * max(0.0, self.scaler.config.betaInertia - 1.0)
+                if self.totalNodes - delta <= estimatedNodes <= self.totalNodes + delta:
                     logger.debug('Difference in new (%s) and previous estimates in number of '
                                  '%s (%s) required is within beta (%s), making no change.',
                                  estimatedNodes, self.nodeTypeString, self.totalNodes, self.scaler.config.betaInertia)


Test patch: