Repository: DataBiosphere/toil

Issue:
Provisioner can't fit jobs it could fit before, and can't explain itself.
I woke up to an 802 megabyte Toil log this morning. It seems that the provisioner can't fit jobs that used to work with node types of "i3.8xlarge,i3.8xlarge:0.90" into any of "i3.8xlarge,r4.8xlarge:0.60".

The log looks like this.

```
DEBUG:toil.provisioners.clusterScaler:Running bin packing for node shapes [<toil.provisioners.abstractProvisioner.Shape object at 0x7fa0441a3dd0>, <toil.provisioners.abstractProvisioner.Shape object at 0x7fa0441a3f50>] and 35 job(s).
2018-06-06 23:22:07,570 - toil.provisioners.clusterScaler - DEBUG - Running bin packing for node shapes [<toil.provisioners.abstractProvisioner.Shape object at 0x7fa0441a3dd0>, <toil.provisioners.abstractProvisioner.Shape object at 0x7fa0441a3f50>] and 35 job(s).
WARNING:toil.provisioners.clusterScaler:Couldn't fit job with requirements <toil.provisioners.abstractProvisioner.Shape object at 0x7fa044094fd0> into any nodes in the nodeTypes list.
2018-06-06 23:22:07,571 - toil.provisioners.clusterScaler - WARNING - Couldn't fit job with requirements <toil.provisioners.abstractProvisioner.Shape object at 0x7fa044094fd0> into any nodes in the nodeTypes list.
ERROR:toil.provisioners.clusterScaler:Exception encountered in scaler thread. Making a best-effort attempt to keep going, but things may go wrong from now on.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 857, in tryRun
    currentNodeCounts)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 460, in getEstimatedNodeCounts
    goalTime=self.targetTime)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 288, in binPacking
    bpf.binPack(jobShapes)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 85, in binPack
    self.addJobShape(jS)
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 104, in addJobShape
    nodeReservations = self.nodeReservations[chosenNodeShape]
KeyError: None
```

It then continued throwing the same KeyError several times a second for many hours, but not shutting down the workflow or the cluster.

I think there are 3 bugs here:
- [x] There are constant and absurdly frequent KeyErrors instead of a graceful shutdown
- [x] Job requirements and node sizes are reported as `<toil.provisioners.abstractProvisioner.Shape object at 0x7fa0441a3dd0>` or similar instead of actual numbers, which prevents me from knowing how big the problem job is and how bit a node I need to provide.
- [ ] The jobs aren't getting scheduled onto the i3.8xlarge instances, where they ought to fit.

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-24)
┆Issue Number: TOIL-24



Discussion:
Thanks for logging this.  Shape objects should print as a custom string however (in Toil 3.16.0):
https://github.com/BD2KGenomics/toil/blob/master/src/toil/provisioners/abstractProvisioner.py#L80

What version of Toil is this?
@DailyDreaming: I think if you use `__str__` on a list it uses `__repr__` on the elements inside rather than `__str__`. (Who knows why...) But that's easy to fix just by adding in a repr.

This is probably on me. I recently messed with the bin packing--it looks like it will end up seizing up if you have a job larger than every single one of your nodes. Well, oops. That's a unit test to add.

Yeah, the thousands of KeyErrors a second is probably not ideal. I can add a throttling bit in there, though I want to push back on the graceful shutdown idea: I have personally had thousands of dollars of wasted compute because the scaler shut down multiple times on stupid transient AWS errors, so I am super jaded about toil's eagerness to shutdown. Toil has played the game of "retry only on these certain exceptions we've seen thrown by boto" and lost every time, because there are about a hundred different exceptions boto can choose to throw. Better IMO to just keep the scaler thread spinning, since scaling is sort of a non-essential "bonus" feature. That way, your cluster can keep chugging along finishing jobs until the scaler recovers (if ever).
@joelarmstrong My mistake, forgot about `__repr__` when that was written.

Maybe we could add the caveat that only if there are no currently active jobs waiting to finish, it should attempt to gracefully shutdown?  That way nothing's cancelled and the user can still specify new instances to fulfill the needs of whatever jobs are holding it up.
About the third point: Toil has some history of underestimating disk space on i3  (https://github.com/BD2KGenomics/toil/issues/2057).  Is it possible the fix for this somehow got reverted in 3.16.0? 
This is definitely 3.16.0.

Maybe shutting down the whole workflow is too dramatic a response? But if the scaler thread is itself having internal errors, and no jobs are running, I'm not sure it makes sense to keep it up, because it's not making progress. On the other hand, we don't want to shut down just because the scaler got a bad reply from Amazon.



Resolving patch:
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -90,6 +90,14 @@ def __str__(self):
                 self.disk,
                 self.preemptable)
 
+    def __repr__(self):
+        return "Shape(wallTime=%s, memory=%s, cores=%s, disk=%s, preemptable=%s)" % \
+               (self.wallTime,
+                self.memory,
+                self.cores,
+                self.disk,
+                self.preemptable)
+
 class AbstractProvisioner(with_metaclass(ABCMeta, object)):
     """
     An abstract base class to represent the interface for provisioning worker nodes to use in a
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -68,8 +68,6 @@ class BinPackedFit(object):
     def __init__(self, nodeShapes, targetTime=defaultTargetTime):
         self.nodeShapes = sorted(nodeShapes)
         self.targetTime = targetTime
-
-        # {_Shape(wallTime=3600, memory=1073741824, cores=1, disk=8589934592, preemptable=False): []}
         self.nodeReservations = {nodeShape:[] for nodeShape in nodeShapes}
 
     def binPack(self, jobShapes):
@@ -99,6 +97,7 @@ def addJobShape(self, jobShape):
         if chosenNodeShape is None:
             logger.warning("Couldn't fit job with requirements %r into any nodes in the nodeTypes "
                            "list." % jobShape)
+            return
 
         # grab current list of job objects appended to this nodeType
         nodeReservations = self.nodeReservations[chosenNodeShape]
@@ -836,8 +835,8 @@ def addCompletedJob(self, job, wallTime):
 
     def tryRun(self):
         while not self.stop:
-            try:
-                with throttle(self.scaler.config.scaleInterval):
+            with throttle(self.scaler.config.scaleInterval):
+                try:
                     queuedJobs = self.scaler.leader.getJobs()
                     queuedJobShapes = [
                         Shape(wallTime=self.scaler.getAverageRuntime(
@@ -858,9 +857,9 @@ def tryRun(self):
                     self.scaler.updateClusterSize(estimatedNodeCounts)
                     if self.stats:
                         self.stats.checkStats()
-            except:
-                logger.exception("Exception encountered in scaler thread. Making a best-effort "
-                                 "attempt to keep going, but things may go wrong from now on.")
+                except:
+                    logger.exception("Exception encountered in scaler thread. Making a best-effort "
+                                     "attempt to keep going, but things may go wrong from now on.")
         self.scaler.shutDown()
 
 class ClusterStats(object):


Test patch:
diff --git a/src/toil/test/provisioners/clusterScalerTest.py b/src/toil/test/provisioners/clusterScalerTest.py
--- a/src/toil/test/provisioners/clusterScalerTest.py
+++ b/src/toil/test/provisioners/clusterScalerTest.py
@@ -248,6 +248,19 @@ def testPathologicalCase(self):
         # Hopefully we didn't assign just one node to cover all those jobs.
         self.assertNotEqual(self.bpf.getRequiredNodes(), {r3_8xlarge: 1, c4_8xlarge_preemptable: 0})
 
+    def testJobTooLargeForAllNodes(self):
+        """
+        If a job is too large for all node types, the scaler should print a
+        warning, but definitely not crash.
+        """
+        # Takes more RAM than an r3.8xlarge
+        largerThanR3 = Shape(wallTime=3600,
+                             memory=h2b('360G'),
+                             cores=32,
+                             disk=h2b('600G'),
+                             preemptable=False)
+        self.bpf.addJobShape(largerThanR3)
+        # If we got here we didn't crash.
 
 class ClusterScalerTest(ToilTest):
     def setUp(self):