Repository: DataBiosphere/toil

Issue:
32MB is too small of a memory requirement for encapsulating jobs and gets them OOM-killed on Slurm
See this issue in Cactus on Slurm: https://github.com/ComparativeGenomicsToolkit/cactus/issues/72

```
Issued job 'Job' d/s/jobQ0PKIE with job batch system ID: 314 and cores: 0, disk: 1.0 M, and memory: 32.0 M
Job ended successfully: 'Job' d/s/jobQ0PKIE
Issued job 'Job' d/s/jobQ0PKIE with job batch system ID: 315 and cores: 0, disk: 1.0 M, and memory: 32.0 M
Job ended successfully: 'Job' d/s/jobQ0PKIE
Issued job 'Job' d/s/jobQ0PKIE with job batch system ID: 316 and cores: 0, disk: 1.0 M, and memory: 32.0 M
Job ended successfully: 'Job' d/s/jobQ0PKIE
Issued job 'Job' d/s/jobQ0PKIE with job batch system ID: 317 and cores: 0, disk: 1.0 M, and memory: 32.0 M
Job failed with exit value 135: 'Job' d/s/jobQ0PKIE
No log file is present, despite job failing: 'Job' d/s/jobQ0PKIE
Due to failure we are reducing the remaining retry count of job 'Job' d/s/jobQ0PKIE with ID d/s/jobQ0PKIE to 5
We have increased the default memory of the failed job 'Job' d/s/jobQ0PKIE to 2147483648 bytes
Issued job 'Job' d/s/jobQ0PKIE with job batch system ID: 318 and cores: 0, disk: 1.0 M, and memory: 2.0 G
Job ended successfully: 'Job' d/s/jobQ0PKIE
Issued job 'mappingQualityRescoring' z/e/jobZTuM4S with job batch system ID: 319 and cores: 13, disk: 2.0 G, and memory: 2.0 G
Got message from job at time 04-05-2019 10:23:24: Input cigar file has 7362 lines
Got message from job at time 04-05-2019 10:23:24: Filtered, non-overlapping primary cigar file has 11273 lines
Got message from job at time 04-05-2019 10:23:24: Filtered, non-overlapping secondary cigar file has 31711 lines
Job ended successfully: 'mappingQualityRescoring' z/e/jobZTuM4S
```

The encapsulating job encapsulates the predecessor of a `mappingQualityRescoring` job. It gets repeatedly scheduled and immediately killed before Toil itself can properly start up or make any sort of log file, and this is interpreted as "success". But the job doesn't get removed, because it didn't actually run, so it gets scheduled again and again. Eventually it manages to report a failure (exit code 135, signal 7, SIGBUS (???)), and gets its memory bumped up to the default, so it can start Toil properly and finish the encapsulating job.

We need to change the requirements [here](https://github.com/DataBiosphere/toil/blob/8057b47f3841f177e040def8ef1b2f1968f0cd94/src/toil/job.py#L1574) to not be less resources than Toil itself needs to start up.

We *should* come up with a way to prevent jobs from being scheduled with fewer resources than Toil itself needs, even if they ask for them.

And we *should* also fix our notion that silence from a job equals success, or at least not report that kind of "success" to the user as success in the log. Toil is launching the same job repeatedly and hearing nothing back from any of the invocations; it should complain that the job got lost, treat it as a failure, and do a retry, not assert that it succeeded.


Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -1571,7 +1571,8 @@ def __init__(self, job):
         self.encapsulatedJob = job
         Job.addChild(self, job)
         # Use small resource requirements for dummy Job instance.
-        self.encapsulatedFollowOn = Job(disk='1M', memory='32M', cores=0.1)
+        # But not too small, or the job won't have enough resources to safely start up Toil.
+        self.encapsulatedFollowOn = Job(disk='100M', memory='512M', cores=0.1)
         Job.addFollowOn(self, self.encapsulatedFollowOn)
 
     def addChild(self, childJob):


Test patch: