Repository: DataBiosphere/toil

Issue:
Mesos and single-machine treat default requirements differently
The following user script

```
import sys
from toil.job import Job

def hello( name ):
    return "Hello, %s!" % name

if __name__ == '__main__':
    options = Job.Runner.getDefaultArgumentParser().parse_args()
    job = Job.wrapFn( hello, "world" )
    result = Job.Runner.startToil( job, options )
    assert result == 'Hello, world!'
```

passes with single machine and hangs with Mesos.

Command lines used:

```
python2.7 hello_world.py --batchSystem=mesos --mesosMaster=mesos-master:5050 aws:us-west-2:foo
python2.7 hello_world.py aws:us-west-2:foo
```

Mesos and single-machine treat default requirements differently
The following user script

```
import sys
from toil.job import Job

def hello( name ):
    return "Hello, %s!" % name

if __name__ == '__main__':
    options = Job.Runner.getDefaultArgumentParser().parse_args()
    job = Job.wrapFn( hello, "world" )
    result = Job.Runner.startToil( job, options )
    assert result == 'Hello, world!'
```

passes with single machine and hangs with Mesos.

Command lines used:

```
python2.7 hello_world.py --batchSystem=mesos --mesosMaster=mesos-master:5050 aws:us-west-2:foo
python2.7 hello_world.py aws:us-west-2:foo
```



Discussion:
Assuming this is on a t2.micro instance, we expect the batch system to hang/fail. Micros have 1 core while the default core value is 2. 

To communicate this behavior to the user we should keep track of the max resource offer per resource type and log a warning if we get a job with a resource requirement greater than that max offer.

Sort of related to #548. That issue describes SingleMachine to actually do the right thing of hanging when a three-core workload is being run on a two-core machine. 

Also, it looks like the default defaultCores is 1, not 2 as @cket claims.

Is it possible that Mesos or the framework subtracts some fraction of a core (say .1) for itself, leaving only .9 for Toil?

The executor and framework currently don't reserve any cores for themselves, although the executor used to reserve 0.1 cores for itself. 

Is this still running on mesos 0.22? The original reason for setting executor resource reqs was that future versions of mesos would require them, but they were so inconvenient that we removed them.

If we updated to a more recent version of mesos, its possible that the executor now automatically reserves cores for itself, causing this behavior.

It looks like we are now using 0.25 but looking through the list of changes here:

http://mesos.apache.org/documentation/latest/upgrades/

doesn't seem to list mandatory executor resource requirements as a change. Probably still worth testing on an actual cluster, however.

Assuming this is on a t2.micro instance, we expect the batch system to hang/fail. Micros have 1 core while the default core value is 2. 

To communicate this behavior to the user we should keep track of the max resource offer per resource type and log a warning if we get a job with a resource requirement greater than that max offer.

Sort of related to #548. That issue describes SingleMachine to actually do the right thing of hanging when a three-core workload is being run on a two-core machine. 

Also, it looks like the default defaultCores is 1, not 2 as @cket claims.

Is it possible that Mesos or the framework subtracts some fraction of a core (say .1) for itself, leaving only .9 for Toil?

The executor and framework currently don't reserve any cores for themselves, although the executor used to reserve 0.1 cores for itself. 

Is this still running on mesos 0.22? The original reason for setting executor resource reqs was that future versions of mesos would require them, but they were so inconvenient that we removed them.

If we updated to a more recent version of mesos, its possible that the executor now automatically reserves cores for itself, causing this behavior.

It looks like we are now using 0.25 but looking through the list of changes here:

http://mesos.apache.org/documentation/latest/upgrades/

doesn't seem to list mandatory executor resource requirements as a change. Probably still worth testing on an actual cluster, however.


Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -23,7 +23,6 @@
 from threading import Thread
 from threading import Lock, Condition
 from Queue import Queue, Empty
-
 from toil.batchSystems.abstractBatchSystem import AbstractBatchSystem
 
 log = logging.getLogger(__name__)
@@ -45,13 +44,19 @@ class SingleMachineBatchSystem(AbstractBatchSystem):
     concurrently.
     """
 
+    physicalMemory = os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES')
+
     def __init__(self, config, maxCores, maxMemory, maxDisk):
         if maxCores > self.numCores:
             log.warn('Limiting maxCores to CPU count of system (%i).', self.numCores)
             maxCores = self.numCores
+        if maxMemory > self.physicalMemory:
+            log.warn('Limiting maxMemory to physically available memory (%i).', self.physicalMemory)
+            maxMemory = self.physicalMemory
         AbstractBatchSystem.__init__(self, config, maxCores, maxMemory, maxDisk)
         assert self.maxCores >= self.minCores
         assert self.maxMemory >= 1
+
         # The scale allows the user to apply a factor to each task's cores requirement, thereby
         # squeezing more tasks onto each core (scale < 1) or stretching tasks over more cores
         # (scale > 1).
@@ -124,8 +129,8 @@ def worker(self, inputQueue):
                                 statusCode = popen.wait()
                                 if 0 != statusCode:
                                     if statusCode != -9 or not info.killIntended:
-                                        log.error( "Got exit code %i (indicating failure) from "
-                                                   "command '%s'.", statusCode, jobCommand)
+                                        log.error("Got exit code %i (indicating failure) from "
+                                                  "command '%s'.", statusCode, jobCommand)
                             finally:
                                 self.runningJobs.pop(jobID)
                         finally:
@@ -148,7 +153,7 @@ def issueBatchJob(self, command, memory, cores, disk):
         assert cores >= self.minCores
         assert memory <= self.maxMemory, ('The job is requesting {} bytes of memory, more than '
                                           'the maximum of {} this batch system was configured '
-                                          'with.'.format( memory, self.maxMemory))
+                                          'with.'.format(memory, self.maxMemory))
 
         self.checkResourceRequest(memory, cores, disk)
         log.debug("Issuing the command: %s with memory: %i, cores: %i, disk: %i" % (


Test patch: