Repository: DataBiosphere/toil

Issue:
singleMachine batchSystem may be really inefficient
Currently, the singleMachine batch system launches a huge number of threads: 10 times the maxCores number. All these threads contend on a single queue waiting for jobs to appear. When they do get a job, they hold onto it and try to contend until a "resource pool" has enough resources to continue, and they can get an exclusive lock on the pool so they can decrement it.

This seems pretty inefficient. We could get the same behavior by just having one thread spin waiting for resources and monitoring the Popen objects. It's not totally clear how much speed we'd get from doing it with fewer threads, but it definitely shouldn't hurt. Plus, this will get rid of the uber-annoying "Recieved termination signal" tens or hundreds of times when exiting a single-machine Toil workflow.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-208)
┆Issue Number: TOIL-208



Discussion:

This can be easily and efficiently addressed by using an
event-oriented programming.

In it's simplest form, a single thread could be responsible for
starting the subprocesses.  Each subprocess would have stdout
pipe coming back to the leader.  Select or another I/O polling
mechanism then waits for any pipe to become read-ready; if an
EOF is returned, then that child process has terminated.

   https://docs.python.org/2/library/select.html

The python-3 async module could be a higher-level way to
implemented it with trollius
(https://pypi.python.org/pypi/trollius) providing an
implementation for python-2.

Probably other approaches, although, while select/poll is a
little tedious to program, it is rather straight forward in what
it does.

Recommend against using SIGCHLD, as the global nature or signals
can be very difficult to implement.

Joel Armstrong <notifications@github.com> writes:
> Currently, the singleMachine batch system launches a huge number of threads: 10
> times the maxCores number. All these threads contend on a single queue waiting
> for jobs to appear. When they do get a job, they hold onto it and try to
> contend until a "resource pool" has enough resources to continue, and they can
> get an exclusive lock on the pool so they can decrement it.
> 
> This seems pretty inefficient. We could get the same behavior by just having
> one thread spin waiting for resources and monitoring the Popen objects. It's
> not totally clear how much speed we'd get from doing it with fewer threads, but
> it definitely shouldn't hurt. Plus, this will get rid of the uber-annoying
> "Recieved termination signal" tens or hundreds of times when exiting a
> single-machine Toil workflow.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.*
> 

I have a quick draft of a better single-machine batch system, though it's not ready yet and I won't have much time to fix it up in the near future. But that lets us run some tests to quantify how inefficient the current system is. When using 64 cores, sorting 10000 lines using the sort test takes 42.5 +/- 13.6 s. Under my experimental batch system without the contention, it takes 29.2 +/- 5.8 s. So for very large systems, this can reduce toil overhead by 30%. Obviously the sort test runtime is pretty much only overhead, so the gains in practice would be quite a bit smaller.

With 8 cores there's very little noticeable difference, less than 10%. (There's almost an order of magnitude less contention.) So fixing this may not do all that much for people running on their laptops.
worth having though.

On Tue, Oct 3, 2017 at 3:46 PM, Joel Armstrong <notifications@github.com>
wrote:

> I have a quick draft of a better single-machine batch system, though it's
> not ready yet and I won't have much time to fix it up in the near future.
> But that lets us run some tests to quantify how inefficient the current
> system is. When using 64 cores, sorting 10000 lines using the sort test
> takes 42.5 +/- 13.6 s. Under my experimental batch system without the
> contention, it takes 29.2 +/- 5.8 s. So for very large systems, this can
> reduce toil overhead by 50%. Obviously the sort test runtime is pretty much
> only overhead, so the gains in practice would be quite a bit smaller.
>
> With 8 cores there's very little noticeable difference, less than 10%.
> (There's almost an order of magnitude less contention.) So fixing this may
> not do all that much for people running on their laptops.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1902#issuecomment-334000977>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AMXKjmiSvy9_JYUPKyfYA4IGGZIwWsEDks5sorlVgaJpZM4PgFJ3>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "cgloud-jenkins" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cgloud-jenkins+unsubscribe@googlegroups.com.
> To post to this group, send email to cgloud-jenkins@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/cgloud-jenkins/BD2KGenomics/toil/issues/1902/334000977%40github.com
> <https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil/issues/1902/334000977%40github.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Benedict
(calendar invites: bpaten@ucsc.edu,
 appointments: Tina Bernard <tibernar@ucsc.edu>)

Is it ready enough for a PR?
It might be time to actually redo the single machine batch system, what with all the issues we get from people complaining it starts too many/the wrong number of threads (https://github.com/DataBiosphere/toil/issues/2308#issuecomment-588353477, #2928, etc.).

@joelarmstrong Did your reimplementation get lost, or is it around somewhere? I could step up and polish it up for inclusion if you wanted to hand it off. Otherwise I can try starting over myself.

Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -25,12 +25,13 @@
 import os
 import time
 import math
-from threading import Thread
+import subprocess
+import traceback
+from threading import Thread, Event
 from threading import Lock, Condition
 from six.moves.queue import Empty, Queue
 
 import toil
-import subprocess
 from toil.batchSystems.abstractBatchSystem import BatchSystemSupport
 from toil.lib.threading import cpu_count
 from toil import worker as toil_worker
@@ -41,8 +42,21 @@
 
 class SingleMachineBatchSystem(BatchSystemSupport):
     """
-    The interface for running jobs on a single machine, runs all the jobs you give it as they
-    come in, but in parallel.
+    The interface for running jobs on a single machine, runs all the jobs you
+    give it as they come in, but in parallel.
+
+    Uses a single "daddy" thread to manage a fleet of child processes.
+    
+    Communication with the daddy thread happens via two queues: one queue of
+    jobs waiting to be run (the input queue), and one queue of jobs that are
+    finished/stopped and need to be returned by getUpdatedBatchJob (the output
+    queue).
+
+    When the batch system is shut down, the daddy thread is stopped.
+
+    If running in debug-worker mode, jobs are run immediately as they are sent
+    to the batch system, in the sending thread, and the daddy thread is not
+    run. But the queues are still used.
     """
 
     @classmethod
@@ -58,9 +72,7 @@ def supportsWorkerCleanup(cls):
     minCores = 0.1
     """
     The minimal fractional CPU. Tasks with a smaller core requirement will be rounded up to this
-    value. One important invariant of this class is that each worker thread represents a CPU
-    requirement of minCores, meaning that we can never run more than numCores / minCores jobs
-    concurrently.
+    value. 
     """
     physicalMemory = toil.physicalMemory()
 
@@ -92,141 +104,370 @@ def __init__(self, config, maxCores, maxMemory, maxDisk):
                 
         self.debugWorker = config.debugWorker
         
-        # Number of worker threads that will be started
-        self.numWorkers = int(old_div(self.maxCores, self.minCores))
         # A counter to generate job IDs and a lock to guard it
         self.jobIndex = 0
         self.jobIndexLock = Lock()
+
         # A dictionary mapping IDs of submitted jobs to the command line
         self.jobs = {}
         """
         :type: dict[str,toil.job.JobNode]
         """
-        # A queue of jobs waiting to be executed. Consumed by the workers.
+
+        # A queue of jobs waiting to be executed. Consumed by the daddy thread.
         self.inputQueue = Queue()
-        # A queue of finished jobs. Produced by the workers.
+
+        # A queue of finished jobs. Produced by the daddy thread.
         self.outputQueue = Queue()
+
         # A dictionary mapping IDs of currently running jobs to their Info objects
         self.runningJobs = {}
         """
         :type: dict[str,Info]
         """
-        # The list of worker threads
-        self.workerThreads = []
+        
+        # These next two are only used outside debug-worker mode
+
+        # A dict mapping PIDs to Popen objects for running jobs.
+        # Jobs that don't fork are executed one at a time in the main thread.
+        self.children = {}
+        """
+        :type: dict[int,subprocess.Popen]
         """
-        :type list[Thread]
+        # A dict mapping child PIDs to the Job IDs they are supposed to be running.
+        self.childToJob = {}
+        """
+        :type: dict[int,str]
         """
-        # Variables involved with non-blocking resource acquisition
-        self.acquisitionTimeout = 5
-        self.acquisitionRetryDelay = 10
-        self.aquisitionCondition = Condition()
 
         # A pool representing available CPU in units of minCores
-        self.coreFractions = ResourcePool(self.numWorkers, 'cores', self.acquisitionTimeout)
-        # A lock to work around the lack of thread-safety in Python's subprocess module
-        self.popenLock = Lock()
+        self.coreFractions = ResourcePool(int(old_div(self.maxCores, self.minCores)), 'cores')
         # A pool representing available memory in bytes
-        self.memory = ResourcePool(self.maxMemory, 'memory', self.acquisitionTimeout)
+        self.memory = ResourcePool(self.maxMemory, 'memory')
         # A pool representing the available space in bytes
-        self.disk = ResourcePool(self.maxDisk, 'disk', self.acquisitionTimeout)
-
-        if not self.debugWorker:
-            log.debug('Setting up the thread pool with %i workers, '
-                     'given a minimum CPU fraction of %f '
-                     'and a maximum CPU value of %i.', self.numWorkers, self.minCores, maxCores)
-            for i in range(self.numWorkers):
-                worker = Thread(target=self.worker, args=(self.inputQueue,), daemon=True)
-                worker.start()
-                self.workerThreads.append(worker)
-        else:
+        self.disk = ResourcePool(self.maxDisk, 'disk')
+
+        # We use this event to signal shutdown
+        self.shuttingDown = Event()
+
+        # A thread in charge of managing all our child processes.
+        # Also takes care of resource accounting.
+        self.daddyThread = None
+        # If it breaks it will fill this in
+        self.daddyException = None
+
+        if self.debugWorker:
             log.debug('Started in worker debug mode.')
+        else:
+            self.daddyThread = Thread(target=self.daddy, daemon=True)
+            self.daddyThread.start()
+            log.debug('Started in normal mode.')
 
-    def _runWorker(self, jobCommand, jobID, environment):
+    def daddy(self):
         """
-        Run the jobCommand using the worker and wait for it to finish.
-        The worker is forked unless it is a '_toil_worker' job and
-        debugWorker is True.
+        Be the "daddy" thread.
+
+        Our job is to look at jobs from the input queue.
+        
+        If a job fits in the available resources, we allocate resources for it
+        and kick off a child process.
+
+        We also check on our children.
+
+        When a child finishes, we reap it, release its resources, and put its
+        information in the output queue.
         """
-        startTime = time.time()  # Time job is started
-        if self.debugWorker and "_toil_worker" in jobCommand:
-            # Run the worker without forking
+
+        try:
+            log.debug('Started daddy thread.')
+
+            while not self.shuttingDown.is_set():
+                # Main loop
+
+                while not self.shuttingDown.is_set():
+                    # Try to start as many jobs as we can try to start
+                    try:
+                        # Grab something from the input queue if available.
+                        args = self.inputQueue.get_nowait()
+                        jobCommand, jobID, jobCores, jobMemory, jobDisk, environment = args
+
+                        coreFractions = int(old_div(jobCores, self.minCores))
+                        
+                        # Try to start the child
+                        result = self._startChild(jobCommand, jobID,
+                            coreFractions, jobMemory, jobDisk, environment)
+
+                        log.debug('Tried to start job %s and got: %s', jobID, str(result))
+                        
+                        if result is None:
+                            # We did not get the resources to run this job.
+                            # Requeue last, so we can look at the next job.
+                            # TODO: Have some kind of condition the job can wait on,
+                            # but without threads (queues for jobs needing
+                            # cores/memory/disk individually)?
+                            self.inputQueue.put(args)
+                            break
+
+                        # Otherwise it's a PID if it succeeded, or False if it couldn't
+                        # start. But we don't care either way here.
+
+                    except Empty:
+                        # Nothing to run. Stop looking in the queue.
+                        break
+
+                # Now check on our children.
+                for done_pid in self._pollForDoneChildrenIn(self.children):
+                    # A child has actually finished.
+                    # Clean up after it.
+                    self._handleChild(done_pid)
+
+                # Then loop again: start and collect more jobs.
+                # TODO: It would be good to be able to wait on a new job or a finished child, whichever comes first.
+                # For now we just sleep and loop.
+                time.sleep(0.01)
+                    
+
+            # When we get here, we are shutting down.
+            
+            for popen in self.children.values():
+                # Kill all the children, going through popen to avoid signaling re-used PIDs.
+                popen.kill()
+            for popen in self.children.values():
+                # Reap all the children
+                popen.wait()
+            
+            # Then exit the thread.
+            return
+        except Exception as e:
+            log.critical('Unhandled exception in daddy thread: %s', traceback.format_exc())
+            # Pass the exception back to the main thread so it can stop the next person who calls into us.
+            self.daddyException = e
+            raise
+
+    def _checkOnDaddy(self):
+        if self.daddyException is not None:
+            # The daddy thread broke and we cannot do our job
+            log.critical('Propagating unhandled exception in daddy thread to main thread')
+            exc = self.daddyException
+            self.daddyException = None
+            raise exc
+
+    def _pollForDoneChildrenIn(self, pid_to_popen):
+        """
+        See if any children represented in the given dict from PID to Popen
+        object have finished.
+        
+        Return a collection of their PIDs.
+        
+        Guarantees that each child's exit code will be gettable via wait() on
+        the child's Popen object (i.e. does not reap the child, unless via
+        Popen).
+        """
+
+        # We keep our found PIDs in a set so we can work around waitid showing
+        # us the same one repeatedly.
+        ready = set()
+        
+        # Find the waitid function
+        waitid = getattr(os, 'waitid', None)
+
+        if callable(waitid):
+            # waitid exists (not Mac)
+
+            while True:
+                # Poll for any child to have exit, but don't reap it. Leave reaping
+                # to the Popen.
+                # TODO: What if someone else in Toil wants to do this syscall?
+                # TODO: Is this one-notification-per-done-child with WNOHANG? Or
+                # can we miss some? Or do we see the same one repeatedly until it
+                # is reaped?
+                try:
+                    siginfo = waitid(os.P_ALL, -1, os.WEXITED | os.WNOWAIT | os.WNOHANG)
+                except ChildProcessError:
+                    # This happens when there is nothing to wait on right now,
+                    # instead of the weird C behavior of overwriting a field in
+                    # a pointed-to struct.
+                    siginfo = None
+                if siginfo is not None and siginfo.si_pid in pid_to_popen and siginfo.si_pid not in ready:
+                    # Something new finished
+                    ready.add(siginfo.si_pid)
+                else:
+                    # Nothing we own that we haven't seen before has finished.
+                    return ready
+        else:
+            # On Mac there's no waitid and no way to wait and not reap.
+            # Fall back on polling all the Popen objects.
+            # To make this vaguely efficient we have to return done children in
+            # batches.
+            for pid, popen in pid_to_popen.items():
+                if popen.poll() is not None:
+                    # Process is done
+                    ready.add(pid)
+                    log.debug('Child %d has stopped', pid)
+            
+            # Return all the done processes we found
+            return ready
+
+    def _runDebugJob(self, jobCommand, jobID, environment):
+        """
+        Run the jobCommand right now, in the current thread.
+        May only be called in debug-worker mode.
+        Assumes resources are available.
+        """
+        
+        assert self.debugWorker
+
+    
+        # TODO: It is not possible to kill running jobs in forkless mode,
+        # because they are run immediately in the main thread.
+        info = Info(time.time(), None, None, killIntended=False)
+        self.runningJobs[jobID] = info
+
+        if jobCommand.startswith("_toil_worker "):
+            # We can actually run in this thread
             jobName, jobStoreLocator, jobStoreID = jobCommand.split()[1:] # Parse command
             jobStore = Toil.resumeJobStore(jobStoreLocator)
-            # TODO: The following does not yet properly populate self.runningJobs so it is not possible to kill
-            # running jobs in forkless mode - see the "None" value in place of popen
-            info = Info(time.time(), None, killIntended=False)
-            try:
-                self.runningJobs[jobID] = info
-                try:
-                    toil_worker.workerScript(jobStore, jobStore.config, jobName, jobStoreID, 
-                                             redirectOutputToLogFile=not self.debugWorker) # Call the worker
-                finally:
-                    self.runningJobs.pop(jobID)
-            finally:
-                if not info.killIntended:
-                    self.outputQueue.put((jobID, 0, time.time() - startTime))
+            toil_worker.workerScript(jobStore, jobStore.config, jobName, jobStoreID, 
+                                     redirectOutputToLogFile=not self.debugWorker) # Call the worker
         else:
-            with self.popenLock:
-                popen = subprocess.Popen(jobCommand,
-                                         shell=True,
-                                         env=dict(os.environ, **environment))
-            info = Info(time.time(), popen, killIntended=False)
-            try:
-                self.runningJobs[jobID] = info
-                try:
-                    statusCode = popen.wait()
-                    if statusCode != 0 and not info.killIntended:
-                        log.error("Got exit code %i (indicating failure) "
-                                  "from job %s.", statusCode, self.jobs[jobID])
-                finally:
-                    self.runningJobs.pop(jobID)
-            finally:
-                if not info.killIntended:
-                    self.outputQueue.put((jobID, statusCode, time.time() - startTime))
+            # Run synchronously. If starting or running the command fails, let the exception stop us.
+            subprocess.check_call(jobCommand,
+                                  shell=True,
+                                  env=dict(os.environ, **environment))
+
+        self.runningJobs.pop(jobID)
+        if not info.killIntended:
+            self.outputQueue.put((jobID, 0, time.time() - info.time))
+
+    def _startChild(self, jobCommand, jobID, coreFractions, jobMemory, jobDisk, environment):
+        """
+        Start a child process for the given job.
         
-    # Note: The input queue is passed as an argument because the corresponding attribute is reset
-    # to None in shutdown()
-
-    def worker(self, inputQueue):
-        while True:
-            if self.debugWorker and inputQueue.empty():
-                return
-            args = inputQueue.get()
-            if args is None:
-                break
-            jobCommand, jobID, jobCores, jobMemory, jobDisk, environment = args
-            while True:
-                try:
-                    coreFractions = int(old_div(jobCores, self.minCores))
-                    log.debug('Acquiring %i bytes of memory from a pool of %s.', jobMemory,
-                              self.memory)
-                    with self.memory.acquisitionOf(jobMemory):
-                        log.debug('Acquiring %i fractional cores from a pool of %s to satisfy a '
-                                  'request of %f cores', coreFractions, self.coreFractions,
-                                  jobCores)
-                        with self.coreFractions.acquisitionOf(coreFractions):
-                            with self.disk.acquisitionOf(jobDisk):
-                                self._runWorker(jobCommand, jobID, environment)
-
-                except ResourcePool.AcquisitionTimeoutException as e:
-                    log.debug('Could not acquire enough (%s) to run job (%s). Requested: (%s), '
-                              'Avaliable: %s. Sleeping for 10s.', e.resource, jobID, e.requested,
-                              e.available)
-                    with self.aquisitionCondition:
-                        # Make threads sleep for the given delay, or until another job finishes.
-                        # Whichever is sooner.
-                        self.aquisitionCondition.wait(timeout=self.acquisitionRetryDelay)
-                    continue
+        Allocate its required resources and save it and save it in our bookkeeping structures.
+
+        If the job is started, returns its PID.
+        If the job fails to start, reports it as failed and returns False.
+        If the job cannot get the resources it needs to start, returns None.
+        """
+
+        # We fill this in if we manage to actually start the child.
+        popen = None
+
+        # This is when we started working on the job.
+        startTime = time.time()
+
+        # See if we can fit the job in our resource pools right now.
+        if self.coreFractions.acquireNow(coreFractions):
+            # We got some cores
+            if self.memory.acquireNow(jobMemory):
+                # We got some memory
+                if self.disk.acquireNow(jobDisk):
+                    # We got the final resource, disk.
+                    # Actually run the job.
+                    # When it finishes we will release what it was using.
+                    # So it is important to not lose track of the child process.
+
+                    try:
+                        # Launch the job
+                        popen = subprocess.Popen(jobCommand,
+                                                 shell=True,
+                                                 env=dict(os.environ, **environment))
+                    except Exception:
+                        # If the job can't start, make sure we release resources now
+                        self.coreFractions.release(coreFractions)
+                        self.memory.release(jobMemory)
+                        self.disk.release(jobDisk)
+
+                        log.error('Could not start job %s: %s', jobID, traceback.format_exc())
+                        
+                        # Report as failed.
+                        # TODO: what should the exit code be?
+                        self.outputQueue.put((jobID, -1, 0))
+
+                        # Free resources
+                        self.coreFractions.release(coreFractions)
+                        self.memory.release(jobMemory)
+                        self.disk.release(jobDisk)
+
+                        # Complain it broke.
+                        return False
+                    else:
+                        # If the job did start, record it
+                        self.children[popen.pid] = popen
+                        # Make sure we can look it up by PID later
+                        self.childToJob[popen.pid] = jobID
+                        # Record that the job is running, and the resources it is using
+                        info = Info(startTime, popen, (coreFractions, jobMemory, jobDisk), killIntended=False)
+                        self.runningJobs[jobID] = info
+
+                        log.debug('Launched job %s as child %d', jobID, popen.pid)
+
+                        # Report success starting the job
+                        # Note that if a PID were somehow 0 it would look like False
+                        assert popen.pid != 0
+                        return popen.pid
                 else:
-                    log.debug('Finished job. self.coreFractions ~ %s and self.memory ~ %s',
-                              self.coreFractions.value, self.memory.value)
-                    with self.aquisitionCondition:
-                        # Wake up sleeping threads
-                        self.aquisitionCondition.notifyAll()
-                    break
+                    # We can't get disk, so free cores and memory
+                    self.coreFractions.release(coreFractions)
+                    self.memory.release(jobMemory)
+                    log.debug('Not enough disk to run job %s', jobID)
+            else:
+                # Free cores, since we can't get memory
+                self.coreFractions.release(coreFractions)
+                log.debug('Not enough memory to run job %s', jobID)
+        else:
+            log.debug('Not enough cores to run job %s', jobID)
+
+        # If we get here, we didn't succeed or fail starting the job.
+        # We didn't manage to get the resources.
+        # Report that.
+        return None
+
+    def _handleChild(self, pid):
+        """
+        Handle a child process PID that has finished.
+        The PID must be for a child job we started.
+        Not thread safe to run at the same time as we are making more children.
+
+        Remove the child from our bookkeeping structures and free its resources.
+        """
+        
+        # Look up the child
+        popen = self.children[pid]
+        jobID = self.childToJob[pid]
+        info = self.runningJobs[jobID]
+
+        # Unpack the job resources
+        (coreFractions, jobMemory, jobDisk) = info.resources
+
+        # Clean up our records of the job.
+        self.runningJobs.pop(jobID)
+        self.childToJob.pop(pid)
+        self.children.pop(pid)
+        
+        # See how the child did, and reap it.
+        statusCode = popen.wait()
+        if statusCode != 0 and not info.killIntended:
+            log.error("Got exit code %i (indicating failure) "
+                      "from job %s.", statusCode, self.jobs[jobID])
+        if not info.killIntended:
+            # Report if the job failed and we didn't kill it.
+            # If we killed it then it shouldn't show up in the queue.
+            self.outputQueue.put((jobID, statusCode, time.time() - info.time))
+
+        # Free up the job's resources.
+        self.coreFractions.release(coreFractions)
+        self.memory.release(jobMemory)
+        self.disk.release(jobDisk)
+
+        log.debug('Child %d for job %s succeeded', pid, jobID)
 
+        
     def issueBatchJob(self, jobNode):
         """Adds the command and resources to a queue to be run."""
+
+        self._checkOnDaddy()
+
         # Round cores to minCores and apply scale
         cores = math.ceil(jobNode.cores * self.scale / self.minCores) * self.minCores
         assert cores <= self.maxCores, ('The job {} is requesting {} cores, more than the maximum of '
@@ -244,21 +485,33 @@ def issueBatchJob(self, jobNode):
             jobID = self.jobIndex
             self.jobIndex += 1
         self.jobs[jobID] = jobNode.command
-        self.inputQueue.put((jobNode.command, jobID, cores, jobNode.memory,
-                             jobNode.disk, self.environment.copy()))
-        if self.debugWorker:  # then run immediately, blocking for return
-            self.worker(self.inputQueue)
+        
+        if self.debugWorker:
+            # Run immediately, blocking for return.
+            # Ignore resource requirements; we run one job at a time
+            self._runDebugJob(jobNode.command, jobID, self.environment.copy())
+        else:
+            # Queue the job for later
+            self.inputQueue.put((jobNode.command, jobID, cores, jobNode.memory,
+                                jobNode.disk, self.environment.copy()))
+            
+
         return jobID
 
     def killBatchJobs(self, jobIDs):
         """Kills jobs by ID."""
+
+        self._checkOnDaddy()
+
         log.debug('Killing jobs: {}'.format(jobIDs))
         for jobID in jobIDs:
             if jobID in self.runningJobs:
                 info = self.runningJobs[jobID]
                 info.killIntended = True
                 if info.popen != None:
-                    os.kill(info.popen.pid, 9)
+                    log.debug('Send kill to PID %s', info.popen.pid)
+                    info.popen.kill()
+                    log.debug('Sent kill to PID %s', info.popen.pid)
                 else:
                     # No popen if running in forkless mode currently 
                     assert self.debugWorker
@@ -268,28 +521,36 @@ def killBatchJobs(self, jobIDs):
 
     def getIssuedBatchJobIDs(self):
         """Just returns all the jobs that have been run, but not yet returned as updated."""
+        
+        self._checkOnDaddy()
+        
         return list(self.jobs.keys())
 
     def getRunningBatchJobIDs(self):
+        
+        self._checkOnDaddy()
+        
         now = time.time()
         return {jobID: now - info.time for jobID, info in list(self.runningJobs.items())}
 
     def shutdown(self):
         """
-        Cleanly terminate worker threads. Add sentinels to inputQueue equal to maxThreads. Join
-        all worker threads.
+        Cleanly terminate and join daddy thread.
         """
-        # Remove reference to inputQueue (raises exception if inputQueue is used after method call)
-        inputQueue = self.inputQueue
-        self.inputQueue = None
-        for i in range(self.numWorkers):
-            inputQueue.put(None)
-        for thread in self.workerThreads:
-            thread.join()
+        
+        if self.daddyThread is not None:
+            # Tell the daddy thread to stop.
+            self.shuttingDown.set()
+            # Wait for it to stop.
+            self.daddyThread.join()
+
         BatchSystemSupport.workerCleanup(self.workerCleanupInfo)
 
     def getUpdatedBatchJob(self, maxWait):
-        """Returns a tuple of a no-longer-running, the return value of its process, and its runtime, or None."""
+        """Returns a tuple of a no-longer-running job, the return value of its process, and its runtime, or None."""
+        
+        self._checkOnDaddy()
+        
         try:
             item = self.outputQueue.get(timeout=maxWait)
         except Empty:
@@ -305,35 +566,74 @@ def setOptions(cls, setOption):
 
 
 class Info(object):
+    """
+    Record for a running job.
+
+    Stores the start time of the job, the Popen object representing its child
+    (or None), the tuple of (coreFractions, memory, disk) it is using (or
+    None), and whether the job is supposed to be being killed.
+    """
     # Can't use namedtuple here since killIntended needs to be mutable
-    def __init__(self, startTime, popen, killIntended):
+    def __init__(self, startTime, popen, resources, killIntended):
         self.time = startTime
         self.popen = popen
+        self.resources = resources
         self.killIntended = killIntended
 
 
 class ResourcePool(object):
-    def __init__(self, initial_value, resourceType, timeout):
+    """
+    Represents an integral amount of a resource (such as memory bytes).
+
+    Amounts can be acquired immediately or with a timeout, and released.
+
+    Provides a context manager to do something with an amount of resource
+    acquired.
+    """
+    def __init__(self, initial_value, resourceType, timeout=5):
         super(ResourcePool, self).__init__()
+        # We use this condition to signal everyone whenever some resource is released.
+        # We use its associated lock to guard value.
         self.condition = Condition()
+        # This records how much resource is available right now.
         self.value = initial_value
         self.resourceType = resourceType
         self.timeout = timeout
 
+    def acquireNow(self, amount):
+        """
+        Reserve the given amount of the given resource.
+
+        Returns True if successful and False if this is not possible immediately.
+        """
+
+        with self.condition:
+            if amount > self.value:
+                return False
+            self.value -= amount
+            self.__validate()
+            return True
+            
+
     def acquire(self, amount):
+        """
+        Reserve the given amount of the given resource.
+
+        Raises AcquisitionTimeoutException if this is not possible in under
+        self.timeout time.
+        """
         with self.condition:
             startTime = time.time()
             while amount > self.value:
                 if time.time() - startTime >= self.timeout:
-                    # This means the thread timed out waiting for the resource.  We exit the nested
-                    # context managers in worker to prevent blocking of a resource due to
-                    # unavailability of a nested resource request.
+                    # This means the thread timed out waiting for the resource.
                     raise self.AcquisitionTimeoutException(resource=self.resourceType,
                                                            requested=amount, available=self.value)
-                # Allow 5 seconds to get the resource, else quit through the above if condition.
-                # This wait + timeout is the last thing in the loop such that a request that takes
-                # longer than 5s due to multiple wakes under the 5 second threshold are still
-                # honored.
+                # Allow self.timeout seconds to get the resource, else quit
+                # through the above if condition. This wait + timeout is the
+                # last thing in the loop such that a request that takes longer
+                # than self.timeout due to multiple wakes under the threshold
+                # are still honored.
                 self.condition.wait(timeout=self.timeout)
             self.value -= amount
             self.__validate()
@@ -342,7 +642,7 @@ def release(self, amount):
         with self.condition:
             self.value += amount
             self.__validate()
-            self.condition.notifyAll()
+            self.condition.notify_all()
 
     def __validate(self):
         assert 0 <= self.value


Test patch:
diff --git a/src/toil/test/__init__.py b/src/toil/test/__init__.py
--- a/src/toil/test/__init__.py
+++ b/src/toil/test/__init__.py
@@ -38,7 +38,7 @@
 from unittest.util import strclass
 
 from toil import subprocess
-from toil import toilPackageDirPath, applianceSelf
+from toil import toilPackageDirPath, applianceSelf, ApplianceImageNotFound
 from toil import which
 from toil.lib.iterables import concat
 from toil.lib.memoize import memoize
@@ -467,6 +467,25 @@ def needs_appliance(test_item):
     return unittest.skip(f"Cannot find appliance {image}. Use 'make test' target to automatically build appliance, or "
                          f"just run 'make push_docker' prior to running this test.")(test_item)
 
+def needs_fetchable_appliance(test_item):
+    """
+    Use as a decorator before test classes or methods to only run them if
+    the Toil appliance Docker image is able to be downloaded from the Internet.
+    """
+
+    test_item = _mark_test('fetchable_appliance', test_item)
+    if os.getenv('TOIL_SKIP_DOCKER', '').lower() == 'true':
+        return unittest.skip('Skipping docker test.')(test_item)
+    try:
+        image = applianceSelf()
+    except ApplianceImageNotFound:
+        # Not downloadable
+        return unittest.skip(f"Cannot see appliance in registry. Use 'make test' target to automatically build appliance, or "
+                             f"just run 'make push_docker' prior to running this test.")(test_item)
+    else:
+        return test_item
+        
+    
 
 def integrative(test_item):
     """
diff --git a/src/toil/test/batchSystems/batchSystemTest.py b/src/toil/test/batchSystems/batchSystemTest.py
--- a/src/toil/test/batchSystems/batchSystemTest.py
+++ b/src/toil/test/batchSystems/batchSystemTest.py
@@ -49,6 +49,7 @@
                        needs_aws_s3,
                        needs_lsf,
                        needs_kubernetes,
+                       needs_fetchable_appliance,
                        needs_mesos,
                        needs_parasol,
                        needs_gridengine,
@@ -345,6 +346,7 @@ def _createConfig(self):
 
 @needs_kubernetes
 @needs_aws_s3
+@needs_fetchable_appliance
 class KubernetesBatchSystemTest(hidden.AbstractBatchSystemTest):
     """
     Tests against the Kubernetes batch system