Repository: DataBiosphere/toil

Issue:
Kubernetes batch system ignores --maxCores, --maxDisk, and --maxMemory and just throws all the jobs at the cluster at once
In theory, `--maxCores`, `--maxDisk`, and `--maxMemory` should limit how much of each resource Toil is willing to request/use for its jobs at a time.

In practice, this is implemented in the single machine batch system, but not in some of the others, including the Kubernetes batch system.

I would like to be able to run my Toil run against our lab cluster without swamping it and making other people's jobs wait for my jobs to finish. So I would like to leave some cluster resources free for others, by restricting the maximum resources Toil will try to use at once. `--maxCores`, `--maxDisk`, and `--maxMemory` seem to be the right way to do this, but they are implemented per batch system, and they are not implemented in the Kubernetes batch system.

Someone should implement these features in the Kubernetes batch system. Perhaps the `ResourcePool` from the single machine batch system could be re-used?

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-457)
┆Issue Number: TOIL-457



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/abstractBatchSystem.py b/src/toil/batchSystems/abstractBatchSystem.py
--- a/src/toil/batchSystems/abstractBatchSystem.py
+++ b/src/toil/batchSystems/abstractBatchSystem.py
@@ -18,15 +18,16 @@
 from abc import ABC, abstractmethod
 from argparse import ArgumentParser, _ArgumentGroup
 from contextlib import contextmanager
+from threading import Condition
+import time
 from typing import (Any,
-                    Callable,
                     ContextManager,
                     Dict,
-                    Iterator,
                     List,
+                    Set,
+                    Iterator,
                     NamedTuple,
                     Optional,
-                    TypeVar,
                     Union,
                     cast)
 
@@ -579,3 +580,171 @@ def __str__(self) -> str:
 
         return ''.join(msg)
 
+
+class AcquisitionTimeoutException(Exception):
+    """To be raised when a resource request times out."""
+    def __init__(self, resource: str, requested: Union[int, float, Set[int]], available: Union[int, float, Set[int]]) -> None:
+        """
+        Creates an instance of this exception that indicates which resource is insufficient for
+        current demands, as well as the resources requested and actually available.
+        :param str resource: string representing the resource type
+        :param requested: the resources requested that resulted in this exception
+        :param available: the resources actually available
+        """
+        self.requested = requested
+        self.available = available
+        self.resource = resource
+
+
+class ResourcePool:
+    """
+    Represents an integral amount of a resource (such as memory bytes).
+    Amounts can be acquired immediately or with a timeout, and released.
+    Provides a context manager to do something with an amount of resource
+    acquired.
+    """
+    def __init__(self, initial_value: int, resource_type: str, timeout: float = 5) -> None:
+        super().__init__()
+        # We use this condition to signal everyone whenever some resource is released.
+        # We use its associated lock to guard value.
+        self.condition = Condition()
+        # This records how much resource is available right now.
+        self.value = initial_value
+        self.resource_type = resource_type
+        self.timeout = timeout
+
+    def acquireNow(self, amount: int) -> bool:
+        """
+        Reserve the given amount of the given resource.
+        Returns True if successful and False if this is not possible immediately.
+        """
+
+        with self.condition:
+            if amount > self.value:
+                return False
+            self.value -= amount
+            self.__validate()
+            return True
+
+    def acquire(self, amount: int) -> None:
+        """
+        Reserve the given amount of the given resource.
+        Raises AcquisitionTimeoutException if this is not possible in under
+        self.timeout time.
+        """
+        with self.condition:
+            startTime = time.time()
+            while amount > self.value:
+                if time.time() - startTime >= self.timeout:
+                    # This means the thread timed out waiting for the resource.
+                    raise AcquisitionTimeoutException(resource=self.resource_type,
+                                                      requested=amount, available=self.value)
+                # Allow self.timeout seconds to get the resource, else quit
+                # through the above if condition. This wait + timeout is the
+                # last thing in the loop such that a request that takes longer
+                # than self.timeout due to multiple wakes under the threshold
+                # are still honored.
+                self.condition.wait(timeout=self.timeout)
+            self.value -= amount
+            self.__validate()
+
+    def release(self, amount: int) -> None:
+        with self.condition:
+            self.value += amount
+            self.__validate()
+            self.condition.notify_all()
+
+    def __validate(self) -> None:
+        assert 0 <= self.value
+
+    def __str__(self) -> str:
+        return str(self.value)
+
+    def __repr__(self) -> str:
+        return "ResourcePool(%i)" % self.value
+
+    @contextmanager
+    def acquisitionOf(self, amount: int) -> Iterator[None]:
+        self.acquire(amount)
+        try:
+            yield
+        finally:
+            self.release(amount)
+
+
+class ResourceSet:
+    """
+    Represents a collection of distinct resources (such as accelerators).
+    Subsets can be acquired immediately or with a timeout, and released.
+    Provides a context manager to do something with a set of of resources
+    acquired.
+    """
+    def __init__(self, initial_value: Set[int], resource_type: str, timeout: float = 5) -> None:
+        super().__init__()
+        # We use this condition to signal everyone whenever some resource is released.
+        # We use its associated lock to guard value.
+        self.condition = Condition()
+        # This records what resources are available right now.
+        self.value = initial_value
+        self.resource_type = resource_type
+        self.timeout = timeout
+
+    def acquireNow(self, subset: Set[int]) -> bool:
+        """
+        Reserve the given amount of the given resource.
+        Returns True if successful and False if this is not possible immediately.
+        """
+
+        with self.condition:
+            if subset > self.value:
+                return False
+            self.value -= subset
+            return True
+
+    def acquire(self, subset: Set[int]) -> None:
+        """
+        Reserve the given amount of the given resource.
+        Raises AcquisitionTimeoutException if this is not possible in under
+        self.timeout time.
+        """
+        with self.condition:
+            startTime = time.time()
+            while subset > self.value:
+                if time.time() - startTime >= self.timeout:
+                    # This means the thread timed out waiting for the resource.
+                    raise AcquisitionTimeoutException(resource=self.resource_type,
+                                                      requested=subset, available=self.value)
+                # Allow self.timeout seconds to get the resource, else quit
+                # through the above if condition. This wait + timeout is the
+                # last thing in the loop such that a request that takes longer
+                # than self.timeout due to multiple wakes under the threshold
+                # are still honored.
+                self.condition.wait(timeout=self.timeout)
+            self.value -= subset
+
+    def release(self, subset: Set[int]) -> None:
+        with self.condition:
+            self.value |= subset
+            self.condition.notify_all()
+
+    def get_free_snapshot(self) -> Set[int]:
+        """
+        Get a snapshot of what items are free right now.
+        May be stale as soon as you get it, but you will need some kind of hint
+        to try and do an acquire.
+        """
+        return set(self.value)
+
+    def __str__(self) -> str:
+        return str(self.value)
+
+    def __repr__(self) -> str:
+        return "ResourceSet(%s)" % self.value
+
+    @contextmanager
+    def acquisitionOf(self, subset: Set[int]) -> Iterator[None]:
+        self.acquire(subset)
+        try:
+            yield
+        finally:
+            self.release(subset)
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -24,9 +24,11 @@
 import logging
 import math
 import os
+from queue import Empty, Queue
 import string
 import sys
 import tempfile
+from threading import Event, Thread, Condition, RLock
 import time
 import uuid
 from argparse import ArgumentParser, _ArgumentGroup
@@ -35,6 +37,7 @@
                     Dict,
                     Iterator,
                     List,
+                    Set,
                     Literal,
                     Optional,
                     Tuple,
@@ -96,11 +99,12 @@
 from toil.batchSystems.abstractBatchSystem import (EXIT_STATUS_UNAVAILABLE_VALUE,
                                                    BatchJobExitReason,
                                                    InsufficientSystemResources,
+                                                   ResourcePool,
                                                    UpdatedBatchJobInfo)
 from toil.batchSystems.cleanup_support import BatchSystemCleanupSupport
 from toil.batchSystems.contained_executor import pack_job
 from toil.batchSystems.options import OptionSetter
-from toil.common import Config, Toil
+from toil.common import Config, Toil, SYS_MAX_SIZE
 from toil.job import JobDescription, Requirer
 from toil.lib.conversions import human2bytes
 from toil.lib.misc import get_user_name, slow_down, utc_now
@@ -230,6 +234,43 @@ def __init__(self, config: Config, maxCores: int, maxMemory: int, maxDisk: int)
         # This will be a label to select all our jobs.
         self.run_id = f'toil-{self.unique_id}'
 
+        # Keep track of available resources.
+        maxMillicores = int(SYS_MAX_SIZE if self.maxCores == SYS_MAX_SIZE else self.maxCores * 1000)
+        self.resource_sources = [
+            # A pool representing available CPU in units of millicores (1 CPU
+            # unit = 1000 millicores)
+            ResourcePool(maxMillicores, 'cores'),
+            # A pool representing available memory in bytes
+            ResourcePool(self.maxMemory, 'memory'),
+            # A pool representing the available space in bytes
+            ResourcePool(self.maxDisk, 'disk'),
+        ]
+
+        # A set of job IDs that are queued (useful for getIssuedBatchJobIDs())
+        self._queued_job_ids: Set[int] = set()
+
+        # Keep track of the acquired resources for each job
+        self._acquired_resources: Dict[str, List[int]] = {}
+
+        # Queue for jobs to be submitted to the Kubernetes cluster
+        self._jobs_queue: Queue[Tuple[int, JobDescription, V1PodSpec]] = Queue()
+
+        # A set of job IDs that should be killed
+        self._killed_queue_jobs: Set[int] = set()
+
+        # We use this event to signal shutdown
+        self._shutting_down = Event()
+
+        # A lock to protect critical regions when working with queued jobs.
+        self._mutex = RLock()
+
+        # A condition set to true when there is more work to do. e.g.: new job
+        # in the queue or any resource becomes available.
+        self._work_available = Condition(lock=self._mutex)
+
+        self.schedulingThread = Thread(target=self._scheduler, daemon=True)
+        self.schedulingThread.start()
+
     def _pretty_print(self, kubernetes_object: Any) -> str:
         """
         Pretty-print a Kubernetes API object to a YAML string. Recursively
@@ -505,6 +546,58 @@ def _stream_until_error(self, method: Callable[P, _ItemsHaver[R]], *args: P.args
                     # Something actually weird is happening.
                     raise
 
+    def _scheduler(self) -> None:
+        """
+        The scheduler thread looks at jobs from the input queue, and submits
+        them to the Kubernetes cluster when there are resources available.
+        """
+        while True:
+            with self._work_available:
+                # Wait until we get notified to do work.
+                self._work_available.wait()
+
+                if self._shutting_down.is_set():
+                    # We're shutting down.
+                    logger.info("Shutting down scheduler thread.")
+                    break
+
+                # Either we have new jobs inside the queue or more resources
+                # have become available.
+
+                # Loop through all jobs inside the queue and see if any of them
+                # could be launched.
+                jobs: Queue[Tuple[int, JobDescription, V1PodSpec]] = Queue()
+                while True:
+                    try:
+                        job = self._jobs_queue.get_nowait()
+                        job_id, job_desc, spec = job
+
+                        # Check if this job has been previously killed.
+                        if job_id in self._killed_queue_jobs:
+                            self._killed_queue_jobs.remove(job_id)
+                            logger.debug(f"Skipping killed job {job_id}")
+                            continue
+
+                        job_name = f'{self.job_prefix}{job_id}'
+                        result = self._launch_job(job_name, job_desc, spec)
+                        if result is False:
+                            # Not enough resources to launch this job.
+                            jobs.put(job)
+                        else:
+                            self._queued_job_ids.remove(job_id)
+
+                    except Empty:
+                        break
+
+                # We've gone over the entire queue and these are the jobs that
+                # have not been scheduled, so put them back to the queue.
+                while True:
+                    try:
+                        self._jobs_queue.put(jobs.get_nowait())
+                    except Empty:
+                        break
+                logger.debug(f"Roughly {self._jobs_queue.qsize} jobs in the queue")
+
     def setUserScript(self, userScript: Resource) -> None:
         logger.info(f'Setting user script for deployment: {userScript}')
         self.user_script = userScript
@@ -797,54 +890,148 @@ def mount_host_path(volume_name: str, host_path: str, mount_path: str, create: b
 
         return pod_spec
 
+    def _release_acquired_resources(self, resources: List[int], notify: bool = False) -> None:
+        """
+        Release all resources acquired for a job.
+
+        :param resources: The resources in the order: core, memory, disk.
+        :param notify: If True, notify the threads that are waiting on the
+                       self._work_available condition.
+        """
+        # TODO: accelerators?
+
+        for resource, request in zip(self.resource_sources, resources):
+            assert isinstance(resource, ResourcePool) and isinstance(request, int)
+            resource.release(request)
+
+        if notify:
+            with self._work_available:
+                self._work_available.notify_all()
+
+    def _launch_job(
+        self,
+        job_name: str,
+        job_desc: JobDescription,
+        pod_spec: V1PodSpec
+    ) -> bool:
+        """
+        Try to launch the given job to the Kubernetes cluster. Return False if
+        we don't have enough resources to submit the job.
+        """
+
+        # Limit the amount of resources requested at a time.
+        resource_requests: List[int] = [int(job_desc.cores * 1000), job_desc.memory, job_desc.disk]
+
+        acquired = []
+        for source, request in zip(self.resource_sources, resource_requests):
+            # For each kind of resource we want, go get it
+            assert ((isinstance(source, ResourcePool) and isinstance(request, int)))
+            if source.acquireNow(request):
+                acquired.append(request)
+            else:
+                # We can't get everything
+                self._release_acquired_resources(acquired,
+                    # Put it back quietly.
+                    notify=False)
+                return False
+
+        self._acquired_resources[job_name] = acquired
+
+        # We have all the resources we need; submit it to the cluster.
+
+        # Make metadata to label the job/pod with info.
+        # Don't let the cluster autoscaler evict any Toil jobs.
+        metadata = V1ObjectMeta(name=job_name,
+                                labels={"toil_run": self.run_id},
+                                annotations={"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"})
+
+        # Wrap the spec in a template
+        template = V1PodTemplateSpec(spec=pod_spec, metadata=metadata)
+
+        # Make another spec for the job, asking to run the template with no
+        # backoff/retry. Specify our own TTL to avoid catching the notice
+        # of over-zealous abandoned job cleanup scripts.
+        job_spec = V1JobSpec(template=template,
+                             backoff_limit=0,
+                             ttl_seconds_after_finished=self.finished_job_ttl)
+
+        # And make the actual job
+        job = V1Job(spec=job_spec,
+                    metadata=metadata,
+                    api_version="batch/v1",
+                    kind="Job")
+
+        # Launch the job
+        launched = self._api('batch', errors=[]).create_namespaced_job(self.namespace, job)
+
+        logger.debug(f"Launched job: {job_name}")
+
+        return True
+
+    def _delete_job(
+        self,
+        job_name: str, *,
+        propagation_policy: Literal["Foreground", "Background"] = "Foreground",
+        gone_ok: bool = False,
+        resource_notify: bool = True
+    ) -> None:
+        """
+        Given the name of a kubernetes job, delete the job and release all
+        acquired resources for the job.
+
+        :param job_name: The Kubernetes job name.
+        :param propagation_policy: The propagation policy for the delete.
+        :param gone_ok: If True, don't raise an error if the job is already gone.
+        :param resource_notify: If True, notify the threads that are waiting on
+            the self._work_available condition.
+        """
+        try:
+            logger.debug(f'Deleting Kubernetes job {job_name}')
+            self._api('batch', errors=[404] if gone_ok else []).delete_namespaced_job(
+                job_name,
+                self.namespace,
+                propagation_policy=propagation_policy
+            )
+        finally:
+            # We should always release the acquired resources.
+            resources = self._acquired_resources.get(job_name)
+            if not resources:
+                logger.warning(f"Cannot get the acquired resources from {job_name}")
+                return
+            self._release_acquired_resources(resources, notify=resource_notify)
+            del self._acquired_resources[job_name]
+
     def issueBatchJob(self, job_desc: JobDescription, job_environment: Optional[Dict[str, str]] = None) -> int:
         # Try the job as local
         localID = self.handleLocalJob(job_desc)
         if localID is not None:
             # It is a local job
             return localID
-        else:
-            # We actually want to send to the cluster
 
-            # Check resource requirements
-            self.check_resource_request(job_desc)
+        # We actually want to send to the cluster
 
-            # Make a pod that describes running the job
-            pod_spec = self._create_pod_spec(job_desc, job_environment=job_environment)
+        # Check resource requirements
+        self.check_resource_request(job_desc)
 
-            # Make a batch system scope job ID
-            jobID = self.getNextJobID()
-            # Make a unique name
-            jobName = self.job_prefix + str(jobID)
+        # Make a pod that describes running the job
+        pod_spec = self._create_pod_spec(job_desc, job_environment=job_environment)
 
-            # Make metadata to label the job/pod with info.
-            # Don't let the cluster autoscaler evict any Toil jobs.
-            metadata = V1ObjectMeta(name=jobName,
-                                                      labels={"toil_run": self.run_id},
-                                                      annotations={"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"})
+        # Make a batch system scope job ID
+        job_id = self.getNextJobID()
 
-            # Wrap the spec in a template
-            template = V1PodTemplateSpec(spec=pod_spec, metadata=metadata)
+        # Put job inside queue to be launched to the cluster
+        self._queued_job_ids.add(job_id)
 
-            # Make another spec for the job, asking to run the template with no
-            # backoff/retry. Specify our own TTL to avoid catching the notice
-            # of over-zealous abandoned job cleanup scripts.
-            job_spec = V1JobSpec(template=template,
-                                                   backoff_limit=0,
-                                                   ttl_seconds_after_finished=self.finished_job_ttl)
+        # Also keep track of the ID for getIssuedBatchJobIDs()
+        self._jobs_queue.put((job_id, job_desc, pod_spec))
 
-            # And make the actual job
-            job = V1Job(spec=job_spec,
-                                          metadata=metadata,
-                                          api_version="batch/v1",
-                                          kind="Job")
+        # Let scheduler know that there is work to do.
+        with self._work_available:
+            self._work_available.notify_all()
 
-            # Make the job
-            launched = self._api('batch', errors=[]).create_namespaced_job(self.namespace, job)
+        logger.debug(f"Enqueued job {job_id}")
 
-            logger.debug('Launched job: %s', jobName)
-
-            return jobID
+        return job_id
 
     class _ArgsDict(TypedDict):
         """
@@ -1231,10 +1418,9 @@ def getUpdatedBatchJob(self, maxWait: float) -> Optional[UpdatedBatchJobInfo]:
 
                     if (exitReason == BatchJobExitReason.FAILED) or (succeeded_pods + failed_pods == totalPods):
                         # Cleanup if job is all finished or there was a pod that failed
-                        logger.debug('Deleting Kubernetes job %s', jobObject.metadata.name)
-                        self._api('batch', errors=[]).delete_namespaced_job(
+                        # TODO: use delete_job() to release acquired resources
+                        self._delete_job(
                             jobObject.metadata.name,
-                            self.namespace,
                             propagation_policy='Foreground'
                         )
                         # Make sure the job is deleted so we won't see it again.
@@ -1431,12 +1617,7 @@ def _getUpdatedBatchJobImmediately(self) -> Optional[UpdatedBatchJobInfo]:
 
         try:
             # Delete the job and all dependents (pods), hoping to get a 404 if it's magically gone
-            logger.debug('Deleting Kubernetes job %s', jobObject.metadata.name)
-            self._api('batch', errors=[404]).delete_namespaced_job(
-                jobObject.metadata.name,
-                self.namespace,
-                propagation_policy='Foreground'
-            )
+            self._delete_job(jobObject.metadata.name, propagation_policy='Foreground', gone_ok=True)
 
             # That just kicks off the deletion process. Foreground doesn't
             # actually block. See
@@ -1488,6 +1669,12 @@ def shutdown(self) -> None:
         # Shutdown local processes first
         self.shutdownLocal()
 
+        # Shutdown scheduling thread
+        self._shutting_down.set()
+        with self._work_available:
+            self._work_available.notify_all()   # Wake it up.
+
+        self.schedulingThread.join()
 
         # Kill all of our jobs and clean up pods that are associated with those jobs
         try:
@@ -1498,6 +1685,7 @@ def shutdown(self) -> None:
                 propagation_policy='Background'
             )
             logger.debug('Killed jobs with delete_collection_namespaced_job; cleaned up')
+            # TODO: should we release all resources? We're shutting down so would it matter?
         except ApiException as e:
             if e.status != 404:
                 # Anything other than a 404 is weird here.
@@ -1505,14 +1693,9 @@ def shutdown(self) -> None:
 
             # If batch delete fails, try to delete all remaining jobs individually.
             logger.debug('Deleting Kubernetes jobs individually for toil_run=%s', self.run_id)
-            for jobID in self._getIssuedNonLocalBatchJobIDs():
-                jobName = f'{self.job_prefix}{jobID}'
-                logger.debug(f'Deleting Kubernetes job {jobName}')
-                self._api('batch', errors=[]).delete_namespaced_job(
-                    jobName,
-                    self.namespace,
-                    propagation_policy='Background'
-                )
+            for job_id in self._getIssuedNonLocalBatchJobIDs():
+                job_name = f'{self.job_prefix}{job_id}'
+                self._delete_job(job_name, propagation_policy='Background', resource_notify=False)
 
             # Aggregate all pods and check if any pod has failed to cleanup or is orphaned.
             ourPods = self._ourPodObject()
@@ -1546,15 +1729,18 @@ def _getIssuedNonLocalBatchJobIDs(self) -> List[int]:
         Get the issued batch job IDs that are not for local jobs.
         """
         jobIDs = []
-        got_list = self._ourJobObject()
-        for job in got_list:
-            # Get the ID for each job
-            jobIDs.append(self._getIDForOurJob(job))
+        with self._mutex:
+            got_list = self._ourJobObject()
+            for job in got_list:
+                # Get the ID for each job
+                jobIDs.append(self._getIDForOurJob(job))
         return jobIDs
 
     def getIssuedBatchJobIDs(self) -> List[int]:
-        # Make sure to send the local jobs also
-        return self._getIssuedNonLocalBatchJobIDs() + list(self.getIssuedLocalJobIDs())
+        # Make sure to send the local jobs and queued jobs also
+        with self._mutex:
+            queued_jobs = list(self._queued_job_ids)
+        return self._getIssuedNonLocalBatchJobIDs() + list(self.getIssuedLocalJobIDs()) + queued_jobs
 
     def _get_start_time(self, pod: Optional[V1Pod] = None, job: Optional[V1Job] = None) -> datetime.datetime:
         """
@@ -1603,32 +1789,40 @@ def killBatchJobs(self, jobIDs: List[int]) -> None:
         # Clears workflow's jobs listed in jobIDs.
 
         # First get the jobs we even issued non-locally
-        issuedOnKubernetes = set(self._getIssuedNonLocalBatchJobIDs())
+        issued_on_kubernetes = set(self._getIssuedNonLocalBatchJobIDs())
+        deleted_jobs: List[str] = []
 
-        for jobID in jobIDs:
+        for job_id in jobIDs:
             # For each job we are supposed to kill
-            if jobID not in issuedOnKubernetes:
+            if job_id not in issued_on_kubernetes:
                 # It never went to Kubernetes (or wasn't there when we just
                 # looked), so we can't kill it on Kubernetes.
+
+                # It is likely that the job is still waiting inside the jobs
+                # queue and hasn't been submitted to Kubernetes. In this case,
+                # it is not reliable to loop through the queue and try to
+                # delete it. Instead, let's keep track of it and don't submit
+                # when we encounter it.
+                with self._mutex:
+                    self._killed_queue_jobs.add(job_id)
+
+                    # Make sure we don't keep track of it for getIssuedBatchJobIDs().
+                    if job_id in self._queued_job_ids:
+                        self._queued_job_ids.remove(job_id)
+
                 continue
             # Work out what the job would be named
-            jobName = self.job_prefix + str(jobID)
+            job_name = self.job_prefix + str(job_id)
 
             # Delete the requested job in the foreground.
             # This doesn't block, but it does delete expeditiously.
-            logger.debug('Deleting Kubernetes job %s', jobName)
-            response = self._api('batch', errors=[]).delete_namespaced_job(
-                jobName,
-                self.namespace,
-                propagation_policy='Foreground'
-            )
-            logger.debug('Killed job by request: %s', jobName)
+            self._delete_job(job_name, propagation_policy='Foreground')
 
-        for jobID in jobIDs:
-            # Now we need to wait for all the jobs we killed to be gone.
+            deleted_jobs.append(job_name)
+            logger.debug('Killed job by request: %s', job_name)
 
-            # Work out what the job would be named
-            jobName = self.job_prefix + str(jobID)
+        for job_name in deleted_jobs:
+            # Now we need to wait for all the jobs we killed to be gone.
 
             # Block until the delete takes.
             # The user code technically might stay running forever, but we push
@@ -1636,7 +1830,7 @@ def killBatchJobs(self, jobIDs: List[int]) -> None:
             # a resource) onto the user code, instead of always hanging
             # whenever we can't certify that a faulty node is no longer running
             # the user code. 
-            self._waitForJobDeath(jobName)
+            self._waitForJobDeath(job_name)
 
     @classmethod
     def get_default_kubernetes_owner(cls) -> str:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -20,14 +20,18 @@
 import time
 import traceback
 from argparse import ArgumentParser, _ArgumentGroup
-from contextlib import contextmanager
 from queue import Empty, Queue
-from threading import Condition, Event, Lock, Thread
+from threading import Event, Lock, Thread
 from typing import Dict, List, Optional, Set, Sequence, Tuple, Union
 
 import toil
 from toil import worker as toil_worker
-from toil.batchSystems.abstractBatchSystem import EXIT_STATUS_UNAVAILABLE_VALUE, BatchSystemSupport, UpdatedBatchJobInfo, InsufficientSystemResources
+from toil.batchSystems.abstractBatchSystem import (EXIT_STATUS_UNAVAILABLE_VALUE,
+                                                   BatchSystemSupport,
+                                                   ResourcePool,
+                                                   ResourceSet,
+                                                   UpdatedBatchJobInfo,
+                                                   InsufficientSystemResources)
 
 from toil.bus import ExternalBatchIdMessage
 from toil.batchSystems.options import OptionSetter
@@ -844,184 +848,3 @@ def __init__(self, startTime, popen, resources, killIntended):
         self.popen = popen
         self.resources = resources
         self.killIntended = killIntended
-
-class AcquisitionTimeoutException(Exception):
-    """To be raised when a resource request times out."""
-    def __init__(self, resource: str, requested: Union[int, float, Set[int]], available: Union[int, float, Set[int]]):
-        """
-        Creates an instance of this exception that indicates which resource is insufficient for
-        current demands, as well as the resources requested and actually available.
-
-        :param str resource: string representing the resource type
-
-        :param requested: the resources requested that resulted in this exception
-
-        :param available: the resources actually available
-        """
-        self.requested = requested
-        self.available = available
-        self.resource = resource
-
-class ResourcePool:
-    """
-    Represents an integral amount of a resource (such as memory bytes).
-
-    Amounts can be acquired immediately or with a timeout, and released.
-
-    Provides a context manager to do something with an amount of resource
-    acquired.
-    """
-    def __init__(self, initial_value: int, resource_type: str, timeout=5):
-        super().__init__()
-        # We use this condition to signal everyone whenever some resource is released.
-        # We use its associated lock to guard value.
-        self.condition = Condition()
-        # This records how much resource is available right now.
-        self.value = initial_value
-        self.resource_type = resource_type
-        self.timeout = timeout
-
-    def acquireNow(self, amount):
-        """
-        Reserve the given amount of the given resource.
-
-        Returns True if successful and False if this is not possible immediately.
-        """
-
-        with self.condition:
-            if amount > self.value:
-                return False
-            self.value -= amount
-            self.__validate()
-            return True
-
-
-    def acquire(self, amount):
-        """
-        Reserve the given amount of the given resource.
-
-        Raises AcquisitionTimeoutException if this is not possible in under
-        self.timeout time.
-        """
-        with self.condition:
-            startTime = time.time()
-            while amount > self.value:
-                if time.time() - startTime >= self.timeout:
-                    # This means the thread timed out waiting for the resource.
-                    raise AcquisitionTimeoutException(resource=self.resource_type,
-                                                      requested=amount, available=self.value)
-                # Allow self.timeout seconds to get the resource, else quit
-                # through the above if condition. This wait + timeout is the
-                # last thing in the loop such that a request that takes longer
-                # than self.timeout due to multiple wakes under the threshold
-                # are still honored.
-                self.condition.wait(timeout=self.timeout)
-            self.value -= amount
-            self.__validate()
-
-    def release(self, amount):
-        with self.condition:
-            self.value += amount
-            self.__validate()
-            self.condition.notify_all()
-
-    def __validate(self):
-        assert 0 <= self.value
-
-    def __str__(self):
-        return str(self.value)
-
-    def __repr__(self):
-        return "ResourcePool(%i)" % self.value
-
-    @contextmanager
-    def acquisitionOf(self, amount):
-        self.acquire(amount)
-        try:
-            yield
-        finally:
-            self.release(amount)
-
-class ResourceSet:
-    """
-    Represents a collection of distinct resources (such as accelerators).
-
-    Subsets can be acquired immediately or with a timeout, and released.
-
-    Provides a context manager to do something with a set of of resources
-    acquired.
-    """
-    def __init__(self, initial_value: Set[int], resource_type: str, timeout: float = 5):
-        super().__init__()
-        # We use this condition to signal everyone whenever some resource is released.
-        # We use its associated lock to guard value.
-        self.condition = Condition()
-        # This records what resources are available right now.
-        self.value = initial_value
-        self.resource_type = resource_type
-        self.timeout = timeout
-
-    def acquireNow(self, subset: Set[int]):
-        """
-        Reserve the given amount of the given resource.
-
-        Returns True if successful and False if this is not possible immediately.
-        """
-
-        with self.condition:
-            if subset > self.value:
-                return False
-            self.value -= subset
-            return True
-
-    def acquire(self, subset: Set[int]):
-        """
-        Reserve the given amount of the given resource.
-
-        Raises AcquisitionTimeoutException if this is not possible in under
-        self.timeout time.
-        """
-        with self.condition:
-            startTime = time.time()
-            while subset > self.value:
-                if time.time() - startTime >= self.timeout:
-                    # This means the thread timed out waiting for the resource.
-                    raise AcquisitionTimeoutException(resource=self.resource_type,
-                                                      requested=subset, available=self.value)
-                # Allow self.timeout seconds to get the resource, else quit
-                # through the above if condition. This wait + timeout is the
-                # last thing in the loop such that a request that takes longer
-                # than self.timeout due to multiple wakes under the threshold
-                # are still honored.
-                self.condition.wait(timeout=self.timeout)
-            self.value -= subset
-
-    def release(self, subset: Set[int]):
-        with self.condition:
-            self.value |= subset
-            self.condition.notify_all()
-
-    def get_free_snapshot(self) -> Set[int]:
-        """
-        Get a snapshot of what items are free right now.
-        May be stale as soon as you get it, but you will need some kind of hint
-        to try and do an acquire.
-        """
-        return set(self.value)
-
-    def __str__(self):
-        return str(self.value)
-
-    def __repr__(self):
-        return "ResourceSet(%s)" % self.value
-
-    @contextmanager
-    def acquisitionOf(self, subset: Set[int]):
-        self.acquire(subset)
-        try:
-            yield
-        finally:
-            self.release(subset)
-
-
-


Test patch: