Repository: DataBiosphere/toil

Issue:
Parasol test hangs with new, more aggressive failure simulation
PR #431 (to be merged soon) makes the failure simulation in sortTest (look for badWorker in worker.py) much more aggressive. It now randomly kills the worker process instead of raising an exception in a fixed place in the code. This **revealed** a new issue which causes the Parasol sort test to hang. It is very likely that this is **caused** not by the changes in PR #431 but by a bug in Parasol or the corresponding batch system.

Parasol test hangs with new, more aggressive failure simulation
PR #431 (to be merged soon) makes the failure simulation in sortTest (look for badWorker in worker.py) much more aggressive. It now randomly kills the worker process instead of raising an exception in a fixed place in the code. This **revealed** a new issue which causes the Parasol sort test to hang. It is very likely that this is **caused** not by the changes in PR #431 but by a bug in Parasol or the corresponding batch system.



Discussion:
I'm having trouble replicating this. I see some warnings that I should fix, but the test always passes. Is it possible Jenkins is terminating the test before it can finish? It does take a long time.

```
(env)alden@thinkpad:toil$ py.test -s -v src/toil/test/sort/sortTest.py::SortTest::testFileParasol
=================================== test session starts ====================================
platform linux2 -- Python 2.7.9, pytest-2.8.1, py-1.4.30, pluggy-0.3.1 -- /home/alden/toil/env/bin/python2
cachedir: .cache
rootdir: /home/alden/toil, inifile: setup.cfg
collected 14 items 

src/toil/test/sort/sortTest.py::SortTest::testFileParasol No handlers could be found for logger "toil.resource"
WARNING:toil.batchSystems.parasol:Could not clear sick status of the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.batchSystems.parasol:Could not flush the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/alden/toil/src', name='toil.job', extension='.py').
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:22 : Splitting range (0..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:38 : Splitting range (0..60) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:40 : Splitting range (60..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 5 and jobWrapper store ID: R/f/job3divGJ failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: R/f/job3divGJ
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job R/f/job3divGJ to 2
WARNING:toil.jobWrapper:We have increased the default memory of the failed job to 2147483648 bytes
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:52 : Splitting range (0..40) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:52 : Sorting range (40..60) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:53 : Splitting range (60..90) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:57 : Sorting range (90..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:57 : Sorting range (60..80) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:58 : Sorting range (80..90) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:11 : Splitting range (0..30) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:13 : Sorting range (30..40) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:15 : Merging a/C/tmp1CXI92.tmp and S/4/tmpUVafQD.tmp to V/f/tmpMdUBiu.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:19 : Sorting range (0..20) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:22 : Sorting range (20..30) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:39 : Merging B/6/tmpzDmLSt.tmp and d/w/tmpoX902l.tmp to O/K/tmpNRwoyK.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:41 : Merging V/f/tmpMdUBiu.tmp and 5/0/tmpXhC0nv.tmp to Q/B/tmp_oph5N.tmp
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 21 and jobWrapper store ID: T/k/jobbaxA4w failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: T/k/jobbaxA4w
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job T/k/jobbaxA4w to 2
WARNING:toil.jobWrapper:We have increased the default memory of the failed job to 2147483648 bytes
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 22 and jobWrapper store ID: w/T/job3WCb1Q failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: w/T/job3WCb1Q
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job w/T/job3WCb1Q to 2
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:26:23 : Merging O/K/tmpNRwoyK.tmp and P/e/tmp4gALqt.tmp to W/C/tmpTLMnd3.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:26:54 : Merging W/C/tmpTLMnd3.tmp and C/K/tmpZ4Ymce.tmp to R/T/tmp8pjYb4.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:27:22 : Merging R/T/tmp8pjYb4.tmp and Q/B/tmp_oph5N.tmp to F/j/tmpmXeubP.tmp
WARNING:toil.batchSystems.parasol:Could not clear sick status of the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.batchSystems.parasol:Could not flush the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.utils.toilClean:Parsed arguments
INFO:toil.utils.toilClean:Checking if we have files for toil
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX
INFO:toil.utils.toilClean:Deleting the JobStore
PASSED
(env)alden@thinkpad:toil$ git branch
* issues/376-caching
  master

```

I'm having trouble replicating this. I see some warnings that I should fix, but the test always passes. Is it possible Jenkins is terminating the test before it can finish? It does take a long time.

```
(env)alden@thinkpad:toil$ py.test -s -v src/toil/test/sort/sortTest.py::SortTest::testFileParasol
=================================== test session starts ====================================
platform linux2 -- Python 2.7.9, pytest-2.8.1, py-1.4.30, pluggy-0.3.1 -- /home/alden/toil/env/bin/python2
cachedir: .cache
rootdir: /home/alden/toil, inifile: setup.cfg
collected 14 items 

src/toil/test/sort/sortTest.py::SortTest::testFileParasol No handlers could be found for logger "toil.resource"
WARNING:toil.batchSystems.parasol:Could not clear sick status of the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.batchSystems.parasol:Could not flush the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/alden/toil/src', name='toil.job', extension='.py').
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:22 : Splitting range (0..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:38 : Splitting range (0..60) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:40 : Splitting range (60..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 5 and jobWrapper store ID: R/f/job3divGJ failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: R/f/job3divGJ
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job R/f/job3divGJ to 2
WARNING:toil.jobWrapper:We have increased the default memory of the failed job to 2147483648 bytes
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:52 : Splitting range (0..40) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:52 : Sorting range (40..60) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:53 : Splitting range (60..90) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:57 : Sorting range (90..100) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:57 : Sorting range (60..80) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:24:58 : Sorting range (80..90) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:11 : Splitting range (0..30) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:13 : Sorting range (30..40) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:15 : Merging a/C/tmp1CXI92.tmp and S/4/tmpUVafQD.tmp to V/f/tmpMdUBiu.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:19 : Sorting range (0..20) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:22 : Sorting range (20..30) of file: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-tempDir-YHV5xb/fileToSort.txt
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:39 : Merging B/6/tmpzDmLSt.tmp and d/w/tmpoX902l.tmp to O/K/tmpNRwoyK.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:25:41 : Merging V/f/tmpMdUBiu.tmp and 5/0/tmpXhC0nv.tmp to Q/B/tmp_oph5N.tmp
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 21 and jobWrapper store ID: T/k/jobbaxA4w failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: T/k/jobbaxA4w
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job T/k/jobbaxA4w to 2
WARNING:toil.jobWrapper:We have increased the default memory of the failed job to 2147483648 bytes
WARNING:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 22 and jobWrapper store ID: w/T/job3WCb1Q failed with exit value 9
WARNING:toil.leader:No log file is present, despite jobWrapper failing: w/T/job3WCb1Q
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job w/T/job3WCb1Q to 2
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:26:23 : Merging O/K/tmpNRwoyK.tmp and P/e/tmp4gALqt.tmp to W/C/tmpTLMnd3.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:26:54 : Merging W/C/tmpTLMnd3.tmp and C/K/tmpZ4Ymce.tmp to R/T/tmp8pjYb4.tmp
WARNING:toil.leader:Got message from job at time: 10-01-2015 21:27:22 : Merging R/T/tmp8pjYb4.tmp and Q/B/tmp_oph5N.tmp to F/j/tmpmXeubP.tmp
WARNING:toil.batchSystems.parasol:Could not clear sick status of the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
WARNING:toil.batchSystems.parasol:Could not flush the parasol batch /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX/results.txt
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.utils.toilClean:Parsed arguments
INFO:toil.utils.toilClean:Checking if we have files for toil
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /tmp/toil-test-toil.test.sort.sortTest.SortTest.testFileParasol-jobstore-FP0sAX
INFO:toil.utils.toilClean:Deleting the JobStore
PASSED
(env)alden@thinkpad:toil$ git branch
* issues/376-caching
  master

```


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,7 @@
     author_email='benedict@soe.usc.edu',
     url="https://github.com/BD2KGenomics/toil",
     install_requires=[
-        'bd2k-python-lib==1.10.dev5'],
+        'bd2k-python-lib==1.10.dev6'],
     tests_require=[
         'mock==1.0.1',
         'pytest==2.8.3'],
diff --git a/src/toil/batchSystems/parasol.py b/src/toil/batchSystems/parasol.py
--- a/src/toil/batchSystems/parasol.py
+++ b/src/toil/batchSystems/parasol.py
@@ -31,19 +31,19 @@
 from toil.batchSystems.abstractBatchSystem import AbstractBatchSystem
 from toil.lib.bioio import getTempFile
 
-logger = logging.getLogger( __name__ )
+logger = logging.getLogger(__name__)
 
 
 class ParasolBatchSystem(AbstractBatchSystem):
     """
     The interface for Parasol.
     """
+
     def __init__(self, config, maxCores, maxMemory, maxDisk):
-        AbstractBatchSystem.__init__(self, config, maxCores, maxMemory, maxDisk) #Call the parent constructor
+        AbstractBatchSystem.__init__(self, config, maxCores, maxMemory, maxDisk)
         if maxMemory != sys.maxint:
-            logger.warn("A max memory has been specified for the parasol batch system class of %i, but currently "
-                        "this batchsystem interface does not support such limiting" % maxMemory)
-        #Keep the name of the results file for the pstat2 command..
+            logger.warn('The Parasol batch system does not support maxMemory.')
+        # Keep the name of the results file for the pstat2 command..
         command = config.parasolCommand
         if os.path.sep not in command:
             try:
@@ -54,31 +54,21 @@ def __init__(self, config, maxCores, maxMemory, maxDisk):
         self.parasolCommand = command
         self.parasolResultsDir = tempfile.mkdtemp(dir=config.jobStore)
 
-        self.queuePattern = re.compile("q\s+([0-9]+)")
-        self.runningPattern = re.compile("r\s+([0-9]+)\s+[\S]+\s+[\S]+\s+([0-9]+)\s+[\S]+")
-
-        logger.info("Going to sleep for a few seconds to kill any existing jobs")
-        time.sleep(5) #Give batch system a second to sort itself out.
-        logger.info("Removed any old jobs from the queue")
-
-        #In Parasol, each results file corresponds to a separate batch, and all
-        #jobs in a batch have the same cpu and memory requirements. The keys to this
-        #dictionary are the (cpu, memory) tuples for each batch. A new batch
-        #is created whenever a job has a new unique combination of cpu and memory
-        #requirements.
+        # In Parasol, each results file corresponds to a separate batch, and all jobs in a batch
+        # have the same cpu and memory requirements. The keys to this dictionary are the (cpu,
+        # memory) tuples for each batch. A new batch is created whenever a job has a new unique
+        # combination of cpu and memory requirements.
         self.resultsFiles = dict()
         self.maxBatches = config.parasolMaxBatches
 
-        #Allows the worker process to send back the IDs of
-        #jobs that have finished, so the batch system can
-        #decrease its used cpus counter
+        # Allows the worker process to send back the IDs of jobs that have finished, so the batch
+        #  system can decrease its used cpus counter
         self.cpuUsageQueue = Queue()
 
-        #Also stores finished job IDs, but is read
-        #by getUpdatedJobIDs().
+        # Also stores finished job IDs, but is read by getUpdatedJobIDs().
         self.updatedJobsQueue = Queue()
 
-        #Use this to stop the worker when shutting down
+        # Use this to stop the worker when shutting down
         self.running = True
 
         self.worker = Thread(target=self.updatedJobWorker, args=())
@@ -86,22 +76,21 @@ def __init__(self, config, maxCores, maxMemory, maxDisk):
         self.usedCpus = 0
         self.jobIDsToCpu = {}
 
-        #Set of jobs that have been issued but aren't known to
-        #have finished or been killed yet. Jobs that end
-        #by themselves are removed in getUpdatedJob, and jobs
-        #that are killed are removed in killBatchJobs.
+        # Set of jobs that have been issued but aren't known to have finished or been killed yet.
+        #  Jobs that end by themselves are removed in getUpdatedJob, and jobs that are killed are
+        #  removed in killBatchJobs.
         self.runningJobs = set()
 
-    def runParasol(self,command,runUntilSuccessful=True):
+    def runParasol(self, command, autoRetry=True):
         """
         Issues a parasol command using popen to capture the output. If the command fails then it
         will try pinging parasol until it gets a response. When it gets a response it will
         recursively call the issue parasol command, repeating this pattern for a maximum of N
         times. The final exit value will reflect this.
         """
+        command = list(concat(self.parasolCommand, command))
         while True:
-            command = list( concat(self.parasolCommand, command) )
-            logger.debug('Running %r',command)
+            logger.debug('Running %r', command)
             process = subprocess.Popen(command,
                                        stdout=subprocess.PIPE,
                                        stderr=subprocess.PIPE,
@@ -112,74 +101,75 @@ def runParasol(self,command,runUntilSuccessful=True):
                 if line: logger.warn(line)
             if status == 0:
                 return 0, stdout.split('\n')
-            message = 'Command %r failed with exit status %i' % (command,status)
-            if not runUntilSuccessful:
+            message = 'Command %r failed with exit status %i' % (command, status)
+            if autoRetry:
+                logger.warn(message)
+            else:
                 logger.error(message)
                 return status, None
-            else:
-                logger.warn(message)
-            # FIXME: This is, of course, insane
+            logger.warn('Waiting for a 10s, before trying again')
             time.sleep(10)
-            logger.warn("Waited for a few seconds, will try again")
 
     parasolOutputPattern = re.compile("your job ([0-9]+).*")
 
     def issueBatchJob(self, command, memory, cores, disk):
-        """Issues parasol with job commands.
+        """
+        Issues parasol with job commands.
         """
         self.checkResourceRequest(memory, cores, disk)
 
         MiB = 1 << 20
-        truncatedMemory = (memory/MiB) * MiB
-        #Look for a batch for jobs with these resource requirements, with
-        #the memory rounded down to the nearest megabyte. Rounding down
-        #meams the new job can't ever decrease the memory requirements
-        #of jobs already in the batch.
-        if (truncatedMemory, cores) in self.resultsFiles.keys():
+        truncatedMemory = (memory / MiB) * MiB
+        # Look for a batch for jobs with these resource requirements, with
+        # the memory rounded down to the nearest megabyte. Rounding down
+        # meams the new job can't ever decrease the memory requirements
+        # of jobs already in the batch.
+        if len(self.resultsFiles) >= self.maxBatches:
+            raise RuntimeError( 'Number of batches reached limit of %i' % self.maxBatches)
+        try:
             results = self.resultsFiles[(truncatedMemory, cores)]
-        else:
+        except KeyError:
             results = getTempFile(rootDir=self.parasolResultsDir)
             self.resultsFiles[(truncatedMemory, cores)] = results
-        if len(self.resultsFiles.values()) > self.maxBatches:
-            raise RuntimeError("Number of parasol batches exceeded the limit of %i" % self.maxBatches)
 
         # Prefix the command with environment overrides, optionally looking them up from the
         # current environment if the value is None
         command = ' '.join(concat('env', self.__environment(), command))
-        parasolCommand = ['-verbose', '-ram=%i' % memory, '-cpu=%i' % cores,
-                          '-results=' + results, 'add', 'job', command]
-        #Deal with the cpus
+        parasolCommand = ['-verbose',
+                          '-ram=%i' % memory,
+                          '-cpu=%i' % cores,
+                          '-results=' + results,
+                          'add', 'job', command]
+        # Deal with the cpus
         self.usedCpus += cores
-        while True: #Process finished results with no wait
+        while True:  # Process finished results with no wait
             try:
-               jobID = self.cpuUsageQueue.get_nowait()
+                jobID = self.cpuUsageQueue.get_nowait()
             except Empty:
                 break
             if jobID in self.jobIDsToCpu.keys():
                 self.usedCpus -= self.jobIDsToCpu.pop(jobID)
             assert self.usedCpus >= 0
-            self.cpuUsageQueue.task_done()
-        while self.usedCpus > self.maxCores: #If we are still waiting
+        while self.usedCpus > self.maxCores:  # If we are still waiting
             jobID = self.cpuUsageQueue.get()
             if jobID in self.jobIDsToCpu.keys():
                 self.usedCpus -= self.jobIDsToCpu.pop(jobID)
             assert self.usedCpus >= 0
-            self.cpuUsageQueue.task_done()
-        #Now keep going
+        # Now keep going
         while True:
-            #time.sleep(0.1) #Sleep to let parasol catch up #Apparently unnecessary
             line = self.runParasol(parasolCommand)[1][0]
             match = self.parasolOutputPattern.match(line)
-            if match is not None: #This is because parasol add job will return success, even if the job was not properly issued!
-                break
-            else:
-                logger.info("We failed to properly add the job, we will try again after a sleep")
+            if match is None:
+                # This is because parasol add job will return success, even if the job was not
+                # properly issued!
+                logger.info('We failed to properly add the job, we will try again after a 5s.')
                 time.sleep(5)
-        jobID = int(match.group(1))
-        self.jobIDsToCpu[jobID] = cores
-        self.runningJobs.add(jobID)
-        logger.debug("Got the parasol job id: %s from line: %s" % (jobID, line))
-        return jobID
+            else:
+                jobID = int(match.group(1))
+                self.jobIDsToCpu[jobID] = cores
+                self.runningJobs.add(jobID)
+                logger.debug("Got the parasol job id: %s from line: %s" % (jobID, line))
+                return jobID
 
     def setEnv(self, name, value=None):
         if value and ' ' in value:
@@ -198,27 +188,30 @@ def killBatchJobs(self, jobIDs):
                 if jobID in self.runningJobs:
                     self.runningJobs.remove(jobID)
                 exitValue = self.runParasol(['remove', 'job', str(jobID)],
-                                            runUntilSuccessful=False)[0]
+                                            autoRetry=False)[0]
                 logger.info("Tried to remove jobID: %i, with exit value: %i" % (jobID, exitValue))
             runningJobs = self.getIssuedBatchJobIDs()
             if set(jobIDs).difference(set(runningJobs)) == set(jobIDs):
                 break
+            logger.warn( 'Tried to kill some jobs, but something happened and they are still '
+                         'going, will try againin 5s.')
             time.sleep(5)
-            logger.warn("Tried to kill some jobs, but something happened and they are still going, so I'll try again")
-        #update the CPU usage, because killed jobs aren't
-        #written to the results file.
+        # Update the CPU usage, because killed jobs aren't written to the results file.
         for jobID in jobIDs:
             if jobID in self.jobIDsToCpu.keys():
                 self.usedCpus -= self.jobIDsToCpu.pop(jobID)
 
+    queuePattern = re.compile(r'q\s+([0-9]+)')
+    runningPattern = re.compile(r'r\s+([0-9]+)\s+[\S]+\s+[\S]+\s+([0-9]+)\s+[\S]+')
+
     def getJobIDsForResultsFile(self, resultsFile):
-        """Get all queued and running jobs for a results file.
+        """
+        Get all queued and running jobs for a results file.
         """
         jobIDs = []
-        for line in self.runParasol(['-results=' + resultsFile,'pstat2'])[1]:
+        for line in self.runParasol(['-results=' + resultsFile, 'pstat2'])[1]:
             runningJobMatch = self.runningPattern.match(line)
             queuedJobMatch = self.queuePattern.match(line)
-            jobID = None
             if runningJobMatch:
                 jobID = runningJobMatch.group(1)
             elif queuedJobMatch:
@@ -229,21 +222,23 @@ def getJobIDsForResultsFile(self, resultsFile):
         return set(jobIDs)
 
     def getIssuedBatchJobIDs(self):
-        """Gets the list of jobs issued to parasol in all results files, but 
-        not including jobs created by other users.
+        """
+        Gets the list of jobs issued to parasol in all results files, but not including jobs
+        created by other users.
         """
         issuedJobs = set()
-        for resultsFile in self.resultsFiles.values():
+        for resultsFile in self.resultsFiles.itervalues():
             issuedJobs.update(self.getJobIDsForResultsFile(resultsFile))
 
         return list(issuedJobs)
 
     def getRunningBatchJobIDs(self):
-        """Returns map of running jobIDs and the time they have been running.
         """
-        #Example lines..
-        #r 5410186 benedictpaten worker 1247029663 localhost
-        #r 5410324 benedictpaten worker 1247030076 localhost
+        Returns map of running jobIDs and the time they have been running.
+        """
+        # Example lines..
+        # r 5410186 benedictpaten worker 1247029663 localhost
+        # r 5410324 benedictpaten worker 1247030076 localhost
         runningJobs = {}
         issuedJobs = self.getIssuedBatchJobIDs()
         for line in self.runParasol(['pstat2'])[1]:
@@ -252,34 +247,35 @@ def getRunningBatchJobIDs(self):
                 if match is not None:
                     jobID = int(match.group(1))
                     startTime = int(match.group(2))
-                    if jobID in issuedJobs: #It's one of our jobs
+                    if jobID in issuedJobs:  # It's one of our jobs
                         runningJobs[jobID] = time.time() - startTime
         return runningJobs
 
     def getUpdatedBatchJob(self, maxWait):
-        try:
-            info = self.updatedJobsQueue.get(timeout=maxWait)
-        except Empty:
-            return None
-        jobID, exitValue = info
-        self.updatedJobsQueue.task_done()
-        if jobID not in self.runningJobs:
-            #we tried to kill this job, but it ended by itself
-            #instead, so skip it.
-            return self.getUpdatedJob(maxWait)
-        else:
-            self.runningJobs.remove(jobID)
-        return jobID, exitValue
+        while True:
+            try:
+                jobID, status = self.updatedJobsQueue.get(timeout=maxWait)
+            except Empty:
+                return None
+            try:
+                self.runningJobs.remove(jobID)
+            except KeyError:
+                # We tried to kill this job, but it ended by itself instead, so skip it.
+                pass
+            else:
+                return jobID, status
 
     @classmethod
     def getRescueBatchJobFrequency(cls):
-        """Parasol leaks jobs, but rescuing jobs involves calls to parasol list jobs and pstat2,
-        making it expensive. 
         """
-        return 5400 #Once every 90 minutes
+        Parasol leaks jobs, but rescuing jobs involves calls to parasol list jobs and pstat2,
+        making it expensive.
+        """
+        return 5400  # Once every 90 minutes
 
     def updatedJobWorker(self):
-        """We use the parasol results to update the status of jobs, adding them
+        """
+        We use the parasol results to update the status of jobs, adding them
         to the list of updated jobs.
 
         Results have the following structure.. (thanks Mark D!)
@@ -302,27 +298,29 @@ def updatedJobWorker(self):
         resultsFileHandles = []
         try:
             while self.running:
-                #Look for any new results files that have been created, and open them
+                # Look for any new results files that have been created, and open them
                 newResultsFiles = set(os.listdir(self.parasolResultsDir)).difference(resultsFiles)
                 for newFile in newResultsFiles:
-                    resultsFiles.add(newFile)
                     newFilePath = os.path.join(self.parasolResultsDir, newFile)
                     resultsFileHandles.append(open(newFilePath, 'r'))
-
+                    resultsFiles.add(newFile)
                 for fileHandle in resultsFileHandles:
-                    line = fileHandle.readline()
-                    if line:
+                    while self.running:
+                        line = fileHandle.readline()
+                        if not line:
+                            break
                         assert line[-1] == '\n'
-                        results = line.split()
-                        result = int(results[0])
-                        # I don't know what Parasol puts in the low-order byte but the hight
-                        # order byte appears to be the status code.
-                        if result > 0xFF:
-                            result >>= 8
-                        jobID = int(results[2])
-                        self.cpuUsageQueue.put(jobID)
-                        self.updatedJobsQueue.put((jobID, result))
-                time.sleep(0.01) #Go to sleep to avoid churning
+                        status, host, jobId, exe, usrTicks, sysTicks, submitTime, startTime, \
+                        endTime, user, errFile, command = line[:-1].split(None, 11)
+                        status = int(status)
+                        jobId = int(jobId)
+                        if os.WIFEXITED(status):
+                            status = os.WEXITSTATUS(status)
+                        else:
+                            status = -status
+                        self.cpuUsageQueue.put(jobId)
+                        self.updatedJobsQueue.put((jobId, status))
+                time.sleep(1)
         except:
             logger.warn("Error occurred while parsing parasol results files.")
             raise
@@ -330,14 +328,15 @@ def updatedJobWorker(self):
             for fileHandle in resultsFileHandles:
                 fileHandle.close()
 
-
     def shutdown(self):
-        self.killBatchJobs(self.getIssuedBatchJobIDs()) #cleanup jobs
-        for results in self.resultsFiles.values():
-            exitValue = self.runParasol(['-results='+results, 'clear', 'sick'], runUntilSuccessful=False)[0]
+        self.killBatchJobs(self.getIssuedBatchJobIDs())  # cleanup jobs
+        for results in self.resultsFiles.itervalues():
+            exitValue = self.runParasol(['-results=' + results, 'clear', 'sick'],
+                                        autoRetry=False)[0]
             if exitValue is not None:
                 logger.warn("Could not clear sick status of the parasol batch %s" % results)
-            exitValue = self.runParasol(['-results='+results, 'flushResults'], runUntilSuccessful=False)[0]
+            exitValue = self.runParasol(['-results=' + results, 'flushResults'],
+                                        autoRetry=False)[0]
             if exitValue is not None:
                 logger.warn("Could not flush the parasol batch %s" % results)
         self.running = False
diff --git a/src/toil/batchSystems/parasolTestSupport.py b/src/toil/batchSystems/parasolTestSupport.py
--- a/src/toil/batchSystems/parasolTestSupport.py
+++ b/src/toil/batchSystems/parasolTestSupport.py
@@ -1,11 +1,13 @@
 from __future__ import absolute_import
-from abc import ABCMeta, abstractmethod
 import logging
+import tempfile
 import threading
 import time
 import subprocess
 import multiprocessing
 import os
+from bd2k.util.files import rm_f
+from bd2k.util.objects import InnerClass
 
 log = logging.getLogger(__name__)
 
@@ -14,19 +16,20 @@ class ParasolTestSupport(object):
     """
     For test cases that need a running Parasol leader and worker on the local host
     """
-    def _startParasol(self, numCores=None):
+
+    def _startParasol(self, numCores=None, memory=None):
         if numCores is None:
             numCores = multiprocessing.cpu_count()
-        self.machineList = os.path.join(os.getcwd(), "machineList.txt")
-        with open(self.machineList, "w") as out:
-            out.write("localhost 4 3000 /tmp /scratch 36000 r1")
-
-        self.leader = self.ParasolLeaderThread(numCores)
+        if memory is None:
+            memory = self._physicalMemory()
+        self.numCores = numCores
+        self.memory = memory
+        self.leader = self.ParasolLeaderThread()
         self.leader.start()
-        self.worker = self.ParasolWorkerThread(numCores)
+        self.worker = self.ParasolWorkerThread()
         self.worker.start()
         while self.leader.popen is None or self.worker.popen is None:
-            log.info("Waiting for leader and worker processes")
+            log.info('Waiting for leader and worker processes')
             time.sleep(.1)
 
     def _stopParasol(self):
@@ -34,43 +37,70 @@ def _stopParasol(self):
         self.worker.join()
         self.leader.popen.kill()
         self.leader.join()
-        os.remove(self.machineList)
-        if os.path.exists("para.results"):
-            os.remove("para.results")
-        if os.path.exists("parasol.jid"):
-            os.remove("parasol.jid")
+        for path in ('para.results', 'parasol.jid'):
+            rm_f(path)
+
+    @classmethod
+    def _physicalMemory(cls):
+        return os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES')
 
     class ParasolThread(threading.Thread):
-        __metaclass__ = ABCMeta
 
         # Lock is used because subprocess is NOT thread safe: http://tinyurl.com/pkp5pgq
         lock = threading.Lock()
 
-        def __init__(self, numCores):
+        def __init__(self):
             threading.Thread.__init__(self)
-            self.numCores = numCores
             self.popen = None
-            
 
-        @abstractmethod
         def parasolCommand(self):
             raise NotImplementedError
 
         def run(self):
+            command = self.parasolCommand()
             with self.lock:
-                self.popen = subprocess.Popen(self.parasolCommand())
-            self.popen.wait()
+                self.popen = subprocess.Popen(command)
+            status = self.popen.wait()
+            if status != 0:
+                log.error("Command '%s' failed with %i.", command, status)
+                raise subprocess.CalledProcessError(status, command)
             log.info('Exiting %s', self.__class__.__name__)
 
+    @InnerClass
     class ParasolLeaderThread(ParasolThread):
+
+        def __init__(self):
+            super(ParasolTestSupport.ParasolLeaderThread, self).__init__()
+            self.machineList = None
+
+        def run(self):
+            with tempfile.NamedTemporaryFile(prefix='machineList.txt', mode='w') as f:
+                self.machineList = f.name
+                # name - Network name
+                # cpus - Number of CPUs we can use
+                # ramSize - Megabytes of memory
+                # tempDir - Location of (local) temp dir
+                # localDir - Location of local data dir
+                # localSize - Megabytes of local disk
+                # switchName - Name of switch this is on
+                f.write('localhost {numCores} {ramSize} {tempDir} {tempDir} 1024 foo'.format(
+                    numCores=self.outer.numCores,
+                    tempDir=tempfile.gettempdir(),
+                    ramSize=self.outer.memory / 1024 / 1024))
+                f.flush()
+                super(ParasolTestSupport.ParasolLeaderThread, self).run()
+
         def parasolCommand(self):
             return ['paraHub',
+                    '-spokes=1',
                     '-debug',
-                    os.path.join(os.getcwd(), "machineList.txt")]
+                    self.machineList]
 
+    @InnerClass
     class ParasolWorkerThread(ParasolThread):
         def parasolCommand(self):
             return ['paraNode',
-                    '-cpu=%i' % self.numCores,
+                    '-cpu=%i' % self.outer.numCores,
+                    '-randomDelay=0',
                     '-debug',
                     'start']


Test patch:
diff --git a/src/toil/test/__init__.py b/src/toil/test/__init__.py
--- a/src/toil/test/__init__.py
+++ b/src/toil/test/__init__.py
@@ -209,8 +209,8 @@ def needs_parasol(test_item):
     """
     test_item = _mark_test('parasol', test_item)
     try:
-        with open('/dev/null', 'a') as dev_null:
-            subprocess.Popen('parasol', stdout=dev_null, stderr=dev_null)
+        with open(os.devnull, 'r+') as devnull:
+            subprocess.Popen('parasol', stdout=devnull, stderr=devnull, stdin=devnull)
     except OSError:
         return unittest.skip("Skipping test. Install Parasol to include this test.")(test_item)
     except:
diff --git a/src/toil/test/batchSystems/batchSystemTest.py b/src/toil/test/batchSystems/batchSystemTest.py
--- a/src/toil/test/batchSystems/batchSystemTest.py
+++ b/src/toil/test/batchSystems/batchSystemTest.py
@@ -397,8 +397,11 @@ def _createDummyConfig(self):
         return config
 
     def createBatchSystem(self):
-        self._startParasol(numCores)
-        return ParasolBatchSystem(config=self.config, maxCores=numCores, maxMemory=3e9,
+        memory = int(3e9)
+        self._startParasol(numCores=numCores, memory=memory)
+        return ParasolBatchSystem(config=self.config,
+                                  maxCores=numCores,
+                                  maxMemory=memory,
                                   maxDisk=1001)
 
     def tearDown(self):
@@ -412,12 +415,10 @@ def testBatchResourceLimits(self):
 
         batches = self._getBatchList()
         self.assertEqual(len(batches), 2)
-        # It would be better to directly check that the batches
-        # have the correct memory and cpu values, but parasol seems
-        # to slightly change the values sometimes.
-        self.assertTrue(batches[0]["ram"] != batches[1]["ram"])
-
-        # need to kill one of the jobs because there are only two cores available
+        # It would be better to directly check that the batches have the correct memory and cpu
+        # values, but Parasol seems to slightly change the values sometimes.
+        self.assertNotEqual(batches[0]['ram'], batches[1]['ram'])
+        # Need to kill one of the jobs because there are only two cores available
         self.batchSystem.killBatchJobs([job2])
         job3 = self.batchSystem.issueBatchJob("sleep 1000", memory=1e9, cores=1, disk=1000)
         batches = self._getBatchList()
@@ -440,7 +441,7 @@ def _parseBatchString(self, batchString):
     def _getBatchList(self):
         exitStatus, batchLines = self.batchSystem.runParasol(['list', 'batches'])
         self.assertEqual(exitStatus, 0)
-        return [self._parseBatchString(line) for line in batchLines[1:] if not line == ""]
+        return [self._parseBatchString(line) for line in batchLines[1:] if line]
 
 
 @needs_gridengine
diff --git a/src/toil/test/sort/sortTest.py b/src/toil/test/sort/sortTest.py
--- a/src/toil/test/sort/sortTest.py
+++ b/src/toil/test/sort/sortTest.py
@@ -189,7 +189,6 @@ def testFileGridEngine(self):
         self._toilSort(jobStore=self._getTestJobStorePath(), batchSystem='gridengine')
 
     @needs_parasol
-    @unittest.skip("skipping until parasol support is less flaky (see github issue #449")
     def testFileParasol(self):
         self._startParasol()
         try: