Repository: DataBiosphere/toil

Issue:
Toil default cache size is too small
The default 2.14G cache is too small for most purposes. Suggestion for enhancement could be to dynamically allocate it to 50%(or whatever fraction) of the free space on the volume that holds the localTempDir

```
import ctypes
import os
import platform
import sys

def get_free_space_mb(dirname):
    """Return folder/drive free space (in megabytes)."""
    if platform.system() == 'Windows':
        free_bytes = ctypes.c_ulonglong(0)
        ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(dirname), None, None, ctypes.pointer(free_bytes))
        return free_bytes.value / 1024 / 1024
    else:
        st = os.statvfs(dirname)
        return st.f_bavail * st.f_frsize / 1024 / 1024

cache_size = max(0.5*get_free_space(self. localTempDir), 2.14G)
```

Code was obtained from 

http://stackoverflow.com/questions/51658/cross-platform-space-remaining-on-volume-using-python

Caching fails for large-ish files being written asynchronously to the jobStore
Following on from Arjun's tests we established that if a file was being written to the file store then it's local file could be cleaned from the cache, resulting in subsequent jobs trying to retrieve an incompletely written file from the jobStore.



Discussion:
#486

Looks got. We can safely delete the windows part since Toil does not support it. When computing the default value for the cache size,  the directory  where the cache will be created should be passed to the above function.

Beware that statvs is deprecated

https://docs.python.org/2/library/statvfs.html

As an alternative, checkout psutil which Toil already depends  on for Mesos.

i think `--workDir` is the cache location.  However, this will have to happen AFTER input arguments have been parsed since we cannot assume what folder the user will use for `--workDir` when the help is being generated and hence cannot print an accurate value in the default field.

Hi Arjun,

Intentionally the cache is set small by default, because a large cache
could exhaust space on the users local disk, if we're not quite careful.
Your heuristic setting in a race condition of multiple workers per node is
pretty dangerous.

My preferred option is to create a system that monitors the total size of
the files on the users disk and dynamically reduces the cached files to
stay within a threshold, but that is substantially more complex than what
is currently implemented. Can you play with the cache size parameter to see
if it can be made to work for you?

On Mon, Oct 12, 2015 at 5:46 PM, Arjun Arkal Rao notifications@github.com
wrote:

> i think --workDir is the cache location. However, this will have to
> happen AFTER input arguments have been parsed since we cannot assume what
> folder the user will use for --workDir when the help is being generated
> and hence cannot print an accurate value in the default field.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/488#issuecomment-147530483.
> 
> ## 
> 
> You received this message because you are subscribed to the Google Groups
> "cgloud-jenkins" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cgloud-jenkins+unsubscribe@googlegroups.com.
> To post to this group, send email to cgloud-jenkins@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil/issues/488/147530483%40github.com
> https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil/issues/488/147530483%40github.com?utm_medium=email&utm_source=footer
> .
> 
> For more options, visit https://groups.google.com/d/optout.



Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -68,6 +68,7 @@ def __init__(self):
         self.defaultMemory = 2147483648
         self.defaultCores = 1
         self.defaultDisk = 2147483648
+        self.defaultCache = 2147483648 #Cache is 2GB
         self.maxCores = sys.maxint
         self.maxMemory = sys.maxint
         self.maxDisk = sys.maxint
@@ -81,7 +82,6 @@ def __init__(self):
         self.maxLogFileSize=50120
         self.sseKey = None
         self.cseKey = None
-        self.cacheSize = 2147483648 #Cache is 2GB
         
         #Debug options
         self.badWorker = 0.0
@@ -147,6 +147,7 @@ def f(x):
         setOption("defaultMemory", h2b, iC(1))
         setOption("defaultCores", h2b, iC(1))
         setOption("defaultDisk", h2b, iC(1))
+        setOption("defaultCache", h2b, iC(0))
         setOption("maxCores", h2b, iC(1))
         setOption("maxMemory", h2b, iC(1))
         setOption("maxDisk", h2b, iC(1))
@@ -163,7 +164,6 @@ def checkSse(sseKey):
                 assert(len(f.readline().rstrip()) == 32)
         setOption("sseKey", checkFn=checkSse)
         setOption("cseKey", checkFn=checkSse)
-        setOption("cacheSize", h2b, iC(0))
         
         #Debug options
         setOption("badWorker", float, iC(0, 1))
@@ -233,9 +233,13 @@ def _addOptions(addGroupFn, config):
                       help=("The default amount of memory to request for a job (in bytes), "
                             "by default is 2^31 = 2 gigabytes, default=%s" % config.defaultMemory))
     addOptionFn("--defaultCores", dest="defaultCores", default=None,
-                      help="The number of cpu cores to dedicate a job. default=%s" % config.defaultCores)
+                      help="The default number of cpu cores to dedicate a job. default=%s" % config.defaultCores)
     addOptionFn("--defaultDisk", dest="defaultDisk", default=None,
-                      help="The amount of disk space to dedicate a job (in bytes). default=%s" % config.defaultDisk)
+                      help="The default amount of disk space to dedicate a job (in bytes). default=%s" % config.defaultDisk)
+    addOptionFn("--defaultCache", dest="defaultCache", default=None,
+                help=("The default amount of disk space to use in caching "
+                      "files shared between jobs. This must be less than the disk requirement "
+                      "for the job default=%s" % config.defaultCache))
     addOptionFn("--maxCores", dest="maxCores", default=None,
                       help=("The maximum number of cpu cores to request from the batch system at any "
                             "one time. default=%s" % config.maxCores))
@@ -279,10 +283,6 @@ def _addOptions(addGroupFn, config):
                 help="Path to file containing 256-bit key to be used for client-side encryption on "
                 "azureJobStore. By default, no encryption is used.")
     
-    addOptionFn("--cacheSize", dest="cacheSize", default=None,
-                help=("The maximum amount of disk space to use in caching "
-                      "files shared between jobs. default=%s" % config.cacheSize))
-    
     #
     #Debug options
     #
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -28,11 +28,10 @@
 import logging
 import shutil
 import stat
-from threading import Thread, Semaphore
+from threading import Thread, Semaphore, Event
 from Queue import Queue, Empty
 from bd2k.util.humanize import human2bytes
 from io import BytesIO
-
 from toil.resource import ModuleDescriptor
 from toil.common import loadJobStore
 
@@ -55,16 +54,27 @@ class Job(object):
     This public functions of this class and its  nested classes are the API 
     to toil.
     """
-    def __init__(self, memory=None, cores=None, disk=None):
+    def __init__(self, memory=None, cores=None, disk=None, cache=None):
         """
         This method must be called by any overiding constructor.
         
-        Memory is the maximum number of bytes of memory the job will
-        require to run. Cores is the number of CPU cores required.
+        memory is the maximum number of bytes of memory the job will
+        require to run. 
+        
+        cores is the number of CPU cores required.
+        
+        disk is the amount of local disk space required by the job, 
+        expressed in bytes.
+        
+        cache is the amount of disk (so that cache <= disk), expressed in bytes,
+        for storing files from previous jobs computed on the worker so 
+        that they can be accessed from a local copy. 
         """
         self.cores = cores
-        self.memory = human2bytes(str(memory)) if memory is not None else memory
-        self.disk = human2bytes(str(disk)) if disk is not None else disk
+        parse = lambda x : human2bytes(str(x)) if x is not None else x
+        self.memory = parse(memory)
+        self.disk = parse(disk)
+        self.cache = parse(cache)
         #Private class variables
 
         #See Job.addChild
@@ -204,7 +214,7 @@ def encapsulate(self):
     #The following function is used for passing return values between
     #job run functions
     ####################################################
-
+    
     def rv(self, argIndex=None):
         """
         Gets a PromisedJobReturnValue, representing the argIndex return
@@ -366,8 +376,15 @@ class FileStore( object ):
         Class used to manage temporary files and log messages, 
         passed as argument to the Job.run method.
         """
-        def __init__(self, jobStore, jobWrapper, localTempDir,
-                     inputBlockFn, jobStoreFileIDToCacheLocation, terminateEvent):
+        #Variables used for synching reads/writes
+        lockFilesLock = Semaphore()
+        lockFiles = set()
+        #For files in jobStore that are on the local disk, 
+        #map of jobStoreFileIDs to locations in localTempDir.
+        jobStoreFileIDToCacheLocation = {}
+        terminateEvent = Event() #Used to signify crashes in threads
+        
+        def __init__(self, jobStore, jobWrapper, localTempDir, inputBlockFn):
             """
             This constructor should not be called by the user, 
             FileStore instances are only provided as arguments 
@@ -382,8 +399,7 @@ def __init__(self, jobStore, jobWrapper, localTempDir,
             #Asynchronous writes stuff
             self.workerNumber = 2
             self.queue = Queue()
-            self.updateSemaphore = Semaphore()
-            self.terminateEvent = terminateEvent
+            self.updateSemaphore = Semaphore() 
             #Function to write files to job store
             def asyncWrite():
                 try:
@@ -392,17 +408,17 @@ def asyncWrite():
                             #Block for up to two seconds waiting for a file
                             args = self.queue.get(timeout=2)
                         except Empty:
-                            #Check if termination event is signaled
+                            #Check if termination event is signaled 
                             #(set in the event of an exception in the worker)
-                            if terminateEvent.isSet():
+                            if self.terminateEvent.isSet():
                                 raise RuntimeError("The termination flag is set, exiting")
                             continue
                         #Normal termination condition is getting None from queue
                         if args == None:
                             break
                         inputFileHandle, jobStoreFileID = args
-                        #We pass in a fileHandle, rather than the file-name, in case
-                        #the file itself is deleted. The fileHandle itself should persist
+                        #We pass in a fileHandle, rather than the file-name, in case 
+                        #the file itself is deleted. The fileHandle itself should persist 
                         #while we maintain the open file handle
                         with jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
                             bufferSize=1000000 #TODO: This buffer number probably needs to be modified/tuned
@@ -412,20 +428,18 @@ def asyncWrite():
                                     break
                                 outputFileHandle.write(copyBuffer)
                         inputFileHandle.close()
+                        #Remove the file from the lock files
+                        with self.lockFilesLock:
+                            self.lockFiles.remove(jobStoreFileID)
                 except:
-                    terminateEvent.set()
+                    self.terminateEvent.set()
                     raise
-
-            self.workers = map(lambda i : Thread(target=asyncWrite),
+                    
+            self.workers = map(lambda i : Thread(target=asyncWrite), 
                                range(self.workerNumber))
             for worker in self.workers:
                 worker.start()
             self.inputBlockFn = inputBlockFn
-            #Caching
-            #
-            #For files in jobStore that are on the local disk,
-            #map of jobStoreFileIDs to locations in localTempDir.
-            self.jobStoreFileIDToCacheLocation = jobStoreFileIDToCacheLocation
 
         def getLocalTempDir(self):
             """
@@ -435,7 +449,7 @@ def getLocalTempDir(self):
             the job terminates, removing all files it contains recursively. 
             """
             return os.path.abspath(tempfile.mkdtemp(prefix="t", dir=self.localTempDir))
-
+        
         def getLocalTempFile(self):
             """
             Get an absolute path to a local temporary file. 
@@ -445,7 +459,7 @@ def getLocalTempFile(self):
             handle, tmpFile = tempfile.mkstemp(prefix="tmp",
                                                suffix=".tmp", dir=self.localTempDir)
             os.close(handle)
-            return os.path.abspath(tmpFile)
+            return os.path.abspath(tmpFile) 
 
         def writeGlobalFile(self, localFileName, cleanup=False):
             """
@@ -471,12 +485,14 @@ def writeGlobalFile(self, localFileName, cleanup=False):
                 self.queue.put((open(absLocalFileName, 'r'), jobStoreFileID))
                 #Chmod to make file read only to try to prevent accidental user modification
                 os.chmod(absLocalFileName, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                with self.lockFilesLock:
+                    self.lockFiles.add(jobStoreFileID)
                 self.jobStoreFileIDToCacheLocation[jobStoreFileID] = absLocalFileName
             else:
                 #Write the file directly to the file store
                 jobStoreFileID = self.jobStore.writeFile(localFileName, cleanupID)
             return jobStoreFileID
-
+        
         def writeGlobalFileStream(self, cleanup=False):
             """
             Similar to writeGlobalFile, but returns a context manager yielding a 
@@ -488,7 +504,7 @@ def writeGlobalFileStream(self, cleanup=False):
             """
             #TODO: Make this work with the caching??
             return self.jobStore.writeFileStream(None if not cleanup else self.jobWrapper.jobStoreID)
-
+        
         def readGlobalFile(self, fileStoreID, userPath=None, cache=True):
             """
             Returns an absolute path to a local, temporary copy of the file 
@@ -523,11 +539,11 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True):
                 userPath = os.path.abspath(userPath) #Make an absolute path
                 #Turn off caching if user file is not in localTempDir
                 if cache and not userPath.startswith(self.localTempDir):
-                    cache = False
+                    cache = False  
             #When requesting a new file from the jobStore first check if fileStoreID
             #is a key in jobStoreFileIDToCacheLocation.
             if fileStoreID in self.jobStoreFileIDToCacheLocation:
-                cachedAbsFilePath = self.jobStoreFileIDToCacheLocation[fileStoreID]
+                cachedAbsFilePath = self.jobStoreFileIDToCacheLocation[fileStoreID]   
                 if cache:
                     #If the user specifies a location and it is not the current location
                     #return a hardlink to the location, else return the original location
@@ -544,7 +560,7 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True):
                     shutil.copyfile(cachedAbsFilePath, localFilePath)
                     return localFilePath
             else:
-                #If it is not in the cache read it from the jobStore to the
+                #If it is not in the cache read it from the jobStore to the 
                 #desired location
                 localFilePath = userPath if userPath != None else self.getLocalTempFile()
                 self.jobStore.readFile(fileStoreID, localFilePath)
@@ -564,11 +580,11 @@ def readGlobalFileStream(self, fileStoreID):
             """
             if fileStoreID in self.filesToDelete:
                 raise RuntimeError("Trying to access a file in the jobStore you've deleted: %s" % fileStoreID)
-
+            
             #If fileStoreID is in the cache provide a handle from the local cache
             if fileStoreID in self.jobStoreFileIDToCacheLocation:
                 #This leaks file handles (but the commented out code does not work properly)
-                return open(self.jobStoreFileIDToCacheLocation[fileStoreID], 'r')
+                return open(self.jobStoreFileIDToCacheLocation[fileStoreID], 'r') 
                 #with open(self.jobStoreFileIDToCacheLocation[fileStoreID], 'r') as fH:
                 #        yield fH
             else:
@@ -596,9 +612,9 @@ def logToMaster(self, string, level=logging.INFO):
             """
             logger.log(level=level, msg=("LOG-TO-MASTER: " + string))
             self.loggingMessages.append((str(string), level))
-
-        #Private methods
-
+            
+        #Private methods 
+        
         def _updateJobWhenDone(self):
             """
             Asynchronously update the status of the job on the disk, first waiting
@@ -610,33 +626,33 @@ def asyncUpdate():
                     #Wait till all file writes have completed
                     for i in xrange(len(self.workers)):
                         self.queue.put(None)
-
+            
                     for thread in self.workers:
                         thread.join()
-
+                    
                     #Wait till input block-fn returns - in the event of an exception
-                    #this will eventually terminate
+                    #this will eventually terminate 
                     self.inputBlockFn()
-
+                    
                     #Check the terminate event, if set we can not guarantee
                     #that the workers ended correctly, therefore we exit without
                     #completing the update
                     if self.terminateEvent.isSet():
                         raise RuntimeError("The termination flag is set, exiting before update")
-
-                    #Indicate any files that should be deleted once the update of
+                    
+                    #Indicate any files that should be deleted once the update of 
                     #the job wrapper is completed.
                     self.jobWrapper.filesToDelete = list(self.filesToDelete)
-
+                    
                     #Complete the job
                     self.jobStore.update(self.jobWrapper)
-
+                    
                     #Delete any remnant jobs
                     map(self.jobStore.delete, self.jobsToDelete)
-
+                    
                     #Delete any remnant files
                     map(self.jobStore.deleteFile, self.filesToDelete)
-
+                    
                     #Remove the files to delete list, having successfully removed the files
                     if len(self.filesToDelete) > 0:
                         self.jobWrapper.filesToDelete = []
@@ -657,7 +673,7 @@ def asyncUpdate():
             except: #This is to ensure that the semaphore is released in a crash to stop a deadlock scenario
                 self.updateSemaphore.release()
                 raise
-
+            
         def _cleanLocalTempDir(self, cacheSize):
             """
             At the end of the job, remove all localTempDir files except those whose value is in
@@ -666,10 +682,12 @@ def _cleanLocalTempDir(self, cacheSize):
             :param cacheSize: the total number of bytes of files allowed in the cache.
             """
             #Remove files so that the total cached files are smaller than a cacheSize
-
+            
             #List of pairs of (fileCreateTime, fileStoreID) for cached files
+            with self.lockFilesLock:
+                deletableCacheFiles = set(self.jobStoreFileIDToCacheLocation.keys()) - self.lockFiles
             cachedFileCreateTimes = map(lambda x : (os.stat(self.jobStoreFileIDToCacheLocation[x]).st_ctime, x),
-                                        self.jobStoreFileIDToCacheLocation.keys())
+                                        deletableCacheFiles)
             #Total number of bytes stored in cached files
             totalCachedFileSizes = sum([os.stat(self.jobStoreFileIDToCacheLocation[x]).st_size for x in
                                         self.jobStoreFileIDToCacheLocation.keys()])
@@ -678,44 +696,44 @@ def _cleanLocalTempDir(self, cacheSize):
             cachedFileCreateTimes.sort()
             cachedFileCreateTimes.reverse()
             #Now do the actual file removal
-            while totalCachedFileSizes > cacheSize:
-                fileCreateTime, fileStoreID =  cachedFileCreateTimes.pop()
+            while totalCachedFileSizes > cacheSize and len(cachedFileCreateTimes) > 0:
+                fileCreateTime, fileStoreID = cachedFileCreateTimes.pop()
                 fileSize = os.stat(self.jobStoreFileIDToCacheLocation[fileStoreID]).st_size
                 filePath = self.jobStoreFileIDToCacheLocation[fileStoreID]
                 self.jobStoreFileIDToCacheLocation.pop(fileStoreID)
                 os.remove(filePath)
                 totalCachedFileSizes -= fileSize
                 assert totalCachedFileSizes >= 0
-
-            #Iterate from the base of localTempDir and remove all
+            
+            #Iterate from the base of localTempDir and remove all 
             #files/empty directories, recursively
             cachedFiles = set(self.jobStoreFileIDToCacheLocation.values())
-
-            def clean(dirOrFile):
-                canRemove = True
+            
+            def clean(dirOrFile, remove=True):
+                canRemove = True 
                 if os.path.isdir(dirOrFile):
                     for f in os.listdir(dirOrFile):
                         canRemove = canRemove and clean(os.path.join(dirOrFile, f))
-                    if canRemove:
+                    if canRemove and remove:
                         os.rmdir(dirOrFile) #Dir should be empty if canRemove is true
                     return canRemove
                 if dirOrFile in cachedFiles:
                     return False
                 os.remove(dirOrFile)
-                return True
-            clean(self.localTempDir)
-
+                return True    
+            clean(self.localTempDir, False)
+        
         def _blockFn(self):
             """
             Blocks while _updateJobWhenDone is running.
-            """
+            """ 
             self.updateSemaphore.acquire()
             self.updateSemaphore.release() #Release so that the block function can be recalled
             #This works, because once acquired the semaphore will not be acquired
             #by _updateJobWhenDone again.
             return
-
-        def __del__(self):
+        
+        def __del__(self): 
             """Cleanup function that is run when destroying the class instance 
             that ensures that all the file writing threads exit.
             """
@@ -836,7 +854,7 @@ def filter_main(module_name, class_name):
 
         unpickler.find_global = filter_main
         return unpickler.load()
-
+    
     def getUserScript(self):
         return self.userModule
 
@@ -912,7 +930,7 @@ def _getImpliedEdges(roots):
                 for descendant in reacheable:
                     extraEdges[descendant] += job._followOns[:]
         return extraEdges
-
+    
     ####################################################
     #The following functions are used to serialise
     #a job graph to the jobStore
@@ -966,15 +984,23 @@ def _createEmptyJobForJob(self, jobStore, command=None,
         """
         Create an empty job for the job.
         """
+        memory=(self.memory if self.memory is not None
+               else float(jobStore.config.defaultMemory))
+        cores=(self.cores if self.cores is not None
+               else float(jobStore.config.defaultCores))
+        disk=(self.disk if self.disk is not None
+              else float(jobStore.config.defaultDisk))
+        cache=(self.cache if self.cache is not None
+              else float(jobStore.config.defaultCache))
+        
+        if cache > disk:
+            raise RuntimeError("Trying to allocate a cache (cache: %s) larger"
+                               " than the disk requirement for the job! (disk: %s)" % (cache, disk))
+
         return jobStore.create(command=command,
-                               memory=(self.memory if self.memory is not None
-                                       else jobStore.config.defaultMemory),
-                               cores=(self.cores if self.cores is not None
-                                    else float(jobStore.config.defaultCores)),
-                               disk=(self.disk if self.disk is not None
-                                    else float(jobStore.config.defaultDisk)),
+                               memory=memory, cores=cores, disk=disk,
                                predecessorNumber=predecessorNumber)
-
+        
     def _makeJobWrappers(self, jobWrapper, jobStore):
         """
         Creates a job for each job in the job graph, recursively.
@@ -998,14 +1024,14 @@ def _makeJobWrappers2(self, jobStore, jobsToJobWrappers):
                 jobWrapper.stack.append(jobs)
         else:
             jobWrapper = jobsToJobWrappers[self]
-        #The return is a tuple stored within a job.stack
+        #The return is a tuple stored within a job.stack 
         #The tuple is jobStoreID, memory, cores, disk, predecessorID
         #The predecessorID is used to establish which predecessors have been
         #completed before running the given Job - it is just a unique ID
         #per predecessor
         return (jobWrapper.jobStoreID, jobWrapper.memory, jobWrapper.cores, jobWrapper.disk,
                 None if jobWrapper.predecessorNumber <= 1 else str(uuid.uuid4()))
-
+        
     def getTopologicalOrderingOfJobs(self):
         """
         Get a list of jobs such that for all pairs of indices i, j for which i < j, 
@@ -1025,7 +1051,7 @@ def getRunOrder(job):
                 map(getRunOrder, job._children + job._followOns)
         getRunOrder(self)
         return ordering
-
+    
     def _serialiseJob(self, jobStore, jobsToJobWrappers, rootJobWrapper):
         """
         Pickle a job and its jobWrapper to disk.
@@ -1053,8 +1079,8 @@ def _serialiseJob(self, jobStore, jobsToJobWrappers, rootJobWrapper):
         jobsToJobWrappers[self].command = ' '.join( ('_toil', fileStoreID) + userScript)
         #Update the status of the jobWrapper on disk
         jobStore.update(jobsToJobWrappers[self])
-
-    def _serialiseJobGraph(self, jobWrapper, jobStore, returnValues, firstJob):
+    
+    def _serialiseJobGraph(self, jobWrapper, jobStore, returnValues, firstJob):  
         """
         Pickle the graph of jobs in the jobStore. The graph is not fully serialised
         until the jobWrapper itself is written to disk, this is not performed by this
@@ -1067,13 +1093,13 @@ def _serialiseJobGraph(self, jobWrapper, jobStore, returnValues, firstJob):
         self.checkJobGraphForDeadlocks()
         #Create the jobWrappers for followOns/children
         jobsToJobWrappers = self._makeJobWrappers(jobWrapper, jobStore)
-        #Get an ordering on the jobs which we use for pickling the jobs in the
+        #Get an ordering on the jobs which we use for pickling the jobs in the 
         #correct order to ensure the promises are properly established
         ordering = self.getTopologicalOrderingOfJobs()
         assert len(ordering) == len(jobsToJobWrappers)
         #Temporarily set the jobStore strings for the promise call back functions
         for job in ordering:
-            job._promiseJobStore = jobStore
+            job._promiseJobStore = jobStore 
         ordering.reverse()
         assert self == ordering[-1]
         if firstJob:
@@ -1103,7 +1129,7 @@ def _serialiseJobGraph(self, jobWrapper, jobStore, returnValues, firstJob):
                 jobWrapper.stack.append(combinedFollowOns)
             if len(combinedChildren) > 0:
                 jobWrapper.stack.append(combinedChildren)
-
+            
     def _serialiseFirstJob(self, jobStore):
         """
         Serialises the root job. Returns the wrapping job.
@@ -1113,7 +1139,7 @@ def _serialiseFirstJob(self, jobStore):
                                                 predecessorNumber=0)
         #Write the graph of jobs to disk
         self._serialiseJobGraph(jobWrapper, jobStore, None, True)
-        jobStore.update(jobWrapper)
+        jobStore.update(jobWrapper) 
         #Store the name of the first job in a file in case of restart
         #Up to this point the root-job is not recoverable
         with jobStore.writeSharedFileStream("rootJobStoreID") as f:
@@ -1141,9 +1167,9 @@ def _execute(self, jobWrapper, stats, localTempDir, jobStore, fileStore):
         #Add the promise files to delete to the list of jobStoreFileIDs to delete
         for jobStoreFileID in promiseFilesToDelete:
             fileStore.deleteGlobalFile(jobStoreFileID)
-        promiseFilesToDelete.clear()
+        promiseFilesToDelete.clear() 
         #Now indicate the asynchronous update of the job can happen
-
+            
         fileStore._updateJobWhenDone()
         #Change dir back to cwd dir, if changed by job (this is a safety issue)
         if os.getcwd() != baseDir:
@@ -1339,8 +1365,8 @@ class PromisedJobReturnValue(object):
     argument to Job before the former Job's run function has been executed.
     """
     def __init__(self, promiseCallBackFunction):
-        self.promiseCallBackFunction = promiseCallBackFunction
-
+        self.promiseCallBackFunction = promiseCallBackFunction 
+        
 def promisedJobReturnValuePickleFunction(promise):
     """
     This function and promisedJobReturnValueUnpickleFunction are used as custom pickle/unpickle 
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -34,7 +34,7 @@
 import xml.etree.cElementTree as ET
 import cPickle
 import shutil
-from threading import Thread, Event
+from threading import Thread
 import signal
 
 logger = logging.getLogger( __name__ )
@@ -207,8 +207,7 @@ def badWorker():
     messageNode = ET.SubElement(elementNode, "messages")
     messages = []
     blockFn = lambda : True
-    jobStoreFileIDToCacheLocation = {}
-    terminateEvent = Event() #This is used to signify crashes in threads
+    cleanCacheFn = lambda x : True
     try:
 
         #Put a message at the top of the log, just to make sure it's working.
@@ -249,6 +248,9 @@ def badWorker():
         jobStore.update(jobWrapper) #Update first, before deleting the file
         if oldLogFile != None:
             jobStore.delete(oldLogFile)
+            
+        #Make a temporary file directory for the jobWrapper
+        localTempDir = makePublicDir(os.path.join(localWorkerTempDir, "localTempDir"))
     
         ##########################################
         #Setup the stats, if requested
@@ -266,35 +268,42 @@ def badWorker():
             
             if jobWrapper.command != None:
                 if jobWrapper.command.startswith( "_toil " ):
-                    #Make a temporary file directory for the jobWrapper
-                    localTempDir = makePublicDir(os.path.join(localWorkerTempDir, "localTempDir"))
+                    #Load the job
+                    job = Job._loadJob(jobWrapper.command, jobStore)
+                    
+                    #Cleanup the cache from the previous job
+                    cleanCacheFn(job.cache if job.cache != None else config.defaultCache)
                     
                     #Create a fileStore object for the job
                     fileStore = Job.FileStore(jobStore, jobWrapper, localTempDir, 
-                                              blockFn, jobStoreFileIDToCacheLocation, 
-                                              terminateEvent)
+                                              blockFn)
                     #Get the next block function and list that will contain any messages
                     blockFn = fileStore._blockFn
                     messages = fileStore.loggingMessages
-                    #Load and run the job
-                    Job._loadJob(jobWrapper.command, 
-                    jobStore)._execute( jobWrapper=jobWrapper,
-                                        stats=elementNode if config.stats else None, 
-                                        localTempDir=localTempDir,
-                                        jobStore=jobStore,
-                                        fileStore=fileStore)
                     
-                    #Cleanup the temporary file directory
-                    fileStore._cleanLocalTempDir(config.cacheSize)
+                    #Run the job
+                    job._execute( jobWrapper=jobWrapper,
+                                  stats=elementNode if config.stats else None, 
+                                  localTempDir=localTempDir, jobStore=jobStore,
+                                  fileStore=fileStore)
+                    
+                    #Set the clean cache function
+                    cleanCacheFn = fileStore._cleanLocalTempDir
+                    
                 else: #Is another command (running outside of jobs may be deprecated)
+                    #Cleanup the cache from the previous job
+                    cleanCacheFn(0)
+                    
                     system(jobWrapper.command)
+                    #Set a dummy clean cache fn
+                    cleanCacheFn = lambda x : None
             else:
                 #The command may be none, in which case
                 #the jobWrapper is either a shell ready to be deleted or has 
                 #been scheduled after a failure to cleanup
                 break
             
-            if terminateEvent.isSet():
+            if Job.FileStore.terminateEvent.isSet():
                 raise RuntimeError("The termination flag is set")
             
             ##########################################
@@ -363,8 +372,7 @@ def badWorker():
             assert jobWrapper.cores >= successorJob.cores
             
             #Build a fileStore to update the job
-            fileStore = Job.FileStore(jobStore, jobWrapper, localTempDir, blockFn, 
-                                      jobStoreFileIDToCacheLocation, terminateEvent)
+            fileStore = Job.FileStore(jobStore, jobWrapper, localTempDir, blockFn)
             
             #Update blockFn
             blockFn = fileStore._blockFn
@@ -400,7 +408,7 @@ def badWorker():
     except: #Case that something goes wrong in worker
         traceback.print_exc()
         logger.error("Exiting the worker because of a failed jobWrapper on host %s", socket.gethostname())
-        terminateEvent.set()
+        Job.FileStore.terminateEvent.set()
     
     ##########################################
     #Wait for the asynchronous chain of writes/updates to finish
@@ -413,7 +421,7 @@ def badWorker():
     #so safe to test if they completed okay
     ########################################## 
     
-    if terminateEvent.isSet():
+    if Job.FileStore.terminateEvent.isSet():
         jobWrapper = jobStore.load(jobStoreID)
         jobWrapper.setupJobAfterFailure(config)
         workerFailed = True


Test patch:
diff --git a/src/toil/test/mesos/helloWorld.py b/src/toil/test/mesos/helloWorld.py
--- a/src/toil/test/mesos/helloWorld.py
+++ b/src/toil/test/mesos/helloWorld.py
@@ -29,7 +29,7 @@ def hello_world(job):
     foo_bam = job.fileStore.writeGlobalFile('foo_bam.txt')
 
     # Spawn child
-    job.addChildJobFn(hello_world_child, foo_bam, memory=100, cores=0.5, disk=2000)
+    job.addChildJobFn(hello_world_child, foo_bam, memory=100, cores=0.5, disk="3G")
 
 
 def hello_world_child(job, hw):
@@ -58,7 +58,7 @@ def main():
 
     print args.jobStore
     # Launch first toil Job
-    i = Job.wrapJobFn(hello_world, memory=100, cores=0.5, disk=2000)
+    i = Job.wrapJobFn(hello_world, memory=100, cores=0.5, disk="3G")
     j = Job.Runner.startToil(i, args)
 
 
diff --git a/src/toil/test/src/helloWorldTest.py b/src/toil/test/src/helloWorldTest.py
--- a/src/toil/test/src/helloWorldTest.py
+++ b/src/toil/test/src/helloWorldTest.py
@@ -16,37 +16,33 @@
 from toil.job import Job
 from toil.test import ToilTest
 
-
 class HelloWorldTest(ToilTest):
     def testHelloWorld(self):
         options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
         options.logLevel = "INFO"
         Job.Runner.startToil(HelloWorld(), options)
 
-
 class HelloWorld(Job):
     def __init__(self):
-        Job.__init__(self, memory=100000, cores=2, disk=20000)
+        Job.__init__(self,  memory=100000, cores=2, disk="3G")
 
     def run(self, fileStore):
-        fileID = self.addChildJobFn(childFn, cores=1, memory="1M", disk="10M").rv()
+        fileID = self.addChildJobFn(childFn, cores=1, memory="1M", disk="3G").rv()
         self.addFollowOn(FollowOn(fileID))
 
-
 def childFn(job):
     with job.fileStore.writeGlobalFileStream() as (fH, fileID):
         fH.write("Hello, World!")
         return fileID
 
-
 class FollowOn(Job):
-    def __init__(self, fileId):
+    def __init__(self,fileId):
         Job.__init__(self)
-        self.fileId = fileId
+        self.fileId=fileId
 
     def run(self, fileStore):
         tempDir = fileStore.getLocalTempDir()
-        tempFilePath = "/".join([tempDir, "LocalCopy"])
+        tempFilePath = "/".join([tempDir,"LocalCopy"])
         with fileStore.readGlobalFileStream(self.fileId) as globalFile:
             with open(tempFilePath, "w") as localFile:
                 localFile.write(globalFile.read())
diff --git a/src/toil/test/src/jobFileStoreTest.py b/src/toil/test/src/jobFileStoreTest.py
--- a/src/toil/test/src/jobFileStoreTest.py
+++ b/src/toil/test/src/jobFileStoreTest.py
@@ -13,7 +13,7 @@
 # limitations under the License.
 from __future__ import absolute_import
 import random
-
+import os
 from toil.job import Job
 from toil.test import ToilTest
 
@@ -23,81 +23,110 @@ class JobFileStoreTest(ToilTest):
     """
     Tests testing the Job.FileStore class
     """
-    def testJobFileStore(self):
+    def testJobFileStoreWithSmallCache(self, retryCount=0, badWorker=0.0, 
+                         stringNo=1, stringLength=1000000, cacheSize=10000, testNo=2):
         """
         Creates a chain of jobs, each reading and writing files using the 
         Job.FileStore interface. Verifies the files written are always what we expect.
         The chain tests the caching behavior. 
         """
-        for test in xrange(10):
+        for test in xrange(testNo):
             #Make a list of random strings, each of 100k chars and hash the first 200 
             #base prefix to the string
             def randomString():
                 chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-                s = "".join(map(lambda i : random.choice(chars), xrange(100000)))
+                s = "".join(map(lambda i : random.choice(chars), xrange(stringLength)))
                 return s[:PREFIX_LENGTH], s
             #Total length is 2 million characters (20 strings of length 100K each) 
-            testStrings = dict(map(lambda i : randomString(), xrange(20)))
+            testStrings = dict(map(lambda i : randomString(), xrange(stringNo)))
             options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
             options.logLevel = "INFO"
-            options.cacheSize = 1000000
-            options.retryCount=100
-            options.badWorker=0.5
+            options.cacheSize = cacheSize
+            options.retryCount=retryCount
+            options.badWorker=badWorker
             options.badWorkerFailInterval = 1.0
             chainLength = 10
             # Run the workflow, the return value being the number of failed jobs
             Job.Runner.startToil(Job.wrapJobFn(fileTestJob, [], 
                                                testStrings, chainLength), 
                                  options)
+            
+    def testJobFileStoreWithOverLargeCache(self):
+        """
+        Tests case that all files are cached.
+        """
+        self.testJobFileStoreWithSmallCache(retryCount=0, badWorker=0.0, 
+                         stringNo=5, stringLength=1000000, 
+                         cacheSize=10000000)
+        
+    def testJobFileStoreWithMediumCache(self):
+        """
+        Tests case that about half the files are cached
+        """
+        self.testJobFileStoreWithSmallCache(retryCount=0, badWorker=0.0, 
+                         stringNo=5, stringLength=1000000, 
+                         cacheSize=3000000)
+    
+    def testJobFileStoreWithMediumCacheAndBadWorker(self):
+        """
+        Tests case that about half the files are cached and the worker is randomly
+        failing.
+        """
+        self.testJobFileStoreWithSmallCache(retryCount=100, badWorker=0.5, 
+                         stringNo=5, stringLength=1000000, 
+                         cacheSize=3000000)
         
 def fileTestJob(job, inputFileStoreIDs, testStrings, chainLength):
     """
     Test job exercises Job.FileStore functions
     """
+    outputFileStoreIds = [] #Strings passed to the next job in the chain
+    
+    #Load the input jobStoreFileIDs and check that they map to the 
+    #same set of random input strings, exercising the different functions in the fileStore interface
     for fileStoreID in inputFileStoreIDs:
-        #Load the input jobStoreFileIDs and check that they map to the 
-        #same set of random input strings
-        if random.random() > 0.666:
+        if random.random() > 0.5:
             #Read the file for the fileStoreID, randomly picking a way to invoke readGlobalFile
             if random.random() > 0.5:
                 tempFile = job.fileStore.readGlobalFile(fileStoreID, 
+                                                        job.fileStore.getLocalTempFile() if 
+                                                        random.random() > 0.5 else None,
                                                         cache=random.random() > 0.5)
+                with open(tempFile, 'r') as fH:
+                    string = fH.readline()
             else:
-                userPath = job.fileStore.getLocalTempFile()
-                tempFile = job.fileStore.readGlobalFile(fileStoreID, userPath, 
-                                                        cache=random.random() > 0.5)
-                assert userPath == tempFile
-            #Check the local file is as we expect
-            with open(tempFile, 'r') as fH:
-                string = fH.readline()
-                assert testStrings[string[:PREFIX_LENGTH]] == string
-        elif random.random() > 0.666:
-            #Try reading the file using the stream method.
-            with job.fileStore.readGlobalFileStream(fileStoreID) as fH:
-                string = fH.readline()
-                assert testStrings[string[:PREFIX_LENGTH]] == string
+                #Check the local file is as we expect
+                with job.fileStore.readGlobalFileStream(fileStoreID) as fH:
+                    string = fH.readline()
+                    
+            #Check the string we get back is what we expect
+            assert testStrings[string[:PREFIX_LENGTH]] == string
+            
+            #This allows the file to be passed to the next job
+            outputFileStoreIds.append(fileStoreID)
         else:
             #This tests deletion
             job.fileStore.deleteGlobalFile(fileStoreID)
+    
+    #Fill out the output strings until we have the same number as the input strings
+    #exercising different ways of writing files to the file store
+    while len(outputFileStoreIds) < len(testStrings):
+        #Pick a string and write it into a file
+        testString = random.choice(testStrings.values())
+        if random.random() > 0.5:
+            #Make a local copy of the file
+            tempFile = job.fileStore.getLocalTempFile() if random.random() > 0.5 \
+            else os.path.join(job.fileStore.getLocalTempDir(), "temp.txt")
+            with open(tempFile, 'w') as fH:
+                fH.write(testString)
+            #Write a local copy of the file using the local file
+            outputFileStoreIds.append(job.fileStore.writeGlobalFile(tempFile))
+        else:
+            #Use the writeGlobalFileStream method to write the file
+            with job.fileStore.writeGlobalFileStream() as (fH, fileStoreID):
+                fH.write(testString)
+                outputFileStoreIds.append(fileStoreID)
 
     if chainLength > 0:
-        #For each job write some these strings into the file store, collecting 
-        #the fileIDs
-        outputFileStoreIds = []
-        for testPrefix in testStrings:
-            if random.random() > 0.5:
-                #Make a local copy of the file
-                tempFile = job.fileStore.getLocalTempFile()
-                with open(tempFile, 'w') as fH:
-                    fH.write(testStrings[testPrefix])
-                #Write a local copy of the file using the local file
-                outputFileStoreIds.append(job.fileStore.writeGlobalFile(tempFile))
-            else:
-                with job.fileStore.writeGlobalFileStream() as (fH, fileStoreID):
-                    fH.write(testStrings[testPrefix])
-                    outputFileStoreIds.append(fileStoreID)
-    
         #Make a child that will read these files and check it gets the same results
         job.addChildJobFn(fileTestJob, outputFileStoreIds, testStrings, chainLength-1)
-        
-        
\ No newline at end of file
diff --git a/src/toil/test/src/userDefinedJobArgTypeTest.py b/src/toil/test/src/userDefinedJobArgTypeTest.py
--- a/src/toil/test/src/userDefinedJobArgTypeTest.py
+++ b/src/toil/test/src/userDefinedJobArgTypeTest.py
@@ -59,14 +59,14 @@ def _testFromMain(self):
 
 class JobClass(Job):
     def __init__(self, level, foo):
-        Job.__init__(self, memory=100000, cores=2, disk=20000)
+        Job.__init__(self, memory=100000, cores=2, disk="3G")
         self.level = level
         self.foo = foo
 
     def run(self, fileStore):
         self.foo.assertIsCopy()
         if self.level < 2:
-            self.addChildJobFn(jobFunction, self.level + 1, Foo(), cores=1, memory="1M", disk="10M")
+            self.addChildJobFn(jobFunction, self.level + 1, Foo(), cores=1, memory="1M", disk="30G")
 
 
 def jobFunction(job, level, foo):