Repository: DataBiosphere/toil

Issue:
Wrong permissions on file in fileJobStore
```
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/cache-bf91efaf-3d0a-441d-8e55-40a040a201d7/.cacheLock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Released lock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/cache-bf91efaf-3d0a-441d-8e55-40a040a201d7/.cacheLock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Released lock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/cache-bf91efaf-3d0a-441d-8e55-40a040a201d7/.cacheLock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Released lock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/cache-bf91efaf-3d0a-441d-8e55-40a040a201d7/.cacheLock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Cache miss on file with ID 'R/q/tmp4jyd3q.tmp'.
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/cache-bf91efaf-3d0a-441d-8e55-40a040a201d7/.cacheLock
T/O/jobugCtyo:  INFO:toil.job:CACHE: Read file with ID 'R/q/tmp4jyd3q.tmp' from the cache.
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Released lock
T/O/jobugCtyo:  DEBUG:toil.job:CACHE: Released lock
T/O/jobugCtyo:  DEBUG:root:Running the command: cactus_workflow_getFlowers CRITICAL '<st_kv_database_conf type="kyoto_tycoon">
T/O/jobugCtyo:                          <kyoto_tycoon database_dir="/tmp/toil-bf91efaf-3d0a-441d-8e55-40a040a201d7/tmp4egzRz/2087b518-a4bc-4613-9299-5c6b2ac71f11/tL3udCx/cactusDB" database_name="Anc2.kch" host="thinkpad" in_me
mory="1" port="1979" snapshot="0" />
T/O/jobugCtyo:                  </st_kv_database_conf>' 0 100000000 50000000
T/O/jobugCtyo:  Exception in thread Thread-25:
T/O/jobugCtyo:  Traceback (most recent call last):
T/O/jobugCtyo:    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
T/O/jobugCtyo:      self.run()
T/O/jobugCtyo:    File "/usr/lib/python2.7/threading.py", line 754, in run
T/O/jobugCtyo:      self.__target(*self.__args, **self.__kwargs)
T/O/jobugCtyo:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 571, in asyncWrite
T/O/jobugCtyo:      with jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
T/O/jobugCtyo:    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
T/O/jobugCtyo:      return self.gen.next()
T/O/jobugCtyo:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 255, in updateFileStream
T/O/jobugCtyo:      with open(self._getAbsPath(jobStoreFileID), 'w') as f:
T/O/jobugCtyo:  IOError: [Errno 13] Permission denied: '/home/alden/alignments/toil/work/toil/tmp/R/q/tmp4jyd3q.tmp'
```

I tried to make a test case for this outside of cactus but wasn't able to recreate it exactly. It looks like the file is ending up with 444 permissions - I'm not sure if this is ever intended to occur or not. This appears to happen with graphs like this:

A --> C 
| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |
B &nbsp;&nbsp;&nbsp;&nbsp; D

where A writes a file to the file store, B downloads it with mutable=True, then modifies it and writes it back to the file store with cleanup=False, and D downloads it. The test case I made with the above graph doesn't trigger the asyncWrite error, but the file store ID does have 444 permissions in job D.



Discussion:
Files read from the file store will always have 444 unless you mention `mutable=True`  in  the readGlobalFile call

Yes, but what's going on here is that the file in the filestore itself has 444 permissions, so asyncWrite isn't allowed to write to it.

Are you putting it back in the jobstore under the same jobstore ID?

@hannes-ucsc didn't we decide that you can't overwrite the file in the job store? If you download a file and modify it, you need to upload it as a new jobstore ID

I don't believe it's possible anymore to modify a JobStore ID through the FileStore interface. So I'm definitely writing to a new ID after modfiying the file. But I do modify the file that was read into the local temp dir (with mutable=True) and then write that same local file back to the file store. Those are the only set of conditions that cause this. Somehow when the ID is created it ends up with 444 permissions and then asyncWrite doesn't work. I've looked through all the chmod calls and none of them directly change the permissions of jobStore IDs, but it's possible that when the cached file is symlinked to the jobStore ID, chmod will traverse the symlink: http://superuser.com/questions/303040/how-do-file-permissions-apply-to-symlinks

I have a feeling that the problem is here: https://github.com/BD2KGenomics/toil/blob/master/src/toil/job.py#L1591

Are you in 514? Can we discuss this in person?

Just had a chat with @adderan. I think this async write may not be working how we except it to in this new caching and a race condition is causing the problem. thought of a quick test that could confirm my suspicion and he's working on recreating it.

<b>j/A/job7tFSbV:  INFO:root:Starting job A: 1465853608.006610</b>                                                                                                                                                                                                           [390/1522]
j/A/job7tFSbV:  INFO:root:Make wrapper jobs: There are 1 flowers
j/A/job7tFSbV:  INFO:root:Adding recursive flower job
<b>j/A/job7tFSbV:  INFO:root:Ending job A: 1465853608.006953</b>
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  INFO:toil.job:LOG-TO-MASTER: Added file with ID 'W/5/tmp9XReyE.tmp' to the list of files to be globally deleted.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
j/A/job7tFSbV:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
j/A/job7tFSbV:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
j/A/job7tFSbV:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
j/A/job7tFSbV:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
j/A/job7tFSbV:  DEBUG:toil.job:Getting defaultdict from module collections.
j/A/job7tFSbV:  DEBUG:toil.job:Getting list from module **builtin**.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
j/A/job7tFSbV:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
j/A/job7tFSbV:  DEBUG:toil.job:Getting set from module **builtin**.
j/A/job7tFSbV:  Starting the next jobWrapper
j/A/job7tFSbV:  DEBUG:toil.worker:Starting the next jobWrapper
j/A/job7tFSbV:  Got a command to run: _toil j/A/job7tFSbV/g/tmpJzzCYj.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
j/A/job7tFSbV:  DEBUG:toil.worker:Got a command to run: _toil j/A/job7tFSbV/g/tmpJzzCYj.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
j/A/job7tFSbV:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
j/A/job7tFSbV:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
j/A/job7tFSbV:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
j/A/job7tFSbV:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
j/A/job7tFSbV:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
j/A/job7tFSbV:  DEBUG:toil.job:Getting defaultdict from module collections.
j/A/job7tFSbV:  DEBUG:toil.job:Getting list from module __builtin__.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
j/A/job7tFSbV:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
j/A/job7tFSbV:  DEBUG:toil.job:Getting set from module **builtin**.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
<b>j/A/job7tFSbV:  INFO:root:Starting job B: 1465853608.015252</b>
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  INFO:toil.job:CACHE: Cache hit on file with ID 'c/V/tmpuhhMZk.tmp'.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
<b>j/A/job7tFSbV:  INFO:root:Ending job B: 1465853608.711008</b>
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  INFO:toil.job:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'c/V/tmpuhhMZk.tmp'.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  INFO:toil.job:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'z/b/tmpQPXzDG.tmp'.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
j/A/job7tFSbV:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
j/A/job7tFSbV:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
j/A/job7tFSbV:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
j/A/job7tFSbV:  DEBUG:toil.job:Getting CactusReferenceRecursion2 from module cactus.pipeline.cactus_workflow.
j/A/job7tFSbV:  DEBUG:toil.job:Getting defaultdict from module collections.
j/A/job7tFSbV:  DEBUG:toil.job:Getting list from module **builtin**.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Promise from module toil.job.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
j/A/job7tFSbV:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
j/A/job7tFSbV:  DEBUG:toil.job:Getting set from module **builtin**.
j/A/job7tFSbV:  Starting the next jobWrapper
j/A/job7tFSbV:  DEBUG:toil.worker:Starting the next jobWrapper
j/A/job7tFSbV:  Got a command to run: _toil j/A/job7tFSbV/g/tmpJrUBjj.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
j/A/job7tFSbV:  DEBUG:toil.worker:Got a command to run: _toil j/A/job7tFSbV/g/tmpJrUBjj.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
j/A/job7tFSbV:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
j/A/job7tFSbV:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
j/A/job7tFSbV:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
j/A/job7tFSbV:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
j/A/job7tFSbV:  DEBUG:toil.job:Getting CactusReferenceRecursion2 from module cactus.pipeline.cactus_workflow.
j/A/job7tFSbV:  DEBUG:toil.job:Getting defaultdict from module collections.
j/A/job7tFSbV:  DEBUG:toil.job:Getting list from module __builtin__.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Promise from module toil.job.
j/A/job7tFSbV:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
j/A/job7tFSbV:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
j/A/job7tFSbV:  DEBUG:toil.job:Getting set from module **builtin**.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Cache miss on file with ID 'z/b/tmpQPXzDG.tmp'.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  INFO:toil.job:CACHE: Read file with ID 'z/b/tmpQPXzDG.tmp' from the cache.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
<b>j/A/job7tFSbV:  INFO:root:Starting job C: 1465853608.718821</b>
j/A/job7tFSbV:  INFO:root:Flower names: 1 0
j/A/job7tFSbV:  DEBUG:root:Running the command: cactus_workflow_getFlowers CRITICAL '<st_kv_database_conf type="kyoto_tycoon">
j/A/job7tFSbV:                          <kyoto_tycoon database_dir="/tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/tmpWExDal/72093bca-a830-4b56-b6f1-a51eb2c86bf6/tNoYkhM/cactusDB" database_name="Anc2.kch" host="thinkpad" in_memory="1" port="1979" snapshot="0" />
j/A/job7tFSbV:                  </st_kv_database_conf>' 0 100000000 50000000
<b>j/A/job7tFSbV:  INFO:toil.job:Trying to asyncWrite to jobStore ID z/b/tmpQPXzDG.tmp at time 1465853608.728162</b>
j/A/job7tFSbV:  Exception in thread Thread-7:
j/A/job7tFSbV:  Traceback (most recent call last):
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 801, in **bootstrap_inner
j/A/job7tFSbV:      self.run()
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 754, in run
j/A/job7tFSbV:      self.__target(_self.__args, *_self.__kwargs)
j/A/job7tFSbV:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 573, in asyncWrite
j/A/job7tFSbV:      with jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
j/A/job7tFSbV:    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter**
j/A/job7tFSbV:      return self.gen.next()
j/A/job7tFSbV:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 255, in updateFileStream
j/A/job7tFSbV:      with open(self._getAbsPath(jobStoreFileID), 'w') as f:
j/A/job7tFSbV:  IOError: [Errno 13] Permission denied: '/home/alden/alignments/toil/work/toil/tmp/z/b/tmpQPXzDG.tmp'
j/A/job7tFSbV:
j/A/job7tFSbV:  Exception in thread Thread-12:
j/A/job7tFSbV:  Traceback (most recent call last):
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
j/A/job7tFSbV:      self.run()
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 754, in run
j/A/job7tFSbV:      self.__target(_self.__args, *_self.__kwargs)
j/A/job7tFSbV:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 868, in asyncUpdate
j/A/job7tFSbV:      raise RuntimeError("The termination flag is set, exiting before update")
j/A/job7tFSbV:  RuntimeError: The termination flag is set, exiting before update
j/A/job7tFSbV:  Exception in thread Thread-9:
j/A/job7tFSbV:  Traceback (most recent call last):
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
j/A/job7tFSbV:      self.run()
j/A/job7tFSbV:    File "/usr/lib/python2.7/threading.py", line 754, in run
j/A/job7tFSbV:      self.__target(_self.__args, *_self.__kwargs)
j/A/job7tFSbV:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 868, in asyncUpdate
j/A/job7tFSbV:      raise RuntimeError("The termination flag is set, exiting before update")
j/A/job7tFSbV:  RuntimeError: The termination flag is set, exiting before update
j/A/job7tFSbV:
j/A/job7tFSbV:
j/A/job7tFSbV:  INFO:root:Make wrapper jobs: There are 1 flowers
j/A/job7tFSbV:  INFO:root:Adding recursive flower job
<b>j/A/job7tFSbV:  INFO:root:Ending job C: 1465853608.735363</b>
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock
j/A/job7tFSbV:  INFO:toil.job:LOG-TO-MASTER: Added file with ID '0/f/tmpM2kXX8.tmp' to the list of files to be globally deleted.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-d9d9bff3-83a4-464d-9704-d41ccd357a8f/cache-d9d9bff3-83a4-464d-9704-d41ccd357a8f/.cacheLock
j/A/job7tFSbV:  INFO:toil.job:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'z/b/tmpQPXzDG.tmp'.
j/A/job7tFSbV:  DEBUG:toil.job:CACHE: Released lock

Well that's dumb. writeFileStream should write to a temp file and rename when done. The rename() sys call is not controlled by the mode bits on the inode of the file, but rather by the mode bits on the containing directory's inode which should allow writing. Right?

If I am right, the person who fixes this should scan FileJobStore for other occurrences of that faux pas.

Pinging @diekhans for giggles.

doesn't explain why the file got changed to mode 444 though, right?

Ok @adderan we brainstormed and we think we might have figured it out. Can we bother you to run 2 more tests for us to confirm our suspicions?
1. have C use more cpu or memory than A/B. This will force C into a new job wrapper and we shouldn't see this issue.
2. Have C download the file as mutable. The error in that case should be cactus realizing the file is not completely written.

Thanks
-A

Okay, here are the results of test 1. I'm setting jobs A and B to 500 MB and job C to 1GB. It looks like asycWrite happens before job C in this case. I tried running a few times and got the same results. 

```
G/y/jobsKXr6V:  INFO:root:Starting job A: 1465933594.958452
G/y/jobsKXr6V:  INFO:root:Job A memory: 500000000
G/y/jobsKXr6V:  INFO:root:Make wrapper jobs: There are 1 flowers
G/y/jobsKXr6V:  INFO:root:Adding recursive flower job
G/y/jobsKXr6V:  INFO:root:Ending job A: 1465933594.959053
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-2ccf3282-0705-41db-8c03-650fc3a6d5a9/cache-2ccf3282-0705-41db-8c03-650fc3a6d5a9/.cacheLock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Released lock
G/y/jobsKXr6V:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
G/y/jobsKXr6V:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
G/y/jobsKXr6V:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
G/y/jobsKXr6V:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
G/y/jobsKXr6V:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting defaultdict from module collections.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting list from module __builtin__.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting set from module __builtin__.
G/y/jobsKXr6V:  Starting the next jobWrapper
G/y/jobsKXr6V:  DEBUG:toil.worker:Starting the next jobWrapper
G/y/jobsKXr6V:  Got a command to run: _toil G/y/jobsKXr6V/g/tmphATaK_.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
G/y/jobsKXr6V:  DEBUG:toil.worker:Got a command to run: _toil G/y/jobsKXr6V/g/tmphATaK_.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
G/y/jobsKXr6V:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
G/y/jobsKXr6V:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
G/y/jobsKXr6V:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
G/y/jobsKXr6V:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
G/y/jobsKXr6V:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting defaultdict from module collections.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting list from module __builtin__.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
G/y/jobsKXr6V:  DEBUG:toil.job:Getting set from module __builtin__.
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-2ccf3282-0705-41db-8c03-650fc3a6d5a9/cache-2ccf3282-0705-41db-8c03-650fc3a6d5a9/.cacheLock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Released lock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-2ccf3282-0705-41db-8c03-650fc3a6d5a9/cache-2ccf3282-0705-41db-8c03-650fc3a6d5a9/.cacheLock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Released lock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-2ccf3282-0705-41db-8c03-650fc3a6d5a9/cache-2ccf3282-0705-41db-8c03-650fc3a6d5a9/.cacheLock
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Released lock
G/y/jobsKXr6V:  INFO:root:Starting job B: 1465933594.967740
G/y/jobsKXr6V:  INFO:root:Job B memory: 500000000
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-2ccf3282-0705-41db-8c03-650fc3a6d5a9/cache-2ccf3282-0705-41db-8c03-650fc3a6d5a9/.cacheLock
G/y/jobsKXr6V:  INFO:toil.job:CACHE: Cache hit on file with ID 'I/0/tmpMbgKQm.tmp'.
G/y/jobsKXr6V:  DEBUG:toil.job:CACHE: Released lock
G/y/jobsKXr6V:  INFO:toil.job:Trying to asyncWrite to jobStore ID I/0/tmpMbgKQm.tmp at time 1465933594.971697
G/y/jobsKXr6V:  Exception in thread Thread-7:
G/y/jobsKXr6V:  Traceback (most recent call last):
G/y/jobsKXr6V:    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
G/y/jobsKXr6V:      self.run()
G/y/jobsKXr6V:    File "/usr/lib/python2.7/threading.py", line 754, in run
G/y/jobsKXr6V:      self.__target(*self.__args, **self.__kwargs)
G/y/jobsKXr6V:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 573, in asyncWrite
G/y/jobsKXr6V:      with jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
G/y/jobsKXr6V:    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
G/y/jobsKXr6V:      return self.gen.next()
G/y/jobsKXr6V:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 255, in updateFileStream
G/y/jobsKXr6V:      with open(self._getAbsPath(jobStoreFileID), 'w') as f:
G/y/jobsKXr6V:  IOError: [Errno 13] Permission denied: '/home/alden/alignments/toil/work/toil/tmp/I/0/tmpMbgKQm.tmp'
```

With mutable=True in job C, Cactus does have trouble parsing the file, but in a follow-on of job C, not job C itself.

```
C/o/jobE6vLfJ:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
C/o/jobE6vLfJ:  DEBUG:toil.job:Getting set from module __builtin__.
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Cache miss on file with ID 'E/T/tmpzymUw7.tmp'.
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  INFO:toil.job:CACHE: Added file with ID 'E/T/tmpzymUw7.tmp' to the cache.
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  INFO:toil.job:Trying to asyncWrite to jobStore ID E/T/tmpzymUw7.tmp at time 1465945035.438969
C/o/jobE6vLfJ:  Read only 0 bytes of string of length 936 when caching substrings from DB
C/o/jobE6vLfJ:
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  INFO:toil.job:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'E/T/tmpzymUw7.tmp'.
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/cache-3bfd5192-3dd8-4281-baaa-38153bd035fc/.cacheLock
C/o/jobE6vLfJ:  DEBUG:toil.job:CACHE: Released lock
C/o/jobE6vLfJ:  Traceback (most recent call last):
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/worker.py", line 339, in main
C/o/jobE6vLfJ:      fileStore=fileStore)
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 2454, in _execute
C/o/jobE6vLfJ:      returnValues = self._run(jobWrapper, fileStore)
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 2443, in _run
C/o/jobE6vLfJ:      return self.run(fileStore)
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/submodules/cactus/pipeline/cactus_workflow.py", line 970, in run
C/o/jobE6vLfJ:      bottomUpPhase=True)
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/submodules/cactus/shared/common.py", line 384, in runCactusAddReferenceCoordinates
C/o/jobE6vLfJ:      popenPush(command, stdinString=flowerNames)
C/o/jobE6vLfJ:    File "/home/alden/progressiveCactus-new/submodules/sonLib/bioio.py", line 224, in popenPush
C/o/jobE6vLfJ:      raise RuntimeError("Command: %s with stdin string '%s' exited with non-zero status %i" % (command, stdinString, sts))
C/o/jobE6vLfJ:  RuntimeError: Command: cactus_addReferenceCoordinates --cactusDisk '<st_kv_database_conf type="kyoto_tycoon">
C/o/jobE6vLfJ:                          <kyoto_tycoon database_dir="/tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/tmpMDPaZd/001bb23b-2c0e-4f9f-a134-677686bc5bf0/tpE2vqB/cactusDB" database_name="Anc2.kch" host="thinkpad" in_memory="1" port="1979" snapshot="0" />
C/o/jobE6vLfJ:                  </st_kv_database_conf>' --secondaryDisk '<st_kv_database_conf type="kyoto_tycoon">
C/o/jobE6vLfJ:                          <kyoto_tycoon database_dir="/tmp/toil-3bfd5192-3dd8-4281-baaa-38153bd035fc/tmpkLSEgg/d30b3ef1-5ca3-4400-a677-56121cf057c9/tvE3aU8/tempDB" database_name="Anc2.kch" host="thinkpad" in_memory="1" port="1980" snapshot="0" />
C/o/jobE6vLfJ:                  </st_kv_database_conf>
C/o/jobE6vLfJ:          ' --logLevel CRITICAL --referenceEventString Anc2  --bottomUpPhase with stdin string '9549 40250921669623609 13 321 13 13 13 13 13 13 13 13 4 386 13 13 4 555 19 4 3259
```

Thanks a bunch @adderan. I think this confirms our suspicions. @hannes-ucsc I'll go ahead and implement the harbinger logic into writeGlobalFile.

@adderan can you clone this branch and tell me if this fixed things?

This is one of two critical bugs that block the 3.2.0 release of Toil which is scheduled to be released next Wed. 

Unfortunately this does not fix it. 

```
A/R/job_CKRwi:  INFO:root:DatabaseID in RefRecursion: W/A/tmpoDfcNo.tmp
A/R/job_CKRwi:  INFO:root:Make wrapper jobs: There are 1 flowers
A/R/job_CKRwi:  INFO:root:Adding recursive flower job
A/R/job_CKRwi:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-ebfd917e-d082-49e2-bd08-df67d1b4f542/cache-ebfd917e-d082-49e2-bd08-df67d1b4f542/.cacheLock
A/R/job_CKRwi:  DEBUG:toil.job:CACHE: Released lock
A/R/job_CKRwi:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
A/R/job_CKRwi:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
A/R/job_CKRwi:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
A/R/job_CKRwi:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
A/R/job_CKRwi:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
A/R/job_CKRwi:  DEBUG:toil.job:Getting defaultdict from module collections.
A/R/job_CKRwi:  DEBUG:toil.job:Getting list from module __builtin__.
A/R/job_CKRwi:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
A/R/job_CKRwi:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
A/R/job_CKRwi:  DEBUG:toil.job:Getting set from module __builtin__.
A/R/job_CKRwi:  Starting the next jobWrapper
A/R/job_CKRwi:  DEBUG:toil.worker:Starting the next jobWrapper
A/R/job_CKRwi:  Got a command to run: _toil A/R/job_CKRwi/g/tmpL_3L95.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
A/R/job_CKRwi:  DEBUG:toil.worker:Got a command to run: _toil A/R/job_CKRwi/g/tmpL_3L95.tmp /home/alden/progressiveCactus-new/submodules cactus.pipeline.cactus_workflow
A/R/job_CKRwi:  DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow').
A/R/job_CKRwi:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
A/R/job_CKRwi:  WARNING:toil.resource:Can't find resource for leader path '/home/alden/progressiveCactus-new/submodules/cactus'
A/R/job_CKRwi:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/alden/progressiveCactus-new/submodules', name='cactus.pipeline.cactus_workflow')
A/R/job_CKRwi:  DEBUG:toil.job:Getting CactusReferenceWrapper from module cactus.pipeline.cactus_workflow.
A/R/job_CKRwi:  DEBUG:toil.job:Getting defaultdict from module collections.
A/R/job_CKRwi:  DEBUG:toil.job:Getting list from module __builtin__.
A/R/job_CKRwi:  DEBUG:toil.job:Getting Element from module xml.etree.ElementTree.
A/R/job_CKRwi:  DEBUG:toil.job:Getting ModuleDescriptor from module toil.resource.
A/R/job_CKRwi:  DEBUG:toil.job:Getting set from module __builtin__.
A/R/job_CKRwi:  DEBUG:toil.job:CACHE: Obtained lock on file /tmp/toil-ebfd917e-d082-49e2-bd08-df67d1b4f542/cache-ebfd917e-d082-49e2-bd08-df67d1b4f542/.cacheLock
A/R/job_CKRwi:  Exception in thread Thread-7:
A/R/job_CKRwi:  Traceback (most recent call last):
A/R/job_CKRwi:    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
A/R/job_CKRwi:      self.run()
A/R/job_CKRwi:    File "/usr/lib/python2.7/threading.py", line 754, in run
A/R/job_CKRwi:      self.__target(*self.__args, **self.__kwargs)
A/R/job_CKRwi:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/job.py", line 1031, in asyncWrite
A/R/job_CKRwi:      with self.jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
A/R/job_CKRwi:    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
A/R/job_CKRwi:      return self.gen.next()
A/R/job_CKRwi:    File "/home/alden/progressiveCactus-new/python/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 255, in updateFileStream
A/R/job_CKRwi:      with open(self._getAbsPath(jobStoreFileID), 'w') as f:
A/R/job_CKRwi:  IOError: [Errno 13] Permission denied: '/home/alden/alignments/toil/work/toil/tmp/W/A/tmpoDfcNo.tmp'
A/R/job_CKRwi:
```

looking into it

@adderan, can you enable @arkal to reproduce this on his own? With the release being just a few days away, we can't afford 24h turn-around time on verification.

I'm working on a unit test that can replicate this

Sorry, I will try to respond more quickly next time, I missed this email before. But just in case, to run Cactus you would:

1) git clone https://github.com/adderan/progressiveCactus
2) git checkout toil
3) git submodule update --init
4) make (you might have to fix your python installation if you're on Ubuntu, using the command listed here: http://stackoverflow.com/questions/15608236/eclipse-and-google-app-engine-importerror-no-module-named-sysconfigdata-nd-u)

You could also use my AWS template for Cactus: https://github.com/adderan/cactusCluster

Once Cactus is installed, it can be run with:

bin/runProgressiveCactus.sh examples/evolverMammalsLoci1.txt <workdir> <OutputAlignmentFile.hal> --logLevel DEBUG --defaultMemory 10000


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -548,47 +548,43 @@ def __init__(self, jobStore, jobWrapper, localTempDir, inputBlockFn):
             self.queue = Queue()
             self.updateSemaphore = Semaphore()
             self.mutable = self.jobStore.config.readGlobalFileMutableByDefault
-            #Function to write files asynchronously to job store
-            def asyncWrite():
-                try:
-                    while True:
-                        try:
-                            #Block for up to two seconds waiting for a file
-                            args = self.queue.get(timeout=2)
-                        except Empty:
-                            #Check if termination event is signaled
-                            #(set in the event of an exception in the worker)
-                            if self._terminateEvent.isSet():
-                                raise RuntimeError("The termination flag is set, exiting")
-                            continue
-                        #Normal termination condition is getting None from queue
-                        if args is None:
-                            break
-                        inputFileHandle, jobStoreFileID = args
-                        #We pass in a fileHandle, rather than the file-name, in case
-                        #the file itself is deleted. The fileHandle itself should persist
-                        #while we maintain the open file handle
-                        with jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
-                            bufferSize=1000000 #TODO: This buffer number probably needs to be modified/tuned
-                            while 1:
-                                copyBuffer = inputFileHandle.read(bufferSize)
-                                if not copyBuffer:
-                                    break
-                                outputFileHandle.write(copyBuffer)
-                        inputFileHandle.close()
-                        #Remove the file from the lock files
-                        with self._pendingFileWritesLock:
-                            self._pendingFileWrites.remove(jobStoreFileID)
-                except:
-                    self._terminateEvent.set()
-                    raise
-
-            self.workers = map(lambda i : Thread(target=asyncWrite),
+            self.workers = map(lambda i : Thread(target=self.asyncWrite),
                                range(self.workerNumber))
             for worker in self.workers:
                 worker.start()
             self.inputBlockFn = inputBlockFn
 
+        #Function to write files asynchronously to job store
+        def asyncWrite(self):
+            try:
+                while True:
+                    try:
+                        # Block for up to two seconds waiting for a file
+                        args = self.queue.get(timeout=2)
+                    except Empty:
+                        # Check if termination event is signaled
+                        # (set in the event of an exception in the worker)
+                        if self._terminateEvent.isSet():
+                            raise RuntimeError("The termination flag is set, exiting")
+                        continue
+                    # Normal termination condition is getting None from queue
+                    if args is None:
+                        break
+                    inputFileHandle, jobStoreFileID = args
+                    # We pass in a fileHandle, rather than the file-name, in case
+                    # the file itself is deleted. The fileHandle itself should persist
+                    # while we maintain the open file handle
+                    with self.jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
+                        shutil.copyfileobj(inputFileHandle, outputFileHandle)
+                    inputFileHandle.close()
+                    # Remove the file from the lock files
+                    with self._pendingFileWritesLock:
+                        self._pendingFileWrites.remove(jobStoreFileID)
+            except:
+                self._terminateEvent.set()
+                raise
+
+
         @contextmanager
         def open(self, job):
             '''
@@ -998,6 +994,49 @@ def __init__(self, jobStore, jobWrapper, localTempDir, inputBlockFn):
 
             self._setupCache()
 
+        # Cache-friendly version of asyncWrite.  This essentially duplicates FileStore.asyncWrite
+        # and adds in a few lines with caching logic.  We deemed it not worth the effort of
+        # attempting to make a template method that can be modified in CachedFileStore since the
+        # parent method will be phased out very soon.
+        def asyncWrite(self):
+            try:
+                while True:
+                    try:
+                        # Block for up to two seconds waiting for a file
+                        args = self.queue.get(timeout=2)
+                    except Empty:
+                        # Check if termination event is signaled
+                        # (set in the event of an exception in the worker)
+                        if self._terminateEvent.isSet():
+                            raise RuntimeError("The termination flag is set, exiting")
+                        continue
+                    # Normal termination condition is getting None from queue
+                    if args is None:
+                        break
+                    inputFileHandle, jobStoreFileID = args
+                    cachedFileName = self.encodedFileID(jobStoreFileID)
+                    # Ensure that the harbinger exists in the cache directory and that the PID
+                    # matches that of this writing thread.
+                    # If asyncWrite is ported to subprocesses instead of threads in the future,
+                    # insert logic here to securely overwrite the harbinger file.
+                    harbingerFile = self.HarbingerFile(self, cachedFileName=cachedFileName)
+                    assert harbingerFile.exists()
+                    assert harbingerFile.read() == int(os.getpid())
+                    # We pass in a fileHandle, rather than the file-name, in case
+                    # the file itself is deleted. The fileHandle itself should persist
+                    # while we maintain the open file handle
+                    with self.jobStore.updateFileStream(jobStoreFileID) as outputFileHandle:
+                        shutil.copyfileobj(inputFileHandle, outputFileHandle)
+                    inputFileHandle.close()
+                    # Remove the file from the lock files
+                    with self._pendingFileWritesLock:
+                        self._pendingFileWrites.remove(jobStoreFileID)
+                    # Remove the harbinger file
+                    harbingerFile.delete()
+            except:
+                self._terminateEvent.set()
+                raise
+
         @contextmanager
         def open(self,job):
             '''
@@ -1074,6 +1113,15 @@ def writeGlobalFile(self, localFileName, cleanup=False):
                 # Check if the user allows asynchronous file writes
                 elif self.jobStore.config.useAsync:
                     jobStoreFileID = self.jobStore.getEmptyFileStoreID(cleanupID)
+                    # Before we can start the async process, we should also create a dummy harbinger
+                    # file in the cache such that any subsequent jobs asking for this file will not
+                    # attempt to download it from the job store till the write is complete.  We do
+                    # this now instead of in the writing thread because there is an edge case where
+                    # readGlobalFile in a subsequent job is called before the writing thread has
+                    # received the message to write the file and has created the dummy harbinger
+                    # (and the file was unable to be cached/was evicted from the cache).
+                    harbingerFile = self.HarbingerFile(self, fileStoreID=jobStoreFileID)
+                    harbingerFile.write()
                     fileHandle = open(absLocalFileName, 'r')
                     with self._pendingFileWritesLock:
                         self._pendingFileWrites.add(jobStoreFileID)
@@ -1142,7 +1190,7 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
             # currently being downloaded by another job and will be in the cache shortly. It is used
             # to prevent multiple jobs from simultaneously downloading the same file from the file
             # store.
-            harbingerFileName = ''.join(['/.'.join(os.path.split(cachedFileName)), '.harbinger'])
+            harbingerFile = self.HarbingerFile(self, cachedFileName=cachedFileName)
             # setup the output filename.  If a name is provided, use it - This makes it a Named
             # Local File. If a name isn't provided, use the base64 encoded name such that we can
             # easily identify the files later on.
@@ -1176,25 +1224,9 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
                 # FileStoreID exists.  If it does, the wait and periodically check for the removal
                 # of the file and the addition of the completed download into cache of the file by
                 # the other job. Then we link to it.
-                elif fileIsLocal and os.path.exists(harbingerFileName):
-                    logger.info('CACHE: Waiting for another worker to download file with ID %s.'
-                                % fileStoreID)
-                    while os.path.exists(harbingerFileName):
-                        # Ensure that the process downloading the file is still alive.  The PID will
-                        # be in the harbinger file.
-                        pid = int(open(harbingerFileName).read())
-                        if self._pidExists(pid):
-                            # Release the file lock and then wait for a bit before repeating.
-                            flock(lockFileHandle, LOCK_UN)
-                            time.sleep(20)
-                            # Grab the file lock before repeating.
-                            flock(lockFileHandle, LOCK_EX)
-                        else:
-                            # The process that was supposed to download the file has died so we need
-                            # to remove the harbinger.
-                            os.remove(harbingerFileName)
-
-                    # If the code reaches here, the partial lock file has been removed. This means
+                elif fileIsLocal and harbingerFile.exists():
+                    harbingerFile.waitOnDownload(lockFileHandle)
+                    # If the code reaches here, the harbinger file has been removed. This means
                     # either the file was successfully downloaded and added to cache, or something
                     # failed. To prevent code duplication, we recursively call readGlobalFile.
                     flock(lockFileHandle, LOCK_UN)
@@ -1204,13 +1236,11 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
                 else:
                     logger.debug('CACHE: Cache miss on file with ID \'%s\'.' % fileStoreID)
                     if fileIsLocal and cache:
-                        # If caching of the downloaded file is desired, First create the .partial
+                        # If caching of the downloaded file is desired, First create the harbinger
                         # file so other jobs know not to redundantly download the same file.  Write
                         # the PID of this process into the file so other jobs know who is carrying
                         # out the download.
-                        with open(harbingerFileName + '.tmp', 'w') as harbingerFile:
-                            harbingerFile.write(str(os.getpid()))
-                        os.rename(harbingerFileName + '.tmp', harbingerFileName)
+                        harbingerFile.write()
                         # Now release the file lock while the file is downloaded as download could
                         # take a while.
                         flock(lockFileHandle, LOCK_UN)
@@ -1234,7 +1264,7 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
                         finally:
                             # Reacquire the file lock and delete the partial file.
                             flock(lockFileHandle, LOCK_EX)
-                            os.remove(harbingerFileName)
+                            harbingerFile.delete()
                     else:
                         # Release the cache lock since the remaining stuff is not cache related.
                         flock(lockFileHandle, LOCK_UN)
@@ -1930,28 +1960,93 @@ def updateJobReqs(self, fileSize, actions):
             def isPopulated(self):
                 return self.__dict__ != {}
 
-        @staticmethod
-        def _pidExists(pid):
+        class HarbingerFile(object):
             """
-            This will return True if the process associated with pid is still running on the
-            machine.
-            This is based on stackoverflow question 568271.
-
-            :param int pid: ID of the process to check for
-            :return: True/False
-            :rtype: bool
+            Represents the placeholder file that harbinges the arrival of a local copy of a file in
+            the job store.
             """
-            assert pid > 0
-            try:
-                os.kill(pid, 0)
-            except OSError as err:
-                if err.errno == errno.ESRCH:
-                    # ESRCH == No such process
-                    return False
+            def __init__(self, fileStore, fileStoreID=None, cachedFileName=None):
+                """
+                Returns the harbinger file name for a cached file, or for a job store ID
+
+                :param class fileStore: The 'self' object of the fileStore class
+                :param str fileStoreID: The file store ID for an input file
+                :param str cachedFileName: The cache file name corresponding to a given file
+                :return: Harbinger file name
+                :rtype: str
+                """
+                # We need either a file store ID, or a cached file name, but not both (XOR).
+                assert (fileStoreID is None) != (cachedFileName is None)
+                if fileStoreID is not None:
+                    self.fileStoreID = fileStoreID
+                    cachedFileName = fileStore.encodedFileID(fileStoreID)
                 else:
-                    raise
-            else:
-                return True
+                    self.fileStoreID = fileStore.decodedFileID(cachedFileName)
+                self.harbingerFileName = '/.'.join(os.path.split(cachedFileName)) + '.harbinger'
+
+            def write(self):
+                with open(self.harbingerFileName + '.tmp', 'w') as harbingerFile:
+                    harbingerFile.write(str(os.getpid()))
+                # Make this File read only to prevent overwrites
+                os.chmod(self.harbingerFileName + '.tmp', 0444)
+                os.rename(self.harbingerFileName + '.tmp', self.harbingerFileName)
+
+            def waitOnDownload(self, lockFileHandle):
+                """
+                This method is called when a readGlobalFile process is waiting on another process to
+                write a file to the cache.
+
+                :param lockFileHandle: The open handle to the cache lock file
+                :return: None
+                """
+                while self.exists():
+                    logger.info('CACHE: Waiting for another worker to download file with ID %s.'
+                                % self.fileStoreID)
+                    # Ensure that the process downloading the file is still alive.  The PID will
+                    # be in the harbinger file.
+                    pid = self.read()
+                    if self._pidExists(pid):
+                        # Release the file lock and then wait for a bit before repeating.
+                        flock(lockFileHandle, LOCK_UN)
+                        time.sleep(20)
+                        # Grab the file lock before repeating.
+                        flock(lockFileHandle, LOCK_EX)
+                    else:
+                        # The process that was supposed to download the file has died so we need
+                        # to remove the harbinger.
+                        harbingerFile.delete()
+
+            def read(self):
+                return int(open(self.harbingerFileName).read())
+
+            def exists(self):
+                return os.path.exists(self.harbingerFileName)
+
+            def delete(self):
+                os.remove(self.harbingerFileName)
+
+            @staticmethod
+            def _pidExists(pid):
+                """
+                This will return True if the process associated with pid is still running on the
+                machine.
+                This is based on stackoverflow question 568271.
+
+                :param int pid: ID of the process to check for
+                :return: True/False
+                :rtype: bool
+                """
+                assert pid > 0
+                try:
+                    os.kill(pid, 0)
+                except OSError as err:
+                    if err.errno == errno.ESRCH:
+                        # ESRCH == No such process
+                        return False
+                    else:
+                        raise
+                else:
+                    return True
 
     class Service:
         """


Test patch:
diff --git a/src/toil/test/src/jobCacheEjectionTest.py b/src/toil/test/src/jobCacheEjectionTest.py
--- a/src/toil/test/src/jobCacheEjectionTest.py
+++ b/src/toil/test/src/jobCacheEjectionTest.py
@@ -39,13 +39,13 @@ def testCacheEjection(self):
             options.retryCount=100
             options.badWorker=0.5
             options.badWorkerFailInterval = 1.0
-            # Create a temp file to write teh test results
+            # Create a temp file to write the test results
             handle, logfile = tempfile.mkstemp(dir=test_dir)
             os.close(handle)
             file_sizes = [50000, 40000, 30000]
             # Randomize to (potentially) test all combinations
             random.shuffle(file_sizes)
-            # Run the workflow. A, B and C do teh cache operations, and D prints test status to tempFile
+            # Run the workflow. A, B and C do the cache operations, and D prints test status to tempFile
             A = Job.wrapJobFn(fileTestJob, file_sizes[0])
             B = Job.wrapJobFn(fileTestJob, file_sizes[0])
             C = Job.wrapJobFn(fileTestJob, file_sizes[0])
diff --git a/src/toil/test/src/jobCacheTest.py b/src/toil/test/src/jobCacheTest.py
--- a/src/toil/test/src/jobCacheTest.py
+++ b/src/toil/test/src/jobCacheTest.py
@@ -233,7 +233,7 @@ def _testCacheEviction(self, file1MB, file2MB, diskRequestMB):
                 expectedResult = 50 - file1MB if diskRequestMB <= file1MB else 0
             try:
                 A = Job.wrapJobFn(self._writeFileToJobStore, isLocalFile=True, fileMB=file1MB)
-                # Sleep for 1 second after writing the first fiel so that their ctimes are
+                # Sleep for 1 second after writing the first file so that their ctimes are
                 # guaranteed to be distinct for the purpose of this test.
                 B = Job.wrapJobFn(self._sleepy, timeToSleep=1)
                 C = Job.wrapJobFn(self._writeFileToJobStore, isLocalFile=True, fileMB=file2MB)
@@ -253,8 +253,8 @@ def _testCacheEviction(self, file1MB, file2MB, diskRequestMB):
             except FailedJobsException as err:
                 self.assertEqual(err.numberOfFailedJobs, 1)
                 errType, errMsg = self._parseAssertionError(self.options.logFile)
-                if (errType == 'AssertionError' 
-                    and errMsg == 'Unable to free up enough space for caching.'):
+                if (errType == 'AssertionError' and
+                        errMsg == 'Unable to free up enough space for caching.'):
                     self.assertEqual(expectedResult, 'Fail')
                 else:
                     self.fail('Toil did not raise the expected AssertionError')
@@ -361,6 +361,62 @@ def _parseAssertionError(logFile):
                             return fields[1][:-1], ' '.join(fields[2:])
             raise RuntimeError('An error was not found in the error log.')
 
+        def testAsyncWriteWithCaching(self):
+            """
+            Ensure the Async Writing of files happens as expected.  The first Job forcefully
+            modifies the cache lock file to 1GB. The second asks for 1GB of disk and  writes a 900MB
+            file into cache and then rewrites it to the job store. The third asks for 1GB of disk
+            and then requests the file.
+
+            The second writeGlobalFile in the writing job should trigger an async write to the job
+            store and the since the first was written to cache, the second won't have space to be.
+            The third job should wait for the second to finish writing to the job store (through the
+            harbinger mechanism) and there should be no read-only error thrown by the async write
+            threads.
+            """
+            self.options.retryCount = 0
+            A = Job.wrapJobFn(self._forceModifyCacheLockFile, newTotalMB=1024, disk='1G')
+            B = Job.wrapJobFn(self._doubleWriteFileToJobStore, fileMB=900, disk='1G')
+            C = Job.wrapJobFn(self._readFromJobStoreWithoutAsssertions, fsID=B.rv(), disk='1G')
+            # Set it to > 2GB such that the cleanup jobs don't die.
+            D = Job.wrapJobFn(self._forceModifyCacheLockFile, newTotalMB=5000, disk='10M')
+            A.addChild(B)
+            B.addChild(C)
+            C.addChild(D)
+            Job.Runner.startToil(A, self.options)
+
+        @staticmethod
+        def _doubleWriteFileToJobStore(job, fileMB):
+            """
+            Write a local file to job store, then write it again.  The second should trigger an
+            async write.
+
+            :param job: job
+            :param fileMB: File Size
+            :return: Job store file ID for second written file
+            """
+            job.fileStore.logToMaster('Double writing a file into job store')
+            work_dir = job.fileStore.getLocalTempDir()
+            with open(os.path.join(work_dir, str(uuid4())), 'w') as testFile:
+                testFile.write(os.urandom(fileMB * 1024 * 1024))
+
+            job.fileStore.writeGlobalFile(testFile.name)
+            return job.fileStore.writeGlobalFile(testFile.name)
+
+        @staticmethod
+        def _readFromJobStoreWithoutAsssertions(job, fsID):
+            """
+            Reads a file from the job store.  That will be all, thank you.
+
+            :param job: job
+            :param fsID: Job store file ID for the read file
+            :return: None
+            """
+            job.fileStore.logToMaster('Reading the written file')
+            assert not job.fileStore._fileIsCached(fsID)
+            assert job.fileStore.HarbingerFile(job.fileStore, fileStoreID=fsID).exists()
+            job.fileStore.readGlobalFile(fsID)
+
         # writeGlobalFile tests
         def testWriteNonLocalFileToJobStore(self):
             """
@@ -732,7 +788,7 @@ def _removeReadFileFn(job, fileToDelete, readAsMutable):
             """
             Accept a file. Run os.remove on it. Then attempt to delete it locally. This will raise
             an error for files read immutably.
-            Then write a new file to teh jobstore and try to do the same. This should always raise
+            Then write a new file to the jobstore and try to do the same. This should always raise
             an error
 
             :param fileToDelete: File written to the job store that is tracked by the cache