Repository: DataBiosphere/toil

Issue:
"database is locked" error when using the new SQLite caching db
The new SQLite-based caching keeps failing when there are several tasks running on the same worker (stack trace below). Looks like SQLite can't handle many concurrent calls (see https://stackoverflow.com/questions/26862809/operational-error-database-is-locked/26864360 ).

@adamnovak not sure if there is an easy way to limit concurrency as workers are on different processes. Perhaps the easiest thing could be to retry with a random exponential backoff and also increase the timeout as suggested in the thread? 

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cwltool/executors.py", line 168, in run_jobs
    for job in jobiter:
  File "/usr/local/lib/python2.7/dist-packages/cwltool/command_line_tool.py", line 455, in job
    reffiles, builder.stagedir, runtimeContext, True)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 274, in make_path_mapper
    runtimeContext.toil_get_file)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 228, in __init__
    referenced_files, basedir, stagedir, separateDirs=separateDirs)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 262, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 319, in setup
    self.visit(fob, stagedir, basedir, copy=fob.get("writable"), staged=True)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 262, in visit
    stagedir, basedir, copy=copy, staged=staged)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 267, in visitlisting
    self.visit(ld, stagedir, basedir, copy=ld.get("writable", copy), staged=staged)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 255, in visit
    resolved = self.get_file(loc) if self.get_file else loc
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 309, in toil_get_file
    src_path = file_store.readGlobalFile(FileID.unpack(file_store_id[7:]))
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStores/cachingFileStore.py", line 1060, in readGlobalFile
    return self._readGlobalFileWithCache(fileStoreID, localFilePath, symlink, readerID)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStores/cachingFileStore.py", line 1427, in _readGlobalFileWithCache
    (fileStoreID, cachedPath, self.getGlobalFileSize(fileStoreID), 'downloading', pid))
OperationalError: database is locked
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 369, in workerScript
    fileStore.startCommit(jobState=False)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStores/cachingFileStore.py", line 959, in open
    self._deallocateSpaceForJob()
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStores/cachingFileStore.py", line 795, in _deallocateSpaceForJob
    self._removeJob(self.con, self.cur, self.jobID)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStores/cachingFileStore.py", line 762, in _removeJob
    for row in cur.execute('SELECT tempdir FROM jobs WHERE id = ?', (jobID,)):
OperationalError: database is locked
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-481)
┆Issue Number: TOIL-481



Discussion:
Actually, increasing the timeout to 60s substantially reduced the errors, so we may be ok by just increasing the timeout for now. But lemme know if you have other ideas around reducing the concurrency and/or number of operations.
The database is *supposed* to be locked, I think; I'm relying on
SQLite's locking and internal concurrency control to keep all the
workers synchronized.

It could be that we have some piece of code that is holding a write
lock on the database for much longer than it should be (i.e. starting
a batch of writes but not committing in a timely fashion because it is
running some other code). I'm not sure how we'd hunt for that, other
than by reading through the code looking for places where that might
happen. It would be touch to pause all the other workers and find the
one holding the lock and what it is doing. Maybe we could get
everybody to announce their locks/unlocks with the real time logger
somehow.

On 1/8/20, Asha Rostamianfar <notifications@github.com> wrote:
> Actually, increasing the timeout to 60s substantially reduced the errors, so
> we may be ok by just increasing the timeout for now. But lemme know if you
> have other ideas around reducing the concurrency and/or number of
> operations.
>
> --
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub:
> https://github.com/DataBiosphere/toil/issues/2910#issuecomment-572303096


In addition to tracking down who is hogging the lock, putting
the write locking in a retry loop would be useful.  It can be
very hard to get a timeout right when process can slow down for
unpredictable reasons.



Adam Novak <notifications@github.com> writes:
> The database is *supposed* to be locked, I think; I'm relying on
> SQLite's locking and internal concurrency control to keep all the
> workers synchronized.
> 
> It could be that we have some piece of code that is holding a write
> lock on the database for much longer than it should be (i.e. starting
> a batch of writes but not committing in a timely fashion because it is
> running some other code). I'm not sure how we'd hunt for that, other
> than by reading through the code looking for places where that might
> happen. It would be touch to pause all the other workers and find the
> one holding the lock and what it is doing. Maybe we could get
> everybody to announce their locks/unlocks with the real time logger
> somehow.
> 
> On 1/8/20, Asha Rostamianfar <notifications@github.com> wrote:
> > Actually, increasing the timeout to 60s substantially reduced the errors, so
> > we may be ok by just increasing the timeout for now. But lemme know if you
> > have other ideas around reducing the concurrency and/or number of
> > operations.
> >
> > --
> > You are receiving this because you were mentioned.
> > Reply to this email directly or view it on GitHub:
> > https://github.com/DataBiosphere/toil/issues/2910#issuecomment-572303096
> 
> 
> -- 
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/DataBiosphere/toil/issues/2910#issuecomment-572308378
> The database is *supposed* to be locked, I think; I'm relying on
> SQLite's locking and internal concurrency control to keep all the
> workers synchronized.
> 
> It could be that we have some piece of code that is holding a write
> lock on the database for much longer than it should be (i.e. starting
> a batch of writes but not committing in a timely fashion because it is
> running some other code). I'm not sure how we'd hunt for that, other
> than by reading through the code looking for places where that might
> happen. It would be touch to pause all the other workers and find the
> one holding the lock and what it is doing. Maybe we could get
> everybody to announce their locks/unlocks with the real time logger
> somehow.
> 
> On 1/8/20, Asha Rostamianfar <notifications@github.com> wrote:
> > Actually, increasing the timeout to 60s substantially reduced the errors, so
> > we may be ok by just increasing the timeout for now. But lemme know if you
> > have other ideas around reducing the concurrency and/or number of
> > operations.
> >
> > --
> > You are receiving this because you were mentioned.
> > Reply to this email directly or view it on GitHub:
> > https://github.com/DataBiosphere/toil/issues/2910#issuecomment-572303096
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.*
> 

Both sound good! Any objections to increasing the timeout to 60s in the meantime? That substantially reduces the errors while we investigate the root cause(s).
That's definitely a good idea @arostamianfar; I suppose you know where it is configured?
yeap; sent PR #2913
One potential cause here is this part of `_giveAwayDownloadingFile` where it makes a change to the database but never commits to close out the transaction: https://github.com/DataBiosphere/toil/blob/07af6913e73199193be7bcf290c2dedbc7f82d2d/src/toil/fileStores/cachingFileStore.py#L1360-L1365
@adamnovak i think we can close this issue? I haven't seen the error since merging both changes. Thanks for your help!
Excellent. I will close this up.
I was able to reliably reproduce this again on a new workflow that operates on large (100GiB+) files. I haven't yet looked into the details on why the exception happens, but I think the reproducible scenario is like:
* Job 1 runs on worker `A` and outputs a large file, `F`.
* After job 1 finishes, job 2 runs on worker `B` and uses `F`. Starts downloading the file from s3 (takes 15+ mins).
* Job 3 runs on worker `B` while `F` is being downloaded.

@adamnovak do see any obvious places where the lock can be held while a file is being downloaded?
Based on Asha's description you have this:

`{ Job_3[Worker_B], Job_2[Worker_B](Job_1[Worker_A] -> F) }`

Usually you want to have the cache travel to get around deadlocks as such:

`{ Job_3[Worker_B](Cache_Copy_3), Job_2[Worker_B](Job_1[Worker_A](Cache_Copy_1) -> F, Cache_Copy_2) }`

Think of it as message-passing a copy of the cache with each executor, rather than have global locks.

Hope it helps,
~p

@pgrosu I don't quite parse your notation here, or what you mean by having the cache "travel". The current design is that each machine (which I think @arostamianfar is calling a "worker") has its own separate cache, shared among all the jobs on that machine.

What is *supposed* to be happening here is that job 2 on worker B should [record in the database that it is going to download the file and commit](https://github.com/DataBiosphere/toil/blob/d07f003a2847455c11d564fad7fdaf321c7ffc4b/src/toil/fileStores/cachingFileStore.py#L1421-L1429), and then [download the file](https://github.com/DataBiosphere/toil/blob/d07f003a2847455c11d564fad7fdaf321c7ffc4b/src/toil/fileStores/cachingFileStore.py#L1441). It shouldn't hold a database write lock while the download is ongoing (although maybe it somehow is holding a read lock?).

Job 3, meanwhile, should run the same `INSERT OR IGNORE...` and `INSERT... WHERE...` write transaction, see that the file is in the process of being downloaded by someone else, and go [here](https://github.com/DataBiosphere/toil/blob/d07f003a2847455c11d564fad7fdaf321c7ffc4b/src/toil/fileStores/cachingFileStore.py#L1278-L1284), where it basically cleans up anything orphaned and then immediately checks again to see if the file is ready yet. It shouldn't be holding any locks during this spin-and-poll process, but maybe it is polling too fast, taking the write lock for its conditional writes, and starving the actual downloader from being able to record that the download is complete?

@arostamianfar Is Job 3 or Job 2 the one that is timing out due to the database being locked? And is it timing out near the beginning of the download, or at the end of the download?

It sounds like Job 3 is probably timing out a the beginning, since you want to look for places where the downloader can hold a lock. It's also interesting that in your stack trace, at the polling code, it can't get a write lock, and then the cleanup can't get a read lock, suggesting it is being starved out by an exclusive writer lock held by somebody else. But, I don't see anywhere where the downloader is going to hold a write lock for an appreciable time, except maybe [here where it records files it deleted in a loop](https://github.com/DataBiosphere/toil/blob/d07f003a2847455c11d564fad7fdaf321c7ffc4b/src/toil/fileStores/cachingFileStore.py#L643-L648). Is it possible that there are multiple Job 3s all banging on the database as fast as they can, and one of them is getting starved out by the others polling (via conditional writes) much more often than they need to?

Things we could try:

1. Add a sleep to the loop that Job 3 is in, polling for the file to be ready.
2. Change [this loop](https://github.com/DataBiosphere/toil/blob/d07f003a2847455c11d564fad7fdaf321c7ffc4b/src/toil/fileStores/cachingFileStore.py#L643-L648) where we execute deletions to free up space to commit after we record each deletion, instead of all of them, in case a lot of files were deleted.
3. Change that loop to poll using reads if the downloading file is still there, instead of going back to the beginning and trying to take ownership of the download on the off chance that the download we saw before was orphaned and got cleaned up.
4. Wrap all our SQL in connection context managers to be super sure that we don't forget to commit.
5. Don't keep connections to the database open persistently, on the theory that they may keep read locks longer than they need to and thus stall writers.

Resolving patch:
diff --git a/src/toil/deferred.py b/src/toil/deferred.py
--- a/src/toil/deferred.py
+++ b/src/toil/deferred.py
@@ -306,8 +306,6 @@ def _runOrphanedDeferredFunctions(self):
                     # So skip it.
                     continue
 
-                logger.debug("Try file %s" % fullFilename)
-
                 fd = None
 
                 try:
diff --git a/src/toil/fileStores/cachingFileStore.py b/src/toil/fileStores/cachingFileStore.py
--- a/src/toil/fileStores/cachingFileStore.py
+++ b/src/toil/fileStores/cachingFileStore.py
@@ -46,6 +46,7 @@
 from toil.lib.humanize import bytes2human
 from toil.lib.misc import mkdir_p, robust_rmtree, atomic_copy, atomic_copyobj
 from toil.lib.objects import abstractclassmethod
+from toil.lib.retry import retry
 from toil.resource import ModuleDescriptor
 from toil.fileStores.abstractFileStore import AbstractFileStore
 from toil.fileStores import FileID
@@ -185,11 +186,22 @@ class CachingFileStore(AbstractFileStore):
 
     """
 
-    def __init__(self, jobStore, jobGraph, localTempDir, waitForPreviousCommit, forceNonFreeCaching=False):
+    def __init__(self, jobStore, jobGraph, localTempDir, waitForPreviousCommit):
         super(CachingFileStore, self).__init__(jobStore, jobGraph, localTempDir, waitForPreviousCommit)
 
         # For testing, we have the ability to force caching to be non-free, by never linking from the file store
-        self.forceNonFreeCaching = forceNonFreeCaching
+        self.forceNonFreeCaching = False
+        # Also for testing, we have the ability to force a delay (in seconds)
+        # during file download from the job store, in order to easily test the
+        # behavior of the system when a download is in progress.
+        self.forceDownloadDelay = None
+
+        # When waiting for other running workers to download a file, or
+        # otherwise progress, how long in seconds should we wait between
+        # polling attempts?  Our mechanism for polling involves an exclusive
+        # lock on the database and conditional writes, so this should be high
+        # enough that everyone isn't constantly contending for the lock.
+        self.contentionBackoff = 15
 
         # Variables related to caching
         # Decide where the cache directory will be. We put it next to the
@@ -217,7 +229,7 @@ def __init__(self, jobStore, jobGraph, localTempDir, waitForPreviousCommit, forc
         # Connect to the cache database in there, or create it if not present
         self.dbPath = os.path.join(self.localCacheDir, 'cache-{}.db'.format(self.workflowAttemptNumber))
         # We need to hold onto both a connection (to commit) and a cursor (to actually use the database)
-        self.con = sqlite3.connect(self.dbPath, timeout=60)
+        self.con = sqlite3.connect(self.dbPath, timeout=60.0)
         self.cur = self.con.cursor()
 
         # Note that sqlite3 automatically starts a transaction when we go to
@@ -244,8 +256,7 @@ def __init__(self, jobStore, jobGraph, localTempDir, waitForPreviousCommit, forc
 
         # Initialize the space accounting properties
         freeSpace, _ = getFileSystemSize(self.localCacheDir)
-        self.cur.execute('INSERT OR IGNORE INTO properties VALUES (?, ?)', ('maxSpace', freeSpace))
-        self.con.commit()
+        self._write([('INSERT OR IGNORE INTO properties VALUES (?, ?)', ('maxSpace', freeSpace))])
 
         # Space used by caching and by jobs is accounted with queries
 
@@ -259,6 +270,84 @@ def __init__(self, jobStore, jobGraph, localTempDir, waitForPreviousCommit, forc
         # time.
         self.commitThread = None
 
+    
+    @staticmethod
+    def _staticWrite(con, cur, operations):
+        """
+        Write to the caching database, using the given connection.
+
+        If we can't get an SQLite write lock on the database, retry with some
+        backoff until we can.
+
+        operations is a list of tuples of (sql string, optional tuple of values
+        to substitute), or bare sql strings.
+
+        All operations are executed in a single transaction, which is
+        committed.
+
+        :param sqlite3.Connection con: Connection to the cache database.
+        :param sqlite3.Cursor cur: Cursor in the cache database.
+        :param list operations: List of sql strings or tuples of (sql, optional values) to execute.
+        :return: Number of rows modified by the last operation
+        :rtype: int
+        """
+
+        for attempt in retry(timeout=float('inf'), predicate=lambda e: isinstance(e, sqlite3.OperationalError) and 'is locked' in str(e)):
+            # Try forever with backoff
+            with attempt:
+                try:
+                    for item in operations:
+                        if not isinstance(item, tuple):
+                            # Must be a single SQL string. Wrap it.
+                            item = (item,)
+                        # Parse out the command and the variables to substitute
+                        command = item[0]
+                        if len(item) < 2:
+                            args = ()
+                        else:
+                            args = item[1]
+                        # Do it
+                        cur.execute(command, args)
+                except Exception as e:
+                    logging.error('Error talking to caching database: %s', str(e))
+
+                    # Try to make sure we don't somehow leave anything part-done if a
+                    # middle operation somehow fails. 
+                    try:
+                        con.rollback()
+                    except:
+                        # But don't stop if we can't roll back.
+                        pass
+
+                    # Raise and maybe retry
+                    raise e
+                else:
+                    # The transaction worked!
+                    # Now commit the transaction.
+                    con.commit()
+
+        return cur.rowcount
+
+    def _write(self, operations):
+        """
+        Write to the caching database, using the instance's connection
+
+        If we can't get an SQLite write lock on the database, retry with some
+        backoff until we can.
+
+        operations is a list of tuples of (sql string, optional tuple of values
+        to substitute), or bare sql strings.
+
+        All operations are executed in a single transaction, which is
+        committed.
+
+        :param list operations: List of sql strings or tuples of (sql, optional values) to execute.
+        :return: Number of rows modified by the last operation
+        :rtype: int
+        """
+
+        return self._staticWrite(self.con, self.cur, operations)
+
     @classmethod
     def _ensureTables(cls, con):
         """
@@ -270,7 +359,7 @@ def _ensureTables(cls, con):
         # Get a cursor
         cur = con.cursor()
 
-        cur.execute("""
+        cls._staticWrite(con, cur, ["""
             CREATE TABLE IF NOT EXISTS files (
                 id TEXT NOT NULL PRIMARY KEY,
                 path TEXT UNIQUE NOT NULL,
@@ -278,8 +367,7 @@ def _ensureTables(cls, con):
                 state TEXT NOT NULL,
                 owner INT
             )
-        """)
-        cur.execute("""
+        """, """
             CREATE TABLE IF NOT EXISTS refs (
                 path TEXT NOT NULL,
                 file_id TEXT NOT NULL,
@@ -287,22 +375,19 @@ def _ensureTables(cls, con):
                 state TEXT NOT NULL,
                 PRIMARY KEY (path, file_id)
             )
-        """)
-        cur.execute("""
+        """, """
             CREATE TABLE IF NOT EXISTS jobs (
                 id TEXT NOT NULL PRIMARY KEY,
                 tempdir TEXT NOT NULL,
                 disk INT NOT NULL,
                 worker INT
             )
-        """)
-        cur.execute("""
+        """, """
             CREATE TABLE IF NOT EXISTS properties (
                 name TEXT NOT NULL PRIMARY KEY,
                 value INT NOT NULL
             )
-        """)
-        con.commit()
+        """])
 
     # Caching-specific API
 
@@ -452,8 +537,7 @@ def adjustCacheLimit(self, newTotalBytes):
         Adjust the total cache size limit to the given number of bytes.
         """
 
-        self.cur.execute('UPDATE properties SET value = ? WHERE name = ?', (newTotalBytes, 'maxSpace'))
-        self.con.commit()
+        self._write([('UPDATE properties SET value = ? WHERE name = ?', (newTotalBytes, 'maxSpace'))])
 
     def fileIsCached(self, fileID):
         """
@@ -524,8 +608,7 @@ def cachingIsFree(self):
             free = 0
 
         # Save to the database if we're the first to work this out
-        self.cur.execute('INSERT OR IGNORE INTO properties VALUES (?, ?)', ('freeCaching', free))
-        self.con.commit()
+        self._write([('INSERT OR IGNORE INTO properties VALUES (?, ?)', ('freeCaching', free))])
 
         # Return true if we said caching was free
         return free == 1
@@ -581,13 +664,9 @@ def _stealWorkFromTheDead(self):
         for owner in deadOwners:
             # Try and adopt all the files that any dead owner had
 
-            # If they were deleting, we delete
-            self.cur.execute('UPDATE files SET owner = ?, state = ? WHERE owner = ? AND state = ?',
-                (pid, 'deleting', owner, 'deleting'))
+            # If they were deleting, we delete.
             # If they were downloading, we delete. Any outstanding references
             # can't be in use since they are from the dead downloader.
-            self.cur.execute('UPDATE files SET owner = ?, state = ? WHERE owner = ? AND state = ?',
-                (pid, 'deleting', owner, 'downloading'))
             # If they were uploading or uploadable, we mark as cached even
             # though it never made it to the job store (and leave it unowned).
             #
@@ -599,9 +678,12 @@ def _stealWorkFromTheDead(self):
             #
             # TODO: if we ever let other PIDs be responsible for writing our
             # files asynchronously, this will need to change.
-            self.cur.execute('UPDATE files SET owner = NULL, state = ? WHERE owner = ? AND (state = ? OR state = ?)',
-                ('cached', owner, 'uploadable', 'uploading'))
-            self.con.commit()
+            self._write([('UPDATE files SET owner = ?, state = ? WHERE owner = ? AND state = ?',
+                (pid, 'deleting', owner, 'deleting')),
+                ('UPDATE files SET owner = ?, state = ? WHERE owner = ? AND state = ?',
+                (pid, 'deleting', owner, 'downloading')),
+                ('UPDATE files SET owner = NULL, state = ? WHERE owner = ? AND (state = ? OR state = ?)',
+                ('cached', owner, 'uploadable', 'uploading'))])
 
             logger.debug('Tried to adopt file operations from dead worker %d', owner)
 
@@ -642,10 +724,9 @@ def _executePendingDeletions(cls, con, cur):
 
         for fileID in deletedFiles:
             # Drop all the files. They should have stayed in deleting state. We move them from there to not present at all.
-            cur.execute('DELETE FROM files WHERE id = ? AND state = ?', (fileID, 'deleting'))
             # Also drop their references, if they had any from dead downloaders.
-            cur.execute('DELETE FROM refs WHERE file_id = ?', (fileID,))
-        con.commit()
+            cls._staticWrite(con, cur, [('DELETE FROM files WHERE id = ? AND state = ?', (fileID, 'deleting')),
+                ('DELETE FROM refs WHERE file_id = ?', (fileID,))])
 
         return len(deletedFiles)
 
@@ -681,9 +762,8 @@ def _executePendingUploads(self, con, cur):
                 break
 
             # We need to set it to uploading in a way that we can detect that *we* won the update race instead of anyone else.
-            cur.execute('UPDATE files SET state = ? WHERE id = ? AND state = ?', ('uploading', fileID, 'uploadable'))
-            con.commit()
-            if cur.rowcount != 1:
+            rowCount = self._staticWrite(con, cur, [('UPDATE files SET state = ? WHERE id = ? AND state = ?', ('uploading', fileID, 'uploadable'))])
+            if rowCount != 1:
                 # We didn't manage to update it. Someone else (a running job if
                 # we are a committing thread, or visa versa) must have grabbed
                 # it.
@@ -699,8 +779,7 @@ def _executePendingUploads(self, con, cur):
             uploadedCount += 1
 
             # Remember that we uploaded it in the database
-            cur.execute('UPDATE files SET state = ?, owner = NULL WHERE id = ?', ('cached', fileID))
-            con.commit()
+            self._staticWrite(con, cur, [('UPDATE files SET state = ?, owner = NULL WHERE id = ?', ('cached', fileID))])
 
         return uploadedCount
 
@@ -726,8 +805,7 @@ def _allocateSpaceForJob(self, newJobReqs):
         # But we won't actually let the job run and use any of this space until
         # the cache has been successfully cleared out.
         pid = os.getpid()
-        self.cur.execute('INSERT INTO jobs VALUES (?, ?, ?, ?)', (self.jobID, self.localTempDir, newJobReqs, pid))
-        self.con.commit()
+        self._write([('INSERT INTO jobs VALUES (?, ?, ?, ?)', (self.jobID, self.localTempDir, newJobReqs, pid))])
 
         # Now we need to make sure that we can fit all currently cached files,
         # and the parts of the total job requirements not currently spent on
@@ -770,8 +848,7 @@ def _removeJob(cls, con, cur, jobID):
                 # May not exist
                 pass
         # And their database entries
-        cur.execute('DELETE FROM refs WHERE job_id = ?', (jobID,))
-        con.commit()
+        cls._staticWrite(con, cur, [('DELETE FROM refs WHERE job_id = ?', (jobID,))])
 
         try:
             # Delete the job's temp directory to the extent that we can.
@@ -780,8 +857,7 @@ def _removeJob(cls, con, cur, jobID):
             pass
 
         # Strike the job from the database
-        cur.execute('DELETE FROM jobs WHERE id = ?', (jobID,))
-        con.commit()
+        cls._staticWrite(con, cur, [('DELETE FROM jobs WHERE id = ?', (jobID,))])
 
     def _deallocateSpaceForJob(self):
         """
@@ -845,14 +921,13 @@ def _tryToFreeUpSpace(self):
         pid = os.getpid()
 
         # Try and grab it for deletion, subject to the condition that nothing has started reading it
-        self.cur.execute("""
+        self._write([("""
             UPDATE files SET owner = ?, state = ? WHERE id = ? AND state = ?
             AND owner IS NULL AND NOT EXISTS (
                 SELECT NULL FROM refs WHERE refs.file_id = files.id AND refs.state != 'mutable'
             )
             """,
-            (pid, 'deleting', fileID, 'cached'))
-        self.con.commit()
+            (pid, 'deleting', fileID, 'cached'))])
 
         logger.debug('Evicting file %s', fileID)
 
@@ -981,9 +1056,8 @@ def writeGlobalFile(self, localFileName, cleanup=False):
 
         # Create a file in uploadable state and a reference, in the same transaction.
         # Say the reference is an immutable reference
-        self.cur.execute('INSERT INTO files VALUES (?, ?, ?, ?, ?)', (fileID, cachePath, fileSize, 'uploadable', pid))
-        self.cur.execute('INSERT INTO refs VALUES (?, ?, ?, ?)', (absLocalFileName, fileID, creatorID, 'immutable'))
-        self.con.commit()
+        self._write([('INSERT INTO files VALUES (?, ?, ?, ?, ?)', (fileID, cachePath, fileSize, 'uploadable', pid)),
+            ('INSERT INTO refs VALUES (?, ?, ?, ?)', (absLocalFileName, fileID, creatorID, 'immutable'))])
 
         if absLocalFileName.startswith(self.localTempDir):
             # We should link into the cache, because the upload is coming from our local temp dir
@@ -1014,11 +1088,10 @@ def writeGlobalFile(self, localFileName, cleanup=False):
             # space to make a full copy in the cache, if we aren't allowed to
             # take this copy away from the writer.
 
-            # Change the reference to 'mutable', which it will be
-            self.cur.execute('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', absLocalFileName, fileID))
-            # And drop the file altogether
-            self.cur.execute('DELETE FROM files WHERE id = ?', (fileID,))
-            self.con.commit()
+            # Change the reference to 'mutable', which it will be.
+            # And drop the file altogether.
+            self._write([('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', absLocalFileName, fileID)),
+                ('DELETE FROM files WHERE id = ?', (fileID,))])
 
             # Save the file to the job store right now
             logger.debug('Actually executing upload for file %s', fileID)
@@ -1028,10 +1101,7 @@ def writeGlobalFile(self, localFileName, cleanup=False):
         return FileID.forPath(fileID, absLocalFileName)
 
     def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=False, symlink=False):
-        if not isinstance(fileStoreID, FileID):
-            # Don't let the user forge File IDs.
-            raise TypeError('Received file ID not of type FileID: {}'.format(fileStoreID))
-
+        
         if str(fileStoreID) in self.filesToDelete:
             # File has already been deleted
             raise FileNotFoundError('Attempted to read deleted file: {}'.format(fileStoreID))
@@ -1082,9 +1152,8 @@ def _readGlobalFileWithoutCache(self, fileStoreID, localFilePath, mutable, symli
         # the backing job store yet.
 
         # Try and make a 'copying' reference to such a file
-        self.cur.execute('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ?)',
-            (localFilePath, readerID, 'copying', fileStoreID, 'uploadable', 'uploading'))
-        self.con.commit()
+        self._write([('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ?)',
+            (localFilePath, readerID, 'copying', fileStoreID, 'uploadable', 'uploading'))])
 
         # See if we got it
         have_reference = False
@@ -1104,11 +1173,14 @@ def _readGlobalFileWithoutCache(self, fileStoreID, localFilePath, mutable, symli
             if cachedPath is None:
                 raise RuntimeError('File %s went away while we had a reference to it!' % fileStoreID)
 
+            if self.forceDownloadDelay is not None:
+                # Wait around to simulate a big file for testing
+                time.sleep(self.forceDownloadDelay)
+            
             atomic_copy(cachedPath, localFilePath)
 
             # Change the reference to mutable
-            self.cur.execute('UPDATE refs SET state = ? WHERE path = ? and file_id = ?', ('mutable', localFilePath, fileStoreID))
-            self.con.commit()
+            self._write([('UPDATE refs SET state = ? WHERE path = ? and file_id = ?', ('mutable', localFilePath, fileStoreID))])
 
         else:
 
@@ -1118,9 +1190,12 @@ def _readGlobalFileWithoutCache(self, fileStoreID, localFilePath, mutable, symli
 
             # Create a 'mutable' reference (even if we end up with a link)
             # so we can see this file in deleteLocalFile.
-            self.cur.execute('INSERT INTO refs VALUES (?, ?, ?, ?)',
-                (localFilePath, fileStoreID, readerID, 'mutable'))
-            self.con.commit()
+            self._write([('INSERT INTO refs VALUES (?, ?, ?, ?)',
+                (localFilePath, fileStoreID, readerID, 'mutable'))])
+
+            if self.forceDownloadDelay is not None:
+                # Wait around to simulate a big file for testing
+                time.sleep(self.forceDownloadDelay)
 
             # Just read directly
             if mutable or self.forceNonFreeCaching:
@@ -1143,6 +1218,11 @@ def _downloadToCache(self, fileStoreID, cachedPath):
         :param toil.fileStores.FileID fileStoreID: job store id for the file
         :param str cachedPath: absolute destination path in the cache. Already known not to exist.
         """
+
+        if self.forceDownloadDelay is not None:
+            # Wait around to simulate a big file for testing
+            time.sleep(self.forceDownloadDelay)
+
         if self.forceNonFreeCaching:
             # Always copy
             with self.jobStore.readFileStream(fileStoreID) as inStream:
@@ -1172,9 +1252,8 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
         while True:
             # Try and create a downloading entry if no entry exists
             logger.debug('Trying to make file record for id %s', fileStoreID)
-            self.cur.execute('INSERT OR IGNORE INTO files VALUES (?, ?, ?, ?, ?)',
-                (fileStoreID, cachedPath, self.getGlobalFileSize(fileStoreID), 'downloading', pid))
-            self.con.commit()
+            self._write([('INSERT OR IGNORE INTO files VALUES (?, ?, ?, ?, ?)',
+                (fileStoreID, cachedPath, self.getGlobalFileSize(fileStoreID), 'downloading', pid))])
 
             # See if we won the race
             self.cur.execute('SELECT COUNT(*) FROM files WHERE id = ? AND state = ? AND owner = ?', (fileStoreID, 'downloading', pid))
@@ -1195,9 +1274,8 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
                 # two readers, one cached copy, and space for two copies total.
 
                 # Make the copying reference
-                self.cur.execute('INSERT INTO refs VALUES (?, ?, ?, ?)',
-                    (localFilePath, fileStoreID, readerID, 'copying'))
-                self.con.commit()
+                self._write([('INSERT INTO refs VALUES (?, ?, ?, ?)',
+                    (localFilePath, fileStoreID, readerID, 'copying'))])
 
                 # Fulfill it with a full copy or by giving away the cached copy
                 self._fulfillCopyingReference(fileStoreID, cachedPath, localFilePath)
@@ -1213,9 +1291,8 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
                 # is in 'cached' or 'uploadable' or 'uploading' state.
                 # It might be uploading because *we* are supposed to be uploading it.
                 logger.debug('Trying to make reference to file %s', fileStoreID)
-                self.cur.execute('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ? OR state = ?)',
-                    (localFilePath, readerID, 'copying', fileStoreID, 'cached', 'uploadable', 'uploading'))
-                self.con.commit()
+                self._write([('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ? OR state = ?)',
+                    (localFilePath, readerID, 'copying', fileStoreID, 'cached', 'uploadable', 'uploading'))])
 
                 # See if we got it
                 self.cur.execute('SELECT COUNT(*) FROM refs WHERE path = ? and file_id = ?', (localFilePath, fileStoreID))
@@ -1240,14 +1317,13 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
 
                         # See if we have no other references and we can give away the file.
                         # Change it to downloading owned by us if we can grab it.
-                        self.cur.execute("""
+                        self._write([("""
                             UPDATE files SET files.owner = ?, files.state = ? WHERE files.id = ? AND files.state = ?
                             AND files.owner IS NULL AND NOT EXISTS (
                                 SELECT NULL FROM refs WHERE refs.file_id = files.id AND refs.state != 'mutable'
                             )
                             """,
-                            (pid, 'downloading', fileStoreID, 'cached'))
-                        self.con.commit()
+                            (pid, 'downloading', fileStoreID, 'cached'))])
 
                         if self._giveAwayDownloadingFile(fileStoreID, cachedPath, localFilePath):
                             # We got ownership of the file and managed to give it away.
@@ -1258,13 +1334,19 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
                         # need to wait for one of those people with references to the file
                         # to finish and give it up.
                         # TODO: work out if that will never happen somehow.
-
-                    # OK, now we have space to make a copy. Do it
+                        time.sleep(self.contentionBackoff)
+
+                    # OK, now we have space to make a copy.
+                    
+                    if self.forceDownloadDelay is not None:
+                        # Wait around to simulate a big file for testing
+                        time.sleep(self.forceDownloadDelay)
+                    
+                    # Make the copy
                     atomic_copy(cachedPath, localFilePath)
 
                     # Change the reference to mutable
-                    self.cur.execute('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID))
-                    self.con.commit()
+                    self._write([('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID))])
 
                     # Now we're done
                     return localFilePath
@@ -1283,6 +1365,9 @@ def _readGlobalFileMutablyWithCache(self, fileStoreID, localFilePath, readerID):
             self._stealWorkFromTheDead()
             self._executePendingDeletions(self.con, self.cur)
 
+            # Wait for other people's downloads to progress before re-polling.
+            time.sleep(self.contentionBackoff)
+
     def _fulfillCopyingReference(self, fileStoreID, cachedPath, localFilePath):
         """
         For use when you own a file in 'downloading' state, and have a
@@ -1311,16 +1396,18 @@ def _fulfillCopyingReference(self, fileStoreID, cachedPath, localFilePath):
         # Expose this file as cached so other people can copy off of it too.
 
         # Change state from downloading to cached
-        self.cur.execute('UPDATE files SET state = ?, owner = NULL WHERE id = ?',
-            ('cached', fileStoreID))
-        self.con.commit()
+        self._write([('UPDATE files SET state = ?, owner = NULL WHERE id = ?',
+            ('cached', fileStoreID))])
+
+        if self.forceDownloadDelay is not None:
+            # Wait around to simulate a big file for testing
+            time.sleep(self.forceDownloadDelay)
 
         # Make our copy
         atomic_copy(cachedPath, localFilePath)
 
         # Change our reference to mutable
-        self.cur.execute('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID))
-        self.con.commit()
+        self._write([('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID))])
 
         # Now we're done
         return
@@ -1350,12 +1437,13 @@ def _giveAwayDownloadingFile(self, fileStoreID, cachedPath, localFilePath):
         if self.cur.fetchone()[0] > 0:
             # Now we have exclusive control of the cached copy of the file, so we can give it away.
 
+            # Don't fake a delay here; this should be a rename always.
+
             # We are giving it away
             shutil.move(cachedPath, localFilePath)
             # Record that.
-            self.cur.execute('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID))
-            self.cur.execute('DELETE FROM files WHERE id = ?', (fileStoreID,))
-            self.con.commit()
+            self._write([('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('mutable', localFilePath, fileStoreID)),
+                ('DELETE FROM files WHERE id = ?', (fileStoreID,))])
 
             # Now we're done
             return True
@@ -1417,16 +1505,14 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
 
         # Start a loop until we can do one of these
         while True:
-            # Try and create a downloading entry if no entry exists
-            logger.debug('Trying to make file record for id %s', fileStoreID)
-            self.cur.execute('INSERT OR IGNORE INTO files VALUES (?, ?, ?, ?, ?)',
-                (fileStoreID, cachedPath, self.getGlobalFileSize(fileStoreID), 'downloading', pid))
+            # Try and create a downloading entry if no entry exists.
             # Make sure to create a reference at the same time if it succeeds, to bill it against our job's space.
             # Don't create the mutable reference yet because we might not necessarily be able to clear that space.
-            logger.debug('Trying to make file reference to %s', fileStoreID)
-            self.cur.execute('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND state = ? AND owner = ?',
-                (localFilePath, readerID, 'immutable', fileStoreID, 'downloading', pid))
-            self.con.commit()
+            logger.debug('Trying to make file record and reference for id %s', fileStoreID)
+            self._write([('INSERT OR IGNORE INTO files VALUES (?, ?, ?, ?, ?)',
+                (fileStoreID, cachedPath, self.getGlobalFileSize(fileStoreID), 'downloading', pid)),
+                ('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND state = ? AND owner = ?',
+                (localFilePath, readerID, 'immutable', fileStoreID, 'downloading', pid))])
 
             # See if we won the race
             self.cur.execute('SELECT COUNT(*) FROM files WHERE id = ? AND state = ? AND owner = ?', (fileStoreID, 'downloading', pid))
@@ -1446,9 +1532,8 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
                     # We made the link!
 
                     # Change file state from downloading to cached so other people can use it
-                    self.cur.execute('UPDATE files SET state = ?, owner = NULL WHERE id = ?',
-                        ('cached', fileStoreID))
-                    self.con.commit()
+                    self._write([('UPDATE files SET state = ?, owner = NULL WHERE id = ?',
+                        ('cached', fileStoreID))])
 
                     # Now we're done!
                     return localFilePath
@@ -1456,8 +1541,7 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
                     # We could not make a link. We need to make a copy.
 
                     # Change the reference to copying.
-                    self.cur.execute('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('copying', localFilePath, fileStoreID))
-                    self.con.commit()
+                    self._write([('UPDATE refs SET state = ? WHERE path = ? AND file_id = ?', ('copying', localFilePath, fileStoreID))])
 
                     # Fulfill it with a full copy or by giving away the cached copy
                     self._fulfillCopyingReference(fileStoreID, cachedPath, localFilePath)
@@ -1473,9 +1557,8 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
                 # is in 'cached' or 'uploadable' or 'uploading' state.
                 # It might be uploading because *we* are supposed to be uploading it.
                 logger.debug('Trying to make reference to file %s', fileStoreID)
-                self.cur.execute('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ? OR state = ?)',
-                    (localFilePath, readerID, 'immutable', fileStoreID, 'cached', 'uploadable', 'uploading'))
-                self.con.commit()
+                self._write([('INSERT INTO refs SELECT ?, id, ?, ? FROM files WHERE id = ? AND (state = ? OR state = ? OR state = ?)',
+                    (localFilePath, readerID, 'immutable', fileStoreID, 'cached', 'uploadable', 'uploading'))])
 
                 # See if we got it
                 self.cur.execute('SELECT COUNT(*) FROM refs WHERE path = ? and file_id = ?', (localFilePath, fileStoreID))
@@ -1499,8 +1582,7 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
                         # we already have code for that for mutable downloads,
                         # so just clear the reference and download mutably.
 
-                        self.cur.execute('DELETE FROM refs WHERE path = ? AND file_id = ?', (localFilePath, fileStoreID))
-                        self.con.commit()
+                        self._write([('DELETE FROM refs WHERE path = ? AND file_id = ?', (localFilePath, fileStoreID))])
 
                         return self._readGlobalFileMutablyWithCache(fileStoreID, localFilePath, readerID)
                 else:
@@ -1513,6 +1595,9 @@ def _readGlobalFileWithCache(self, fileStoreID, localFilePath, symlink, readerID
                     self._stealWorkFromTheDead()
                     self._executePendingDeletions(self.con, self.cur)
 
+                    # Wait for other people's downloads to progress.
+                    time.sleep(self.contentionBackoff)
+
     def readGlobalFileStream(self, fileStoreID):
         if str(fileStoreID) in self.filesToDelete:
             # File has already been deleted
@@ -1552,8 +1637,7 @@ def deleteLocalFile(self, fileStoreID):
 
         for path in deleted:
             # Drop the references
-            self.cur.execute('DELETE FROM refs WHERE file_id = ? AND job_id = ? AND path = ?', (fileStoreID, jobID, path))
-            self.con.commit()
+            self._write([('DELETE FROM refs WHERE file_id = ? AND job_id = ? AND path = ?', (fileStoreID, jobID, path))])
 
         # Now space has been revoked from the cache because that job needs its space back.
         # That might result in stuff having to be evicted.
@@ -1580,8 +1664,7 @@ def deleteGlobalFile(self, fileStoreID):
         # it gets evicted, and only delete at the back end?
 
         # Pop the file into deleting state owned by us if it exists
-        self.cur.execute('UPDATE files SET state = ?, owner = ? WHERE id = ?', ('deleting', pid, fileStoreID))
-        self.con.commit()
+        self._write([('UPDATE files SET state = ?, owner = ? WHERE id = ?', ('deleting', pid, fileStoreID))])
 
         # Finish the delete if the file is present
         self._executePendingDeletions(self.con, self.cur)
@@ -1751,8 +1834,7 @@ def _removeDeadJobs(cls, con):
         # Now we know which workers are dead.
         # Clear them off of the jobs they had.
         for deadWorker in deadWorkers:
-            cur.execute('UPDATE jobs SET worker = NULL WHERE worker = ?', (deadWorker,))
-            con.commit()
+            cls._staticWrite(con, cur, [('UPDATE jobs SET worker = NULL WHERE worker = ?', (deadWorker,))])
         if len(deadWorkers) > 0:
             logger.debug('Reaped %d dead workers', len(deadWorkers))
 
@@ -1768,8 +1850,7 @@ def _removeDeadJobs(cls, con):
             jobID = row[0]
 
             # Try to own this job
-            cur.execute('UPDATE jobs SET worker = ? WHERE id = ? AND worker IS NULL', (pid, jobID))
-            con.commit()
+            cls._staticWrite(con, cur, [('UPDATE jobs SET worker = ? WHERE id = ? AND worker IS NULL', (pid, jobID))])
 
             # See if we won the race
             cur.execute('SELECT id, tempdir FROM jobs WHERE id = ? AND worker = ?', (jobID, pid))


Test patch:
diff --git a/src/toil/test/src/fileStoreTest.py b/src/toil/test/src/fileStoreTest.py
--- a/src/toil/test/src/fileStoreTest.py
+++ b/src/toil/test/src/fileStoreTest.py
@@ -32,6 +32,7 @@
 from toil.realtimeLogger import RealtimeLogger
 
 import collections
+import datetime
 import inspect
 import logging
 import os
@@ -1071,10 +1072,12 @@ def _deleteLocalFileFn(job, nonLocalDir):
             job.fileStore.deleteLocalFile(localFsID)
             assert not os.path.exists(readBackFile1)
             assert not os.path.exists(readBackFile2)
-            # Try to get a non-FileID
+            # Try to get a non-FileID that doesn't exist.
             try:
                 job.fileStore.readGlobalFile('bogus')
-            except TypeError:
+            except NoSuchFileException:
+                # TODO: We would like to require TypeError, but for Cactus
+                # support we have to accept non-FileIDs.
                 pass
             else:
                 raise RuntimeError("Managed to get a file from a non-FileID")
@@ -1086,6 +1089,67 @@ def _deleteLocalFileFn(job, nonLocalDir):
             else:
                 raise RuntimeError("Managed to read a non-existent file")
 
+        @travis_test
+        def testSimultaneousReadsUncachedStream(self):
+            """
+            Test many simultaneous read attempts on a file created via a stream
+            directly to the job store.
+            """
+            self.options.retryCount = 0
+            self.options.disableChaining = True
+            
+            # Make a file
+            parent = Job.wrapJobFn(self._createUncachedFileStream)
+            # Now make a bunch of children fight over it
+            for i in range(30):
+                parent.addChildJobFn(self._readFileWithDelay, parent.rv())
+
+            Job.Runner.startToil(parent, self.options)
+
+        @staticmethod
+        def _createUncachedFileStream(job):
+            """
+            Create and return a FileID for a non-cached file written via a stream.
+            """
+
+            messageBytes = 'This is a test file\n'.encode('utf-8')
+
+            with job.fileStore.jobStore.writeFileStream() as (out, idString):
+                # Write directly to the job store so the caching file store doesn't even see it.
+                # TODO: If we ever change how the caching file store does its IDs we will have to change this.
+                out.write(messageBytes)
+
+            # Now make a file ID
+            fileID = FileID(idString, len(messageBytes))
+
+            return fileID
+
+        @staticmethod
+        def _readFileWithDelay(job, fileID, cores=0.1, memory=50 * 1024 * 1024, disk=50 * 1024 * 1024):
+            """
+            Read a file from the CachingFileStore with a delay imposed on the download.
+            Should create contention.
+
+            Has low requirements so we can run a lot of copies at once.
+            """
+
+            # Make sure the file store delays
+            # Delay needs to be longer than the timeout for sqlite locking in the file store.
+            job.fileStore.forceDownloadDelay = 120
+
+            readStart = datetime.datetime.now()
+            logger.debug('Begin read at %s', str(readStart))
+            
+            localPath = job.fileStore.readGlobalFile(fileID, cache=True, mutable=True)
+
+            readEnd = datetime.datetime.now()
+            logger.debug('End read at %s: took %f seconds', str(readEnd), (readEnd - readStart).total_seconds())
+
+            with open(localPath, 'rb') as fh:
+                text = fh.read().decode('utf-8').strip()
+            logger.debug('Got file contents: %s', text)
+                
+
 
 class NonCachingFileStoreTestWithFileJobStore(hidden.AbstractNonCachingFileStoreTest):
     jobStoreType = 'file'