Repository: DataBiosphere/toil

Issue:
AWS jobStore's getEmptyFileStoreID() doesn't create an empty file
This isn't in the public API, I think, but it's necessary for some things.

getEmptyFileStoreID() does return an ID, but trying to actually read that empty file gives this:
```
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/tmp/tmpuWaLIg/6424e67a3d26124c4ae9b187785a8c18/cactus/pipeline/ktserverToil.py", line 28, in start
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        path = job.fileStore.readGlobalFile(snapshotExportID)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1658, in readGlobalFile
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        self.jobStore.readFile(fileStoreID, localFilePath)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 589, in readFile
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        info.download(localFilePath)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1242, in download
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        assert False
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e    AssertionError
```

In FileJobStore this just "downloads" an empty file--which is the correct behavior IMO. I'll add this to the tests and see if any other jobstores have this issue as well.
AWS jobStore's getEmptyFileStoreID() doesn't create an empty file
This isn't in the public API, I think, but it's necessary for some things.

getEmptyFileStoreID() does return an ID, but trying to actually read that empty file gives this:
```
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/tmp/tmpuWaLIg/6424e67a3d26124c4ae9b187785a8c18/cactus/pipeline/ktserverToil.py", line 28, in start
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        path = job.fileStore.readGlobalFile(snapshotExportID)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1658, in readGlobalFile
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        self.jobStore.readFile(fileStoreID, localFilePath)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 589, in readFile
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        info.download(localFilePath)
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e      File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1242, in download
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e        assert False
d1f10c49-bcf8-4f55-aff3-7dd807be3d2e    AssertionError
```

In FileJobStore this just "downloads" an empty file--which is the correct behavior IMO. I'll add this to the tests and see if any other jobstores have this issue as well.


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -344,6 +344,9 @@ def delete(self, jobStoreID):
 
     def getEmptyFileStoreID(self, jobStoreID=None):
         info = self.FileInfo.create(jobStoreID)
+        with info.uploadStream() as _:
+            # Empty
+            pass
         info.save()
         log.debug("Created %r.", info)
         return info.fileID
diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -375,10 +375,8 @@ def updateFileStream(self, jobStoreFileID):
 
     def getEmptyFileStoreID(self, jobStoreID=None):
         jobStoreFileID = self._newFileID()
-        for attempt in retry_azure(timeout=45):
-            with attempt:
-                self.files.put_blob(blob_name=jobStoreFileID, blob='',
-                                    x_ms_blob_type='BlockBlob')
+        with self._uploadStream(jobStoreFileID, self.files) as _:
+            pass
         self._associateFileWithJob(jobStoreFileID, jobStoreID)
         return jobStoreFileID
 


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -738,6 +738,16 @@ def testDestructionIdempotence(self):
             cleaner = self._createJobStore()
             cleaner.destroy()
 
+        def testEmptyFileStoreIDIsReadable(self):
+            """Simply creates an empty fileStoreID and attempts to read from it."""
+            id = self.master.getEmptyFileStoreID()
+            fh, path = tempfile.mkstemp()
+            try:
+                self.master.readFile(id, path)
+                self.assertTrue(os.path.isfile(path))
+            finally:
+                os.unlink(path)
+
         def _largeLogEntrySize(self):
             """
             Sub-classes may want to override these in order to maximize test coverage