Repository: DataBiosphere/toil

Issue:
S3 multipart uploader crashed for large file
I tried to use the AWS jobStore's exportFile() to export a 30G file to an S3 bucket.  Got this error (same code worked for smaller files)

https://github.com/BD2KGenomics/toil-vg/issues/110#issuecomment-276095314
S3 multipart uploader crashed for large file
I tried to use the AWS jobStore's exportFile() to export a 30G file to an S3 bucket.  Got this error (same code worked for smaller files)

https://github.com/BD2KGenomics/toil-vg/issues/110#issuecomment-276095314


Discussion:
Note: boto3 has an API to do multipart uploads robustly in one call. 
If I recall correctly toil actually uses boto's multipart upload. It looks from your stack trace like this is a problem with the simpledb part that's needed to make the job store strongly consistent.
I was trying to export an arbitrary file from the job to an S3 store inside
the job.  Perhaps that's an abuse of exportFile.  I blamed it on multipart
because it was only crashing while uploading huge files.  My apologies if
that's not the issue at heart -- I didn't really read the toil internals
too closely.

Do you have any recommendations on how to copy a file from, say, a
temporary work dir within a toil job to an arbitrary output store?  There's
code in toil_vg to do this (IOStore class, which was inherited from Adam a
while back), but I was hoping to replace it with Toil code (as IOStore
didn't support multipart uploads).  I want to checkpoint certain
intermediate files during the workflow without waiting for toil to finish
running (when I can get them with a toil.export).

Anyway, I ended up adding multipart support to IOStore in the meantime very
simply using boto3, hence my API comment.  Toil indeed uses the boto but
still takes dozens of lines of finicky looking code.  boto3 gives a higher
level API to do it in a one-liner.

https://github.com/glennhickey/toil-vg/blob/issues/54-refactor-toillib/src/toil_vg/iostore.py#L636

As an AWS newbie, it was appealing to me, but if the current version aint'
broke....


On Tue, Jan 31, 2017 at 8:08 PM, Joel Armstrong <notifications@github.com>
wrote:

> If I recall correctly toil actually uses boto's multipart upload. It looks
> from your stack trace like this is a problem with the simpledb part that's
> needed to make the job store strongly consistent.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1469#issuecomment-276544508>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA2_7h-drLFIxDpBkAJ2M-1_fABudTC8ks5rX9shgaJpZM4Ly-Sr>
> .
>

Well, the current version seems to be broke. I just meant, given where it failed, it looks like the multipart upload part actually technically *worked*, or would have worked, but it failed because of metadata issues.

(Toil devs--is the simpledb consistency stuff even needed now, given that file updates aren't allowed anymore? I think S3 has read-after-creation consistency.)

Do you have a version of the code that failed that you can link to? I'm interested in getting this tracked down, because I want to do exactly the same type of file-checkpointing thing for Cactus.

edit: not so I can run the pipeline to reproduce the bug--just so I can see the control flow.
IIRC we still need SDB since job files can still be updated. An S3-only job store might be possible, we probably won't be exploring that in the near future though.

I'm also interested in the code that caused this error, hopefully the bug isn't too hard to track down.
Code's here
https://github.com/glennhickey/toil-vg/blob/issues/110-fix-iostore/src/toil_vg/vg_common.py#L264-L270

Was run on a 30G local file that was written to the job store a few lines
above that link.   This code worked fine for smaller files (tried about 3G
successfully I think).  The sleep call is a bit hokey but was necessary
(and maybe related to my error?).  Run on toil 3.5.0a1.dev251 on an
r3.8xlarge cluster in mesos mode with --workDir on an EBS volume.

But yeah, a simple, standard way for checkpointing intermediate files would
be extremely helpful.

On Wed, Feb 1, 2017 at 1:56 PM, Christopher Ketchum <
notifications@github.com> wrote:

> IIRC we still need SDB since job files can still be updated. An S3-only
> job store might be possible, we probably won't be exploring that in the
> near future though.
>
> I'm also interested in the code that caused this error, hopefully the bug
> isn't too hard to track down.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1469#issuecomment-276746666>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA2_7nu8pcYhmup8Ufk8QOXtotg_Mokdks5rYNVAgaJpZM4Ly-Sr>
> .
>

Thanks, I'll look into this. Out of curiosity, why are you checkpointing files?
Another thought - you aren't using `--disableCaching` are you? With caching enabled writeGlobalFile actually just queues the upload, the actual upload is asynchronous. I don't think there is any logic to consolidate caching with the imprt/exportFile methods, so that code probably works when the upload finishes in < 10 seconds.
Note: boto3 has an API to do multipart uploads robustly in one call. 
If I recall correctly toil actually uses boto's multipart upload. It looks from your stack trace like this is a problem with the simpledb part that's needed to make the job store strongly consistent.
I was trying to export an arbitrary file from the job to an S3 store inside
the job.  Perhaps that's an abuse of exportFile.  I blamed it on multipart
because it was only crashing while uploading huge files.  My apologies if
that's not the issue at heart -- I didn't really read the toil internals
too closely.

Do you have any recommendations on how to copy a file from, say, a
temporary work dir within a toil job to an arbitrary output store?  There's
code in toil_vg to do this (IOStore class, which was inherited from Adam a
while back), but I was hoping to replace it with Toil code (as IOStore
didn't support multipart uploads).  I want to checkpoint certain
intermediate files during the workflow without waiting for toil to finish
running (when I can get them with a toil.export).

Anyway, I ended up adding multipart support to IOStore in the meantime very
simply using boto3, hence my API comment.  Toil indeed uses the boto but
still takes dozens of lines of finicky looking code.  boto3 gives a higher
level API to do it in a one-liner.

https://github.com/glennhickey/toil-vg/blob/issues/54-refactor-toillib/src/toil_vg/iostore.py#L636

As an AWS newbie, it was appealing to me, but if the current version aint'
broke....


On Tue, Jan 31, 2017 at 8:08 PM, Joel Armstrong <notifications@github.com>
wrote:

> If I recall correctly toil actually uses boto's multipart upload. It looks
> from your stack trace like this is a problem with the simpledb part that's
> needed to make the job store strongly consistent.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1469#issuecomment-276544508>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA2_7h-drLFIxDpBkAJ2M-1_fABudTC8ks5rX9shgaJpZM4Ly-Sr>
> .
>

Well, the current version seems to be broke. I just meant, given where it failed, it looks like the multipart upload part actually technically *worked*, or would have worked, but it failed because of metadata issues.

(Toil devs--is the simpledb consistency stuff even needed now, given that file updates aren't allowed anymore? I think S3 has read-after-creation consistency.)

Do you have a version of the code that failed that you can link to? I'm interested in getting this tracked down, because I want to do exactly the same type of file-checkpointing thing for Cactus.

edit: not so I can run the pipeline to reproduce the bug--just so I can see the control flow.
IIRC we still need SDB since job files can still be updated. An S3-only job store might be possible, we probably won't be exploring that in the near future though.

I'm also interested in the code that caused this error, hopefully the bug isn't too hard to track down.
Code's here
https://github.com/glennhickey/toil-vg/blob/issues/110-fix-iostore/src/toil_vg/vg_common.py#L264-L270

Was run on a 30G local file that was written to the job store a few lines
above that link.   This code worked fine for smaller files (tried about 3G
successfully I think).  The sleep call is a bit hokey but was necessary
(and maybe related to my error?).  Run on toil 3.5.0a1.dev251 on an
r3.8xlarge cluster in mesos mode with --workDir on an EBS volume.

But yeah, a simple, standard way for checkpointing intermediate files would
be extremely helpful.

On Wed, Feb 1, 2017 at 1:56 PM, Christopher Ketchum <
notifications@github.com> wrote:

> IIRC we still need SDB since job files can still be updated. An S3-only
> job store might be possible, we probably won't be exploring that in the
> near future though.
>
> I'm also interested in the code that caused this error, hopefully the bug
> isn't too hard to track down.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1469#issuecomment-276746666>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA2_7nu8pcYhmup8Ufk8QOXtotg_Mokdks5rYNVAgaJpZM4Ly-Sr>
> .
>

Thanks, I'll look into this. Out of curiosity, why are you checkpointing files?
Another thought - you aren't using `--disableCaching` are you? With caching enabled writeGlobalFile actually just queues the upload, the actual upload is asynchronous. I don't think there is any logic to consolidate caching with the imprt/exportFile methods, so that code probably works when the upload finishes in < 10 seconds.

Resolving patch:
diff --git a/src/toil/fileStore.py b/src/toil/fileStore.py
--- a/src/toil/fileStore.py
+++ b/src/toil/fileStore.py
@@ -248,7 +248,7 @@ def importFile(self, srcUrl, sharedFileName=None):
         return self.jobStore.importFile(srcUrl, sharedFileName=sharedFileName)
 
     def exportFile(self, jobStoreFileID, dstUrl):
-        self.jobStore.exportFile(jobStoreFileID, dstUrl)
+        raise NotImplementedError()
 
     # A utility method for accessing filenames
     def _resolveAbsoluteLocalPath(self, filePath):
@@ -721,6 +721,13 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
                                                               0.0, False)
         return localFilePath
 
+    def exportFile(self, jobStoreFileID, dstUrl):
+        while jobStoreFileID in self._pendingFileWrites:
+            # The file is still being writting to the job store - wait for this process to finish prior to
+            # exporting it
+            time.sleep(1)
+        self.jobStore.exportFile(jobStoreFileID, dstUrl)
+
     def readGlobalFileStream(self, fileStoreID):
         if fileStoreID in self.filesToDelete:
             raise RuntimeError(
@@ -1657,6 +1664,9 @@ def readGlobalFileStream(self, fileStoreID):
         with self.jobStore.readFileStream(fileStoreID) as f:
             yield f
 
+    def exportFile(self, jobStoreFileID, dstUrl):
+        self.jobStore.exportFile(jobStoreFileID, dstUrl)
+
     def deleteLocalFile(self, fileStoreID):
         try:
             localFilePaths = self.localFileMap.pop(fileStoreID)


Test patch:
diff --git a/src/toil/test/src/fileStoreTest.py b/src/toil/test/src/fileStoreTest.py
--- a/src/toil/test/src/fileStoreTest.py
+++ b/src/toil/test/src/fileStoreTest.py
@@ -14,6 +14,8 @@
 
 from __future__ import absolute_import
 from __future__ import print_function
+
+import filecmp
 from abc import abstractmethod, ABCMeta
 from struct import pack, unpack
 from uuid import uuid4
@@ -946,6 +948,21 @@ def _multipleFileReader(job, diskMB, fsID, maxWriteFile):
             # the file
             time.sleep(3)
 
+        @staticmethod
+        def _writeExportGlobalFile(job):
+            fileName = os.path.join(job.fileStore.getLocalTempDir(), 'testfile')
+            with open(fileName, 'wb') as f:
+                f.write(os.urandom(1024 * 30000)) # 30 Mb
+            outputFile = os.path.join(job.fileStore.getLocalTempDir(), 'exportedFile')
+            job.fileStore.exportFile(job.fileStore.writeGlobalFile(fileName), 'File://' + outputFile)
+            assert filecmp.cmp(fileName, outputFile)
+
+        def testFileStoreExportFile(self):
+            # Tests that files written to job store can be immediately exported
+            # motivated by https://github.com/BD2KGenomics/toil/issues/1469
+            root = Job.wrapJobFn(self._writeExportGlobalFile)
+            Job.Runner.startToil(root, self.options)
+
         # Testing for the return of file sizes to the sigma job pool.
         def testReturnFileSizes(self):
             """