Repository: DataBiosphere/toil

Issue:
Allow cleaning the S3 jobstore even if there are versioning inconsistencies
We currently check that the versioning of the s3 bucket is consistent when binding the bucket [here](https://github.com/DataBiosphere/toil/blob/3e950e4326ed55bd969eda15217aeab88a9d3d26/src/toil/jobStores/aws/jobStore.py#L744). However, this check can fail if the s3 bucket is in a 'strange' state (I've seen this occasionally happening where the s3 bucket has not been fully deleted; perhaps an internal API call failed?). When this happens, then it's not possible to clean the jobstore and one needs to manually delete the bucket outside of Toil.

I don't think this check is needed when cleaning the s3 jobstore as we're about to delete the bucket anyway.

Proposal is to add a flag to ignore this check only when cleaning the jobstore.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-439)
┆Issue Number: TOIL-439



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -152,7 +152,7 @@ def resume(self):
         self._bind(create=False)
         super(AWSJobStore, self).resume()
 
-    def _bind(self, create=False, block=True):
+    def _bind(self, create=False, block=True, check_versioning_consistency=True):
         def qualify(name):
             assert len(name) <= self.maxNameLen
             return self.namePrefix + self.nameSeparator + name
@@ -168,7 +168,8 @@ def qualify(name):
             self.filesBucket = self._bindBucket(qualify('files'),
                                                 create=create,
                                                 block=block,
-                                                versioning=True)
+                                                versioning=True,
+                                                check_versioning_consistency=check_versioning_consistency)
 
     @property
     def _registered(self):
@@ -671,7 +672,8 @@ def _connectS3(self):
             raise ValueError("Could not connect to S3. Make sure '%s' is a valid S3 region." % self.region)
         return s3
 
-    def _bindBucket(self, bucket_name, create=False, block=True, versioning=False):
+    def _bindBucket(self, bucket_name, create=False, block=True, versioning=False,
+                    check_versioning_consistency=True):
         """
         Return the Boto Bucket object representing the S3 bucket with the given name. If the
         bucket does not exist and `create` is True, it will be created.
@@ -737,7 +739,7 @@ def bucket_creation_pending(e):
                 if versioning and not bucketExisted:
                     # only call this method on bucket creation
                     bucket.configure_versioning(True)
-                else:
+                elif check_versioning_consistency:
                     # now test for versioning consistency
                     # we should never see any of these errors since 'versioning' should always be true
                     bucket_versioning = self.__getBucketVersioning(bucket)
@@ -1264,7 +1266,7 @@ def destroy(self):
         # FIXME: Destruction of encrypted stores only works after initialize() or .resume()
         # See https://github.com/BD2KGenomics/toil/issues/1041
         try:
-            self._bind(create=False, block=False)
+            self._bind(create=False, block=False, check_versioning_consistency=False)
         except BucketLocationConflictException:
             # If the unique jobstore bucket name existed, _bind would have raised a
             # BucketLocationConflictException before calling destroy.  Calling _bind here again


Test patch: