Repository: DataBiosphere/toil

Issue:
AWS job store bind bucket can fail with 409 error
Thanks to @joelarmstrong for reporting this issue.
```
DEBUG:toil.batchSystems.mesos.executor:Invoking command: '_toil_worker aws:us-west-2:cactus-jcarmstr-hmrd-mr-pp2 28e435d0-df43-49db-9420-a83fb9491038'
INFO:toil.common:Created the workflow directory at /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S214/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0022/executors/toil-6073/runs/f22c0f68-fb72-4f42-b70e-7c525e347c47/toil-43764132-2f0d-41c5-9984-a2f78d9eb7e2
Traceback (most recent call last):
  File "/usr/local/bin/_toil_worker", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 103, in main
    jobStore = Toil.resumeJobStore(jobStoreLocator)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 766, in resumeJobStore
    jobStore.resume()
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 222, in resume
    self._bind(create=False)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 241, in _bind
    versioning=True)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 738, in _bindBucket
    bucket.configure_versioning(True)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 1306, in configure_versioning
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>OperationAborted</Code><Message>A conflicting conditional operation is currently in progress against this resource. Please try again.</Message><RequestId>ABFB7931AAFEB086</RequestId><HostId>O6ur1eYDtMByINPox7L4BsPXAdSfEAOE6Fr0Pu207bBnhsMQoXDU2mxkH8IE4y0kzHZze/Bt3aE=</HostId></Error>
DEBUG:toil.batchSystems.mesos.executor:Sending task status update ...
```
AWS job store bind bucket can fail with 409 error
Thanks to @joelarmstrong for reporting this issue.
```
DEBUG:toil.batchSystems.mesos.executor:Invoking command: '_toil_worker aws:us-west-2:cactus-jcarmstr-hmrd-mr-pp2 28e435d0-df43-49db-9420-a83fb9491038'
INFO:toil.common:Created the workflow directory at /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S214/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0022/executors/toil-6073/runs/f22c0f68-fb72-4f42-b70e-7c525e347c47/toil-43764132-2f0d-41c5-9984-a2f78d9eb7e2
Traceback (most recent call last):
  File "/usr/local/bin/_toil_worker", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 103, in main
    jobStore = Toil.resumeJobStore(jobStoreLocator)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 766, in resumeJobStore
    jobStore.resume()
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 222, in resume
    self._bind(create=False)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 241, in _bind
    versioning=True)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 738, in _bindBucket
    bucket.configure_versioning(True)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 1306, in configure_versioning
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>OperationAborted</Code><Message>A conflicting conditional operation is currently in progress against this resource. Please try again.</Message><RequestId>ABFB7931AAFEB086</RequestId><HostId>O6ur1eYDtMByINPox7L4BsPXAdSfEAOE6Fr0Pu207bBnhsMQoXDU2mxkH8IE4y0kzHZze/Bt3aE=</HostId></Error>
DEBUG:toil.batchSystems.mesos.executor:Sending task status update ...
```


Discussion:
2 potential solutions to this: 
1) exponential back-off to make sure the configure_versioning call will eventually succeed
2) since this call is in the worker (where the job store has already been created) we should be able to just skip this call.

Unless I'm missing something 2 is the way to go since the call doesn't seem necessary 
2 potential solutions to this: 
1) exponential back-off to make sure the configure_versioning call will eventually succeed
2) since this call is in the worker (where the job store has already been created) we should be able to just skip this call.

Unless I'm missing something 2 is the way to go since the call doesn't seem necessary 

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -697,12 +697,15 @@ def bucket_creation_pending(e):
                 else:
                     if self.__getBucketRegion(bucket) != self.region:
                         raise BucketLocationConflictException(self.__getBucketRegion(bucket))
-                if versioning:
+                if versioning and not bucketExisted:
+                    # only call this method on bucket creation
                     bucket.configure_versioning(True)
                 else:
+                    # now test for versioning consistency
+                    # we should never see any of these errors since 'versioning' should always be true
                     bucket_versioning = self.__getBucketVersioning(bucket)
-                    if bucket_versioning is True:
-                        assert False, 'Cannot disable bucket versioning if it is already enabled'
+                    if bucket_versioning != versioning:
+                        assert False, 'Cannot modify versioning on existing bucket'
                     elif bucket_versioning is None:
                         assert False, 'Cannot use a bucket with versioning suspended'
                 if bucketExisted:


Test patch: