Repository: DataBiosphere/toil

Issue:
Flaky versioning on S3 buckets is not handled well
We try to make sure our S3 buckets have versioning enabled before uploading stuff to them, but I observed a test failure in https://ucsc-ci.com/databiosphere/toil/-/jobs/17308 where that didn't work. We were told that versioning was enabled, but we got back something without a `'VersionId'`:

```
Test 58 failed: toil-cwl-runner --clean=always --logDebug --statusWait=10 --setEnv=SINGULARITY_DOCKER_HUB_MIRROR --batchSystem=kubernetes --outdir=/tmp/tmpzgzt8ji7 tests/iwdr-entry.cwl tests/string-job.json
Test if trailing newline is present in file entry in InitialWorkDir
Returned non-zero
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'oauthlib', 'requests_oauthlib', 'google', 'requests', 'bcdocs', 'cachecontrol', 'salad', 'kubernetes', 'prov', 'dill', 's3transfer', 'docker', 'pkg_resources', 'pyasn1', 'concurrent', 'boto', 'boto3', 'humanfriendly', 'charset_normalizer', 'rsa', 'setuptools', 'botocore', 'galaxy', 'urllib3', 'websocket', 'rdflib'}
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.statsAndLogging] Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Instantiating <class 'toil.jobStores.aws.jobStore.AWSJobStore'> for region us-west-2 and name prefix 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5'
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil] Attempting to read cached credentials from /root/.cache/aws/cached_temporary_credentials.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil] /root/.cache/aws/cached_temporary_credentials is empty. Credentials are not temporary.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Binding to job store domain 'toil-registry'.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Binding to job store domain 'toil-registry'.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Binding to job store domain 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--jobs'.
[2022-06-08T04:20:03+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Binding to job store domain 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files'.
[2022-06-08T04:20:04+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Binding to job store bucket 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files'.
[2022-06-08T04:20:04+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Bucket 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files' does not exist.
[2022-06-08T04:20:04+0000] [MainThread] [D] [toil.lib.aws.utils] Creating bucket 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files' in region us-west-2.
[2022-06-08T04:20:06+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Created new job store bucket 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files' with versioning state True.
[2022-06-08T04:20:06+0000] [MainThread] [D] [toil.jobStores.abstractJobStore] The workflow ID is: '2d85cf1a-36d4-4d9a-a3a0-d9515e349456'
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Starting sha1 checksum to match None
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Updating checksum with 2001 bytes
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Starting multipart upload
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Multipart upload started as PTiHHgZZ0fQYuhH5quTUVX7GqeofHvoUu.Q4qw4eoZLqytgADomvrsNiVGhivW._SmjCEW2RaMVHFOBumXknehDD7fOc3B4ohOfl68gIw5NkRcfp8x.I27cS2noYsaCo
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Multipart upload visible as PTiHHgZZ0fQYuhH5quTUVX7GqeofHvoUu.Q4qw4eoZLqytgADomvrsNiVGhivW._SmjCEW2RaMVHFOBumXknehDD7fOc3B4ohOfl68gIw5NkRcfp8x.I27cS2noYsaCo
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Uploading part 1 of 2001 bytes to PTiHHgZZ0fQYuhH5quTUVX7GqeofHvoUu.Q4qw4eoZLqytgADomvrsNiVGhivW._SmjCEW2RaMVHFOBumXknehDD7fOc3B4ohOfl68gIw5NkRcfp8x.I27cS2noYsaCo
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Completed checksum with hash edad22402d05c7d3eb106a750a11e697ef7cbb03 vs. expected None
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Attempting to complete upload...
[2022-06-08T04:20:06+0000] [Thread-1  ] [D] [toil.jobStores.aws.jobStore] Completed upload object of type <class 'dict'>: {'ResponseMetadata': {'RequestId': 'NVZ0R9WBP34MTWTZ', 'HostId': '2mWw2qViGO0mVhT5iTUzTY2t+Bo/WEeUz9E/SP4mYKSI/NMoo6XNfbwU+BcEUWgKz4wyxXOHlOU=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '2mWw2qViGO0mVhT5iTUzTY2t+Bo/WEeUz9E/SP4mYKSI/NMoo6XNfbwU+BcEUWgKz4wyxXOHlOU=', 'x-amz-request-id': 'NVZ0R9WBP34MTWTZ', 'date': 'Wed, 08 Jun 2022 04:20:07 GMT', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'Location': 'https://toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files.s3.us-west-2.amazonaws.com/a77581f6-ad82-5142-acb7-5a6b01eab3fb', 'Bucket': 'toil-cwl-17ecd083-2900-4ad5-aa8a-7320cadac1d5--files', 'Key': 'a77581f6-ad82-5142-acb7-5a6b01eab3fb', 'ETag': '"6db92527d4f8262be04e13eec4394038-1"'}
[2022-06-08T04:20:06+0000] [Thread-1  ] [E] [toil.lib.retry] Got a <class 'KeyError'>: 'VersionId' which is not retriable according to <function retryable_s3_errors at 0x7ff30e34d310>
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/builds/databiosphere/toil/src/toil/lib/threading.py", line 71, in run
    self.tryRun()
  File "/builds/databiosphere/toil/src/toil/lib/threading.py", line 77, in tryRun
    super().run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/builds/databiosphere/toil/src/toil/jobStores/utils.py", line 92, in _reader
    self.readFrom(readable)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 1282, in readFrom
    info.version = completed['VersionId']
KeyError: 'VersionId'
Traceback (most recent call last):
  File "/builds/databiosphere/toil/venv/bin/toil-cwl-runner", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil-cwl-runner')())
  File "/builds/databiosphere/toil/src/toil/cwl/cwltoil.py", line 3435, in main
    with Toil(options) as toil:
  File "/builds/databiosphere/toil/src/toil/common.py", line 873, in __enter__
    jobStore.initialize(config)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 160, in initialize
    super().initialize(config)
  File "/builds/databiosphere/toil/src/toil/jobStores/abstractJobStore.py", line 192, in initialize
    self.write_config()
  File "/builds/databiosphere/toil/src/toil/jobStores/abstractJobStore.py", line 204, in write_config
    pickle.dump(self.__config, fileHandle, pickle.HIGHEST_PROTOCOL)
  File "/usr/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 531, in write_shared_file_stream
    yield writable
  File "/usr/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 1363, in uploadStream
    yield writable
  File "/builds/databiosphere/toil/src/toil/jobStores/utils.py", line 128, in __exit__
    self.thread.join()
  File "/builds/databiosphere/toil/src/toil/lib/threading.py", line 84, in join
    raise_(exc_type, exc_value, traceback)
  File "/builds/databiosphere/toil/src/toil/lib/exceptions.py", line 61, in raise_
    raise exc.with_traceback(traceback)
  File "/builds/databiosphere/toil/src/toil/lib/threading.py", line 71, in run
    self.tryRun()
  File "/builds/databiosphere/toil/src/toil/lib/threading.py", line 77, in tryRun
    super().run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/builds/databiosphere/toil/src/toil/jobStores/utils.py", line 92, in _reader
    self.readFrom(readable)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 1282, in readFrom
    info.version = completed['VersionId']
KeyError: 'VersionId'
```

We're supposed to handle the case where we don't get a version ID and sit there and poll for the version ID to appear:

https://github.com/DataBiosphere/toil/blob/017c962bda289af0c488c773f36e249cd9a40c0b/src/toil/jobStores/aws/jobStore.py#L1285-L1295

But that doesn't actually happen because we crash when the key isn't present. We need to be using `get()` here.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1180)
┆friendlyId: TOIL-1180



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -1279,7 +1279,7 @@ def readFrom(self, readable):
 
                                     logger.debug('Completed upload object of type %s: %s', str(type(completed)),
                                                  repr(completed))
-                                    info.version = completed['VersionId']
+                                    info.version = completed.get('VersionId')
                                     logger.debug('Completed upload with version %s', str(info.version))
 
                             if info.version is None:


Test patch: