Repository: DataBiosphere/toil

Issue:
AWSJobStoreTest.testSharedFiles can't pass anymore because Amazon turned off public access and ACL support on new buckets
It has been [observed](https://ucsc-ci.com/databiosphere/toil/-/jobs/39488) to fail thusly:

```
_______________________ AWSJobStoreTest.testSharedFiles ________________________
[gw2] linux -- Python 3.9.15 /builds/databiosphere/toil/venv/bin/python
Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/jobStores/jobStoreTest.py", line 374, in testSharedFiles
    self.assertUrl(jobstore1.get_shared_public_url('nonEncrypted'))
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 660, in get_shared_public_url
    return self.get_public_url(self._shared_file_id(shared_file_name))
  File "/builds/databiosphere/toil/src/toil/lib/retry.py", line 292, in call
    return func(*args, **kwargs)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 634, in get_public_url
    self.filesBucket.Object(compat_bytes(jobStoreFileID)).Acl().put(ACL='public-read')
  File "/builds/databiosphere/toil/venv/lib/python3.9/site-packages/boto3/resources/factory.py", line 580, in do_action
    response = action(self, *args, **kwargs)
  File "/builds/databiosphere/toil/venv/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/builds/databiosphere/toil/venv/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/builds/databiosphere/toil/venv/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1323)
┆Issue Number: TOIL-1323



Discussion:
This is #4235 back again to haunt us I think?
No, it looks like this is a new problem, caused by https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/

Amazon now turns on the disable-public-access feature for new buckets, and also sets them to not use ACLs anymore (like the ACLs we use to mark files public so we can get public links to them) and instead wants you to use IAM policies.

We're almost certainly not going to revise the AWS job store logic to write custom policies that open access to particular objects. I think we need to add code to make the AWS API calls to restore the old behavior here. That is probably:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_public_access_block.html#S3.Client.delete_public_access_block

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_bucket_ownership_controls.html


Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -58,6 +58,7 @@
                                   WritablePipe)
 from toil.lib.aws.session import establish_boto3_session
 from toil.lib.aws.utils import (create_s3_bucket,
+                                enable_public_objects,
                                 flatten_tags,
                                 get_bucket_region,
                                 get_object_for_url,
@@ -752,6 +753,10 @@ def bucket_retry_predicate(error):
                                 flat_tags = flatten_tags(tags)
                                 bucket_tagging = self.s3_resource.BucketTagging(bucket_name)
                                 bucket_tagging.put(Tagging={'TagSet': flat_tags})
+
+                            # Configure bucket so that we can make objects in
+                            # it public, which was the historical default. 
+                            enable_public_objects(bucket_name)
                         elif block:
                             raise
                         else:
diff --git a/src/toil/lib/aws/utils.py b/src/toil/lib/aws/utils.py
--- a/src/toil/lib/aws/utils.py
+++ b/src/toil/lib/aws/utils.py
@@ -56,6 +56,7 @@
     from mypy_boto3_sdb import SimpleDBClient
 except ImportError:
     BotoServerError = None  # type: ignore
+    ClientError = None  # type: ignore
     # AWS/boto extra is not installed
 
 logger = logging.getLogger(__name__)
@@ -217,6 +218,43 @@ def create_s3_bucket(
         )
     return bucket
 
+@retry(errors=[ClientError])
+def enable_public_objects(bucket_name: str) -> None:
+    """
+    Enable a bucket to contain objects which are public.
+
+    This adjusts the bucket's Public Access Block setting to not block all
+    public access, and also adjusts the bucket's Object Ownership setting to a
+    setting which enables object ACLs.
+
+    Does *not* touch the *account*'s Public Access Block setting, which can
+    also interfere here. That is probably best left to the account
+    administrator.
+
+    This configuration used to be the default, and is what most of Toil's code
+    is written to expect, but it was changed so that new buckets default to the
+    more restrictive setting
+    <https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>,
+    with the expectation that people would write IAM policies for the buckets
+    to allow public access if needed. Toil expects to be able to make arbitrary
+    objects in arbitrary places public, and naming them all in an IAM policy
+    would be a very awkward way to do it. So we restore the old behavior.
+    """
+
+    s3_client = cast(S3Client, session.client('s3'))
+
+    # Even though the new default is for public access to be prohibited, this
+    # is implemented by adding new things attached to the bucket. If we remove
+    # those things the bucket will default to the old defaults. See
+    # <https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/>.
+
+    # Stop blocking public access
+    s3_client.delete_public_access_block(Bucket=bucket_name)
+
+    # Stop using an ownership controls setting that prohibits ACLs.
+    s3_client.delete_bucket_ownership_controls(Bucket=bucket_name)
+
+
 def get_bucket_region(bucket_name: str, endpoint_url: Optional[str] = None, only_strategies: Optional[Set[int]] = None) -> str:
     """
     Get the AWS region name associated with the given S3 bucket.


Test patch: