Repository: cloud-custodian/cloud-custodian

Issue:
aws - s3 block public object ACLs erroring out
After deploying [Block Public S3 Objects ACLs](https://cloudcustodian.io/docs/usecases/s3denypublicobjectacls.html) successfully and confirming that a lambda function named "custodian-s3-deny-public-object-acl" was created, I'm getting the below error in CloudWatch logs. The bucket itself is also still publicly accessible.
 

> [ERROR]	2019-01-14T18:46:57.643Z	67ff7b4d-f27b-43ac-b899-09ccdab191c2	Error while executing policy
> Traceback (most recent call last):
> File "/var/task/c7n/policy.py", line 273, in run
> results = a.process(resources)
> File "/var/task/c7n/resources/s3.py", line 725, in process
> results += filter(None, [f.result()])
> File "/var/runtime/concurrent/futures/_base.py", line 455, in result
> return self.__get_result()
> File "/var/runtime/concurrent/futures/thread.py", line 63, in run
> result = self.fn(*self.args, **self.kwargs)
> File "/var/task/c7n/resources/s3.py", line 1084, in process_bucket
> current = {s['Sid']: s for s in policy.get('Statement', [])}
> File "/var/task/c7n/resources/s3.py", line 1084, in <dictcomp>
> current = {s['Sid']: s for s in policy.get('Statement', [])}
> KeyError: u'Sid'

Here is the policy that was deployed:
```
policies:
  - name: s3-deny-public-object-acl
    comment: Identifies S3 buckets and ensures that they have a restriction in them to not allow objects to be written with public access
    resource: s3
    description: |
      Appends a bucket policy statement to all existing s3 buckets to
      deny anyone from setting s3 objects in the bucket to public-read,
      public-read-write, or any authenticated AWS user.
    actions:
      - type: set-statements
        statements:
           - Sid: "DenyS3PublicObjectACL"
             Effect: "Deny"
             Action: "s3:PutObjectAcl"
             Principal: "*"
             Resource:
                - "arn:aws:s3:::{bucket_name}/*"
                - "arn:aws:s3:::{bucket_name}"
             Condition:
               StringEqualsIgnoreCaseIfExists:
                  's3:x-amz-acl':
                      - "public-read"
                      - "public-read-write"
                      - "authenticated-read"
```



Discussion:
I am seeing the exact same issue
what form of public block are you deploying? there's four options and they can be applied at bucket or account level.

- ignore acls
- don't allow new acls

- block public policies put
- ignore public bucket policies

I'm deploying all 4 options at the account level.

On Thu, Jan 31, 2019, 4:40 PM Kapil Thangavelu <notifications@github.com
wrote:

> what form of public block are you deploying? there's four options and they
> can be applied at bucket or account level.
>
>    -
>
>    ignore acls
>    -
>
>    don't allow new acls
>    -
>
>    block public policies put
>    -
>
>    ignore public bucket policies
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/cloud-custodian/cloud-custodian/issues/3362#issuecomment-459516603>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AqtZ8G6x2w-zzaj0qMJjhWNcsrrLOoWGks5vI2LUgaJpZM4Z_pRL>
> .
>

I'm running into this too, just trying to add a statement to a bucket policy.  It looks like it's failing because the existing bucket policy statements don't necessarily all have the `Sid` field.  I can save a bucket policy in the AWS console with two statements, neither of which have a `Sid` field.   So, while you might require a `Sid` field in the cloud custodian policy, you can't assume it will have one in the existing bucket policy.

Resolving patch:
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -1101,19 +1101,23 @@ class SetPolicyStatement(BucketActionBase):
     def process_bucket(self, bucket):
         policy = bucket.get('Policy') or '{}'
 
-        fmtargs = self.get_std_format_args(bucket)
+        target_statements = format_string_values(
+            copy.deepcopy({s['Sid']: s for s in self.data.get('statements', [])}),
+            **self.get_std_format_args(bucket))
 
         policy = json.loads(policy)
-        current = {s['Sid']: s for s in policy.get('Statement', [])}
-        new = copy.deepcopy(current)
-        additional = {s['Sid']: s for s in self.data.get('statements', [])}
-        additional = format_string_values(additional, **fmtargs)
-        new.update(additional)
-        if new == current:
+        bucket_statements = policy.setdefault('Statement', [])
+
+        for s in bucket_statements:
+            if s.get('Sid') not in target_statements:
+                continue
+            if s == target_statements[s['Sid']]:
+                target_statements.pop(s['Sid'])
+
+        if not target_statements:
             return
 
-        statements = list(new.values())
-        policy['Statement'] = statements
+        bucket_statements.extend(target_statements.values())
         policy = json.dumps(policy)
 
         s3 = bucket_client(local_session(self.manager.session_factory), bucket)


Test patch: