Repository: cloud-custodian/cloud-custodian

Issue:
aws - account enable-cloudtrail action govcloud compatibility
This is cli only I am not using lambda.

Running policy cloudtrail-enable-GovCloud.yml against account ACCOUNT

17:36:46			2019-02-13 22:36:46,765: custodian.policy:INFO policy: cloudtrail-enable resource:account region:us-gov-west-1 count:1 time:0.99
17:36:47			2019-02-13 22:36:47,440: custodian.output:ERROR Error while executing policy
17:36:47			Traceback (most recent call last):
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
17:36:47			    results = a.process(resources)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 702, in process
17:36:47			    s3client.put_bucket_policy(Bucket=bucket_name, Policy=policy_json)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 357, in _api_call
17:36:47			    return self._make_api_call(operation_name, kwargs)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 661, in _make_api_call
17:36:47			    raise error_class(parsed_response, operation_name)
17:36:47			ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: Policy has invalid resource
17:36:47			2019-02-13 22:36:47,441: custodian.commands:ERROR Error while executing policy cloudtrail-enable, continuing
17:36:47			Traceback (most recent call last):
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/commands.py", line 237, in run
17:36:47			    policy()
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 870, in __call__
17:36:47			    resources = mode.run()
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
17:36:47			    results = a.process(resources)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 702, in process
17:36:47			    s3client.put_bucket_policy(Bucket=bucket_name, Policy=policy_json)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 357, in _api_call
17:36:47			    return self._make_api_call(operation_name, kwargs)
17:36:47			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 661, in _make_api_call
17:36:47			    raise error_class(parsed_response, operation_name)
17:36:47			ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: Policy has invalid resource
17:36:49			2019-02-13 22:36:49,621: custodian.policy:INFO policy: cloudtrail-enable resource:account region:us-gov-east-1 count:1 time:0.83
17:36:49			2019-02-13 22:36:49,919: custodian.output:ERROR Error while executing policy
17:36:49			Traceback (most recent call last):
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
17:36:49			    results = a.process(resources)
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 692, in process
17:36:49			    raise ce
17:36:49			ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-gov-west-1 location constraint is incompatible for the region specific endpoint this request was sent to.
17:36:49			2019-02-13 22:36:49,920: custodian.commands:ERROR Error while executing policy cloudtrail-enable, continuing
17:36:49			Traceback (most recent call last):
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/commands.py", line 237, in run
17:36:49			    policy()
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 870, in __call__
17:36:49			    resources = mode.run()
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
17:36:49			    results = a.process(resources)
17:36:49			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 692, in process
17:36:49			    raise ce
17:36:49			ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-gov-west-1 location constraint is incompatible for the region specific endpoint this request was sent to.




The Policy:
policies:
  - name: "cloudtrail-enable"
    resource: account
    comments: |
      This policy will determine if cloudtrail is enabled on the
      account and enable it if it isnt.
    filters:
      - type: check-cloudtrail
        #global-events: true
        multi-region: true
        running: true
    actions:
      - type: enable-cloudtrail
        trail: cc-trail
        bucket: "${ACCOUNT}-cloudtrail"
        bucket-region: us-gov-west-1


Discussion:
hmm.. so which region was the lambda executing in? was it executing in the same region and partition (gov-cloud) that it was trying to create the bucket in? typically within a partition  (say default), you can address and create buckets across regions with location constraint afaicr. However cross partition addressing typically doesn't work.  its possible we need to explicitly construct an s3 client to the end point destination for the bucket, but the default region (if the lambda was executing in the target region) should have also worked, hence the questions.
Hi Kapil,

I am not using lambda at all. I am executing via c7n cli to both region us-gov-west-1 us-gov-east-1
The s3 bucket lives in  us-gov-west-1

I also notice that GovCloud is very picky and needs region_name if I was generating the STS token via script.  

conn = boto3.client('sts', region_name=region)
iam = boto3.client('iam', aws_access_key_id=token['Credentials']['AccessKeyId'],
                aws_secret_access_key=token['Credentials']['SecretAccessKey'],
                aws_session_token=token['Credentials']['SessionToken'], region_name=region)

My custodian version is 0.8.33.1 I haven't updated to the latest yet. I can if you think thats the issue.
there are some other govcloud compatibility changes in trunk (mostly taking into account partitions when doing arn generation), but that wouldn't affect the issue that your seeing here. so in this context when the error happens, is it executing in us-gov-east-1 when it throws the error, if you execute against just us-gov-west-1 does it error. What's the cli command your using here (no account ids pls)?

Hi Kapil,

I made the script execute on one of the GovCloud and region  us-gov-west-1 I used the same policy as I mention at the beginning. it looks like it creates the s3 bucket fine as I can see it in s3 but then fails to create the trail.

custodian run --profile ${ACCOUNT} --region ${REGION} --cache ${CACHE_DIR}/${POLICY_NAME}/${ACCOUNT}-cache.file --output-dir ${OUTPUT_DIR}/${POLICY_NAME}/${ACCOUNT} <(ACCOUNT=${ACCOUNT} envsubst < ${POLICY_DIR}/${POLICY})


Running policy cloudtrail-enable-GovCloud.yml against account ACCOUNT
07:48:02			2019-02-14 12:48:02,369: custodian.policy:INFO policy: cloudtrail-enable resource:account region:us-gov-west-1 count:1 time:0.89
07:48:03			2019-02-14 12:48:03,043: custodian.output:ERROR Error while executing policy
07:48:03			Traceback (most recent call last):
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
07:48:03			    results = a.process(resources)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 702, in process
07:48:03			    s3client.put_bucket_policy(Bucket=bucket_name, Policy=policy_json)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 357, in _api_call
07:48:03			    return self._make_api_call(operation_name, kwargs)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 661, in _make_api_call
07:48:03			    raise error_class(parsed_response, operation_name)
07:48:03			ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: Policy has invalid resource
07:48:03			2019-02-14 12:48:03,044: custodian.commands:ERROR Error while executing policy cloudtrail-enable, continuing
07:48:03			Traceback (most recent call last):
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/commands.py", line 237, in run
07:48:03			    policy()
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 870, in __call__
07:48:03			    resources = mode.run()
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/policy.py", line 264, in run
07:48:03			    results = a.process(resources)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/c7n/resources/account.py", line 702, in process
07:48:03			    s3client.put_bucket_policy(Bucket=bucket_name, Policy=policy_json)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 357, in _api_call
07:48:03			    return self._make_api_call(operation_name, kwargs)
07:48:03			  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 661, in _make_api_call
07:48:03			    raise error_class(parsed_response, operation_name)
07:48:03			ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: Policy has invalid resource
thanks, that's a slightly different issue, and a bit more obvious to rectify, in that when we do our s3 policy gen for the trail bucket it  wasn't using our arn generators (which are now partition aware) but statically defining the s3 arn prefix. I can send a pr later today that can attempt to address both of these issues. for a multi-region trail, I think you'll need mark the policy as being pinned to `region: us-gov-west-1` as a key at the policy level. so that when its not executed against us-gov-east-1.

Resolving patch:
diff --git a/c7n/resources/account.py b/c7n/resources/account.py
--- a/c7n/resources/account.py
+++ b/c7n/resources/account.py
@@ -28,7 +28,7 @@
 from c7n.filters import Filter, FilterRegistry, ValueFilter
 from c7n.filters.missing import Missing
 from c7n.manager import ResourceManager, resources
-from c7n.utils import local_session, type_schema
+from c7n.utils import local_session, type_schema, generate_arn
 
 from c7n.resources.iam import CredentialReport
 
@@ -577,14 +577,15 @@ def process(self, resources):
                 ccEmailAddresses=self.data.get('notify', []))
 
 
-def cloudtrail_policy(original, bucket_name, account_id):
+def cloudtrail_policy(original, bucket_name, account_id, bucket_region):
     '''add CloudTrail permissions to an S3 policy, preserving existing'''
     ct_actions = [
         {
             'Action': 's3:GetBucketAcl',
             'Effect': 'Allow',
             'Principal': {'Service': 'cloudtrail.amazonaws.com'},
-            'Resource': 'arn:aws:s3:::' + bucket_name,
+            'Resource': generate_arn(
+                service='s3', resource=bucket_name, region=bucket_region),
             'Sid': 'AWSCloudTrailAclCheck20150319',
         },
         {
@@ -595,9 +596,8 @@ def cloudtrail_policy(original, bucket_name, account_id):
             },
             'Effect': 'Allow',
             'Principal': {'Service': 'cloudtrail.amazonaws.com'},
-            'Resource': 'arn:aws:s3:::%s/AWSLogs/%s/*' % (
-                bucket_name, account_id
-            ),
+            'Resource': generate_arn(
+                service='s3', resource=bucket_name, region=bucket_region),
             'Sid': 'AWSCloudTrailWrite20150319',
         },
     ]
@@ -680,7 +680,7 @@ def process(self, accounts):
         kms = self.data.get('kms', False)
         kms_key = self.data.get('kms-key', '')
 
-        s3client = session.client('s3')
+        s3client = session.client('s3', region_name=bucket_region)
         try:
             s3client.create_bucket(
                 Bucket=bucket_name,
@@ -697,7 +697,8 @@ def process(self, accounts):
             current_policy = None
 
         policy_json = cloudtrail_policy(
-            current_policy, bucket_name, self.manager.config.account_id)
+            current_policy, bucket_name,
+            self.manager.config.account_id, bucket_region)
 
         s3client.put_bucket_policy(Bucket=bucket_name, Policy=policy_json)
         trails = client.describe_trails().get('trailList', ())
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -329,6 +329,8 @@ def generate_arn(
     """
     if region and region in REGION_PARTITION_MAP:
         partition = REGION_PARTITION_MAP[region]
+    if service == 's3':
+        region = ''
     arn = 'arn:%s:%s:%s:%s:' % (
         partition, service, region if region else '', account_id if account_id else '')
     if resource_type:


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -222,6 +222,12 @@ def test_generate_arn(self):
         self.assertEqual(
             utils.generate_arn("s3", "my_bucket"), "arn:aws:s3:::my_bucket"
         )
+
+        self.assertEqual(
+            utils.generate_arn("s3", "my_bucket", region="us-gov-west-1"),
+            "arn:aws-us-gov:s3:::my_bucket"
+        )
+
         self.assertEqual(
             utils.generate_arn(
                 "cloudformation",