Repository: ansible/ansible

Issue:
cloudtrail module does not support China region
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
cloudtrail

##### ANSIBLE VERSION
```
$ ansible --version
ansible 2.5.0 (devel b88304f211) last updated 2018/01/10 15:31:29 (GMT +100)
  config file = /Users/nbailey/automation/roles-aws/aws_cloudtrail/tests/ansible.cfg
  configured module search path = [u'/Users/nbailey/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/nbailey/virtualenv-ansible-devel/src/ansible/lib/ansible
  executable location = /Users/nbailey/virtualenv-ansible-devel/bin/ansible
  python version = 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
```

##### CONFIGURATION
nothing relevant.

##### OS / ENVIRONMENT

OSX, running against AWS China.

##### SUMMARY

Boto3 itself fully supports CloudTrail in the AWS China region. Ansible's implementation does not seem to - CreateTrail API calls return with an HTTP 400 and the following: 

"An error occurred (UnsupportedOperationException) when calling the CreateTrail operation: The operation requested is not supported in the region."

I can't see anything to cause this. Behaviour is the same when explicitly setting the documented China Cloudtrail endpoint (https://cloudtrail.cn-north-1.amazonaws.com.cn) and when allowing Boto3 to evaluate endpoints itself.

##### STEPS TO REPRODUCE

```yaml
- name: create test cloudtrail in china
  hosts: test-trail-1
  connection: local
  tags: [ china ]

  pre_tasks:
  - name: CreateTrail API call
    cloudtrail:
      state: enabled
      name: test-trail-1
      s3_bucket_name: "{{ aws.bucket_name }}"
      region: cn-north-1
      profile: "{{ aws.boto_profile }}"
```

##### EXPECTED RESULTS
The module creates a trail, just as they do when creating a boto3 cloudtrail client and making the API calls directly:

```
$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> import botocore
>>> session = boto3.Session(profile_name='<redacted>')
>>> session = boto3.Session(profile_name='<redacted>')
>>> preprod = session.client('cloudtrail', region_name='cn-north-1')
>>> response = preprod.delete_trail(Name='test')
>>> response
{'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'b289adf1-3b5e-47bd-b883-d29ee3a6d216', 'HTTPHeaders': {'x-amzn-requestid': 'b289adf1-3b5e-47bd-b883-d29ee3a6d216', 'date': 'Wed, 10 Jan 2018 15:35:20 GMT', 'content-length': '2', 'content-type': 'application/x-amz-json-1.1'}}}
>>> response = preprod.create_trail(Name='test',S3BucketName='cn-sh-cnn1-cloud-trail')
>>> response
{u'IncludeGlobalServiceEvents': True, u'Name': u'test', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '708da52e-e384-403f-aa16-0fa53c1933bd', 'HTTPHeaders': {'x-amzn-requestid': '708da52e-e384-403f-aa16-0fa53c1933bd', 'date': 'Wed, 10 Jan 2018 15:35:31 GMT', 'content-length': '186', 'content-type': 'application/x-amz-json-1.1'}}, u'TrailARN': u'arn:aws-cn:cloudtrail:cn-north-1:640673851126:trail/test', u'IsMultiRegionTrail': False, u'S3BucketName': u'cn-sh-cnn1-cloud-trail'}
>>>
```

##### ACTUAL RESULTS

 The module appears to connect to the AWS service endpoint in cn-north-1 successfully (the region parameter is correct and it is an AWS service response). However the module output reports that the CreateTrail operation is not supported in this region, even though it works with the AWS CLI or with a boto3 API call not wrapped by Ansible.

Here's full output:
 
```
RUNNING HANDLER [aws_cloudtrail : manage cloudtrail trails] ****************************************************************************************************************************************************************************************************************************************
Using module file /Users/nbailey/automation/roles-aws/aws_modules/library/cloudtrail.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: nbailey
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100 `" && echo ansible-tmp-1515598328.47-33693836653100="` echo /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100 `" ) && sleep 0'
<localhost> PUT /var/folders/3k/n0flxzzx3tn15hd44_0l4qt40038gw/T/tmpvORY0v TO /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100/cloudtrail.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100/ /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100/cloudtrail.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100/cloudtrail.py; rm -rf "/Users/nbailey/.ansible/tmp/ansible-tmp-1515598328.47-33693836653100/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/3k/n0flxzzx3tn15hd44_0l4qt40038gw/T/ansible_MCTZx7/ansible_module_cloudtrail.py", line 270, in create_trail
    resp = client.create_trail(**ct_params)
  File "/Users/nbailey/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/nbailey/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
UnsupportedOperationException: An error occurred (UnsupportedOperationException) when calling the CreateTrail operation: The operation requested is not supported in the region.

fatal: [test-trail-1 -> localhost]: FAILED! => {
    "attempts": 1,
    "changed": false,
    "error": {
        "code": "UnsupportedOperationException",
        "message": "The operation requested is not supported in the region."
    },
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "cloudwatch_logs_log_group_arn": null,
            "cloudwatch_logs_role_arn": null,
            "ec2_url": null,
            "enable_log_file_validation": true,
            "enable_logging": false,
            "include_global_events": true,
            "is_multi_region_trail": true,
            "kms_key_id": null,
            "name": "test-trail-1",
            "profile": "<redacted>",
            "region": "cn-north-1",
            "s3_bucket_name": "aws-cloudtrail-test-bucket-a",
            "s3_key_prefix": null,
            "security_token": null,
            "sns_topic_name": null,
            "state": "present",
            "tags": {},
            "validate_certs": true
        }
    },
    "msg": "An error occurred (UnsupportedOperationException) when calling the CreateTrail operation: The operation requested is not supported in the region.",
    "response_metadata": {
        "http_headers": {
            "connection": "close",
            "content-length": "110",
            "content-type": "application/x-amz-json-1.1",
            "date": "Wed, 10 Jan 2018 15:32:10 GMT",
            "x-amzn-requestid": "<redacted>"
        },
        "http_status_code": 400,
        "request_id": "<redacted>",
        "retry_attempts": 0
    }
}
```

Not sure how to debug this further - I'm no great expert on the boto framework, but I don't see anything weird in either the module (create_trail function is here: https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudtrail.py#L266
 and fairly straightforward) or in the boto_conn logic (here: https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/ec2.py#L116). Any insight into it much appreciated!



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/cloudtrail.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudtrail.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @shepdelacreme @tedder @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I know this is a bigger problem than Cloudtrail, I'm not sure where it is on the roadmap. Some related tickets: #21532 #30158 #28704 #34145 
Thanks for looking around. As far as I can tell none of those are directly related - the lambda ones (#28704, #34145) are fixes for parsing ARNs other than `aws`, the AWS dynamic region detection PR (#21532) is merged and was a slightly different problem, and #30158 was breaking other things while adding support for China... 

To boil it down, this bug is about Ansible's implementation of a Boto3 CloudTrail client in the cn-north-1 region producing different results from a simple hand-rolled implementation. I couldn't see any existing reports of an issue like this in GovCloud or China, and I've not hit this in the dozen or so other modules I use in China.

Normally when Ansible modules breaks in AWS China because it's been because the endpoint is misconstructed (building endpoints out of servicename + 'amazonaws.com' when China resources end with 'com.cn'), or because of issues with hardcoding `aws` as a partition name. This looks like something a bit more subtle. Well - either that or I'm doing something really stupid, but if so I've looked pretty hard and can't see what the stupid thing could be!
How sure are you that it's the cloudtrail implementation specifically? I believe it is fairly widespread, as it uses the common connection framework, which places the error somewhere in or around boto3_conn.
I've used a lot of the other AWS Ansible modules in China and never seen this failure mode before. I'm not sure how many of those are using this boto3 code path, but I think a fair number.

My suspicion is that there's some weirdness in the CloudTrail service, and it doesn't handle the input that the module passes to boto3_conn correctly. I don't think it's 'the module is buggy', I think it's 'AWS China is doing something nonstandard and Ansible needs to interact with it in specific ways'. 

Things that might shed more light:

1. is there a way to get the contents of the connection **kwargs that get passed to boto3_conn?
2. Is there a way to get the HTTP request/complete API call boto3 makes?

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/cloudtrail.py b/lib/ansible/modules/cloud/amazon/cloudtrail.py
--- a/lib/ansible/modules/cloud/amazon/cloudtrail.py
+++ b/lib/ansible/modules/cloud/amazon/cloudtrail.py
@@ -62,7 +62,6 @@
     description:
       - Specifies whether log file integrity validation is enabled.
       - CloudTrail will create a hash for every log file delivered and produce a signed digest file that can be used to ensure log files have not been tampered.
-    default: false
     version_added: "2.4"
     aliases: [ "log_file_validation_enabled" ]
   include_global_events:
@@ -444,7 +443,7 @@ def main():
         s3_key_prefix=dict(),
         sns_topic_name=dict(),
         is_multi_region_trail=dict(default=False, type='bool'),
-        enable_log_file_validation=dict(default=False, type='bool', aliases=['log_file_validation_enabled']),
+        enable_log_file_validation=dict(type='bool', aliases=['log_file_validation_enabled']),
         include_global_events=dict(default=True, type='bool', aliases=['include_global_service_events']),
         cloudwatch_logs_role_arn=dict(),
         cloudwatch_logs_log_group_arn=dict(),
@@ -472,12 +471,6 @@ def main():
         S3BucketName=module.params['s3_bucket_name'],
         IncludeGlobalServiceEvents=module.params['include_global_events'],
         IsMultiRegionTrail=module.params['is_multi_region_trail'],
-        EnableLogFileValidation=module.params['enable_log_file_validation'],
-        S3KeyPrefix='',
-        SnsTopicName='',
-        CloudWatchLogsRoleArn='',
-        CloudWatchLogsLogGroupArn='',
-        KmsKeyId=''
     )
 
     if module.params['s3_key_prefix']:
@@ -492,6 +485,9 @@ def main():
     if module.params['cloudwatch_logs_log_group_arn']:
         ct_params['CloudWatchLogsLogGroupArn'] = module.params['cloudwatch_logs_log_group_arn']
 
+    if module.params['enable_log_file_validation'] is not None:
+        ct_params['EnableLogFileValidation'] = module.params['enable_log_file_validation']
+
     if module.params['kms_key_id']:
         ct_params['KmsKeyId'] = module.params['kms_key_id']
 
@@ -599,7 +595,9 @@ def main():
                 pass
             trail = dict()
             trail.update(ct_params)
-            trail['LogFileValidationEnabled'] = ct_params['EnableLogFileValidation']
+            if 'EnableLogFileValidation' not in ct_params:
+                ct_params['EnableLogFileValidation'] = False
+            trail['EnableLogFileValidation'] = ct_params['EnableLogFileValidation']
             trail.pop('EnableLogFileValidation')
             fake_arn = 'arn:aws:cloudtrail:' + region + ':' + acct_id + ':trail/' + ct_params['Name']
             trail['HasCustomEventSelectors'] = False


Test patch: