Repository: ansible/ansible

Issue:
cloudfront_distribution_module fails on new required parameter
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->
It appears that AWS has added a new required parameter to the cloudfront APIs.

##### SUMMARY
<!--- Explain the problem briefly -->
Running a playbook with the cloudfront_distribution_module fails.  It was working as recently as May 14.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
cloudfront_distribution_module

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.2
  config file = /home/kdelong/hatch-devops/ansible/ansible.cfg
  configured module search path = [u'/home/kdelong/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.13 (default, Jan 31 2018, 00:17:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
no env vars: `env | grep ANSIBLE` is empty

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Amazon Linux 2017.09

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
    - name: create Cloudfront distribution
      cloudfront_distribution:
        state: present
        alias: "{{ item.alias }}"
        default_root_object: index.html
        origins:
            - id: "{{ item.origin_id }}"
              domain_name: "{{ item.domain_name }}"
              origin_path: ''
              s3_origin_access_identity_enabled: yes
        default_cache_behavior:
          target_origin_id: "{{ item.origin_id }}"
          viewer_protocol_policy: redirect-to-https
          allowed_methods:
            items:
              - GET
              - HEAD
            cached_methods:
              - GET
              - HEAD
          forwarded_values:
            query_string: false
            cookies:
              forward: none
          smooth_streaming: false
          compress: false
        custom_error_responses:
          - error_code: 403
            response_page_path: '/index.html' 
            response_code: 200
            error_caching_min_ttl: 300
        logging:
          enabled: true
          include_cookies: false
          bucket: hatch-logs.s3.amazonaws.com
          prefix: "{{ item.log_prefix }}"
        enabled: true
        price_class: PriceClass_100
        viewer_certificate:
          acm_certificate_arn: arn:aws:acm:us-east-1:1234567890:certificate/XXX
          minimum_protocol_version: TLSv1.1_2016
          ssl_support_method: sni-only
        http_version: http2
        ipv6_enabled: yes
      register: cf
      loop:
        - alias: ourdomain.ourco.com
          origin_id: S3-our-origin
          domain_name: our-bucket.s3.amazonaws.com
          log_prefix: sub-web-cloudfront/prod/
```
Etc, there are more items in the loop, they all fail with the identical message

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Task should succeed. It was working on May 14, which is when I wrote and tested this playbook.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
<!--- Paste verbatim command output between quotes below -->
```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: InvalidArgument: An error occurred (InvalidArgument) when calling the UpdateDistribution operation: The parameter Field level encryption id is required.
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/cloud/amazon/cloudfront_distribution.py
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774 `" && echo ansible-tmp-1527269181.25
-141910903271774="` echo /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774 `" ) && sleep 0'
<127.0.0.1> PUT /home/kdelong/.ansible/tmp/ansible-local-316195O2hYZ/tmptVNofO TO /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774/cloudfront_dis
tribution.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774/ /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141
910903271774/cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2.7 /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774/cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/kdelong/.ansible/tmp/ansible-tmp-1527269181.25-141910903271774/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_OmI3VZ/ansible_module_cloudfront_distribution.py", line 1190, in update_distribution
    return client.update_distribution(DistributionConfig=config, Id=distribution_id, IfMatch=e_tag)['Distribution']
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
InvalidArgument: An error occurred (InvalidArgument) when calling the UpdateDistribution operation: The parameter Field level encryption id is required.

failed: [localhost] (item={u'alias': u'ourdomain.ourco.com', u'log_prefix': u'sub-web-cloudfront/prod/', u'domain_name': u'our-bucket.s3.amazonaws.com', u'
origin_id': u'S3-our-origin'}) => {
    "changed": false,
    "error": {
        "code": "InvalidArgument",
        "message": "The parameter Field level encryption id is required.",
        "type": "Sender"
    },
    "invocation": {
        "module_args": {
            "alias": "ourdomain.ourco.com",
            "aliases": [
                "ourdomain.ourco.com"
            ],
            "aws_access_key": null,
            "aws_secret_key": null,
            "cache_behaviors": null,
            "caller_reference": null,
            "comment": null,
            "custom_error_responses": [
                {
                    "error_caching_min_t_t_l": 300,
                    "error_code": 403,
                    "response_code": "200",
                    "response_page_path": "/index.html"
                }
            ],
            "default_cache_behavior": {
                "allowed_methods": {
                    "cached_methods": {
                        "items": [
                            "HEAD",
                            "GET"
                        ],
                        "quantity": 2
                    },
                    "items": [
                        "HEAD",
                        "GET"
                    ],
                    "quantity": 2
                },
                "compress": false,
                "default_t_t_l": 86400,
                "forwarded_values": {
                    "cookies": {
                        "forward": "none"
                    },
                    "headers": {
                        "quantity": 0
                    },
                    "query_string": false,
                    "query_string_cache_keys": {
                        "quantity": 0
                    }
                },
                "lambda_function_associations": {
                    "quantity": 0
                },
                "max_t_t_l": 31536000,
                "min_t_t_l": 0,
                "smooth_streaming": false,
                "target_origin_id": "S3-my-bucket",
                "trusted_signers": {
                    "enabled": false,
                    "quantity": 0
                },
                "viewer_protocol_policy": "redirect-to-https"
            },
            "default_origin_domain_name": null,
            "default_origin_path": null,
            "default_root_object": "index.html",
            "distribution_id": null,
            "e_tag": null,
            "ec2_url": null,
            "enabled": true,
            "http_version": "http2",
            "ipv6_enabled": true,
            "logging": {
                "bucket": "our-logs.s3.amazonaws.com",
                "enabled": true,
                "include_cookies": false,
                "prefix": "sub-web-cloudfront/prod/"
            },
            "origins": [
                {
                    "custom_headers": {
                        "quantity": 0
                    },
                    "domain_name": "our-bucket.s3.amazonaws.com",
                    "id": "S3-our-origin",
                    "origin_path": "",
                    "s3_origin_config": {
                        "origin_access_identity": "origin-access-identity/cloudfront/XXX"
                    }
                }
            ],
            "price_class": "PriceClass_100",
            "profile": null,
            "purge_aliases": false,
            "purge_cache_behaviors": false,
            "purge_custom_error_responses": false,
            "purge_origins": false,
            "purge_tags": false,
            "region": null,
            "restrictions": null,
            "security_token": null,
            "state": "present",
            "tags": {},
            "validate_certs": true,
            "viewer_certificate": {
                "a_c_m_certificate_arn": "arn:aws:acm:us-east-1:1234567890:certificate/XXX",
                "minimum_protocol_version": "TLSv1.1_2016",
                "s_s_l_support_method": "sni-only"
            },
            "wait": false,
            "wait_timeout": 1800,
            "web_acl_id": null
        }
    },
    "item": {
        "alias": "ourdomain.ourco.com",
        "domain_name": "our-bucket.s3.amazonaws.com",
        "log_prefix": "sub-web-cloudfront/prod/",
        "origin_id": "S3-our-origin"
    },
    "msg": "Error updating distribution to {u'Comment': '', u'Restrictions': {u'GeoRestriction': {u'RestrictionType': 'none', u'Quantity': 0}}, u'Logging': {'Enabled': True, 'Prefix': 'sub-web-cloudfront/prod/', 'Bucket': 'our-logs.s3.amazonaws.com', 'IncludeCookies': False}, u'WebACLId': '', u'Origins': {'Items': [{'OriginPath': '', 'CustomHeaders': {'Quantity': 0}, 'S3OriginConfig': {'OriginAccessIdentity': 'origin-access-identity/cloudfront/XXX'}, 'Id': 'S3-our-origin', 'DomainName': 'our-bucket.s3.amazonaws.com'}], 'Quantity': 1}, u'PriceClass': 'PriceClass_100', u'DefaultRootObject': 'index.html', u'Enabled': True, u'DefaultCacheBehavior': {'TrustedSigners': {'Enabled': False, 'Quantity': 0}, 'LambdaFunctionAssociations': {u'Quantity': 0}, 'TargetOriginId': 'S3-our-origin', 'ViewerProtocolPolicy': 'redirect-to-https', 'ForwardedValues': {'Headers': {'Quantity': 0}, 'Cookies': {'Forward': 'none'}, 'QueryStringCacheKeys': {'Quantity': 0}, 'QueryString': False}, 'MaxTTL': 31536000, 'SmoothStreaming': False, 'DefaultTTL': 86400, 'AllowedMethods': {u'Items': ['HEAD', 'GET'], u'Quantity': 2, u'CachedMethods': {u'Items': ['HEAD', 'GET'], u'Quantity': 2}}, 'MinTTL': 0, 'Compress': False}, u'IsIPV6Enabled': True, u'ViewerCertificate': {u'SSLSupportMethod': 'sni-only', u'MinimumProtocolVersion': 'TLSv1.1_2016', u'ACMCertificateArn': 'arn:aws:acm:us-east-1:1234567890:certificate/XXX', u'Certificate': 'arn:aws:acm:us-east-1:1234567890:certificate/XXX', u'CertificateSource': 'acm'}, u'CustomErrorResponses': {'Items': [{'ErrorCode': 403, 'ResponsePagePath': '/index.html', 'ResponseCode': '200', 'ErrorCachingMinTTL': 300}], 'Quantity': 1}, u'Aliases': {'Items': ['ourdomain.ourco.com'], 'Quantity': 1}, u'CallerReference': '1519340531311', u'CacheBehaviors': {'Quantity': 0}, u'HttpVersion': 'http2'}: An error occurred (InvalidArgument) when calling the UpdateDistribution operation: The parameter Field level encryption id is required.",
    "response_metadata": {
        "http_headers": {
            "connection": "close",
            "content-length": "301",
            "content-type": "text/xml",
            "date": "Fri, 25 May 2018 17:26:22 GMT",
            "x-amzn-requestid": "be68c1fd-6040-11e8-b867-bd8b2f06ee63"
        },
        "http_status_code": 400,
        "request_id": "be68c1fd-6040-11e8-b867-bd8b2f06ee63",
        "retry_attempts": 0
    }
}
```


I have also opened an issue with AWS: this is actually their bug for adding a new required parameter to the API.


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/cloudfront_distribution.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudfront_distribution.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wilvk
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Thank you for reporting this. I am facing the exact same issue!

@s-hertel why have you removed the needs_triage label?
My AWS support contact just reported:

> I was in constant contact with our internal team regarding the "FieldLevelEncryptionID" parameter being mandatory and they have responded that the value is not necessary when creating a CF distribution, however, all the parameters are necessary when updating the distribution. The team updated that "FieldLevelEncryptionID" needs to be there, however, you can leave the parameter blank.

Hopefully that helps.
Same here :(
#41770 should fix this issue.
@willthames thanks! I would very much like this PR to be merged asap, as the cloudfront module is pretty much unusable now :/
Worth adding in 2.5.6 imho.

👍 
@pierreant-p it's definitely a backport candidate. 

But it's not the only way to fix a broken module without updating Ansible: https://willthames.github.io/2017/12/12/using-updated-modules-with-stable-ansible.html
@willthames oh sweet thanks !
I know this was already worked on, but I thought I'd pass along what my AWS support contact just told me:

> The ansible client is using a model from an older API revision and sending it to a newer API endpoint. That is not supported for updates.
> 
> The CloudFront API is document based, and the correct flow is as follows:
> 
> GetDistributionConfig => retrieves the current configuration for the API revision that is hit
> 
> <user modifies configuration to update/replace existing values>
> 
> PutDistributionConfig => updates the configuration against the same API revision endpoint that was used to Get the config in step 1.
> 
> The internal team mentioned that the ansible client is stripping unrecognized parameters from the configuration document retrieved, or using an older API revision to get the configuration, mutating the model and then trying to put it back. However, that won't work as it violates our schema.
Thanks @kenwdelong - that's interesting, it might be enough then to simply ensure that the FieldLevelEncryptionId is not stripped if it is returned.

If I create a second version that attempts this behaviour, can anyone test this for me?
https://github.com/ansible/ansible/commit/501a8c4d78af5ab5369350b96b996b487dc00a17 is an alternative implementation that should work based on the new information from @kenwdelong 
@willthames I tested it and it works for me without any issue. It successfully updated the distribution. Thank you very much for all your hard work!
Thanks @bmamouri, I'll update the PR with the new branch and then look at getting it merged and backported.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/cloudfront_distribution.py b/lib/ansible/modules/cloud/amazon/cloudfront_distribution.py
--- a/lib/ansible/modules/cloud/amazon/cloudfront_distribution.py
+++ b/lib/ansible/modules/cloud/amazon/cloudfront_distribution.py
@@ -152,6 +152,7 @@
               I(lambda_function_associations[])
                 I(lambda_function_arn)
                 I(event_type)
+              I(field_level_encryption_id)
 
     cache_behaviors:
       description:
@@ -180,6 +181,7 @@
             I(max_ttl)
             I(compress)
             I(lambda_function_associations[])
+            I(field_level_encryption_id)
 
     purge_cache_behaviors:
       description: Whether to remove any cache behaviors that aren't listed in I(cache_behaviors). This switch
@@ -1505,6 +1507,7 @@ def validate_cache_behavior(self, config, cache_behavior, valid_origins, is_defa
         cache_behavior = self.validate_allowed_methods(config, cache_behavior.get('allowed_methods'), cache_behavior)
         cache_behavior = self.validate_lambda_function_associations(config, cache_behavior.get('lambda_function_associations'), cache_behavior)
         cache_behavior = self.validate_trusted_signers(config, cache_behavior.get('trusted_signers'), cache_behavior)
+        cache_behavior = self.validate_field_level_encryption_id(config, cache_behavior.get('field_level_encryption_id'), cache_behavior)
         return cache_behavior
 
     def validate_cache_behavior_first_level_keys(self, config, cache_behavior, valid_origins, is_default_cache):
@@ -1587,6 +1590,14 @@ def validate_lambda_function_associations(self, config, lambda_function_associat
         except Exception as e:
             self.module.fail_json_aws(e, msg="Error validating lambda function associations")
 
+    def validate_field_level_encryption_id(self, config, field_level_encryption_id, cache_behavior):
+        # only set field_level_encryption_id if it's already set or if it was passed
+        if field_level_encryption_id is not None:
+            cache_behavior['field_level_encryption_id'] = field_level_encryption_id
+        elif 'field_level_encryption_id' in config:
+            cache_behavior['field_level_encryption_id'] = config.get('field_level_encryption_id')
+        return cache_behavior
+
     def validate_allowed_methods(self, config, allowed_methods, cache_behavior):
         try:
             if allowed_methods is not None:


Test patch:
diff --git a/test/integration/targets/cloudfront_distribution/meta/main.yml b/test/integration/targets/cloudfront_distribution/meta/main.yml
--- a/test/integration/targets/cloudfront_distribution/meta/main.yml
+++ b/test/integration/targets/cloudfront_distribution/meta/main.yml
@@ -1,3 +1 @@
-dependencies:
-  - prepare_tests
-  - setup_ec2
+dependencies: []
diff --git a/test/integration/targets/cloudfront_distribution/tasks/main.yml b/test/integration/targets/cloudfront_distribution/tasks/main.yml
--- a/test/integration/targets/cloudfront_distribution/tasks/main.yml
+++ b/test/integration/targets/cloudfront_distribution/tasks/main.yml
@@ -124,7 +124,6 @@
         custom_origin_config:
           http_port: 8080
       - domain_name: "{{ resource_prefix }}2.example.com"
-      - domain_name: "{{ test_identifier }}2.example.com"
       default_root_object: index.html
       wait: yes
       state: present
@@ -178,7 +177,7 @@
     cloudfront_distribution:
       alias: "{{ cloudfront_alias }}"
       origins:
-      - domain_name: "{{ test_identifier }}2.example.com"
+      - domain_name: "{{ resource_prefix }}2.example.com"
       default_root_object: index.php
       state: present
       <<: *aws_connection_info