Repository: ansible-collections/community.aws

Issue:
Cloudfront - Update Distribution - max retries exceeded
I'm using your module to update the path of an origin for a cloudfront distribution. It was working fine but now I get this error:
```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: 
botocore.exceptions.ClientError: An error occurred (Throttling) when calling the UpdateDistribution operation (reached max retries: 4): Rate exceeded
fatal: [10.63.56.237]: FAILED! => {"boto3_version": "1.16.13", "botocore_version": "1.19.13", "changed": false, "error": {"code": "Throttling", "message": "Rate exceeded"
```
 I contacted AWS Support but the issue can not be fixed increasing any limit, apparently we are calling UpdateDistribution 2 or 3 times in less than one minute. In my side I only do one call, like this:

```
- name: "Update onorigin_path from cloudfront distribution"
  community.aws.cloudfront_distribution:
    state: present
    distribution_id: "{{ cdn_cloudfront }}"
    origins:
      - id: "{{ cdn_cloudfront_origin_id }}"
        domain_name: "{{ cdn_cloudfront_origin_domain }}"
        origin_path: "/{{ deploy_helper.new_release }}"
    default_cache_behavior:
      target_origin_id: "{{ cdn_cloudfront_origin_id }}"
```

Any idea of how can be fixed?



##### ISSUE TYPE
- Bug Report

##### ANSIBLE VERSION

2.10
##### COMPONENT NAME

cloudfront_distribution


Discussion:
Programmatically the way around this is to add the AWSRetry decorator to the 'client' calls:
An example of this can be found in:
https://github.com/ansible-collections/amazon.aws/pull/103/files#diff-17496fe80d361b6e7fa1af8cbdefaa96e1003031e89f6b4767fb73171f822c83
(this example calls the variable 'connection' rather than 'client' but it's the same sort of boto3 object)

The reason there are multiple update calls is that boto3 (the library used to connect to AWS) has a very simplistic retry model built in, however with busy AWS accounts it's often insufficient.
Thanks Mark, that's very helpful 👍 That decorator can be added in my yaml file or I need to modify the python library?
You need to modify the ansible python module (plugins/modules/cloudfront_distribution.py).  If you're able to get it working and are able to open a pull request I should be able to review and get it merged.
Hi Mark, I did the changes here https://github.com/ansible-collections/community.aws/pull/297 the retry works because now it takes much longer to throw the error but I still get the same error from AWS. Any idea what can be happening?
It's tough to say, one option is to increase the initial delay (`delay=3` in your PR currently), it's also worth making sure that it's more than just the update call that you retry on.

The trouble is that most of the time when you're hitting this problem it's because you've got a very busy account.  To reduce issues like this we (${DAYJOB}) have been spreading our services across multiple accounts (See also AWS Organisations) and using SAML or Cross-Account trusts for authentication.  This has the added benefit of helping to segregate some of the access controls.
Hi Mark, I spoke with AWS support and share screen and look at logs with them. Even implementing jittered_backoff or exponential_backoff still can see 2 requests at the same time in less than one second. Then those retries have the gaps in between as the backoff works fine. From our side we will execute "update-distribution" using the aws cli from command shell. I don't know if still useful the opened PR or should we see why there are always 2 calls in less than 2 seconds.

Thank you very much
In my opinion PRs going through and adding the retries are valuable, if you're willing to add the extra `aws_retry=True` parameters I'd be happy to get them merged...
sure! I will, I just need to put few things together before ;)
@fnavarrodev: Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:
- ansible version
- component name

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE/bug_report.md

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: issue_missing_data --->
cc @jillr @s-hertel @willthames @wilvk @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
##### COMPONENT NAME
community.aws.cloudfront_distribution

##### ANSIBLE VERSION
```
ansible 2.9.15
```

Resolving patch:
diff --git a/plugins/modules/cloudfront_distribution.py b/plugins/modules/cloudfront_distribution.py
--- a/plugins/modules/cloudfront_distribution.py
+++ b/plugins/modules/cloudfront_distribution.py
@@ -1378,7 +1378,7 @@
 from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
 from ansible_collections.amazon.aws.plugins.module_utils.cloudfront_facts import CloudFrontFactsServiceManager
 from ansible.module_utils.common.dict_transformations import recursive_diff
-from ansible_collections.amazon.aws.plugins.module_utils.ec2 import compare_aws_tags, ansible_dict_to_boto3_tag_list, boto3_tag_list_to_ansible_dict
+from ansible_collections.amazon.aws.plugins.module_utils.ec2 import AWSRetry, compare_aws_tags, ansible_dict_to_boto3_tag_list, boto3_tag_list_to_ansible_dict
 from ansible_collections.amazon.aws.plugins.module_utils.ec2 import camel_dict_to_snake_dict, snake_dict_to_camel_dict
 import datetime
 
@@ -1433,7 +1433,7 @@ def ansible_list_to_cloudfront_list(list_items=None, include_quantity=True):
 def create_distribution(client, module, config, tags):
     try:
         if not tags:
-            return client.create_distribution(DistributionConfig=config)['Distribution']
+            return client.create_distribution(aws_retry=True, DistributionConfig=config)['Distribution']
         else:
             distribution_config_with_tags = {
                 'DistributionConfig': config,
@@ -1441,42 +1441,42 @@ def create_distribution(client, module, config, tags):
                     'Items': tags
                 }
             }
-            return client.create_distribution_with_tags(DistributionConfigWithTags=distribution_config_with_tags)['Distribution']
+            return client.create_distribution_with_tags(aws_retry=True, DistributionConfigWithTags=distribution_config_with_tags)['Distribution']
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error creating distribution")
 
 
 def delete_distribution(client, module, distribution):
     try:
-        return client.delete_distribution(Id=distribution['Distribution']['Id'], IfMatch=distribution['ETag'])
+        return client.delete_distribution(aws_retry=True, Id=distribution['Distribution']['Id'], IfMatch=distribution['ETag'])
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error deleting distribution %s" % to_native(distribution['Distribution']))
 
 
 def update_distribution(client, module, config, distribution_id, e_tag):
     try:
-        return client.update_distribution(DistributionConfig=config, Id=distribution_id, IfMatch=e_tag)['Distribution']
+        return client.update_distribution(aws_retry=True, DistributionConfig=config, Id=distribution_id, IfMatch=e_tag)['Distribution']
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error updating distribution to %s" % to_native(config))
 
 
 def tag_resource(client, module, arn, tags):
     try:
-        return client.tag_resource(Resource=arn, Tags=dict(Items=tags))
+        return client.tag_resource(aws_retry=True, Resource=arn, Tags=dict(Items=tags))
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error tagging resource")
 
 
 def untag_resource(client, module, arn, tag_keys):
     try:
-        return client.untag_resource(Resource=arn, TagKeys=dict(Items=tag_keys))
+        return client.untag_resource(aws_retry=True, Resource=arn, TagKeys=dict(Items=tag_keys))
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error untagging resource")
 
 
 def list_tags_for_resource(client, module, arn):
     try:
-        response = client.list_tags_for_resource(Resource=arn)
+        response = client.list_tags_for_resource(aws_retry=True, Resource=arn)
         return boto3_tag_list_to_ansible_dict(response.get('Tags').get('Items'))
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg="Error listing tags for resource")
@@ -2152,7 +2152,7 @@ def main():
         ]
     )
 
-    client = module.client('cloudfront')
+    client = module.client('cloudfront', retry_decorator=AWSRetry.jittered_backoff())
 
     validation_mgr = CloudFrontValidationManager(module)
 


Test patch:
diff --git a/tests/integration/targets/cloudfront_distribution/tasks/main.yml b/tests/integration/targets/cloudfront_distribution/tasks/main.yml
--- a/tests/integration/targets/cloudfront_distribution/tasks/main.yml
+++ b/tests/integration/targets/cloudfront_distribution/tasks/main.yml
@@ -240,6 +240,17 @@
       state: present
     register: cf_update_default_root_object
 
+  - name: update default_root_object of existing distribution with retries
+    cloudfront_distribution:
+      distribution_id: "{{ distribution_id }}"
+      origins:
+      - domain_name: "{{ resource_prefix }}2.example.com"
+      default_root_object: index.php
+      state: present
+    retries: 3
+    delay: 3  
+    register: cf_update_default_root_object    
+
   - name: ensure origin was updated
     assert:
       that: