Repository: ansible-collections/community.aws

Issue:
[BUG] `self.paginated_response` does not return the aggregated list of distributions (cloudfront_info)
### Summary

AWS paginates distributions by default at 100 items per page.

Trying the following can cause an error if the distribution appears in the first page of the results:
```yaml
- name: "Check if distribution already exists for {{ domainName }}"
  cloudfront_info:
    <<: *AWS_CREDENTIALS
    distribution: true
    domain_name_alias: "{{ domainName }}"
  register: distribution_data
```

--- 

The error lies at the line 502, in the function `paginated_response`: https://github.com/ansible-collections/community.aws/blob/3ae2b3efa4588ac1284c565d6f73543a555a5edd/plugins/modules/cloudfront_info.py#L502

For distributions, AWS returns an object as follows:
```js
{ 
  "IsTruncated": true,
  "Items": [{ 
     "ARN": "REDACTED",
     // ... distribution data 
   }, {
     "ARN": "REDACTED",
     // ... distribution data 
  }],
  "Marker": "",
  "MaxItems": 100,
  "NextMarker": "REDACTED_MARKER",
  "Quantity": 100
}
```

Which means `results.update(result)` on the Nth page overwrites the previous results, causing the returned data to only include the distributions included in the last page -- as the `items` arrays are not concatenated together

In my case the 2nd page shows the following object:
```js
{ 
  "IsTruncated": true,
  "Items": [{ 
     "ARN": "REDACTED",
     // ... distribution data 
   }, {
     "ARN": "REDACTED",
     // ... distribution data 
  }],
  "Marker": "REDACTED_MARKER",
  "MaxItems": 100,
  "NextMarker": "",
  "Quantity": 59
}
```

And printing the length of results returned by `service_mgr.get_distribution_id_from_domain_name(domain_name_alias)` yields `59`, which is incorrect, and should be 159 as can be seen from the following screenshot

![image](https://user-images.githubusercontent.com/2607260/137746793-f32242e2-5485-4c54-bf8f-306b49e5647a.png)


### Issue Type

Bug Report

### Component Name

cloudfront_info

### Ansible Version

![image](https://user-images.githubusercontent.com/2607260/137741051-806d003b-59ca-4cfd-9209-bc8e543e6414.png)


### Collection Versions

The bug exists in both
- the embedded core of Ansible
- the latest revision of `cloudfront_unfo` taken directly from https://github.com/ansible-collections/community.aws/blob/3ae2b3efa4588ac1284c565d6f73543a555a5edd/plugins/modules/cloudfront_info.py (3ae2b3efa4588ac1284c565d6f73543a555a5edd being the latest as of the this writing)

### AWS SDK versions

![image](https://user-images.githubusercontent.com/2607260/137741051-806d003b-59ca-4cfd-9209-bc8e543e6414.png)


### Configuration

```console (paste below)
$ ansible-config dump --only-changed

ANY_ERRORS_FATAL(~/workspace/ansible/ansible.cfg) = True
DEFAULT_STDOUT_CALLBACK(~/workspace/ansible/ansible.cfg) = debug
DUPLICATE_YAML_DICT_KEY(~/workspace/ansible/ansible.cfg) = warn
HOST_KEY_CHECKING(~/workspace/ansible/ansible.cfg) = False
INJECT_FACTS_AS_VARS(~/workspace/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(~/workspace/ansible/ansible.cfg) = auto
RETRY_FILES_ENABLED(~/workspace/ansible/ansible.cfg) = False
TRANSFORM_INVALID_GROUP_CHARS(~/workspace/ansible/ansible.cfg) = never
```


### OS / Environment

Debian 4.19.194-3 (2021-07-18) x86_64

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
1. Have over 100 distributions (or adjust the pagination of distribtions down enough to have multiple pages)
2. Try to fetch the info of a distribution that would appear on the first of multiple pages
3. Lo and behold: "Error unable to source a distribution id from domain_name_alias"
```

Sample fetch:
```yaml
- name: "Check if distribution already exists for {{ domainName }}"
  cloudfront_info:
    <<: *AWS_CREDENTIALS
    distribution: true
    domain_name_alias: "{{ domainName }}"
  register: distribution_data
```

### Expected Results

**EXPECTATIONS**: 
- Have the corresponding distribution data fetched disregarding on which page it may be

### Actual Results

```console (paste below)
fatal: [REDACTED]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "all_lists": false,
            "aws_access_key": "REDACTED",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "debug_botocore_endpoint_logs": false,
            "distribution": true,
            "distribution_config": false,
            "distribution_id": null,
            "domain_name_alias": "REDACTED",
            "ec2_url": null,
            "invalidation": false,
            "invalidation_id": null,
            "list_distributions": false,
            "list_distributions_by_web_acl_id": false,
            "list_invalidations": false,
            "list_origin_access_identities": false,
            "list_streaming_distributions": false,
            "origin_access_identity": false,
            "origin_access_identity_config": false,
            "origin_access_identity_id": null,
            "profile": null,
            "region": "eu-west-3",
            "security_token": null,
            "streaming_distribution": false,
            "streaming_distribution_config": false,
            "summary": false,
            "validate_certs": true
        }
    }
}

MSG:

Error unable to source a distribution id from domain_name_alias
```

Thrown by: https://github.com/ansible-collections/community.aws/blob/3ae2b3efa4588ac1284c565d6f73543a555a5edd/plugins/modules/cloudfront_info.py#L607-L611

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/cloudfront_info.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/cloudfront_info.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jillr @markuman @s-hertel @tremble @wilvk @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
An ugly-yet-simple fix would be:

```python
    # Only necessary until https://github.com/ansible-collections/community.aws/issues/769 gets fixed
    def paginated_response(self, func, result_key=""):
        '''
        Returns expanded response for paginated operations.
        The 'result_key' is used to define the concatenated results that are combined from each paginated response.
        '''
        args = dict()
        results = dict()
        items = []
        loop = True
        while loop:
            response = func(**args)
            if result_key == "":
                result = response
                result.pop('ResponseMetadata', None)
            else:
                result = response.get(result_key)
            results.update(result)
            if 'Items' in result.keys():
              items = items + result.get('Items')
              results['Items'] = items
            args['Marker'] = response.get('NextMarker')
            for key in response.keys():
                if key.endswith('List'):
                    args['Marker'] = response[key].get('NextMarker')
                    break
            loop = args['Marker'] is not None
        return results
```
Hi @TheOptimisticFactory,

There's actually a simpler option: boto3 has paginator helpers (only for some calls)
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudfront.html#CloudFront.Paginator.ListDistributions

You can see an example of how to use them in #472 :
https://github.com/ansible-collections/community.aws/pull/472/files#diff-5b6740d7309875806709fffb540ad4344cf7621a6bcb580e28cdee673a453a95R162

Would you be interested in submitting a Pull Request?


Mark
While my problem is purely on the distribution listing, it _may_ also occur on any of the calls to `self.paginated_response()` which [covers an array of different AWS functions](https://github.com/ansible-collections/community.aws/blob/3ae2b3efa4588ac1284c565d6f73543a555a5edd/plugins/modules/cloudfront_info.py)
1. `self.client.get_distribution`
2. `self.client.get_distribution_config`
3. `self.client.get_cloud_front_origin_access_identity`
4. `self.client.get_cloud_front_origin_access_identity_config`
5. `self.client.get_invalidation`
6. `self.client.get_streaming_distribution`
7. `self.client.get_streaming_distribution_config`
8. `self.client.list_cloud_front_origin_access_identities`
9. `self.client.list_distributions`
10. `self.client.list_distributions_by_web_acl_id`
11. `self.client.list_invalidations`
12. `self.client.list_streaming_distributions`
13. `self.client.list_distributions_by_web_acl_id`

---

The method `get_distribution_id_from_domain_name` itself uses both distributions and streaming distributions (L462-L463)

https://github.com/ansible-collections/community.aws/blob/3ae2b3efa4588ac1284c565d6f73543a555a5edd/plugins/modules/cloudfront_info.py#L459-L472

---

Switching to boto3 iterators implies adjusting all the above calls, and I would rather wait for an experienced maintainer to carry out that task safely :)

Resolving patch:
diff --git a/plugins/modules/cloudfront_info.py b/plugins/modules/cloudfront_info.py
--- a/plugins/modules/cloudfront_info.py
+++ b/plugins/modules/cloudfront_info.py
@@ -269,6 +269,7 @@
 
 from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
 from ansible_collections.amazon.aws.plugins.module_utils.ec2 import boto3_tag_list_to_ansible_dict
+from ansible_collections.amazon.aws.plugins.module_utils.ec2 import AWSRetry
 
 
 class CloudFrontServiceManager:
@@ -278,64 +279,72 @@ def __init__(self, module):
         self.module = module
 
         try:
-            self.client = module.client('cloudfront')
+            self.client = module.client('cloudfront', retry_decorator=AWSRetry.jittered_backoff())
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             module.fail_json_aws(e, msg='Failed to connect to AWS')
 
     def get_distribution(self, distribution_id):
         try:
-            func = partial(self.client.get_distribution, Id=distribution_id)
-            return self.paginated_response(func)
+            distribution = self.client.get_distribution(aws_retry=True, Id=distribution_id)
+            return distribution
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing distribution")
 
     def get_distribution_config(self, distribution_id):
         try:
-            func = partial(self.client.get_distribution_config, Id=distribution_id)
-            return self.paginated_response(func)
+            distribution = self.client.get_distribution_config(aws_retry=True, Id=distribution_id)
+            return distribution
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing distribution configuration")
 
     def get_origin_access_identity(self, origin_access_identity_id):
         try:
-            func = partial(self.client.get_cloud_front_origin_access_identity, Id=origin_access_identity_id)
-            return self.paginated_response(func)
+            origin_access_identity = self.client.get_cloud_front_origin_access_identity(aws_retry=True, Id=origin_access_identity_id)
+            return origin_access_identity
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing origin access identity")
 
     def get_origin_access_identity_config(self, origin_access_identity_id):
         try:
-            func = partial(self.client.get_cloud_front_origin_access_identity_config, Id=origin_access_identity_id)
-            return self.paginated_response(func)
+            origin_access_identity = self.client.get_cloud_front_origin_access_identity_config(aws_retry=True, Id=origin_access_identity_id)
+            return origin_access_identity
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing origin access identity configuration")
 
     def get_invalidation(self, distribution_id, invalidation_id):
         try:
-            func = partial(self.client.get_invalidation, DistributionId=distribution_id, Id=invalidation_id)
-            return self.paginated_response(func)
+            invalidation = self.client.get_invalidation(aws_retry=True, DistributionId=distribution_id, Id=invalidation_id)
+            return invalidation
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing invalidation")
 
     def get_streaming_distribution(self, distribution_id):
         try:
-            func = partial(self.client.get_streaming_distribution, Id=distribution_id)
-            return self.paginated_response(func)
+            streaming_distribution = self.client.get_streaming_distribution(aws_retry=True, Id=distribution_id)
+            return streaming_distribution
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing streaming distribution")
 
     def get_streaming_distribution_config(self, distribution_id):
         try:
-            func = partial(self.client.get_streaming_distribution_config, Id=distribution_id)
-            return self.paginated_response(func)
+            streaming_distribution = self.client.get_streaming_distribution_config(aws_retry=True, Id=distribution_id)
+            return streaming_distribution
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error describing streaming distribution")
 
+    # Split out paginator to allow for the backoff decorator to function
+    @AWSRetry.jittered_backoff()
+    def _paginated_result(self, paginator_name, **params):
+        paginator = self.client.get_paginator(paginator_name)
+        results = paginator.paginate(**params).build_full_result()
+        return results
+
     def list_origin_access_identities(self):
         try:
-            func = partial(self.client.list_cloud_front_origin_access_identities)
-            origin_access_identity_list = self.paginated_response(func, 'CloudFrontOriginAccessIdentityList')
-            if origin_access_identity_list['Quantity'] > 0:
+            results = self._paginated_result('list_cloud_front_origin_access_identities')
+            origin_access_identity_list = results.get('CloudFrontOriginAccessIdentityList', {'Items': []})
+
+            if len(origin_access_identity_list['Items']) > 0:
                 return origin_access_identity_list['Items']
             return {}
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
@@ -343,12 +352,14 @@ def list_origin_access_identities(self):
 
     def list_distributions(self, keyed=True):
         try:
-            func = partial(self.client.list_distributions)
-            distribution_list = self.paginated_response(func, 'DistributionList')
-            if distribution_list['Quantity'] == 0:
-                return {}
-            else:
+            results = self._paginated_result('list_distributions')
+            distribution_list = results.get('DistributionList', {'Items': []})
+
+            if len(distribution_list['Items']) > 0:
                 distribution_list = distribution_list['Items']
+            else:
+                return {}
+
             if not keyed:
                 return distribution_list
             return self.keyed_list_helper(distribution_list)
@@ -357,21 +368,23 @@ def list_distributions(self, keyed=True):
 
     def list_distributions_by_web_acl_id(self, web_acl_id):
         try:
-            func = partial(self.client.list_distributions_by_web_acl_id, WebAclId=web_acl_id)
-            distribution_list = self.paginated_response(func, 'DistributionList')
-            if distribution_list['Quantity'] == 0:
-                return {}
-            else:
+            results = self._paginated_result('list_cloud_front_origin_access_identities', WebAclId=web_acl_id)
+            distribution_list = results.get('DistributionList', {'Items': []})
+
+            if len(distribution_list['Items']) > 0:
                 distribution_list = distribution_list['Items']
+            else:
+                return {}
             return self.keyed_list_helper(distribution_list)
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error listing distributions by web acl id")
 
     def list_invalidations(self, distribution_id):
         try:
-            func = partial(self.client.list_invalidations, DistributionId=distribution_id)
-            invalidation_list = self.paginated_response(func, 'InvalidationList')
-            if invalidation_list['Quantity'] > 0:
+            results = self._paginated_result('list_invalidations', DistributionId=distribution_id)
+            invalidation_list = results.get('InvalidationList', {'Items': []})
+
+            if len(invalidation_list['Items']) > 0:
                 return invalidation_list['Items']
             return {}
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
@@ -379,12 +392,14 @@ def list_invalidations(self, distribution_id):
 
     def list_streaming_distributions(self, keyed=True):
         try:
-            func = partial(self.client.list_streaming_distributions)
-            streaming_distribution_list = self.paginated_response(func, 'StreamingDistributionList')
-            if streaming_distribution_list['Quantity'] == 0:
-                return {}
-            else:
+            results = self._paginated_result('list_streaming_distributions')
+            streaming_distribution_list = results.get('StreamingDistributionList', {'Items': []})
+
+            if len(streaming_distribution_list['Items']) > 0:
                 streaming_distribution_list = streaming_distribution_list['Items']
+            else:
+                return {}
+
             if not keyed:
                 return streaming_distribution_list
             return self.keyed_list_helper(streaming_distribution_list)
@@ -484,30 +499,6 @@ def get_aliases_from_distribution_id(self, distribution_id):
         except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
             self.module.fail_json_aws(e, msg="Error getting list of aliases from distribution_id")
 
-    def paginated_response(self, func, result_key=""):
-        '''
-        Returns expanded response for paginated operations.
-        The 'result_key' is used to define the concatenated results that are combined from each paginated response.
-        '''
-        args = dict()
-        results = dict()
-        loop = True
-        while loop:
-            response = func(**args)
-            if result_key == "":
-                result = response
-                result.pop('ResponseMetadata', None)
-            else:
-                result = response.get(result_key)
-            results.update(result)
-            args['Marker'] = response.get('NextMarker')
-            for key in response.keys():
-                if key.endswith('List'):
-                    args['Marker'] = response[key].get('NextMarker')
-                    break
-            loop = args['Marker'] is not None
-        return results
-
     def keyed_list_helper(self, list_to_key):
         keyed_list = dict()
         for item in list_to_key:


Test patch: