Repository: ansible/ansible

Issue:
cloudfront_facts pagination not working
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
cloudfront_facts

##### ANSIBLE VERSION
```
ansible 2.5.0
  config file = /Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg
  configured module search path = [u'/Users/gennadii_aleksandrov/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/gennadii_aleksandrov/Library/Python/2.7/lib/python/site-packages/ansible
  executable location = /Users/Gennadii_Aleksandrov/Library/Python/2.7/bin/ansible
  python version = 2.7.14 (default, Mar  9 2018, 23:57:12) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
```

##### CONFIGURATION
```
DEFAULT_HOST_LIST(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = [u'/Users/gennadii_aleksandrov/r/ansible-control-repo/inventory/local']
DEFAULT_REMOTE_USER(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = ubuntu
HOST_KEY_CHECKING(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
RETRY_FILES_ENABLED(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
cloudfront_facts returns only first 100 CloudFront distributions with `summary: true`.
Also, it returns `Error unable to source a distribution id from domain_name_alias` when trying to find distribution with `domain_name_alias` if that distribution is not on the first page of Amazon API response.

##### STEPS TO REPRODUCE
0. Create more than 100 CloudFront distributions :)
Then there are two cases here:
1. First - distributions summary:
<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        summary: true
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```
2. Second - try to fetch info on distribution which is not in first 100:
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        distribution: true
        domain_name_alias: '{{ fqdn }}'
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```



##### EXPECTED RESULTS
1. Full list of distributions with more than 100 elements.
2. `cloudfront` dict containing distribution information.

##### ACTUAL RESULTS
1. You will only see first 100 distributions.
2. You will see `Error unable to source a distribution id from domain_name_alias`

cloudfront_facts pagination not working
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
cloudfront_facts

##### ANSIBLE VERSION
```
ansible 2.5.0
  config file = /Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg
  configured module search path = [u'/Users/gennadii_aleksandrov/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/gennadii_aleksandrov/Library/Python/2.7/lib/python/site-packages/ansible
  executable location = /Users/Gennadii_Aleksandrov/Library/Python/2.7/bin/ansible
  python version = 2.7.14 (default, Mar  9 2018, 23:57:12) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
```

##### CONFIGURATION
```
DEFAULT_HOST_LIST(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = [u'/Users/gennadii_aleksandrov/r/ansible-control-repo/inventory/local']
DEFAULT_REMOTE_USER(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = ubuntu
HOST_KEY_CHECKING(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
RETRY_FILES_ENABLED(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
cloudfront_facts returns only first 100 CloudFront distributions with `summary: true`.
Also, it returns `Error unable to source a distribution id from domain_name_alias` when trying to find distribution with `domain_name_alias` if that distribution is not on the first page of Amazon API response.

##### STEPS TO REPRODUCE
0. Create more than 100 CloudFront distributions :)
Then there are two cases here:
1. First - distributions summary:
<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        summary: true
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```
2. Second - try to fetch info on distribution which is not in first 100:
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        distribution: true
        domain_name_alias: '{{ fqdn }}'
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```



##### EXPECTED RESULTS
1. Full list of distributions with more than 100 elements.
2. `cloudfront` dict containing distribution information.

##### ACTUAL RESULTS
1. You will only see first 100 distributions.
2. You will see `Error unable to source a distribution id from domain_name_alias`

cloudfront_facts pagination not working
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
cloudfront_facts

##### ANSIBLE VERSION
```
ansible 2.5.0
  config file = /Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg
  configured module search path = [u'/Users/gennadii_aleksandrov/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/gennadii_aleksandrov/Library/Python/2.7/lib/python/site-packages/ansible
  executable location = /Users/Gennadii_Aleksandrov/Library/Python/2.7/bin/ansible
  python version = 2.7.14 (default, Mar  9 2018, 23:57:12) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
```

##### CONFIGURATION
```
DEFAULT_HOST_LIST(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = [u'/Users/gennadii_aleksandrov/r/ansible-control-repo/inventory/local']
DEFAULT_REMOTE_USER(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = ubuntu
HOST_KEY_CHECKING(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
RETRY_FILES_ENABLED(/Users/gennadii_aleksandrov/r/ansible-control-repo/ansible.cfg) = False
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
cloudfront_facts returns only first 100 CloudFront distributions with `summary: true`.
Also, it returns `Error unable to source a distribution id from domain_name_alias` when trying to find distribution with `domain_name_alias` if that distribution is not on the first page of Amazon API response.

##### STEPS TO REPRODUCE
0. Create more than 100 CloudFront distributions :)
Then there are two cases here:
1. First - distributions summary:
<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        summary: true
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```
2. Second - try to fetch info on distribution which is not in first 100:
```yaml
---
- hosts: localhost
  tasks:
    - name: Check if Cloudfront distribution exists for the site
      tags: cloudfront
      cloudfront_facts:
        distribution: true
        domain_name_alias: '{{ fqdn }}'
      ignore_errors: true
      register: cloudfront_facts_result
    - debug: msg={{ cloudfront_facts_result }}
```



##### EXPECTED RESULTS
1. Full list of distributions with more than 100 elements.
2. `cloudfront` dict containing distribution information.

##### ACTUAL RESULTS
1. You will only see first 100 distributions.
2. You will see `Error unable to source a distribution id from domain_name_alias`



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/cloudfront_facts.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudfront_facts.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wilvk
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've taken a glance at `boto3` docs and it seems that CloudFront is using `NextMarker` instead of `NextToken` for pagination.
Line 512 in `cloudfront_facts.py`:
```
            args['NextMarker'] = response.get('NextMarker')
```
The issue is that `response` never has `NextMarker`. This key belongs to `DistributionList` (or `StreamingDistributionList` etc.) key of `response`
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/cloudfront_facts.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudfront_facts.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wilvk
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've taken a glance at `boto3` docs and it seems that CloudFront is using `NextMarker` instead of `NextToken` for pagination.
Line 512 in `cloudfront_facts.py`:
```
            args['NextMarker'] = response.get('NextMarker')
```
The issue is that `response` never has `NextMarker`. This key belongs to `DistributionList` (or `StreamingDistributionList` etc.) key of `response`
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/cloudfront_facts.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/cloudfront_facts.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @willthames @wilvk
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've taken a glance at `boto3` docs and it seems that CloudFront is using `NextMarker` instead of `NextToken` for pagination.
Line 512 in `cloudfront_facts.py`:
```
            args['NextMarker'] = response.get('NextMarker')
```
The issue is that `response` never has `NextMarker`. This key belongs to `DistributionList` (or `StreamingDistributionList` etc.) key of `response`

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/cloudfront_facts.py b/lib/ansible/modules/cloud/amazon/cloudfront_facts.py
--- a/lib/ansible/modules/cloud/amazon/cloudfront_facts.py
+++ b/lib/ansible/modules/cloud/amazon/cloudfront_facts.py
@@ -509,8 +509,12 @@ def paginated_response(self, func, result_key=""):
             else:
                 result = response.get(result_key)
             results.update(result)
-            args['NextToken'] = response.get('NextToken')
-            loop = args['NextToken'] is not None
+            args['Marker'] = response.get('NextMarker')
+            for key in response.keys():
+                if key.endswith('List'):
+                    args['Marker'] = response[key].get('NextMarker')
+                    break
+            loop = args['Marker'] is not None
         return results
 
     def keyed_list_helper(self, list_to_key):


Test patch: