Repository: ansible/ansible

Issue:
ec2.py limits elasticache nodes to first 100 nodes.
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
AWS dynamic inventory "ec2.py" shows only first 100 nodes for the Elasticache service. Rest of the nodes and their respective replication_group output get discarded making them unable to use in play.
This happens because of the boto "ElastiCache.Client.describe_cache_clusters" which limit the output to 100.
Ref: http://boto3.readthedocs.io/en/latest/reference/services/elasticache.html#ElastiCache.Client.describe_cache_clusters

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->

Inventory
```
/etc/ansible/inventory/ec2.py
```
##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
2.3
2.4
2.5
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
For Ansible 2.5
inventory      = /etc/ansible/inventory
library        = /usr/local/lib/python2.7/dist-packages/ansible#/usr/share/ansible

For Ansible 2.4/2.3
hostfile       = /etc/ansible/inventory
library        = /usr/local/lib/python2.7/dist-packages/ansible#/usr/share/ansible

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Centos 7
##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
/etc/ansible/inventory/ec2.py --refresh-cache
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Should list all the Elasticache nodes and the replication groups.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
List only first 100 instances/nodes of Elasticache and the respective replication groups, rest get truncated.
<!--- Paste verbatim command output between quotes below -->



Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @lwade @tgerla
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I have updated the component, It is related to AWS dynamic inventory file "ec2.py".

Resolving patch:
diff --git a/contrib/inventory/ec2.py b/contrib/inventory/ec2.py
--- a/contrib/inventory/ec2.py
+++ b/contrib/inventory/ec2.py
@@ -782,13 +782,26 @@ def get_elasticache_clusters_by_region(self, region):
         # ElastiCache boto module doesn't provide a get_all_instances method,
         # that's why we need to call describe directly (it would be called by
         # the shorthand method anyway...)
+        clusters = []
         try:
             conn = self.connect_to_aws(elasticache, region)
             if conn:
                 # show_cache_node_info = True
                 # because we also want nodes' information
-                response = conn.describe_cache_clusters(None, None, None, True)
-
+                _marker = 1
+                while _marker:
+                    if _marker == 1:
+                        _marker = None
+                    response = conn.describe_cache_clusters(None, None, _marker, True)
+                    _marker = response['DescribeCacheClustersResponse']['DescribeCacheClustersResult']['Marker']
+                    try:
+                        # Boto also doesn't provide wrapper classes to CacheClusters or
+                        # CacheNodes. Because of that we can't make use of the get_list
+                        # method in the AWSQueryConnection. Let's do the work manually
+                        clusters = clusters + response['DescribeCacheClustersResponse']['DescribeCacheClustersResult']['CacheClusters']
+                    except KeyError as e:
+                        error = "ElastiCache query to AWS failed (unexpected format)."
+                        self.fail_with_error(error, 'getting ElastiCache clusters')
         except boto.exception.BotoServerError as e:
             error = e.reason
 
@@ -802,16 +815,6 @@ def get_elasticache_clusters_by_region(self, region):
                 error = "Looks like AWS ElastiCache is down:\n%s" % e.message
             self.fail_with_error(error, 'getting ElastiCache clusters')
 
-        try:
-            # Boto also doesn't provide wrapper classes to CacheClusters or
-            # CacheNodes. Because of that we can't make use of the get_list
-            # method in the AWSQueryConnection. Let's do the work manually
-            clusters = response['DescribeCacheClustersResponse']['DescribeCacheClustersResult']['CacheClusters']
-
-        except KeyError as e:
-            error = "ElastiCache query to AWS failed (unexpected format)."
-            self.fail_with_error(error, 'getting ElastiCache clusters')
-
         for cluster in clusters:
             self.add_elasticache_cluster(cluster, region)
 


Test patch: