Repository: ansible-collections/amazon.aws

Issue:
aws_ec2 use_ssm_inventory fails with a large number of instances
### Summary

After upgrading to version 6.0.0 and enabling `use_ssm_inventory`, dynamic inventory graphing fails with a warning. Our setup has many instances, and so the list of instance ids is well over the 40 character limit for the aws ssm get-inventory api endpoint.

### Issue Type

Bug Report

### Component Name

aws_ec2

### Ansible Version

```console
❯ ansible --version
ansible [core 2.15.3]
```


### Collection Versions

```console
❯ ansible-galaxy collection list

# /opt/homebrew/lib/python3.11/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.0.0  
```


### AWS SDK versions

```console
❯ pip3 show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.152
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.29.152
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer
```


### Configuration

_No response_

### OS / Environment

_No response_

### Steps to Reproduce

inventory/aws_ec2.yml
```yaml
---
##
# AWS EC2 Inventory
##

# Dynamic AWS inventory: Uses the aws_ec2 plugin to generate a dynamic inventory.
# For more info and examples see:
#  - https://docs.ansible.com/ansible/latest/collections/amazon/aws/docsite/aws_ec2_guide.html
#  - https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html#id3
#
# Usage:
# - Run `ansible-inventory --graph` to list the available hosts and groups.

# Enable the aws_ec2 plugin
plugin: aws_ec2

# Enables fetching additional EC2 instance information from the AWS Systems Manager (SSM) inventory service into hostvars.
use_ssm_inventory: true

##
# Hosts
##

# This controls how we display hostnames.
hostnames:

  # Using the Hostname tag is more aligned with the legacy static inventory and easier to reference for humans. However using `instance-id` might be more practical in the future with dynamic groups.
  - tag:Hostname

##
# Groups
##

# Generate dynamic groups to run playbooks against.
keyed_groups:
  # Using tags to generate groups. This gives something similar to our legacy static inventory with familiar names to identify instances with.

  # By Application name
  - key: tags.Application
    prefix: App

  # By Environment
  - key: tags.Environment
    prefix: Env


##
# Compose
##

# Compose creates vars from Jinja2 templates but also allows us to set some connection details for the dynamic inventory.
compose:
  # Use SSM Agent to connect to the instance ID. This means no ssh keys are needed..
  ansible_host: instance_id
  ansible_connection: 'community.aws.aws_ssm'
```
command used:
```command
❯ ansible-inventory --graph
``` 


### Expected Results

I expect to get a graph output of instances grouped appropriately. This occurs as expected with the above inventory/aws_ec2.yml with amazon.aws collection version 5.5.0 and earlier.

### Actual Results

```console
❯ ansible-inventory --graph
[WARNING]:  * Failed to parse /Users/<>/inventory/aws_ec2.yaml with ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin: An error occurred
(ValidationException) when calling the GetInventory operation: 1 validation error detected: Value '[<instance ids list>]' at 'filters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 40
[WARNING]: Unable to parse /Users/<>/inventory/aws_ec2.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/inventory/aws_ec2.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/aws_ec2.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
The exact same thing is happening to me:

...at 'filters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 40
same issue here
@abikouo The option `use_ssm_inventory` limits to use only if the number of instances in the syncing account is less than the allowed limit of instances for SSM `send_command`. Is it possible to sync the inventory in a batch with the option to apply the count of instances so that we can use it to keep it under the allowed limit of instances for SSM `send_command`?

Resolving patch:
diff --git a/plugins/inventory/aws_ec2.py b/plugins/inventory/aws_ec2.py
--- a/plugins/inventory/aws_ec2.py
+++ b/plugins/inventory/aws_ec2.py
@@ -649,8 +649,8 @@ def _query(self, regions, include_filters, exclude_filters, strict_permissions,
         return {"aws_ec2": instances}
 
     def _add_ssm_information(self, connection, instances):
-        filters = [{"Key": "AWS:InstanceInformation.InstanceId", "Values": [x["InstanceId"] for x in instances]}]
-        result = _get_ssm_information(connection, filters)
+        instance_ids = [x["InstanceId"] for x in instances]
+        result = self._get_multiple_ssm_inventories(connection, instance_ids)
         for entity in result.get("Entities", []):
             for x in instances:
                 if x["InstanceId"] == entity["Id"]:
@@ -659,6 +659,19 @@ def _add_ssm_information(self, connection, instances):
                         x["SsmInventory"] = content[0]
                     break
 
+    def _get_multiple_ssm_inventories(self, connection, instance_ids):
+        result = {}
+        # SSM inventory filters Values list can contain a maximum of 40 items so we need to retrieve 40 at a time
+        # https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_InventoryFilter.html
+        while len(instance_ids) > 40:
+            filters = [{"Key": "AWS:InstanceInformation.InstanceId", "Values": instance_ids[:40]}]
+            result.update(_get_ssm_information(connection, filters))
+            instance_ids = instance_ids[40:]
+        if instance_ids:
+            filters = [{"Key": "AWS:InstanceInformation.InstanceId", "Values": instance_ids}]
+            result.update(_get_ssm_information(connection, filters))
+        return result
+
     def _populate(
         self,
         groups,


Test patch:
diff --git a/tests/unit/plugins/inventory/test_aws_ec2.py b/tests/unit/plugins/inventory/test_aws_ec2.py
--- a/tests/unit/plugins/inventory/test_aws_ec2.py
+++ b/tests/unit/plugins/inventory/test_aws_ec2.py
@@ -649,3 +649,39 @@ def test_inventory__add_ssm_information(m_get_ssm_information, inventory):
 
     filters = [{"Key": "AWS:InstanceInformation.InstanceId", "Values": [x["InstanceId"] for x in instances]}]
     m_get_ssm_information.assert_called_once_with(connection, filters)
+
+
+@patch("ansible_collections.amazon.aws.plugins.inventory.aws_ec2._get_ssm_information")
+def test_inventory__get_multiple_ssm_inventories(m_get_ssm_information, inventory):
+    instances = [{"InstanceId": f"i-00{i}", "Name": f"instance {i}"} for i in range(41)]
+    result = {
+        "StatusCode": 200,
+        "Entities": [
+            {
+                "Id": f"i-00{i}",
+                "Data": {
+                    "AWS:InstanceInformation": {
+                        "Content": [{"os_type": "Linux", "os_name": "Fedora", "os_version": 37}]
+                    }
+                },
+            }
+            for i in range(41)
+        ],
+    }
+    m_get_ssm_information.return_value = result
+
+    connection = MagicMock()
+
+    expected = [
+        {
+            "InstanceId": f"i-00{i}",
+            "Name": f"instance {i}",
+            "SsmInventory": {"os_type": "Linux", "os_name": "Fedora", "os_version": 37},
+        }
+        for i in range(41)
+    ]
+
+    inventory._add_ssm_information(connection, instances)
+    assert expected == instances
+
+    assert 2 == m_get_ssm_information.call_count