Repository: ansible-collections/amazon.aws

Issue:
ec2_instance module broken in Python 3.8 - dict keys modified during iteration
### Summary

The `ec2_instance` AWS module doesn't work in Python 3.8. This is due to the fact that the `find_instances` method pops keys from the `filters` dictionary while it is being iterated:

``` console
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RuntimeError: dictionary keys changed during iteration
fatal: [rmahroua-test-routing-AzB]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n 
 File \"/home/rmahroua/.ansible/tmp/ansible-tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", 
line 102, in <module>\n    _ansiballz_main()\n  File \"/home/rmahroua/.ansible/tmp/ansible-
tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", line 94, in _ansiballz_main\n    
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/rmahroua/.ansible/tmp/ansible-
tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", line 40, in invoke_module\n    
runpy.run_module(mod_name='ansible.modules.cloud.amazon.ec2_instance', init_globals=None, run_name='__main__', 
alter_sys=True)\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, 
init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    
_run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, 
run_globals)\n  File \"/tmp/ansible_ec2_instance_payload_j_rhever/ansible_ec2_instance_payload.zip/ansible/modules
/cloud/amazon/ec2_instance.py\", line 1710, in <module>\n  File \"/tmp/ansible_ec2_instance_payload_j_rhever
/ansible_ec2_instance_payload.zip/ansible/modules/cloud/amazon/ec2_instance.py\", line 1681, in main\n  File \"/tmp
/ansible_ec2_instance_payload_j_rhever/ansible_ec2_instance_payload.zip/ansible/modules/cloud/amazon
/ec2_instance.py\", line 1297, 
in find_instances
RuntimeError: dictionary keys changed during iteration", 
"module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
```
The fix consists in adding the sanitized keys to a new dictionary and passing it to the paginator instead of modifying the existing one as follows:

``` python
def find_instances(ec2, ids=None, filters=None):
    paginator = ec2.get_paginator('describe_instances')
    if ids:
        return list(paginator.paginate(
            InstanceIds=ids,
        ).search('Reservations[].Instances[]'))
    elif filters is None:
        module.fail_json(msg="No filters provided when they were required")
    elif filters is not None:
        sanitized_filters = dict()
        for key in filters.keys():
            if not key.startswith("tag:"):
                sanitized_filters[key.replace("_", "-")] = filters[key]
        return list(paginator.paginate(
            Filters=ansible_dict_to_boto3_filter_list(sanitized_filters)
        ).search('Reservations[].Instances[]'))
    return []
```

### Issue Type

Bug Report

### Component Name

ec2_instance

### Ansible Version

```console (paste below)
ansible 2.9.2
  config file = /mnt/ansicloud/ansicloud-full/playbooks/aws/ansible.cfg
  configured module search path = ['/home/rmahroua/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rmahroua/.local/lib/python3.8/site-packages/ansible
  executable location = /home/rmahroua/.local/bin/ansible
  python version = 3.8.10 (default, May  4 2021, 00:00:00) [GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]
```


### Collection Versions

``` console (paste below)
$ ansible-galaxy collection list
```


### AWS SDK versions

```console (paste below)
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /home/rmahroua/.local/lib/python3.8/site-packages
Requires:
Required-by: aws-automation
---
Name: boto3
Version: 1.21.11
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: aws-automation, aws-shell
---
Name: botocore
Version: 1.24.11
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: aws-automation, awscli, boto3, s3transfer
```


### Configuration

```console (paste below)
ANSIBLE_PIPELINING(/mnt/ansicloud/ansicloud-full/playbooks/aws/ansible.cfg) = False
ANSIBLE_SSH_ARGS(/mnt/ansicloud/ansicloud-full/playbooks/aws/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=120s -o PreferredAuthentications=publickey
DEFAULT_CALLBACK_WHITELIST(/mnt/ansicloud/ansicloud-full/playbooks/aws/ansible.cfg) = ['json', 'timer']
DEFAULT_ROLES_PATH(/mnt/ansicloud/ansicloud-full/playbooks/aws/ansible.cfg) = ['/mnt/ansicloud/ansicloud-full/playbooks/aws/roles']

```


### OS / Environment

Fedora 33

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
``` yaml (paste below)
- ec2:
    key_name: mykey
    instance_type: t2.micro
    image: ami-123456
    wait: yes
    group: webserver
    count: 3
    vpc_subnet_id: subnet-29e63245
    assign_public_ip: yes
```


### Expected Results

EC2 instances should provision instead of the module failing with an error of `RuntimeError: dictionary keys changed during iteration"`

### Actual Results

```console (paste below)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RuntimeError: dictionary keys changed during iteration
fatal: [rmahroua-test-routing-AzB]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n 
 File \"/home/rmahroua/.ansible/tmp/ansible-tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", 
line 102, in <module>\n    _ansiballz_main()\n  File \"/home/rmahroua/.ansible/tmp/ansible-
tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", line 94, in _ansiballz_main\n    
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/rmahroua/.ansible/tmp/ansible-
tmp-1647283103.3464394-212411615216081/AnsiballZ_ec2_instance.py\", line 40, in invoke_module\n    
runpy.run_module(mod_name='ansible.modules.cloud.amazon.ec2_instance', init_globals=None, run_name='__main__', 
alter_sys=True)\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, 
init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    
_run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, 
run_globals)\n  File \"/tmp/ansible_ec2_instance_payload_j_rhever/ansible_ec2_instance_payload.zip/ansible/modules
/cloud/amazon/ec2_instance.py\", line 1710, in <module>\n  File \"/tmp/ansible_ec2_instance_payload_j_rhever
/ansible_ec2_instance_payload.zip/ansible/modules/cloud/amazon/ec2_instance.py\", line 1681, in main\n  File \"/tmp
/ansible_ec2_instance_payload_j_rhever/ansible_ec2_instance_payload.zip/ansible/modules/cloud/amazon
/ec2_instance.py\", line 1297, 
in find_instances
RuntimeError: dictionary keys changed during iteration", 
"module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
```

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/ec2_instance.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/ec2_instance.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jillr @ryansb @s-hertel @tremble
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@Razique Thank you for raising this. Would you be willing to open a pull request with the patch?
Yup! I can work on this. I'll have to see whether contribution guide differs between AWS community collections and "core".
@Razique Awesome! Let me know if you need any help. If you want, you can also reach out to us on IRC #ansible-aws.

What you need is a changelog fragment https://docs.ansible.com/ansible/latest/community/development_process.html#id11 and also an integration test for this change.

Resolving patch:
diff --git a/plugins/modules/ec2_instance.py b/plugins/modules/ec2_instance.py
--- a/plugins/modules/ec2_instance.py
+++ b/plugins/modules/ec2_instance.py
@@ -1463,6 +1463,8 @@ def change_network_attachments(instance, params):
 
 def find_instances(ids=None, filters=None):
     paginator = client.get_paginator('describe_instances')
+    sanitized_filters = dict()
+
     if ids:
         params = dict(InstanceIds=ids)
     elif filters is None:
@@ -1470,13 +1472,16 @@ def find_instances(ids=None, filters=None):
     else:
         for key in list(filters.keys()):
             if not key.startswith("tag:"):
-                filters[key.replace("_", "-")] = filters.pop(key)
-        params = dict(Filters=ansible_dict_to_boto3_filter_list(filters))
+                sanitized_filters[key.replace("_", "-")] = filters[key]
+            else:
+                sanitized_filters[key] = filters[key]
+        params = dict(Filters=ansible_dict_to_boto3_filter_list(sanitized_filters))
 
     try:
         results = _describe_instances(**params)
     except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:
         module.fail_json_aws(e, msg="Could not describe instances")
+
     retval = list(results)
     return retval
 


Test patch: