Repository: ansible/ansible
Pull request: Allow botocore configuration for aws modules
##### SUMMARY
~WIP while still looking at a possible solution for boto modules.~

This PR allows UserAgent to be set for boto modules and allows general configuration for boto3 modules. Many boto configurations can be set by a boto config file like connection timeout and proxy settings. UserAgent is not able to be configured in that way. In boto3 most of those options have been removed and are only modifiable via code. This addresses both problems.

Tested with CloudTrail.

Fix #55182

##### ISSUE TYPE
- Feature Pull Request

##### COMPONENT NAME
AWS modules

Example use:
```
    - name: create bucket
      aws_s3:
        aws_config:
          user_agent: "{{ custom_user_agent }}"
        bucket: "{{ bucket }}"
        region: us-east-1
        mode: create
```

Problem statement:
Add User-Agent Config for AWS Modules
<!--- Verify first that your feature was not already discussed on GitHub -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Describe the new feature/improvement briefly below -->
Add the ability to specify a user agent string for boto/3

##### ISSUE TYPE
- Feature Idea

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
aws

##### ADDITIONAL INFORMATION
<!--- Describe how the feature would be used, why it is needed and what it would solve -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
aws_access_key: redact
aws_secret_key: redact
aws_useragent: "Mozilla/5.0 (X11; Linux x86_64)"
```

<!--- HINT: You can also paste gist.github.com links for larger files -->

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/module_utils/aws/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/aws/__init__.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->

Commit message: play nicely with modules that might be modifying config

Patch to review:
diff --git a/lib/ansible/module_utils/ec2.py b/lib/ansible/module_utils/ec2.py
index d8d986dc90315a..4c5b5c94131c9b 100644
--- a/lib/ansible/module_utils/ec2.py
+++ b/lib/ansible/module_utils/ec2.py
@@ -28,6 +28,7 @@
 
 import os
 import re
+import sys
 import traceback
 
 from ansible.module_utils.ansible_release import __version__
@@ -134,13 +135,15 @@ def _boto3_conn(conn_type=None, resource=None, region=None, endpoint=None, **par
                          'the conn_type parameter in the boto3_conn function '
                          'call')
 
-    if params.get('config'):
-        config = params.pop('config')
-        config.user_agent_extra = 'Ansible/{0}'.format(__version__)
-    else:
-        config = botocore.config.Config(
-            user_agent_extra='Ansible/{0}'.format(__version__),
-        )
+    config = botocore.config.Config(
+        user_agent_extra='Ansible/{0}'.format(__version__),
+    )
+
+    if params.get('config') is not None:
+        config = config.merge(params.pop('config'))
+    if params.get('aws_config') is not None:
+        config = config.merge(params.pop('aws_config'))
+
     session = boto3.session.Session(
         profile_name=profile,
     )
@@ -184,6 +187,7 @@ def aws_common_argument_spec():
         validate_certs=dict(default=True, type='bool'),
         security_token=dict(aliases=['access_token'], no_log=True),
         profile=dict(),
+        aws_config=dict(type='dict'),
     )
 
 
@@ -209,6 +213,7 @@ def get_aws_connection_info(module, boto3=False):
     region = module.params.get('region')
     profile_name = module.params.get('profile')
     validate_certs = module.params.get('validate_certs')
+    config = module.params.get('aws_config')
 
     if not ec2_url:
         if 'AWS_URL' in os.environ:
@@ -307,6 +312,13 @@ def get_aws_connection_info(module, boto3=False):
 
         boto_params['validate_certs'] = validate_certs
 
+    if config is not None:
+        if HAS_BOTO3 and boto3:
+            boto_params['aws_config'] = botocore.config.Config(**config)
+        elif HAS_BOTO and not boto3:
+            if 'user_agent' in config:
+                sys.modules["boto.connection"].UserAgent = config['user_agent']
+
     for param, value in boto_params.items():
         if isinstance(value, binary_type):
             boto_params[param] = text_type(value, 'utf-8', 'strict')
diff --git a/lib/ansible/plugins/doc_fragments/aws.py b/lib/ansible/plugins/doc_fragments/aws.py
index 84c3a4048a0086..c37b582401dd93 100644
--- a/lib/ansible/plugins/doc_fragments/aws.py
+++ b/lib/ansible/plugins/doc_fragments/aws.py
@@ -50,6 +50,13 @@ class ModuleDocFragment(object):
       - Uses a boto profile. Only works with boto >= 2.24.0.
     type: str
     version_added: "1.6"
+  aws_config:
+    description:
+      - A dictionary to modify the botocore configuration.
+      - Parameters can be found at U(https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore.config.Config).
+      - Only the 'user_agent' key is used for boto modules. See U(http://boto.cloudhackers.com/en/latest/boto_config_tut.html#boto) for more boto configuration.
+    type: dict
+    version_added: "2.9"
 requirements:
   - python >= 2.6
   - boto


Code review comments:
diff_hunk: @@ -50,6 +50,13 @@ class ModuleDocFragment(object):
       - Uses a boto profile. Only works with boto >= 2.24.0.
     type: str
     version_added: "1.6"
+  aws_config:
+    description:
+      - A dictionary to modify the botocore configuration.
+      - Parameters can be found at U(https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore.config.Config).
+      - Only the 'user_agent' key is used for boto modules. See U(http://boto.cloudhackers.com/en/latest/boto_config_tut.html#boto) for more boto configuration.
+    type: dict
+    version_added: "2.9"
path: lib/ansible/plugins/doc_fragments/aws.py
text: @user1:
```suggestion
    version_added: "2.10"
```