Repository: ansible/ansible

Issue:
AWS boto3 - profile support
**Issue Type:**

Bug Report

**Ansible Version**
ansible 2.0.0 (detached HEAD de54f26376) last updated 2015/11/03 08:46:06 (GMT +1000)
  lib/ansible/modules/core: (detached HEAD e318be30ff) last updated 2015/11/03 08:45:42 (GMT +1000)
  lib/ansible/modules/extras: (detached HEAD 80f198a82f) last updated 2015/11/03 08:45:42 (GMT +1000)

Running Ansible from source - v2.0.0-0.4.beta2

**Environment:**
OSX 10.10.5
Running Ansible from source - v2.0.0-0.4.beta2
Running boto (2.38.0) and boto3 (1.1.4)
Python 2.7.10

**Summary:**

When utilising boto3_conn from module_utils/ec2.py and specifying a boto profile_name, the connection fails due to profile_name not being a valid boto3.session.Session().client() parameter.

_TypeError: client() got an unexpected keyword argument 'profile_name'_

**Steps To Reproduce:**

Creating a new module using boto3 for listing Route53 entries, host zones etc and allowing the user to provide a boto profile - same behaviour as in boto (version 2).

Connection setup in new module:

```
try:
    region, ec2_url, aws_connect_kwargs = get_aws_connection_info(module, boto3=True)
    route53 = boto3_conn(module, conn_type='client', resource='route53', **aws_connect_kwargs)
except boto.exception.NoAuthHandlerFound, e:
    module.fail_json(msg="Can't authorize connection - "+str(e))
```

boto3_conn section in question:

```
elif conn_type == 'client':
    client = boto3.session.Session().client(resource, region_name=region, endpoint_url=endpoint, **params)
return client
```

Setup a play using the new module and specify a profile: _name_

**Expected Results:**

```
TASK [route53_entries : Testing custom module] *********************************
 ok: [127.0.0.1] => {"HostedZones": [{ dictionary of returned hosted zones from AWS }}
```

**Actual Results:**

```
TASK [route53_entries : Testing custom module] *********************************
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1446523327.09-207574321397669/route53_facts", line 2208, in <module>
main()
  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1446523327.09-207574321397669/route53_facts", line 67, in main
    route53 = boto3_conn(module, conn_type='client', resource='route53', **aws_connect_kwargs)
  File "/Users/Etherdaemon/.ansible/tmp/ansible-tmp-1446523327.09-207574321397669/route53_facts", line 2033, in boto3_conn
    client = boto3.session.Session().client(resource, region_name=region, endpoint_url=endpoint, profile_name=profile_name, **params)
TypeError: client() got an unexpected keyword argument 'profile_name'

fatal: [127.0.0.1]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"profile": "personal", "type": "A"}, "module_name": "route53_facts"}, "parsed": false}
```

**Proposed fix:**

My initial thoughts are to do the following - I've tested this for just my own scenario so obviously a lot more thought has to go in for everyone's use cases.

Update module_utils/ec2.py with the following:
- method: get_aws_connection_info
  - Remove profile_name from boto_params dictionary for boto3 requests
  - move existing return region, ec2_url, boto_params to be directly underneath the boto/boto3 else statement (to cater for boto 2 connections)
  - add a new return statement under the boto3 if statement to include the new profile_name argument separately - return region, ec2_url, profile_name, boto_params
- method: boto3_conn
  - add a new profile_name=None input parameter
  - update the boto3.session statement to create the session with the profile and the client with the rest of the arguments.

This should then allow users to call the boto3_conn like so...

```
try:
    region, ec2_url, profile_name, aws_connect_kwargs = get_aws_connection_info(module, boto3=True)
    route53 = boto3_conn(module, conn_type='client', resource='route53', profile_name=profile_name, **aws_connect_kwargs)
except boto.exception.NoAuthHandlerFound, e:
    module.fail_json(msg="Can't authorize connection - "+str(e))
```



Discussion:
@willthames has brought up a good point on the fact that this does change the specification of the reply for existing modules calling the get_aws_connection_info method. It seems a bit hacky but I might update the merge request to return the additonal argument only if a profile_name is passed in. This should then allow boto profiles to be fixed while not affecting the existing setups that people are using.

@Etherdaemon, another alternative is to keep `boto_params` as is (and thus keep the existing API for `get_aws_connection_info`) and fix the `boto3_conn` to remove `profile_name` from `boto_params` and pass it into `Session`. That seems a simpler change, but relies on people using `boto3_conn`. 

@willthames Yes - fair point. I think you've done more contributions than me on Ansible - would that be the better way to go for this change? I'm happy to update accordingly. It does mean that if someone wants to use the return of get_aws_connection_info later with a different boto3 connection call, they would need to remove it out in their own methods as well.

@willthames, having a further think about it, I think the best option is what you suggested, otherwise writing modules for it would be quite messy with catering for either 3 values and 4 values being returned from that method. I'll work on the update and submit an updated PR.

I was considering how to add boto3 support to different aws modules, since boto3 supports a lot of things Ansible cant do currently (like tag elb instances, elasticache replication etc).

I think for existing modules the battle would be calling boto3 methods versus existing boto methods as the can be so different. It might be easier to pull the existing boto(2) code out of the main function and into a separate boto(2) function, and also have a separate boto(3) function. Then have a try and import both versions up the top to set the constants, then in main have an if else statement where boto3 is preferred over boto2. Another consideration would be the acceptable parameters required for boto2 versus boto3 and how to handle that. For the moment, my initial requirements can be done in a separate module, so I'm sort of cheating.

```
try:
    HAS_BOTO = True
except ImportError:
    HAS_BOTO = False

try:
    import boto3
    HAS_BOTO3 = True
except ImportError:
    HAS_BOTO3 = False


 def boto2_connection(module):
     try:
         conn = Route53Connection(**aws_connect_kwargs)
     except boto.exception.BotoServerError, e:
         module.fail_json(msg = e.error_message)

    #Go do boto2 functions

 def boto3_connection(module):
     try:
         region, ec2_url, aws_connect_kwargs = get_aws_connection_info(module, boto3=True)
         route53 = boto3_conn(module, conn_type='client', resource='route53', region=region, endpoint=ec2_url, **aws_connect_kwargs)
     except boto.exception.NoAuthHandlerFound, e:
         module.fail_json(msg="Can't authorize connection - "+str(e))

     #Go do boto3 functions

 def main():
     # Validate Requirements
     if not (HAS_BOTO or HAS_BOTO3):
         module.fail_json(msg='boto3 or boto is required.')

     if HAS_BOTO3:
         boto3_connection(module)
     elif HAS_BOTO:
         boto2_connection(module)
```


Resolving patch:
diff --git a/lib/ansible/module_utils/ec2.py b/lib/ansible/module_utils/ec2.py
--- a/lib/ansible/module_utils/ec2.py
+++ b/lib/ansible/module_utils/ec2.py
@@ -41,20 +41,25 @@
 
 
 def boto3_conn(module, conn_type=None, resource=None, region=None, endpoint=None, **params):
+    profile = params.pop('profile_name', None)
+    params['aws_session_token'] = params.pop('security_token', None)
+    params['verify'] = params.pop('validate_certs', None)
+
     if conn_type not in ['both', 'resource', 'client']:
         module.fail_json(msg='There is an issue in the code of the module. You must specify either both, resource or client to the conn_type parameter in the boto3_conn function call')
 
     if conn_type == 'resource':
-        resource = boto3.session.Session().resource(resource, region_name=region, endpoint_url=endpoint, **params)
+        resource = boto3.session.Session(profile_name=profile).resource(resource, region_name=region, endpoint_url=endpoint, **params)
         return resource
     elif conn_type == 'client':
-        client = boto3.session.Session().client(resource, region_name=region, endpoint_url=endpoint, **params)
+        client = boto3.session.Session(profile_name=profile).client(resource, region_name=region, endpoint_url=endpoint, **params)
         return client
     else:
-        resource = boto3.session.Session().resource(resource, region_name=region, endpoint_url=endpoint, **params)
-        client = boto3.session.Session().client(resource, region_name=region, endpoint_url=endpoint, **params)
+        resource = boto3.session.Session(profile_name=profile).resource(resource, region_name=region, endpoint_url=endpoint, **params)
+        client = boto3.session.Session(profile_name=profile).client(resource, region_name=region, endpoint_url=endpoint, **params)
         return client, resource
 
+
 def aws_common_argument_spec():
     return dict(
         ec2_url=dict(),
@@ -153,7 +158,6 @@ def get_aws_connection_info(module, boto3=False):
         if profile_name:
             boto_params['profile_name'] = profile_name
 
-
     else:
         boto_params = dict(aws_access_key_id=access_key,
                            aws_secret_access_key=secret_key,


Test patch: