Repository: ansible/ansible

Issue:
EC2 user-data error when spot_price != null
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ec2

##### ANSIBLE VERSION
```
$ ansible --version
ansible 2.4.2.0
  config file = /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg
  configured module search path = ['/Users/romanvg/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible
  executable location = /Users/romanvg/.miniconda/envs/ansible24/bin/ansible
  python version = 3.6.3 | packaged by conda-forge | (default, Nov  4 2017, 10:13:32) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
```

##### CONFIGURATION
$ ansible-config dump --only-changed
ANSIBLE_SSH_ARGS(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = -o ControlMaster=auto
HOST_KEY_CHECKING(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = False
INVENTORY_ENABLED(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = ['host_list', 'script', 'yaml', 'ini', 'constructed']
RETRY_FILES_ENABLED(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = False

##### OS / ENVIRONMENT
N/A

##### SUMMARY

There's a parsing error on the base64 encoding of `user-data` when spot-price has a defined price.

https://github.com/brainstorm/pcgr-deploy/blob/master/ansible/aws.yml#L16

##### STEPS TO REPRODUCE

`ansible-playbook aws.yaml`

```yaml
---
- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Launch EC2 instance
      ec2:
         group: "{{ aws.security_group }}"
         instance_type: "{{ aws.instance_type }}"
         instance_profile_name: "{{ aws.instance_profile_name }}"
         instance_tags:
             Name: pcgr "{{ pcgr.release }}"
             name: pcgr
             pcgrexecid: "{{ pcgr.execution_id }}"
         spot_price: "{{ aws.spot_price }}"
         user_data: "{{ lookup('file', 'files/s3_user_data.sh') }}"
         image: "{{ aws.image_id }}"
         keypair: "{{ aws.keypair }}"
         region: "{{ aws.region }}"
         zone: "{{ aws.zone }}"
         wait: yes
         volumes:
           - device_name: "{{ aws.volume_device }}"
             volume_type: gp2
             volume_size: "{{ aws.volume_size }}"
             delete_on_termination: true
         count_tag:
           name: pcgr
         exact_count: 1
      register: ec2

    - name: Add new instance to host group
      add_host:
        hostname: "{{ item.public_dns_name }}"
        groupname: launched
      with_items: "{{ ec2.instances }}"

    - name: Wait for SSH to come up
      wait_for:
        host: "{{ item.public_dns_name }}"
        port: 22
        delay: 60
        timeout: 320
        state: started
      with_items: "{{ ec2.instances }}"
```

##### EXPECTED RESULTS
Clean instantiation of an ec2 instance, loading its corresponding user-data:

https://github.com/brainstorm/pcgr-deploy/blob/master/ansible/files/s3_user_data.sh

##### ACTUAL RESULTS
```
(ansible24) brainstorm:ansible romanvg$ ansible-playbook aws.yml -vvvv
ansible-playbook 2.4.2.0
  config file = /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg
  configured module search path = ['/Users/romanvg/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible
  executable location = /Users/romanvg/.miniconda/envs/ansible24/bin/ansible-playbook
  python version = 3.6.3 | packaged by conda-forge | (default, Nov  4 2017, 10:13:32) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
Using /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg as config file
setting up inventory plugins
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available

Loading callback plugin default of type stdout, v2.0 from /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible/plugins/callback/__init__.py

PLAYBOOK: aws.yml ************************************************************************************************************************************************************************************************************************************************
1 plays in aws.yml

PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Launch EC2 instance] ***************************************************************************************************************************************************************************************************************************************
task path: /Users/romanvg/dev/pcgr-deploy/ansible/aws.yml:6
Using module file /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible/modules/cloud/amazon/ec2.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: romanvg
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460 `" && echo ansible-tmp-1516170269.445788-236322726281460="` echo /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/tmpk8nz061i TO /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/romanvg/.miniconda/envs/ansible24/bin/python /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py; rm -rf "/Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1725, in <module>
    main()
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1719, in main
    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 963, in enforce_count
    = create_instances(module, ec2, vpc, override_count=to_create)
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1253, in create_instances
    res = ec2.request_spot_instances(spot_price, **params)
  File "/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/boto/ec2/connection.py", line 1616, in request_spot_instances
    params['%s.UserData' % ls] = base64.b64encode(user_data)
  File "/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1725, in <module>\n    main()\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1719, in main\n    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 963, in enforce_count\n    = create_instances(module, ec2, vpc, override_count=to_create)\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1253, in create_instances\n    res = ec2.request_spot_instances(spot_price, **params)\n  File \"/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/boto/ec2/connection.py\", line 1616, in request_spot_instances\n    params['%s.UserData' % ls] = base64.b64encode(user_data)\n  File \"/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/base64.py\", line 58, in b64encode\n    encoded = binascii.b2a_base64(s, newline=False)\nTypeError: a bytes-like object is required, not 'str'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 0
}

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1
```

/cc @vladsaveliev


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @lwade @tgerla @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Can anyone confirm this bug?
@brainstorm I don't use ec2 but I do fix Python3 bugs.  Can you test whether this PR fixes your issue? https://github.com/ansible/ansible/pull/37628

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2.py b/lib/ansible/modules/cloud/amazon/ec2.py
--- a/lib/ansible/modules/cloud/amazon/ec2.py
+++ b/lib/ansible/modules/cloud/amazon/ec2.py
@@ -620,15 +620,15 @@
 
 '''
 
-import traceback
 import time
+import traceback
 from ast import literal_eval
-from ansible.module_utils.six import get_function_code, string_types
-from ansible.module_utils._text import to_text
+from distutils.version import LooseVersion
+
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.ec2 import get_aws_connection_info, ec2_argument_spec, ec2_connect
-from distutils.version import LooseVersion
-from ansible.module_utils.six import string_types
+from ansible.module_utils.six import get_function_code, string_types
+from ansible.module_utils._text import to_bytes, to_text
 
 try:
     import boto.ec2
@@ -1121,7 +1121,7 @@ def create_instances(module, ec2, vpc, override_count=None):
                       'instance_type': instance_type,
                       'kernel_id': kernel,
                       'ramdisk_id': ramdisk,
-                      'user_data': user_data}
+                      'user_data': to_bytes(user_data, errors='surrogate_or_strict')}
 
             if ebs_optimized:
                 params['ebs_optimized'] = ebs_optimized


Test patch: