Repository: ansible/ansible

Issue:
ec2_facts throws str error in Python 3.5+
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
ec2_facts

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
```
# inventory file that I use via ansible-playbook -i
[all]
localhost

[all:vars]
ansible_python_interpreter=/usr/bin/python3
```

##### OS / ENVIRONMENT
Ubuntu 16.04.2 LTS
Python 3.5.2

##### SUMMARY
When trying to run `ec2_facts`, I get this error: `TypeError: a bytes-like object is required, not 'str'`
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
# in any playbook
  - name: Gather facts
    action: ec2_facts
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
That ec2_facts would work.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
It errors out with `a bytes-like object is required, not 'str'`

Just as a note, I thought this was connected to #17038, but perhaps not.
ec2_facts throws str error in Python 3.5+
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
ec2_facts

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
```
# inventory file that I use via ansible-playbook -i
[all]
localhost

[all:vars]
ansible_python_interpreter=/usr/bin/python3
```

##### OS / ENVIRONMENT
Ubuntu 16.04.2 LTS
Python 3.5.2

##### SUMMARY
When trying to run `ec2_facts`, I get this error: `TypeError: a bytes-like object is required, not 'str'`
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
# in any playbook
  - name: Gather facts
    action: ec2_facts
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
That ec2_facts would work.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
It errors out with `a bytes-like object is required, not 'str'`

Just as a note, I thought this was connected to #17038, but perhaps not.


Discussion:
cc @silviud
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
reproduced, and my output is:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 185, in <module>
    main()
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 175, in main
    ec2_facts = Ec2Metadata(module).run()
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 159, in run
    data = self._mangle_fields(self._data, self.uri_meta)
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 95, in _mangle_fields
    for key, value in fields.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'
```

In Python 3, dict.items() returns iterator object. dict.iteritems() is removed.
@ansibot python3 issue
https://docs.ansible.com/ansible/dev_guide/developing_python3.html
Just figured out a way to fix this bug, since it is in core module, I've got no idea how to fix it so submitted the other issue to ask around #23612.
This has been fixed in 2.3.
Fixed in #18859
@snowsky @s-hertel it look like the bug is still present in 2.3 : 

OS Version:
```
Mac OS X 10.11.6
```

Python version:
```
Python 2.7.12
```

Ansible version:
```
ansible --version
ansible 2.3.0.0
  config file = /Users/jchampseix/Documents/GitSpace/provisionning-cloud/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
```

Error output:
```
fatal: [XXX.XXX.XXX.XXX]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Shared connection to XXX.XXX.XXX.XXX closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 195, in <module>\r\n    main()\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 184, in main\r\n    ec2_facts = Ec2Metadata(module).run()\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 167, in run\r\n    self.fetch(self.uri_meta)  # populate _data\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 123, in fetch\r\n    subfields = raw_subfields.split('\\n')\r\nTypeError: a bytes-like object is required, not 'str'\r\n",
    "msg": "MODULE FAILURE",
    "rc": 0
}
```
@JulienChampseix, it's meant to be used on an EC2 instance. Per [the docs](http://docs.ansible.com/ansible/ec2_facts_module.html):
> This module fetches data from the metadata servers in ec2 (aws) as per http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html. **The module must be called from within the EC2 instance itself.**
cc @silviud
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
reproduced, and my output is:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 185, in <module>
    main()
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 175, in main
    ec2_facts = Ec2Metadata(module).run()
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 159, in run
    data = self._mangle_fields(self._data, self.uri_meta)
  File "/tmp/ansible_49xrua_1/ansible_module_ec2_facts.py", line 95, in _mangle_fields
    for key, value in fields.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'
```

In Python 3, dict.items() returns iterator object. dict.iteritems() is removed.
@ansibot python3 issue
https://docs.ansible.com/ansible/dev_guide/developing_python3.html
Just figured out a way to fix this bug, since it is in core module, I've got no idea how to fix it so submitted the other issue to ask around #23612.
This has been fixed in 2.3.
Fixed in #18859
@snowsky @s-hertel it look like the bug is still present in 2.3 : 

OS Version:
```
Mac OS X 10.11.6
```

Python version:
```
Python 2.7.12
```

Ansible version:
```
ansible --version
ansible 2.3.0.0
  config file = /Users/jchampseix/Documents/GitSpace/provisionning-cloud/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
```

Error output:
```
fatal: [XXX.XXX.XXX.XXX]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Shared connection to XXX.XXX.XXX.XXX closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 195, in <module>\r\n    main()\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 184, in main\r\n    ec2_facts = Ec2Metadata(module).run()\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 167, in run\r\n    self.fetch(self.uri_meta)  # populate _data\r\n  File \"/tmp/ansible_w95qd_uy/ansible_module_ec2_facts.py\", line 123, in fetch\r\n    subfields = raw_subfields.split('\\n')\r\nTypeError: a bytes-like object is required, not 'str'\r\n",
    "msg": "MODULE FAILURE",
    "rc": 0
}
```
@JulienChampseix, it's meant to be used on an EC2 instance. Per [the docs](http://docs.ansible.com/ansible/ec2_facts_module.html):
> This module fetches data from the metadata servers in ec2 (aws) as per http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html. **The module must be called from within the EC2 instance itself.**

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_facts.py b/lib/ansible/modules/cloud/amazon/ec2_facts.py
--- a/lib/ansible/modules/cloud/amazon/ec2_facts.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_facts.py
@@ -57,7 +57,7 @@
 
 import socket
 import re
-
+from ansible.module_utils._text import to_text
 socket.setdefaulttimeout(5)
 
 
@@ -97,7 +97,7 @@ def _fetch(self, url):
             data = response.read()
         else:
             data = None
-        return data
+        return to_text(data)
 
     def _mangle_fields(self, fields, uri, filter_patterns=['public-keys-0']):
         new_fields = {}
@@ -110,7 +110,7 @@ def _mangle_fields(self, fields, uri, filter_patterns=['public-keys-0']):
                 new_key = "".join(split_fields)
                 new_fields[self._prefix % new_key] = value
         for pattern in filter_patterns:
-            for key in new_fields.keys():
+            for key in dict(new_fields):
                 match = re.search(pattern, key)
                 if match:
                     new_fields.pop(key)
@@ -138,11 +138,10 @@ def fetch(self, uri, recurse=True):
 
     def fix_invalid_varnames(self, data):
         """Change ':'' and '-' to '_' to ensure valid template variable names"""
-        for (key, value) in data.items():
+        for key in data:
             if ':' in key or '-' in key:
                 newkey = key.replace(':', '_').replace('-', '_')
-                del data[key]
-                data[newkey] = value
+                data[newkey] = data.pop(key)
 
     def add_ec2_region(self, data):
         """Use the 'ansible_ec2_placement_availability_zone' key/value


Test patch: