Repository: ansible/ansible

Issue:
File name extension encoding issues in python 3 ansible 2.6.1
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->

I was trying to migrate from Ansible 2.4 on Python 2.7 to Ansible 2.6 on Python 3.7. Ansible stopped parsing my `inventory.config` which had a simple `constructed` plugin. Using Ansible 2.6 on Python 2.7 works fine.

From the debugging output:
```
inventory.config did not meet constructed requirements
```

My `inventory.config`, which worked great on Ansible 2.4 looked like:
```
---
plugin: constructed
groups:
  <group name>: <group definition>
  <group name>: <group definition>
  <group name>: <group definition>
```

I dug around in the `constructed.py` and found that it was failing on:
```
ext in ['.config'] + C.YAML_FILENAME_EXTENSIONS
```

I printed `ext`, and got `b'.config'`. If I changed the test to `ext in [b'.config'] + C.YAML_FILENAME_EXTENSIONS`, it worked.

@agaffney said on IRC:
>it's almost certainly byte vs. unicode. I just did a quick test in an interactive py3 interpreter, and was able to reproduce the 'in' keyword not matching for byte vs. unicode

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->

`constructed` plugin
`/usr/local/lib/python3.7/site-packages/ansible/plugins/inventory/constructed.py`

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.6.1
  config file = None
  configured module search path = ['/Users/mwraith/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

My `ansible.cfg` has the following inventory plugins:
```
[inventory]
enable_plugins = constructed, host_list, script, yaml, ini
```

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

macOS High Sierra 10.13.6

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
inventory.config:
```yaml
---
plugin: constructed
groups: []
```
playbook:
```yaml
- debug: msg="test"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

I'd expect it to not fail on parsing inventory.config.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
27634 1532372792.39836: Considering b'inventory.config'
 27634 1532372792.39843: Examining possible inventory source: b'/path/to/testinv/inventory.config'
 27634 1532372792.39849: Attempting to use plugin constructed (/usr/local/lib/python3.7/site-packages/ansible/plugins/inventory/constructed.py)
 27634 1532372792.39859: /path/to/testinv/inventory.config did not meet constructed requirements
```


Discussion:

Resolving patch:
diff --git a/lib/ansible/inventory/manager.py b/lib/ansible/inventory/manager.py
--- a/lib/ansible/inventory/manager.py
+++ b/lib/ansible/inventory/manager.py
@@ -223,7 +223,9 @@ def parse_source(self, source, cache=False):
         parsed = False
         display.debug(u'Examining possible inventory source: %s' % source)
 
+        # use binary for path functions
         b_source = to_bytes(source)
+
         # process directories as a collection of inventories
         if os.path.isdir(b_source):
             display.debug(u'Searching for inventory files in directory: %s' % source)
@@ -235,9 +237,9 @@ def parse_source(self, source, cache=False):
                     continue
 
                 # recursively deal with directory entries
-                b_fullpath = os.path.join(b_source, i)
-                parsed_this_one = self.parse_source(b_fullpath, cache=cache)
-                display.debug(u'parsed %s as %s' % (to_text(b_fullpath), parsed_this_one))
+                fullpath = to_text(os.path.join(b_source, i), errors='surrogate_or_strict')
+                parsed_this_one = self.parse_source(fullpath, cache=cache)
+                display.debug(u'parsed %s as %s' % (fullpath, parsed_this_one))
                 if not parsed:
                     parsed = parsed_this_one
         else:
@@ -258,7 +260,7 @@ def parse_source(self, source, cache=False):
 
                 # initialize and figure out if plugin wants to attempt parsing this file
                 try:
-                    plugin_wants = bool(plugin.verify_file(to_text(source)))
+                    plugin_wants = bool(plugin.verify_file(source))
                 except Exception:
                     plugin_wants = False
 
@@ -267,16 +269,16 @@ def parse_source(self, source, cache=False):
                         # in case plugin fails 1/2 way we dont want partial inventory
                         plugin.parse(self._inventory, self._loader, source, cache=cache)
                         parsed = True
-                        display.vvv('Parsed %s inventory source with %s plugin' % (to_text(source), plugin_name))
+                        display.vvv('Parsed %s inventory source with %s plugin' % (source, plugin_name))
                         break
                     except AnsibleParserError as e:
-                        display.debug('%s was not parsable by %s' % (to_text(source), plugin_name))
+                        display.debug('%s was not parsable by %s' % (source, plugin_name))
                         failures.append({'src': source, 'plugin': plugin_name, 'exc': e})
                     except Exception as e:
-                        display.debug('%s failed to parse %s' % (plugin_name, to_text(source)))
+                        display.debug('%s failed to parse %s' % (plugin_name, source))
                         failures.append({'src': source, 'plugin': plugin_name, 'exc': AnsibleError(e)})
                 else:
-                    display.debug('%s did not meet %s requirements' % (to_text(source), plugin_name))
+                    display.debug('%s did not meet %s requirements' % (source, plugin_name))
             else:
                 if not parsed and failures:
                     # only if no plugin processed files should we show errors.
@@ -285,7 +287,7 @@ def parse_source(self, source, cache=False):
                         if hasattr(fail['exc'], 'tb'):
                             display.vvv(to_text(fail['exc'].tb))
         if not parsed:
-            display.warning("Unable to parse %s as an inventory source" % to_text(source))
+            display.warning("Unable to parse %s as an inventory source" % source)
 
         # clear up, jic
         self._inventory.current_source = None


Test patch: