Repository: ansible/ansible

Issue:
Unable to parse inventory script when cache plugin specified
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
- Inventory Plugins
- Cache Plugins

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file =
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
N/A

##### SUMMARY
Running an ad hoc command using an inventory script is successful. If a cache plugin is specified, however, the same script cannot be parsed.

##### STEPS TO REPRODUCE
Confirm that ansible can run an ad-hoc command using an inventory script:

```
cat << EOF > fake_inv_script.py
#!/bin/env python

print("""{"all": {"hosts": ["localhost"]}, "_meta": {"hostvars": {"localhost": {"ansible_connection": "local"}}}}""")
EOF

ansible localhost -i fake_inv_script.py -m ping
```

Export information specifying a cache plugin and re-run the command:
```
export ANSIBLE_LIBRARY=/var/lib/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/library
export ANSIBLE_CACHE_PLUGINS=/var/lib/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/fact_caching
export ANSIBLE_CACHE_PLUGIN=tower
export ANSIBLE_CACHE_PLUGIN_CONNECTION=tcp://127.0.0.1:6564
export CALLBACK_CONNECTION='FAKE'
export FACT_QUEUE='FAKE'
```
(where the tower fact cache plugin is https://github.com/ansible/ansible-tower/blob/release_3.1.3/awx/plugins/fact_caching/tower.py)

##### EXPECTED RESULTS
Expect both ad-hoc commands to be successful. 

##### ACTUAL RESULTS
The second command with fact caching is unable to parse the inventory script results:

```
 10481 1497310098.63414: Examining possible inventory source: /root/tmp/fake_inv_script.py
 10481 1497310098.63421: Attempting to use plugin host_list
 10481 1497310098.63426: /root/tmp/fake_inv_script.py did not meet host_list requirements
 10481 1497310098.63431: Attempting to use plugin script
 10481 1497310098.66391: Attempting to use plugin yaml
 10481 1497310098.66404: /root/tmp/fake_inv_script.py did not meet yaml requirements
 10481 1497310098.66411: Attempting to use plugin ini
 [WARNING]:  * Failed to parse /root/tmp/fake_inv_script.py with script inventory plugin: 'unicode' object has no attribute 'keys'
```

Full output:
https://gist.github.com/jladdjr/bf7e8b8e0ba9d55e6fd3bf96f3e3a5a2#file-gistfile1-txt-L12-L27


Discussion:
Git bisect shows issue was introduced by https://github.com/ansible/ansible/commit/8f97aef1a365cbbbb822d6d09f96af17a076b295

ping @bcoca 
Prevents Tower 3.1.3 (latest release) from running scan jobs using Ansible devel.
I cannot reproduce with other cache plugins:
```
ansible -i ./fake_inv.py -m ping all
localhost | SUCCESS => {
    "changed": false,
    "failed": false,
    "ping": "pong"
}

```
can you show -vvvv output instead of debug?
FWIW, I cannot reproduce using a slightly modified version of the tower.py callback, that comments out everything related to kombu.  So effectively other than the `self.producer.publish`, everything else executes.

I also tested this by running the scan modules also, although against an Ubuntu box.

Of note here, I did this outside of tower, via ansible-playbook directly.
> can you show -vvvv output instead of debug?

@bcoca: https://gist.github.com/jladdjr/f740cd7992430dd0ff7ecca439706590
Talked with @chrismeyersfsu on the Tower team - he's looking at how we implement the fact caching plug-in to see if it could be the source of the issue
it seems we are getting a string back instead of a dictionary/cacheclass, I would expect a dictionary since that is basically what the cache plugin derives from.
Pinging @chrismeyersfsu for update
Tower's fact cache plugin assumes that value in FactCache `set` calls are dictionaries, or behave like dictionaries (instance of) `set(self, key, value)`. 

This commit https://github.com/ansible/ansible/commit/110fd917d6b140ee064cbf68c28d9b630d016af9#diff-d01d986987d9cd0f682890b51b98b04dR111 sends in a unicode value.

Maybe the value set in the cache should be set to the value after the loader processes it `processed = self.loader.load(data)`

@bcoca 
i can make that change, i was trying to keep the script's JSON output so it would be easy to debug. 

~~Also loader caches that raw data already so I did not see the need to do it x2~~ this might not be true anymore, i eliminated many unused  caches.
does this fix your issue?
```diff
diff --git a/lib/ansible/plugins/inventory/script.py b/lib/ansible/plugins/inventory/script.py
index 44088057b1..3c8739ee49 100644
--- a/lib/ansible/plugins/inventory/script.py
+++ b/lib/ansible/plugins/inventory/script.py
@@ -84,9 +84,7 @@ class InventoryModule(BaseInventoryPlugin):

         try:
             cache_key = self.get_cache_prefix(path)
-            if cache and cache_key in inventory.cache:
-                data = inventory.cache[cache_key]
-            else:
+            if not cache or cache_key not in inventory.cache:
                 try:
                     sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                 except OSError as e:
@@ -107,14 +105,12 @@ class InventoryModule(BaseInventoryPlugin):
                 except Exception as e:
                     raise AnsibleError("Inventory {0} contained characters that cannot be interpreted as UTF-8: {1}".format(path, to_native(e)))

-                if cache:
-                    inventory.cache[cache_key] = data
-
-            try:
-                processed = self.loader.load(data)
-            except Exception as e:
-                raise AnsibleError("failed to parse executable inventory script results from {0}: {1}\n{2}".format(path, to_native(e), err))
+                try:
+                    inventory.cache[cache_key] = self.loader.load(data, file_name=path)
+                except Exception as e:
+                    raise AnsibleError("failed to parse executable inventory script results from {0}: {1}\n{2}".format(path, to_native(e), err))

+            processed = inventory.cache[cache_key]
             if not isinstance(processed, Mapping):
                 raise AnsibleError("failed to parse executable inventory script results from {0}: needs to be a json dict\n{1}".format(path, err))

@@ -189,6 +185,6 @@ class InventoryModule(BaseInventoryPlugin):
         if out.strip() == '':
             return {}
         try:
-            return json_dict_bytes_to_unicode(self.loader.load(out))
+            return json_dict_bytes_to_unicode(self.loader.load(out, file_name=path))
         except ValueError:
             raise AnsibleError("could not parse post variable response: %s, %s" % (cmd, out))
```
@chrismeyersfsu in any case, it is a bad assumption that every key in the cache is a dict, there is no reason they cannot be strings

Resolving patch:
diff --git a/lib/ansible/plugins/inventory/script.py b/lib/ansible/plugins/inventory/script.py
--- a/lib/ansible/plugins/inventory/script.py
+++ b/lib/ansible/plugins/inventory/script.py
@@ -84,9 +84,7 @@ def parse(self, inventory, loader, path, cache=True):
 
         try:
             cache_key = self.get_cache_prefix(path)
-            if cache and cache_key in inventory.cache:
-                data = inventory.cache[cache_key]
-            else:
+            if not cache or cache_key not in inventory.cache:
                 try:
                     sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                 except OSError as e:
@@ -107,14 +105,12 @@ def parse(self, inventory, loader, path, cache=True):
                 except Exception as e:
                     raise AnsibleError("Inventory {0} contained characters that cannot be interpreted as UTF-8: {1}".format(path, to_native(e)))
 
-                if cache:
-                    inventory.cache[cache_key] = data
-
-            try:
-                processed = self.loader.load(data)
-            except Exception as e:
-                raise AnsibleError("failed to parse executable inventory script results from {0}: {1}\n{2}".format(path, to_native(e), err))
+                try:
+                    inventory.cache[cache_key] = self.loader.load(data, file_name=path)
+                except Exception as e:
+                    raise AnsibleError("failed to parse executable inventory script results from {0}: {1}\n{2}".format(path, to_native(e), err))
 
+            processed = inventory.cache[cache_key]
             if not isinstance(processed, Mapping):
                 raise AnsibleError("failed to parse executable inventory script results from {0}: needs to be a json dict\n{1}".format(path, err))
 
@@ -189,6 +185,6 @@ def get_host_variables(self, path, host):
         if out.strip() == '':
             return {}
         try:
-            return json_dict_bytes_to_unicode(self.loader.load(out))
+            return json_dict_bytes_to_unicode(self.loader.load(out, file_name=path))
         except ValueError:
             raise AnsibleError("could not parse post variable response: %s, %s" % (cmd, out))


Test patch: