Repository: ansible/ansible

Issue:
script inventory plugin: group_vars and host_vars files ignored when relative to inventory (not playbook), and inventory script is a symlink
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
`script` inventory plugin

##### ANSIBLE VERSION
```
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/florian/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
```

##### CONFIGURATION
`ansible-config dump --only-changed` comes back empty.

##### OS / ENVIRONMENT
Ubuntu 17.04

##### SUMMARY
If I run `ansible-playbook` with a static inventory (`-i /path/to/hosts`), and I use `group_vars` and `host_vars` directories relative to that inventory file (as opposed to relative to the playbook file, as documented [here](http://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable)), then those variables get parsed correctly.

In Ansible 2.3, if I did the same with a *dynamic* inventory script (`-i /path/to/some-script.py`), the behavior would be identical.

In Ansible 2.4, that is no longer the case, and static and dynamic inventory behaves quite differently:

- with `-i /path/to/hosts`, variables in `/path/to/group_vars` and `/path/to/host_vars` are correctly added to the inventory.
- with `-i /path/to/some-script.py`, variables in `/path/to/group_vars` and `/path/to/host_vars` are ignored, and I need to symlink those directories into the playbook directory to be picked up.

* * *
I'm not sure what the correct behavior is, for Ansible 2.4, considering the move from inventory scripts to inventory plugins. But it would seem like static and dynamic inventory ought to behave identically with regard to variable parsing, as it was the case pre-2.4.


Discussion:
In my tests both static and dynamic inventories have host_vars and group_vars picked up relative to the inventory file itself.

with test1 in `~/inventories/group_vars/all`
static:
```
#>ansible -m debug -a 'var=test1' all -i ~/inventories/inv.yml
keras.server.com | SUCCESS => {
    "test1": "from inv group vars"
}
```

dynamic:
```
#> ansible -m debug -a 'var=test1' all -i ~/inventories/test_inv.py
MOCK_1032 | SUCCESS => {
    "test1": "from inv group vars"
}
```
Thanks for looking into this @bcoca. 

Hmm, weird. I use my own `lxc_inventory.py` script, and it's behaving differently since I upgraded to 2.4.

All my `host_vars` files live in `../ansible-laptop-config/`, relative to my playbook directory. Every `host_vars/*` file contains a dictionary, `packages`. 

I have one physical box, called `aspireone`, that I connect to via SSH, and that is defined in a static inventory file, `../ansible-laptop-config/hosts`. Its `host_vars` file, `../ansible-laptop-config/host_vars/aspireone`, contains:

```yaml
packages:
  - name: aptitude
  - name: chromium-browser
...
```

When I run the `debug` module against this host, I get the expected output for the `packages` variable:

```
$ ansible -i ../ansible-laptop-config/hosts -m debug -a "var=hostvars[inventory_hostname]['packages']" aspireone
aspireone | SUCCESS => {
    "hostvars[inventory_hostname]['packages']": [
        {
            "name": "aptitude"
        }, 
        {
            "name": "chromium-browser"
        }, 
...
```

I also have a locally-running LXC container, `zesty-firefox-beta`, that is looked up via the `lxc_inventory.py` script. Its `host_vars` file, `../ansible-laptop-config/host_vars/zesty-firefox-beta`, contains:

```yaml
packages:
  - name: aptitude
  - name: dbus-x11
  - name: firefox
```

But when I run `debug` against *this* host, the `packages` variable is never picked up:

```
 $ ansible -i ../ansible-laptop-config/lxc_inventory.py -m debug -a "var=hostvars[inventory_hostname]['packages']" zesty-firefox-beta
zesty-firefox-beta | SUCCESS => {
    "hostvars[inventory_hostname]['packages']": "VARIABLE IS NOT DEFINED!"
}
```

Yet when I symlink `group_vars` and `host_vars` into my playbook directory, I get:

```
$ ansible -i ../ansible-laptop-config/lxc_inventory.py -m debug -a "var=hostvars[inventory_hostname]['packages']" zesty-firefox-beta
zesty-firefox-beta | SUCCESS => {
    "hostvars[inventory_hostname]['packages']": [
        {
            "name": "aptitude"
        }, 
        {
            "name": "dbus-x11"
        }, 
        {
            "name": "firefox"
        }, 
...
```
Like I said, this behavior changed between 2.3 and 2.4; in 2.3, the variable lookup would work fine for the LXC containers.

I'm not sure what to do about this. I'm happy to fix my contributed script, I just don't exactly know how (or, for that matter, what if anything it's doing wrong). Can you suggest something?
The behaviour would be a bug, but as I posted above, i cannot reproduce it, both my static and dynamic inventories are in the same directory and i can pick up both group_vars and host_vars w/o issues.

Can you run both with ANSIBLE_DEBUG=1 and post the output?
Yes, I believe you're looking for the `getting variables` output from the `debug` module? Here it is:

Static inventory (picks `host_vars` up correctly):
```
  8647 1506972162.47818: Loading ActionModule 'debug' from /usr/lib/python2.7/dist-packages/ansible/plugins/action/debug.py (found_in_cache=False, class_only=True)
  8647 1506972162.47822: getting variables
  8647 1506972162.47824: in VariableManager get_vars()
  8647 1506972162.47847: Loading FilterModule 'core' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
  8647 1506972162.47850: Loading FilterModule 'ipaddr' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
  8647 1506972162.47853: Loading FilterModule 'json_query' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
  8647 1506972162.47856: Loading FilterModule 'mathstuff' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
  8647 1506972162.47858: Loading FilterModule 'network' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/network.py (found_in_cache=True, class_only=False)
  8647 1506972162.47861: Loading FilterModule 'urlsplit' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/urlsplit.py (found_in_cache=True, class_only=False)
  8647 1506972162.47874: Loading TestModule 'core' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
  8647 1506972162.47878: Loading TestModule 'files' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
  8647 1506972162.47881: Loading TestModule 'mathstuff' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
  8647 1506972162.47938: Calling all_inventory to load vars for aspireone
  8647 1506972162.47943: Calling groups_inventory to load vars for aspireone
  8647 1506972162.47947: Calling all_plugins_inventory to load vars for aspireone
  8647 1506972162.47995: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py
  8647 1506972162.48026: 	processing dir /home/florian/git/ansible-laptop-config/group_vars
  8647 1506972162.48049: Loading data from /home/florian/git/ansible-laptop-config/group_vars/all
  8647 1506972162.48110: Calling all_plugins_play to load vars for aspireone
  8647 1506972162.48126: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8647 1506972162.48142: Calling groups_plugins_inventory to load vars for aspireone
  8647 1506972162.48156: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8647 1506972162.48168: 	processing dir /home/florian/git/ansible-laptop-config/group_vars
  8647 1506972162.48179: 	processing dir /home/florian/git/ansible-laptop-config/group_vars
  8647 1506972162.48186: Calling groups_plugins_play to load vars for aspireone
  8647 1506972162.48200: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8647 1506972162.48226: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8647 1506972162.48235: 	processing dir /home/florian/git/ansible-laptop-config/host_vars
  8647 1506972162.48278: Loading data from /home/florian/git/ansible-laptop-config/host_vars/aspireone
  8647 1506972162.48580: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8647 1506972162.48602: done with get_vars()
```

LXC inventory script (misses the `host_vars` file):

```
  8559 1506972113.80857: Loading ActionModule 'debug' from /usr/lib/python2.7/dist-packages/ansible/plugins/action/debug.py (found_in_cache=False, class_only=True)
  8559 1506972113.80859: getting variables
  8559 1506972113.80861: in VariableManager get_vars()
  8559 1506972113.80890: Loading FilterModule 'core' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
  8559 1506972113.80894: Loading FilterModule 'ipaddr' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
  8559 1506972113.80896: Loading FilterModule 'json_query' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
  8559 1506972113.80898: Loading FilterModule 'mathstuff' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
  8559 1506972113.80901: Loading FilterModule 'network' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/network.py (found_in_cache=True, class_only=False)
  8559 1506972113.80903: Loading FilterModule 'urlsplit' from /usr/lib/python2.7/dist-packages/ansible/plugins/filter/urlsplit.py (found_in_cache=True, class_only=False)
  8559 1506972113.80915: Loading TestModule 'core' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
  8559 1506972113.80918: Loading TestModule 'files' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
  8559 1506972113.80920: Loading TestModule 'mathstuff' from /usr/lib/python2.7/dist-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
  8559 1506972113.80973: Calling all_inventory to load vars for zesty-firefox-beta
  8559 1506972113.80977: Calling groups_inventory to load vars for zesty-firefox-beta
  8559 1506972113.80980: Calling all_plugins_inventory to load vars for zesty-firefox-beta
  8559 1506972113.81024: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py
  8559 1506972113.81037: Calling all_plugins_play to load vars for zesty-firefox-beta
  8559 1506972113.81047: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8559 1506972113.81057: Calling groups_plugins_inventory to load vars for zesty-firefox-beta
  8559 1506972113.81066: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8559 1506972113.81077: Calling groups_plugins_play to load vars for zesty-firefox-beta
  8559 1506972113.81086: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8559 1506972113.81104: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8559 1506972113.81120: Loading VarsModule 'host_group_vars' from /usr/lib/python2.7/dist-packages/ansible/plugins/vars/host_group_vars.py (found_in_cache=True, class_only=False)
  8559 1506972113.81137: done with get_vars()
```

So it looks like `host_group_vars.py` is doing the right thing with the static inventory, but is tripping up over the LXC inventory script.
@fghaas Are you by chance using a symlink to link the dynamic script?  If so, we have confirmation on it since that's the issue I isolated in my environment after updating.  In our case, we have:


- i inventory/\<environment\>/ec2.py  and only one of the ec2.py scripts is real, the rest symlink as to reduce the amount of duplicate files/code.  ie:  -i inventory/tst/ec2.py exists, -i inventory/stg/ec2.py is a symlink, but was never evaluated before, it was instead treated as a relative file (because of group_vars and host_vars).  -- Would like to confirm this instead of creating a possible duplicate bug.  If it is indeed different I will open another ticket, otherwise we may be able to change this one a bit so that it reflects the issue with symlink evaluation in inventory terms.
@Daemoen Bullseye! That is indeed a symlink (it points to the `lxc_inventory.py` script in my `ansible` git checkout). If I replace the symlink with a copied file, then things work all right. Thanks a lot, I never would have thought of that.

Strikes me as bizarre, though. Why would it do that? Do you have another bug for this that you can point me to?

Resolving patch:
diff --git a/lib/ansible/cli/__init__.py b/lib/ansible/cli/__init__.py
--- a/lib/ansible/cli/__init__.py
+++ b/lib/ansible/cli/__init__.py
@@ -615,7 +615,7 @@ def parse(self):
                     self.options.inventory = [self.options.inventory]
 
                 # Ensure full paths when needed
-                self.options.inventory = [unfrackpath(opt) if ',' not in opt else opt for opt in self.options.inventory]
+                self.options.inventory = [unfrackpath(opt, follow=False) if ',' not in opt else opt for opt in self.options.inventory]
 
             else:
                 self.options.inventory = C.DEFAULT_HOST_LIST


Test patch: