Repository: ansible/ansible

Issue:
backup files in host_vars override vars.yml
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

host_vars configuration
##### ANSIBLE VERSION

```
ansible 2.1.1.0
  config file = /home/rod/deltoid/servers/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

Added vault password file path to ansible.cfg, nothing else.
##### OS / ENVIRONMENT

Arch linux (x86_64)
##### SUMMARY

When reading host_vars, ansible reads vars from a file name "vars.yml~" with priority over a file name "vars.yml". The "vars.yml~" file was created by editing the original file with emacs, which saves the backup by appending a "~" to the filename.
##### STEPS TO REPRODUCE

Create host_vars in a folder as follows:

```
ansible-folder
- host_vars
  - my-server-01
    - vars.yml
        ansible_user: root
        ansible_port: 22
    - vars.yml~
        ansible_user: someone_else
        ansible_port: 9999
```

When connecting to a host (e.g. ansible -i hosts all -m ping), ansible will use the vars saved in vars.yml~.
##### EXPECTED RESULTS

Expected to connect with 'root' on port 22, etc.

```
Using /home/rod/deltoid/servers/ansible/ansible.cfg as config file
SSH password: 
Loaded callback minimal of type stdout, v2.0
<dlt-tx-02> ESTABLISH SSH CONNECTION FOR USER: root
<dlt-tx-02> SSH: EXEC sshpass -d13 ssh -C -vvv -o ControlMaster=no -o ControlPersist=no -o Port=22 -o User=root -o ConnectTimeout=10 -o ControlPath=/dev/null dlt-tx-02 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1472954078.5-25255107732476 `" && echo ansible-tmp-1472954078.5-25255107732476="` echo $HOME/.ansible/tmp/ansible-tmp-1472954078.5-25255107732476 `" ) && sleep 0'"'"''
<dlt-tx-02> PUT /tmp/tmpldCjDh TO /root/.ansible/tmp/ansible-tmp-1472954078.5-25255107732476/ping
<dlt-tx-02> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -C -vvv -o ControlMaster=no -o ControlPersist=no -o Port=22 -o User=root -o ConnectTimeout=10 -o ControlPath=/dev/null '[dlt-tx-02]'
<dlt-tx-02> ESTABLISH SSH CONNECTION FOR USER: root
<dlt-tx-02> SSH: EXEC sshpass -d13 ssh -C -vvv -o ControlMaster=no -o ControlPersist=no -o Port=22 -o User=root -o ConnectTimeout=10 -o ControlPath=/dev/null -tt dlt-tx-02 '/bin/sh -c '"'"'LANG=en_AU.UTF-8 LC_ALL=en_AU.UTF-8 LC_MESSAGES=en_AU.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1472954078.5-25255107732476/ping; rm -rf "/root/.ansible/tmp/ansible-tmp-1472954078.5-25255107732476/" > /dev/null 2>&1 && sleep 0'"'"''
dlt-tx-02 | SUCCESS => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "data": null
        }, 
        "module_name": "ping"
    }, 
    "ping": "pong"
}
```
##### ACTUAL RESULTS

Ansible generated an ssh command for the incorrect user/port.

```
Using /home/rod/deltoid/servers/ansible/ansible.cfg as config file
SSH password: 
Loaded callback minimal of type stdout, v2.0
<dlt-tx-02> ESTABLISH SSH CONNECTION FOR USER: someone_else
<dlt-tx-02> SSH: EXEC sshpass -d13 ssh -C -vvv -o ControlMaster=no -o ControlPersist=no -o Port=9999 -o User=someone_else -o ConnectTimeout=10 -o ControlPath=/dev/null dlt-tx-02 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1472954135.94-221184064468334 `" && echo ansible-tmp-1472954135.94-221184064468334="` echo $HOME/.ansible/tmp/ansible-tmp-1472954135.94-221184064468334 `" ) && sleep 0'"'"''
dlt-tx-02 | UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
```



Discussion:
I've been able to replicate this. I think it's due to the behaviour of the `_load_inventory_file()` function in `lib/ansible/vars/__init__.py`.
The list of files to be loaded is sorted using sort(), which leaves a filename appended with a `~` character to be loaded later:
`[u'localhost.yml', u'localhost.yml~']`

You can avoid those files being loaded with an extra condition on the for loop that follows, but I don't know if that's desirable behaviour.
e.g.

```
# do not parse hidden files or dirs, e.g. .svn/
paths = [os.path.join(path, name) for name in names if not name.startswith('.') and not name.endswith('~')]
```

Same bug here with group_vars.
Thanks @george-buckerfield, that's exactly the behaviour I'd be wanting. I've looked at that part of the code and agree with your take on it.

My take on desirable behaviour: saving with a "~" extension for backup files is the default behaviour for both vim & emacs, so makes sense to exclude these when parsing hostvars and groupvars.

Alternatively might be to reverse the sort order, so that backup files were parsed first and over-written by variables in the actual file. (Personally think that's less transparent than just not parsing it at all though.)

Does anyone want a pull request for this, or is more discussion needed about whether or not this is desireable behaviour?
@rodmanning I had a PR ready to go a while back, so I've submitted it. Happy to modify it based on any feedback though.

Resolving patch:
diff --git a/lib/ansible/vars/__init__.py b/lib/ansible/vars/__init__.py
--- a/lib/ansible/vars/__init__.py
+++ b/lib/ansible/vars/__init__.py
@@ -550,7 +550,7 @@ def _get_inventory_basename(self, path):
         else:
             return name
 
-    def _load_inventory_file(self, path, loader):
+    def _load_inventory_file(self, path, loader, filter_ext=False):
         '''
         helper function, which loads the file and gets the
         basename of the file without the extension
@@ -569,24 +569,26 @@ def _load_inventory_file(self, path, loader):
             names.sort()
 
             # do not parse hidden files or dirs, e.g. .svn/
-            paths = [os.path.join(path, name) for name in names if not name.startswith('.')]
+            paths = [os.path.join(path, name) for name in names if not (name.startswith('.') or name.endswith('~'))]
             for p in paths:
-                results = self._load_inventory_file(path=p, loader=loader)
+                results = self._load_inventory_file(path=p, loader=loader, filter_ext=True)
                 if results is not None:
                     data = combine_vars(data, results)
 
         else:
             file_name, ext = os.path.splitext(path)
             data = None
-            if not ext or ext not in C.YAML_FILENAME_EXTENSIONS:
-                for test_ext in C.YAML_FILENAME_EXTENSIONS:
-                    new_path = path + test_ext
-                    if loader.path_exists(new_path):
-                        data = loader.load_from_file(new_path)
-                        break
-            else:
+            if not filter_ext or ext in C.YAML_FILENAME_EXTENSIONS:
                 if loader.path_exists(path):
                     data = loader.load_from_file(path)
+                else:
+                    # try appending yaml extenstion to find valid files
+                    # avoid empty extensions otherwise all files would be tried
+                    for test_ext in (ext for ext in C.YAML_FILENAME_EXTENSIONS if ext):
+                        new_path = path + test_ext
+                        if loader.path_exists(new_path):
+                            data = loader.load_from_file(new_path)
+                            break
 
         rval = AnsibleInventoryVarsData()
         rval.path = path


Test patch: