Repository: ansible/ansible

Issue:
Text backups (xxx.yml~) in host_vars override variables
##### Issue Type:

Bug Report
##### Ansible Version:

ansible 1.9.1 from ppa:ansible/ansible
##### Ansible Configuration:

default
##### Environment:

Ubuntu 15.04
##### Summary:

Text backup files ending on tilde as created e.g. by joe or gedit, in `host_vars/<hostname>/` override the actual variable data.
##### Steps To Reproduce:
- Write samle data into `host_vars/<hostname>/test.yml`
- Edit `host_vars/<hostname>/test.yml` using gedit, gedit will create `host_vars/<hostname>/test.yml~`
- Ensure `test.yml` and `test.yml~` differ

Now ansible will override a variable defined in `test.yml` using the data from `test.yml~`.
##### Expected Results:

Only `*.yml` files are recognized by ansible in `host_vars`.
##### Actual Results:

`*.yml~` files are recognized by ansible in `host_vars`.



Discussion:
~~host/group_vars import have no extension limitations or requirements, they just have to match host/group name.~~
so I'm totally wrong, it should only load files with the following extensions:
 "", ".yml", ".yaml", ".json", so from the code I cannot see how its loading yml~

Could you add a link to the code you are referring to? Thanks

https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/__init__.py#L1520

Let's do the math together.

Let

```
YAML_FILENAME_EXTENSIONS = [ "", ".yml", ".yaml", ".json" ]
basepath = "/home/me/serverconfig/host_vars/example.com"
paths_to_check = [ "".join([basepath, ext]) for ext in YAML_FILENAME_EXTENSIONS ]
```

then

```
paths_to_check =
['/home/me/serverconfig/host_vars/example.com',
'/home/me/serverconfig/host_vars/example.com.yml',
'/home/me/serverconfig/host_vars/example.com.yaml',
'/home/me/serverconfig/host_vars/example.com.json']
```

Now
`_load_vars_from_path` is called on
`/home/me/serverconfig/host_vars/example.com`
which is a directory.

Then `_load_vars_from_folder` is called on
`/home/me/serverconfig/host_vars/example.com`
which will return all files inside, that are not "hidden". But "hidden is defines as starting with "." (https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/__init__.py#L1620)

ah, yes, you did not mention it was a folder, if using a folder for group
vars, everything not hidden gets slurped.

## 

Brian Coca

Is that desired? Does a user need/want to parse `/home/me/serverconfig/host_vars/example.com/holiday.JPEG` as YAML?

Assuming it is desired that all files in a host's host_vars are parsed as variables, one could argue about what a hidden file is on Linux. Nautilus on Ubuntu for example does not display text backup files ending on ~ while `ls` does.

When loading vars from a dir, only hidden file or subdirs are skipped (https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/__init__.py#L1619-L1620)

When checking the base paths to load files from, https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/__init__.py#L1521, `C.YAML_FILENAME_EXTENSIONS` is respected, however when using a directory in group/host_vars, this happens only for the directory name.

So that check should also happen when scanning a dirctory at https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/__init__.py#L1619-L1620

@bcoca you marked this issues a a bug_report, so I assume such a fix is acceptable?

@svg, considering it, want to sync with rest of core before deciding

@bcoca, @jimi-c, and I discussed this and we all think that it would be a bugfix to make loading from a directory obey the same rules as when checking the base paths.


Resolving patch:
diff --git a/lib/ansible/utils/__init__.py b/lib/ansible/utils/__init__.py
--- a/lib/ansible/utils/__init__.py
+++ b/lib/ansible/utils/__init__.py
@@ -1617,7 +1617,9 @@ def _load_vars_from_folder(folder_path, results, vault_password=None):
     names.sort()
 
     # do not parse hidden files or dirs, e.g. .svn/
-    paths = [os.path.join(folder_path, name) for name in names if not name.startswith('.')]
+    paths = [os.path.join(folder_path, name) for name in names
+            if not name.startswith('.')
+            and os.path.splitext(name)[1] in C.YAML_FILENAME_EXTENSIONS]
     for path in paths:
         _found, results = _load_vars_from_path(path, results, vault_password=vault_password)
     return results


Test patch:
diff --git a/test/units/TestInventory.py b/test/units/TestInventory.py
--- a/test/units/TestInventory.py
+++ b/test/units/TestInventory.py
@@ -56,7 +56,7 @@ def dir_inventory(self):
             'thrudgelmir0', 'thrudgelmir1', 'thrudgelmir2',
             'thrudgelmir3', 'thrudgelmir4', 'thrudgelmir5',
             'Hotep-a', 'Hotep-b', 'Hotep-c',
-            'BastC', 'BastD', 'neptun', ]
+            'BastC', 'BastD', 'neptun', 'goldorak', ]
 
     #####################################
     ### Empty inventory format tests
@@ -222,6 +222,11 @@ def test_subset_filename(self):
         inventory.subset('@' + os.path.join(self.test_dir, 'restrict_pattern'))
         self.assertEqual(sorted(inventory.list_hosts()),  sorted(['thor','odin']))
 
+    def test_vars_yaml_extension(self):
+        inventory = self.simple_inventory()
+        vars = inventory.get_variables('goldorak')
+        assert vars['YAML_FILENAME_EXTENSIONS_TEST']
+
     @raises(errors.AnsibleError)
     def testinvalid_entry(self):
        Inventory('1234')
diff --git a/test/units/inventory_test_data/group_vars/noparse/all.yml~ b/test/units/inventory_test_data/group_vars/noparse/all.yml~
new file mode 100644
--- /dev/null
+++ b/test/units/inventory_test_data/group_vars/noparse/all.yml~
@@ -0,0 +1,2 @@
+---
+YAML_FILENAME_EXTENSIONS_TEST: False
diff --git a/test/units/inventory_test_data/group_vars/noparse/file.txt b/test/units/inventory_test_data/group_vars/noparse/file.txt
new file mode 100644
--- /dev/null
+++ b/test/units/inventory_test_data/group_vars/noparse/file.txt
@@ -0,0 +1,2 @@
+---
+YAML_FILENAME_EXTENSIONS_TEST: False
diff --git a/test/units/inventory_test_data/group_vars/parse/all.yml b/test/units/inventory_test_data/group_vars/parse/all.yml
new file mode 100644
--- /dev/null
+++ b/test/units/inventory_test_data/group_vars/parse/all.yml
@@ -0,0 +1,2 @@
+---
+YAML_FILENAME_EXTENSIONS_TEST: True
diff --git a/test/units/inventory_test_data/simple_hosts b/test/units/inventory_test_data/simple_hosts
--- a/test/units/inventory_test_data/simple_hosts
+++ b/test/units/inventory_test_data/simple_hosts
@@ -20,3 +20,9 @@ Bast[C:D]
 
 [auth]
 neptun auth="YWRtaW46YWRtaW4="
+
+[parse:children]
+noparse
+
+[noparse]
+goldorak