Repository: ansible/ansible

Issue:
Multiple files in role's defaults directory
##### ISSUE TYPE
Feature Idea

##### COMPONENT NAME
roles

##### ANSIBLE VERSION
N/A

##### SUMMARY
I have a role that manages a complex piece of software with many options. Right now, the defaults/main.yml is growing larger and larger with more variables.

It would be nice if in the defaults folder I could have various YAML files for each group of variables that are semantically related.

I've tried adding more files to this folder but it doesn't seem their definitions are picked up (as expected). Only main.yml is loaded. 

I believe this would be a nice feature.



Discussion:
you can have a `main.yml` in `tasks`, that includes other files, even in subdirectories of `tasks`, eg to configure your main two components foo and bar of that large role:

```
---
- include: foo/main.yml
- include: bar/main.yml
```

Using an `include_vars` module for default role variables defeats the reason default variables exist in the first place, namely being used to provide "last resort defaults" which can be easily overwritten by other parts of the role/playbook.

The idea of multiple files in `defaults/` directory was previously being discussed on IRC, we concluded that parsing the `defaults/` directory similarly to `inventory/group_vars/` and `inventory/host_vars/` directories would be nice to have. The needed code already exists, and hopefully can be easily adapted for role defaults.

+1

+1

Would be nice to have it!

+1

Yeah just assumed that defaults/\* were found and loaded automagically as mentioned above.  It seems the suggested workaround (including additional defaults files in tasks) will override inventory vars due to [variable precedence](http://docs.ansible.com/ansible/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable), so it seems we may have to bundle in 1 file to avoid this issue until addressed.

Ansible 2.x variable precedence

```
    role defaults [1] (loading all role defaults here critical)
    inventory vars [2]
    inventory group_vars
    inventory host_vars
    playbook group_vars
    playbook host_vars
    host facts
    registered vars
    set_facts
    play vars
    play vars_prompt
    play vars_files
    role and include vars (hack coming in here will override everything above)
    block vars (only for tasks in block)
    task vars (only for the task)
    extra vars (always win precedence)
```

This may be related to #8121

+1

+1

Definitely a must have feature due to variable precedence constraints.
This also allows to define a defaults file per class of variables and keep a clean separation; for instance, in networking, this would allow me to define one defaults file for IPv4, one for IPv6, one for SNMP and so on...
At last, It seems necessary to be able to put these files into different folders under "defaults".

Just hit this one myself, so, definitely +1

This really is a PITA.
If I could suggest a solution, it would be to add an option, similar to `include_vars`, but in `meta/main.yml`, to specify load-order.

```
- dependencies: []
- default_vars:
  - "main.yml"
  - "{{ ansible_os_family }}-{{ ansible_distribution_major_version }}.yml"
```

This would allow the option to default to `["main.yml"]` and not break existing code, while giving developers the ability to define granular variables based on contexts of their choosing.

Weird. I totally suspected this would work out of the box. One of the few times ansible actually violates the principle of the least surprise a bit...

Definitely +1

For `vars/` the same, but seems like #2958 was deciding against this

@cornfeedhobo's example exactly matches my use case.  By using include_vars, I'm preventing easy overrides of any individual variables by plays.  

Another interface option would be: 
- Add an option to the include_vars module, 'as_default', which processes the included file as if it were in defaults.

For me it really looks weird/counter-intuitive that there's a directory "defaults" with a single file in it.
I mean, have you ever seen a python module with a single `__main__.py` file in it?

Which is precisely why I intuitively thought the functionality talked about on this ticket was already present. "Of course I can have multiple defaults files, there is a directory for it" 

`include_role` allows you to specify 'alternate' files now http://docs.ansible.com/ansible/include_role_module.html via `defaults_from` option
Just got into a situation when `defaults/main.yml` split is desired. Added my 👍 to the issue.
Same here, I need it as well 
+1
+1
@bcoca that is a nice addition and adds flexibility, but does not address the issue here, and puts the onus on the playbook writer, when the intention is to empower the role to set sane defaults easily.

Please review [my comment above](https://github.com/ansible/ansible/issues/14248#issuecomment-254059046)
Yes this is very helpful feature to avoid files with 2k lines. 
+1
+1 
+1
👍 
+1
+1
+1
Same issue here.  main.yml too large, split into multiple files.  We assumed it would "just work" and load all the vars in the files under defaults/ since it is a dir.  nojoy.
+1
+1
+1
👍 
+1
+1
👍 
You guys can thumbs down every +1 comment, but I highly doubt that you can track the number of subscribers, so adding +1 is still a decent way to show desired support. 

Until issue attention is clearly tied to the number of subscribers to an issue, forever expect to see +1 comments. 
What you can expect is that the ansible team locks the issue comments instead, so don't be a spammer and dont defend them please.
> What you can expect is that the ansible team locks the issue comments instead, so don't be a spammer and dont defend them please.

They could do that but this is a core issue to being able to properly develop roles especially since the introduction of `include_role` in 2.x alongside the old `roles` include method. If you design your role to be compatible with `include_role`, which is preferable since it makes development a hell of lot more modular and flexible than with the old `roles` method alone, then you are left with having to track either including defaults yourself as tasks in your playbooks, pass the buck on to the end user who shouldnt have to know how to do this for your role to work properly, or track the platform/distro/release as a nested level of your role variables which complicates the hell out of development (although it works as Ive created a role that can support 20 different platform versions [here](https://github.com/ju2wheels/ansible-galaxy-ju2wheels.pyenv)). Have you actually tried writing a role that actually supports more than a few platforms? Its no surprise why most roles on Ansible Galaxy dont, this issue is probably number one on that list IMO.

So if they went to lock this, not respond at all to a year old issue, and signal to the community that they are not interested in discussing critical issues that are preventing people from creating real solutions, then they can definitely expect people to not stick around and continue using the product. If they reply and give a sense of whether they agree its an issue, then maybe people will stop +1'ing it in order to draw attention to the problem.

This is only an issue because they released `include_role` without thinking through how actually using it in practice would affect defaults. Forcing someone to choose between using `roles` and `include_role` in order to make a role work properly or add extra work to enable both to work simultaneous is the core of the issue here.

My 2 cents.

PS If you are downvoting +1's because you dont want the notifications.... hit that unsubscribe button on the top right and you wont get them anymore.

OK, this issue is important for you. I get the point. For me too. That's why I'm subscribed, BTW, and that's why I don't want to unsubscribe, and that's why I downvote spammers, and that's why I upvote [the original comment](https://github.com/ansible/ansible/issues/14248#issue-130464648), which is **the civilized way** to show _lazy_ support for it.
I'd like to add my voice to the request for this ability and detail my use-case.  In addition to the simple ability to break-up large main.yml, the ability to do something like this would be great:
```
- name: Some name
  include_default_vars:
  with_first_found:
    - "{{ ansible_os_family }}-{{ ansible_distribution_major_version }}.yml"
   - main.yml
```

What I'm hoping to accomplish with this is [as an example] an openssh role that supports *all* ssh_config and sshd_config options, and supports multiple OS's and versions [ie. Debian 8/9, EL6/7, etc] but which if called with no vars set by the user, will safely build the config to the OS_majorversion default settings, but which can have ANY available option overridden by the user. 

As it stands now, if I put those OS defaults in include_vars, the precedence is too high, and the user can't override those settings in the inventory, or in group_vars/all or in group_vars/groupname or in host_vars etc. 

I'm sure there's a *way* to do this now, but anything I can think of is wildly ungraceful and ugly and would be difficult to maintain. In addition, at least up to this point, nobody I've encountered  #ansible has any better ideas on how to do this in a graceful, maintainable way. 

Adding this feature would allow for this and may help result in higher quality roles being made available in github/galaxy.
@ralphie02 no, thats not anywhere near a solution to what this thread is about. That is just normal host based variables which is different from the requirement of setting defaults for a multi platform role.
Similar vars/ feature_idea: https://github.com/ansible/ansible/issues/11639

Resolving patch:
diff --git a/lib/ansible/parsing/dataloader.py b/lib/ansible/parsing/dataloader.py
--- a/lib/ansible/parsing/dataloader.py
+++ b/lib/ansible/parsing/dataloader.py
@@ -12,6 +12,7 @@
 import re
 import tempfile
 
+from ansible import constants as C
 from ansible.errors import AnsibleFileNotFound, AnsibleParserError
 from ansible.module_utils.basic import is_executable
 from ansible.module_utils.six import binary_type, text_type
@@ -393,3 +394,53 @@ def cleanup_all_tmp_files(self):
                 self.cleanup_tmp_file(f)
             except Exception as e:
                 display.warning("Unable to cleanup temp files: %s" % to_native(e))
+
+    def find_vars_files(self, path, name, extensions=None, allow_dir=True):
+        """
+        Find vars files in a given path with specified name. This will find
+        files in a dir named <name>/ or a file called <name> ending in known
+        extensions.
+        """
+
+        b_path = to_bytes(os.path.join(path, name))
+        found = []
+
+        if extensions is None:
+            # Look for file with no extension first to find dir before file
+            extensions = [''] + C.YAML_FILENAME_EXTENSIONS
+        # add valid extensions to name
+        for ext in extensions:
+
+            if '.' in ext:
+                full_path = b_path + to_bytes(ext)
+            elif ext:
+                full_path = b'.'.join([b_path, to_bytes(ext)])
+            else:
+                full_path = b_path
+
+            if self.path_exists(full_path):
+                if self.is_directory(full_path):
+                    if allow_dir:
+                        found.extend(self._get_dir_vars_files(to_text(full_path), extensions))
+                    else:
+                        next
+                else:
+                    found.append(full_path)
+                break
+        return found
+
+    def _get_dir_vars_files(self, path, extensions):
+        found = []
+        for spath in sorted(self.list_directory(path)):
+            if not spath.startswith(u'.') and not spath.endswith(u'~'):  # skip hidden and backups
+
+                ext = os.path.splitext(spath)[-1]
+                full_spath = os.path.join(path, spath)
+
+                if self.is_directory(full_spath) and not ext:  # recursive search if dir
+                    found.extend(self._get_dir_vars_files(full_spath, extensions))
+                elif self.is_file(full_spath) and (not ext or to_text(ext) in extensions):
+                    # only consider files with valid extensions or no extension
+                    found.append(full_spath)
+
+        return found
diff --git a/lib/ansible/playbook/role/__init__.py b/lib/ansible/playbook/role/__init__.py
--- a/lib/ansible/playbook/role/__init__.py
+++ b/lib/ansible/playbook/role/__init__.py
@@ -223,58 +223,46 @@ def _load_role_data(self, role_include, parent_role=None):
                                          obj=handler_data, orig_exc=e)
 
         # vars and default vars are regular dictionaries
-        self._role_vars = self._load_role_yaml('vars', main=self._from_files.get('vars'))
+        self._role_vars = self._load_role_yaml('vars', main=self._from_files.get('vars'), allow_dir=True)
         if self._role_vars is None:
             self._role_vars = dict()
         elif not isinstance(self._role_vars, dict):
             raise AnsibleParserError("The vars/main.yml file for role '%s' must contain a dictionary of variables" % self._role_name)
 
-        self._default_vars = self._load_role_yaml('defaults', main=self._from_files.get('defaults'))
+        self._default_vars = self._load_role_yaml('defaults', main=self._from_files.get('defaults'), allow_dir=True)
         if self._default_vars is None:
             self._default_vars = dict()
         elif not isinstance(self._default_vars, dict):
             raise AnsibleParserError("The defaults/main.yml file for role '%s' must contain a dictionary of variables" % self._role_name)
 
-    def _load_role_yaml(self, subdir, main=None):
+    def _load_role_yaml(self, subdir, main=None, allow_dir=False):
         file_path = os.path.join(self._role_path, subdir)
         if self._loader.path_exists(file_path) and self._loader.is_directory(file_path):
-            main_file = self._resolve_main(file_path, main)
-            if self._loader.path_exists(main_file):
-                return self._loader.load_from_file(main_file)
+            # Valid extensions and ordering for roles is hard-coded to maintain
+            # role portability
+            extensions = ['.yml', '.yaml', '.json']
+            # If no <main> is specified by the user, look for files with
+            # extensions before bare name. Otherwise, look for bare name first.
+            if main is None:
+                _main = 'main'
+                extensions.append('')
+            else:
+                _main = main
+                extensions.insert(0, '')
+            found_files = self._loader.find_vars_files(file_path, _main, extensions, allow_dir)
+            if found_files:
+                data = {}
+                for found in found_files:
+                    new_data = self._loader.load_from_file(found)
+                    if new_data and allow_dir:
+                        data = combine_vars(data, new_data)
+                    else:
+                        data = new_data
+                    return data
             elif main is not None:
                 raise AnsibleParserError("Could not find specified file in role: %s/%s" % (subdir, main))
         return None
 
-    def _resolve_main(self, basepath, main=None):
-        ''' flexibly handle variations in main filenames '''
-
-        post = False
-        # allow override if set, otherwise use default
-        if main is None:
-            main = 'main'
-            post = True
-
-        bare_main = os.path.join(basepath, main)
-
-        possible_mains = (
-            os.path.join(basepath, '%s.yml' % main),
-            os.path.join(basepath, '%s.yaml' % main),
-            os.path.join(basepath, '%s.json' % main),
-        )
-
-        if post:
-            possible_mains = possible_mains + (bare_main,)
-        else:
-            possible_mains = (bare_main,) + possible_mains
-
-        if sum([self._loader.is_file(x) for x in possible_mains]) > 1:
-            raise AnsibleError("found multiple main files at %s, only one allowed" % (basepath))
-        else:
-            for m in possible_mains:
-                if self._loader.is_file(m):
-                    return m  # exactly one main file
-            return possible_mains[0]  # zero mains (we still need to return something)
-
     def _load_dependencies(self):
         '''
         Recursively loads role dependencies from the metadata list of
diff --git a/lib/ansible/plugins/vars/host_group_vars.py b/lib/ansible/plugins/vars/host_group_vars.py
--- a/lib/ansible/plugins/vars/host_group_vars.py
+++ b/lib/ansible/plugins/vars/host_group_vars.py
@@ -89,7 +89,7 @@ def get_vars(self, loader, path, entities, cache=True):
                         if os.path.exists(b_opath):
                             if os.path.isdir(b_opath):
                                 self._display.debug("\tprocessing dir %s" % opath)
-                                found_files = self._find_vars_files(opath, entity.name)
+                                found_files = loader.find_vars_files(opath, entity.name)
                                 FOUND[key] = found_files
                             else:
                                 self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))
@@ -102,48 +102,3 @@ def get_vars(self, loader, path, entities, cache=True):
                 except Exception as e:
                     raise AnsibleParserError(to_native(e))
         return data
-
-    def _find_vars_files(self, path, name):
-        """ Find {group,host}_vars files """
-
-        b_path = to_bytes(os.path.join(path, name))
-        found = []
-
-        # first look for w/o extensions
-        if os.path.exists(b_path):
-            if os.path.isdir(b_path):
-                found.extend(self._get_dir_files(to_text(b_path)))
-            else:
-                found.append(b_path)
-        else:
-            # add valid extensions to name
-            for ext in C.YAML_FILENAME_EXTENSIONS:
-
-                if '.' in ext:
-                    full_path = b_path + to_bytes(ext)
-                elif ext:
-                    full_path = b'.'.join([b_path, to_bytes(ext)])
-                else:
-                    full_path = b_path
-
-                if os.path.exists(full_path) and os.path.isfile(full_path):
-                    found.append(full_path)
-                    break
-        return found
-
-    def _get_dir_files(self, path):
-
-        found = []
-        for spath in sorted(os.listdir(path)):
-            if not spath.startswith(u'.') and not spath.endswith(u'~'):  # skip hidden and backups
-
-                ext = os.path.splitext(spath)[-1]
-                full_spath = os.path.join(path, spath)
-
-                if os.path.isdir(full_spath) and not ext:  # recursive search if dir
-                    found.extend(self._get_dir_files(full_spath))
-                elif os.path.isfile(full_spath) and (not ext or to_text(ext) in C.YAML_FILENAME_EXTENSIONS):
-                    # only consider files with valid extensions or no extension
-                    found.append(full_spath)
-
-        return found


Test patch:
diff --git a/test/units/mock/loader.py b/test/units/mock/loader.py
--- a/test/units/mock/loader.py
+++ b/test/units/mock/loader.py
@@ -23,7 +23,7 @@
 
 from ansible.errors import AnsibleParserError
 from ansible.parsing.dataloader import DataLoader
-from ansible.module_utils._text import to_bytes
+from ansible.module_utils._text import to_bytes, to_text
 
 
 class DictDataLoader(DataLoader):
@@ -39,6 +39,7 @@ def __init__(self, file_mapping=None):
         self._vault_secrets = None
 
     def load_from_file(self, path, unsafe=False):
+        path = to_text(path)
         if path in self._file_mapping:
             return self.load(self._file_mapping[path], path)
         return None
@@ -46,22 +47,32 @@ def load_from_file(self, path, unsafe=False):
     # TODO: the real _get_file_contents returns a bytestring, so we actually convert the
     #       unicode/text it's created with to utf-8
     def _get_file_contents(self, path):
+        path = to_text(path)
         if path in self._file_mapping:
             return (to_bytes(self._file_mapping[path]), False)
         else:
             raise AnsibleParserError("file not found: %s" % path)
 
     def path_exists(self, path):
+        path = to_text(path)
         return path in self._file_mapping or path in self._known_directories
 
     def is_file(self, path):
+        path = to_text(path)
         return path in self._file_mapping
 
     def is_directory(self, path):
+        path = to_text(path)
         return path in self._known_directories
 
     def list_directory(self, path):
-        return [x for x in self._known_directories]
+        ret = []
+        path = to_text(path)
+        for x in (list(self._file_mapping.keys()) + self._known_directories):
+            if x.startswith(path):
+                if os.path.dirname(x) == path:
+                    ret.append(os.path.basename(x))
+        return ret
 
     def is_executable(self, path):
         # FIXME: figure out a way to make paths return true for this
diff --git a/test/units/playbook/role/test_role.py b/test/units/playbook/role/test_role.py
--- a/test/units/playbook/role/test_role.py
+++ b/test/units/playbook/role/test_role.py
@@ -187,6 +187,68 @@ def test_load_role_with_vars(self):
         self.assertEqual(r._default_vars, dict(foo='bar'))
         self.assertEqual(r._role_vars, dict(foo='bam'))
 
+    @patch('ansible.playbook.role.definition.unfrackpath', mock_unfrackpath_noop)
+    def test_load_role_with_vars_dirs(self):
+
+        fake_loader = DictDataLoader({
+            "/etc/ansible/roles/foo_vars/defaults/main/foo.yml": """
+            foo: bar
+            """,
+            "/etc/ansible/roles/foo_vars/vars/main/bar.yml": """
+            foo: bam
+            """,
+        })
+
+        mock_play = MagicMock()
+        mock_play.ROLE_CACHE = {}
+
+        i = RoleInclude.load('foo_vars', play=mock_play, loader=fake_loader)
+        r = Role.load(i, play=mock_play)
+
+        self.assertEqual(r._default_vars, dict(foo='bar'))
+        self.assertEqual(r._role_vars, dict(foo='bam'))
+
+    @patch('ansible.playbook.role.definition.unfrackpath', mock_unfrackpath_noop)
+    def test_load_role_with_vars_nested_dirs(self):
+
+        fake_loader = DictDataLoader({
+            "/etc/ansible/roles/foo_vars/defaults/main/foo/bar.yml": """
+            foo: bar
+            """,
+            "/etc/ansible/roles/foo_vars/vars/main/bar/foo.yml": """
+            foo: bam
+            """,
+        })
+
+        mock_play = MagicMock()
+        mock_play.ROLE_CACHE = {}
+
+        i = RoleInclude.load('foo_vars', play=mock_play, loader=fake_loader)
+        r = Role.load(i, play=mock_play)
+
+        self.assertEqual(r._default_vars, dict(foo='bar'))
+        self.assertEqual(r._role_vars, dict(foo='bam'))
+
+    @patch('ansible.playbook.role.definition.unfrackpath', mock_unfrackpath_noop)
+    def test_load_role_with_vars_dir_vs_file(self):
+
+        fake_loader = DictDataLoader({
+            "/etc/ansible/roles/foo_vars/vars/main/foo.yml": """
+            foo: bar
+            """,
+            "/etc/ansible/roles/foo_vars/vars/main.yml": """
+            foo: bam
+            """,
+        })
+
+        mock_play = MagicMock()
+        mock_play.ROLE_CACHE = {}
+
+        i = RoleInclude.load('foo_vars', play=mock_play, loader=fake_loader)
+        r = Role.load(i, play=mock_play)
+
+        self.assertEqual(r._role_vars, dict(foo='bam'))
+
     @patch('ansible.playbook.role.definition.unfrackpath', mock_unfrackpath_noop)
     def test_load_role_with_metadata(self):