Repository: ansible/ansible

Issue:
New ansible-inventory option to return variables at group level
##### ISSUE TYPE
 - Feature Idea

##### COMPONENT NAME
`ansible-inventory` CLI, found at https://github.com/ansible/ansible/blob/devel/lib/ansible/cli/inventory.py

##### ANSIBLE VERSION
2.4 or 2.5

##### CONFIGURATION
Any, functionality of output should be consistent with any valid Ansible configurations.

##### OS / ENVIRONMENT
N/A

##### SUMMARY
`ansible-inventory` will return all variables at the host level. This is consistent with behavior inside of playbooks, but can lead to very large outputs, because variables defined at the group level will be replicated with each host inside of it.

I previously made an attempt at this in https://github.com/bcoca/ansible/pull/4, but that solution needed more work to respect the users existing configuration customizations.

##### STEPS TO REPRODUCE

Copy this folder to your local machine, cd into it.

https://github.com/AlanCoding/Ansible-inventory-file-examples/tree/master/issues/gvars

`ansible-inventory -i inventory.ini --list`

##### EXPECTED RESULTS

Perhaps by adding another flag, like `--optimized`, we could get output without the `airport_code` variable duplicated, like so:

```
{
    "_meta": {
        "hostvars": {
            "host1": {}, 
            "host2": {}
        }
    }, 
    "all": {
        "children": [
            "raleigh", 
            "ungrouped"
        ]
    }, 
    "raleigh": {
        "hosts": [
            "host1", 
            "host2"
        ],
       "vars": {
           "airport_code": "RDU"
       }
    }, 
    "ungrouped": {}
}
```

##### ACTUAL RESULTS

The only currently available way of using this command will put `airport_code` (a variable specific to the group `raleigh`) in hostvars.

```
{
    "_meta": {
        "hostvars": {
            "host1": {
                "airport_code": "RDU"
            }, 
            "host2": {
                "airport_code": "RDU"
            }
        }
    }, 
    "all": {
        "children": [
            "raleigh", 
            "ungrouped"
        ]
    }, 
    "raleigh": {
        "hosts": [
            "host1", 
            "host2"
        ]
    }, 
    "ungrouped": {}
}
```

RESOLVED: ~Note that `ansible_inventory_sources` appears to be some other unrelated thing, and I may raise a separate issue about that.~


Discussion:
@bcoca I've run into this too while going through our Tower tests.

I feel like having Ansible return variables in this way is more intuitive to the user.
@simfarm yet misleading as they would get the impression that ansible itself will be able to access by group
Can you explain @bcoca? Why would we not want to present full group variables under host variables?
we DO present them, they are in the hostvars

we DONT present them in the group that set them as the groups 'go away' once inventory is processed.
It is really not intuitive to me that we'd show group variables partially in host variables.
intuitive or not, the current output represents what Ansible itself sees when the play starts, not the original inventory layout.

If we present the variables by group we are not showing the final flattening/resolution, aka the value you would be using in the playbook. 

I do want to add this option, but it should not be the default and clearly state that this is NOT what is available in the playbooks.
> I do want to add this option, but it should not be the default and clearly state that this is NOT what is available in the playbooks.

Agreed, it should be some flag specifically for tools that want to use this (more memory-efficient) data model. AWX is a prime example.

----
Also consider that current group vars behavior is not consistent. Example:

https://github.com/AlanCoding/Ansible-inventory-file-examples/blob/master/static/group_vars.ini

https://gist.github.com/AlanCoding/a07cc14e96f5267336ebefe1692994be

```json
    "southeast": {
        "children": [
            "atlanta", 
            "raleigh"
        ], 
        "vars": {
            "escape_pods": 2, 
            "halon_system_timeout": 30, 
            "self_destruct_countdown": 60, 
            "some_server": "foo.southeast.example.com"
        }
```

you can see that `escape_pods` _are_ defined on the group level, but they are also defined on the hostvars level.

As a counter-example, if you run the file `static/vars_dirs/inv` in that repo, you find that group var `halon_system_timeout` is _only_ defined in hostvars (it comes from a `group_vars` folder).

I expect that `escape_pods` will be removed from the group-level vars in the normal "flat" use as a side-effect of this enhancement. This would make the behavior fully consistent.
yep, that is a bug
In addition to Alan's concern about memory efficiency, the big issue here for Tower / AWX is that in the UI and API prior to running a playbook, we actually do need to distinguish between host vars and group vars. If you have a group var defined, it makes no sense for Tower to display it as a host var on each of your 10,000 hosts, and have an empty "group vars" section in the UI that never gets populated.

I completely understand that when you're actually running a playbook, Ansible flattens this and there is no difference between a host and a group var - each host just gets what it gets in the the host loop.  But that's not the only use case for the ansible-inventory script, and if all the dynamic inventory sources are going to be in core now, Tower / AWX needs some way of picking up group vars from them.

(@matburt please comment on the above and let me know what parts I got wrong)
Thanks for clarifying - I'm going to look at porting parts of Alan's start on this to the version of ansible-inventory that's on devel. 
Looking at the implementation options for this, it's going to require some deep changes to how values are discovered (and track their source) for hosts. It would be along the lines of https://github.com/ansible/ansible/compare/devel...alikins:var_man_changed_inv_plugins , which definitely can't be done after the 2.5 core freeze, so this fix will have to wait until the next feature cycle. 
@ryansb I tend to agree with pushing this to 2.6, 
Ignore the invasiveness of https://github.com/ansible/ansible/compare/devel...alikins:var_man_changed_inv_plugins , the other issue would that it is based
on tracking scope/context of vars as setup by a playbook.  ansible-inventory does
it own thing for building up the data structures it uses for dumping the output that
mostly bypasses VariableManager.get_vars() and combine_vars() so that branch
doesn't know anything about the results.
i think we can still have this in 2.5, just diff approach, just use some of the functions from varsmanager directly instead of consuming the results of `get_vars`

Resolving patch:
diff --git a/lib/ansible/cli/inventory.py b/lib/ansible/cli/inventory.py
--- a/lib/ansible/cli/inventory.py
+++ b/lib/ansible/cli/inventory.py
@@ -20,9 +20,13 @@
 import optparse
 from operator import attrgetter
 
+from ansible import constants as C
 from ansible.cli import CLI
-from ansible.errors import AnsibleOptionsError
+from ansible.errors import AnsibleError, AnsibleOptionsError
+from ansible.inventory.host import Host
+from ansible.plugins.loader import vars_loader
 from ansible.parsing.dataloader import DataLoader
+from ansible.utils.vars import combine_vars
 
 try:
     from __main__ import display
@@ -83,11 +87,20 @@ def parse(self):
         self.parser.add_option_group(action_group)
 
         # Options
+
+        # graph
         self.parser.add_option("-y", "--yaml", action="store_true", default=False, dest='yaml',
                                help='Use YAML format instead of default JSON, ignored for --graph')
         self.parser.add_option("--vars", action="store_true", default=False, dest='show_vars',
                                help='Add vars to graph display, ignored unless used with --graph')
 
+        # list
+        self.parser.add_option("--export", action="store_true", default=C.INVENTORY_EXPORT, dest='export',
+                               help="When doing an --list, represent in a way that is optimized for export,"
+                                    "not as an accurate representation of how Ansible has processed it")
+        # self.parser.add_option("--ignore-vars-plugins", action="store_true", default=False, dest='ignore_vars_plugins',
+        #                       help="When doing an --list, skip vars data from vars plugins, by default, this would include group_vars/ and host_vars/")
+
         super(InventoryCLI, self).parse()
 
         display.verbosity = self.options.verbosity
@@ -183,11 +196,63 @@ def dump(self, stuff):
 
         return results
 
+    # FIXME: refactor to use same for VM
+    def get_plugin_vars(self, path, entity):
+
+        data = {}
+
+        def _get_plugin_vars(plugin, path, entities):
+            data = {}
+            try:
+                data = plugin.get_vars(self.loader, path, entity)
+            except AttributeError:
+                try:
+                    if isinstance(entity, Host):
+                        data.update(plugin.get_host_vars(entity.name))
+                    else:
+                        data.update(plugin.get_group_vars(entity.name))
+                except AttributeError:
+                    if hasattr(plugin, 'run'):
+                        raise AnsibleError("Cannot use v1 type vars plugin %s from %s" % (plugin._load_name, plugin._original_path))
+                    else:
+                        raise AnsibleError("Invalid vars plugin %s from %s" % (plugin._load_name, plugin._original_path))
+            return data
+
+        for plugin in vars_loader.all():
+            data = combine_vars(data, _get_plugin_vars(plugin, path, entity))
+
+        return data
+
+    def _get_group_variables(self, group):
+
+        # get info from inventory source
+        res = group.get_vars()
+
+        # FIXME: add switch to skip vars plugins
+        # add vars plugin info
+        for inventory_dir in self.inventory._sources:
+            res = combine_vars(res, self.get_plugin_vars(inventory_dir, group))
+
+        if group.priority != 1:
+            res['ansible_group_priority'] = group.priority
+
+        return res
+
     def _get_host_variables(self, host):
-        if self._new_api:
-            hostvars = self.vm.get_vars(host=host)
+
+        if self.options.export:
+            hostvars = host.get_vars()
+
+            # FIXME: add switch to skip vars plugins
+            # add vars plugin info
+            for inventory_dir in self.inventory._sources:
+                hostvars = combine_vars(hostvars, self.get_plugin_vars(inventory_dir, host))
         else:
-            hostvars = self.vm.get_vars(self.loader, host=host)
+            if self._new_api:
+                hostvars = self.vm.get_vars(host=host, include_hostvars=False)
+            else:
+                hostvars = self.vm.get_vars(self.loader, host=host, include_hostvars=False)
+
         return hostvars
 
     def _get_group(self, gname):
@@ -257,8 +322,11 @@ def format_group(group):
             for subgroup in sorted(group.child_groups, key=attrgetter('name')):
                 results[group.name]['children'].append(subgroup.name)
                 results.update(format_group(subgroup))
+            if self.options.export:
+                results[group.name]['vars'] = self._get_group_variables(group)
 
             self._remove_empty(results[group.name])
+
             return results
 
         results = format_group(top)
@@ -267,8 +335,10 @@ def format_group(group):
         results['_meta'] = {'hostvars': {}}
         hosts = self.inventory.get_hosts()
         for host in hosts:
-            results['_meta']['hostvars'][host.name] = self._get_host_variables(host=host)
-            self._remove_internal(results['_meta']['hostvars'][host.name])
+            hvars = self._get_host_variables(host)
+            if hvars:
+                self._remove_internal(hvars)
+                results['_meta']['hostvars'][host.name] = hvars
 
         return results
 
@@ -299,7 +369,14 @@ def format_group(group):
                         self._remove_internal(myvars)
                     results[group.name]['hosts'][h.name] = myvars
 
+            if self.options.export:
+
+                gvars = self._get_group_variables(group)
+                if gvars:
+                    results[group.name]['vars'] = gvars
+
             self._remove_empty(results[group.name])
+
             return results
 
         return format_group(top)


Test patch: