Repository: ansible-collections/community.general

Issue:
OpenNebula dynamic inventory plugin does not support keyed_groups
### Summary

The documentation for the OpenNebula dynamic inventory plugin [0] refers to an option to created keyed_groups (because the documentation extends it).

```
    extends_documentation_fragment:
        - constructed
```

However, this functionality is not implemented.  Even though the `InventoryModule` inherits from `Constructable`, `self._add_host_to_keyed_groups()` is never called.

[0] https://docs.ansible.com/ansible/latest/collections/community/general/opennebula_inventory.html

### Issue Type

Bug Report

### Component Name

opennebula dynamic inventory plugin

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.4]
  config file = /code/rafiki/ansible.cfg
  configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vagrant/.pyenv/versions/3.10.3/lib/python3.10/site-packages/ansible
  ansible collection location = /home/vagrant/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/vagrant/.pyenv/versions/3.10.3/bin/ansible
  python version = 3.10.3 (main, Apr  6 2022, 22:36:07) [GCC 11.2.0]
  jinja version = 3.1.1
  libyaml = True

```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general

# /home/vagrant/.pyenv/versions/3.10.3/lib/python3.10/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 4.7.0  

```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/code/rafiki/ansible.cfg) = False

```


### OS / Environment

```
vagrant@ubuntu-jammy:/code/rafiki$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu Jammy Jellyfish (development branch)"
```

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->

opennebula.yml:

```yaml (paste below)
plugin: community.general.opennebula
api_url: "RPC_URL"
# specify these here, or in the form username:password in ~/.one/one_auth or export ONE_AUTH=/some/other/path
#api_username:
#api_password:
keyed_groups:
  - prefix: distro
    key: ansible_distribution

```

`ansible-inventory --list -i opennebula.yml`

### Expected Results

I expected a grouping of hosts by distribution.  Note that the plugin does perform grouping by LABEL, which is seen below with the `srv` and `foo` labels/groups.

### Actual Results

```console (paste below)
```console
ansible-inventory --list -i opennebula.yml
{
    "_meta": {
        "hostvars": {
            "terraform_demo_00": {
                "INFO": "",
                "INPUTS_ORDER": "",
                "LABELS": [],
                "LOGO": "images/logos/logo.png",
                "LXD_SECURITY_PRIVILEGED": "true",
                "MEMORY_UNIT_COST": "MB",
                "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
                "ansible_host": "192.168.11.235",
                "name": "terraform_demo_00",
                "v4_first_ip": "192.168.11.235",
                "v6_first_ip": false
            },
            "terraform_demo_01": {
                "INFO": "",
                "INPUTS_ORDER": "",
                "LABELS": [
                    "srv",
                    "foo"
                ],
                "LOGO": "images/logos/logo.png",
                "LXD_SECURITY_PRIVILEGED": "true",
                "MEMORY_UNIT_COST": "MB",
                "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
                "ansible_host": "192.168.11.241",
                "name": "terraform_demo_01",
                "v4_first_ip": "192.168.11.241",
                "v6_first_ip": false
            }
        }
    },
    "all": {
        "children": [
            "srv",
            "foo",
            "ungrouped"
        ]
    },
    "srv": {
        "hosts": [
            "terraform_demo_01"
        ]
    },
    "foo": {
        "hosts": [
            "terraform_demo_01"
        ]
    },
    "ungrouped": {
        "hosts": [
            "terraform_demo_00"
        ]
    }
}
```
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* `contrib/inventory`
* `lib/ansible/plugins/inventory`

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
!component =plugins/inventory/opennebula.py
Files identified in the description:
* [`plugins/inventory/opennebula.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/opennebula.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @feldsam
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Hi @bsanders , I don;t know how keyed groups work on backgroud, would you share xcepted result as json code? From what variables you want to get distribution? I don't see it in opennebula user template
I was not super clear on how the ansible_distribution was supposed to turn into keyed_groups either.  I was looking at this [0], specifically the "examples".  I guess I was assuming ansible would reach out to get the facts, including distro, *then* sort things into groups.  But that wasn't happening.

To figure out what actually happens, I hacked the plugin and I've stepped through the code with pdb.  With my changes, I'm able to say `keyed_groups: key:some_opennebula_attribute` and it works.  Not sure if `key: ansible_distribution` is supposed to work or not.

Would you be willing to look at a PR to add this functionality?  I may attempt to implement the other `Constructable` features, though I'm not sure I'll need them myself.

[0] https://docs.ansible.com/ansible/latest/collections/ansible/builtin/constructed_inventory.html
This isn't a bug report, but a feature request. Inventory plugins have to opt-in to support keyed groups (and similar features).
The current documentation for the plugin says it supports it, but it's not implemented. I spent an hour trying to figure out what I was doing wrong before looking at the code and realizing it just wasn't implemented. 
This is bug in documentation. So we can fix it, but how, when it is extended?
Ah, I completely missed the `extends_documentation_fragment`. Sorry.
@feldsam basically what you need to do is: https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/nmap.py#L86-L93
I have the code working.  Currently looking into adding some unit tests.

Resolving patch:
diff --git a/plugins/inventory/opennebula.py b/plugins/inventory/opennebula.py
--- a/plugins/inventory/opennebula.py
+++ b/plugins/inventory/opennebula.py
@@ -206,28 +206,40 @@ def _retrieve_servers(self, label_filter=None):
     def _populate(self):
         hostname_preference = self.get_option('hostname')
         group_by_labels = self.get_option('group_by_labels')
+        strict = self.get_option('strict')
 
         # Add a top group 'one'
         self.inventory.add_group(group='all')
 
         filter_by_label = self.get_option('filter_by_label')
-        for server in self._retrieve_servers(filter_by_label):
+        servers = self._retrieve_servers(filter_by_label)
+        for server in servers:
+            hostname = server['name']
             # check for labels
             if group_by_labels and server['LABELS']:
                 for label in server['LABELS']:
                     self.inventory.add_group(group=label)
-                    self.inventory.add_host(host=server['name'], group=label)
+                    self.inventory.add_host(host=hostname, group=label)
 
-            self.inventory.add_host(host=server['name'], group='all')
+            self.inventory.add_host(host=hostname, group='all')
 
             for attribute, value in server.items():
-                self.inventory.set_variable(server['name'], attribute, value)
+                self.inventory.set_variable(hostname, attribute, value)
 
             if hostname_preference != 'name':
-                self.inventory.set_variable(server['name'], 'ansible_host', server[hostname_preference])
+                self.inventory.set_variable(hostname, 'ansible_host', server[hostname_preference])
 
             if server.get('SSH_PORT'):
-                self.inventory.set_variable(server['name'], 'ansible_port', server['SSH_PORT'])
+                self.inventory.set_variable(hostname, 'ansible_port', server['SSH_PORT'])
+
+            # handle construcable implementation: get composed variables if any
+            self._set_composite_vars(self.get_option('compose'), server, hostname, strict=strict)
+
+            # groups based on jinja conditionals get added to specific groups
+            self._add_host_to_composed_groups(self.get_option('groups'), server, hostname, strict=strict)
+
+            # groups based on variables associated with them in the inventory
+            self._add_host_to_keyed_groups(self.get_option('keyed_groups'), server, hostname, strict=strict)
 
     def parse(self, inventory, loader, path, cache=True):
         if not HAS_PYONE:


Test patch:
diff --git a/tests/unit/plugins/inventory/fixtures/opennebula_inventory.json b/tests/unit/plugins/inventory/fixtures/opennebula_inventory.json
new file mode 100644
--- /dev/null
+++ b/tests/unit/plugins/inventory/fixtures/opennebula_inventory.json
@@ -0,0 +1,222 @@
+[
+    {
+        "DEPLOY_ID": "bcfec9d9-c0d0-4523-b5e7-62993947e94c",
+        "ETIME": 0,
+        "GID": 105,
+        "GNAME": "SW",
+        "HISTORY_RECORDS": {},
+        "ID": 451,
+        "LAST_POLL": 0,
+        "LCM_STATE": 3,
+        "MONITORING": {},
+        "NAME": "terraform_demo_00",
+        "RESCHED": 0,
+        "STATE": 3,
+        "STIME": 1649886492,
+        "TEMPLATE": {
+            "NIC": [
+                {
+                    "AR_ID": "0",
+                    "BRIDGE": "mgmt0",
+                    "BRIDGE_TYPE": "linux",
+                    "CLUSTER_ID": "0",
+                    "IP": "192.168.11.248",
+                    "MAC": "02:00:c0:a8:2b:bb",
+                    "MODEL": "virtio",
+                    "NAME": "NIC0",
+                    "NETWORK": "Infrastructure",
+                    "NETWORK_ID": "0",
+                    "NIC_ID": "0",
+                    "SECURITY_GROUPS": "0,101",
+                    "TARGET": "one-453-0",
+                    "VLAN_ID": "12",
+                    "VN_MAD": "802.1Q"
+                }
+            ],
+            "NIC_DEFAULT": {
+                "MODEL": "virtio"
+            },
+            "TEMPLATE_ID": "28",
+            "TM_MAD_SYSTEM": "shared",
+            "VCPU": "4",
+            "VMID": "453"
+        },
+        "USER_TEMPLATE": {
+            "GUEST_OS": "linux",
+            "INPUTS_ORDER": "",
+            "LABELS": "foo,bench",
+            "LOGO": "images/logos/linux.png",
+            "MEMORY_UNIT_COST": "MB",
+            "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
+            "TGROUP": "bench_clients"
+        }
+    },
+    {
+        "DEPLOY_ID": "25895435-5e3a-4d50-a025-e03a7a463abd",
+        "ETIME": 0,
+        "GID": 105,
+        "GNAME": "SW",
+        "HISTORY_RECORDS": {},
+        "ID": 451,
+        "LAST_POLL": 0,
+        "LCM_STATE": 3,
+        "MONITORING": {},
+        "NAME": "terraform_demo_01",
+        "RESCHED": 0,
+        "STATE": 3,
+        "STIME": 1649886492,
+        "TEMPLATE": {
+            "NIC": [
+                {
+                    "AR_ID": "0",
+                    "BRIDGE": "mgmt0",
+                    "BRIDGE_TYPE": "linux",
+                    "CLUSTER_ID": "0",
+                    "IP": "192.168.11.241",
+                    "MAC": "02:00:c0:a8:4b:bb",
+                    "MODEL": "virtio",
+                    "NAME": "NIC0",
+                    "NETWORK": "Infrastructure",
+                    "NETWORK_ID": "0",
+                    "NIC_ID": "0",
+                    "SECURITY_GROUPS": "0,101",
+                    "TARGET": "one-451-0",
+                    "VLAN_ID": "12",
+                    "VN_MAD": "802.1Q"
+                }
+            ],
+            "NIC_DEFAULT": {
+                "MODEL": "virtio"
+            },
+            "TEMPLATE_ID": "28",
+            "TM_MAD_SYSTEM": "shared",
+            "VCPU": "4",
+            "VMID": "451"
+        },
+        "USER_TEMPLATE": {
+            "GUEST_OS": "linux",
+            "INPUTS_ORDER": "",
+            "LABELS": "foo,bench",
+            "LOGO": "images/logos/linux.png",
+            "MEMORY_UNIT_COST": "MB",
+            "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
+            "TESTATTR": "testvar",
+            "TGROUP": "bench_clients"
+        }
+    },
+    {
+        "DEPLOY_ID": "2b00c379-3601-45ee-acf5-e7b3ff2b7bca",
+        "ETIME": 0,
+        "GID": 105,
+        "GNAME": "SW",
+        "HISTORY_RECORDS": {},
+        "ID": 451,
+        "LAST_POLL": 0,
+        "LCM_STATE": 3,
+        "MONITORING": {},
+        "NAME": "terraform_demo_srv_00",
+        "RESCHED": 0,
+        "STATE": 3,
+        "STIME": 1649886492,
+        "TEMPLATE": {
+            "NIC": [
+                {
+                    "AR_ID": "0",
+                    "BRIDGE": "mgmt0",
+                    "BRIDGE_TYPE": "linux",
+                    "CLUSTER_ID": "0",
+                    "IP": "192.168.11.247",
+                    "MAC": "02:00:c0:a8:0b:cc",
+                    "MODEL": "virtio",
+                    "NAME": "NIC0",
+                    "NETWORK": "Infrastructure",
+                    "NETWORK_ID": "0",
+                    "NIC_ID": "0",
+                    "SECURITY_GROUPS": "0,101",
+                    "TARGET": "one-452-0",
+                    "VLAN_ID": "12",
+                    "VN_MAD": "802.1Q"
+                }
+            ],
+            "NIC_DEFAULT": {
+                "MODEL": "virtio"
+            },
+            "TEMPLATE_ID": "28",
+            "TM_MAD_SYSTEM": "shared",
+            "VCPU": "4",
+            "VMID": "452"
+        },
+        "USER_TEMPLATE": {
+            "GUEST_OS": "linux",
+            "INPUTS_ORDER": "",
+            "LABELS": "serv,bench",
+            "LOGO": "images/logos/linux.png",
+            "MEMORY_UNIT_COST": "MB",
+            "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
+            "TGROUP": "bench_server"
+        }
+    },
+    {
+        "DEPLOY_ID": "97037f55-dd2c-4549-8d24-561a6569e870",
+        "ETIME": 0,
+        "GID": 105,
+        "GNAME": "SW",
+        "HISTORY_RECORDS": {},
+        "ID": 311,
+        "LAST_POLL": 0,
+        "LCM_STATE": 3,
+        "MONITORING": {},
+        "NAME": "bs-windows",
+        "RESCHED": 0,
+        "STATE": 3,
+        "STIME": 1648076254,
+        "TEMPLATE": {
+            "NIC": [
+                {
+                    "AR_ID": "0",
+                    "BRIDGE": "mgmt0",
+                    "BRIDGE_TYPE": "linux",
+                    "CLUSTER_ID": "0",
+                    "IP": "192.168.11.209",
+                    "MAC": "02:00:c0:a8:0b:dd",
+                    "MODEL": "virtio",
+                    "NAME": "NIC0",
+                    "NETWORK": "Infrastructure",
+                    "NETWORK_ID": "0",
+                    "NETWORK_UNAME": "admin",
+                    "NIC_ID": "0",
+                    "SECURITY_GROUPS": "0,101",
+                    "TARGET": "one-311-0",
+                    "VLAN_ID": "12",
+                    "VN_MAD": "802.1Q"
+                },
+                [
+                    "TEMPLATE_ID",
+                    "23"
+                ],
+                [
+                    "TM_MAD_SYSTEM",
+                    "shared"
+                ],
+                [
+                    "VCPU",
+                    "4"
+                ],
+                [
+                    "VMID",
+                    "311"
+                ]
+            ]
+        },
+        "UID": 22,
+        "UNAME": "bsanders",
+        "USER_TEMPLATE": {
+            "GUEST_OS": "windows",
+            "INPUTS_ORDER": "",
+            "LABELS": "serv",
+            "HYPERVISOR": "kvm",
+            "SCHED_REQUIREMENTS": "ARCH=\"x86_64\"",
+            "SET_HOSTNAME": "windows"
+        }
+    }
+]
diff --git a/tests/unit/plugins/inventory/test_opennebula.py b/tests/unit/plugins/inventory/test_opennebula.py
--- a/tests/unit/plugins/inventory/test_opennebula.py
+++ b/tests/unit/plugins/inventory/test_opennebula.py
@@ -9,14 +9,18 @@
 __metaclass__ = type
 
 from collections import OrderedDict
+import json
 
 import pytest
 
 from ansible.inventory.data import InventoryData
+from ansible.parsing.dataloader import DataLoader
+from ansible.template import Templar
 from ansible_collections.community.general.plugins.inventory.opennebula import InventoryModule
+from ansible_collections.community.general.tests.unit.compat.mock import create_autospec
 
 
-@pytest.fixture(scope="module")
+@pytest.fixture
 def inventory():
     r = InventoryModule()
     r.inventory = InventoryData()
@@ -33,6 +37,18 @@ def test_verify_file_bad_config(inventory):
     assert inventory.verify_file('foobar.opennebula.yml') is False
 
 
+def get_vm_pool_json():
+    with open('tests/unit/plugins/inventory/fixtures/opennebula_inventory.json', 'r') as json_file:
+        jsondata = json.load(json_file)
+
+    data = type('pyone.bindings.VM_POOLSub', (object,), {'VM': []})()
+
+    for fake_server in jsondata:
+        data.VM.append(type('pyone.bindings.VMType90Sub', (object,), fake_server)())
+
+    return data
+
+
 def get_vm_pool():
     data = type('pyone.bindings.VM_POOLSub', (object,), {'VM': []})()
 
@@ -195,36 +211,99 @@ def get_vm_pool():
     return data
 
 
-def get_option(option):
-    if option == 'api_url':
-        return 'https://opennebula:2633/RPC2'
-    if option == 'api_username':
-        return 'username'
-    elif option == 'api_password':
-        return 'password'
-    elif option == 'api_authfile':
-        return '~/.one/one_auth'
-    elif option == 'hostname':
-        return 'v4_first_ip'
-    elif option == 'group_by_labels':
-        return True
-    elif option == 'filter_by_label':
-        return None
-    else:
-        return False
+options_base_test = {
+    'api_url': 'https://opennebula:2633/RPC2',
+    'api_username': 'username',
+    'api_password': 'password',
+    'api_authfile': '~/.one/one_auth',
+    'hostname': 'v4_first_ip',
+    'group_by_labels': True,
+    'filter_by_label': None,
+}
+
+options_constructable_test = options_base_test.copy()
+options_constructable_test.update({
+    'compose': {'is_linux': "GUEST_OS == 'linux'"},
+    'filter_by_label': 'bench',
+    'groups': {
+        'benchmark_clients': "TGROUP.endswith('clients')",
+        'lin': 'is_linux == True'
+    },
+    'keyed_groups': [{'key': 'TGROUP', 'prefix': 'tgroup'}],
+
+})
+
+
+# given a dictionary `opts_dict`, return a function that behaves like ansible's inventory get_options
+def mk_get_options(opts_dict):
+    def inner(opt):
+        return opts_dict.get(opt, False)
+    return inner
 
 
 def test_get_connection_info(inventory, mocker):
-    inventory.get_option = mocker.MagicMock(side_effect=get_option)
+    inventory.get_option = mocker.MagicMock(side_effect=mk_get_options(options_base_test))
 
     auth = inventory._get_connection_info()
     assert (auth.username and auth.password)
 
 
+def test_populate_constructable_templating(inventory, mocker):
+    # bypass API fetch call
+    inventory._get_vm_pool = mocker.MagicMock(side_effect=get_vm_pool_json)
+    inventory.get_option = mocker.MagicMock(side_effect=mk_get_options(options_constructable_test))
+
+    # the templating engine is needed for the constructable groups/vars
+    # so give that some fake data and instantiate it.
+    fake_config_filepath = '/fake/opennebula.yml'
+    fake_cache = {fake_config_filepath: options_constructable_test.copy()}
+    fake_cache[fake_config_filepath]['plugin'] = 'community.general.opennebula'
+    dataloader = create_autospec(DataLoader, instance=True)
+    dataloader._FILE_CACHE = fake_cache
+    inventory.templar = Templar(loader=dataloader)
+
+    inventory._populate()
+
+    # note the vm_pool (and json data file) has four hosts,
+    # but options_constructable_test asks ansible to filter it out
+    assert len(get_vm_pool_json().VM) == 4
+    assert set([vm.NAME for vm in get_vm_pool_json().VM]) == set([
+        'terraform_demo_00',
+        'terraform_demo_01',
+        'terraform_demo_srv_00',
+        'bs-windows',
+    ])
+    assert set(inventory.inventory.hosts) == set(['terraform_demo_00', 'terraform_demo_01', 'terraform_demo_srv_00'])
+
+    host_demo00 = inventory.inventory.get_host('terraform_demo_00')
+    host_demo01 = inventory.inventory.get_host('terraform_demo_01')
+    host_demosrv = inventory.inventory.get_host('terraform_demo_srv_00')
+
+    assert 'benchmark_clients' in inventory.inventory.groups
+    assert 'lin' in inventory.inventory.groups
+    assert inventory.inventory.groups['benchmark_clients'].hosts == [host_demo00, host_demo01]
+    assert inventory.inventory.groups['lin'].hosts == [host_demo00, host_demo01, host_demosrv]
+
+    # test group by label:
+    assert 'bench' in inventory.inventory.groups
+    assert 'foo' in inventory.inventory.groups
+    assert inventory.inventory.groups['bench'].hosts == [host_demo00, host_demo01, host_demosrv]
+    assert inventory.inventory.groups['serv'].hosts == [host_demosrv]
+    assert inventory.inventory.groups['foo'].hosts == [host_demo00, host_demo01]
+
+    # test `compose` transforms GUEST_OS=Linux to is_linux == True
+    assert host_demo00.get_vars()['GUEST_OS'] == 'linux'
+    assert host_demo00.get_vars()['is_linux'] is True
+
+    # test `keyed_groups`
+    assert inventory.inventory.groups['tgroup_bench_clients'].hosts == [host_demo00, host_demo01]
+    assert inventory.inventory.groups['tgroup_bench_server'].hosts == [host_demosrv]
+
+
 def test_populate(inventory, mocker):
     # bypass API fetch call
     inventory._get_vm_pool = mocker.MagicMock(side_effect=get_vm_pool)
-    inventory.get_option = mocker.MagicMock(side_effect=get_option)
+    inventory.get_option = mocker.MagicMock(side_effect=mk_get_options(options_base_test))
     inventory._populate()
 
     # get different hosts