Repository: ansible-collections/community.general

Issue:
ansible_collections.community.general.plugins.inventory.icinga2 plugin: HTTP Error 404: Not Found
### Summary

When I try to use the icinga2 inventory plugin, ansible-inventory reports a 404: Not found.


### Issue Type

Bug Report

### Component Name
community.general.icinga2
icinga2.py

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.11.4]

```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general
community.general 4.1.0

```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
INVENTORY_ENABLED(/etc/ansible/ansible.cfg) = ['community.general.icinga2', 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml']

```


### OS / Environment

Ubuntu 20.04

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
root @ ~ # ansible-inventory -i my.icinga2.yml --list
[WARNING]:  * Failed to parse /root/my.icinga2.yml with ansible_collections.community.general.plugins.inventory.icinga2 plugin: HTTP Error 404: Not Found

cat my.icinga2.yml
# my.icinga2.yml
plugin: community.general.icinga2
url: https://excample.domain.com:5665
user: ansible
password: secret
host_filter: None
validate_certs: false
```


### Expected Results

A valid inventory

### Actual Results

```console (paste below)
Nothing
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
None

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* ``

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [`plugins/inventory/icinga2.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/icinga2.py)
* [`plugins/modules/monitoring/icinga2_feature.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/monitoring/icinga2_feature.py)
* [`plugins/modules/monitoring/icinga2_host.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/monitoring/icinga2_host.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @bongoeadgc6 @nerzhul @t794104
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Good morning,

You must modify your my.icinga2.yml file to reflect your instance of Icinga

Does navigating to https://excample.domain.com:5665 lead to your Icinga server?
Good morning @BongoEADGC6 
I have anonymized my snippet. Of course, I configured it to my icinga master server and I also tested the API access via curl, which works,
In your example, the api is accessed via http. But my api is only accessable via https. could this be the problem?

@MarcusCaepio Can you remove the following from your config file `host_filter: None`? This is causing no hosts to be found. Likely we need to provide a better handling if the filter given doesn't provide any hosts.
Comment out the host_filter, but got the same result. Also trying a valid filter does not help.
```
ansible-inventory [core 2.11.4]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.8.10 (default, Jun  2 2021, 10:49:15) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /root/my.icinga2.yml as it did not pass its verify_file() method
script declined parsing /root/my.icinga2.yml as it did not pass its verify_file() method
toml declined parsing /root/my.icinga2.yml as it did not pass its verify_file() method
[WARNING]:  * Failed to parse /root/my.icinga2.yml with ansible_collections.community.general.plugins.inventory.icinga2 plugin: HTTP Error 404: Not Found
  File "/root/.local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/root/.ansible/collections/ansible_collections/community/general/plugins/inventory/icinga2.py", line 219, in parse
    self._api_connect()
  File "/root/.ansible/collections/ansible_collections/community/general/plugins/inventory/icinga2.py", line 104, in _api_connect
    open_url(api_status_url, **request_args)
  File "/root/.local/lib/python3.8/site-packages/ansible/module_utils/urls.py", line 1535, in open_url
    return Request().open(method, url, data=data, headers=headers, use_proxy=use_proxy,
  File "/root/.local/lib/python3.8/site-packages/ansible/module_utils/urls.py", line 1446, in open
    return urllib_request.urlopen(request, None, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1062, in http_error_401
    response = self.http_error_auth_reqed('www-authenticate',
  File "/usr/lib/python3.8/urllib/request.py", line 1011, in http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
  File "/usr/lib/python3.8/urllib/request.py", line 1026, in retry_http_basic_auth
    return self.parent.open(req, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
[WARNING]:  * Failed to parse /root/my.icinga2.yml with auto plugin: HTTP Error 404: Not Found
  File "/root/.local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/root/.local/lib/python3.8/site-packages/ansible/plugins/inventory/auto.py", line 58, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/root/.ansible/collections/ansible_collections/community/general/plugins/inventory/icinga2.py", line 219, in parse
    self._api_connect()
  File "/root/.ansible/collections/ansible_collections/community/general/plugins/inventory/icinga2.py", line 104, in _api_connect
    open_url(api_status_url, **request_args)
  File "/root/.local/lib/python3.8/site-packages/ansible/module_utils/urls.py", line 1535, in open_url
    return Request().open(method, url, data=data, headers=headers, use_proxy=use_proxy,
  File "/root/.local/lib/python3.8/site-packages/ansible/module_utils/urls.py", line 1446, in open
    return urllib_request.urlopen(request, None, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1062, in http_error_401
    response = self.http_error_auth_reqed('www-authenticate',
  File "/usr/lib/python3.8/urllib/request.py", line 1011, in http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
  File "/usr/lib/python3.8/urllib/request.py", line 1026, in retry_http_basic_auth
    return self.parent.open(req, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
[WARNING]:  * Failed to parse /root/my.icinga2.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
  File "/root/.local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/root/.local/lib/python3.8/site-packages/ansible/plugins/inventory/yaml.py", line 112, in parse
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]:  * Failed to parse /root/my.icinga2.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.
  File "/root/.local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/root/.local/lib/python3.8/site-packages/ansible/plugins/inventory/ini.py", line 136, in parse
    raise AnsibleParserError(e)
[WARNING]: Unable to parse /root/my.icinga2.yml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

```
A curl running `curl -k -s -S -i -u ansible:secret 'https://example.domain.com:5665/v1/objects/hosts'` results all hosts
The plugin does a POST request, maybe that's a potential reason for the differences?
Ok I am one step further: The API user needs permissions for ""status/query". This should be a hint in the docs ;)
Now I am struggling with 
```
File "/root/.local/lib/python3.8/site-packages/ansible/inventory/data.py", line 232, in add_host
    raise AnsibleError("Invalid empty host name provided: %s" % host)

```
So, I got it running, but:
The dictionaries are stored with the host_addr and not with the host_name. But address is not mandatory in icinga (see: https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#host), a host object name is. The inventory plugin fails, when a host object got no address. So we replaced the host_addr variables with host_name and the inventory creation also works without a filter and for hosts without an address (one may discuss, if it makes sense to add hosts, which do not have an address):

```
176     def _convert_inv(self, json_data):
177         """Convert Icinga2 API data to JSON format for Ansible"""
178         groups_dict = {"_meta": {"hostvars": {}}}
179         for entry in json_data:
180             host_name = entry['name']
181             host_attrs = entry['attrs']
182             if host_attrs['state'] == 0:
183                 host_attrs['state'] = 'on'
184             else:
185                 host_attrs['state'] = 'off'
186             host_groups = host_attrs['groups']
187             host_addr = host_attrs['address']
188             self.inventory.add_host(host_name)
189             for group in host_groups:
190                 if group not in self.inventory.groups.keys():
191                     self.inventory.add_group(group)
192                 self.inventory.add_child(group, host_name)
193             self.inventory.set_variable(host_name, 'address', host_addr)
194             self.inventory.set_variable(host_name, 'hostname', host_name)
195             self.inventory.set_variable(host_name, 'state',
196                                         host_attrs['state'])
197             self.inventory.set_variable(host_name, 'state_type',
198                                         host_attrs['state_type'])
199         return groups_dict
```

Another advantage of this is, that you can filter and limit your playbooks with host_names instead of IPs.
I can fork and create a PR, if you wish. 
Very interesting. What version of Icinga2 are you using? Icinga2 v2.12.2-1, when you query for a host without a configured address field the address attribute is still configured with the name field from what I'm seeing. This also allows the host to be seen by the plugin.

Icinga Config
```
object Host "applications.example.com" {
    import "generic-host"
    display_name = "Apps Server"
    notes = "Checks for access to applications.example.com"
}

```

Icinga API Results
```
{
    "results": [
            "attrs": {
                "__name": "applications.example.com",
                "acknowledgement": 0.0,
                "acknowledgement_expiry": 0.0,
                "acknowledgement_last_change": 0.0,
                "action_url": "",
                "active": true,
                "address": "applications.example.com",
                "address6": "",
                "check_attempt": 1.0,
                "check_command": "hostalive",
                ...
            }
            ...
            "joins": {},
            "meta": {},
            "name": "applications.example.com",
            "type": "Host"
    ]
}
```

Is this not the case for you?
I am running r2.13.2-1 and address is empty, if it is not set in the config,e.g:
```
object Host "dummy" {
  notes = "Virtual Host"
  check_command = "dummy"
   [.... a few vars.... ]
}
```

Results in

```
{
  "results": [
    {
      "attrs": {
        "__name": "dummy",
        "acknowledgement": 0,
        "acknowledgement_expiry": 0,
        "acknowledgement_last_change": 0,
        "action_url": "",
        "active": true,
        "address": "",
        "address6": "",
        [ ...]
```
@MarcusCaepio Fascinating. Going to review the changelog to see what happened. Likely we can just look for address field. If it's populated, use that, if not, fall back to the host object name.

Resolving patch:
diff --git a/plugins/inventory/icinga2.py b/plugins/inventory/icinga2.py
--- a/plugins/inventory/icinga2.py
+++ b/plugins/inventory/icinga2.py
@@ -35,13 +35,23 @@
         type: string
         required: true
       host_filter:
-        description: An Icinga2 API valid host filter.
+        description:
+          - An Icinga2 API valid host filter. Leave blank for no filtering
         type: string
         required: false
       validate_certs:
         description: Enables or disables SSL certificate verification.
         type: boolean
         default: true
+      inventory_attr:
+        description:
+          - Allows the override of the inventory name based on different attributes.
+          - This allows for changing the way limits are used.
+          - The current default, C(address), is sometimes not unique or present. We recommend to use C(name) instead.
+        type: string
+        default: address
+        choices: ['name', 'display_name', 'address']
+        version_added: 4.2.0
 '''
 
 EXAMPLES = r'''
@@ -52,6 +62,7 @@
 password: secure
 host_filter: \"linux-servers\" in host.groups
 validate_certs: false
+inventory_attr: name
 '''
 
 import json
@@ -59,6 +70,7 @@
 from ansible.errors import AnsibleParserError
 from ansible.plugins.inventory import BaseInventoryPlugin, Constructable
 from ansible.module_utils.urls import open_url
+from ansible.module_utils.six.moves.urllib.error import HTTPError
 
 
 class InventoryModule(BaseInventoryPlugin, Constructable):
@@ -76,6 +88,7 @@ def __init__(self):
         self.icinga2_password = None
         self.ssl_verify = None
         self.host_filter = None
+        self.inventory_attr = None
 
         self.cache_key = None
         self.use_cache = None
@@ -114,9 +127,21 @@ def _post_request(self, request_url, data=None):
         if data is not None:
             request_args['data'] = json.dumps(data)
         self.display.vvv("Request Args: %s" % request_args)
-        response = open_url(request_url, **request_args)
+        try:
+            response = open_url(request_url, **request_args)
+        except HTTPError as e:
+            try:
+                error_body = json.loads(e.read().decode())
+                self.display.vvv("Error returned: {0}".format(error_body))
+            except Exception:
+                error_body = {"status": None}
+            if e.code == 404 and error_body.get('status') == "No objects found.":
+                raise AnsibleParserError("Host filter returned no data. Please confirm your host_filter value is valid")
+            raise AnsibleParserError("Unexpected data returned: {0} -- {1}".format(e, error_body))
+
         response_body = response.read()
         json_data = json.loads(response_body.decode('utf-8'))
+        self.display.vvv("Returned Data: %s" % json.dumps(json_data, indent=4, sort_keys=True))
         if 200 <= response.status <= 299:
             return json_data
         if response.status == 404 and json_data['status'] == "No objects found.":
@@ -155,7 +180,7 @@ def get_inventory_from_icinga(self):
         """Query for all hosts """
         self.display.vvv("Querying Icinga2 for inventory")
         query_args = {
-            "attrs": ["address", "state_type", "state", "groups"],
+            "attrs": ["address", "display_name", "state_type", "state", "groups"],
         }
         if self.host_filter is not None:
             query_args['host_filter'] = self.host_filter
@@ -177,24 +202,35 @@ def _convert_inv(self, json_data):
         """Convert Icinga2 API data to JSON format for Ansible"""
         groups_dict = {"_meta": {"hostvars": {}}}
         for entry in json_data:
-            host_name = entry['name']
             host_attrs = entry['attrs']
+            if self.inventory_attr == "name":
+                host_name = entry.get('name')
+            if self.inventory_attr == "address":
+                # When looking for address for inventory, if missing fallback to object name
+                if host_attrs.get('address', '') != '':
+                    host_name = host_attrs.get('address')
+                else:
+                    host_name = entry.get('name')
+            if self.inventory_attr == "display_name":
+                host_name = host_attrs.get('display_name')
             if host_attrs['state'] == 0:
                 host_attrs['state'] = 'on'
             else:
                 host_attrs['state'] = 'off'
-            host_groups = host_attrs['groups']
-            host_addr = host_attrs['address']
-            self.inventory.add_host(host_addr)
+            host_groups = host_attrs.get('groups')
+            self.inventory.add_host(host_name)
             for group in host_groups:
                 if group not in self.inventory.groups.keys():
                     self.inventory.add_group(group)
-                self.inventory.add_child(group, host_addr)
-            self.inventory.set_variable(host_addr, 'address', host_addr)
-            self.inventory.set_variable(host_addr, 'hostname', host_name)
-            self.inventory.set_variable(host_addr, 'state',
+                self.inventory.add_child(group, host_name)
+            # If the address attribute is populated, override ansible_host with the value
+            if host_attrs.get('address') != '':
+                self.inventory.set_variable(host_name, 'ansible_host', host_attrs.get('address'))
+            self.inventory.set_variable(host_name, 'hostname', entry.get('name'))
+            self.inventory.set_variable(host_name, 'display_name', host_attrs.get('display_name'))
+            self.inventory.set_variable(host_name, 'state',
                                         host_attrs['state'])
-            self.inventory.set_variable(host_addr, 'state_type',
+            self.inventory.set_variable(host_name, 'state_type',
                                         host_attrs['state_type'])
         return groups_dict
 
@@ -211,6 +247,7 @@ def parse(self, inventory, loader, path, cache=True):
         self.icinga2_password = self.get_option('password')
         self.ssl_verify = self.get_option('validate_certs')
         self.host_filter = self.get_option('host_filter')
+        self.inventory_attr = self.get_option('inventory_attr')
         # Not currently enabled
         # self.cache_key = self.get_cache_key(path)
         # self.use_cache = cache and self.get_option('cache')


Test patch:
diff --git a/tests/unit/plugins/inventory/test_icinga2.py b/tests/unit/plugins/inventory/test_icinga2.py
--- a/tests/unit/plugins/inventory/test_icinga2.py
+++ b/tests/unit/plugins/inventory/test_icinga2.py
@@ -37,6 +37,7 @@ def query_hosts(hosts=None, attrs=None, joins=None, host_filter=None):
             'attrs': {
                 'address': 'test-host1.home.local',
                 'groups': ['home_servers', 'servers_dell'],
+                'display_name': 'Test Host 1',
                 'state': 0.0,
                 'state_type': 1.0
             },
@@ -48,6 +49,7 @@ def query_hosts(hosts=None, attrs=None, joins=None, host_filter=None):
         {
             'attrs': {
                 'address': 'test-host2.home.local',
+                'display_name': 'Test Host 2',
                 'groups': ['home_servers', 'servers_hp'],
                 'state': 1.0,
                 'state_type': 1.0
@@ -56,6 +58,19 @@ def query_hosts(hosts=None, attrs=None, joins=None, host_filter=None):
             'meta': {},
             'name': 'test-host2',
             'type': 'Host'
+        },
+        {
+            'attrs': {
+                'address': '',
+                'display_name': 'Test Host 3',
+                'groups': ['not_home_servers', 'servers_hp'],
+                'state': 1.0,
+                'state_type': 1.0
+            },
+            'joins': {},
+            'meta': {},
+            'name': 'test-host3.example.com',
+            'type': 'Host'
         }
     ]
     return json_host_data
@@ -66,6 +81,7 @@ def test_populate(inventory, mocker):
     inventory.icinga2_user = 'ansible'
     inventory.icinga2_password = 'password'
     inventory.icinga2_url = 'https://localhost:5665' + '/v1'
+    inventory.inventory_attr = "address"
 
     # bypass authentication and API fetch calls
     inventory._check_api = mocker.MagicMock(side_effect=check_api)
@@ -77,6 +93,9 @@ def test_populate(inventory, mocker):
     print(host1_info)
     host2_info = inventory.inventory.get_host('test-host2.home.local')
     print(host2_info)
+    host3_info = inventory.inventory.get_host('test-host3.example.com')
+    assert inventory.inventory.get_host('test-host3.example.com') is not None
+    print(host3_info)
 
     # check if host in the home_servers group
     assert 'home_servers' in inventory.inventory.groups
@@ -87,11 +106,29 @@ def test_populate(inventory, mocker):
     assert group1_data.hosts == group1_test_data
     # Test servers_hp group
     group2_data = inventory.inventory.groups['servers_hp']
-    group2_test_data = [host2_info]
+    group2_test_data = [host2_info, host3_info]
     print(group2_data.hosts)
     print(group2_test_data)
     assert group2_data.hosts == group2_test_data
 
-    # check if host state rules apply properyl
+    # check if host state rules apply properly
     assert host1_info.get_vars()['state'] == 'on'
+    assert host1_info.get_vars()['display_name'] == "Test Host 1"
     assert host2_info.get_vars()['state'] == 'off'
+    assert host3_info.get_vars().get('ansible_host') is None
+
+    # Confirm attribute options switcher
+    inventory.inventory_attr = "name"
+    inventory._populate()
+    assert inventory.inventory.get_host('test-host3.example.com') is not None
+    host2_info = inventory.inventory.get_host('test-host2')
+    assert host2_info is not None
+    assert host2_info.get_vars().get('ansible_host') == 'test-host2.home.local'
+
+    # Confirm attribute options switcher
+    inventory.inventory_attr = "display_name"
+    inventory._populate()
+    assert inventory.inventory.get_host('Test Host 3') is not None
+    host2_info = inventory.inventory.get_host('Test Host 2')
+    assert host2_info is not None
+    assert host2_info.get_vars().get('ansible_host') == 'test-host2.home.local'