Repository: ansible-collections/community.general

Issue:
community.general.proxmox plugin: Missing support for templating in inventory file
### Summary

When using the proxmox inventory plugin, it crashed with the error mentioned in the title.

### Issue Type

Bug Report

### Component Name

proxmox

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.1]
  config file = /home/robin/Checkmk/Ansible/kubernetes/ansible.cfg
  configured module search path = ['/home/robin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  ansible collection location = /home/robin/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
  jinja version = 3.0.3
  libyaml = True
```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general
# /usr/local/lib/python3.8/dist-packages/ansible_collections
Collection        Version
----------------- -------
community.general 4.0.2  

# /home/$USER/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 4.2.0
```

### Configuration

```console (paste below)
$ ansible-config dump --only-changed
DEFAULT_GATHERING(/home/$USER/$PROJECT/ansible.cfg) = smart
DEFAULT_MANAGED_STR(/home/$USER/$PROJECT/ansible.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_ROLES_PATH(/home/$USER/$PROJECT/ansible.cfg) = ['/home/$USER/$PROJECT/roles']
INVENTORY_ENABLED(/home/$USER/$PROJECT/ansible.cfg) = ['community.general.proxmox']
SHOW_CUSTOM_STATS(/home/$USER/$PROJECT/ansible.cfg) = True
```

### OS / Environment

- Ubuntu 20.04 on control host
- Proxmox 7.1-8 not clustered on remote end

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
plugin: community.general.proxmox
url: "https://$PROXMOX:8006"
user: "{{ lookup('env','PM_USER') }}"
password: "{{ lookup('env','PM_PASS') }}"
validate_certs: true
want_facts: true
```
Run `ansible-inventory -i inventory/dev.proxmox.yml --graph`.

### Expected Results

A normal inventory output should be provided.

### Actual Results

```console (paste below)
$ ansible-inventory -i inventory/dev.proxmox.yml --graph -vvv 
ansible-inventory [core 2.12.1]
  config file = /home/$USER/$PROJECT/ansible.cfg
  configured module search path = ['/home/$USER/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  ansible collection location = /home/$USER/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
  jinja version = 3.0.3
  libyaml = True
Using /home/$USER/$PROJECT/ansible.cfg as config file
[WARNING]:  * Failed to parse /home/$USER/$PROJECT/inventory/dev.proxmox.yml with
ansible_collections.community.general.plugins.inventory.proxmox plugin: 'NoneType' object is not
subscriptable
  File "/usr/local/lib/python3.8/dist-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/$USER/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 478, in parse
    self._populate()
  File "/home/$USER/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 364, in _populate
    self._get_auth()
  File "/home/$USER/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 179, in _get_auth
    'ticket': json['data']['ticket'],
[WARNING]: Unable to parse /home/$USER/$PROJECT/inventory/dev.proxmox.yml as an
inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* `plugins/modules/cloud/misc/proxmox`

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Ajpantuso @Thulium-Drake @joshainglis @karmab @tleguern
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
!component =plugins/inventory/proxmox.py
Files identified in the description:
* [`plugins/inventory/proxmox.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/proxmox.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @ilijamt
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
You probably have to create API Token in Proxmox Datacenter settings
btw this exception should be caught and the right error message displayed, generic NoneType exception is confusing.
That would mean, that password authentication is not supported?
Then the docs should state that I think.
Did you check the permissions of the API user? I have the following permission (in ```/etc/pve/user.cfg```)

```
user:search@pve:1:0::::::

acl:1:/:search@pve:PVEAuditor:
```

Make sure that propagation is set.
Thanks for your patience, I was caught up in other projects.
@Thulium-Drake: Well I can provision VMs using terraform in the Proxmox environment, so I should be able to pull an inventory, right? Or does this plugin only work, when it can see everything in the cluster?
Nope, the account I use on my home system has no read permissions on specific test systems I don't want to control through Ansible :-)

Looking at the config, did you try setting the credentials normally (you can protect them using Ansible Vault)
I was afraid, you would say something like that. ;)

I will fiddle around some more, but time is rare. Will keep you guys posted in here.

Thanks for the hints nevertheless!
Son of a ... nice old lady!
Just tried to put the credentials directly into the file as @Thulium-Drake recommended, and it works.
As the lookup is performed as outlined in documentation, I am unsure if this is an issue with the Proxmox module, or with Ansible itself?
My guess that it has something to do with Ansible's way of working here. I've actually never tried using lookups in the inventory (outside of the vars files ofcourse), but I do know that Ansible Vault looks like some sort of macro, so maybe it's behavior is different (as it does work like a charm)
Do you have any idea, how or where to report this appropriately?
Because I see the other methods of doing this, but for me the environment is the most suitable way to do this.
Not entirely sure, maybe @felixfontein can help point in the right direction :-)
Inventory files are not templated by Ansible. If an inventory plugin wants to template it, or parts of it, it has to do so manually. The linode inventory plugin for example allows to template the access token: https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/linode.py#L83-L87

The implementation is in https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/linode.py#L150-L151
Okay, so it sounds to me, like my use case would be a feature request towards this collection, right?
How should we move forward? Keeping this issue open? Creating a new one as a dedicated feature request?
I am no developer, so I can only provide information and test, but I would love to see this feature happen.
I'd keep this issue open, but adjust the title. There's no need to create another issue IMO :)

Implementing this shouldn't be hard (since there's an example of how it can be done). Maybe @Thulium-Drake has some spare time for this ;-) One question is which fields to allow templating in. `user`, `password` and `url` might be good choices here.
I had some time, so here is a PR for this.
@robin-tribe29 Could you give the PR a spin? I don't have a setup that I can use right now :-) and yours should be good to go ;-)
Did not realize, that I could do that, sorry. :sweat_smile: 
I have no idea what I am doing, but I copied the contents of @ilijamt's fork to the installation directory of the collection, and it is looking good! :tada:
```
$ ansible-inventory -i inventory/dev.proxmox.yml --graph    
@all:
  |--@proxmox_all_lxc:
  |--@proxmox_all_qemu:
  |--@proxmox_all_running:
  |--@proxmox_all_stopped:
  |--@proxmox_nodes:
  |  |--node-001
  |--@proxmox_node_001_lxc:
  |--@proxmox_node_001_qemu:
  |--@ungrouped:
```

Resolving patch:
diff --git a/plugins/inventory/proxmox.py b/plugins/inventory/proxmox.py
--- a/plugins/inventory/proxmox.py
+++ b/plugins/inventory/proxmox.py
@@ -31,6 +31,7 @@
         description:
           - URL to Proxmox cluster.
           - If the value is not specified in the inventory configuration, the value of environment variable C(PROXMOX_URL) will be used instead.
+          - Since community.general 4.7.0 you can also use templating to specify the value of the I(url).
         default: 'http://localhost:8006'
         type: str
         env:
@@ -40,6 +41,7 @@
         description:
           - Proxmox authentication user.
           - If the value is not specified in the inventory configuration, the value of environment variable C(PROXMOX_USER) will be used instead.
+          - Since community.general 4.7.0 you can also use templating to specify the value of the I(user).
         required: yes
         type: str
         env:
@@ -49,6 +51,7 @@
         description:
           - Proxmox authentication password.
           - If the value is not specified in the inventory configuration, the value of environment variable C(PROXMOX_PASSWORD) will be used instead.
+          - Since community.general 4.7.0 you can also use templating to specify the value of the I(password).
         required: yes
         type: str
         env:
@@ -136,6 +139,14 @@
   my_inv_var_1: "'my_var1_value'"
   my_inv_var_2: >
     "my_var_2_value"
+
+# Specify the url, user and password using templating
+# my.proxmox.yml
+plugin: community.general.proxmox
+url: "{{ lookup('ansible.builtin.ini', 'url', section='proxmox', file='file.ini') }}"
+user: "{{ lookup('ansible.builtin.env','PM_USER') | default('ansible@pve') }}"
+password: "{{ lookup('community.general.random_string', base64=True) }}"
+
 '''
 
 import itertools
@@ -148,6 +159,7 @@
 from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.six.moves.urllib.parse import urlencode
 from ansible.utils.display import Display
+from ansible.template import Templar
 
 from ansible_collections.community.general.plugins.module_utils.version import LooseVersion
 
@@ -500,10 +512,24 @@ def parse(self, inventory, loader, path, cache=True):
         # read config from file, this sets 'options'
         self._read_config_data(path)
 
+        t = Templar(loader=loader)
+
         # read options
-        self.proxmox_url = self.get_option('url').rstrip('/')
-        self.proxmox_user = self.get_option('user')
-        self.proxmox_password = self.get_option('password')
+        proxmox_url = self.get_option('url')
+        if t.is_template(proxmox_url):
+            proxmox_url = t.template(variable=proxmox_url, disable_lookups=False)
+        self.proxmox_url = proxmox_url.rstrip('/')
+
+        proxmox_user = self.get_option('user')
+        if t.is_template(proxmox_user):
+            proxmox_user = t.template(variable=proxmox_user, disable_lookups=False)
+        self.proxmox_user = proxmox_user
+
+        proxmox_password = self.get_option('password')
+        if t.is_template(proxmox_password):
+            proxmox_password = t.template(variable=proxmox_password, disable_lookups=False)
+        self.proxmox_password = proxmox_password
+
         self.cache_key = self.get_cache_key(path)
         self.use_cache = cache and self.get_option('cache')
         self.host_filters = self.get_option('filters')


Test patch: