Repository: ansible-collections/community.vmware

Issue:
Error adding custom attributes to newly created VM
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
The module *vmware_guest_custom_attributes* fails in creating the custom attributes if the VM doesn't already have them.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
*vmware_guest_custom_attributes*

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible [core 2.11.0] 
  config file = None
  configured module search path = ['/home/_/repos/my_project/ansible/plugins/modules']
  ansible python module location = /home/_/repos/my_project/venv/lib/python3.9/site-packages/ansible
  ansible collection location = /home/_/repos/my_project/ansible/collections
  executable location = /home/_/repos/my_project/venv/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True
```

##### COLLECTION VERSION
<!--- Paste verbatim output from "ansible-galaxy collection list <namespace>.<collection>"  between the quotes
for example: ansible-galaxy collection list community.general
-->
```paste below
# /home/_/repos/my_project/ansible/collections/ansible_collections
Collection        Version
----------------- -------
amazon.aws        5.1.0  
ansible.posix     1.4.0  
ansible.utils     2.8.0  
community.general 6.0.1  
community.vmware  3.2.0  
netbox.netbox     3.9.0
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
debian 11

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Create VMs on vSphere
  community.vmware.vmware_guest:
    hostname: "{{ hostname }}"
    username: "{{ username }}"
    password: "{{ password }}"
    datacenter: "{{ datacenter }}"
    name: "{{ host }}"
    esxi_hostname: "{{ esxi_hostname }}"
    folder: "{{ vm_folder }}"
    state: "{{ vm_state }}"
    annotation: "{{ vm_annotation | default(omit) }}"
    disk: "{{ disks_details }}"
    networks:
      - name: "{{ vm_network }}"
        ip: "{{ vm_ip }}"
        netmask: "{{ vm_netmask }}"
        gateway: "{{ vm_gateway }}"
    hardware:
      memory_mb: "{{ vm_ram }}"
      num_cpus: "{{ vm_vcpu }}"
      num_cpu_cores_per_socket: "{{ vm_vcpu }}"
    template: "{{ vm_template }}"
    wait_for_ip_address: yes

- name: Configure custom attributes
  community.vmware.vmware_guest_custom_attributes:
    hostname: "{{ hostname }}"
    username: "{{ username }}"
    password: "{{ password }}"
    name: "{{ host }}"
    folder: "{{ vm_folder }}"
    state: present
    attributes: "{{ vm_attributes }}"
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
Custom attributes added to the newly created VM

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
Module fails with the error below

<!--- Paste verbatim command output between quotes -->
```paste below
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'value'
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/_/.ansible/tmp/ansible-tmp-1670252175.1850512-14501-53110338757278/AnsiballZ_vmware_guest_custom_attributes.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/home/_/.ansible/tmp/ansible-tmp-1670252175.1850512-14501-53110338757278/AnsiballZ_vmware_guest_custom_attributes.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/_/.ansible/tmp/ansible-tmp-1670252175.1850512-14501-53110338757278/AnsiballZ_vmware_guest_custom_attributes.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.vmware.plugins.modules.vmware_guest_custom_attributes', init_globals=dict(_module_fqn='ansible_collections.community.vmware.plugins.modules.vmware_guest_custom_attributes', _modlib_path=modlib_path),\n  File \"/usr/lib/python3.9/runpy.py\", line 210, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_community.vmware.vmware_guest_custom_attributes_payload_62z6z4_f/ansible_community.vmware.vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py\", line 358, in <module>\n  File \"/tmp/ansible_community.vmware.vmware_guest_custom_attributes_payload_62z6z4_f/ansible_community.vmware.vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py\", line 346, in main\n  File \"/tmp/ansible_community.vmware.vmware_guest_custom_attributes_payload_62z6z4_f/ansible_community.vmware.vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py\", line 189, in set_custom_field\n  File \"/tmp/ansible_community.vmware.vmware_guest_custom_attributes_payload_62z6z4_f/ansible_community.vmware.vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py\", line 271, in check_exists\nKeyError: 'value'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
```


The problem seems related to the fact that the *vm.customValue* attribute is empty after creation, so the elements of the *existing_custom_attributes* in the function *check_exists* in the *vmware_guest_custom_attributes.py* are without the *value* field. It should be enough to add an empty value field when creating the *existing_custom_attributes* and eventually overwrite it after. I'm going to create a pull request to address this issue. 



Discussion:
I just spent the day troubleshooting this same error. Brand new VM with no existing attributes set fails with the error specified here, but if the VM had an attribute set at some point, it runs without error.
While waiting for the fix, downgrading to version 3.1.0 solves the issue.
> While waiting for the fix, downgrading to version 3.1.0 solves the issue.

That may be, but also be aware that the python module pyvmomi 8.0.0.1 is only compatible with this collection 3.2.0 
@sky-joker Do you think you could have a look at this? Since the issue appears in 3.2.0 it looks like it is related to your changes in #1532. But I wasn't able to reproduce the issue in our environment :-/

Maybe you see something I don't. I'm also confused since the integration tests look very similar to the "steps to reproduce" (create a new VM and add a custom attribute) but don't fail in the CI pipeline:

https://github.com/ansible-collections/community.vmware/blob/8ce49d9952fb2979b0ba2c5420f15caccd65b80c/tests/integration/targets/vmware_guest_custom_attributes/tasks/main.yml#L5-L36

@tom-ph @MallocArray FYI Since 3.2.0 there's also [vmware_custom_attribute_manager](https://github.com/ansible-collections/community.vmware/blob/58fe030cfe072e978d0e0d1c5fb064c09309ce4a/docs/community.vmware.vmware_custom_attribute_manager_module.rst) that can be used to manage custom attributes on vSphere objects including VMs. But beware of an important difference: I contrast to `vmware_guest_custom_attributes` this new module will not silently create the custom attribute if it doesn't exist yet. It has to exist already, but there's [vmware_custom_attribute](https://github.com/ansible-collections/community.vmware/blob/58fe030cfe072e978d0e0d1c5fb064c09309ce4a/docs/community.vmware.vmware_custom_attribute_module.rst) (also new in 3.2.0) to do this. Maybe that's an alternative to downgrading to 3.1.0.
I also ran into this bug. Maybe I can add some info as to why it fails for me but not in the CI tests.

The VM inherits empty values for some preset custom attributes from it's template and when the values are supposed to be assigned they fail since they don't appear to be added as a key. Making the value non-blank, e.g. adding a space, does work as a workaround. This isn't feasible for all of them but works for testing. I do hope this gets fixed.

Resolving patch:
diff --git a/plugins/modules/vmware_guest_custom_attributes.py b/plugins/modules/vmware_guest_custom_attributes.py
--- a/plugins/modules/vmware_guest_custom_attributes.py
+++ b/plugins/modules/vmware_guest_custom_attributes.py
@@ -254,15 +254,15 @@ def check_exists(self, vm, user_fields):
             })
 
         # Gather the values of set the custom attribute.
-        for v in vm.customValue:
-            for e in existing_custom_attributes:
+        for e in existing_custom_attributes:
+            for v in vm.customValue:
                 if e['key'] == v.key:
                     e['value'] = v.value
 
-                # When add custom attribute as a new one, it has not the value key.
-                # Add the value key to avoid unintended behavior in the difference check.
-                if 'value' not in e:
-                    e['value'] = ''
+            # When add custom attribute as a new one, it has not the value key.
+            # Add the value key to avoid unintended behavior in the difference check.
+            if 'value' not in e:
+                e['value'] = ''
 
         # Select the custom attribute and value to update the configuration.
         _user_fields_for_diff = []


Test patch: