Repository: ansible-collections/community.vmware

Issue:
vmware_vmkernel has inconsistent behavior with multiple vSAN interfaces
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
When using vmware_vmkernel to configure several vmk interfaces in a loop, it is extremely hard to keep a consistent set of configuration across runs :
- It is not possible to define two interfaces (say, vmk0 or vmk3) with `enable_vsan: True`, only the last one (vmk3) defined is reflected 
- Not specifying the argument enable_vsan results in `enable_vsan: False` being processed, which means upon next run even if vmk0 is manually set as a vSAN interface, the flag will be removed.

This can be extremely problematic for migrations where one would need to have multiple vSAN interfaces active at the same time.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
vmware_vmkernel

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/users/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python/3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.3 (default, Dec  5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /home/users/ansible/.vault_pass
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Red Hqt Enterprise Linux release 8.2 (Ootpa)
Attempted on a ESXi 7.0.2 host

I am in an air-gapped environment and stuck with this specific version of Ansible, unable to use collections.

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: "Configure vmkernel port {{ item.device }}"
  vmware_vmkernel:
    hostname: "{{ esxi_hostname }}"
    username: "{{ esxi_username }}"
    esxi_hostname: "{{ esxi_hostname }}"
    password: "{{ esxi_password }}"
    vswitch_name: "{{ item.vswitch_name }}"
    portgroup_name: "{{ item.portgroup_name }}"
    network: "{{ item.network }}"
    device: "{{ item.device }}"
    state: present
    enable_mgmt: "{{ item.enable_mgmt|default(omit) }}"
    enable_vsan: "{{ item.enable_vsan|default(omit) }}"
  delegate_to: localhost
  with_items:
    - { vswitch_name: "vSwitch0", portgroup_name: "Management Network", network: { type: "static", ip_address: "10.0.0.1", subnet_mask: "255.255.255.0" }, device: "vmk0", enable_mgmt: True, enable_vsan: True }
    - { vswitch_name: "vSwitch1", portgroup_name: "vSAN", network: { type: "static", ip_address: "192.168.0.1", subnet_mask: "255.255.255.0" }, device: "vmk3", enable_vsan: True }
  tags:
    - esxi_vmk_interfaces
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
After running the playbook I would expect the following command on the host to return this result, having both interfaces enabled as vSAN.
```
[root@esxi_host:~] esxcli vsan network list | grep VmkNic
   VmkNic Name: vmk0
   VmkNic Name: vmk3
```

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
The module itself executes successfully (no Ansible error), but :
- With no interfaces configured
```
$ ansible-playbook -i development ./provision.yml -v -t esxi_vmk_interfaces -l esxi_host
PLAY [vsan]

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk0]
changed: [esxi_host -> localhost] => {"changed": true, "device": "vmk0", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "10.0.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter services updated", "mtu": 1500, "portgroup": "Management Network", "services": "Mgmt, vSAN", "services_previous": "Mgmt", "switch": "vSwitch0", "tcpip_stack": "default", "vsan": null}

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk3]
changed: [esxi_host -> localhost] => {"changed": false, "device": "vmk3", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "192.168.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter already configured properly", "mtu": 1500, "portgroup": "vSAN", "services": "VSAN", "switch": "vSwitch1", "tcpip_stack": "default", "vsan": null}
```
- Running again (with vmk0 configured as per above)
```
$ ansible-playbook -i development ./provision.yml -v -t esxi_vmk_interfaces -l esxi_host
PLAY [vsan]

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk0]
changed: [esxi_host -> localhost] => {"changed": false, "device": "vmk0", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "10.0.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter already configured properly", "mtu": 1500, "portgroup": "Management Network", "services": "Mgmt,VSAN", "switch": "vSwitch0", "tcpip_stack": "default", "vsan": null}

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk3]
changed: [esxi_host -> localhost] => {"changed": true, "device": "vmk3", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "192.168.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter services updated", "mtu": 1500, "portgroup": "vSAN", "services": "VSAN", "services_previous": "", "switch": "vSwitch1", "tcpip_stack": "default", "vsan": null}
```
- Running again (with vmk3 configured as per above)
```
$ ansible-playbook -i development ./provision.yml -v -t esxi_vmk_interfaces -l esxi_host
PLAY [vsan]

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk0]
changed: [esxi_host -> localhost] => {"changed": true, "device": "vmk0", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "10.0.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter services updated", "mtu": 1500, "portgroup": "Management Network", "services": "Mgmt, VSAN", "services_previous": "Mgmt", "switch": "vSwitch0", "tcpip_stack": "default", "vsan": null}

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk3]
changed: [esxi_host -> localhost] => {"changed": false, "device": "vmk3", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "192.168.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter already configured properly", "mtu": 1500, "portgroup": "vSAN", "services": "VSAN", "switch": "vSwitch1", "tcpip_stack": "default", "vsan": null}
```
- Running again (with vmk0 configured as per above)
```
$ ansible-playbook -i development ./provision.yml -v -t esxi_vmk_interfaces -l esxi_host
PLAY [vsan]

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk0]
changed: [esxi_host -> localhost] => {"changed": false, "device": "vmk0", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "10.0.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter already configured properly", "mtu": 1500, "portgroup": "Management Network", "services": "Mgmt, VSAN", "switch": "vSwitch0", "tcpip_stack": "default", "vsan": null}

TASK [esxi_vmk_intrefaces : Configure vmkernel port vmk3]
changed: [esxi_host -> localhost] => {"changed": true, "device": "vmk3", "ipv4": "static", "ipv4_gw": "No override", "ipv4_ip": "192.168.0.1", "ipv4_sm": "255.255.255.0", "msg": "VMkernel Adapter services updated", "mtu": 1500, "portgroup": "vSAN", "services": "VSAN", "services_previous": "", "switch": "vSwitch1", "tcpip_stack": "default", "vsan": null}
```

When confirming the result on the ESXi host itself, it becomes obvious that enabling vSAN on one interface disables vSAN on the others :
```
[root@esxi_host:~] esxcli vsan network list | grep VmkNic
   VmkNic Name: vmk3
```

In a live vSAN cluster, this has the result of splitting the cluster since esxi_host's vSAN service is now unable to talk to other hosts. I have to manually recover by running `esxcli vsan network ip add -i vmk0`.

It should also be noted execution is not consistent across runs of identical configuration because it only checks the "current" status of service once in the loop before applying it for all interfaces.

I am trying to work around by setting a default dominant interface which I expect the module to set up for me, and running commands over SSH via the shell module to enforce the vSAN flag for other interfaces I need, however his leads to another bug (line 758 where local variable operation is referenced before assignment because that bit of code never initializes it), making it impossible to manually override the behavior of the module for vSAN services, since it will insist on touching the interface.

It would be nice if : 
- the behavior could be made consistent
- the module could be used merely for creating a vmk interface, and letting another task configure the services (as of now, I understand it works in a all-in-one-fell-swoop fashion)

Looking at the code, I assume it is because the `set_vsan_service_type` method crafts a new port configuration while ignoring any potentially previously existing configuration in the vSAN system, which conveniently also "removed" interfaces.

The code on line 758 seems to be pasted from other services, but does not do proper removal : the "operation" variable is never defined, so when it reaches this line (should it need to remove the flag from an interface that had it), it fails with an "unassigned variable" error. Replacing "operation=operation" with "operation='deselect'" does not work, it fails with a "Cannot change the host configuration" error, indicating this is not the proper way to proceed.

The proper way to proceed would likely to make set_vsan_service_type generic, capable of handling both adding/removal of interfaces.


Discussion:
Files identified in the description:
* [`plugins/modules/vmware_vmkernel.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/vmware_vmkernel.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Akasurde @Tomorrow9 @goneri @lparkes @pgbidkar @warthog9
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/plugins/modules/vmware_vmkernel.py b/plugins/modules/vmware_vmkernel.py
--- a/plugins/modules/vmware_vmkernel.py
+++ b/plugins/modules/vmware_vmkernel.py
@@ -751,12 +751,7 @@ def host_vmk_update(self):
                 if changed_service_vsan:
                     if self.vnic.device in service_type_vmks['vsan']:
                         services_previous.append('VSAN')
-                    if self.enable_vsan:
-                        results['vsan'] = self.set_vsan_service_type()
-                    else:
-                        self.set_service_type(
-                            vnic_manager=vnic_manager, vmk=self.vnic, service_type='vsan', operation=operation
-                        )
+                    results['vsan'] = self.set_vsan_service_type(self.enable_vsan)
 
                 results['services_previous'] = ', '.join(services_previous)
         else:
@@ -792,7 +787,7 @@ def find_dvspg_by_key(self, dv_switch, portgroup_key):
 
         return None
 
-    def set_vsan_service_type(self):
+    def set_vsan_service_type(self, enable_vsan):
         """
         Set VSAN service type
         Returns: result of UpdateVsan_Task
@@ -801,20 +796,43 @@ def set_vsan_service_type(self):
         result = None
         vsan_system = self.esxi_host_obj.configManager.vsanSystem
 
-        vsan_port_config = vim.vsan.host.ConfigInfo.NetworkInfo.PortConfig()
-        vsan_port_config.device = self.vnic.device
-
+        vsan_system_config = vsan_system.config
         vsan_config = vim.vsan.host.ConfigInfo()
-        vsan_config.networkInfo = vim.vsan.host.ConfigInfo.NetworkInfo()
-        vsan_config.networkInfo.port = [vsan_port_config]
-        if not self.module.check_mode:
+
+        vsan_config.networkInfo = vsan_system_config.networkInfo
+        current_vsan_vnics = [portConfig.device for portConfig in vsan_system_config.networkInfo.port]
+        changed = False
+        result = "%s NIC %s (currently enabled NICs: %s) : " % ("Enable" if enable_vsan else "Disable", self.vnic.device, current_vsan_vnics)
+        if not enable_vsan:
+            if self.vnic.device in current_vsan_vnics:
+                vsan_config.networkInfo.port = list(filter(lambda portConfig: portConfig.device != self.vnic.device, vsan_config.networkInfo.port))
+                changed = True
+        else:
+            if self.vnic.device not in current_vsan_vnics:
+                vsan_port_config = vim.vsan.host.ConfigInfo.NetworkInfo.PortConfig()
+                vsan_port_config.device = self.vnic.device
+
+                if vsan_config.networkInfo is None:
+                    vsan_config.networkInfo = vim.vsan.host.ConfigInfo.NetworkInfo()
+                    vsan_config.networkInfo.port = [vsan_port_config]
+                else:
+                    vsan_config.networkInfo.port.append(vsan_port_config)
+                changed = True
+
+        if not self.module.check_mode and changed:
             try:
                 vsan_task = vsan_system.UpdateVsan_Task(vsan_config)
-                wait_for_task(vsan_task)
+                task_result = wait_for_task(vsan_task)
+                if task_result[0]:
+                    result += "Success"
+                else:
+                    result += "Failed"
             except TaskError as task_err:
                 self.module.fail_json(
                     msg="Failed to set service type to vsan for %s : %s" % (self.vnic.device, to_native(task_err))
                 )
+        if self.module.check_mode:
+            result += "Dry-run"
         return result
 
     def host_vmk_create(self):
@@ -909,7 +927,7 @@ def host_vmk_create(self):
 
             # VSAN
             if self.enable_vsan:
-                results['vsan'] = self.set_vsan_service_type()
+                results['vsan'] = self.set_vsan_service_type(self.enable_vsan)
 
             # Other service type
             host_vnic_manager = self.esxi_host_obj.configManager.virtualNicManager


Test patch: