Repository: ansible-collections/community.vmware

Issue:
vmware_cluster_(ha|drs) - Addition of Advanced Options to HA vCenter cluster throws errors due to incompatible types
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
Unable to add Advanced Options to vCenter HA Enabled cluster due to incompatible types being sent to the API

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
vmware_cluster_ha
##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
None
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
**OS**: Photon OS
**vCenter version**: 6.7

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Simply copy/paste the example of vmware_cluster_ha module documentation, fill in the correct values and make sure you are using the collection, not the official module from version 2.9
<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Enable HA without admission control
  vmware_cluster_ha:
    hostname: '{{ vcenter_hostname }}'
    username: '{{ vcenter_username }}'
    password: '{{ vcenter_password }}'
    datacenter_name: '{{ datacenter }}'
    cluster_name: '{{ cluster_name }}'
    enable_ha: yes
    validate_certs: False
    advanced_settings:
      'das.includeFTcomplianceChecks': 'false'
  delegate_to: localhost
  tags:
    - vcenter_configuration
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
```
{
    "changed": true,
    "result": null,
    "invocation": {
        "module_args": {
            "hostname": "vcenter_host",
            "username": "vcenter_user",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "datacenter_name": "datacenter_name",
            "cluster_name": "cluster_name",
            "enable_ha": true,
            "validate_certs": false,
            "advanced_settings": {
                "das.includeFTcomplianceChecks": "false"
            },
            "datacenter": "datacenter_name",
            "port": 443,
            "ha_host_monitoring": "enabled",
            "host_isolation_response": "none",
            "ha_vm_monitoring": "vmMonitoringDisabled",
            "ha_vm_failure_interval": 30,
            "ha_vm_min_up_time": 120,
            "ha_vm_max_failures": 3,
            "ha_vm_max_failure_window": -1,
            "ha_restart_priority": "medium",
            "proxy_host": null,
            "proxy_port": null,
            "slot_based_admission_control": null,
            "reservation_based_admission_control": null,
            "failover_host_admission_control": null
        }
    },
    "_ansible_no_log": false,
    "_ansible_delegated_vars": {
        "ansible_host": "localhost"
    }
}
```
##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
```paste below
{
    "msg": "('The request refers to an unexpected or unknown type.', None)",
    "exception": "  File \"/tmp/ansible_community.vmware.vmware_cluster_ha_payload_brhahxm0/ansible_community.vmware.vmware_cluster_ha_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_cluster_ha.py\", line 409, in configure_ha\n  File \"/tmp/ansible_community.vmware.vmware_cluster_ha_payload_brhahxm0/ansible_community.vmware.vmware_cluster_ha_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py\", line 80, in wait_for_task\n    raise_from(TaskError(error_msg, host_thumbprint), task.info.error)\n  File \"<string>\", line 3, in raise_from\n",
    "invocation": {
        "module_args": {
            "hostname": "vcenter_host",
            "username": "vcenter_user",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "datacenter_name": "datacenter_name",
            "cluster_name": "cluster_name",
            "enable_ha": true,
            "validate_certs": false,
            "advanced_settings": {
                "das.includeFTcomplianceChecks": "no"
            },
            "datacenter": "datacenter_name",
            "port": 443,
            "ha_host_monitoring": "enabled",
            "host_isolation_response": "none",
            "ha_vm_monitoring": "vmMonitoringDisabled",
            "ha_vm_failure_interval": 30,
            "ha_vm_min_up_time": 120,
            "ha_vm_max_failures": 3,
            "ha_vm_max_failure_window": -1,
            "ha_restart_priority": "medium",
            "proxy_host": null,
            "proxy_port": null,
            "slot_based_admission_control": null,
            "reservation_based_admission_control": null,
            "failover_host_admission_control": null
        }
    },
    "_ansible_no_log": false,
    "changed": false,
    "_ansible_delegated_vars": {
        "ansible_host": "localhost"
    }
}
```



Discussion:
The problem comes from https://github.com/ansible-collections/vmware/blob/main/plugins/module_utils/vmware.py#L864.
It tries to parse it to boolean type, but the API only accepts String values.

We have monkey patched it the following way in order to suit our case:
```
for option_key, option_value in options.items():
    if self.is_boolean(option_value):
        option_value = VmomiSupport.vmodlTypes['string'](option_value)
```

I.e. since we are only passing boolean values, we only changed this line, but it will probably throw the same error for other types different than strings.
@Akasurde @goneri  may you have a look at it, please?
> The problem comes from https://github.com/ansible-collections/vmware/blob/main/plugins/module_utils/vmware.py#L864.
> It tries to parse it to boolean type, but the API only accepts String values.
> 
> We have monkey patched it the following way in order to suit our case:
> 
> ```
> for option_key, option_value in options.items():
>     if self.is_boolean(option_value):
>         option_value = VmomiSupport.vmodlTypes['string'](option_value)
> ```

The problem I see here is that if someone wants to set `das.includeFTcomplianceChecks` that expects a string `false`, and also `das.someOtherSetting` that expects a boolean `false` your solution would set a string value although `das.someOtherSetting` requires a bool. I can't think of a solution for this at the moment.

For ESXi hosts, we can query the API for supported advanced settings including their type. Unfortunately, it looks like we can't do this for advanced cluster settings... at least, I haven’t found a way to do this.

I've more or less stolen this from vmware_host_config_manager (ansible/ansible#62801 and ansible/ansible#65675) in the hope it would just work. Obviously, I was wrong :-(

For the record: I don't mention this to blame the authors of vmware_host_config_manager (I don't know about any problems with this module, so the code works well there), it's just that we lost a lot of commit history when these modules were migrated to a collection. And linking to the ansible PRs makes it easier for people to understand how and why this code was written... if nothing else, it will make it easier for _me_ to understand what I did and why ^^
@aleksandar-kinanov @mariolenz Do you think we can add an extra parameter to override boolean check-in `option_diff` function? 

From VMware documentation - https://pubs.vmware.com/vi3/sdk/ReferenceGuide/vim.option.OptionValue.html -

```
The value of the option. The Any data object type enables you to define any value for the option. Typically, however, the value of an option is of type String or Integer. 
```

Let me know.

> Do you think we can add an extra parameter to override boolean check-in `option_diff` function?

The problem is: How do we distinguish between advanced settings that require a string `true` / `false` and those that require a boolean `true` / `false`? Anyway, I think we should give it a try and hope HA advanced settings always work with strings and never with booleans. I'll work on a PR and then we'll just see what happens, OK?

Personally, I think this is a case of bad API design. If it's a question of `true` or `false`, why use strings instead of booleans? Feel free to kick VMware for this... and if they complain, tell me and I'll kick 'em again.
@mariolenz Are you working on this?

Resolving patch:
diff --git a/plugins/module_utils/vmware.py b/plugins/module_utils/vmware.py
--- a/plugins/module_utils/vmware.py
+++ b/plugins/module_utils/vmware.py
@@ -855,14 +855,16 @@ def is_truthy(value):
 
 # options is the dict as defined in the module parameters, current_options is
 # the list of the currently set options as returned by the vSphere API.
-def option_diff(options, current_options):
+# When truthy_strings_as_bool is True, strings like 'true', 'off' or 'yes'
+# are converted to booleans.
+def option_diff(options, current_options, truthy_strings_as_bool=True):
     current_options_dict = {}
     for option in current_options:
         current_options_dict[option.key] = option.value
 
     change_option_list = []
     for option_key, option_value in options.items():
-        if is_boolean(option_value):
+        if truthy_strings_as_bool and is_boolean(option_value):
             option_value = VmomiSupport.vmodlTypes['bool'](is_truthy(option_value))
         elif isinstance(option_value, int):
             option_value = VmomiSupport.vmodlTypes['int'](option_value)
diff --git a/plugins/modules/vmware_cluster_ha.py b/plugins/modules/vmware_cluster_ha.py
--- a/plugins/modules/vmware_cluster_ha.py
+++ b/plugins/modules/vmware_cluster_ha.py
@@ -266,7 +266,7 @@ def __init__(self, module):
 
         self.advanced_settings = self.params.get('advanced_settings')
         if self.advanced_settings:
-            self.changed_advanced_settings = option_diff(self.advanced_settings, self.cluster.configurationEx.dasConfig.option)
+            self.changed_advanced_settings = option_diff(self.advanced_settings, self.cluster.configurationEx.dasConfig.option, False)
         else:
             self.changed_advanced_settings = None
 


Test patch:
diff --git a/tests/integration/targets/vmware_cluster_ha/tasks/main.yml b/tests/integration/targets/vmware_cluster_ha/tasks/main.yml
--- a/tests/integration/targets/vmware_cluster_ha/tasks/main.yml
+++ b/tests/integration/targets/vmware_cluster_ha/tasks/main.yml
@@ -162,6 +162,39 @@
       that:
         - not change_num_heartbeat_ds_again.changed
 
+  - name: Change advanced setting "das.includeFTcomplianceChecks" (check-mode)
+    vmware_cluster_ha: &change_includeFTcomplianceChecks
+      validate_certs: False
+      hostname: "{{ vcenter_hostname }}"
+      username: "{{ vcenter_username }}"
+      password: "{{ vcenter_password }}"
+      datacenter_name: "{{ dc1 }}"
+      cluster_name: test_cluster_ha
+      advanced_settings:
+          'das.includeFTcomplianceChecks': 'false'
+    check_mode: yes
+    register: change_includeFTcomplianceChecks_check
+
+  - assert:
+      that:
+        - change_includeFTcomplianceChecks_check.changed
+
+  - name: Change advanced setting "das.includeFTcomplianceChecks"
+    vmware_cluster_ha: *change_includeFTcomplianceChecks
+    register: change_includeFTcomplianceChecks
+
+  - assert:
+      that:
+        - change_includeFTcomplianceChecks.changed
+
+  - name: Change advanced setting "das.includeFTcomplianceChecks" again
+    vmware_cluster_ha: *change_includeFTcomplianceChecks
+    register: change_includeFTcomplianceChecks_again
+
+  - assert:
+      that:
+        - not change_includeFTcomplianceChecks_again.changed
+
   # Delete test cluster
   - name: Delete test cluster
     vmware_cluster:
diff --git a/tests/integration/targets/vmware_host_config_manager/tasks/main.yml b/tests/integration/targets/vmware_host_config_manager/tasks/main.yml
--- a/tests/integration/targets/vmware_host_config_manager/tasks/main.yml
+++ b/tests/integration/targets/vmware_host_config_manager/tasks/main.yml
@@ -88,3 +88,39 @@
     assert:
       that:
           - all_hosts_result_check_mode.changed
+
+  # Test that PR 332 doesn't break boolean settings for this module.
+  - name: Change a boolean setting for a given host (check-mode)
+    vmware_host_config_manager: &change_logDirUnique
+      hostname: "{{ vcenter_hostname }}"
+      username: "{{ vcenter_username }}"
+      password: "{{ vcenter_password }}"
+      esxi_hostname: '{{ esxi1 }}'
+      options:
+        'Syslog.global.logDirUnique': true
+      validate_certs: no
+    check_mode: yes
+    register: change_logDirUnique_check
+
+  - name: ensure changes would be done to given host
+    assert:
+      that:
+          - change_logDirUnique_check.changed
+
+  - name: Change a boolean setting for a given host
+    vmware_host_config_manager: *change_logDirUnique
+    register: change_logDirUnique
+
+  - name: ensure changes are done to given host
+    assert:
+      that:
+          - change_logDirUnique.changed
+
+  - name: Change a boolean setting for a given host again
+    vmware_host_config_manager: *change_logDirUnique
+    register: change_logDirUnique_again
+
+  - name: ensure changes are not done to given host
+    assert:
+      that:
+          - not change_logDirUnique_again.changed