Repository: ansible/ansible

Issue:
vmware_guest - Cluster option stopped working
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
vmware_guest

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
ansible 2.4.0.0
  config file = /Users/xxxx/.ansible.cfg
  configured module search path = [u'/Users/xxxx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Python/2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.10 (default, Feb  7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = [u'/Users/xxx/workspace/devops/platform-roles', u'/Users
HOST_KEY_CHECKING(/Users/xxx/.ansible.cfg) = False
RETRY_FILES_ENABLED(/Users/xxx/.ansible.cfg) = False
##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Darwin xxxxx.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
##### SUMMARY
<!--- Explain the problem briefly -->
vmware_guest option "Cluster" does not seem to be returning the correct hosts in the specified Cluster. Using esxi-hostname to a specific host does work.

This used to work and may have broken in 2.4

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```
- name: "Create DHCP VM {{ inventory_hostname }}"
  vmware_guest:
    hostname: "{{ vc_hostname }}"
    username: "{{ vc_user }}"
    password: "{{ vc_pass }}"
    cluster: "{{ vc_cluster }}"
    datacenter: "{{ vc_datacenter }}"
    template: "{{ vm_template }}"
    name: "{{ inventory_hostname }}"
    networks:
      - vlan: "{{ vm_vlan }}"
        type: "dhcp"
    annotation: "{{ vm_notes }}"
    folder: "{{ vc_folder }}"
    disk:
      - size_gb: 30
        type: thin
        datastore: "{{ vm_datastore }}"
    hardware:
      memory_mb: "{{ vm_memory }}"
      num_cpus: "{{ vm_cpu }}"
    customization:
      hostname: "{{ inventory_hostname }}"  
      domain: "{{ vm_domain }}"
    customvalues:
      - { key: "Created By", value: "Ansible" } 
      - { key: "Department", value: "Engineering" } 
      - { key: "Requested By", value: "Jeff Yates" } 
      - { key: "Date Created", value: "{{ ansible_date_time.date }}" } 
    state: poweredon
    validate_certs: no
    force: yes
    wait_for_ip_address: "{{ vm_wait_for | default('yes') }}"
  register: newvm   
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expected it to use a host in the specified Cluster.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

Since "Cluster" is not finding the correct Host with the correct Datastore it fails since it can't find the template. If I use esxi_hostname instead of cluster it works just fine.

<!--- Paste verbatim command output between quotes below -->
Does not find the correct host in the Cluster.
```
fatal: [jaytest]: FAILED! => {
    "changed": true, 
    "clone_method": "Clone", 
    "clonespec": {
        "config": {
            "alternateGuestName": null, 
            "annotation": null, 
            "bootOptions": null, 
            "changeTrackingEnabled": null, 
            "changeVersion": null, 
            "consolePreferences": null, 
            "cpuAffinity": null, 
            "cpuAllocation": null, 
            "cpuFeatureMask": [], 
            "cpuHotAddEnabled": true, 
            "cpuHotRemoveEnabled": null, 
            "crypto": null, 
            "deviceChange": [
                {
                    "backing": null, 
                    "device": "<class 'pyVmomi.VmomiSupport.vim.vm.device.VirtualE1000'>", 
                    "dynamicProperty": [], 
                    "dynamicType": null, 
                    "fileOperation": null, 
                    "operation": "edit", 
                    "profile": []
                }
            ], 
            "dynamicProperty": [], 
            "dynamicType": null, 
            "extraConfig": [], 
            "files": null, 
            "firmware": null, 
            "flags": null, 
            "ftInfo": null, 
            "guestAutoLockEnabled": null, 
            "guestId": null, 
            "instanceUuid": null, 
            "latencySensitivity": null, 
            "locationId": null, 
            "managedBy": null, 
            "maxMksConnections": null, 
            "memoryAffinity": null, 
            "memoryAllocation": null, 
            "memoryHotAddEnabled": true, 
            "memoryMB": 2048, 
            "memoryReservationLockedToMax": null, 
            "messageBusTunnelEnabled": null, 
            "migrateEncryption": null, 
            "name": null, 
            "nestedHVEnabled": null, 
            "networkShaper": null, 
            "npivDesiredNodeWwns": null, 
            "npivDesiredPortWwns": null, 
            "npivNodeWorldWideName": [], 
            "npivOnNonRdmDisks": null, 
            "npivPortWorldWideName": [], 
            "npivTemporaryDisabled": null, 
            "npivWorldWideNameOp": null, 
            "npivWorldWideNameType": null, 
            "numCPUs": 2, 
            "numCoresPerSocket": null, 
            "powerOpInfo": null, 
            "repConfig": null, 
            "scheduledHardwareUpgradeInfo": null, 
            "swapPlacement": null, 
            "tools": null, 
            "uuid": null, 
            "vAppConfig": null, 
            "vAppConfigRemoved": null, 
            "vAssertsEnabled": null, 
            "vPMCEnabled": null, 
            "version": null, 
            "virtualICH7MPresent": null, 
            "virtualSMCPresent": null, 
            "vmProfile": []
        }, 
        "customization": "<class 'pyVmomi.VmomiSupport.vim.vm.customization.Specification'>", 
        "dynamicProperty": [], 
        "dynamicType": null, 
        "location": {
            "datastore": "'vim.Datastore:datastore-1749':Tintri-Anchor", 
            "deviceChange": [], 
            "disk": [], 
            "diskMoveType": null, 
            "dynamicProperty": [], 
            "dynamicType": null, 
            "folder": null, 
            "host": null, 
            "pool": "'vim.ResourcePool:resgroup-716':Resources", 
            "profile": [], 
            "service": null, 
            "transform": null
        }, 
        "memory": null, 
        "powerOn": false, 
        "snapshot": null, 
        "template": false
    }, 
    "configspec": {
        "alternateGuestName": null, 
        "annotation": null, 
        "bootOptions": null, 
        "changeTrackingEnabled": null, 
        "changeVersion": null, 
        "consolePreferences": null, 
        "cpuAffinity": null, 
        "cpuAllocation": null, 
        "cpuFeatureMask": [], 
        "cpuHotAddEnabled": true, 
        "cpuHotRemoveEnabled": null, 
        "crypto": null, 
        "deviceChange": [
            {
                "backing": null, 
                "device": "<class 'pyVmomi.VmomiSupport.vim.vm.device.VirtualE1000'>", 
                "dynamicProperty": [], 
                "dynamicType": null, 
                "fileOperation": null, 
                "operation": "edit", 
                "profile": []
            }
        ], 
        "dynamicProperty": [], 
        "dynamicType": null, 
        "extraConfig": [], 
        "files": null, 
        "firmware": null, 
        "flags": null, 
        "ftInfo": null, 
        "guestAutoLockEnabled": null, 
        "guestId": null, 
        "instanceUuid": null, 
        "latencySensitivity": null, 
        "locationId": null, 
        "managedBy": null, 
        "maxMksConnections": null, 
        "memoryAffinity": null, 
        "memoryAllocation": null, 
        "memoryHotAddEnabled": true, 
        "memoryMB": 2048, 
        "memoryReservationLockedToMax": null, 
        "messageBusTunnelEnabled": null, 
        "migrateEncryption": null, 
        "name": null, 
        "nestedHVEnabled": null, 
        "networkShaper": null, 
        "npivDesiredNodeWwns": null, 
        "npivDesiredPortWwns": null, 
        "npivNodeWorldWideName": [], 
        "npivOnNonRdmDisks": null, 
        "npivPortWorldWideName": [], 
        "npivTemporaryDisabled": null, 
        "npivWorldWideNameOp": null, 
        "npivWorldWideNameType": null, 
        "numCPUs": 2, 
        "numCoresPerSocket": null, 
        "powerOpInfo": null, 
        "repConfig": null, 
        "scheduledHardwareUpgradeInfo": null, 
        "swapPlacement": null, 
        "tools": null, 
        "uuid": null, 
        "vAppConfig": null, 
        "vAppConfigRemoved": null, 
        "vAssertsEnabled": null, 
        "vPMCEnabled": null, 
        "version": null, 
        "virtualICH7MPresent": null, 
        "virtualSMCPresent": null, 
        "vmProfile": []
    }, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "annotation": "Platform Weblogic", 
            "cluster": "RPM", 
            "customization": {
                "domain": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "hostname": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
            }, 
            "customvalues": [
                {
                    "key": "Created By", 
                    "value": "Ansible"
                }, 
                {
                    "key": "Department", 
                    "value": "Engineering"
                }, 
                {
                    "key": "Requested By", 
                    "value": "Jeff"
                }, 
                {
                    "key": "Date Created", 
                    "value": "2017-09-25"
                }
            ], 
            "datacenter": "LV - Prod", 
            "disk": [
                {
                    "datastore": "/Tintri/Tintri-UAT-RPM", 
                    "size_gb": 30, 
                    "type": "thin"
                }
            ], 
            "esxi_hostname": null, 
            "folder": "/Platform-Uat/rpm", 
            "force": true, 
            "guest_id": null, 
            "hardware": {
                "memory_mb": 2048, 
                "num_cpus": 2
            }, 
            "hostname": "a1vcenter01.********", 
            "is_template": false, 
            "linked_clone": false, 
            "name": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "name_match": "first", 
            "networks": [
                {
                    "name": "dvPort-uat-10.10.114", 
                    "type": "dhcp", 
                    "vlan": "dvPort-uat-10.10.114"
                }
            ], 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "resource_pool": null, 
            "snapshot_src": null, 
            "state": "poweredon", 
            "template": "RHMaster7.x-1", 
            "username": "root", 
            "uuid": null, 
            "validate_certs": false, 
            "wait_for_ip_address": true
        }
    }, 
    "msg": "Unable to access the virtual machine configuration: Unable to access file [Tintri-Anchor] RHMaster7.x-1/RHMaster7.x-1.vmtx"
}
```





Discussion:
cc @akasurde @dagwieers @dav1x @jctanner @nerzhul
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Same behavior on RHEL 7.3.
Looking at the source code for vmware_guest I think the problem is that if cluster is selected it never sets the hostname only if esxi_hostname is set.
This seems to be specific to a template creation.
In the function deploy_vm i changed the following and it seems to be working now

old -

if self.params['esxi_hostname']:
  relospec.host = self.select_host()

new -

if self.params['esxi_hostname'] or self.params['cluster']:
  relospec.host = self.select_host()

Looks very similar to https://github.com/ansible/ansible/issues/28649
@coltsixshooter could you send a PR for that change so that it would run through the integration tests?

needs_info
I agree that #28649 seems to be the same thing. I will do a PR this weekend for the change and do a few more tests to verify.
Glad I am testing this, actually had to make two changes. Pull request this weekend.
PR Submitted: https://github.com/ansible/ansible/pull/31139

jscaltreto's fix for the resource pool is the one to go with.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/vmware/vmware_guest.py b/lib/ansible/modules/cloud/vmware/vmware_guest.py
--- a/lib/ansible/modules/cloud/vmware/vmware_guest.py
+++ b/lib/ansible/modules/cloud/vmware/vmware_guest.py
@@ -1198,17 +1198,24 @@ def find_folder(self, searchpath):
         return root
 
     def get_resource_pool(self):
-
         resource_pool = None
-
-        if self.params['esxi_hostname']:
+        # highest priority, resource_pool given.
+        if self.params['resource_pool']:
+            resource_pool = self.select_resource_pool_by_name(self.params['resource_pool'])
+        # next priority, esxi hostname given.
+        elif self.params['esxi_hostname']:
             host = self.select_host()
             resource_pool = self.select_resource_pool_by_host(host)
+        # next priority, cluster given, take the root of the pool
+        elif self.params['cluster']:
+            cluster = self.cache.get_cluster(self.params['cluster'])
+            resource_pool = cluster.resourcePool
+        # fallback, pick any RP
         else:
             resource_pool = self.select_resource_pool_by_name(self.params['resource_pool'])
 
         if resource_pool is None:
-            self.module.fail_json(msg='Unable to find resource pool "%(resource_pool)s"' % self.params)
+            self.module.fail_json(msg='Unable to find resource pool, need esxi_hostname, resource_pool, or cluster')
 
         return resource_pool
 


Test patch:
diff --git a/test/integration/targets/vmware_guest/tasks/create_rp_d1_c1_f0.yml b/test/integration/targets/vmware_guest/tasks/create_rp_d1_c1_f0.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/vmware_guest/tasks/create_rp_d1_c1_f0.yml
@@ -0,0 +1,259 @@
+- name: Wait for Flask controller to come up online
+  wait_for:
+    host: "{{ vcsim }}"
+    port: 5000
+    state: started
+
+- name: kill vcsim
+  uri:
+    url: http://{{ vcsim }}:5000/killall
+- name: start vcsim with no folders
+  uri:
+    url: http://{{ vcsim }}:5000/spawn?datacenter=1&cluster=2&pool=2&folder=0
+  register: vcsim_instance
+
+- name: Wait for Flask controller to come up online
+  wait_for:
+    host: "{{ vcsim }}"
+    port: 443
+    state: started
+
+- name: get a list of VMS from vcsim
+  uri:
+    url: http://{{ vcsim }}:5000/govc_find?filter=VM
+  register: vmlist
+
+- name: get a list of clusters from vcsim
+  uri:
+    url: http://{{ vcsim }}:5000/govc_find?filter=CCR
+  register: clusts
+
+- name: get a list of resource pools from vcsim
+  uri:
+    url: http://{{ vcsim }}:5000/govc_find?filter=RP
+  register: res_pools
+
+- name: get a list of hosts from vcsim
+  uri:
+    url: http://{{ vcsim }}:5000/govc_find?filter=H
+  register: hosts
+
+- debug: var=vcsim_instance
+- debug: var=vmlist
+- debug: var=res_pools
+- debug: var=clusts
+- debug: var=hosts
+
+# Create one with the defaults
+- name: create new VM with default resource pool
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    hardware:
+        num_cpus: 1
+        memory_mb: 512
+    disk:
+        - size: 0gb
+          type: thin
+          autoselect_datastore: True
+    state: poweredoff
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rp_d1_c1_f0
+
+- debug: var=clone_rp_d1_c1_f0
+
+- name: assert that changes were made
+  assert:
+    that:
+      - "clone_rp_d1_c1_f0.results|map(attribute='changed')|unique|list == [true]"
+
+- name: delete the new VMs
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    #guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    state: absent
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rp_d1_c1_f0_delete
+
+- debug: var=clone_rp_d1_c1_f0_delete
+
+- name: assert that changes were made with deletion
+  assert:
+    that:
+        - "clone_rp_d1_c1_f0_delete.results|map(attribute='changed')|unique|list == [true]"
+
+# now create with just a cluster
+- name: create new VM with default resource pool in cluster
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    cluster: "{{ clusts['json'][0]|basename }}"
+    hardware:
+        num_cpus: 1
+        memory_mb: 512
+    disk:
+        - size: 0gb
+          type: thin
+          autoselect_datastore: True
+    state: poweredoff
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rpc_d1_c1_f0
+
+- debug: var=clone_rpc_d1_c1_f0
+
+- name: assert that changes were made
+  assert:
+    that:
+      - "clone_rpc_d1_c1_f0.results|map(attribute='changed')|unique|list == [true]"
+
+- name: delete the new VMs
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    #guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    cluster: "{{ clusts['json'][0]|basename }}"
+    state: absent
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rpc_d1_c1_f0_delete
+
+- debug: var=clone_rpc_d1_c1_f0_delete
+
+- name: assert that changes were made with deletion
+  assert:
+    that:
+        - "clone_rpc_d1_c1_f0_delete.results|map(attribute='changed')|unique|list == [true]"
+
+# now create with a specific resource pool
+- name: create new VM with specific resource pool in cluster
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    cluster: "{{ clusts['json'][0]|basename }}"
+    resource_pool: "{{ res_pools['json'][2]|basename }}"
+    hardware:
+        num_cpus: 1
+        memory_mb: 512
+    disk:
+        - size: 0gb
+          type: thin
+          autoselect_datastore: True
+    state: poweredoff
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rpcp_d1_c1_f0
+
+- debug: var=clone_rpcp_d1_c1_f0
+
+- name: assert that changes were made
+  assert:
+    that:
+      - "clone_rpcp_d1_c1_f0.results|map(attribute='changed')|unique|list == [true]"
+
+- name: delete the new VMs
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    #guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    cluster: "{{ clusts['json'][0]|basename }}"
+    state: absent
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rpcp_d1_c1_f0_delete
+
+- debug: var=clone_rpcp_d1_c1_f0_delete
+
+- name: assert that changes were made with deletion
+  assert:
+    that:
+        - "clone_rpcp_d1_c1_f0_delete.results|map(attribute='changed')|unique|list == [true]"
+
+# now create with a specific host
+- name: create new VM with specific host
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    esxi_hostname: "{{ hosts['json'][0]|basename }}"
+    hardware:
+        num_cpus: 1
+        memory_mb: 512
+    disk:
+        - size: 0gb
+          type: thin
+          autoselect_datastore: True
+    state: poweredoff
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rph_d1_c1_f0
+
+- debug: var=clone_rph_d1_c1_f0
+
+- name: assert that changes were made
+  assert:
+    that:
+      - "clone_rph_d1_c1_f0.results|map(attribute='changed')|unique|list == [true]"
+
+- name: delete the new VMs
+  vmware_guest:
+    validate_certs: False
+    hostname: "{{ vcsim }}"
+    username: "{{ vcsim_instance['json']['username'] }}"
+    password: "{{ vcsim_instance['json']['password'] }}"
+    name: "{{ 'newvm_' + item|basename }}"
+    #template: "{{ item|basename }}"
+    #guest_id: centos64Guest
+    datacenter: "{{ (item|basename).split('_')[0] }}"
+    state: absent
+    folder: "{{ item|dirname }}"
+  with_items: "{{ vmlist['json'][0] }}"
+  register: clone_rph_d1_c1_f0_delete
+
+- debug: var=clone_rph_d1_c1_f0_delete
+
+- name: assert that changes were made with deletion
+  assert:
+    that:
+        - "clone_rph_d1_c1_f0_delete.results|map(attribute='changed')|unique|list == [true]"
diff --git a/test/integration/targets/vmware_guest/tasks/main.yml b/test/integration/targets/vmware_guest/tasks/main.yml
--- a/test/integration/targets/vmware_guest/tasks/main.yml
+++ b/test/integration/targets/vmware_guest/tasks/main.yml
@@ -14,3 +14,4 @@
 - include: clone_d1_c1_f0.yml
 - include: create_d1_c1_f0.yml
 - include: cdrom_d1_c1_f0.yml
+- include: create_rp_d1_c1_f0.yml