Repository: ansible/ansible

Issue:
vmware_guest - Cluster option stopped working 
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
modules/cloud/vmware/vmware_guest.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.1.0
  config file = /home/jzimmett/.ansible.cfg
  configured module search path = [u'/home/jzimmett/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

```

##### CONFIGURATION
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = [u'/home/xxxx/workspace/devops/platform-roles
HOST_KEY_CHECKING(/home/xxxx/.ansible.cfg) = False
RETRY_FILES_ENABLED(/home/xxxx/.ansible.cfg) = False

##### OS / ENVIRONMENT
Red Hat 7.4

##### SUMMARY
Same issue as #30879. vmware_guest is not handling the "cluster" option correctly and fails that it can not find the data store.

I have modified the vmware_guest.py to work properly and will do a pull request.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: "Create DHCP VM {{ vm_name }}"
  vmware_guest:
    hostname: "{{ vc_hostname }}"
    username: "{{ vc_user }}"
    password: "{{ vc_pass }}"
    cluster: "{{ vc_cluster }}"
    datacenter: "{{ vc_datacenter }}"
    template: "{{ vm_template }}"
    name: "{{ vm_name }}"
    networks:
      - vlan: "{{ vm_vlan }}"
        type: "dhcp"
    annotation: "{{ vm_notes }}"
    folder: "{{ vc_folder }}"
    disk:
      - size_gb: 30
        type: thin
        datastore: "{{ vm_datastore }}"
    hardware:
      memory_mb: "{{ vm_memory }}"
      num_cpus: "{{ vm_cpu }}"
    customization:
      hostname: "{{ vm_name }}"
      domain: "{{ vm_domain }}"
    customvalues:
      - { key: "Created By", value: "Ansible" }
      - { key: "Department", value: "Engineering" }
      - { key: "Requested By", value: "Jeff Yates" }
      - { key: "Date Created", value: "{{ ansible_date_time.date }}" }
    state: poweredon
    validate_certs: no
    force: yes
    wait_for_ip_address: "{{ vm_wait_for | default('yes') }}"
  register: newvm

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS

Handle the Cluster option properly

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
Unable to access the virtual machine configuration: Unable to access file [Tintri-UAT-RPM]

```



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/vmware/vmware_guest.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/vmware/vmware_guest.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @akasurde @dagwieers @dav1x @jctanner @nerzhul @pdellaert
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I'm also experiencing this issue in 2.4.1  The cluster parameter seems ignored.
This is currently impacting an engagement.
As a workaround, use the esxi_hostname: parameter.  You can use a group variable if you have multiple clusters to make it work until the clusters parameter gets fixed.
@coltsixshooter @mike45379 @one-t Could you please re-try again with the latest version and let me know if it works for you ?

needs_info
Yes, I will pull it down and test on 12/27.

Thanks

On Sat, Dec 16, 2017 at 8:14 AM, Abhijeet Kasurde <notifications@github.com>
wrote:

> @coltsixshooter <https://github.com/coltsixshooter> @mike45379
> <https://github.com/mike45379> @one-t <https://github.com/one-t> Could
> you please re-try again with the latest version and let me know if it works
> for you ?
>
> needs_info
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33156#issuecomment-352192942>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGsRqPZU78HebbYqTpJjgeiFKYdesSupks5tA-xfgaJpZM4QmW6B>
> .
>

I am still experiencing the issue with 2.4.2.  The workaround using esxi_hostname still works.  
I have made code changes and uploaded but they don't seem to get into the build -

vmware_guest.py

Function "get_resource_pool"

Update this line

if self.params['esxi_hostname']:

to

if self.params['esxi_hostname'] or self.params['cluster']:

Function "deploy_vm"

if self.params['esxi_hostname']:

to

if self.params['esxi_hostname'] or self.params['cluster']:
That fix worked.  Thanks. 
Can the ansible team please confirm this fix will get into the next build?

Thanks
@coltsixshooter Does this fix work with multiple clusters i.e, If I have template in one cluster and I would like to deploy VM to another cluster ... I still see the error?
Not sure, but will test and try to fix if broken.

On Fri, Jan 19, 2018 at 10:26 PM, NoOne <notifications@github.com> wrote:

> @coltsixshooter <https://github.com/coltsixshooter> Does this fix work
> with multiple clusters i.e, If I have template in one cluster and I would
> like to deploy VM to another cluster ... I still see the error?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33156#issuecomment-359149544>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGsRqBxx9t_zdteR5I6mUset5olvtbvgks5tMYcpgaJpZM4QmW6B>
> .
>

Yeah it’s not working if I have a data store clusters. And as I said it’s not working across different clusters 
If you could able to look into it and fix the issue that would be great full.
This problem still exist in 2.5.0, can someone please make the one line fix.

Function "deploy_vm"

if self.params['esxi_hostname']:
to
if self.params['esxi_hostname'] or self.params['cluster']:
@coltsixshooter Could you please elaborate more on the scenario you are trying here ? I will try to reproduce this in my environment.
Without the "or" statement for the cluster option it seems to fail finding
a vmware host in the cluster. This has been a problem since 2.4

On Wed, Apr 11, 2018 at 9:10 PM, Abhijeet Kasurde <notifications@github.com>
wrote:

> Could you please elaborate more on the scenario you are trying here ? I
> will try to reproduce this in my environment.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33156#issuecomment-380671961>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGsRqMIjd9ED7YCqz_rVhQcGh7JlEBI-ks5tntOdgaJpZM4QmW6B>
> .
>

We have a similar issue. Problem description: we have a VMware setup with multiple clusters in one datacenter. The VM (created from a template) gets assigned to a random cluster/esxi host with ansible 2.5.2. We already noticed that when we use a VMware user with no access permissions on the other clusters, the vm creation works perfectly without any code changes. (It also works fine with Ansible 2.3)
cc @warthog9
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Additional checks showed us that this is probably related to the resource pool configuration. We have the same resource pools configured on all clusters and the vmware_guest module ignores the cluster selection when you select a resource pool. A random resource pool with a matching name is selected and that can be on any cluster...
@stroobl Thanks for providing debugging information. There is a PR open for this - https://github.com/ansible/ansible/pull/39470. 

Resolving patch:
diff --git a/lib/ansible/module_utils/vmware.py b/lib/ansible/module_utils/vmware.py
--- a/lib/ansible/module_utils/vmware.py
+++ b/lib/ansible/module_utils/vmware.py
@@ -76,28 +76,20 @@ def wait_for_vm_ip(content, vm, timeout=300):
     return facts
 
 
-def find_obj(content, vimtype, name, first=True):
-    container = content.viewManager.CreateContainerView(container=content.rootFolder, recursive=True, type=vimtype)
-    obj_list = container.view
+def find_obj(content, vimtype, name, first=True, folder=None):
+    container = content.viewManager.CreateContainerView(folder or content.rootFolder, recursive=True, type=vimtype)
+    # Get all objects matching type (and name if given)
+    obj_list = [obj for obj in container.view if not name or to_text(obj.name) == to_text(name)]
     container.Destroy()
 
-    # Backward compatible with former get_obj() function
-    if name is None:
+    # Return first match or None
+    if first:
         if obj_list:
             return obj_list[0]
         return None
 
-    # Select the first match
-    if first is True:
-        for obj in obj_list:
-            if to_text(obj.name) == to_text(name):
-                return obj
-
-        # If no object found, return None
-        return None
-
-    # Return all matching objects if needed
-    return [obj for obj in obj_list if obj.name == name]
+    # Return all matching objects or empty list
+    return obj_list
 
 
 def find_dvspg_by_name(dv_switch, portgroup_name):
diff --git a/lib/ansible/modules/cloud/vmware/vmware_guest.py b/lib/ansible/modules/cloud/vmware/vmware_guest.py
--- a/lib/ansible/modules/cloud/vmware/vmware_guest.py
+++ b/lib/ansible/modules/cloud/vmware/vmware_guest.py
@@ -1850,33 +1850,6 @@ def obj_has_parent(self, obj, parent):
             if current_parent is None:
                 return False
 
-    def select_resource_pool_by_name(self, resource_pool_name):
-        resource_pool = self.cache.find_obj(self.content, [vim.ResourcePool], resource_pool_name)
-        if resource_pool is None:
-            self.module.fail_json(msg='Could not find resource_pool "%s"' % resource_pool_name)
-        return resource_pool
-
-    def select_resource_pool_by_host(self, host):
-        resource_pools = self.cache.get_all_objs(self.content, [vim.ResourcePool])
-        for rp in resource_pools.items():
-            if not rp[0]:
-                continue
-
-            if not hasattr(rp[0], 'parent') or not rp[0].parent:
-                continue
-
-            # Find resource pool on host
-            if self.obj_has_parent(rp[0].parent, host.parent):
-                # If no resource_pool selected or it's the selected pool, return it
-                if self.module.params['resource_pool'] is None or rp[0].name == self.module.params['resource_pool']:
-                    return rp[0]
-
-        if self.module.params['resource_pool'] is not None:
-            self.module.fail_json(msg="Could not find resource_pool %s for selected host %s"
-                                  % (self.module.params['resource_pool'], host.name))
-        else:
-            self.module.fail_json(msg="Failed to find a resource group for %s" % host.name)
-
     def get_scsi_type(self):
         disk_controller_type = "paravirtual"
         # set cpu/memory/etc
@@ -1921,28 +1894,39 @@ def find_folder(self, searchpath):
 
         return root
 
-    def get_resource_pool(self):
-        resource_pool = None
-        # highest priority, resource_pool given.
-        if self.params['resource_pool']:
-            resource_pool = self.select_resource_pool_by_name(self.params['resource_pool'])
-        # next priority, esxi hostname given.
-        elif self.params['esxi_hostname']:
-            host = self.select_host()
-            resource_pool = self.select_resource_pool_by_host(host)
-        # next priority, cluster given, take the root of the pool
-        elif self.params['cluster']:
-            cluster = self.cache.get_cluster(self.params['cluster'])
-            if cluster is None:
-                self.module.fail_json(msg="Unable to find cluster '%(cluster)s'" % self.params)
-            resource_pool = cluster.resourcePool
-        # fallback, pick any RP
+    def get_resource_pool(self, cluster=None, host=None, resource_pool=None):
+        """ Get a resource pool, filter on cluster, esxi_hostname or resource_pool if given """
+
+        cluster_name = cluster or self.params.get('cluster', None)
+        host_name = host or self.params.get('esxi_hostname', None)
+        resource_pool_name = resource_pool or self.params.get('resource_pool', None)
+
+        # get the datacenter object
+        datacenter = find_obj(self.content, [vim.Datacenter], self.params['datacenter'])
+        if not datacenter:
+            self.module.fail_json(msg='Unable to find datacenter "%s"' % self.params['datacenter'])
+
+        # if cluster is given, get the cluster object
+        if cluster_name:
+            cluster = find_obj(self.content, [vim.ComputeResource], cluster_name, folder=datacenter)
+            if not cluster:
+                self.module.fail_json(msg='Unable to find cluster "%s"' % cluster_name)
+        # if host is given, get the cluster object using the host
+        elif host_name:
+            host = find_obj(self.content, [vim.HostSystem], host_name, folder=datacenter)
+            if not host:
+                self.module.fail_json(msg='Unable to find host "%s"' % host_name)
+            cluster = host.parent
         else:
-            resource_pool = self.select_resource_pool_by_name(self.params['resource_pool'])
-
-        if resource_pool is None:
-            self.module.fail_json(msg='Unable to find resource pool, need esxi_hostname, resource_pool, or cluster')
+            cluster = None
 
+        # get resource pools limiting search to cluster or datacenter
+        resource_pool = find_obj(self.content, [vim.ResourcePool], resource_pool_name, folder=cluster or datacenter)
+        if not resource_pool:
+            if resource_pool_name:
+                self.module.fail_json(msg='Unable to find resource_pool "%s"' % resource_pool_name)
+            else:
+                self.module.fail_json(msg='Unable to find resource pool, need esxi_hostname, resource_pool, or cluster')
         return resource_pool
 
     def deploy_vm(self):
@@ -2189,12 +2173,9 @@ def reconfigure_vm(self):
 
         relospec = vim.vm.RelocateSpec()
         if self.params['resource_pool']:
-            relospec.pool = self.select_resource_pool_by_name(self.params['resource_pool'])
-
-            if relospec.pool is None:
-                self.module.fail_json(msg='Unable to find resource pool "%(resource_pool)s"' % self.params)
+            relospec.pool = self.get_resource_pool()
 
-            elif relospec.pool != self.current_vm_obj.resourcePool:
+            if relospec.pool != self.current_vm_obj.resourcePool:
                 task = self.current_vm_obj.RelocateVM_Task(spec=relospec)
                 self.wait_for_task(task)
                 change_applied = True


Test patch:
diff --git a/test/integration/targets/vmware_guest/tasks/cdrom_d1_c1_f0.yml b/test/integration/targets/vmware_guest/tasks/cdrom_d1_c1_f0.yml
--- a/test/integration/targets/vmware_guest/tasks/cdrom_d1_c1_f0.yml
+++ b/test/integration/targets/vmware_guest/tasks/cdrom_d1_c1_f0.yml
@@ -35,7 +35,7 @@
     folder: "/{{ (clusterlist['json'][0]|basename).split('_')[0] }}/vm"
     name: CDROM-Test
     datacenter: "{{ (clusterlist['json'][0]|basename).split('_')[0] }}"
-    cluster: "{{ clusterlist['json'][0] }}"
+    cluster: "{{ clusterlist['json'][0]|basename }}"
     resource_pool: Resources
     guest_id: centos64Guest
     hardware:
diff --git a/test/integration/targets/vmware_guest/tasks/vapp_d1_c1_f0.yml b/test/integration/targets/vmware_guest/tasks/vapp_d1_c1_f0.yml
--- a/test/integration/targets/vmware_guest/tasks/vapp_d1_c1_f0.yml
+++ b/test/integration/targets/vmware_guest/tasks/vapp_d1_c1_f0.yml
@@ -35,7 +35,7 @@
     folder: "/{{ (clusterlist['json'][0]|basename).split('_')[0] }}/vm"
     name: vApp-Test
     datacenter: "{{ (clusterlist['json'][0]|basename).split('_')[0] }}"
-    cluster: "{{ clusterlist['json'][0] }}"
+    cluster: "{{ clusterlist['json'][0]|basename }}"
     resource_pool: Resources
     guest_id: centos64Guest
     hardware: