Repository: ansible-collections/community.vmware

Issue:
Inventory plugin is extremely slow
##### SUMMARY
It takes about 1.5min to get ~350 hosts from vCenter
Related issue: https://github.com/ansible/ansible/issues/56786

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
community.vmware.vmware_vm_inventory

##### ANSIBLE VERSION
```
ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/lib/python-exec/python3.7/ansible
  python version = 3.7.9 (default, Aug 21 2020, 01:37:20) [GCC 9.3.0]
```

##### CONFIGURATION
```
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_legacy_silent
INVENTORY_ENABLED(/etc/ansible/ansible.cfg) = ['host_list', 'yaml', 'constructed', 'vmware_vm_inventory']
RETRY_FILES_SAVE_PATH(/etc/ansible/ansible.cfg) = /tmp/ansible/retry-files
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Gentoo GNU/Linux

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
Plugin configuration:
```yaml
plugin: community.vmware.vmware_vm_inventory
strict: False
hostname: "server"
username: "user"
password: "*****"
validate_certs: False
with_tags: False
properties:
  - 'name'
  - 'guest.ipAddress'
  - 'config.name'
  - 'config.uuid'
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->

Python script from https://github.com/vmware/vsphere-automation-sdk-python#connect-to-a-vcenter-server
do the work in ~3sec.

```python
import requests
import urllib3
from vmware.vapi.vsphere.client import create_vsphere_client
session = requests.session()

session.verify = False
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
client = create_vsphere_client(server=hostname,
                               username=username,
                               password=password,
                               session=session)

print(client.vcenter.VM.list())
```
```
real	0m3.198s
user	0m0.366s
sys	0m0.037s
API invocations: 4
```

##### ACTUAL RESULTS
Ansible inventory - almost 1.5min

<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
time ansible-inventory -vvvv --list -i inventory.vmware.yml

<!--- Paste verbatim command output between quotes -->
```paste below
real	1m26.505s
user	0m6.451s
sys	0m0.180s
API invocations: 5
```



Discussion:
@Perlovka Thanks for reporting this issue.

@dacrystal Would you like to provide your views on this? Thanks.
I have run it on a ~100vms vCenter using same plugin configuration and it didn't take that much:

```sh
real    0m3.818s
user    0m1.999s
sys     0m0.172s
```

@Perlovka I assume when you said 350 hosts, you also meant 350 guests(vms) of vCenter, right?

From your time, the process is waiting on I/O(blocking) for  **79.874s** while  the actually process time is only (**6.631s**). It seems there is a huge delay in getting the data from vCenter.


Would you mind generate a profile stats and share it. You can run the following:
```sh
python -m cProfile -o profile.pstats $(command -v ansible-inventory) -i inventory.vmware.yml --list
```

As a workaround, you can enable `cache` and set a `cache_timeout`.





@dacrystal yes, 350 VMs, not ESXi hosts )

It looks like it makes query to vCenter for every VM, and because connection is slow, total time is so big.
Is there a way to get info about all VMs in one API call? Or VMWare API does not allowing this?
```
Tue Oct 13 09:08:03 2020    profile.pstats

         16868969 function calls (16677172 primitive calls) in 78.814 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1184   55.276    0.047   55.276    0.047 {method 'read' of '_ssl._SSLSocket' objects}
   189660    2.177    0.000    6.865    0.000 SoapAdapter.py:660(StartElementHandler)
   189661    1.864    0.000    5.398    0.000 SoapAdapter.py:721(EndElementHandler)
      388    1.705    0.004   14.575    0.038 {method 'ParseFile' of 'pyexpat.xmlparser' objects}
   539869    1.170    0.000    1.170    0.000 VmomiSupport.py:461(GetPropertyInfo)
    47597    1.148    0.000    2.330    0.000 VmomiSupport.py:624(__init__)
        4    1.113    0.278    1.113    0.278 {method 'do_handshake' of '_ssl._SSLSocket' objects}
  2676028    0.914    0.000    0.990    0.000 {built-in method builtins.isinstance}
```
From the number it seems your connection to vCenter is slow. You can see each **read** of `SSLSocket` takes **0.047s**.

I'm running a test over VPN (to a nearby Data Center):
```
708    2.491    0.004    2.491    0.004 {method 'read' of '_ssl._SSLSocket' objects}
```

The plugin is calling vCenter in one pyVmomi API call.

```py
return self.content.propertyCollector.RetrieveContents([filter_spec])
```

However there is seems to be rogue calls that triggered for each VM. Would you mind remove the following line in `vmware/plugins/inventory/vmware_vm_inventory.py` (751-753):
```py
            if not vm_obj.obj.config:
                # Sometime orphaned VMs return no configurations
                continue
``` 

And report the profile of `{method 'read' of '_ssl._SSLSocket' objects}`, this should improve it.


---

@Akasurde I'm planing on refactor for optimization, Here a list of improvements I have in mind:
- Remove orphaned VMs checks (TBH, I'm not sure why it is there :sweat_smile: )
- Optimize `to_nested_dict`. Maybe exploring `VmomiSupport.VmomiJSONEncoder`.
- Use `SmartConnect` and `SmartConnectNoSSL`, instead of providing `SSLContext` manually.



- Orphaned VM check is added for VMs which are not completely deleted.
- OK
- OK
For that case, we can internally add `runtime.connectionState` to `properties` if not added and then check for `orphaned `:

```py
if properties['runtime.connectionState'] == 'orphaned':
                continue
```


Resolving patch:
diff --git a/plugins/inventory/vmware_vm_inventory.py b/plugins/inventory/vmware_vm_inventory.py
--- a/plugins/inventory/vmware_vm_inventory.py
+++ b/plugins/inventory/vmware_vm_inventory.py
@@ -280,7 +280,6 @@
 import base64
 from ansible.errors import AnsibleError, AnsibleParserError
 from ansible.module_utils._text import to_text, to_native
-from ansible.module_utils.common.dict_transformations import dict_merge
 from ansible.module_utils.common.dict_transformations import camel_dict_to_snake_dict
 from ansible.module_utils.common.dict_transformations import _snake_to_camel
 from ansible.utils.display import Display
@@ -388,6 +387,7 @@ def _login(self):
             service_instance = connect.SmartConnect(host=self.hostname, user=self.username,
                                                     pwd=self.password, sslContext=ssl_context,
                                                     port=self.port)
+
         except vim.fault.InvalidLogin as e:
             raise AnsibleParserError("Unable to log on to vCenter or ESXi API at %s:%s as %s: %s" % (self.hostname, self.port, self.username, e.msg))
         except vim.fault.NoPermission as e:
@@ -554,6 +554,21 @@ def build_containers(containers, vim_type, names, filters):
         return []
 
 
+def in_place_merge(a, b):
+    """
+        Recursively merges second dict into the first.
+
+    """
+    if not isinstance(b, dict):
+        return b
+    for k, v in b.items():
+        if k in a and isinstance(a[k], dict):
+            a[k] = in_place_merge(a[k], v)
+        else:
+            a[k] = v
+    return a
+
+
 def to_nested_dict(vm_properties):
     """
     Parse properties from dot notation to dict
@@ -568,7 +583,7 @@ def to_nested_dict(vm_properties):
 
         for k in prop_parents:
             prop_dict = {k: prop_dict}
-        host_properties = dict_merge(host_properties, prop_dict)
+        host_properties = in_place_merge(host_properties, prop_dict)
 
     return host_properties
 
@@ -721,6 +736,8 @@ def _populate_from_source(self):
             query_props = None
             vm_properties.remove('all')
         else:
+            if 'runtime.connectionState' not in vm_properties:
+                vm_properties.append('runtime.connectionState')
             query_props = [x for x in vm_properties if x != "customValue"]
 
         objects = self.pyv.get_managed_objects_properties(
@@ -743,14 +760,13 @@ def _populate_from_source(self):
         hostnames = self.get_option('hostnames')
 
         for vm_obj in objects:
-            if not vm_obj.obj.config:
-                # Sometime orphaned VMs return no configurations
-                continue
-
             properties = dict()
             for vm_obj_property in vm_obj.propSet:
                 properties[vm_obj_property.name] = vm_obj_property.val
 
+            if (properties.get('runtime.connectionState') or properties['runtime'].connectionState) == 'orphaned':
+                continue
+
             # Custom values
             if 'customValue' in vm_properties:
                 field_mgr = []


Test patch: