Repository: ansible-collections/community.vmware

Issue:
vmware_guest_cross_vc_clone: a long running task fails for "NotAuthenticated"
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
vmware_guest_cross_vc_clone fails when cloning a rather huge image.

The cloning of a ~17GB image takes something like ~1h.  The clone operation continues in vCenter and is succesfull, only the ansible task fails. See ACTUAL RESULTS for the logs. Seems like some sort of a timeout issue for the authentication/session in the ansible module.

A 4GB image takes some 17 mins in our env, and is successful.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
vmware_guest_cross_vc_clone

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below

```

##### COLLECTION VERSION
<!--- Paste verbatim output from "ansible-galaxy collection list <namespace>.<collection>"  between the quotes
for example: ansible-galaxy collection list community.general
-->
```paste below
community-vmware-1.17.1.tar.gz
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below

```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->


##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
```An exception occurred during task execution. To see the full traceback, use -vvv. The error was: }
fatal: [my-esx-host -> localhost]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 40, in invoke_module
  File "/usr/lib64/python2.7/runpy.py", line 176, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 396, in <module>
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 387, in main
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 253, in clone
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 229, in get_new_vm_info
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py", line 365, in find_vm_by_name
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py", line 207, in find_object_by_name
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_yJuy2n/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py", line 794, in get_all_objs
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1397, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.NotAuthenticated: (vim.fault.NotAuthenticated) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'The session is not authenticated.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   object = 'vim.Folder:group-d1',
   privilegeId = 'System.View'
}
```



Discussion:
A bit different timeout exception, quite exactly after 1 hour:
```
TASK [clone a VM across VC] ****************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.community.vmware.plugins.module_utils.vmware.TaskError: Timeout
fatal: [my-precious-esx -> localhost]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 40, in invoke_module
  File "/usr/lib64/python2.7/runpy.py", line 176, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_QsIiA3/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 396, in <module>
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_QsIiA3/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 387, in main
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_QsIiA3/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_cross_vc_clone.py", line 249, in clone
  File "/tmp/ansible_community.vmware.vmware_guest_cross_vc_clone_payload_QsIiA3/ansible_community.vmware.vmware_guest_cross_vc_clone_payload.zip/ansible_collections/community/vmware/plugins/module_utils/vmware.py", line 140, in wait_for_task
ansible_collections.community.vmware.plugins.module_utils.vmware.TaskError: Timeout

```
Uff...this latter timeout seems come from here: https://github.com/ansible-collections/community.vmware/blob/023a5f7e886356f75f3bf0de144e11bc9b0b9373/plugins/module_utils/vmware.py#L118

None of the calling modules, including vmware_guest_cross_vc_clone is passing a timeout-parameter so the default  of "only" 1h is being used. Also, there's no way to configure the timeout from outside - so I'm doomed with this. For cross vcenter cloning, which often means cross site cloning, the transfer will take quite a long time.

Would the original "NotAuthenticated" be related to this task-timeout? I'm pretty sure that NonAutehnticated happened in much shorter time that this 1h - unfortunately I can't tell what was the timing for that failure.
Hello,

For your first issue, it came from the function get_new_vm_info. At the end of the clone, the line "find_vm_by_name(content=self.destination_content, vm_name=vm)" use self.destination_content which use a session created before the clone in "sanitize_params" function. We need to recreate a session after the clone to get the new vm info, because the session could be expired depending on the time of the clone. I did it locally and it works. I will push the fix soon.

For your second issue, you're right. We should be able to override the timeout value like it has been done in the here : https://github.com/ansible-collections/community.vmware/commit/bbf7b4d50bac82f9b9d2c74dabc38294326c8375

Again I tested it and it works, I will push it soon.

Alban


Resolving patch:
diff --git a/plugins/modules/vmware_guest_cross_vc_clone.py b/plugins/modules/vmware_guest_cross_vc_clone.py
--- a/plugins/modules/vmware_guest_cross_vc_clone.py
+++ b/plugins/modules/vmware_guest_cross_vc_clone.py
@@ -111,7 +111,11 @@
     required: false
     default: 'present'
     choices: [ 'present', 'poweredon' ]
-
+  timeout:
+    description:
+    - The timeout in seconds. When the timeout is reached, the module will fail.
+    type: int
+    default: 3600
 extends_documentation_fragment:
 - community.vmware.vmware.documentation
 
@@ -219,12 +223,21 @@ def __init__(self, module):
         self.destination_vcenter_password = self.params['destination_vcenter_password']
         self.destination_vcenter_port = self.params.get('port', 443)
         self.destination_vcenter_validate_certs = self.params.get('destination_vcenter_validate_certs', None)
+        self.timeout = self.params.get('timeout')
 
     def get_new_vm_info(self, vm):
         # to check if vm has been cloned in the destination vc
+        # connect to destination VC
         # query for the vm in destination vc
         # get the host and datastore info
         # get the power status of the newly cloned vm
+        self.destination_content = connect_to_api(
+            self.module,
+            hostname=self.destination_vcenter,
+            username=self.destination_vcenter_username,
+            password=self.destination_vcenter_password,
+            port=self.destination_vcenter_port,
+            validate_certs=self.destination_vcenter_validate_certs)
         info = {}
         vm_obj = find_vm_by_name(content=self.destination_content, vm_name=vm)
         if vm_obj is None:
@@ -246,7 +259,7 @@ def clone(self):
             self.module.fail_json(msg="Destination folder does not exist. Please refer to the documentation to correctly specify the folder.")
         vm_name = self.params['destination_vm_name']
         task = self.vm_obj.Clone(folder=vm_folder, name=vm_name, spec=self.clone_spec)
-        wait_for_task(task)
+        wait_for_task(task, timeout=self.timeout)
         if task.info.state == 'error':
             result = {'changed': False, 'failed': True, 'msg': task.info.error.msg}
         else:
@@ -349,7 +362,8 @@ def main():
         destination_resource_pool=dict(type='str', default=None),
         is_template=dict(type='bool', default=False),
         state=dict(type='str', default='present',
-                   choices=['present', 'poweredon'])
+                   choices=['present', 'poweredon']),
+        timeout=dict(type='int', default=3600)
     )
 
     module = AnsibleModule(


Test patch: