Repository: ansible/ansible

Issue:
vmware_guest_file_operation does not work with ESXi directly
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
vmware_guest_file_operation

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.0
  config file = None
  configured module search path = [u'/home/wojtek/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Red Hat 4.8.5-14
##### SUMMARY
<!--- Explain the problem briefly -->
File operations do not work if ESXi is used directly. 
According to the documentation(https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.vm.guest.FileManager.html):
_The host part of the URL is returned as '*' if the hostname to be used is the name of the server to which the call was made._
This means that in case ESXi is contacted directly, URL returned by InitiateFileTransferFromGuest etc. will contain '*' that should be replaced with hostname.

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->


<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: fetch file from vm
  vmware_guest_file_operation:
    hostname: myVSphere
    username: root
    password: mySecret
    validate_certs: True
    vm_id: my_vm
    vm_username: root
    vm_password: superSecret
    copy:
        src: "/tmp/src_file"
        dest: "/tmp/copied_file"
  delegate_to: localhost
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
File is fetched.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
{"msg": "initiateFileTransferToGuest : problem during file transfer", 
"failed": true, 
"uuid": "564d4ed5-9ac8-06b6-d8d4-11b97335963f", 
"invocation": 
{"module_args": 
{"username": "root", 
"datacenter": null, 
"vm_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
"copy": {"dest": "/tmp/copied_file", "src": "/tmp/src_file", "overwrite": false}, 
"vm_id": "my_vm", 
"hostname": "192.168.0.51", 
"vm_username": "root", 
"cluster": null, 
"vm_id_type": "vm_name", 
"directory": null, 
"folder": null, 
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
"validate_certs": true, 
"fetch": null}}}

```



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @akasurde @dav1x @stravassac @warthog9
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
The same issue is in module_utils.vmware.push_file_to_guest module_utils.vmware.fetch_file_from_guest. This applies to older versions.
Files identified in the description:
* [contrib/inventory/vmware.py](https://github.com/ansible/ansible/blob/devel/contrib/inventory/vmware.py)
* [lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/module_utils/vmware.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/vmware.py)
* [lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
@wojciechkopciewicz I cannot reproduce your error but I not use ESXI directly but the Vcenter(6.0U3) for hostname.
Now in your example you copy a file from the Ansible Host to the VM my_vm, you not fetch a file from the vm.
In my case:
```ansible 2.5.2
  config file = None
  configured module search path = ['/home/stravassac/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/stravassac/virtualenvs/ansible/lib/python3.5/site-packages/ansible
  executable location = /home/stravassac/virtualenvs/ansible/bin/ansible
  python version = 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
```

```
tasks:
  - name: file
    vmware_guest_file_operation:
        hostname: XXXXXX
        username: XXXXXX
        password: XXXXXX
        datacenter: XXXXX
        validate_certs: False
        vm_id: XXXXXXXX
        vm_username: root
        vm_password: XXXXXXX
        copy:
          src: "/tmp/toto.txt"
          dest: "/tmp/toto.txt"
    delegate_to: localhost
```


```
<localhost> EXEC /bin/sh -c '/usr/bin/env python /home/stravassac/.ansible/tmp/ansible-tmp-1526282928.7500546-11795477499492/vmware_guest_file_operation.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/stravassac/.ansible/tmp/ansible-tmp-1526282928.7500546-11795477499492/ > /dev/null 2>&1 && sleep 0'
changed: [127.0.0.1 -> localhost] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "cluster": null,
            "copy": {
                "dest": "/tmp/toto.txt",
                "overwrite": false,
                "src": "/tmp/toto.txt"
            },
            "datacenter": "XXXXXXX",
            "directory": null,
            "fetch": null,
            "folder": null,
            "hostname": "XXXXXXXXXXX",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "username": "XXXXXXXXXX",
            "validate_certs": false,
            "vm_id": "XXXXXXXXXX",
            "vm_id_type": "vm_name",
            "vm_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "vm_username": "root"
        }
    },
    "uuid": "564d1723-77f3-06c2-e70f-f52d9440f2ef"
}

```
The problem only exists when communicating with ESXi directly. 
It is caused by the fact that when vCenter is used, the url returned will contain the url of the ESXi to contact.
 If ESXi is communicated directly, url will contain '*' to indicate that the same host should be contacted again.
The simplest solution would be:
url = fileTransferInfo.url
url = url.replace('*', self.params['hostname'])
resp, info = urls.fetch_url(self.module, url, method="GET")

The problem exists both for copy and fetch operations - I mixed sample playbook for copy with result from fetch operation - sorry for that.
@stravassac 
Here are the playbooks and results again both for fail scenarios:
Copy:
```
- name: copy file to vm
      vmware_guest_file_operation:
        copy:
          src: /tmp/sample
          dest: /tmp/sample
        hostname: 192.168.0.51  # this is esxi ip
        password: xxxxxxx
        username: root
        vm_username: root
        vm_id: test_vm
        vm_password: xxxxxxx
        validate_certs: false
```

Result:
```
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "cluster": null, 
            "copy": {
                "dest": "/tmp/sample", 
                "overwrite": false, 
                "src": "/tmp/sample"
            }, 
            "datacenter": null, 
            "directory": null, 
            "fetch": null, 
            "folder": null, 
            "hostname": "192.168.0.51", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "username": "root", 
            "validate_certs": false, 
            "vm_id": "test_vm", 
            "vm_id_type": "vm_name", 
            "vm_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "vm_username": "root"
        }
    }, 
    "msg": "initiateFileTransferToGuest : problem during file transfer", 
    "uuid": "564d4ed5-9ac8-06b6-d8d4-11b97335963f"
}
```

Fetch:
```
    - name: fetch file from vm
      vmware_guest_file_operation:
        fetch:
          src: /tmp/sample
          dest: /tmp/sample
        hostname: 192.168.0.51  # this is esxi ip
        password: xxxxxxx
        username: root
        vm_username: root
        vm_id: test_vm
        vm_password: xxxxxxx
        validate_certs: false

```
Result:

```
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "cluster": null, 
            "copy": null, 
            "datacenter": null, 
            "directory": null, 
            "fetch": {
                "dest": "/tmp/sample", 
                "src": "/tmp/sample"
            }, 
            "folder": null, 
            "hostname": "192.168.0.51", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "username": "root", 
            "validate_certs": false, 
            "vm_id": "test_vm", 
            "vm_id_type": "vm_name", 
            "vm_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "vm_username": "root"
        }
    }, 
    "msg": "local file write exception : 'NoneType' object has no attribute 'read'", 
    "uuid": "564d4ed5-9ac8-06b6-d8d4-11b97335963f"
}
```

I can see that asterisks are not present in my previous comment.
The code that solves the problem looks as follows:
```
url = fileTransferInfo.url
url = url.replace("*", self.params['hostname'])
resp, info = urls.fetch_url(self.module, url, method="GET")
```

Resolving patch:
diff --git a/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py b/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py
--- a/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py
+++ b/lib/ansible/modules/cloud/vmware/vmware_guest_file_operation.py
@@ -277,6 +277,7 @@ def fetch(self):
         result = dict(changed=True, uuid=self.vm.summary.config.uuid)
         vm_username = self.module.params['vm_username']
         vm_password = self.module.params['vm_password']
+        hostname = self.module.params['hostname']
         dest = self.module.params["fetch"]['dest']
         src = self.module.params['fetch']['src']
         creds = vim.vm.guest.NamePasswordAuthentication(username=vm_username, password=vm_password)
@@ -286,6 +287,7 @@ def fetch(self):
             fileTransferInfo = file_manager.InitiateFileTransferFromGuest(vm=self.vm, auth=creds,
                                                                           guestFilePath=src)
             url = fileTransferInfo.url
+            url = url.replace("*", hostname)
             resp, info = urls.fetch_url(self.module, url, method="GET")
             try:
                 with open(dest, "wb") as local_file:
@@ -316,6 +318,7 @@ def copy(self):
         result = dict(changed=True, uuid=self.vm.summary.config.uuid)
         vm_username = self.module.params['vm_username']
         vm_password = self.module.params['vm_password']
+        hostname = self.module.params['hostname']
         overwrite = self.module.params["copy"]["overwrite"]
         dest = self.module.params["copy"]['dest']
         src = self.module.params['copy']['src']
@@ -340,6 +343,7 @@ def copy(self):
             url = file_manager.InitiateFileTransferToGuest(vm=self.vm, auth=creds, guestFilePath=dest,
                                                            fileAttributes=file_attributes, overwrite=overwrite,
                                                            fileSize=file_size)
+            url = url.replace("*", hostname)
             resp, info = urls.fetch_url(self.module, url, data=data, method="PUT")
 
             status_code = info["status"]


Test patch: