Repository: ansible/ansible

Issue:
unarchive: unable to unpack tar.gz from URI with python3 interpreter
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
unarchive

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
$ ansible --version
ansible 2.4.0 (devel 313591f8b8) last updated 2017/04/13 21:20:44 (GMT +200)
  config file = 
  configured module search path = [u'/home/pvdputte/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pvdputte/VC/vendor/ansible-master/lib/ansible
  executable location = /home/pvdputte/VC/vendor/ansible-master/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```

##### CONFIGURATION
Default

##### OS / ENVIRONMENT
Running from ubuntu 14.04LTS
Managing ubuntu 17.04 (only python 3 after base install)

##### SUMMARY
<!--- Explain the problem briefly -->
Attempting to run the unarchive task to download and extract an http URL yields an error "Source 'http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz' failed to transfer"

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
$ ansible-d targethost -e 'ansible_python_interpreter=/usr/bin/python3' -m unarchive -a "src=http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz dest=/tmp remote_src=true"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
tar.gz file is fetched and extracted in the expected location. E.g. with python 2:
```
$ ansible-d targethost -m unarchive -a "src=http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz dest=/tmp remote_src=true"
targethost | SUCCESS => {
    "changed": true, 
    "dest": "/tmp", 
    "extract_results": {
        "cmd": [
            "/bin/tar", 
            "--extract", 
            "-C", 
            "/tmp", 
            "-z", 
            "-f", 
            "/tmp/ansible_3W9Kei/SoapUI-5.3.0-linux-bin.tar.gz"
        ], 
        "err": "", 
        "out": "", 
        "rc": 0
    }, 
    "gid": 0, 
    "group": "root", 
    "handler": "TgzArchive", 
    "mode": "01777", 
    "owner": "root", 
    "size": 4096, 
    "src": "/tmp/ansible_3W9Kei/SoapUI-5.3.0-linux-bin.tar.gz", 
    "state": "directory", 
    "uid": 0
}
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

ansible 2.3:
```
targethost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/tmp", 
            "directory_mode": null, 
            "exclude": [], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "SoapUI-5.3.0-linux-bin.tar.gz", 
            "owner": null, 
            "regexp": null, 
            "remote_src": true, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "msg": "Failure downloading http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz, write() argument must be str, not bytes"
}
```
ansible 2.4:
```
targethost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/usr/local", 
            "directory_mode": null, 
            "exclude": [], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "SoapUI-5.3.0-linux-bin.tar.gz", 
            "owner": null, 
            "regexp": null, 
            "remote_src": false, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "msg": "Source 'http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz' failed to transfer"
}
```



Using ansible + python 2 to download on the target system works fine.


At first I thought it would be similar to #19710, but apparently something related has already changed in master, as the 2.4 output looks different.


Discussion:
cc @dagwieers @pileofrogs
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I'm having the same issue ( with python 2.7 though) I think it's coming fromn the fact that `remote_src` is somehow set incorrectly.
If you look at the ansible 2.4 log, it says `"remote_src": false,` 
This appears to be an inconsistency in the way that `remote_src` works in comparison to `copy`.

Specifically due to these lines:

```python
    # did tar file arrive?
    if not os.path.exists(src):
        if not remote_src and copy:
            module.fail_json(msg="Source '%s' failed to transfer" % src)
```

The change that likely caused this is 6aeea1fe

The work around currently is to also supply `copy=false`.  The documentation lists that `copy` is deprecated in favor of `remote_src` but this code path will not allow you to use a URL, without setting `copy=false`
git bisect tells me that it hasn't been working since 7b197d823ee775b0138495bcd3954dff28851dd5 

probably because of this : 

    # remove action plugin only key
    for key in ('remote_src', 'decrypt'):
        if key in new_module_args:
            del new_module_args[key]

in ansible/lib/ansible/plugins/action/unarchive.py

Using `copy: false` instead of `remote_src: true` worked for me

Resolving patch:
diff --git a/lib/ansible/modules/files/unarchive.py b/lib/ansible/modules/files/unarchive.py
--- a/lib/ansible/modules/files/unarchive.py
+++ b/lib/ansible/modules/files/unarchive.py
@@ -770,7 +770,6 @@ def main():
             src               = dict(required=True, type='path'),
             original_basename = dict(required=False, type='str'), # used to handle 'dest is a directory' via template, a slight hack
             dest              = dict(required=True, type='path'),
-            copy              = dict(required=False, default=True, type='bool'),
             remote_src        = dict(required=False, default=False, type='bool'),
             creates           = dict(required=False, type='path'),
             list_files        = dict(required=False, default=False, type='bool'),
@@ -780,21 +779,19 @@ def main():
             validate_certs    = dict(required=False, default=True, type='bool'),
         ),
         add_file_common_args = True,
-        mutually_exclusive   = [("copy", "remote_src"),],
         # check-mode only works for zip files, we cover that later
         supports_check_mode = True,
     )
 
     src        = module.params['src']
     dest       = module.params['dest']
-    copy       = module.params['copy']
     remote_src = module.params['remote_src']
     file_args = module.load_file_common_arguments(module.params)
     # did tar file arrive?
     if not os.path.exists(src):
-        if not remote_src and copy:
+        if not remote_src:
             module.fail_json(msg="Source '%s' failed to transfer" % src)
-        # If copy=false, and src= contains ://, try and download the file to a temp directory.
+        # If remote_src=true, and src= contains ://, try and download the file to a temp directory.
         elif '://' in src:
             tempdir = os.path.dirname(os.path.realpath(__file__))
             package = os.path.join(tempdir, str(src.rsplit('/', 1)[1]))
diff --git a/lib/ansible/plugins/action/unarchive.py b/lib/ansible/plugins/action/unarchive.py
--- a/lib/ansible/plugins/action/unarchive.py
+++ b/lib/ansible/plugins/action/unarchive.py
@@ -53,7 +53,7 @@ def run(self, tmp=None, task_vars=None):
                 return result
             # We will take the information from copy and store it in
             # the remote_src var to use later in this file.
-            remote_src = not boolean(self._task.args.get('copy'))
+            self._task.args['remote_src'] = remote_src = not boolean(self._task.args.pop('copy'))
 
         if source is None or dest is None:
             result['failed'] = True
@@ -128,7 +128,7 @@ def run(self, tmp=None, task_vars=None):
             )
 
         # remove action plugin only key
-        for key in ('remote_src', 'decrypt'):
+        for key in ('decrypt',):
             if key in new_module_args:
                 del new_module_args[key]
 


Test patch: