Repository: ansible/ansible-modules-core

Issue:
Unarchive fails verifying non-ascii named files in zip archives
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

files/unarchive.py
##### ANSIBLE VERSION

```
ansible 2.2.0 (devel cb5675a29f) last updated 2016/08/12 17:26:42 (GMT +000)
  lib/ansible/modules/core: (devel 09cf141bc2) last updated 2016/08/12 16:49:29 (GMT +000)
  lib/ansible/modules/extras: (detached HEAD 61d5fe148c) last updated 2016/08/12 16:49:32 (GMT +000)
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

default
##### OS / ENVIRONMENT

Probably N/A

Controller: MacOS 10.11.6 & Ubuntu 14.04
Managed: Ubuntu 14.04
##### SUMMARY

The unarchive module fails while trying to verify the files in an unpacked archive if those files have non-ascii characters. This was discovered trying to download the GitHub Ansible repo zip archive, originally mis-reported as https://github.com/ansible/ansible/issues/17022.

The problem has been isolated and can be recreated with the attached [snowman-of-doom.zip](https://github.com/ansible/ansible-modules-core/files/415857/snowman-of-doom.zip) zip archive. 

This does not happen with tar archives.

If I'm reading the output correctly, it appears the `unzip` lists the included file (`snowman-of-doom/☃.txt`) as `snowman-of-doom/#U2603.txt`, but later ansible is looking for `snowman-of-doom/\\xe2\\x98\\x83.txt`.
##### STEPS TO REPRODUCE

Download the attached [snowman-of-doom.zip](https://github.com/ansible/ansible-modules-core/files/415857/snowman-of-doom.zip) and throw it in `/tmp`. 

Run the following playbook:

```

---

- hosts: all

  tasks:
    - name: Unpack zip archive
      unarchive: 
        src: /tmp/snowman-of-doom.zip
        dest: /tmp/

```
##### EXPECTED RESULTS

Archive unpacks.
##### ACTUAL RESULTS

Module failure.

```
Using /etc/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: ansible-17022.yml ****************************************************
1 plays in /vagrant/ansible-17022.yml

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
Using module file /tmp/ansible/lib/ansible/modules/core/system/setup.py
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464 `" && echo ansible-tmp-1471023184.68-228458518492464="` echo $HOME/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464 `" ) && sleep 0'"'"''
<172.28.128.11> PUT /tmp/tmpHFjAWs TO /home/vagrant/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464/setup.py
<172.28.128.11> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r '[172.28.128.11]'
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464/ /home/vagrant/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464/setup.py && sleep 0'"'"''
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 172.28.128.11 '/bin/sh -c '"'"'/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464/setup.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1471023184.68-228458518492464/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [172.28.128.11]

TASK [Unpack zip archive] ******************************************************
task path: /vagrant/ansible-17022.yml:6
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877 `" && echo ansible-tmp-1471023185.34-257893414845877="` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877 `" ) && sleep 0'"'"''
Using module file /tmp/ansible/lib/ansible/modules/core/files/stat.py
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888 `" && echo ansible-tmp-1471023185.43-14079921266888="` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888 `" ) && sleep 0'"'"''
<172.28.128.11> PUT /tmp/tmpgpURHY TO /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888/stat.py
<172.28.128.11> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r '[172.28.128.11]'
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888/ /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888/stat.py && sleep 0'"'"''
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 172.28.128.11 '/bin/sh -c '"'"'/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888/stat.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1471023185.43-14079921266888/" > /dev/null 2>&1 && sleep 0'"'"''
<172.28.128.11> PUT /tmp/snowman-of-doom.zip TO /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source
<172.28.128.11> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r '[172.28.128.11]'
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/ /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source && sleep 0'"'"''
Using module file /tmp/ansible/lib/ansible/modules/core/files/unarchive.py
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760 `" && echo ansible-tmp-1471023185.78-244527971992760="` echo $HOME/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760 `" ) && sleep 0'"'"''
<172.28.128.11> PUT /tmp/tmpFL4nnr TO /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760/unarchive.py
<172.28.128.11> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r '[172.28.128.11]'
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760/ /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760/unarchive.py && sleep 0'"'"''
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 172.28.128.11 '/bin/sh -c '"'"'/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760/unarchive.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1471023185.78-244527971992760/" > /dev/null 2>&1 && sleep 0'"'"''
<172.28.128.11> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.128.11> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r 172.28.128.11 '/bin/sh -c '"'"'rm -f -r /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/ > /dev/null 2>&1 && sleep 0'"'"''
fatal: [172.28.128.11]: FAILED! => {
    "changed": true, 
    "dest": "/tmp/", 
    "diff": {
        "prepared": ">d++++++.?? snowman-of-doom/\n>f++++++.?? snowman-of-doom/#U2603.txt\n"
    }, 
    "extract_results": {
        "cmd": "/usr/bin/unzip -o \"/home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source\" \"snowman-of-doom/\" \"snowman-of-doom/#U2603.txt\" -d \"/tmp/\"", 
        "err": "", 
        "out": "Archive:  /home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source\n   creating: /tmp/snowman-of-doom/\n extracting: /tmp/snowman-of-doom/#U2603.txt  \n", 
        "rc": 0
    }, 
    "failed": true, 
    "gid": 0, 
    "group": "root", 
    "handler": "ZipArchive", 
    "invocation": {
        "module_args": {
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/tmp/", 
            "directory_mode": null, 
            "exclude": [], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "snowman-of-doom.zip", 
            "owner": null, 
            "regexp": null, 
            "remote_src": false, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "mode": "01777", 
    "msg": "Unexpected error when accessing exploded file: [Errno 2] No such file or directory: '/tmp/snowman-of-doom/\\xe2\\x98\\x83.txt'", 
    "owner": "root", 
    "size": 4096, 
    "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1471023185.34-257893414845877/source", 
    "state": "directory", 
    "uid": 0
}

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @/vagrant/ansible-17022.retry

PLAY RECAP *********************************************************************
172.28.128.11              : ok=1    changed=0    unreachable=0    failed=1   
```



Discussion:
<!-- @joemaller: Thanks for filing this bug report! -->

@dagwieers, @pileofrogs, ping. This issue is waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

Again, the problem here has to do with the fact that we use external commands and scrape the output (rather than use a pure-python implementation). The reason why there is a difference in interpretation is because listing files in the ZIP archive is done using pure-python means (**zipfile**) whereas the extracting is performed using the `unzip` command.

The right solution is to reimplement unarchive as a pure-python module, the workaround is to ensure that the scraped `unzip` output is converted into proper unicode filenames. (so #U2603 becomes &#x2603;).


Resolving patch:
diff --git a/files/unarchive.py b/files/unarchive.py
--- a/files/unarchive.py
+++ b/files/unarchive.py
@@ -133,6 +133,7 @@
 import binascii
 import codecs
 from zipfile import ZipFile, BadZipfile
+from ansible.module_utils._text import to_text
 
 try:  # python 3.3+
     from shlex import quote
@@ -347,7 +348,7 @@ def is_unarchived(self):
             version = pcs[1]
             ostype = pcs[2]
             size = int(pcs[3])
-            path = pcs[7]
+            path = to_text(pcs[7], errors='surrogate_or_strict')
 
             # Skip excluded files
             if path in self.excludes:
@@ -592,7 +593,7 @@ def files_in_archive(self, force_refresh=False):
         if self.excludes:
             cmd.extend([ '--exclude=' + quote(f) for f in self.excludes ])
         cmd.extend([ '-f', self.src ])
-        rc, out, err = self.module.run_command(cmd)
+        rc, out, err = self.module.run_command(cmd, cwd=self.dest, environ_update=dict(LANG='C', LC_ALL='C', LC_MESSAGES='C'))
         if rc != 0:
             raise UnarchiveError('Unable to list files in the archive')
 
@@ -621,7 +622,7 @@ def is_unarchived(self):
         if self.excludes:
             cmd.extend([ '--exclude=' + quote(f) for f in self.excludes ])
         cmd.extend([ '-f', self.src ])
-        rc, out, err = self.module.run_command(cmd)
+        rc, out, err = self.module.run_command(cmd, cwd=self.dest, environ_update=dict(LANG='C', LC_ALL='C', LC_MESSAGES='C'))
 
         # Check whether the differences are in something that we're
         # setting anyway
@@ -670,7 +671,7 @@ def unarchive(self):
         if self.excludes:
             cmd.extend([ '--exclude=' + quote(f) for f in self.excludes ])
         cmd.extend([ '-f', self.src ])
-        rc, out, err = self.module.run_command(cmd, cwd=self.dest)
+        rc, out, err = self.module.run_command(cmd, cwd=self.dest, environ_update=dict(LANG='C', LC_ALL='C', LC_MESSAGES='C'))
         return dict(cmd=cmd, rc=rc, out=out, err=err)
 
     def can_handle_archive(self):
@@ -741,9 +742,6 @@ def main():
         supports_check_mode = True,
     )
 
-    # We screenscrape a huge amount of commands so use C locale anytime we do
-    module.run_command_environ_update = dict(LANG='C', LC_ALL='C', LC_MESSAGES='C', LC_CTYPE='C')
-
     src        = os.path.expanduser(module.params['src'])
     dest       = os.path.expanduser(module.params['dest'])
     copy       = module.params['copy']


Test patch: