Repository: ansible/ansible-modules-core

Issue:
Failure to Handle ZipFile Issues with Older Versions of Python
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

unarchive
##### ANSIBLE VERSION

```
ansible 2.0.1.0
  config file = /opt/ansible/sysadmin/ansible.cfg
  configured module search path = /usr/share/ansible/
```
##### CONFIGURATION

Nothing noteworthy.
##### OS / ENVIRONMENT

Ansible is running from a CentOS 7 machine
Affected clients are running RHEL 5
##### SUMMARY

We are building code on a RHEL 6 system, using a relatively contemporary version of zip and unzip utilities. After building, the resulting artifacts exists as a zip file which we use Ansible to deploy.

When deploying to RHEL 5 systems, we first ran into an issue where the shipped version of unzip choked on the file. We built a RHEL 5 package for unzip 6.0 (taken from a RHEL 6 source RPM), and then ran into an issue where Python 2.4 (as shipped with RHEL 5) was also failing to validate the ZipFile, as when called from the files_in_archive function:

```
    def files_in_archive(self, force_refresh=False):
        if self._files_in_archive and not force_refresh:
            return self._files_in_archive

        archive = ZipFile(self.src)
        try:
            self._files_in_archive = archive.namelist()
        except:
            raise UnarchiveError('Unable to list files in the archive')

        return self._files_in_archive
```

This causes our code deployment to fail.
##### STEPS TO REPRODUCE
1. Create a zip file on a host running a modern version of zip
2. Use the unarchive module to deploy the file to an older system (that has a newer version of unzip).
3. Wait for failure

```
  - name: Unpack app on host
    unarchive: src={{ destinationDirectory }}/{{ project }}-{{ version }}.zip 
                 dest={{ destinationDirectory }}
                 copy=no
```
##### EXPECTED RESULTS

The file explodes properly
##### ACTUAL RESULTS

The explode fails

```
TASK [Unpack app on host] ******************************************************
fatal: [nw-hweb-d10]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n  File \"/apps/home/exampleuser/.ansible/tmp/ansible-tmp-1461759972.63-110735257718247/unarchive\", line 3108, in ?\r\n    main()\r\n  File \"/apps/home/exampleuser/.ansible/tmp/ansible-tmp-1461759972.63-110735257718247/unarchive\", line 336, in main\r\n    for filename in handler.files_in_archive:\r\n  File \"/apps/home/engadmin/.ansible/tmp/ansible-tmp-1461759972.63-110735257718247/unarchive\", line 120, in files_in_archive\r\n    archive = ZipFile(self.src)\r\n  File \"/usr/lib64/python2.4/zipfile.py\", line 210, in __init__\r\n    self._GetContents()\r\n  File \"/usr/lib64/python2.4/zipfile.py\", line 230, in _GetContents\r\n    self._RealGetContents()\r\n  File \"/usr/lib64/python2.4/zipfile.py\", line 262, in _RealGetContents\r\n    raise BadZipfile, \"Bad magic number for central directory\"\r\nzipfile.BadZipfile: Bad magic number for central directory\r\n", "msg": "MODULE FAILURE", "parsed": false}
```



Discussion:
It's worth noting that we tried using a .tar.gz file with the same contents in place of the zip file, as a means to avoid having to add my (admittedly dirty) patch to Ansible, and ran into a similar problem, though this time, the actual problem with the tarball made itself far more obvious:

```
Unexpected error when accessing exploded file: [Errno 2] No such file or directory: '/apps/rc-static/./vehicles/rcc/C/renault_m\\\\303\\\\243\\\\302\\\\251gane_diesel.jpg'
```

Python 2.4 does not like these special/accented characters. My patch works for this as well, as it were, but I wasn't expecting the problem to persist with a different archive file format.

@selble I thought the tar.gz problem might be the same as https://github.com/ansible/ansible/issues/14283  Which I've been unable to reproduce.  I wasn't able to reproduce it using the tar.gz from there to a rhel5 box, though so I'm not sure if the tar.gz problem is the same or different.  If you can get me a tar.gz with maybe just that filename in it I could try and reproduce with yours.

The zip problem might be separate.. I'll try and reproduce that tomorrow.

Thanks @abadger, I appreciate you looking into this one.

I can trigger it with the following tarball:

[python_pain.tar.gz](https://github.com/ansible/ansible-modules-core/files/242431/python_pain.tar.gz)

With the following command:

```
ansible -vvv "a-test-host" -m unarchive -s -a "src=/home/selble/python_pain.tar.gz dest=/home/selble/tmp"
```

I get the following result:

```
<a-test-host> ESTABLISH SSH CONNECTION FOR USER: adminuser
<a-test-host> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=1800s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=adminuser -o ConnectTimeout=10 -o ControlPath=/home/selble/.ansible/cp/ansible-ssh-%h-%p-%r -tt a-test-host '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-csryggdkjdcenbwkdemtmqtwzmpzrwhw; /bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'rc=flag; [ -r /home/selble/tmp ] || rc=2; [ -f /home/selble/tmp ] || rc=1; [ -d /home/selble/tmp ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc}  "/home/selble/tmp && exit 0; (python -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();
afile = open("'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'/home/selble/tmp'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha();
afile = open("'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'/home/selble/tmp'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'0  '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'/home/selble/tmp)'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"''"'"''
<a-test-host> PUT /home/selble/python_pain.tar.gz TO /tmp/.ansible-adminuser/ansible-tmp-1461933836.69-99913198566058/source
<a-test-host> SSH: EXEC scp -C -o ControlMaster=auto -o ControlPersist=1800s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=adminuser -o ConnectTimeout=10 -o ControlPath=/home/selble/.ansible/cp/ansible-ssh-%h-%p-%r /home/selble/python_pain.tar.gz '[a-test-host]:/tmp/.ansible-adminuser/ansible-tmp-1461933836.69-99913198566058/source'
<a-test-host> ESTABLISH SSH CONNECTION FOR USER: adminuser
<a-test-host> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=1800s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=adminuser -o ConnectTimeout=10 -o ControlPath=/home/selble/.ansible/cp/ansible-ssh-%h-%p-%r a-test-host '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-yjcrfcyjbbjtvcuikebolljvnvuyogtt; /bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"''"'"''
a-test-host | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/home/selble/tmp", 
            "directory_mode": null, 
            "follow": false, 
            "force": null, 
            "group": null, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "python_pain.tar.gz", 
            "owner": null, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "/tmp/.ansible-adminuser/ansible-tmp-1461933836.69-99913198566058/source"
        }
    },
    "msg": "Unexpected error when accessing exploded file: [Errno 2] No such file or directory: '/home/selble/tmp/python/pain/renault_m\\\\303\\\\243\\\\302\\\\251gane_diesel.jpg'"
}
```

In both cases, the problem is triggered when the native Python modules are invoked in unarchive.py, and not when the system binaries are called (at least after upgrading unzip). One could certainly argue that this is a flaw in Python 2.4 that Red Hat almost certainly won't change at this point, but it'd be nice if Ansible could accommodate it.

My proposed patch is quite quick and dirty, but it is better than failing on older systems that would otherwise work, if it weren't for the old version of Python.

As a temporary workaround for the tarball traceback -- if you set your locale (LANG in the environment or module_lang in the ansible.cfg) to a utf8 locale, I believe it will work.  I was able to get your tarball to work like this:

```
$ LANG=en_US.utf8 ansible -vvv -i 'rhel5,' '*' -m unarchive -a "src=python_pain.tar.gz dest=/var/tmp/unarchive"
```

and fail like this:

```
$ LANG=C ansible -vvv -i 'rhel5,' '*' -m unarchive -a "src=python_pain.tar.gz dest=/var/tmp/unarchive"
```

Looking for a solution in code, now.

For the tar problem, looks like there's several ways to fix it for gtar:
- LC_ALL=*utf8 will output the proper strings
- --quoting-style=literal will output the raw bytes even if the locale doesn't support it
- inside of python, detect the escaped strings and convert them from octal to utf8 bytes

For bsdtar (checked on freebsd10) only the latter appears to be an option.

Thanks to @hsoft, the tar problem should now be fixed in devel and the stable-2.1 branches.  Next task -- look further into the problem with zipfiles.

@selble could you provide me with a sample zipfile that fails on rhel5 as well?  I tested a few zip files created on Fedora 23 but so far all of those work with RHEL5's python-2.4 (and with RHEL5's zip as well).  So I'm obviously doing something wrong in creating the data to test with.

@abadger, I apologize for the delay in replying. I've attempted to create a redacted zip file that would fail (thinking perhaps it was the non-ASCII characters that were causing issues), but have been unable to do so, and i cannot provide the actual file that's failing, on account of the data within it.

Okay, I might have a lead: https://bugs.python.org/issue3997   Possibly it's the number of files in the archive that's provoking the bad behaviour.


Resolving patch:
diff --git a/files/unarchive.py b/files/unarchive.py
--- a/files/unarchive.py
+++ b/files/unarchive.py
@@ -123,7 +123,7 @@
 import datetime
 import time
 import binascii
-from zipfile import ZipFile
+from zipfile import ZipFile, BadZipfile
 import tarfile
 import subprocess
 
@@ -174,17 +174,41 @@ def _permstr_to_octal(self, modestr, umask):
 #                    mode += 2**(9+j)
         return ( mode & ~umask )
 
+    def _legacy_file_list(self, force_refresh=False):
+        unzip_bin = self.module.get_bin_path('unzip')
+        if not unzip_bin:
+            raise UnarchiveError('Python Zipfile cannot read %s and unzip not found' % self.src)
+
+        rc, out, err = self.module.run_command([unzip_bin, '-v', self.src])
+        if rc:
+            raise UnarchiveError('Neither python zipfile nor unzip can read %s' % self.src)
+
+        for line in out.splitlines()[3:-2]:
+            fields = line.split(None, 7)
+            self._files_in_archive.append(fields[7])
+            self._infodict[fields[7]] = long(fields[6])
+
     def _crc32(self, path):
         if self._infodict:
             return self._infodict[path]
 
-        archive = ZipFile(self.src)
         try:
-            for item in archive.infolist():
-                self._infodict[item.filename] = long(item.CRC)
-        except:
-            archive.close()
-            raise UnarchiveError('Unable to list files in the archive')
+            archive = ZipFile(self.src)
+        except BadZipfile:
+            e = get_exception()
+            if e.args[0].lower().startswith('bad magic number'):
+                # Python2.4 can't handle zipfiles with > 64K files.  Try using
+                # /usr/bin/unzip instead
+                self._legacy_file_list()
+            else:
+                raise
+        else:
+            try:
+                for item in archive.infolist():
+                    self._infodict[item.filename] = long(item.CRC)
+            except:
+                archive.close()
+                raise UnarchiveError('Unable to list files in the archive')
 
         return self._infodict[path]
 
@@ -194,16 +218,26 @@ def files_in_archive(self, force_refresh=False):
             return self._files_in_archive
 
         self._files_in_archive = []
-        archive = ZipFile(self.src)
         try:
-            for member in archive.namelist():
-                if member not in self.excludes:
-                    self._files_in_archive.append(member)
-        except:
-            archive.close()
-            raise UnarchiveError('Unable to list files in the archive')
+            archive = ZipFile(self.src)
+        except BadZipfile:
+            e = get_exception()
+            if e.args[0].lower().startswith('bad magic number'):
+                # Python2.4 can't handle zipfiles with > 64K files.  Try using
+                # /usr/bin/unzip instead
+                self._legacy_file_list(force_refresh)
+            else:
+                raise
+        else:
+            try:
+                for member in archive.namelist():
+                    if member not in self.excludes:
+                        self._files_in_archive.append(member)
+            except:
+                archive.close()
+                raise UnarchiveError('Unable to list files in the archive')
 
-        archive.close()
+            archive.close()
         return self._files_in_archive
 
     def is_unarchived(self):


Test patch: