Repository: ansible/ansible

Issue:
Ansible unarchive bug using exclude
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
unarchive
/usr/lib/python2.7/dist-packages/ansible/modules/files/unarchive.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
Xubuntu 14.04

##### SUMMARY
The unarchive module does not work correctly with the exclude parameter.  When I set the exclude parameter to a directory name followed by slash asterisk, the err property is an empty string and the exclusions are applied correctly, but the module fails with "Unexpected error when accessing exploded file: [Errno 2] No such file or directory:", followed by the first of the filenames that match the excluded wildcard, but are no longer in the destination directory.  

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1. Create a zip file called tester.zip with the following structure in the input directory (attempts to attach the sample zip repeatedly failed!)
\tester
\tester\top_level
\tester\top_level\bar
\tester\top_level\bar\baz.txt
\tester\top_level\foo
\tester\top_level\foo\static
\tester\top_level\foo\static\0.png
\tester\top_level\foo\static\1.png
\tester\top_level\foo\static\2.png
\tester\top_level\foo\static\3.png
2. Run the following playbook WITHOUT THE EXCLUDE to unarchive the zip into the output folder.
3. Delete one of the files in the output/tester/top_level/foo/static directory, e.g. 0.png
4. Run the following playbook AS-IS

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- name: Unpack zip file excluding one folder.
  hosts: localhost
  tasks:
    - name: Unpack zip file excluding one folder.
      unarchive: src=/input/tester.zip dest=output copy=no mode=0755 exclude="tester/top_level/foo/static/*
```

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expect the tester.zip to be extracted to the output directory, and NOT change the contents of the output/tester/top_level/static directory

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The final steps of unarchive.py are attempting to set permissions on files using the list of files in the archive, without considering the list of files that have been excluded, leading to the error.




<!--- Paste verbatim command output between quotes below -->
```
Using /etc/ansible/ansible.cfg as config file
 [WARNING]: provided hosts list is empty, only localhost is available


PLAYBOOK: unpacker.yml **********************************************************************************************************************************************************************************************************************
1 plays in unpacker.yml

PLAY [Unpack zip file excluding one folder.] ************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: vagrant
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323 `" && echo ansible-tmp-1498816759.12-19610887468323="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpqCUzfg TO /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323/ /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323/setup.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1498816759.12-19610887468323/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Unpack zip file excluding one folder.] ************************************************************************************************************************************************************************************************
task path: /home/vagrant/ansible_testing/unarchiver/unpacker.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: vagrant
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.6-57515879616580 `" && echo ansible-tmp-1498816759.6-57515879616580="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.6-57515879616580 `" ) && sleep 0'
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/files/stat.py
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600 `" && echo ansible-tmp-1498816759.71-144634990416600="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpk9Pkc8 TO /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600/stat.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600/ /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600/stat.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600/stat.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1498816759.71-144634990416600/" > /dev/null 2>&1 && sleep 0'
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/files/unarchive.py
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544 `" && echo ansible-tmp-1498816759.99-259119137497544="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmppQ8ul_ TO /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544/unarchive.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544/ /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544/unarchive.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544/unarchive.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1498816759.99-259119137497544/" > /dev/null 2>&1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/vagrant/.ansible/tmp/ansible-tmp-1498816759.6-57515879616580/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true, 
    "dest": "output", 
    "diff": {
        "prepared": ">f++++++.?? tester/top_level/foo/static/0.png\n"
    }, 
    "extract_results": {
        "cmd": [
            "/usr/bin/unzip", 
            "-o", 
            "input/tester.zip", 
            "-x", 
            "tester/top_level/foo/static/*", 
            "-d", 
            "output"
        ], 
        "err": "", 
        "out": "Archive:  input/tester.zip\n extracting: output/tester/top_level/bar/baz.txt  \n", 
        "rc": 0
    }, 
    "failed": true, 
    "gid": 1000, 
    "group": "vagrant", 
    "handler": "ZipArchive", 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "copy": false, 
            "creates": null, 
            "delimiter": null, 
            "dest": "output", 
            "directory_mode": null, 
            "exclude": [
                "tester/top_level/foo/static/*"
            ], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": "0755", 
            "original_basename": "tester.zip", 
            "owner": null, 
            "regexp": null, 
            "remote_src": false, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "input/tester.zip", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "mode": "0775", 
    "msg": "Unexpected error when accessing exploded file: [Errno 2] No such file or directory: 'output/tester/top_level/foo/static/0.png'", 
    "owner": "vagrant", 
    "size": 4096, 
    "src": "input/tester.zip", 
    "state": "directory", 
    "uid": 1000
}
	to retry, use: --limit @/home/vagrant/ansible_testing/unarchiver/unpacker.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   

```



Discussion:
cc @dagwieers @pileofrogs
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/modules/files/unarchive.py b/lib/ansible/modules/files/unarchive.py
--- a/lib/ansible/modules/files/unarchive.py
+++ b/lib/ansible/modules/files/unarchive.py
@@ -131,6 +131,7 @@
 import binascii
 import codecs
 import datetime
+import fnmatch
 import grp
 import os
 import platform
@@ -262,7 +263,11 @@ def files_in_archive(self, force_refresh=False):
         else:
             try:
                 for member in archive.namelist():
-                    if member not in self.excludes:
+                    if self.excludes:
+                        for exclude in self.excludes:
+                            if not fnmatch.fnmatch(member, exclude):
+                                self._files_in_archive.append(to_native(member))
+                    else:
                         self._files_in_archive.append(to_native(member))
             except:
                 archive.close()


Test patch:
diff --git a/test/integration/targets/unarchive/tasks/main.yml b/test/integration/targets/unarchive/tasks/main.yml
--- a/test/integration/targets/unarchive/tasks/main.yml
+++ b/test/integration/targets/unarchive/tasks/main.yml
@@ -17,7 +17,7 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 # Make sure we start fresh
 
-# Need unzup for unarchive module, and zip for archive creation.
+# Need unzip for unarchive module, and zip for archive creation.
 - name: Ensure zip and unzip is present to create test archive (yum)
   yum: name=zip,unzip state=latest
   when: ansible_pkg_mgr  ==  'yum'
@@ -26,11 +26,11 @@
   #  dnf: name=zip state=latest
   #  when: ansible_pkg_mgr  ==  'dnf'
 
-- name: Ensure zip & unzup is present to create test archive (apt)
+- name: Ensure zip & unzip is present to create test archive (apt)
   apt: name=zip,unzip state=latest
   when: ansible_pkg_mgr  ==  'apt'
 
-- name: Ensure zip & unzup is present to create test archive (pkg)
+- name: Ensure zip & unzip is present to create test archive (pkg)
   pkgng: name=zip,unzip state=present
   when: ansible_pkg_mgr  ==  'pkgng'
 
@@ -51,18 +51,18 @@
 
 # This gets around an unzip timestamp bug in some distributions
 # Recent unzip on Ubuntu and BSD will randomly round some timestamps up.
-# But that doesn't seem to happen when the timestamp has an even second. 
+# But that doesn't seem to happen when the timestamp has an even second.
 - name: Bug work around
   command: touch -t "201705111530.00" {{output_dir}}/foo-unarchive.txt {{output_dir}}/foo-unarchive-777.txt {{output_dir}}/FOO-UNAR.TXT
 # See Ubuntu bug 1691636: https://bugs.launchpad.net/ubuntu/+source/unzip/+bug/1691636
-# When these are fixed, this code should be removed.  
+# When these are fixed, this code should be removed.
 
 - name: prep a zip file
   shell: zip test-unarchive.zip foo-unarchive.txt foo-unarchive-777.txt chdir={{output_dir}}
 
 - name: add a file with Windows permissions to zip file
   shell: zip -k test-unarchive.zip FOO-UNAR.TXT chdir={{output_dir}}
-  
+
 - name: prep a subdirectory
   file: path={{output_dir}}/unarchive-dir state=directory
 
@@ -178,12 +178,32 @@
 - name: repeat the last request to verify no changes
   unarchive: src={{output_dir}}/test-unarchive.zip dest={{output_dir | expanduser}}/test-unarchive-zip remote_src=yes list_files=True
   register: unarchive03b
-  
+
 - name: verify that the task was not marked as changed
   assert:
     that:
       - "unarchive03b.changed == false"
 
+- name: "Create {{ output_dir }}/exclude directory"
+  file:
+    state: directory
+    path: "{{ output_dir }}/exclude"
+
+- name: Unpack zip file excluding one file.
+  unarchive:
+    src: "{{ output_dir }}/test-unarchive.zip"
+    dest: "{{ output_dir }}/exclude"
+    exclude: "foo-unarchive-*.txt"
+
+- name: verify that the file was unarchived
+  shell: find {{ output_dir }}/exclude chdir={{ output_dir }}
+  register: unarchive_dir01
+
+- name: verify that zip extraction excluded file
+  assert:
+    that:
+      - "'foo-unarchive-777.txt' not in unarchive_dir01.stdout"
+
 - name: remove our zip unarchive destination
   file: path={{output_dir}}/test-unarchive-zip state=absent
 
@@ -204,7 +224,7 @@
   when: unarchive04.stat.exists
 
 - name: try unarchiving to /tmp
-  unarchive: src={{output_dir}}/test-unarchive.tar.gz dest=/tmp remote_src=yes 
+  unarchive: src={{output_dir}}/test-unarchive.tar.gz dest=/tmp remote_src=yes
   register: unarchive05
 
 - name: verify that the file was marked as changed
@@ -251,7 +271,6 @@
 - name: create our unarchive destination
   file: path={{output_dir}}/test-unarchive-tar-gz state=directory
 
-
 - name: unarchive over existing extraction and set mode to 0644
   unarchive:
     src: "{{ output_dir }}/test-unarchive.tar.gz"
@@ -293,7 +312,6 @@
 - name: remove our tar.gz unarchive destination
   file: path={{ output_dir }}/test-unarchive-tar-gz state=absent
 
-
 - name: create our unarchive destination
   file: path={{output_dir}}/test-unarchive-zip state=directory
 
@@ -358,11 +376,9 @@
 - name: remove our zip unarchive destination
   file: path={{ output_dir }}/test-unarchive-zip state=absent
 
-
 - name: create our unarchive destination
   file: path={{output_dir}}/test-unarchive-tar-gz state=directory
 
-
 - name: create a directory with quotable chars
   file: path="{{ output_dir }}/test-quotes~root" state=directory
 
@@ -503,7 +519,6 @@
 - name: remove our tar.gz unarchive destination
   file: path={{ output_dir }}/test-unarchive-tar-gz state=absent
 
-
 - name: Create a file
   file:
     path: "{{ output_dir }}/test-unarchive-tar-gz"