Repository: ansible-collections/community.general

Issue:
archive module reports no change when it changes the zip
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY

There are some circumstances where the `archive` module will report that the Ansible task has not changed, but the module has correctly changed the contents of the `.zip` file.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
archive

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```
ansible 2.10.1
  config file = /home/ec2-user/.ansible.cfg
  configured module search path = ['/home/ec2-user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ec2-user/env/local/lib/python3.6/dist-packages/ansible
  executable location = /home/ec2-user/env/bin/ansible
  python version = 3.6.12 (default, Aug 31 2020, 18:56:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```
ANSIBLE_PIPELINING(/home/ec2-user/.ansible.cfg) = True
DEFAULT_LOCAL_TMP(/home/ec2-user/.ansible.cfg) = /dev/shm/ansible/tmp_local/ansible-local-1069740w52z4_
```

##### OS / ENVIRONMENT

Amazon Linux


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```
---
- hosts: localhost
  connection: local
  vars:
    zip_dest: "/tmp/zip.zip"
    zip_src: "/tmp/zip_source"
    zip_extracted: "/tmp/zip_extracted"
  tasks:
  - name: reset after previous run
    with_items:
    - "{{ zip_dest }}"
    - "{{ zip_src }}"
    - "{{ zip_extracted }}"
    file:
      state: absent
      path: "{{ item }}"
    
  
  - name: create directory to be zipped up
    file:
      state: directory
      path: "{{ zip_src }}"
      
  - name: create a file to be zipped up
    copy:
      content: "hello"
      dest: "{{ zip_src }}/hello_1.txt"
      
  - name: create zip the first time
    archive:
      path: "{{ zip_src }}"
      dest: "{{ zip_dest }}"
      
  - name: rename the source file
    shell: mv {{ zip_src }}/hello_1.txt {{ zip_src }}/hello_2.txt
    
  - name: zip again with renamed contents
    archive:
      path: "{{ zip_src }}"
      dest: "{{ zip_dest }}"
    register: zip2
    
  - name: create directory to unzip into
    file:
      state: directory
      path: "{{ zip_extracted }}"
    
  - name: unzip to check the change is there
    unarchive:
      src: "{{ zip_dest }}"
      dest: "{{ zip_extracted }}"
    
  - name: check the contents of the zip has changed
    assert:
      that:
      - '"{{ zip_extracted }}/zip_source/hello_2.txt" is exists'
      - '"{{ zip_extracted }}/zip_source/hello_1.txt" is not exists'
    
  - name: check previous task is marked as changed
    assert:
      that:
      - zip2 is changed
      
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS

The playbook should run without error.

##### ACTUAL RESULTS

The playbook fails because of the final assertion.
The contents of the zip has changed, but the task is not marked as changed.

Note that if you add a trailing slash to the path being zipped up, the expected result is observed in this MWE. (Although I have seen it happen in my full, non-minimal playbook, with the trailing slash.)


Discussion:
Files identified in the description:
* [`plugins/modules/files/archive.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/files/archive.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @bendoh
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Archive module checks if the file has been changed by these methods:
Size:
https://github.com/ansible-collections/community.general/blob/f7656ac0d3a82845dd12accc804cbe0a66d8b9c1/plugins/modules/files/archive.py#L459-L461
https://github.com/ansible-collections/community.general/blob/f7656ac0d3a82845dd12accc804cbe0a66d8b9c1/plugins/modules/files/archive.py#L538-L540
File change:
https://github.com/ansible-collections/community.general/blob/f7656ac0d3a82845dd12accc804cbe0a66d8b9c1/plugins/modules/files/archive.py#L562-L563
which are `secontext`, `owner`, `group`, `mode`, `attributes`: 
https://github.com/ansible/ansible/blob/caba47dd3f3f37e989357bce676e420ff8b6a86f/lib/ansible/module_utils/basic.py#L1400-L1417

So the contents of the archive isn't being checked right now and I don't know if it's worthy to check the contents or not.
Ok, so a rename of a source file means the size of the archive hasn't changed, and that's why it's marked as unchanged?

I do feel strongly that it would be great to modify the module to detect at least file renames. This bug has caused me many hours of grief. (e.g. find bug in my code, change my code, run my playbook to deploy the change, bug still exists, I think my bug fix didn't work, but actually it would have, but my playbook didn't actually deploy the new code.)

Would there be an easy way to compare the list of files within the zip, before and after?

The module still does do the change, it just doesn't report it. So how does it know to do the change? Am I reading this bit correctly:

https://github.com/ansible-collections/community.general/blob/f7656ac0d3a82845dd12accc804cbe0a66d8b9c1/plugins/modules/files/archive.py#L395-L398

Are the files re-written into the zip even if they did not change?

Could we hash the zip before and after? Or would that be too slow?

> So the contents of the archive isn't being checked right now and I don't know if it's worthy to check the contents or not.

It's probably a good idea to at least compare the list of files in it, or even the file sizes. That information is easy to get. Also I think zip files have CRC32 checksums, which can also be quickly compared.

Doing a full binary comparsion is something that should not be done by default IMO, but it would be nice to be able to control this with an option.
Ok. I'll try to write a PR(s) to:

1. Compare the file list
2. Compare the hash, but not by default
3. Add a note to the documentation that `changed` has some false negatives.
> Ok, so a rename of a source file means the size of the archive hasn't changed, and that's why it's marked as unchanged?

There is no difference in size and file owner, group, secontext, etc. of the archive that has been created in your examples (`hello_1` & `hello_2`), so yes that's why it's marked as unchanged. I checked locally and if you just rename it to `hello` instead of `hello_1` it would mark it as changed because the size of created archive will change.

> Are the files re-written into the zip even if they did not change?

I think this module just archives the files and then check if the size and other attributes of the created archive has been changed or not, it would mark it as unchanged when these are not changed, but it would definitely archive the sources. You can check the created archives in your example to verify this.
I'm trying to write a PR for this, but I can't figure out how to run a playbook using my local, modified checkout of this repo, rather than the stable pre-installed version.

```
cd ~
mkdir collections
cd collections
git clone git@github.com:mdavis-xyz/community.general.git
cd ~
export COLLECTIONS_PATH=$(pwd)/collections
vim collections/community.general/.../archive.py # add an `assert False` line to main()
vim test.yaml # paste in playbook above
ansible-playbook test.yaml
```

It still runs as if it hasn't picked up my modifications.

How do I *use* a local, modified copy of `collections.general`?
you need to check out the repository as ~/collections/ansible_collections/community/general/, not as ~/collections/community.general/ as you're doing right now. (See also https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#contributing-to-collections)
Ah thanks. Yep that works now.

I've raised #1231 to improve the documentation for this.

Resolving patch:
diff --git a/plugins/modules/files/archive.py b/plugins/modules/files/archive.py
--- a/plugins/modules/files/archive.py
+++ b/plugins/modules/files/archive.py
@@ -182,6 +182,7 @@
 from fnmatch import fnmatch
 from sys import version_info
 from traceback import format_exc
+from zlib import crc32
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
 from ansible.module_utils.common.text.converters import to_bytes, to_native
@@ -234,10 +235,6 @@ def expand_paths(paths):
     return expanded_path, is_globby
 
 
-def legacy_filter(path, exclusion_patterns):
-    return matches_exclusion_patterns(path, exclusion_patterns)
-
-
 def matches_exclusion_patterns(path, exclusion_patterns):
     return any(fnmatch(path, p) for p in exclusion_patterns)
 
@@ -313,6 +310,7 @@ def __init__(self, module):
         if self.remove:
             self._check_removal_safety()
 
+        self.original_checksums = self.destination_checksums()
         self.original_size = self.destination_size()
 
     def add(self, path, archive_name):
@@ -377,8 +375,16 @@ def add_targets(self):
                 msg='Errors when writing archive at %s: %s' % (_to_native(self.destination), '; '.join(self.errors))
             )
 
-    def compare_with_original(self):
-        self.changed |= self.original_size != self.destination_size()
+    def is_different_from_original(self):
+        if self.original_checksums is None:
+            return self.original_size != self.destination_size()
+        else:
+            return self.original_checksums != self.destination_checksums()
+
+    def destination_checksums(self):
+        if self.destination_exists() and self.destination_readable():
+            return self._get_checksums(self.destination)
+        return None
 
     def destination_exists(self):
         return self.destination and os.path.exists(self.destination)
@@ -494,6 +500,10 @@ def open(self):
     def _add(self, path, archive_name):
         pass
 
+    @abc.abstractmethod
+    def _get_checksums(self, path):
+        pass
+
 
 class ZipArchive(Archive):
     def __init__(self, module):
@@ -513,9 +523,18 @@ def open(self):
         self.file = zipfile.ZipFile(_to_native_ascii(self.destination), 'w', zipfile.ZIP_DEFLATED, True)
 
     def _add(self, path, archive_name):
-        if not legacy_filter(path, self.exclusion_patterns):
+        if not matches_exclusion_patterns(path, self.exclusion_patterns):
             self.file.write(path, archive_name)
 
+    def _get_checksums(self, path):
+        try:
+            archive = zipfile.ZipFile(_to_native_ascii(path), 'r')
+            checksums = set((info.filename, info.CRC) for info in archive.infolist())
+            archive.close()
+        except zipfile.BadZipfile:
+            checksums = set()
+        return checksums
+
 
 class TarArchive(Archive):
     def __init__(self, module):
@@ -554,13 +573,35 @@ def py27_filter(tarinfo):
             return None if matches_exclusion_patterns(tarinfo.name, self.exclusion_patterns) else tarinfo
 
         def py26_filter(path):
-            return legacy_filter(path, self.exclusion_patterns)
+            return matches_exclusion_patterns(path, self.exclusion_patterns)
 
         if PY27:
             self.file.add(path, archive_name, recursive=False, filter=py27_filter)
         else:
             self.file.add(path, archive_name, recursive=False, exclude=py26_filter)
 
+    def _get_checksums(self, path):
+        try:
+            if self.format == 'xz':
+                with lzma.open(_to_native_ascii(path), 'r') as f:
+                    archive = tarfile.open(fileobj=f)
+                    checksums = set((info.name, info.chksum) for info in archive.getmembers())
+                    archive.close()
+            else:
+                archive = tarfile.open(_to_native_ascii(path), 'r|' + self.format)
+                checksums = set((info.name, info.chksum) for info in archive.getmembers())
+                archive.close()
+        except (lzma.LZMAError, tarfile.ReadError, tarfile.CompressionError):
+            try:
+                # The python implementations of gzip, bz2, and lzma do not support restoring compressed files
+                # to their original names so only file checksum is returned
+                f = self._open_compressed_file(_to_native_ascii(path), 'r')
+                checksums = set([(b'', crc32(f.read()))])
+                f.close()
+            except Exception:
+                checksums = set()
+        return checksums
+
 
 def get_archive(module):
     if module.params['format'] == 'zip':
@@ -603,7 +644,7 @@ def main():
         else:
             archive.add_targets()
             archive.destination_state = STATE_INCOMPLETE if archive.has_unfound_targets() else STATE_ARCHIVED
-            archive.compare_with_original()
+            archive.changed |= archive.is_different_from_original()
             if archive.remove:
                 archive.remove_targets()
     else:
@@ -613,7 +654,7 @@ def main():
         else:
             path = archive.paths[0]
             archive.add_single_target(path)
-            archive.compare_with_original()
+            archive.changed |= archive.is_different_from_original()
             if archive.remove:
                 archive.remove_single_target(path)
 


Test patch:
diff --git a/tests/integration/targets/archive/tests/idempotency.yml b/tests/integration/targets/archive/tests/idempotency.yml
--- a/tests/integration/targets/archive/tests/idempotency.yml
+++ b/tests/integration/targets/archive/tests/idempotency.yml
@@ -19,12 +19,12 @@
     format: "{{ format }}"
   register: file_content_idempotency_after
 
-# After idempotency fix result will be reliably changed for all formats
 - name: Assert task status is changed - file content idempotency ({{ format }})
   assert:
     that:
-      - file_content_idempotency_after is not changed
-  when: "format in ('tar', 'zip')"
+      - file_content_idempotency_after is changed
+  # Only ``zip`` archives are guaranteed to compare file content checksums rather than header checksums
+  when: "format == 'zip'"
 
 - name: Remove archive - file content idempotency ({{ format }})
   file:
@@ -54,12 +54,10 @@
     format: "{{ format }}"
   register: file_name_idempotency_after
 
-# After idempotency fix result will be reliably changed for all formats
 - name: Check task status - file name idempotency ({{ format }})
   assert:
     that:
-      - file_name_idempotency_after is not changed
-  when: "format in ('tar', 'zip')"
+      - file_name_idempotency_after is changed
 
 - name: Remove archive - file name idempotency ({{ format }})
   file:
@@ -89,12 +87,12 @@
     format: "{{ format }}"
   register: single_file_content_idempotency_after
 
-# After idempotency fix result will be reliably changed for all formats
 - name: Assert task status is changed - single file content idempotency ({{ format }})
   assert:
     that:
-      - single_file_content_idempotency_after is not changed
-  when: "format in ('tar', 'zip')"
+      - single_file_content_idempotency_after is changed
+  # ``tar`` archives are not guaranteed to identify changes to file content if the file meta properties are unchanged.
+  when: "format != 'tar'"
 
 - name: Remove archive - single file content idempotency ({{ format }})
   file:
@@ -125,11 +123,12 @@
   register: single_file_name_idempotency_after
 
 
-# After idempotency fix result will be reliably changed for all formats
+# The gz, bz2, and xz formats do not store the original file name
+# so it is not possible to identify a change in this scenario.
 - name: Check task status - single file name idempotency ({{ format }})
   assert:
     that:
-      - single_file_name_idempotency_after is not changed
+      - single_file_name_idempotency_after is changed
   when: "format in ('tar', 'zip')"
 
 - name: Remove archive - single file name idempotency ({{ format }})