Repository: ansible-collections/community.aws

Issue:
s3_sync etag calculation is wrong because use hardcoded chunk_size
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
The s3_sync module reports wrong etags compared to the one calculated from AWS S3 on server side.
The etag algorithm is based on the chunk_size used during multipart upload. The s3_sync module uses the boto3 defaults for the S3 Transfer, but the function that calculates the local etags does not use those defaults (it uses an hardcoded value instead)


##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
plugins/modules/s3_sync.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.10.2
  config file = None
  configured module search path = ['/Users/chiesa/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/chiesa/.local/pipx/venvs/ansible/lib/python3.8/site-packages/ansible
  executable location = /Users/chiesa/.local/bin/ansible
  python version = 3.8.2 (default, May  1 2020, 13:24:17) [Clang 11.0.3 (clang-1103.0.32.59)]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
no changes 

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Mac OS 10.15.7

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Upload some folder with s3_sync module and compare the local_etag calculated by the module with the S3 remote tags (calculated with the chunk_size used by boto3 - underlying library to tranfer the file)


<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
The local_etags of the file just uploaded should match the etags on S3



##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
The local_etags of the files just uploaded are mismatching S3 etags due to the wrong chunk_size hardcoded in the module:
<!--- Paste verbatim command output between quotes -->
```paste below
DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024
```



Discussion:

Resolving patch:
diff --git a/plugins/modules/s3_sync.py b/plugins/modules/s3_sync.py
--- a/plugins/modules/s3_sync.py
+++ b/plugins/modules/s3_sync.py
@@ -237,7 +237,10 @@
 
 try:
     import botocore
+    from boto3.s3.transfer import TransferConfig
+    DEFAULT_CHUNK_SIZE = TransferConfig().multipart_chunksize
 except ImportError:
+    DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024
     pass  # Handled by AnsibleAWSModule
 
 from ansible.module_utils._text import to_text
@@ -270,10 +273,6 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with calculate_multipart_etag.  If not, see <http://www.gnu.org/licenses/>.
-
-DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024
-
-
 def calculate_multipart_etag(source_path, chunk_size=DEFAULT_CHUNK_SIZE):
     """
     calculates a multipart upload etag for amazon s3


Test patch:
diff --git a/tests/integration/targets/s3_sync/aliases b/tests/integration/targets/s3_sync/aliases
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/aliases
@@ -0,0 +1,3 @@
+cloud/aws
+shippable/aws/group1
+
diff --git a/tests/integration/targets/s3_sync/files/test1.txt b/tests/integration/targets/s3_sync/files/test1.txt
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/files/test1.txt
@@ -0,0 +1 @@
+test1
\ No newline at end of file
diff --git a/tests/integration/targets/s3_sync/files/test2.yml b/tests/integration/targets/s3_sync/files/test2.yml
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/files/test2.yml
@@ -0,0 +1,2 @@
+---
+test2: example
diff --git a/tests/integration/targets/s3_sync/files/test3.json b/tests/integration/targets/s3_sync/files/test3.json
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/files/test3.json
@@ -0,0 +1,3 @@
+{
+  "test3": "value"
+}
\ No newline at end of file
diff --git a/tests/integration/targets/s3_sync/meta/main.yml b/tests/integration/targets/s3_sync/meta/main.yml
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/meta/main.yml
@@ -0,0 +1,3 @@
+dependencies:
+  - prepare_tests
+  - setup_ec2
diff --git a/tests/integration/targets/s3_sync/tasks/main.yml b/tests/integration/targets/s3_sync/tasks/main.yml
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/s3_sync/tasks/main.yml
@@ -0,0 +1,108 @@
+---
+- name: S3 bucket creation
+  collections:
+    - amazon.aws
+    - community.general
+  module_defaults:
+    group/aws:
+      aws_access_key: '{{ aws_access_key }}'
+      aws_secret_key: '{{ aws_secret_key }}'
+      security_token: '{{ security_token | default(omit) }}'
+      region: '{{ aws_region }}'
+  block:
+    # ============================================================
+    - name: Create simple s3_bucket
+      s3_bucket:
+        name: "{{ resource_prefix }}-testbucket-ansible"
+        state: present
+      register: output
+
+    - assert:
+        that:
+          - output.changed
+          - output.name == '{{ resource_prefix }}-testbucket-ansible'
+          - not output.requester_pays
+    # ============================================================
+    - name: Prepare fixtures folder
+      file:
+        path: "{{ output_dir }}/s3_sync"
+        state: directory
+        mode: '0755'
+
+    - name: Prepare files to sync
+      copy:
+        src: "{{ item }}"
+        dest: "{{ output_dir }}/s3_sync/{{ item }}"
+        mode: preserve
+      with_items:
+      - test1.txt
+      - test2.yml
+      - test3.json
+
+    - name: Prepare file with size bigger than chunk size
+      shell: |
+        dd if=/dev/zero of=test4.txt bs=1M count=10
+      args:
+        chdir: "{{ output_dir }}/s3_sync"
+
+    - name: Sync files with remote bucket
+      s3_sync:
+        bucket: "{{ resource_prefix }}-testbucket-ansible"
+        file_root: "{{ output_dir }}/s3_sync"
+      register: output
+    - assert:
+        that:
+          - output is changed
+
+    # ============================================================
+    - name: Sync files already present
+      s3_sync:
+        bucket: "{{ resource_prefix }}-testbucket-ansible"
+        file_root: "{{ output_dir }}/s3_sync"
+      register: output
+    - assert:
+        that:
+          - output is not changed
+
+    # ============================================================
+    - name: Sync files with etag calculation
+      s3_sync:
+        bucket: "{{ resource_prefix }}-testbucket-ansible"
+        file_root: "{{ output_dir }}/s3_sync"
+        file_change_strategy: checksum
+      register: output
+    - assert:
+        that:
+          - output is not changed
+
+    # ============================================================
+    # DOCUMENTATION EXAMPLES
+    # ============================================================
+    - name: all the options
+      s3_sync:
+        bucket: "{{ resource_prefix }}-testbucket-ansible"
+        file_root: "{{ output_dir }}/s3_sync"
+        mime_map:
+          .yml: application/text
+          .json: application/text
+        key_prefix: config_files/web
+        file_change_strategy: force
+        permission: public-read
+        cache_control: "public, max-age=31536000"
+        include: "*"
+        exclude: "*.txt,.*"
+      register: output
+
+    - assert:
+        that:
+          - output is changed
+
+  always:
+    - name: Ensure all buckets are deleted
+      s3_bucket:
+        name: "{{item}}"
+        state: absent
+        force: true
+      ignore_errors: yes
+      with_items:
+        - "{{ resource_prefix }}-testbucket-ansible"