Repository: getmoto/moto

Issue:
Error in S3 compute_checksum()
There is an issue with the way CRC32 and CRC32C checksums are computed in `compute_checksum()`. They are treated as the same, which is incorrect. See [this line](https://github.com/getmoto/moto/blob/6b2ee153e61def632808015d35ea3e32ac585483/moto/s3/utils.py#L202). Consequently, the tests for `compute_checksum()` are incorrect, see
https://github.com/getmoto/moto/blob/6b2ee153e61def632808015d35ea3e32ac585483/tests/test_s3/test_s3_utils.py#L139C1-L144

The correct value of **CRC32** for `somedata` is `Uwy90A==`
The correct value of **CRC3C** for `somedata` is `dB9qBQ==`

CRC32C computation is best handled with `crc32c`.



Discussion:
Hi @badmetacoder, welcome to Moto! That is indeed not implemented yet, for a few reasons:

 - I wanted to avoid introducing a new dependency for functionality that will be unnoticed/unused for 99% for users
 - Libraries that compute `crc32c` are either a license-mess, and/or do not support ARM architectures

Do you have a specific test case were you need Moto to support the `CRC32C` checksum?
> Hi @badmetacoder, welcome to Moto! That is indeed not implemented yet, for a few reasons:
> 
> * I wanted to avoid introducing a new dependency for functionality that will be unnoticed/unused for 99% for users
> * Libraries that compute `crc32c` are either a license-mess, and/or do not support ARM architectures
> 
> Do you have a specific test case were you need Moto to support the `CRC32C` checksum?

I am guessing this could be an [optional] install for those who need it. My use cases are 

1. clients uploading data via pre-signed URLs and wanting to ensure data integrity (large files containing important data);
2. processes generating reports and uploading them to S3 and then allowing external parties to download and verify integrity of downloaded data.

I have tests for these cases and right now can only verify that I am setting the Checksum algorithm to a specific type. It's something, but not ideal and doesn't test for checksum validation failures.

Resolving patch:
diff --git a/moto/s3/models.py b/moto/s3/models.py
--- a/moto/s3/models.py
+++ b/moto/s3/models.py
@@ -1523,12 +1523,20 @@ def test_my_custom_endpoint():
 
     Note that this only works if the environment variable is set **before** the mock is initialized.
 
+    ------------------------------------
+
     When using the MultiPart-API manually, the minimum part size is 5MB, just as with AWS. Use the following environment variable to lower this:
 
     .. sourcecode:: bash
 
         S3_UPLOAD_PART_MIN_SIZE=256
 
+    ------------------------------------
+
+    Install `moto[s3crc32c]` if you use the CRC32C algorithm, and absolutely need the correct value. Alternatively, you can install the `crc32c` dependency manually.
+
+    If this dependency is not installed, Moto will fall-back to the CRC32-computation when computing checksums.
+
     """
 
     def __init__(self, region_name: str, account_id: str):
diff --git a/moto/s3/utils.py b/moto/s3/utils.py
--- a/moto/s3/utils.py
+++ b/moto/s3/utils.py
@@ -199,7 +199,15 @@ def _self_iterable(self) -> Dict[str, Any]:
 def compute_checksum(body: bytes, algorithm: str) -> bytes:
     if algorithm == "SHA1":
         hashed_body = _hash(hashlib.sha1, (body,))
-    elif algorithm == "CRC32" or algorithm == "CRC32C":
+    elif algorithm == "CRC32C":
+        try:
+            import crc32c
+
+            hashed_body = crc32c.crc32c(body).to_bytes(4, "big")
+        except:  # noqa: E722 Do not use bare except
+            # Optional library Can't be found - just revert to CRC32
+            hashed_body = binascii.crc32(body).to_bytes(4, "big")
+    elif algorithm == "CRC32":
         hashed_body = binascii.crc32(body).to_bytes(4, "big")
     else:
         hashed_body = _hash(hashlib.sha256, (body,))


Test patch:
diff --git a/tests/test_s3/test_s3_utils.py b/tests/test_s3/test_s3_utils.py
--- a/tests/test_s3/test_s3_utils.py
+++ b/tests/test_s3/test_s3_utils.py
@@ -139,7 +139,13 @@ def test_checksum_crc32():
 
 
 def test_checksum_crc32c():
-    compute_checksum(b"somedata", "CRC32C").should.equal(b"Uwy90A==")
+    try:
+        import crc32c  # noqa # pylint: disable=unused-import
+
+        compute_checksum(b"somedata", "CRC32C").should.equal(b"dB9qBQ==")
+    except:  # noqa: E722 Do not use bare except
+        # Optional library Can't be found - just revert to CRC32
+        compute_checksum(b"somedata", "CRC32C").should.equal(b"Uwy90A==")
 
 
 def test_cors_utils():