Repository: scrapy/scrapy

Issue:
Scrapy does not decode base64 MD5 checksum from GCS
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

Incorrect GCS Checksum processing

### Steps to Reproduce

1. Obtain the checksum for an up-to-date file.

**Expected behavior:** [What you expect to happen]
matches the checksum of the file downloaded
**Actual behavior:** [What actually happens]
NOT matches the checksum of the file downloaded
**Reproduces how often:** [What percentage of the time does it reproduce?]
Always
### Versions
current

### Additional context

https://cloud.google.com/storage/docs/json_api/v1/objects
> MD5 hash of the data, encoded using [base64](https://datatracker.ietf.org/doc/html/rfc4648#section-4).

But, Scrapy dose not decode MD5 from GCS.



Discussion:
Hi! Can you please provide more info? 
For instance, I downloaded and saved a file from Google Cloud Storage using Scrapy, and its MD5 is "42bff117f2040df9c0fbf24752da0023". However, when I attempted to scrap the file again, the Scrapy output showed "checksum": "Qr/xF/IEDfnA+/JHUtoAIw==", "status": "uptodate". This is because, as indicated in the URL I provided earlier, Google's API (and the client) encodes the MD5 in base64 for some unknown reason. By decoding "Qr/xF/IEDfnA+/JHUtoAIw==" with base64 and using byte.hex(), the original MD5 can be obtained.
I see. The relevant code seems to be https://github.com/scrapy/scrapy/blob/d60b4edd11436e61284615ec7ce89f8ac7e46d9a/scrapy/pipelines/files.py#L231
me and @tstauder can work on this issue.

Is it possible to have it assigned to one of us? 
You don't really need that to work on it or provide a PR.
Hi @Gallaecio , from what I understand from the discussion above and the source code, it seems that all that is required to fix this is to change `checksum = blob.md5_hash` to `checksum = base64.b64decode(blob.md5_hash).hex()`. Let me know if there's anything more to it, I'll be more than happy to raise a PR
@heppymxm I believe that’s it. And adding a test for it, which is probably harder than the change itself :sweat_smile: 
This [line](https://github.com/scrapy/scrapy/blob/c7730627a0f99afccca11437b0775757c50ca9e8/tests/test_pipeline_files.py#L604) is comparing the checksum to a b64 encoded string. I'm curious where this even came from? And if this is valid, can we not simply extract out the md5 hash from this and replace it here?
Thanks!
Oh, updating that value to the right expectation should be test enough, nice! I assume the current base is the base64-encoded MD5 of https://github.com/scrapy/scrapy/blob/c7730627a0f99afccca11437b0775757c50ca9e8/tests/test_pipeline_files.py#L593
Shall I raise a PR with these changes?
That would be awesome!

Resolving patch:
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -3,6 +3,7 @@
 
 See documentation in topics/media-pipeline.rst
 """
+import base64
 import functools
 import hashlib
 import logging
@@ -228,7 +229,7 @@ def __init__(self, uri):
     def stat_file(self, path, info):
         def _onsuccess(blob):
             if blob:
-                checksum = blob.md5_hash
+                checksum = base64.b64decode(blob.md5_hash).hex()
                 last_modified = time.mktime(blob.updated.timetuple())
                 return {"checksum": checksum, "last_modified": last_modified}
             return {}


Test patch:
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -601,7 +601,7 @@ def test_persist(self):
         s = yield store.stat_file(path, info=None)
         self.assertIn("last_modified", s)
         self.assertIn("checksum", s)
-        self.assertEqual(s["checksum"], "zc2oVgXkbQr2EQdSdw3OPA==")
+        self.assertEqual(s["checksum"], "cdcda85605e46d0af6110752770dce3c")
         u = urlparse(uri)
         content, acl, blob = get_gcs_content_and_delete(u.hostname, u.path[1:] + path)
         self.assertEqual(content, data)