Repository: drivendataorg/cloudpathlib
Pull request #490 (2024-11-30 00:12:18)

Add md5 hash property to GSPath (#483)
Live tests for #483



github-actions[bot]: <!-- NETLIFY DEPLOY COMMENT GENERATED BY ACTIONS_NETLIFY - APP ID SHA256: 9d0998be292ddace9530fffad95434f4e2279530e25cf278a25f07954faa426e -->
🚀 Deployed on https://deploy-preview-490--gallant-agnesi-5f7bb3.netlify.app
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) Report
Attention: Patch coverage is `80.00000%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 91.6%. Comparing base [(`7f1a5dc`)](https://app.codecov.io/gh/drivendataorg/cloudpathlib/commit/7f1a5dce4277eb2bcd1611389241b818651f9f3a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) to head [(`b3b4b34`)](https://app.codecov.io/gh/drivendataorg/cloudpathlib/commit/b3b4b342f84f8d71afd75e3bbeae06e14d685ee5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg).

| [Files with missing lines](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) | Patch % | Lines |
|---|---|---|
| [cloudpathlib/gs/gspath.py](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&filepath=cloudpathlib%2Fgs%2Fgspath.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-Y2xvdWRwYXRobGliL2dzL2dzcGF0aC5weQ==) | 85.7% | [1 Missing :warning: ](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) |
| [cloudpathlib/local/implementations/gs.py](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&filepath=cloudpathlib%2Flocal%2Fimplementations%2Fgs.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-Y2xvdWRwYXRobGliL2xvY2FsL2ltcGxlbWVudGF0aW9ucy9ncy5weQ==) | 66.6% | [1 Missing :warning: ](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) |

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff            @@
##           master    #490     +/-   ##
========================================
- Coverage    94.4%   91.6%   -2.9%     
========================================
  Files          23      23             
  Lines        1776    1785      +9     
========================================
- Hits         1678    1636     -42     
- Misses         98     149     +51     
```

| [Files with missing lines](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) | Coverage Δ | |
|---|---|---|
| [cloudpathlib/gs/gsclient.py](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&filepath=cloudpathlib%2Fgs%2Fgsclient.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-Y2xvdWRwYXRobGliL2dzL2dzY2xpZW50LnB5) | `90.0% <ø> (-1.5%)` | :arrow_down: |
| [cloudpathlib/gs/gspath.py](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&filepath=cloudpathlib%2Fgs%2Fgspath.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-Y2xvdWRwYXRobGliL2dzL2dzcGF0aC5weQ==) | `92.4% <85.7%> (-1.2%)` | :arrow_down: |
| [cloudpathlib/local/implementations/gs.py](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490?src=pr&el=tree&filepath=cloudpathlib%2Flocal%2Fimplementations%2Fgs.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-Y2xvdWRwYXRobGliL2xvY2FsL2ltcGxlbWVudGF0aW9ucy9ncy5weQ==) | `97.1% <66.6%> (-2.9%)` | :arrow_down: |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/drivendataorg/cloudpathlib/pull/490/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg)

</details>
pjbull: @fafnirZ The hashes do not match in the live tests. Do you know why that would be?

```python
=================================== FAILURES ===================================
_______ test_md5_property[hello world-5eb63bbbe01eeed093cb22bb8f5acdc3] ________
[gw1] linux -- Python 3.11.10 /home/runner/.venv/bin/python

content = 'hello world', expected_hash = '5eb63bbbe01eeed093cb22bb8f5acdc3'
gs_rig = <tests.conftest.CloudProviderTestRig object at 0x7f4eacb26e10>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f4ead623e50>

    @pytest.mark.parametrize(
        "content, expected_hash",
        [
            ("hello world", "5eb63bbbe01eeed093cb22bb8f5acdc3"),
            ("another test case", "4f8182cd9856777ebe3c4f5dc58dacea"),
        ],
    )
    def test_md5_property(content, expected_hash, gs_rig, monkeypatch):
        # if USE_LIVE_CLOUD this doesnt have any effect
        monkeypatch.setenv("MOCK_EXPECTED_MD5_HASH", expected_hash)
    
        p: GSPath = gs_rig.create_cloud_path("dir_0/file0_0.txt")
        p.write_text(content)
>       assert p.md5 == expected_hash
E       AssertionError: assert 'XrY7u+Ae7tCTyyK7j1rNww==' == '5eb63bbbe01eeed093cb22bb8f5acdc3'
E         
E         - 5eb63bbbe01eeed093cb22bb8f5acdc3
E         + XrY7u+Ae7tCTyyK7j1rNww==

tests/test_gs_specific.py:67: AssertionError
---------------------------- Captured stdout setup -----------------------------
Test directory name is: RkUnPnTDZxRHxQeE72hydb-test_gs_specific-test_md5_property
____ test_md5_property[another test case-4f8182cd9856777ebe3c4f5dc58dacea] _____
[gw1] linux -- Python 3.11.10 /home/runner/.venv/bin/python

content = 'another test case'
expected_hash = '4f8182cd9856777ebe3c4f5dc58dacea'
gs_rig = <tests.conftest.CloudProviderTestRig object at 0x7f4ead3eca10>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f4eace25650>

    @pytest.mark.parametrize(
        "content, expected_hash",
        [
            ("hello world", "5eb63bbbe01eeed093cb22bb8f5acdc3"),
            ("another test case", "4f8182cd9856777ebe3c4f5dc58dacea"),
        ],
    )
    def test_md5_property(content, expected_hash, gs_rig, monkeypatch):
        # if USE_LIVE_CLOUD this doesnt have any effect
        monkeypatch.setenv("MOCK_EXPECTED_MD5_HASH", expected_hash)
    
        p: GSPath = gs_rig.create_cloud_path("dir_0/file0_0.txt")
        p.write_text(content)
>       assert p.md5 == expected_hash
E       AssertionError: assert 'T4GCzZhWd36+PE9dxY2s6g==' == '4f8182cd9856777ebe3c4f5dc58dacea'
E         
E         - 4f8182cd9856777ebe3c4f5dc58dacea
E         + T4GCzZhWd36+PE9dxY2s6g==

tests/test_gs_specific.py:67: AssertionError
```
pjbull: Thanks @fafnirZ, I just grabbed the change directly. Will merge if tests pass

Patch:
diff --git a/HISTORY.md b/HISTORY.md
index 2e379a6..74f862e 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -3,6 +3,7 @@
 ## Unreleased
 
 - Fixed `CloudPath(...) / other` to correctly attempt to fall back on `other`'s `__rtruediv__` implementation, in order to support classes that explicitly support the `/` with a `CloudPath` instance. Previously, this would always raise a `TypeError` if `other` were not a `str` or `PurePosixPath`. (PR [#479](https://github.com/drivendataorg/cloudpathlib/pull/479))
+- Add `md5` property to `GSPath`, updated LocalGSPath to include `md5` property, updated mock_gs.MockBlob to include `md5_hash` property.
 - Fixed an uncaught exception on Azure Gen2 storage accounts with HNS enabled when used with `DefaultAzureCredential`. (Issue [#486](https://github.com/drivendataorg/cloudpathlib/issues/486))
 
 ## v0.20.0 (2024-10-18)
diff --git a/README.md b/README.md
index c421ee6..2013f74 100644
--- a/README.md
+++ b/README.md
@@ -205,7 +205,7 @@ Most methods and properties from `pathlib.Path` are supported except for the one
 | `bucket`               | ❌                | ✅         | ✅         |
 | `container`            | ✅                | ❌         | ❌         |
 | `key`                  | ❌                | ✅         | ❌         |
-| `md5`                  | ✅                | ❌         | ❌         |
+| `md5`                  | ✅                | ❌         | ✅         |
 
 ----
 
diff --git a/cloudpathlib/gs/gsclient.py b/cloudpathlib/gs/gsclient.py
index edd5b88..c66e661 100644
--- a/cloudpathlib/gs/gsclient.py
+++ b/cloudpathlib/gs/gsclient.py
@@ -121,6 +121,7 @@ class GSClient(Client):
                 "size": blob.size,
                 "updated": blob.updated,
                 "content_type": blob.content_type,
+                "md5_hash": blob.md5_hash,
             }
 
     def _download_file(self, cloud_path: GSPath, local_path: Union[str, os.PathLike]) -> Path:
diff --git a/cloudpathlib/gs/gspath.py b/cloudpathlib/gs/gspath.py
index bf085a7..a651a41 100644
--- a/cloudpathlib/gs/gspath.py
+++ b/cloudpathlib/gs/gspath.py
@@ -1,7 +1,7 @@
 import os
 from pathlib import Path
 from tempfile import TemporaryDirectory
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Optional
 
 from ..cloudpath import CloudPath, NoStatError, register_path_class
 
@@ -95,3 +95,10 @@ class GSPath(CloudPath):
     @property
     def etag(self):
         return self.client._get_metadata(self).get("etag")
+
+    @property
+    def md5(self) -> Optional[str]:
+        meta = self.client._get_metadata(self)
+        if not meta:
+            return None
+        return meta.get("md5_hash", None)
diff --git a/cloudpathlib/local/implementations/gs.py b/cloudpathlib/local/implementations/gs.py
index 5e63084..a5673c0 100644
--- a/cloudpathlib/local/implementations/gs.py
+++ b/cloudpathlib/local/implementations/gs.py
@@ -53,6 +53,10 @@ class LocalGSPath(LocalPath):
     def etag(self):
         return self.client._md5(self)
 
+    @property
+    def md5(self) -> str:
+        return self.client._md5(self)
+
 
 LocalGSPath.__name__ = "GSPath"
 


Test patch:
diff --git a/tests/mock_clients/mock_gs.py b/tests/mock_clients/mock_gs.py
index 48aae60..e576365 100644
--- a/tests/mock_clients/mock_gs.py
+++ b/tests/mock_clients/mock_gs.py
@@ -1,4 +1,5 @@
 from datetime import datetime, timedelta
+import os
 from pathlib import Path, PurePosixPath
 import shutil
 from tempfile import TemporaryDirectory
@@ -95,6 +96,10 @@ class MockBlob:
     def etag(self):
         return "etag"
 
+    @property
+    def md5_hash(self):
+        return os.environ.get("MOCK_EXPECTED_MD5_HASH", "md5_hash")
+
     @property
     def size(self):
         path = self.bucket / self.name
diff --git a/tests/test_gs_specific.py b/tests/test_gs_specific.py
index 1632aca..6af6c46 100644
--- a/tests/test_gs_specific.py
+++ b/tests/test_gs_specific.py
@@ -49,3 +49,29 @@ def test_as_url(gs_rig):
     assert "X-Goog-Date" in query_params
     assert "X-Goog-SignedHeaders" in query_params
     assert "X-Goog-Signature" in query_params
+
+
+@pytest.mark.parametrize(
+    "contents",
+    [
+        "hello world",
+        "another test case",
+    ],
+)
+def test_md5_property(contents, gs_rig, monkeypatch):
+    def _calculate_b64_wrapped_md5_hash(contents: str) -> str:
+        # https://cloud.google.com/storage/docs/json_api/v1/objects
+        from base64 import b64encode
+        from hashlib import md5
+
+        contents_md5_bytes = md5(contents.encode()).digest()
+        b64string = b64encode(contents_md5_bytes).decode()
+        return b64string
+
+    # if USE_LIVE_CLOUD this doesnt have any effect
+    expected_hash = _calculate_b64_wrapped_md5_hash(contents)
+    monkeypatch.setenv("MOCK_EXPECTED_MD5_HASH", expected_hash)
+
+    p: GSPath = gs_rig.create_cloud_path("dir_0/file0_0.txt")
+    p.write_text(contents)
+    assert p.md5 == expected_hash