Repository: jschneier/django-storages

Issue:
GS_BLOB_CHUNK_SIZE not respected for reuploading existing files
Using the Gcloud backend, if a file with a given name already exists in the bucket, the `GS_BLOB_CHUNK_SIZE` setting is not taken into account for the upload. This is clearly visible in this snippet;

https://github.com/jschneier/django-storages/blob/620a13336f23a8c658914b2daa28661e2d350e8f/storages/backends/gcloud.py#L37-L41

For now, I've just worked around it by deleting any existing files with the same name prior to the upload. Seems like Gcloud's `Bucket.get_blob` method does support the `chunk_size` parameter though, so it should be a quick fix.

https://github.com/googleapis/python-storage/blob/0e5c082fc6baf16477c23a3d6fb647597625aee1/google/cloud/storage/bucket.py#L838



Discussion:
Can you open a PR to fix it? I think you can just change line 37 to pass in `chunk_size=storage.blob_chunk_size` given `get_blob` takes in `**kwargs` that are forwarded on to the constructor. Can you test that out and see if it fixes the issue.

> On Jun 28, 2022, at 02:34, wigeria ***@***.***> wrote:
> 
> 
> Using the Gcloud backend, if a file with a given name already exists in the bucket, the GS_BLOB_CHUNK_SIZE setting is not taken into account for the upload. This is clearly visible in this snippet;
> 
> https://github.com/jschneier/django-storages/blob/620a13336f23a8c658914b2daa28661e2d350e8f/storages/backends/gcloud.py#L37-L41 <https://github.com/jschneier/django-storages/blob/620a13336f23a8c658914b2daa28661e2d350e8f/storages/backends/gcloud.py#L37-L41>
> For now, I've just worked around it by deleting any existing files with the same name prior to the upload.
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/jschneier/django-storages/issues/1152>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAREDWEHXSWR6RC4Y7P66NTVRKMFVANCNFSM52A665RQ>.
> You are receiving this because you are subscribed to this thread.
> 



Resolving patch:
diff --git a/storages/backends/gcloud.py b/storages/backends/gcloud.py
--- a/storages/backends/gcloud.py
+++ b/storages/backends/gcloud.py
@@ -34,7 +34,8 @@ def __init__(self, name, mode, storage):
         self.mime_type = mimetypes.guess_type(name)[0]
         self._mode = mode
         self._storage = storage
-        self.blob = storage.bucket.get_blob(name)
+        self.blob = storage.bucket.get_blob(
+            name, chunk_size=storage.blob_chunk_size)
         if not self.blob and 'w' in mode:
             self.blob = Blob(
                 self.name, storage.bucket,


Test patch:
diff --git a/tests/test_gcloud.py b/tests/test_gcloud.py
--- a/tests/test_gcloud.py
+++ b/tests/test_gcloud.py
@@ -38,7 +38,8 @@ def test_open_read(self):
 
         f = self.storage.open(self.filename)
         self.storage._client.bucket.assert_called_with(self.bucket_name)
-        self.storage._bucket.get_blob.assert_called_with(self.filename)
+        self.storage._bucket.get_blob.assert_called_with(
+            self.filename, chunk_size=None)
 
         f.blob.download_to_file = lambda tmpfile: tmpfile.write(data)
         self.assertEqual(f.read(), data)
@@ -49,7 +50,8 @@ def test_open_read_num_bytes(self):
 
         f = self.storage.open(self.filename)
         self.storage._client.bucket.assert_called_with(self.bucket_name)
-        self.storage._bucket.get_blob.assert_called_with(self.filename)
+        self.storage._bucket.get_blob.assert_called_with(
+            self.filename, chunk_size=None)
 
         f.blob.download_to_file = lambda tmpfile: tmpfile.write(data)
         self.assertEqual(f.read(num_bytes), data[0:num_bytes])
@@ -59,7 +61,8 @@ def test_open_read_nonexistent(self):
         self.storage._bucket.get_blob.return_value = None
 
         self.assertRaises(FileNotFoundError, self.storage.open, self.filename)
-        self.storage._bucket.get_blob.assert_called_with(self.filename)
+        self.storage._bucket.get_blob.assert_called_with(
+            self.filename, chunk_size=None)
 
     def test_open_read_nonexistent_unicode(self):
         filename = 'ủⓝï℅ⅆℇ.txt'
@@ -531,3 +534,22 @@ def test_override_init_argument(self):
         self.assertEqual(storage.location, 'foo1')
         storage = gcloud.GoogleCloudStorage(location='foo2')
         self.assertEqual(storage.location, 'foo2')
+
+    def test_dupe_file_chunk_size(self):
+        """
+        Tests that recreating a file that already exists in the bucket
+        respects the `GS_BLOB_CHUNK_SIZE` setting
+        """
+        chunk_size = 1024*256
+
+        with override_settings(GS_BLOB_CHUNK_SIZE=chunk_size):
+            # Creating a new storage here since chunk-size is set as an
+            # attribute on init
+            storage = gcloud.GoogleCloudStorage()
+            storage._bucket = mock.MagicMock()
+            # Confirms that `get_blob` always returns a truthy value
+            storage._bucket.get_blob.return_value = True
+
+            storage.open(self.filename, 'wb')
+            storage._bucket.get_blob.assert_called_with(
+                self.filename, chunk_size=chunk_size)