Repository: jschneier/django-storages

Issue:
Google Cloud backend not a very good replacement for Django's Media or Static handling
The Google cloud backend isn't a very good candidate for Django's Media or Static files storage backend without some additional configuration and studying the sources. Ideally, the docs would talk a little bit about issues such as #463 #413 

Here's my solution:

```python
"""
GoogleCloudStorage extensions suitable for handing Django's
Static and Media files.

Requires following settings:
MEDIA_URL, GS_MEDIA_BUCKET_NAME
STATIC_URL, GS_STATIC_BUCKET_NAME

In addition to
https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
"""
from django.conf import settings
from storages.backends.gcloud import GoogleCloudStorage
from storages.utils import setting
from urllib.parse import urljoin


class GoogleCloudMediaStorage(GoogleCloudStorage):
    """GoogleCloudStorage suitable for Django's Media files."""

    def __init__(self, *args, **kwargs):
        if not settings.MEDIA_URL:
            raise Exception('MEDIA_URL has not been configured')
        kwargs['bucket_name'] = setting('GS_MEDIA_BUCKET_NAME', strict=True)
        super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)

    def url(self, name):
        """.url that doesn't call Google."""
        return urljoin(settings.MEDIA_URL, name)


class GoogleCloudStaticStorage(GoogleCloudStorage):
    """GoogleCloudStorage suitable for Django's Static files"""

    def __init__(self, *args, **kwargs):
        if not settings.STATIC_URL:
            raise Exception('STATIC_URL has not been configured')
        kwargs['bucket_name'] = setting('GS_STATIC_BUCKET_NAME', strict=True)
        super(GoogleCloudStaticStorage, self).__init__(*args, **kwargs)

    def url(self, name):
        """.url that doesn't call Google."""
        return urljoin(settings.STATIC_URL, name)
```


Discussion:
Thanks for doing the legwork. I tried implementing this snippet but I keep getting nulls. Am I missing something?

in settings.py:
```
DEFAULT_FILE_STORAGE = 'api.storage.backends.GoogleCloudStaticStorage',                                    
GS_BUCKET_NAME = my-bucket-name,                                                                                      
GS_STATIC_BUCKET_NAME = my-bucket-name 

STATIC_URL  = ''                                                                 
```

And my images are in `gs://my-bucket-name/images` and `gs://my-bucket-name/thumbnail`

Edit: Alright, after a good night's sleep, I realized I should be using my bucket's URL for `STATIC_URL`. Ex: `https://my-bucket-name.storage.googleapis.com`.

Again, thanks for doing the legwork!
Maybe a PR to the docs would move this along? Thanks for your work.
@elnygren And where does your file get stored?
n/m - I realized you reference these from `settings.py`

```python
STATICFILES_STORAGE = 'googlecloudstorage.GoogleCloudStaticStorage'
DEFAULT_FILE_STORAGE = 'googlecloudstorage.GoogleCloudMediaStorage'
```

So you just need to store them in such a way as python can find them.
Okay, so as currently written we hit Google every time `.url` is called. This is obviously substandard for performance reasons. This is really easy to update this is taken from the Google Cloud library, PR anyone?

```    @property
    def public_url(self):
        """The public URL for this blob's object.

        :rtype: `string`
        :returns: The public URL for this blob.
        """
        return '{storage_base_url}/{bucket_name}/{quoted_name}'.format(
            storage_base_url=_API_ACCESS_ENDPOINT,
            bucket_name=self.bucket.name,
            quoted_name=quote(self.name.encode('utf-8')))
```

No one has asked for signed urls yet but we'll need them eventually I think. Amazon is nice because it's on the bucket whereas for this we need to fetch the blob.
Not sure if this is the place, as it seems like I'm hijacking another issue, but..

> No one has asked for signed urls yet but we'll need them eventually I think.

I'd like signed URLs, please! See My PR https://github.com/jschneier/django-storages/pull/326 or the one that extends it further https://github.com/jschneier/django-storages/pull/399

I know for mine, I've just not had the time or experience to see how to fit it in with the tests, but could help add documentation, if it would help
As someone looking to implement this in a django project, is it really the way forward with these issues that have been raised? The lack of documentation made me uneasy in the first place as I am new to Django but I can find anything else.

Im struggling to see at first how I implement this but I can get my head around it if its worth it.
@xtechgamer735 I use this project in production on with Google Cloud Storage. There are some issues that are being worked, but the project is actively maintained.
 
Personally, I do not use GCS for static file storage (try this instead: http://whitenoise.evans.io/en/stable/) is simplier without the fragile deployment to a 3rd party network storage solution. Instead, I use it for media storage and it works fine with only a few configuration settings.
@sww314 Ah thanks for the link! That looks great.

I think GCS is the best way forward for me, it was the comment regarding inefficient url usage that got me most worried. Is that nothing really to be concerned about?

If you don't mind me asking what type of data are you storing?
The url call makes the listing of many-many files slow. So in particular, it makes collectstatic pretty slow.
 For typical, flow of getting and saving a media file (images, docs etc) it works fine - since Google replies are fast.

Okay so I think what we want is essentially what we have in the boto backends:

1) if Custom URL: Return that
2) if Signed URL: Fetch the blob and return the generated signed url
3) else: 
```python
name = self._encode_name(self._normalize_name(clean_name(name)))
return 'https://storage.googleapis.com/{}/{}'.format(self.bucket_name, quote(name, safe=''))
``` 

I ripped that out of the Google Cloud storage library, can someone try that out and confirm it at least passes a sanity check?

Then we can move onto points 1 & 2 after which I can ship 1.7
@jschneier for 3, you can just use self.bucket.blob(name).public_url()
No need to move code from the Google library.

Yep, you are right.

> On Aug 26, 2018, at 11:24 AM, Scott White <notifications@github.com> wrote:
> 
> @jschneier <https://github.com/jschneier> for 3, you can just use self.bucket.blob(name).public_url()
> No need to move code from the Google library.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/jschneier/django-storages/issues/491#issuecomment-416046535>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJB2IjGElyezo5eIPR_UY5Id3eF5RDtks5uUr26gaJpZM4Tj8DZ>.
> 



Resolving patch:
diff --git a/storages/backends/gcloud.py b/storages/backends/gcloud.py
--- a/storages/backends/gcloud.py
+++ b/storages/backends/gcloud.py
@@ -1,4 +1,5 @@
 import mimetypes
+from datetime import timedelta
 from tempfile import SpooledTemporaryFile
 
 from django.core.exceptions import ImproperlyConfigured, SuspiciousOperation
@@ -87,6 +88,8 @@ class GoogleCloudStorage(Storage):
     auto_create_acl = setting('GS_AUTO_CREATE_ACL', 'projectPrivate')
     default_acl = setting('GS_DEFAULT_ACL')
 
+    expiration = setting('GS_EXPIRATION', timedelta(seconds=86400))
+
     file_name_charset = setting('GS_FILE_NAME_CHARSET', 'utf-8')
     file_overwrite = setting('GS_FILE_OVERWRITE', True)
     cache_control = setting('GS_CACHE_CONTROL')
@@ -249,10 +252,17 @@ def get_created_time(self, name):
         return created if setting('USE_TZ') else timezone.make_naive(created)
 
     def url(self, name):
-        # Preserve the trailing slash after normalizing the path.
+        """
+        Return public url or a signed url for the Blob.
+        This DOES NOT check for existance of Blob - that makes codes too slow
+        for many use cases.
+        """
         name = self._normalize_name(clean_name(name))
-        blob = self._get_blob(self._encode_name(name))
-        return blob.public_url
+        blob = self.bucket.blob(self._encode_name(name))
+
+        if self.default_acl == 'publicRead':
+            return blob.public_url
+        return blob.generate_signed_url(self.expiration)
 
     def get_available_name(self, name, max_length=None):
         if self.file_overwrite:


Test patch:
diff --git a/tests/test_gcloud.py b/tests/test_gcloud.py
--- a/tests/test_gcloud.py
+++ b/tests/test_gcloud.py
@@ -5,8 +5,8 @@
 except ImportError:  # Python 3.2 and below
     import mock
 
-import datetime
 import mimetypes
+from datetime import datetime, timedelta
 
 from django.core.exceptions import ImproperlyConfigured
 from django.core.files.base import ContentFile
@@ -251,7 +251,7 @@ def test_size_no_file(self):
         self.assertRaises(NotFound, self.storage.size, self.filename)
 
     def test_modified_time(self):
-        naive_date = datetime.datetime(2017, 1, 2, 3, 4, 5, 678)
+        naive_date = datetime(2017, 1, 2, 3, 4, 5, 678)
         aware_date = timezone.make_aware(naive_date, timezone.utc)
 
         self.storage._bucket = mock.MagicMock()
@@ -266,7 +266,7 @@ def test_modified_time(self):
             self.storage._bucket.get_blob.assert_called_with(self.filename)
 
     def test_get_modified_time(self):
-        naive_date = datetime.datetime(2017, 1, 2, 3, 4, 5, 678)
+        naive_date = datetime(2017, 1, 2, 3, 4, 5, 678)
         aware_date = timezone.make_aware(naive_date, timezone.utc)
 
         self.storage._bucket = mock.MagicMock()
@@ -288,7 +288,7 @@ def test_get_modified_time(self):
             self.storage._bucket.get_blob.assert_called_with(self.filename)
 
     def test_get_created_time(self):
-        naive_date = datetime.datetime(2017, 1, 2, 3, 4, 5, 678)
+        naive_date = datetime(2017, 1, 2, 3, 4, 5, 678)
         aware_date = timezone.make_aware(naive_date, timezone.utc)
 
         self.storage._bucket = mock.MagicMock()
@@ -315,22 +315,47 @@ def test_modified_time_no_file(self):
 
         self.assertRaises(NotFound, self.storage.modified_time, self.filename)
 
-    def test_url(self):
+    def test_url_public_object(self):
         url = 'https://example.com/mah-bukkit/{}'.format(self.filename)
+        self.storage.default_acl = 'publicRead'
 
         self.storage._bucket = mock.MagicMock()
         blob = mock.MagicMock()
         blob.public_url = url
-        self.storage._bucket.get_blob.return_value = blob
+        blob.generate_signed_url = 'not called'
+        self.storage._bucket.blob.return_value = blob
 
         self.assertEqual(self.storage.url(self.filename), url)
-        self.storage._bucket.get_blob.assert_called_with(self.filename)
+        self.storage._bucket.blob.assert_called_with(self.filename)
 
-    def test_url_no_file(self):
+    def test_url_not_public_file(self):
+        secret_filename = 'secret_file.txt'
         self.storage._bucket = mock.MagicMock()
-        self.storage._bucket.get_blob.return_value = None
+        blob = mock.MagicMock()
+        generate_signed_url = mock.MagicMock(return_value='http://signed_url')
+        blob.public_url = 'http://this_is_public_url'
+        blob.generate_signed_url = generate_signed_url
+        self.storage._bucket.blob.return_value = blob
+
+        url = self.storage.url(secret_filename)
+        self.storage._bucket.blob.assert_called_with(secret_filename)
+        self.assertEqual(url, 'http://signed_url')
+        blob.generate_signed_url.assert_called_with(timedelta(seconds=86400))
+
+    def test_url_not_public_file_with_custom_expires(self):
+        secret_filename = 'secret_file.txt'
+        self.storage._bucket = mock.MagicMock()
+        blob = mock.MagicMock()
+        generate_signed_url = mock.MagicMock(return_value='http://signed_url')
+        blob.generate_signed_url = generate_signed_url
+        self.storage._bucket.blob.return_value = blob
+
+        self.storage.expiration = timedelta(seconds=3600)
 
-        self.assertRaises(NotFound, self.storage.url, self.filename)
+        url = self.storage.url(secret_filename)
+        self.storage._bucket.blob.assert_called_with(secret_filename)
+        self.assertEqual(url, 'http://signed_url')
+        blob.generate_signed_url.assert_called_with(timedelta(seconds=3600))
 
     def test_get_available_name(self):
         self.storage.file_overwrite = True