Repository: googleapis/google-cloud-python

Issue:
Storage: Blob.compose doesn't use default content_type
[`Blob.compose`](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/51ba7e148fc76e07018dc60192219fb28ff628b0/storage/google/cloud/storage/blob.py#L1363-L1381) does not use the [`self._get_content_type` method](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/51ba7e148fc76e07018dc60192219fb28ff628b0/storage/google/cloud/storage/blob.py#L605-L633), which means Blob.compose raises an error if the `content_type` is not explicitly specified. This is [documented behavior](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/51ba7e148fc76e07018dc60192219fb28ff628b0/storage/google/cloud/storage/blob.py#L1377-L1381), but is it intended behavior? Should compose use `self._get_content_type` to use the default type, or perhaps infer the type from the blobs being composed?



Discussion:
@frankyn I'm not sure how the GCS team would like to shape this feature.
Acking. Adding to GCS client library weekly.
@frankyn -- any updates?
Apologies on the wait.

content-type is optional and defaults to `application/octet-stream` based on [Storage.Objects Ref docs](https://cloud.google.com/storage/docs/json_api/v1/objects#resource). I don't think we should fail the method if it's not set.

Additionally, I'm looking at `upload_*` and these methods provide a parameter for content_type. Including this parameter would follow the pattern that exists for these methods and would be easier to understand.

@tseaver do you have objections to surfacing content_type as an optional parameter for compose()?
Great! content-type as an optional parameter (but falling back to the default through `self._get_content_type`) seems ideal. 

Could this lead to unexpected behavior? For example, composing blobs of some non-default type but not specifying that type in compose leads them to be composed as `application/octet-stream`. I suppose the type can also be inferred from the types of the objects being composed, but this seems more involved.
@rasmi that sounds like a fair condition. 
```python
if content_type is not None:
  # use this
if self._get_content_type is not `None`
  # use this
else:
  # it's okay to not provide content_type as it will default to application/octet-stream.
```

I think inferring the content-type might be not be as helpful as simply surfacing a parameter for the content type. I think it would feel magical _which is not a bad thing_, but it may or may not be the desired result for everyone.


Okay, agreed.

I can submit a PR. However, I have an implementation question. Content type is [not explicitly used in `self.compose`](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/dc5d144567ffee60a7438d54a4516d3016a5bfa3/storage/google/cloud/storage/blob.py#L1381-L1400), but it does pass `self` when making a request, so presumably the blob object's content type must be set before the request is made (which is why the error is raised if it's not set).

Additionally, `self._get_content_type()` handles arguments/None values correctly. So I think we can simply add the `content_type` argument to the `compose` function and do:
```
self.content_type = self._get_content_type(content_type)
```
at the top of `compose` to ensure that the content type is set. If the argument is None, it will use the blob's content type, and if the blob has no content type, it will use the default. Is that a reasonable implementation?
@frankyn as long as the [`objects.compose` API](https://cloud.google.com/storage/docs/json_api/v1/objects/compose) honors the `destination.content_type` field, it should work just to set `content_type` on the `Blob` instance through which we are invoking it (we already require that, in fact, because we copy the destination blob's properties into the request body).

```python
sources = [bucket.blob(name) for name in ['source1.txt', 'source2.txt']]
destination = bucket.blob('destination.txt)
destination.content_type = 'text/plain'
# set other destination properties here
destination.compose(sources)
```

I don't really see the value in adding Another Way to Do It(TM).

This issue is a request that, if the destination blob does *not* have `content_type` set, we deduce the correct value by looking at the content types of the sources.  Given that we may not have loaded them (e.g., in the example I just showed), I don't know how feasible it would be.  I guess we could do something like:

```python
    def _deduce_content_type_for_compose(self, sources):
        if self.content_type is None:
            source_types = set(
                src.content_type for src in sources if src.content_type is not None)
            if len(source_types) == 1:
                return source_types[0]
            return self.content_type
```
Thanks @tseaver, your comment helps.
I don't think we should infer content types from a list of objects and instead depend on the destination blob content type.

How about not failing the compose if a content_type is not found?


> How about not failing the compose if a content_type is not found?

That would be fine.  It has been a while, but I wonder if I added that check because the `objects.compose` API used to fail if it wasn't set.
Ah good question. I do recall this came up for [Node.js](https://github.com/googleapis/nodejs-storage/issues/166) and the GCS API has since changed this requirement. The code used to be correct but now the restriction has been removed.

Resolving patch:
diff --git a/storage/google/cloud/storage/blob.py b/storage/google/cloud/storage/blob.py
--- a/storage/google/cloud/storage/blob.py
+++ b/storage/google/cloud/storage/blob.py
@@ -1374,13 +1374,7 @@ def compose(self, sources, client=None):
                       ``NoneType``
         :param client: Optional. The client to use.  If not passed, falls back
                        to the ``client`` stored on the blob's bucket.
-
-        :raises: :exc:`ValueError` if this blob does not have its
-                 :attr:`content_type` set.
         """
-        if self.content_type is None:
-            raise ValueError("Destination 'content_type' not set.")
-
         client = self._require_client(client)
         query_params = {}
 


Test patch:
diff --git a/storage/tests/system.py b/storage/tests/system.py
--- a/storage/tests/system.py
+++ b/storage/tests/system.py
@@ -779,6 +779,26 @@ def test_compose_create_new_blob(self):
         composed = destination.download_as_string()
         self.assertEqual(composed, SOURCE_1 + SOURCE_2)
 
+    def test_compose_create_new_blob_wo_content_type(self):
+        SOURCE_1 = b'AAA\n'
+        source_1 = self.bucket.blob('source-1')
+        source_1.upload_from_string(SOURCE_1)
+        self.case_blobs_to_delete.append(source_1)
+
+        SOURCE_2 = b'BBB\n'
+        source_2 = self.bucket.blob('source-2')
+        source_2.upload_from_string(SOURCE_2)
+        self.case_blobs_to_delete.append(source_2)
+
+        destination = self.bucket.blob('destination')
+
+        destination.compose([source_1, source_2])
+        self.case_blobs_to_delete.append(destination)
+
+        self.assertIsNone(destination.content_type)
+        composed = destination.download_as_string()
+        self.assertEqual(composed, SOURCE_1 + SOURCE_2)
+
     def test_compose_replace_existing_blob(self):
         BEFORE = b'AAA\n'
         original = self.bucket.blob('original')
diff --git a/storage/tests/unit/test_blob.py b/storage/tests/unit/test_blob.py
--- a/storage/tests/unit/test_blob.py
+++ b/storage/tests/unit/test_blob.py
@@ -2113,15 +2113,35 @@ def test_compose_wo_content_type_set(self):
         SOURCE_1 = 'source-1'
         SOURCE_2 = 'source-2'
         DESTINATION = 'destinaton'
-        connection = _Connection()
+        RESOURCE = {}
+        after = ({'status': http_client.OK}, RESOURCE)
+        connection = _Connection(after)
         client = _Client(connection)
         bucket = _Bucket(client=client)
         source_1 = self._make_one(SOURCE_1, bucket=bucket)
         source_2 = self._make_one(SOURCE_2, bucket=bucket)
         destination = self._make_one(DESTINATION, bucket=bucket)
+        # no destination.content_type set
 
-        with self.assertRaises(ValueError):
-            destination.compose(sources=[source_1, source_2])
+        destination.compose(sources=[source_1, source_2])
+
+        self.assertIsNone(destination.content_type)
+
+        kw = connection._requested
+        self.assertEqual(len(kw), 1)
+        self.assertEqual(kw[0], {
+            'method': 'POST',
+            'path': '/b/name/o/%s/compose' % DESTINATION,
+            'query_params': {},
+            'data': {
+                'sourceObjects': [
+                    {'name': source_1.name},
+                    {'name': source_2.name},
+                ],
+                'destination': {},
+            },
+            '_target_object': destination,
+        })
 
     def test_compose_minimal_w_user_project(self):
         SOURCE_1 = 'source-1'