Repository: googleapis/google-cloud-python

Issue:
blob.py is unable to handle a partial rewrite and fails with KeyError on lack of resource 
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/storage/google/cloud/storage/blob.py#L871
```
871:        self._set_properties(api_response['resource'])
```

This relies on the API to always return 'resource'.  However, it is optional as per the doc.
https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite
"""
This property is present in the response only when copying completes.
"""

As such, when the write was partial, done: false is returned and resource not present.  Thus failing with the following KeyError.
```
Traceback (most recent call last): 
File "xxx.py", line 304, in test_011_CopyFileFromBucketToBucketNewName 
new_destination_file_name=new_file_name) 
File "xxx.py, line 276, in copyFileFromBucketToBucket 
token, rewritten, total = destination_blob.rewrite(source_blob, token=token) 
File "/home/jenkins/venv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 681, in rewrite 
self._set_properties(api_response['resource']) 
KeyError: 'resource' 
```

The most easy fix would be to push _set_properties down under after done: false check:

```
E.g.;

       api_response = client._connection.api_request(
            method='POST', path=source.path + '/rewriteTo' + self.path,
            query_params=query_params, data=self._properties, headers=headers,
            _target_object=self)
        # self._set_properties(api_response['resource'])  ## Comment out this line
        rewritten = int(api_response['totalBytesRewritten'])
        size = int(api_response['objectSize'])

        if api_response['done']:
            return None, rewritten, size

        self._set_properties(api_response['resource']) # and move here 
        return api_response['rewriteToken'], rewritten, size
```


Discussion:
I don't think your solution will work. I came across the same issue, but when it is *not* done, it falls through. You probably want to move it inside the if `api_response['done']`.
Ah, you are totally right. Scrap my suggestion.
I wasn't sure how _set_properties is used in the API and I mistook the "done" check in the wrong way, of course.

Thanks for reporting @80nashi!

If I understand correctly from the docs, the `resource` is only available when the copy is complete?
>A resource containing the metadata for the copied-to object. This property is present in the response only when copying completes.

@tseaver, @dhermes maybe we should be raising an exception here?

I'm not sure from the docs, but is there polling that should be available for rewriting so you can check if the copy is done?
As I understand the method and the API, the method is expected to return the rewrite token without raising an exception, in case of partial write (`done: false` and `resource` non-existent).
Then the user program can call the method again with the non-null, non empty string rewrite token, repeat it until `done: true` and `resource` present.


So yeah, the following can be a possible fix as verba suggested.
However, I'm not sure what _set_properties is expected to do in case of `done: false`.

```
        # self._set_properties(api_response['resource'])  ## Comment out this line
        rewritten = int(api_response['totalBytesRewritten'])
        size = int(api_response['objectSize'])

        if api_response['done']:
            self._set_properties(api_response['resource']) # and move here
            return None, rewritten, size

        return api_response['rewriteToken'], rewritten, size
```

Resolving patch:
diff --git a/storage/google/cloud/storage/blob.py b/storage/google/cloud/storage/blob.py
--- a/storage/google/cloud/storage/blob.py
+++ b/storage/google/cloud/storage/blob.py
@@ -868,11 +868,14 @@ def rewrite(self, source, token=None, client=None):
             method='POST', path=source.path + '/rewriteTo' + self.path,
             query_params=query_params, data=self._properties, headers=headers,
             _target_object=self)
-        self._set_properties(api_response['resource'])
         rewritten = int(api_response['totalBytesRewritten'])
         size = int(api_response['objectSize'])
 
+        # The resource key is set if and only if the API response is
+        # completely done. Additionally, there is no rewrite token to return
+        # in this case.
         if api_response['done']:
+            self._set_properties(api_response['resource'])
             return None, rewritten, size
 
         return api_response['rewriteToken'], rewritten, size


Test patch:
diff --git a/storage/unit_tests/test_blob.py b/storage/unit_tests/test_blob.py
--- a/storage/unit_tests/test_blob.py
+++ b/storage/unit_tests/test_blob.py
@@ -1288,6 +1288,33 @@ def test_compose_w_additional_property_changes(self):
         self.assertEqual(kw[0]['path'], '/b/name/o/%s/compose' % DESTINATION)
         self.assertEqual(kw[0]['data'], SENT)
 
+    def test_rewrite_response_without_resource(self):
+        from six.moves.http_client import OK
+
+        SOURCE_BLOB = 'source'
+        DEST_BLOB = 'dest'
+        DEST_BUCKET = 'other-bucket'
+        TOKEN = 'TOKEN'
+        RESPONSE = {
+            'totalBytesRewritten': 33,
+            'objectSize': 42,
+            'done': False,
+            'rewriteToken': TOKEN,
+        }
+        response = ({'status': OK}, RESPONSE)
+        connection = _Connection(response)
+        client = _Client(connection)
+        source_bucket = _Bucket(client=client)
+        source_blob = self._make_one(SOURCE_BLOB, bucket=source_bucket)
+        dest_bucket = _Bucket(client=client, name=DEST_BUCKET)
+        dest_blob = self._make_one(DEST_BLOB, bucket=dest_bucket)
+
+        token, rewritten, size = dest_blob.rewrite(source_blob)
+
+        self.assertEqual(token, TOKEN)
+        self.assertEqual(rewritten, 33)
+        self.assertEqual(size, 42)
+
     def test_rewrite_other_bucket_other_name_no_encryption_partial(self):
         from six.moves.http_client import OK