Repository: googleapis/google-cloud-python

Issue:
PubSub JSON Double base64 Encode 
We recently upgraded from 0.18.1 to 0.21.0 and we do *not* use gRPC/GAX/Protobuf fun times, but we use the JSONPublisherApi.

Maybe we are using the topic.Batch().Commit() method wrong, but we have been using it in our retry strategy to just simply call Commit() again.

With the change in #2577 it seems that Commit() will double encode *if* you just call Commit() again on a network failure.

Is this the intended design? Are we supposed to keep another copy of the message batch or create another message batch?


Discussion:
@HakShak You reference "the change in #2577" but that was actually a bug report of this same issue, which presumably was fixed in #2578. Can you give a snippet of code showing the issue?
```python
topic_batch = topic.batch()
topic_batch.publish(somedata, someattributes)

try:
    topic_batch.commit()
except NetworkErrorOfSomeKind:
    topic_batch.commit()
# Imagine a proper retry loop here.
```

Ultimately this: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0.18.1/gcloud/pubsub/topic.py#L442

Changed to this: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub/_http.py#L206

The base64 encoding was done on `Batch.Publish()`, but now is done in `Batch.Commit()`
I can test this more later on today.

We could return a copy of the messages from `_transform_messages_base64()`, instead of editing them in place, or we might have to flag the messages that are encoded already.
We ended up copying the Batch.messages before trying commit. The flag seems more "efficient".

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub/_http.py b/pubsub/google/cloud/pubsub/_http.py
--- a/pubsub/google/cloud/pubsub/_http.py
+++ b/pubsub/google/cloud/pubsub/_http.py
@@ -15,6 +15,7 @@
 """Create / interact with Google Cloud Pub/Sub connections."""
 
 import base64
+import copy
 import functools
 import os
 
@@ -203,9 +204,10 @@ def topic_publish(self, topic_path, messages):
         :rtype: list of string
         :returns: list of opaque IDs for published messages.
         """
-        _transform_messages_base64(messages, _base64_unicode)
+        messages_to_send = copy.deepcopy(messages)
+        _transform_messages_base64(messages_to_send, _base64_unicode)
         conn = self._connection
-        data = {'messages': messages}
+        data = {'messages': messages_to_send}
         response = conn.api_request(
             method='POST', path='/%s:publish' % (topic_path,), data=data)
         return response['messageIds']


Test patch:
diff --git a/pubsub/unit_tests/test__http.py b/pubsub/unit_tests/test__http.py
--- a/pubsub/unit_tests/test__http.py
+++ b/pubsub/unit_tests/test__http.py
@@ -284,10 +284,32 @@ def test_topic_publish_hit(self):
         msg_data = connection._called_with['data']['messages'][0]['data']
         self.assertEqual(msg_data, B64_PAYLOAD)
 
+    def test_topic_publish_twice(self):
+        import base64
+
+        PAYLOAD = b'This is the message text'
+        B64_PAYLOAD = base64.b64encode(PAYLOAD).decode('ascii')
+        MESSAGE = {'data': PAYLOAD, 'attributes': {}}
+        RETURNED = {'messageIds': []}
+        connection = _Connection(RETURNED, RETURNED)
+        client = _Client(connection, self.PROJECT)
+        api = self._make_one(client)
+
+        api.topic_publish(self.TOPIC_PATH, [MESSAGE])
+        api.topic_publish(self.TOPIC_PATH, [MESSAGE])
+
+        messages = connection._called_with['data']['messages']
+        self.assertEqual(len(messages), 1)
+        self.assertEqual(messages[0]['data'], B64_PAYLOAD)
+
     def test_topic_publish_miss(self):
+        import base64
         from google.cloud.exceptions import NotFound
+
         PAYLOAD = b'This is the message text'
+        B64_PAYLOAD = base64.b64encode(PAYLOAD).decode('ascii')
         MESSAGE = {'data': PAYLOAD, 'attributes': {}}
+        B64MSG = {'data': B64_PAYLOAD, 'attributes': {}}
         connection = _Connection()
         client = _Client(connection, self.PROJECT)
         api = self._make_one(client)
@@ -299,7 +321,7 @@ def test_topic_publish_miss(self):
         path = '/%s:publish' % (self.TOPIC_PATH,)
         self.assertEqual(connection._called_with['path'], path)
         self.assertEqual(connection._called_with['data'],
-                         {'messages': [MESSAGE]})
+                         {'messages': [B64MSG]})
 
     def test_topic_list_subscriptions_no_paging(self):
         from google.cloud.pubsub.topic import Topic