Repository: googleapis/google-cloud-python

Issue:
GAX API hangs during (second) 'publish'
/cc @bjwatson 

``` python
======================================================================
ERROR: test_message_pull_mode_e2e (pubsub.TestPubsub)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/system_tests/pubsub.py", line 157, in test_message_pull_mode_e2e
    topic.publish(MESSAGE_1, extra=EXTRA_1)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/pubsub/topic.py", line 246, in publish
    message_ids = api.topic_publish(self.full_name, [message_data])
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/pubsub/_gax.py", line 171, in topic_publish
    event.wait()
AttributeError: 'exceptions.AttributeError' object has no attribute 'message_ids'
```



Discussion:
To reproduce:

``` bash
$ git clone git@github.com:tseaver/gcloud
$ cd gcloud
$ git checkout 1855-moar_gax_paging_fixes
$ tox -e system-tests --notest
$ PYTHONPATH= .tox/system-tests/bin/python system_tests/run_system_test.py --package=pubsub
```

Seems like the [PublishResponse](http://pythonhosted.org/gax-google-pubsub-v1/google.pubsub.v1.pubsub_pb2.html#google.pubsub.v1.pubsub_pb2.PublishResponse) object is being returned without `message_ids` nor an exception.

Have you gotten the API `publish(...)` call working in any context? IOW, is this a problem is general, or a problem in this specific context?

@bjwatson I had an earlier error in my system test masking this problem, so I have never actually seen `publish()` succeed with GAX.

@bjwatson One thing different from the other Pubsub APIs (which are working AFAICS, except `pull()` which is masked by this one) is that `push()` gets bundled:  the `AttributeError` gets set by the code which unpacks the response during `event.wait()`.

@bjwatson could it be an issue that the event unpacking code isn't handling repeated elements (e.g., `PublishResponse.message_ids`) properly?

@bjwatson To reproduce (updated):

``` bash
$ git clone git@github.com:GoogleCloudPlatform/gcloud
$ cd gcloud
$ tox -e system-tests --notest
$ GCLOUD_ENABLE_GAX=1 PYTHONPATH= .tox/system-tests/bin/python system_tests/run_system_test.py --package=pubsub
```

cc @tbetbetbe

Hi @tseaver. Sorry for the delay; I was looking into a few other things. I'll try what you suggested and let you know what I find.

I have successfully reproduced this issue on my workstation and am looking into it.

So what I've discovered so far is that `event.result` is assigned to `AttributeError: Assignment not allowed to repeated field "message_ids" in protocol message object.`

Based on code analysis, I believe this is being thrown by https://github.com/googleapis/gax-python/blob/master/google/gax/bundling.py#L193. I'm not sure why we haven't discovered this before. Is this the first time the `bundle_descriptor.subresponse_field` is a repeated field, or should that always be the case? (I don't understand bundling that well, yet)

The bundling configuration for the `publish(...)` call is:

```
    bundling:
      thresholds:
        element_count_threshold: 10
        element_count_limit: 1000 # TO BE REMOVED LATER
        request_byte_threshold: 1024 # 1 Kb
        request_byte_limit: 10485760 # TO BE REMOVED LATER
        delay_threshold_millis: 10
      bundle_descriptor:
        bundled_field: messages
        discriminator_fields:
        - topic
        subresponse_field: message_ids
```

I'm about to sign off for the night. @tbetbetbe Do you have some time to look into this during your morning shift?

@tseaver This is a bug in gax-python. I created issue https://github.com/googleapis/gax-python/issues/110 to track it, and plan to fix it today.

@bjwatson Thanks very much!

@tseaver No problem. I have a fix in code review now: https://github.com/googleapis/gax-python/pull/111

Once this is merged, I will publish new versions of gax-python, pubsub, and logging. And then I will verify that this issue is closed. This should be finished tomorrow.

@tseaver I have fixed this issue with https://pypi.python.org/pypi/google-gax/0.12.1 and https://pypi.python.org/pypi/gax-google-pubsub-v1/0.7.10.

Now the behavior I'm seeing is that the `test_message_pull_mode_e2e` test hangs indefinitely on the [_second_ call to `publish(...)`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/system_tests/pubsub.py#L158). I'm not sure where in the stack the problem is occurring.

I'm about to wrap up for the day. Can you take a look and see what you think?

I also updated logging, although I have not yet looked into https://github.com/GoogleCloudPlatform/gcloud-python/issues/1889

@tseaver I got caught up in some other things today. Can you or @daspecster look into the indefinite hang on the _second_ call to `publish(...)`, and let me know whether you think it's at the gcloud, gapic, or service layer?

This seems to be a [gax-python](https://github.com/googleapis/gax-python issue). The hang is happening at https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/pubsub/_gax.py#L170, but only happens for the second call.

I'll look at it some more on Monday, unless @tbetbetbe or @geigerj get to it first.

- This is indeed a gax-python issue. The problem is that the delay threshold for bundling is handled using a timer object, but we forget to reset the timer object after it fires the first time... Therefore, [this condition](https://github.com/googleapis/gax-python/blob/master/google/gax/bundling.py#L335) will only ever succeed once. I've filed googleapis/gax-python#113 for this.
- https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/pubsub/_gax.py#L170 is somewhat roundabout... The intended way of achieving the same effect in GAX is just

```
try:
  result = self._gax_api.publish(topic_path, message_pbs,
      options=CallOptions(is_bundling=False))
  return result.message_ids
except:
  ...
```


Resolving patch:
diff --git a/gcloud/pubsub/_gax.py b/gcloud/pubsub/_gax.py
--- a/gcloud/pubsub/_gax.py
+++ b/gcloud/pubsub/_gax.py
@@ -162,17 +162,17 @@ def topic_publish(self, topic_path, messages):
         :raises: :exc:`gcloud.exceptions.NotFound` if the topic does not
                     exist
         """
+        options = CallOptions(is_bundling=False)
         message_pbs = [_message_pb_from_dict(message)
                        for message in messages]
         try:
-            event = self._gax_api.publish(topic_path, message_pbs)
-            if not event.is_set():
-                event.wait()
+            result = self._gax_api.publish(topic_path, message_pbs,
+                                           options=options)
         except GaxError as exc:
             if exc_to_code(exc.cause) == StatusCode.NOT_FOUND:
                 raise NotFound(topic_path)
             raise
-        return event.result.message_ids
+        return result.message_ids
 
     def topic_list_subscriptions(self, topic_path, page_size=0,
                                  page_token=None):


Test patch:
diff --git a/gcloud/_testing.py b/gcloud/_testing.py
--- a/gcloud/_testing.py
+++ b/gcloud/_testing.py
@@ -59,17 +59,3 @@ def __init__(self, items, page_token):
     def next(self):
         items, self._items = self._items, None
         return items
-
-
-class _GAXBundlingEvent(object):
-
-    result = None
-
-    def __init__(self, result):
-        self._result = result
-
-    def is_set(self):
-        return self.result is not None
-
-    def wait(self, *_):
-        self.result = self._result
diff --git a/gcloud/pubsub/test__gax.py b/gcloud/pubsub/test__gax.py
--- a/gcloud/pubsub/test__gax.py
+++ b/gcloud/pubsub/test__gax.py
@@ -204,15 +204,12 @@ def test_topic_delete_error(self):
 
     def test_topic_publish_hit(self):
         import base64
-        from gcloud._testing import _GAXBundlingEvent
         PAYLOAD = b'This is the message text'
         B64 = base64.b64encode(PAYLOAD).decode('ascii')
         MSGID = 'DEADBEEF'
         MESSAGE = {'data': B64, 'attributes': {}}
         response = _PublishResponsePB([MSGID])
-        event = _GAXBundlingEvent(response)
-        event.wait()  # already received result
-        gax_api = _GAXPublisherAPI(_publish_response=event)
+        gax_api = _GAXPublisherAPI(_publish_response=response)
         api = self._makeOne(gax_api)
 
         resource = api.topic_publish(self.TOPIC_PATH, [MESSAGE])
@@ -223,29 +220,7 @@ def test_topic_publish_hit(self):
         message_pb, = message_pbs
         self.assertEqual(message_pb.data, B64)
         self.assertEqual(message_pb.attributes, {})
-        self.assertEqual(options, None)
-
-    def test_topic_publish_hit_with_wait(self):
-        import base64
-        from gcloud._testing import _GAXBundlingEvent
-        PAYLOAD = b'This is the message text'
-        B64 = base64.b64encode(PAYLOAD).decode('ascii')
-        MSGID = 'DEADBEEF'
-        MESSAGE = {'data': B64, 'attributes': {}}
-        response = _PublishResponsePB([MSGID])
-        event = _GAXBundlingEvent(response)
-        gax_api = _GAXPublisherAPI(_publish_response=event)
-        api = self._makeOne(gax_api)
-
-        resource = api.topic_publish(self.TOPIC_PATH, [MESSAGE])
-
-        self.assertEqual(resource, [MSGID])
-        topic_path, message_pbs, options = gax_api._publish_called_with
-        self.assertEqual(topic_path, self.TOPIC_PATH)
-        message_pb, = message_pbs
-        self.assertEqual(message_pb.data, B64)
-        self.assertEqual(message_pb.attributes, {})
-        self.assertEqual(options, None)
+        self.assertEqual(options.is_bundling, False)
 
     def test_topic_publish_miss_w_attrs_w_bytes_payload(self):
         import base64
@@ -264,7 +239,7 @@ def test_topic_publish_miss_w_attrs_w_bytes_payload(self):
         message_pb, = message_pbs
         self.assertEqual(message_pb.data, B64)
         self.assertEqual(message_pb.attributes, {'foo': 'bar'})
-        self.assertEqual(options, None)
+        self.assertEqual(options.is_bundling, False)
 
     def test_topic_publish_error(self):
         import base64
@@ -283,7 +258,7 @@ def test_topic_publish_error(self):
         message_pb, = message_pbs
         self.assertEqual(message_pb.data, B64)
         self.assertEqual(message_pb.attributes, {})
-        self.assertEqual(options, None)
+        self.assertEqual(options.is_bundling, False)
 
     def test_topic_list_subscriptions_no_paging(self):
         from google.gax import INITIAL_PAGE