Repository: googleapis/google-cloud-python

Issue:
Every error that is not catchable by the user should be logged via the logging module
Currently, at the very least helper threads for pubsub seem to not and they let the exception bubble up and get printed to stderr. 

I think this is bad form, there's nothing _I_ can do to catch this error, but it'd be nice if I could get it logged in the format I want.


Discussion:
We handle this by explicitly adding a handler to the root logger.

https://googlecloudplatform.github.io/google-cloud-python/latest/logging/usage.html#integration-with-python-logging-module
I think you misunderstood the problem. Firstly, we do have logging set up. Everything above INFO on the root logger goes through our handler. And, yet, we end up errors in the logs that don't have a jsonPayload and none of our metadata, only a textPayload. The errrors I'm talking about look like this:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 54, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 40, in dynamic_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Deadline Exceeded)>
```
and

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 239, in monitor
    return self._commit()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 204, in _commit
    self._messages,
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 325, in publish
    return self._publish(request, retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.6/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Deadline Exceeded
```

As you see, the last one bubbles up all the way to the thread and gets handled there. There's no way for us to catch these and log them correctly. An artefact of the lack of logging support the fact that they're not linked together, since the parser seems to be unable to figure they're linked since we have more than one worker logging there and I guess another process is logging around the same time. In the containers where we only have one process, they get grouped together correctly:

```
Exception in thread Thread-MonitorBatchPublisher:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 54, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 40, in dynamic_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Deadline Exceeded)>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 239, in monitor
    return self._commit()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 204, in _commit
    self._messages,
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 325, in publish
    return self._publish(request, retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.6/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Deadline Exceeded
```


Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/publisher/batch/thread.py b/pubsub/google/cloud/pubsub_v1/publisher/batch/thread.py
--- a/pubsub/google/cloud/pubsub_v1/publisher/batch/thread.py
+++ b/pubsub/google/cloud/pubsub_v1/publisher/batch/thread.py
@@ -20,6 +20,7 @@
 
 import six
 
+import google.api_core.exceptions
 from google.cloud.pubsub_v1 import types
 from google.cloud.pubsub_v1.publisher import exceptions
 from google.cloud.pubsub_v1.publisher import futures
@@ -199,10 +200,24 @@ def _commit(self):
             # Begin the request to publish these messages.
             # Log how long the underlying request takes.
             start = time.time()
-            response = self._client.api.publish(
-                self._topic,
-                self._messages,
-            )
+
+            try:
+                response = self._client.api.publish(
+                    self._topic,
+                    self._messages,
+                )
+            except google.api_core.exceptions.GoogleAPICallError as exc:
+                # We failed to publish, set the exception on all futures and
+                # exit.
+                self._status = base.BatchStatus.ERROR
+
+                for future in self._futures:
+                    future.set_exception(exc)
+
+                _LOGGER.exception(
+                    'Failed to publish %s messages.', len(self._futures))
+                return
+
             end = time.time()
             _LOGGER.debug('gRPC Publish took %s seconds.', end - start)
 
@@ -220,9 +235,14 @@ def _commit(self):
                 self._status = base.BatchStatus.ERROR
                 exception = exceptions.PublishError(
                     'Some messages were not successfully published.')
+
                 for future in self._futures:
                     future.set_exception(exception)
 
+                _LOGGER.error(
+                    'Only %s of %s messages were published.',
+                    len(response.message_ids), len(self._futures))
+
     def monitor(self):
         """Commit this batch after sufficient time has elapsed.
 


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py b/pubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py
--- a/pubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py
+++ b/pubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py
@@ -17,6 +17,7 @@
 
 import mock
 
+import google.api_core.exceptions
 from google.auth import credentials
 from google.cloud.pubsub_v1 import publisher
 from google.cloud.pubsub_v1 import types
@@ -201,6 +202,26 @@ def test_blocking__commit_wrong_messageid_length():
         assert isinstance(future.exception(), exceptions.PublishError)
 
 
+def test_block__commmit_api_error():
+    batch = create_batch()
+    futures = (
+        batch.publish({'data': b'blah blah blah'}),
+        batch.publish({'data': b'blah blah blah blah'}),
+    )
+
+    # Make the API throw an error when publishing.
+    error = google.api_core.exceptions.InternalServerError('uh oh')
+    patch = mock.patch.object(
+        type(batch.client.api), 'publish', side_effect=error)
+
+    with patch:
+        batch._commit()
+
+    for future in futures:
+        assert future.done()
+        assert future.exception() == error
+
+
 def test_monitor():
     batch = create_batch(max_latency=5.0)
     with mock.patch.object(time, 'sleep') as sleep: