Repository: googleapis/google-cloud-python

Issue:
PubSub: Subscribing to a Pub/Sub topic requires additional permissions as a result of the release of google-cloud-pubsub 1.0.1
Prior to todays release of the pubsub client, we were able to subscribe with only `roles/pubsub.subscriber` on the subscription we were trying to subscribe to. Now there is a call to get the subscription, which isn't permitted by that role.

It looks like this commit introduced the issue:
https://github.com/googleapis/google-cloud-python/pull/9268/files#diff-1004b3646a787f6b0438604df260553aR387

So after that release, our application broke (we happened to do a release a few hours after yours that picked up the latest pubsub library from pypi) because it did not have permission to get the subscription's metadata. After investigating for a while we were able to resolve the issue by adding `roles/pubsub.viewer` to the subscription in question. We had to enable Pub/Sub audit logs to get enough details to identify what was going on. Here's an example of the log that pointed us to the permission we were missing:

```
 {
   "protoPayload": {
     "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
     "status": {
       "code": 7,
       "message": "PERMISSION_DENIED"
     },
     "authenticationInfo": {
       "principalEmail": "xxx@xxx.iam.gserviceaccount.com"
     },
     "requestMetadata": {
       "callerIp": "xxx",
       "callerSuppliedUserAgent": "grpc-python/1.24.0 grpc-c/8.0.0 (manylinux; chttp2; ganges),gzip(gfe)",
       "callerNetwork": "//compute.googleapis.com/projects/xxx/global/networks/__unknown__",
       "requestAttributes": {
         "time": "2019-09-27T22:41:46.042805867Z",
         "auth": {}
       },
       "destinationAttributes": {}
     },
     "serviceName": "pubsub.googleapis.com",
     "methodName": "google.pubsub.v1.Subscriber.GetSubscription",
     "authorizationInfo": [
       {
         "resource": "projects/xxx/subscriptions/xxx",
         "permission": "pubsub.subscriptions.get",
         "resourceAttributes": {}
       }
     ],
     "resourceName": "projects/xxx/subscriptions/xxx",
     "request": {
       "subscription": "projects/xxx/subscriptions/xxx",
       "@type": "type.googleapis.com/google.pubsub.v1.GetSubscriptionRequest"
     }
   },
   "insertId": "xxx",
   "resource": {
     "type": "pubsub_subscription",
     "labels": {
       "project_id": "xxx",
       "subscription_id": "projects/xxx/subscriptions/xxx"
     }
   },
   "severity": "ERROR",
   "logName": "projects/xxx/logs/cloudaudit.googleapis.com%2Fdata_access",
 }
```


Discussion:
@jceresini Thank you for the report, and I'm glad to hear that you managed to find a solution.

The [linked change](https://github.com/googleapis/google-cloud-python/pull/9268/files#diff-1004b3646a787f6b0438604df260553aR387) is needed to determine the default message acknowledge timeout set on the subscription before establishing a message stream. Previously, the client overrode that setting with a default value, which sometimes resulted in the ACK timeout hitting too soon, resulting in unnecessary premature re-delivery of the messages the client has not processed yet (issue [#9252](https://github.com/googleapis/google-cloud-python/issues/9252)).

This should probably be documented better in the [permissions and roles](https://cloud.google.com/pubsub/docs/access-control#permissions_and_roles) section of the docs, will escalate this thread.
@plamut Would it be possible to catch the 403 error, emit a warning for the developer, and then fallback to the previous incorrect behavior?  This'd make the required cloud change discoverable without digging through Cloud Audit.
@rvandegrift We were just discussing this issue, and the quickest short term solution is to change the [problematic line](https://github.com/googleapis/google-cloud-python/pull/9268/files#diff-1004b3646a787f6b0438604df260553aR387) - instead of fetching the subscription and read its ACK deadline, we can use a reasonable static default, e.g. 60 seconds.

That will, of course, re-introduce #9252, but the latter is a lesser evil than having surprising permission issues. And using e.g. 60 seconds instead of the previous 10 should mitigate #9252 to a degree.

(the goal here is that subscriptions should work out of the box with the default `pubsub.subscriber` role, i.e. without any custom permissions)

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -50,6 +50,13 @@
 _RESUME_THRESHOLD = 0.8
 """The load threshold below which to resume the incoming message stream."""
 
+_DEFAULT_STREAM_ACK_DEADLINE = 60
+"""The default message acknowledge deadline in seconds for incoming message stream.
+
+This default deadline is dynamically modified for the messages that are added
+to the lease management.
+"""
+
 
 def _maybe_wrap_exception(exception):
     """Wraps a gRPC exception class, if needed."""
@@ -384,8 +391,17 @@ def open(self, callback, on_callback_error):
         )
 
         # Create the RPC
-        subscription = self._client.api.get_subscription(self._subscription)
-        stream_ack_deadline_seconds = subscription.ack_deadline_seconds
+
+        # We must use a fixed value for the ACK deadline, as we cannot read it
+        # from the subscription. The latter would require `pubsub.subscriptions.get`
+        # permission, which is not granted to the default subscriber role
+        # `roles/pubsub.subscriber`.
+        # See also https://github.com/googleapis/google-cloud-python/issues/9339
+        #
+        # When dynamic lease management is enabled for the "on hold" messages,
+        # the default stream ACK deadline should again be set based on the
+        # historic ACK timing data, i.e. `self.ack_histogram.percentile(99)`.
+        stream_ack_deadline_seconds = _DEFAULT_STREAM_ACK_DEADLINE
 
         get_initial_request = functools.partial(
             self._get_initial_request, stream_ack_deadline_seconds


Test patch:
diff --git a/pubsub/tests/system.py b/pubsub/tests/system.py
--- a/pubsub/tests/system.py
+++ b/pubsub/tests/system.py
@@ -381,6 +381,10 @@ class CallbackError(Exception):
         with pytest.raises(CallbackError):
             future.result(timeout=30)
 
+    @pytest.mark.xfail(
+        reason="The default stream ACK deadline is static and received messages "
+        "exceeding FlowControl.max_messages are currently not lease managed."
+    )
     def test_streaming_pull_ack_deadline(
         self, publisher, subscriber, project, topic_path, subscription_path, cleanup
     ):
@@ -395,7 +399,7 @@ def test_streaming_pull_ack_deadline(
         # Subscribe to the topic. This must happen before the messages
         # are published.
         subscriber.create_subscription(
-            subscription_path, topic_path, ack_deadline_seconds=60
+            subscription_path, topic_path, ack_deadline_seconds=240
         )
 
         # publish some messages and wait for completion
@@ -403,7 +407,7 @@ def test_streaming_pull_ack_deadline(
 
         # subscribe to the topic
         callback = StreamingPullCallback(
-            processing_time=15,  # more than the default ACK deadline of 10 seconds
+            processing_time=70,  # more than the default stream ACK deadline (60s)
             resolve_at_msg_count=3,  # one more than the published messages count
         )
         flow_control = types.FlowControl(max_messages=1)
@@ -411,13 +415,13 @@ def test_streaming_pull_ack_deadline(
             subscription_path, callback, flow_control=flow_control
         )
 
-        # We expect to process the first two messages in 2 * 15 seconds, and
+        # We expect to process the first two messages in 2 * 70 seconds, and
         # any duplicate message that is re-sent by the backend in additional
-        # 15 seconds, totalling 45 seconds (+ overhead) --> if there have been
-        # no duplicates in 60 seconds, we can reasonably assume that there
+        # 70 seconds, totalling 210 seconds (+ overhead) --> if there have been
+        # no duplicates in 240 seconds, we can reasonably assume that there
         # won't be any.
         try:
-            callback.done_future.result(timeout=60)
+            callback.done_future.result(timeout=240)
         except exceptions.TimeoutError:
             # future timed out, because we received no excessive messages
             assert sorted(callback.seen_message_ids) == [1, 2]
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
@@ -404,12 +404,9 @@ def test_heartbeat_inactive():
     "google.cloud.pubsub_v1.subscriber._protocol.heartbeater.Heartbeater", autospec=True
 )
 def test_open(heartbeater, dispatcher, leaser, background_consumer, resumable_bidi_rpc):
+    stream_ack_deadline = streaming_pull_manager._DEFAULT_STREAM_ACK_DEADLINE
+
     manager = make_manager()
-    manager._client.api.get_subscription.return_value = types.Subscription(
-        name="projects/foo/subscriptions/bar",
-        topic="projects/foo/topics/baz",
-        ack_deadline_seconds=123,
-    )
 
     manager.open(mock.sentinel.callback, mock.sentinel.on_callback_error)
 
@@ -437,7 +434,8 @@ def test_open(heartbeater, dispatcher, leaser, background_consumer, resumable_bi
     )
     initial_request_arg = resumable_bidi_rpc.call_args.kwargs["initial_request"]
     assert initial_request_arg.func == manager._get_initial_request
-    assert initial_request_arg.args[0] == 123
+    assert initial_request_arg.args[0] == stream_ack_deadline
+    assert not manager._client.api.get_subscription.called
 
     resumable_bidi_rpc.return_value.add_done_callback.assert_called_once_with(
         manager._on_rpc_done