Repository: googleapis/google-cloud-python

Issue:
pubsub should have a default flow controller
re: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5058#issuecomment-378412550

PubSub should come configured with a default flow controller and some arbitrary default max messages (several other languages use 1000 as the number), with documentation around tuning it for higher-throughput workloads (which requires more memory).


Discussion:
What happens for you? Does it just stop pulling for you?

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/client.py b/pubsub/google/cloud/pubsub_v1/subscriber/client.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/client.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/client.py
@@ -160,8 +160,13 @@ def subscribe_experimental(
         the callback during processing, the exception is logged and the message
         is ``nack()`` ed.
 
-        The ``flow_control`` argument can be used to control the rate of
-        message processing.
+        The ``flow_control`` argument can be used to control the rate of at
+        which messages are pulled. The settings are relatively conservative by
+        default to prevent "message hoarding" - a situation where the client
+        pulls a large number of messages but can not process them fast enough
+        leading it to "starve" other clients of messages. Increasing these
+        settings may lead to faster throughput for messages that do not take
+        a long time to process.
 
         This method starts the receiver in the background and returns a
         *Future* representing its execution. Waiting on the future (calling
diff --git a/pubsub/google/cloud/pubsub_v1/types.py b/pubsub/google/cloud/pubsub_v1/types.py
--- a/pubsub/google/cloud/pubsub_v1/types.py
+++ b/pubsub/google/cloud/pubsub_v1/types.py
@@ -14,7 +14,6 @@
 
 from __future__ import absolute_import
 import collections
-import psutil
 import sys
 
 from google.api_core.protobuf_helpers import get_messages
@@ -58,13 +57,13 @@
      'max_lease_duration'],
 )
 FlowControl.__new__.__defaults__ = (
-    psutil.virtual_memory().total * 0.2,  # max_bytes: 20% of total RAM
-    float('inf'),                         # max_messages: no limit
-    0.8,                                  # resume_threshold: 80%
-    100,                                  # max_requests: 100
-    100,                                  # max_request_batch_size: 100
-    0.01,                                 # max_request_batch_latency: 0.01s
-    2 * 60 * 60,                          # max_lease_duration: 2 hours.
+    100 * 1024 * 1024,    # max_bytes: 100mb
+    100,                  # max_messages: 100
+    0.8,                  # resume_threshold: 80%
+    100,                  # max_requests: 100
+    100,                  # max_request_batch_size: 100
+    0.01,                 # max_request_batch_latency: 0.01s
+    2 * 60 * 60,          # max_lease_duration: 2 hours.
 )
 
 
diff --git a/pubsub/setup.py b/pubsub/setup.py
--- a/pubsub/setup.py
+++ b/pubsub/setup.py
@@ -31,7 +31,6 @@
 dependencies = [
     'google-api-core[grpc]<2.0.0dev,>=0.1.3',
     'grpc-google-iam-v1<0.12dev,>=0.11.1',
-    'psutil<6.0dev,>=5.2.2',
 ]
 extras = {
 }


Test patch: