Package: gcloud-aio-pubsub-fsg
Installation$pipinstall--upgradegcloud-aio-pubsubUsageThis Pub/Sub implementation is based ongoogle-cloud-pubsub>= 0.29.4Currently we have only implemented an asyncio version ofSubscriberClientas the subscription pattern does not work with asyncio by default. The official
Google publisher returns a future which is mostly useable as-is. We’ve not yet
seen a need to build a non-asyncio threadsafe version of the library – the
upstream Google libraries have this well-handled.Here’s the rough usage pattern for subscribing:fromgcloud.aio.pubsubimportSubscriberClientfromgoogle.cloud.pubsub_v1.subscriber.messageimportMessageclient=SubscriberClient()# create subscription if it doesn't already existclient.create_subscription('subscription_name','topic_name')asyncdefmessage_callback(message:Message)->None:try:# just an example: process the message however you need to here...result=handle(message)awaitupload_result(result)exceptException:message.nack()else:message.ack()# subscribe to the subscription, receiving a Future that acts as a keepalivekeep_alive=client.subscribe('subscription_name',message_callback)# have the client run forever, pulling messages from this subscription,# passing them to the specified callback function, and wrapping it in an# asyncio task.client.run_forever(keep_alive)ConfigurationOur create_subscription method is a thing wrapper and thus supports all keyword
configuration arguments from the official pubsub client which you can find in
theofficial Google documentation.When subscribing to a subscription you can optionally pass in aFlowControland/orSchedulerinstance.example_flow_control=FlowControl(max_messages=1,resume_threshold=0.8,max_request_batch_size=1,max_request_batch_latency=0.1,max_lease_duration=10,)keep_alive=client.subscribe('subscription_name',message_callback,flow_control=example_flow_control)Understanding how modifyingFlowControlaffects how your pubsub runtime
will operate can be confusing so here’s a handy dandy guide!Welcome to @TheKevJames’s guide to configuring Google Pubsub Subscription
policies! Settle in, grab a drink, and stay a while.The Subscriber is controlled by a FlowControl configuration tuple definedhere:
that configuration objectfgets used by the Subscriber in the following
ways:Max ConcurrencyThe subscriber is allowed to lease new tasks whenever its currently leased
tasksxsatisfy:((len(x)<f.resume_threshold*f.max_messages)and(sum(x.bytes)<f.resume_threshold*f.max_bytes))In practice, this means we should set these values with the following
restrictions:the maximum number of concurrently leased tasks at peak is:= (f.max_messages * f.resume_threshold) + f.max_request_batch_sizethe maximum memory usage of our leased tasks at peak is:= (f.max_bytes * f.resume_threshold) + (f.max_request_batch_size * bytes_per_task)these values are constrain each other, ie. we limit ourselves to the lesser
of these values given:max_tasks * bytes_per_task <> max_memoryAside: it seems like OCNs on Pubsub are ~1538 bytes eachLeasing RequestsWhen leasing new tasks, theSubscriberuses the following algorithm:deflease_more_tasks():start=time.now()yieldqueue.Queue.get(block=True)# always returns >=1for_inrange(f.max_request_batch_size-1):elapsed=time.now()-startyieldqueue.Queue.get(block=False,timeout=f.max_request_batch_latency-elapsed)ifelapsed>=f.max_request_batch_latency:breakIn practice, this means we should setf.max_request_batch_sizegiven the
above concurrent concerns and setf.max_request_batch_latencygiven
whatever latency ratio we are willing to accept.The expected best-case time forQueue.get()off a full queue is no worse
than 0.3ms. This Queue should be filling up as fast as grpc can make requests
to Google Pubsub, which should be Fast Enough(tm) to keep it filled, giventhoserequests are batched.Therefore, we can expect:avg_lease_latency:~= f.max_request_batch_size * 0.0003worst_case_latency:~= f.max_request_batch_latencyNote that leasing occurs based onf.resume_threshold, so some of this
latency is concurrent with task execution.Task ExpiryAny task which has not been acked or nacked counts against the current leased
task count. Our worker thread should ensure all tasks are acked or nacked, but
theFlowControlconfig allows us to handle any other cases. Note that
leasing works as follows:When a subscriber leases a task, Google Pubsub will not re-lease that
task untilsubscription.ack_deadline_seconds = 10(configurable
per-subscription) seconds have passed.If a client callsack()on a task, it is immediately removed from Google
Pubsub.If a client callsnack()on a task, it immediately allows Google Pubsub
to re-lease that task to a new client. The client drops the task from its
memory.Iff.max_lease_durationpasses between a message being leased and acked,
the client will send anack(see above workflow). It will NOT drop the
task from its memory – eg. theworker(task)process may still be run.Notes:all steps are best-effort, eg. read “a task will be deleted” as “a task will
probably get deleted, if the distributed-system luck is with you”in the above workflow “Google Pubsub” refers to the server-side system, eg.
managed by Google where the tasks are actually stored.In practice, we should thus setf.max_lease_durationto no lower than
our 95% percentile task latency at high load. The lower this value is,
the better our throughput will be in extreme cases.Confusionf.max_requestsis defined, but seems to be unused.ContributingPlease see ourcontributing guide.