Package: psq
Notethis is not an official Google product, experimental or otherwise and
is provided without support. It is intended as a sample library for
demonstrating a set of use cases for Google Cloud Pub/Sub. The official
Pub/Sub client library should be used for production applications.psqis an example Python implementation of a simple distributed task
queue usingGoogle Cloud Pub/Sub.psqrequires minimal configuration and relies on Cloud Pub/Sub to
provide scalable and reliable messaging.psqis intentionally similar torqandsimpleq, and takes some
inspiration fromceleryandthis
blog
post.InstallationInstall viapip:pip install psqPrerequisitesA project on theGoogle Developers
Console.TheGoogle Cloud SDKinstalled
locally.You will need theCloud Pub/Sub API
enabledon your project. The link will walk you through enabling the API.You will need to rungcloud authbefore running these examples so
that authentication to Google Cloud Platform services is handled
transparently.UsageFirst, create a task:defadder(a,b):returna+bThen, create a pubsub client and a queue:fromgoogle.cloudimportpubsub_v1importpsqproject='your-project-id'publisher=pubsub_v1.PublisherClient()subscriber=pubsub_v1.SubscriberClient()q=psq.Queue(publisher,subscriber,project)Now you can enqueue tasks:fromtasksimportadderq.enqueue(adder)In order to get task results, you have to configure storage:fromgoogle.cloudimportpubsub_v1fromgoogle.cloudimportdatastoreimportpsqproject='your-project-id'publisher=pubsub_v1.PublisherClient()subscriber=pubsub_v1.SubscriberClient()ds_client=datastore.Client()q=psq.Queue(publisher,subscriber,project,storage=psq.DatastoreStorage(ds_client))With storage configured, you can get the result of a task:r=q.enqueue(adder,5,6)r.result()# -> 11You can also define multiple queues:fast=psq.Queue(publisher,subscriber,project,'fast')slow=psq.Queue(publisher,subscriber,project,'slow')Things to noteBecausepsqis largely similar torq, similar rules around tasks
apply. You can put any Python function call on a queue, provided:The function is importable by the worker. This means the__module__that the function lives in must be importable.
Notably, you can’t enqueue functions that are declared in themainmodule - such as tasks defined in a file that is run
directly withpythonor via the interactive interpreter.The function can be a string, but it must be the absolutely importable path
to a function that the worker can import. Otherwise, the task will fail.The worker and the applications queuing tasks must share exactly the
same source code.The function can’t depend on global context such as global variables,
current_request, etc. Pass any needed context into the worker at
queue time.Delivery guaranteesPub/sub guarantees your tasks will be delivered to the workers, butpsqdoesn’t presently guarantee that a task completes execution or
exactly-once semantics, though it does allow you to provide your own
mechanisms for this. This is similar to Celery’sdefaultconfiguration.Task completion guarantees can be provided via late ack support. Late
ack is possible with Cloud Pub/sub, but it currently not implemented in
this library. SeeCONTRIBUTING.md.There are many approaches for exactly-once semantics, such as
distributed locks. This is possible in systems such aszookeeperandredis.Running a workerExecutepsqworkerin thesame directory where you tasks are
defined:psqworker.py config.qpsqworkeronly operates on one queue at a time. If you want a server
to listen to multiple queues, use something likesupervisordto run multiplepsqworkerprocesses.Broadcast queuesA normal queue will send a single task to a single worker, spreading
your tasks over all workers listening to the same queue. There are also
broadcast queues, which will deliver a copy of the task toeveryworker. This is useful in situations where you want every worker to
execute the same task, such as installing or upgrading software on every
server.broadcast_q=psq.BroadcastQueue(publisher,subscriber,project)defrestart_apache_task():call(["apachectl","restart"])broadcast_q.enqueue(restart_apache_task)Broadcast queues provide an implementation of the solution described inReliable Task Scheduling on Google Compute
Engine.Note: broadcast queues do not currently support any form of storage
and do not support return values.RetriesRaisingpsq.Retryin your task will cause it to be retried.frompsqimportRetrydefretry_if_fail(self):try:r=requests.get('http://some.flaky.service.com')exceptExceptionase:logging.error(e)raiseRetry()Flask & other contextsYou can bind an extra context manager to the queue.app=Flask(__name__)q=psq.Queue(extra_context=app.app_context)This will ensure that the context is available in your tasks, which is
useful for things such as database connections, etc.:fromflaskimportcurrent_appdefflasky_task():backend=current_app.config['BACKEND']Bypassing workers for testingDuring unit tests you most certainly don’t want to spin up workers, but instead
execute the enqueued functions immediately and synchronously. To do this, passasynchronous=Falseto the Queue’s constructor (default is True). Also, you don’t have
to provide a publisher, subscriber or project arguments in this case,
just pass None for all them to the queue.q=psq.Queue(None,None,project=None,asynchronous=False)r=q.enqueue(adder,1,2)# Will be run immediatelyIdeas for improvementssome sort of storage solution for broadcast queues.Memcache/redis value store.@task decorator that adds a delay/defer function.Task chaining / groups / chords.Late ack.Gevent worker.batch support for queueing.Contributing changesSeeCONTRIBUTING.mdLicensingApache 2.0 - SeeLICENSE