Package: pymq
PyMQPyMQ is a simple message-oriented middleware library for implementing Python IPC across machine boundaries. The API
enables different styles of remoting via Pub/Sub, Queues, and synchronous RPC.With PyMQ, developers can integrate Python applications running on different machines in a loosely coupled way over
existing transport mechanisms.
PyMQ currently provides
a Redis backend,
an AWS backend using SQS and SNS,
a POSIX IPC backend for single-machine IPC, and
an in-memory backend for testing.
The API is extensible and other transports can be plugged in.Using PyMQInstallStarting from pymq 0.5, all providers are separated into setuptools extras.
If you want to use the redis backend, then runpip install pymq[redis]Available providers:pymq[redis]Redis providerpymq[aws]AWS provider using SNS and SQSpymq[ipc]Linux IPC providerpymq[full]install all providersInitialize PyMQThe core module manages a global eventbus instance that provides the remoting primitives. The default Redis
implementation uses an event loop over a pubsub object. The global eventbus is initialized viapymq.initand by
passing a provider factory.importpymqfrompymq.provider.redisimportRedisConfig# starts a new thread with a Redis event looppymq.init(RedisConfig())# main application control looppymq.shutdown()This will create an eventbus instance on top of a local Redis server.Pub/SubPub/Sub allows asynchronous event-based communication. Event classes are used to transport state and identify channels.importpymq# common codeclassMyEvent:pass# subscribe code@pymq.subscriberdefon_event(event:MyEvent):print('event received')# publisher codepymq.publish(MyEvent())QueuesQueues are straight forward, as they are compatible to the Pythonqueue.Queuespecification.importpymqqueue=pymq.queue('my_queue')queue.put('obj')print(queue.get())# outputs 'obj'RPCServer codeimportpymq@pymq.remote('product_remote')defproduct(a:int,b:int)->int:# pymq relies on type hints for marshallingreturna*bClient codeimportpymqproduct=pymq.stub('product_remote')product(2,4)# 8With a shared code-base, methods can also be exposed and called by passing the callable. For example,importpymq# common codeclassRemote:defecho(self,param)->None:return'echo: '+param# serverobj=Remote()pymq.expose(obj.echo)# clientecho=pymq.stub(Remote.echo)echo('pymq')# "echo: pymq"If there are multiple providers of the same object, then a stub can be initialized withmulti=Trueto get a list of
results. It may be useful to use a timeout in this case.remote=pymq.stub('remote_method',multi=True,timeout=2)result=remote()# result will be a list containing the results of all invocations of available remote objectsProvidersSimpleEventBusused for testing and rudimentary single-thread dispatchingRedisEventBusworks across network and process boundaries but requires a running redis instanceAwsEventBususes AWS Simple Notification Service (SNS) and Simple Queuing Service (SQS)IpcEventBususesposix_ipcmessage queues as event loops and maintains a tree of topic subscriptions in/run/shm. Useful for eventing across process boundaries without an additional server component.CompatibilityPython 3.8+Known LimitationsJSON serialization relies heavily on type hints. Sending complex types without type hints will cause type errors.There is currently no support for polymorphism with JSON serializationPattern-based topic matching does not work for the in-memory eventbus or the IPC event busYou can only have a limited number of Queues when using the IPC provider, as the kernel limits the number of file
descriptors per processSubscriptions by foreign components to RPC channels will cause issues in multi-call scenariosUsing thepymqsingleton in multiprocessing scenarios may not work as expected because the module holds a Thread in
a global variable. A workaround is to re-start the bus by callingshutdown()andinit()in the forked Process.IPC provider only works for LinuxMulti-RPC does not work for the AWS providerAWS provider cannot automatically delete all queues and topics.BackgroundOriginally part of theSymmetryproject, was extracted as a standalone
library.