Package: persizmq
persizmqpersizmq provides persistence to zeromq. Messages are received in background and stored on disk before further
manipulation.Currently, we only support the zeromq subscriber. Adding support for other classes can be easily done; we simply have
not had need for them so far.UsageSubscriberThe persistent subscriber wraps a zeromq subscriber. We split up the persistence subscription in two components:
a threaded subscriber that listens on the messages in background, and a persistence component that stores the messages
on disk.Threaded SubscriberThe threaded subscriber is implemented aspersizmq.ThreadedSubscriber. You need to specify a callback which is
called upon each received message.You also need to specify on-exception callback in order to handle exceptions raised in the listening thread.Example:importtimeimportzmqimportpersizmqcontext=zmq.Context()subscriber=context.socket(zmq.SUB)subscriber.setsockopt_string(zmq.SUBSCRIBE,"")subscriber.connect("ipc:///some-queue.zeromq")defcallback(msg:bytes)->None:print("received a message:{}".format(msg))defon_exception(exception:Exception)->None:print("an exception was raised in the listening thread:{}".format(exception))withpersizmq.ThreadedSubscriber(callback=callback,subscriber=subscriber,on_exception=on_exception):# do something while we are listening on messages...time.sleep(10)StorageWe provide two storage modes for the received messages:persizmq.PersistentStorage: stores messages in a FIFO queue on disk.persizmq.PersistentLatestStorage: solely stores the newest message on disk.The storage component is passed directly to the threaded subscriber as a callback.Example:importpathlibimportzmqimportpersizmqcontext=zmq.Context()subscriber=context.socket(zmq.SUB)subscriber.setsockopt_string(zmq.SUBSCRIBE,"")subscriber.connect("ipc:///some-queue.zeromq")persistent_dir=pathlib.Path("/some/dir")storage=persizmq.PersistentStorage(persistent_dir=persistent_dir)defon_exception(exception:Exception)->None:print("an exception was raised in the listening thread:{}".format(exception))withpersizmq.ThreadedSubscriber(callback=storage.add_message,subscriber=subscriber,on_exception=on_exception):msg=storage.front()# non-blockingifmsgisnotNone:print("Received a persistent message:{}".format(msg))storage.pop_front()FilteringWe also provide filtering components which can be chained on the threaded subscriber. The filtering chains are
particularly handy if you want to persist only a small amount of messages and ignore the rest.The filters are implemented inpersizmq.filtermodule.Example:importpathlibimportzmqimportpersizmqimportpersizmq.filtercontext=zmq.Context()subscriber=context.socket(zmq.SUB)subscriber.setsockopt_string(zmq.SUBSCRIBE,"")subscriber.connect("ipc:///some-queue.zeromq")persistent_dir=pathlib.Path("/some/dir")storage=persizmq.PersistentStorage(persistent_dir=persistent_dir)defon_exception(exception:Exception)->None:print("an exception was raised in the listening thread:{}".format(exception))withpersizmq.ThreadedSubscriber(lambdamsg:storage.add_message(persizmq.filter.MaxSize(max_size=1000)(msg)),subscriber=subscriber,on_exception=on_exception):msg=storage.front()# non-blockingifmsgisnotNone:print("Received a persistent message:{}".format(msg))storage.pop_front()InstallationCreate a virtual environment:python3-mvenvvenv3Activate it:sourcevenv3/bin/activateInstall persizmq with pip:pip3installpersizmqDevelopmentCheck out the repository.In the repository root, create the virtual environment:python3-mvenvvenv3Activate the virtual environment:sourcevenv3/bin/activateInstall the development dependencies:pip3install-e.[dev]We use tox for testing and packaging the distribution. Assuming that the virtual environment has been activated and
the development dependencies have been installed, run:toxWe also provide a set of pre-commit checks that lint and check code for formatting. Run them locally from an activated
virtual environment with development dependencies:./precommit.pyThe pre-commit script can also automatically format the code:./precommit.py--overwriteVersioningWe followSemantic Versioning. The version X.Y.Z indicates:X is the major version (backward-incompatible),Y is the minor version (backward-compatible), andZ is the patch version (backward-compatible bug fix).