Package: tg-redis-queue
Thorgate :: Redis QueueRedis queue, that can be easily used to store json-compatible dictionaries and is aimed to be race-condition proof.InstallationTo use the synchronous version:python3-mpipinstalltg-redis-queue[synchronous]To use asynchronous version:python3-mpipinstalltg-redis-queue[asyncio]Example usagefromtg_redis_queue.sync_redis_queueimportRedisObjectQueueclassMyRedisObjectQueue(RedisObjectQueue):# Default expiry time is 60 seconds, set it to 5 minutes for example purposes# so that queue doesn't expire while we're playing around with it.## It is possible to set this to None as well, then queue will never expire and# you are responsible for cleaning it up with `.prune()`EXPIRY_TIME=300def_get_redis_url(self):# Can alternatively use base RedisObjectQueue and provide# redis_url to constructor, like# RedisObjectQueue(name="test", redis_url="redis://localhost:6379")return"redis://localhost:6379"# Add items to the queuequeue=MyRedisObjectQueue(name='example_que')queue.add({'key':1})queue.add({'key':2})queue.add({'key':3})queue.add({'key':4})queue.add({'key':5})# Can be in separate thread or processqueue=MyRedisObjectQueue(name='example_que')# Get items with offset 0, 1 and 2items=queue.get_items(end=2)print([item.dataforiteminitems])# [{'key': 1}, {'key': 2}, {'key': 3}]print(queue.remove(items))# 3 - number of items removed# Can use pop as wellitem=queue.pop()print(item.data)# {'key': 4}print(queue.get_total_size())# 1 - only {'key': 5} is left# Can prune all the dataqueue.prune()print(queue.get_total_size())# 0 - nothing leftIt is possible to use it the queue with async redis (with use of aioredis package).importasynciofromtg_redis_queue.async_redis_queimportAsyncRedisObjectQueueasyncdefenqueue_data():queue=awaitAsyncRedisObjectQueue.create(name="my_queue",redis_url="redis://localhost:6379",)awaitasyncio.gather(queue.add({"id":1}),queue.add({"id":2}),queue.add({"id":3}),queue.add({"id":4}),queue.add({"id":5}),)awaitqueue.cleanup_connection()asyncdefconsume_queue_data():queue=awaitAsyncRedisObjectQueue.create(name="my_queue",redis_url="redis://localhost:6379",)print(awaitqueue.pop())awaitqueue.cleanup_connection()loop=asyncio.get_event_loop()loop.run_until_complete(enqueue_data())# Queue now has 5 itemsloop.run_until_complete(consume_queue_data())# Prints first item in the queueIt is possible to manually set the key for queue items. Items with the same key
can only exist in the queue once, and if an item is added to the queue again with
the same key, item data will be updated but item score (position in the queue)
will be preserved - this way you can update the item in the queue without resetting
it's queue position:fromtg_redis_queue.sync_redis_queueimportRedisObjectQueuefromtg_redis_queueimportRedisQueItemqueue=RedisObjectQueue(name="my-queue",redis_url="redis://localhost:6379")# Add item with explicit key to the queuequeue.add(RedisQueItem(key=f"item-1",data={"id":1,"message":"test"},))# Queue now: ["item-1"]queue.add(RedisQueItem(key=f"item-2",data={"id":2,"message":"test"},))# Queue now: ["item-1", "item-2"]queue.add(RedisQueItem(key=f"item-1",data={"id":42,"message":"test again"},))# Queue ordering is unchanged: ["item-1", "item-2"]item=queue.pop()# Data is updatedassertitem.data=={"id":42,"message":"test again"}AuthorsThis package is developed and maintained byThorgateas
part of our effort to change the world with the power of technology.See our other projects:https://github.com/thorgatehttps://gitlab.com/thorgate-publicContributingTo start development, clone the repository and runmake setup. It expects you to
have python3.8 and poetry installed.You will need to setREDIS_URLenvironment variable to run the tests:exportREDIS_URL=redis://localhost:6379The easiest way to run redis is to run it with Docker:dockerrun--namemy-redis-container-p6379:6379-dredisCode-formatters are available to make the code more readable and uniform, as well as
linters to make sure the code is good quality. See Makefile for details.The following command will re-format the codemakeblack-format-allisort-fixThe following command will check the code with linters and testsmakequalitycoverageFor testing in different environments, tox is used. For convenience, tox is ran in
gitlab pipeline.Please make sure your commit passes all the checks before opening a merge request.Please consider adding yourself to authors inpyptoject.tomlif your contribution
is beyond trivial changes.ChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a Changelog,
and this project adheres toSemantic Versioning.1.1.3- 2021-06-30ChangedReordered pyproject.toml so that poetry sections are on the topMade package homepage to be git repository for easier accessFixedTypo in readmeWrong dates in this changelog (we were lost in time, 2020->2021)Remove MANIFEST.in since pyproject.toml made it obsoleteAdded missing dataclasses dependency1.1.2- 2021-06-10AddedMore examples in readme on how to manually override the key for itemFixedTypo in readme in aioredis exampleLinks in changelog to show diffsExamples providing not accurate data on how get_items upper limit works1.1.1- 2021-06-10FixedPackage description gone in pypi1.1.0- 2021-06-10AddedAsynchronous redis queue implementationAbility to override the serialization logic in a subclass (dump_dataandload_data) to
allow using some alternative to jsonThis changelogChangedPackage no longer depends on redis directly (since asynchronous version uses aioredis), to
install with proper dependencies usesynchronousorasyncioextra requirements (seeREADME)Due to different implementations requiring different packages, is no longer possible to import directly from
tg_redis_queue:from tg_redis_queue.sync_redis_queue import RedisObjectQueueinstead offrom tg_redis_queue import RedisObjectQueue1.0.0- 2021-05-28AddedSynchronous redis queue implementation, extracted from existing non-library codeTests, and pipeline configuration for running the tests in CICode quality checks and formatters (isort, black, prospector)