Package: aio-redis-mq
aio_redis_mqLightweight Message Queue & Broker base on async python redis streamsSuitable Application EnvironmentModern software applications have moved from being a single monolithic unit to loosely coupled collections of services.
While this new architecture brings many benefits, those services still need to interact with each other,
creating the need for robust and efficient messaging solutions.The following problems are suitable for using message queuing：Asynchronous processingFlow controlService decouplingConnect flow computingAs a publish / subscribe systemInstallationpipinstallaio-redis-mqQuick Startimportasyncioimporttimefromaio_redis_mqimportMQProducer,MQConsumer_redis_url='redis://root:xxxxx@localhost/1'asyncdefproducer_task(producer):for_inrange(0,10):awaitasyncio.sleep(1)send_msg_id=awaitproducer.send_message({'msg':f'msg_{_}','content':time.strftime("%Y-%m-%d%H:%M:%S")})print(f'producer_task time at{time.strftime("%Y-%m-%d%H:%M:%S")}',f'message id={send_msg_id}')asyncdefconsumer_task(consumer:MQConsumer,consumer_index:int):for_inrange(0,10):msg=awaitconsumer.block_read_messages(block=1500)print(f'consumer_{consumer_index}block read message',msg)asyncdefmain():# one producerproducer=MQProducer('pub_stream',redis_name='_redis_local',redis_url=_redis_url)# three consumerconsumer1=MQConsumer('pub_stream',redis_name='_redis_local',redis_url=_redis_url)consumer2=MQConsumer('pub_stream',redis_name='_redis_local',redis_url=_redis_url)consumer3=MQConsumer('pub_stream',redis_name='_redis_local',redis_url=_redis_url)awaitasyncio.gather(producer_task(producer),consumer_task(consumer1,1),consumer_task(consumer2,2),consumer_task(consumer3,3))if__name__=='__main__':asyncio.run(main())Group Consumerimportasyncioimporttimefromaio_redis_mqimportMQProducer,GroupManager,Group,GroupConsumer_redis_url='redis://root:xxxxx@localhost/1'asyncdefproducer_task(producer):for_inrange(0,10):awaitasyncio.sleep(1)print(f'-------------------------------------{_}-------------------------------------')send_msg_id=awaitproducer.send_message({'msg':f'msg_{_}','content':time.strftime("%Y-%m-%d%H:%M:%S")})print(f'group_producer send_message time at{time.strftime("%Y-%m-%d%H:%M:%S")}',f'message id={send_msg_id}')asyncdefconsumer_task(consumer:GroupConsumer):for_inrange(0,10):# Here we use a low-level read message API and do not detect pending messages or handle idle messagesmsg=awaitconsumer.block_read_messages(count=1,block=1500)awaitasyncio.sleep(0.05)print(f'group_consumer{consumer.consumer_id}group={consumer.group_name}block read message',msg)iflen(msg)>0andlen(msg[0][1])>0:msg_id=msg[0][1][0][0]ack_result=awaitconsumer.ack_message(msg_id)print(f'group_consumer{consumer.consumer_id}group={consumer.group_name}ack message id='f'{msg_id}{"successful"ifack_resultelse"failed"}.')# show infoasyncdefshow_groups_infor(group:Group):print(f'-----------------------------{group.group_name}---------- groups info ------------------------------------')group_info=awaitgroup.get_groups_info()print(f'group name:{group.group_name}groups info :{group_info}')print(f'-----------------------------{group.group_name}--------- consumer info -----------------------------------')consumer_info=awaitgroup.get_consumers_info()print(f'group name:{group.group_name}consumer info :{consumer_info}')print(f'-----------------------------{group.group_name}-------- pending info -------------------------------------')pending_info=awaitgroup.get_pending_info()print(f'group name:{group.group_name}pending info :{pending_info}')asyncdefmain():# create one producerproducer=MQProducer('group_stream1',redis_name='_group_redis_',redis_url=_redis_url)# create group manager , via same stream key, same redis_namegroup_manager=GroupManager('group_stream1',redis_name='_group_redis_',redis_url=_redis_url)# create first groupgroup1=awaitgroup_manager.create_group('group1')# create two consumers in the same groupconsumer1=awaitgroup1.create_consumer('consumer1')consumer2=awaitgroup1.create_consumer('consumer2')# create second groupgroup2=awaitgroup_manager.create_group('group2')# create three consumers in the same groupconsumer3=awaitgroup2.create_consumer('consumer3')consumer4=awaitgroup2.create_consumer('consumer4')consumer5=awaitgroup2.create_consumer('consumer5')awaitasyncio.gather(producer_task(producer),consumer_task(consumer1),consumer_task(consumer2),consumer_task(consumer3),consumer_task(consumer4),consumer_task(consumer5))print('------------------------------------- show total infor -------------------------------------')stream_info=awaitgroup_manager.get_stream_info(group_manager.stream_key)print(f'stream_key:{group_manager.stream_key}stream info :{stream_info}')awaitshow_groups_infor(group1)awaitshow_groups_infor(group2)if__name__=='__main__':asyncio.run(main())More ExampleFor more examples, please query the example folder.About Redis streamsThe Redis Stream is a new data type introduced with Redis 5.0, which models a log data structure in a more abstract way.
Redis Streams doubles as a communication channel for building streaming architectures and as a log-like data structure
for persisting data, making Streams the perfect solution for event sourcing.The stream type published in redis5.0 is also used to implement typical message queues.
The emergence of this stream type meets almost all the requirements of message queues,
including but not limited to:Serialization generation of message IDMessage traversalBlocking and non blocking reading of messagesGroup consumption of messagesUnfinished message processingMessage queue monitoringComparison of basic conceptsCommon distributed message system, including RabbitMQ 、 RocketMQ 、 Kafka 、Pulsar 、Redis streamsRedis streams vs KafkaKafkaRedis StreamsDescriptionRecordMessageObjects to be processed in the message engineProducerProducerClients that publish new messages to topicsConsumerConsumerClients that subscribe to new messages from topicsConsumer GroupConsumer GroupA group composed of multiple consumer instances can consume the same topic at the same time to achieve high throughput.BrokerCluster Nodeservers form the storage layer. Leader-Follower replicaTopicStream Data typeTopics are logical containers that carry messagespartitionsDifferent Redis keysRedis StreamsDifferences with Kafka (TM) partitionsPerformanceYou can use the following tools for performance testing.OpenMessaging Benchmark FrameworkAPI ReferenceMQClientclient for message system, can manage and query messages.__init__(redis_name: Optional[str] = None, redis_url: Optional[str] = None, redis_pool: aioredis.client.Redis = None, **kwargs)create MQ Client instanceredis_name: name for cache redis clientredis_url:  redis server urlredis_pool: aioredis.client.Redis instance, defaults to Noneget_stream_length(stream_key: KeyT)Returns the number of elements in a given stream.stream_key: key of the stream.query_messages(stream_key: KeyT, min_id: StreamIdT = "-", max_id: StreamIdT = "+", count: Optional[int] = None)query message value from min_id to max_id with count limit in a given stream.stream_key: key of the stream.min_id: first stream ID. defaults to '-', meaning the earliest available.max_id: last stream ID. defaults to '+', meaning the latest available.count: if set, only return this many items, beginning with the earliest available.reverse_query_messages(stream_key: KeyT, min_id: StreamIdT = "-", max_id: StreamIdT = "+", count: Optional[int] = None)query message value in reverse order from min_id to max_id with count limit in a given stream.get_stream_info(stream_key: KeyT)Returns general information about the stream.delete_message(stream_key: KeyT,  *ids: StreamIdT)Deletes one or more messages from a stream.stream_key: key of the stream.*ids: message ids to delete.trim_stream(stream_key: KeyT,  maxlen: int, approximate: bool = True)Deletes one or more messages from a stream.stream_key: key of the stream.maxlen: truncate old stream messages beyond this sizemaxlen: actual stream length may be slightly more than maxlenclient=MQClient(redis_name='my_redis',redis_url='redis://root:xxxxx@localhost/0')# get stream lengthstream_length=awaitclient.get_stream_length('_test_stream1')# get stream infostream_info=awaitclient.get_stream_info('_test_stream1')assertstream_info.get('length')==stream_length# get first_message_infofirst_message_info=awaitclient.query_messages('_test_stream1',count=1)# get last_message_infolast_message_info=awaitclient.reverse_query_messages('_test_stream1',count=1)assertfirst_message_info[0]==stream_info.get('first-entry')assertlast_message_info[0]==stream_info.get('last-entry')MQProducer <- MQClientmessage producer, MQClient with a specific stream key__init__(stream_key: KeyT, redis_name: str = None, redis_pool: aioredis.client.Redis = None, **kwargs)message producer in message system based on a specific stream key.stream_key: key of streamredis_name: name for cache redis clientredis_url:  redis server urlredis_pool: aioredis.client.Redis instance, defaults to Nonesend_message(message: Dict[FieldT, EncodableT], msg_id: StreamIdT = "*", maxlen: int = None, approximate: bool = True)Coroutine. send message content to a stream which is a message container, and return message id.message:  dict of field/value pairs to insert into the streammsg_id:  Location to insert this record. By default it is appended.maxlen:  max number of messages, truncate old stream members beyond this sizeapproximate:  actual stream length may be slightly more than maxlenproducer=MQProducer('pub_stream',redis_name='my_redis',redis_url='redis://root:xxxxx@localhost/0')send_msg_id=awaitproducer.send_message({'msg_key1':'value1','msg_key2':'value2'})MQConsumer <- MQClientmessage consumer, MQClient with a specific stream key__init__(stream_key: KeyT, redis_name: str = None, redis_pool: aioredis.client.Redis = None, **kwargs)message consumer in message system based on a specific stream key.stream_key: key of streamredis_name: name for cache redis clientredis_url:  redis server urlredis_pool: aioredis.client.Redis instance, defaults to Noneread_messages(streams: Dict[KeyT, StreamIdT], count: Optional[int] = None)Coroutine. read messages from streams as message containersstreams:  a dict of stream keys to stream IDs, where IDs indicate the last ID already seen.count:  if set, only return this many items, beginning with the earliest available.block_read_messages(*stream_key: KeyT, count: Optional[int] = None, block: Optional[int] = None,)Coroutine. Block and monitor multiple streams for new data.stream_key:  key of the stream.count:  if set, only return this many items, beginning with the earliest available.block:  number of milliseconds to wait, if nothing already present.consumer=MQConsumer('pub_stream',redis_name='my_redis',redis_url='redis://root:xxxxx@localhost/0')# block read new messagenew_msg=awaitconsumer.block_read_messages(block=1500)# read messages from msg_id(0 or other id)  in single stream (pub_stream)read_msgs=awaitconsumer.read_messages({'pub_stream':0},count=10)GroupManager__init__(stream_key: KeyT, redis_name: str = None, **kwargs)group manager in message system based on a specific stream key.stream_key: key of streamredis_name: name for cache redis clientredis_url:  redis server urlcreate_group(group_name: GroupT, msg_id: StreamIdT = "$", mkstream: bool = True)Create a new group consumer associated with a streamgroup_name: name of the consumer groupmsg_id: ID of the last item in the stream to consider already delivered.mkstream:  a boolean indicating whether to create new streamdestroy_group(group_name: GroupT)Destroy a consumer groupgroup_name: name of the consumer groupget_groups_info()Returns general information about the consumer groups of the stream.group_manager=GroupManager('pub_stream',redis_name='my_redis',redis_url='redis://root:xxxxx@localhost/0')# create groupgroup=awaitgroup_manager.create_group('group')Groupcreate_consumer(consumer_id: ConsumerT)create a consumer instance in groupconsumer_id: id of consumer.delete_consumer(consumer_id: ConsumerT)Remove a specific consumer from a consumer group.consumer_id: id of consumer.set_msg_id(msg_id: StreamIdT)Set the consumer group last delivered ID to something else.msg_id: ID of the last item in the stream to consider already deliveredget_groups_info()Returns general information about the consumer groups of the stream.get_consumers_info()Returns general information about the consumers in the group. only return consumer which has read messageget_pending_info()Returns information about pending messages of a group.query_pending_messages(min_msg_id: Optional[StreamIdT], max_msg_id: Optional[StreamIdT], count: Optional[int], consumer_id: Optional[ConsumerT] = None)Returns information about pending messages, in a range.min_msg_id: minimum message IDmax_msg_id: maximum message IDcount: number of messages to returnconsumer_id: id of a consumer to filter by (optional)ack_message(*msg_id: StreamIdT)Acknowledges the successful processing of one or more messages.msg_id: message ids to acknowledge.claim_message(consumer_id: ConsumerT, min_idle_time: int, msg_ids: Union[List[StreamIdT], Tuple[StreamIdT]], idle: Optional[int] = None, time: Optional[int] = None, retrycount: Optional[int] = None, force: bool = False, justid: bool = False)Changes the ownership of a pending message. In the context of a stream consumer group,
this command changes the ownership of a pending message,
so that the new owner is the consumer specified as the command argument.consumer_id: name of a consumer that claims the message.min_idle_time: filter messages that were idle less than this amount of millisecondsmsg_ids: non-empty list or tuple of message IDs to claimidle: Set the idle time (last time it was delivered) of the message in mstime: optional integer. This is the same as idle but instead of a relative amount of milliseconds,
it sets the idle time to a specific Unix time (in milliseconds).retrycount: optional integer. set the retry counter to the specified value.
This counter is incremented every time a message is delivered again.force: optional boolean, false by default. Creates the pending message entry in the PEL
even if certain specified IDs are not already in the PEL assigned to a different client.justid: optional boolean, false by default. Return just an array of IDs of messages successfully
claimed, without returning the actual messageGroupConsumerread_messages(streams: Dict[KeyT, StreamIdT], count: Optional[int] = None, noack: bool = False)Read from a stream via a consumer group.streams: a dict of stream names to stream IDs, where IDs indicate the last ID already seen.count: if set, only return this many items, beginning with the earliest availablenoack: do not add messages to the PEL (Pending Entries List)block_read_messages(*stream_key: KeyT, block: Optional[int] = None, count: Optional[int] = None, noack: bool = False)Block read from a stream via a consumer group.stream_key: a list of stream keyblock: number of milliseconds to wait, if nothing already present.count: if set, only return this many items, beginning with the earliest availablenoack: do not add messages to the PEL (Pending Entries List)query_pending_messages(min_msg_id: Optional[StreamIdT], max_msg_id: Optional[StreamIdT], count: Optional[int])Returns information about pending messages, in a range.min_msg_id: minimum message IDmax_msg_id: maximum message IDcount: number of messages to returnack_message(*msg_id: StreamIdT)Acknowledges the successful processing of one or more messages.msg_id: message ids to acknowledge.Developerkavinbj