Package: rstream
RabbitMQ Stream Python ClientA Python asyncio-based client forRabbitMQ StreamsThe RabbitMQ stream plug-in is required. See thedocumentationfor enabling it.Table of ContentsInstallationExamplesClient CodecsAMQP 1.0 codec  vs BinaryPublishing messagesPublishing with confirmationSub-Entry Batching and CompressionDeduplicationConsuming messagesServer-side offset trackingSuper StreamsSingle Active ConsumerConnecting with SSLSasl MechanismsManaging disconnectionsReconnectLoad BalancerClient PerformancesTest caseBuild and TestProject NotesInstallationThe RabbitMQ stream plug-in is required. See thedocumentationfor enabling it.The client is distributed viaPIP:pipinstallrstreamExamplesHereyou can find different examples.Client CodecsBefore start using the client is important to read this section.
The client supports two codecs to store the messages to the server:AMQP 1.0BinaryBy default you should useAMQP 1.0codec:amqp_message=AMQPMessage(body="hello:{}".format(i),)AMQP 1.0 codec  vs BinaryYou need to use theAMQP 1.0codec to exchange messages with other stream clients likeJava,.NET,Rust,Goor if you want to use theAMQP 0.9.1clients.You can use theBinaryversion if you need to exchange messages from Python to Python.Note: The messages stored inBinaryare not compatible with the other clients and with AMQP 0.9.1 clients.Once the messages are stored to the server, you can't change them.Read also theClient PerformancessectionPublishing messagesYou can publish messages with four different methods:send: asynchronous, messages are automatically buffered internally and sent at once after a timeout expires.send_batch: synchronous, the user buffers the messages and sends them. This is the fastest publishing method.send_wait: synchronous, the caller wait till the message is confirmed. This is the slowest publishing method.send_sub_entry: asynchronous. SeeSub-entry batching and compression.On theexamplesdirectory you can find diffent way to send the messages:producer using sendproducer using send_waitproducer using send_batchproducer using sub_entry_batchPublishing with confirmationThe Send method takes as parameter an handle function that will be called asynchronously when the message sent will be notified from the server to have been published.Example:producer using send and handling confirmationproducer using send_batch and handling confirmationWithsend_waitinstead will wait until the confirmation from the server is received.Sub-Entry Batching and CompressionRabbitMQ Stream provides a special mode to publish, store, and dispatch messages: sub-entry batching. This mode increases throughput at the cost of increased latency and potential duplicated messages even when deduplication is enabled. It also allows using compression to reduce bandwidth and storage if messages are reasonably similar, at the cost of increasing CPU usage on the client side.Sub-entry batching consists in squeezing several messages – a batch – in the slot that is usually used for one message. This means outbound messages are not only batched in publishing frames, but in sub-entries as well.# sending with compressionawaitproducer.send_sub_entry(STREAM,compression_type=CompressionType.Gzip,sub_entry_messages=messages)Full example producer using sub-entry batchConsumer side is automatic, so no need configurations.The client is shipped with No Compression (CompressionType.No) and Gzip Compression (CompressionType.Gzip) the other compressions (Snappy,Lz4,Zstd) can be used implementing theICompressionCodecclass.DeduplicationRabbitMQ Stream can detect and filter out duplicated messages, based on 2 client-side elements: the producer name and the message publishing ID.
All the producer methods to send messages (send, send_batch, send_wait) takes a publisher_name parameter while the message publishing id can be set in the AMQP message.Example:producer with deduplicationConsuming messagesSeeconsumer examplesfor basic consumer and consumers with different offsets.Server-side offset trackingRabbitMQ Streams provides server-side offset tracking for consumers. This features allows a consuming application to restart consuming where it left off in a previous run.
You can use the store_offset (to store an offset in the server) and query_offset (to query it) methods of the consumer class like in this example:server side offset trackingSuperstreamsA super stream is a logical stream made of individual, regular streams. It is a way to scale out publishing and consuming with RabbitMQ Streams: a large logical stream is divided into partition streams, splitting up the storage and the traffic on several cluster nodes.See theblog postfor more info.You can usesuperstream_producerandsuperstream_consumerclasses which internally uses producers and consumers to operate on the componsing streams.See theSuper Stream exampleSingle Active ConsumerSingle active consumer provides exclusive consumption and consumption continuity on a stream.See theblog postfor more info.
See examples in:See thesingle active consumer exampleFilteringFiltering is a new streaming feature enabled from RabbitMQ 3.13 based on Bloom filter.
RabbitMQ Stream provides a server-side filtering feature that avoids reading all the messages of a stream and filtering
only on the client side. This helps to save network bandwidth when a consuming application needs only a subset of
messages.https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/#filteringSee thefiltering examplesConnecting with SSLYou can enable ssl/tls.
See example here:tls exampleSasl MechanismsYou can use the following sasl mechanisms:PLAINEXTERNALThe client usesPLAINmechanism by default.TheEXTERNALmechanism is used to authenticate a user based on a certificate presented by the client.
Example:ssl_context=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)# put the root certificate of the cassl_context.load_verify_locations("certs/ca_certificate.pem")ssl_context.load_cert_chain("certs/client_HOSTNAME_certificate.pem","certs/client_HOSTNAME_key.pem",)asyncwithProducer("HOSTNAME",username="not_important",password="not_important",port=5551,ssl_context=ssl_context,sasl_configuration_mechanism=SlasMechanism.MechanismExternal## <--- here EXTERNAL configurationThe pluginrabbitmq_auth_mechanism_sslneeds to be enabled on the server side, andssl_options.fail_if_no_peer_certneeds to set totrueconfig example:auth_mechanisms.3 = PLAIN
auth_mechanisms.2 = AMQPLAIN
auth_mechanisms.1 = EXTERNAL

ssl_options.cacertfile = certs/ca_certificate.pem
ssl_options.certfile = certs/server_certificate.pem
ssl_options.keyfile = certs/server_key.pem
listeners.ssl.default = 5671
stream.listeners.ssl.default = 5551
ssl_options.verify               = verify_peer
ssl_options.fail_if_no_peer_cert = trueManaging disconnectionsThe client supports auto-reconnect just for Producer and SuperstreamProducer at the moment.When the TCP connection is disconnected unexpectedly, the Producer and the SuperstreamProducer will try to automatically
reconnect while in case of the Consumer/SuperstreamConsumer the client raises an event that needs to be managed:asyncdefon_connection_closed(disconnection_info:DisconnectionErrorInfo)->None:print("connection has been closed from stream: "+str(disconnection_info.streams)+" for reason: "+str(disconnection_info.reason))consumer=Consumer(..on_close_handler=on_connection_closed,)ReconnectWhen theon_close_handlerevent is raised, you can close the Consumers by doing a correct clean-up or try reconnecting using the reconnect stream.Example:asyncdefon_connection_closed(disconnection_info:OnClosedErrorInfo)->None:print("connection has been closed from stream: "+str(disconnection_info.streams)+" for reason: "+str(disconnection_info.reason))forstreamindisconnection_info.streams:print("reconnecting stream: "+stream)awaitproducer.reconnect_stream(stream)Please take a look at the complete exampleshereMetadata UpdateIf the streams topology changes (ex:Stream deleted or add/remove follower), The server removes the producers and consumers
linked to the stream and then it sends the Metadata update event.
the behaviour is similar to what we have for disconnections. In case of the Producer/Superstream Producer
the Client will try to automatically reconnect while the Consumer needs to manage the on_close_handler event.Please take a look at the complete exampleshereLoad BalancerIn order to handle load balancers, you can use theload_balancer_modeparameter for producers and consumers. This will always attempt to create a connection via the load balancer, discarding connections that are inappropriate for the client type.Producers must connect to the leader node, while consumers can connect to any, prioritizing replicas if available.Client PerformancesThe RabbitMQ Stream queues can handle high throughput. Currently, the client cannot reach the maximum throughput the server can handle.We found some bottlenecks; one of them is the current AMQP 1.0 marshal and unmarshal message format.This one:foriinrange(1_000_000):amqp_message=AMQPMessage(body="hello:{}".format(i),)# send is asynchronousawaitproducer.send(stream=STREAM,message=amqp_message)is more or less ~55% slower than:foriinrange(1_000_000):# send is asynchronousawaitproducer.send(stream=STREAM,message=b"hello")You can use thebatch_sendto test the performances.We are evaluating rewriting theAMQP 1.0 codecoptimized for the stream use case.Test caseLinux Ubuntu 4 cores and 8 GB of RamRabbitMQ installed to the serverSend batch with AMQP 1.0 codec:$python3docs/examples/basic_producers/producer_send_batch.pySent1.000.000messagesin9.3218seconds.107.275,5970messagespersecondSend batch with binary codec:$python3docs/examples/basic_producers/producer_send_batch_binary.pySent1.000.000messagesin2.9930seconds.334.116,5639messagespersecondBuild and TestTo run the tests, you need to have a running RabbitMQ Stream server.
You can use the docker official image.Run the server with the following command:dockerrun-it--rm--namerabbitmq-p5552:5552-p5672:5672-p15672:15672\-eRABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbitmq_stream advertised_host localhost'\rabbitmq:3.12-managementenable the plugin:dockerexecrabbitmqrabbitmq-pluginsenablerabbitmq_streamand run the tests:poetryrunpytestProject NotesThe project is in development and stabilization phase. Features and API are subject to change, but breaking changes will be kept to a minimum.Any feedback or contribution is welcome