Package: quixstreams
Quix Streams v2Quix Streams v2 is a cloud native library for processing data in Kafka using pure Python. It’s designed to give you the power of a distributed system in a lightweight library by combining the low-level scalability and resiliency features of Kafka with an easy to use Python interface.Quix Streams has the following benefits:No JVM, no orchestrator, no server-side engine.Easily integrates with the entire Python ecosystem (pandas, scikit-learn, TensorFlow, PyTorch etc).Support for many serialization formats, including JSON (and Quix-specific).Support for stateful operations using RocksDB.Support for aggregations over tumbling and hopping time windowsA simple framework with Pandas-like interface to ease newcomers to streaming."At-least-once" Kafka processing guarantees.Designed to run and scale resiliently via container orchestration (like Kubernetes).Easily runs locally and in Jupyter Notebook for convenient development and debugging.Seamless integration with the Quix platform.Use Quix Streams to build event-driven, machine learning/AI or physics-based applications that depend on real-time data from Kafka.Getting started 🏄Install Quix Streamspython-mpipinstallquixstreamsRequirementsPython 3.8+, Apache Kafka 0.10+Seerequirements.txtfor the full list of requirementsExample ApplicationHere's an example of how toprocessdata from a Kafka Topic with Quix Streams:fromquixstreamsimportApplication,State# Define an applicationapp=Application(broker_address="localhost:9092",# Kafka broker addressconsumer_group="consumer-group-name",# Kafka consumer group)# Define the input and output topics. By default, "json" serialization will be usedinput_topic=app.topic("my_input_topic")output_topic=app.topic("my_output_topic")defcount(data:dict,state:State):# Get a value from state for the current Kafka message keytotal=state.get('total',default=0)total+=1# Set a value back to the statestate.set('total',total)# Update your message data with a value from the statedata['total']=total# Create a StreamingDataFrame instance# StreamingDataFrame is a primary interface to define the message processing pipelinesdf=app.dataframe(topic=input_topic)# Print the incoming messagessdf=sdf.update(lambdavalue:print('Received a message:',value))# Select fields from incoming messagessdf=sdf[["field_1","field_2","field_3"]]# Filter only messages with "field_0" > 10 and "field_2" != "test"sdf=sdf[(sdf["field_1"]>10)&(sdf["field_2"]!="test")]# Filter messages using custom functionssdf=sdf[sdf.apply(lambdavalue:0<(value['field_1']+value['field_3'])<1000)]# Generate a new value based on the current onesdf=sdf.apply(lambdavalue:{**value,'new_field':'new_value'})# Update a value based on the entire message contentsdf['field_4']=sdf.apply(lambdavalue:value['field_1']+value['field_3'])# Use a stateful function to persist data to the state store and update the value in placesdf=sdf.update(count,stateful=True)# Print the result before producing itsdf=sdf.update(lambdavalue,ctx:print('Producing a message:',value))# Produce the result to the output topicsdf=sdf.to_topic(output_topic)if__name__=="__main__":# Run the streaming applicationapp.run(sdf)How It WorksThere are two primary components:StreamingDataFrame- a predefined declarative pipeline to process and transform incoming messages.Application- to manage the Kafka-related setup & teardown and message lifecycle (consuming, committing). It processes each message with the dataframe you provide it.Under the hood, theApplicationwill:Consume a message.Deserialize it.Process it with yourStreamingDataFrame.Produce it to the output topic.Automatically commit the topic offset and state updates after the message is processed.React to Kafka rebalancing updates and manage the topic partitions.Manage the State store.Handle OS signals and gracefully exit the application.More ExamplesYou may find more examples in theexamplesfolderhere.Advanced UsageFor more in-depth description of Quix Streams components, please
follow these links:StreamingDataFrame.Serialization.Stateful Processing.Usage with Quix SaaS Platform.Upgrading from Quix Streams <2.0.Using theQuix Platform-Application.Quix()This library doesn't have any dependency on any commercial products, but if you use it together with Quix SaaS Platform you will get some advantages out of the box during your development process such as:Auto-configuration.Monitoring.Data explorer.Data persistence.Pipeline visualization.Metrics.and more.Quix Streams provides a seamless integration with Quix Platform viaApplication.Quix()class.
This class will automatically configure the Application using Quix SDK Token.If you are running this within the Quix platform it will be configured
automatically.Otherwise, please seeQuix Platform Configuration.What's NextThis library is being actively developed.Here are some of the planned improvements:Windowed aggregations over Tumbling & Hopping windowsState recovery based on Kafka changelog topicsWindowed aggregations over Sliding windowsGroup-bys and joins (for merging topics/keys)Support for "exactly-once" Kafka processing (aka transactions)Support for Avro and Protobuf formatsSchema Registry supportTo find out when the next version is ready, make sure you watch this repo
and join ourQuix Community on Slack!Contribution GuideContributing is a great way to learn and we especially welcome those who haven't contributed to an OSS project before.We're very open to any feedback or code contributions to this OSS project ❤️.Before contributing, please read ourContributingfile for how you can best give feedback and contribute.Need help?If you run into any problems, please create anissueor ask in #quix-help in ourQuix Community on Slack.Community 👭Join other software engineers inThe Stream, an online community of people interested in all things data streaming. This is a space to both listen to and share learnings.🙌Join our Slack community!LicenseQuix Streams is licensed under the Apache 2.0 license. View a copy of the License filehere.Stay in touch 👋You can follow us onTwitterandLinkedinwhere we share our latest tutorials, forthcoming community events and the occasional meme.If you have any questions or feedback - write to us atsupport@quix.io!