Package: bowline-streaming
BowlineEasily build performant data stream processing pipelines in Python.Bowline is a Python library that simplifies creating data pipelines that perform sequential computations
on data streams.Key features of Bowline include:Performance: Each BowlineProcessorruns in its own process, meaning that Bowline is ideal for high
throughput and computationally heavy workloads.Simplicity: Bowline abstracts away the complexity ofmultiprocessingby handling process creation,
inter-process communication, and clean process shutdown.Typing: Data inputs and ouputs are validated withpydantic.Create Complex Pipelines: Bowline provides aProcessorChainthat chains processes together, and aProcessGraphto enable the creation of more complex, branching pipelines.Installationpython -m pip install bowline-streamingUsageThe following section describe theProcessor,ProcessorChain, andProcessorGraphclasses.ProcessorsProcessors are the "building blocks" of the process chains.
A Processor defines the function to be executed, data input and output formats, and a processor name.
Processors are executed in a background process, and data is transferred via process-safe queues.ExampleThe following example shows creating and using aProcessorthat adds two numbers and prints the results.frompydanticimportBaseModelfrombowlineimportProcessorclassAddInputModel(BaseModel):x:inty:intclassAddOutputModel(BaseModel):result:intdefadd_and_print(input:AddInputModel)->AddOutputModel:result=input.x+input.yreturnAddOutputModel(result=result)if__name__=='__main__':# Create and start processoraddition_processor=Processor(target_function=add_and_print,name="add",input_model=AddInputModel,output_model=AddOutputModel)addition_processor.start()# Push data to theaddition_processor.push_input(AddInputModel(x=2,y=2))addition_processor.push_input(AddInputModel(x=3,y=4))addition_processor.push_input(AddInputModel(x=123,y=456))# Get the resultsprint("Results: ")for_inrange(3):# We pushed 3 inputs, we expect 3 outputs# Wait for results to be readywhilenotaddition_processor.has_output():passresult=addition_processor.get_output()print(result)# Stop the processoraddition_processor.shutdown()Processor ChainsProcessor Chains connect multiple Processors in a chain, such that the output of a Processor is passed as the input to the next.
This allows for the definition of pipelines that can run in sequence on streaming data.ExampleThe following example shows creating and using aProcessorChainthat adds two numbers, then squares the result.frompydanticimportBaseModelfrombowlineimportProcessorfrombowlineimportProcessorChainclassAddInputModel(BaseModel):x:inty:intclassAddOutputModel(BaseModel):result:intclassSquareOutputModel(BaseModel):result:intdefadd_two_numbers(input:AddInputModel)->AddOutputModel:result=input.x+input.yreturnAddOutputModel(result=result)defsquare_number(input:AddOutputModel)->SquareOutputModel:result=input.result*input.resultreturnSquareOutputModel(result=result)if__name__=='__main__':# Create processorsadd_two_numbers_processor=Processor(target_function=add_two_numbers,name="add",input_model=AddInputModel,output_model=AddOutputModel)square_number_processor=Processor(target_function=square_number,name="square",input_model=AddOutputModel,output_model=SquareOutputModel)# Create process chainprocessor_chain=ProcessorChain()processor_chain.add_processor(add_two_numbers_processor)processor_chain.add_processor(square_number_processor)# Start the processor chainprocessor_chain.start()# Push some data to the chain. This will add the numbers, then square them.processor_chain.push_input(AddInputModel(x=2,y=2))processor_chain.push_input(AddInputModel(x=3,y=4))processor_chain.push_input(AddInputModel(x=123,y=456))# Get the resultsprint(f"Results: ")for_inrange(3):# Wait for outputwhilenotprocessor_chain.has_output():passprint(processor_chain.get_output())# Shut down the processor chainprocessor_chain.shutdown()Processor GraphsProcessor Graphs connect multiple Processors in a graph, such that the output of a Processor can be passed to multiple subsequent Processors.
This allows for the definition of pipelines that can run with an arbitrary number of branching processors within them.ExampleThe following example creates aProcessorGraphin which two numbers are added together, then the result of that
calculation is passed to two subsequent processors: one which squares that result, and the other of which computes its
square root.importmathfrompydanticimportBaseModelfrombowlineimportProcessor,ProcessorGraphclassAddInputModel(BaseModel):x:inty:intclassAddOutputModel(BaseModel):result:intclassSquareOutputModel(BaseModel):result:intclassSquareRootOutputModel(BaseModel):result:floatdefadd_two_numbers(input:AddInputModel)->AddOutputModel:result=input.x+input.yprint(f"{input.x}+{input.y}={result}")returnAddOutputModel(result=result)defsquare_number(input:AddOutputModel)->SquareOutputModel:result=input.result*input.resultprint(f"{input.result}squared is{result}")returnSquareOutputModel(result=result)defsquare_root(input:AddOutputModel)->SquareRootOutputModel:result=math.sqrt(input.result)print(f"The square root of{input.result}is{result}")returnSquareRootOutputModel(result=result)if__name__=='__main__':# Create processorsadd_two_numbers_processor=Processor(target_function=add_two_numbers,name="addition",input_model=AddInputModel,output_model=AddOutputModel)square_number_processor=Processor(target_function=square_number,name="square",input_model=AddOutputModel,output_model=SquareOutputModel)square_root_processor=Processor(target_function=square_root,name="sqrt",input_model=AddOutputModel,output_model=SquareRootOutputModel)# Create ProcessorGraph# This processor graph will run add_two_numbers(), then run square_number() and square_root() on its result.processor_graph=ProcessorGraph()processor_graph.add_processor(add_two_numbers_processor)processor_graph.add_processor(square_number_processor,add_two_numbers_processor)processor_graph.add_processor(square_root_processor,add_two_numbers_processor)# Start the ProcessorGraphprocessor_graph.start()# Push input to graphprocessor_graph.push_input(AddInputModel(x=2,y=2))processor_graph.push_input(AddInputModel(x=3,y=4))processor_graph.push_input(AddInputModel(x=123,y=456))# Get resultsfor_inrange(6):# We provided 3 inputs, and there are 2 terminal Processors, so 6 total results# Wait for results to be availablewhilenotprocessor_graph.has_output():passresult=processor_graph.get_output()print(f"Received output{result.output}from processor{result.processor}")# Shut down processorsprocessor_graph.shutdown()Configuration-Driven PipelinesYou can also create pipelines via a yaml configuration file, for bothProcessorChains andProcessorGraphs.ProcessorChainExampleyamlconfiguration file, in which theaddProcessor provides input to thesquareProcessor.chain:processors:-add:target_function:simple_chain.add_two_numbersinput_model:simple_chain.AddInputModeloutput_model:simple_chain.AddOutputModel-square:target_function:simple_chain.square_numberinput_model:simple_chain.AddOutputModeloutput_model:simple_chain.SquareOutputModelExample code:frombowline.utils.configimportProcessorConfig# Create process chain from config fileconfig_file_path="examples/chain-config.yml"config=ProcessorConfig(config_file_path)processor_chain=config.generate_processors()# Start the processor chainprocessor_chain.start()ProcessorGraphExampleyamlconfiguration file, which definesa graph in which theadditionProcessor provides inputs to thesquareandsqrtProcessors:graph:processors:-addition:target_function:simple_graph.add_two_numbersinput_model:simple_graph.AddInputModeloutput_model:simple_graph.AddOutputModelprocessors:-square:target_function:simple_graph.square_numberinput_model:simple_graph.AddOutputModeloutput_model:simple_graph.SquareOutputModel-sqrt:target_function:simple_graph.square_rootinput_model:simple_graph.AddOutputModeloutput_model:simple_graph.SquareRootOutputModelExample code:frombowline.utils.configimportProcessorConfig# Create process chain from config fileconfig_file_path="examples/graph-config.yml"config=ProcessorConfig(config_file_path)processor_graph=config.generate_processors()# Start the ProcessorGraphprocessor_graph.start()ConsiderationsBecause Bowline usesmultiprocessingbehind the scenes, all data models must be serializable.Local DevelopmentClone the repository:git clone git@github.com:scottbarnesg/bowline.gitInstallbowlineas an interactive package:python -m pip install -e .Running the testsInstall the test dependencies:python -m pip install -e .[dev]Run the tests:python -m pytest test/Or, to view log output while running tests:python -m pytest --capture=no --log-cli-level=INFO test/test_processor.pyFAQWhy doesn't Bowline use asyncio?Bowline is built on themultiprocessinglibrary, so eachProcessorinstance runs in its own process.
This means that async functionality is generally not needed (although you are welcome to implement it in the functions you want Bowline to run).Bowline's primary use case is creating high-throughput, low-latency pipelines for streaming data that you want to perform computationally-heavy operations on.
If you are looking for an async-first library that runs tasks in a single process and is designed for lighter workloads, a tool like Faust may be a better
fit for you use case.