Package: grugstream
Grugstreampip install grugstreamSee thedocumentationIntroductionThe grugstream library provides an easy way to work with asynchronous/ reactive / streaming programming in Python.Set up data processing pipelines that are faster, use less memory and are easy to understand.Works with anyio - which means it works with asyncio and trio.This library is inspired byRxpy, aioreactivemonixPhilosophy - when you hit dot on your keyboard - you should have everything you need.Also - everything is statically typed!Getting StartedBasic ExampleWhat we always do isCreate an observable (A stream that is not running yet)Transform it things likemaporfilterRun it.For example,to_listwill run the observable and collect the results into a list.run_to_completionwill run the observable until it completesto_file_appendingwill run the observable and write the results to a fileimportanyiofromgrugstreamimportObservable# Mock async function simulating an HTTP call to Googleasyncdefmock_http_call_to_google(item:str)->str:# Simulate the asynchronous delay of an HTTP requestawaitanyio.sleep(1)returnf"Response from Google{item}"asyncdefmain():# Create an observable, and call google for each itemobservable=(Observable.from_iterable(["one","two","three"])# this is the same as map, but it's for async functions.map_async(lambdaitem:mock_http_call_to_google(item)))# Actually start the stream and collect the results into a listresults=awaitobservable.to_list()forresponseinresults:print(response)anyio.run(main)Map operators - Parallel ExampleRunning things in parallel is as simple as callingmap_async_parinstead ofmap_async:importanyiofromgrugstreamimportObservable# Mock async function simulating an HTTP call to Googleasyncdefmock_http_call_to_google(item:str)->str:# Simulate the asynchronous delay of an HTTP requestawaitanyio.sleep(1)returnf"Response from Google{item}"asyncdefmain():# Create an observable, and call google for each itemobservable=(# repeat every 0.1 secondsObservable.from_repeat("one",0.1)# at any given time, there will be at most 50 concurrent calls to google.map_async_par(lambdaitem:mock_http_call_to_google(item),max_par=50))# Actually start the stream - results into a list# Let's take only 20 resultsresults=awaitobservable.take(20).to_list()forresponseinresults:print(response)anyio.run(main)Chaining other api callsSuppose you have multiple api calls to run.You  want all the api calls to be run in parallel with each other -
the items doing the 2nd api call don't need to wait for all the items for the 1st api call to complete.
And maybe you want to stream to a file while it completes.
Thats when streaming really shines.importrandomfrompathlibimportPathfromtypingimportList,OptionalimportanyiofromgrugstreamimportObservable# Mock async function simulating an HTTP call to Googleasyncdefmock_http_call_to_google(item:str)->str:awaitanyio.sleep(1)returnf"Google Response for{item}"# Mock async function simulating an API call that returns a list of itemsasyncdefmock_api_call_that_returns_list(item:str)->List[str]:awaitanyio.sleep(0.5)return[f"Item{i}from{item}"foriinrange(3)]# Mock async function simulating an API call that returns an Optional valueasyncdefmock_api_call_that_returns_optional(item:str)->Optional[str]:awaitanyio.sleep(0.2)maybe_yes=random.choice([True,False])returnitemifmaybe_yeselseNoneasyncdefmain():observable=(Observable.from_repeat("query",0.1).map_async_par(lambdaitem:mock_http_call_to_google(item)).map_async_par(lambdaitem:mock_api_call_that_returns_list(item)).flatten_iterable()# Flatten the list into individual items.map_async_par(lambdaitem:mock_api_call_that_returns_optional(item)).print().flatten_optional()# Remove None values)# Write the results to a fileawaitobservable.take(100).to_file(Path("results.txt"))anyio.run(main)Logging and debugging, print and tqdmWe provide aprintandtqdmoperator to help you debug your streams.importanyiofromtqdmimporttqdmfromgrugstreamimportObservable# Mock async function simulating an HTTP call to Googleasyncdefmock_http_call_to_google(item:str)->str:awaitanyio.sleep(0.1)returnf"Google Response for{item}"asyncdefmain():observable=(Observable.from_repeat("query",0.1).throttle(1)# don't spam google too much!.map_async_par(lambdaitem:mock_http_call_to_google(item))# Show a progress bar that should show ~1 it/s.tqdm(tqdm_bar=tqdm(desc="Google observable"))# Print the elements.print())awaitobservable.take(1000).run_to_completion()anyio.run(main)for_each operator - side effectsSometimes you want to do something with the elements of the stream, but you don't want to change the stream itself.
For example, you might want to write some intermediate items to a file.importanyiofrompathlibimportPathfromgrugstreamimportObservable# Mock async function simulating an HTTP call to Googleasyncdefmock_http_call_to_google(item:str)->str:awaitanyio.sleep(0.1)returnf"Google Response for{item}"asyncdefmain():my_list=[]observable=(Observable.from_repeat("query",0.1).map_async_par(lambdaitem:mock_http_call_to_google(item))# What's google's response? Let's write it to a file.for_each_to_file(file_path=Path("results.txt"),)# Let's also append it to a list to print.for_each(lambdaitem:my_list.append(item)).map(lambdaitem:item.upper()).print())awaitobservable.take(1000).run_to_completion()print(my_list)anyio.run(main)Building an ObservableThis library provides several utility methods for creating observables:From Existing Datafrom_iterable(iterable): Create an observable from a Python iterable like a list or a tuple.from_async_iterable(iterable): Create an observable from an asynchronous iterable.from_one(value): Create an observable that emits a single value.from_one_option(value): Create an observable that emits a single value or nothing if the value isNone.Example:fromgrugstreamimportObservableobservable=Observable.from_iterable([1,2,3])Transforming ObservablesmapApplies a function to all elements in the source observable.observable=Observable.from_iterable([1,2,3])new_observable=observable.map(lambdax:x*2)filterFilters out elements that do not match a given predicate.observable=Observable.from_iterable([1,2,3])filtered_observable=observable.filter(lambdax:x>1)flatten_iterableTransforms an observable of iterables into an observable of the individual items.observable=Observable.from_iterable([[1,2],[3,4]])flattened_observable=observable.flatten_iterable()Back-pressure, Buffearing, ThrottlingThe library supports back-pressure to ensure that the producer and consumer are in sync. There are also methods likethrottle(seconds)to control the rate of emissions.throttled_observable=observable.throttle(1.0)# Emits at most one item per secondSubscription and Error HandlingYou can use theSubscriberclass to define custom subscribers. It has three methods:on_next(value): Called when a new value is emitted.on_error(error): Called when an error occurs.on_completed(): Called when the observable completes.In general, I hope that you wouldn't have to implement your own subscriber.
Most things you want to do can be done by chaining operators such asmapandfilter.