Package: sequential-functions
Sequential FunctionsCompose functions into a sequence that are called sequentially.Build fast readable code.Break your problem into small functional steps and let sequential functions run them sequentially.Want to go faster? Simply increase the number of threads or processes.ExamplesToy ExampleToy example that highlights the syntax.importsequential_functionsassfdefmain():# Compose an easy to read list of stepssequence=sf.Compose(square,plus_one,)# Use list to pull items through the sequenceoutputs=list(sequence(range(5)))print(outputs)defsquare(x):returnx*xdefplus_one(x):returnx+1if__name__=="__main__":main()Output[1,2,5,10,17]Under the HoodCompose uses generator chaining to run items through each of the functions.
Both of these methods produce the same outputimportsequential_functionsassfdefmain():# Method 1sequence=sf.Compose(square,plus_one,)outputs=list(sequence(range(5)))print(outputs,"Method 1 - Composed Sequence")# Method 2generator_chain=range(5)generator_chain=(square(x)forxingenerator_chain)generator_chain=(plus_one(x)forxingenerator_chain)output=list(generator_chain)print(outputs,"Method 2 - Generator Chain")defsquare(x):returnx*xdefplus_one(x):returnx+1if__name__=="__main__":main()Output[1,2,5,10,17]Method1-ComposedSequence[1,2,5,10,17]Method2-GeneratorChainBest PracticeIt's best practice to pass a dict in and out of each function.
Each function can modify the dict as they complete their computation.
This design seems the most readable and extensible.importsequential_functionsassfdefmain():sequence=sf.Compose(create_item_dict,load_image,preprocess_image,detect_objects,)paths=["cat.jpg","dog.jpg"]foriteminsequence(paths):print(f"Results:{item['image_path']}")print(item["detections"])print()defcreate_item_dict(path):print(f"Item Dict:{path}")item={"image_path":path}returnitemdefload_image(item):print(f"Loading:{item['image_path']}")item["image"]="e.g. numpy array"returnitemdefpreprocess_image(item):print(f"Preprocessing:{item['image_path']}")item["tensor"]="e.g. torch tensor"returnitemdefdetect_objects(item):print(f"Detecting:{item['image_path']}")item["detections"]=["box 1","box 2"]returnitemif__name__=="__main__":main()OutputItemDict:cat.jpg
Loading:cat.jpg
Preprocessing:cat.jpg
Detecting:cat.jpg
Results:cat.jpg['box 1','box 2']ItemDict:dog.jpg
Loading:dog.jpg
Preprocessing:dog.jpg
Detecting:dog.jpg
Results:dog.jpg['box 1','box 2']Multi ProcessingIt's trivial to distribute work to multiple processes by providing the num_processes argument.
Order is not preserved with multiprocessing.
Use multiprocessing when computation is the bottle neck.importsequential_functionsassfimporttimeimportosdefmain():sequence=sf.Compose(slow_task,record_process_id,num_processes=5,# Simply choose the number of processes)start_time=time.perf_counter()forxinsequence(range(5)):print(x)end_time=time.perf_counter()print(f"total time:{end_time-start_time}")defslow_task(x):time.sleep(1)# sleep 1 secondreturnxdefrecord_process_id(x):returnf"Task{x}completed by process{os.getpid()}"if__name__=="__main__":main()OutputTask1completedbyprocess1253843Task0completedbyprocess1253837Task2completedbyprocess1253838Task3completedbyprocess1253846Task4completedbyprocess1253840totaltime:1.241348128998652Multi ThreadingIt's trivial to distribute work to multiple threads by providing the num_threads argument.
Order is not preserved with multithreading.
Use threading when IO is the bottle neck. e.g loading urls.importsequential_functionsassfimporttimeimportthreadingdefmain():sequence=sf.Compose(slow_task,record_thread_name,num_threads=5,# Simply choose the number of thread)start_time=time.perf_counter()forxinsequence(range(5)):print(x)end_time=time.perf_counter()print(f"total time:{end_time-start_time}")defslow_task(x):time.sleep(1)# sleep 1 secondreturnxdefrecord_thread_name(x):name=threading.current_thread().namereturnf"Task{x}completed by thread{name}"if__name__=="__main__":main()OutputTask0completedbythreadThreadPoolExecutor-0_2
Task1completedbythreadThreadPoolExecutor-0_3
Task4completedbythreadThreadPoolExecutor-0_1
Task2completedbythreadThreadPoolExecutor-0_0
Task3completedbythreadThreadPoolExecutor-0_4
totaltime:1.2234888289822266NestingCompose returns a callable that can be nesting inside another Compose.
Each compose can use threads and processes independently.importsequential_functionsassfimportthreadingimporttimeimportosdefmain():sequence=sf.Compose(function_a,sf.Compose(function_b,num_threads=3,),sf.Compose(function_c,num_processes=3,),)outputs=list(sequence(range(3)))print(outputs)deffunction_a(x):print(f"function_a({x}) ran in main thread")returnxdeffunction_b(x):time.sleep(1)# sleep 1 secondprint(f"function_b({x}) ran in thread{threading.current_thread().name}")returnxdeffunction_c(x):time.sleep(1)# sleep 1 secondprint(f"function_c({x}) ran in process{os.getpid()}")returnxif__name__=="__main__":main()Outputfunction_c(0)raninprocess1253950function_c(1)raninprocess1253958function_c(2)raninprocess1253948function_a(0)raninmainthread
function_a(1)raninmainthread
function_a(2)raninmainthread
function_b(0)raninthreadThreadPoolExecutor-0_1
function_b(2)raninthreadThreadPoolExecutor-0_0
function_b(1)raninthreadThreadPoolExecutor-0_2[0,1,2]BatchingUse batching to collate multiple items into a batch.
A machine learning model may more efficient on batches.importsequential_functionsassfimporttimedefmain():sequence=sf.Compose(# Build the batches in background processes.sf.Compose(load_image,sf.Batch(batch_size=3),collate_images,num_processes=3,),# Detect in the main process.detect_objects,debatch_detections,)image_paths=(f"image_{i}.jpg"foriinrange(10))results=list(sequence(image_paths))forresultinresults:print(result)defload_image(path):returnpath.replace("image","tensor").replace(".jpg","")defcollate_images(x_list):# Ideally you would stack images into a tensorreturn",".join(x_list)defdetect_objects(x_batch):print(f"Detecting on Batch:{x_batch}")# Ideally your detection runs faster with a batch of images.returnx_batch.replace("tensor","Detections tensor")defdebatch_detections(x_batch):yield fromx_batch.split(",")if__name__=="__main__":main()OutputDetectingonBatch:tensor_3,tensor_4,tensor_5
DetectingonBatch:tensor_0,tensor_1,tensor_2
DetectingonBatch:tensor_6,tensor_7,tensor_9
DetectingonBatch:tensor_8
Detectionstensor_3
Detectionstensor_4
Detectionstensor_5
Detectionstensor_0
Detectionstensor_1
Detectionstensor_2
Detectionstensor_6
Detectionstensor_7
Detectionstensor_9
Detectionstensor_8CallablesFunctions can be any type of callable.
Use closures and callable objects to change the behaviour of functionsimportsequential_functionsassfdefmain():sequence=sf.Compose(to_string,append_string(" Hello"),append_string(" World!"),EncloseString("**"),EncloseString(".."),)forxinsequence(range(5)):print(x)defto_string(x):returnstr(x)defappend_string(s):# create new function on the flydefclosure(x):returnx+s# return this new functionreturnclosureclassEncloseString():# Callable classdef__init__(self,s):self.s=sdef__call__(self,x):returnself.s+x+self.sif__name__=="__main__":main()Output..**0HelloWorld!**..
..**1HelloWorld!**..
..**2HelloWorld!**..
..**3HelloWorld!**..
..**4HelloWorld!**..Item GrowthFunctions can yield out more items than they take in.importsequential_functionsassfdefmain():sequence=sf.Compose(yield_video_frames,detect_objects,)forxinsequence(range(3)):print(x)defyield_video_frames(x):num_frames=3foriinrange(num_frames):yieldf"Video{x}, Frame{i}"defdetect_objects(x):returnf" Detecting objects in{x}"if__name__=="__main__":main()OutputDetectingobjectsinVideo0,Frame0DetectingobjectsinVideo0,Frame1DetectingobjectsinVideo0,Frame2DetectingobjectsinVideo1,Frame0DetectingobjectsinVideo1,Frame1DetectingobjectsinVideo1,Frame2DetectingobjectsinVideo2,Frame0DetectingobjectsinVideo2,Frame1DetectingobjectsinVideo2,Frame2