Package: dataloop-upipe
uPipeMicro-pipe is asyncio based framework for runningsimple,high performance, parallel and synchronized python
processes within a single machine.uPipe comes with auto-scaler for maximum throughput and hardware utilization and a monitoring tool.a=Processor('a',entry='./a.py',config=config)b=Processor('b',entry='./b.py',config=config)pipe=Pipe('plus-one')pipe.add(a).add(b)awaitpipe.start()awaitpipe.wait_for_completion()print("Running")Why uPipe ?In the world of machine learning python has taken the stage as the go-to programming language, yet python has few
fundamental issues preventing it from being production friendlyParallel processingPython has a built-in weakness for parallel data processing, this weakness is sourced in the core of python at
theGILThe GlobalInterpreterLock prevents from different threads to access memory (and therefore variables) in parallel. This
essentially means that a multi-threaded python program executes slower compared to single thread on CPU intensive
workloads. Python also offers multiprocessing capabilities that are not blocked by GIL since the workloads are running
in different processes, yet using multiprocessor does not allow sharing memory and communication between process
implicitly usesPicklingfor data sharing, which in turn significantly
degrades data throughput.Parallel optimizationAny given processing workload is bounded by critical hardware resources where the leading resources are I/O (Network and
disk), Memory and CPU. Machine learning also introduces additional resources like GPU cores, CPU memory and other AI
specific accelerators. Optimizing the overall data on a given machine is a process of trial and error, often includes
non-trivial technical challenges.uPipe implementsAutoScalingby process
into the single machine with maximum pipeline throughput as goal.SimpleuPipe exposes a simple API for creating data pipelines from existing packages and library's run with fews line of code.The basics entities:DataFrame: Arbitrary binary data that is shared between processors.Processor: A python code that can receive, process and generate(emit) data frames. Processor is a logical entities
that can encapsulate many (os) processes of the same processor code.Queue: Two processors will communicate DataFrames using queue, usually memory based queue.Pipe: The Pipeline the defines the data flow between Processors and manages the processing execution.Read moreuPipe basicsuPipe APIuPipe architectureuPipe designInstallpipinstalldataloop-upipeImportfrom dataloop.upipe import Processor, PipeUsageProcessor declarationProcessor can be defined inline methods(func) or as external scripts (entry) in order to be used in pipeline
definition, the name for each processor must be a unique string and used to identify the processor.a=Processor('my name is A',func=processor_a)b=Processor('b',entry='./b.py')Pipe definitionProcessor can be chained to pipe and to each other by theaddmethodpipe=Pipe('some-pipe')pipe.add(a).add(b)#a,b are processorsJoining a Processor to a pipe###This code is declaring the Worker's code andjoinin order to be able to receive and emit data frames.proc=Worker("b")awaitproc.join()# Processor is now ready to ingest and emit dataAcquiring data by processorWorker can get data using thegetandget_syncmethodsproc=Worker("b")awaitproc.join()# Processor is now ready to ingest and emit data#get data frame, return None if no data availabledata_frame=awaitproc.get()#get data frame, wait until data availabledata_frame=awaitproc.get_sync()Emitting data by processorWorker can emit data frame for the next Processor in Pipe using theemitandemit_syncmethodsproc=Worker("b")awaitproc.join()# Processor is now ready to ingest and emit data#emit data frame, return None if no space (back pressure from next processor)awaitproc.emit()#emit data frame, wait if no space (back pressure)awaitproc.emit_sync()start pipeawaitpipe.start()awaitpipe.wait_for_completion()terminate pipeawaitpipe.terminate()awaitpipe.wait_for_completion()# wait for existing data to complete