Package: py-entangle
This version: 0.2.3Current development version is here:0.2.4A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs, supporting bothcontrol flowanddataflowexecution paradigms as well as de-centralized CPU & GPU scheduling.For a quick look at what makes Entangle special, take a look atDesign Goals.New In This ReleaseBug fixes to process.py, ssh.pyDistributed dataflow exampleDataflow decorator re-write. Now works with ssh for distributed dataflow. Fixes prior issues with local dataflows.Retry usage exampleDockerfile provided for quick and easy experimentation.Workflows can now return the call graph structure upon completion. SeeGraph ExampleSupport for workflow futures (if that's your thing) SeeWorkflow Future ExampleQuick UsageWith Entangle you can run simple, hardware parallelized code with conditional logic that looks like this.result=add(add(num(6),two()ifFalseelseone()),subtract(five(),two()))print(result())or train two AI models in parallel using tensorflow container utilizing dedicated CPU and GPU usage.@process@docker(image="tensorflow/tensorflow:latest-gpu",packages=['tensorflow_datasets'])deftrain_modelA():# train itreturn@process@docker(image="tensorflow/tensorflow:latest-gpu",packages=['tensorflow_datasets'])deftrain_modelB():# train itreturn@workflowdeftrain_models(*args):# I'm training a bunch of models in parallel!returnworkflow=train_models(train_modelA(),train_modelB())result=workflow()DockerTo quickly get started with Entangle, build and run a docker container from the included Dockerfile.$dockerbuild-tentangle.
$dockerrun-itentangle:latest
root@9579336b3e34:/#python-mentangle.examples.exampleOr if you have theNVIDIA Docker Environmentsetup you can test the numba GPU vector example.$dockerrun-it--gpusallentangle
root@13428af4a37b:/#python-mentangle.examples.example3(0.2957176749914652,0.41134210501331836)Wiki ArticlesHow Does Entangle Compare to Other Parallel Compute Frameworks?Development Roadmap & ContributionsOutlineOverviewWhat does "Entangle" mean?Important NotesInstallationDesign GoalsArchitectureDeclarative InfrastructureExecutionWorkflowsProcess BehaviorCompositionDecorator CompositionWorkflow CompositionContainersDataflowsDataflow vs WorkflowsDAG DataflowResults ComparisonAdvantages of Strict DataflowSchedulersDistributed FlowsExamplesGPU ExampleShared Memory ExampleAI ExampleDataflow ExamplesData-Driven BranchingDistributed DataflowDocker ExampleScheduler ExampleGraph ExampleWorkflow Future ExampleLoggingDesign ToolOverviewEntangle is adifferentkind of parallel compute framework for multi-CPU/GPU environments.
It allows for simple workflow design usingplain old pythonand special decorators that control the type of parallel compute and infrastructure needed.One key feature of entangle is fine-grained control over individual functions in a workflow. You could easily describe multiple functions running across multiple compute environments all interacting as if they were simple local python functions.
No central scheduler or workflow manager is needed allowing you to choose where and how functions operate withdeclarative infrastructure.Another unique quality is the use of composition to build parallel workflows dynamically.What does "Entangle" mean?The term is derived from a quantum physics phenomena calledquantum entanglementwhich involves the state of a pair or group of particles affecting one anotherat a distanceinstantaneously.In this context, it is a metaphor for how tasks send data (particles) to one another in the context of a connected microflow.IMPORTANT NOTES!Please keep in mind that Entangle isin developmentand is classified asPre-Alpha. Some of the functionality shown here is incomplete. If you clone this repo and want to experiment be sure to update often as things break, improve, get fixed etc. quite frequently. Themainbranch will always contain the most current release. All development for the next version is done on the development branch for the next released listed at the top of this document.InstallationNOTE: At the moment entangle only works with python 3.8 due to how coroutines work there and also shared memory features.From PyPi$pipinstall--upgradepy-entangle
$python-mentangle.examples.exampleFrom repo rootpython3.8$virtualenv--python=python3.8venv
$sourcevenv/bin/activate(venv)$pythonsetup.pyinstall(venv)$python-mentangle.examples.exampleminiconda3Installminiconda3with python3.8 for linux$condainit$pythonsetup.pyinstall
$python-mentangle.examples.exampleInstalling NumbaNOTE: Numba package is disabled by default insetup.py. If you want this package, just uncomment it; however some OS specific steps might be required.On some systems you might encounter the following error when trying to installnumba.RuntimeError:Couldnotfinda`llvm-config`binary.Try the following remedy (for ubuntu systems)$sudoapt-getinstall-y--no-install-recommendsllvm-10llvm-10-dev
$exportLLVM_CONFIG=/usr/bin/llvm-config-10
$pip3installnumbaTesting$pytest--verbose--color=yes--disable-pytest-warnings--no-summary--pyargsentangle.testsor if you don't have GPU$pytest--verbose--color=yes--pyargsentangle.tests.test_entangleor just do this$pythonsetup.pytestCleaningClean all build files, directories, temp files and any files created by examples and tests.$pythonsetup.pycleanMinicondaIf you are planning to run or use GPU enabled code it is recommended to set up aminiconda3virtualenv.Design GoalsSmall & SimpleEasy to UnderstandAPI-lessPlain Old PythonTrue ParallelismPluggable & FlexibleComposition BasedShared-NothingServerless & ThreadlessTrue Dataflow SupportCPU/GPU SchedulingDistributed DataflowArchitectureEntangle is designed without a central scheduler or workflow manager. Rather, each function is decorated with special descriptors that turn them into their own workflow managers.
Thesedecoratorsimplement logic to parallelize andgathervalues from its dependent arguments, which are executed as separate processes. As each function is assigned a dedicated CPU, the workflow is thus an ensemble of parallel, independent micro-flows that resolve themselves and pass their values into queues until the workflow completes.This offers an extremeshared nothingdesign that maximizes CPU usage in a multi-CPU environment.Each function (or task) is given a process and scheduled to a CPU by the operating system. Since python Processes are native bound OS processes, this inherits the benefit of the operating system scheduler which is optimized for the underlying hardware.
Arguments that satisfy the function are run in parallel in the same fashion. The parent function then uses asyncio coroutines to monitor queues for the results from the processes. This keeps the CPU usage down while the dependent functions produce their results and eliminates the need for monitor threads.As a workflow executes, it fans out over CPUs. Each process acting as it's own scheduler to spawn new processes and resolve arguments, while also monitoring queues for incoming results asynchronously.
This makes the workflow a truly emergent, dynamic computing construct vs a monolithic service managing all the pieces. Of course, this is not to say one approach is better, just that entangle takes a different approach based on its preferred tradeoffs.TradeoffsEvery design approach is a balance of tradeoffs. Entangle favors CPU utilization andtrueparallelism over resource managers, centralized (which is to say network centric) schedulers or other shared services.
It favors simplicity over behavior - leaving specific extensions to you, attempting to be minimal and un-opinionated. It tries to beinvisibleto the end user as much as possible. It strives for the basic principle that,"if it looks like it should work, it should work."Entangle leans on the OS scheduler to prioritize processes based on the behavior of those processes and underlying resource utilizations. It therefore does not provide its own redundant (which is to saycentralized) scheduler or task manager. Because of this, top-down visibility or control of workflow processes is not as easy as with centralized task managers.Entangle prefers the non-API approach, where it looks like regular python expressions, over strict API's or invocation idioms. This makes it easier to pick up and use and plays well with 3rd party frameworks too.Use CasesBecause of these tradeoffs, there are certain use cases that align with entangle and others that probably do not.If you want top-down visibility & control of workflows and tasks, Entangle is probably not ready for you.If you have lots of CPUs, entangle could be for you! If you want easy python workflows that span local and remote cloud resources, entangle could be for you.
If you want to write custom handlers that enrich or execute code in custom ways for your needs, entangle makes this easy for you.OrchestrationOne focused use case for entangle is when you want to orchestrate across different compute nodes, remote APIs and other disparate endpoints in a single workflow, with inherent parallelism.Each step of the workflow has different parameters, needs and protocols used to communicate with it.
Such a workflow might simply look like:data=data_refinement(get_source_data())result=measure_vectors(vector1(data("vector1")),vector2(data("vector2")),vector3(data("vector3")))GPU ProcessingAnother use case is the need to run multiple parallel tasks that operate on matrix data using a GPU.
Entangle makes this quite easy as seen inGPU Example,Docker ExampleandShared Memory ExampleDevOpsFor devops use cases Entangle allows you to write simple, parallel workflow graphs usingplain old python. This let's you write efficient parallel devops pipelines with ease. Build simple workflows that do powerful things like orchestrating across multiple clouds, services, repositories etc in an efficient dataflow parallel design.What Entangle is notHere are some things entangle is not,out-of-the-box. This isn't to say entangle can't do these things. In fact, entangle is designed to be a low level framework for implementing these kinds of things.Entangle does not yet perform fail over (TBD)Entangle is not a batch process framework (TBD)Entangle is not map/reduceEntangle is not a centralized task managerDeclarative InfrastructureEntangle allows you to target specific infrastructure environments or needs using simple decorators.For example, to specify a process run on local hardware you can use the @local decorator@process@localdefmyfunc():returnIf you want to execute a function in AWS EC2 or fargate, you could write it as:@process@aws(keys=[])@ec2(ami='ami-12345')defmyfunc():return@process@aws(keys=[])@fargate(ram='2GB',cpu='Xeon')defmyfunc():returnor using docker containers@process@docker(image="tensorflow/tensorflow:latest-gpu")defreduce_sum():importtensorflowastfreturntf.reduce_sum(tf.random.normal([1000,1000]))If you have a custom on-prem environment you can write a simple decorator that deploys the task to that and use it alongside other infrastructure decorators.Where, What & How: Using MixinsEntangle uses the concept ofmixinsto associate infrastructure needs (where) with compute (what) and concurrency needs (how).
Thus, it allows you to mix and match combinations of these within a single workflow or dataflow.For example, you might need to get data fromAWS Lambda, runGPU Algorithm on that data inside a containerthen send those results to 10 CPUs in acompute cloudforparallel analysis, then gather those results and send them to aweb serviceon your network for storage or rendering.
All these steps have different locations, infrastructure requirements, compute needs, processing times, and protocols.ExecutionAs we mentioned above, entangle workflows will fan out during execution and occupy CPUs throughout the workflow. The OS will determine the priority of processes based on their resource needs at the time of execution.
Here is a simple workflow and diagram showing how the parallel execution unfolds.result=add(mult(one(),two()),three())This execution order applies to workflows in entange. If you use@dataflowdecorator the execution follows that of a dataflow compute model. Refer to the sectionDataflowsfor more information.Threads vs ProcessesIn Python, threads do not execute in parallel to one another, it only gives the illusion of such. Python handles the context switching between threads and is limited by the GIL.
Processes on the other hand, are not controlled by a GIL and can thus truly run in parallel. The host operating system governs the sheduling of processes and entangle is designed to exploit this benefit.WorkflowsA workflow in Entangle is just a fancy term for a call graph of function processes. The example above inExecutionis a simple workflow.
Workflows execute innatural dependency orderingthat you'd expect from any python function call. The inner most dependencies are invoked first so they can return their values to parent functions.Note that this paradigm is pretty much the way most imperative languages operate today, but it does differ fromdataflowswhich we talk about down belowDataflows.Imperative vs Structured DeclarationIn Entangle, there are two ways you can write your workflows, depending which is more convenient for you. Both produce in the same execution sequence and results.Let's look at the example below:result=add(add(num(6),two()ifFalseelseone()),subtract(five(),two()))This represents thestructuredparadigm, based off a more strict type of lambda math notation where functions invoke functions until the top-most value is produced.We can also write this as a sequence ofimperativedeclarations_five=five()_two=two()_sub=subtract(_five,_two)_num=num(6)_two2=two()ifFalseelseone()_add1=add(_num,_two2)result=add(_add1,_sub)Process BehaviorKeyword parameters on the@processdecorator allow you to control some meta-behavior of the process.WaitWait indicates how long a function should wait before its arguments arrive. It is a sibling totimeouthowever it is different.@process(wait=20)defvalues(*args):values=[argforarginargs]returnvalueso=values(one(),train())In the above example, it is saying that thevaluesfunction will wait up to 20 seconds forbothone()andtrain()functions to complete and return values otherwise it will throw aProcessTimeoutException.TimeoutTimeout is more self-evident. It is the wait period in seconds, entangle will allow a process to run.# Wait at most, 3 seconds for this task to complete@process(timeout=3)deftask():returnTrue# Wait indefinitely for this task to complete@processdeftaskB():returnFalseWhen a process times out, aProcessTimeoutExceptionwill be thrown by Entangle and the process will be terminated if it is still alive.CompositionEntangle offers a couple different ways to use composition effectively: withdecoratorsand withworkflows.Decorator CompositionYou can compose your tasks by combining process and infrastructure decorators.Again, in the example below, we are declaring aprocessandlocalinfrastructure for our task to run by composing two decorators together.@process@localdeftaskA():returnor, specifying that the task run as a process inside AWS fargate, unchanged.@process@aws(keys=[])@fargate(ram='2GB',cpu=4)deftaskA():returnWorkflow CompositionComposing workflows is just as simple. You can write code that itself constructs workflows on the fly easily.fromentangle.processimportprocessfromentangle.httpimportrequestfromentangle.workflowimportworkflow@process@request(url='https://datausa.io/api/data',method='GET')defmydata(data):importjsondata=json.loads(data)returnint(data['data'][0]['Year'])@processdeftwo():return2@processdefadd(a,b):v=int(a)+int(b)print("ADD: *"+str(v)+"*")returnv@workflowdefworkflow1():returnadd(mydata(drilldowns='Nation',measures='Population'),two())@workflowdefworkflow2(value):returnadd(value(),two())result=workflow2(workflow1)print(result())The key to making this work is thedeferring of executiontrait of Entangle which we will discuss in a later post.
But essentially it allows for separation of workflowdeclarationfromexecution. Doing this allows you to treat workflows as objects and pass them around anywhere a normal python function (or workflow) is expected. Prior to execution.ContainersEntangle supports two container technologies:dockerandsingularity(TBD). These are used with the associated decorators@dockerand@singularity.
Using containers allows you to run functions that have complex OS or python depdendencies not native to your hosting environment.For a complete example, please seeDocker ExampleDataflowsEntangle supports two kinds of execution flow,dataflow[8]andworkflow(or what is more traditionally calledcontrol flow). They both complete a DAG-based execution graph but in slightly different ways and with different advantages to the programmer.As wikipedia states[8]:Dataflow is a software paradigm based on the idea of disconnecting computational actors into stages (pipelines) that can execute concurrently. Dataflow can also be called stream processing or reactive programming.[1]However, Merriam-Webster's simple definition[9]illuminates a key trait of dataflows - "...as data becomes available": a computer architecture that utilizes multiple parallel processors to perform simultaneous operations as data becomes availableData ReadinessIn many parallel data computations the arrival or readiness of some data might lag behind other data, perhaps coming from longer computations or farther away.
True dataflow models allow the computation to proceed on a parallel pathas far as it can gowith the currently available data.
This means dependent operations are not held up by control flow execution order in some cases and the overall computation is optimized.Dataflows vs WorkflowsAuthor, slikts[1], descibes these differences very nicely (from[1]).Control flow refers to the path the point of execution takes in a program, and sequential programming that focuses on explicit control flow using control structures like loops or conditionals is called imperative programming. In an imperative model, data may follow the control flow, but the main question is about the order of execution.Dataflow abstracts over explicit control flow by placing the emphasis on the routing and transformation of data and is part of the declarative programming paradigm. In a dataflow model, control follows data and computations are executed implicitly based on data availability.Concurrency control refers to the use of explicit mechanisms like locks to synchronize interdependent concurrent computations. Dataflow is also used to abstract over explicit concurrency control.Simple ExampleLet's start with a simple workflow example:A(B(),C())In traditionalcontrol flowor what I calllambda based[10]execution, the programming language'sdependency analysiswill determine the order of execution. In this exampleB()andC()are dependencies ofA()and thus need to completebeforeA()can be executed. In other words, they areinputstoA(). Basic stuff.This means the execution of each compute function is aware of the specific dependent functions it must resolve first.
We call thiscontrol depedency[2].Let's say the dependency was reversed. Whereby, a value computed byA()was a dependency ofbothB()andC(). How would we write this in conventionalcontrol flow?We might do something like this.B(A())C(A())Breaking our previous single expression into multiple expressions. However, in this case,A()is being invoked twice, which could produce different values.
So we might introduce a variablea=A()B=B(a)C=C(a)Now we have 3 expressions that must run in a proper order. We have done some of the work by making a separate expression for our dependent valuea. But for large dataflows this can be a bigger burden on the programmer to unravel all the dependencies and put them in proper order.What if the execution of an expression was not computed using the traditionaldependency analysismost languages use today but instead was defined by stricterdataflowsemantics?DAG DataflowIn dataflow, a DAG represents the flow of values from compute nodes where each node computes its value once and the value isemittedor sent to directionally connected nodes in the DAG.This paradigm makes it easier to express our intentions of sharing values fromA()by computing it once and sending the results toB()andC(). NeitherB()norC()explicitly depend onA(). The dataflow DAG provides the dependency structure for all the compute nodes.Now let's rewrite our expression if it were executed in strictdatafloworder.A(B(),C())Here, the dataflow engine executing this expression understands the intention to computeA()first, thenin parallelcomputeB()andC()with thesameresult computed only once fromA()as their input.
Written imperatively, this would equate to:a=A()B(a)C(a)whereB(a)andC(a)run in parallel.IMPORTANT!The dataflow syntax provides the necessary graph structure for a dataflow engine to know explicitly which functions can operate in parallel.Results ComparisonSo what are the differences in the results from ourworkflowversion and ourdataflowversion? It should be clear that the workflow version takes as input 2 values (B(),C()) and produces 1 value, A().However, our dataflow version is different. It takes as input 1 value A() and produces two results, B() and C(), in parallel. So the computations are different!Advantages of Strict DataflowAs was pointed out in the intro to this section, dataflow provides declarativedata dependencymodelling for a computation. This is sometimes a more natural way of thinking about a problem for the human programmer.
It allows a clean separation between the initial state of a dataflow and various desired outcomes that would be more difficult to model usingcontrol flowprogramming, as the programmer will have to use multiple imperative steps to introduce the proper execution order and indicating which computations are parallel is unclear from linear ordering.Dataflow has improved efficiencies when it comes to data-centric computations as well because it only computes nodes once per DAG execution.
This approach requires nocachingorvariablesthat might be required with imperative-based control flow.Naturally ParallelA dataflow DAG is a naturally and implicitly parallel model - by its declarative structure. For CPU-bound, data centric tasks it is simple and easy to understand for this reason.Detailed ExampleFor a more detailed example of using@dataflowin entangle seeDataflow Examples.ReferencesConcurrency Glossary -https://slikts.github.io/concurrency-glossary/Dependency Graphs -https://en.wikipedia.org/wiki/Dependency_graphDataflow Programming -https://en.wikipedia.org/wiki/Dataflow_programmingData-Flow vs Control-Flow for Extreme Level Computing -https://ieeexplore.ieee.org/document/6919190Advances in Dataflow Programming Languages -https://futureofcoding.org/notes/dataflow/advances-in-dataflow-programming-langauges.htmlData dependency -https://en.wikipedia.org/wiki/Data_dependencyAn introduction to a formal theory of dependence analysis -https://link.springer.com/article/10.1007/BF00128174Dataflow -https://en.wikipedia.org/wiki/DataflowDataflow -https://www.merriam-webster.com/dictionary/dataflowFunctional Programming/Lambda Calculus -https://www.tutorialspoint.com/functional_programming/functional_programming_lambda_calculus.htmSchedulersEntangle supports a composition based mechanism for attaching schedulers to workflows and functions.
The scheduler class will control access to CPU resources based on its constraints. For example, if you want to run a workflow with potentially 20 parallel tasks, but only want to allocate 4 CPUs to execute the workflow, the scheduler class can ensure entangle doesn't spawn more processes than requested.
Schedulers wrap individual functions and pull CPU "cookies" off a scheduler queue to hand off to processes. Each cookie contains a CPU identifier that the process then binds to. When a cookie is placed on the queue by a process it means that CPU (id) is available for use.Parallel processes thus use the queue mechanism toself-organizearound the allocated CPUs by requesting cookies, assigning their CPU affinity to that cpu id, running their behaviors and returning the cookie to the queue when complete.
This approach requires no centralized scheduler server as the workfow processes all use the same multiprocessing.Queue to retrieve CPU cookies.Pluggable SchedulersEntangle allows you to provide your own scheduler class using the decorator.@scheduler(cpus=4,impl='my.package.MyScheduler'}defmyfunc():returnCurrently, the scheduler class need only implement one method.def register(self, f, cpus=12):and return afunctionorpartialthat wraps the provided function with scheduler behavior.To see the implementation ofDefaultSchedulerclickhere.For a workflow example using scheduler seeScheduler Examplebelow.Distributed FlowsEntangle allows you to pass a workflow (or dataflow) to a remote machine for execution. When combined with@schedulerdecorators this also forwards scheduler behavior to the remote machine where it manages the received workflow there.
This type of propagation requires no centralized (i.e. shared) scheduler or services and thus scales very well.Moreover, parts of a workflow can be sent to different machines for a truly distributed workflow.SSH DecoratorFunctions or flows (graph of functions) are remoted by using the@sshdecorator like the example below.@ssh(user='me',host='radiant',key='/home/me/.ssh/id_rsa.pub',python='/home/me/venv/bin/python')@scheduler(**scheduler_config)@threaddefworkflow2():passIn this example, we have declared a (contrived) workflow that adds the return values of 2 embedded functions and returns it.
The@sshdecorator indicates that this workflow is to be copied and executed on the remote servermyserveras usermeusing the python executable at/home/me/venv/bin/python.
Entangle will marshall the codes to the remote server and execute them there.Scheduler PropagationIf any dependent or subsequent functions are invoked on the remote server, any decorators that apply to those will be enforced.
If you use@schedulerthen it will utilize thescheduler queueto request CPU cookies. If you also use another@sshdecorator then that dependent function will be shipped to a 3rd remote server and the process repeated there.diagram hereEach time a workflow decorated with@scheduleris sent to a remote machine, that scheduler then manages its portion of the workflow and any dependent functions that it might resolve.
This pattern forms a sort ofdistributed treeof schedulers that work in parallel across multiple machines, yet fully resolve to complete the root workflow.Let's take a closer look at this example, which uses 3 different machines to solve its workflow.@ssh(user='darren',host='miko',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@scheduler(**config)@processdeftwo():# run some codesreturn2@ssh(user='darren',host='radiant',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@scheduler(**config)@processdefthree():# run some codesreturn3@scheduler(**config)@processdefadd(a,b):v=int(a.get_result())+int(b)returnv@ssh(user='darren',host='phoenix',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@scheduler(**config)@processdefworkflow():_add=add(three(),two())return_add()In the above example, theworkflow()is first sent to machinephoenixand executed there. It wraps the functionaddwhich also executes onphoenixbecause it has no@sshdecorator and the workflow is already there.Theadd()function requires the functionsthree()andtwo()be solved first. These two functions are sent to machinesmikoandradiantto be solved.
The results are returned to theaddfunction running onphoenixand the result of theworkflow()is returned to the calling machine, or the machine where the workflow was executed on.Once theworkflow()reachesphoenixthe@schedulerattached to the workflow manages the CPU's there according to its constraints.
Since theadd()function has two dependencies that can run in parallel the@schedularcan request 2 CPUs and run them in parallel.diagramExamplesGPU ExampleShared Memory ExampleAI ExampleDocker ExampleDataflow ExamplesData-Driven BranchingDistributed DataflowScheduler ExampleGraph ExampleWorkflow Future ExampleRetry ExampleGeneral ExampleThere are a variety of example workflows and dataflows you can run. In addition to the sample code provided below you can run these using the following commands.$python-mentangle.examples.example
$python-mentangle.examples.example2
$python-mentangle.examples.example3
$python-mentangle.examples.example4
$python-mentangle.examples.example5
$python-mentangle.examples.example6
$python-mentangle.examples.example_graph.py
$python-mentangle.examples.example_graph_future.py
$python-mentangle.examples.example_with_future.py
$python-mentangle.examples.lambdaexample
$python-mentangle.examples.listexample
$python-mentangle.examples.listexample2
$python-mentangle.examples.dataflowexample
$python-mentangle.examples.dataflowexample2
$python-mentangle.examples.dockerexample
$python-mentangle.examples.aiexample
$python-mentangle.examples.retry_example
$python-mentangle.examples.schedulerexample
$python-mentangle.examples.schedulerexample2
$python-mentangle.examples.sshdatafloweexample
$python-mentangle.examples.sshschedulerexample
$python-mentangle.examples.timeoutexampleFor a complete list of the examples source code and binders to run them please visit the wiki.GPU ExampleThis example assumes you have installednvidia-cuda-toolkitand associated python packages along withnumba.In this example, two vectorized functions with different sized matrices are run in parallel, and their times are gathered.importnumpyasnpfromentangle.processimportprocessfromtimeitimportdefault_timerastimerfromnumbaimportvectorize@processdefdovectors1():@vectorize(['float32(float32, float32)'],target='cuda')defpow(a,b):returna**bvec_size=100a=b=np.array(np.random.sample(vec_size),dtype=np.float32)c=np.zeros(vec_size,dtype=np.float32)start=timer()pow(a,b)duration=timer()-startreturnduration@processdefdovectors2():@vectorize(['float32(float32, float32)'],target='cuda')defpow(a,b):returna**bvec_size=100000000a=b=np.array(np.random.sample(vec_size),dtype=np.float32)c=np.zeros(vec_size,dtype=np.float32)start=timer()pow(a,b)duration=timer()-startreturnduration@processdefdurations(*args):times=[argforarginargs]returntimesdp=durations(dovectors1(),dovectors2())print(dp())Which outputs something like[0.21504536108113825,0.3445616390090436]Shared Memory ExampleThe default return value conduit in Entangle is the Queue. Task return values are marshalled back through queues where they are gathered and provided as function parameters to the parent process task.
This method is not desirable for very large data sets such as matrices in GPU computations. The below example shows how Entangle uses python 3.8's shared memory feature to implicitly share volatile memory across native parallel processes.importnumpyasnpfromentangle.processimportprocessfromtimeitimportdefault_timerastimerfromnumbaimportvectorize@process(shared_memory=True)defdopow(names,smm=None,sm=None):(namea,nameb,shapea,shapeb,typea,typeb)=namesstart=timer()shma=sm(namea)shmb=sm(nameb)# Get matrixes from shared memorynp_shma=np.frombuffer(shma.buf,dtype=typea)np_shmb=np.frombuffer(shmb.buf,dtype=typeb)@vectorize(['float32(float32, float32)'],target='cuda')defpow(a,b):returna**bpow(np_shma,np_shmb)duration=timer()-startprint("Powers Time: ",duration)@process(shared_memory=True)defcreatevectors(smm=None,sm=None):vec_size=100000000start=timer()a=b=np.array(np.random.sample(vec_size),dtype=np.float32)c=np.zeros(vec_size,dtype=np.float32)# create shared memory for matricesshma=smm.SharedMemory(a.nbytes)shmb=smm.SharedMemory(b.nbytes)names=(shma.name,shmb.name,a.shape,b.shape,a.dtype,b.dtype)duration=timer()-startprint("Create Vectors Time: ",duration)returnnamesdp=dopow(createvectors())dp()OutputsCreateVectorsTime:0.8577492530457675
PowersTime:0.8135421359911561SharedMemoryManager & SharedMemoryIn the example above, you will notice two special keywords being passed into the functions,def createvectors(smm=None, sm=None):smmis a handle to theSharedMemoryManagerbeing used for this workflow andsmis a handle to theSharedMemoryclass needed to acquire the shared memory segments by name.
If you setshared_memory=Truethen youmustinclude these keyword arguments in your method or an error will occur.Now, you might be asking yourself, if one of the design goals wasshared-nothingthen why are we talking aboutshared memory?
When we say "shared" (in shared-nothing) we refer to resources that have to be synchronized or locked when accessed by parallel processes, thereby creating bottlenecks in the execution.
The shared memory example here does not introduce any contention, rather, it is used in a pipeline fashion.
In this approach, a given shared memory address is only updated by one process at a time (e.g. using it to return its data to the waiting process). Multiple shared memory segments can be created during the course of a workflow
for parallel running processes.AI ExampleHere is an example that uses tensorflow to train a model and return the summary.fromentangle.logging.debugimportloggingfromentangle.dockerimportdockerfromentangle.processimportprocess@process@docker(image="tensorflow/tensorflow:latest-gpu",packages=['tensorflow_datasets'])deftrain():importtensorflowastfimporttensorflow_datasetsastfds(ds_train,ds_test),ds_info=tfds.load('mnist',split=['train','test'],shuffle_files=True,as_supervised=True,with_info=True,)defnormalize_img(image,label):returntf.cast(image,tf.float32)/255.,labelds_train=ds_train.map(normalize_img,num_parallel_calls=tf.data.experimental.AUTOTUNE)ds_train=ds_train.cache()ds_train=ds_train.shuffle(ds_info.splits['train'].num_examples)ds_train=ds_train.batch(128)ds_train=ds_train.prefetch(tf.data.experimental.AUTOTUNE)ds_test=ds_test.map(normalize_img,num_parallel_calls=tf.data.experimental.AUTOTUNE)ds_test=ds_test.batch(128)ds_test=ds_test.cache()ds_test=ds_test.prefetch(tf.data.experimental.AUTOTUNE)model=tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),tf.keras.layers.Dense(128,activation='relu'),tf.keras.layers.Dense(10)])model.compile(optimizer=tf.keras.optimizers.Adam(0.001),loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],)model.fit(ds_train,epochs=6,verbose=0,validation_data=ds_test,)returnmodel.summary()model=train()print(model())Docker ExampleIn this example we are running a process that spawns the decorated function inside a docker container and waits for the result.
We compose this using the@processand@dockerdecorators to achieve the design. The functionreduce_sumis runinsidethe docker container using imagetensorflow/tensorflow:latest-gpuand the result is returned seamlessly to the workflow.fromentangle.dockerimportdockerfromentangle.processimportprocessimportlogginglogging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',level=logging.INFO)@process@docker(image="tensorflow/tensorflow:latest-gpu")defreduce_sum():importtensorflowastfreturntf.reduce_sum(tf.random.normal([1000,1000]))rs=reduce_sum()print(rs())Tensorflow GPU ContainerThe above example launches a GPU enabled docker on thenvidia dockerplatform (running on your local machine). Tensorflow by default will consume the entire GPU for its processing, however if you want to run parallel GPU jobs that only consume GPU memoryas needed, then you need to use:@docker(image="tensorflow/tensorflow:latest-gpu", consume_gpu=False)Dataflow ExamplesThe example below demonstrates the dataflow capability of Entangle. This is a different compute paradigm from workflows. Please read the section onDataflows vs Workflowsfor complete explanation of the difference.NOTE: We use threads as our execution unit in this example as it makes seeing the output possible. With@processyou won't see the aggregate output on your console, instead it will be logged toentangle.logfile.
With Entangle you decide whether to use concurrency (threads) or parallelism (processes). Entangle is itself, threadless.importthreadingimporttimefromentangle.logging.debugimportloggingfromentangle.dataflowimportthreadfromentangle.dataflowimportprocessfromentangle.dataflowimportdataflowdeftriggered(func,result):print("triggered:{}{}".format(func.__name__,result))@dataflow(callback=triggered)@threaddefprintx(x):print('printx:{}'.format(threading.current_thread().name))return("X:{}".format(x))@dataflow(callback=triggered)@threaddefprinty(y):print('printy:{}'.format(threading.current_thread().name))return("Y:{}".format(y))@dataflow(callback=triggered)@threaddefprintz(z):print('printz:{}'.format(threading.current_thread().name))return("Z:{}".format(z))@dataflow(callback=triggered)@threaddefecho(e):print('echo:{}'.format(threading.current_thread().name))return"Echo!{}".format(e)@dataflow(executor='thread',callback=triggered,maxworkers=3)defemit(a,**kwargs):print('emit:{}'.format(threading.current_thread().name))returna+"!"# Create the dataflow graphflow=emit(printx(printz(echo())),printy(printz()),printy())# Invoke the dataflow graph with initial inputflow('emit')time.sleep(2)# Call flow again with different input valueflow('HELLO')Data-Driven BranchingIt's useful to have a data flow that routes to different paths depending on input data.
Entangle makes this relatively easy. The example below embeds a lambda expression directly in the dataflow structure that chooses eitherprintx()orprinty()as the next compute node depending on what the input value is -after emit has generated the value.In the snippet belowemitfirst produces a value based on some input, the result is emitted to eitherprintx()orprinty()depending on the value of the result.
Note that this is computed during the execution of the DAG, not at declaration time.flow=emit(lambdax:printx()ifx=='emit'elseprinty())flow('emit')Full example:importthreadingimporttimefromentangle.dataflowimportthreadfromentangle.dataflowimportdataflowdeftriggered(func,result):print("triggered:{}{}".format(func.__name__,result))@dataflow(callback=triggered)@threaddefprintx(x):print('printx:{}'.format(threading.current_thread().name))return("X:{}".format(x))@dataflow(callback=triggered)@threaddefprinty(y):print('printy:{}'.format(threading.current_thread().name))return("Y:{}".format(y))@dataflow(executor='thread',callback=triggered,maxworkers=3)defemit(a,**kwargs):print('emit:{}'.format(threading.current_thread().name))returna+"!"# Create the dataflow graph# Defer whether we will forward to printx() or printy() depending# on the result receive from emit. This won't be known until the data is ready.flow=emit(lambdax:printx()ifx=='emit'elseprinty())# Invoke the dataflow graph with initial inputflow('emit')time.sleep(2)# Call flow again with different input valueflow('HELLO')Which outputs:printx: emit MainThread
triggered: inner X: emit
printz: ThreadPoolExecutor-3_0
triggered: inner Z: X: emit
----------------------------
printy: MainThread
triggered: inner Y: HELLODistributed DataflowIn the example below, we combine@dataflowwith@sshto get instant distributed dataflow!importthreadingimporttimefromentangle.logging.debugimportloggingfromentangle.sshimportsshfromentangle.processimportprocessfromentangle.dataflowimportdataflowdeftriggered(func,result):print("triggered:{}{}".format(func.__name__,result))@dataflow(callback=triggered)@ssh(user='darren',host='miko',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@processdefprintz(z):print('printz:{}'.format(threading.current_thread().name))withopen('/tmp/printz.out','w')aspr:pr.write("Z:{}".format(z))return"Z:{}".format(z)@dataflow(callback=triggered)@ssh(user='darren',host='radiant',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@processdefprintx(x):print('printx:{}'.format(threading.current_thread().name))withopen('/tmp/printx.out','w')aspr:pr.write("X:{}".format(x))return"X:{}".format(x)@dataflow(callback=triggered)@processdefprinty(y):print('printy:{}'.format(threading.current_thread().name))return"Y:{}".format(y)@dataflow(callback=triggered)@ssh(user='darren',host='radiant',key='/home/darren/.ssh/id_rsa.pub',python='/home/darren/venv/bin/python')@processdefecho(e):print('echo:{}'.format(threading.current_thread().name))withopen('/tmp/echo.out','w')aspr:pr.write("Echo!{}".format(e))return"Echo!{}".format(e)@dataflow(callback=triggered,maxworkers=3)defemit(value):print('emit:{}'.format(threading.current_thread().name))returnvalue+"!"if__name__=='__main__':results=[]# Create the dataflow graphflow=emit(printx(printz(echo())),printy(printz()),printy())result=flow('emit')Scheduler Examplefromentangle.logging.debugimportloggingfromentangle.processimportprocessfromentangle.httpimportrequestfromentangle.workflowimportworkflowfromentangle.schedulerimportschedulerscheduler_config={'cpus':2,'impl':'entangle.scheduler.DefaultScheduler'}@scheduler(**scheduler_config)@processdeftwo():return2@scheduler(**scheduler_config)@processdefthree():return3@scheduler(**scheduler_config)@processdefadd(a,b):print("add:{}{}".format(a,b))v=int(a)+int(b)print("ADD: *"+str(v)+"*")returnv@scheduler(**scheduler_config)@workflowdefworkflow2():returnadd(three(),two())result=workflow2()print(result())Graph ExampleThe follow example code shows how we can collect the call graph trace for our workflow and display it.importjsonimporttimeimportasynciofromentangle.logging.debugimportloggingfromentangle.processimportprocess@processdefone():return1@processdeftwo():return2@processdeffive():return5@processdefnum(n):returnn@processdefadd(a,b):v=int(a)+int(b)print("ADD: *"+str(v)+"*")returnv@processdefsubtract(a,b):returnint(a)-int(b)if__name__=='__main__':workflow=add(add(num(6),two()ifFalseelseone()),subtract(five(),add(subtract(num(8),two()),one())))result=workflow()print(result)graph=workflow.graph(wait=True)print("GRAPH:",json.dumps(graph,indent=4))Which outputs this graph structureGRAPH:{"add":[{"add":{"num":{"6":[]},"one":[]}},{"subtract":{"five":[],"add":{"subtract":{"num":{"8":[]},"two":[]},"one":[]}}}]}We can also use futures to wait for the graph data to arrive as a callback.future=workflow.graph(wait=False)defshow_graph(graph):print("GRAPH:",graph.result())future.add_done_callback(show_graph)future.entangle()Workflow Future ExampleEntangle allows you to use future results for workflows if the blocking method doesn't meet your use case.
To do this, we alter the invocation of the workflow slightly.defcallback(result):print("CALLBACK:",result.result())# set up future callbacksfuture=workflow.future(callback=callback)print('Future:',future)# Trigger workflow. Does not blockworkflow(proc=True)# Notify results when availablefuture.entangle()# Does not blockRetry ExampleTo specify how many times a function should be retried before throwing an exception with a sleep value in seconds between retries.@process(retry=5.sleep=1)deffive():importtimeval=int(str(time.time()).split('.')[1])%5ifval!=0:raiseException("Not a FIVE!")return5General ExampleAn example of how entangle will be used (still in development)fromentangle.processimportprocessfromentangle.threadimportthreadfromentangle.taskimporttaskfromentangle.localimportlocalfromentangle.awsimportec2fromentangle.awsimportlmbdafromentangle.httpimportrequest@process(timeout=60)@local(cpus=4)defadd(a,b):returna+b@process(cache=True)@aws(keys=[])@ec2defone():return1@thread@localdeftwo():return2@lmbda(function='name')@aws(keys=[])defproxy():# lambda proxypass@processdefsubtract(a,b):returnint(a)-int(b)@processdeffive():return5@processdefnum(n):returnn@process@request(url='http://..../',method='POST')defrequest(data):# Manipulate http response data here and return new resultreturndata# 1,2,3 get passed to lambda function and result returnedresult=proxy(1,2,3)# Pass key:value params and get result from your functionresult=request(key1=value,key2=value)# parallel workflow is just "plain old python"result=add(add(num(6),two()ifFalseelseone()),subtract(five(),two()))print(result())LoggingLogging in Entangle is intended to be convenient and provide some useful out-of-the-box defaults that "just work".There are 3 default loggers you can import.fromentangle.logging.infoimportloggingfromentangle.logging.debugimportloggingfromentangle.logging.fileimportloggingAnd the details of each are:infoimportlogginglogging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',level=logging.INFO)debugimportlogginglogging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',level=logging.DEBUG)fileimportlogginglogging.basicConfig(filename='entangle.log',format='%(asctime)s:%(levelname)s:%(message)s',level=logging.DEBUG)You can of course provide your own logging configuration, but be sure to include it at the top of your file so the various entangle modules pick it up.Design ToolA prototype visual design tool for Entangle is shown below. More details will be posted on thye wikihere.