Package: magnus
Hello from magnusMagnus is a simplified workflow definition language that helps in:Streamlined Design Process:Magnus enables users to efficiently plan their pipelines withstubbed nodes, along with offering support for various structures such astasks,parallel branches, andloops or map branchesin bothyamlor apython SDKfor maximum flexibility.Incremental Development:Build your pipeline piece by piece with Magnus, which allows for the
implementation of tasks aspython functions,notebooks, orshell scripts,
adapting to the developer's preferred tools and methods.Robust Testing:Ensure your pipeline performs as expected with the ability to test using sampled data. Magnus
also provides the capability tomock and patch tasksfor thorough evaluation before full-scale deployment.Seamless Deployment:Transition from the development stage to production with ease.
Magnus simplifies the process by requiringonly configuration changesto adapt to different environments, including support forargo workflows.Efficient Debugging:Quickly identify and resolve issues in pipeline execution with Magnus's local
debugging features. Retrieve data from failed tasks andretry failuresusing your chosen debugging tools to maintain a smooth development experience.Along with the developer friendly features, magnus also acts as an interface to production grade concepts
such asdata catalog,reproducibility,experiment trackingand secureaccess to secrets.What does it do?DocumentationMore details about the project and how to use it available here.InstallationThe minimum python version that magnus supports is 3.8pipinstallmagnusPlease look at theinstallation guidefor more information.ExampleYour application code. Use pydantic models as DTO.Assumed to be present atfunctions.pyfrompydanticimportBaseModelclassInnerModel(BaseModel):"""A pydantic model representing a group of related parameters."""foo:intbar:strclassParameter(BaseModel):"""A pydantic model representing the parameters of the whole pipeline."""x:inty:InnerModeldefreturn_parameter()->Parameter:"""The annotation of the return type of the function is not mandatorybut it is a good practice.Returns:Parameter: The parameters that should be used in downstream steps."""# Return type of a function should be a pydantic modelreturnParameter(x=1,y=InnerModel(foo=10,bar="hello world"))defdisplay_parameter(x:int,y:InnerModel):"""Annotating the arguments of the function is important formagnus to understand the type of parameters you want.Input args can be a pydantic model or the individual attributes."""print(x)# >>> prints 1print(y)# >>> prints InnerModel(foo=10, bar="hello world")Application code using driver functions.The code is runnable without any orchestration framework.fromfunctionsimportreturn_parameter,display_parametermy_param=return_parameter()display_parameter(my_param.x,my_param.y)Orchestration using magnuspython SDKyamlExample present at:examples/python-tasks.pyRun it as:python examples/python-tasks.pyfrommagnusimportPipeline,Taskdefmain():step1=Task(name="step1",command="examples.functions.return_parameter",)step2=Task(name="step2",command="examples.functions.display_parameter",terminate_with_success=True,)step1>>step2pipeline=Pipeline(start_at=step1,steps=[step1,step2],add_terminal_nodes=True,)pipeline.execute()if__name__=="__main__":main()Example present at:examples/python-tasks.yamlExecute via the cli:magnus execute -f examples/python-tasks.yamldag:description:|This is a simple pipeline that does 3 steps in sequence.In this example:1. First step: returns a "parameter" x as a Pydantic model2. Second step: Consumes that parameter and prints itThis pipeline demonstrates one way to pass small data from one step to another.start_at:step 1steps:step 1:type:taskcommand_type:python# (2)command:examples.functions.return_parameter# (1)next:step 2step 2:type:taskcommand_type:pythoncommand:examples.functions.display_parameternext:successsuccess:type:successfail:type:failTranspile to argo workflowsNo code change, just change the configuration.executor:type:"argo"config:image:magnus:demopersistent_volumes:-name:magnus-volumemount_path:/mntrun_log_store:type:file-systemconfig:log_folder:/mnt/run_log_storeMore details can be found inargo configuration.Execute the code asmagnus execute -f examples/python-tasks.yaml -c examples/configs/argo-config.yamExpandapiVersion:argoproj.io/v1alpha1kind:Workflowmetadata:generateName:magnus-dag-annotations:{}labels:{}spec:activeDeadlineSeconds:172800entrypoint:magnus-dagpodGC:strategy:OnPodCompletionretryStrategy:limit:'0'retryPolicy:Alwaysbackoff:duration:'120'factor:2maxDuration:'3600'serviceAccountName:default-editortemplates:-name:magnus-dagfailFast:truedag:tasks:-name:step-1-task-uvdp7htemplate:step-1-task-uvdp7hdepends:''-name:step-2-task-772vg3template:step-2-task-772vg3depends:step-1-task-uvdp7h.Succeeded-name:success-success-igzq2etemplate:success-success-igzq2edepends:step-2-task-772vg3.Succeeded-name:step-1-task-uvdp7hcontainer:image:magnus:democommand:-magnus-execute_single_node-'{{workflow.parameters.run_id}}'-step%1---log-level-WARNING---file-examples/python-tasks.yaml---config-file-examples/configs/argo-config.yamlvolumeMounts:-name:executor-0mountPath:/mntimagePullPolicy:''resources:limits:memory:1Gicpu:250mrequests:memory:1Gicpu:250m-name:step-2-task-772vg3container:image:magnus:democommand:-magnus-execute_single_node-'{{workflow.parameters.run_id}}'-step%2---log-level-WARNING---file-examples/python-tasks.yaml---config-file-examples/configs/argo-config.yamlvolumeMounts:-name:executor-0mountPath:/mntimagePullPolicy:''resources:limits:memory:1Gicpu:250mrequests:memory:1Gicpu:250m-name:success-success-igzq2econtainer:image:magnus:democommand:-magnus-execute_single_node-'{{workflow.parameters.run_id}}'-success---log-level-WARNING---file-examples/python-tasks.yaml---config-file-examples/configs/argo-config.yamlvolumeMounts:-name:executor-0mountPath:/mntimagePullPolicy:''resources:limits:memory:1Gicpu:250mrequests:memory:1Gicpu:250mtemplateDefaults:activeDeadlineSeconds:7200timeout:10800sarguments:parameters:-name:run_idvalue:'{{workflow.uid}}'volumes:-name:executor-0persistentVolumeClaim:claimName:magnus-volumePipelines can be:LinearA simple linear pipeline with tasks eitherpython functions,notebooks, orshell scriptsParallel branchesExecute branches in parallelloops or mapExecute a pipeline over an iterable parameter.Arbitrary nestingAny nesting of parallel within map and so on.