Package: sap-ai-core-metaflow
Metaflow library for SAP AI CoreThesap-ai-core-metaflowpackage is aMetaflow pluginthat generatesArgo Workflows.It adds Argo Workflows capabilities using Metaflow's@kubernetesdecorator.Table of ContentsPlugin InstallationInstall and configure Argo WorkflowsConfigure MetaflowRun the Argo WorkflowTemplateMetaflow-Argo DecoratorsCommand-line Parameters and Environment VariablesOther Supported FeaturesUse with SAP AICorePlugin Installationpipinstallsap-ai-core-metaflowInstall and configure Argo WorkflowsWith the Metaflow-Argo plugin you can execute flows in a K8S cluster.Note: you can also install a local K8S cluster using docker desktop (Mac, Windows) or K3D, minikube (Linux).InstallArgo Workflowsin the K8S cluster.Optionally exposeargo-serverto access ArgoWorkflows UI.Configure MetaflowFor a quick-start follow the steps below.(Find complete documentation here:Configuring Metaflow.)The MetaflowconfigFileAn S3 bucket can be configured to store training data and Metaflow code packages.Create the config file:~/.metaflowconfig/config.jsoncontaining your S3bucketandprefix:{"METAFLOW_DEFAULT_DATASTORE":"s3","METAFLOW_DATASTORE_SYSROOT_S3":"s3://<bucket>/<prefix>"}In addition, your localaws clihas to be configured for the same bucket.Create the K8S SecretThis secret allows an Argo Workflow to access the S3 bucket when running in the cluster:kubectlcreatesecretgenericdefault-object-store-secret--from-literal=AWS_ACCESS_KEY_ID=XXX--from-literal=AWS_SECRET_ACCESS_KEY=YYYRun the Argo WorkflowTemplateHere we usehelloworld.pyfrom theMetaflow Tutorials.Deploy the Argo WorkflowTemplatepythonhelloworld.py--withkubernetes:secrets=default-object-store-secretargocreateYou should see the following output:Metaflow2.5.0executingHelloFlowforuser:<xyz>Validatingyourflow...Thegraphlooksgood!Runningpylint...Pylintishappy!DeployinghelloflowtoArgoWorkflowTemplates...WorkflowTemplatehelloflowispushedtoArgoWorkflowssuccessfully.Trigger the Execution of the WorkflowTemplatepythonhelloworld.pyargotriggerYou should see arun-idafter a successful start of a workflow:Metaflow2.5.0executingHelloFlowforuser:<xyz>Validatingyourflow...Thegraphlooksgood!Runningpylint...Pylintishappy!WorkflowhelloflowistriggeredonArgoWorkflows(run-idhelloflow-92vbn).Check the Execution Statuspythonhelloworld.pyargolist-runsThe flow will display the stateRunningfor some time and finally the flow showsSucceeded:Metaflow2.5.0executingHelloFlowforuser:<xyz>Validatingyourflow...Thegraphlooksgood!Runningpylint...Pylintishappy!helloflow-92vbnstartedAt:'2022-06-15T14:00:27Z'SucceededMetaflow-Argo DecoratorsIn addition to general Metaflow decorators, i.e.@kubernetes,@resources,@environment, etc. the Metaflow-Argo plugin supports
additional Argo-specific customization for steps which support
input_artifacts ,output_artifacts, labels and shared_memory.The@argoDecorator on Step LevelA Metaflowstepcan override the default values specified on the
flow level or add extra configurations.LabelsLabels are attached as metadata to the pod executing astep:...@argo(labels={'plan':'basic'})@stepdeftraining():...Thesteplabels override the flow labels.Input ArtifactsThe artifacts configuration is inserted into the generated
WorkflowTemplate as-is.@argo(input_artifacts=[{'name':'input_dataset','path':'/tmp/train.csv','s3':{'endpoint':'s3.amazonaws.com','bucket':'my-bucket-name','key':'path/in/bucket','accessKeySecret':{'name':'my-s3-credentials','key':'accessKey'},'secretKeySecret':{'name':'my-s3-credentials','key':'secretKey'}}}])@stepdefstart(self):ds=pd.read_csv('/tmp/train.csv')If theDefault Artifact Repositoryis configured, the credentials can be ommited:@argo(input_artifacts=[{'name':'input_dataset','path':'/tmp/spam.csv','s3':{'key':'path/in/bucket'}}])Output ArtifactsPrerequisite: TheDefault Artifact Repositoryis configuredIf the configuration below is added to astep, Argo will upload the output artifact to the specified S3 bucket.@argo(output_artifacts=[{'name':'model','globalName':'model','path':'/tmp/model','archive':{'none':{}}}])@stepdeftrain_model(self):...withopen('/tmp/model','wb')asf:pickle.dump(self.model,f)Configure additional Disk Space using Persistent Volume Claim (PVC)If the pod's disk space is not sufficient, one can mount extra storage using aPersistent Volume Claim.The example below creates a pvc in WorkflowTemplate with a default size of 50GiB:@argo(mount_pvc="/mnt/data")@stepdefstart(self):# use /mnt/data for huge filesTo set the size for thePVC, add a value for the command-line parametervolumen-claimin MiB:pythonhelloflow.pyargocreate--volume-claim=100000Shared Memory AllocationFor adding moreshared memoryin MiB use the following configuration:@argo(shared_memory=100)@stepdeftrain_model(self):...Command-line Parameters and Environment VariablesThe Metaflow-Argo plugin allows overriding defaultflowvalues. This done through environment variables or command-line parameters. Command-line parameters naturally have precedence overenvvars.Command-line ParametersList Command-line ParametersUsage:helloworld.pyargocreate[OPTIONS]DeployanewversionofthisworkflowtoArgoWorkflowTemplates.

Options:--image-pull-secretTEXTDockerimagepullsecret.--labelTEXTLabeltoattachtotheworkflowtemplate.--annotationTEXTAnnotationtoattachtotheworkflowtemplate.--volume-claimINTEGERSizeofthevolumeClaiminMiB--k8s-namespaceTEXTDeployintothespecifiedkubernetesnamespace.--embeddedDon'tdownloadcodepackageintostepcontainers.Dockerimagesshouldhaveaflowandalldependenciesembedded.--max-workersINTEGERMaximumnumberofparallelpods.[default:100]--workflow-timeoutINTEGERWorkflowtimeoutinseconds.--only-jsonOnlyprintoutJSONsenttoArgo.Donotdeployanything.--helpShowthismessageandexit.Set the Flow NameThe Argo WorkflowTemplate name is taken by default from class name of the the Metaflow script. If you want to change the flow name, use:pythonhelloworld.pyargo--name=hellocreate@kubernetesDecorator Command Line ParameterThe Metaflow command-line parameter--with=kubernetesallows for setting configurations when using a kubernetes cluster:secretsto access object store, multiple secrets can be passed separated by,pythonhelloworld.py--with=kubernetes:secrets=default-object-store-secret,s3-secretargocreateimageto specify a docker imagepythonhelloworld.py--with=kubernetes:secrets=default-object-store-secret,image=demisto/sklearn:1.0.0.29944argocreatediskis used to reserve ephemeral storage in MiB for the node.pythonhelloworld.py--with=kubernetes:secrets=default-object-store-secret,disk=1024argocreateEmbedded code packageThe flag--embeddedgenerates steps' commands without a code package download; expects a code	package	to be alreadyembeddedinto a docker image.Environment VariablesPrefix for the Flow NameTo add a prefix to the flow name set the environment variableARGO_WORKFLOW_PREFIX.Default K8S NamespaceThe kubernetes namespace to store templates and execute workflows can be
set with envvarMETAFLOW_KUBERNETES_NAMESPACE.Default Environment Variables for Command-line ParametersMetaflow provides envvars for the correspondent command-line parameters by setting this prefix:METAFLOW_ARGO_CREATE_e.g.:exportMETAFLOW_ARGO_CREATE_MAX_WORKERS=4Other Supported FeaturesThe Metaflow-Argo plugin supports also many Metaflow features.@kubernetesDecoratorThe Metaflow-Argo plugin makes use of the specifications defined in@kubernetesdecorator to configure container resources.@kubernetes(image='tensorflow/tensorflow:2.6.0',cpu=8,memory=60000)@stepdeftraining():...@resourcesDecoratorFull support forRequesting resources with @resources
decorator.
If similar resources are specified in both@kubernetesand@resources, the maximum is taken.@kubernetes(image='tensorflow/tensorflow:2.4.2-gpu',node_selector='gpu=nvidia-tesla-k80')@resources(memory=60000,cpu=2)@stepdeftraining():...@environmentDecoratorIn addition toenvandenvFromenvvars can be specified in the decorator:@environment(vars={'var':os.getenv('var_1')})@stepdefstart(self):print(os.getenv('var'))Display Logs for a StepMetaflow stores stdout and stderr of every step/task and provides a
convenient way to access them, passing run-id and step-name:python00-helloworld/helloworld.pylogshelloflow-bdmmw/startOptionally specify a task-id, useful when have many instances of the
same step, e.g. when use foreach:python00-helloworld/helloworld.pylogshelloflow-bdmmw/hello/helloflow-bdmmw-3938807173Resume flowsSeeHow to debug failed
flowsfor details.python00-helloworld/helloworld.pyresume--origin-run-id=helloflow-bdmmwSchedule flowsMetaflow's@scheduledecorator is fully supported. E.g. a workflow template with the@schedule(hourly=True)will automatically trigger workflow execution
on every hour start.Automatically rerun failed steps@retrydecorator helps to deal with transient failures. SeeRetrying
Tasks with the retry
Decorator.Handle failures programmaticallyWhat if a default behavior to stop a whole flow if a particular step
fails is too limiting? With a help of@catchdecorator, it's possible
to handle step's failure in a next step.Catching Exceptions with the
catch
Decorator.Limit the Step Execution TimeUse@timeoutdecorator to guard yourself from accidental resources
consumption. ReadTiming out with the timeout
Decoratorfor more details.Use with SAP AICoreSAP AI Core does not allow access to its K8s cluster or argo-server. Workflow Templates are generated in json/yaml format and synced to K8s via the registered Git repo. The Workflows can then be triggered viaSAP AI API.The mandatory information for SAP AI Core is provided through labels and annotations.pythonhelloworld.py--with=kubernetes:secrets=default-object-store-secret,image=docker.io/<my-repo>/helloworldargocreate--image-pull-secret=<docker-secret>--label="scenarios.ai.sap.com/id:ml_usecase_name"--label="ai.sap.com/version:0.1.0"--annotation="scenarios.ai.sap.com/name:Hello-World"--annotation=""executables.ai.sap.com/name":"helloworld"--only-json > hello-world.jsonHereyou can find a step-by-step example for SAP AI Core.