Creating Containerized Workflows with Argo | by George Novack | Towards Data Science

In a previous article, I explored Kubeflow Pipelines and walked through the process of creating and executing a simple Machine Learning Pipeline. In this article, I will take a closer look at Argo, the open-source workflow orchestration engine that is used as the default orchestration platform for Kubeflow Pipelines.
In order to start playing with Argo, we will first need a running Kubernetes cluster. I will be using EKS, the managed Kubernetes service on AWS, and will walk through the steps required to get Argo running on EKS. Any Kubernetes cluster will work, so long as Argo is deployed and has the permissions it needs to run Workflows on the cluster.
The easiest way to get started with EKS, is to use the eksctl command-line tool. The instructions for installing eksctl can be found here: https://github.com/weaveworks/eksctl/blob/main/README.md
Once eksctl is installed, we’ll create the Kubernetes cluster by running the following command:
eksctl create cluster
Note: The command above creates AWS resources that will incur costs on your account. By default, the eksctl create cluster command creates a Kubernetes cluster with 2 m5.large type EC2 Instances, each of which, at the time of this writing, cost around $0.10 per hour.
This command may take 15–20 minutes to finish provisioning the cluster. Once it is complete, the first thing we will do is create a Namespace that will house our Argo resources. We’ll use kubectl to do this:
kubectl create namespace argo
Next, we will download the Argo installation manifest from the Argo GitHub repository:
wget https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/install.yaml .
This will download a YAML manifest file called install.yaml that describes all of the Kubernetes resources that we will need to get Argo up and running.
We’ll have to make a few modifications to this file in order to access the Argo server UI.
Update the argo-server Service to be of type LoadBalancer. This will provision a Network Load Balancer in AWS that will route requests to the argo-server Service. The resulting resource definition should look like this:
Update the argo-server Service to be of type LoadBalancer. This will provision a Network Load Balancer in AWS that will route requests to the argo-server Service. The resulting resource definition should look like this:
2. Update the argo-server deployment to add an environment variable called BASE_HREF with a value of “/”. The result should look like this (the resource definition is truncated for brevity):
After making these modifications, we can create the resources defined in this manifest:
kubectl apply -n argo -f install.yaml
Next, we will need to create an Ingress resource:
Finally, in order for Argo to be able to access other Kubernetes resources, we will need to assign a role to the Argo Service Account. On a real production cluster, we would want to assign this Service Account to a role with specific limited permissions, but for now, we will just grant admin access to this account. We can accomplish this with the following command:
kubectl -n argo create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default
And that’s all the setup we will need to perform to get started. Now, we will get the public URL for the Argo server UI by running the following:
kubectl get svc argo-server -n argo
The public URL will be returned by this command in the EXTERNAL-IP column. It should end in .elb.amazonaws.com.
Now open a browser and navigate to: https://{YOUR_EXTERNAL_IP}:2746
Note: If you are using Chrome on MacOS and are having difficulty bypassing the security warning raised by your browser when navigating to the Argo Server UI, there is a somewhat hidden workaround in Chrome: Simply click anywhere in the browser window to ensure the window is in focus, then type thisisunsafe. For more details on this workaround, see This is unsafe — Bypassing the Google Chrome “Your connection is not private” Warning
If everything is working properly, you should see an Argo Server landing page like the one shown below:
With the setup out of the way, let’s execute our first workflow with Argo.
Navigate to Workflows (this is the top icon in the sidebar) > Submit New Workflow > Edit using full workflow options
This will open the Workflow creation experience, which includes a text editor populated with a basic Workflow YAML manifest, and additional tabs for Workflow parameters and metadata.
Let’s replace this default Workflow manifest with a simple Workflow that runs the Whalesay container as shown below, then click Create to run the Workflow:
Once you kick off the Workflow, you will be shown a Workflow diagram that will update as the workflow progresses (in this case, it will just be a single step). From this page, you will be able to view the details, inputs, outputs, and logs of your Workflow. There isn’t much to see here with our barebones Hello World Workflow, but you can navigate to Main Logs to confirm that the Whalesay ASCII art was logged.
Now that we know how to execute a Workflow, let’s take a closer look at the YAML manifest that we used to define the Workflow.
The first few lines of the manifest define Workflow metadata, specifically the generateName field, which will be combined with a randomly generated string to form the name of our Workflow (we could also have used the name field to explicitly define the name, but this name would have to be unique), and the namespace field, which defines the Kubernetes namespace in which the Workflow will be executed.
The rest of the manifest defines the Workflow Spec, which is where the actually Workflow logic is defined. In the example above, the Workflow Spec has 2 top-level fields:
entrypoint: This specifies the template that will be executed first.
templates: This defines all of the templates used in the Workflow. As we will see shortly, template is a bit of a loaded term in Argo, but here template simply means a unit of Workflow execution. In our Whalesay Workflow, we have just a single template named whalesay of type Container. Since this template defines a container to execute, we specify the container image to use, the command to execute, and the arguments to pass in.
Now let’s introduce a more complicated Workflow that executes a few different steps in sequence. This Workflow will accept 2 arguments, a salutation, and a username, and it will end by executing the Whalesay container using the provided arguments to create a greeting message.
Although we could easily accomplish this with just the Whalesay container, we will take a more roundabout approach here for the sake of demonstrating some of the other features that Argo offers.
Here is the YAML manifest for our new Workflow:
Let’s break this down:
The first difference you’ll notice is the addition of the arguments field on line 6. This is where we define the salutation and username parameters that will be passed into our Workflow.
The first template defined in this Workflow is on line 14. This template is called greeting, accepts a parameter called salutation, and is of type Script. Script templates are similar to Container templates in that they require a container image and command; however, Script templates allow us to pass in an inline script to execute on the container. This template executes a simple Python script that retrieves the value of the salutation parameter, then prints it out.
The next template is called user and is also of type Script. This template is just like the salutation template, except for that this template uses a shell script instead of a Python script (There is no particular reason not to use Python here; this is just to illustrate that we are not limited to Python as our scripting language).
The third template is the same Whalesay Container template that we used earlier, except now it accepts 2 parameters, salutation and username, and uses these parameters to create the greeting.
The fourth template is a Steps template. Steps templates allow us to chain together a series of other templates, creating a multi-step workflow where the outputs of one template can be used as the inputs of another. The first step uses the greeting template and passes in the Workflow parameter salutation, the second step uses the user template and passes in the Workflow parameter username, and the final template uses the whalesay template and passes in the outputs of the previous two steps.
Here is what the workflow diagram looks like for this workflow:
Note: If you executed this workflow (or just looked closely at the diagram) you may have noticed that the get-title and get-greeting steps are executed in parallel, and then after both complete the whalesay step runs. This is because the Steps template actually accepts a list of lists of steps. Steps in the inner lists will be run in parallel, and steps in the outer list will be run in sequence. For example, the list of lists structure for the Workflow above looks like:
[[get-title, get-greeting], [whalesay]]
This is why get-title and get-greeting are run in parallel, and whalesay is run after both complete.
One of the most important features of a workflow orchestration platform is the ability to share frequently-used steps across multiple workflows. This allows users of the platform to quickly get started creating workflows by using a library of common steps that have already been implemented by other users. For example, you might create reusable steps for common data transformations or for integrations with external services.
The construct in Argo that represents a reusable step (or set of steps) is called a Workflow Template. This is not to be confused with the templates that we have discussed earlier when defining our Workflow YAML manifests. For clarity, here is the difference:
template: represents a single unit of execution in a Workflow. There are a number of different types of templates, including Container, Script, and Steps.
Workflow Template: A Workflow definition that is saved in your cluster, which can then be executed as a standalone Workflow, or as part of any other Workflow.
Let’s create a Workflow Template to see how this works. Our Workflow Template will be a random number generator that accepts a min and max value, and returns a random integer in the specified range.
The Workflow Templates tab in the Argo Server UI is where we can view and create Workflow Templates.
Creating a Workflow Template is very similar to creating a Workflow, only when we create a Workflow Template, it is saved for later use rather than executed immediately in the way that a Workflow is.
Here is the YAML manifest for the Workflow Template we will be working with:
As you can see, this manifest is just like the Workflow manifests that we created earlier: it takes 2 parameters, min and max, and has a single Python script template which generates a random number in the specified range.
Once this Workflow Template is created, we will be able to view it from the Workflow Templates tab in the UI, and submit it as a standalone Workflow by selecting it and clicking Submit
We can also use this Workflow Template from within other Workflows. Let’s see how this works. We’ll go back to the Workflows tab and choose Submit New Workflow, then enter the following YAML manifest:
This Workflow is a lot like the multi-step Workflow that we already discussed, only this time the Steps template is not just chaining together templates defined directly in this Workflow. The first step in the Step template (line 19 above) uses the templateRef field instead of the template field to reference a template defined in a Workflow Template. In this particular case, we are referencing the generator template, which is defined in the random-number-generator Workflow Template.
In case this is still a little confusing, here is a diagram that illustrates the relationships between the Workflow, Workflow Template, and templates in this example:
When we execute this Workflow, the call-random-number-generator step will run just like it is a template defined directly within the Workflow, when in reality it’s implementation is encapsulated in the random-number-generator Workflow Template.
This means that we can create many other Workflows (or other Workflow Templates) that consume this functionality without concern about the details of its implementation, and if we ever need to modify this functionality (for example, the current random range excludes the max value, this may or may not be the desired behavior; or we may want to refactor this into a container image rather than an inline script in order to more easily test this function), we can do so in the Workflow Template, knowing that all consumers will start to use these modification without any additional effort on their end.
In this article, we saw how to get started with Argo Workflows by creating a few simple Workflows. We looked at how to pass parameters between multiple templates in a Workflow, and then how to encapsulate shared logic using Workflow Templates.
Although the examples we explored here were very simplistic, these same core concepts can be used to build and execute powerful containerized workflows on Kubernetes, and they will act as a strong foundation to build upon if you decide to continue exploring the other features that Argo has to offer.
You can find all of the YAML manifests that we created in the argo-workflows-getting-started respository: https://github.com/gnovack/argo-workflows-getting-started
https://argoproj.github.io/argo-workflows/quick-start/
https://www.eksworkshop.com/advanced/410_batch/
Thanks for reading. Feel free to reach out with any questions or comments.