A Simple MLOps Pipeline on Your Local Machine | by Kyle Gallatin | Towards Data Science

It can be difficult to get any kind of ops experience outside of enterprise without turning to the cloud. The simple reason is that there’s little benefit in not turning to the cloud. In order to provide a zero-cost entry point into MLOps for those new to the space, I wanted to see if I could set up a simple MLOps pipeline in a reasonable amount of time on my personal laptop.
This is a “tip of the iceberg” tutorial for MLOps. Much of it will be modeled loosely after the famous Continuous Delivery for Machine Learning paper (Sato et al, 2019) — but with much less depth considering I’m just one dude putting in a few hours here and there. Instead of a real “pipeline”...think of it more like Michael Scott’s Tube City.
Some of technologies used will be:
Python
Docker/Kubernetes
Mlflow
Minio
Seldon Core
Github
Jenkins
ArgoCD
None of this is fit for production as outlined, but many of the principles and technologies are widely applicable. You’ll really only need Docker Desktop to follow along. This will enable you to run Kubernetes. However, you should also clone this repo, as it has all of the additional scripts we’ll need.
One of the core tenets of data science that differentiates it from software engineering is its focus on experimentation. With software, you develop, you test, and you push features that are primarily code-based. In data science, on the other hand, you conduct heaps of experiments while making changes in configuration, data, features, etc. The output isn’t even necessarily “completed code,” but artifacts such as model weights.
Git-oriented source control isn’t the right tool for this problem. To create a robust training pipeline, you’d have to build quite a bit. Here, we’ll just ignore any restrictions on the model training approach and offload the results of our experiments in a central, shared repository.
Mlflow has become one of the most popular tools for experiment management. While it does offer a lot more than simple tracking functions, we’ll be focusing on those for the purpose of this exercise.
First, create a new folder in the mlops-pipeline repo for the storage of our data.
mkdir -p ./experiment-tracking/buckets/mlflow
Then, you can use the docker-compose file in experiment-tracking (outlined below) to start our required services.
Some credit for this approach goes to this post on mlflow and the minio docs, but I removed nginx here since we don’t really need it. Note: sql volumes were omitted due to a hack-y workaround for this error.
We provide the environment variables in a .env file and build with docker-compose --env-file ./.env up. Give it a minute, but this will start three services:
Minio to simulate S3 storageSQL to store mlflow dataMlflow itself for both server and UI
Minio to simulate S3 storage
SQL to store mlflow data
Mlflow itself for both server and UI
These three services are now running inside a docker network, but Mlflow and minio have also been made available locally. Check the following URLs on your local machine:
MLflow: http://localhost:5000
Minio: http://localhost:9000
Now we can log metrics and models to the file storage using Mlflow. Since minio is available as “S3” inside our docker network, this is easiest to run from the container.
docker exec -it mlflow_server bash
Running the code above in a terminal will start a session in the mlflow_server container. You can check which other containers are running with docker ps.
Now that we’re inside the container, we have access to the files within. If you’re using the GitHub repo, the following training script should already be available in the container.
Pretty low-effort Python, but this is a quick and easy way to make sure that things are working. Ideally, our training pipelines would extend to run anywhere and log artifacts back to mlflow/minio running in the cloud. However, the fact that minio is running locally inside docker makes things trickier. This is the simplest way for now.
python train.py
If it ran successfully, you can navigate back to the minio and Mlflow UIs to check the performance and files logged by our run.
While this path contains the artifacts necessary to deploy with an Mlflow server, we’ll be following a different approach just using the serialized model: model.pkl. To take this path further, you could work to integrate DVC, a popular command line tool for data versioning that works well with git.
A popular way to serve machine learning artifacts, like pickles, is to package them with flask APIs and serve them with a production ready web server like gunicorn.
To break the cycle a bit, I’m going to leverage the machine learning framework seldon core to serve our model. This will contribute the additional benefit of making it ready for kubernetes.
Seldon provides out-of-the-box inference servers for libraries like sklearn and tensorflow, but you can also use seldon-core to deploy a model with our own Docker container and code. If you look at mlops-pipeline/ml-app, you’ll be able to view some of the code for Seldon. The only thing we’re missing is our model.pkl, which we can get by copying from our local minio bucket.
cp experiment-tracking/buckets/mlflow/<INT>/<HASH>/artifacts/rf-regressor/model.pkl ml-app
Normally, we’d pull this right from the cloud when we deploy (I have a download.py file for this if I ever need it), but this example is just to illustrate how you can test a deployment using Seldon.
Now, we just need three things:
requirements.txt for our model runtime:
requirements.txt for our model runtime:
sklearnseldon-core
2. Model.py that defines a Model class:
3. Dockerfile to create our serving image:
Note the Seldon specific env variables and command at the bottom. Now, we can try running our model server:
docker build -t seldon-app .docker run -p 5001:5000 -it seldon-app # port 5001 in case of mlflow
This should start the server locally and make it reachable at port 5001. You can send a curl request, but it’s probably easier to run python tests.py in the directory. It’s a short script I made to grab a dataframe row and send it over HTTP to our service for a prediction.
It should return something like this (the Github version has a placeholder for Seldon metrics too):
{"data":{"names":[],"ndarray":[268788.9666666667]},"meta":{"metrics":[{"key":"mycounter","type":"COUNTER","value":1},{"key":"mygauge","type":"GAUGE","value":100},{"key":"mytimer","type":"TIMER","value":20.2}]}}
It works! Or at least it should.
Now, everything in this section really has nothing to do with ops. While our choice of serving framework has an effect on our pipeline, it’s nothing to do with CI/CD.
We definitely don’t want to run things manually every time we update our code or model, so let’s welcome some automation.
CI/CD refer to continuous integration and continuous delivery. The former involves frequent and iterative merge/build/testing of code to mitigate the risk of things breaking. The latter aims to ensure rapid and easy production deployments. There’s probably some overlap and ambiguity in the responsibilities of each, at least from my persepctive. I recommend looking it up if you’re interested in the specifics.
Jenkins is typically used in tandem with a source control management (SCM) tool like Github. Jenkins projects are often configured to trigger builds or run scripts when people push or merge branches in their repo. However, there are a number of available hooks. It can also be used to automate:
Building/tagging/pushing of software (Docker images)
Unit tests, integration tests, etc...
Deployment to dev/stage/prod environments (if you use it for CD)
Like most things, we can set it up with docker. Łukasz Pawłowski has an amazing post here about getting Jenkins setup on Ubuntu. Since I’m setting it up on macOS, I have to do things a bit differently.
If you go into the mlops-pipeline/jenkins directory, you should see these three files:
.envdocker-compose.yamlDockerfile
.env
docker-compose.yaml
Dockerfile
First, let’s create a place for Jenkins to store data.
mkdir ~/jenkins_home
Then, as we did earlier with Mlflow, we can use docker-compose up to start the server. While Pawłowski’s article would have us create additional users and grant them Jenkins permissions, I leave the user as root in the Jenkins container since I’m less concerned about security with these local services.
I also changed the permissions on /var/run/docker.sock so that anyone can run it (a super reckless software engineering move), including Jenkins when it’s mounted to the container:
chmod 777 /var/run/docker.sock
With Jenkins started up, you can access the UI at localhost:8180. You’ll have to copy the admin password Jenkins printed in the terminal (although it’s also written to ~/jenkins_home if you need it again). After some config steps, you can see the UI:
To create a build for our project, we’ll go to “New Item” -> “Freestyle Project.” While features like “Pipeline” are probably better for configuring large-scale production projects, our project is simple example and doesn’t need to adhere to those considerations.
Usually, people build from source code management tools like Github. We can provide our repo here. If it’s private, you’ll need credentials.
Make sure to delete the branch specifier and leave it empty. For professional projects, you’d want to configure Github hooks (idk Google it) so that pushes to master automatically trigger builds of our project.
Finally, we have a simple shell script that performs our operations when triggered.
If everything is successfully configured, Jenkins will build the image and update it in your local Docker registry! This is a dramatically oversimplified example, but you may also want to automate:
Unit and integration tests
Service/load tests
Image tagging/pushing
Automated deployment (CD)
If you want a real solution as opposed to the one I’ve outlined in this hacky blog, you can check out the Seldon docs.
One thing I had to consider when writing this is that machine learning is more artifact driven than it is code driven. Say you have an ML pipeline that runs daily and trains a new model that you want to deploy to production. There would be no code changes to your ML app, but the artifact will have changed. In that case, we’d want to trigger builds when new model.pkl files are created.
This can be accomplished with bucket notifications in most cloud storage offerings, including minio. To set this up, we can follow a combination of the instructions in this blog I found and the minio guide for establishing bucket notifications for webhooks.
In your Jenkins instance, you first need to install the generic webhook trigger plugin. I had a huge pain trying to install it locally, so I gave in and used Jenkins from the AWS marketplace instead. Then I ran this sketchy groovy script I found on stack overflow in the Jenkins Script Console:
You then need to restart the instance. Once enabled, you can configure the hook in your Jenkins build.
To test, go to http://JENKINS_URL/generic-webhook-trigger/invoke and you should see Jenkins kick off another build of your project! Now, we move onto minio.
First, install the minio client. I just used brew on mac.
brew install minio/stable/mc
Then, we can use the client to access our buckets (make sure you still have the minio/mlflow containers running).
mc alias set s3 http://localhost:9000mc ls s3/mlflow
Right on. Now we can use the client to set up your notifications. I still haven’t totally nailed this bit and it isn’t working completely... but I’m hoping to update soon. Let me know if you figure it out first.
mc admin config set s3 notify_webhook:1 queue_limit="0" queue_dir='.' endpoint="http://JENKINS_URL/generic-webhook-trigger/invoke"mc admin service restart s3mc event add s3/mlflow arn:minio:sqs::1:webhook --suffix .pkl
This next part, hypothetically, should trigger a build of my project but (like I said) I’m still running into issues over here. This is probably much easier to configure on AWS or GCP in a real setting, but the principle remains. You’d want to:
Configure Jenkins to build when new models are trained and pushed to S3 or GCSConfigure ml-app code so that it always pulls the latest modelBuild and test the model container with the new artifact so that it’s verified to escalate to production!
Configure Jenkins to build when new models are trained and pushed to S3 or GCS
Configure ml-app code so that it always pulls the latest model
Build and test the model container with the new artifact so that it’s verified to escalate to production!
Not my best work. I rarely bail on a half-baked idea, but when I tell you I’ve already lost too many hours on this one, I’m understating the amount of time I’ve spent here. So, moving on.
We have services to log our experiments and a pipeline to build an app that serves our model. Now we need to deploy it. A common paradigm is to scale workloads on kubernetes (k8s) so we’re gonna do that.
With Docker Desktop installed, you can simply enable k8s in preferences.
This will also set up other tools for you, like the k8s CLI, kubectl. Once started, you can open a terminal and interact with your “cluster” using the CLI. For instance, we can check what namespaces we have by running:
kubectl get ns
I won’t be expounding much about the different k8s resources and objects here, so feel free to investigate those separately.
Since we’re using Seldon, we also want to install the Seldon k8s components and Ambassador for ingress to our cluster. Per the Seldon documentation, we can use helm (installed with brew install helm on mac). Helm is a package manager for k8s.
helm repo add datawire https://www.getambassador.iokubectl create namespace ambassador && helm install ambassador --namespace ambassador datawire/ambassadorkubectl create ns seldon-systemhelm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set ambassador.enabled=true --set usageMetrics.enabled=true --namespace seldon-system
You can use kubectl to check on the status of your deployments for each namespace.
kubectl get deploy -n seldon-system
On kubernetes, deployments are declarative objects that manage functional units called pods. Pods are groups of one or more containers. In our case, we want to deploy our docker container to kubernetes using Seldon.
The other thing we need to install is argo CD. We could’ve done the deployment to k8s using Jenkins, but I really wanted to try this one out.
kubectl create ns argocdkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
This will create a bunch more resources.
... This article is getting way too long, huh.
Seldon comes with custom resource definitions (CRDs) for k8s objects. This means we can create Seldon-specific objects for our deployment very easily.
Here is an example SeldonDeployment YAML that I created for the image we’ve built locally:
To deploy this to kubernetes, run the following:
kubectl apply -f ml-app/deploy/base-deployment.yaml
You should see something like this:
seldondeployment.machinelearning.seldon.io/seldon-app created
Now, our Seldon app is running! If we have ambassador configured successfully, we should be able to make requests to our application.
It was at this point my 2015 Macbook really started to complain and I was quickly running out of CPU. If you need to, change the amount of resources allocated to Docker in your preferences.
Now, let’s do a port forward for ambassador. To get the pods:
kubectl get po -n ambassador
To forward that pod:
kubectl port-forward POD_NAME 8877 -n ambassador
Now, you can look view ambassador by going to http://localhost:8877/ambassador/v0/diag/. If successful, your deployment will be available at http://<ambassadorEndpoint>/seldon/<deploymentName>/api/v1.0/predictions
If you have trouble with the default ambassador configuration like I did, you can also forward the Seldon service directly to make requests to it:
kubectl port-forward -n seldon-system svc/seldon-app-default-app 9000:9000
That will forward the Seldon-created service to your local machine, which you can confirm by running curl localhost:9000/predict (which will give you a bad data error since you didn’t pass anything to predict).
To actually confirm with curl, here’s a command:
curl -X POST localhost:9000/predict -H 'Content-Type: application/json' -d '{"data":{"ndarray":[[3,1,1180,1180,7,1,0,5650,1,0,98178]]}}'
Note that if I had installed minio on kubernetes instead of separately with Docker, we could’ve followed Seldon’s guide on using a prepackage inference server and pulling from minio storage. This path is probably recommended since it doesn’t require baking a model into the docker image itself.
Now that our app works, we can configure argo to run deployments for us instead of kubectl apply -f. The goal here to always have our deployment in sync with the representation of our deployment (YAML).
We’re going to make it so we can access Argo via HTTP. In a new terminal, run:
kubectl port-forward -n argocd svc/argocd-server 8080:80
Wait until the pods come up. You can go to localhost:8080 to access Argo. By default, the user is “admin” and the password is the name of the server pod. You can get this by running: kubectl get po -n argocd | grep argocd-server
Once in the UI, we do something similar to Jenkins. We specify our git repo. Go to “+ New App” and configure as below:
Once you click CREATE, you can see the app creating and syncing! This uses our base-deployment.yaml file on Github and syncs the resources with that application definition. Changes to Github should result in application changes.
This shows that our single YAML file, base-deployment.yaml , generates a ton of resources through the help of the Seldon Controller. We can monitor the health of all these resources and easily sync YAML changes we make in Github to the actual state of our running application in kubernetes.
Plus, it looks pretty cool.
A real pipeline would be far more robust, automated, and require many checks and balances. I also omitted any kind of monitoring/observability which is a must for a production system. Given more time, I would’ve liked to iron those out — maybe even demo submitting ML jobs to kubernetes as part of the process. Honestly, though, I’m burnt out. This is it.
While I definitely prioritized breadth over depth here, hopefully this gives you a sense of all the possibilities for MLOps and I didn’t burn a couple weekends for no reason. I know the title included the word “simple,” but every component of any system comes with the risk of becoming a rabbit hole.
Per usual, hit me up on LinkedIn or something. Hopefully, I’ll have more focused MLOps content soon.