MLflow Part 2: Deploying a Tracking Server to Minikube! | by David Hundley | Towards Data Science

10/15/20 Update: In writing my next post in this series, I found several bugs that prevented me from appropriately deploying to Minikube. To that end, I’ve updated a number of things to get you up and going with a WORKING instance! 😃
Welcome back, friends! We’re back with our continued mini-series on MLflow. In case you missed out part one, be sure to check it out here. The first post was a super basic introduction to log basic parameters, metrics, and artifacts with MLflow. That was just having us log those items to a spot on our local machine, which is not an ideal practice. In a company context, you ideally want to have all those things logged to a central, reusable location. That’s we’ll be tackling in today’s post! And of course, you can find all my code on GitHub at this link.
So to be clear, we’re going to be covering some advanced topics that require a bit of foreknowledge about Docker and Kubernetes. I personally plan to write posts on those at a later date, but for now, I’d recommend the following resources if you want to get a quick start on working with Docker and Kubernetes:
Docker 101 Tutorial
Learn Kubernetes Basics
Now if you know Kubernetes, chances are that you are familiar with Minikube, but in case you aren’t, Minikube is basically a small VM you can run on your local machine to start a sandbox environment to test out Kubernetes concepts. Once Minikube is up and running, it’ll look very familiar to those of you who have worked in legit Kubernetes environments. The instructions to set up Minikube are nicely documented in this page, BUT in order to get Minikube working, we need to get a couple additional things added later on down this post.
Before going further, I think a picture is worth a thousand words, so below is a tiny picture of the architecture we’ll be building here.
Alrighty, so on the right there we have our Minikube environment. Again, Minikube is highly representative of a legit Kubernetes environment, so the pieces inside Minikube are all things we’d see in any Kubernetes workspace. As such, we can see that MLflow’s tracking server is deployed inside a Deployment. That Deployment interacts with the outside world by connecting a service to an ingress (which is why the ingress spans both the inside and outside in our picture), and then we can view the tracking server interface inside our web browser. Simple enough, right?
Okay, so step 1 is going to be to create a Docker image that builds the MLflow tracking server. This is really simple, and I personally have uploaded my public image in case you want to skip this first step. (Here is that image in my personal Docker Hub.) The Dockerfile is simply going to build on top of a basic Python image, install MLflow, and set the proper entrypoint command. That looks like this:
# Defining base imageFROM python:3.8.2-slim# Installing packages from PyPiRUN pip install mlflow[extras]==1.9.1 && \    pip install psycopg2-binary==2.8.5 && \    pip install boto3==1.15.16# Defining start up commandEXPOSE 5000ENTRYPOINT ["mlflow", "server"]
You know the drill from here: build and push out to Docker Hub! (Or just use mine.)
Where our 10–15–20 update begins!
Okay, in the previous iteration of this post, I attempted to use a simple PVC for storage of the metadata and artifacts. Turns out that it is not so easy. Instead, we’re going to have to do a little extra legwork to get this going on Minikube. To that end, we’re going to configure a Postgres store for the backend metadata and an object store called Minio for our artifacts. (More on Minio below in case you haven’t heard of it.) If both of these things sound daunting to you, that’s okay! You can simply use my code to get up and going.
Alrighty, so let’s tackle the Postgres deployment. Here is the K8s manifest code for that:
apiVersion: v1kind: ConfigMapmetadata:  name: mlflow-postgres-config  labels:    app: mlflow-postgresdata:  POSTGRES_DB: mlflow_db  POSTGRES_USER: mlflow_user  POSTGRES_PASSWORD: mlflow_pwd  PGDATA: /var/lib/postgresql/mlflow/data---apiVersion: apps/v1kind: StatefulSetmetadata:  name: mlflow-postgres  labels:    app: mlflow-postgresspec:  selector:    matchLabels:      app: mlflow-postgres  serviceName: "mlflow-postgres-service"  replicas: 1  template:    metadata:      labels:        app: mlflow-postgres    spec:      containers:      - name: mlflow-postgres        image: postgres:11        ports:        - containerPort: 5432          protocol: TCP        envFrom:        - configMapRef:            name: mlflow-postgres-config        resources:          requests:            memory: "1Gi"            cpu: "500m"        volumeMounts:        - name: mlflow-pvc          mountPath: /var/lib/postgresql/mlflow  volumeClaimTemplates:  - metadata:      name: mlflow-pvc    spec:      accessModes: [ "ReadWriteOnce" ]      resources:        requests:          storage: 100Mi---apiVersion: v1kind: Servicemetadata:  name: mlflow-postgres-service  labels:    svc: mlflow-postgres-servicespec:  type: NodePort  ports:  - port: 5432    targetPort: 5432    protocol: TCP  selector:    app: mlflow-postgres
So I’m not going to go line by line with everything, but at a thousand foot level, this is going to spin up a Postgres instance with 100Mi of storage and the appropriate config information defined at the top of the code. You can change those variables if you’d like. Remember, we’re just learning here, so these variables are obviously exposed. In the real world, this is a HUGE security concern, so don’t follow my lead here if you’re going to use this for a legit deployment.
Alright, with that deployed, we’re ready to tackle our object store: Minio. Now if you’re totally new to Minio like me, it basically is an object store you can deploy to K8s that basically emulates Amazon Web Service’s (AWS’s) S3 service. The deployment syntax for that looks like this:
apiVersion: apps/v1kind: Deploymentmetadata:  name: mlflow-miniospec:  selector:    matchLabels:      app: mlflow-minio  template:    metadata:      labels:        app: mlflow-minio    spec:      volumes:      - name: mlflow-pvc        persistentVolumeClaim:          claimName: mlflow-pvc      containers:      - name: mlflow-minio        image: minio/minio:latest        args:        - server        - /data        volumeMounts:        - name: mlflow-pvc          mountPath: '/data'        env:        - name: MINIO_ACCESS_KEY          value: "minio"        - name: MINIO_SECRET_KEY          value: "minio123"        ports:        - containerPort: 9000---apiVersion: v1kind: Servicemetadata:  name: mlflow-minio-servicespec:  type: NodePort  ports:  - port: 9000    targetPort: 9000    protocol: TCP  selector:    app: mlflow-minio---apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata:  name: mlflow-minio-ingress  annotations:    kubernetes.io/ingress.class: nginx    nginx.ingress.kubernetes.il/add-base-url: "true"    nginx.ingress.kubernetes.io/ssl-redirect: "false"spec:  rules:  - host: mlflow-minio.local    http:      paths:        - backend:            serviceName: mlflow-minio-service            servicePort: 9000          path: /---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: mlflow-pvcspec:  accessModes:    - ReadWriteMany  resources:    requests:      storage: 100Mi
So again at a high level, we’re deploying a Minio object store backed by a PVC with 100Mi of data. You can also see in the environment variables of the deployment that we define the access key ID and secret access key. These very much correlate to AWS’s AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Now before moving onto the next step, you’ll have to properly configure your machine’s ingress, and in my original post, I share how to do that down below. (I’m too lazy to re-type that here, so do a CTRL+F for ingress and I’m sure you’ll find it!)
Alrighty, if you configured your ingress properly and navigate on over to mlflow-minio.local in a browser, you should be greeted with this splash screen.
In the respective fields, type in the access key and secret key we defined in the Minio deployment. (If you kept it the same as me, those respectively are “minio” and “minio123”.) Hit enter to be greeted with this next screen.
Okay, so in my case, I already created the “bucket” we’ll be using to store our artifacts. For you to do it, it’s as simple as clicking the orangish plus sign in the bottom right of the UI, selecting “Create New Bucket”, and naming your new bucket “mlflow”.
Phew! Alright, we have our backend stuff set up! Time to get the actual server itself rolling!
I’m primarily going to stick to the Deployment manifest here. Most of this syntax will look pretty familiar to you. The only thing to be mindful of here are the arguments we’ll pass to our building Docker image. Let me show you what my Deployment manifest looks like first.
# Creating MLflow deploymentapiVersion: apps/v1kind: Deploymentmetadata:  name: mlflow-deploymentspec:  replicas: 1  selector:    matchLabels:      app: mlflow-deployment  template:    metadata:      labels:        app: mlflow-deployment    spec:      containers:      - name: mlflow-deployment        image: dkhundley/mlflow-server:1.0.3        imagePullPolicy: Always        args:        - --host=0.0.0.0        - --port=5000        - --backend-store-uri=postgresql://mlflow_user:mlflow_pwd@10.98.160.170:5432/mlflow_db        - --default-artifact-root=s3://mlflow/        - --workers=2        env:        - name: MLFLOW_S3_ENDPOINT_URL          value: http://mlflow-minio.local/        - name: AWS_ACCESS_KEY_ID          value: "minio"        - name: AWS_SECRET_ACCESS_KEY          value: "minio123"        ports:        - name: http          containerPort: 5000          protocol: TCP---apiVersion: v1kind: Servicemetadata:  name: mlflow-servicespec:  type: NodePort  ports:    - port: 5000      targetPort: 5000      protocol: TCP      name: http  selector:    app: mlflow-deployment---apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata:  name: mlflow-ingress  annotations:    kubernetes.io/ingress.class: nginx    nginx.ingress.kubernetes.il/add-base-url: "true"spec:  rules:  - host: mlflow-server.local    http:      paths:        - backend:            serviceName: mlflow-service            servicePort: 5000          path: /
A few key things to point out in here. First, the Postgres instance is referenced in the “args” with its IP address. I obtained that IP address by running the following command:
kubectl get services
That will give you this screen:
You’ll notice that the CLUSTER-IP for the mlflow-postgres-service directly correlates to what is in my server deployment manifest. You’ll need to update your IP with whatever your service shows as it will likely not be the same as me. (And truth be told... I feel like there’s a programatic way to do this, but I honestly don’t know how to do that.) Notice also how we reference Minio as the backend server. It might look weird to you that we’re indeed using AWS-like environment variables, but hey, that’s just how it works!
Okay, so now that we have everything successfully deployed, it’s time to get our Minikube ingress working. You probably won’t have this issue if you’re working in a legit Kubernetes environment, but Minikube can be a bit tricky here. Honestly, this last part took me several days to figure out, so I’m glad to finally pass this knowledge along to you!
Let’s take a glance at the ingress YAML again:
# Creating the Minikube ingressapiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata:  name: mlflow-ingress  annotations:    kubernetes.io/ingress.class: nginx    nginx.ingress.kubernetes.il/add-base-url: "true"spec:  rules:  - host: mlflow-server.local    http:      paths:        - backend:            serviceName: mlflow-service            servicePort: 5000          path: /
Most of this should be familiar to you. In our example here, we’ll be serving out the MLflow tracking server’s UI at mlflow-server.local. One thing that might be new to you are those annotations, and they are absolutely necessary. Without them, your ingress will not work properly. I specifically posted the image below to Twitter to try getting folks to help me out with my blank screen issue. It was quite frustrating.
Bleh, talk about a mess! After much trial and error, I finally figured out that the specific annotation configuration provided above worked. I honestly can’t tell you why though.  ̄\_(ツ)_/ ̄
But wait, there’s more! By default, Minikube isn’t set up to handle ingress right out of the box. In order to do that, you’ll need to do a few things. First up, after your Minikube server is running, run the following command:
minikube addons enable ingress
Easy enough. Now, you need to set up your computer to reference the Minikube cluster’s IP through the mlflow-server.local host we’ve set up in the ingress. To get your Minikube’s IP address, simply run this command:
minikube ip
Copy that to your clipboard. Now, this next part might be totally new to you. (At least, it was to me!) Just like you can create alias commands for Linux, you can also apparently create alias ties from IP addresses to web addresses. It’s very interesting because I learned that this is the place where your browser translates “localhost” to your local IP address.
To navigate to where you need to do that, run the following command:
sudo nano /etc/hosts
You should be greeted with a screen that looks like this:
So you can see here at the top what I was just referencing with the localhost thing. With this interface open, paste in your Minikube’s IP address (which is 192.168.64.4 in my case) followed by BOTH host names for the MLflow server and Minio artifact store, which are respectively mlflow-server.local and mlflow-minio.local in our case.
Alright, if you did everything properly, you should be pretty much all set! Navigate on over to your browser of choice and open up http://mlflow-server.local. If all goes well, you should see a familiar looking screen.
That’s it for this post, folks! I don’t want to overload you all with too much, so in our next post, we’ll take off from here by logging a practice model or two to this shared tracking server just to see that it’s working. And in two posts from now, we’ll keep the ball rolling even further by showing how to deploy models for usage out from this tracking server. So to be honest, the content of this post might not have been that glamorous, but we’re laying down the train tracks that are going to make everything really fly in the next couple posts.
Until then, thanks for reading this post! Be sure to check out my former ones on other data science-related topics, and we’ll see you next week for more MLflow content!