Running Apache Flink with RocksDB on Azure Kubernetes Service | by Hunter Kempf | Towards Data Science

Recently I was looking into how to deploy an Apache Flink cluster that uses RocksDB as the backend state and found a lack of detailed documentation on the subject. I was able to piece together how to deploy this from the Flink documentation and some stack overflow posts but there wasn’t a clear how-to guide anywhere. My goal with this post is to show a step by step guide for how to do this as of Flink 1.11.0. This guide can be followed with minimal modifications for AWS deployments. I will note the changes.
Before you start on this guide it is expected that you have an Azure Subscription with AKS and a Storage Account with a blob store running. If you need help getting that up and running see the links section at the bottom of this article with links to the Azure guides for how to set those services up.
It is also expected that you have flink, the azure-cli and docker downloaded see the bottom for links to those if you don't have them installed.
Using Azure Blob storage or AWS S3 storage for Flink backend state requires an extra plugin jar that is not built into the base Flink docker image. To add the needed jar you first need to download the apache/flink-docker repo from GitHub.
github.com
Once you have downloaded the repo navigate to the folder with the version of Flink you want to deploy. In my case, I want to use 1.11 then if you plan on using scala code, navigate to the directory that has the version of scala you are using. I use scala 2.12 so for me that folder is scala_2.12-debian. If you use java or python I believe you can use either version. Inside that folder you should see two files: Dockerfile and docker-entrypoint.sh. The file we will be editing is Dockerfile. Open that up in a text editor and add the following lines just above the COPY docker-entrypoint.sh line.
For Azure Blob Storage:
# install Flink Azure FS Hadoop pluginRUN mkdir ./plugins/azure-fs-hadoopCOPY ./flink-azure-fs-hadoop-1.11.0.jar ./plugins/azure-fs-hadoop/
For AWS S3 you have 2 options Presto and Hadoop:
Presto:
# install Flink S3 FS Presto pluginRUN mkdir ./plugins/s3-fs-prestoCOPY ./flink-s3-fs-presto-1.11.0.jar ./plugins/s3-fs-presto/
Hadoop:
# install Flink S3 FS Hadoop pluginRUN mkdir ./plugins/s3-fs-hadoopCOPY ./flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/
Next from the Flink 1.11.0 folder navigate to the /opt/ directory and copy the jar you need for your backend to the same directory as the Dockerfile you edited. For Azure that jar is flink-azure-fs-hadoop-1.11.0.jar
Now Build and Push the Docker Image to DockerHub using the following command replacing hunterkempf with your DockerHub username and changing the tag to reflect the version of flink, scala and the plugin you added to it:
docker build --tag hunterkempf/flink:1.11.0_scala_2.12_fs_azure .docker push hunterkempf/flink
ci.apache.org
First, go to the Flink Kubernetes setup page and create the following .yaml files on your computer using a text editor and copying/pasting from the Appendix.
flink-configuration-configmap.yaml
jobmanager-service.yaml
jobmanager-session-deployment.yaml
taskmanager-session-deployment.yaml
flink-configuration-configmap.yaml
You will need to add the following lines to the flink-conf.yaml section:
This example is for a blob store named: flinkblob in a storage account named: flinkstorage. You will also need to get the Access Key to your storage account and paste it after the fs.azure.account.key.flinkstorage.blob.core.windows.net:
state.backend: rocksdbstate.checkpoints.dir: wasbs://flinkblob@flinkstorage.blob.core.windows.net/flink/fs.azure.account.key.flinkstorage.blob.core.windows.net: 
jobmanager-session-deployment.yaml and taskmanager-session-deployment.yaml
Replace the image: flink:1.11.0-scala_2.11 with the name of the docker image you created in part 1 for both jobmanager-session-deployment and taskmanager-session-deployment.
Connect to the AKS Cluster
Run these two commands to connect to your AKS Cluster
az aks install-cliaz aks get-credentials --resource-group myResourceGroup --name myAKSCluster
Deploy Flink to AKS
From the directory that you created the yaml files run the following commands
kubectl create -f flink-configuration-configmap.yamlkubectl create -f jobmanager-service.yamlkubectl create -f jobmanager-session-deployment.yamlkubectl create -f taskmanager-session-deployment.yaml
Now you should have a Flink Cluster running on AKS using Azure Blob Storage as a RocksDB Backend.
View Flink Dashboard
To access the Flink Dashboard run the following commands:
kubectl get pods
this will return something similar to what is below
NAME                               READY   STATUS    RESTARTS   AGEflink-jobmanager-5f857bf45d-n4mbt    1/1     Running   0        38sflink-taskmanager-68854b7998-2zkd6   1/1     Running   0        31sflink-taskmanager-68854b7998-vvm5m   1/1     Running   0        31s
copy the name of the flink-jobmanager pod and use that for the following command replacing the 5f857bf45d-n4mbtwith your value
kubectl port-forward flink-jobmanager-5f857bf45d-n4mbt 8081:8081
Now open up http://localhost:8081 and you should see the Flink Dashboard
To check your configurations click on the Job Manager tab on the left
you should see something similar with your settings listed out in key value notation.
To test the Deployment you can upload one of the test jar’s from the Flink 1.11.0 folder located in the examples/streaming/ folder
Running the Streaming WordCount job took 1 second and returned no errors for me. If you get an Internal Server Error check the Job Manager Logs to see the error stack trace.
Here are some useful links for further reading