[TUTORIAL]: Deploy ML model as REST API on Azure and protect it with SSL | by Ivan Zidov | Towards Data Science

Data Scientists often forget that their new awesome models aren’t so useful if they can't be used in production. In this tutorial, I will show you how you can deploy Tensorflow model to Azure Container Instances and protect it with SSL.
There will be 3 types of deployment:
Deployment #1: Local deployment:Model will be put in a docker container and exposed as an API on localhost.
Deployment #2: Global deployment without encryption:Docker containing model will be uploaded to Azure Container Registry. The stored container will be exposed thought an API that you can call. The downside to that is that everyone can access it if they have the IP address and it is not encrypted.
Deployment #3: Global deployment encrypted with SSL:Docker containing model will be uploaded to Azure Container Registry. We will create Certificate Authority (CA) and self-signed certificates for API usage.
Linux machine with sudo options
Docker installed on Linux
Azure CLI installed on Linux
openssl installed on Linux
Valid subscription for Azure
NOTE: If you don’t have valid subscription, you can create a free account for Azure and receive 2 weeks of Trial period.
Train Tensorflow model and save it with:tf.saved_model.save(model, "path / to / model")
OR:
Here is a demo model that represents f(x)= 2*x -1. It was saved to “./demo_model/”
import os import numpy as np import tensorflow as tf  xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float) ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)  model = tf.keras.Sequential([tf.keras.layers.Dense(units=1, input_shape=[1])]) model.compile(optimizer='sgd', loss='mean_squared_error') history = model.fit(xs, ys, epochs=500, verbose=0) print("Finished training the model")  MODEL_DIR = "./demo_model" version = 1 export_path = os.path.join(MODEL_DIR, str(version)) model.save(export_path, save_format="tf") print('\nexport_path = {}'.format(export_path))
I won’t go into the details of Tensorflow Serving because there is a great Coursera course. This code is used above is code used from this course, but only compressed for this demo. You can learn more about TFServing from here.
Open terminal and navigate to the directory where you have saved the model.
$ MODEL_NAME=demo_model #REPLACE with your model$ MODEL_PATH="./$MODEL_NAME"$ CONTAINER_NAME="$MODEL_NAME"$ sudo docker run -d --name serving_base tensorflow/serving$ sudo docker cp "$MODEL_PATH" serving_base:"/models/$MODEL_NAME"$ sudo docker commit --change "ENV MODEL_NAME $MODEL_NAME" serving_base "$CONTAINER_NAME"$ sudo docker run -d -p 8501:8501 "$MODEL_NAME"
Now you have your model exposed thought an API on localhost, and you can be call it from terminal:
$ curl -d '{"signature_name": "serving_default", "instances": [[10.0],[9.0]]}' -X POST "http://localhost:8501/v1/models/$MODEL_NAME:predict"
Or from python:
import requests import json api_url = "http://localhost:8501/v1/models/demo_model:predict" data = json.dumps({"signature_name": "serving_default", "instances": [[10.0],[9.0]]}) headers = {"content-type": "application/json"} json_response = requests.post(api_url, data=data, headers=headers) print(json_response.text)
#STOP docker imagesudo docker container stop IMAGE_ID  #DELETE docker image sudo docker container rm IMAGE_ID
Deployment #1: COMPLETED!
You can now use your model in localhost.
BONUS TIP: Another option is port forwarding. You connect to the machine with SSH.
ssh -N -f -L localhost:8501:localhost:8501 user@IP.ADDRESS
DOWNSIDES:
You can only use it locally
You need a VPN access to use port forwarding
Not scalable
Used to store containers with models.Go to Azure portal and find Container Registries. Click on Add new registry button to create a new registry.
IMPORTANT:
Pick a valid Subscription and Resource group.
Registry name has to be unique, and we will also be needing it later so take note of it. (TIP: use lowercase only)
Location is mandatory and you should pick the one closest to you
Basic SKU should be enough for most cases.
You can now click on Review and Create this registry button.
IMPORTANT: Navigate to your resource and enable admin user. Take a note of username and password.
$ CONTAINER_REGISTRY_NAME=blogdemo #REPLACE (lower letters only)$ sudo az login #login to Azure$ sudo az acr login --name "$CONTAINER_REGISTRY_NAME" $ sudo docker images
$ VERSION_NAME="$CONTAINER_REGISTRY_NAME.azurecr.io/version1"$ sudo docker tag 2b458f67dac3 "$VERSION_NAME" #REPLACE$ sudo docker push "$VERSION_NAME"
Congrats:You have successfully stored the container with the model
After storing the container, it is fairly easy to expose the model.
The easiest way is to log in to Azure portal, navigate to Container Instances and click Add button. After naming the container, pick a desired region and choose the registry where you have uploaded the model.
NOTE:Size really depends on your model size. The size used in this example is more than enough for this model. I have deployed the model with ~150M parameters (600MB) and it needed 2 vcpu and 4 GB of memory.
Now click on Next: Networking button.
NOTE: DNS name is optional and it needs to be unique.
Click on Review and Create and your model is ready to be used. (To delete an instance just navigate to resource and click on Delete button)
USAGE:
$ curl -d '{"signature_name": "serving_default", "instances": [[10.0],[9.0]]}' -X POST "http://blogdemo.westeurope.azurecontainer.io:8501/v1/models/$MODEL_NAME:predict" #REPLACE blogdemo.westeurope.azurecontainer.io WITH DNS#OR:$ curl -d '{"signature_name": "serving_default", "instances": [[10.0],[9.0]]}' -X POST "http://IP.ADDRESS:8501/v1/models/$MODEL_NAME:predict" #REPLACE IP.ADDRESS WITH IP
Using the model is the same as using the one exposed locally, only difference being the URL. You can use a DNS URL or an IP URL.
Deployment #2: COMPLETED!
You have deployed a model that can be used over the Internet. You can monitor usage of model and upgrade the size of the container if needed, which makes it scalable.
DOWNSIDES:
Everyone who knows the IP-ADDRESS can access the model
Not encrypted
To protect API, we will need encryption. If you use commercial Certificate Authority, you can skip this step. We will create our own and self-sign certificates.
$ mkdir certs $ cd certs $ openssl ecparam -genkey -name secp256r1 | openssl ec -out ca.key $ openssl req -new -x509 -days 3650 -key ca.key -out ca.pem
You will be prompted to fill-out the form. Fill it out with your information.
NOTE:This is an example of elliptic encryption for simplicity. I recommend using an RSA key. This is a great gist that explains how to do it.
$ CLIENT_ID=”client”$ CLIENT_SERIAL=01 #when creating new user make sure that serial is unique$ openssl ecparam -genkey -name secp256r1 | openssl ec -out “${CLIENT_ID}.key”$ openssl req -new -key “${CLIENT_ID}.key” -out “${CLIENT_ID}.csr” #password should be empty$ openssl x509 -req -days 3650 -in “${CLIENT_ID}.csr” -CA ca.pem -CAkey ca.key -set_serial “${CLIENT_SERIAL}” -out “${CLIENT_ID}.pem”$ cat “${CLIENT_ID}.key” “${CLIENT_ID}.pem” ca.pem > “${CLIENT_ID}.full.pem”#OPTIONAL:$ openssl pkcs12 -export -out “${CLIENT_ID}.full.pfx” -inkey “${CLIENT_ID}.key” -in “${CLIENT_ID}.pem” -certfile ca.pem#remember passoword and you will pass it with pfx file
Why do we need NGINX?
To reject all requests that don’t have a valid certificate.
Create nginx.config file as follows:
$ cd ..$ cat > nginx.conf
Copy everything and paste it to a file. After pasting use CTRL+D to close editor.
user nginx;worker_processes auto;events {  worker_connections 1024;}pid        /var/run/nginx.pid;http {server {        listen [::]:443 ssl;        listen 443 ssl;server_name localhost;ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;ssl_ciphers                ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK;        ssl_prefer_server_ciphers  on;ssl_session_cache    shared:SSL:10m; # a 1mb cache can hold about 4000 sessions, so we can hold 40000 sessions        ssl_session_timeout  24h;keepalive_timeout 300; # up from 75 secs defaultadd_header Strict-Transport-Security 'max-age=31536000; includeSubDomains';ssl_certificate      /etc/nginx/server.pem;   ssl_certificate_key /etc/nginx/server.key;   ssl_client_certificate /etc/nginx/ca.pem;           ssl_verify_client on;                location / {            proxy_pass http://localhost:8501; # TODO: replace with correct port                        proxy_set_header Connection "";            proxy_set_header Host $host;            proxy_set_header X-Real-IP $remote_addr;            proxy_set_header X-Forwarded-For $remote_addr;        }    }}
I won’t go into the details of NGINX, but here is the link for the details. This configuration is a mixture of this blog and this article. In a nutshell, it only listens to https (port 443) and validates certificate that it receives. If a certificate is valid, it redirects decrypted request to container with model and encrypts response that is sent back to client.
Create test-deploy.yaml file for Azure:
$ cat > test-deploy.yaml
Copy everything and paste it to a file. After pasting use CTRL+D to close editor.
location: LOCATIONname: NAMEproperties:  containers:  - name: model    properties:      image: IMAGE.azurecr.io/version1:latest      ports:      - port: 8501        protocol: TCP      resources:        requests:          cpu: 1.0          memoryInGB: 1.5  - name: nginx-with-ssl    properties:      image: nginx      ports:      - port: 443        protocol: TCP      resources:        requests:          cpu: 1.0          memoryInGB: 1.5      volumeMounts:      - name: nginx-config        mountPath: /etc/nginx  imageRegistryCredentials:  - server: SERVER.azurecr.io    username: USERNAME    password: PASSWORD  volumes:  - secret:      server.pem: PEM      server.key: KEY      ca.pem: CA      nginx.conf: CONF    name: nginx-config  ipAddress:    ports:    - port: 443      protocol: TCP    type: Public    dnsNameLabel: DNS  osType: Linuxtags: nulltype: Microsoft.ContainerInstance/containerGroups
We will use the sed command to replace parts of this template. These are the commands that you can run without editing.
$ sed -i -e "s/name: NAME/name: $MODEL_NAME/" azure.yaml$ sed -i -e "s/image: IMAGE/image: $CONTAINER_REGISTRY_NAME/" azure.yaml$ sed -i -e "s/server: SERVER/server: $CONTAINER_REGISTRY_NAME/" azure.yaml$ sed -i -e "s/username: USERNAME/username: $CONTAINER_REGISTRY_NAME/" azure.yaml$ sed -i -e "s/server.pem: PEM/server.pem: $(cat ./certs/ca.pem | base64 -w 0)/" azure.yaml$ sed -i -e "s/server.key: KEY/server.key: $(cat ./certs/ca.key| base64 -w 0)/" azure.yaml$ sed -i -e "s/ca.pem: CA/ca.pem: $(cat ./certs/ca.pem | base64 -w 0)/" azure.yaml$ sed -i -e "s/nginx.conf: CONF/nginx.conf: $(cat ./nginx.conf | base64 -w 0)/" azure.yaml
These commands have to be edited. Replace the bolded parts with location and password from the registry and use your DNS to replace DNS_NAME.
$ sed -i -e "s/location: LOCATION/location: westeurope/" azure.yaml$ sed -i -e "s/password: PASSWORD/password: REGISTRY_PASS/" azure.yaml#TIP: If your generated password has some special characters#like / you will have to manually put \/ infront$ sed -i -e "s/dnsNameLabel: DNS/dnsNameLabel: DNS_NAME/" azure.yaml
Replace <AZURE RESOURCE GROUP> with your valid resource group and you are ready to go.
$ az container create --resource-group <AZURE RESOURCE GROUP> --name "$MODEL_NAME" -f azure.yaml
Make sure that you have a proper path to client.key and client.pem and you can proceed to call the API like this:
$ curl -d '{"signature_name": "serving_default", "instances": [[10.0],[9.0]]}' -v --key "./certs/client.key" --cert "./certs/client.full.pem" -X POST -k https://blogdemo.westeurope.azurecontainer.io/v1/models/demo_model:predict#REPLACE blogdemo.westeurope.azurecontainer.io WITH DNS#OR:$ curl -d '{"signature_name": "serving_default", "instances": [[10.0],[9.0]]}' -v --key "./certs/client.key" --cert "./certs/client.full.pem" -X POST -k https://IP.ADDRESS/v1/models/demo_model:predict#REPLACE IP.ADDRESS WITH IP
If everything works fine you will get a response like this one:
IMPORTANT: Since this is a self-signed certificate, you must add -k flag in cURL.
Python version:
import requests import jsonapi_url = "https://blogdemo.westeurope.azurecontainer.io/v1/models/demo_model:predict"data = json.dumps({"signature_name": "serving_default", "instances": [[10.0],[9.0]]})headers = {"content-type": "application/json"}json_response = requests.post(api_url, data=data, headers=headers,verify=False,cert=("./certs/client.pem","./certs/client.key"),)print(json_response.text)
IMPORTANT: Since this is a self-signed certificate, you must set verify=False
When called without a valid certificate:
Deployment #3: COMPLETED!
You can monitor the usage of the model in Container Instance and you can see request made in nginx logs.
Q: Can I deploy a custom model (without TF Serving)?A: Yes. You have to dockerize the model and if you use the right ports, everything else should work as expected.
Q: Can I deploy it to AWS?A: Yes, by using nginx.config and CA configuration seen in this blog.
Q: Can I store the container on Docker Hub?A: Yes. Just replace image with username/repo:version from Docker Hub and remove imageRegistryCredentials from YAML file.