Python data automation in Azure ecosystem by connecting JupyterHub on a VM, Blob Storage and PowerBI dashboard
Tags: Azure, Python, Power Bi, Blob Storage, Jupyter Notebook

Python data automation in Azure ecosystem by connecting JupyterHub on a VM, Blob Storage and PowerBI dashboard

For any Data professional, automating data is key. Once we set up the process of data capture, data analysis, data storage and data visualisation, we want to be able to schedule periodic refreshes of the resulting dashboard to prevent the need for any manual intervention.

This tutorial was created based on a real world need to architect an automated data flow. Essentially, external data needs to be captured, analyzed and published to a PowerBI dashboard on a daily basis. It can be described as follows:

Access external data (public API’s or websites) with Python script running on Azure VM. Virtual Machine needs to run completely in the cloud and have the ability to be scheduled via cronjobs. Clean, analyze, run ML algorithms and output data in a simple format for consumption into PowerBI. VM that is running a python script / JupyterHub notebook needs to ingest external data on a regular basis. Machine learning and analysis is then executed on this data set. The output of this step is a cleaned data set with business insights that need to flow into PowerBI. Yes, PowerBI has the capability to do data cleaning, processing and wrangling — yet for many analysts, it is inferior to the power of Python. Adding this layer of Python analysis gives the analyst much more power and flexibility for data analysis/machine learning. Store data as an intermediary and connect to PowerBI. The output of the above step is then uploaded to a Blob storage container in Azure, which is then connected to PowerBI for use in a dashboard. Publish PowerBI dashboard and set up automatic refresh schedule. The dashboard is published to a public report which is consumed within Teams by business stakeholders.

The following steps outline the process of building this data flow.

Step 1: Create the Virtual Machine in Azure

Set up VM. Click on ‘Virtual Machines’ section and click ‘Add’.

Specify the VM settings. Azure conveniently has preset configurations for common use cases, allowing the user to specify memory requirements, specs, etc.

Use SSH keys to connect to VM. Mac and Linux users can connect using the terminal, otherwise use PuTTY if accessing through Windows.

Step 2: Install JupyterHub on the Virtual Machine

The Littlest JupyterHub is a simple distribution for less than 100 users. It is ideal for this type of project.

After SSH’ing into the VM, enter the following code to ensure that python3 is installed:

sudo apt install python3 python3-dev git curl

Install the JupyterHub and create your own admin username:

curl -L https://tljh.jupyter.org/bootstrap.py | sudo -E python3 - --admin <admin-user-name>

It should take 5–10 minutes to complete the install.

Be sure to set up SSH tunneling — this will allow you to securely connect to the specific port on your server that is running the notebook.

Add the JupyterHub port (usually 8888) as an ‘inbound port rule’ in the Azure VM. This was a tricky step to figure out — contact your network admin if you don’t have proper privileges to adjust these settings.

Return to the VM command line and activate the virtual environment, then launch the notebook:

source /opt/tljh/user/bin/activate jupyter notebook

You should receive a message indicating the location of the server:

The Jupyter Notebook is running at: http://localhost:8888/?token=[ unique_token_id]

The first time you navigate to the URL, input the unique token and the notebook should be up and running.

Note: While writing this article, I discovered this detailed guide from Jupyter.org on how to set up a new VM in Azure specifically for JupyterHub. If anyone has used this implementation, let me know how it worked out in the comments.

Step 3: Upload data into Blob storage through Python

For this tutorial, we are using Azure Blob storage as the intermediary to get our data to flow into PowerBI. The first step is to create the Blob storage that we will be using for this project. Be sure to select ‘Blob’ under ‘Public access level’.