A Python AzureML SDK workflow with VSCode devcontainers | by Luuk van der Velden | Towards Data Science

Microsoft is making great strides in conquering our hearts and mingling with the open source Ecosystem. The same ecosystem they openly worked against in the 00s. I confess I am in the process of being assimilated into the Microsoft world and am feeling good about it. In this blog I describe a development workflow for launching ML models in the cloud (AzureML) and developing code in VSCode remotely.
See also: Part 2: Unifying remote and local AzureML environments
Being fairly late to VSCode it took me by surprise after my usual period of denial. The remote development functionality introduced in Q2 2019 (vscode-dev-blog) really sealed the deal for me. There are three flavors to this:
Deploy the VSCode server within the Windows subsystem for Linux (WSL) and run the client on the Windows desktopDeploy the VSCode server within a Docker container and run the client on the Windows desktopDeploy the VSCode server to an SSH target and run the client locally
Deploy the VSCode server within the Windows subsystem for Linux (WSL) and run the client on the Windows desktop
Deploy the VSCode server within a Docker container and run the client on the Windows desktop
Deploy the VSCode server to an SSH target and run the client locally
This is very rich functionality which does not disappoint in reliability. Let us take a closer look.
Windows subsystem for Linux (WSL) has been around for a number of years. WSL2 unifies running Linux based docker containers and WSL Linux on the same architecture (docker-wsl2-backend). Importantly it removed complex virtualization from the architecture and WSL Linux runs pretty much natively. VSCode can work on code on the windows filesystem and run its server on WSL; integrating seamlessly with Python interpreters and shells.
In this blog we suggest running the VSCode server inside a docker container on the WSL2 backend with the GUI running on the Windows desktop. WSL2 is the future and has benefits, such as Windows Home support (docker-windows-home), resource management and the potential for GPU support (WSL2-dev-blog). What we discuss also works on the historic Docker backend.
Microsoft provides well maintained Dockerfiles containing common development tools for language and language agnostic scenarios (vscode-dev-containers). We selected the azure-machine-learning-python-3 container (azureml-dev-container), because of its Conda Python interpreter with the azureml-sdk package installed. The image derives from Ubuntu. If we include the “./.devcontainer” folder in any of our projects VSCode will:
detect the presence of “./.devcontainer” and its contentsbuild a container from the “./.devcontainer/Dockerfile`install the VSCode server and optional extensions defined in “decontainer.json”run the container as configured in “devcontainer.json”; defining a.o. bind mounts and startup scripts
detect the presence of “./.devcontainer” and its contents
build a container from the “./.devcontainer/Dockerfile`
install the VSCode server and optional extensions defined in “decontainer.json”
run the container as configured in “devcontainer.json”; defining a.o. bind mounts and startup scripts
The settings for VSCode container integration are stored in “./.devcontainer/devcontainer.json” and allow any Dockerfile to be integrated without alteration.
We can add our favorite development tools to these devcontainers as additional layers in the Dockerfile. Or we can build on top of a base image by calling “FROM” in our custom “Dockerfile”. For instance, adding the Azure CLI with extensions and Terraform with “RUN” commands (my-branch). An image with your favorite development environment can be a good companion to have. We can manage our custom image in one place and provide symbolic links to it from each of our projects.
Install docker desktop on windowsInstall git for windows (I suggest adding it to powershell during install)Install VSCode and its “Remote — Containers” extensionChange directory to a suitable dev directory (here “dev”) on the windows filesystemgit clone https://github.com/microsoft/vscode-dev-containers/tree/master/containers/azure-machine-learning-python-3within an existing Python project symlink the .devcontainer in Powershell using New-Item -ItemType Junction -Path “c:\dev\path\to\your\project\.devcontainer\” -Target “c:\dev\vscode-dev-containers\containers\azure-machine-learning-python-3\.devcontainer\”Within the project base folder launch vscode with “code .”VSCode detects the “.devcontainer” and asks to re-open in container -> yesVSCode builds the container and installs its server and re-opensConfirm your environment in the lower left corner, a green square says “Dev Container: Azure Machine Learning”
Install docker desktop on windows
Install git for windows (I suggest adding it to powershell during install)
Install VSCode and its “Remote — Containers” extension
Change directory to a suitable dev directory (here “dev”) on the windows filesystem
git clone https://github.com/microsoft/vscode-dev-containers/tree/master/containers/azure-machine-learning-python-3
within an existing Python project symlink the .devcontainer in Powershell using New-Item -ItemType Junction -Path “c:\dev\path\to\your\project\.devcontainer\” -Target “c:\dev\vscode-dev-containers\containers\azure-machine-learning-python-3\.devcontainer\”
Within the project base folder launch vscode with “code .”
VSCode detects the “.devcontainer” and asks to re-open in container -> yes
VSCode builds the container and installs its server and re-opens
Confirm your environment in the lower left corner, a green square says “Dev Container: Azure Machine Learning”
We can develop using fully reproducible Linux tool chains and still work on a windows laptop. This is probably not convincing to a Linux or MacOS user, but if you become comfortable with these tools you will have them available almost anywhere. This is valuable if you work for companies that provide you with the infrastructure and a laptop. Another scenario would be training deep learning models with GPUs with your gaming desktop windows computer, as GPU support for WSL2 has been announced (WSL2-dev-blog).
A prerequisite for this section is an active Azure subscription. You can easily get a free azure account with starting credits, if you can get hold of a credit card.
Our VSCode runs within a devcontainer featuring Python and the AzureML Python SDK. The SDK contains an elaborate API spanning the entire lifecycle of an ML model. From exploration in notebooks, training and tracking the model to artifact management and deployment. This includes the provisioning and configuration of the compute for model training and the deployment targets for model scoring. Some of the tools (Azure CLI) I will reference were added in my own fork (my-branch).
The following diagram shows the workflow we are aiming at. We launch an experiment from VSCode using the AzureML SDK inside our devcontainer and the SDK executes our code on the compute target. The compute target runs our code within a docker container as well. The specifics of the runtime environment are discussed in part 2/2 of this blog.
Open a terminal in VSCode: menu Terminal -> New TerminalExecute “az login” on the shell. Go to www.microsoft.com/devicelogin and fill in the secret provided in the console outputWe have now linked our environment with an Azure subscriptionThe account information is stored in “~/.azure/azureProfile.json” within the container.
Open a terminal in VSCode: menu Terminal -> New Terminal
Execute “az login” on the shell. Go to www.microsoft.com/devicelogin and fill in the secret provided in the console output
We have now linked our environment with an Azure subscription
The account information is stored in “~/.azure/azureProfile.json” within the container.
Let us look at Python AzureML SDK code to:
Create an AzureML WorkspaceCreate a compute cluster as a training targetRun a Python script on the compute target
Create an AzureML Workspace
Create a compute cluster as a training target
Run a Python script on the compute target
An AzureML workspace consists of a storage account, a docker image registry and the actual workspace with a rich UI on portal.azure.com. A new related Machine Learning studio is in open Beta and looks really good (https://ml.azure.com). The following code creates the workspace and its required services and attaches them to a new resource group.
from azureml.core import Workspaceworkspace = Workspace.create(    name="learnml",    subscription_id="<UUID>",    resource_group="ml",    create_resource_group=True,    location="<location>",    sku="basic",)
We need compute resources to train our ML model. We can create, orchestrate the compute clusters using the AzureML Python SDK. The workspace to create the resources in is defined by config.json file in the working directory (configure-workspace). AmlCompute stands for Azure Machine Learning Compute and it is the configuration class for the cluster. If the cluster does not exist we create one scaling from 0 to a defined maximum number of virtual machines (VMs). These VMs are running docker. We pass the provisioning configuration to the create method of the ComputeTarget class, which returns a ComputeTarget object. This object is used whenever we want to perform computations. The following code can be wrapped into a function to get the compute_target.
from azureml.core import Workspacefrom azureml.core.compute import AmlCompute, ComputeTargetfrom azureml.core.compute_target import ComputeTargetExceptionCLUSTER_NAME = "cpucluster"VM_SIZE = "Standard_D2_V2"MAX_NODES = 1workspace = Workspace.from_config()  # local config.json with subscription info# retrieve existing cluster or create a new onetry:    cpu_cluster = ComputeTarget(workspace=workspace, name=CLUSTER_NAME)    print("Found existing cluster, use it.")except ComputeTargetException:    compute_config = AmlCompute.provisioning_configuration(        vm_size=VM_SIZE, max_nodes=MAX_NODES    )cpu_cluster = ComputeTarget.create(workspace, CLUSTER_NAME, compute_config)
We want to run a Python script on our compute cluster. In part 2/2 of this series we will go into configuring the runtime Environment in detail. Here we show the minimum code needed to run a Pyton script on a compute_target (amlcompute-docs).
from azureml.core import Experiment, RunConfiguration, ScriptRunConfig, Workspacefrom tools import get_compute_targetCLUSTER_NAME = "cpucluster"EXPERIMENT_FOLDER = "/workspaces/azureml/code"workspace = Workspace.from_config()# get_compute_cluster wraps the code under section 2.2.2cpu_cluster = get_compute_target(workspace, CLUSTER_NAME)# Create a new runconfig objectrun_amlcompute = RunConfiguration()"""Skipping Environment details, treated in Blog Part 2/2, code is still valid"""# Set the compute target for the runrun_amlcompute.target = cpu_cluster# Create a script configscript_config = ScriptRunConfig(    source_directory=EXPERIMENT_FOLDER,    script="run_experiment.py",    run_config=run_amlcompute,)# Submit the experimentexperiment = Experiment(workspace=workspace, name="my-experiment")run = experiment.submit(config=script_config)
We have seen how easy it is to create an Azure ML workspace, spinning up a compute cluster and running a Python script on it. All this is tracked as an experiment within the ML studio.
Microsoft is all-in on Machine Learning and Python and this shows. For our workflow we have added Docker/WSL2 into the mix. From a developer perspective it is important that all the pieces work together reliably to allow actual work getting done. I am impressed with the quality of the various parts of the eco-system: Docker/WSL2, VSCode and AzureML and their integration into a workflow. Of course, there are some pitfalls along the way which are summarized below.
File permission mapping between Linux and Windows filesystem is odd (wsl-chmod-dev-blog); Bind mounts are under root:root
Crontab users might have to touch windows Task Scheduler (wsl-scheduling)
GPU support is on the way but not there yet (2020)
Linux GUI application is on the roadmap but not there yet (2021)
Originally published at https://codebeez.nl.