Package: airflow-provider-azure-machinelearning
Airflow Provider for Azure Machine LearningSource Code|Package_PyPI|Example DAGs|Example Docker ContainersThis package enables you to submit workflows to Azure Machine Learning from Apache Airflow.Pre-requisitesAzure AccountandAzure Machine LearningworkspaceTo verfiy your workspace is set up successfully, you can try to access your workspace atAzure Machine Learning Studio, and try to perform basic actions like allocating compute clusters and submittnig a training job, etc.A runningApache Airflowinstance.InstallationIn you Apache Airflow instance, run:pip install airflow-provider-azure-machinelearningOr, try it out by following examples in thedev folder, or Airflow'sHow-to-Guideto set up Airflow in Docker containers.Configure Azure Machine Learning Connections in AirflowTo send workload to your Azure Machine Learning workspace from Airflow, you need to set up an "Azure Machine Learning" Connection in your Airflow instance:Make sure this package is installed to your Airflow instance. Without this, you will not see "Azure Machine Learning" in the drop down in step 3 and will not be able to add this type of connections.On Airflow web portal, navigate toAdmin-->Connections, and click on+to add a new connection.From the "Connection Type" dropdown, select "Azure Machine Learning". You should see a form like belowConnection Idis a unique identifier for your connection. You will also need to pass this string into AzureML Airflow operators. Check out thoseexample dags.Descriptionis optional. All other fields are required.Tenant ID. You can followthis instructionto retrieve it.Subscription ID,Resource Group Name, andWorkspace Namecan uniquely identify your workspace in Azure Machine Learning. After openingAzure Machine Learning Studio, select the desired workspace, then click the "Change workspace" on the upper-right corner of the website (to the left of the profile icon). Here you can find theWorkspace Name. Now, click "View All Properties in Azure Portal'. This is Azure resource page of your workspace. From there you can retrieveSubscription ID, andResource Group Name.Client IDandSecretare a pair. They are basically 'username' and 'password' to the service principle based authentification process. You need to generate them in Azure Portal, and give it 'Contributor' permissions to the resource group of your workspace. That ensures your Airflow connection can read/write your Azure ML resources to facilitate workloads. Please follow the 3 simple steps below to set them up.To create a service principal, you need to follow 3 simple steps:Create aClient ID. Follow instruction from the "Register an application with Azure AD and create a service principal" section of Azure guidehowto-create-service-principal-portal.Application ID, akaClient ID, is the unique identifier of this service principal.Create aSecret. You can create aSecretunder this application in the Azure Portal following the instructions in the "Option 2: Create a new application secret" section ofthis instruction. Once asecretis successfully created, you will not be able to see the value. So we recommend you store your secret into Azure Key Vault, followingthis instruction.Give this Service PrincipalContribtoraccess to your Azure Machine LearningResource Group. Repeat the instruction form the item 7 above and land on your workspaces' resource page and click on theResource Group. From the left hand panel, selectAccess Control (IAM)and assignContributorrole to the the Application from above. This step is important. Without it, your Airflow will not have the necessary write access to necessary resources to create compute clusters, to execute training workloads, or to upload data, etc. Here isan instruction to assign roles.NoteIf "Azure Machine Learning" is missing from the dropdown in step 3 above, it meansairflow-providers-azure-machinelearningpackage is not successfully installed. You can follow instructions in theInstallation sectionto install it, and use commands like ``pip show airflow-provider-azure-machinelearning``` in the Airflow webserver container/machine to verify the package is installed correctly.You can have many connections in one Airflow instance for different Azure Machine Learning workspaces. You can do this to:Orchestrate workloads across multiple workspace/subscription from 1 single DAG.Achieve isolation between different engineers' workload.Achieve isolation between experimental and production environments.The instructions above are for adding a connection via the Airflow UI. You can also do so via the Airflow Cli. You can find more examples of how to do this via Cli atAirflow Documentation. Below is an example Airflow command:airflowconnectionsadd\--conn-type"azure_machine_learning"\--conn-description"[Description]"\--conn-host"schema"\--conn-login"[Client-ID]"\--conn-password"[Secret]"\--conn-extra'{"extra__azure_machine_learning__tenantId": "[Tenant-ID]", "extra__azure_machine_learning__subscriptionId": "[Subscription-ID]", "extra__azure_machine_learning__resource_group_name": "[Resource-Group-Name]", "extra__azure_machine_learning__workspace_name": "[Workspace-Name]"}'\"[Connection-ID]"ExamplesCheck outexample_dagson how to make use of this provider package. If you do not have a running Airflow instance, please refer toexample docker containers, or [Apache Airflow documentations)https://airflow.apache.org/).Dev EnvironmentTo build this package, run its tests, run its linting tools, etc, you will need following:Via pip:pip install -r dev/requirements.txtVia conda:conda env create -f dev/environment.ymlRunning the tests and lintersAll tests are intestsfolder. To run them, from this folder, runpytestThis repo usesblack,flake8, andisortto keep coding format consistent. From this folder, runblack .,isort ., andflake8.IssuesPlease submit issues and pull requests in our official repo:https://github.com/azure/airflow-provider-azure-machinelearning.ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.TrademarksThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must followMicrosoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.Release History0.0.1Features AddedFirst preview.