Package: airflow-hdinsight
airflow-hdinsightA set of airflow hooks, operators and sensors to allow airflow DAGs to operate with the Azure HDInsight platform, for cluster creation and monitoring as well as job submission and monitoring. Also included are some enhanced Azure Blob and Data Lake sensors.This project is both an amalgamation and enhancement of existing open source airflow extensions, plus new extensions to solve the problem.Installationpip install airflow-hdinsightExtensionsairflowhdiTypeNameWhat it doesHookAzureHDInsightHookUses the HDInsightManagementClient from theHDInsight SDK for Pythonto expose several operations on an HDInsight cluster - get cluster state, create, delete.OperatorAzureHDInsightCreateClusterOperatorUse the AzureHDInsightHook to create a clusterOperatorAzureHDInsightDeleteClusterOperatorUse the AzureHDInsightHook to delete a clusterOperatorConnectedAzureHDInsightCreateClusterOperatorExtends the AzureHDInsightCreateClusterOperator to allow fetching of the security credentials and cluster creation spec from an airflow connectionOperatorAzureHDInsightSshOperatorUses the AzureHDInsightHook and SSHHook to run an SSH command on the master node of the given HDInsight clusterSensorAzureHDInsightClusterSensorA sensor to monitor the provisioning state or running state (can switch between either mode) of a given HDInsight cluster. Uses the AzureHDInsightHook.SensorWasbWildcardPrefixSensorAn enhancement to theWasbPrefixSensorto support sensing on a wildcard prefixSensorAzureDataLakeStorageGen1WebHdfsSensorUses airflow'sAzureDataLakeHookto sense a glob path (which implicitly supports wildcards) on ADLS Gen 1. ADLS Gen 2 is not yet supported in airflow.airflowlivyTypeNameWhat it doesHookLivyBatchHookUses the Apache LivyBatch APIto submit spark jobs to a livy server, get batch state, verify batch state by quering either the spark history server or yarn resource manager, spill the logs of the spark job post completion, etc.OperatorLivyBatchOperatorUses the LivyBatchHook to submit a spark job to a livy serverSensorLivyBatchSensorUses the LivyBatchHook to sense termination and verify completion, spill logs of a spark job submitted earlier to a livy serverOrigins of the HDinsight operator workThe HDInsight operator work is loosely inspired fromalikemalocalan/airflow-hdinsight-operators, however that has a huge number of defects, as to why it wasnever acceptedto bemergedinto airflow in the first place. This project solves all of those issues and more, and is frankly a full rewrite.Origins of the livy workThe livy batch operator is based on the work bypanovvv's projectairfllow-livy-operators. It does some necessary changes:Seperates the operator into a hook (LivyBatchHook), an operator (LivyBatchOperator) and a sensor (LivyBatchSensor)Adds additional verification and log spilling to the sensor (the original sensor does not)Removes additional verifiation and log spilling from the operator - hence alllowing a async pattern akin to the EMR add step operator and step sensor.Creates livy, spark and YARN airflow connections dynamically from an Azure HDInsight connectionReturns the batch ID from the operator so that a sensor can use it after being passed through XComChanges logging to LoggingMixin callsAllows templatization of fieldsState of airflow livy operators in the wild..As it stands today (June of 2020), there are multiple airflow livy operator projects out there:panovvv/airflow-livy-operators: the project which this project bases its work ontheofficial livy providerin airflow 2.0, with a backport available for airflow 1.1.x: alas the official provider has very limited functionality - it does not spill the job's logs, and it does not do additional verification for job completion using spark history server or yarn resource manager, amongst other limitationsrssanders3/airflow-spark-operator-plugin: this is the oldest livy operator, which only supports livy sessions and not batches. there's a copy of this inalikemalocalan/airflow-hdinsight-operators.