Package: airflow-metrics
airflow-metricsairflow-metricsis an Airflow plugin for automatically sending metrics from Airflow to Datadog.Tested For:apache-airflow>=1.10.2, <=1.10.3Installationpipinstallairflow-metricsOptionalIf you want to the metrics fromBigQueryOperatorandGoogleCloudStorageToBigQueryOperator, then make sure the necessary dependencies are installed.pipinstallapache-airflow[gcp_api]Setupairflow-metricswill report all metrics to Datadog, so create anairflowconnection with your Datadog api key.airflowconnections--add--conn_iddatadog_default--conn_typeHTTP--conn_extr'{"api_key": "<your api key>"}'Note: If you skip this step, yourairflowinstallation should still work but no metrics will be reported.UsageThat's it!airflow-metricswill now begin sending metrics from Airflow to Datadog automatically.Metricsairflow-metricswill automatically begin reporting the following metricsairflow.task.stateThe total number of tasks in a state where the state is stored as a tag.airflow.task.state.bqThe current number of big query tasks in a state where the state is stored as a tag.airflow.dag.durationThe duration of a DAG in ms.airflow.task.durationThe duration of a task in ms.airflow.request.durationThe duration of a HTTP request in ms.airflow.request.status.successThe current number of HTTP requests with successful status codes (<400)airflow.request.status.failureThe current number of HTTP requests with unsuccessful status codes (>=400)airflow.task.upserted.bqThe number of rows upserted by a BigQueryOperator.airflow.task.delay.bqThe time taken for the big query job from a BigQueryOperator to start in ms.airflow.task.duration.bqThe time taken for the big query job from a BigQueryOperator to finish in ms.airflow.task.upserted.gcs_to_bqThe number of rows upserted by a GoogleCloudStorageToBigQueryOperator.airflow.task.delay.gcs_to_bqThe time taken for the big query from a GoogleCloudStorageToBigQueryOperator to start in ms.airflow.task.duration.gcs_to_bqThe time taken for the big query from a GoogleCloudStorageToBigQueryOperator to finish in ms.ConfigurationBy default,airflow-metricswill begin extracting metrics from Airflow as you run your DAGs and send them to Datadog. You can opt out of it entirely or opt out of a subset of the metrics by setting these configurations in yourairflow.cfg[airflow_metrics]

airflow_metrics_enabled = True
airflow_metrics_tasks_enabled = True
airflow_metrics_bq_enabled = True
airflow_metrics_gcs_to_bq_enabled = True
airflow_metrics_requests_enabled = True
airflow_metrics_thread_enabled = True`Limitationsairflow-metricsstarts a thread to report some metrics, and is not supported when using sqlite as your database.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Getting StartedSet up your virtual environment for python3 however you like.pipinstall-e.
airflowinitdb
airflowconnections--add--conn_iddatadog_default--conn_typeHTTP--conn_extr'{"api_key": ""}'Note: The last step is necessary, otherwise the plugin will not initialize correctly and will not collect metrics. But you are free to add a dummy key for development purposes.Running Testspipinstall-rrequirements-dev.txt
pytest