Package: awehflow
AwehflowConfiguration based Airflow pipelines with metric logging and alerting out the box.PrerequisitesDevelopment environmentIn order to develop awehflow for a given version of Airflow follow these stepsInstall and configure minicondaOn Mac, if running ARM create an x86 version of conda using the snippet below
```bash
### add this to ~/.zshrc (or ~/.bashrc if you're using Bash)
create_x86_conda_environment () {create a conda environment using x86 architecturefirst argument is environment name, all subsequent arguments will be passed toconda createexample usage: create_x86_conda_environment myenv_x86 python=3.9CONDA_SUBDIR=osx-64 conda create $@
conda activate $2
conda config --env --set subdir osx-64
}1. Define the version that you'd like to install
```bash
export AIRFLOW_VERSION="2.1.4"Create a conda environment for your version of Airflow, the bash belowcreate_x86_conda_environment-n"airflow_$AIRFLOW_VERSION""python=3.8.12"Configure the AIRFLOW_HOME directorycondadeactivate
condaactivate"airflow_$AIRFLOW_VERSION"condaenvconfigvarssetAIRFLOW_HOME="$HOME/airflow/airflow_$AIRFLOW_VERSION"condadeactivate
condaactivateairflow_"$AIRFLOW_VERSION"echo"$AIRFLOW_HOME"Install airflow usingpipcondaactivateairflow_$AIRFLOW_VERSIONpipinstall--no-cache-dir"apache-airflow==$AIRFLOW_VERSION"Install required providerscondaactivateairflow_$AIRFLOW_VERSIONpipinstall--no-cache-dir"apache-airflow[google]==$AIRFLOW_VERSION"pipinstall--no-cache-dir"apache-airflow[postgres]==$AIRFLOW_VERSION"On MacOS ARM install the psycop binarybash     pip install --no-cache-dir "psycopg2-binary==`pip list | grep -i 'psycopg2 ' | tr -s ' ' | cut -d' ' -f 2`"Customisation per versionFor2.2.31. force the MarkupSafe package versionpipinstall--no-cache-dirmarkupsafe==2.0.1Init the airflow dbairflowdbinitYou will need the following to run this code:Python 3Installationpip install awehflow[default]If you are installing on Google Cloud Composer with Airflow 1.10.2:pip install awehflow[composer]Event & metric tablesCreate apostgresqldatabase that can be referenced via Airflow connection. In the DB create the following tablesJobs data tableCREATETABLEpublic.jobs(idserial4NOTNULL,run_idvarcharNOTNULL,dag_idvarcharNULL,"name"varcharNULL,projectvarcharNULL,statusvarcharNULL,engineersjsonNULL,errorjsonNULL,start_timetimestamptzNULL,end_timetimestamptzNULL,reference_timetimestamptzNULL,CONSTRAINTjob_id_pkeyPRIMARYKEY(id),CONSTRAINTrun_id_dag_id_uniqueUNIQUE(run_id,dag_id));Task metrics tableCREATETABLEpublic.task_metrics(idserial4NOTNULL,run_idvarcharNULL,dag_idvarcharNULL,task_idvarcharNULL,job_namevarcharNULL,valuejsonNULL,created_timetimestamptzNULL,reference_timetimestamptzNULL,CONSTRAINTtask_metrics_id_pkeyPRIMARYKEY(id));Data metrics tableCREATETABLEpublic.data_metrics(idserial4NOTNULL,platformvarcharNULL,"source"varcharNULL,"key"varcharNULL,valuejsonNULL,reference_timetimestamptzNULL,CONSTRAINTdata_metrics_pkeyPRIMARYKEY(id),CONSTRAINTunique_metricUNIQUE(platform,source,key,reference_time));UsageUsage ofawehflowcan be broken up into two parts: bootstrapping and configuration of pipelinesBootstrapIn order to expose the generated pipelines (airflowDAGs) forairflowto pick up when scanning forDAGs, one has to create aDagLoaderthat points to a folder where the pipeline configuration files will be located:importosfromawehflow.alerts.slackimportSlackAlerterfromawehflow.coreimportDagLoaderfromawehflow.events.postgresimportPostgresMetricsEventHandler"""airflow doesn't pick up DAGs in files unlessthe words 'airflow' and 'DAG' features"""configs_path=os.path.join(os.path.dirname(os.path.abspath(__file__)),'configs')metrics_handler=PostgresMetricsEventHandler(jobs_table='jobs',task_metrics_table='task_metrics')slack_alerter=SlackAlerter(channel='#airflow')loader=DagLoader(project="awehflow-demo",configs_path=configs_path,event_handlers=[metrics_handler],alerters=[slack_alerter])dags=loader.load(global_symbol_table=globals())As seen in the code snippet, one can also pass in"event handlers"and"alerters"to perform actions on certain pipeline events and potentially alert the user of certain events on a given channel. See the sections below for more detail.
The global symbol table needs to be passed to theloadersinceairflowscans it for objects of typeDAG, and then synchronises the state with its own internal state store.*caveat:airflowignorespythonfiles that don't contain the words"airflow"and"DAG". It is thus advised to put those words in a comment to ensure the generatedDAGsget picked up when theDagBagis getting filled.Event HandlersAs a pipeline generated usingawehflowis running, certain events get emitted. An event handler gives the user the option of running code when these events occur.The following events are (potentially) potentially emitted as a pipeline runs:startsuccessfailuretask_metricExisting event handlers include:PostgresMetricsEventHandler: persists pipeline metrics to a Postgres databasePublishToGooglePubSubEventHandler: events get passed straight to a Google Pub/Sub topicAnAlertsEventHandlergets automatically added to a pipeline. Events get passed along to registered alerters.AlertersAnAlerteris merely a class that implements analertmethod. By default aSlackAlerteris configured in thedags/PROJECT/bootstrap.pyfile of an awehflow project.  awehflow supports the addition of multiple alerters, which allows success or failure events to be sent to mutliple channelsYAML configurationIn order to add alerts to an awehflow DAG add the following to the root space of the configurationalert_on:-'failure'# Send out a formatted message if a task in the DAG fails. This is optional-'success'# Send out a formatted message once the DAG completes successfully. This is optionalAvailable alertersSlackAlerter-awehflow.alerts.slack.SlackAlerterSends an alert to a specified slack channel via the Slack webhook functionalityParameterschannel- The name of the channel that the alerts should be sent toslack_conn_id- The name of the airflow connection that contains the token information, default:slack_defaultConnection requirements - Create a HTTP connection with the name specified forslack_conn_id, the required HTTP fields are:password- The slack token issued by your admin team, which allows for the sending of messages via the slack python APIGoogleChatAlerter-awehflow.alerts.googlechat.GoogleChatAlerterSends an alert to the configured Google Chat spaceParametersgchat_conn_id- The name of the airflow connection that contains the GChat space information, default:gchat_defaultConnection requirements - Create a HTTP connection with the name specified for thegchat_conn_id, the requried HTTP fields are:host- The GChat spaces URLhttps://chat.googleapis.com/v1/spacespassword- The GChat spaces key configuration information, exhttps://chat.googleapis.com/v1/spaces/SPACES_ID?key=SPACES_KEYSPACES_ID- Should be supplied by your GChat admin teamSPACES_KEY- Should be supplied by your GChat admin teamConfigurationAwehflow configuration files can be written as .yml OR .hocon files either formats are supportedShown below is sample hocon configuration file{
  name: my_first_dag,
  version: 1,
  description: "This is my first dag",
  owner: The name of the owner of the DAG
  schedule: "10 0 * * *",
  start_date: 2022-01-01,
  end_date: 2022-01-01,
  catchup: true,
  concurrency: 1 // Defaults to airflow configuration
  max_active_tasks: 1 // Defaults to airflow configuration
  max_active_runs: 1 // Defaults to airflow configuration
  dagrun_timeout: None
  doc_md: The DAG documentation markdown
  access_control: None // A dict of roles that have specific permissions
  is_paused_upon_creation: None // Defaults to airflow configuration
  tags: [
    'tag one',
    'tag two'
  ],
  dag_params: { 
    /* This dict will define DAG parameters and defaulted when triggering a DAG manually with CONF,
    Values are accessible as template values {{ dag_run.conf["config_value_1"] }}
    */
    'config_value_1': 'SOME TEXT',
    'config_value_2': 1234
  },
  alert_on:[ // Whether the events alert should send a message on success OR failure
    success,
    failure
  ],
  params: { // parameter values that will be passed in to each task for rendering
    default: {
      source_folder: /tmp
    },
    production: {
      source_folder: /data
    }
  },
  default_dag_args: { //The default DAG arguments whichis also passed to each task in the dag
    retries: 1
  },
  pre_hooks: [ // Pre hook sensors are executed BEFORE the start task
    {
      id: 'pre_hook_ping_sensor'
      operator: 'airflow.sensors.bash.BashSensor'
      params: {
        bash_command: 'echo ping'
        mode: 'reschedule'
      }
    }
  ],
  dependencies: [ // Dependencies sensors are executed AFTER the start task to the DAG start time being logged
    {
      id: 'dependenciy_ping_sensor'
      operator: 'airflow.sensors.bash.BashSensor'
      params: {
        bash_command: 'echo ping'
        mode: 'reschedule'
      }
    }
  ],
  tasks: [ // The array of Tasks that defines the DAG
      {
        id: first_dummy_task, // The task ID that will be shown in the task bubble or tree
        operator: airflow.operators.dummy.DummyOperator, // The fully qualified path to the Class of the Operator
      },
      {
        id: first_bash_task, // The task ID that will be shown in the task bubble or tree
        operator: airflow.operators.bash.BashOperator, // The fully qualified path to the Class of the Operator
        params: { 
          /* 
          The dictionary of parameters that will be passed to the Operator, the "default_dag_args" dict will be merged with this.
          Any parameter of the Operator Class can be added to this dict, template rending of values depends on the specific Operator
          */
          bash_command: 'echo "Hello World"'
        },
        upstream: [ // The list of tasks that must be executed prior to this task
          first_dummy_task
        ]
      }
    ]
}This configuration does the following:Creates a DAG calledmy_first_dagScheduled to run daily 10min past midnightCatchup has been enabled to ensure all runs of the DAG since 2022-01-01 are executedPre hooksCheck if the commandecho pingsucceedsDependenciesCheck if the commandecho pingsucceedsTasksFirst run a dummy task that does nothingIf the dummy task succeeds, execute the bash commandRunning the testsTests may be run withpython-munittestdiscovertestsor to run code coverage too:coveragerun-munittestdiscovertests&&coveragehtml