Package: ewah
ewahEwah: ELT With Airflow Helper - Classes and functions to make apache airflow life easier.Functions to create all DAGs required for ELT using only a simple config file.DWHs ImplementedSnowflakePostgreSQLBigqueryOperatorsEWAH currently supports the following operators:AircallBigQueryDynamoDBFacebook (partially, so far: ads insights; incremental only)FX Rates (from Yahoo Finance)Google AdsGoogle Analytics (incremental only)Google Maps (location data from an address)Google SheetsHubspotMailchimpMailingworkMongoDBMySQLOracleSQLPipedrivePostgreSQL / RedshiftRecurlyS3 (for CSV or JSON files stored in an S3 bucket, e.g. from Kinesis Firehose)SalesforceShopifyStripeZendeskUniversal operator argumentsThe following arguments are accepted by all operators, unless explicitly stated otherwise:argumentrequiredtypedefaultdescriptionsource_conn_idyesstringn.a.name of the airflow connection with source credentialsdwh_engineyesstringn.a.DWH type - e.g. postgres - usuallydwh_conn_idyesstringn.a.name of the airflow connection of the DWHtarget_table_nameimplicitstringn.a.name of the table in the DWH; the target table name is the name given in the table configtarget_schema_nameyesstringname of the schema in the DWH where the table will livetarget_schema_name_suffixnostring_nextwhen loading new data, how to suffix the schema name during the loading processtarget_database_nameyes for Snowflake DWHstringn.a.name of the database (only for Snowflake, illegal argument for non-Snowflake DWHs)drop_and_replacenobooleansame as DAG-level settingwhether a table is loading as full refresh or incrementally. Normally set by the DAG level config. Incremental loads can overwrite this setting to fully refresh some small tables (e.g. if they are small and have noupdated_atcolumn)primary_keyoperator-dependentstring or list of stringsn.a.name of the primary key column(s); if given, EWAH will set the column as primary key in the DWH and use it when applicable during upsert operationsadd_metadatanobooleanTruesome operators may add metadata to the tables; this behavior can be turned off (e.g. shop name for the shopify operator)Operator: Google AdsThese arguments are specific to the Google Ads operator.argumentrequiredtypedefaultdescriptionfieldsyesdictn.a.most important argument; excludes metrics; detailed belowmetricsyeslist of stringsn.a.list of all metrics to load, must load at least one metricresourceyesstringn.a.name of the report, e.g.keyword_viewclient_idyesstringn.a.10-digit number, often written with hyphens, e.g.123-123-1234(acceptable with or without hyphens)conditionsnolist of stringsn.a.list of strings of condition to include in the query, all conditions will be combined usingANDoperatordata_fromnodatetime, timedelta or airflow-template-stringdata_interval_start of task instancestart date of particular airflow task instance OR timedelta -> calculate delta from data_untildata_untilnodatetime or airflow-template-stringdata_interval_end of task instanceget data from google_ads until this pointarguments: fields and metricsthefieldsandmetricsarguments are the most important for this operator. Themetricsare separated fromfieldsbecause thefieldsare simultaneously the updated_on_columns. When creating the google ads query, they are combined. Themetricsargument is simply a list of metrics to be requested from Google Ads. Thefieldsargument is a bit more complex, due to the nature of Google Ad's API. It is essentially a nested json.Because the query may look something like this:SELECTcampaign.id,campaign.name,ad_group_criterion.criterion_id,ad_group_criterion.keyword.text,ad_group_criterion.keyword.match_type,segments.date,metrics.impressions,metrics.clicks,metrics.cost_microsFROMkeyword_viewWHEREsegments.dateBETWEEN2020-08-01AND2020-08-08i.e., there are nested object structures, thefieldsstructure must reflect the same. Take a look at the example config for the correct configuration of abovementioned Google Ads query. Note in addition, that the fields will be uploaded with the same names to the DWH, excepts that the periods will be replaced by underscored. i.e., the tablekeyword_view_datain the example below will have the columnscampaign_id,ad_group_criterion_keyword_text, etc.Finally, note thatsegments.dateis always required in thefieldsargument.Oracle operator particularitiesThe Oracle operator utilizes thecx_Oraclepython library. To make it work, you need to install additional packages, seeherefor details.ExampleSample configuration indags.yamlfile:EL_Google_Ads:incremental:Trueel_operator:google_adstarget_schema_name:raw_google_adsoperator_config:general_config:source_conn_id:google_adsclient_id:"123-123-1234"# conditions: noneincremental_config:data_from:!!python/object/apply:datetime.timedelta-3# days as time interval before the execution date of the DAG# only use this for normal runs! backfills: use execution date context insteadtables:keyword_view_data:resource:keyword_viewfields:campaign:-id-namead_group_criterion:-keyword:-text-match_type-criterion_idsegments:-datemetrics:-impressions-clicks-cost_microsPhilosophyThis package strictly follows an ELT Philosophy:Business value is created by infusing business logic into the data and making great analyses and usable data available to stakeholders, not by building data pipelinesAirflow solely orchestrates loading raw data into a central DWHData is either loaded as full refresh (all data at every load) or incrementally, exploiting airflow's catchup and execution logicThe only additional DAGs are dbt DAGs and utility DAGsWithin that DWH, each data source lives in its own schema (e.g.raw_salesforce)Irrespective of full refresh or incremental loading, DAGs always load into a separate schema (e.g.raw_salesforce_next) and at the end replace the schema with the old data with the schema with the new data, to avoid data corruption due to errors in DAG executionAny data transformation is defined using SQL, ideally usingdbtSeriously, dbt is awesome, give it a shot!(Non-SQL) Code contains no transformationsUsageIn your airflow Dags folder, define the DAGs by invoking either the incremental loading or full refresh DAG factory. The incremental loading DAG factory returns three DAGs in a tuple, make sure to call it like so:dag1, dag2, dag3 = dag_factory_incremental_loading()or add the dag IDs to your namespace like so:dags=dag_factory_incremental_loading()fordagindags:globals()[dag._dag_id]=dagOtherwise, airflow will not recognize the DAGs. Most arguments should be self-explanatory. The two noteworthy arguments areel_operatorandoperator_config.
The former must be a child object ofewah.operators.base.EWAHBaseOperator. Ideally, The required operator is already available for your use. Please feel free to fork and commit your own operators to this project! The latter is a dictionary containing the entire configuration of the operator. This is where you define what tables to load, how to load them, if loading specific columns only, and any other detail related to your EL job.Full refresh factoryAfilename.pyfile in your airflow/dags folder may look something like this:fromewah.utils.dag_factory_full_refreshimportdag_factory_drop_and_replacefromewah.constantsimportEWAHConstantsasECfromewah.operators.postgresimportEWAHPostgresOperatorfromdatetimeimportdatetime,timedeltadag=dag_factory_drop_and_replace(dag_name='EL_production_postgres_database',# Name of the DAGdwh_engine=EC.DWH_ENGINE_POSTGRES,# Implemented DWH Enginedwh_conn_id='dwh',# Airflow connection ID with connection details to the DWHel_operator=EWAHPostgresOperator,# Ewah Operator (or custom child class of EWAHBaseOperator)target_schema_name='raw_production',# Name of the raw schema where data will end up in the DWHtarget_schema_suffix='_next',# suffix of the schema containing the data before replacing the production data schema with the temporary loading schema# target_database_name='raw', # Only Snowflakestart_date=datetime(2019,10,23),# As per airflow standardschedule_interval=timedelta(hours=1),# Only timedelta is allowed!default_args={# Default args for DAG as per airflow standard'owner':'Data Engineering','retries':1,'retry_delay':timedelta(minutes=5),'email_on_retry':False,'email_on_failure':True,'email':['email@address.com'],},operator_config={'general_config':{'source_conn_id':'production_postgres','source_schema_name':'public',},'tables':{'table_name':{},# ...# Additional optional kwargs at the table level:#   primary_key#   + any operator specific arguments},},)For all kwargs of the operator config, the general config can be overwritten by supplying specific kwargs at the table level.Configure all DAGs in a single YAML fileStandard data loading DAGs should be just a configuration. Thus, you can
configure the DAGs using a simple YAML file. Yourdags.pyfile in your$AIRFLOW_HOME/dagsfolder may then look like that, and nothing more:importosfromairflowimportDAG# This module must be imported for airflow to see DAGsfromairflow.configurationimportconffromewah.dag_factoriesimportdags_from_yml_filefolder=os.environ.get('AIRFLOW__CORE__DAGS_FOLDER',None)folder=folderorconf.get("core","dags_folder")dags=dags_from_yml_file(folder+os.sep+'dags.yml',True,True)fordagindags:# Must add the individual DAGs to the global namespaceglobals()[dag._dag_id]=dagAnd the YAML file may look like this:---base_config:# applied to all DAGs unless overwrittendwh_engine:postgresdwh_conn_id:dwhairflow_conn_id:airflowstart_date:2019-10-23 00:00:00+00:00schedule_interval:!!python/object/apply:datetime.timedelta-0# days-3600# secondsschedule_interval_backfill:!!python/object/apply:datetime.timedelta-7schedule_interval_future:!!python/object/apply:datetime.timedelta-0-3600additional_task_args:retries:1retry_delay:!!python/object/apply:datetime.timedelta-0-300email_on_retry:Falseemail_on_failure:Trueemail:['me+airflowerror@mail.com']el_dags:EL_Production:# equals the name of the DAGincremental:Falseel_operator:postgrestarget_schema_name:raw_productionoperator_config:general_config:source_conn_id:production_postgressource_schema_name:publictables:users:source_table_name:Userstransactions:source_table_name:UserTransactionssource_schema_name:transaction_schema# Overwrite general_config args as neededEL_Facebook:incremental:Trueel_operator:fbstart_date:2019-07-01 00:00:00+00:00target_schema_name:raw_facebookoperator_config:general_config:source_conn_id:facebookaccount_ids:-123-987data_from:'{{data_interval_start}}'# Some fields allow airflow templating, depending on the operatordata_until:'{{data_interval_end}}'level:adtables:ads_data_age_gender:insight_fields:-adset_id-adset_name-campaign_name-campaign_id-spendbreackdowns:-age-gender...Developing EWAH locally with DockerIt is easy to develop EWAH with Docker. Here's how:Step 1: clone the repository locallyStep 2: configure your local secretsInewah/airflow/docker/secrets, you should at least have a file calledsecret_airflow_connections.ymlwhich looks exactly likeewah/airflow/docker/airflow_connections.ymland contains any additional credentials you may need during development -> this file can also contain connections defined in theairflow_connections.yml, in which case your connections will overwrite them.You can add any other files in this folder that you may need, e.g. private key filesFor example, you can add a file calledgoogle_service_acc.jsonthat contains service account credentials that can be loaded as extra into an airflow connection used for a Google Analytics DAGSample files are shown belowStep 3: rundocker-compose upto start the postgres, webserver and scheduler containersYou can stop them withCTRL+CIf you want to free up the ports again or if you are generally done developing, you should additionally rundocker-compose downStep 4: Make any changes you wish to files inewah/ewah/orairflow/dags/If you add new dependencies, makes sure to add them in theewah/setup.pyfile; you may need to restart the containers to include the new dependencies in the ewah installations running in your containersStep 5: Commit, Push and open a PRWhen pushing, feel free to include configurations in yourewah/airflow/dags/dags.ymlthat utilize your new feature, if applicableSamplesecret_airflow_connections.yml---connections:-id:ssh_tunnelhost:[SSH Server IP]port:[SSH Port]# usually 22login:[SSH username]password:[private key password OR SSH server password]extra:!text_from_file/opt/airflow/docker/secrets/ssh_rsa_private# example, with a file called `ssh_rsa_private` in your `ewah/airflow/docker/secrets/` folder-id:shopwarehost:[host]# "localhost" if tunnelling into the server running the MySQL databaseschema:[database_name]port:3306login:[username]password:[password]-id:google_service_accountextra:!text_from_file/opt/airflow/docker/secrets/google_service_acc.json# example, with a file called `google_service_acc.json` in your `ewah/airflow/docker/secrets/` folder...Samplegoogle_service_acc.json{"client_secrets":{"type":"service_account","project_id":"my-project-id","private_key_id":"abcdefghij1234567890abcdefghij1234567890","private_key":"-----BEGIN PRIVATE KEY-----\n[...]\n-----END PRIVATE KEY-----\n","client_email":"xxx@my-project-id.iam.gserviceaccount.com","client_id":"012345678901234567890","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/xxx%40my-project-id.iam.gserviceaccount.com"}}Using EWAH with AstronomerTo avoid all devops troubles, it is particularly easy to use EWAH with astronomer.
Your astronomer project requires the following:addewahto therequirements.txtaddlibstdc++to thepackages.txthave adags.pyfile and adags.ymlfile in your dags folderin production, you may need to request your airflow metadata postgres database password from the support for incremental loading DAGs