GDI/Equity Landscape/Airflow

==Equity Landscape Airflow==
===DAGs===
All the Airflow DAGs for Equity Landscape can be found under the Data Engineering repo.
We currently have four types of DAGs for the project namely. 
* equity_landscape_api_dag (API)
* equity_landscape_csv_dag (CSV)
* equity_landscape_hql_dag (HQL)
* equity_landscape_app_dag (APP) 
====Scheduling====
* The DAGs are not scheduled, they are run on an ad-hoc basis.
* This is due to the pipeline pooling data from various external sources, some of which their update schedules cannot be determined.
====DAG execution====
* The order of execution is:
** API and CSV can run independently.
** HQL needs to run after both API and CSV have run.
** APP can only run after HQL is run. 
====API DAG====
* Ideal location for all data that needs to be stored onto HDFS via API calls.
* Uses a job_config property to specify a task and it's parameters. 
** The job_config property consists of:
*** task_name (dict) which contains the below properties
*** application : The python program to execute (string)
*** args: The arguments that the python program expects (dict)
**** Example DAG variable
**** 
**** 
* The DAG consists of the following tasks:
** load_lua (currently not in use and is removed from the code but it's there should it be required)
*** This uses load_lua_data from the project repo to load data into organizational_info_input_metrics. We aren't using this data currently as it has not matured enough to be used to be used as an alternative for Affiliate data. 
*** 
**load_world_bank_data
*** This uses the load_wbgapi from the project repo
****It takes in the db number and series as arguments. Uses the world bank python package to fetch the data and load them into Hive (world_bank_data_input_metrics)
====CSV DAG====
* Responsible for loading CSV files into the respective Hive Tables. 
* This also uses a job_config dictionary that consists of the following properties: 
** A list of dictionaries that consist of: 
** table_name - name of the Hive table to insert into 
** file_location - filename of the csv file (in the DAG, this value is concatenated to base_csv_location variable that can also be specified as part of the DAG variables. 
** There is an optional args dictionary that is used to specify additional parameters to load_csv
** Example DAG variable
** 
** 
====HQL DAG====
DAG graph
* This is responsible for the first part of data transformations that uses HQL files to load data into Hive tables. 
** The files are stored on the refinery repo.
* The DAG uses TaskGroups to define the metrics each set of HQL files represent.
** geometrics
*** alt= Image of Geometrics Task Group| Geometrics TaskGroup|795x795px
*** editorship
**** Includes the following tasks:
***** geoeditor_input_metrics
****** Takes in data from canonical_data.wikis and wiki_db_map_input_metrics that we loaded using the CSV DAG and lastly wmf.geoeditor_monthly to load the data into geoeditor_input_metrics.
****** It is executed twice (has the years: [] config and uses the a_year_ago and two_years_ago flag)
***** geoeditor_input_metrics_pivot
****** Takes in data from geoeditor_input_metrics and pivots it.
***** geoeditor_online_input_metrics
****** Used for aggregating data from wmf.geoeditors_monthly and country_meta_data to calculate active editor data.
***** brief_projects_edited_metrics
****** Calculates the average active editors per wiki_db, project_label by country.
*** readership
**** georeadership_input_metrics
***** Used for storing the following partitions:
****** pageviews
****** unique_devices
****** yoy_metrics for the above metrics.
** grants_leadership
*** grants_leadership_input_metrics
**** Takes in data from grants_input_metrics and calculates various grants_metrics including total_historical_grants_to_data, total_calendar_year_grants amongst others.
** population
*** population_leadership_input_metrics
**** Uses data from population_data_input_metrics that was loaded using the CSV DAG. 
*** access' has been moved to the APP DAG.
* Each metric is represented by a dictionary that contains a list of HQL tasks and properties.
** Each HQL task contains it's own properties, namely:
*** pos used for ordering each task in the DAG list to ensure that it runs before/after another task.
*** hql_path an hdfs location specifiying the location of the hql file.
*** query_parameters the parameters passed into the hql file.
**** Note: year='-' is replaced by the a_year_ago parameter from the DAG's variables.
*** The years: [] property that uses the a_year_ago and two_years_ago parameters to create a range of year that the HQL task needs to run for.
* Example DAG variable
** 
====APP DAG====
* The second and final part of data transformations which comprises of two portions, the inputs and outputs.
* We use a job_config dictionary to define the tasks and their properties.
* This DAG is responsible for the hdfs outputs which are the final output metrics and dashboard files. 
* For the inputs we currently have:
** access_input_metrics
** affiliate_input_metrics
** programs_and_events_input_metrics
** freedom_leadership_input_metrics
* For the outputs we currently have:
** input_metrics
** regional_input_metrics
** output_metrics
** dashboard_output
** languages_output
* dashboard_output and languages_output are tasks that generate files to be used by the dashboard.
* Example DAG
*