Package: airflow-pentaho-plugin
Pentaho Airflow pluginThis plugins runs Jobs and Transformations through Carte servers.
It allows to orchestrate a massive number of trans/jobs taking care
of the dependencies between them, even between different instances.
This is done by usingCarteJobOperatorandCarteTransOperatorIt also runs Pan (transformations) and Kitchen (Jobs) in local mode,
both from repository and local XML files. For this approach, useKitchenOperatorandPanOperatorRequirementsA Apache Airflow system deployed.One or many working PDI CE installations.A Carte server for Carte Operators.SetupThe same setup process must be performed on webserver, scheduler
and workers (that runs this tasks) to get it working. If you want to
deploy specific workers to run this kind of tasks, seeQueues,
inAirflowConceptssection.Pip packageFirst of all, the package should be installed viapip installcommand.pipinstallairflow-pentaho-pluginAirflow connectionThen, a new connection needs to be added to Airflow Connections, to do this,
go to Airflow web UI, and click onAdmin -> Connectionson the top menu.
Now, click onCreatetab.Use HTTP connection type. Enter theConn Id, this plugin usespdi_defaultby default, the username and the password for your Pentaho Repository.At the bottom of the form, fill theExtrafield withpentaho_home, the
path where your pdi-ce is placed, andrep, the repository name for this
connection, using a json formatted string like it follows.{"pentaho_home":"/opt/pentaho","rep":"Default"}CarteIn order to useCarteJobOperator, the connection should be set different. Fillhost(includinghttp://orhttps://) andportfor Carte hostname and port,usernameandpasswordfor PDI repository, andextraas it follows.{"rep":"Default","carte_username":"cluster","carte_password":"cluster"}UsageCarteJobOperatorCarteJobOperator is responsible for running jobs in remote slave servers. Here
it is an example ofCarteJobOperatorusage.# For versions before 2.0# from airflow.operators.airflow_pentaho import CarteJobOperatorfromairflow_pentaho.operators.carteimportCarteJobOperator# ... ## Define the task using the CarteJobOperatoravg_spent=CarteJobOperator(conn_id='pdi_default',task_id="average_spent",job="/home/bi/average_spent",params={"date":"{{ ds }}"},# Date in yyyy-mm-dd formatdag=dag)# ... #some_task>>avg_spent>>another_taskKitchenOperatorKitchen operator is responsible for running Jobs. Lets suppose that we have
a definedJobsaved on/home/bi/average_spentin our repository with
the argumentdateas input parameter. Lets define the task using theKitchenOperator.# For versions before 2.0# from airflow.operators.airflow_pentaho import KitchenOperatorfromairflow_pentaho.operators.kettleimportKitchenOperator# ... ## Define the task using the KitchenOperatoravg_spent=KitchenOperator(conn_id='pdi_default',queue="pdi",task_id="average_spent",directory="/home/bi",job="average_spent",params={"date":"{{ ds }}"},# Date in yyyy-mm-dd formatdag=dag)# ... #some_task>>avg_spent>>another_taskCarteTransOperatorCarteTransOperator is responsible for running transformations in remote slave
servers. Here it is an example ofCarteTransOperatorusage.# For versions before 2.0# from airflow.operators.airflow_pentaho import CarteTransOperatorfromairflow_pentaho.operators.carteimportCarteTransOperator# ... ## Define the task using the CarteJobOperatorenriche_customers=CarteTransOperator(conn_id='pdi_default',task_id="enrich_customer_data",job="/home/bi/enrich_customer_data",params={"date":"{{ ds }}"},# Date in yyyy-mm-dd formatdag=dag)# ... #some_task>>enrich_customers>>another_taskPanOperatorPan operator is responsible for running transformations. Lets suppose that
we have one saved on/home/bi/clean_somedata. Lets define the task using thePanOperator. In this case, the transformation receives a parameter that
determines the file to be cleaned.# For versions before 2.0# from airflow.operators.airflow_pentaho import PanOperatorfromairflow_pentaho.operators.kettleimportPanOperator# ... ## Define the task using the PanOperatorclean_input=PanOperator(conn_id='pdi_default',queue="pdi",task_id="cleanup",directory="/home/bi",trans="clean_somedata",params={"file":"/tmp/input_data/{{ ds }}/sells.csv"},dag=dag)# ... #some_task>>clean_input>>another_taskFor more information, please seesample_dags/pdi_flow.py