Package: airflow-kube-job-operator
Airflow Kubernetes Job OperatorWhat is this?An Airflow Operator that manages creation, watching, and deletion of a Kubernetes Job. It assumes the client passes in a path to a yaml file that may have Jinja templated fields.Who is it for?This package makes the assumption that you're using Kubernetes somehow. Airflow itself may be deployed in Kubernetes (in_cluster mode) or you may just want it to manage Jobs running remotely on a cluster (give Airflow a kube config).Why would I use this?In our use of Airflow we struggled a lot with binding our business logic via many different custom Operators and Plugins directly to Airflow. Instead, we found Airflow to be a great manager of execution of code but not the best tool for writing the ETL/ML code itself.Ideally this should be one of the only Airflow Operators you need.How do I use it?Here are the parameters.ParameterDescriptionTypeyaml_file_nameThe name of the yaml file, could be a full pathstryaml_write_pathIf you want the rendered yaml file written, where should it be?stryaml_write_filenameIf you want the rendered yaml file written, what is the filename?stryaml_template_fieldsIf you have variables in your yaml file you want filled outdicttail_logsWhether to output a log tail of the pods to airflow, will only do it at an end statebool (F)tail_logs_everyevery x seconds to wait to begin a new log dump (nearest 5 sec)inttail_logs_line_countnum of lines from end to outputintlog_yamlWhether to log the rendered yamlbool (T)in_clusterWhether or not Airflow has cluster permissions to create and manage Jobsbool (F)config_fileThe path to the kube configfilestrcluster_contextIf you are using a kube config file include the cluster contextstrdelete_completed_jobAutodelete Jobs that completed without errorsbool (F)Step 1. Install the packagepipinstallairflow-kube-job-operatorStep 1.5 (Optional) Add Role to your Airflow deploymentIf you want the Jobs to get created without having to bundle your kubeconfig file somehow into your Airflow pods, you'll need to deploy Airflow in kubernetes and give Airflow some extra RBAC permissions to handle Jobs within your cluster.** This is needed if you want to use the optionin_cluster=True**Here's an example of what you may needkind:RoleapiVersion:rbac.authorization.k8s.io/v1metadata:name:airflowrules:-verbs:-create-list-get-watch-delete-update-patchapiGroups:-''-batchresources:-pods-jobs-jobs/status-verbs:-getapiGroups:-''resources:-pods/log-verbs:-create-getapiGroups:-''resources:-pods/execIf you want to give Airflow power to run Jobs Cluster-wide modify the ClusterRole instead.Alternatively, just give Airflow your kube cluster config. (A.ii.)Step 2. Create a template folder for your yaml filesThis template folder can be anywhere. It's up to you. But here's a suggestion.If you have...~/airflow/dagsthen~/airflow/kubernetes/jobCould be a valid choice.Lets create a very simple job and put it there.apiVersion:batch/v1kind:Jobmetadata:name:countdownnamespace:<WRITE YOUR AIRFLOW NAMESPACE HERE>spec:template:metadata:name:countdownspec:containers:-name:counterimage:centos:7command:-"bin/bash"-"-c"-"foriin987654321;doecho$i;done"restartPolicy:NeverSave the above at~/airflow/kubernetes/job/countdown.yamlStep 3. Create your DagFirst some questions to ask yourself...A. How do I want my Dag to have access to kubernetes? 
    i. My Airflow has the above RBAC permissions to make Jobs
    ii. I rather just use my kube config file. It's accessible somewhere in Airflow already (web, worker, and scheduler)

B. What does my yaml look like? 
    i. I have a simple yaml file. Just create my Job. (The yaml, 'countdown.yaml' above is like this)
    ii. I have a single yaml file for my Job but I want some templated fields filled out. 
    iii. I'm hardcore. I have multiple yaml files templated in the Jinja style so I can reuse my templates across tasks and dags.A.i. Usingin_cluster=TruefromairflowimportDAGfromdatetimeimportdatetime,timedeltafromairflow_kjoimportKubernetesJobOperatordefault_args={'owner':'airflow','depends_on_past':False,'email':['airflow@example.com'],'email_on_failure':False,'email_on_retry':False,'retries':1,# the number of times the pod will retry, can pass in per-task'retry_delay':timedelta(minutes=5),'start_date':datetime(2021,2,24,12,0),}withDAG('kubernetes_job_operator',default_args=default_args,description='KJO example DAG',schedule_interval=None,catchup=False)asdag:task_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='/path/to/airflow/kubernetes/job/countdown.yaml',in_cluster=True)A.ii. Usingconfig_file=/path/to/.kube/configtask_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='/path/to/airflow/kubernetes/job/countdown.yaml',config_file='/path/to/.kube/config',cluster_context='my_kube_config_context')What is this "my_kube_config_context" business?
Read about it in the kubernetes config documentationhereB.i. Simple yaml file executionIn addition to the above Dag styles you could also make use of Airflow's nativetemplate_searchpathfield to clean up the Dag a bit.fromairflowimportDAGfromdatetimeimportdatetime,timedeltafromairflow_kjoimportKubernetesJobOperatordefault_args={'owner':'airflow','depends_on_past':False,'email':['airflow@example.com'],'email_on_failure':False,'email_on_retry':False,'retries':1,'retry_delay':timedelta(minutes=5),'start_date':datetime(2021,2,24,12,0),}withDAG('kubernetes_job_operator',default_args=default_args,description='KJO example DAG',schedule_interval=None,template_searchpath='/path/to/airflow/kubernetes/job'catchup=False)asdag:task_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown.yaml',in_cluster=True)B.ii. Simple yaml templatingLet's make the yaml a little more interesting.apiVersion:batch/v1kind:Jobmetadata:name:countdown-templated-{{task_num}}namespace:<WRITE YOUR AIRFLOW NAMESPACE HERE>spec:template:metadata:name:countdownspec:containers:-name:counterimage:centos:7command:-"bin/bash"-"-c"-"{{command}}"restartPolicy:NeverAnd save this as~/airflow/kubernetes/job/countdown.yaml.tmplWe now have the fieldscommandandtask_numas variables in our yaml file.
Here's how our Dag looks now...withDAG('kubernetes_job_operator',default_args=default_args,description='KJO example DAG',schedule_interval=None,template_searchpath='/path/to/airflow/kubernetes/job'catchup=False)asdag:command='sleep 60; for i in 5 4 3 2 1 ; do echo $i ; done'task_num=1task_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown.yaml.tmpl',yaml_template_fields={'command':command,'task_num':task_num},in_cluster=True)B.iii. Multiple yaml templatesThis is very much up to you how you want your Jinja templates separated, if its valid yaml and valid Jinja, it will render and apply just fine...Heres an example use case.Create a 'header' template at~/airflow/kubernetes/job/countdown_header.yaml.tmplapiVersion:batch/v1kind:Jobmetadata:name:countdown-templated-separatednamespace:<WRITE YOUR AIRFLOW NAMESPACE HERE>{%block spec %}{%endblock %}Create a 'body' template at~/airflow/kubernetes/job/countdown_body.yaml.tmpl{%extends 'countdown_header.yaml.tmpl' %}{%block spec %}spec:template:metadata:name:countdownspec:containers:-name:counterimage:centos:7command:-"bin/bash"-"-c"-"{{command}}"restartPolicy:Never{%endblock %}Here's the Dag changes nowtask_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown_body.yaml.tmpl',yaml_template_fields={'command':command},in_cluster=True)In this situation it may be useful to have Airflow write out the rendered yaml file somewhere.task_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown_body.yaml.tmpl',yaml_template_fields={'command':command},yaml_write_path='/tmp',yaml_write_filename='rendered.yaml',# will be on the worker podin_cluster=True)It could be very useful to have an NFS to share the same filestore across pods for writing these rendered yaml files out.LoggingIf you're using Kubernetes you should have a logging solution of some sort to aggregate and provide searchability of all your logs. However, here are some use cases for forwarding the logs using the KJO.1. I just want a simple tail of the logs, I don't care about extra behavior configuration
2. I only want logs tailed out when the pods are in an end state; Completed, Errored
3. I want to specify how many lines are tailed out and/or how frequently its tailed outAdd 'tail_logs' to our task from above.task_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown_body.yaml.tmpl',yaml_template_fields={'command':command},in_cluster=True,tail_logs=True)If any tail_logs* parameter is set, 'tail_logs' does not need to be set.Configure the behavior of the log tailtask_1=KubernetesJobOperator(task_id='example_kubernetes_job_operator',yaml_file_name='countdown_body.yaml.tmpl',yaml_template_fields={'command':command},in_cluster=True,tail_logs_every=60,# secondstail_logs_line_count=100)This could get to be quite noisy so be mindful of your particular use case.Notes....We need to think about how to add PVC support. If a client's Task relies on a PVC being created, they need a way to add it to their DAG and have it created and deleted as a part of the Job flow. Maybe a KubernetesPVCOperator is better than a parameter solution.ContributingThis is a young project and not yet battle tested. Contributions, suggestions, etc. appreciated.