Package: kannon
kannonKannon is a wrapper for thegokartlibrary that allows gokart tasks to be easily executed in a distributed and parallel manner on multiplekubernetesjobs.InstallKannon can be installed viapip.pipinstallkannonQuick StarterAn easy and self-contained tutorial can be found inhere!UsageIt is required for users to prepare following two scripts and copy them into a docker container:A script to start task pipeline on master job.A script for child jobs to run assigned tasks.Easy and self-contained quick starter will be available soon!Give custom permissions to the service accountIf you have a service accountmy-batchwithin a namespacemynamespace, the following permissions onjobsandjobs/statusshould be added.# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: mynamespace
  name: job-admin
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["batch"]
    resources: ["jobs/status"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: job-admin-binding
  namespace: mynamespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: job-admin
subjects:
  - namespace: mynamespace
    kind: ServiceAccount
    name: my-batchHow to define tasksThe way to define tasks is almost the same as for gokart.
All you have to do is replace the parent class of the task class you want to distribute withkannon.TaskOnBulletfromgokart.TaskOnKart.To get detailed explanations on how to define agokart.TaskOnKart, please refer togokart docs.Here is an example to define the following task pipeline with gokart and kannon.Two parts (Task B0 + Task C0) and (Task B1 + Task C1) can be run in a distributed and parallel manner.importgokartimportluigiimportkannonclassTaskA(gokart.TaskOnKart):param=luigi.Parameter()defrun(self):self.dump("A")# TODO: Change gokart.TaskOnKart -> kannon.TaskOnBulletclassTaskB(kannon.TaskOnBullet):param=luigi.Parameter()parent=gokart.TaskInstanceParameter()defrequires(self):return{"parent":self.parent}defrun(self):self.dump("B")# TODO: Change gokart.TaskOnKart -> kannon.TaskOnBulletclassTaskC(kannon.TaskOnBullet):param=luigi.Parameter()parent=gokart.TaskInstanceParameter()defrequires(self):return{"parent":self.parent}defrun(self):self.dump("C")classTaskD(gokart.TaskOnKart):param=luigi.Parameter()parent_0=gokart.TaskInstanceParameter()parent_1=gokart.TaskInstanceParameter()parent_2=gokart.TaskInstanceParameter()defrequires(self):return{"parent_0":self.parent_0,"parent_1":self.parent_1,"parent_2":self.parent_2}defrun(self):self.dump("D")task_a=TaskA(param="a")# b0 and b1 are executed in paralleltask_b0=TaskB(param="b0",parent=task_a)task_b1=TaskB(param="b1",parent=task_a)# c0 and c1 are executed in paralleltask_c0=TaskC(param="c0",parent=task_b0)task_c1=TaskC(param="c1",parent=task_b1)task_d=TaskD(param="d",parent_0=task_c0,parent_1=task_c1,parent_2=task_a)A script to run job on master jobSteps:Import module wheregokart.TaskOnKartandkannon.TaskOnBulletclasses are defined.Load luigi and k8s configs.Create a task instance.Create a template job for child jobs.RunKannon.build.""" This script runs on master job. """importloggingimportgokartimportluigifromkubernetesimportconfig,clientimportfire# TODO: Import task definition here!importexample_tasksfromkannonimportKannonlogging.basicConfig(level=logging.INFO)defmain(container_name:str,image_name:str,):# TODO: Load luigi config here!luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini")# TODO: Load kube config here!config.load_incluster_config()v1=client.BatchV1Api()# TODO: Create task instance here!task_root=[CREATETASKINSTANCEHERE]# TODO: create a template for child job from yaml file or using k8s API.template_job=utils.create_from_yaml(v1,PATH_TO_CHILD_JOB_YAML)# TODO: Run Kannon.build!Kannon(api_instance=v1,template_job=template_job,job_prefix="template-child-job",path_child_script="./run_child.py",).build(task_root)if__name__=="__main__":Fire.fire(main)Note thatjob.spec.template.containers[i].commandandjob.metadata.nameare replaced withinKannon.build.A script for child jobs to run assigned tasksFor now, it is required for users to prepare the following script. In the future release, it will not be required.Steps:Import module wheregokart.TaskOnKartandkannon.TaskOnBulletclasses are defined.Load luigi config.Parse a serialized task instance.Rungokart.build.""" This script requires to be defined by user. """importgokartimportluigiimportlogging# TODO: Import task definitions here!importexample_tasksimportfirelogging.basicConfig(level=logging.INFO)defmain(serialized_task:str):# TODO: Load luigi config here!luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini")# TODO: Parse a serialized gokart.TaskOnKart here!task:gokart.TaskOnKart=gokart.TaskInstanceParameter().parse(serialized_task)# TODO: Run gokart.build!gokart.build(task)if__name__=="__main__":fire.Fire(main)ThanksKannon is a wrapper for gokart. Thanks to gokart and dependent projects!gokart