Package: kernelpipes
Kaggle Kernel Pipelineskernelpipes is a Python 3.x project that supports running a series of Kaggle kernels through the Kaggle API.InstallationThe easiest way to install is via pip and PyPIpip install kernelpipesYou can also install kernelpipes directly from GitHub. Using a Python Virtual Environment is recommended.pip install git+https://github.com/neuml/kernelpipesPython 3.6+ is supportedThis package assumes a Kaggle token has been generated and installed. See thekaggle-apiAPI credentials section for more information.ExampleThe following YAML script is an example pipeline script. Pipelines require a name and a series of steps to execute.# Pipeline namename:CORD-19 Papers# Pipeline execution stepssteps:-kernel:davidmezzetti/cord-19-papers-status:2.5m-output:davidmezzetti/cord-19-papersThe above pipeline executes a kernel named cord-19-papers, checking every 2.5 minutes for completion. Once the kernel is complete, output files are downloaded locally.Below is a more complex example demonstrating scheduling and multiple status steps.# Pipeline namename:CORD-19 Pipeline# Schedule job to run every 15 minutes between 10pm - 12am local timeschedule:"*/1522-23***"# Pipeline execution stepssteps:-check:allen-institute-for-ai/CORD-19-research-challenge-kernel:davidmezzetti/cord-19-article-entry-dates-status:1m-kernel:davidmezzetti/cord-19-analysis-with-sentence-embeddings-status:15m-kernel:davidmezzetti/cord-19-patient-descriptions-kernel:davidmezzetti/cord-19-risk-factors-status:2.5m-kernel:davidmezzetti/cord-19-report-builder-kernel:davidmezzetti/cord-19-study-metadata-export-kernel:davidmezzetti/cord-19-task-csv-exports-status:2.5mThe script above is named "CORD-19 Pipeline" and has a series of sequential steps. Kernel steps will execute a kernel and status steps will poll the status of preceding kernel steps, waiting for completion. For example, the first status command above will run a Status API call every minute checking for completion. If any of the steps return an error status, the pipeline is halted and the program exits.Kernels are started until they reach a status step. In the example above, the last status call executes cord-19-report-builder, cord-19-study-metadata-export and cord-19-task-csv-exports in parallel and waits for all 3 to complete.Assuming the YAML file above is stored at pipeline.yml, the pipeline can be executed via:kernelpipespipeline.ymlNote: When running on Windows, it may be necessary to set an environment variable to force UTF-8 file operations before running the command above e.g. 'set PYTHONUTF8=1'Basic pipeline configurationnamename:<pipeline name>Required field, names the pipelinescheduleschedule:cron stringOptional field to enable running jobs through a scheduler. System cron can be used in place of this, depending on preference. One advantage of this method is that new jobs won't be spawned while a prior job is running. For example if a job is scheduled to run every hour and a run takes 1.5 hours, it will skip the 2nd run and start again on the 3rd hour.Stepscheckcheck:/kaggle/dataset/pathAllows conditionally running a pipeline based on dataset update status. Retrieves dataset metadata and compares the latest version against the last run version and only allows processing to proceed if the dataset has been updated. If there is no local metadata for the dataset, the run will proceed.kernelkernel:/kaggle/kernel/pathReturns the kernel specified at /kaggle/kernel/pathoutputoutput:/kaggle/kernel/pathDownloads kernel output files for the kernel specified at /kaggle/kernel/pathstatusstatus:<seconds|s|m|h>Checks the status of preceding kernel steps at the specified duration.Example durations: 10 for 10 seconds, 30s for 30 seconds, 1m for 1 minute and 1h for 1 hour.