Package: dkist-processing-core
OverviewThe dkist-processing-core package provides an abstraction layer between the dkist data processing code, the workflow
engine that supports it (Airflow), and the logging infrastructure. By providing the abstraction layer to Airflow
specifically a versioning system is implemented.There are 4 main entities which implement the abstraction which are described below.Task: The Task defines the interface used by a processing pipeline for a step in a workflow.
By conforming to this interface (i.e. subclassing) the processing pipelines can remain agnostic of how the tasks will ultimately be run.
The Task additionally implements some methods that should be global for all dkist processing tasks based on the infrastructure it will run on (e.g. application performance monitoring infrastructure).Node: The job of the Node is to translate a Task into code that can instantiate that task.
Instantiations of a Task can vary depending on the target environment e.g. a virtual environment with a BashOperator for Airflow vs. straight python for a notebook.Workflow: The Workflow defines the interface used by the processing pipeline to chain tasks together in a directed graph.
The Workflow transforms this graph into the workflow engine format by providing any wrapping boilerplate, task ordering, and selecting the appropriate Node instantiation.Build Utils: The Build Utils are the capstone layer which aims to ease the transformation process for multiple workflows at a time during a processing pipeline’s build process.UsageThe Workflow and Task are the primary objects used by client libraries.
The Task is used as a base class and the subclass must at a minimum implement run.
A Workflow is used to give the tasks an order of execution and a name for the flow.fromdkist_processing_coreimportTaskBasefromdkist_processing_coreimportWorkflow# Task definitionsclassMyTask1(TaskBase):defrun(self):print("Running MyTask1")classMyTask2(TaskBase):defrun(self):print("Running MyTask2")# Workflow definition# MyTask1 -> MyTask2w=Workflow(process_category="My",process_name="Workflow",workflow_package=__package__,workflow_version="dev")w.add_node(MyTask1,upstreams=None)w.add_node(MyTask2,upstreams=MyTask1)Using dkist-processing-core for data processing with Airflow involves a project structure and
build process that results in code artifacts deployed toPyPIand a
zip of workflow artifacts deployed to artifactory.The client dkist data processing libraries should implement a structure and build pipeline usingdkist-processing-testas an example.  The build pipelines for a client repo can leverage thebuild_utilsfor test and export.Specifically for Airflow, the resulting deployment has the versioned workflow artifacts all available to the scheduler
and the versioned code artifacts available to workers for task executionBuilddkist-processing-core is built usingbitbucket-pipelinesDeploymentdkist-processing-core is deployed toPyPIEnvironment VariablesVariableDescriptionTypeDefaultBUILD_VERSIONBuild/Export pipelines only.  This is the value that will be appended to all artifacts and represents their unique versionSTRdevMESH_CONFIGProvides the dkistdc cloud mesh configuration.  Specifically the location of the message brokerJSONISB_USERNAMEMessage broker user nameSTRISB_PASSWORDMessage broker passwordSTRDevelopmentgitclonegit@bitbucket.org:dkistdc/dkist-processing-core.gitcddkist-processing-corepre-commitinstallpipinstall-e.[test]pytest-v--covdkist_processing_coreChangelogWhen you makeanychange to this repository itMUSTbe accompanied by a changelog file.
The changelog for this repository uses thetowncrierpackage.
Entries in the changelog for the next release are added as individual files (one per change) to thechangelog/directory.Writing a Changelog EntryA changelog entry accompanying a change should be added to thechangelog/directory.
The name of a file in this directory follows a specific template:<PULL REQUEST NUMBER>.<TYPE>[.<COUNTER>].rstThe fields have the following meanings:<PULL REQUEST NUMBER>: This is the number of the pull request, so people can jump from the changelog entry to the diff on BitBucket.<TYPE>: This is the type of the change and must be one of the values described below.<COUNTER>: This is an optional field, if you make more than one change of the same type you can append a counter to the subsequent changes, i.e.100.bugfix.rstand100.bugfix.1.rstfor two bugfix changes in the same PR.The list of possible types is defined the the towncrier section ofpyproject.toml, the types are:feature: This change is a new code feature.bugfix: This is a change which fixes a bug.doc: A documentation change.removal: A deprecation or removal of public API.misc: Any small change which doesn’t fit anywhere else, such as a change to the package infrastructure.Rendering the Changelog at Release TimeWhen you are about to tag a release first you must runtowncrierto render the changelog.
The steps for this are as follows:Runtowncrier build –version vx.y.zusing the version number you want to tag.Agree to have towncrier remove the fragments.Add and commit your changes.Tag the release.NOTE:If you forget to add a Changelog entry to a tagged release (either manually or automatically withtowncrier)
then the Bitbucket pipeline will fail. To be able to use the same tag you must delete it locally and on the remote branch:# First, actually update the CHANGELOG and commit the updategitcommit# Delete tagsgittag-dvWHATEVER.THE.VERSIONgitpush--deleteoriginvWHATEVER.THE.VERSION# Re-tag with the same versiongittagvWHATEVER.THE.VERSIONgitpush--tagsoriginmain