Package: dane-workflows
IntroductionPython library for creating "processing workflows" that useDANE environments, which in a nutshell offer, depending on the setup of each environment, an API for some kind of multi-media processing, e.g.:Automatic Speech RecognitionNamed Entity ExtractionComputer Vision algorithmsAny kind of Machine Learning algorithmThis Python library is however not limited to usingDANE, but cannot also be used to hook up any API that does something with generating certain data from certain input data.AchitectureThe following image illustrates the dane-workflows architecture:The following section details more about concepts illustrated in the image.Definition of a workflowA workflow is able to iteratively:obtain input/source data from aDataProvidersend it to aProcessingEnvironment(e.g. DANE environment)wait for the processing environment to complete its workobtain results from the processing environmentpass results to anExporter, which typically reconsiles the processed data with the source dataAs mentioned in the definition of a workflow, this Python library works with the following components/concepts:TaskSchedulerMain process that handles all the steps described in theDefinition of a workflowStatusHandlerKeeps track of the workflow status, esuring recovery after crashes. By default the status is persisted to a SQLite database file, using theSQLiteStatusHandlerbut other implementations can be made by subclassingStatusHandler.StatusMonitorNote: This component is currently implemented and not yet available.Runs on top of the StatusHandler database and visualises the overall progress of a workflow in a human-readable manner (e.g. show the % of successfully/failed processed items)DataProviderIteratively called by theTaskSchedulerto obtain a new batch of source data. No default implementations are available (yet), since there are many possible ways one would want to supply data to a system. Simply subclass fromDataProviderto have full control over your input flow.DataProcessingEnvironmentIteratively called by theTaskSchedulerto submit batches of data to an (external) processing environment. Also takes care of obtaining the output of finished processes from such an environment.This library contains a full implementation,DANEEnvironment, for interacting withDANE environments, but other environments/APIs can be supported by subclassing fromProcessingEnvironment.ExporterCalled by theTaskSchedulerwith output data from a processing environment. No default implementation is available (yet), since this is typically the most use-case sensitive part of any workflow, meaning you should decide what to do with the output data (by subclassingExporter).Getting startedPrerequisitesPython >= 3.8 <= 3.10PoetryInstallationInstall via pypi.org, using e.g.pip install dane-workflowslocal developmentRunpoetry install. After completion run:poetry shellTo test the contents of this repository works well, run:./scripts/check-project.shTODO finaliseUsageAfter installing dane-workflows in your local environment, you can run an example workflow with:python main.pyThis example script usesconfig-example.ymlto configure and run a workflow using the following implementations:DataProvider: ExampleDataProvider (with two dummy input documents)DataProcessingEnvironment: ExampleDataProcessingEnvironment (mocks processing environment)StatusHandler: SQLiteStatusHandler (writes output to./proc_stats/all_stats.db)Exporter: ExampleExporter (does nothing with results)To setup a workflow for your own purposes, consider the following:What data do I want to process?We've provided theExampleDataProviderto easily feed a workflow with a couple of files (via config.yml). This is mostly for testing out your workflow.Mostly likely you'll need to implement your ownDataProviderby subclassing it. This way you can e.g. load your input data from a database, spreadsheet or whatever else you need.Which processing environment will I use?Since this project is developed to at least interface with runningDANE environmentswe've providedDANEEnvironmentas a default implementation ofDataProcessingEnvironment.In case you'd like to call any other tool for processing your data, you're required to implement a subclass ofDataProcessingEnvironment.What I will I do with the output of the processing environment?After yourDataProcessingEnvironmenthas processed a batch of items from yourDataProvidertheTaskSchedulerhands over the output data to your subclass ofExporter.Since this is the most use-case dependant part of any workflow, we do not provide any useful default implementation.Note:ExampleExporteris only used as a placeholder for tests or dry runs.RoadmapImplement more advanced recoveryAdd example workflows (refer in README)Finalise initial READMEAddPython docstringSee theopen issuesfor a full list of proposed features, known issues and user questions.LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.ContactUse theissue trackerfor any questions concerning this repositoryProject Link:https://github.com/beeldengeluid/dane-workflowsCodemeta.json requirements:https://github.com/CLARIAH/clariah-plus/blob/main/requirements/software-metadata-requirements.md