Package: data-validation-framework
Data Validation FrameworkThis project provides simple tools to create data validation workflows.
The workflows are based on theluigilibrary.The main objective of this framework is to gather in a same place both the specifications that the
data must follow and the code that actually tests the data. This avoids having multiple documents
to store the specifications and a repository to store the code.InstallationThis package should be installed using pip:pipinstalldata-validation-frameworkUsageBuilding a workflowBuilding a new workflow is simple, as you can see in the following example:importluigiimportdata_validation_frameworkasdvfclassValidationTask1(dvf.task.ElementValidationTask):"""Use the class docstring to describe the specifications of the ValidationTask1."""output_columns={"col_name":None}@staticmethoddefvalidation_function(row,output_path,*args,**kwargs):# Return the validation result for one row of the datasetifrow["col_name"]<=10:returndvf.result.ValidationResult(is_valid=True)else:returndvf.result.ValidationResult(is_valid=False,ret_code=1,comment="The value should always be <= 10")defexternal_validation_function(df,output_path,*args,**kwargs):# Update the dataset inplace here by setting values to the 'is_valid' column.# The 'ret_code' and 'comment' values are optional, they will be added to the report# in order to help the user to understand why the dataset did not pass the validation.# We can use the value from kwargs["param_value"] here.ifint(kwargs["param_value"])<=10:df["is_valid"]=Trueelse:df["is_valid"]=Falsedf["ret_code"]=1df["comment"]="The value should always be <= 10"classValidationTask2(dvf.task.SetValidationTask):"""In some cases you might want to keep the docstring to describe what a developerneeds to know, not the end-user. In this case, you can use the ``__specifications__``attribute to store the specifications."""a_parameter=luigi.Parameter()__specifications__="""Use the __specifications__ to describe the specifications of theValidationTask2."""definputs(self):return{ValidationTask1:{"col_name":"new_col_name_in_current_task"}}defkwargs(self):return{"param_value":self.a_parameter}validation_function=staticmethod(external_validation_function)classValidationWorkflow(dvf.task.ValidationWorkflow):"""Use the global workflow specifications to give general context to the end-user."""definputs(self):return{ValidationTask1:{},ValidationTask2:{},}Where theValidationWorkflowclass only defines the sub-tasks that should be called for the
validation. The sub-tasks can be either advf.task.ElementValidationTaskor advf.task.SetValidationTask. In both cases, you can define relations between these sub-tasks
since one could need the result of another one to run properly. This is defined in two steps:in the required task, aoutput_columnsattribute should be defined so that the next tasks
can know what data is available, as shown in the previous example for theValidationTask1.in the task that requires another task, ainputsmethod should be defined, as shown in the
previous example for theValidationTask2.The sub-classes ofdvf.task.ElementValidationTaskshould return advf.result.ValidationResultobject. The sub-classes ofdvf.task.SetValidationTaskshould
return aPandas.DataFrameobject with at least the following columns["is_valid", "ret_code", "comment", "exception"]and with the same index as the input dataset.Generate the specifications of a workflowThe specifications that the data should follow can be generated with the following luigi command:luigi--moduletest_validationValidationWorkflow--log-levelINFO--local-scheduler--result-pathout--ValidationTask2-a-parameter15--specifications-onlyRunning a workflowThe workflow can be run with the following luigi command (note that the moduletest_validationmust be available in yoursys.path):luigi--moduletest_validationValidationWorkflow--log-levelINFO--local-scheduler--dataset-dfdataset.csv--result-pathout--ValidationTask2-a-parameter15This workflow will generate the following files:out/report_ValidationWorkflow.pdf: the PDF validation report.out/ValidationTask1/report.csv: The CSV containing the validity values of the taskValidationTask1.out/ValidationTask2/report.csv: The CSV containing the validity values of the taskValidationTask2.out/ValidationWorkflow/report.csv: The CSV containing the validity values of the complete
workflow... note::As any `luigi <https://luigi.readthedocs.io/en/stable>`_ workflow, the values can be stored
into a `luigi.cfg` file instead of being passed to the CLI.Advanced featuresRequire a regular Luigi taskIn some cases, one want to execute a regular Luigi task in a validation workflow. In this case, it
is possible to use theextra_requires()method to pass these extra requirements. In the
validation task it is then possible to get the targets of these extra requirements using theextra_input()method.classTestTaskA(luigi.Task):defrun(self):# Do something and write the 'target.file'defoutput(self):returntarget.OutputLocalTarget("target.file")classTestTaskB(task.SetValidationTask):output_columns={"extra_target_path":None}defkwargs(self):return{"extra_task_target_path":self.extra_input().path}defextra_requires(self):returnTestTaskA()@staticmethoddefvalidation_function(df,output_path,*args,**kwargs):df["is_valid"]=Truedf["extra_target_path"]=kwargs["extra_task_target_path"]Funding & AcknowledgmentThe development of this software was supported by funding to the Blue Brain Project, a research
center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH
Board of the Swiss Federal Institutes of Technology.For license and authors, seeLICENSE.txtandAUTHORS.mdrespectively.Copyright © 2022-2023 Blue Brain Project/EPFL