Package: data-linter
Data LinterA python package to to allow automatic validation of data as part of a Data Engineering pipeline. It is designed to read in and validate tabular data against a given schema for the data. The schemas provided adhere toour metadata schemas standardsfor data. This package can also be used to manage movement of data from a landing area (s3 or locally) to a new location based on the result of the validation.This package uses own lightweight pandas dataframe operations to run simple validation tests on the columns based on the datatype and additional tags in the metadata. Utilisesmojap-arrow-pd-parserfor reading data.Installationpipinstalldata_linterUsageThis package takes ayamlbased config file written by the user (see example below), and validates data in the specified s3 folder path against specified metadata. If the data conforms to the metadata, it is moved to the specified s3 folder path for the next step in the pipeline (note can also provide local paths). Any failed checks are passed to a separate location for testing. The package also generates logs to allow you to explore issues in more detail.Simple UseTo run the validation, at most simple you can use the following:In Python:fromdata_linter.validationimportrun_validationconfig_path="config.yaml"run_validation(config_path)Via command line:data_linter--config_pathconfig.yamlExample config fileland-base-path:s3://testing-bucket/land/# Where to get the data fromfail-base-path:s3://testing-bucket/fail/# Where to write the data if failedpass-base-path:s3://testing-bucket/pass/# Where to write the data if passedlog-base-path:s3://testing-bucket/logs/# Where to write logscompress-data:true# Compress data when moving elsewhere (only applicable from CSV/JSON)remove-tables-on-pass:true# Delete the tables in land if validation passesall-must-pass:true# Only move data if all tables have passedfail-unknown-files:exceptions:-additional_file.txt-another_additional_file.txtvalidator-engine-params:log_verbosity:2# how many samples of incorrect data to include in logs, 0 means all. Pandas validator only.# Tables to validatetables:table1:required:true# Does the table have to existpattern:null# Assumes file is called table1 (same as key)metadata:meta_data/table1.json# local path to metadatalog_verbosity:5# overrides the validator-engine-params log verbosity for this table onlyallow-missing-cols:False# allows there to be data in the metadata but not actual datatable2:required:truepattern:^table2metadata:meta_data/table2.jsonrow-limit:10000# for big tables - only take the first x rowsallow-unexpected-data:True# allows there to be columns present in the data but not the metadataunexpected data and missing columnsTo allow flexibilty in what is validated in the data, the parametersallow-unexpected-dataandallow-missing-colshas been added. These can be described neatly in one diagram:You can also run the validator as part of a python script, where you might want to dynamically generate your config:fromdata_linter.validationimportrun_validationbase_config={"land-base-path":"s3://my-bucket/land/","fail-base-path":"s3://my-bucket/fail/","pass-base-path":"s3://my-bucket/pass/","log-base-path":"s3://my-bucket/log/","compress-data":False,"remove-tables-on-pass":False,"all-must-pass":False,"tables":{}}defget_table_config(table_name):d={"required":False,"expect-header":True,"metadata":f"metadata/{table_name}.json","pattern":r"^{}\.jsonl$".format(table_name),"headers-ignore-case":True,}returndfortablein["table1","table2"]:base_config["tables"][table_name]=get_table_config(table_name)run_validation(base_config)# Then watch that log go...Validating a single fileWithout all the bells and whistlesIf you do not needdata_linterto match files to a specified config, log the process and then move data around based on the outcome of the validation you can just use the validators themselves:# Example using simple pandas validatior (without added data_linter features)importjsonfromdata_linter.validatorsimportPandasValidatorfilepath="tests/data/end_to_end1/land/table1.csv"table_params={"expect-header":True}withopen("tests/data/end_to_end1/meta_data/table1.json")asf:metadata=json.load(f)pv=PandasValidator(filepath,table_params,metadata)pv.read_data_and_validate()pv.valid# True (data in table1.csv is valid against metadata)pv.response.get_result()# Returns dict of all tests ran against data# The response object of the PandasValidator in itself, and has it's own functionspv.response.get_names_of_column_failures()# [], i.e. no cols failedParallel RunningData Linter can also work in parallel to trigger multiple validations at once (only supports use of S3 atm). An example below:In this scenario we use the parallisation process to init the process split the job into 4 validators and then run the closedown.The init stagesplits the config into 4 chunks, based on the file size of the data sitting in the specified land path. It split configs are written to a temporary path in S3 for each validator to pick up and run in parallel.The validator stagecan be ran in parallel (for simplicity they are run sequentially in the example below). Each validator will take the config in the temp folder path and process the files given in that subsetting config.The closedown stagewill take all the logs all validator runs, conbine them then move the data based on the validators results. It will then finally clean up the temp folder.# Simple example running DL with multiple validators (in this case 4)# [init] -> [validator]x4 -> [closedown]importyamlfromdata_linterimportvalidationfromdataengineeringutils3.s3importget_filepaths_from_s3_foldersimple_yaml_config="""land-base-path: s3://land/fail-base-path: s3://fail/pass-base-path: s3://pass/log-base-path: s3://log/compress-data: falseremove-tables-on-pass: trueall-must-pass: true# Tables to validatetables:table1:required: truemetadata: tests/data/end_to_end1/meta_data/table1.jsonexpect-header: truetable2:required: truepattern: ^table2metadata: tests/data/end_to_end1/meta_data/table2.json"""test_folder="tests/data/end_to_end1/land/"config=yaml.safe_load(simple_yaml_config)# Init stagevalidation.para_run_init(4,config)# Validation stage (although ran sequentially this can be ran in parallel)foriinrange(4):validation.para_run_validation(i,config)# Closedown stagevalidation.para_collect_all_status(config)validation.para_collect_all_logs(config)There are more parallelisation examples, which can be found in thetest_simple_examples.py test moduleValidatorsPandasThis is the default validator used by data_linter as of the version 5 release.Dealing with timestamps and datesTimestamps are always a pain to deal with especially when using different file types. The Pandas Validator has tried to keep true to the file types based on the tests it runs.If the file type stores date/timestamp information as a string (i.e. CSV and JSONL) then the pandas Validator will read in the timestamp / date columns as strings. It will then apply validation tests against those columns checking if the string representation of the dates in the data is a valid date. For timestamp and date types these tests assume ISO standard string representation%Y-%m-%d %H:%M:%Sand%Y-%m-%d. If your timestamp/date types are comming in as strings that do not conform to the ISO standard format then you can provide you column in the metadata with adatetime_formatproperty that specifies the exected format e.g...."columns":[{"name":"date_in_uk","type":"date64","datetime_format":"%d/%m/%Y"},...Often you might recieve data that is exported from a system that might encode your timestamp as a date but is written to a format that encodes the data as a timestamp. In this scenario you would expect your dates (in a str timestamp format) to always have a time component of00:00:00. You can also use data_linter to validate this by specifing the datetime_format of your column as the expected timestamp in format but still specify that the data type is a date e.g...."columns":[{"name":"date_in_uk","type":"date64","datetime_format":"%d/%m/%Y 00:00:00"},...In the above data_linter will attempt to fist parse the column with the specifieddatetime_formatand then as the column type is date it will check that it it truely a date (and not have a time component).If the file_format isparquetthen timestamps are encoded in the filetype and there are just read in as is. Currently data_linter doesn't support minimum and maximum tests for timestamps/dates and also does not currently have tests for time types.Process DiagramHow logic worksHow to updateWe have tests that run on the current state of thepoetry.lockfile (i.e. the current dependencies). We also run tests based on the most up to date dependencies allowed inpyproject.toml. This allows us to see if there will be any issues when updating dependences. These can be run locally in thetestsfolder.When updating this package, make sure to change the version number inpyproject.tomland describe the change in CHANGELOG.md.If you have changed any dependencies inpyproject.toml, runpoetry updateto updatepoetry.lock.Once you have created a release in GitHub, to publish the latest version to PyPI, run:poetrybuild
poetrypublish-u<username>Here, you should substitute  for your PyPI username. In order to publish to PyPI, you must be an owner of the project.