Package: goodtables
goodtables-pyGoodtables is a framework to validate tabular data. It can check the structure
of your data (e.g. all rows have the same number of columns), and its contents
(e.g. all dates are valid).[Important Notice]goodtableswas renamed tofrictionlesssince version 3. The framework got various improvements and was extended to be a complete data solution. The change in not breaking for the existing software so no actions are required. Please read theMigration Guideto start working with Frictionless for Python.we continue to bug-fixgoodtables@2.xin thisbranchas well as it's available onPyPias it was beforeplease note thatfrictionless@3.xversion's API, we're working on at the moment, is not stablewe will releasefrictionless@4.xby the end of 2020 to be the first SemVer/stable versionFeaturesStructural checks: Ensure that there are no empty rows, no blank headers, etc.Content checks: Ensure that the values have the correct types ("string", "number", "date", etc.), that their format is valid ("string must be an e-mail"), and that they respect the constraints ("age must be a number greater than 18").Support for multiple tabular formats: CSV, Excel files, LibreOffice, Data Package, etc.Parallelized validations for multi-table datasetsCommand line interfaceContentsGetting StartedInstallingRunning on CLIRunning on PythonDocumentationReportChecksPresetsData Quality ErrorsFrequently Asked QuestionsAPI ReferenceclivalidatepresetcheckErrorspecGoodtablesExceptionContributingChangelogGetting StartedFor faster goodtables-combatible Pandas dataframes validation take a look athttps://github.com/ezwelty/goodtables-pandas-pyInstallingpip install goodtables
pip install goodtables[ods]  # If you need LibreOffice's ODS file supportRunning on CLIgoodtables data.csvUsegoodtables --helpto see the different options.Running on Pythonfromgoodtablesimportvalidatereport=validate('invalid.csv')report['valid']# falsereport['table-count']# 1report['error-count']# 3report['tables'][0]['valid']# falsereport['tables'][0]['source']# 'invalid.csv'report['tables'][0]['errors'][0]['code']# 'blank-header'You can read a more in depth explanation on using goodtables with Python on
thedeveloper documentationsection. Check also
theexamplesfolder for other examples.DocumentationGoodtables validates your tabular dataset to find structural and content
errors. Consider you have a file namedinvalid.csv. Let's validate it:report=validate('invalid.csv')We could also pass a remote URI instead of a local path. It supports CSV, XLS,
XLSX, ODS, JSON, and all other formats supported by thetabulatorlibrary.ReportThe validation report follows the JSON Schema defined ongoodtables/schemas/report.json.The output of thevalidate()method is a report dictionary. It includes
information if the data was valid, count of errors, list of table reports, which
individual checks failed, etc. A report will be looking like this:{"time":0.009,"error-count":1,"warnings":["Table \"data/invalid.csv\" inspection has reached 1 error(s) limit"],"preset":"table","valid":false,"tables":[{"errors":[{"row-number":null,"message":"Header in column 3 is blank","row":null,"column-number":3,"code":"blank-header"}],"error-count":1,"headers":["id","name","","name"],"scheme":"file","row-count":2,"valid":false,"encoding":"utf-8","time":0.007,"schema":null,"format":"csv","source":"data/invalid"}],"table-count":1}The errors are divided in one of the following categories:source- data can't be loaded or parsedstructure- general tabular errors like duplicate headersschema- error of checks againstTable Schemacustom- custom checks errorsChecksCheck is a main validation actor in goodtables. The list of enabled checks can
be changed usingchecksandskip_checksarguments. Let's explore the options
on an example:report=validate('data.csv')# by default structure and schema (if available) checksreport=validate('data.csv',checks=['structure'])# only structure checksreport=validate('data.csv',checks=['schema'])# only schema (if available) checksreport=validate('data.csv',checks=['bad-headers'])# check only 'bad-headers'report=validate('data.csv',skip_checks=['bad-headers'])# exclude 'bad-headers'By default a dataset will be validated against all available Data Quality Spec
errors. Some checks can be unavailable for validation. For example, if the
schema isn't provided, only thestructurechecks will be done.PresetsGoodtables support different formats of tabular datasets. They're called
presets. A tabular dataset is some data that can be split in a list of data
tables, as:We can change the preset using thepresetargument forvalidate(). By
default, it'll be inferred from the source, falling back totable. To validate
adata package, we can do:report=validate('datapackage.json')# implicit presetreport=validate('datapackage.json',preset='datapackage')# explicit presetThis will validate all tabular resources in the datapackage.It's also possible to validate a list of files using the "nested" preset. To do
so, the first argument tovalidate()should be a list of dictionaries, where
each key in the dictionary is named after a parameter onvalidate(). For example:report=validate([{'source':'data1.csv'},{'source':'data2.csv'}])# implicit presetreport=validate([{'source':'data1.csv'},{'source':'data2.csv'}],preset='nested')# explicit presetIs similar to:report_data1=validate('data1.csv')report_data2=validate('data2.csv')The difference is that goodtables validates multiple tables in parallel, so
calling using the "nested" preset should run faster.Data Quality ErrorsBase report errors are standardized and described inData Quality Spec.Source errorsThe basic checks can't be disabled, as they deal with goodtables being able to read the files.checkdescriptionio-errorData reading error because of IO error.http-errorData reading error because of HTTP error.source-errorData reading error because of not supported or inconsistent contents.scheme-errorData reading error because of incorrect scheme.format-errorData reading error because of incorrect format.encoding-errorData reading error because of an encoding problem.Structure errorsThese checks validate that the structure of the file are valid.checkdescriptionblank-headerThere is a blank header name. All cells in the header row must have a value.duplicate-headerThere are multiple columns with the same name. All column names must be unique.blank-rowRows must have at least one non-blank cell.duplicate-rowRows can't be duplicated.extra-valueA row has more columns than the header.missing-valueA row has less columns than the header.Schema errorsThese checks validate the contents of the file. To use them, you need to pass aTable Schema. If you don't have a schema, goodtables can infer it if you use theinfer_schemaoption.If your schema only covers part of the data, you can use theinfer_fieldsto infer the remaining fields.Lastly, if the order of the fields in the data is different than in your schema, enable theorder_fieldsoption.checkdescriptionschema-errorSchema is not valid.non-matching-headerThe header's name in the schema is different from what's in the data.extra-headerThe data contains a header not defined in the schema.missing-headerThe data doesn't contain a header defined in the schema.type-or-format-errorThe value can’t be cast based on the schema type and format for this field.required-constraintThis field is a required field, but it contains no value.pattern-constraintThis field value's should conform to the defined pattern.unique-constraintThis field is a unique field but it contains a value that has been used in another row.enumerable-constraintThis field value should be equal to one of the values in the enumeration constraint.minimum-constraintThis field value should be greater or equal than constraint value.maximum-constraintThis field value should be less or equal than constraint value.minimum-length-constraintA length of this field value should be greater or equal than schema constraint value.maximum-length-constraintA length of this field value should be less or equal than schema constraint value.Custom errorscheckdescriptionblacklisted-valueEnsure there are no cells with the blacklisted values.deviated-valueEnsure numbers are within a number of standard deviations from the average.foreign-keyEnsure foreign keys are valid within a data packagesequential-valueEnsure numbers are sequential.truncated-valueDetect values that were potentially truncated.custom-constraintDefines a constraint based on the values of other columns (e.g.value * quantity == total).blacklisted-valueSometimes we have to check for some values we don't want to have in out dataset. It accepts following options:optiontypedescriptioncolumnint/strColumn number or nameblacklistlist of strList of blacklisted valuesConsider the following CSV file:id,name
1,John
2,bug
3,bad
5,AlexLet's check that thenamecolumn doesn't contain rows withbugorbad:fromgoodtablesimportvalidatereport=validate('data.csv',checks=[{'blacklisted-value':{'column':'name','blacklist':['bug','bad']}},])# error on row 3 with code "blacklisted-value"# error on row 4 with code "blacklisted-value"deviated-valueThis check helps to find outlines in a column containing positive numbers. It accepts following options:optiontypedescriptioncolumnint/strColumn number or nameaveragestrAverage type, either "mean", "median" or "mode"intervalintValues must be inside rangeaverage ± standard deviation * intervalConsider the following CSV file:temperature
1
-2
7
0
1
2
5
-4
100
8
3We usemedianto get an average of the column values and allow interval of 3 standard deviations. For our case median is2.0and standard deviation is29.73so all valid values must be inside the[-87.19, 91.19]interval.report=validate('data.csv',checks=[{'deviated-value':{'column':'temperature','average':'median','interval':3}},])# error on row 10 with code "deviated-value"foreign-keyWe support here relative paths. It MUST be used only for trusted data sources.This check validate foreign keys within a data package. Consider we have a data package defined below:DESCRIPTOR={'resources':[{'name':'cities','data':[['id','name','next_id'],[1,'london',2],[2,'paris',3],[3,'rome',4],# [4, 'rio', None],],'schema':{'fields':[{'name':'id','type':'integer'},{'name':'name','type':'string'},{'name':'next_id','type':'integer'},],'foreignKeys':[{'fields':'next_id','reference':{'resource':'','fields':'id'},},{'fields':'id','reference':{'resource':'people','fields':'label'},},],},},{'name':'people','data':[['label','population'],[1,8],[2,2],# [3, 3],# [4, 6],],},],}Runninggoodtableson it will raise a fewforeign-keyerrors because we have commented some rows in the data package's data:report=validate(DESCRIPTOR,checks=['structure','schema','foreign-key'])print(report){'error-count': 2,
 'preset': 'datapackage',
 'table-count': 2,
 'tables': [{'datapackage': '...',
             'error-count': 2,
             'errors': [{'code': 'foreign-key',
                         'message': 'Foreign key "[\'next_id\']" violation in '
                                    'row 4',
                         'message-data': {'fields': ['next_id']},
                         'row-number': 4},
                        {'code': 'foreign-key',
                         'message': 'Foreign key "[\'id\']" violation in row 4',
                         'message-data': {'fields': ['id']},
                         'row-number': 4}],
             'format': 'inline',
             'headers': ['id', 'name', 'next_id'],
             'resource-name': 'cities',
             'row-count': 4,
             'schema': 'table-schema',
             'source': 'inline',
             'time': 0.031,
             'valid': False},
            {'datapackage': '...',
             'error-count': 0,
             'errors': [],
             'format': 'inline',
             'headers': ['label', 'population'],
             'resource-name': 'people',
             'row-count': 3,
             'source': 'inline',
             'time': 0.038,
             'valid': True}],
 'time': 0.117,
 'valid': False,
 'warnings': []}It experimetally supports external resource checks, for example, for aforeignKeydefinition like these:{"package":"../people/datapackage.json","resource":"people","fields":"label"}{"package":"http:/example.com/datapackage.json","resource":"people","fields":"label"}sequential-valueThis checks is for pretty common case when a column should have integers that sequentially increment.  It accepts following options:optiontypedescriptioncolumnint/strColumn number or nameConsider the following CSV file:id,name
1,one
2,two
3,three
5,fiveLet's check if theidcolumn contains sequential integers:fromgoodtablesimportvalidatereport=validate('data.csv',checks=[{'sequential-value':{'column':'id'}},])# error on row 5 with code "sequential-value"truncated-valueSome database or spreadsheet software (like MySQL or Excel) could cutoff values on saving. There are some well-known heuristics to find this bad values. Seehttps://github.com/propublica/guides/blob/master/data-bulletproofing.mdfor more detailed information.Consider the following CSV file:id,amount,comment
1,14000000,good
2,2147483647,bad
3,32767,bad
4,234234234,bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbadTo detect all probably truncated values we could usetruncated-valuecheck:report=validate('data.csv',checks=['truncated-value',])# error on row 3 with code "truncated-value"# error on row 4 with code "truncated-value"# error on row 5 with code "truncated-value"custom-constraintWith Table Schema we could create constraints for an individual field but sometimes it's not enough. With a custom constraint check every row could be checked against given limited python expression in which variable names resolve to column values. See list ofavailable operators. It accepts following options:constraint (str)Constraint definition (e.g.col1 + col2 == col3)Consider csv file like this:id,name,salary,bonus
1,Alex,1000,200
2,Sam,2500,500
3,Ray,1350,500
4,John,5000,1000Let's say our business rule is to be shy on bonuses:report=validate('data.csv',checks=[{'custom-constraint':{'constraint':'salary > bonus * 4'}},])# error on row 4 with code "custom-constraint"Frequently Asked QuestionsHow can I add a new custom check?To create a custom check user could use acheckdecorator. This way the builtin check could be overridden (use the spec error code likeduplicate-row) or could be added a check for a custom error (usetype,contextandpositionarguments):fromgoodtablesimportvalidate,check,Error@check('custom-check',type='custom',context='body')defcustom_check(cells):errors=[]forcellincells:message='Custom error on column{column_number}and row{row_number}'error=Error('custom-error',cell,message)errors.append(error)returnerrorsreport=validate('data.csv',checks=['custom-check'])Recommended steps:let's discuss in comment proposed checks firstselect name for a new check likepossible-noise-textcopyhttps://github.com/frictionlessdata/goodtables-py/blob/master/goodtables/contrib/checks/blacklisted_value.pyto new check moduleadd new check module to configuration -https://github.com/frictionlessdata/goodtables-py/blob/master/goodtables/config.pywrite actual code for the new checkwrite tests and readme for the new checkHow can I add support for a new tabular file type?To create a custom preset user could use apresetdecorator. This way the builtin preset could be overridden or could be added a custom preset.fromtabulatorimportStreamfromtableschemaimportSchemafromgoodtablesimportvalidate@preset('custom-preset')defcustom_preset(source,**options):warnings=[]tables=[]fortableinsource:try:tables.append({'source':str(source),'stream':Stream(...),'schema':Schema(...),'extra':{...},})exceptException:warnings.append('Warning message')returnwarnings,tablesreport=validate(source,preset='custom-preset')For now this documentation section is incomplete. Please see builtin presets to learn more about the dataset extraction protocol.API Referenceclicli()Command-line interfaceUsage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  validate*  Validate tabular files (default).
  init       Init data package from list of files.validatevalidate(source,**options)Validates a source file and returns a report.Argumentssource (Union[str, Dict, List[Dict], IO]):
The source to be validated.
It can be a local file path, URL, dict, list of dicts, or a
file-like object. If it's a list of dicts and thepresetis
"nested", each of the dict key's will be used as if it was passed
as a keyword argument to this method.The file can be a CSV, XLS, JSON, and any other format supported by
  `tabulator`_.checks (List[str]):
List of checks names to be enabled. They can be
individual check names (e.g.blank-headers), or check types (e.g.structure).skip_checks (List[str]):
List of checks names to be skipped. They can
be individual check names (e.g.blank-headers), or check types
(e.g.structure).infer_schema (bool):
Infer schema if one wasn't passed as an argument.infer_fields (bool):
Infer schema for columns not present in the received schema.order_fields (bool):
Order source columns based on schema fields order.
This is useful when you don't want to validate that the data
columns' order is the same as the schema's.error_limit (int):
Stop validation if the number of errors per table exceeds this value.table_limit (int):
Maximum number of tables to validate.row_limit (int):
Maximum number of rows to validate.preset (str):
Dataset type could betable(default),datapackage,nestedor custom. Usually, the preset can be inferred from the
source, so you don't need to define it.Any (Any):
Any additional arguments not defined here will be passed on,
depending on the chosenpreset. If thepresetistable, the
extra arguments will be passed on totabulator, if it isdatapackage, they will be passed on to thedatapackageconstructor.RaisesGoodtablesException: Raised on any non-tabular error.Returnsdict: The validation report.presetpreset(name)Register a custom preset (decorator)Example@preset('custom-preset')defcustom_preset(source,**options):# ...Argumentsname (str): preset namecheckcheck(name,type=None,context=None,position=None)Register a custom check (decorator)Example@check('custom-check',type='custom',context='body')defcustom_check(cells):# ...Argumentsname (str): preset nametype (str): has to becustomcontext (str): has to beheadorbodyposition (str): has to bebefore:<check-name>orafter:<check-name>ErrorError(self,code,cell=None,row_number=None,message=None,message_substitutions=None)Describes a validation check errorArgumentscode (str): The error code. Must be one in the spec.cell (dict, optional): The cell where the error occurred.row_number (int, optional): The row number where the error occurs.message (str, optional):
The error message. Defaults to the message from the Data Quality Spec.message_substitutions (dict, optional):
Dictionary with substitutions to be used when
generating the error message and description.RaisesKeyError: Raised if the error code isn't known.specdict() -> new empty dictionary
dict(mapping) -> new dictionary initialized from a mapping object's
(key, value) pairs
dict(iterable) -> new dictionary initialized as if via:
d = {}
for k, v in iterable:
d[k] = v
dict(**kwargs) -> new dictionary initialized with the name=value pairs
in the keyword argument list.  For example:  dict(one=1, two=2)GoodtablesExceptionGoodtablesException(self,/,*args,**kwargs)Base goodtables exceptionContributingThe project follows theOpen Knowledge International coding standards.Recommended way to get started is to create and activate a project virtual environment.
To install package and development dependencies into active environment:$makeinstallTo run tests with linting and coverage:$maketestChangelogHere described only breaking and the most important changes. The full changelog and documentation for all released versions could be found in nicely formattedcommit history.v2.5Addedcheck.check_headers_hookto support headers check for body-contexted checks (seehttps://github.com/frictionlessdata/goodtables-py/tree/v3for native support)v2.4Added integrity checks for data packages. Ifresource.bytesorresource.hash(sha256) is provided it will be verified against actual valuesv2.3Added aforeign keys checkv2.2Improved missing/non-matching-headers detection (#298)v2.1A new key added to theerror.to_dictreturn:message-datav2.0Breaking changes:Checks method signature now only receives the current row'scellslistChecks raise errors by returning an array ofErrorobjectsCells have the row number in therow-numberkeyFiles with ZIP extension are presumed to be datapackages, sogoodtables mydatapackage.zipworksImprovements to goodtables CLI (#233)Newgoodtables init <data paths>command to create a newdatapackage.jsonwith the files passed as parameters and their inferred schemas.Bug fixes:Fix bug withtruncated-valuescheck on date fields (#250)v1.5New API added:Validationsourcenow could be apathlib.Pathv1.4Improved behaviour:rebased on Data Quality Spec v1rebased on Data Package Spec v1rebased on Table Schema Spec v1treat primary key as required/unique fieldv1.3New advanced checks added:blacklisted-valuecustom-constraintdeviated-valuesequential-valuetruncated-valuev1.2New API added:report.presetreport.tables[].schemav1.1New API added:report.tables[].schemereport.tables[].formatreport.tables[].encodingv1.0This version includes various big changes. A migration guide is under development and will be published here.v0.6First version ofgoodtables.