Package: datatest
Datatest helps to speed up and formalize data-wrangling and data
validation tasks. It implements a system of validation methods,
difference classes, and acceptance managers. Datatest can help you:Clean and wrangle data faster and more accurately.Maintain a record of checks and decisions regarding important data sets.Distinguish between ideal criteria and acceptible deviation.Validate the input and output of data pipeline components.Measure progress of data preparation tasks.On-board new team members with an explicit and structured process.Datatest can be used directly in your own projects or as part of a testing
framework likepytestorunittest. It has no hard dependencies; it’s
tested on Python 2.6, 2.7, 3.2 through 3.10, PyPy, and PyPy3; and is freely
available under the Apache License, version 2.Documentation:https://datatest.readthedocs.io/(stable)https://datatest.readthedocs.io/en/latest/(latest)Official:https://pypi.org/project/datatest/Code ExamplesValidating a Dictionary of Listsfromdatatestimportvalidate,accepted,Invaliddata={'A':[1,2,3,4],'B':['x','y','x','x'],'C':['foo','bar','baz','EMPTY']}validate(data.keys(),{'A','B','C'})validate(data['A'],int)validate(data['B'],{'x','y'})withaccepted(Invalid('EMPTY')):validate(data['C'],str.islower)Validating a Pandas DataFrameimportpandasaspdfromdatatestimportregister_accessors,accepted,Invalidregister_accessors()df=pd.read_csv('data.csv')df.columns.validate({'A','B','C'})df['A'].validate(int)df['B'].validate({'x','y'})withaccepted(Invalid('EMPTY')):df['C'].validate(str.islower)InstallationThe easiest way to install datatest is to usepip:pip install datatestIf you are upgrading from version 0.11.0 or newer, use the--upgradeoption:pip install --upgrade datatestUpgrading From Version 0.9.6If you have an existing codebase of older datatest scripts, you should
upgrade using the following steps:Install datatest 0.10.0 first:pip install --force-reinstall datatest==0.10.0Run your existing code and check for DeprecationWarnings.Update the parts of your code that use deprecated features.Once your code is running without DeprecationWarnings,
install the latest version of datatest:pip install --upgrade datatestStuntman MikeIf you need bug-fixes or features that are not available
in the current stable release, you can “pip install” the
development version directly from GitHub:pip install --upgrade https://github.com/shawnbrown/datatest/archive/master.zipAll of the usual caveats for a development install should
apply—only use this version if you can risk some instability
or if you know exactly what you’re doing. While care is taken
to never break the build, it can happen.Safety-first ClydeIf you need to review and test packages before installing, you can
install datatest manually.Download the latestsourcedistribution from the Python Package
Index (PyPI):https://pypi.org/project/datatest/#filesUnpack the file (replacing X.Y.Z with the appropriate version number)
and review the source code:tar xvfz datatest-X.Y.Z.tar.gzChange to the unpacked directory and run the tests:cd datatest-X.Y.Z
python setup.py testDon’t worry if some of the tests are skipped. Tests for optional data
sources (like pandas DataFrames or NumPy arrays) are skipped when the
related third-party packages are not installed.If the source code and test results are satisfactory, install the
package:python setup.py installSupported VersionsTested on Python 2.6, 2.7, 3.2 through 3.10, PyPy, and PyPy3.
Datatest is pure Python and may also run on other implementations
as well (check using “setup.py test” before installing).Backward CompatibilityIf you have existing tests that use API features which have
changed since 0.9.0, you can still run your old code by
adding the following import to the beginning of each file:fromdatatest.__past__importapi09To maintain existing test code, this project makes a best-effort
attempt to provide backward compatibility support for older
features. The API will be improved in the future but only in
measured and sustainable ways.All of the data used at theNational Committee for an Effective
Congresshas been checked with
datatest for several years so there is, already, a large and
growing codebase that relies on current features and must be
maintained into the future.Soft DependenciesDatatest has no hard, third-party dependencies. But if you want
to interface with pandas DataFrames, NumPy arrays, or other
optional data sources, you will need to install the relevant
packages (pandas,numpy, etc.).Development RepositoryThe development repository fordatatestis hosted onGitHub.Freely licensed under the Apache License, Version 2.0Copyright 2014 - 2021 National Committee for an Effective Congress, et al.