Package: ydata-quality
ydata-qualityydata_quality is an open-source python library for assessing Data Quality throughout the multiple stages of a data pipeline development.A holistic view of the data can only be captured through a look at data from multiple dimensions andydata_qualityevaluates it in a modular way wrapped into a single Data Quality engine. This repository contains the core python source scripts and walkthrough tutorials.QuickstartThe source code is currently hosted on GitHub at:https://github.com/ydataai/ydata-qualityBinary installers for the latest released version are available at thePython Package Index (PyPI).pip install ydata-qualityComprehensive quality check in few lines of codefromydata_qualityimportDataQualityimportpandasaspd#Load in the datadf=pd.read_csv('./datasets/transformed/census_10k.csv')# create a DataQuality object from the main class that holds all quality modulesdq=DataQuality(df=df)# run the tests and outputs a summary of the quality testsresults=dq.evaluate()Warnings:
	TOTAL: 5 warning(s)
	Priority 1: 1 warning(s)
	Priority 2: 4 warning(s)

Priority 1 - heavy impact expected:
	* [DUPLICATES - DUPLICATE COLUMNS] Found 1 columns with exactly the same feature values as other columns.
Priority 2 - usage allowed, limited human intelligibility:
	* [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3 numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove the highest VIF variables.
	* [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1960 ED values in the dataset.
	* [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 10 categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and sorted descending according to propensity. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove variables following the provided order.
	* [DUPLICATES - EXACT DUPLICATES] Found 3 instances with exact duplicate feature values.On top of the summary, you can retrieve a list of detected warnings for detailed inspection.# retrieve a list of data quality warningswarnings=dq.get_warnings()ExamplesHere you can find walkthrough tutorials and examples to familarize with different modules ofydata_qualityStart Here for Quick and Overall WalkthroughTo dive into any focussed module, and to understand how they work, here are tutorial notebooks:Bias and FairnessData ExpectationsData RelationsDrift AnalysisDuplicatesLabelling:CategoricalsandNumericalsMissingsErroneous DataContributingWe are open to collaboration! If you want to start contributing you only need to:Search for an issue in which you would like to work. Issues for newcomers are labeled with good first issue.Create a PR solving the issue.We would review every PRs and either accept or ask for revisions.You can also join the discussions on the #data-quality channel on ourSlackand request features/bug fixes by opening issues on our repository.SupportFor support in using this library, please join the #help Slack channel. The Slack community is very friendly and great about quickly answering questions about the use and development of the library.Click here to join our Slack community!LicenseGNU General Public License v3.0