Package: datachecks
Open Source Data Quality Monitoring.⭐️ If you like it, star the repo⭐|Documentations|Slack Community|Why Data Monitoring?APM (Application Performance Monitoring) tools are used to monitor the performance of applications. APM tools are mandatory part of dev stack. Without AMP tools, it is very difficult to monitor the performance of applications.But for Data products regular APM tools are not enough. We need a new kind of tools that can monitor the performance of Data applications.
Data monitoring tools are used to monitor the data quality of databases and data pipelines. It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.What isdatachecks?Datachecks is an open-source data monitoring tool that helps to monitor the data quality of databases and data pipelines.
It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.Datachecks can generate several reliability, uniqueness, completeness metrics from several data sourcesReports: Data Quality VisualisationYou can generate with just one command. It generates a beautiful data quality report with all the metrics.
This html report can be shared with the team.CLI: Data Quality Visualisation in BashData quality report can be generated in the terminal. It is very useful for debugging. All it takes is one command.Getting StartedInstalldatacheckswith the command that is specific to the database.Install DatachecksTo install all datachecks dependencies, use the below command.pipinstalldatachecks-UCreate the config fileWith a simple config file, you can generate data quality reports for your data sources. Below is the sample config example.
For more details, please visit theconfig guideRun from CLIGenerate Report in Terminaldatachecksinspect-Cconfig.yamlGenerate HTML Reportdatachecksinspect-Cconfig.yaml--html-reportPlease visit theQuick Start GuideSupported Data SourcesDatachecks supports sql and search data sources. Below are the list of supported data sources.Data SourceTypeSupportedPostgresTransactional Database:thumbsup:MySqlTransactional Database:thumbsup:MS SQL ServerTransactional Database:soon:OpenSearchSearch Engine:thumbsup:ElasticsearchSearch Engine:thumbsup:GCP BigQueryData Warehouse:thumbsup:DataBricksData Warehouse:thumbsup:SnowflakeData Warehouse:soon:AWS RedShiftData Warehouse:thumbsup:Metric TypesMetricDescriptionReliability MetricsReliability metrics detect whether tables/indices/collections are updating with timely dataNumeric Distribution MetricsNumeric Distribution metrics detect changes in the numeric distributions i.e. of values, variance, skew and moreUniqueness MetricsUniqueness metrics detect when data constraints are breached like duplicates, number of distinct values etcCompleteness MetricsCompleteness metrics detect when there are missing values in datasets i.e. Null, empty valueValidity MetricsValidity metrics detect whether data is formatted correctly and represents a valid valueOverviewWhat Datacheck does not do?Community & SupportFor additional information and help, you can use one of these channels:Slack(Live chat with the team, support, discussions, etc.)GitHub issues(Bug reports, feature requests)Contributions:raised_hands: We greatly appreciate contributions - be it a bug fix, new feature, or documentation!Check out thecontributions guideandopen issues.Datachecks contributors: :blue_heart:TelemetryUsage Analytics & Data PrivacyLicenseThis project is licensed under the terms of theAPACHE 2 License.