Package: datacockpit
DataCockpitDataCockpit is a Python toolkit that leverages the data, usage logs, and associated meta-data
within a data lake and provisions two kinds of auxiliary information:quality – the validity and appropriateness of data required to perform certain analytic tasks.usage – the historical utilization characteristics of data across multiple users.ArchitectureInstallDataCockpit is written in Python 3. Please ensure you have a Python 3 environment already installed.Installing DataCockpit is as simple as making microwave popcorn! Justpip install datacockpitand then sit back and let it do the work while you put your feet up.How to UsefromdatacockpitimportDataCockpit# Setup database connection with SQL engine and log-file CSVdcp_obj=DataCockpit(engine=your_sqlalchemy_engine,logs_path="your/logs/path.csv")# Compute and persist quality & usage metricsdcp_obj.compute_quality(levels=None,metrics=None)dcp_obj.compute_usage(levels=None,metrics=None)First, we initialize aSQLAlchemyenginethat is used to connect to the database.Then we create an object,dcp_obj, of theDataCockpitclass, by passing theengineandlogs_pathas arguments.logs_pathpoints to the location where the historical usage logs (SQL queries and metadata such as the user who ran them and the timestamp) are saved in a CSV file.The next two lines call methods to compute and persist quality and usage with parameters to support differentlevels(e.g., ['attribute', 'record', 'all']) andmetrics(e.g., ['correctness', 'completeness', 'all']).Thecompute_commands persist the computed metrics to database tables.You can retrieve the computed metrics for use in downstream applications through the belowget_commands.# Retrieve computed information for use in downstream applicationsquality_info=dcp_obj.get_quality()usage_info=dcp_obj.get_usage()Depending on your data and query patterns, theget_quality()andget_usage()functions will return the following quality and usage information.Data Quality InformationAttribute metrics table (attribute_metrics):Completenessis the percentage of non-missing values for an attribute.Correctnessis the percentage of correct values for an attribute based on pre-defined constraints.Objectivityis the amount of distortion in the data distribution.Uniquenessis the percentage of unique values for an attribute.Record metrics table (record_metrics):Completenessis the percentage of non-missing values in each dataset record.Correctnessis the percentage of correct values in each dataset record.Uniquenessis the percentage of unique values in each dataset record.Data Usage InformationThe usage metrics are fairly self explanatory. The SQL queries are parsed to get the Metadata Table that shows usage statistics for every attribute in the datasets (analogous to tables). The Aggregate Table and the Dataset Usage Tables are rolled up from the Metadata Table. Other analyses such as timeseries analyses are shown in the Jupyter notebooks in theassets/notebooksdirectory. It runs on a sample usage file available inassets/data/query_logs.csv.Metadata table (dcp_metadata):Aggregate table (dcp_aggr):Data usage (dcp_dataset_usage)BuildDataCockpit can be installed as a Python package and imported in your own awesome applications!DataCockpit is written in Python 3. Please ensure you have a Python 3 environment already installed.Clone this repository (review branch) and enter (cd) into it.Create a new virtual environment,virtualenv --python=python3 venvActivate it using,source venv/bin/activate(MacOSX/Linux) orvenv\Scripts\activate.bat(Windows)Install dependencies,python -m pip install -r requirements.txt<make your changes>Bump up the version in setup.py and create a Python distributable,python setup.py sdistThis will create a new file insidedatacockpit-..*.tar.gzinside thedistdirectory.Install the above file in your Python environment using,python -m pip install <PATH-TO-datacockpit-*.*.*.tar.gz>Verify by opening your Python console and importing it:>>>fromdatacockpitimportDataCockpitEnjoy, DataCockpit is now available for use as a Python package!AnalysisLook atassetsfor
examples of how to use the metrics obtained, such as visualizing the temporal trends in data
or finding the most critical attributes.CreditsDataCockpit was created by Arpit Narechania, Fan Du, Atanu R. Sinha, Ryan A. Rossi, Jane Hoffswell, Shunan Guo, Eunyee Koh, Surya Chakraborty, Shivam Agarwal, Shamkant B. Navathe, and Alex Endert.LicenseThe software is available under theMIT License.ContactIf you have any questions, feel free toopen an issueor contactArpit Narechania,Surya Chakraborty(chakraborty [at] gatech.edu), or Shivam Agarwal (s.agarwal [at] gatech.edu).