Package: datatracer
An open source project from Data to AI Lab at MIT.DataTracerData Lineage Tracing LibraryLicense:MITDevelopment Status:Pre-AlphaHomepage:https://github.com/data-dev/DataTracerOverviewDataTracer is a Python library for solving Data Lineage problems using statistical
methods, machine learning techniques, and hand-crafted heuristics.Currently the Data Tracer library implements discovery of the following properties:Primary Key: Identify which column is the primary key in each table.Foreign Key: Find which relationships exist between the tables.Column Mapping: Given a field in a table, deduce which other fields, from the same table
or other tables, are more related or contributed the most in generating the given field.REST APIThe DataTracer library also incorporates a REST API that enables interaction with the DataTracer
Solvers via HTTP communication. You can check ithereInstallRequirementsDataTracerhas been developed and tested onPython 3.5 and 3.6, 3.7Also, although it is not strictly required, the usage of avirtualenvis highly recommended in order to avoid
interfering with other software installed in the system whereDataTraceris run.Install with pipThe easiest and recommended way to installDataTraceris usingpip:pipinstalldatatracerThis will pull and install the latest stable release fromPyPi.If you want to install from source or contribute to the project please read theContributing Guide.Data Format: Datasets and MetadataThe DataTracer library is prepared to work using datasets, which are a collection of tables
loaded aspandas.DataFramesand a MetaData JSON which provides information about the
dataset structure.You can find more information about the MetaData format in theMetaData repository.The DataTracer also includes a fewdemo datasetswhich you can easily
download to your computer using thedatatracer.get_demo_datafunction:fromdatatracerimportget_demo_dataget_demo_data()This will create a folder calleddatatracer_demoin your working directory with a few
datasets ready to use inside it.QuickstartIn this short tutorial we will guide you through a series of steps that will help you
getting started withData Tracer.Load dataThe first step will be to load the data in the format expected by DataTracer.For this, we can use thedatatracer.load_datasetfunction passing the path to
the dataset folder.For example, if we want to use theclassicmodelsdataset included in the demo folder
that we just created we can load it using:fromdatatracerimportload_datasetmetadata,tables=load_dataset('datatracer_demo/classicmodels')This will return a tuple which contains:AMetaDatainstance with details about the dataset.Adictwith all the tables of the dataset loaded as apandas.DataFrame.Select a SolverIn the DataTracer project, the different Data Lineage problems are solved using what we
callsolvers.We can see the list of available solvers using theget_solversfunction:fromdatatracerimportget_solversget_solvers()which will return a list with their names:['datatracer.column_map',
 'datatracer.foreign_key.basic',
 'datatracer.foreign_key.standard',
 'datatracer.primary_key.basic']Use a DataTracer instance to find table relationshipsIn order to use the selected solver you will need to load it using theDataTracerclass.In this example, we will try to figure out the relationships between the tables in our dataset
by using the solverdatatracer.foreign_key.standard.fromdatatracerimportDataTracer# Load the Solversolver=DataTracer.load('datatracer.foreign_key.standard')# Solve the Data Lineage problemforeign_keys=solver.solve(tables)The result will be a dictionary containing the foreign key candidates:[{'table': 'products',
  'field': 'productLine',
  'ref_table': 'productlines',
  'ref_field': 'productLine'},
 {'table': 'payments',
  'field': 'customerNumber',
  'ref_table': 'customers',
  'ref_field': 'customerNumber'},
 {'table': 'orders',
  'field': 'customerNumber',
  'ref_table': 'customers',
  'ref_field': 'customerNumber'},
 {'table': 'orderdetails',
  'field': 'productCode',
  'ref_table': 'products',
  'ref_field': 'productCode'},
 {'table': 'orderdetails',
  'field': 'orderNumber',
  'ref_table': 'orders',
  'ref_field': 'orderNumber'},
 {'table': 'employees',
  'field': 'officeCode',
  'ref_table': 'offices',
  'ref_field': 'officeCode'}]What's next?You can learn more about the DataTracer features in thenotebook tutorials.Also don't forget to have a look at the DataTracerREST API.History0.0.6 - 2020-06-19Addupdate_metadataprimitives and pipelines.Upgrade to MetaData v0.0.20.0.5 - 2020-06-12Add newupdate_metadataendpoint to the REST API.New demo dataset and new tutorial.0.0.4 - 2020-06-05Add initial version of pretrained solversReorganize ColumnMapSolver code treeAdd REST API to access DataTracer solvers via HTTP0.0.3 - 2020-05-28Finish Column Mapping and add tutorialMinor refactoring and adding docstringsFix testing config0.0.2 - 2020-05-26Curate configuration and dependencies0.0.1 - 2020-05-22First release.Features:Primary Key DetectionForeign Key DetectionColumn Mapping