Package: driftdb
DriftdbDriftdbis a historical metric storefromdriftdb.connectorsimportGithubConnectorfromgithubimportGithubgithub_connector=GithubConnector(github_client=Github("gh_token"),github_repository_name="org/repo")dataframe=bigquery.Client().query(query).to_dataframe(){"unique_key":['2022-01-01_FR','2022-01-01_GB'...github_connector.snapshot_table(table_dataframe=dataframe,table_name="revenue")'🎉 data/act_metrics_finance/mrr.csv Successfully stored!''💩 Historical data change detected, Ammy was assigned to it'PurposeNon-moving data is a journey, in reality, the data moves and it has many impacts (Data Integrity and Reconciliation, Predictive Modeling, Historical Data Accuracy)
The purpose of this library is:to snapshot the data, and parse the diff in chunks (schema update, new data collection, data duplication, drift...)to store it using a connectorto trigger alertsGetting Started (with Github as a store, it's free)To get started with Driftdb, follow these steps:Create a new repository on GitHub calleddatadrift(or whatever other name you prefer) with a README file.Generate a personal access token on GitHub that has access to thedatadriftrepository. You can do this by going to your GitHub settings, selecting "Developer settings", and then "Personal access tokens". Click "Generate new token" and give it the necessary permissions (content and pull requests).In your data pipelines, when relevant, callsnapshot_tablewith the following parametersa connector (in this example a github connector)your table in a dataframe formatthe name of the table: "kpi/my_kpi"For instance>>>fromdriftdb.connectorsimportGithubConnector>>>github_connector=GithubConnector(github_client=Github("gh_token"),github_repository_name="org/repo")>>>github_connector.snapshot_table(table_dataframe=dataframe,table_name="revenue")That's it! With these steps, you can start using Driftdb to store and track your metrics over time.DataframeDriftdb is base on the standard dataframe format fromPandas.
One can use any library to get the data as long as the format fits the following requirements:The first column of the dataframe must beunique_keyThe first columns must have only unique keysThe second column must be a date (which is the collection date: the booking_date, the order_date etc)The granularity of the dataframe depends on every use case:it can be at very low level (like transaction) or aggregated (like a metric)it can contain all the dimension, or none1st column: Unique keyThe unique_key is used to detect a modification in historical dataIn case you have duplicated lines, driftdb will automatically rename them with-duplicate-nunique_key  value
0          A     10
1          B     20
2          C     30
3          B     40
4          C     50
5          C     60
6          D     70unique_key  value
0                A     10
1                B     20
2                C     30
3    B-duplicate-1     40
4    C-duplicate-1     50
5    C-duplicate-2     60
6                D     702nd column: DateThe date key is used to detect new historical data, or deleted historical data. And differentiate if a new batch is being collected (which won't be a drift)Large DatasetPartitionningIn case of more than 1M rows, partitionning is recomanded using thepartition_and_store_tablefunction.>>>fromdriftdb.connectors.workflowimportpartition_and_snapshot_table>>>very_large_dataframe=bigquery.Client().query(query).to_dataframe(){"unique_key":['2022-01-01_FR','2022-01-01_GB'...>>>connector.partition_and_snapshot_table(table_dataframe=very_large_dataframe,table_name="act_metrics_finance/mrr")'🎁 Partitionning data/act_metrics_finance/mrr.csv...'AlertingDriftA drift is a modification of historical data. It can be a modification, addition or deletion in a table that is supposed to be "non-moving data".Drift HandlerA drift handler is a function that conforms the typeDriftHandler:DriftHandler=Callable[[DriftEvaluatorContext],DriftEvaluation]# With DriftEvaluatorContext and DriftEvaluation beingclassDriftEvaluatorContext:def__init__(self,before:pd.DataFrame,after:pd.DataFrame,summary:DriftSummary):self.before=beforeself.after=afterself.summary=summaryclassDriftEvaluation(TypedDict):should_alert:boolmessage:strDefault Drift HandlerThe default drift evaluator never triggers any alert, it returnsshould_alert = False.Alert Drift HandlerThealert_drift_handlerwill trigger an alert if there is a drifts, and an alert message containing the summary of the drift, example:Drift detected:
- 🆕 0 addition
- ♻️ 2 modifications
- 🗑️ 0 deletionTo use thealert_drift_handler, add it when you call snapshot_table like this:fromdriftdb.alertingimportalert_drift_handlerconnector.snapshot_table(table_dataframe,table_name,drift_handler=alert_drift_handler)Threshold Drift HandlerThe Threshold Drift Handler is designed to monitor changes in numerical values. It triggers an alert when a numerical value is updated and the absolute difference, when divided by the old value, exceeds a specified threshold.Here's how you can use the Threshold Drift Handler:fromdriftdb.alertingimportTresholdDriftHandlerFactory# Set your desired thresholdthreshold=0.1# Alert if the change is over 10%threshold_handler=TresholdDriftHandlerFactory(numerical_cols=['metric_column_name'],threshold=threshold)connector.snapshot_table(table_dataframe,table_name,drift_handler=threshold_handler)Custom Drift HandlerYou can provide a custom evaluator which is a function with a DriftEvaluatorContext containing the following properties:classDriftEvaluatorContext(TypedDict):before:pd.DataFrameafter:pd.DataFramesummary:DriftSummaryclassDriftSummary(TypedDict):added_rows:pd.DataFramedeleted_rows:pd.DataFramemodified_rows_unique_keys:pd.Indexmodified_patterns:pd.DataFrameThen implement your handler, and use it in snapshot_table.defmy_drift_handler(data_drift_context:DriftEvaluatorContext,)->DriftEvaluation:# do what you wantifthere_is_something_I_dont_like:return{"should_alert":True,"message":"No this should not happen"}return{"should_alert":False,"message":""}New DataWhen there is a new batch of data in a table, e.g. the results of last week. This addition is considered new data. It should not be confused with a new row entry of historical data. For instance, if a new transaction with a paying_date from a month ago is registered today, it will be considered a drift, not a new data.New Data HandlerYou can also add alerting when inserting new data in the table.Detect Outlier HandlerYou can create adetect_outlier_handlerwith theDetectOutlierHandlerFactorythat takes 2 arguments, the numerical columns and the category columns.
For numerical columns, if the new data is an outlier (using theinterquartil method) it will trigger an alert.
For category columns, if a new category is detected, it will trigger an alert.To use thedetect_outlier_handler, add it when you call snapshot_table like this:fromdriftdb.alertingimportDetectOutlierHandlerFactorynew_data_handler=DetectOutlierHandlerFactory(numerical_cols=["age"],categorical_cols=[])connector.snapshot_table(table_dataframe,table_name,new_data_handler=new_data_handler)Custom New Data HandlerYou can provide a custom handler which is a function with a NewDataEvaluatorContext containing the following properties:classNewDataEvaluatorContext:def__init__(self,before:pd.DataFrame,after:pd.DataFrame,added_rows:pd.DataFrame):self.before=beforeself.after=afterself.added_rows=added_rowsThen implement your handler, and use it in snapshot_table.defmy_new_data_handler(new_data_context:NewDataEvaluatorContext,)->DriftEvaluation:# do what you wantifthere_is_something_I_dont_like:return{"should_alert":True,"message":"No this should not happen"}return{"should_alert":False,"message":""}CLIInstead of storing data on github, you can store data locally and explore it with the cli.Getting startedFrom dbt snapshot (dbt >= 1.6)pipinstalldriftdb

driftdbdbtsnapshot
driftdbstartFrom generated seedspipinstalldriftdb

driftdbseedcreate
driftdbseedupdate

driftdbstartFeaturesMetricsLoad a csvdriftdb load-csv path/to/csvData visualizationdriftdb startStart the driftdb, and navigate tolocalhost:9741/tables.
Visualize how a metric evolved, given a period, in a waterfall chart.