Package: sdmetrics
This repository is part ofThe Synthetic Data Vault Project, a project fromDataCebo.OverviewThe SDMetrics library evaluates synthetic data by comparing it to the real data that you're trying to mimic. It includes a variety of metrics to capture different aspects of the data, for examplequality and privacy. It also includes reports that you can run to generate insights, visualize data and share with your team.The SDMetrics library ismodel-agnostic, meaning you can use any synthetic data. The library does not need to know how you created the data.InstallInstall SDMetrics using pip or conda. We recommend using a virtual environment to avoid conflicts with other software on your device.pipinstallsdmetricscondainstall-cconda-forgesdmetricsFor more information about using SDMetrics, visit theSDMetrics Documentation.UsageGet started withSDMetrics Reportsusing some demo data,fromsdmetricsimportload_demofromsdmetrics.reports.single_tableimportQualityReportreal_data,synthetic_data,metadata=load_demo(modality='single_table')my_report=QualityReport()my_report.generate(real_data,synthetic_data,metadata)Creating report: 100%|██████████| 4/4 [00:00<00:00,  5.22it/s]

Overall Quality Score: 82.84%

Properties:
Column Shapes: 82.78%
Column Pair Trends: 82.9%Once you generate the report, you can drill down on the details and visualize the results.my_report.get_visualization(property_name='Column Pair Trends')Save the report and share it with your team.my_report.save(filepath='demo_data_quality_report.pkl')# load it at any point in the futuremy_report=QualityReport.load(filepath='demo_data_quality_report.pkl')Want more metrics?You can also manually apply any of the metrics in this library to your data.# calculate whether the synthetic data respects the min/max bounds# set by the real datafromsdmetrics.single_columnimportBoundaryAdherenceBoundaryAdherence.compute(real_data['start_date'],synthetic_data['start_date'])0.8503937007874016# calculate whether the synthetic data is new or whether it's an exact copy of the real datafromsdmetrics.single_tableimportNewRowSynthesisNewRowSynthesis.compute(real_data,synthetic_data,metadata)1.0What's next?To learn more about the reports and metrics, visit theSDMetrics Documentation.The Synthetic Data Vault Projectwas first created at MIT'sData to AI Labin 2016. After 4 years of research and traction with enterprise, we
createdDataCeboin 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.Get started using the SDV package-- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.Historyv0.13.0 - 2023-12-04This release makes significant improvements to the Diagnostic Reports! The report now runs a diagnostic to calculate scores for three basic but important properties of your data: data validity, data structure and in the multi table case, relationship validity. Data validity checks that the columns of your data are valid (eg. correct range or values). Data structure makes sure the synthetic data has the correct columns. Relationship validity checks to make sure key references are correct and the cardinality is within ranges seen in the real data. These changes are meant to make theDiagnosticReporta quick way for you to see if there are any major problems with your synthetic data.Additionally, some general improvements were made and bugs were resolved. TheLogisticDetectionandSVCDetectionmetrics were fixed to only use boolean, categorical, datetime and numeric columns in their calculations. A bug that prevented visualizations from displaying on Jupyter notebooks was patched. The cardinality property in the multi tableQualityReportcan now handle multiple foreign keys to the same parent. Finally, a new visualization was added for sequential/timeseries data calledget_column_line_plot.New FeaturesDetection metrics should only use statistically modeled columns (filter out the rest) - Issue#286by @lajohn4747Add visualization for timeseries / sequential data  - Issue#376by @lajohn4747Multi table quality report should handle multi-foreign keys (to same parent) - Issue#406by @R-PalazzoAddKeyUniquenessmetric - Issue#460by @R-PalazzoAddReferentialIntegritymetric - Issue#461by @R-PalazzoAddCategoryAdherencemetric - Issue#462by @R-PalazzoAddTableFormatmetric - Issue#463by @R-PalazzoAddCardinalityBoundaryAdherencemetric - Issue#464by @frances-hAddDataValidityproperty - Issue#467by @R-PalazzoAddStructureproperty - Issue#468by @R-PalazzoAddRelationship Validityproperty - Issue#469by @R-PalazzoUpdateDiagnosticReportto calculate base correctness of synthetic data - Issue#471by @R-PalazzoUpdate the synthetic data that's available for the multi-table demo - Issue#501by @R-PalazzoUpdate the synthetic data that's available for the single-table demo - Issue#502by @R-PalazzoUpdateTableFormatmetric toTableStructure+ fix its computation - Issue#518by @R-PalazzoBugs FixedSometimes graphs don't show when using Jupyter notebook - Issue#322by @pvk-developerFix ReferentialIntegrity NaN handling - Issue#494by @R-PalazzoKeyUniqueness metric should only be applied to primary and alternate keys - Issue#503by @R-PalazzoSingle table Structure property should not have visualization - Issue#504by @R-PalazzoMulti table Structure property visualization has incorrect styling - Issue#505by @R-PalazzoUserWarning: KeyError: 'relationships'in DiagnosticReport if metadata missing relationships - Issue#506by @R-PalazzoReportvalidatemethod should be private - Issue#507by @R-PalazzoValueErrorin DiagnosticReport if synthetic data does not match metadata - Issue#508by @R-PalazzoCheck if QualityReport needs the synthetic data to match the metadata - Issue#509by @R-PalazzoRunning single table report on multi table data (or vice versa) results in confusing error - Issue#510by @R-PalazzoAdd metadata validation - Issue#526by @R-Palazzov0.12.1 - 2023-11-01This release fixes a bug with the new Intertable Trends property and older pandas versions and a bug with how the ML Efficacy metric handled train and test data. Reports handle missing relationships more gracefully.Bugs FixedMultiple FutureWarning lines printed out when running the Quality Report (Intertable Trends property) - Issue#490by @frances-hTransformer should not be fit on test data - Issue#291by @fealhoReports should not crash if there are no relationships - Issue#481by @lajohn4747v0.12.0 - 2023-10-31This release adds a new property, InterTable Trends. Several plots were moved from the reports module into the new visualizations module.  Themetadataparameter was removed for these plots, and theplot_typesparameter was added.plot_typeslets the user control which plot type is used. Several crashes have been resolved.New FeaturesProvide meta information about the reports - Pull#472by @frances-hValidate that the metadata is always a dict - Issue#428by @R-PalazzoExpose reports module in top-level init - Pull#459by @frances-hAdd new get_column_pair_plot - Issue#444by @pvk-developerAdd InterTable Trends property - Issue#451by @frances-hAdd new get_column_plot - Issue#443by @pvk-developerAdd new get_cardinality_plot - Issue#445by @frances-hCreate visualizations module - Issue#442by @frances-h, @pvk-developerBugs FixedFixNewRowSynthesison datetime columns without formats - Issue#473by @fealhoIntertable trends property crashes if a table has no statistical columns - Issue#476by @lajohn4747Fix BoundaryAdherence NaN handling - Issue#470by @frances-hThe Intertable Trends visualization is mislabeled as 'Column Shapes' - Issue#477by @lajohn4747ValueError when using get_cardinality_plot on some schemas - Issue#447by @frances-hInternalSwitch default branch from master to main - Issue#420by @amontanez24v0.11.1 - 2023-09-14This release makes multiple changes to better handle errors that get raised from theDiagnosticReport. The report should be able to run to completion now and have any errors that it encounters reported in a column on the details that can be observed from runningget_details. It also resolves many warnings that were interrupting the printing of the report's results and progress.New FeaturesCreate single table coverage property - Issue#389by @R-PalazzoCreate single table synthesis property - Issue#390by @R-PalazzoCreate single table Boundaries property - Issue#391by @R-PalazzoAdd multi table Coverage, Synthesis and Boundaries property - Issue#393by @R-PalazzoBugs FixedEnsure that theSynthesisproperty score doesn't change - Issue#425by @amontanez24The Error column contains a mix ofNaNandNonevalues - Issue#427by @pvk-developerAlways show theTablecolumn inget_details- Issue#429by @frances-hDiagnostic explanations should not repeat if I generate multiple times - Issue#430by @amontanez24RangeCoverage errors on datetime columns in DiagnosticReport - Issue#431by @frances-hThe coverage visualization shows empty bar graph for nan values - Issue#432by @frances-hDiagnostic report should skip over all NaN columns - Issue#433by @pvk-developerQuality report is printing out a long warning message (hundreds of lines) - Issue#448by @amontanez24InternalUse property classes in single table DiagnosticReport - Issue#392by @R-PalazzoUse property classes in multi table DiagnosticReport - Issue#394by @R-Palazzov0.11.0 - 2023-08-10This release adds a function that allows users to plot the cardinality of foreign and primary keys in synthetic data. More specifically, it graphs the frequency that each number of children per parent row occurs in the parent table.Additionally, architectural changes are made to improve the efficiency and error handling of theQualityReport! The progress bar is also enhanced to be more informative when the report is generating.This release also adds support for Python 3.11 and drops support for Python 3.7.New FeaturesVisualize cardinality of foreign key columns - Issue#283by @R-PalazzoCreate single table BaseProperty class - Issue#354by @amontanez24Create single table column shapes property - Issue#355by @R-PalazzoCreate single table column pair trends property - Issue#356by @R-PalazzoCreate multi table BaseProperty class - Issue#357by @pvk-developerCreate multi table column shapes and column pair trends properties - Issue#358by @R-PalazzoCreate Parent Child Relationships property class - Issue#359by @pvk-developerIn Multi Table Quality Report: Rename "Table Relationships" property to "Cardinality" - Issue#360by @frances-hMore accurate progress bar for single table Quality Report - Issue#361by @R-PalazzoMore accurate progress bar for multi table Quality Report - Issue#362by @fealhoRaise error in CorrelationSimilarity if either column is constant - Issue#407by @fealhoBug FixesIssue in building the denormalized table inside the Parent-Child Detection metrics - Issue#328by @fealhoDon't modify the rounding in the quality report - Issue#401by @R-PalazzoThe Cardinality property is missing some relationships - Issue#404by @pvk-developerThe Cardinality property is not returning a DataFrame - Issue#405by @fealhoOverall property score should be the average across all breakdowns - Issue#415by @amontanez24InternalUse property classes in single table QualityReport - Issue#370by @R-PalazzoUse property classes in multi table QualityReport - Issue#371by @fealhoAdd add-on detection for premium metrics - Issue#388by @amontanez24MaintenanceAdd support for Python 3.11 - Issue#353by @amontanez24Drop support for Python 3.7 - Issue#380by @amontanez24v0.10.1 - 2023-06-06This release fixes a bug that was causing theDiagnosticReportto crash on theNewRowSynthesismetric. It also adds support for PyTorch 2.0!Bug FixesValueError: multi-line expressions (NewRowSynthesis metric in DiagnosticReport) - Issue#327by @R-PalazzoMaintenanceUpgrade to torch 2.0 - Issue#347by @fealhov0.10.0 - 2023-05-03This release makes theDiagnosticReportmore fault tolerant by preventing it from crashing if a metric it uses fails. It also adds support for Pandas 2.0!Additionally, support for the oldSDVmetadata format (preSDV1.0) has been dropped.New FeaturesCleanup SDMetrics to only accept SDV 1.0 metadata format - Issue#331by @amontanez24Make the diagnostic report more fault-tolerant - Issue#332by @frances-hMaintenanceRemove upper bound for pandas - Issue#338by @pvk-developerv0.9.3 - 2023-04-12This release improves the clarity of warning/error messages. We also add a version add-on, update the workflow to optimize the runtime and fix a bug in theNewRowSynthesismetric when computing thesynthetic_sample_sizefor multi-table.### New FeaturesAdd functionality to find version add-on - Issue#321by @frances-hMore detailed warning in QualityReport when there is a constant input - Issue#316by @pvk-developerMake error more informative in QualityReport when tables cannot be merged - Issue#317by @frances-hMore detailed warning in QualityReport for unexpected category values - Issue#315by @frances-hBug FixesMulti table DiagnosticReport sets synthetic_sample_size too low for NewRowSynthesis - Issue#320by @pvk-developerv0.9.2 - 2023-03-08This release fixes bugs in theNewRowSynthesismetric when too many columns were present. It also fixes bugs around datetime columns that are formatted as strings in bothget_column_pair_plotandget_column_plot.Bug FixesMethod get_column_pair_plot: Does not plot synthetic data if datetime column is formatted as a string - Issue [#310] (https://github.com/sdv-dev/SDMetrics/issues/310) by @frances-hMethod get_column_plot: ValueError if a datetime column is formatted as a string - Issue#309by @frances-hFix ValueError in the NewRowSynthesis metric (also impacts DiagnosticReport) - Issue#307by @frances-hv0.9.1 - 2023-02-17This release fixes bugs in the existing metrics and reports.Bug FixesFix issue-296 for discrete and continuous columns - Issue#296by @R-PalazzoSupport new metadata for datetime_format - Issue#303by @frances-hv0.9.0 - 2023-01-18This release supports Python 3.10 and drops support for Python 3.6. We also add a verbosity argument to report generation.New FeaturesSilent mode when creating reports. - Issue#269by @katxiaoSupport Python versions >=3.7 and <3.11 - Issue287by @katxiaov0.8.1 - 2022-12-09This release fixes bugs in the existing metrics and reports. We also make the reports compatible with future SDV versions.New FeaturesFilter out additional sdtypes that will be available in future versions of SDV - Issue#265by @katxiaoNewRowSynthesis should ignore PrimaryKey column - Issue#260by @katxiaoBug FixesVisualization crashes if there are metric errors - Issue#272by @katxiaoScore for TVComplement if synthetic data only has missing values - Issue#271by @katxiaoFix 'timestamp' column metadata in the multi table demo - Issue#267by @katxiaoFix 'duration' column in the single table demo - Issue#266by @katxiaoREADME.md example has a bug - Issue#262by @katxiaoUpdate README.md to fix a bug - Issue#263by @katxiaoVisualization get_column_pair_plot: update parameter name to column_names - Issue#258by @katxiao"Column Shapes" and "Column Pair Trends" Calculation Inconsistency - Issue#254by @katxiaoDiagnostic Report missing RangeCoverage for numerical columns - Issue#255by @katxiaov0.8.0 - 2022-11-02This release introduces theDiagnosticReport, which helps a user verify – at a quick glance – that their data is valid. We also fix an existing bug with detection metrics.New FeaturesFixes for new metadata - Issue#253by @katxiaoAdd default synthetic sample size to DiagnosticReport - Issue#248by @katxiaoExclude pii columns from single table metrics - Issue#245by @katxiaoAccept both old and new metadata - Issue#244by @katxiaoAddress Diagnostic Report and metric edge cases - Issue#243by @katxiaoUpdate visualization average per table - Issue#242by @katxiaoAdd save and load functionality to multi-table DiagnosticReport - Issue#218by @katxiaoVisualization methods for the multi-table DiagnosticReport - Issue#217by @katxiaoAdd getter methods to multi-table DiagnosticReport - Issue#216by @katxiaoCreate multi-table DiagnosticReport - Issue#215by @katxiaoVisualization methods for the single-table DiagnosticReport - Issue#211by @katxiaoAdd getter methods to single-table DiagnosticReport - Issue#210by @katxiaoCreate single-table DiagnosticReport - Issue#209by @katxiaoAdd save and load functionality to single-table DiagnosticReport - Issue#212by @katxiaoAdd single table diagnostic report - Issue#237by @katxiaoBug FixesDetection test test doesn't look at metadata when determining which columns to use - Issue#119by @R-PalazzoInternal ImprovementsRemove torch dependency - Issue#233by @katxiaoUpdate README - Issue#250by @katxiaov0.7.0 - 2022-09-27This release introduces theQualityReport, which evaluates how well synthetic data captures mathematical properties from the real data. TheQualityReportincorporates the new metrics introduced in the previous release, and allows users to get detailed results, visualize the scores, and save the report for future viewing. We also add utility methods for visualizing columns and pairs of columns.New FeaturesCatch typeerror in new row synthesis query - Issue#234by @katxiaoAdd NewRowSynthesis Metric - Issue#207by @katxiaoUpdate plot utilities API - Issue#228by @katxiaoFix column pairs visualization bug - Issue#230by @katxiaoSave version - Issue#229by @katxiaoUpdate efficacy metrics API - Issue#227by @katxiaoAdd RangeCoverage Metric - Issue#208by @katxiaoAdd get_column_pairs_plot utility method - Issue#223by @katxiaoParse date as datetime - Issue#222by @katxiaoUpdate error handling for reports - Issue#221by @katxiaoVisualization API update - Issue#220by @katxiaoBug fixes for QualityReport - Issue#219by @katxiaoUpdate column pair metric calculation - Issue#214by @katxiaoAdd get score methods for multi table QualityReport - Issue#190by @katxiaoAdd multi table QualityReport visualization methods - Issue#192by @katxiaoAdd plot_column visualization utility method - Issue#193by @katxiaoAdd save and load behavior to multi table QualityReport - Issue#188by @katxiaoCreate multi-table QualityReport - Issue#186by @katxiaoAdd single table QualityReport visualization methods - Issue#191by @katxiaoAdd save and load behavior to single table QualityReport - Issue#187by @katxiaoAdd get score methods for single table Quality Report - Issue#189by @katxiaoCreate single-table QualityReport - Issue#185by @katxiaoInternal ImprovementsAuto apply "new" label instead of "pending review" - Issue#164by @katxiaofix typo - Issue#195by @fealhov0.6.0 - 2022-08-12This release removes SDMetric's dependency on the RDT library, and also introduces new quality and diagnostic metrics. Additionally, we introduce a newcompute_breakdownmethod that returns a breakdown of metric results.New FeaturesHandle null values correctly - Issue#194by @katxiaoAdd wrapper classes for new single and multi table metrics - Issue#169by @katxiaoAdd CorrelationSimilarity metric - Issue#143by @katxiaoAdd CardinalityShapeSimilarity metric - Issue#160by @katxiaoAdd CardinalityStatisticSimilarity metric - Issue#145by @katxiaoAdd ContingencySimilarity Metric - Issue#159by @katxiaoAdd TVComplement metric - Issue#142by @katxiaoAdd MissingValueSimilarity metric - Issue#139by @katxiaoAdd CategoryCoverage metric - Issue#140by @katxiaoAdd compute breakdown column for single column - Issue#152by @katxiaoAdd BoundaryAdherence metric - Issue#138by @katxiaoGet KSComplement Score Breakdown - Issue#130by @katxiaoAdd StatisticSimilarity Metric - Issue#137by @katxiaoNew features for KSTest.compute - Issue#129by @amontanez24Internal ImprovementsAdd integration tests and fixes - Issue#183by @katxiaoRemove rdt hypertransformer dependency in timeseries metrics - Issue#176by @katxiaoReplace rdt LabelEncoder with sklearn - Issue#178by @katxiaoRemove rdt as a dependency - Issue#182by @katxiaoUse sklearn's OneHotEncoder instead of rdt - Issue#170by @katxiaoRemove KSTestExtended - Issue#180by @katxiaoRemove TSFClassifierEfficacy and TSFCDetection metrics - Issue#171by @katxiaoUpdate the default tags for a feature request - Issue#172by @katxiaoBump github macos version - Issue#174by @katxiaoFix pydocstyle to check sdmetrics - Issue#153by @pvk-developerUpdate the RDT version to 1.0 - Issue#150by @pvk-developerUpdate slack invite link - Issue#132by @pvk-developerv0.5.0 - 2022-05-11This release fixes an error  where the relationalKSTestcrashes if a table doesn't have numerical columns.
It also includes some housekeeping, updating the pomegranate and copulas version requirements.Issues closedCap pomegranate to <0.14.7 - Issue#116by @csalaRelational KSTest crashes with IncomputableMetricError if a table doesn't have numerical columns - Issue#109by @katxiaov0.4.1 - 2021-12-09This release improves the handling of metric errors, and updates the default transformer behavior used in SDMetrics.Issues closedReport metric errors from compute_metrics - Issue#107by @katxiaoSpecify default categorical transformers - Issue#105by @katxiaov0.4.0 - 2021-11-16This release adds support for Python 3.9 and updates dependencies to ensure compatibility with the
rest of the SDV ecosystem, and upgrades to the latestsRDTrelease.Issues closedReplacesktimeforpyts- Issue#103by @pvk-developerAdd support for Python 3.9 - Issue#102by @pvk-developerIncrease code style lint - Issue#80by @fealhoAddpip checktoCIworkflows - Issue#79by @pvk-developerUpgrade dependency ranges - Issue#69by @katxiaov0.3.2 - 2021-08-16This release makespomegranatean optional dependency.Issues closedMake pomegranate an optional dependency - Issue#63by @fealhov0.3.1 - 2021-07-12This release fixes a bug to make the privacy metrics available in the API docs.
It also updates dependencies to ensure compatibility with the rest of the SDV ecosystem.Issues closedCategoricalSVMnot being imported - Issue#65by @csalav0.3.0 - 2021-03-30This release includes privacy metrics to evaluate if the real data could be obtained or
deduced from the synthetic samples. Additionally all the metrics have anormalizemethod
which takes theraw_scoregenerated by the metric and returns a value between0and1.Issues closedAdd normalize method to metrics - Issue#51by @csala and @fealhoImplement privacy metrics - Issue#36by @ZhuofanXie and @fealhov0.2.0 - 2021-02-24Dependency upgrades to ensure compatibility with the rest of the SDV ecosystem.v0.1.3 - 2021-02-13Updates the required dependecies to facilitate a conda release.Issues closedUpgrade sktime - Issue#49by @fealhov0.1.2 - 2021-01-27Big fixing release that addresses several minor errors.Issues closedMore splits than classes - Issue#46by @fealhoScipy 1.6.0 causes an AttributeError - Issue#44by @fealhoTime series metrics fails with variable length timeseries - Issue#42by @fealhoParentChildDetection metrics KeyError - Issue#39by @csalav0.1.1 - 2020-12-30This version adds Time Series Detection and Efficacy metrics, as well as a fix
to ensure that Single Table binary classification efficacy metrics work well
with binary targets which are not boolean.Issues closedTimeseries efficacy metrics - Issue#35by @csalaTimeseries detection metrics - Issue#34by @csalaEnsure binary classification targets are bool - Issue#33by @csalav0.1.0 - 2020-12-18This release introduces a new project organization and API, with metrics
grouped by data modality, with a common API:Single ColumnColumn PairSingle TableMulti TableTime SeriesWithin each data modality, different families of metrics have been implemented:StatisticalDetectionBayesian Network and Gaussian Mixture LikelihoodMachine Learning Efficacyv0.0.4 - 2020-11-27Patch release to relax dependencies and avoid conflicts when using the latest SDV version.v0.0.3 - 2020-11-20Fix error on detection metrics when input data contains infinity or NaN values.Issues closedValueError: Input contains infinity or a value too large for dtype('float64') - Issue#11by @csalav0.0.2 - 2020-08-08Add support for Python 3.8 and a broader range of dependencies.v0.0.1 - 2020-06-26First release to PyPI.