Package: sdgym
This repository is part ofThe Synthetic Data Vault Project, a project fromDataCebo.OverviewThe Synthetic Data Gym (SDGym) is a benchmarking framework for modeling and generating
synthetic data. Measure performance and memory usage across different synthetic data modeling
techniques – classical statistics, deep learning and more!The SDGym library integrates with the Synthetic Data Vault ecosystem. You can use any of its
synthesizers, datasets or metrics for benchmarking. You can also customize the process to include
your own work.Datasets: Select any of the publicly available datasets from the SDV project, or input your own data.Synthesizers: Choose from any of the SDV synthesizers and baselines. Or write your own custom
machine learning model.Evaluation: In addition to performance and memory usage, you can also measure synthetic data
quality and privacy through a variety of metrics.InstallInstall SDGym using pip or conda. We recommend using a virtual environment to avoid conflicts with other software on your device.pipinstallsdgymcondainstall-cpytorch-cconda-forgesdgymFor more information about using SDGym, visit theSDGym Documentation.UsageLet's benchmark synthetic data generation for single tables. First, let's define which modeling
techniques we want to use. Let's choose a few synthesizers from the SDV library and a few others
to use as baselines.# these synthesizers come from the SDV library# each one uses different modeling techniquessdv_synthesizers=['GaussianCopulaSynthesizer','CTGANSynthesizer']# these basic synthesizers are available in SDGym# as baselinesbaseline_synthesizers=['UniformSynthesizer']Now, we can benchmark the different techniques:importsdgymsdgym.benchmark_single_table(synthesizers=(sdv_synthesizers+baseline_synthesizers))The result is a detailed performance, memory and quality evaluation across the synthesizers
on a variety of publicly available datasets.Supplying a custom synthesizerBenchmark your own synthetic data generation techniques. Define your synthesizer by
specifying the training logic (using machine learning) and the sampling logic.defmy_training_logic(data,metadata):# create an object to represent your synthesizer# train it using the datareturnsynthesizerdefmy_sampling_logic(trained_synthesizer,num_rows):# use the trained synthesizer to create# num_rows of synthetic datareturnsynthetic_dataLearn more in theCustom Synthesizers Guide.Customizing your datasetsThe SDGym library includes many publicly available datasets that you can include right away.
List these using theget_available_datasetsfeature.sdgym.get_available_datasets()dataset_name   size_MB     num_tables
KRK_v1         0.072128    1
adult          3.907448    1
alarm          4.520128    1
asia           1.280128    1
...You can also include any custom, private datasets that are stored on your computer on an
Amazon S3 bucket.my_datasets_folder = 's3://my-datasets-bucket'For more information, see the docs forCustomized Datasets.What's next?Visit theSDGym Documentationto learn more!The Synthetic Data Vault Projectwas first created at MIT'sData to AI Labin 2016. After 4 years of research and traction with enterprise, we
createdDataCeboin 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.Get started using the SDV package-- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.Historyv0.7.0 - 2023-06-13This release adds support for SDV 1.0 and PyTorch 2.0!New FeaturesAdd functions to top level import - Issue#229by @fealhoCleanup SDGym to the new SDV 1.0 metadata and synthesizers - Issue#212by @fealhoBugs Fixedlimit_dataset_size causes sdgym to crash - Issue#231by @fealhobenchmark_single_table crashes with metadata dict - Issue#232by @fealhoPassing None as synthesizers runs all of them - Issue#233by @fealhotimeout parameter causes sdgym to crash - Issue#234by @pvk-developerSDGym is not working with latest torch - Issue#210by @amontanez24Fix sdgym --help - Issue#206by @katxiaoInternalIncrease code style lint - Issue#123by @fealhoRemove code support for synthesizers that are not strings/classes - PR#236by @fealhoCode Refactoring - Issue#215by @fealhoMaintenanceRemove pomegranate - Issue#230by @amontanez24v0.6.0 - 2023-02-01This release introduces methods for benchmarking single table data and creating custom synthesizers, which can be based on existing SDGym-defined synthesizers or on user-defined functions. This release also adds support for Python 3.10 and drops support for Python 3.6.New FeaturesBenchmarking progress bar should update on one line - Issue#204by @katxiaoSupport local additional datasets folder with zip files - Issue#186by @katxiaoEnforce that each synthesizer is unique in benchmark_single_table - Issue#190by @katxiaoSimplify the file names inside the detailed_results_folder - Issue#191by @katxiaoUse SDMetrics silent report generation - Issue#179by @katxiaoRemove arguments in get_available_datasets - Issue#197by @katxiaoAccept metadata.json as valid metadata file - Issue#194by @katxiaoCheck if file or folder exists before writing benchmarking results - Issue#196by @katxiaoRename benchmarking argument "evaluate_quality" to "compute_quality_score" - Issue#195by @katxiaoAdd option to disable sdmetrics in benchmarking - Issue#182by @katxiaoPrefix remote bucket with 's3' - Issue#183by @katxiaoBenchmarking error handling - Issue#177by @katxiaoAllow users to specify custom synthesizers' display names - Issue#174by @katxiaoUpdate benchmarking results columns - Issue#172by @katxiaoAllow custom datasets - Issue#166by @katxiaoUse new datasets s3 bucket - Issue#161by @katxiaoCreate benchmark_single_table method - Issue#151by @katxiaoUpdate summary metrics - Issue#134by @katxiaoBenchmark individual methods - Issue#159by @katxiaoAdd method to create a sdv variant synthesizer - Issue#152by @katxiaoAdd method to generate a multi table synthesizer - Issue#149by @katxiaoAdd method to create single table synthesizers - Issue#148by @katxiaoUpdating existing synthesizers to new API - Issue#154by @katxiaoBug FixesPip encounters dependency issues with ipython - Issue#187by @katxiaoIndependentSynthesizer is printing out ConvergeWarning too many times - Issue#192by @katxiaoSize values in benchmarking results seems inaccurate - Issue#184by @katxiaoImport error in the example for benchmarking the synthesizers - Issue#139by @katxiaoUpdates and bugfixes - Issue#132by @csalaMaintenanceUpdate README - Issue#203by @katxiaoSupport Python Versions >=3.7 and <3.11 - Issue#170by @katxiaoSDGym Package Maintenance Updates documentation  - Issue#163by @katxiaoRemove YData - Issue#168by @katxiaoUpdate to newest SDV - Issue#157by @katxiaoUpdate slack invite link. - Issue#144by @pvk-developerupdating workflows to work with windows - Issue#136by @amontanez24Update conda dependencies - Issue#130by @katxiaov0.5.0 - 2021-12-13This release adds support for Python 3.9, and updates dependencies to accept the latest versions when possible.Issues closedAdd support for Python 3.9 -Issue #127by @katxiaoAdd pip check worflow -Issue #124by @pvk-developerFix meta.yaml dependencies -PR #119by @fealhoUpgrade dependency ranges -Issue #118by @katxiaov0.4.1 - 2021-08-20This release fixed a bug where passing ajsonfile as configuration for a multi-table synthesizer crashed the model.
It also adds a number of fixes and enhancements, including: (1) a function and CLI command to list the available synthesizer names,
(2) a curate set of dependencies and makingGretelinto an optional dependency, (3) updatingGretelto use temp directories,
(4) usingnvidia-smito get the number of gpus and (5) multipledockerfileupdates to improve functionality.Issues closedBug when using JSON configuration for multiple multi-table evaluation -Issue #115by @pvk-developerUse nvidia-smi to get number of gpus -PR #113by @katxiaoList synthesizer names -Issue #82by @fealhoUse nvidia base for dockerfile -PR #108by @katxiaoAdd Makefile target to install gretel and ydata -PR #107by @katxiaoCurate dependencies and make Gretel optional -PR #106by @csalaUpdate gretel checkpoints to use temp directory -PR #105by @katxiaoInitialize variable before reference -PR #104by @katxiaov0.4.0 - 2021-06-17This release adds new synthesizers for Gretel and ydata, and creates a Docker image for SDGym.
It also includes enhancements to the accepted SDGym arguments, adds a summary command to aggregate
metrics, and adds the normalized score to the benchmark results.New FeaturesAdd normalized score to benchmark results -Issue #102by @katxiaoAdd max rows and max columns args -Issue #96by @katxiaoAutomatically detect number of workers -Issue #97by @katxiaoAdd summary function and command -Issue #92by @amontanez24Allow jobs list/JSON to be passed -Issue #93by @fealhoAdd ydata to sdgym -Issue #90by @fealhoAdd dockerfile for sdgym -Issue #88by @katxiaoAdd Gretel to SDGym synthesizer -Issue #87by @amontanez24v0.3.1 - 2021-05-20This release adds new features to store results and cache contents into an S3 bucket
as well as a script to collect results from a cache dir and compile a single results
CSV file.Issues closedCollect cached results from s3 bucket -Issue #85by @katxiaoStore cache contents into an S3 bucket -Issue #81by @katxiaoStore SDGym results into an S3 bucket -Issue #80by @katxiaoAdd a way to collect cached results -Issue #79by @katxiaoAllow reading datasets from private s3 bucket -Issue #74by @katxiaoTypos in the sdgym.run function docstring documentation -Issue #69by @sbrugmanv0.3.0 - 2021-01-27Major rework of the SDGym functionality to support a collection of new features:Add relational and timeseries model benchmarking.Use SDMetrics for model scoring.Update datasets format to match SDV metadata based storage format.Centralize default datasets collection in thesdv-datasetsS3 bucket.Add options to download and use datasets from different S3 buckets.Rename synthesizers to baselines and adapt to the new metadata format.Add model execution and metric computation time logging.Add optional synthetic data and error traceback caching.v0.2.2 - 2020-10-17This version adds a rework of the benchmark function and a few new synthesizers.New FeaturesNew CLI withrun,make-leaderboardandmake-summarycommandsParallel execution via Dask or MultiprocessingDownload datasets without executing the benchmarkSupport for python from 3.6 to 3.8New Synthesizerssdv.tabular.CTGANsdv.tabular.CopulaGANsdv.tabular.GaussianCopulaOneHotsdv.tabular.GaussianCopulaCategoricalsdv.tabular.GaussianCopulaCategoricalFuzzyv0.2.1 - 2020-05-12New updated leaderboard and minor improvements.New FeaturesAdd parameters for PrivBNSynthesizer -Issue #37by @csalav0.2.0 - 2020-04-10New Becnhmark API and lots of improved documentation.New FeaturesThe benchmark function now returns a complete leaderboard instead of only one scoreClass Synthesizers can be directly passed to the benchmark functionBug FixesOne hot encoding errors in the Independent, VEEGAN and Medgan Synthesizers.Proper usage of theevalmode during sampling.Fix improperly configured datasets.v0.1.0 - 2019-08-07First release to PyPi