Package: dp-cgans
👯 DP-CGANS (Differentially Private - Conditional Generative Adversarial NetworkS)Abstract: This repository presents a Conditional Generative Adversary Networks (GANs) on tabular data (and RDF data) combining with Differential Privacy techniques. Our pre-print publication:Generating synthetic personal health data using conditional generative adversarial networks combining with differential privacy.Author: Chang Sun, Institute of Data Science, Maastricht UniversityStart date: Nov-2021Status: Under developmentNote: "Standing on the shoulders of giants". This repository is inspired by the excellent work ofCTGANfromSynthetic Data Vault (SDV),Tensorflow Privacy, andRdfPdans. Highly appreciate they shared the ideas and implementations, made code publicly available, well-written documentation. More related work can be found in the References below.This package is extended from SDV (https://github.com/sdv-dev/SDV), CTGAN (https://github.com/sdv-dev/CTGAN), and Differential Privacy in GANs (https://github.com/civisanalytics/dpwgan). The author modified the conditional matrix and cost functions to emphasize on the relations between variables. The main changes are in ctgan/synthesizers/ctgan.py ../data_sampler.py ../data_transformer.py📥️ InstallationYou will need Python >=3.8+ and <3.10pipinstalldp-cgans🪄 Usage⌨️ Use as a command-line interfaceYou can easily generate synthetic data for a file using your terminal after installingdp-cganswith pip.To quickly run our example, you can download theexample data:wgethttps://raw.githubusercontent.com/sunchang0124/dp_cgans/main/resources/example_tabular_data_UCIAdult.csvThen rundp-cgans:dp-cgansgenexample_tabular_data_UCIAdult.csv--epochs2--outputout.csv--gen-size100Get a full rundown of the available options for generating synthetic data with:dp-cgansgen--help🐍 Use with pythonThis library can also be used directly in python scriptsIf your input is tabular data (e.g., csv):fromdp_cgansimportDP_CGANimportpandasaspdtabular_data=pd.read_csv("../resources/example_tabular_data_UCIAdult.csv")# We adjusted the original CTGAN model from SDV. Instead of looking at the distribution of individual variable, we extended to two variables and keep their corrllmodel=DP_CGAN(epochs=100,# number of training epochsbatch_size=1000,# the size of each batchlog_frequency=True,verbose=True,generator_dim=(128,128,128),discriminator_dim=(128,128,128),generator_lr=2e-4,discriminator_lr=2e-4,discriminator_steps=1,private=False,)print("Start training model")model.fit(tabular_data)model.save("generator.pkl")# Generate 100 synthetic rowssyn_data=model.sample(100)syn_data.to_csv("syn_data_file.csv")🧑‍💻 Development setupFor development, we recommend to install and useHatch, as it will automatically install and sync the dependencies when running development scripts. But you can also directly create a virtual environment and install the library withpip install -e .InstallClone the repository:gitclonehttps://github.com/sunchang0124/dp_cganscddp_cgansWhen working in development thehatchtool will automatically install and sync the dependencies when running a script. But you can also directlyRunRun the library with the CLI:hatch-vrundp-cgansgen--helpYou can also enter a new shell with the virtual environments automatically activated:hatchshell
dp-cgansgen--helpTestsRun the tests locally:hatchrunpytest-sFormatRun formatting and linting (black and ruff):hatchrunfmtReset the virtual environmentsIn case the virtual environments is not updating as expected you can easily reset it with:hatchenvprune📦️ New release processThe deployment of new releases is done automatically by a GitHub Action workflow when a new release is created on GitHub. To release a new version:Make sure thePYPI_API_TOKENsecret has been defined in the GitHub repository (in Settings > Secrets > Actions). You can get an API token from PyPIhere.Increment theversionnumber insrc/dp_cgans/__init__.pyfile:hatchversionfix# Bump from 0.0.1 to 0.0.2hatchversionminor# Bump from 0.0.1 to 0.1.0hatchversion0.1.1# Bump to the specified versionCreate a new release on GitHub, which will automatically trigger the publish workflow, and publish the new release to PyPI.You can also manually build and publish from you laptop:hatchbuild
hatchpublish📚️ References / Further readingThere are many excellent work on generating synthetic data using GANS and other methods. We list the studies that made great conbributions for the field and inspiring for our work.GANSSynthetic Data Vault (SDV) [Paper] [Github]Modeling Tabular Data using Conditional GAN (a part of SDV) [Paper] [Github]Wasserstein GAN [Paper]Improved Training of Wasserstein GANs [Paper]Synthesising Tabular Data using Wasserstein Conditional GANs with Gradient Penalty (WCGAN-GP) [Paper]PacGAN: The power of two samples in generative adversarial networks [Paper]CTAB-GAN: Effective Table Data Synthesizing [Paper]Conditional Tabular GAN-Based Two-Stage Data Generation Scheme for Short-Term Load Forecasting [Paper]TabFairGAN: Fair Tabular Data Generation with Generative Adversarial Networks [Paper]Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning [Paper]Differential PrivacyTensorflow Privacy [Github]Renyi Differential Privacy [Paper]DP-CGAN : Differentially Private Synthetic Data and Label Generation [Paper]Differentially Private Generative Adversarial Network [Paper] [Github] Another implementation [Github]Private Data Generation Toolbox [Github]autodp: Automating differential privacy computation [Github]Differentially Private Synthetic Medical Data Generation using Convolutional GANs [Paper]DTGAN: Differential Private Training for Tabular GANs [Paper]DIFFERENTIALLY PRIVATE SYNTHETIC DATA: APPLIED EVALUATIONS AND ENHANCEMENTS [Paper]FFPDG: FAST, FAIR AND PRIVATE DATA GENERATION [Paper]OthersEvoGen: a Generator for Synthetic Versioned RDF [Paper]Generation and evaluation of synthetic patient data [Paper]Fake It Till You Make It: Guidelines for Effective Synthetic Data Generation [Paper]Generating and evaluating cross-sectional synthetic electronic healthcare data: Preserving data utility and patient privacy [Paper]Synthetic data for open and reproducible methodological research in social sciences and official statistics [Paper]A Study of the Impact of Synthetic Data Generation Techniques on Data Utility using the 1991 UK Samples of Anonymised Records [Paper]