Package: ctgan
This repository is part ofThe Synthetic Data Vault Project, a project fromDataCebo.OverviewCTGAN is a collection of Deep Learning based synthetic data generators for single table data, which are able to learn from real data and generate synthetic data with high fidelity.Important Links:computer:WebsiteCheck out the SDV Website for more information about our overall synthetic data ecosystem.:orange_book:BlogA deeper look at open source, synthetic data creation and evaluation.:book:DocumentationQuickstarts, User and Development Guides, and API Reference.:octocat:RepositoryThe link to the Github Repository of this library.:keyboard:Development StatusThis software is in its Pre-Alpha stage.CommunityJoin our Slack Workspace for announcements and discussions.Currently, this library implements theCTGANandTVAEmodels described in theModeling Tabular data using Conditional GANpaper, presented at the 2019 NeurIPS conference.InstallUse CTGAN through the SDV library:warning: If you're just getting started with synthetic data, we recommend installing the SDV library which provides user-friendly APIs for accessing CTGAN. :warning:The SDV library provides wrappers for preprocessing your data as well as additional usability features like constraints. See theSDV documentationto get started.Use the CTGAN standalone libraryAlternatively, you can also install and useCTGANdirectly, as a standalone library:Usingpip:pipinstallctganUsingconda:condainstall-cpytorch-cconda-forgectganWhen using the CTGAN library directly, you may need to manually preprocess your data into the correct format, for example:Continuous data must be represented as floatsDiscrete data must be represented as ints or stringsThe data should not contain any missing valuesUsage ExampleIn this example we load theAdult Census Dataset* which is a built-in demo dataset. We use CTGAN to learn from the real data and then generate some synthetic data.fromctganimportCTGANfromctganimportload_demoreal_data=load_demo()# Names of the columns that are discretediscrete_columns=['workclass','education','marital-status','occupation','relationship','race','sex','native-country','income']ctgan=CTGAN(epochs=10)ctgan.fit(real_data,discrete_columns)# Create synthetic datasynthetic_data=ctgan.sample(1000)*For more information about the dataset see:
Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml].
Irvine, CA: University of California, School of Information and Computer Science.Join our communityJoin ourSlack channelto discuss more about CTGAN and synthetic data. If you find a bug or have a feature request, you can alsoopen an issueon our GitHub.Interested in contributing to CTGAN?Read ourContribution Guideto get started.Citing CTGANIf you use CTGAN, please cite the following work:Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, Kalyan Veeramachaneni.Modeling Tabular data using Conditional GAN. NeurIPS, 2019.@inproceedings{ctgan,
  title={Modeling Tabular data using Conditional GAN},
  author={Xu, Lei and Skoularidou, Maria and Cuesta-Infante, Alfredo and Veeramachaneni, Kalyan},
  booktitle={Advances in Neural Information Processing Systems},
  year={2019}}Related ProjectsPlease note that these projects are external to the SDV Ecosystem. They are not affiliated with or maintained by DataCebo.R Interface for CTGAN: A wrapper aroundCTGANthat brings the functionalities toRusers.
More details can be found in the corresponding repository:https://github.com/kasaai/ctganCTGAN Server CLI: A package to easily deploy CTGAN onto a remote server. Created by Timothy Pillow @oregonpillow at:https://github.com/oregonpillow/ctgan-server-cliThe Synthetic Data Vault Projectwas first created at MIT'sData to AI Labin 2016. After 4 years of research and traction with enterprise, we
createdDataCeboin 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.Get started using the SDV package-- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.Historyv0.9.0 - 2024-02-13This release makes CTGAN sampling more efficient by saving the frequency of each categorical value.New FeaturesImprove DataSampler efficiency - Issue [#327] ((https://github.com/sdv-dev/CTGAN/issue/327)) by @fealhov0.8.0 - 2023-11-13This release adds a progress bar that will show when setting theverboseparameter toTruewhen initializingTVAE.New FeaturesAdd verbosity TVAE (progress bar + save the loss values) - Issue [#300]((https://github.com/sdv-dev/CTGAN/issues/300) by @frances-hv0.7.5 - 2023-10-05This release adds a progress bar that will show when setting theverboseparameter to True when initializingCTGAN. It also removes a warning that was showing.MaintenanceRemove model_missing_values from ClusterBasedNormalizer call - PR#310by @fealhoSwitch default branch from master to main - Issue#311by @amontanez24Remove or implement CTGAN tests - Issue#312by @fealhoNew FeaturesAdd progress bar for CTGAN fitting (+ save the loss values) - Issue#298by @frances-hv0.7.4 - 2023-07-25This release adds support for Python 3.11 and drops support for Python 3.7.MaintenanceWhy is there an upper bound in the packaging requirement? (packaging<22) - Issue#276by @fealhoAdd support for Python 3.11 - Issue#296by @fealhoDrop support for Python 3.7 - Issue#302by @fealhov0.7.3 - 2023-05-25This release adds support for Torch 2.0!Bugs FixedTorch 2.0 fails with cuda=False - Issue#288by @amontanez24MaintenanceUpgrade to torch 2.0 - Issue#280by @frances-hv0.7.2 - 2023-05-09This release adds support for Pandas 2.0! It also fixes a bug in theload_demofunction.Bugs Fixedload_demo raises urllib.error.HTTPError: HTTP Error 403: Forbidden - Issue#284by @amontanez24MaintenanceRemove upper bound for pandas - Issue#282by @frances-hv0.7.1 - 2023-02-23This release fixes a bug that prevented theCTGANmodel from being saved after sampling.Bugs FixedCannot save CTGANSynthesizer after sampling (TypeError) - Issue#270by @pvk-developerv0.7.0 - 2023-01-20This release adds support for python 3.10 and drops support for python 3.6. It also fixes a couple of the most common warnings that were surfacing.New FeaturesSupport Python 3.10 and 3.11 - Issue#259by @pvk-developerBugs FixedFix SettingWithCopyWarning (may be leading to a numerical calculation bug) - Issue#215by @amontanez24FutureWarning in data_transformer with pandas 1.5.0 - Issue#246by @amontanez24MaintenanceCTGAN Package Maintenance Updates - Issue#257by @amontanez24v0.6.0 - 2022-10-07This release renames the models in CTGAN.CTGANSynthesizeris now calledCTGANandTVAESynthesizeris now calledTVAE.New FeaturesRename synthesizers - Issue#243by @amontanez24v0.5.2 - 2022-08-18This release updates CTGAN to use the latest version of RDT. It also includes performance and robustness updates to the data transformer.Issues closedBump rdt version - Issue#242by @katxiaoSingle thread data transform is slow for huge table - Issue#151by @mfhbreeFix RDT api - Issue#232by @pvk-developerUpdate macos to use latest version. - Issue#237by @pvk-developerUpdate the RDT version to 1.0 - Issue#224by @pvk-developerUpdate slack invite link. - Issue#222by @pvk-developerrobustness fix, when data have less rows than the default number of cl… - Issue#211by @Deathn0tv0.5.1 - 2022-02-25This release fixes a bug with the decoder instantiation, and also allows users to set a random state for the model
fitting and sampling.Issues closedUpdate self.decoder with correct variable name - Issue#203by @tejuafonjaAdd random state - Issue#204by @katxiaov0.5.0 - 2021-11-18This release adds support for Python 3.9 and updates dependencies to ensure compatibility with the
rest of the SDV ecosystem, and upgrades to the latestsRDTrelease.Issues closedAdd support for Python 3.9 - Issue#177by @pvk-developerAdd pip check to CI workflows - Issue#174by @pvk-developerTypo inCTGANcode - Issue#158by @ori-katz100 and @fealhov0.4.3 - 2021-07-12Dependency upgrades to ensure compatibility with the rest of the SDV ecosystem.v0.4.2 - 2021-04-27In this release, the way in which the loss function of the TVAE model was computed has been fixed.
In addition, the default value of thediscriminator_decayhas been changed to a more optimal
value. Also some improvements to the tests were added.Issues closedTVAE: loss function - Issue#143by @fealho and @DingfanChenSetdiscriminator_decayto1e-6- Pull request#145by @fealhoAdds unit tests - Pull requests#140by @fealhov0.4.1 - 2021-03-30This release exposes all the hyperparameters which the user may find useful for bothCTGANandTVAE. AlsoTVAEcan now be fitted on datasets that are shorter than the batch
size and drops the last batch only if the data size is not divisible by the batch size.Issues closedTVAE: Adaptbatch_sizeto data size - Issue#135by @fealho and @csalaValueErrorfromvalidate_discre_columnswithuniqueCombinationConstraint- Issue133by @fealho and @MLjunggv0.4.0 - 2021-02-24Maintenance relese to upgrade dependencies to ensure compatibility with the rest
of the SDV libraries.Also add a validation on the CTGANcondition_columnandcondition_valueinputs.ImprovementsValidate condition_column and condition_value - Issue#124by @fealhov0.3.1 - 2021-01-27ImprovementsCheck discrete_columns valid before fitting -Issue #35by @fealhoBugs fixedValueError: max() arg is an empty sequence -Issue #115by @fealhov0.3.0 - 2020-12-18In this release we add a new TVAE model which was presented in the original CTGAN paper.
It also exposes more hyperparameters and moves epochs and log_frequency from fit to the constructor.A new verbose argument has been added to optionally disable unnecessary printing, and a new hyperparameter
calleddiscriminator_stepshas been added to CTGAN to control the number of optimization steps performed
in the discriminator for each generator epoch.The code has also been reorganized and cleaned up for better readability and interpretability.Special thanks to @Baukebrenninkmeijer @fealho @leix28 @csala for the contributions!ImprovementsAdd TVAE -Issue #111by @fealhoMovelog_frequencyto__init__-Issue #102by @fealhoAdd discriminator steps hyperparameter -Issue #101by @BaukebrenninkmeijerCode cleanup / Expose hyperparameters -Issue #59by @fealho and @leix28Publish to conda repo -Issue #54by @fealhoBugs fixedFixed NaN != NaN counting bug. -Issue #100by @fealhoUpdate dependencies and testing -Issue #90by @csalav0.2.2 - 2020-11-13In this release we introduce several minor improvements to make CTGAN more versatile and
propertly support new types of data, such as categorical NaN values, as well as conditional
sampling and features to save and load models.Additionally, the dependency ranges and python versions have been updated to support up
to date runtimes.Many thanks @fealho @leix28 @csala @oregonpillow and @lurosenb for working on making this release possible!ImprovementsDrop Python 3.5 support -Issue #79by @fealhoSupport NaN values in categorical variables -Issue #78by @fealhoSample synthetic data conditioning on a discrete column -Issue #69by @leix28Support recent versions of pandas -Issue #57by @csalaEasy solution for restoring original dtypes -Issue #26by @oregonpillowBugs fixedLoss to nan -Issue #73by @fealhoSwapped the sklearn utils testing import statement -Issue #53by @lurosenbv0.2.1 - 2020-01-27Minor version including changes to ensure the logs are properly printed and
the option to disable the log transformation to the discrete column frequencies.Special thanks to @kevinykuo for the contributions!Issues Resolved:Option to sample from true data frequency instead of logged frequency -Issue #16by @kevinykuoFlush stdout buffer for epoch updates -Issue #14by @kevinykuov0.2.0 - 2019-12-18Reorganization of the project structure with a new Python API, new Command Line Interface
and increased data format support.Issues Resolved:Reorganize the project structure -Issue #10by @csalaMove epochs to the fit method -Issue #5by @csalav0.1.0 - 2019-11-07First Release - NeurIPS 2019 Version.