Package: sdgx
🚀 Synthetic Data GeneratorSynthetic Data Generator (SDG) is a framework focused on quickly generating high-quality structured tabular data. It supports many single-table and multi-table data synthesis algorithms, achieving up to 120 times performance improvement, and supports differential privacy and other methods to enhance the security of synthesized data.Synthetic data is generated by machines based on real data and algorithms, it does not contain sensitive information, but can retain the characteristics of real data.
There is no correspondence between synthetic data and real data, and it is not subject to privacy regulations such as GDPR and ADPPA.
In practical applications, there is no need to worry about the risk of privacy leakage.
High-quality synthetic data can also be used in various fields such as data opening, model training and debugging, system development and testing, etc.🎉 Featureshigh performanceSupports a wide range of statistical data synthesis algorithms to achieve up to 120x performance improvement, without the need for GPU devices;Optimised for big data scenarios, effectively reducing memory consumption;Continuously tracking the latest advances in academia and industry, and introducing support for excellent algorithms and models in a timely manner.Provide distributed training support for deep learning models with frameworks such as torch.Privacy enhancements:SDG supports differential privacy, anonymization and other methods to enhance the security of synthetic data.Easy to extendSupports expansion of models, data processing, data connectors, etc. in the form of plug-in packagesReadthe latest API docsfor more details.🔛 Quick StartPre-build imageYou can use pre-built images to quickly experience the latest features.dockerpullidsteam/sdgx:latestLocal Install (Recommended)At present, the code of this project is updated very quickly. We recommend that you use SDG by installing it through the source code.gitclonegit@github.com:hitsz-ids/synthetic-data-generator.git
pipinstall.# Or install from gitpipinstallgit+https://github.com/hitsz-ids/synthetic-data-generator.gitInstall from PyPipipinstallsdgxQuick Demo of Single Table Data Generation and MetricDemo codefromsdgx.data_connectors.csv_connectorimportCsvConnectorfromsdgx.models.ml.single_table.ctganimportCTGANSynthesizerModelfromsdgx.synthesizerimportSynthesizerfromsdgx.utilsimportdownload_demo_data# This will download demo data to ./datasetdataset_csv=download_demo_data()# Create data connector for csv filedata_connector=CsvConnector(path=dataset_csv)# Initialize synthesizer, use CTGAN modelsynthesizer=Synthesizer(model=CTGANSynthesizerModel(epochs=1),# For quick demodata_connector=data_connector,)# Fit the modelsynthesizer.fit()# Samplesampled_data=synthesizer.sample(1000)print(sampled_data)ComparisonReal data are as follows：>>>data_connector.read()ageworkclassfnlwgteducation...capitallosshoursperweeknative-countryclass0        2State-gov77516Bachelors...02United-States<=50K13Self-emp-not-inc83311Bachelors...00United-States<=50K22Private215646HS-grad...02United-States<=50K33Private23472111th...02United-States<=50K41Private338409Bachelors...02Cuba<=50K..............................488372Private215419Bachelors...02United-States<=50K488384NaN321403HS-grad...02United-States<=50K488392Private374983Bachelors...03United-States<=50K488402Private83891Bachelors...02United-States<=50K488411Self-emp-inc182148Bachelors...03United-States>50K[48842rowsx15columns]Synthetic data are as follows：>>>sampled_dataageworkclassfnlwgteducation...capitallosshoursperweeknative-countryclass0      1NaN28219Some-college...02Puerto-Rico<=50K12Private250166HS-grad...02United-States>50K22Private50304HS-grad...02United-States<=50K34Private89318Bachelors...02Puerto-Rico>50K41Private172149Bachelors...03United-States<=50K.............................9952NaN208938Bachelors...01United-States<=50K9962Private166416Bachelors...22United-States<=50K9972NaN336022HS-grad...01United-States<=50K9983Private198051Masters...02United-States>50K9991NaN41973HS-grad...02United-States<=50K[1000rowsx15columns]🤝 Join CommunityThe SDG project was initiated byInstitute of Data Security, Harbin Institute of Technology. If you are interested in out project, welcome to join our community. We welcome organizations, teams, and individuals who share our commitment to data protection and security through open source:ReadCONTRIBUTINGbefore draft a pull request.Submit an issue by viewingView First Good Issueor submit a Pull Request.ContributorsZhongsheng Ji💻MoooCat💻YUAN KAIWEN💻sjh120💻Z712023💻👩‍🎓 Related WorkResearch PaperCTGAN：Modeling Tabular Data using Conditional GANTVAE：Modeling Tabular Data using Conditional GANtable-GAN：Data Synthesis based on Generative Adversarial NetworksCTAB-GAN:CTAB-GAN: Effective Table Data SynthesizingOCT-GAN:OCT-GAN: Neural ODE-based Conditional Tabular GANsDatasetAdultSatelliteRossmannTelstra📄 LicenseThe SDG open source project uses Apache-2.0 license, please refer to theLICENSE.Star History