Package: sdv
This repository is part ofThe Synthetic Data Vault Project, a project fromDataCebo.OverviewTheSynthetic Data Vault(SDV) is a Python library designed to be your one-stop shop for
creating tabular synthetic data. The SDV uses a variety of machine learning algorithms to learn
patterns from your real data and emulate them in synthetic data.Features:brain:Create synthetic data using machine learning.The SDV offers multiple models, ranging
from classical statistical methods (GaussianCopula) to deep learning methods (CTGAN). Generate
data for single tables, multiple connected tables or sequential tables.:bar_chart:Evaluate and visualize data.Compare the synthetic data to the real data against a
variety of measures. Diagnose problems and generate a quality report to get more insights.:arrows_counterclockwise:Preprocess, anonymize and define constraints.Control data
processing to improve the quality of synthetic data, choose from different types of anonymization
and define business rules in the form of logical constraints.Important LinksTutorialsGet some hands-on experience with the SDV. Launch the tutorial notebooks and run the code yourself.:book:DocsLearn how to use the SDV library with user guides and API references.:orange_book:BlogGet more insights about using the SDV, deploying models and our synthetic data community.CommunityJoin our Slack workspace for announcements and discussions.:computer:WebsiteCheck out the SDV website for more information about the project.InstallThe SDV is publicly available under theBusiness Source License.
Install SDV using pip or conda. We recommend using a virtual environment to avoid conflicts with
other software on your device.pipinstallsdvcondainstall-cpytorch-cconda-forgesdvGetting StartedLoad a demo dataset to get started. This dataset is a single table describing guests staying at a
fictional hotel.fromsdv.datasets.demoimportdownload_demoreal_data,metadata=download_demo(modality='single_table',dataset_name='fake_hotel_guests')The demo also includesmetadata, a description of the dataset, including the data types in each
column and the primary key (guest_email).Synthesizing DataNext, we can create anSDV synthesizer,  an object that you can use to create synthetic data.
It learns patterns from the real data and replicates them to generate synthetic data. Let's use
theFAST_MLpreset synthesizer, which is optimized for performance.fromsdv.liteimportSingleTablePresetsynthesizer=SingleTablePreset(metadata,name='FAST_ML')synthesizer.fit(data=real_data)And now the synthesizer is ready to create synthetic data!synthetic_data=synthesizer.sample(num_rows=500)The synthetic data will have the following properties:Sensitive columns are fully anonymized.The email, billing address and credit card number
columns contain new data so you don't expose the real values.Other columns follow statistical patterns.For example, the proportion of room types, the
distribution of check in dates and the correlations between room rate and room type are preserved.Keys and other relationships are intact.The primary key (guest email) is unique for each row.
If you have multiple tables, the connection between a primary and foreign keys makes sense.Evaluating Synthetic DataThe SDV library allows you to evaluate the synthetic data by comparing it to the real data. Get
started by generating a quality report.fromsdv.evaluation.single_tableimportevaluate_qualityquality_report=evaluate_quality(real_data,synthetic_data,metadata)Creating report: 100%|██████████| 4/4 [00:00<00:00, 19.30it/s]
Overall Quality Score: 89.12%
Properties:
Column Shapes: 90.27%
Column Pair Trends: 87.97%This object computes an overall quality score on a scale of 0 to 100% (100 being the best) as well
as detailed breakdowns. For more insights, you can also visualize the synthetic vs. real data.fromsdv.evaluation.single_tableimportget_column_plotfig=get_column_plot(real_data=real_data,synthetic_data=synthetic_data,column_name='amenities_fee',metadata=metadata)fig.show()What's Next?Using the SDV library, you can synthesize single table, multi table and sequential data. You can
also customize the full synthetic data workflow, including preprocessing, anonymization and adding
constraints.To learn more, visit theSDV Demo page.CreditsThank you to our team of contributors who have built and maintained the SDV ecosystem over the
years!View ContributorsCitationIf you use SDV for your research, please cite the following paper:Neha Patki, Roy Wedge, Kalyan Veeramachaneni.The Synthetic Data Vault.IEEE DSAA 2016.@inproceedings{
    SDV,
    title={The Synthetic data vault},
    author={Patki, Neha and Wedge, Roy and Veeramachaneni, Kalyan},
    booktitle={IEEE International Conference on Data Science and Advanced Analytics (DSAA)},
    year={2016},
    pages={399-410},
    doi={10.1109/DSAA.2016.49},
    month={Oct}
}The Synthetic Data Vault Projectwas first created at MIT'sData to AI Labin 2016. After 4 years of research and traction with enterprise, we
createdDataCeboin 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.Get started using the SDV package-- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.Release Notes1.10.0 - 2024-02-15This release adds multiple improvements to handling premium transformers and column relationships, including using premium transformers even if the PII flag is set to true. Additionally, the SDV now warns users to save the metadata after auto-detection has been used. Semantic sdtype detection has also been improved to tokenize column names to prevent unexpected substring matches.This release also fixes a few warning bugs and fixes an issue that would causemetadata.to_dictto fail for metadata loaded from older versions of the SDV. A few synthesizer bugs were also resolved. The quality of the sequence_index for thePARSynthesizerhas been improved, and an issue that would causeCTGANSynthesizer,TVAESynthesizer, andCopulaGANSynthesizerto crash if all columns were to be generated from scratch has been fixed.Bugs FixedHMASynthesizer sometimes creates null values (out-of-bounds parameters synthesized) - Issue#1691by @fealhoUnable to conditionally sample some rows when using aScalarRangeconstraint - Issue#1737by @fealhoMetadata.to_dict fails on metadata instances pre 1.9.0 - Issue#1739by @amontanez24Metadata auto-detection should not assign a primary key if there are NaN values in it - Issue#1740by @R-Palazzo'' object has no attribute '_model' - Issue#1741by @fealhoColumn relationship warning should be raised during synthesizer initialization only - Issue#1750by @R-PalazzoImprove quality ofsequence_index: Move the start dates into the context model - Issue#1760by @frances-hAdd-ons warning is raised twice for multi table synthesizers.  - Issue#1768by @R-PalazzoNew FeaturesMetadata auto-detection should tokenize words before determining PII - Issue#1725by @fealhoProvide a friendlier error if data is stored as dtype'category'(CTGAN, TVAE) - Issue#1735by @frances-hAllow the ability to easily remove primary keys - Issue#1742by @frances-hConstraint should not be set on columns inside a gps relationship - Issue#1748by @R-PalazzoSet the default transformer for GPS column relationship - Issue#1749by @R-PalazzoAdd aversionmodule to align with SDV Enterprise - Issue#1761by @R-PalazzoWarn users to save their metadata file after auto-detecting/updating it - Issue#1762by @R-PalazzoSet the GPSNoiser as default transformer for GPS column relationship  - Issue#1767by @R-PalazzoUpdate transformer assignment logic for handling pii - Issue#1775by @R-Palazzo1.9.0 - 2024-01-11This release makes a number of improvements. It introduces a new concept to the metadata known as column relationships! Column relationships can be used to define when certain groups of columns in a table should be treated as a special concept (eg. address). You can add a column relationship by using the newadd_column_relationshipmethod. The metadata detection was also improved by allowing semantic sdtypes (eg. 'email', 'phone_number') to be detected as primary keys.This release also patches some bugs. An issue messing up the likelihood matching in theHMASynthesizerwas resolved. TheCTGANSynthesizerno longer fails when using theFixedCombinationsconstraint. TheInequalityconstraint was also patched to handle datetimes better.DeprecationsTheset_address_columnsmethod is deprecated in favor ofadd_column_relationship.New FeaturesImprove error messages for composite keys - Issue#1684by @frances-hAdd column relationship validation to single table metadata - Issue#1698by @frances-hAdd add_column_relationship method to single table metadata - Issue#1699by @frances-hMake synthesizers work with column_relationships - Issue#1700by @R-PalazzoMetadata auto-detection should find primary keys of semantic sdtypes - Issue#1724by @fealhoBugs FixedInvalidDataError for Inequality constraint (even though data is valid) - Issue#1692by @fealhoBaseIndependentSamplercrashes because it tries to cast id columns - Issue#1712by @pvk-developerKeyError inCTGANSynthesizerwhen applyingFixedCombinationsconstraint - Issue#1717by @pvk-developerFix _get_likelihoods not generating likelihood values - Issue#1720by @frances-h1.8.0 - 2023-12-05This release adds support for the new Diagnostic Report from SDMetrics. This report calculates scores for three basic but important properties of your data: data validity, data structure and in the multi table case, relationship validity. Data validity checks that the columns of your data are valid (eg. correct range or values). Data structure makes sure the synthetic data has the correct columns. Relationship validity checks to make sure key references are correct and the cardinality is within ranges seen in the real data.Additionally, a few bugs were fixed and functionality was improved around synthesizers. It is now possible to access the loss values for theTVAESynthesizerandCTGANSynthesizerby using theget_loss_valuesmethod. Theget_parametersmethod is now more detailed and returns all the parameters used to make a synthesizer. The metadata is now capable of detecting some common pii sdtypes. Finally, a bug that made every parent row generated by theHMASynthesizerhave at least one child row was patched. This should improve cardinality.MaintenanceAddressSettingWithCopyWarning(HMASynthesizer) - Issue#1557by @pvk-developerBump SDMetrics version - Issue#1702by @amontanez24New FeaturesAllow me to access loss values for GAN-based synthesizers  - Issue#1671by @frances-hCreate a unifiedget_parametersmethod for all multi-table synthesizers - Issue#1674by @frances-hSet credentials key as variables - Issue#1680by @R-PalazzoIdentifying PII Sdtypes in Metadata - Issue#1683by @R-PalazzoMake SDV compatible with the latest SDMetrics - Issue#1687by @fealhoSingleTablePreset uses FrequencyEncoder - Issue#1695by @fealhoBugs FixedHMASynthesizer creates too much synthetic data (always creates a child for every parent row) - Issue#1673by @frances-h1.7.0 - 2023-11-16This release adds an alert to theCTGANSynthesizerduring preprocessing. The alert informs the user if the fitting of the synthesizer is likely to be slow on their schema. Additionally, it is now possible to enforce that sampled datetime values stay within the range of the fitted data!This release also makes internal changes to support address data in SDV Enterprise.New FeaturesAdd set_address_columns method - Issue#1593by @R-PalazzoUpdate_transformers should raise error on address columns - Issue#1594by @R-Palazzoadd_constraints should raise error on address columns - Issue#1595by @R-PalazzoPrint alert if CTGANSynthesizer is likely to be slow - Issue#1658by @fealhoSet enforce_min_max_values to True for datetime transformers - Issue#1676by @R-PalazzoBugs FixedUnable to visualize metadata (Error: bad label formatandCalledProcessError) - Issue#1625by @fealhoCan't set address columns after fitting - Issue#1661by @R-Palazzo1.6.0 - 2023-11-07This release improves user messaging in multiple ways. The most notable is that users will now see an alert if theHMASynthesizeris likely to be slow for their data's schema. Additionally, the logger messaging for constraints and the error messaging when setting distributions on non-parametric models was made more detailed.The visualization plots in thesdv.evaluationsub-package all got a new parameter calledplot_type, allowing the users to specify the plot type to use if the one being inferred is not useful. Thesdv.datasets.local.load_csvsmethod now has a parameter calledread_csv_parameters, that allow users to specify how the csvs should be read during loading. The same change was also made to thesdv.metadata.multi_table.detect_table_from_csv,sdv.metadata.multi_table.detect_from_csvsandsdv.metadata.single_table.detect_from_csvmethods.Multiple bugs were resolved including one that caused new categories to be created during the sample step ofCTGANSynthesizer.New FeaturesImprove debug messages when a constraint falls back to reject sampling approach - Issue#1478by @amontanez24Constraints should work with timezone-aware datetime columns - Issue#1576by @fealhoBetter error message when trying to get distributions from non-parametric models - PR#1633by @frances-hAdd options to read CSV files - Issue#1644by @lajohn4747Print alert if HMASynthesizer is likely to be slow - Issue#1646by @lajohn4747Make SDV compatible with SDMetrics 0.12.1 - Issue#1650by @pvk-developerMake function to estimate number of columns CTGAN produces - Issue#1657by @fealhoBugs FixedIn get_available_demos, the num_tables column should be an int - Issue#1420by @lajohn4747AttributeError when using specific locale strings (es_AR, fr_BE) - Issue#1439by @lajohn4747Confusing error when passing in an empty dataframe (with constraints) - Issue#1455by @lajohn4747HMASynthesizer: Better error message for learned distributions (misleading fit error) - Issue#1579by @fealhoFix tests in SDV after update in RDT 1.7.1 - Issue#1638by @lajohn4747CTGAN sometimes creates new categories (int data) - Issue#1647by @pvk-developerCTGAN sometimes creates new categories (object data) - Issue#1648by @pvk-developerBetter error message if I provide an incompatible sdtype/locale combo - Issue#1653by @pvk-developer1.5.0 - 2023-10-13Several improvements and bug fixes were made in this release. Most notably, the metadata detection was substantially improved. Support for the 'unknown' sdtype was added, providing more flexibility in data representation. The software now attempts to intelligently detect primary keys and identify parent-child relationships in the metadata, streamlining the metadata creation process.Additionally, issues related to conditional sampling with negative float values, the inability to update transformers for columns created by constraints, and compatibility with numpy version 1.25 and higher were addressed. The default branch was also switched from 'master' to 'main' for better development practices. Various bugs and errors, including those involving HMA and datetime format detection, were also resolved.New FeaturesImprove metadata detection - Issue#1515by @R-PalazzoSupport 'unknown' sdtype - Issue#1516by @R-PalazzoDetect primary keys in metadata - Issue#1521by @frances-hDetect relationships in MultiTableMetadata - Issue#1522by @frances-hMake function to estimate number of columns HMA produces. - Issue#1572by @fealhoAdd wrapper for get_cardinalty_plot - Issue#1573by @frances-h[Metadata detection] Add a cardinality cap when choosing between categorical vs. numerical  - Issue#1584by @pvk-developer[Metadata Detection] Only make primary/foreign keys sdtypeid(leave others asunknown) - Issue#1598by @amontanez24Check and supply a more descriptive error when trying to use'gaussian_kde'with HMA - Issue#1604by @frances-hBugs FixedConditional sampling with negative float values doesn't work - Issue#1161by @fealhoCannot update transformers for columns that get created by constraints (KeyError) - Issue#1454by @frances-hHMA produces KeyError for a schema with 3+ levels of depth - Issue#1558by @fealhoColumns consisting of only Nones are being detected as datetime - Issue#1589by @pvk-developerHMASynthesizer throws an error when sampling multi table models with three levels of depths - Issue#1600by @amontanez24ValueError: Invalid distribution specificationwhen setting numerical_distributions on child table (HMA) - Issue#1605by @fealhoBug: updating transformers in DataProcessor resets warning filters - Issue#1618by @rwedgeMaintenanceInvestigate how to get numpy >1.25 to pass - Issue#1501by @rwedgeSwitch default branch from master to main - Issue#1550by @amontanez241.4.0 - 2023-08-23This release makes multiple improvements to the metadata. Both the single and multi table metadata classes now have avalidate_datamethod. This method runs checks to validate the data against the current specifications in the metadata. TheSingleTableMetadata.visualizeis also improved. The sequence index is now shown in the same section as the sequence key. It also now shows all key and index information (eg. sequence key, primary key, sequence index) in one section.TheCTGANSynthesizerhas been made more efficient in the following ways:Boolean columns are now being skipped duringpreprocesslike categorial columns are.It is possible to apply other transformations to categorical columns and haveCTGANskip the one-hot encoding step.Additional changes include that the columns labeled with the sdtypeidwill now go through theIDGeneratortransformer by default and constraint transformations that were being overwritten during sampling will now be respected.New FeaturesAdd validate_data method to Metadata - Issue#1518by @fealhoUse IDGenerator for ID columns - Issue#1519by @frances-hMetadata visualization for sequential data: Only create 2 sections - Issue#1543by @frances-hBugs FixedInefficient CTGAN modeling when adding categorical transformers - Issue#1450by @fealhoCTGANSynthesizer is assigning LabelEncoder to boolean columns (instead of None) - Issue#1530by @fealhoMetadata visualization for sequential data: Missing sequence index - Issue#1542by @frances-hConstraint outputs are being overwritten in DataProcessor.reverse_transform - Issue#1551by @amontanez241.3.0 - 2023-08-14This release adds two new methods to theMultiTableMetadata:detect_from_csvsanddetect_From_dataframes. These methods allow you to detect metadata for a whole dataset at once by either loading them from a folder or a dictionary mapping table names to thepandas.DataFrames. TheSingleTableMetadatacan now be visualized! Additionally, there is now asummarizedoption in theshow_table_detailsparameter of thevisualizemethods. This will print each sdtype in the table and the number of columns that have that sdtype.Additionally, this release patches a bug that prevented custom constraints from working on columns that were primary or alternate keys. It also adds support for Python 3.11!New FeaturesAlign default transformers between SDV and RDT - Issue#1484by @R-PalazzoAdd visualize method to SingleTableMetadata - Issue#1517by @pvk-developerAdd detect_from_csvs and detect_from_dataframes methods to MultiTableMetadata - Issue#1520by @R-PalazzoAllow empty tables to be fitted using fit_processed_data - Issue#1524by @fealhoSummarized metadata visualization - Issue#1525by @pvk-developerBugs FixedCannot use custom constraint transforms for certain columns (inconsistent ordering in forward vs. reverse) - Issue#1476by @fealhoCannot create custom constraint with primary key - Issue#1514by @amontanez24MaintenanceAdd support for Python 3.11 - Issue#1459by @fealho1.2.1 - 2023-07-13This release fixes a bug that caused theInequalityconstraint and others to fail if there were None values in a datetime column.Bugs FixedInequality fails with None and datetime - Issue#1471by @pvk-developerMaintenanceDrop support for Python 3.7 - Issue#1487by @pvk-developerInternalMake HMA use hierarchical sampling mixin - Issue#1428by @frances-hMove progress bar out of base multi table synthesizer - Issue#1486by @R-Palazzo1.2.0 - 2023-06-07This release adds a parameter calledverboseto theHMASynthesizer. Setting it to True will show progress bars during the fitting steps. Additionally, performance optimizations were made to the modeling and initialization of theHMASynthesizer.Multiple changes were made to enhance constraints. TheRangeconstraint was improved to be able to generate more accurate data when null values are provided. Constraints are also now validated against the data when runningvalidate()on any synthesizer.Finally, some warnings were resolved.New FeaturesReport fitting progress for the HMASynthesizer - Issue#1440by @pvk-developerBugs FixedRange constraint does not produce cases of missing values & may create violative data - Issue#1393by @R-PalazzoSynthesizers don't validate constraints during validate() - Issue#1402by @pvk-developerConfusing error during metadata validation - Issue#1417by @frances-hSettingWithCopyWarning when conditional sampling -#1436by @pvk-developerHMASynthesizer is modeling child tables - Issue#1442by @pvk-developerValueError when sampling PII columns - Issue#1445by @pvk-developerInternalAdd BaseHierarchicalSampler Mixin - Issue#1394by @frances-hAdd BaseIndependentSampler Mixin - Issue#1395by @frances-hSynthesizers created twice during HMA init - Issue#1418by @frances-hGet rid of unnecessary methods for single table sampling - Issue#1430by @amontanez24Detect all addons from top levelinit- PR#1453by @frances-hMaintenanceUpgrade to torch 2.0 - Issue#1365by @fealhoDuring fit, there is a FutureWarning (due to RDT 1.5.0) - Issue#1456by @amontanez241.1.0 - 2023-05-10This release adds a new initialization parameter to synthesizers calledlocalesthat allows users to set the locales to use for all columns that have a locale basedsdtype(eg.addressorphone_number). Additionally, it adds support for Pandas 2.0!Multiple enhancements were made to improve the performance of data and metadata validation in synthesizers. TheInequalityconstraint was improved to be able to generate more scenarios of data concerning the presence of NaNs. Finally, many warnings have been resolved.New FeaturesAdd add-on detection for new constraints - Issue#1397by @frances-hAdd add-on detection for multi and single table synthesizers - Issue#1385by @frances-hSetting a locale for all my anonymized (PII) columns - Issue#1371by @frances-hBugs FixedSkip checking for Faker function if its a default sdtype - PR#1410by @pvk-developerInequality constraint does not produce all possibilities of missing values - Issue#1392by @R-PalazzoDeprecated locale warning - Issue#1400by @frances-hMaintenanceUse cached Faker instance to discover if an sdtype is a Faker function. - Issue#1405by @pvk-developerUpgrade to pandas 2.0 - Issue#1366by @pvk-developerInternalRefactor Multi Table Modeling - Issue#1387by @pvk-developerPytestConfigWarning: Unknown config option: collect_ignore - Issue#1376by @amontanez24Pandas FutureWarning: Could not cast to int64 - Issue#1357by @R-PalazzoRuntimeWarning: invalid value encountered in cast. - Issue#1369by @amontanez241.0.1 - 2023-04-20This release improves theload_custom_constraint_classesmethod by removing thetable_nameparameter and just loading the constraint
for all tables instead. It also improves some error messages as well as removes some of the warnings that have been surfacing.Support for sdtypes is enhanced by resolving a bug that was incorrecttly specifying Faker functions for some of them. Support for datetime formats has also been improved. Finally, thepathargument in somesaveandloadmethods was changed tofilepathfor consistency.New FeaturesMethod load_custom_constraint_classes does not need table_name parameter - Issue#1354by @R-PalazzoImprove error message for invalid primary keys - Issue#1341by @R-PalazzoAdd functionality to find version add-on - Issue#1309by @frances-hBugs FixedCertain sdtypes cause Faker to raise error - Issue#1346by @frances-hChange path to filepath for load and save methods - Issue#1352by @fealhoSome datetime formats cause InvalidDataError, even if the datetime matches the format - Issue#1136by @amontanez24InternalInequality constraint raises RuntimeWarning (invalid value encountered in log) - Issue#1275by @frances-hPandas FutureWarning: Default dtype for Empty Series will be 'object' - Issue#1355by @R-PalazzoPandas FutureWarning: Length 1 tuple will be returned - Issue#1356by @R-Palazzo1.0.0 - 2023-03-28This is a major release that introduces a new API to theSDVaimed at streamlining the process of synthetic data generation! To achieve this, this release includes the addition of several large features.MetadataSome of the most notable additions are the newSingleTableMetadataandMultiTableMetadataclasses. These classes enable a number of features that make it easier to synthesize your data correctly such as:Automatic data detection - Callingmetadata.detect_from_dataframe()ormetadata.detect_from_csv()will populate the metadata autonomously with values it thinks represent the data.Easy updating - Once an instance of the metadata is created, values can be easily updated using a number of methods defined in the API. For more info, view thedocs.Metadata validation - Callingmetadata.validate()will return a report of any invalid definitions in the metadata specification.Upgrading - Users with the previous metadata format can easily update to the new specification using theupgrade_metadata()method.Saving and loading - The metadata itself can easily be saved to a json file and loaded back up later.Class and Module NamesAnother major change is the renaming of our core modeling classes and modules. The name changes are meant to highlight the difference between the underlying machine learning models, and the objects responsible for the end-to-end workflow of generating synthetic data. The main name changes are as follows:tabular->single_tablerelational->multi_tabletimeseries->sequentialBaseTabularModel->BaseSingleTableSynthesizerGaussianCopula->GaussianCopulaSynthesizerCTGAN->CTGANSynthesizerTVAE->TVAESynthesizerCopulaGan->CopulaGANSynthesizerPAR->PARSynthesizerHMA1->HMASynthesizerInSDV1.0, synthesizers are classes that take in metadata and handle data preprocessing, model training and model sampling. This is similar to the previousBaseTabularModelinSDV<1.0.Synthetic Data WorkflowSynthesizersinSDV1.0 define a clear workflow for generating synthetic data.Synthesizers are initialized with a metadata class.They can then be used to transform the data and apply constraints using thesynthesizer.preprocess()method. This step also validates that the data matches the provided metadata to avoid errors in fitting or sampling.The processed data can then be fed into the underlying machine learning model usingsynthesizer.fit_processed_data(). (Alternatively, data can be preprocessed and fit to the model usingsynthesizer.fit().)Data can then be sampled usingsynthesizer.sample().Each synthesizer class also provides a series of methods to help users customize the transformations their data goes through. Read more about thathere.Notice that the preprocessing and model fitting steps can now be separated. This can be helpful if preprocessing is time consuming or if the data has been processed externally.Other Highly Requested FeaturesAnother major addition is control over randomization. InSDV<1.0, users could set a seed to control the randomization for only some columns. InSDV1.0, randomization is controlled for all columns. Every new call to sample generates new data, but the synthesizer's seed can be reset to the original state usingsynthesizer.reset_randomization(), enabling reproducibility.SDV 1.0adds accessibility and transparency into the transformers used for preprocessing and underlying machine learning models.Using thesynthesizer.get_transformers()method, you can access the transformers used to preprocess each column and view their properties. This can be useful for debugging and accessing privacy information like mappings used to mask data.Distribution parameters learned by copula models can be accessed using thesynthesizer.get_learned_distributions()method.PII handling is improved by the following features:Primary keys can be set to natural sdtypes (eg. SSN, email, name). Previously they could only be numerical or text.ThePseudoAnonymizedFakercan be used to provide consistent mapping to PII columns. As mentioned before, the mapping itself can be accessed by viewing the transformers for the column usingsynthesizer.get_transformers().A bug causing PII columns to slow down modeling is patched.Finally, the synthetic data can now be easily evaluated using theevaluate_quality()andrun_diagnostic()methods. The data can be compared visually to the actual data using theget_column_plot()andget_column_pair_plot()methods. For more info on how to visualize or interpret the synthetic data evaluation, read the docshere.Issues ResolvedNew FeaturesChange auto_assign_transformers to handle id types - Issue#1325by @pvk-developerChange 'text' sdtype to 'id' - Issue#1324by @frances-hInupgrade_metadata, return the object instead of writing it to a JSON file - Issue#1319by @frances-hInupgrade_metadataindex primary keys should be converted totext- Issue#1318by @amontanez24Addload_from_dictto SingleTableMetadata and MultiTableMetadata - Issue#1314by @amontanez24Throw aSynthesizerInputErrorifFixedCombinationsconstraint is applied to a column that is notbooleanorcategorical- Issue#1306by @frances-hMissingsaveandloadmethods forHMASynthesizer- Issue#1262by @amontanez24Better input validation when creating single and multi table synthesizers - Issue#1242by @fealhoBetter input validation onHMASynthesizer.sample- Issue#1241by @R-PalazzoValidate that relationship must be between aprimary keyandforeign key- Issue#1236by @fealhoImproveupdate_columnvalidation forpiiattribute - Issue#1226by @pvk-developerOrder the output ofget_transformers()based on the metadata - Issue#1222by @pvk-developerLog if anynumerical_distributionswill not be applied - Issue#1212by @fealhoImprove error handling forGaussianCopulaSynthesizer:numerical_distributions- Issue#1211by @fealhoImprove error handling when validatingconstraints- Issue#1210by @fealhoAddfake_companiesdemo - Issue#1209by @amontanez24Allow me to create a custom constraint class and use it in the same file - Issue#1205by @amontanez24Sampling should reset after retraining the model - Issue#1201by @pvk-developerChange function nameHMASynthesizer.update_table_parameters-->set_table_parameters- Issue#1200by @pvk-developerAddget_infomethod to synthesizers - Issue#1199by @fealhoAdd evaluation methods to synthesizer - Issue#1190by @fealhoUpdateevaluate.pyto work with the newmetadata- Issue#1186by @fealhoRemove old code - Issue#1181by @pvk-developerDrop support for python 3.6 and add support for 3.10 - Issue#1176by @fealhoAdd constraint methods to MultiTableSynthesizers - Issue#1171by @fealhoUpdate custom constraint workflow - Issue#1169by @pvk-developerAdd get_constraints method to synthesizers - Issue#1168by @pvk-developerMigrate adding and validating constraints to BaseSynthesizer - Issue#1163by @pvk-developerChange metadata"SCHEMA_VERSION"-->"METADATA_SPEC_VERSION"- Issue#1139by @amontanez24Add ability to reset random sampling - Issue#1130by @pvk-developerAdd get_available_demos - Issue#1129by @fealhoAdd demo loading functionality - Issue#1128by @fealhoUse logging instead of printing in detect methods - Issue#1107by @fealhoAdd save and load methods to synthesizers - Issue#1106by @pvk-developerAdd sampling methods to PARSynthesizer - Issue#1083by @amontanez24Add transformer methods to PARSynthesizer - Issue#1082by @fealhoAdd validate to PARSynthesizer - Issue#1081by @amontanez24Add preprocess and fit methods to PARSynthesizer - Issue#1080by @amontanez24Create SingleTablePreset - Issue#1079by @amontanez24Add sample method to multi-table synthesizers - Issue#1078by @pvk-developerAdd get_learned_distributions method to synthesizers - Issue#1075by @pvk-developerAdd preprocess and fit methods to multi-table synthesizers - Issue#1074by @pvk-developerAdd transformer related methods to BaseMultiTableSynthesizer - Issue#1072by @fealhoAdd validate method toBaseMultiTableSynthesizer- Issue#1071by @pvk-developerCreate BaseMultiTableSynthesizer and HMASynthesizer classes - Issue#1070by @pvk-developerCreate PARSynthesizer - Issue#1055by @amontanez24Raise an error if an invalid sdtype is provided to the metadata - Issue#1042by @amontanez24Only allow datetime and numerical sdtypes to be set as the sequence index - Issue#1030by @amontanez24Change set_alternate_keys to add_alternate_keys and add error handling - Issue#1029by @amontanez24CreateMultiTableMetadata.add_tablemethod - Issue#1024by @amontanez24Add update_transformers to synthesizers - Issue#1021by @fealhoAdd assign_transformers and get_transformers methods to synthesizers - Issue#1020by @pvk-developerAdd fit and fit_processed_data methods to synthesizers - Issue#1019by @pvk-developerAdd preprocess method to synthesizers - Issue#1018by @pvk-developerAdd sampling to synthesizer classes - Issue#1015by @pvk-developerAdd validate method to synthesizer - Issue#1014by @fealhoCreate GaussianCopula, CTGAN, TVAE and CopulaGAN synthesizer classes - Issue#1013by @pvk-developerCreate BaseSynthesizer class - Issue#1012by @pvk-developerAdd constraint conversion to upgrade_metadata - Issue#1005by @amontanez24Add method to generate keys to DataProcessor - Issue#994by @pvk-developerCreate formatter - Issue#970by @fealhoCreate a utility to load multiple CSV files at once - Issue#969by @amontanez24Create a utility to convert old --> new metadata format - Issue#966by @amontanez24Add validation check thatprimary_key,alternate_keysandsequence_keycannot be sdtype categorical - Issue#963by @fealhoAdd anonymization to DataProcessor - Issue#950by @pvk-developerAdd utility methods to DataProcessor - Issue#948by @fealhoAdd fit, transform and reverse_transform to DataProcessor - Issue#947by @amontanez24Create DataProcessor class - Issue#946by @amontanez24Add add_constraint method to MultiTableMetadata - Issue#895by @amontanez24Add key related methods to MultiTableMetadata - Issue#894by @fealhoAdd update_column and add_column methods to MultiTableMetadata - Issue#893by @amontanez24Add detect methods to MultiTableMetadata - Issue#892by @amontanez24Add load_from_json and save_to_json methods to the MultiTableMetadata - Issue#891by @fealhoAdd add_relationship method to MultiTableMetadata - Issue#890by @pvk-developerAdd validate method to MultiTableMetadata - Issue#888by @pvk-developerAdd visualize method to MultiTableMetadata class - Issue#884by @amontanez24Create MultiTableMetadata class - Issue#883by @pvk-developerAdd add_constraint method to SingleTableMetadata - Issue#881by @amontanez24Add key related methods to SingleTableMetadata - Issue#880by @fealhoAdd validate method to SingleTableMetadata - Issue#879by @fealhoAdd _validate_inputs class method to each constraint - Issue#878by @fealhoAdd update_column and add_column methods to SingleTableMetadata - Issue#877by @pvk-developerAdd detect methods to SingleTableMetadata - Issue#876by @pvk-developerAdd load_from_json and save_to_json methods to SingleTableMetadata - Issue#874by @pvk-developerCreate SingleTableMetadata class - Issue#873by @pvk-developerBugs FixedInupgrade_metadata, PII values are being converted to generic categorical columns - Issue#1317by @frances-hPARSynthesizeris missingsaveandloadmethods - Issue#1289by @amontanez24Confusing warning when updating transformers - Issue#1272by @frances-hWhen adding constraints,auto_assign_transformersis showing columns that should no longer exist - Issue#1260by @pvk-developerCannot fit twice if I modify transformers:ValueError: There are non-numerical values in your data.- Issue#1259by @frances-hCannot fit twice if I add constraints:ValueError: There are non-numerical values in your data.- Issue#1258by @frances-hHMASynthesizererrors out when fitting a dataset that has a table which holds primary key and foreign keys only - Issue#1257by @pvk-developerChange ValueErrors to InvalidMetadataErrors - Issue#1251by @frances-hMulti-table should show foreign key transformers as None - Issue#1249by @frances-hCannot useHMASynthesizer.fit_processed_datamore than once (KeyError) - Issue#1240by @frances-hFunctionget_available_demoscrashes if a dataset'snum-tablesorsize-MBcannot be found - Issue#1215by @amontanez24Cannot supply a natural key toHMASynthesizer(wheresdtypeis custom): Error insample- Issue#1214by @pvk-developerUnable to sample when using aPseudoAnonymizedFaker- Issue#1207by @pvk-developerIncorrectsdtypespecified in demo datasetstudent_placements_pii- Issue#1206by @amontanez24Auto assigned transformers for datetime columns don't have the right parameters - Issue#1204by @pvk-developerCannot applyInequalityconstraint on demo dataset's datetime columns - Issue#1203by @pvk-developerpii should not be required to auto-assign faker transformers - Issue#1194by @pvk-developerMisc. bug fixes for SDV 1.0.0 - Issue#1193by @pvk-developerSmall bug fixes in demo module - Issue#1192by @pvk-developerForeign Keys are added as Alternate Keys when upgrading - Issue#1143by @pvk-developerAlternate keys not unique when assigned to a semantic type - Issue#1111by @pvk-developerSynthesizer errors if column is semantic type and pii is False - Issue#1110by @fealhoSampled values not unique if primary key is numerical - Issue#1109by @pvk-developerValidate not called during synthesizer creation - Issue#1105by @pvk-developerSingleTableSynthesizer fit doesn't update rounding - Issue#1104by @amontanez24Methodauto_assign_tranformersalways setsenforce_min_max_values=True- Issue#1095by @fealhoSampled context columns in PAR must be in the same order - Issue#1052by @amontanez24Incorrect schema version printing during detect_table_from_dataframe - Issue#1038by @amontanez24Same relationship can be added twice to MultiTableMetadata - Issue#1031by @amontanez24Miscellaneous metadata bugs - Issue#1026by @amontanez24MaintenanceSDV Package Maintenance Updates - Issue#1140by @amontanez24InternalAdd integration tests for 'Synthesize Sequences' demo - Issue#1295by @pvk-developerAdd integration tests for 'Adding Constraints' demo - Issue#1280by @pvk-developerAdd integration tests to the 'Use Your Own Data' demo - Issue#1278by @frances-hAdd integration tests for 'Synthesize Multi Tables' demo - Issue#1277by @pvk-developerAdd integration tests for 'Synthesize a Table' demo - Issue#1276by @frances-hUpdateget_available_demostests - Issue#1247by @fealhoMake private attributes public in the metadata - Issue#1245by @fealho0.18.0 - 2023-01-24This release adds suppport for Python 3.10 and drops support for 3.6.MaintenanceDrop support for python 3.6 - Issue#1177by @amontanez24Support for python 3.10 - Issue#939by @amontanez24Support Python >=3.10,<4 - Issue#1000by @amontanez240.17.2 - 2022-12-08This release fixes a bug in the demo module related to loading the demo data with constraints. It also adds a name to the demo datasets. Finally, it bumps the version ofSDMetricsused.MaintenanceUpgrade SDMetrics requirement to 0.8.0 - Issue#1125by @katxiaoNew FeaturesProvide a name for the default demo datasets - Issue#1124by @amontanez24Bugs FixedCannot load_tabular_demo with metadata - Issue#1123by @amontanez240.17.1 - 2022-09-29This release bumps the dependency requirements to use the latest version ofSDMetrics.MaintenancePatch release: Bump required version for SDMetrics - Issue#1010by @katxiao0.17.0 - 2022-09-09This release updates the code to use RDT version 1.2.0 and greater, so that those new features are now available in SDV. This changes the transformers that are available in SDV models to be those that are in RDT version 1.2.0. As a result, some arguments for initializing models have changed.Additionally, this release fixes bugs related to loading models with custom constraints. It also fixes a bug that addedNaNsto the index of sampled data when usingsample_remaining_columns.Bugs FixedIncorrect rounding in Custom Constraint example - Issue#941by @amontanez24Can't save the model if use the custom constraint - Issue#928by @pvk-developerUser Guide code fixes - Issue#983by @amontanez24Index contains NaNs when using sample_remaining_columns - Issue#985by @amontanez24Cannot sample after loading a model with custom constraints: TypeError - Issue#984by @pvk-developerSet HyperTransformer config manually, based on Metadata if given - Issue#982by @pvk-developerNew FeaturesChange default metrics for evaluate - Issue#949by @fealhoMaintenanceUpdate the RDT version to 1.0 - Issue#897by @pvk-developer0.16.0 - 2022-07-21This release brings user friendly improvements and bug fixes on theSDVconstraints, to help
users generate their synthetic data easily.Some predefined constraints have been renamed and redefined to be more user friendly & consistent.
The custom constraint API has also been updated for usability. The SDV now automatically determines
the besthandling_strategyto use for each constraint, attemptingtransformby default and
falling back toreject_samplingotherwise. Thehandling_strategyparameters are no longer
included in the API.Finally, this version ofSDValso unifies the parameters for all sampling related methods for
all models (including TabularPreset).Changes to ConstraintsGreatherThanconstraint is now separated in two new constraints:Inequality, which is
intended to be used between two columns, andScalarInequality, which is intended to be used
between a column and a scalar.Betweenconstraint is now separated in two new constraints:Range, which is intended to
be used between three columns, andScalarRange, which is intended to be used between a column
and low and high scalar values.FixedIncrementsa new constraint that makes the data increment by a certain value.Newcreate_custom_constraintfunction available to create custom constraints.Removed ConstraintsRoundingRounding is automatically being handled by therdt.HyperTransformer.ColumnFormulathecreate_custom_constrainttakes place over this one and allows more
advanced usage for the end users.New FeaturesImprove error message for invalid constraints - Issue#801by @fealhoNumerical Instability in Constrained GaussianCopula - Issue#806by @fealhoUnify sampling params for reject sampling - Issue#809by @amontanez24SplitGreaterThanconstraint intoInequalityandScalarInequality- Issue#814by @fealhoSplitBetweenconstraint intoRangeandScalarRange- Issue#815@pvk-developerChangecolumnstocolumn_namesinOneHotEncodingandUniqueconstraints - Issue#816by @amontanez24Update columns parameter inPositiveandNegativeconstraint - Issue#817by @fealhoCreateFixedIncrementsconstraint - Issue#818by @amontanez24Improve datetime handling inScalarInequalityandScalarRangeconstraints - Issue#819by @pvk-developerSupport strict boundaries even when transform strategy is used - Issue#820by @fealhoAddcreate_custom_constraintfactory method - Issue#836by @fealhoInternal ImprovementsRemovehandling_strategyparameter - Issue#833by @amontanez24Removefit_columns_modelparameter - Issue#834by @pvk-developerRemove theColumnFormulaconstraint - Issue#837by @amontanez24Movetable_data.copyto base class of constraints - Issue#845by @fealhoBugs FixedNumerical Instability in Constrained GaussianCopula - Issue#801by @tlranda and @fealhoFix error message forFixedIncrements- Issue#865by @pvk-developerFix constraints with conditional sampling - Issue#866by @amontanez24Fix error message inScalarInequality- Issue#868by @pvk-developerCannot usemax_tries_per_batchon sample:TypeError: sample() got an unexpected keyword argument 'max_tries_per_batch'- Issue#885by @amontanez24Conditional sampling + batch size:ValueError: Length of values (1) does not match length of index (5)- Issue#886by @amontanez24TabularPresetdoesn't support new sampling parameters - Issue#887by @fealhoConditional Sampling:batch_sizeis being set toNoneby default? - Issue#889by @amontanez24Conditional sampling using GaussianCopula inefficient when categories are noised - Issue#910by @amontanez24Documentation ChangesShow theAPIforTabularPresetmodels - Issue#854by @katxiaoUpdate handling constraints doc - Pull Request#856by @amontanez24Update custom costraints documentation - Pull Request#857by @pvk-developer0.15.0 - 2022-05-25This release improves the speed of theGaussianCopulamodel by removing logic that previously searched for the appropriate distribution to
use. It also fixes a bug that was happening when conditional sampling was used with theTabularPreset.The rest of the release focuses on making changes to improve constraints including changing theUniqueCombinationsconstraint toFixedCombinations,
making theUniqueconstraint work with missing values and erroring when null values are seen in theOneHotEncodingconstraint.New FeaturesSilence warnings coming from univariate fit in copulas - Issue#769by @pvk-developerRemove parameters related to distribution search and change default - Issue#767by @fealhoUpdate the UniqueCombinations constraint - Issue#793by @fealhoMake Unique constraint works with nans - Issue#797by @fealhoError out if nans in OneHotEncoding - Issue#800by @amontanez24Bugs FixedUnable to sample conditionally in Tabular_Preset model - Issue#796by @katxiaoDocumentation ChangesSupport GPU computing and progress track? - Issue#478by @fealho0.14.1 - 2022-05-03This release adds aTabularPreset, available in thesdv.litemodule, which allows users to easily optimize a tabular model for speed.
In this release, we also include bug fixes for sampling with conditions, an unresolved warning, and setting field distributions. Finally,
we include documentation updates for sampling and the newTabularPreset.Bugs FixedSampling with conditions={column: 0.0} for float columns doesn't work - Issue#525by @shlomihod and @tssbasresolved FutureWarning with Pandas replaced append by concat - Issue#759by @Deathn0tField distributions bug in CopulaGAN - Issue#747by @katxiaoField distributions bug in GaussianCopula - Issue#746by @katxiaoNew FeaturesSet default transformer to categorical_fuzzy - Issue#768by @amontanez24Model nulls normally when tabular preset has constraints - Issue#764by @katxiaoDon't modify my metadata object - Issue#754by @amontanez24Presets should be able to handle constraints - Issue#753by @katxiaoChange preset optimize_for --> name - Issue#749by @katxiaoCreate a speed optimized Preset - Issue#716by @katxiaoDocumentation ChangesAdd tabular preset docs - Issue#777by @katxiaosdv.sampling module is missing from the API - Issue#740by @katxiao0.14.0 - 2022-03-21This release updates the sampling API and splits the existing functionality into three methods -sample,sample_conditions,
andsample_remaining_columns. We also add support for sampling in batches, displaying a progress bar when sampling with more than one batch,
sampling deterministically, and writing the sampled results to an output file. Finally, we include fixes for sampling with conditions
and updates to the documentation.Bugs FixedFix write to file in sampling - Issue#732by @katxiaoConditional sampling doesn't work if the model has a CustomConstraint - Issue#696by @katxiaoNew FeaturesUpdates to GaussianCopula conditional sampling methods - Issue#729by @katxiaoUpdate conditional sampling errors - Issue#730by @katxiaoEnable Batch Sampling + Progress Bar - Issue#693by @katxiaoCreate sample_remaining_columns() method - Issue#692by @katxiaoCreate sample_conditions() method - Issue#691by @katxiaoImprove sample() method - Issue#690by @katxiaoCreate Condition object - Issue#689by @katxiaoIs it possible to generate data with new set of primary keys? - Issue#686by @katxiaoNo way to fix the random seed? - Issue#157by @katxiaoCan you set a random state for the sdv.tabular.ctgan.CTGAN.sample method? - Issue#515by @katxiaogenerating different synthetic data while training the model multiple times. - Issue#299by @katxiaoDocumentation ChangesTypo in the document documentation - Issue#680by @katxiao0.13.1 - 2021-12-22This release adds support for passing tabular constraints to the HMA1 model, and adds more explicit error handling for
metric evaluation. It also includes a fix for using categorical columns in the PAR model and documentation updates
for metadata and HMA1.Bugs FixedCategorical column after sequence_index column - Issue#314by @fealhoNew FeaturesSupport passing tabular constraints to the HMA1 model - Issue#296by @katxiaoMetric evaluation error handling metrics - Issue#638by @katxiaoDocumentation ChangesMake true/false values lowercase in Metadata Schema specification - Issue#664by @katxiaoUpdate docstrings for hma1 methods - Issue#642by @katxiao0.13.0 - 2021-11-22This release makes multiple improvements to differentConstraintclasses. TheUniqueconstraint can now
handle columns with the nameindexand no longer crashes on subsets of the original data. TheBetweenconstraint can now handle columns with nulls properly. The memory of all constraints was also improved.Various other features and fixes were added. Conditional sampling no longer crashes when thenum_rowsargument
is not provided. Multiple localizations can now be used for PII fields. Scaffolding for integration tests was added
and the workflows now runpip check.Additionally, this release adds support for Python 3.9!Bugs FixedGaussian Copula – Memory Issue in Release 0.10.0 - Issue#459by @xammApplying Unique Constraint errors when calling model.fit() on a subset of data - Issue#610by @xammCalling sampling with conditions and without num_rows crashes - Issue#614by @xammMetadata.visualize with path parameter throws AttributeError - Issue#634by @xammThe Unique constraint crashes when the data contains a column called index - Issue#616by @xammThe Unique constraint cannot handle non-default index - Issue#617by @xammConstraintsNotMetError when applying Between constraint on datetime columns containing null values - Issue#632by @katxiaoNew FeaturesAdds Multi localisations feature for PII fields defined in #308 - PR#609by @xammHousekeeping TasksSupport latest version of Faker - Issue#621by @katxiaoAdd scaffolding for Metadata integration tests - Issue#624by @katxiaoAdd support for Python 3.9 - Issue#631by @amontanez24Internal ImprovementsAdd pip check to CI workflows - Issue#626by @pvk-developerDocumentation ChangesAnonymizing PII in single table tutorials states address field as e-mail type - Issue#604by @xammSpecial thanks to @xamm, @katxiao, @pvk-developer and @amontanez24 for all the work that made this release possible!0.12.1 - 2021-10-12This release fixes bugs in constraints, metadata behavior, and SDV documentation. Specifically, we added
proper handling of data containing null values for constraints and timeseries data, and updated the
default metadata detection behavior.Bugs FixedValueError: The parameter loc has invalid values - Issue#353by @fealhoGaussian Copula is generating different data with metadata and without metadata - Issue#576by @katxiaoMake pomegranate an optional dependency - Issue#567by @katxiaoSmall wording change for Question Issue Template - Issue#571by @katxiaoConstraintsNotMetError when using GreaterThan constraint with datetime - Issue#590by @katxiaoGreaterThan constraint crashing with NaN values - Issue#592by @katxiaoNull values in GreaterThan constraint raises error - Issue#589by @katxiaoColumnFormula raises ConstraintsNotMetError when checking NaN values - Issue#593by @katxiaoGreaterThan constraint raises TypeError when using datetime - Issue#596by @katxiaoFix repository language - Issue#464by @fealhoUpdateinit.py - Issue#578by @dyuliuIndexingError: Unalignable boolean - Issue#446by @fealho0.12.0 - 2021-08-17This release focuses on improving and expanding upon the existing constraints. More specifically, the users can now
(1) specify multiple columns inPositiveandNegativeconstraints, (2) use the newUniqueconstraint and
(3) use datetime data with theBetweenconstraint. Additionaly, error messages have been added and updated
to provide more useful feedback to the user.Besides the added features, several bugs regarding theUniqueCombinationsandColumnFormulaconstraints have been fixed,
and an error in the metadata.json for thestudent_placementsdataset was corrected. The release also added documentation
for thefit_columns_modelwhich affects the majority of the available constraints.New FeaturesChange default fit_columns_model to False - Issue#550by @katxiaoSupport multi-column specification for positive and negative constraint - Issue#545by @sarahmishRaise error when multiple constraints can't be enforced - Issue#541by @amontanez24Create Unique Constraint - Issue#532by @amontanez24Passing invalid conditions when using constraints produces unreadable errors - Issue#511by @katxiaoImprove error message for ColumnFormula constraint when constraint column used in formula - Issue#508by @katxiaoAdd datetime functionality to Between constraint - Issue#504by @katxiaoBugs FixedUniqueCombinations constraint with handling_strategy = 'transform' yields synthetic data with nan values - Issue#521by @katxiao and @csalaUniqueCombinations constraint outputting wrong data type - Issue#510by @katxiao and @csalaUniqueCombinations constraint on only one column gets stuck in an infinite loop - Issue#509by @katxiaoConditioning on a non-constraint column using the ColumnFormula constraint - Issue#507by @katxiaoConditioning on the constraint column of the ColumnFormula constraint - Issue#506by @katxiaoUpdate metadata.json for duration of student_placements dataset - Issue#503by @amontanez24Unit test for HMA1 when working with a single child row per parent row - Issue#497by @pvk-developerUniqueCombinations constraint for more than 2 columns - Issue#494by @katxiao and @csalaDocumentation ChangesAdd explanation of fit_columns_model to API docs - Issue#517by @katxiao0.11.0 - 2021-07-12This release primarily addresses bugs and feature requests related to using constraints for the single-table models.
Users can now enforce scalar comparison with the existingGreaterThanconstraint and apply 5 new constraints:OneHotEncoding,Positive,Negative,BetweenandRounding.
Additionally, the SDV will now auto-apply constraints for rounding numerical values, and for keeping the data within the observed bounds.
All related user guides are updated with the new functionality.New FeaturesAdd OneHotEncoding Constraint - Issue#303by @fealhoGreaterThan Constraint should apply to scalars - Issue#410by @amontanez24Improve GreaterThan constraint - Issue#368by @amontanez24Add Non-negative and Positive constraints across multiple columns- Issue#409by @amontanez24Add Between values constraint - Issue#367by @fealhoEnsure values fall within the specified range - Issue#423by @amontanez24Add Rounding constraint - Issue#482by @katxiaoAdd rounding and min/max arguments that are passed down to the NumericalTransformer - Issue#491by @amontanez24Bugs FixedGreaterThan constraint between Date columns rasises TypeError - Issue#421by @amontanez24GreaterThan constraint's transform strategy fails on columns that are not float - Issue#448by @amontanez24AttributeError on UniqueCombinations constraint with non-strings - Issue#196by @katxiaoUse reject sampling to sample missing columns for constraints - Issue#435by @amontanez24Documentation ChangesEnsure privacy metrics are available in the API docs - Issue#458by @fealhoEnsure forumla constraint is called ColumnFormula everywhere in the docs - Issue#449by @fealho0.10.1 - 2021-06-10This release changes the way we sample conditions to not only group by the conditions passed by the user, but also by the transformed conditions that result from them.Issues resolvedConditionally sampling on variable in constraint should have variety for other variables - Issue#440by @amontanez240.10.0 - 2021-05-21This release improves the constraint functionality by allowing constraints and conditions
at the same time. Additional changes were made to update tutorials.Issues resolvedNot able to use constraints and conditions in the same time - Issue#379by @amontanez24Update benchmarking user guide for reading private datasets - Issue#427by @katxiao0.9.1 - 2021-04-29This release broadens the constraint functionality by allowing for theColumnFormulaconstraint to take lambda functions and returned functions as an input for its formula.It also improves conditional sampling by ensuring that anyidfields generated by the
model remain unique throughout the sampled data.TheCTGANmodel was improved by adjusting a default parameter to be more mathematically
correct.Additional changes were made to improve tutorials as well as fix fragile tests.Issues resolvedTutorials test sometimes fails - Issue#355by @fealhoDuplicate IDs when using reject-sampling - Issue#331by @amontanez24 and @csaladiscriminator_decay should be initialized at 1e-6 but it's 0 - Issue#401by @fealho and @YoucefZemmouriTutorial typo - Issue#380by @fealhoRequest for sdv.constraint.ColumnFormula for a wider range of function - Issue#373by @amontanez24 and @JetfiRex0.9.0 - 2021-03-31This release brings new privacy metrics to the evaluate framework which help to determine
if the real data could be obtained or deduced from the synthetic samples.
Additionally, now there is a normalized score for the metrics, which stays between0and1.There are improvements that reduce the usage of memory ram when sampling new data. Also there
is a new parameter to control the reject sampling crash,graceful_reject_sampling, which if
set toTrueand if it's not possible to generate all the requested rows, it will just issue a
warning and return whatever it was able to generate.TheMetadataobject can now be visualized using different combinations ofnamesanddetails,
which can be set toTrueorFalsein order to display only the table names with details or
without. There is also an improvement on thevalidation, which now will display all the errors
found at the end of the validation instead of only the first one.This version also exposes all the hyperparameters of the modelsCTGANandTVAEto allow a more
advanced usage. There is also a fix for theTVAEmodel on small datasets and it's performance
withNaNvalues has been improved. There is a fix for when usingUniqueCombinationConstraintwith thetransformstrategy.Issues resolvedMemory Usage Gaussian Copula Trained Model consuming high memory when generating synthetic data - Issue#304by @pvk-developer and @AnupamaGangadharAdd option to visualize metadata with only table names - Issue#347by @csalaAdd sample parameter to control reject sampling crash - Issue#343by @fealhoVerbose metadata validation - Issue#348by @csalaMissing the introduction of custom specification for hyperparameters in the TVAE model - Issue#344by @imkhoa99 and @pvk-developer0.8.0 - 2021-02-24This version adds conditional sampling for tabular models by combining a reject-sampling
strategy with the native conditional sampling capabilities from the gaussian copulas.It also introduces several upgrades on the HMA1 algorithm that improve data quality and
robustness in the multi-table scenarios by making changes in how the parameters of the child
tables are aggregated on the parent tables, including a complete rework of how the correlation
matrices are modeled and rebuild after sampling.Issues resolvedFix probabilities contain NaN error - Issue#326by @csalaConditional Sampling for tabular models - Issue#316by @fealho and @csalaHMA1: LinAlgError: SVD did not converge - Issue#240by @csala0.7.0 - 2021-01-27This release introduces a few changes in the HMA1 relational algorithm to decrease modeling
and sampling times, while also ensuring that correlations are properly kept across tables
and also adding support for some relational schemas that were not supported before.A few changes in constraints and tabular models also ensure that situations that produced
errors before now work without errors.Issues resolvedFix unique key generation - Issue#306by @fealhoEnsure tables that contain nothing but ids can be modeled - Issue#302by @csalaMetadata visualization improvements - Issue#301by @csalaMulti-parent re-model and re-sample issue - Issue#298by @csalaSupport datetimes in GreaterThan constraint - Issue#266by @rollervanSupport for multiple foreign keys in one table - Issue#185by @csala0.6.1 - 2020-12-31SDMetrics version is updated to include the new Time Series metrics, which have also
been added to the API Reference and User Guides documentation. Additionally,
a few code has been refactored to reduce external dependencies and a few minor bugs
related to single table constraints have been fixedIssues resolvedAdd timeseries metrics and user guides -Issue #289by @csalaAdd functions to generate regex ids -Issue #288by @csalaSaving a fitted tabular model with UniqueCombinations constraint raises PicklingError -Issue #286by @csalaConstraints:handling_strategy='reject_sampling'causes'ZeroDivisionError: division by zero'-Issue #285by @csala0.6.0 - 2020-12-22This release updates to the latest CTGAN, RDT and SDMetrics libraries to introduce a
new TVAE model, multiple new metrics for single table and multi table, and fixes
issues in the re-creation of tabular models from a metadata dict.Issues resolvedUpgrade to SDMetrics v0.1.0 and addsdv.metricsmodule -Issue #281by @csalaUpgrade to CTGAN 0.3.0 and add TVAE model -Issue #278by @fealhoAdddtype_transformerstoTable.from_dict-Issue #276by @csalaFix Metadatafrom_dictbehavior -Issue #275by @csala0.5.0 - 2020-11-25This version updates the dependencies and makes a few internal changes in order
to ensure that SDV works properly on Windows Systems, making this the first
release to be officially supported on Windows.Apart from this, some more internal changes have been made to solve a few minor
issues from the older versions while also improving the processing speed when
processing relational datasets with the default parameters.API breaking changesThedistributionargument of theGaussianCopulahas been renamed tofield_distributions.TheHMA1andSDVclasses now use thecategorical_fuzzytransformer by default instead of
theone_hot_encodingone.Issues resolvedGaussianCopula: renamedistributionargument tofield_distributions-Issue #237by @csalaGaussianCopula: Improve error message if an invalid distribution name is passed -Issue #220by csalaImport urllib.request explicitly -Issue #227by @csalaTypeError: cannot astype a datetimelike from [datetime64[ns]] to [int32] -Issue #218by @csalaChange default categorical transformer tocategorical_fuzzyin HMA1 -Issue #214by @csalaInteger categoricals being sampled as strings instead of integer values -Issue #194by @csala0.4.5 - 2020-10-17In this version a new family of models for Synthetic Time Series Generation is introduced
under thesdv.timeseriessub-package. The new family of models now includes a new class
calledPAR, which implements aProbabilistic AutoRegressivemodel.This version also adds support for composite primary keys and regex based generation of id
fields in tabular models and drops Python 3.5 support.Issues resolvedDrop python 3.5 support -Issue #204by @csalaSupport composite primary keys in tabular models -Issue #207by @csalaAdd the option to generate stringidfields based on regex on tabular models -Issue #208by @csalaSynthetic Time Series -Issue #142by @csala0.4.4 - 2020-10-06This version adds a new tabular model based on combining the CTGAN model with the reversible
transformation applied in the GaussianCopula model that converts random variables with
arbitrary distributions to new random variables with standard normal distribution.The reversible transformation is handled by the GaussianCopulaTransformer recently added to RDT.Issues resolvedAdd CopulaGAN Model -Issue #202by @csala0.4.3 - 2020-09-28This release moves the models and algorithms related to generation of synthetic
relational data to a newsdv.relationalsubpackage (Issue #198)As part of the change, also the oldsdv.modelshave been removed and now
relational model is based on the recently introducedsdv.tabularmodels.0.4.2 - 2020-09-19In this release thesdv.evaluationmodule has been reworked to include 4 different
metrics and in all cases return a normalized score between 0 and 1.Included metrics are:cstestkstestlogistic_detectionsvc_detection0.4.1 - 2020-09-07This release fixes a couple of minor issues and introduces an important rework of the
User Guides section of the documentation.Issues fixedError Message: "make sure the Graphviz executables are on your systems' PATH" -Issue #182by @csalaAnonymization mappings leak -Issue #187by @csala0.4.0 - 2020-08-08In this release SDV gets new documentation, new tutorials, improvements to the Tabular API
and broader python and dependency support.Complete list of changes:New Documentation site based on thepydata-sphinx-theme.New User Guides and Notebook tutorials.New Developer Guides section within the docs with details about the SDV architecture,
the ecosystem libraries and how to extend and contribute to the project.Improved API for the Tabular models with focus on ease of use.Support for Python 3.8 and the newest versions of pandas, scipy and scikit-learn.New Slack Workspace for development discussions and community support.0.3.6 - 2020-07-23This release introduces a new concept ofConstraints, which allow the user to define
special relationships between columns that will not be handled via modeling.This is done via a newsdv.constraintssubpackage which defines some well-known pre-defined
constraints, as well as a generic framework that allows the user to customize the constraints
to their needs as much as necessary.New FeaturesSupport for Constraints -Issue #169by @csala0.3.5 - 2020-07-09This release introduces a new subpackagesdv.tabularwith models designed specifically
for single table modeling, while still providing all the usual conveniences from SDV, such
as:Seamless multi-type supportMissing data handlingPII anonymizationCurrently implemented models are:GaussianCopula: Multivariate distributions modeled using copula functions. This is stronger
version, with more marginal distributions and options, than the one used to model multi-table
datasets.CTGAN: GAN-based data synthesizer that can generate synthetic tabular data with high fidelity.0.3.4 - 2020-07-04New FeaturesSupport for Multiple Parents -Issue #162by @csalaSample by default the same number of rows as in the original table -Issue #163by @csalaGeneral ImprovementsAdd benchmark -Issue #165by @csala0.3.3 - 2020-06-26General ImprovementsUse SDMetrics for evaluation -Issue #159by @csala0.3.2 - 2020-02-03General ImprovementsImprove metadata visualization -Issue #151by @csala @JDTheRipperPC0.3.1 - 2020-01-22New FeaturesAdd Metadata Validation -Issue #134by @csala @JDTheRipperPCAdd Metadata Visualization -Issue #135by @JDTheRipperPCGeneral ImprovementsAdd path to metadata JSON -Issue #143by @JDTheRipperPCUse new Copulas and RDT versions -Issue #147by @csala @JDTheRipperPC0.3.0 - 2019-12-23New FeaturesCreate sdv.models subpackage -Issue #141by @JDTheRipperPC0.2.2 - 2019-12-10New FeaturesAdapt evaluation to the different data types -Issue #128by @csala @JDTheRipperPCExtendload_demofunctionality to load other datasets -Issue #136by @JDTheRipperPC0.2.1 - 2019-11-25New FeaturesMethods to generate Metadata from DataFrames -Issue #126by @csala @JDTheRipperPC0.2.0 - 2019-10-11New Featurescompatibility with rdt issue 72 -Issue #120by @csala @JDTheRipperPCGeneral ImprovementsError docstring sampler.__fill_text_columns -Issue #144by @JDTheRipperPCReach 90% coverage -Issue #112by @JDTheRipperPCReview unittests -Issue #111by @JDTheRipperPCBugs FixedTime required for sample_all function? -Issue #118by @csala @JDTheRipperPC0.1.2 - 2019-09-18New FeaturesAdd option to model the amount of child rows - Issue93by @ManuelAlvarezCGeneral ImprovementsAdd Evaluation Metrics - Issue52by @ManuelAlvarezCEnsure unicity on primary keys on different calls - Issue63by @ManuelAlvarezCBugs fixedexecuting readme: 'not supported between instances of 'int' and 'NoneType' - Issue104by @csala0.1.1 - Anonymization of dataAdd warnings when trying to model an unsupported dataset structure.GH#73Add option to anonymize data.GH#51Add support for modeling data with different distributions, when usingGaussianMultivariatemodel.GH#68Add support forVineCopulasas a model.GH#71ImproveGaussianMultivariateparameter sampling, avoiding warnings and unvalid parameters.GH#58Fix issue that caused that sampled categorical values sometimes got numerical values mixed.GH#81Improve the validation of extensions.GH#69Update examples.GH#61ReplacedTableclass with aNamedTuple.GH#92Fix inconsistent dependencies and add upper bound to dependencies.GH#96Fix error when merging extension inModeler.CPAwhen running examples.GH#860.1.0 - First ReleaseFirst release on PyPI.