Package: rdt
This repository is part ofThe Synthetic Data Vault Project, a project fromDataCebo.OverviewRDT (Reversible Data Transforms) is a Python library that transforms raw data into fully numerical
data, ready for data science. The transforms are reversible, allowing you to convert from numerical
data back into your original format.InstallInstallRDTusingpiporconda. We recommend using a virtual environment to avoid
conflicts with other software on your device.pipinstallrdtcondainstall-cconda-forgerdtFor more information about using reversible data transformations, visit theRDT Documentation.QuickstartIn this short series of tutorials we will guide you through a series of steps that will
help you getting started usingRDTto transform columns, tables and datasets.Load the demo dataAfter you have installed RDT, you can get started using the demo dataset.fromrdtimportget_democustomers=get_demo()This dataset contains some randomly generated values that describe the customers of an online
marketplace.last_login email_optin credit_card  age  dollars_spent
0 2021-06-26       False        VISA   29          99.99
1 2021-02-10       False        VISA   18            NaN
2        NaT       False        AMEX   21           2.50
3 2020-09-26        True         NaN   45          25.00
4 2020-12-22         NaN    DISCOVER   32          19.99Let's transform this data so that each column is converted to full, numerical data ready for data
science.Creating the HyperTransformer & configTheHyperTransformeris capable of transforming multi-column datasets.fromrdtimportHyperTransformerht=HyperTransformer()TheHyperTransformerneeds to know about the columns in your dataset and which transformers to
apply to each. These are described by a config. We can ask theHyperTransformerto automatically
detect it based on the data we plan to use.ht.detect_initial_config(data=customers)This will create and set the config.Config:
{
    "sdtypes": {
        "last_login": "datetime",
        "email_optin": "boolean",
        "credit_card": "categorical",
        "age": "numerical",
        "dollars_spent": "numerical"
    },
    "transformers": {
        "last_login": "UnixTimestampEncoder()",
        "email_optin": "BinaryEncoder()",
        "credit_card": "FrequencyEncoder()",
        "age": "FloatFormatter()",
        "dollars_spent": "FloatFormatter()"
    }
}Thesdtypesdictionary describes the semantic data types of each of your columns and thetransformersdictionary describes which transformer to use for each column. You can customize the
transformers and their settings. (See theTransformers Glossaryfor more information).Fitting & using the HyperTransformerTheHyperTransformerreferences the config while learning the data during thefitstage.ht.fit(customers)Once the transformer is fit, it's ready to use. Use the transform method to transform all columns
of your dataset at once.transformed_data=ht.transform(customers)last_login.value  email_optin.value  credit_card.value  age.value  dollars_spent.value
0      1.624666e+18                0.0                0.2         29                99.99
1      1.612915e+18                0.0                0.2         18                36.87
2      1.611814e+18                0.0                0.5         21                 2.50
3      1.601078e+18                1.0                0.7         45                25.00
4      1.608595e+18                0.0                0.9         32                19.99TheHyperTransformerapplied the assigned transformer to each individual column. Each column
now contains fully numerical data that you can use for your project!When you're done with your project, you can also transform the data back to the original format
using thereverse_transformmethod.original_format_data=ht.reverse_transform(transformed_data)last_login email_optin credit_card  age  dollars_spent
0        NaT       False        VISA   29          99.99
1 2021-02-10       False        VISA   18            NaN
2        NaT       False        AMEX   21            NaN
3 2020-09-26        True         NaN   45          25.00
4 2020-12-22       False    DISCOVER   32          19.99What's Next?To learn more about reversible data transformations, visit theRDT Documentation.The Synthetic Data Vault Projectwas first created at MIT'sData to AI Labin 2016. After 4 years of research and traction with enterprise, we
createdDataCeboin 2020 with the goal of growing the project.
Today, DataCebo is the proud developer of SDV, the largest ecosystem for
synthetic data generation & evaluation. It is home to multiple libraries that support synthetic
data, including:🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular,
multi table and time series data.📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data
generation models.Get started using the SDV package-- a fully
integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries
for specific needs.History1.9.2 - 2024-02-13This release makes a couple improvements to theRegexGenerator. Error messaging is improved and it is now capable of generating an unlimited amount of rows even when theenforce_uniquenessflag is True. It does this by adding suffixes if the max amount of combinations for the provided regex is met.Additionally, this release resolves a few bugs. TheOneHotEncodershould no longer crash on thecategoricaldtype and theUniformEncoderwas improved to support more dtypes.Bugs FixedCategorical reverse transform may crash withValueErrorfor certain dtypes (int64) - Issue#747by @R-PalazzoRegexGenerator gives a confusing message: # of possibilities are shown as an imaginary number - Issue#748by @R-PalazzoOneHotEncoder doesn't support dtype'category'- Issue#751by @fealhoNew FeaturesRegexGenerator should create unlimited regexes, even if unique enforcement is on - Issue#749by @fealhoAdd a _update_multi_column_transformer method - Issue#757by @R-PalazzoInternalMove the _learn_rounding_digits of the FloatFormatter into a helper - Issue#750by @fealho1.9.1 - 2024-01-10This release fixes a bug that caused theAnonymizedFakerto crash with provider/function combinations that return tuples.Bugs FixedAnonymizedFaker crashes with ValueError for specific provider/function pairs (eg. currency) - Issue#743by @ R-Palazzo1.9.0 - 2023-11-14This release adds a parameter to theUnixTimestampEncoderandOptimizedTimestampEncoder, calledenforce_min_max_values. When this is set to True, it clips all values in the reverse transformed data to the min and max datetimes seen in the fitted data.This release also internally adds support for multi-column transformers!New FeaturesSupport multi-column transformers - Issue#683by @R-PalazzoImprove user warnings and logic for update_sdtype - Issue#684by @R-PalazzoImprove user warnings and logic for update_transformers and update_transformers_by_sdtype - Issue#685by @R-PalazzoImprove user warnings and logic for remove_transformers and remove_transformers_by_sdtype - Issue#686by @R-PalazzoAdd enforce_min_max_values to datetime transformers - Issue#740by @R-PalazzoInternalSupport multi-column transformers - Issue#683by @R-PalazzoBugs FixedMulti column transformers crash when assigned to single column - Issue#734by @R-Palazzo1.8.0 - 2023-10-31This release adds the 'random' missing value replacement strategy, which uses random values of the dataset to fill in missing values.
Additionally users are now able to use theUniformUnivariatedistribution within the Gaussian Normalizer with this update.This release contains fixes for theClusterBasedNormalizerwhich crashes in the reverse transform caused by values being out of bounds
and a patch for the randomization issue dealing with different values after applyingreset_randomization.Anonymization has been moved into RDT library from SDV as it was found to self contained module for RDT and would reduce dependencies needed in SDV.FeaturesMake the default missing value imputation 'mean' - Issue#730by @R-PalazzoWhen no rounding scheme is detected, log the info instead of showing a warning - Issue#709by @frances-hThe GaussianNormalizer should accept distribution names that are consistent with scipy - Issue#656by @fealhoThe GaussianNormalizer should accept uniform distributions - Issue#655by @fealhoRemove psutil - Issue#615by @fealhoConsider deprecating the FrequencyEncoder - Issue#614by @fealhoReplace missing values with variable (random) values from the dataset - Issue#606BugsRDT Uniform Encoder creates nan Value bug - Issue#719by @lajohn4747HyperTransformer transforms while fitting and messes up the random seed - Issue#716by @pvk-developerResolve locales warning for specific sdtype/locale combos (eg. en_US with postcode) - Issue#701by @pvk-developerThe OrderedLabelEncoder should not accept duplicate categories - Issue#673by @frances-hClusterBasedNormalizer crashes on reverse transform (IndexError) - Issue#672by @fealhoUnnecessary warning in OneHotEncoder when there are nan values - Issue#616by @fealhoMaintenanceRemove performance tests - Issue#707by @fealhoClusterBasedNormalizer code cleanup - Issue#696by @fealhoSwitch default branch from master to main - Issue#687by @amontanez24DeprecationsThefrequencyEncodertransformer will no longer be supported in future versions of RDT. Please use theUniformEncodertransformer instead.GaussianNormalizerdistribution option names have been updated to be consistent with scipy.gaussian->norm,student_t->t, andtruncated_gaussian->truncnorm1.7.0 - 2023-08-22This release adds 3 new transformers:UniformEncoder- A categorical and boolean transformer that converts the column into a uniform distribution.OrderedUniformEncoder- The same as above, but the order for the categories can be specified, changing which range in the uniform distribution each category belongs to.IDGenerator- A text transformer that drops the input column during transform and returns IDs during reverse transform. The IDs all take the form <prefix><number><suffix> and can be configured with a custom prefix, suffix and starting point.Additionally, theAnonymizedFakeris enhanced to support the text sdtype.DeprecationsTheget_input_sdtypemethod is being deprecated in favor ofget_supported_sdtypes.New FeaturesCreate IDGenerator transformer - Issue#675by @R-PalazzoAdd UniformEncoder (and its ordered version) - Issue#678by @R-PalazzoAllow me to use AnonymizedFaker with sdtype text columns - Issue#688by @amontanez24MaintenanceDeprecate get_input_sdtype - Issue#682by @R-Palazzo1.6.1 - 2023-08-02This release updates the default transformers used for certain sdtypes. It also enables theAnonymizedFakerandPseudoAnonymizedFakerto work with any sdtype besides boolean, categorical, datetime, numerical or text.Bugs[Enterprise Usage] Unable to assign generic PII transformers (eg. AnonymizedFaker) - Issue#674by @amontanez24New FeaturesUpdate the default transformers that HyperTransformer assigns to each sdtype - Issue#664by @amontanez241.6.0 - 2023-07-12This release adds the ability to generate missing values to theAnonymizedFaker. Users can now provide themissing_value_generationparameter during initialization. They can set it toNoneto not generate any missing values, or'random'to generate random missing values in the same proportion as the fitted data.Additionally, this release improves theNullTransformerby allowing nulls to be replaced on the forward transform even ifmissing_value_generationis set to None. It also fixes a bug that was causing theUnixTimestampEncoderto return a different dtype than the input onreverse_transform. This was particularly problematic when datetime columns are represented as ints.New FeaturesAnonymizedFaker should be able to model and generate missing values - Issue#660by @R-PalazzoBugsThe datetime transformers don't give me back the same dtype sometimes - Issue#657by @frances-hRDT NullTransformer doesn't replace nulls if missing_value_generation is None - Issue#658by @amontanez24MaintenanceRemove python 3.7 builds - Issue#663by @amontanez24Drop support for Python 3.7 - Issue#666by @amontanez24InternalAdd add-on modules to sys.modules - Issue#653by @amontanez241.5.0 - 2023-06-01This release adds a new parameter calledmissing_value_generationto the initialization of certain transformers to specify how missing values should be created. The parameter can be used in theFloatFormatter,BinaryEncoder,UnixTimestampEncoder,OptimizedTimestampEncoder,GaussianNormalizerandClusterBasedNormalizer. Additionally, it fixes a bug that was causing every column that had nulls to generate them in the same place.DeprecationsThemodel_missing_valuesparameter is being deprecated in favor of the newmissing_value_generationparameter.BugsFix randomization when creating null values - Issue#639by @fealhoNew FeaturesAllow a no-op handling strategy for missing values (nulls) - Issue#644by @pvk-developerAdd add-on detection for premium transformers - Issue#646by @frances-hMaintenancePerformance tests still fragile - Issue#641by @fealhoInvestigate removing quality tests - Issue#642by @amontanez241.4.2 - 2023-05-02This release fixes a bug that caused datetime and numerical transformers to crash if a column was all NaNs. Additionally, it adds support for Pandas 2.0!BugsNumerical & datetime transformers crash if the entire column is null - Issue#637by @fraces-hMaintenanceRemove upper bound for pandas - Issue#633by @pvk-developer1.4.1 - 2023-04-25This release patches an issue that prevented theRegexGeneratorfrom working with regexes that had a very large number of possible combinations.BugsRegexGenerator continues to have problems if there are too many possibilities - Issue#635by @pvk-developer1.4.0 - 2023-04-13This release adds a couple of new features including adding theOrderedLabelEncoderand deprecating theCustomLabelEncoder. It also adds a change that makes all generator type transformers in theHyperTransformeruse a different random seed.Additionally, bugs were patched in theRegexGeneratorthat caused it to crash or take too long in certain cases. Finally, this release improved the detection of Faker functions in theAnonymizedFaker.BugsFind nested Faker provider submodules - PR#630by @frances-hRegexGenerator fails to generate values if there are too many possibilities - Issue#623by @R-PalazzoRegexGenerator takes too much time and runs out of memory if there are too many possibilities - Issue#624by @R-PalazzoNew FeaturesChoose a different seed for each transformer - Issue#619by @fealhoRename CustomLabelEncoder to OrderedLabelEncoder - Issue#621by @R-PalazzoAdd functionality to find version add-on - Issue#620by @frances-h1.3.0 - 2023-01-18This release makes changes to the way that individual transformers are stored in theHyperTransformer. When accessing the config viaHyperTransformer.get_config(), the transformers listed in the config are now the actual transformer instances used during fitting and transforming. These instances can now be accessed and used to examine their properties post fitting. For example, you can now view the mapping for aPseudoAnonymizedFakerinstance usingPseudoAnonymizedFaker.get_mapping()on the instance retrieved from the config.Additionally, the output ofreverse_tranformno longer appends the.valuesuffix to every unnamed output column. Only output columns that are created from context extracted from the input columns will have suffixes (eg..normalizedin theClusterBasedNormalizer).TheAnonymizedFakerandRegexGeneratornow have anenforce_uniquenessparameter, which controls whether the data returned byreverse_transformshould be unique. TheHyperTransformernow has a method calledcreate_anonymized_columnsthat can be used to generate columns that are matched with anonymizing transformers likeAnonymizedFakerandRegexGenerator. The method can be used as follows:HyperTransformer.create_anonymized_columns(num_rows=5, column_names=['email_optin', 'credit_card'])Another major change in this release is the ability to control randomization. Every time aHyperTransformeris initialized, its randomness will be reset to the same seed, and it will yield the same results forreverse_transformif given the same input. Every subsequent call toreverse_transformyields a different result. If a user desires to reset the seed, they can callHyperTransformer.reset_randomization.Finally, this release adds support for Python 3.10 and drops support for 3.6.BugsThe reset_randomization should also apply to fit and transform - Issue#608by @amontanez24Cannot print CustomLabelEncoder: ValueError - Issue#607by @amontanez24Float formatter learn_rounding_scheme doesn't work on all digits - Issue#556by @fealhoWarnings not showing on update_transformers_by_sdtype - Issue#582by @amontanez24OneHotEncoder doesn't work with boolean sdtype - Issue#583by @pvk-developerSetting config on HyperTransformer does not read supported_sdtypes - Issue#560by @pvk-developerhttps://github.com/sdv-dev/RDT/issues/545- Issue#545by @pvk-developerAdd error to NullTransformer when data only contains nans - PR#567by @fealhoUpdate update_transformers validation - PR#563by @fealhoMaintenanceSupport Python 3.10 - Issue#593by @pvk-developerRDT 1.3 Package Maintenance Updates - Issue#594by @pvk-developerNew FeaturesUpdate errors - Issue#599by @amontanez24Add ability to control randomness - Issue#584by @amontanez24Printing and error improvements - Issue#581by @amontanez24Make RegexGenerator not to reset itself - Issue#558by @pvk-developerAdd a reset_anonymization method - Issue#559by @pvk-developerDon't copy instances of tranformer - Issue#541by @fealhoRemove '.value' suffix - Issue#533by @fealhoChange the NEXT_TRANSFORMERS logic - Issue#557by @fealhoAdd utility functions to AnonymizedFaker - Issue#561by @pvk-developerUpdate API for update_transformers_by_sdtype to be more explicit about instances vs. copies - Issue#540by @fealhoAdd create_anonymized_columns method to anonymize data from scratch - Issue#546by @pvk-developerAdd parameter to AnonymizedFaker() and RegexGenerator() to generate only unique values - Issue#542by @pvk-developer1.2.1 - 2022-9-12This release fixes a bug that caused theUnixTimestampEncoderto return data with the incorrect datetime format. It also fixes a bug that caused the null column
not to be reverse transformed when using theUnixTimestampEncoderwhen themissing_value_replacementwas not set.BugsInconsistency in date format after reverse transform - Issue#515by @pvk-developerFix calling null_transformer with model_missing_values. - PR#550by @pvk-developer1.2.0 - 2022-8-17This release adds a new transformer called thePseudoAnonymizedFaker. This transformer enables the pseudo-anonymization of your data by mapping all of a column's original values to fake values that get returned during the reverse transformation process. Each original value is always mapped to the same fake value.Additionally, this release enables theHyperTransformerto use categorical transformers on boolean columns. It also introduces a new parameter calledcomputer_representationto theFloatFormatterthat will allow for values to be clipped to certain bounds based on the computer type used for a numerical column.Finally, this release patches a bug that caused unpredicatable results from thereverse_transformmethod of theFrequencyEncoderwhenadd_noiseis enabled.New FeaturesAdd PseudoAnonymizedFaker transformer - Issue#517by @pvk-developerBoolean columns should be able to use any of the categorical transformers - Issue#527by @pvk-developerUpdate FloatFormatter with parameters for the computer representation - Issue#521by @fealhoBugsUnpredictable results for FrequencyEncoder(add_noise=True) - Issue#528by @fealhoInternalPerformance Tests update - Issue#524by @pvk-developer1.1.0 - 2022-6-9This release adds multiple new transformers: theCustomLabelEncoderand theRegexGenerator. TheCustomLabelEncoderworks similarly
to theLabelEncoder, except it allows users to provide the order of the categories. TheRegexGeneratorallows users to specify a regex
pattern and will generate values that match that pattern.This release also improves current transformers. TheLabelEncodernow has a parameter calledorder_bythat allows users to specify the
ordering scheme for their data (eg. order numerically or alphabetically). TheLabelEncoderalso now has a parameter calledadd_noisethat allows users to specify whether or not uniform noise should be added to the transformed data. Performance enhancements were made for theGaussianNormalizerby removing an unnecessary distribution search and theFloatFormatterwill no longer round values to any place higher
than the ones place by default.New FeaturesAdd noise parameter to LabelEncoder - Issue#500by @fealhoRemove parameters related to distribution search and change default for GaussianNormalizer - Issue#499by @amontanez24Add order_by parameter to LabelEncoder - Issue#510by @amontanez24Only round to decimal places in FloatFormatter - Issue#508by @fealhoAdd CustomLabelEncoder transformer - Issue#507by @amontanez24Add RegexGenerator Transformer - Issue#505by @pvk-developer1.0.0 - 2022-4-25The main update of this release is the introduction of aconfig, which describes thesdtypesandtransformersthat will be used by theHyperTransformerfor each column of the data, wheresdtypestands for thesemanticorstatisticalmeaning of a datatype. The user can interact with this config through the newly created methodsupdate_sdtypes,get_config,set_config,update_transformers,update_transformers_by_sdtypeandremove_transformer_by_sdtype.This release also included various new features and updates, including:Users can now transform subsets of the data using its own methods,transform_subsetandreverse_transform_subset.User validation was added for the following methods:transform,reverse_transform,update_sdtypes,update_transformers,set_config.Unnecessary warnings were removed fromGaussianNormalizer.fitandFrequencyEncoder.transform.The user can now set a transformers as None.Transformers that cannot work with missing values will automatically fill them in.Added support for additional datetime formats.Settingmodel_missing_values = Falsein a transformer was updated to keep track of the percentage of missing values, instead of producing data containingNaN's.All parameters were removed from theHyperTransformer.The demo datasetget_demowas improved to be more intuitive.Finally, a number of transformers were redesigned to be more user friendly. Among them, the following transformers have also been renamed:BayesGMMTransformer->ClusterBasedNormalizerGaussianCopulaTransformer->GaussianNormalizerDateTimeRoundedTransformer->OptimizedTimestampEncoderDateTimeTransformer->UnixTimestampEncoderNumericalTransformer->FloatFormatterLabelEncodingTransformer->LabelEncoderOneHotEncodingTransformer->OneHotEncoderCategoricalTransformer->FrequencyEncoderBooleanTransformer->BinaryEncoderPIIAnonymizer->AnonymizedFakerNew FeaturesFix using None as transformer when update_transformers_by_sdtype - Issue#496by @pvk-developerRename PIIAnonymizer --> AnonymizedFaker - Issue#483by @pvk-developerUser validation for reverse_transform - Issue#480by @amontanez24User validation for transform - Issue#479by @fealho\User validation for set_config - Issue#478by @fealhoUser validation for update_transformers_by_sdtype - Issue#477by @amontanez24User validation for update_transformers - Issue#475by @fealhoUser validation for update_sdtypes - Issue#474by @fealhoAllow columns to not have a transformer - Issue#473by @pvk-developerCreate methods to transform a subset of the data (& reverse transform it) - Issue#472by @amontanez24Throw a warning if you use set_config on a HyperTransformer that's already fit - Issue#466by @amontanez24Update README for RDT 1.0 - Issue#454by @amontanez24Issue with printing PIIAnonymizer in HyperTransformer - Issue#452by @pvk-developerPretty print get_config - Issue#450by @pvk-developerSilence warning for GaussianNormalizer.fit - Issue#443by @pvk-developerTransformers that cannot work with missing values should automatically fill them in - Issue#442by @amontanez24More descriptive error message in PIIAnonymizer when provider_name and function_name don't align - Issue#440by @pvk-developerCan we support additional datetime formats? - Issue#439by @pvk-developerUpdate FrequencyEncoder.transform so that pandas won't throw a warning - Issue#436by @pvk-developerUpdate functionality when model_missing_values=False - Issue#435by @amontanez24Create methods for getting and setting a config - Issue#418by @amontanez24Input validation & error handling in HyperTransformer - Issue#408by @fealho and @amontanez24Remove unneeded params from HyperTransformer - Issue#407by @pvk-developerRename property: _valid_output_sdtypes - Issue#406by @amontanez24Add pii as a new sdtype in HyperTransformer - Issue#404by @pvk-developerUpdate transformers by data type (in HyperTransformer) - Issue#403by @pvk-developerUpdate transformers by column name in HyperTransformer - Issue#402by @pvk-developerImprove updating field_data_types in HyperTransformer - Issue#400by @amontanez24Create method to auto detect HyperTransformer config from data - Issue#399by @fealhoUpdate HyperTransformer default transformers - Issue#398by @fealhoAdd PIIAnonymizer - Issue#397by @pvk-developerImprove the way we print an individual transformer - Issue#395by @amontanez24Rename columns parameter in fit for each individual transformer - Issue#376by @fealho and @pvk-developerCreate a more descriptive demo dataset - Issue#374by @fealhoDelete unnecessary transformers - Issue#373by @fealhoUpdate NullTransformer to make it user friendly - Issue#372by @pvk-developerUpdate BayesGMMTransformer to make it user friendly - Issue#371by @amontanez24Update GaussianCopulaTransformer to make it user friendly - Issue#370by @amontanez24Update DateTimeRoundedTransformer to make it user friendly - Issue#369by @amontanez24Update DateTimeTransformer to make it user friendly - Issue#368by @amontanez24Update NumericalTransformer to make it user friendly - Issue#367by @amontanez24Update LabelEncodingTransformer to make it user friendly - Issue#366by @fealhoUpdate OneHotEncodingTransformer to make it user friendly - Issue#365by @fealhoUpdate CategoricalTransformer to make it user friendly - Issue#364by @fealhoUpdate BooleanTransformer to make it user friendly - Issue#363by @fealhoUpdate names & functionality for handling missing values - Issue#362by @pvk-developerBugsChecking keys of config as set - Issue#497by @amontanez24Only update transformer used when necessary for update_sdtypes - Issue#469by @amontanez24Fix how get_config prints transformers - Issue#468by @pvk-developerNullTransformer reverse_transform alters input data due to not copying - Issue#455by @amontanez24Attempting to transform a subset of the data should lead to an Error - Issue#451by @amontanez24Detect_initial_config isn't detecting sdtype "numerical" - Issue#449by @pvk-developerPIIAnonymizer not generating multiple locales - Issue#447by @pvk-developerError when printing ClusterBasedNormalizer and GaussianNormalizer - Issue#441by @pvk-developerDatetime reverse transform crashes if datetime_format is specified - Issue#438by @amontanez24Correct datetime format is not recovered on reverse_transform - Issue#437by @pvk-developerUse numpy NaN values in BinaryEncoder - Issue#434by @pvk-developerDuplicate _output_columns during fitting - Issue#423by @fealhoInternal ImprovementsMaking methods that aren't part of API private - Issue#489by @amontanez24Fix columns missing in config and update transformers to None - Issue#495by @pvk-developer0.6.4 - 2022-3-7This release fixes multiple bugs concerning theHyperTransformer. One is that theget_transformer_tree_yamlmethod no longer crashes on
every call. Another is that calling theupdate_field_data_typesandupdate_default_data_type_transformersafter fitting no longer breaks thetransformmethod.TheHyperTransformernow sorts its outputs for bothtransformandreverse_transformbased on the order of the input's columns. It is also now possible
to create transformers that simply drops columns duringtransformand don't return any new columns.New FeaturesSupport dropping a column trough a transformer - Issue#393by @pvk-developerHyperTransformer should sort columns after transform and reverse_transform - Issue#405by @fealhoBugsget_transformer_tree_yaml fails - Issue#389by @amontanez24HyperTransformer _unfit method not working correctly - Issue#390by @amontanez24Blank dataframe after updating the data types - Issue#401by @amontanez240.6.3 - 2022-2-4This release adds a new module to theRDTlibrary calledperformance. This module can be used to evaluate the speed and peak memory usage
of any transformer in RDT. This release also increases the maximum acceptable version of scikit-learn to make it more compatible with other libraries
in theSDVecosystem. On top of that, it fixes a bug related to a new version ofpandas.New FeaturesMove profiling functions into RDT library - Issue#353by @amontanez24HousekeepingIncrease scikit-learn dependency range - Issue#351by @amontanez24pandas 1.4.0 release causes a small error - Issue#358by @fealhoBugsPerformance tests get stuck on Unix if multiprocessing is involved - Issue#337by @amontanez240.6.2 - 2021-12-28This release adds a newBayesGMMTransformer. This transformer can be used to convert a numerical column into two
columns: a discrete column indicating the selectedcomponentof the GMM for each row, and a continuous column containing
the normalized value of each row based on themeanandstdof the selectedcomponent. It is useful when the column being transformed
came from multiple distributions.This release also adds multiple new methods to theHyperTransformerAPI. These allow for users to access the specfic
transformers used on each input field, as well as view the entire tree of transformers that are used when runningtransform.
The exact methods are:BaseTransformer.get_input_columns()- Return list of input columns for a transformer.BaseTransformer.get_output_columns()- Return list of output columns for a transformer.HyperTransformer.get_transformer(field)- Return the transformer instance used for a field.HyperTransformer.get_output_transformers(field)- Return dictionary mapping output columns of a field to the transformers used on them.HyperTransformer.get_final_output_columns(field)- Return list of all final output columns related to a field.HyperTransformer.get_transformer_tree_yaml()- Return YAML representation of transformers tree.Additionally, this release fixes a bug where theHyperTransformerwas incorrectly raising aNotFittedError. It also improved theDatetimeTransformerby autonomously detecting if a column needs to be converted fromdtypeobjecttodtypedatetime.New FeaturesCast column to datetime if specified in field transformers - Issue#321by @amontanez24Add a BayesianGMM Transformer - Issue#183by @fealhoAdd transformer tree structure and traversal methods - Issue#330by @amontanez24Bugs fixedHyperTransformer raises NotFittedError after fitting - Issue#332by @amontanez240.6.1 - 2021-11-10This release adds support for Python 3.9! It also removes unused document files.Internal ImprovementsAdd support for Python 3.9 - Issue#323by @amontanez24Remove docs - PR#322by @pvk-developer0.6.0 - 2021-10-29This release makes major changes to the underlying code for RDT as well as the API for both theHyperTransformerandBaseTransformer.
The changes enable the following functionality:TheHyperTransformercan now apply a sequence of transformers to a column.Transformers can now take multiple columns as an input.RDT has been expanded to allow for infinite data types to be added instead of being restricted topandas.dtypes.Users can define acceptable output types for runningHyperTransformer.transform.TheHyperTransformerwill continuously apply transformations to the input fields until only acceptable data types are in the output.Transformers can return data of any data type.Transformers now have named outputs and output types.Transformers can suggest which transformer to use on any of their outputs.To take advantage of this functionality, the following API changes were made:TheHyperTransformerhas new initialization parameters that allow users to specify data types for any field in their data as well as
specify which transformer to use for a field or data type. The parameters are:field_transformers- A dictionary allowing users to specify which transformer to use for a field or derived field. Derived fields
are fields created by runningtransformon the input data.field_data_types- A dictionary allowing users to specify the data type of a field.default_data_type_transformers- A dictionary allowing users to specify the default transformer to use for a data type.transform_output_types- A dictionary allowing users to specify which data types are acceptable for the output oftransform.
This is a result of the fact that transformers can now be applied in a sequence, and not every transformer will return numeric data.Methods were also added to theHyperTransformerto allow these parameters to be modified. These includeget_field_data_types,update_field_data_types,get_default_data_type_transformers,update_default_data_type_transformersandset_first_transformers_for_fields.TheBaseTransformernow requires the column names it will transform to be provided tofit,transformandreverse_transform.TheBaseTransformeradded the following method to allow for users to see its output fields and output types:get_output_types.TheBaseTransformeradded the following method to allow for users to see the next suggested transformer for each output field:get_next_transformers.On top of the changes to the API and the capabilities of RDT, many automated checks and tests were also added to ensure that contributions
to the library abide by the current code style, stay performant and result in data of a high quality. These tests run on every push to the
repository. They can also be run locally via the following functions:validate_transformer_code_style- Checks that new code follows the code style.validate_transformer_quality- Tests that new transformers yield data that maintains relationships between columns.validate_transformer_performance- Tests that new transformers don't take too much time or memory.validate_transformer_unit_tests- Checks that the unit tests cover all new code, follow naming conventions and pass.validate_transformer_integration- Checks that the integration tests follow naming conventions and pass.New FeaturesUpdate HyperTransformer API - Issue#298by @amontanez24Create validate_pull_request function - Issue#254by @pvk-developerCreate validate_transformer_unit_tests function - Issue#249by @pvk-developerCreate validate_transformer_performance function - Issue#251by @katxiaoCreate validate_transformer_quality function - Issue#253by @amontanez24Create validate_transformer_code_style function - Issue#248by @pvk-developerCreate validate_transformer_integration function - Issue#250by @katxiaoEnable users to specify transformers to use in HyperTransformer - Issue#233by @amontanez24 and @csalaAddons implementation - Issue#225by @pvk-developerCreate ways for HyperTransformer to know which transformers to apply to each data type - Issue#232by @amontanez24 and @csalaUpdate categorical transformers - PR#231by @fealhoUpdate numerical transformer - PR#227by @fealhoUpdate datetime transformer - PR#230by @fealhoUpdate boolean transformer - PR#228by @fealhoUpdate null transformer - PR#229by @fealhoUpdate the baseclass - PR#224by @fealhoBugs fixedIf the input data has a different index, the reverse transformed data may be out of order - Issue#277by @amontanez24Documentation changesRDT contributing guide - Issue#301by @katxiao and @amontanez24Internal improvementsAdd PR template for new transformers - Issue#307by @katxiaoImplement Quality Tests for Transformers - Issue#252by @amontanez24Update performance test structure - Issue#257by @katxiaoAutomated integration test for transformers - Issue#223by @katxiaoMove datasets to its own module - Issue#235by @katxiaoFix missing coverage in rdt unit tests - Issue#219by @fealhoAdd repo-wide automation - Issue#309by @katxiaoOther issues closedDeprecationWarning: np.float is a deprecated alias for the builtin float - Issue#304by @csalaAdd pip check to CI workflows - Issue#290by @csalaShould Transformers subclasses exist for specific configurations? - Issue#243by @fealho0.5.3 - 2021-10-07This release fixes a bug with learning rounding digits in theNumericalTransformer,
and includes a few housekeeping improvements.Issues closedUpdate learn rounding digits to handle all nan data - Issue#244by @katxiaoAdapt to latest PyLint housekeeping - Issue#216by @fealho0.5.2 - 2021-08-16This release fixes a couple of bugs introduced by the previous release regarding theOneHotEncodingTransformerand theBooleanTransformer.Issues closedBooleanTransformer.reverse_transform sometimes crashes with TypeError - Issue#210by @katxiaoOneHotEncodingTransformer causing shape misalignment in CopulaGAN, CTGAN, and TVAE - Issue#208by @sarahmishBoolean.transformer.reverse_transform modifies the input data - Issue#211by @katxiao0.5.1 - 2021-08-11This release improves the overall performance of the library, both in terms of memory and time consumption.
More specifically, it makes the following modules more efficient:NullTransformer,DatetimeTransformer,LabelEncodingTransformer,NumericalTransformer,CategoricalTransformer,BooleanTransformerandOneHotEncodingTransformer.It also adds performance-based testing and a script for profiling the performance.Issues closedAdd performance-based testing - Issue#194by @amontanez24Audit the NullTransformer - Issue#192by @amontanez24Audit DatetimeTransformer - Issue#189by @sarahmishAudit the LabelEncodingTransformer - Issue#184by @amontanez24Audit the NumericalTransformer - Issue#181by @fealhoAudit CategoricalTransformer - Issue#180by @katxiaoAudit BooleanTransformer - Issue#179by @katxiaoAuditing OneHotEncodingTransformer - Issue#178by @sarahmishCreate script for profiling - Issue#176by @amontanez24Create folder structure for performance testing - Issue#174by @amontanez240.5.0 - 2021-07-12This release updates theNumericalTransformerby adding a newroundingargument.
Users can now obtain numerical values with precision, either pre-specified or automatically computed from the given data.Issues closedAddroundingargument toNumericalTransformer- Issue#166by @amontanez24 and @csalaNumericalTransformerrounding error with infinity - Issue#169by @amontanez24Add min and max arguments to NumericalTransformer - Issue#106by @amontanez240.4.2 - 2021-06-08This release adds a new method to theCategoricalTransformerto solve a bug where
the transformer becomes unusable after being pickled and unpickled if it hadNaNvalues in the data which it was fit on.It also fixes some grammar mistakes in the documentation.Issues closedCategoricalTransformer with NaN values cannot be pickled bug - Issue#164by @pvk-developer and @csalaDocumentation changesdocs: fix typo - PR#163by @sbrugman0.4.1 - 2021-03-29This release improves theHyperTransformermemory usage when working with a
high number of columns or a high number of categorical values when using one hot encoding.Issues closedBoolean,DatetimeandLabelEncodingtransformers fail with 2Dndarray- Issue#160by @pvk-developerHyperTransformer: Memory usage increase whenreverse_transformis called - Issue#156by @pvk-developer and @AnupamaGangadhar0.4.0 - 2021-02-24In this release a change in the HyperTransformer allows using it to transform and
reverse transform a subset of the columns seen during training.The anonymization functionality which was deprecated and not being used has also
been removed along with the Faker dependency.Issues closedAllow the HyperTransformer to be used on a subset of the columns - Issue#152by @csalaRemove faker - Issue#150by @csala0.3.0 - 2021-01-27This release changes the behavior of theHyperTransformerto prevent it from
modifying any column in the givenDataFrameif thetransformersdictionary
is passed empty.Issues closedIf transformers is an empty dict, do nothing - Issue#149by @csala0.2.10 - 2020-12-18This release adds a new argument to theHyperTransformerwhich gives control over
which transformers to use by default for eachdtypeif no specific transformer
has been specified for the field.This is also the first version to be officially released on conda.Issues closedAdddtype_transformersargument to HyperTransformer - Issue#148by @csalaMakes Copulas an optional dependency - Issue#144by @fealho0.2.9 - 2020-11-27This release fixes a bug that prevented theCategoricalTransformerfrom working properly
when being passed data that contained numerical data only, without any strings, but also
containedNoneorNaNvalues.Issues closedKeyError: nan - CategoricalTransformer fails on numerical + nan data only - Issue#142by @csala0.2.8 - 2020-11-20This release fixes a few minor bugs, including some which prevented RDT from fully working
on Windows systems.Thanks to this fixes, as well as a new testing infrastructure that has been set up, from now
on RDT is officially supported on Windows systems, as well as on the Linux and macOS systems
which were previously supported.Issues closedTypeError: unsupported operand type(s) for: 'NoneType' and 'int' - Issue#132by @csalaExample does not work on Windows - Issue#114by @csalaOneHotEncodingTransformer producing all zeros - Issue#135by @fealhoOneHotEncodingTransformer support for lists and lists of lists - Issue#137by @fealho0.2.7 - 2020-10-16In this release we drop the support for the now officially dead Python 3.5
and introduce a new feature in the DatetimeTransformer which reduces the dimensionality
of the generated numerical values while also ensuring that the reverted datetimes
maintain the same level as time unit precision as the original ones.Drop Py35 support - Issue#129by @csalaAdd option to drop constant parts of the datetimes - Issue#130by @csala0.2.6 - 2020-10-05Add GaussianCopulaTransformer - Issue#125by @csaladtype category error - Issue#124by @csala0.2.5 - 2020-09-18Miunor bugfixing release.Bugs FixedHandle NaNs in OneHotEncodingTransformer - Issue#118by @csalaOneHotEncodingTransformer fails if there is only one category - Issue#119by @csalaAll NaN column produces NaN values enhancement - Issue#121by @csalaMake the CategoricalTransformer learn the column dtype and restore it back - Issue#122by @csala0.2.4 - 2020-08-08General ImprovementsSupport Python 3.8 - Issue#117by @csalaSupport pandas >1 - Issue#116by @csala0.2.3 - 2020-07-09Implement OneHot and Label encoding as transformers - Issue#112by @csala0.2.2 - 2020-06-26Bugs FixedEscapecolumn_namein hypertransformer - Issue#110by @csala0.2.1 - 2020-01-17Bugs FixedBoolean Transformer fails to revert when there are NO nulls - Issue#103by @JDTheRipperPC0.2.0 - 2019-10-15This version comes with a brand new API and internal implementation, removing the old
metadata JSON from the user provided arguments, and making each transformer work only
withpandas.Seriesof their corresponding data type.As part of this change, several transformer names have been changed and a new BooleanTransformer
and a feature to automatically decide which transformers to use based on dtypes have been added.Unit test coverage has also been increased to 100%.Special thanks to @JDTheRipperPC and @csala for the big efforts put in making this
release possible.IssuesDrop the usage of meta - Issue#72by @JDTheRipperPCMake CatTransformer.probability_map deterministic - Issue#25by @csala0.1.3 - 2019-09-24New FeaturesAdd attributes NullTransformer and col_meta - Issue#30by @ManuelAlvarezCGeneral ImprovementsIntegrate with CodeCov - Issue#89by @csalaRemake Sphinx Documentation - Issue#96by @JDTheRipperPCImprove README - Issue#92by @JDTheRipperPCDocument RELEASE workflow - Issue#93by @JDTheRipperPCAdd support to Python 3.7 - Issue#38by @ManuelAlvarezCCreate way to pass HyperTransformer table dict - Issue#45by @ManuelAlvarezC0.1.2Add a numerical transformer for positive numbers.Add option to anonymize data on categorical transformer.Move thecol_metaargument from method-level to class-level.Move the logic for missing values from the transformers into theHyperTransformer.Removed unreacheble lines inNullTransformer.Numbertransfomerto set default value to 0 when the column is null.Add a CLA for collaborators.Refactor performance-wise the transformers.0.1.1Improve handling of NaN in NumberTransformer and CatTransformer.Add unittests for HyperTransformer.Remove unused methodsget_typesandimpute_tablefrom HyperTransformer.Make NumberTransformer enforce dtype int on integer data.Make DTTransformer check data format before transforming.Add minimal API Reference.Mergerdt.utilsintoHyperTransformerclass.0.1.0First release on PyPI.