Invertible Tabular GANs: Killing Two Birds with One Stone for Tabular Data Synthesis

Tabular data synthesis has received wide attention in the literature . This is because available data is often limited , incomplete , or can not be obtained easily , and data privacy is becoming increasingly important . In this work , we present a generalized GAN framework for tabular synthesis , which combines the adversarial training of GANs and the negative log-density regularization of invertible neural networks . The proposed framework can be used for two distinctive objectives . First , we can further improve the synthesis quality , by decreasing the negative log-density of real records in the process of adversarial training . On the other hand , by increasing the negative log-density of real records , realistic fake records can be synthesized in a way that they are not too much close to real records and reduce the chance of potential information leakage . We conduct experiments with real-world datasets for classification , regression , and privacy attacks . In general , the proposed method demonstrates the best synthesis quality ( in terms of task-oriented evaluation metrics , e.g. , F1 ) when decreasing the negative log-density during the adversarial training . If increasing the negative log-density , our experimental results show that the distance between real and fake records increases , enhancing robustness against privacy attacks . 1 Introduction . Generative models , such as generative adversarial networks ( GANs ) and variational autoencoders ( VAEs ) , have proliferated over the past several years [ 24 , 15 , 31 , 2 , 18 , 1 , 16 ] . GANs are one of the most successful models among generative models , and tabular data synthesis is one of the many GAN applications [ 7 , 4 , 28 , 27 , 21 , 38 ] . However , tabular data synthesis is challenging due to the following two problems : i ) Tabular data frequently contains sensitive information . ii ) It is required to share tabular data with people , some of whom are trustworthy while others are not . Therefore , generating fake records as similar as possible to real ones , which is commonly accepted to enhance the synthesis quality , is not always preferred in tabular data synthesis , e.g. , sharing with unverified people [ 27 , 5 ] . In [ 5 ] , it was revealed that one can effectively extract privacy-related information from a pre-trained GAN model if its log-densities are high enough . To this end , we propose a generalized framework , called invertible tabular GAN ( IT-GAN ) , where we integrate the adversarial training of GANs and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the negative log-density training of invertible neural networks . In our framework , we can improve1 or sacrifice the negative log-density during the adversarial training to trade off between synthesis quality and privacy protection ( cf . Figure 1 ) . Our generator based on neural ordinary differential equations ( NODEs [ 6 ] ) is invertible and enables the proposed training concept . For NODEs , there exists an efficient unbiased estimation technique of their Jacobian-determinants [ 6 , 16 ] . By using the unbiased Hutchinson estimator , therefore , we can efficiently estimate the negative log-density . After that , this negative log-density can be used for the following two opposite objectives : i ) decreasing the negative log-density to further increase the synthesis quality ( cf . IT-GAN ( Q ) in Figure 1 ) , or ii ) increasing the negative log-density to synthesize realistic fake records that are not too much similar to real records ( cf . IT-GAN ( L ) in Figure 1 ) . In particular , the second objective to make the log-density worse after little sacrificing the synthesis quality is closely related to the information leakage issue of tabular data . However , this invertible generator has one limitation – the dimensionality of hidden layers can not be changed . To overcome this limitation , we propose a joint architecture of an autoencoder ( AE ) and a GAN . The motivation behind the proposed joint architecture is twofold : i ) The role of the AE is to create a hidden representation space , on which the generator and the discriminator work . The hidden representation space has the same dimensionality as that of the latent input vector of the generator . Therefore , the input and output sizes are the same in our generator , which meets the invariant dimensionality requirement of NODEs . ii ) Separating the labor between the AE and the generator can improve the training process . In general , tabular data contains a large number of columns , which makes the synthesis more difficult . In our joint architecture , the generator shares its task with the AE ; it does not directly synthesize fake records but fake hidden representations . The decoder ( recovery network ) of the AE converts them into human-readable fake records . Therefore , our final training consists of the GAN training , the AE training , and the negative log-density regularization . We conduct experiments with 6 real-world tabular datasets and compare our method with 9 baseline methods . In many evaluation cases , our methods outperform all other baselines . Our contributions can be summarized as below : 1 . We propose a general framework where we can trade-off between synthesis quality and information leakage . 2 . To this end , we combine the adversarial training of GANs and the negative log-density training of invertible neural networks . 3 . We conduct thorough experiments with 6 real-world tabular datasets and our methods outperform existing methods in almost all cases . 1It is well known that VAEs generate blurred samples but achieve a better log-density than GANs . Therefore , there exists a room to improve the log-density of fake samples by GANs . 2 Related Work . We introduce various tabular data synthesis methods and invertible neural networks . In particular , we review the invertible characteristic of NODEs . 2.1 Table Data Synthesis . LetX1 : N be a tabular data which consists of N columns . Each column ofX1 : N is either discrete or continuous numerical . Let x be a record ofX1 : N . The goal of tabular data synthesis is i ) to learn a density p̂ ( x ) that best approximates the data distribution p ( x ) and ii ) to generate a fake tabular data X̂1 : N . For simplicity without loss of generality , we assume |X1 : N | = |X̂1 : N | , i.e. , the real and fake tabular data have the same number of records . This take can be accomplished by various approaches , e.g. , VAEs , GANs , and so forth , to name a few . We introduce a couple of seminal models in this field . Tabular data synthesis , which generates a realistic synthetic table by modeling a joint probability distribution of columns in a table , encompasses many different methods depending on the types of data . For instance , Bayesian networks [ 3 , 42 ] and decision trees [ 30 ] are used to generate discrete variables . A recursive modeling of tables using the Gaussian copula is used to generate continuous variables [ 29 ] . A differentially private algorithm for decomposition is used to synthesize spatial data [ 9 , 41 ] . However , some constraints that these models have such as the type of distributions and computational problems have hampered high-fidelity data synthesis . In recent years , several data generation methods based on GANs have been introduced to synthesize tabular data , which mostly handle continuous/discrete numerical records . RGAN [ 13 ] generates continuous time-series healthcare records while MedGAN [ 7 ] , CorrGAN [ 28 ] generate discrete discrete records . EhrGAN [ 4 ] generates plausible labeled records using semi-supervised learning to augment limited training data . PATE-GAN [ 21 ] generates synthetic data without endangering the privacy of original data . TableGAN [ 27 ] improved tabular data synthesis using convolutional neural networks to maximize the prediction accuracy on the label column . TGAN [ 38 ] is one of the most recent conditional GAN-based models . It suggested a couple of important directions toward high-fidelity tabular data synthesis , e.g. , a preprocessing mechanism to convert each record into a form suitable for GANs . 2.2 Invertible Neural Networks and Neural Ordinary Differential Equations . Invertible neural networks are typically bijective . Owing to this property and the change of variable theorem , we can efficiently calculate the exact log-density of data sample x as follows : log px ( x ) = log pz ( z ) − log det ∣∣∂f ( z ) ∂z ∣∣ , ( 1 ) where f : Rdim ( z ) → Rdim ( x ) is an invertible function , and z ∼ pz ( z ) . ∂f ( z ) ∂z is the Jacobian of f , which is the most computationally demanding part to calculate — it has a cubic time complexity . Therefore , invertible neural networks typically restrict the Jacobian matrix definition into a form that can be efficiently calculated [ 31 , 36 , 25 , 26 , 10 , 11 ] . However , FFJORD [ 16 ] recently proposed a NODE-based invertible architecture where we can use any form of Jacobian and we also rely on this technique . In NODEs [ 6 ] , let h ( t ) be a hidden vector at time ( or layer ) t in a neural network . NODEs solve the following integral problem to calculate h ( ti+1 ) from h ( ti ) [ 6 ] : h ( ti+1 ) = h ( ti ) + ∫ ti+1 ti f ( h ( t ) , t ; θf ) dt , ( 2 ) where f ( h ( t ) , t ; θf ) , which we call ODE function , is a neural network to approximate ḣ def = dh ( t ) dt . To solve the integral problem , NODEs rely on ODE solvers , e.g. , the explicit Euler method , the Dormand–Prince method , and so forth [ 12 ] . h ( ti ) is easily reconstructed from h ( ti+1 ) with a reverse-time ODE as follows : h ( ti ) = h ( ti+1 ) + ∫ ti ti+1 f ( h ( t ) , t ; θf ) dt . ( 3 ) Two other related papers are FlowGAN [ 17 ] and TimeGAN [ 39 ] although they do not synthesize tabular data . FlowGAN combines the adversarial training with NICE [ 10 ] or RealNVP [ 11 ] . Grover et al . showed in the paper that likelihood-based training does not show reliable synthesis for highdimensional space and they attempt to combine them [ 17 ] . TimeGAN also combines , given time-series data , the adversarial training and the supervised likelihood training of predicting a next value from past values . This supervised training is available because they deal with time-series data . In our case , we combine the adversarial training and the negative log-density regularization of NODEs , which are considered as more general than NICE and RealNVP [ 16 ] . 3 Proposed Method . We propose a more advanced setting for tabular data synthesis than those of existing methods . Our goal is to integrate the adversarial training of GANs and the log-density training of invertible neural networks . Therefore , one can easily trade off between synthesis quality and privacy protection . We describe our design in this section . The two key points in our design are that i ) we use an invertible neural network architecture to design our generator , and ii ) we integrate an AE into our framework to ii-a ) enable the isometric architecture of the generator , i.e. , the dimensions of hidden layers do not vary , and ii-b ) distribute the workload of the generator . 3.1 Overall Architecture The overall architecture is in Figure 2 . Our architecture can be classified into the following four data paths : AE-path ) The AE data path , highlighted in red , is related to an AE model to generate , given a real record xreal , a hidden representation hreal and reconstruct x̂real . Logdensity-path ) There are two different data paths related to the generator . The log-density path , highlighted in blue , is related to an invertible model to calcu- late the log-density of the hidden representation hreal , i.e. , log p̂g ( hreal ) . Therefore , we can consider it as the log-density of the real record xreal because the hidden representation is from the real record . GAN-path ) The second data path related to the generator , highlighted in orange , is for the adversarial training . The discriminator reads hreal and hfake , which is generated by the generator from the latent vector z , to distinguish them . Synthesis-path ) The last data path , highlighted in green , is used after finishing training our model . Using the generator and the decoder , we synthesize many fake records .

Summary: The paper proposes a new architecture for tabular data synthesis. In particular, the proposed architecture utilizes a combination of autoencoders (AEs), generative adversarial networks (GANs), and invertible models. The architecture can also consider aspects of privacy via a hyperparameter which controls the likelihood of the generator producing data from the training set.  Main Contributions:   - The proposed architecture, utilizes an AE to first change inputs into a latent dimension for the GAN to then process.   - The invertible model is utilized in the generator of the GAN -> This ensures that the latent representation of the AE agrees with the invertible generator (such that there exists an input to the generator which will produce the true latent representation of an AE input)   - Furthermore, the random generator inputs induce a pdf -> which gives a pdf on the (reconstructed) AE latent space generated by the generator network (via invertibility). This can thus be used as a regularization condition to penalise/reward the likelihood of training data's latent representations generated by the generator network.   - The following is experimentally tested using a number of datasets and against a number of baselines, with better reported performance.   - A privacy experiment is done. Here, the closeness of generated data to training data is evaluated.