How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data
Authors: Mihaela C Stoian
Salijona Dyrmishi
Maxime Cordy
Thomas Lukasiewicz
Eleonora Giunchiglia
Venue: ICLR 2024 poster (2024)
Keywords: Synthetic Data Generation
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Deep Generative Models (DGMs) have been shown to be powerful tools for generating tabular data, as they have been increasingly able to capture the complex distributions that characterize them. However, to generate realistic synthetic data, it is often not enough to have a good approximation of their distribution, as it also requires compliance with constraints that encode essential background knowledge on the problem at hand. In this paper, we address this limitation and show how DGMs for tabular data can be transformed into Constrained Deep Generative Models (C-DGMs), whose generated samples are guaranteed to be compliant with the given constraints. This is achieved by automatically parsing the constraints and transforming them into a Constraint Layer (CL) seamlessly integrated with the DGM. Our extensive experimental analysis with various DGMs and tasks reveals that standard DGMs often violate constraints, some exceeding 95% non-compliance, while their corresponding C-DGMs are never non-compliant. Then, we quantitatively demonstrate that, at training time, C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their standard counterparts with up to 4.5% improvement in utility and detection. Further, we show how our CL does not necessarily need to be integrated at training time, as it can be also used as a guardrail at inference time, still producing some improvements in the overall performance of the models. Finally, we show that our CL does not hinder the sample generation time of the models.

Published as a conference paper at ICLR 2024
How REALISTIC Is YOUR SYNTHETIC DATA? CONSTRAINING DEEP GENERATIVE MODELS FOR TABULAR DATA
Mihaela Cätälina Stoian Salijona Dyrmishi Maxime Cordy?, Thomas Lukasiewicz Eleonora Giunchiglia
University of Oxford, University of Luxembourg, Vienna University of Technology
ABSTRACT
Deep Generative Models (DGMs) have been shown to be powerful tools for gen- erating tabular data, as they have been increasingly able to capture the complex distributions that characterize them. However, generate realistic synthetic data, it is often not enough to have a good approximation their distribution, as it also requires compliance with constraints that encode essential background knowl- edge on the problem at hand. In this paper, we address this limitation and show how DGMs for tabular data can be transformed into Constrained Deep Genera- tive Models (C-DGMs), whose generated samples are guaranteed to be compli- ant with the given constraints. This achieved by automatically parsing the con- straints and transforming them into Constraint Layer (CL) seamlessly integrated with the DGM. Our extensive experimental analysis with various DGMs and tasks reveals that standard DGMs often violate constraints, some exceeding 95% non- compliance, while their corresponding C-DGMs are never non-compliant. Then, we quantitatively demonstrate that, training time, C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their stan- dard counterparts with up 6.5% improvement in utility and detection. Further, we show how our CL does not necessarily need to be integrated at training time, it can be also used as a guardrail inference time, still producing some im- provements in the overall performance of the models. Finally, we show that our CL does not hinder the sample generation time of the models.
INTRODUCTION
Synthetic data generation represents important area of machine learning (ML) due to its numerous applications in the real world. Indeed, synthetic data have been increasingly used to augment real data to improve the predictive performance of ML models (see, e.g., Han et al. (2005), ), remedy data scarcity (see, e.g., Choi et al. (2017)), ) and to promote fairness (see, e.g., van Breugel et al. (202T)), and they are now even used generate brand-new datasets ensure privacy in sensitive settings (see, e.g., Jordon et al. (2019); Yoon et al. (2020); Lee et al. (2021)).
Deep Generative Models (DGMs) have been shown be powerful tools for generating tabular data, as they have been progressively more able to capture the complex distributions that characterize such data (see, e.g., Kim al. (2023)). However, for generating realistic tabular data, it is insuf- ficient to just learn a good distribution approximation; it is necessary to create samples that obey a set of constraints expressing background knowledge about known characteristics of the features and/or existing relationships among them. For example, if we are generating data from a clinical trial dataset, then, for every synthetic sample, we surely want the value associated with the "maxi- mum level of hemoglobin recorded" column to be greater than equal to the one associated with the "minimum level of hemoglobin recorded" column. Indeed, any sample violating such constraint is not realistic. Existing methods, while excellent at capturing complex distributions, able to learn even from this simple background knowledge, and thus do not provide any guaran- tee of constraint satisfaction. In addition, currently, work incorporates background knowledge in
still not
"qqual contribution.

---

Published as a conference paper at ICLR 2024
DGMs for tabular data except for GOGGLE (Liu et al. 2022 which, however, is only able to inject very simple background knowledge about existing correlations among features.
In this paper, we address this limitation, and show how integrate any background knowledge that can be expressed as a set of linear inequalities into different standard DGMs for tabular data. To this end, we introduce novel approach able to transform different DGMs into corresponding Constrained Deep Generative Models (C-DGMs), i.e., DGMs whose generated samples are guar- anteed to be compliant with the of user-defined constraints. Our method takes as input linear inequality constraints and a DGM, and then automatically parses the constraints and generates a differentiable Constraint Layer (CL) that can be seamlessly integrated with the DGM. Our method thus returns novel model, the C-DGM, whose sample space is guaranteed to be compliant with the constraints. To evaluate our approach, we conduct an extensive experimental analysis that in- volves testing Wasserstein GAN (WGAN) (Arjovsky et all],2111111 2017), CTGAN (Xu et al. 2019). Table- GAN (Park et al. 2018), TVAE (Xu et al. 2019). and GOGGLE (Liu et al. 2022) on a collection of six tasks for which rich background knowledge is available: our datasets are annotated with up to 31 constraints, each including up to 17 different features. To this end, we first show that these models often violate the constraints, with WGAN even generating 100% of non-compliant samples for one dataset, and five models generating more than 95% con-compliant samples for another one (see T.ble Then, we quantitatively demonstrate that the addition of the constraints in the topology of the network improves the performance both in terms of detection (i.e., how well the generated data matches the real data distribution) and in terms of utility (i.e., how well the generated data can replace real data to train ML models), with up to 6.5% improvement over all datasets. This validates the principle that generating compliant samples contributes to improving their quality: a challeng- ing problem for which the state-of-the-ar progresses only by small successive improvements (Liu al., 2022] Kim et al. 2023). Further, we show how CL does not necessarily need to be added at training time, but it can simply be incorporated at inference time as a guardrail. This is important when the model is available only as a black-box system without any possibility to act on Finally, show how CL does not hinder the sample generation time of the models. This paper thus follows the same principles outlined in Giunchiglia et al. (2023a), where the authors advocate for a more sgquirements-driven machine learning. where performance is just of the requirements defining a model has to satisfy (others might include safety, fairness, robustness etc.).
Contributions: (i) We show that standard DGMs often generate synthetic data that are not aligned with the available background knowledge. (ii) We develop a method able to take as input any set of constraints expressed as linear inequalities and automatically create a differentiable constraint layer that can be seamlessly integrated with DGMs. (iii) We prove that the samples generated with C- DGMs are guaranteed to be compliant with the given constraints. This is the first method that is able to incorporate complex background knowledge into DGMs and guarantee that it is always satisfied. (iv) We show that C-DGMs outperform their standard counterparts in terms of both utility and detection. (v) We display how CL can be used as a guardrail at inference time. (vi) We quantitatively demonstrate that CL has negligible impact on the samples' generation time.
PROBLEM STATEMENT
Let px be an unknown distribution over X E R,,, and let 'D be the training dataset consisting of N i.i.d. samples drawn from px. The goal of standard generative modeling is to learn from D the parameters 0 of a generative model such that the model distribution po approximates px.
In constrained generative modeling, we assume to have access to a set of constraints expressing some background knowledge about the sample space of px, i.e., stating which samples are admissible and which are not. The goal is to learn the parameters / of a generative model such that (i) the model distribution pe approximates px. and (ii) the sample space of pa is compliant with the constraints,
Let II be a finite set of constraints expressing such background knowledge, where each constraint is linear inequality over set of variables X' {Øx l = 1...., D}. each variable uniquely corresponding to a feature of the dataset, and for this reason in the following we do not distinguish between variables and their corresponding features. Each constraint has the following form:
Σ Wk Th + b20.
(1)
2

---

Published as a conference paper at ICLR 2024
with wk R. R, {2,>) >}. and F&k representing continuous feature. Indeed, any set of linear inequalities using E E {<, S. -,=, can be easily converted into an equivalent one in which 2 (2,>). In ① when 2 is we say that the linear inequality is strict.
Example 2.1. II = { 22 2 0, F2 5 > 0} expresses that for every generated sample ž, the 1st feature must be greater than or equal to the 2nd, and that its 2nd feature must be greater than 5.
Linear constraints represent important class of constraints -ss underlined by the existence of en- tire fields dedicated to its study (see c.g., linear programming) and they possess many favourable properties, such as: (i) they are logically complete, as inconsistencies can be determined by comput- ing linear combinations of the constraints themselves (Farkas, 1902). (ii) it is possible to compile them in backtrack free representation that allows for computing satisfying samples in linear time in the size of the compiled representation (Dechter) 1999), and (iii) they always define a convex space.
A sample generated by DGM is an assignment to the variables in X. A sample è satisfies:
1. the constraint ! if Σx wx.Tx + bz 0, where 7y is the value associated with the feature Th.
2. a set II of constraints if it satisfies all the constraints in II.
Contrarily, if è does not satisfy a constraint (resp., II) then 2 violates the constraint (resp., IT). A set of constraints is satisfiable if there exists sample satisfying II. A DGM model m is compliant with II if all its generated samples satisfy II.
Given a DGM m generating samples that possibly do not satisfy II, our goal is to create a C-DGM noted C-m such that: (i) C-m is compliant with II, and (ii) for each sample ă generated by m, C-m generates H' such that i' satisfies II and that, intuitively, is optimal in the sense that it minimally differs from ã (while taking into account the user preferences on which features should be changed).
In the following, given a constraint & of form I a variable Ty appears positively (resp., negatively) in ø we > 0 (resp., wi < 0). A variable appears in ø if appears positively or negatively in ø.
CONSTRAINED DEEP GENERATIVE MODELS
To achieve our goal, we build differentiable constraint layer (CL) that (i) can be seamlessly inte- grated with multiple DGMs, (ii) guarantees the satisfaction of the constraints, and (iii) guarantees a possibly optimal output that minimally changes the initial DGM predictions. Given generic DGM, CL can be added at training time right after the layer generating the samples. CL allows the gradients to backpropagate through it and thus the network can learn to exploit the background knowledge encodes. To better ground the discussion, consider Figure where we can see an overview of how to integrate our layer with a based model. In the Figure, the noise vector z is fed into the gen- erator, which in turn outputs a sample ö. Often, ī needs to be transformed using pre-defined map- ping f before using it, as the output space of the generator may not coincide with the feature space of the real data. Then, f(ā) is passed to CL. which returns the corrected /' now compliant with the constraints Il-and then (ā' is passed to the discriminator together with f (r). where r D is real data point for our dataset. In the follow- ing, we focus on the construction of CL, and, for ease of presentation and without loss of generality, we assume that the output space of the generator coincides with the feature space of the real dataset. Thus, assume that the generator generates sample ē, which then gets fed directly to CL, which in turn returns CL))), the output of the C-DGM.
(r)
=1 ( 2,,
12,
(()
Generator
Constraint Layer
'xy
Discriminator
Figure 1: Overview on how to integrate CL into a GAN -bsed model.
Given sample ë. the basic idea of the constraint layer is to incrementally compute the value ã; of the ith feature in the sample, minimally changing it whenever à; is not compatible with the values of the already analyzed features and the given constraints. To this end, let A X + [1, DJ be variable ordering function injectively mapping each variable in X' to an integer in [1, D). Such ordering is user-defined and defines the order of computation of the features. To ease the notation, and without
3

---

Published as a conference paper at ICLR 2024
loss of generality, from Then, given A, we associate to each variable Dy a set of constraints II, inductively defined as follows:
denote with z; the variable with order (and thus X(x;) i).
1. if i D, then II; II, and
2. i < D and j = i + 1, then
I1; = 1, (L) Unf) U fred, ,²²) oo EII + )
where (i) 11 (resp.. 11") ) is the subset of 11, where t, appears negatively (resp., positively) and (ii) red, x²2) is the reduction of and d² over defined, assuming o² Σx w
and Φ3² EE wfax + 12 p² 0 with 22. E (2,>}. J.as
Σ (w + w w3)) + + + 68|w 20,
in which E. is D if b=3 and 2 is > otherwise.
Example 3.1. Continuing with the example, = {z] - -2 2 0, 22 - 5 > 0}. 112 = 11, and
Ih fredz (21 22 0, 5 > 0)} = {a1 5 0}.
Consider now a generic model m and a sample ō generated by m. To define CL))). we first compute the upper bounds (resp., lower bounds) associated with ã; given the constraints and the computed values CL()), associated with the features I, with j < i. Indeed, each constraint & of the form 1p in 11;" (resp.. 11,) defines lower (resp., upper) bound on I; given by the expression =f Exz( (w//w, ) b/ w,, as is equivalent z; - _2 (resp.. -2, + + (20) 0).
Then, for each I,..., D, the values associated with the least upper bound (ub;) and greatest lower bound (1b;) for the feature given the values of CL(T))...... CL(T); are defined to be
ub. = min( U (CL()))
lb; max U E (CL())))
(2)
DETT
TENT
where (CL())) corresponds to ef instantiated with the values of CL(ă). Clearly, the fact that Il is satisfiable and the definition of ub; and lb; in (2) which incorporates the values of CL()), for j < i, ensures lb; S ub, for any = I..... D (as formally stated below). However, as- suming, e.g,, that for sample λ, I; S lb,, it may not be possible to set the value of the fea- ture z; to be exactly lbs, because there exists a strict linear inequality z; > lb; in II Indeed, in these cases, there does not exist a value v for with > lb, and minimum |0 7./. and the only way out is to select u = lb; + e, where e > 0 is an arbitrarily chosen small value. For this reason, for every i 1,..., D. CL()), is defined as CL(ā); = min't (max' lb,), ubi),
C CLE3.)
(3)
where max' lb:)) is equal to v = max( lb;) there does not exist a strict inequality á in II." such that v = E (CL( and is equal to + e with e > small enough to ensure e < ub, - 1b, otherwise. The analogous intuition applies to min'. Example 3.2. Continuing with the example. =I= {01 T2 > 0, 12 5 0). 1I, IL. 11, {z1 5 0]. If 7 and 22 = 3, then CL((); = 7 and CL(ā)2 5.1, while if ān 82 3, then CL(()) = CL(;)2 5.1, such values computed with 0.1.
CL(i); = min(max((,,..1) ,
CL( = 5.1(
x
Figure 2: CL in Example 3.2
Notice that for every sample it always possible to compute, at least in theory, a value for c such that CL( satisfies the constraints in II. In practice, in the implementation, we fix to be the minimum representable positive float.
Theorem 3.3. Let be deep generative model. Let 1I be satisfiable and finite set of constraints over the sample space. Let X be a variable ordering and CL be the constraint layer built from X and II. Then, the model C-m obtained by incorporating CL in m is compliant with II.
We assume the function min(V) over finite set V of values in IR to be defined as min(0) = +o0, and min(fu) U V') vif S min(V') and min(V") otherwise. Analogously for the function max(V).
4

---

Published as a conference paper at ICLR 2024
which contains also the proof of the following corollary. Corollary 3.4. the hypotheses of the theorem, for every sample produced by m and for every
See proof in Appendix A.1
i= == D, it is always the case that ub; 2 lb;, and thus also CI(3); = min' (mnx" (ā:, (b:), ub:) max (min' (i;, ub;), W.))
(4)
In addition to satisfying the constraints, our goal was to guarantee CL(Ø) optimality, capturing the intuition that it has to minimally differ from I, i.e., that it is not possible to reduce the distance (CI((); - 32l for each i= = I,..,, D. while satisfying the constraints. Such intuition is formalized by saying that (T( is optimal (with respect to I1) if
1. (T. satisfies IT, and
2. there does not exist another sample N' different from (ă) which satisfies II and such that for each i = 1,..., D. 1 2, < [CL( - Ez/-
Indeed, if a sample i satisfies the constraints, we expect CL( to return ã and thus be optimal. Notice that more than one value might satisfy the above definition of optimality. Indeed, given a sample E. the proposed definition has the advantageous properties that (i) if E satisfies the constraints then CL( = 3 is the unique optimal solution, and (ii) if does not satisfy the constraints then not possible get "closer" to ह along all the dimensions while satisfying the constraints. This rather relaxed definition of optimality allows us to consider different methods to compute CL()), each corresponding
preference about the set of features which we want to minimally change. In our case, we want to minimally change those features whose distribution is better approximated by the unconstrained DGM (see Appendix C.3 for the definition of two different policies to compute CL)))). Indeed, though in different setting. Giunchiglia et al. (2023b) showed that computing correction without taking into account the confidence the predictions consistently leads to worse performance.
Theorem 3.5. Let m be a deep generative model. Let II be a satisfiable and finite set of constraints over the sample space. Let X be variable ordering and CL be the constraint layer built from X and II. Let T a sample produced by m. If satisfies II, then CL(ā)= = and CL(ā) is optimal.
The proof is in Appendix A.2 Excluding the lucky cases in which the sample i already satisfies the constraints, we will show that CL(@) is ensured to be optimal, e.g., when II does not contain strict inequalities. In the general case, however, CL(Ø) may not be optimal since an optimal value for CL()) may not exist. This is due to the presence in II of strict linear inequalities (see, e.g, in Example
3.2) 3.2). However, for such cases we can prove that CL(ā) can take values as close as needed to the original sample ă while satisfying the constraints. In order to make these notions precise, we introduce CL -1.2(6 (Ø) as the value computed by CL for the constrained generative modeling problem in which II is replaced with 2I2 In II2 each strict linear inequality Σx WgEE + b > 0 in II is replaced with the corresponding non-strict one, i.e., Σ. W2 + b Z O. By definition, the samples satisfying IT also satisfy IT²; and ( ) is optimal when considering ITz but may not be optimal when considering II (as (z) violates some strict linear inequality in IT). In such cases, (T) can be considered as the limit optimal solution when considering II and prove that CL(Ø) can get arbitrarily close CL 2
(?) while satisfying the constraints in II.
Theorem 3.6. Let m be a deep generative model, Let II be a satisfiable and finite set of constraints over the sample space. Let X be variable ordering and CL be the constraint layer built from X and TI. Let à be a sample produced by m.
1. CL(T) is optimal if (CC(() CL and
2. CL(ī) tends to CL² CL²(3) as the values used to compute CL(i) tend to 0, otherwise.
According to the theorem, CL(ā) is guaranteed to be optimal when no values are introduced for computing CL) and an e value has to be introduced because of a strict linear inequality, (Z) can still be as close as needed to the optimal solution CL.² (i) of the relaxed problem with IT=.
CL.²(G)
EXPERIMENTAL ANALYSIS
We quantitatively evaluate different aspects of our approach support the claims of our paper:
5

---

Published as a conference paper at ICLR 2024
Table 1: Constraint violation rate for each model and dataset.
Model/Dataset
URL
WiDDS
LCLD
Heloc
FSP
News
WGAN
11.1+1.6
98.2+-0.2
100.0+0.00
57.0+13.0
70.7++.3
45.6+-9.66
TableGAN
1.9
96.4+2.4
6.1+0.9
45.6+16.3
71.6+8.7
72.6+:5.3
CTGAN
3.1=2.66
99.9+0.00
11.8+2.77
41.6+12.1
74.3+
54.3+=1011
TVAE
3.0=0.77
99.910000
3,910.55
1.
66.41:300
50.3:1399
GOGGLE
5.916.66
78.2
13.1#2.9
47.3+7.0
63.7:117.6
44.8:1..2
All C-models
0.0.1..0
0.0.10.0
0.0.00.0
0.0:10.0
0.0 +0.0
0.0
1. Background knowledge alignment: How often do constraint violations occur? Section 4.2
shows that standard DGMs often violate the constraints expressing background knowledge.
2. Synthefic data quality; Does background knowledge improve the synthetic data quality? Sec- tion shows that C-DGMs outperform DGMs in terms of two metrics: utility and detection.
3. Post-processing ability: Can CL. act as a guardrail at inference time? Section 4.4 shows that CL. can be used as guardrail while slightly improving the overall performance of the DGMs. Generation time: Does background knowledge injection affect generation time? Section 4.5
shows that the sample generation time for C-DGMs is comparable with that of DGMs.
4.1 EXPERIMENTAL ANALYSIS SETTINGS
Models: We experimented with five DGM models, namely WGAN (Arjovsky er al.) 2017), Table- GAN Park et al. 2018), CTGAN (Xu et al. 2019), TVAE (Xu et al. 2019), and GOGGLE (Liu ctt al. 2022). Each model was augmented by our CL, resulting in C-WGAN, C-TableGAN, C-
CIGAN, C-TVAE, and C-GOGGLE models. Refer to Appendix B.1 for details the models.
To evaluate the models, we selected real-world datasets for which a clear description of the feature relationships either was available has been derived from our domain expertise. We limited our selection to datasets having at least known constraints. As a result, we found 6 such datasets, 4 for binary classification tasks, for multi-class classification, and 1 for regression. The datasets vary not only in size (2K to 1M rows), but also in feature number (24 to 109) and feature type (continuous, categorical, or mixed). The constraints between features are equally varied in number (from 4 to 31) as well as number of features appearing in each constraint (from I to 17). Additional details about the datasets and constraints statistics can be found in Appendix B.2 and respectively.
Datasets:
B.3
Quality Evaluation: We quantitatively evaluate two characteristics of the quality of generated sam- ples: (i) utility, i.e., whether our synthetic data can be used as an alternative fake dataset to train other models, and (ii) detection, i.e., whether classifier can be trained tell apart the synthetic samples from the real data. In order to evaluate the utility of the generated samples, follow the Train Synthetic, Test on Real (TSTR) framework (Esteban et al.) 2017 Jordon et al. 2019) which is a paradigm used in different papers (see, e.g., Kim al. (2023)) Following this framework, we train multiple models (e.g., Random Forest, XGBoost etc.), validate them with original training data, and test them real test data. To evaluate such models,
report the average Fi-score, AUROC, and weighted F1 F1-score for the classification datasets, and explained variance and mean absolute error for regression. On the other hand, evaluate in terms of detection, we follow the popular evaluation method for tabular data generation (see, c.g., Liu et al. (2022)) and train six models (e.g.. Decision Tree, AdaBoost etc.) to distinguish between the real and synthetic datasets. Again, we report average F1-score, AUROC and weighted F1-score. All the metrics are reported over 5 runs. For more details on the evaluation procedure, refer to Appendix B.4 Our full code provided on GitHub
4.2 BACKGROUND KNOWLEDGE ALIGNMENT
How often do constraint violations occur? To assess the ability of standard DGMs create samples that are aligned with the available background knowledge, we measure the constraints' violation rate (CVR) which, given set of synthetic samples S and a set of constraints II, represents the
Thhe code is available at wttp:///chh.cccmmmimiioooooaoonn

---

Published as a conference paper at ICLR 2024
18
Max hemoglobin
16
14
12
10
Real
TableGAN
C-TableGAN
8
8
12
14
12 14 16
8
10
16
Min hemoglobin
Min hemaglobin
Min hemogiobin
Figure 3: Real data and samples generated by TableGAN and C-TableGAN for WiDS.
percentage of samples in S violating II. Table 1 shows the CVR for each tested model and dataset. In particular, the first five rows report the CVR for the standard DGMs, while the last row reports the CVR for all their respective constrained versions (i.e., C-WGAN, C-TableGAN, etc.) it is always equal to zero for all models, datasets and runs. This is expected, as our C-DGMs offer theoretical guarantees to always generate samples compliant with the constraints. Let us now focus on the results obtained for the standard DGMs. Firstly, no standard DGM guarantees to produce only samples satisfying the constraints. Further, in more than half of the reported experiments, we have CVR > 50% (i.e., more than half of the generated samples violate the constraints), with peaks of CVR 100% for WGAN tested on LCLD and CVR 95% for four out of five DGMs tested on WiDS. Additionally, to give better overview of the phenomenon, in Appendix C.I we present the results according to two other metrics: (i) constraints violation coverage (CVC), which, given S and II. represents the percentage of constraints in II that have been violated at least once by any of the samples in S, and (ii) samplewise constraints violation coverage (sCVC), which represents the average
the samples in S of the percentage of the constraints violated by each sample. Finally, to visually demonstrate the difference between DGMs and the C-DGMs, we select three constraints where only two variables appear, and create two-dimensional scatter-plots of the (i) real data, (ii) the samples generated by the DGMs and (iii) the ones generated by the C-DGMs, with the variables appearing in the constraint on the axes. In Figure we present only the plots for TableGAN, the other figures can be found in Appendix C.I In Figure 3 we consider the constraint Lew 2 MinHemoollobnn from the WIDS dataset, and we highlight red the region violating the constraint. The Figure confirms our quantitative analysis, as TableGAN often violates the constraint, while C-TableGAN never does. Further, we can see how the distribution of the samples generated by C- C-TableGAN more closely matches the one of the real data.
4.3 SYNTIETIC DATA QUALITY
Does background knowledge im- prove the synthetic data quality? To validate our hypothesis, we com- pare the performance the standard DGMs with their respective C-DGMs in terms of utility and detection, as specified in Section 4.1 The results of our experiments are summarised in Table 2 where we report the av- erage utility performance (left) and the average detection performance (right) over all datasets with respect to: Fl-score (F1), weighted F1-score (wF1), and Area Under the ROC Curve (AUC). As we can see from Table 2 except for two cases (Fl util- ity of C-CTGAN and AUROC detec- tion of C-TVAE), our C-DGMs out- perform their standard counterparts
Table 2: Results for every C-DGM and their respective stan- dard version. The best results are in bold.
Utility (1)
Detection (L)
F1
wF1
AUC
F1
wF1
AUC
WGAN C-WGAN
0.463
0.488
0.730
0.945
0.943
0.954
0.483
0.502
0.745
0.915
0.912
0.934
TableGAN
0.330
0.400
0.704
0,908
0.907
0.926
C-TableGAN
0.375
0.432
0.714
0.898
0.895
0.917
CTGAN
0.517
0.532
0.771
0.902
0.901
0.920
C-CTGAN
0.516
0.537
0.773
0.894
0.891
0.919
TVAE
0.497
0.527
0.767
0.869
0.868
0.892
C-TVAE
0.507
0.537
0.773
0.868
0.867
0.898
GOGGLE
0.344
0.373
0.624
0.926
0.926
0.943
C-GOGGLE
0.409
0.427
0.667
0.925
0.916
0.937
both in terms of utility and detection according all metrics almost always, and never have worse

---

Published as a conference paper at ICLR 2024
performance. Often, such difference in performance is non-negligible, as in the case of GOGGLE where we register a 6.5% difference in terms of utility when calculated in terms of Fl-score.
Notice that for utility results had to leave out of the average News regression dataset, there we use metries with different scales (i.e., Explained Variance and Mean Absolute Error). The utility results obtained on News reported in the Appendix C.4.1 in Table 20 For News, in all C-DGMs we see an improvement or no change in the utility performance according to at least one of the two metrics. We report the utility and detection for individual datasets in Appendix C.4.I C.4.1 hyperparameters used in each experiment are reported in Appendix B.5 Table
and C.4.2 The
4.4 POST- POST-SROCESSING ABILITY
Can CL act as guardrail at infer- ence time? In many practical scenar- ios, users might not want to modify and retrain their model to add a con- straint layer. For example, this might be the case for companies that al- ready have their (possibly black-bor. box) models and/or already generated data that simply needs to aligned with the available background knowledge. In this Subsection, we thus show that CL can be added at inference time as a guardrail on any model without hindering the quality of the gener- ated data. The results of our analysis shown in Table where P-DGM stands for a standard DGM with CL added as a post post-processing step at ference time. The results demonstrate that P-DGMs outperform their standard counterparts 17 times out of 30 (in 1 other case having equal performance), thus showing the potential of using CL. as post-processor. As expected, the differ- ence in results is often very little, as the CL layer is only applied at inference time, On the contrary, if we compare the P-DGMs models with the C-DGMs from Table 2 we can see that the C-DGMs almost always perform better (there are only two exceptions where the difference in performance is as little as 0.001). This is again expected, as C-DGMs are injected with the background knowledge at training time, thus making them able to exploit it.
Table 3: Results for every P-DGM and their respective stan- dard version. The best results are in bold.
Utility (t)
Detection (4)
F1
wF1
AUC
F1
wF1
AUC
WGAN
0.463
0.488
0.730
0.945
0.943
0.954
P-WGAN
0.462
0.489
0.732
0.930
0.929
0.946
TableGAN P-TableGAN
0.330
0.400
0.704
0.908
0.907
0.926
0.328
0.399
0.707
0.901
0.898
0.922
CTGAN
0.517
0.532
0.771
0.902
0.901
0.920
P-CTGAN
0.512
0.528
0.770
0.897
0.900
0.926
TVAE
0.497
0.527
0.767
0.869
0.868
0.892
P-TVAE
0.495
0.524
0.767
0.875
0.876
0.902
GOGGLE
0.344
0.373
0.624
0.926
0.926
0.943
P-GOGGLE
0.348
0.374
0.626
0.925
0.924
0.942
4.5 IMPACT ON SAMPLES GENERATION TIME
Does background knowledge injec- tion affect generation time? Another important aspect of synthetic data generators is their sample generation time (see, e.g., Kim et al. (023);
Table 4: Sample generation time in seconds.
URL.
WIDS
LCLD
Heloc
FSP
News
WGAN
0.02
0.03
0.01
0.00
0.00
0.01
C-WGAN
0.02
0.04
0.01
0.01
0.01
0.02
Xiao et al. (. In order to show that the addition of CL has basically no impact on speed, for each dataset and model we measure the generation time of 1000 samples and report the average results over runs in Table The results show that the constrained versions are as fast or at most 0.03s slower than the unconstrained version for 15 and 14 cases (out of 30 cases), respectively, with only one case 0.27s slower than the unconstrained version. This means that the constrained layer introduces almost no overhead to the sampling process in many practical scenarios. This result is particularly interesting
TableGAN
0.18
3.21
0.17
0.17
0.18
0.20
C-TableGAN
0.19
3.19
0.18
0.18
0.18
0.19
CTGAN
0.13
0.26
0.08
0.06
0.08
0.14
C-CTGAN
0.14
0.27
0.08
0.06
0.08
0.14
TVAE
0.12
0.27
0.06
0.06
0.06
0.12
C-TVAE
0.13
0.27
0.07
0.06
0.07
0.13
GOGGLE
0.71
3.99
9.91
0.16
0.06
2.01
C-GOGGLE
0.71
3.86
10.18
0.16
0.06
2.04
8

---

Published as a conference paper at ICLR 2024
given that representation of the constraints may have exponential size (Dechter 1999). In- deed, we did not experience any exponential blow-up, which happens in the worst case, when the constraints have many variables in common.
RELATED WORK
Our work lies at the intersection of standard tabular data synthesis and neuro-symbolic AI for its ability to incorporate background knowledge into neural architectures.
Tabular Data Synthesis. Several approaches based on DGMs have been specifically designed to address particular challenges in generating tabular data such as mixed types of features, and im- balanced categorical data. Notable among these are GAN-based approaches like TableGAN (Park) et al. 2018), CTGAN (Xu et al. 2019), OCT-GAN (Kim et al. 2021). and IT-GAN (Lee et al.) 2021. These methods leverage the power of GANs to model the underlying data distribution and generate synthetic samples that closely resemble real-world tabular data. Few approaches focus es- pecially on particular domains like healthcare where privacy is important (see, e.g., (Choi et al, 2017 Che et al., 2017)). Following privacy concerns, two approaches, i.e., DPGAN (Xie et al., 2018 and IATE-GAN (Jordon et al.] 2019), incorporate differential privacy techniques to ensure that the generated synthetic data does not reveal sensitive information about the individuals in the original dataset. Alternative to GANs, Xu et al. (019) proposed TVAE as a variation of the stan- dard Variational AutoEncoder, while TabDDPM (Kotelnikov et al.] 2023 and STaSy (Kim et al. 2023) were proposed following the achievements of score- score-based models. Finally, [Liu et al. (2022) proposed GOGGLE, model that uses graph learning to infer relational structure from the data.
Neuro-symbolic AI Methods. Neuro-symbolic AI raised great interest in the recent past for many reasons (see, e.g., (Raedt et al.) [2018] Avvvia Garcez & Lamb 2023), among which there is the abil- ity to incorporate complex background knowledge in deep learning models. Standard approaches in the field incorporate logical constraints in the training of neural network by introducing addi- tional terms in the loss function that penalize the network for violating them (see, e.g., (Diligenti al., 2012 2017 Xu et 2018 Fischer et al.) 2019 Badreddine al. 2022 Stoian al. (023)) These approaches, while often easy to incorporate into neural models, do not give any guarantee that the constraints are actually satisfied. Alternative approaches, e.g., DeepProbL.og (Manhaeve et al. 2018), rely on solver to inject background knowledge as a set of definite clauses both at training and inference time. A more recent work (van Kricken et al., 2023) in this line proposed using neural net- works for performing approximate inference in polynomial time to address the scalability problem of probabilistic neuro-syne cuurosymmolic learning frameworks. More closely to our approach, we find the recent methods that are able to incorporate the background knowledge into the topology of the network it- self. However, these methods are either only able to deal with constraints expressed in propositional logic (Ahmed et al., (022) or even less expressive constraints (Grunchiglia & Lukasiewicz (02T) or
become quickly intractable for even moderately complex logical constraints (Hoernle et al. 2022).
Regarding the application of veuro-symbolic AI in generative tasks: [iello al. (2020) incorporates
propositional logic constraints on GANs for structured objects generation, M et al. (2022)
shows how to integrate ProbLog (Raedt al., 2007) with VAEs. Our work differs from the last two both in the application domain and in the type of background knowledge we incorporate.
6 DISCUSSION AND CONCLUSIONS
Our work introduces a novel method able to translate complex constraints, expressed as linear in- equalities, into seamlessly integrated layer within Deep Generative Models (DGMs), thereby yield- ing Constrained Deep Generative Models (C-DGMs). This infusion of domain-specific knowledge directly into the network's architecture facilitates the generation of more realistic tabular data sam- ples, C-DGMs are guaranteed to comply with the specified constraints. Furthermore, ments reveal that including our layer during the training phase enhances the quality of the generated samples, thus underlining the importance of constraint integration not only during data generation but also throughout the model's 's learning process. Limitations: In this paper, we focus on constraints that can be expressed as linear inequalities. While this covers a wide range of scenarios, some rela- tionships among features may demand more expressive constraint representations. In the future, we expect many developments in this work where even more complex constraints are considered.
experi-

---

Published as a conference paper at ICLR 2024
AUTHOR CONTRIBUTIONS
Mihaela Cätälina Stoian: conceptualization, methodology, writing, constraint layer and software implementation, experimental analysis, model training, debugging; Salijona Dyrmishi: conceptual- ization, experimental design and analysis, writing, software implementation, model training, debug- ging; Maxime Cordy: supervision, editing, writing reviewing: Thomas Lukasiewiez: supervision, editing, writing reviewing; Eleonora Giunchiglia: conceptualization, methodology, formalization, writing, supervision.
ACKNOWLEDGMENTS
Mihaela Cätälina Stoian is supported by the EPSRC under the grant EBP/T517811/1 Salijona Dyr- mishi is supported by the Luxembourg National Research Funds (FNR) AFR Grant 14585105. This work was also supported by the Alan Turing Institute under the EPSRC grant EP/N5101 29/1, by the AXA Research Fund, by the EPSRC grant EP/R013667/ and by the EU TAILOR grant. We would like thank Andrew Ryzhikov, Salah Ghamizi, and Thibault Simonetto for the useful discussions. We also thank Thibault Simonetto for his contribution to the scaler and exploration of domain con- straints. We also acknowledge the use of the EPSRC-funded Tier facility JADE (EP/PO20275/1), GPU computing support by Scan Computers International Ltd. and the HPC facilities of the Univer- sity of Luxembourg.
ETHICS STATEMENT
While our method enables the generation of synthetic data, we recognize the importance of re- sponsible and ethical use. It is conceivable that individuals could misuse approach to create high-quality fake data for deceptive purposes, such as unauthorized access or selling counterfeit information. However, synthetic data generation also provide many societal benefits, such as privacy protection as shown in (Park et al. 2018 Yoon et al. 2020)).
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this paper, we include all the necessary details in the Appendix. Appendix Aopendix/Ajjinluuee inciuuue detailed proofs for the technical statements presented in the paper. Appendix B
provides all the details on the experimental analysis settings, in particular: (i) Section B.2 provides the descriptions and links to the datasets, (ii) Section (ectiaonB.4jdettalls 4jdetails the evaluation protocol we followed, and (iii) Section B.5 reports how the hyperparameters search conducted, with Table containing the best hyperparameter configuration for each dataset and model. The code is publicly available at https://gthhub.com/mmbaeee. CooaamnCCostraieeDDGMM
T
REFERENCES
Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Se- mantic probabilistic layers for neuro-symbolic learning. In Proceedings of Neural Information Processing Systems, 2022.
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. CoRR, abs/ 1701.07875 2017.
Samy Badreddine, Artur Avila Garcez, Luciano Serafini, and Michael Spranger. networks. Artificial Intelligence, 303, 2022.
Logic tensor
Massimo Buscema. MetaNet The theory of independent judges. 1998.
Substance use & misuse. 33,
Ramiro Camino, Christian A. Hammerschmidt, and Radu State. Generating multi-categorical sam- ples with generative adversarial networks. CoRR, 0bs//887.01120 2018
Zhengping Che, Yu Cheng, Shuangfei Zhai, Zhaonan Sun, and Yan Liu. Boosting deep learning risk prediction with generative adversarial networks for electronic health records. Proceedings of IEEE International Conference Data Mining, 2017.
10

---

Published as a conference paper at ICLR 2024
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Edward Choi, Siddharth Biswal, Bradley A. Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks. Proceed- ings of the Machine Learning for Health Care Conference, 2017.
David R. Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20, 1958.
Rina Dechter. Bucket elimination: unifying framework for reasoning. Artif. Intell., 113, 1999.
Michelangelo Diligenti, Marco Gori, Marco Maggini, and Leonardo Rigutini. Bridging logic and kernel machines. Machine Learning, 2012.
Michelangelo Diligenti, Soumali Roychowdhury, and Marco Gori. Integrating prior knowledge into deep learning. In Proceedings of International Conference Machine Learning and Applica- tions, 2017.
Artur d' Avila Garcez and Luis C. Lamb. Neurosymbolic AI: The 3rd wave. Artificial Intelligence Review, 2023.
Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. CoRR, abs/1706.00233, 2017.
Julius Farkas. Theorie der einfachen ungleichungen. Journal für die reine und angewandte Mathe- matik, 124, 1902.
Kelwin Fernandes, Pedro Vinagre, Paulo Cortez, and Pedro Sernadela. Online News Popularity. UCI Machine Learning Repository, 2015.
Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. DL2: Training and querying neural networks with logic. In Proceedings of International Conference on Machine Learning, 2019.
Eleonora Giunchiglia and Thomas Lukasiewicz. Multi-label classification neural networks with hard logical constraints. Journal of Artificial Intelligence Research, 72, 2021.
Eleonora Giunchiglia, Fergus Imrie, Mihaela der Schaar, and Thomas Lukasiewicz. Machine Learning with Requirements: a Manifesto. CoRR, abs/2304.00674, 2023a.
Elconora Giunchiglia, Mihacla Cätälina Stoian, Salman Khan, Fabio Cuzzolin, and Thomas Lukasicwicz. ROAD-R: The autonomous driving dataset for learning with requirements. Ma- chine Learning Journal, 2023b.
Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In Proceedings of Advances in Intelligent Computing, International Conference on Intelligent Computing, 2005.
Abdelhakim Hannousse and Salima Yahiouche. Towards benchmark datasets for machine learning based website phishing detection: An experimental study. Engineering Applications of Artificial Intelligence, 104, 2021.
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994.
Geoffrey Hinton. Lecture notes in neural networks for machine learning, 2014.
Tin Kam Ho. Random decision forests. In Proceedings of International Conference Document Analysis and Recognition, volume 1, 1995.
Nicholas Hoernle, Rafael-Michat Karampatsis, Vaishak Belle, and Kobi Gal. MultiplexNet: To- wards fully satisfied logical constraints in neural networks. In Proceedings of Association for the Advancement of Artificial Intelligence, 2022.

---

Published as a conference paper at ICLR 2024
James Jordon, Jinsung Yoon, and Mihaela van der Schaar. PATE-GAN: generating synthetic data with differential privacy guarantees. In Proceedings of Interaational Conference on Learning Representations, 2019.
Jayoung Kim, Jinsung Jeon, Jachoon Lee, Jihyeon Hyeong, and Noseong Park. OCT-GAN: Neural ODE-based Conditional Tabular GANs. In Proceedings of the Web Conference, 2021.
Jayoung Kim, Chaejeong Lee, and Noseong Park. STaSy: Score-based Tabular data Synthesis. In Proceedings of International Conference on Learning Representations, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations, 2015.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling Tabular Data with Diffusion Models. In Proceedings of International Conference on Machine Learning, 2023.
Jaehoon Lee, Jihyeon Hyeong, Jinsung Jeon, Noseong Park, and Jihoon Cho. Invertible tabular GANs: Killing two birds with one stone for tabular data synthesis. In Proceedings of Neural Information Processing Systems, 2021.
Luca Di Liello. Pierfrancesco Ardino, Jacopo Gobbi, Paolo Morettin, Stefano Teso, and Andrea Passerini. Efficient generation of structured objects with constrained adversarial networks. In Proceedings of Neural Information Processing Systems, 2020.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN:
The power of two samples in generative adversarial networks. In Proceedings of Neural Information Processing Systems, 2018.
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela der Schaar. GOGGLE: Genera- tive modelling for tabular data by learning relational structure. In Proceedings of International Conference on Learning Representations, 2022.
Robin Manhaeve, Sehastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. DeepProbLogg Neural probabilistic logic programming. In Proceedings of Neural Information Processing Systems, 2018
Eleonora Misino, Giuseppe Marra, and Emanuele Sansone. VAEL: Bridging Variational Autoen- coders and Probabilistic Logic Programming. In Proceedings of Neural Information Processing Systems, 2022.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Young- min Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11, 2018.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2011.
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: A Probabilistic Prolog and Its Application in Link Discovery. In Proceedings of International Joint Conference Artificial Intelligence, 2007.
Luc De Raedt, Andrea Passerini, and Stefano Teso Learning constraints from examples. In Pro- ceedings of Association for the Advancement of Artificial Intelligence, 2018.
Sebastian Ruder. overview of gradient descent optimization algorithms. arXiv preprint
arXiv:1609.04747, 2016.
Robert E. Schapire. Explaining AdaBoost. In Empirical inference. Springer, 2013.
Thibault Simonetto, Salijona Dyrmishi, Salah Ghamizi, Maxime Cordy, and Yves Le Traon. A unified framework for adversarial attack and defense in constrained feature space. In Proceedings of International Joint Conference on Artificial Intelligence, 2022.
12

---

Published as a conference paper at ICLR 2024
Mihaela C. Stoian, Eleonora Giunchiglia, and Thomas Lukasiewiez. Exploiting t-norms for deep learning in autonomous driving. Proceedings of the International Workshop on Neural- Symbolic Learning and Reasoning, 2023.
Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. DECAF: gen- erating fair synthetic data using causally-aware generative networks. In Proceedings of Neural Information Processing Systems, 2021.
Emile van Krieken, Thiviyan Thanapalasingam, Jakub M. Tomezak, Frank Van Harmelen, and An- nette Ten Teije. A-neSI: A scalable approximate method for probabilistic neurosymbolic infer- ence. In Pruceedings of Neural Information Prucessing Systems, 2023.
Xindong Wu, Vipin Kumar, J. Ross Quinlan, Joydeep Ghosh, Qiang Yang, Hiroshi Motoda, Geof- frey J. McLachlan, Angus Ng, Bing Liu, S Yu Philip, et al. Top 10 algorithms in data mining. Knowledge and information systems, 14, 2008.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In Proceedings of International Conference on Learning Represen- tations, 2022.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint 1802.06880.06666 2018.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In Proceedings of International Conference on Machine Learning. 2018.
Lei Xu, Maria Skoularidou, Alfredo Infaate, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Proceedings of Neural Information Processing Systems, 2019.
Jinsung Yoon, Lydia N. Drumright, and Mihaela van der Schaar. Anonymization through data synthesis using generative adversarial networks (ADS-GAN), IEEE Journal of Biomedical and Health Informatics, 24, 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG Structure Learning with Graph Neural Networks. In Proceedings of International Conference Machine Learning, 2019.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y. Chen. CTAB-GAN: effective table data synthesizing. In Proceedings of Asian Conference on Machine Learning, 2021.
13

---

Published as a conference paper at ICLR 2024
A THEOREMS
A.1 PROOF OF THEOREM 3.3
Proof. The proof is by induction on the number n. of variables in IT. We recall that ( TDD is the set of variables and that we assumed, w.l.o.g., A(x;) =
For the base case n 0, II does not contain variables and, since II is satisfiable, for each constraint >x 2g + b in II, (i) each w, = 0, (ii) b E and (iii) each sample satisfies II.
Assume that n > variables appear in II. Let 2y be the variable with the highest A(z+) value occurring in
Then, DD, ., Fk+1 2k+11 do not occur in II, IT = IT, = II, and
11 H& (IT UII U fredk(d', ,69) 10' m.
(5)
Consider arbitrary sample R.
In I 11, by construction, less than n variables appear. Thus, for the inductive hypothesis, assume CL(@) satisfies 4 Proving that CL(@) satisfies I is equivalent to proving that CL(2) satisfies both II and n since we know from eq. (5) that II, S II,-1 II UII.".
The proof is in two steps. We first prove by contradiction that either (i) 1b, < ubx or (ii) lby = ubk at and there exist two constraints Φ1 E I1g and Φ2 II such that: ub, Ek (CL(E)) and lbg Eg (CL( and both o² and Ø² are not strict inequalities. Then, in the second step, we prove that (z) satisfies each constraint @ II Ull.
First step. Assume that either (i) lb > uby or (ii) lb, = uby and at least one between o' and o² is strict inequality. If lbg > ubg or 1b,- = ubg, then lbg * oc and ubx & too and thus both II and It are empty. Let o² (resp. x²²) be the constraint in 0I (resp. I1,)) such that ubn - x (CL(T)) (resp. lbg- Ex (CL(E))). Such constraints Φ1 and d2 exist since II is finite. Then, by definition, redu(á" 2d²) is equivalent to E c 2 0 if both Øl and ø² are non-strict inequalities, and to Ek -sh > if at least one between
Ø² and o² is a strict inequality.
We know that (od ,0) E II--1 and that, by the inductive hypothesis, CL(i) satisfies IIg-1- Thus, (T( satisfies (1)" o²²), which (taken together with the definition of redy (Ø+, (2) above) implies that e (CL($)) Ek (CL(Ø)) 2 0 if both o' and d² are non-strict inequalities, and that & (CL(T)) - 5x (CL(T) > if at least one between Ø² and s²² is a strict inequality. However, by dd
our assumption, uby dt (CL( and lbg = cx (CL( Thus, we have that uby 2 lb, if both 2² and d² are non-strict inequalities, and that ubn 1b, least one between di and d² is strict inequality, deriving contradiction.
Second step. We now prove that Cā) satisfies each constraint & E U.g. The statement holds since
1. if Φ is a non-strict inequality, by definition, we have that CL(T); 2 1b, 2 c (CL(x) (.)) and
2. if Φ is a strict inequality, we have two cases. In the first one, lbg = = (ā)) and since lb ubv, it is possible to choose an > 0 such that CL(T); lbe + ubg and thus
CL()); lb e (CL())). the second case, lb of (CL(E) and then CL())) 2
lb > (CL(
Analogously, CL( satisfies each constraint & E II.
14

---

Published as a conference paper at ICLR 2024
A.2 PROOF OF THEOREM 3.5
Let स be a sample satisfying the constraint in II. We recall that, w.l.o.g,, we assume ((x;) = i. We will prove by contradiction that if satisfies IT, then CL(()) =
Assume CL(E)) * R. Let be the lowest index such that CL()); * 5.. Then, CL(();: = min' (max' (ã;, lb;), ub;) * Cs and thus either i; < lb; or I; ub;, both cases being impossi- ble given
1. the definitions of lb, and ub;
2. the hypothesis that 2 satisfies the constraints in II, and
3. the fact that all the constraints in IT UTT are entailed by IT and thus satisfied by A.
A.3 PROOF OF THEOREM 3.6
We prove the statements of the theorem in separate lemmas, after first introductory lemma.
Lemma A.1. In the hypotheses of Theorem 3.6 CL² optimal with respect to II.
Proof. Let ã be sample. We recall that, w.l.o.g., we assume X(x;) = i.
We show that for every ..... D, there does not exist another sample स' satisfying 11 such that for each S 33J
CL 2 (ō), and 1 E.l (c): -: The proof by induction on
For the base case (i = 1), we know that in II; the only variable that can appear is x1, thus we have four cases:
1. If is either empty contains constraint 2 which is always satisfied since II (and thus IT2) ) is satisfiable. In this case, CL (L)(()) o1 and the statement trivially holds;
2. I is equivalent to single constraint {aı + 0} in which case if ã then CL 2 (x)1 ž otherwise CL - (T)1 = -a and the statement trivially holds:
3. _1 is equivalent to single constraint {- + 2 0} in which case if 5i S b then CL 2 (ø) 1 क1 otherwise CL (8)1 b and the statement trivially holds;
4. I1 equivalent a pair of constraints {a1 ta 01 + 0}. Since Is satisfiable, a - 2 0. Then, if ēj < -a then (z) -a, if - I ã S then CL² = Z1, and 2 > then CL 2 (क) = b. For each of the three cases, the statement trivially holds.
Assume i = j + 1. Assume by contradiction that there exists another sample 7' satisfying I12 such that for each 1 <
T CLL(()) (m) and [ 3. < CL² -:.- By construction of 11.. we know that only 21 ........ appear in 11,- If we substitute each variable zy with j < i with CL) in 7TF. then (i) in the resulting of constraints the only variable is wt, (ii) back the base case, and (iii) the statement follows.
Lemma A.2. In the hypotheses of Theorem 3.6 CL(E) optimal if CL(()) CL)
Proof. The proof is a direct consequence of the facts that (i) CL(3) satisfies the constraints in II. (ii) @L (z), (iii) ( optimal wrt 112 (from Lemma A.l and (iv) the samples satisfying 1I are subset of the samples satisfying 112
Lemma A.3. In the hypotheses of Theorem 3.6 CL(ā) tends to CL2 (Ø) as the e values used to
compute (l(@)) tend to O.
15

---

Published as a conference paper at ICLR 2024
Proof. Let E1, ................................ Cj. (k 2 0) be the values (assumed, w.l.o.g., to be distinct) in CL If then CI((#) CL (8), and the statement trivially holds.
Consider C i i,....D, D, and substitute 6j with a newly introduced variable 0y. CL(.); is composition of continuous functions in the newly introduced variables and, thus, it is continuous. For an arbitrary continuous function f I + we know that limz-1zo f(z) = f(r)). Thus,
lim CL(B); (T),
Mi mn. (0.....)
and hence
lim
CL (z)
[ Ub 0
B EXPERIMENTAL ANALYSIS SETTINGS
B. MODELS
In our experimental analysis, five base models:
WGAN (Arjovsky et al. 2017) a GAN model trained with Wasserstein loss in a typical generator discriminator GAN-based architecture. In implementation, WGAN uses MinMax transformer for the continuous features and one-hot encoding for categorical ones. It has not been designed specifically for tabular data.
TableGAN (Park et al., 2018) among the first GAN-based approaches proposed for tab- ular data generation. In addition the typical generator and discriminator architecture for GANs, the authors proposed adding a classifier trained to learn the relationship between the labels and the other features. The classifier ensures higher number of semantically correct produced records. TableGAN uses a MinMax transformer for the features.
CTGAN (Xu et al. 2019) uses conditional generator and training-by--aee saining-by-sampling strategy in GAN architecture to model tabular data. The conditional gener-
a generator-diseriminn ator generates synthetic rows conditioned on one of the discrete columns. The training-by- sampling ensures that the data are sampled according to the log-frequency of each category. Both help better model the imbalanced categorical columns. CTGAN transforms discrete features using one-hot encoding and a mode-based normalization for continuous features. A variational Gaussian mixture model (Camino et al. 2018) is used to estimate the num- ber of modes and a Gaussian mixture. For each continuous value, a mode sampled based on probability densities, and its mean and standard deviation are used normalize the value.
TVAE (Xu et al. 2019) was proposed variation of the standard Variational AutoEn- coder to handle tabular data. It uses the same transformations data as CTGAN and trains the mncoder-decoder architecture using evidence lower-bound (ELBO) loss.
graph-based approach learning the relational struc- ture of the data well as functional relationships (dependencies between features). The relational structure of the data is learned by building a graph where nodes are variables and edges indicate dependencies between them. The functional dependencies are learned through message passing neural network (MPNN). The generative model generates each variable considering its surrounding neighborhood.
GOGGLE (Liu et al. 2022)
B.2 DATASETS
We use 6 real-world datasets covering both classification and regression tasks. An overview of these datasets' statistics can be found in Table 5 For the selection, we focused datasets with least three feature relationship constraints that either were provided with the description of the datasets or could derive with domain expertise. The selected datasets are listed below:
16

---

Published as a conference paper at ICLR 2024
Table 5: Datasets statistics.
Dataset
# Train
# Val
# Test
# Features
# Cat.
# Cont.
Task (# classes)
URL
7K
2K
2K
64
20
44
Binary elassification
WiDS
22K
6K
7K
109
9
100
Binary classification
LCLD
494K
199K
431K
29
8
Binary classification
Heloc
8K
2K
0.2K
24
8
16
Binary classification
FSP
2K
0.2K
0.2K
28
28
Multi-class class. (7)
News
31K
7K
1K
60
14
46
Regression
URL Hannousse Yahiouche 2021) is used to perform webpage phishing detection with
features describing statistical properties of the URL itself as well as the content of the page.
WiDS is used to predict if patient is diagnosed with a particular type of diabetes named Diabetes Mellitus, using data from the first 24 hours of intensive care.
LCLD used to predict whether the debt lent is unlikely to be collected. In particular, use the feature-engineered dataset from Simonetto et al. (022), inspired from the Lend- ingClub loan data. The dataset captures features related to the loan well as client history.
FICO's Home Equity Line of Credit dataset (Helod from the FICO xML Challenge is used to predict whether customers will repay their credit lines within 2 years. Similarly to LCLD, the dataset has features related to the credit line and the client's history.
FSF Buscema 1998) used to predict 7 types of surface defects stainless steel plates.
The features approximately capture the geometric shape of the defect and its outline.
News (Fernandes et al. 2015) is used to predict the number of times news article will shared on social networks. The features capture properties of the text, as well as publishing time.
For URL, WiDS, and LCLD, we used the train-val-test splits provided by Simonetto et al. 2022)) For Heloc we used the train-test split of 80-20 and 20% of the training set was later split for validation. Finally, for FSP and News split the data into 80-10-109 sets, and for the former (which is multiclass classification dataset) we preserved the class imbalance.
B.3 CONSTRAINTS DATASHERT
Here we give an overview of the structure of our constraints. To this end, we define F to be the number of features appearing in at least one constraint, and we remind that D is the total number of features. Then, given constraint Φ, we define F (resp. Fø to be the number of features appearing positively (resp. negatively) in ó, and F6 = F + F to be the number of features appearing in 6.
As we can see from 6 the constraints characteristics greatly vary from one dataset to the other. Indeed, we can have from to 31 constraints annotated for each dataset, with as little as 15% of the variables appearing in at least constraint in News to as much as 56.88% in WiDS. Further, we can see that for almost all datasets, the average number of features appearing per constraint is 2.00, except for URL, which has constraint where as many 17 different features appear, and LCLD where we have constraints where a single variable appears.
B.4 EVALUATION PROTOCOL
For evaluating the utility of the DGM/C-DGM models presented in paper, followed closely the protocol from Kim et al. (2023) which we also reproduce here.
Link to dataset:
*Link to dataset: http://wwwwwagggg.mmmmmmmmtooosa
$ "Link to dataset: http://hhghrrcc.oom///ssasaalcog
"Link to the dataset: htnps///nngggiiffecccooddocccccs
Link to dataset: http://wwww.aaggoooommmmmmetettt
Link dataset: http:///iiiiiiieeeeeeedddddeeeee
17

---

Published as a conference paper at ICLR 2024
Table 6: Constraints statistics.
Dataset
# Constr.
FID D
Avg. P
Avg. F
Avg. F'
URL
8
24 / 64
4.25
1.00
3.25
WiDS
31
62 / 109
2.00
1.00
1.00
LCLD
4
5/
1.50
0.75
0.75
Heloc
10/24
2.00
1.00
1.00
FSP
4
7/28
2.00
1.00
1.00
News
5
9/60
2.00
1.00
1.00
First, we generate synthetic dataset, split into training, validation and test partitions using the same proportions as the real dataset.
2. Then, perform hyperparameter search using the synthetic training data partition to train different classificts/regressooss.
For the binary classification datasets (i.e.. URL, WiDS, LCLD, and Heloc) we use: Decision Tree (Wu et al. 2008). AdaBoost (Schapire 2013), Multi-layer Perceptron (MLP) (Haykin 1994). Random Forest (Ho 1995). XGBoost (Chen & Guestrin 2016), and Logistic Re- gression (Cox 1958) classifiers. For multi-class classification datasets, (i.e., FSP) we use Decision Tree, MLP, Random Forest, and XGBoost classifiers. For the regression dataset, (i.e., News) we use MLP, XGBoost, and Random Forest regressors and linear regression. For all the classifiers and regressors above, considered the same hyperparameter set- tings as those from Table 26 of (Kim et al. 2023) and picked the best hyperparameter configuration using the real validation according to the Fl-score.
3. Finally, we tested the selected best models the real test and averaged the results across all the classifiersrregressors to get performance measurements for the DGM/C-DGM predictions according three different metries: F1-score, weighted F1-score, and Area Under the ROC Curve.
The above procedure was repeated times for each DGM/C-DGM model, and the results were averaged separately for each of the metrics.
For evaluating the DGM/C-DGM models in terms detection, we slightly adapted the procedure presented by Kim al. (2023). We first created the training, validation, and test sets by concatenating real and synthetic data, including their targets as usual features, and adding new target column that specifies whether the data is real not. By construction, these datasets are binary classification datasets and, thus, are suitable for the hyperparameter search procedure presented in Kim et al. (2023) using the 6 different binary classifiers mentioned above. We proceeded to pick the best model using the cewly-created validation set, and then we obtained the final detection performance on the newly-created test data (which combines the real and the synthetic data).
B.5 HYPERPARAMETER SEARCH
Prior to experimenting with our C-DGM methods, we conducted extensive hyperparameter search to reveal the best configurations. We always chose the best settings according to the utility perfor- mance measured either by the average over the Fl-score, weighted Fl-score, and Area Under the ROC Curve for the binary and multi-class classification datasets by the Mean Absolute Error for the regression dataset.
Initial phase. For GOGGLE, used the same optimiser and learning rate set Liu et al. 2022 Specifically, we used Adam (Kingma & Ba 2015) with set 5 different learning rates: {1 used Adam with set of 5 different learning rates:
10-3 5 10 10-3 1, 10-2). For TVAE, (5 x 10-6, x 1 1 x 4 2 x 10-4, x 10-3), And for each of the other DGM models, we used three different optimizers, Adam, RMSProp (Hinton 2014), SGD (Ruder) 2016), each with different set of learning rates:
for WGAN (1 (1x10-T 4,1010 -300033,,000 (5x10 11x10 11x10-3 3), and {1x10 10-1 1, 3), respectively.
18

---

the Published as a conference paper at ICLR 2024
10 for TableGAN, {5 x 10 10x100111xx00 1001xx00+.22 1 1001 111x000 1111000111111111111111000111 10
and {1 x 10 x 10 10-3), respetivvely
for CTGAN, {5 10-5, 1 10 ,2 10- {1 1,2 10 x 10 and {1 x 10-1,1 x 10 10-3), respectively.
Then, for each of the above optinizer-learnnng rate pairs, we tested three different batch sizes, depending on the DGM model: {64, 128, 256} for WGAN, {128, 256, 512] for TableGAN, 280, 500} for CTGAN and TVAE, and (64, 128] for GOGGLE. The batch sizes for CTGAN are multiples of 10, to allow for using the CTGAN's recommended PAC (Lin al. 2018) value of 10, among other values.
Further search. The initial phase of hyperparameter search allowed us to narrow down the space of configurations and focus on further investigating the most promising one. For the second phase, we varied the following parameters for each DGM, keeping fixed the rest from the best model of the initial phase:
for WGAN varied initially PAC values 4, 8, and 16 and then discriminator iterations to 1,2, and 10.
for TableGAN we varied separately i) generator layers dimensions 128 from 100 initially
and ii) embedding dimensions by doubling the value.
for CTGAN we varied the value of PAC within {1,5, 15). for TVAE we varied the multiplier the loss within (1,2,3,4)
The best hyperparameters settings are presented in Table 7 We used these configurations for the experiments presented in our paper. The same hyperparameters were then used for C-DGMs. Fur- thermore, for C-DGMs we reported the variable ordering hyperparameter that needs to be given to our CL. We give full description of the orderings and of the impact they have on the performance in Appendix C.3
C RESULTS
C.I BACKGROUND KNOWLEDGE ALIGNMENT
Besides constraints violation rate (CVR) presented the main text, we measure two additional metrics: (i) constraints violation coverage (CVC), which, given set of samples S and II, represents the percentage constraints in II that have been violated at least once by any of the samples in S, and (ii) samplewise constraints violation coverage (sCVC), which represents the average samples in S of the percentage of the constraints violated by each sample.
Table shows that for 5 out of 6 datasets, all the samples generated by unconstrained DGMs violate at least 50% of the constraints, with CVC reaching up to 100% for WiDS, FSPand News. This entails that the models actually struggle with the majority of the constraints specified, and that the problem cannot be solved by just fixing a very small subset of the available constraints. Mcanwhile, all our C-DGMs guarantee that not a single constraint violated by any of the samples.
Regarding SCVC, the results in Table g show that for all models and datasets, on average each sample can violate up to 29.8% of the constraints. As reminder, even a single constraint violated indicates unfeasible example in real setting. The unconstrained DGM models learn different representations that produce different rankings for sCVC. For example, WGAN violates on average the most constraints per samples in LCLD. But, for TableGAN, CTGAN, TVAE and GOGGLE, LCLD is among the datasets with the lowest SCVC. Again, all our C-DGMs ensure that sCVC 0.0, meaning not single constraint is violated for all the samples.
We complement our quantitative analysis of background knowledge alignment for the generated synthetic samples with some visualizations. In particular, consider three different constraints where only two variables appear, and then for each one of them, we create four woo-iimensional scatter plots with the variables appearing in the constraint the axes. The first scatter plot represents the real data, while the other three represent the samples generated by each of the DGMs and the C-DGMs. We now consider them one by one.
19

---

Published as a conference paper at ICLR 2024
Table Best hyperparameter settings used for DGMs/C-DGMs in our experiments.
Model/Dataset
Hyperparameter
URI.
WiDS
LCL.D
Heloc
FSP
News
Batch size
64
128
128
64
128
128
Optimiser
Adam
RMSProp
SGD
Adam
RMSProp
RMSProp
Learning rate
0.0001
0.001
0.001
0.0001
0.001
0.001
WGAN
Epochs
300
80
30
200
50
Discriminator iters
10
10
5
10
10
10
Ordering
Rnd
KDE
KDE
KDE
KDE
KDE
Batch size
128
128
128
128
128
128
Optimiser
Adam
RMSProp
Adam
Adam
Adam
RMSProp
TableGAN
Learning rate
0.001
0,001
0.001
0.001
0.001
0.001
Epochs Ordering
300
50
200
200
50
Corr
Corr
KDE
Corr
KDE
KDE
Batch size
500
500
500
Optimiser
Adam
RMSProp
Adam
Adam
Adam
Adam
CTGAN
Learning rate
0.0002
0.0005
0.0002
0.0002
0.0002
0,0002
Epochs Ordering
150
50
20
500
300
100
KDE
Corr
Rnd
Corr
Corr
Corr
Batch size
70
70
500
500
70
70
Optimiser
Adam
Adam
Adam
Adam
Adam
Adam
TVAE
Learning rate
0.0002
0.000005
0.00001
0.000005
0.00001
0.00001
Epochs
150
50
40
150
500
50
Loss factor Ordering
2
2
4
2
1
3
KDE
KDE
Corr
Corr
Cor
Corr
Batch size Optimiser
128
128
128
64
128
64
Adam
Adam
Adam
Adam
Adam
Adam
GOGGLE
Learning rate
0.005
0.01
0.001
0.001
0.005
0.001
Epochs
1000
200
200
1000
1000
250
Patience
50
50
50
50
50
50
Ordering
Random
Corr
Corr
Random
Corr
KDE
Table 8: Constraints violation coverage (CVC) for datasets and models under study.
Model/Dataset
URL
WIDS
LCLD
HELÓC
Faults
News
WGAN
11.1 1.6
100.0 0.0
75.0 0.0
100.0 0 0
75.0 + 0.0
84.8 8.7
TableGAN
100.0 3.7
5.00
100.0 0
7.00
100.0 0.0
CTGAN
16.5 3.8
100.0 0.0
99.4
75.0
100.0 1 0.0
TVAE
12.5 0.0
100 +:: 0.0
5.0
100.0 - 0.0
7.00
100.0 +: 0.0
GOGGLE
17.5 + 6.9
99.2 1.7
50.0
1.0
75.0
+.000
All C-models
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
0.0 = 0.0
0.0 = 0.0
0.0 + 0.0
1. In Figure we consider again the constraint MaxHHemoglobinLevel MinHHemoglobinLeeell > 0 appearing in WIDS dataset from Section 4.2 in the main text. We visualize the outputs for the remaining models WGAN, CTGAN, TVAE and GOGGLE and their constrained counterparts C-WGAN, C-CTGAN, C-TVAE and C-GOGGLE.
2. In Figure 5 we illustrate the behavior of the real and synthetic samples with respect to constraint from the Heloc dataset which requires that the number of trades that have been insolvent for 60 days must be greater than the number of trades that have been insolvent for 90 days, i.e., haap:/oooeee//aadeeeeeeee/aaaaaa
3. In Figure 6
we illustrate the behavior of the real and synthetic samples with respect to straint from the FSP dataset that requires X Maximum 2 minimum, where X minimum and X maximum refer to the value for the X coordinate in images captured of steel plates.
20

---

Published as a conference paper at ICLR 2024
Table Samplewise constraints violation coverage (sCVC) for all models and datasets under study.
Model/Dataset
URL
WIDS
L.CLD
HELOO
Faults
News
WGAN
1.4 0.2
21.2 +
29.8 + 0.2
10.5 + 2.5
23.3 + 4.3
12.1
+ 2,4
TableGAN
0.6 0.2
1
1.2
9.0
22.9 1 3.2
24.1
+ 3.3
CTGAN
0.4
21.6 0.5
3.0 0.7
7.6.0.3.
24.4
15.8 a 3.9
TVAE
0.1
21.0 0.2
1.1
9.2
21.5 1.5
14.3 = 1.1
GOGGLE
0,8
10.6 2.4
3.3 3.7 0.7
17.2
18.8 + 5.3
10.7 = 1.6
All C-models
0.0 0.0
0.0 + 0.0
0.0 = 0.0
0.0 + 0.0
0.0 0.0
0.0 = 0.0
The Figures visually demonstrate that the models that incorporate our constrained layer (C-WGAN, C-CTGAN, C-TableGAN, C-TVAE and C-GOGGLE) are guaranteed to produce outputs only within the feasible regions of the real data.
Finally, to further investigate the properties the generated data by C-DGMs, we study the impact of constraint reparation the boundary population for the three cases considered above. We defined a band around the boundary whose width w we set to be proportional to the range of the values the considered features in the real dataset. As in all the considered constraints only two features appear, set / (rip))²+ (rap)?, where T1 (resp. r2) represents the range of the first (resp. second) feature, while p represents the proportion of the range values each feature can assume. If p = 1, then the width is equal the diagonal of the rectangle defined by the two points with minimum and maximum coordinates.
The results in Table T0 show the percentage of generated samples that lay the boundary when (1%, 5%, 10%} for the real dataset (last row), individual DGMs and C-DGMs, as well as their averages (third and second to last row, respectively). Notice that C-DGMs populate the boundary at much closer rate to the real data than their unconstrained counterparts. Indeed the maximum difference between the percentage of real data points laying the boundary and the average number of samples generated by C-DGMs is equal to 14.7% (for dataset WiDS and p 1%). On the contrary, the maximum difference between the number of real data points laying at the boundary and the average number of samples generated by DGMs is equal to 51.5% (for dataset FSP and p = 1%)
C.2 FEATURE DISTRIBUTION ANALYSIS
Following Kotelnikov et al. 2023 and Zhao et al. (2021) we perform comparative analysis of the distributions of data generated by the models under study and the real data. The results are presented in Table 11 []]frr for continuous features, for which we measure the difference between the real data and generated data distributions using the Wasserstein distance, and in Table 12 for categorical features, for which measure the difference between the real data and generated data distributions using the divergence.
As we can see from Table 11 for every model and dataset there is very little difference in the Wasserstein distance obtained with the standard DGMs, the P-DGMs, and the C-DGMs. The only big gap is registered for GOGGLE with C-GOGGLE equal to 0.05 while the one obtained with GOGGLE and P-GOGGLE is equal to 0.22. Regarding the results for the categorical features in Table we can see that the differences between DGMs and C-DGMs are very small, while, as expected, there is no difference hetween the results obtained with the DGMs and the P-DGMs. Indeed, our datasets are annotated with no constraints over the categorical features.
the WiDS dataset, where the Wasserstein distance obtained
C.3 VARIABLE ORDERING STUDY
In this subsection, first discuss how picked the variable orderings tested in experimental analysis, and then we conduct an in-detail experimental analysis how the different orderings perform with each model.
21

---

Published as a conference paper at ICLR 2024
18
18
Max hemoglobin
16
Max hemoglobin
14
12
12
10
8
WGAN
C-WGAN
6
10
Min hemoglobin
Min hemoglobin
18
Max hemoglobin
16
Max hemoglobin
16
14
12
12
10
10
CTGAN
C-CTGAN
6
8
10
12
14
16
6
8
10
16
Min hemoglobin
Min hemoglobin
18
Max hemoglobin
16
Max hemoglobbi
16
14
14
12
10
10
8
TVAE
C-TVAE
8
12 Min hemoglobin
10
14
16
6
8
12 Min hemoglobin
10
14
16
18
18
Max hemoglobin
16
Max hemoglobin
14
14
12
12
10
8
GOGGLE
C-GOGGLE
10
Min hemoglobin
Min hemoglobin
Figure 4: Samples generated by WGAN, CTGAN and TVAE, GOGGLE and their constrained ver- sions for WiDS. The real and TableGAN distributions are given in Section 4.2 in the main text.
easy to see that given sample ž, the value of CL(T) may depend on the selected variable ordering, i.e., that different variable orderings may lead to different CL(T). Ideally, we would like the orderings to help in generating the highest quality samples possible. Thus, our intuition has been that features whose distributions that their value can be fruitfully used to compute the value associated with features with higher ranking. With such intuition, designed two different heuristics to choose such orderings. To
easier learn should be assigned lower ranking, so
22

---

Published as a conference paper at ICLR 2024
Insolvent 60 days
10
Insolvent >= 60 days
10
6
WGAN C-WGAN
2
Real
2 Insolvent days
10
Insolvent 90 days
Insolvent >= 60 days
10
Insolvent >= 60 days
8
8
4
TableGAN
CTGAN
2
C- C-baleGAAN
C-CTGAN
2
4
8
10
Insolvent >= 90 days
Insolvent >= 90 days
Insolvent >= 60 days
10
Insolvent >= days
10
8
6
TVAE
GOGGLE C-GOGGLE
C-TVAE
2
4
6
10
2
8
10
Insolvent >= 90 days
Insolvent >= 90 days
Figure 5: Real data and samples generated by WGAN, TableGAN, CTGAN, TVAE, GOGGLE and their constrained versions for Heloc.
obtain the two orderings described below, for each C-DGM, we first need to train respective standard unconstrained DGM, and then generate set of samples S.
The first ordering we propose dataset D and S, for each variable I; compute the score a; |Ejz cor CP) Ejt: cor((C, C)),, where CP (resp. CS) is the vector the values of the jihh column in D (resp. S). The lower the value of a; is, the lower the position I; gets in the variable ordering, and the sooner its value is computed (as its relations with the other features should be easier to compute for the DGM).
correlation-based (Corr.) ordering. Given the initial real data
The second ordering propose is kernel density estimation (KDE)-based ordering, which can be obtained by following the protocol outlined below:
1. Using seikit-learn's PPddreoosa et al. 2011) implementation of KDE, we estimated probability density function of the multidimensional real data.
2. We then computed the log-likelihood of each sample belonging to D and S under the estimated model, using again scikit-learn. is worth noting that seikit-learn returns proba- bility values normalised over the sample spaces under evaluation. Using these, we treated
23

---

Published as a conference paper at ICLR 2024
1500
1500
X. Maximum
X Maximum
1000
1000
500
500
WGAN C-WGAN
Real
1000
1500
1000
1500
X Minimum
Minimum
1500
1500
X Maximum
X Maximum
1000
1000
500
TableGAN
500
CTGAN C-CTGAN
C- TableGAN
500
1000
1500
500
1000
1500
X Minimum
X Minimum
1500
1500
Maximum
X Maximum
1000
1000
500
TVAE
Goggle C-Goggle
C-TVAE
500
1000
1500
500
1000
1500
X Minimum
X
Minimum
Figure 6: Real data and samples generated by WGAN, TableGAN, CTGAN, TVAE, GOGGLE and their constrained versions for FSP.
the problem in discrete setting and approximated the marginal probability mass function each variable. More specifically, for each feature 2;, we determined its unique values from D U and for each unique value tly summed the KDE scores of the samples for which x; = Uij. separately for and S. In case no sample was observed with I; = Wij- the value of the marginal probability mass function of x;; to 0.
3. Hence, obtain two marginal probability mass functions for each feature x, (one for the real data D and one for the generated samples S) and compute the Kullback-Leiller divergence (KL) between them to get single value d,
4. Finally, we rank the features based on their KL scores d; in ascending order.
One disadvantage of this method is that the marginals are computed work, interesting direction would be use higher-fideling approximation of the marginal distributions, based on which the KL. divergence scores, and hence the variable ranks in the ordering, computed.
discrete setting. As future
To assess the impact of using different orderings for each C-DGM model, we compared the two orderings proposed above with random ordering (Rnd). Table 13 summarises the average utility
24

---

Published as a conference paper at ICLR 2024
Table 10: Percentage of the generated samples that lic on the boundary.
WIDS
Heloc
FSP
p 1% p= 5% p= = 10% = 1%
5%
p= 10%
p 1%
p 5% = 10%
WGAN
28.51:.4
45.2
67.010.5
22.510..
83.9
99.4.10.1
14.8
15.611.7 81.2
C-WGAN
62.110.3
72.110.2
83.110.2
100.010.0
100.010.0
100.010.0
76.8.1.0
83.110.8 89.910.5
TableGAN
19.710.1 71.910.2
88.7.1.11
37.410.2
94.710011
99.8100.0
25.311.2
76.010.9 95.810.5
C-TableGAN
72.2+0.2
81.1+0.3
88.7+0.3
83.8+0.3
96.1=0.2
99.1+0.1
75.4=0.5
86.2=0.5
94.3+0.3
CTGAN
6.5
30.5+0.3
53.240.2
80.2+0.4
93.1
96.8+0.3
2.6+0.3
16,420.4
36.7+1..
C-CTGAN
54.2+0.6
62.440.6
73.8+0.5
83.2+0.4
96.0=0.1
98.340.1
49.2=0.8
62.3+1.1 72.9+1.4
TVAE
20.2+0.1
42.0+0.2
62.44..3
69.44..5
90.1+0.4
96.5+u.2
6.5
1..6#1.6 56.7.11.8
C-TVAE
31.0+0.4
56.940.5
69.74.u.3
79.2+0.7
92.7=0.4
97.140.2
38.4=1.2
57.5#0.4 77.4+0.9
GOGGLE
0.9
29.1
33.1+...3
46.340.4
99.620.1
100.0:0.00
35.3=0.0
82.2+0.0 98.910.0
G-GOGGLE
7.7
77.2+0.2
99.1+0.1
83.9
92.4=0.3
97.040.1
81.1=0.0
85.4 89.7+0.0
DGMs
15.2
43.810.3
60.910.33
51.110.4
92.3
98.510.11
16.9
53.5 73.910.9
C-DGMs
45.4
69.910.4
82.910.3
86.010.3
95.410.2
98.3100.1
64.2.10.7
14.9 84.810.6
Real
60.1
73.310.0 85.1 85.710.0
96.510.0
98.911000
68.410.0 82.810.0 18.9
Table 11: Wasserstein distance between numerical features.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.00
0.04-10.0
0.02+10.00
0.03
0.02:10.00 0.04110.00
P-WGAN
0.02=0.00
0,04:10.00
0.02+t.000
0.03
0.07=1.00 000
0,02+t0.00
C-WGAN
0.03#0000
0.05=0000
0.02+10.00
0.03
0.07:10.00
0.02+t0.00
TableGAN
0.01
0.02:10.00
0.01100.00
0.01
0.02:10.00
0,02::0.00
P-TableGAN
0.02+0..00
0.02:10.00
0.02:10.00
0.01 10.00
0.03 00
0.02:10.00
C-TableGAN
0.01
0.02:10.00
0.0110..00
0.02:10.00
0.02:10.00 000
0.02:10.00
CTGAN
0..00
0.000
0.03
0.02.10.00
0.06.10.01
1.000
P-CTGAN
0.02 00
0.05=0.00
0.03+0.00
0.02+0.00
0.08 00
0.01
C-CTGAN
0.01
0.04=0.00
0.03+0.00
0.02
0.06# 01
0,02++0.00
TVAE
0.01
0.01
0.02+0.00
0.01
0.02: 00
0.01
P-TVAE
0.02=0000
0.02=0.00
0.02+0000
0.02+0000
0.04 00
0.01+00000
C-TVAR
0.01
0.02-0.00
0.01
0.0110.00
0.03
0.01
GOGGLE
0.03
0.22+0.10
0.04
0.05+10.01
0.12+0.01
0.08++0.01
P-GOGGLE
0.03
0.22+0.10
0.04
0.05:10.01
0.12:0.02
0.08+00001
C-GOGGLE
0.04
0.05+0.01
0.08+0002
0.07+0.02
0.14+0.02
0.1340.03
performance over the binary and multiclass classification datasets (i.e., all except the News) and the average detection performance over all datasets.
For detection, we can see that the KDE-based ordering performs best for C-WGAN, C-TableGAN and C-CTGAN - according to all three metrics (i.e., Fl-score, weighted Fl-score, and Area Under the ROC Curve) for the first two, and according to two out of three metrics (i.e. weighted Fl- score and Area Under the ROC Curve) for C-CTGAN. On the other hand. the correlation-based ordering yields better results for C-TVAE across all three metries. Only C-GOGGLE gets best de- tection results across all three metrics using the random ordering. For utility, we see again that different C-DGM models work best with different orderings. For instance, the KDE-bbased ordering performs best w.r.t. all three utility metrics for C-WGAN, C-TVAE and C-CTGAN. However, for C-TableGAN and C-GOGGLE, the correlation-base ordering outperforms the other orderings.
Overall, the utility and detection results we obtained using KDE-based or correlation-based order- ings as opposed to random ordering highlight the importance of choosing an ordering that takes into account the data distribution and/or the feature relations captured by the C-DGM models in their predictions. Additionally, we can see that a trend emerges for C- WGAN, C-TableGAN, and C-TVAE: each of these models has clear preference for the ordering that gives the highest overall
25

---

Published as a conference paper at ICLR 2024
Table 12: Sensen-Shannor divergence between categorical features.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.76::0.0
0.79=00.00
0.60+1..00
0.58::..01
0.33+0.001
0.77=10.00
P-WGAN
0.76::0.00
0.79=0000
0.60:1.00
0.5811..01
0.33110.01
0.77:00.00
C-WGAN
0.77+0.00
0.80=0.01
0.60+0.00
0.58+0.00
0.33+0.01
0.77=0.00
TableGAN
0.78+0.00
0.80=0.00
0.69
0.62:00002
0.35+0.01
0.77=0.00
P-TableGAN
0.78+0.00
0.81=0.00
0.6940.01
0.62:
0.35+
0.77=0.00
C-TableGAN
0.78+0.00
0.80=0.00
0.60#a.00
0,59:0.00
0.37=00.02
0.77=0.00
CTGAN
0.76+0.00
0.75=0.00
0.4940001
0.56+
0.33+00.00
0.77=0.00
P-CTGAN
0.76:10.00
0.75=0000
0.49
0.56:10.00
0.33
0.77
C-CTGAN
0.76+00000
0.75=0.00
0.49
0.56:+0.01
0.33+00.00
0.77=0.00
TVAE
0.77+0000
0.79=0.00
0.50
0.57
0.33:00.00
0.77=0000
P-TVAE
0.77+00.00
0.79=0.00
0.50
0.57=00.0
0.33+00.00
0.77=00.00
C-TVAE
0.7710000
0.80=0.00
0.50:10.00
0.57:10.01
0.33
0.77=00.00
GOGGLE
0.002
0.76
0.49.10.01
0.58.10.01
0.370
0.76
P-GOGGLE
0.71100.02
0.76_00.03
0.49
0.58.10.01
0.371.0.03
0.76
C-GOGGLE
0.7110001
0.82
0.4910.01
0.57.10.01
0.39
0.7610.001
Table 13: Variable orderings comparison for C-DGMs. The best results are in bold.
Utilliy (1)
Detection (L)
wFL
AUC
wFL
AUC
Rnd
0.453
0.477
0.735
0.936
0.933
0.950
C-WGAN
Corr
0.475
0.497
0.746
0.919
0.915
0.936
KDE
0.485
0.504
0.748
0.917
0.915
0,935
Rnd
0.346
0.409
0.710
0,908
0.904
0.926
C-TableGAN
Cor
0.361
0.422
0.717
0.897
0.893
0,916
KDE
0.349
0.413
0.706
0.895
0.893
0.916
Rnd
0.509
0.530
0.770
0.894
0.896
0.924
C-CTGAN
Corr
0.513
0,536
0.773
0.887
0.891
0.920
KDE
0.516
0.537
0.773
0.888
0.889
0.919
Rnd
0.487
0.522
0.766
0,873
0.870
0.901
C-TVAE
Corr
0.504
0.534
0.771
0,868
0.868
0.898
KDE
0,504
0.536
0.774
0.875
0.875
0.905
Rnd 0.384
0.406
0.653
0.922
0.916
0.937
C-GOGGLE
Cor 0.409
0.424
0.661
0.929
0.918
0.940
KDE
0.393
0.416
0.661
0.928
0.926
0.941
improvements w.r.t. utility and detection. is also worth noticing that it not always the case that an ordering improves both utility and detection, as we have seen for C-TableGAN and C-TVAE.
We also show effect of using different orderings on P-DGM models, Table 14 to the trends noticed for our C-DGMs, here find that it is harder to establish clear patterns in the preference of the P-DGMs towards any of the orderings. This mainly due to the very small differences between the results, as it can be seen from the Table.
As opposed
Exploring other variable orderings. In our work, we explored two variable orderings and con- ducted extensive experiments with them. However, there are various other ways to define these orderings. One advantage of customizing the order is that it can be tailored to the user's needs. For example, if users require orderings that consider how closely the generated data distribution matches the real data distribution, there are numerous ways achieve this. In our detailed analysis, exam- ined KDE-base ordering that compares the joint distributions of the features. However, a simpler alternative method would be the Wasserstein distance, metric discussed in Section
26

---

Published as a conference paper at ICLR 2024
Table 14: Variable orderings comparison for P-DGM. Best results are in bold.
Utility (1)
Detection ())
wF1
AUC
F1
wF1
AUC
Rnd
0.456
0,484
0.731
0.930
0.928
0.945
P-WGAN
Corr
0.462
0.489
0.732
0.931
0,930
0,947
KDE
0.463
0.489
0.731
0.933
0.931
0.948
Rnd
0.328
0.398
0.705
0.902
0,901
0,925
P-TableGAN
Corr
0.328
0.399
0.703
0.900
0.900
0.922
KDE
0.326
0.398
0.706
0.901
0.901
0.924
Rnd
0,507
0.524
0.769
0.898
0.901
0.926
P-CTGAN
Corr
0.508
0.527
0.770
0.899
0.902
0.928
KDE
0.507
0.524
0.769
0.899
0.903
0.927
Rnd
0,490
0,521
0.764
0.879
0.879
0.905
P-TVAE
Con
0.493
0.523
0.763
0.876
0.879
0.904
KDB
0.494
0.524
0.767
0.876
0.875
0.901
Rnd
0.347
0.373
0.626
0.925
0.925
0.943
P-GOGGLE
Corr
0.348
0.375
0.625
0.929
0,926
0.945
KDE
0,347
0.374
0.626
0.929
0.927
0.944
which differs from the KDE approach in that compares individual distributions of the features rather than joint ones. In yet another scenario, users could define orderings without relying the outputs of the unconstrained model by considering relations between the features in the real data, which is different approach to all the previously-mentioned orderings. For instance, one way to do this is to order the variables based on the cause-effect relations hetween them, which we refer to as causal-based ordering. More precisely, in such an ordering, there is no instance of cause-effect pair of features (I,, xz) in which zj appears before in the ordering.
To assess the performance of our layer when constructed using different orderings, conducted experiments with Wasserstein- and causal- based orderings on the WIDS dataset. For the Wasserstein- based ordering, for each feature we calculated the Wasserstein distance between the real data distri- bution and the generated data distribution obtained from an unconstrained DGM. We then arranged the features in ascending order based on the calculated distances, such that the features with gen- erated data distributions furthest from the real data distributions would be corrected last by our constraint layer.
We note that the Wasserstein distance can only be defined on metric space, making it impractical for use with categorical features without additional modifications (i.e. is possible define distance metrics for each of the categorical features and then apply Wasserstein distance these features). However, here we assume that the constraint layer can only be applied on continuous features and, thus, the position of the categorical features in the orderings will not impact the final results, as our constraints exclude such features. As an alternative, we investigated using the Jensen-Shannon divergence derive another ordering that compares individual feature distributions. And while this method offers has the advantage that it can be applied on both continuous and categorical features, found that unstable in results, aligning with the observations made by Zhao al. (22TT)
To compute the causal-based ordering, we obtained a directed acyclic graph (DAG), capturing the cause-effect relations between the features, and topologically sorted the features. Since none of the datasets we experimented with provided any information on causal relations between features, we utilized DAG-GNN (Yu et al., (019)) to obtain such relations the training partition of each dataset. DAG-GNN is gradient-based causal discovery method employing graph neural networks to learn DAG structure which encodes cause-effect relations. Importantly for our application, we require that the graph has topological ordering, implying the graph should be DAG. However, determining a DAG structure poses a challenge in the causality domain. In fact, the DAG-GNN method does not guarantee that its output structure is DAG, as the DAG constraint is embedded in loss term minimised during training. our experiments, we observed that DAG-GNN produced a few self- loops (edges connecting node itself) for most datasets and cycle for half of the datasets.
27

---

Published as a conference paper at ICLR 2024
Table 15: Comparing DGMs with their C-DGM versions using 5 different orderings, when trained on the WiDS dataset. Best results are in bold.
Utility (1)
Detection (L)
F1
wF1
AUC
F1
wF1
AUC
Rnd
0.303
0.360
0.797
0.995
0.996
0.999
Cor
0.284
0.343
0.796
0.975
0.976
0.989
C. C-WGAN
KDE
0.316
0.372
0.815
0.975
0.975
0,989
WD
0.273
0.331
0.770
0.932
0.925
0,948
Caus
0.275
0.334
0.775
0.944
0.929
0.956
Rnd
0.213
0.279
0.777
0.984
0.984
0.996
Corr
0.246
0.309
0.775
0.956
0.957
0.974
C-TableGAN
KDE
0.208
0.274
0.767
0.962
0.963
0.979
WD
0.242
0.305
0.770
0.961
0.962
0.977
Caus
0,225
0.289
0.770
0.962
0.963
0.979
Rnd
0.364
0.408
0.836
0.990
0.990
0.997
Corr
0.365
0.409
0.826
0.988
0.988
0.995
C-CTGAN
KDE
0,360
0.403
0.832
0,986
0.986
0,994
WD
0.368
0.411
0.842
0.953
0.955
0.970
Caus
0.365
0.409
0.838
0.954
0.956
0.970
Rnd
0.248
0.311
0.773
0.965
0.965
0.982
Corr
0.305
0.363
0.804
0.959
0.960
0.977
C-TVAE
KDE
0.321
0.378
0.816
0.961
0.962
0.979
WD
0.297
0.356
0.794
0.958
0.959
0.976
Caus
0,316
0.373
0.808
0,958
0.959
0.976
Rnd
0.139
0.210
0.643
0.965
0.965
0,979
Corr
0.185
0.253
0.675
0,972
0.971
0.984
C-GOGGLE
KDE
0.171
0.239
0.678
0.975
0.975
0.984
WD
0.207
0.273
0.705
0.980
0.980
0.990
Caus
0.175
0.244
0.681
0.966
0.965
0.979
To address these issues, we eliminated the self-loops and broke the cycles by randomly selecting an edge from each cycle and removing it from the graph. This approach enabled us to obtain DAG structure and, consequently, a topological ordering that we eventually used as the causal-bass based ordering in experiments.
For our experiments we considered the medium-sizee datasets and selected WiDS. Our choice was mainly guided by the levels of difficulty involved when manually checking the causality relations found by the DAG-GNN model. Compared to the other datasets, WiDS' columns had a clearer and more explanatory description, which allowed us determine whether the cause-effect relations between the columns were sensible. In Tables 15 and 16 we compared the different orderings on the WiDS dataset for all C-DGMs and P-DGMs, respectively. As can in Table [.]] both the causal- and Wasserstein distance- based orderings achieved better results than the random ordering in 12 (and 11) out of 15 cases for detection (and utility). On the other hand, we notice that when used during postprocessing, these two new orderings can only bring small performance improvements. However, similarly the other orderings we explored, it is harder distinguish trends where the new orderings might be more helpful than the random ordering.
Future work on variable orderings. As we can see, using custom-made orderings can improve the performance of C-DGMs which use random ordering of the variables w.r.t. both utility and detection. However, it is not straightforward to determine which ordering works best in which sce- nario. To leverage the constraint layer's capabilities to the maximum, for future work plan to investigate when different feature orderings might work best and uncover patterns in the models' and datasets' preferences towards certain orderings.
28

---

Published as a conference paper at ICLR 2024
Table 16: Comparing DGMs with their P-DGM versions using 5 different orderings, when trained on the WIDS dataset. Best results are in bold.
Utility (1)
Detection (4)
F1
wF1
AUC
F1
wF1
AUC
Rnd
0.319
0.372
0.777
0.979
0.980
0.992
Cor
0.332
0.384
0.781
0.976
0.977
0.991
P-WGAN
KDE
0.334
0.386
0.783
0.978
0.978
0.991
WD
0.332
0.384
0.776
0.979
0.979
0.992
Caus
0.332
0.383
0.774
0.980
0.981
0.992
Rnd
0.173
0.242
0.749
0.962
0.963
0,980
Corr
0.174
0.242
0.731
0.963
0.964
0.981
P-TableGAN
KDE
0.171
0.240
0.735
0.962
0.963
0.980
WD
0.181
0.250
0.738
0.962
0.963
0.979
Caus
0.171
0,240
0.741
0.963
0.964
0,980
Rnd
0.364
0.407
0.837
0.991
0.991
0.997
Corr
0.365
0.407
0.835
0.989
0.989
0.997
P-CTGAN
KDE
0.357
0.400
0.835
0.991
0.991
0.997
WD
0.375
0.419
0.836
0.990
0.990
0.997
Caus
0.363
0.406
0.835
0.990
0.990
0.997
Rnd
0.266
0.327
0.785
0.962
0.963
0.978
Corr
0.282
0.342
0.787
0.962
0.962
0.979
P-TVAE
KDE
0.285
0.345
0.797
0.963
0.964
0.979
WD
0.280
0,340
0.778
0.962
0,962
0.979
Caus
0,269
0.329
0.793
0,962
0.962
0.979
Rnd
0.192
0.202
0.665
0.988
0.988
0.993
Com
0.191
0,201
0.667
0.988
0,988
0,993
P-GOGGLE
KDE
0.189
0.197
0.667
0.987
0.987
0.993
WD 0.191
0.200
0.663
0.988
0.988
0.993
Caus
0.191
0.200
0.669
0.988
0.988
0.994
C.4 FULL RESULTS ON DGMs vs. C-DGMS
To assess the impact of constraining DGMs with our method, we conducted an extensive exper- imental analysis where we compared the performance of our C-DGM models with the baseline DGM models in terms of utility and detection, as specified in Section 4.1 In Tables [7[23] we present full utility and detection results. More specifically, in each table we report results for each dataset, each DGM and each C-DGM, using different orderings (i.e., random, correlation and KDF-based). Additionally, for each result in the table, report the standard deviation from the mean. As reminder, we obtained the results by running each experiment with 5 different seeds. As shown, in most cases least one of the orderings used for the C-DGMs outperforms the DGMs, particularly for detection.
Comparing WGAN with C-WGAN, we note that the only dataset for which we do not get an im- provement in utility with either of the three different metrics is LCLD. However, the gap between the WGAN and the best C-WGAN is small for all the 3 different metries: 0.7%, 0. 11,,, 0.2% for F1-score, weighted F1 and Area Under the ROC Curve, respectively. In most cases, with C- WGAN we see significant utility improvements, e.g. 7.7%, 5% and 3.8% for Heloc in FI-score, weighted Fi-score and Area Under the ROC Curve, respectively.
For TableGAN, we see that C-TableGAN outperforms the unconstrained models for all binary and multiclass classification datasets in terms of utility, according to at least 2 out of 3 metries. For the cases where the performance is not improved under some metric, we notice that the difference in performance between TableGAN and C-TableGAN is small, e.g., 0.5% for the FSP dataset and 0.2% for the Heloc dataset in Area Under the ROC Curve. It is worth noticing that out of all the C-DGMs, C-TableGAN brings the highest improvements overall. For example, C-TableGAN outperforms the
29

---

Published as a conference paper at ICLR 2024
unconstrained TableGAN by least 4.5% according to the Fl-score in 4 datasets, with the highest improvement, of 7.5%, recorded for WiDS.
As opposed to C-TableGAN, C-CTGAN does not show the same trend, with most models giving either moderate improvements or performing close to the unconstrained CTGAN, However, it is still the case that most of the datasets (four out of six) show improvements over all three metrics (with the remaining two datasets showing improvements on two out three metrics). Notably, with C-CTGAN did observe large improvement of 64.9 points in the mean absolute error for utility when training on the News dataset, which is the highest improvement we observed with any of the C-DGM models.
Similarly, to the C-WGAN case, there is only one dataset for which C-TVAE does not improve the utility performance in any of the three metrics, namely the Heloc dataset. Nevertheless, the difference between the overall best C-TVAE model (i.e. using the KDE ordering) and TVAE is small for all the 3 different metrics: 0.4%, 0.2%, 0.1% for Fi-score, weighted F1 FI-score and Area Under the ROC Curve, respectively. In most cases, with C-TVAE we see major utility improvements, e.g. 4.0%, 3.6% and 1.5% for WiDS in Fl-score, weighted Fl-score and Area Under the ROC Curve, respectively, and 2.6%, 2.9% and 1.4% for FSP in Fl-score, weighted Fl-score and Area Under the ROC Curve, respectively.
For C-GOGGLE, we that five out of six datasets outperform the standard GOGGLE results on all three metrics, giving major improvements (with the largest improvement being 17.1% in the F1-score for utility when training the URL dataset). Only the FSP dataset does not show any improvements on any of the orderings when using the C-GOGGLE version.
For detection, with C-WGAN get improvement (or do not change the performance, as it hap- pens in a few cases) over all datasets in all metrics, the only exception being the Area Under the ROC Curve for the URL dataset. With C-TableGAN, C-CTGAN, and C-TVAE,
improvement in 4 out of 6 datasets, where we outperform the unconstrained models (i) according to all metrics for 3 datasets and (ii) according at least two out of three metries for dataset, respectively. With C-GOGGLE we see again an improvement in 4 out of 6 datasets, where we outperform the uncon- strained models according to all metrics for 2 datasets and (ii) according at least one out of three metrics for 2 datasets, respectively.
C.4.1 DGMs vs. C-DGMs: UTILITY RESULTS
In Tables 17 19 we present the utility results on all datasets using 3 metrics in the following order: Fl-score, weighted Fl-score, and Area Under the ROC Curve. Separately, in Table report the performance of real and synthetic data for the News dataset (which is a regression dataset), using different metries: Explained Variance (XV) and Mean Absolute Error (MAE).
C.4.2 DGMs vs. C-DGMs: DETECTION RESULTS
In Tables 21 23 we present the detection results on all datasets using 3 metrics in the following order: F1 Fi-score, weighted F1-score and Area Under the ROC Curve.
C.5 FULL RESULTS ON DGMs VS. P-DGMS
Our constrained layer uses the constraints to correct the predictions both at training and inference time, however, the constraints can simply be used inference time to correct the predictions only once. This latter approach, which we call P-DGM, is a way of putting guardrails on the output space of the DGMs and could be the preferred option for users who do not want to make any modifications to their models or retrain them. Both methods guarantee that the constraints are satisfied by the pre- dictions, however, their impact on the models' performance might differ. To see this, we compared DGMs with P-DGMs whose predictions have been corrected according to three different label derings: random, correlation-bat and KDE- based. In each the following tables, we report the mean and standard deviation from the mean.
30

---

Published as a conference paper at ICLR 2024
C.5.1 DGMs vs. P-DGMs: UTILITY RESULTS
In Tables 24 25 and 26 we compare the performance of the DGM and P-DGM models terms of utility and report the results w.r.t. 3 different metrics: Fl-score, weighted Fl-score, and Area Under the ROC Curve. Separately, in Table 27 we report the performance of the synthetic data for the News dataset, using two metrics: XV and MAE.
As we can see, putting guardrails on the output space of the models can help increase the perfor- mance, however, the overall gains are not as high as those used by our C-DGM models which correct the outputs training time.
C.5.2 DGMs vs. P-DGMs: DETECTION RESULTS
In Tables 28
29 and we compare the performance of the DGM and P-DGM models in terms of detection. Similarly to the utility case, post proseesre the outputs of the unconstrained models can help slightly increase the overall performance.
Table 17: Binary (macro) Fi-score utility results with their corresponding stddevs for binary (multi- class) classification datasets.
URL
WIDS
LCLD
Heloc
FSP
WGAN
0.756_20.031
0.329 10.09
0.239 1000026
0.634.10.099
0.357.10.02
Rnd
0.792=0.021
0.302 075
0.196= 026
0.643++0.00 060
0.333++0.028
C-WGAN
Corr
0.790=0.022
0.284 040
0.232
0.704
0.367:10.02
KDE
0.801=0.004
0.316
0.232
0.711+00.08
0.367:+0.027
TableGAN
0.562=0.051
0.171
0.123
0.593:10.058
0.119
Rnd 0.534=0.096
0.213 027
0.148
0.636::u.00
0.199
C-TableGAN
Corr
0.611=0.11 1111
0.246 0447
0.130+0.00 036
0.638 061
0. 0.17990.006
KDE
0.576=0.074
0.20740.043
0.174
0.582++0. 114
0.208
CTGAN
0.822=00.017
0.362
0,247 000077
0.736:10.035
0.374
Rnd 0.833=0.006
0.365 019
0.235 057
0.744+t0.010
0.370+0.000
C-CTGAN
Corr
0.830=0.009
0.365 020
0.260 081
0.730 027
0.383
KDE
0.836=0.000
0.360 022
0.265
0.736++0.010
0.381
TVAE
0,810=0.000
0,282 029
0.185
0.735
0,473+0.016
Rnd
0.824=0.000
0.249 038
0.143+00 018
0.720+0000
0.501+0.012
C-TVAE
Corr
0,826=0.007 007
0.305 021
0.158 011
0,733-+0.017
0,49640.018
KDE
0.824=0.004
0.322
0.146-10.023
0.73110.0006
0.498
GOGGLE
0.622=0.094
4.18990.008
0.163 119
0,596:10.072
0.152100000
Rnd 0.782=0.035
0.139 068
0.157=+0.08
0.723+t0.018
0.117+u.000
C-GOGGLE
Corr
0.793=0.013
0.185 =. 108
0.219
0.71440.013
0.13640.024
KDE
0.788=0.014
0.171
0.167+0.082
0.708++0.022
0.13440.016
31

---

Published as a conference paper at ICLR 2024
Table 18: Weighted Fi-score utility results with their corresponding stddevs for binary and multi- class classification datasets.
URL
WiDS
L.CDD
Heloc
FSP
WGAN
0.764=0.018
0.381 057
0.359=+0.019
0.599 050
0,339:40002
Rnd
0.782=0.025
0.360=0.000 008
0.339 020
0.585+0.007 0799
0.316#0.029
C-WGAN
Corr
0.785=0.011
0.344 036
0.358+00.03
037
0.648+0.002
0.349 0.8
KDE
0.790
0.373
0.358
0.649.10.011
0.349 10.028
TableGAN
0.659=0.035
0.240 034
0.286
0.613:+0.030
0.199
Rnd 0.644=0.061
0,280 025
0.306 043
0.618400.00 0-12
0.198+0.000
C-TableGAN
Corr 0.695
0.309
0.292
0.633
0.180:10.034
KDE
0.670=0.052
0.274
0.314=10.029
0.596:10.048
0.209
CTGAN
0.799=0.033
0.405
0.379
0.675:10.015
0.372:10.034
Rnd 0.818=0.008
0.408
0.369+0.044
0.684
0.369+0.0077
C-CTGAN
Corr
0.816=0.012
0.409=0.019
0.388 060
0.688++0.00
0.379
KDE
0.820
0.403
0.392
0.690.1..010
0.380
TVAE
0.802=0.012
0.342 027
0.330
0.696
0,463
Rnd
0.817=0.007
0.31 034
0.301 012
0.687+0 006
0.492+0.013
C-TVAE
Corr
0.81620.007
0.3.3.10.019
0.31 1:00011
0.690.10.008
0.488100.019
KDE
0.81620.000
0.378 10.038
10.011
0.694.1..003
0.490110.014
GOGGLE
0.648=0.074
0.198=00.067
0.315
0,566::0.00
0.13940.002
Rnd 0.741=0.082
0.210=400061
0.314:10.065
0.663:10.01
0.103410.008
C-GOGGLE
Corr 0.752=0.024
0,253 098
0.357
0.637:10.023
0.121 10.024
KDE
0.749
0.239
0.318
0.656:10.014
0.11910.017
Table 19: Area under the ROC curve utility results with their corresponding stddevs for binary and multiclass classification datasets.
URL
WiDS
LCLD
Heloc
FSP
WGAN
0.839
0.775
0.618 10.011
0.677.10.033
0.742.10.007
Rnd 0.860=0.018
0.798
0.616.10.009
0.672.10.037
0.72810.010
C-WGAN
Corr
0.865=0.011
0.796=10.011 017
0.612+0.016
0.714
0.742+0.022
KDE
0.858=0.011
0.816 015
0.612
0.715:10.011
0.742+0.022
TableGAN
0.843=0.020
0.740=1.0021
0.587+0.027
0.707
0.64210.026
Rnd 0.853=0.018
0.778=t0.017
0.593
0.6924tu.222
0,637+10.222
C-TableGAN
Corr
0.868=0.007
0.775 010
0.605
0.705
0.630+t.005
KDE
0.865=0.010
0.767:10.01
0,587-10.017
0.676:10.037
0.635+10.026
CTGAN
0.859
0.835
0.651 00.000
0.744.10.00
0.700
Rnd 0,880 00.004
0.837 00000
0.626 6626:10.028
0.749 7749.10.008
0.757:10.006
C-CTGAN
Corr
0.877=1.000
0.827 .000033
0.643.10.015
0.755.10.007
0.760
KDE
0.880=0.00
0.833=0.000 007
0.641
0.751+0.011
0.759+0.012
TVAE
0.863
0.800
0,631 1.0.004
0.752.1...0
0.789
Rnd
0.876
0.773 0....00003 036
0.630
0.750
0.80310.009
C-TVAB
Corr
0.878=0.00
0.803 014
0.633++0. 003
0.749- 005
0.791+0.015
KDE
0.879=0.007
0.815=10.022
0.632++0.00 003
0.751+00.002
0.794
GOGGLE
0.742=0.071
0.656
0,543 039
0.600-+u 056
0.577+0.010
Rnd .800==0.02
0.643=10.086
0.569-10.055
0.719
0.535:t0.016
C-GOGGLE
Corr 0.794=20003
0.675 :00001
0.593:10.043 0.06
0.696
0.546+0.011
KDE
0.802=0.016
0.678=0.0029
0.572++0.051
0.713+00.22
0.538+0.020
32

---

Published as a conference paper at ICLR 2024
Table 20: Utility performance comparison between DGM and C-DGM models trained News, using XV and MAE. In the first row, we report the real data performance.
XV
MAE
Real data
=0.001
3001.
WGAN
-0.006++0.00 003
3133.6#242,
Rnd
-0.008+0.002
3093.0 202.33
C-WGAN
Corr
-0.002++0 002
3014.0
KDE
-0.004.10.00
3070.0 98.4
TableGAN
-0.001110.001
2992.2
Rnd
-0.001 1:0.001
3033, 153.9
C-TableGAN
Corr
-0.002.1.0.00
3015.9 :00
KDE
-0.002 002
3016.8=70.8
CTGAN
-0.002:1:.00
3043,8 #37.8
Rnd
-0.002::0.00
3034.6 2294
C-CTGAN
Com
-0.002:10.00
2978.9
KDE
-0.003-+00000
3029. 722.77
TVAE
-0.001
3021.3
Rnd
-0.0024+0
3067. 77..
C-TVAE
Corr
-0.001::0 001
3005.0=566
KDE
-0.002.100.00
301 1.7 1:4.22
GOGGLE
-0.004.10.000
6054.51144.
Rnd -0.001110.00
3042.615444
C-GOGGLE
Corr 10.0011:0.001
3026.01:34.6
KDE
-0.001+:00000
2999.0#27..
Table 21: Binary Fl-score detection results with their corresponding stddevs for all datasets.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.865+10.02
0.975:60002
0.999:0.001
0.96412.00x
0.914:0.018
0.954:10022
Rnd 0.864=1.046
0.996110.003
0.916=0.013
0.970=0.024
0.900+0.078
0.969=0.015
C-WGAN
Cor
0.879#0.027
0.975+000006
0.923=0.005
0.964=0.007
0.843:0.037
0.928:00.023
KDE
0.87740.016
0.975+0.002
0.923=0.005
0.957
0.843+0.037
0.926:20.011
TableGAN
0.831
0,963++.000
0.895=0.024
0.923=0.011
0.90940.021
0.927+0.009
Rnd 0.840=0.022
0.984+0.000
0.870=0.011
0.963+0.02
0.83940.061
0.953+0.0110
C-TableGAN
Cor
0.849#0.00 025
0.956:2.000 004
0.874=0.011
0,952+0.010
0.818
0,933=t0.011
KDE
0.828.10.00
0.962.10.007
0.869 10.014
0.948
0.830.10.015
0.903
CTGAN
0.850 027
0,990+0.000 002
0.914+0.021 0.848:0.016
0.926
0.901=0.019
Rnd 0.834-10.026
0.990.10.001
0.863
0.910
0.859
0.9091
C-CTGAN
Corr 0.820
0.987-10.003
0.845
0.903
0..01
0.905
KDE
0.838-10.033
0.986110002
0.848
0.89710.023
0.85741.002
0.90210.014
TVAE
0.813
0.926:10.001
0.84210.019
0.91420.011
0.843:10.027
0.877+0.013
Rnd 0.826=40.037
0.96540.000
0.796=0.030
0.90540.022
0.874+0.015
0.874400011
C-TVAE
Cor
0.815-40.037
0.95944.0000
0.808
0.905
0.855-10.020 020
0.863-10.008
KDE
0.814:10.030
0.96110..02
0.799
0.907
0.879-10.00
0.890:10.019
GOGGLE
0.892+4.0019
0.98740.018
0.890-0.017
0.924
0.912+0.010
0.949:10.023
Rnd 0.890=0.044 0.444
0.965+0.006 006
0.904=0.011
0.939=0.000 008
0,866=20.020 020
0.967=0.009
C-GOGGLE
Corr 0.898.10.021
0..02
0.91010.0017
0.939-100010
0.884:10.009
0.968
KDE
0.903-10.019
0.9051
0.91110.0017
0.008
0.88710.014
0.95710.020
33

---

Published as a conference paper at ICLR 2024
Table 22: Weighted Fl-score detection results with their corresponding stddevs for all datasets.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.856=0.000 008
0.976+0.000
0.999=0.001
0.964=0.00
0,910+0.022
0.954==0.020
Rnd 0.85110.0117
0.996.10.00
0.912.10013
0.970
0.900.1.0072
0.9691000016
C-WGAN
Corr
0.97510.000 10.016
0.917.10007
0.964
0.84510.022
0.929 10.022
KDE
0..01
0.975.10.002
0.917
10.019
0.845.10.022
0.0051
TableGAN
0.822:10.007 0.964-1..000
0.895
0.025
0.906:10.028
0.029
Rnd 0.830=0.011
0.984-10.002
00001
0.963
0.834-10.054
0.953:10.011
C-TableGAN
Cor
0.835
0.957:10.000
0.866=1.022
0.952200010
0.813:1..020
0.936:1:0010
KDE
0.827=1.0006
0.963+t0.007
0.871=0.013
0.947=1.0119
0.817:+0.021
0.934:10.011
CTGAN
0.862:1.014
0,990+0.000
0.850=0.019
0.915
0.927=00016
0.896:10.023
Rnd 0.843=2.0177 0.990
0.867=0.015
0.861 0.011
0.904=:0.018
C-CTGAN
Corr
0.839 0.98810.002
0.846
0.90411.0013
0.86510.025
0.0022
KDE
0.84910.020
0,986110.002
0.837 10.014
0.897.10.22
0.86811..009
1..041
TVAE
0.831
0.927+00.00 001
0.916=0.010 0.802=00.010
0.847
0.855=00.019
Rnd 0.809.10.114
0,966.10.00
0.795 10.025
0.90810.020
1.8681100018
0.856110.012
C-TVAE
Corr
0.832.10.011
0.96011..003
0.795
0.00810.013
0.857:10.011
0.85410.000
KDE
0.829:1.014
0.962
0.797
0.909
0.873.10.014
0.882
GOGGLE
0.810
0.987:100088
0.092
0.92620.005
0.916:10.011
0.955
Rnd 0.89110.030
0.965
0.905=0.010
0.940
0.823++0.030
0.970110.008
C-GOGGLE
Corr 0.89810.014 0.971
0.912=0.017
0.939
0.817410004
0.97210.012
KDE
0.902:+0.017
0.975+t0.013
0.913=000188
0.939=
0.865-t0.035
0.963:t0.015
Table 23: Area under the ROC curve detection results with their corresponding stddevs for all datasets.
URL
WiDS
L.CLD
Heloc
FSP
News
WGAN
0.872=#.000
0.989
1.000=0.000
0.983=0.004
0.916:0.016
0.964=000221
Rnd
0.877=0.017
0.999+0.00
0.941=0.009
0.983=0.018
0.918+0.000
0.981=00011
C-WGAN
Cor
0,879=0 009
0.98942.000 002
0,946=0.000 006
0.982=0.00 007
0.874+0.022 020
0,945=10.020
KDE
0.883.10.009
0.989
0.9461000006
0..051100188
0.874.1.002
0.941:10.00
TableGAN
0.850#0.00 007
0,980+ 001
0.926=20.020
0.953 009
0,90740.02.
0.940=:0.011
Rnd 0.851#0.000 006
0,995 0010
0.900=0.012
0.978=0.00m 016
0,866
0.964=00011
C-TableGAN
Corr 0.856110.008
0.974
0.904100011
0.970
0.84510.005
0.950110.010
KDE
0.849:0.005
0.978
0.909
0.965
0.849:10002
0.94710.010
CTGAN
0.879=
0,996:+0.00
0.8961
0,932 0.953
0.912:0.021
Rnd 0.865-20.113
0.997-4.000
0.908
0.950
0.896+0.011
0.925++0.021
C-CTGAN
Cor
0.86410.000
0.995-10.01
0.900=0.020
0.946=
0.894-1.002
0.922+00019
KDE
0.86740.015
0.995:10.001
0.897=0.012
0.943
0.893+40.007
0.919+0.022
TVAE
0.85440.007
0.935+40.00 002
0.861 009
0.947
0.872+ 008
0.881
Rnd 0.854+0.00 008
0,982++0.00 001
0.844
0.942=0.015
0.904+0.015
0.883+0.012
C-TVAE
Cor
0.853=0.000
0,977+0.000
0.840=0.011
0.943=0.011
0.896+0.000
0.880=0.004
KDE
0.84910.013
0.979
0.843
0.94410.014
0.90411.003
0.909
GOGGLE
0.891 009
0,993+40.013
0.928=0.010
0.949=0.000
0,920+0.000
0.973+0.013
Rnd 0.898 0.002
0.979
0.93810.010
0.958
0.871
0.977
C-GOGGLE
Corr 0.906:10.020
0.984-10.005
0.943
0.960
0.865
0.980-10.012
KDE
0.907=0.018
0.985
0.944-10.013
0.960
0.880-1.014
0.973-10.009
34

---

Published as a conference paper at ICLR 2024
Table 24: Binary (macro) Fl-score utility results with their corresponding stddevs for binary (multi- class) classification datasets when postprocessing the unconstrained predictions.
URI.
WiDS
L.CLD
Heloc
FSP
WGAN
0.756=20.030
0.329 0559
0.239 026
0.63440.099
0.357=00.20
Rnd
0.767=0.08 034
0.318+0.053
0.201+0.030
0.641#0.113
0.355=0.020
P-WGAN
Com
0.766= 036
0.332 050
0.214:00002
0.641
0.358=0.021
KDE
0.76710.035
0.334.0..059
0.214:10.021
0.641 10.112
0.358.10.00
TableGAN
0.562:+0.051
0.171
0.123
0.593+0.058
0.199=0.011
Rnd 0.565 048
0.173++0.015
0.115+0.002 0.08
0.594=00058
0.105
P-TableGAN
Corr 0.553:10.050
0.174:1.0045
0.125::0.032
0.594100:00
0.393770737
KDB
0.556=20.000
0.171
0.1 0.119::0.025
0.592+0.060
0.193=0..02
CTGAN
0.82240.017
0.362:10.033
0.247:400087
0.736:10.035
0.37440.034
Rnd 0.819#0.014
0.364+0.0022
0.255:+0.089
0.735=t0.038
0.365=0.021
P-CTGAN
Con
0.814:100008
0.365 0.338
0.246+0.008
0.743+t0.03
0.372=0.033
KDE
0.823:10.017
0.357:10.040
0.248
0.736:10.031
0.371
TVAE
0.810=0.008
0,282 029
0.185 021
0.735+0.010
0.473=0.016
Rnd
0.810#0.011
0,266+0.00 020
0.172+00001
0.739
0,464=0.019
P-TVAE
Cor
0.815 100009
0.28310.022
0.176.10.028
0.730.0..00
0,462
KDE
0,815 10.099
0.285.10.08
0.10110.030
0.735
10.021
GOGGLE
0.622:00094
0,18940.008 .038
0.163 119
0.596+0.077
0.152==0.008
Rnd 0.626:10.098
0.193110.02
0. 0.164:10.121
0.600
0.15110.000
P.GOGGLE
Com 0.623110.089
0.191 1:0038
0.169
0.601
0.151
KDE
0.626:10.096
0.189:10.099
0.169
0.597
0.155
Table 25: Weighted Fl-score utility results with their corresponding stddevs for binary and multi- class classification datasets when post-processing the unconstrained predictions.
URL
WIDS
LCLD
Heloc
FSP
WGAN
0.76410.018
00.014
0.359:0.009
0.599
0.339
Rnd
0.768:1.028
0.372+t.0022
0.342+0.019
0.600+0.057
0.337=0.022
P-WGAN
Com 0.769:1.002
0.384:10.044
0.352:4..014
0.5981
0.341 :0.02
KDE
0.762:10.028
0.386:10.057
0.352:0.014
0..039
0.341 =0.026
TableGAN
0.659.1.0035
0.240:10.034
0.286:10.029
0.61510.030
0.199:10.043
Rnd 0.660.0.003
0.243 10.045
0.281-10.022 1.0.022
0.614
0.195 10.046
P-TableGAN
Com 0.657.10.035
0.20310.004
0.285
0.61410.030
0.195:10.038
KDE
0.658++0..10
0,240+0.038
0.285+0.011
0.613+0.029
0.193=#00022
CTGAN
0.79910.033
0.405.10.034
0.37910.006
0.372 0..75
Rnd 0.794-10.031
10.033
0.383
0.671 1.0015
0.363
P-CTGAN
Cor 0,801#0.000 0,407=0.0000
0.379+0.00 .668
0.678=00.010
0.370=0.003 003
KDE
0,802#00.02
0,399+4.0012
0.380+0.058
0.671+0.010
0.368
TVAE
80240.012
0,342+t.0027
0.330-10.016
0.696=00.006
0..63
Rnd 802++0.015
0.327+t..015
0.324+
0.696:00.001
0.454:20.020
P-TVAE
Com
0.806:10.008 0.342-t0.022
0.325
0.691
0.452=20.019
KDE
0.806:20.008 0.344:10.081
0.322++0.022
0.69410.004
0.453=+0.02
GOGGLE
0..08
0.198.10.067
0.315
0.566:10.00
0.139
Rnd 0.645
0.202.0.000
0.315 1.315110.086
0.5667
0.139.10.00
P-GOGGLE
Corr
0.645-20.070
0,201 +0.067
0.318 090
0.56620.047
0.142= 0.0
KDE
0.650=0.072
0.197+4.006
0.318++0.090
0.565+0.041
0.141
35

---

Published as a conference paper at ICLR 2024
Table 26: Area under the ROC curve utility results with their corresponding stddevs for binary and multiclass classification datasets when post-processing the unconstrained predictions.
URL.
WiDS
L.CLD
Heloc
FSP
WGAN
0.839
0.775#0.008
0.618+0.011
0.677=0.083
0.742=00.07
Rnd
0.840=0.019
0.777=0.007
0.618+0.010
0.682+0.012
0.736=0.011
P-WGAN
Com
0.839:#..000
0.782 035
0,617+0.009
0.683+00008
0.740=0.010
KDE
0.83410.021
0.78310.039
0.617.10.08
0.682
0.740.1..010
TableGAN
0.843=0.002
0.740#0.002
0.587
0.707=00007
0,642=0.026
Rnd
0.848:0.019
0.749
0.585+:0.00 0.077
0.703++0000
0.641#0.030
P-TableGAN
Corr 0.846.10.020
0.7012
0.587:10.022
0.707:10.01
0.645
KDE
0.855+0.016
0.73640.016
0.587+10.027
0.707=00001
0.64420.035
CTGAN
0.85940.040
0.835:0.0112
0.651
0.744-10.009
0.760:20.014
Rnd 0.859+0.038
0.837=400022
0.652:+0.024
0.743
0.753=0.008
P-CTGAN
Cor
0.866:10.007
0.835+00.012
0.650:t0.023
0.745
0.754=0.001
KDE
0.86310.035
0.836110.012
0.651:10.015
0.743
0.703
TVAE
1.86+00.011
0,800
0.631
0.752
0.789=0.007
Rnd
0.866=40.03
0.785+0.011
0.630+ 0.00
0.751
0.787=0.011
P-TVAE
Corr
0.800
0.787.10.000
0.629 1..09400004
1..004
0.779
KDE
0..0010.0077
0.797.0.016
0.630.10.002
1..000
0.788 10.010
GOGGLE
0.742::0.01
0.656#0.0045
0.543 039
0.600+0.056
0.577=0.010
Rnd 0.738
0.665.10.033
0.548100.035
0.602
0.575=0.008
P.GOGGLE
Cor 0.74010.063
0.667100.04
0.542:10.036
00.05
0.576=0.0010
KDE
0.741
0.667110.000
0.54240.036
0.603
0.5014
Table 27: Utility performance comparison between DGM and P-DGM models trained on News.
XV
MAE
WGAN
-0.006_0.0033
3133.6.142..
Rnd
-0.005
3151.34211.8
P-WGAN
Com
-0.012=00011
3109.2+220.8
KDE
-0.006=0.003
3159.7+210.7
TableGAN
-0.001=00001
2992.24335.
Rnd
-0.001=0.001
2981.9 32.33
P.TableGAN
Corr
-0.001=0.001
3015.3
KDE
-0.001=0.001
3022.7
CTGAN
-0.002
3043 8
Rnd -0.003=0.004
3013 36.2
P-CTGAN
Cor
-0.006
2999.3 127.3
KDE
-0.003
3004.8 477.4
TVAE
-0.001
3021.3 1021.334:0.00
Rnd -0.001-0.000
3040.5
P-TVAE
Cor
-0.002=0.001
3003.8
KDE
00.001000000
3032.2+:99. 39.3933
GOGGLE
-0.004=0.003
3054.5 1:44.7
Rnd
-0.006=0.0005
3024.
1:37222
P-GOGGLE
Cor
-0.005=0.002
0077.9
KDE
-0.004=0.003
3012.3+36..6
36

---

Published as a conference paper at ICLR 2024
Table 28: Binary Fl-score detection results with their corresponding stddevs for all datasets when post-processing the unconstrained predictions.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0,865 0335
0.975=0.002
0,999
0.964
0.914+0.018
0.954
Rnd
0,856=0.033
0.979=0.000
0.916=0.008 008
0.964=0.004
0.910=0.015
0.954
P-WGAN
Com
0.857=0 033
0.977=0.000
0.921
0.968 005
0.911
0.954=00026
KDE
0.86210..037
0.97810.003
0.92110.004
0.96711.004
0.91111:018
0.957
TableGAN
0.831 020
0.963=0.000
0.895 024
0.923
0.909=00022
0.927
Rnd 0.843=0.034 003
0.962=
0.896 026
0,923=0.001
0.858+ 081
0,932=#0.010
P-TableGAN
Corr 0.833:10.035
0.963
0.900
0,922:10.011
0.860110.029
0.920:1.000
KDE
0.835::0.041
0.962=0.006
0.901
0.921
0.857+4.0029
0.928
CTGAN
0.850=10.027
0.990
0.848
0.914110002
0.926:0:011
0.901
Rnd 0.845=000028
0.990=0.002
0.839
0.916:#0002
0,902:+00222
0.091
P-CTGAN
Cor
0.855:00.014
0.989=0.003
0.845
0.913 02002
0.895:000000
0.098
KDE
0.850::0.023
0.99110.002
0.84110.0099
0.913
0.895
0.90210.000
TVAE
0.813 037
0.926=0.00 001
0.842
0.914 0011
0.843 027
0.877=0.013
Rnd
0.806#0.08 039
0.962
0.835=00000
0.914 007
0.881+0.000 000007
0.873=0.012
P-TVAE
Cor
0.80610.044
0.002
0.829 10.022
0,912.10.011
0.881100001
0.866 10.014
KDE
0.80610.044
0.064
0.8371100010
0.909 10.011
0.874110.012
0.868 10.012
GOGGLE
0.892:t0.015 .001
0.987=0.018
0.890=2.0017
0,924 .006
0.912=+0.010
0.949
Rnd 0.8841:0.025
0.98810.017
0.89210.012
0.9271100006
0.907=100005
0.952
P.GOGGLE
Cor 0.897:10.010
0.988
0.902
0.926:10.006
0.909::0.006
0.95111.002
KDE
0.89840.015
0.987=0.017
0.902+0.005
0.928
0.907=10000
0.951100.00
Table 29: Weighted Fl-score detection results with their corresponding stddevs for all datasets when post-processing the unconstrained predictions.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.856420.008
0.976=0.002
0.999
0.96411.004
0.91010.027
0.954
Rnd
0.852:t..011
0.979=0.003
0.907
0.965
0.907:10.00
0.957
P-WGAN
Com
0.853:10.000 0.977=0.002
0.914 002
0,968110.00
0.004
0.912+:0.01
0.955
KDE
0.855:1..010
0.978=0.003
0.914
0.967:10.004
0.912::0.118
0.959:0.0117
TableGAN
0.822:10.007
0.96411.0066
0.895110.0g
0.9251:0001
0.906
1..011
Rnd 0..05
0.003
0.888 10.031
0.924110.011
0..011
0.934:100012
P-TableGAN
Com
0.826:10.006 0.964
0.898
0.924110.011
0.86810.018
0.923 100000
KDE
0.824++0000
0.963=0.006
0.896=20.02
0.923+0.011
0,873+0.02
0.928
CTGAN
0.862:10.014 0.990
0.850
0.915
0.92711.016
0.896
Rnd 0.85910.017
0.900
0.840 10.014
0.91710.027
0.905
0.894
P-CTGAN
Cor 0.860=2.012 0.989=0.003
0.850 009
0,914 020
0.902
0.894
KDE
0.861+0.017
0.991#0.002
0.847 006
0.913=10007
0.905+0.025
0.899=0.011
TVAE
0.831+0.013 0.927=0.001
0.832+100000 010
0.916#2.010
0.847+0.000
+0.09
Rnd 0.828 0.963
0.832=+0.0m
0.915=000007
0.877+0.010
0.856=20.017
P-TVAE
Cor
0.829:10.012 0.963
0,825
0.914:1.011
0,882:1
0.860
KDE
0.829:100012
0.96411.001
0.83110.014
0.910=10.011
0.870:1.0066
0.848
GOGGLE
0.880:10.012
0.987
0.892
0.926:10.00
0.91611.011
0.955
Rnd 0..08
0.98810.011
10.015
0.928
0.91110.007
1.9571
P.GOGGLE
Cor 0.875+0.00 0211
0.988=0.016
0.901
006
0,927 005
0.907+0.012
0.957=0.017
KDE
0.883+0.011
0.987=0.017
0.901 006
0,929#0.000 0000
0.906=#00011
0.956=0.017
37

---

Published as a conference paper at ICLR 2024
Table 30: Area under the ROC curve detection results with their corresponding stddevs for all datasets when post-processing the unconstrained predictions.
URL
WiDS
LCLD
Heloc
FSP
News
WGAN
0.872=00.00 0008
0.989=0.002
1.000=0.000
0.983=10.00 004
0.916+
0.964
Rnd
0.867=0.003
0.992=0.002
0.936=0.010
0.985=#00002
0.927=0.006
0.966=0.002
P-WGAN
Com
0.869=0.00 001
0.991=0.002
0,942=0.000
0.985 003
0.931=0.013
0.966=0.024
KDE
0.87010.003
0.99110.000
0.942.0.001
0.985
0.0011
0.96910.011
TableGAN
0.850=0.007
0.980=0.001
0.926 020
0.953 009
0.907=00002
0.940=0.011
Rnd 0,850=0.008
0.980=0.001
0.925 024
0,953 000
0.89410.016
0,946=
P-TableGAN
Corr 0.84810.000
0.980
0.929
0,953:10.008
0.886110.018
0.937
KDE
0.852+00.009
0.979=0.003
0.933
0,952
0.890+0.022
0.939
CTGAN
0.879:1.0008
0.996100001
0.896:10.011
0.953:100016
0.932+00007
0.912:10.021
Rnd 0.876#0.012
0.997=00001
0.892 008
0.953=000021
0.925:000199
0.915
P-CTGAN
Cor
0.878 0.004
0.997=0.001
0.896=0.0000 000
0.952:10.015
0.922:00013
0.921
KDE
0.876:10.013
0.998
0.893 004
0.952:10.022
0.92511.017
0.91910.021
TVAE
0.854 007
0.935=0.000
0.861
0.947 005
0.872
0,881
Rnd 0.849
0.978=0.000
0.862 000 008
0, 0,948 004
0,907
0.887=0.021
P-TVAE
Cor
0.850.1.0008
0.97910.001
0.858 10.007
0,94610.007
0.910.10.011
0.884
KDE
0.85010.0009
0.97910.001
0.861
0.861100.008
0,94410.006
0.895110.013
0,874 10.018
GOGGLE
0.891#:.0008 .009
0.993=20013
0.928=
0.949 003
0.920+0.000
0.973=00013
Rnd 0.889
0.99410.013
19223.0.008
0.952.10.004
0.92410.011
0.976
P.GOGGLE
Com
0.8951000110
0.99410.012
0.931110.000
0.952:1.003
0.919#0.011
0.977
KDF
0.89410.017
0.993
0.931 100000
0.952+10.003
0.917:00010
0.976=200012
C.6 REAL DATA PERFORMANCE
To ensure that the comparisons be- tween our C-DGM models and the baseline unconstrained DGMs are meaningful, we conducted hyperpa- rameter search as detailed earlier in Section B.3 allowing us to get close to (and sometimes even surpass) the real data utility performance. We re- port the latter in Table 31 using the same three metrics used for mea- suring the synthetic data utility per- formance (i.e., F1-score, weighted F1-score, and Area Under the ROC Curve) following the same protocol as the one described in Appendix B.4 Comparing the results here with those for the synthetic data Tables 17 19 we can see that, overall, C-WGAN and C-CTGAN models got utility scores similar to those obtained on the real data. For C-TableGAN and C-TVAE we notice several cases where our method helped in bringing the performance of the synthetic data closer to the real data. In particular, for LCLD we notice that the real data got an F1-score 4.8% higher than the unconstrained TableGAN, but our C-TableGAN model was able to match the real data performance (scoring slightly better than it, i.e., by 0.3%). It also worth noticing that the gap between real and synthetic performance was reduced the most for the LCLD and WIDS datasets, both of which are highly unbalanced. For instance, all C-WGAN, WGAN, C-CTGAN, and CTGAN models yield a higher Fl-score than the real LCLD data. We notice similar trends for the other metrics, weighted Fl and Area Under the ROC Curve. On the other hand, none of the DGM or C-DGM models get close to the real FSP data. One possible reason is the datasets' small size and multiclass nature, which makes it harder for the DGM models to capture patterns that lead to the correct different targets.
Table 31: Utility scores calculated on real data.
F1
wF1
AUC
URL
0.007
0.87510.014
0.903=0.009
WIDS
0.083
0.43410.020
0.032
LCLD
0.030
0.31610.013
0.645
Heloc FSP
0.772=0.003
0.662+0.011
0.707=0.008
0.662=0.011
0.659+0.009
0.848=0.010
38


Reviews:
[{"review_id": "hudTO1Aiqn", "reviewer": "Reviewer_DAiC", "summary": "The main contribution of this paper is to add constraints on generating synthetic data so that it is aligned with available background knowledge. The paper introduces constraint layers in order to enforce a set of linear constraints that encode the background knowledge. They also prove the correctness of the constraint layers introduced", "questions": "This isn't really a question but I think the use of linear constraints can be advantageous as they are logically complete. Any inconsistency can be identified easily through Farkas's lemma. I think the authors can potentially frame the use of linear constraints in a better light to highlight how its logical completeness can be useful when specifying background knowledge.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation behind the paper is clear and easy to follow. The paper also offers some theoretical justification for their method which makes their paper compelling. The proofs from what I can see are correct. The experiments are well formulated and support the claims of the paper.", "weaknesses": "The issue with the paper is the proofs can be difficult to follow. It is possible that the authors can spend more time rewording it to make it easier to flow. This also makes it a bit hard to flow and check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KxcpefK3HT", "reviewer": "Reviewer_Yiht", "summary": "Generating realistic tabular data requires compliance with constraints that encode essential background knowledge on the problem. In this paper, the authors address the limitation and show how deep generative models for tabular data can be transformed into constrained deep generative models, whose generative samples are guaranteed to be compliant with the given constraints. This is achieved by automatically parsing the constraints and transforming them into a constraint layer seamlessly integrated with the dgm. The authors shows the effectiveness of the proposed model with experiments on 6 datasets.", "questions": "How was the performance improvement of the state-of-the-art methods after applying the proposed method?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The first to handle with the constraints of tabular data, and the method addressed the problem well.", "weaknesses": "Tabular data synthesis methods used for the experiments are outdated. Please consider [1] GOGGLE, [2] GReaT, [3] STaSy, [4] CoDi, and [5] TabDDPM if possible.\n\n\n[1] GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure, ICLR 2022\n\n[2] Language Models are Realistic Tabular Data Generators, NeurIPS 2021\n\n[3] Stasy: Score-based tabular data synthesis, ICLR 2022\n\n[4] CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis, ICML 2023\n\n[5] Tabddpm: Modelling tabular data with diffusion models, ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EnMou140hz", "reviewer": "Reviewer_ikQ7", "summary": "This paper investigates a lesser-explored challenge in the generation of tabular data—adherence to specific rules or constraints for data entries. It highlights a common issue where existing generative models often fail to respect constraints such as linear inequalities (e.g., one column being less than or equal to another), as they are not designed to fulfill these conditions.\n\nTo address this, the paper introduces a novel approach to adding a constraint layer to standard DGMs, transforming them into constrained DGMs. The study demonstrates that these constrained DGMs are more effective in producing realistic data that adheres to the specified constraints, further improving downstream performance. Moreover, it shows that even applying these constraint satisfaction layers to a pre-trained DGM can significantly enhance the realism of the generated data.\n\nWhile the paper tackles an interesting issue, given the marginal novelty of the paper and the limited number of constraints in the datasets under consideration, I will not accept this paper. I would be willing to increase the score if the authors can provide reasonable answers to my concerns.", "questions": "1) Even though GANs have achieved popularity for image generation, they are known to fail drastically for tabular data generation. That said, is there any reason why other DGMs such as TVAEs, STaSy, and TabDDPM are not considered in this study given the current limitations of GANs? For a more compelling story, it would be good to include other types of generative models as well. Especially in the post hoc experiments.\n\n2) The reported charts and tables are thorough, but I wasn’t able to find any source code for reproducibility and a footnote claims that the code will be released upon publishing; however, I didn’t find anything to run in the supplementary material.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 4, "contribution": 1, "strengths": "1) The paper is well-written and the idea is laid out with sufficient examples to follow through.\n2) Pointing out the fact that many DGMs violate obvious constraints in tabular data is very important and studying it is valuable.\n3) The experimental results seem thorough and the theory checks out.", "weaknesses": "1) **(Important)** The paper mentions the choice of $\\lambda(i) = i$ is for ease of notation, while different orderings can drastically change the performance of a C-DGM. As an illustrative example, consider tabular data with three outputs $\\langle x_1, x_2, x_3 \\rangle$ with constraints: $x_2 \\le 10$ and $x_1 \\le x_2 \\le x_3$. Now assume the generative model only produces $\\langle 10, 5, 5 \\rangle$. In this case, if the ordering $\\lambda = \\langle 1, 2, 3 \\rangle$ is considered, then $CL(\\tilde{x})$ would be $\\langle 10, 10, 10 \\rangle$ and if the ordering $\\lambda = \\langle 2, 1, 3 \\rangle$ is considered, then $ CL(\\tilde{x}) = \\langle 5,5,5 \\rangle$ with one being significantly closer to the generated distribution than the other. In fact, the notion of “optimality” is not well defined, as each ordering can produce a different optimal with none of them being comparable for defining an optimum. Moreover, the notion of optimality should also consider the discrepancy between the $CL(\\cdot)$ outputs and the generative samples to be minimal.\n2) **(Important)** I might have missed something but the reduction defined for constraints $\\Pi$ can easily turn the number of constraints exponential, meaning that $|\\Pi_1| \\in \\mathcal{O}(exp(|\\Pi|))$. The only reason the current experiments do not hinder the performance is that the number of conditions is fairly small, to begin with. I would require datasets with a much larger number of conditions to be convinced that the post hoc method does not impact the sample generation time.\n3) The paper only considers linear constraints. Even though it is pointed out as a limitation some extensions to non-linear constraints are quite simple. For example, by introducing polynomial features, one can add polynomial constraints to the current approach. Having one entire paper on linear constraints seems rather limited in novelty. I would suggest adding simple experiments as proof-of-concept for such extensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7rNiYdK4P", "reviewer": "Reviewer_72Ff", "summary": "The authors propose a method for generating tabular data which respects some domain-specific conditions, by introducing the so-called Constraint Layers (CL), which can enforce linear constraints on the features of the generated data. CLs are tested on GAN models and the results on a selection of tabular problems with constraints show how they are effective at preventing generation of data which violates such constraints.", "questions": "- Perhaps a point that I missed from the paper, but how do CLs affect the training? Can gradients backpropagate through these layers? Can CLs be applied to other Generative Models?\n- How does the application of CLs shift the distribution of the generated data? Does it result in an \"overpopulation\" of the regions on the boundaries? If that's the case, the resulting distribution would be skewed from the true distribution especially when the baseline model generates many samples which violate the constraints. Adding other metrics (see weaknesses) would help with investigating this matter.\n- Can CLs impose constraints between categorical variables, or between numerical and categorical variables? For example, if x1 = \"category 1\", then x2 > 5, or similar?\n- Are there cases in which assigning a valid variable ordering is not feasible?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an effective way to enforce linear constraints on (and between) variables for tabular data generated with deep Generative models. The manuscript is generally very clear and presents the contributions in great detail. The method is tested on a comprehensive set of problems and compared with three strong baselines.", "weaknesses": "From reading the paper, it is not immediately clear how the training procedure with CLs works, and whether CLs can be applied to other generative models, such as Variational Auto Encoders, Normalizing Flows or Diffusion Models. A comparison with methods such as TVAE and TabDDPM (mentioned in the related work) and the application of CLs to them would significantly strengthen the experimental section. A metric to compare the data distribution from the generated distribution is missing from the experiments. Would it be possible to include a metric such as negative log-likelihood or Wasserstein distance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)