Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Authors: Zhengyu Fang
Zhimeng Jiang
Huiyuan Chen
Xiao Li
Jing Li
Venue: ICML 2025 poster (2025)
Keywords: Memorization
Tabular Data
Diffusion Models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Tabular data generation has attracted significant research interest in recent years, with the tabular diffusion models greatly improving the quality of synthetic data. However, while memorization—where models inadvertently replicate exact or near-identical training data—has been thoroughly investigated in image and text generation, its effects on tabular data remain largely unexplored. In this paper, we conduct the first comprehensive investigation of memorization phenomena in diffusion models for tabular data. Our empirical analysis reveals that memorization appears in tabular diffusion models and increases with larger training epochs. We further examine the influence of factors such as dataset sizes, feature dimensions, and different diffusion models on memorization. Additionally, we provide a theoretical explanation for why memorization occurs in tabular diffusion models. To address this issue, we propose TabCutMix, a simple yet effective data augmentation technique that exchanges randomly selected feature segments between random same-class training sample pairs. Building upon this, we introduce TabCutMixPlus, an enhanced method that clusters features based on feature correlations and ensures that features within the same cluster are exchanged together during augmentation. This clustering mechanism mitigates out-of-distribution (OOD) generation issues by maintaining feature coherence. Experimental results across various datasets and diffusion models demonstrate that TabCutMix effectively mitigates memorization while maintaining high-quality data generation. Our code is available at https://github.com/fangzy96/TabCutMix.

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Zhengyu Fang
Zhimeng Jiang"
Huiyuan Chen
Xiao Li4345 Jing Li!
Abstract
demonstrate that TabCutMix effectively mitigates memorization while maintaining high-quality data generation. Our code is available at ht https: / //github. wtwpaau.ooo///ooooooomoo...o wwt.huuu..oooffagggggeagaa httm///aaggg//aatttttot
Tabular data generation has attracted significant research interest in recent years, with the tabu- lar diffusion models greatly improving the qual- ity of synthetic data. However, while memoriza- tion-where models inadvertently replicate ex- act or near-identical training data-has been thor- oughly investigated in image and text generation. its effects on tabular data remain largely unex- plored. In this paper, we conduct the first compre- hensive investigation of memorization phenom- ena in diffusion models for tabular data. Our empirical analysis reveals that memorization ap- pears in tabular diffusion models and increases with larger training epochs. We further examine the influence of factors such as dataset sizes, fea- ture dimensions, and different diffusion models memorization. Additionally, we provide the- oretical explanation for why memorization occurs in tabular diffusion models. To address this issue, we propose TabCutMix, a simple yet effective data augmentation technique that exchanges ran- domly selected feature segments between random same-class training sample pairs. Building upon this, introduce TabCutMixPlus, an enhanced method that clusters features based feature correlations and ensures that features within the same cluster are exchanged together during aug- mentation. This clustering mechanism mitigates out-of-disttibution (OOD) generation issues by maintaining feature coherence. Experimental re- sults across various datasets and diffusion models
1. Introduction
Tabular data generation has gained increasing attention due to its broad applications, such as data imputation (Zheng & Charoenphakdee, 2022; Liu et al., 2024; Villaizán-Vallelar et al., 2024), data augmentation (Fonseca & Bacao, 2023), and data privacy protection (Zhu et al., 2024; Assefa et al., 2020). Unlike image or text data, tabular data consists of structured datasets commonly found fields such as health- care (Hernandez et al., 2022), finance (Assefa et al., 2020), and e-commerce (Cheng al., 2023). Its heterogeneous and mixed-type feature space often poses unique challenges for generative models (Yang et al., 2024b; Zhang et al., 2023b), Recent advances have led to the development of various methods aimed at improving the quality synthetic tabular data, with diffusion models emerging as a particularly effec- tive approach (Zhang et al., 2023a; Kotelnikov et al., 2023). These models have demonstrated significant improvements in generating high-quality tabular data, making them a pow- erful tool for a wide range of applications.
TabDDPM
WS TACUUM
TabSyn
a-Precision
MLE
a-Precision
MLE
f-Recall
Mem. Ratio
f-Recal
Ratio
Equal contribution 'Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, USA Department of Computer Science Engineering, Texas A&M University, College Station, USA 'Department of Biochemistry, Case Western Reserve University, Cleveland, USA *Center for RNA Science and Therapeutics, Case Westem Reserve University, Cleveland, USA Depparment of Biomedical Engineering, Case Western Reserve University, Cleveland, USA. Correspondence to: Jing Li <jingli ewru.edu>.
Shape Score
Trend
Shape Score
Trend Score
Score
Figure 1. The overview performance of TahCutMix in TabDDPM and TahSyn for Default dataset. "Mem. Ratio" represents the memorization ratio.
Despite these advancements, an often-overlooked issue is the phenomenon memorization, where diffusion mod- els unintentionally replicate exact or nearly identical sam- ples from the training data. This not only introduces pri-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
vacy concerns but also hampers model generalization (Yoon et al., 2023; Kandpal al., 2022). While this phenomenon has been extensively investigated in image and text gener- ation (Karras et al., 2022; Carlini et al., 2021; Song et al., 2021; Ho et al., 2020), its occurrence and impact in tabular data generation remain relatively unexplored. This gap in understanding leads to a key question:
TabSyn (Zhang et al., 2023a), and balanced tabular diffu- sion (Yang et al., 2024b).
Memorization in Generative Models. Memorization has been widely studied in image and language do- mains (van den Burg & Williams, 2021; Gu et al., 2023; Huang et al., 2024). In image generation, re- seasrchers (Somepalli et al., 2023a; Carlini et al., 2021) found that diffusion models, like Stable Diffusion (Rom- bach et al., 2022) and DDPM (Ho et al., 2020), memorize portions of their training data at varying levels. Concept ablation (Kumari et al., 2023) is proposed to mitigate mem- orization via fine-tuning of pre-trained models minimize output disparity. AMG (Chen et al., 2024) uses real-time similarity metries to selectively apply guidance to likely duplicates. For text generation, text conditioning amplifies memorization risks, especially in large-scale language mod- els (Somepalli et al., 2023a;b; Huang et al., 2024). Gold- fish loss (Hans et al., 2024) randomly drops a subset of tokens from the training loss computation to prevent the model from memorizing. Memorization prediction (Bider- man al., 2024), i... predicting which sequences will be memorized before full-scale training, is investigated by an- alyzing the memorization patterns of lower-compute trial runs for early intervention. Although these patterns are evident in image and text generation, the impact of memo- rization on tabular data remains underexplored.
Does memorization occur in tabular diffusion models, and if so, how can it be effectively mitigated?
In this paper, aim to address this gap by conducting the first comprehensive investigation into memorization be- haviors within tabular diffusion models. Through rigorous empirical analysis, we examine how various factors-suu as training dataset sizes, feature dimensions, and model ar- chitecture- the extent of memorization. Additionally, we provide a theoretical exploration of memorization in tabular diffusion models, shedding light on the underlying mechanisms that lead to the issue of memorization in tabular data.
To mitigate memorization, we firstly introduce TabCut- Mix, a simple yet effective data augmentation technique that swaps randomly selected feature segments between training samples within the same class. Building on this, we propose TabCutMixPlus, an enhanced augmentation method that clusters features based on feature correlations and ensures that features within the same cluster are exchanged together. This clustering mechanism not only mitigates memorization but also mitigates OOD generation challenges by main- taining feature coherence during augmentation. Extensive experiments across multiple datasets and diffusion models demonstrate that TabCutMixPlus outperforms TabCutMix and other baseline methods in reducing memorization (See Figure. 1) without compromising the quality of the synthetic data, making it a practical solution for improving tabular data generation in real-world scenarios.
3. Memorization in Tabular Diffusion Models
Despite the development of numerous tigh-performing dif- fusion models for tabular data generation, it remains un- clear whether these models are susceptible to memorization. In this section, we introduce a criterion for detecting and quantifying the intensity of memorization in tabular data, Using this criterion, we explore memorization behaviors across various diffusion models under different dataset sizes and feature dimensions. We choose two state-of-the-arr (SOTA) generative models: TabSyn (Zhang et al., 2023a) and TabDDPM (Kotelnikov et al., 2023) for our preliminary memorization analysis. Furthermore four real-world tabu- lar datasets-Adull Default, Shoppers, and Magic-each containing both numerical and categorical features are in- cluded. The details of the datasets can be found in Section 5. Additionally, we provide a theoretical analysis to explain the mechanisms behind memorization in tabular diffusion models.
2. Related Work
Tabular Generative Models. Generative models for tab- ular data have gained attention due to their broad appli- cability. Early approaches like CTGAN and TVAE (Xu et al., 2019) leveraged Generative Adversarial Networks (GANs) (Goodfellow et al,, 2020) and VAES (Kingma, 2013) for handling imbalanced features. GOGGLE (Liu et al., 2023) advanced this by modeling feature dependencies us- ing graph neural networks. Inspired by NLP advancements, GReaT (Borisov et al., 2023) transformed rows into natu- ral language sequences to capture table-level distributions. More recently. diffusion models, originally successful in image generation (Ho et al., 2020), have been adapted for tabular data, as demonstrated by STaSy (Kim et al., 2023), TabDDPM (Kotelnikov et al., 2023), CoDi (Lee et al., 2023),
3.1. Memorization Detection Criterion
A quantitative criterion is essential for quantifying the mem- orization ratio-i.e the proportion of generated samples that are memorized by a model. In natural language pro- cessing, memorization is typically identified when a model
2

---

Understanding and Mitigating Mcmorization in Diffusion Models for Tabular Data
can reproduce verbatim sequences from the training set in response adversarial prompt (Carlini et al., 2021; Kandpal et al., 2022). However, such a verbatim definition is not directly applicable to image and tabular data, where the intrinsic continuous nature of pixels and features makes exact replication less meaningful.
Shoppers, and Magic). For each dataset, check the memorization ratio over the course of training of TabSSn and TabDDPM. Figure 2 illustrates the memorization ratio for both models. Based on our experiments, we make the following observations:
Obs.1: TabSyn exhibits faster convergence with more stable memorization ratios across all datasets compared to TabD- DPM. This trend is particularly prominent for the Default and Adult datasets, where TabSyn stabilizes its memoriza- tion rate after approximately 500 epochs, while TabDDPM continues to fluctuate over a much longer training duration, up to 4000 epochs.
Inspired by prior work in image generation (Yoon et al., 2023; Gu et al., 2023), we adopt the "relative distance ratio" criterion to detect whether generated sample is memo- rized replica from training data D in tabular dataset. Specif- ically, 2' is considered memorized if d( (z, (æ, D)) < d(xr, NN2 (xx, D)), where d(.,,) is the distance metric in the input sample space, (æ, D) represents i-th near- est neighbor of æ in training data D based on the distance '(.,)).
Obs.2: Although the converged memorization rates vary between datasets, the final memorization levels are rela- tively similar across both diffusion models. For instance, in TabSyn, the memorization ratio for Magic can reach up to 80%, indicating high memorization, whereas it stabilizes at 20% in Default, showing lower memorization. Similar trends are observed in TabDDPM, suggesting that while the training dynamics differ, the overall memorization capacity converges to comparable levels across models for the same dataset.
In the image generation domain, 12 norm is commonly adopted as the distance metric measure the sample simi- larity in the input space. However, this metric is not suitable for tabular data generation due to the mix-typed (categorical and numerical) input features. To address this, and inspired from mixed-type data clustering literature (Ji et al., 2013; Ahmad & Khan, 2019), we define a mixed distance .((,-) between generated sample x and real training sample x' as follows:
Mugeraar may
Tduus
Defect: 3Stym
Magis Teags
Shoporra Teua
d(0, x²)
M
1
norm
iEJnum num
Σ
(x: - xz1² xx1)3
Execch
0PM00M
1000
2.0
Dafauik 3P000MM
E0oth
130
Exech
1050
Mumer cuteooudoon Falle
Shoppon 70200T
Epech
+ Σ 1a2 & adj)
100
JEFear
2383
2000
4030
2307
5000
5000
30000
where Frum and Feat represent the index sets for numerical and categorical features, respectively; norm (dx) represents max-min normalization rescaling the distance values to
Figure 2. Memorization ratio curve of TabSyn and TabDDPM w.r.t. training epochs.
da dm))))(dx))
[0, 1] range using norm(d+) = mpx((d))-mi((4))) where & is sample pair distance index; M is the total number of features, such that [Fauml + [Fenel = M. In this equa- tion. (x{) represents i-th feature value for sample x(x'), 12x # x²)) is an indicator function that equals if æ; + x'j and 0 otherwise. In this paper, we use Eq. (1) to measure sample similarity and to quantify the memorization ratio in tabular data generation.
3.3. Impact of Training Dataset Size
Building on the findings from Section 3.2, where TabSyn demonstrated high training stability, we use TabSyn as the backbone model to explore the impact of training dataset size memorization in tabular data. We conduct exper- iment with four datasets (Default, Shoppers, Magic, and Adult), randomly downsampling the training samples to five different sizes: 0.1 %, 1%, 10%, 50%, and 100% of the original dataset. Figure 3 shows the memorization ratio for each dataset size over the training epochs. We make the following observations:
3.2. Effect of Different Diffuusin Models
In this subsection, we focus on examining the behavior of the two diffusion models (TabSyn (Zhang al., 2023a) and TabDDPM (Kotelnikov et al., 2023)) the memoriza- tion ratio across the four tabular datasets (Adult, Default,
Obs.1: Smaller training datasets consistently exhibit higher memorization ratios, as observed across all datasets when the training size is reduced to 0.1 1.. For some datasets, such as Shoppers, even moderate reductions in training size (e.g., 10%) lead to noticeable increases in memorization, whereas for others, such as Magic, the effect becomes prominent
'The factor & is an empirical threshold and widely adopted in image generation literature. Mem-AUC is also defined in Ap- pendix D.6
3

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
only at extremely small sizes (e.g., 0.1%)
function vz, log =0). The denoising score-matching
training process is formulated as:
Obs.2: The memorization ratio generally increases over training epochs before stabilizing, The final converged memorization ratio demonstrates strong dependency on training dataset size when the size is extremely small (e.g- 0.1%). For larger sizes, such as 10%, the dependency less pronounced for datasets like Magic and Shoppers, possibly due to the relatively larger sample pool. This observation suggests that the impact of dataset size memorization becomes increasingly critical as the dataset size decreases.
min E. Zorpp(Zo) (z. ( (2) - Vz, log log
(4)
where Vz. log logp(=z(zz) can be calculated according to 22, logp(zz -(1)
Regarding memorization of synthetic data in the latent space, have
Defeah
Adst
Phapzere
Proposition 3.1 (Guu et al., 2023)). Assume that the neural network can perfectly approximate the optimal score func- tion (z,, t) given by Eq. (6) and perfect SDE solver is applied in backward SDE. The generated sample in latent space zo will exactly replicate the latent embedding of the real sample in training data.
sic
Leoe
5do
3000
5:0
1000
Figure 3. Impact of dataset size among different datasets for Tab-
Syn model.
See proof in Appendix. B. Proposition. 3.1 demonstrates that under ideal conditions, the generated sample in latent space is an exact representation of a real training sample, which contradicts the empirical observation in TabSyn (i.e., not 100% memorization). There are several possible rea- sons for this discrepancy. First, the practical score-matching function learned by the neural network may not perfectly approximate the optimal score due to insufficient optimiza- tion or limited model capacity. Additionally, TabSyn uses a VAE to handle mixed-type tabular data, followed by latent diffusion for generation. As a result, even if the generated sample in latent space is identical a training sample, the final generated sample may differ due the randomness introduced by the VAE decoder.
3.4. Theoretical Analysis
In the previous section, we empirically investigate the mem- orization phenomenon in existing tabular diffusion models. However, the underlying cause of memorization in tabular diffusion models remains unclear. To bridge the gap, we applied theoretical analysis from image generation (Gu et al., 2023) to rationalize why memorization occurs in TabSyn (Zhang et al., 2023a), one of the SOTA tabular generative models.
In TabSyn, a variational autoencoder (VAE) is used map the input features æ into embedding = Encoder((z) in latent space. Subsequently, a latent diffusion is applied to generate samples in the latent space. The final synthetic data is generated via the decoder of VAE. For simplicity, we only consider latent diffusion in the analysis. Specifically, the following forward and backward stochastic differential equations adopted in the latent diffusion:
4. Methodology
Building on the memorization study presented in Section 3, we identify that the memorization in tabular diffusion mod- els remains a significant yet underexplored issue, limiting the diversity and utility of generated data. To address this problem, we propose two novel data augmentation strategies tailored for tabular data generation ?: TabCutMix and its enhanced version, TabCutMixPlus The pseudo-code of our proposed algorithm is in Appendix C.
2t #o + (t)e,, ((0,,),
(2)
dz, = (2)(()( t)dt + (t)
where 20- z represents the initial embedding from the encoder, 2, is the diffused embedding at time 6, and (t) is the noise level at time t. The score function s(z,, t) is defined as s(z,, t) = Zz, log log))(zz). and wi is the standard Wiener process.
4.1. Tab
TabCutMix generates a new training sample (ær, 9) by com- bining two samples (æ 34) and (x MR) that belong to the same class. In tabular data, "same class" refers to in- stances that share the same categorical target label, ensuring that the generated sample remains consistent with its origi- nal class and prevents label mismatches. The newly gener-
When the score function s(z,,t) t) is known, synthetic data can be sampled by reversing the diffusion process. In prac- tice, diffusion models train a neural network s0(z,, t) ap- proximate the score function s a(2,,1) !). However, score func- tion Vz log (z(z) is intractable since the marginal distribu- tion (z) = p(z,) is unknown. Fortunately, the conditional distribution (()))z) is tractable and can be used to train the denoising function to approximate the conditional score
Thheee strategies are inspired by the CutMix (Yun et al., 2019) data augmentation technique used in the image domain
4

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
ated sample is defined using the following mix operation:
5.1. Experimental Setup
Datasets. We use four real-world tabular datasets contain- ing both numerical and categorical features: Adult Default, Shoppers, and Magic. The detailed descriptions and overall statistics of these datasets are provided in Appendix D.1.
x- M GOEA M) DB
(5)
where M E [0.1] is a binary mask matrix indicating which features to swap between the two samples, 1 is mask filled with ones, and @ represents element-wise multi- plication. The portion of exchanged features À is sampled from the uniform distribution (4( 1). Each element of M is sampled independently from a Bernoulli distribution (A). In each training iteration, we first sample the class index E (1, C} using the class prior distribution and then randomly select two samples from that class.
Diffusion Models. We integrate TabCutMix with three ex- isting SOTA liffusion-based tabular data generative models, including TabDDPM (Kotelnikov et al., 2023). STaSy (Kim et al., 2023), and TabSyn (Zhang et al., 2023a). To the best of our knowledge, this work is the first to comprehen- sively evaluate both generation quality and memorization performance for these models.
4.2. TabCutMixPlus
Evaluation Metrics. We evaluate the performance of syn- thetic data generation from two perspectives: memorization and synthetic data quality. For memorization evaluation, we generate the same number of synthetic samples as the train- ing dataset and use Eq. (1) to calculate the distance between the generated and real samples. The generated sample is considered memorized if its closest neighbor in the train- ing data is less than 3/2 of the distance its second closest neighbor (Yoon al., 2023; Gu et al., 2023). Memorization ratio is defined as the proportion of generative samples that are memorized, using fixed threshold of 3. To comple- ment this, we introduce Mem-AUC in Appendix D.6, which summarizes memorization behavior by averaging the mem- orization ratio across a continuous range of thresholds. This metric provides a more comprehensive and robust evalua- tion, especially when the memorization behavior may vary under different threshold settings. To validate the use of the fixed 3 threshold in practice, we compute both Mem-AUC and the memorization ratio at 1. and analyze their correla- tion in Appendix E.9. The results reveal a strong positive correlation, indicating that the -hee-shre metric serves as a reliable proxy for the more holistic Mem-AUC. Further- more, as shown in Figure 11, we also compute the corre- lations among memorization ratios under thresholds 2. 1 and 1. and observe consistently high correlations across all threshold pairs. This further supports the robustness of the memorization metric under different threshold choices, and highlights 1/2 as a representative and stable threshold that bal- ances simplicity and practical effectiveness. For synthetic data quality evaluation, we consider 1) low-order statisties (i.e., column-wise density and pair-wise column correlation) measured by shape score* and trend score 5; 2) high-order metrics a e-prccision and 3-recall scores measuring the over-
While TabCutMix significantly reduces memorization, it may inadvertently disrupt inter-feature relationships, partic- ularly in highly correlated features. Such disruptions can lead to the generation of autoof-distribution (OOD) samples, thereby reducing the reliability of the synthetic data. To overcome this limitation, we propose TabCutMixPlus, an advanced augmentation strategy that preserves structural integrity by clustering features based on their correlations and performing swaps within clusters.
TabCutMixPlus identifies clusters of highly correlated fea- tures using domain-specific correlation measures and hier- archical clustering algorithm'. For numerical features, we use the Pearson correlation coefficient, while for categori- cal features, we employ Cramér's V (Cramér, 1999). For numerical-caategriccl feature pairs, calculate the squared ETA coefficient (Richardson, 2011) 1). Each cluster is treated as an atomic unit during the augmentation process, ensuring that only features within the same cluster are exchanged together. This clustering approach maintains the relation- ships among highly correlated features, thereby mitigating the risk of generating OOD samples.
By ensuring feature coherence during augmentation, Tab- CutMixPlus strikes a balance between reducing memoriza- tion and maintaining high-quality synthetic data. Extensive experiments on OOD detection, detailed in Appendix D.4.6, demonstrate that TabCutMixPlus significantly outperforms TabCutMix and generates superior data utility.
5. Experiments
In this section, we extensively evaluate the effectiveness of TabCutMix and TabCutMixPlus across several SOTA tabular diffusion models in various datasets and compared other augmentation methods Mixup (Zhang, 2017; Takase, 2023) and SMOTE (Chawla et al., 2002).
'Shape Score measures how closely the synthetic data matches the distribution of individual columns the real data using Kolmogorov-Smimne (KS) test.
'Trend Score assesses whether the relationships or correlations between pairs of columns in the synthetic data similar those in the real data
5

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Table The overview performance comparison for tabular diffusion models on more datasets. "TCM" represents our proposed TabCut- Mix and "TCMP" represents TabCutMixPlus "Mem. Ratio" represents memorization ratio. "Improv" represents the improvement ratio on memorization.
Methods
Mem. Rato (%)
Improx:
ML
o-Precisions5)) It
e- A-Recall((k3)
Shape Scoeer%)
Trend (oore(5))
DCR(S)
STaSy
17:57= = 0.53
76,48 + 1.18
87.78 + 5.20
35.94 + 5.45
90.27 + 2.45
89.58 = 1,35
67.68
6.89
50.30 + 0.36
STaSy+Mihup
17.80 a 0.99
- 1.80%
15,69 1.26
82:65 10.01
37.94 2.57
85.77 1: 4.02
86.49 1 4,66
50.81 1: 6,01
50,66 1.39
STASSASMOTE
15.98 =: 0.04
9.07%
15,41 a: 0.95
86.75 1: 5.80
32,95 2.93
$7.89 4: 5.17
32.54 a 0,91
48.57 2: 5.90
51.39 2.23
STISy-TCM
14.51 0.46
17.44%%
75.33
1.30
86.04 11.55
30,13
5.07
90,30 3.88
89.85 3.16
49.51 6:33
50.19 0.99
STAS,4TCMP
10.53 2.00
11.59%
76.30 0.57
90.83 + 4.51
32.81
1.37
91.40 + 0.77
92.08 + 2.04
50.43 + 2,00
50.70 + 1.94
TUNDDEM
19.50 0.45
16.79 0.69
98.15 1: 1.45
44.41 0,70
97.58.1 0.95
94,46 1 0.68
91.85 1. 6.94
49.12 = 0.94
Defoalt
THHDPMM-Mrup
18:40 - 0.71
4.50%
77.18 0.35
93.30 4.16
42.59 1.13
95.34
90.32
92.59
52.36 1.57
TOOOPM--MOTRE
17.46
9.00%
76.92 0.35
91.19 +: 0.68
40.52 0.65
94.89 + 1.46
28.63 2.28
72.73 0.60
50.95 0.38
TANDDEM-TCM
16.70
13.26%
76.47 0.60
97.30 0.46
97.27
50.23 0.53
TANDPM-TTCMP
18:00-0:34 0.34
0.88%
76.92 0.17
0.25
0.55
97.37 0.09
96.64
49.75 0.32
TahSyn
00.03
77.00 0.33
0.13
46.76 0.55
98,96
96.50
98:27
$1.09 0.50
Musuy
0.33
2.65%
77.24 0.42
99:05 0.45
46,94 0.19
97.84 0.16
96.52
0.17
1..5
6.93%
17.24.10.4
93.00 0.29
42.78 0.64
96,59 1
32,70 1 0.23
$1.38 0.90
50.79 0,66
TahSyn-TCM
16.50 + 1,36
10.165
76.84 + 0.34
96 16 1.24
40.00 + 2.46
98.02 + 1.62
96.51 + 1.42
97.65 0.65
51,16 1.82
TCMS
17.90 =: 0,28
12.48%
77.17 d: 0.51
97.61 à: 0.27
44,46 2 0,00
99,03 à: 0.08
96.30 :: 1,48
98.16 a: 0,65
$1.20 0,50
STaSy
26,02 à: 0.89
90.54 2: 0.17
85.79 +: 7.85
34.35 2.4.
89.14 3: 2.29
86,00 :: 2.90
51.59 14.87
51,46 d: 0.39
MTrSy+Mixup
24,80 1: 1,30
4.37% 1
90.74 1. 0.06
90.00 1 191
2.44
90.28 1: 1.69
87,56 1: 1.06
52.61 1. 6.52
50.08 0.59
22.92 1.77
11.915 L
90,50 + 0.34
85.81 + 11.30
$2.11 5.113
86.91 + 0.81
84.36 2,36
45.12 8.52
50.46 0.30
ST1SyIICM
20.80 4: 1.33
19.71%
90.45
0.30
85.39 A: 1.61
31.24 0.97
58.33 1: 3.63
85.39 1: 4.00
45.49 4.78
50.92 4 0.39
STSS/+TCMP
21.45 2.60
17.00%
90.72 0.06
86.71 1: 4.12
32.63-1:1111
89.60 1.55
$6.05 :: 2.44
49.12 9.95
50,751. 0.59
TRNDDPM
+
91.09 0.07
93.58 + 199
98.84 21.52
97.78
94.63
51.56 0.54
TaNDDMM-Mixp
0.41
3.14%
90.82 0.12
95.78 + 0.68
47,65
98.02 + 1.08
96.78 1.38
93.65
50 0.96
TONDDMM-SMOTE
28.98 a
6.00%
90.41 0.36
94.93 1.72
0.06
93,40 1.12
90.76 1.76
80.75
51.82 à: 0.56
TANDUMM-TCM
0..19
11.16%
91.15
96.97 0.06
16.03
50,99 0.65
TAAD
2.13
15.88%
90.54 0.17
92.
6.97
3:7.4
4:21
91.50 =
34.76 10.12
0,50
TubSyn
0.223
91.13 0.09
99.31 0.39
48,00 0.222
99.33 4
98.19 11
98.68 0.41
50,42 0.27
TahSyn-Miup
26.29 0:28
3.305
90.75 + 0.24
98 + 0.81
45.73 2.66
98.30 0.90
97.91 + 0.12
98.06 + 2.22
50.97 1.10
SMOTE
2.11
7.30%
89.97 0.76
98.60 0.50
44.72 H 0.45
94.47 à: 0,57
91.74 =: 0.42
$2.55 0.71
48.42 1.78
TalSyn-TMM
27.03 +
7.60%6 1
91.09 + 0.17
99.04 + 0.42
44.95 0.42
99,40 0.07
98.51 = 0.08
89.18 1.94
50.67 0.11
TCAS
25.99 a 0.52
11.17%
90,96 a: 0.16
98:43 1 1.04
43.23 2.96
98.58 1 0.91
96.53 4: 1.47
93.39 1.
50.30 1. 0.78
STISy
25.51 0.3.
91.26 + 0.23
88:02 + 3.54
34.58 + 1.84
68.15 0.20
89.10 19.10 0.53
47.85 8:48
51.68 + 0.56
MisSs-
24,90 1,20
2.815% L
91,79 t: 0.58
87:03 +: 5.46
38,48 4.54
67,14 1.87
88.72 1.42
47.42 4.84
50.36 2.45
STASSASMOTE
22.52 +:2::110 1.51
11.78% L
91.31 + 1.21
K5.22 + 3:20
30.53 + 1.65
K1.22 + 2.25
84.74 0.78
38.92 + 2.63
46.47 = 0.95
ST1Sy+ICM
22.78 1. 0.69
10.71%
90.56 4 0.44
86.66 1 4.18
34.08 1 1.46
3.77
$6.56 1. 4.26
50.08 1 6.30
50.61 1. 0.41
STVS/+TCMP
22.10 1.21
13.08%
91.37
0.65
85
2.66
34.11 2.08
87.38 0.90
88.61 + 1.64
$2.42 2.65
51.19 0.95
TRADDPM
31.37 +
92.17 0.32
M3.16 1.58
52.57
97.05 à: 0.46
92.92 =
86.74 0.63
51.36 0.63
Stappos
TaNDDFM-Mixup
27.45
12.00%
91.44 1.37
94.80 0.68
92.14
89.31.1 3.91
$2.34
46.85 5.1
TADDPM-SMOTR
15.07%
60.96 0.95
94.41
45.22 3.26
90.75
83.00+ 2.47
64.05
51.94 1.52
TRDDFM--TCM
20.36 à
18.51%
92.17 0.25
94.41 1.49
50,05
97,18 0.34
93.951 0.51
36.96 0.50
47.52
TUMP
à
9.12%
92.119 0,99
93.45 1.65
97.31 0.73
94.791 0.30
87.02 2.04
50.83 0,59
TahSyn
27.65 0,110
91,76 0,66
99.20 0.29
47,79 1.77
98,54 0.19
97.83 1 0.10
95.44 0.39
52.50 0.44
TabSyn-Misup
28.01 + 0.46
-1.18%
92.02 + 0.29
98.57 + 0.32
48.17 0.84
97.99 + 0.09
97.08 + 0.14
98.37 0.47
$1.50 2.63
TMOSE
26,43 a 0.85
4.54%
91.96 à: 1.02
95.27 a: 0.9
44.57 1 0.24
94,55 à: 0.48
94.59 a 0.08
79.89 1.22
49.99 0.81
TubSyn-TCM
25.55 0.18
8.30%
91.43 0.36
99 11 0:28
45.98 0.90
98.56 0.10
97.85 0.06
97.28 1: 2.41
49.92 1.59
TrlSyu-TCMP
25.93 4t 0.23
6.58%
91.75 0.47
99.34 + 0.5
46.48 + 0.77
98.60 0.14
97.77 = 0.00
97.40 :: 0.57
50.21 1:
all fidelity and diversity of synthetic data;5 3) downstream tasks performance machine learning efficiency (MLE)?. We
tion. The reported results are averaged over 5 independent experimental runs. More details on evaluation metrics can be found in Appendix D.4.
report AUC in Table 1, i.e., the testing performance (e.g., AUC) on real data when trained only on synthetically gener- ated tabular datasets; 4) C2ST (Classifier Sample Test) evaluates data quality by measuring how well classifier can distinguish real from synthetic lower accuracy sug- gests better distributional alignment; 5) DCR (Distance to Closest Record) measures privacy risk by quantifying how closely synthetic sample resembles training vs. holdout samples differences indicate better privacy preserva-
5.2. Memorization and Data Quality: Overall Evaluation
To thoroughly compare the memorization and data gen- eration quality, we incorporate several metrics, including the memorization ratio, MIFE, a-precision, f-reeall, shape score, and trend score. We report these metrics results of applying TabCutMix and TabCutMixPlus to three SOTA generative models (i.e., STaSy, TabDDPM, and TabSyn) across four datasets in Table. 1. We observe that:
'Plcsse see more details on high-order metrics in Ap- pendix D.4.3
Plcass see more details on MLE in Appendix D.4.2
Obs.1: TabCutMix and TabCutMixPlus significantly reduce
6

---

Understanding and Mitigating Mcmorization in Diffusion Models for Tabular Data
Adult
Default
Shoppers
Magic
0
5yb (Ratie: 29.26%)
Tab5yn (Ratio: 20.11%;
Tab5yn (Ratio: 27.68%)
0.
TobSyn (Ratio: 80.01%)
TabCurtiiaPuus (Ratio: 75.99%)
TaiCuutMxPlus (Ratio: 17.60%)
TabCurMixxlus (Ratio: 25.93%)
TabCurMixPlus (Ratio: 76.46%)
Density
Density
0.2
Density
Density
0.50
0.2
0.2
0.1
0.1
0.25
0.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
0.0
0.0
0.0
0.5
1.0
0.00
0.0
0.5
1.0
Distance Ratio
Distance Ratio
Distance Ratio
Distance Ratio
Figure 4. The nearest-neighhor distance ratio distributions TahSyn with and without TabCutMixPlus across different datasets.
a further reduction in memorization. For example, in the Magic dataset, TabCutMixPlus reduces the memorization ratio from 80.01% % to 76.46%, generating samples that are less tightly aligned with the real data D.
Tt
Obs.2: The distance ratio distributions with TabCutMix- Plus exhibit a bipolar pattern, with high probabilities near 0 and 1. However, TabCutMixPlus improves the spread of the distribution by reducing the probability mass near 0 and increasing it near 1. This indicates that TabCutMix- Plus better balances memorization reduction and diversity improvement.
Figure 5.
The visualization of real and generated samples of Tab- Syn with and without TabCutMix across different datascts.
the memorization ratio across all models and datasets. For example, in Shoppers dataset, TabCutMix and TabCutMix- Plus reduce the memorization ratio by 8.30% and 6.33% for TabSyn model. Although the actual reduction rate varies over dataset and model combination, the overall results in- dicate that TabCutMix is more effective in mitigating mem- orization than TabCutMixPlus.
5.3.2. VISUALIZATION OF REAL AND GENERATION SAMPLES
We visualize the distribution of real and generative samples for four datasets (i.e., Adult, Default, Shoppers, and Magic) in Figure. 5. For each dataset, we sample 100 generated sam- ples while preserving the memorization ratio consistent with that of the entire generated dataset. For each of these 100 samples, then select their nearest and second-nearest real samples from the training set to visualize. Using t-SNE, we embed both the generative samples and their corresponding nearest and second-nearest real samples from the training data. We make the following observations:
Obs.2: TabCutMixPlus demonstrates higher data quality compared to TabCutMix across various metrics, datasets, and diffusion models. For example, in the Default dataset, TabCutMix achieves an MLE of 76.84%, a-precision of 96.16%. while TabCutMixPlus slightly improves it to 77.17% and 97.61% 31% on TabSyn model, suggesting its supe- rior ability to generate high data utility.
Obs.1: In the TabSyn model, memorized generative samples (marked with x) are tightly clustered around their nearest real samples (shown in blue), indicating high level of memorization. This clustering is particularly pronounced in the Magic dataset, where most generative samples are con- centrated near their nearest neighbors, corresponding to a memorization ratio of 80.01%. %. In contrast, non-memorized samples are more dispersed, demonstrating better diversity.
5.3. A Closer Look at Memorization
5.3.1. 1. DISTANCE RATIO DISTRIBUTION
We analyze the distribution of the nearest-neighhor distance
ratio, defined as r(æ) NN, NNv( to assess the severity of memorization. more zero-concentrated ratio distribution indicates a more severe memorization issue, as the generated sample x is closer to real sample in the training set D. Figure 4 illustrates the distance ratio distribution for both the original TabSyn and TabSyn with TabCutMixPlus, and we observe the following:
D)
Obs.2: While the visual impact of TabCutMix is subtle, we observe that the generative samples exhibit a slightly broader distribution, particularly in datasets like Default and Shoppers. This suggests a reduction in tight clustering around real samples, which correlates with the reduction in memorization ratios. However, in some datasets like Magic, the visual distinction remains modest, indicating that TabCutMix quantitatively reduces memorization.
Obs.1: TabCutMixPlus shifts the distance ratio distribution further away from zero compared to TabSyn, indicating
7

---

Understanding and Mitigating Mcmorization in Diffusion Models for Tabular Data
Table 2. The real and generative samples by TabSyn and TabSyn with Mix and TabCutMixPlus in Adult dataset. TCM and TCMP
represent TabCutMix and TabCutMixPlus, respectively.
Sangles
Age
frleg
Educaions
Edocelass
Maxal Botts
Occoptiaas
Retatiolii
Rac
Sex
Ciptall Giin
Caplal Lros
Heurype
Natio Cuunity
Takkyn
45.0
Priste
307
RS gl
9.0
Dieased
Sales
Unmanial
Fonale
0.0
0.0
45.9
Unieel-Sutts
555
TCM
36.0
Privae
7037
35 gial
9.0
Dinnsed
Sda
Uaranial
Foralo
0.0
0.0
50.0
Centaty
Heal
2003
Prvass
Seme cellggs
Neve-mamed
Deacchil
Female
23.9
Unnen-Stacc
4403
Preaze
226.0
Scar collegs
100
Nev murned
Canheoaar
Owa-child
whit
Female
40.9
Heat
5733
7530
Mamaa choppex
sata
Britza
201112
010
R0.37
Unaett-
Landan
500
Scli-cmp-ent
151815.17 17
Dectome
Mamadlone croppennc
Sala
Hadeadd
Maic
150340
Unzea-Suatt Staic
5.4. Case Study on Adult Dataset: Real vs. Generated
5.5. Hyperparameter Study: Impact of Augmented Ratio
Samples
Table. 2 provides comparison between real samples, syn- thetic samples generated by TabSyn, and synthetic samples generated with TabSyn and TabCutMix (w/ TCM) for the Adult dataset. We report key feature (e.g., age, Workelass, education, marital status, occupation, income, etc.) values of two real samples and the corresponding nearest genera- tive samples to study the quality and characteristies of the generated data.
In this section, we investigate the effect of the augmented ratio in TabCutMix on the memorization rate. Figure 6 and Table 3 present the memorization ratio for different aug- mented ratios across two datasets, Default and Shoppers. We test various augmented ratios, including 0%, 10%, 20%, 30%, and 100%, to analyze their impact on the memoriza- tion behavior over training epochs.
We observe that the memorization ratio decreases con- sistently as the aug- mented ratio in- creases. Without
Obs.1: : The results 2 suggest that TabSyn alone tends to generate samples that closely resemble real data, raising concerns about memorization. For instance, the top real sample has an age of 47.0 years. TabSyn generates a sam- ple with an age of 48.0 years, which is nearly identical. Similarly, other features like workelass, marital status, and occupation are also closely reproduced.
Table 3. Different augmentation ratios (Aug. Ratio) for TabCutMix.
Aug. Ratio
Default
Shoppers
0%
20. 60.11% 11%
27.68%
augmentation (i.e.,
10%
19.45%
27.20%
0% augmented ra-
20%
17.82%
26.05%
tio), the memoriza-
Obs.2: When TabCutMix 2 is applied, the generated age for the top sample changes to 36.0 while the key relationships between other features such as marital status, occupation, and workelass are preserved. For instance, for the work- class feature, all samples across real data, TabSyn, and TabSyn+TCM show "Private," and for the relationship fea- ture, they show "Unmarried" "Own-child," depending on the context. For the bottom sample, prior to applying TabCutMix, the distance ratio is 0.17, which is less than the threshold of 1 and thus considered memorized. However, after applying TabCutMix, the closest sample achieves a distance ratio of 0.88, significantly exceeding the 1 thresh- old, indicating much lower likelihood of memorization. This demonstrates that TabCutMix can introduce diversity in specific features like age while preserving categorical feature relationships.
30%
16.86%
25.38%
tion ratio is higher, 100%
15.34%
22.06%
stabilizing around
20. 11% for Default and 27.68% for Shoppers. In contrast, the 100% augmented ratio (purple curve) yields the lowest memorization ratio, stabilizing at approximately 15.34% for Default and 22.06% for Shoppers. This suggests that higher augmented ratios introduce more data diversity, ef- fectively reducing overfitting and preventing the model from memorizing specific training samples.
6. Conclusions
In this study, first investigate memorization phenom- ena in diffusion models for tabular data using quantitative metrics. Our findings reveal the prevalent memorization be- haviors in existing tabular diffusion models, with the memo- rization ratio increasing as training epochs grow. We further study the effects the diffusion model instantiation, dataset size, and feature dimensions through the lens of memoriza- tion ratio and observe the heterogeneous trend dependent on the dataset. The theoretical analysis provides new insights into why memorization occurs within the SOTA model Tab- Syn. To address this issue, we propose TabCutMix, which reduces memorization by swapping feature segments be- tween samples, and TabCutMixPlus, which improves upon this by clustering correlated features to preserve feature relationships and address duitrf-distribution challenges. Ex-
Default
Shoppers
Memorization Ratio
0.2
Memorization Ratio
0.2
Augmented Ratio
0.1
0%
10%
0.1
20%
30%
100%
0.0
0
500 Epoch
1000
500
1000
Epoch
Figure 6. The memorization ratio v.s. training epochs with differ- ent augmented ratios for TabSyn.
8

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
periments demonstrate that both TabCutMix and TabCut- MixPlus significantly mitigate memorization while main- taining high-quality synthetic data generation. Our work not only highlights the critical issue of memorization in tabular diffusion models but also offers effective solutions with TabCutMix and TabCutMixPlus.
generators. In The Eleventh International Conference on Learning Representations, 2023.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security ) pp. 2633-2650, 2021.
Impact Statement
Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, C., Liu, D., and Xu, C. Towards memorization- free diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8425 8434, 2024.
Acknowledgements
This work is supported in part by NSF CCF-2200255, NSF CCF-2006780, NSF IIS-2027661 NIH UOLAG073323, NIH R01HG009658, NIH 1R01HL159170 and NIH IROINR02010SH This work also made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing Case Western Reserve University.
Cheng, K., Li, X., Wang. Z., Zhang, C., Huang, B., Xu, Y. E., Dong, X. L.., and Sun, Y. Tab-cleaner: Weakly supervised tabular data cleaning via pre-training for commerce catalog. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp- 172-185, 2023.
References
Cramér, H. Mathematical methods of statistics, volume 26. Princeton university press, 1999.
Ahmad, A. and Khan, S. S. Survey of state-of-the-are mixed data clustering algorithms. IEEE Access, 7:31883-31902, 2019. doi: 10. C0L109/AACCESSS2092229333
Fonseca, J. and Bacao, F. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1):115, 2023.
Alaa, A.. Van Breugel, B., Saveliev, S., and van der Schaar, M. How faithful is your synthetic data? sample- level metries for evaluating and auditing generative mod- els. In International Conference Machine Learning, pp. 290-306. PMLR, 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M.. Xu, B. Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(1 1):1111:1444494 2020.
Assefa, S. A., Dervovic, D., Mahfouz, M., Tillman, R. E., Reddy, P., and Veloso, M. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Pro- ceedings of the First ACM International Conference on Al in Finance, pp. 1-8, 2020.
Gu, X., Du, C., Pang, T., Li, C., Lin, M., and Wang, On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.
Hans, A., Wen, Y., Jain, N., Kirchenbauer, J., Kazemi, H., Singhania, P., Singh, S., Somepalli, G., Geiping, J., Bhatele, A., et al. Be like goldfish, don't memorize! mit- igating memorization in generative llms. arXiv preprint urXiv:2406.10209, 2024.
Azizmalayeri, M., Abu-Hanna, A., and Ciná, G. Un- masking the chameleons: A benchmark for out-of- distribution detection in medical tabular data. arXiv preprint arXiv:2309 16220, 2023.
Biderman, S., Prashanth, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Hernandez, M., Epelde, G., Alberdi, A., Cilla, R., and Rankin, D. Synthetic data generation for tabular health records: A systematic review. Neurocomputing, 493: 28 45, 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 18.6840-68511 2020.
Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M., and Kasneci, Language models are realistic tabular data

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Huang, J., Yang, D., and Potts, C. Demystifying verbatim memorization in large language models. arXiv preprint MrXiv:2407.17817, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ji, J., Bai, T., Zhou, C., Ma, C., and Wang, Z. An improved k-prototypes clustering algorithm for mixed numeric and categorical data. Neurocomputing, 120:590-596. 2013.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion art digital forgery? investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6055 2023a.
Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMLR, 2022.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Understanding and mitigating copying in diffusion models. Advances in Neural Information Pro- cessing Systems, 36:47783-47803 2023b.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-basee generative models. Advances in neural information processing systems, 35: 26565 26577, 2022.
Song, Y., Sohl-Dickstein, J.. Kingma. D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Kim, J., Lee, C., and Park, N. Stasy: Score-based tabular data synthesis. In The Eleventh International Conference on Learning Representations, 2023.
Kingma, D. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Takase, T. Feature combination mixup: novel mixup method using feature combination for neural networks. Neu- ral Computing and Applications, 35(17):12763-12774 2023.
Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tabddpm: Modelling tabular data with diffusion mod- cls. In International Conference on Machine Learning, PMLR, 2023.
Ulmer, D.,, Meijerink, L.., and Cinà, G. Trust issues: Uncer- tainty estimation does not enable reliable ood detection medical tabular data. In Machine Learning for Health, pp. 341-354. PMLR, 2020.
pp.
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y. Ablating concepts in text-to-image image dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 22691-22702, 2023.
van den Burg, G. and Williams, C. On memorization in probabilistic deep generative models. Advances in Neural Information Processing Systems, 34:27916-27928, 2021.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Villaizán Vallelado, M., Salvatori, M., Segura, C., and Arapakis, I. Diffusion models for tabular data impu- tation and synthetic data generation. arXiv preprint 2rXiv:2407.02544 2024.
Lee, C., Kim, J., and Park, N. Codi: Co-evolving con- trastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp. 18940-18956. PMLR, 2023.
Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veera- machaneni, Modeling tabular data using conditional gan. Advances in neural information processing systems, 32, 2019.
Liu, T., Qian, Z., Berrevoets. J., and van der Schaar, M. Goggle: Generative modelling for tabular data by learn- ing relational structure. In The Eleventh International Conference on Learning Representations, 2023.
Yang. J., Zhou, K., Li, Y., and Liu, Generalized out-of- distribution detection: A survey. International Journal of Computer Vision, pp. 1-28, 2024a.
Yang, Z., Guo, P., Zanna, K., and Sano, A. Balanced mixed- type tabular data synthesis with diffusion models. arXiv preprint arXiv:2404.0822. 2404.044444 2024b.
Liu, Y., Ajanthan, T., Husain, H., and Nguyen, Self- supervision improves diffusion models for tabular data imputation. arXiv preprint arXiv:2407.180133 2024.
Yoon, T., Choi, J. Y., Kwon, S., and Ryu, E. K. Diffusion probabilistic models generalize when they fail to memo- rize. In ICML 2023 Workshop on Structured Probabilistic Inference {{&} Generative Modeling, 2023.
Richardson, J. T. Eta squared and partial eta squared as mea- sures of effect size in educational research. Educational research review, 6(2):135-14) 2011.
10

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy train strong clas- sifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision,
pp. 6023-6032, 2019.
Zhang. H. mixup: Beyond empirical risk minimization. arXiv preprint 171.:1710.09412, 2017.
Zhang. H., Zhang, J., Srinivasan, B., Shen, Z.,, Qin, X., Faloutsos, C., Rangwala, H., and Karypis, G. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.0940 2023a.
Zhang, Q. Wu, C., Xia, S., Zhao, F., Gao, M., Cheng, Y., and Wang. G. Incremental learning based on granular ball rough sets for classification in dynamic mixed-type decision system. IEEE Transactions Knowledge and Data Engineering, 35(9):9319-993) 93129333, 2023b.
Zheng, S. and Charoenphakdee, N. Diffusion models for missing value imputation in tabular data, arXiv preprint a2X 17128, 2022.
Zhu, C., Tang, J., Brouwer, H., Pérez, F,, van Dijk, M., and Chen, L. Y. Quantifying and mitigating pri- vacy risks for tabular generative models. arXiv preprint arXiv:2403.07842, 2024.

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
A. Proposition A.1
For the denoising score matching objective, we have the following result":
Proposition A.1. For empirical denoising score matching objective Eq. (4) with training data {Zn|n 2, ,.,, ,NN.. the optimal score function is given by
N
21
(z,,)
=
Σ
exp
20²
n=1
ス, - 2ell2 2t
2n -
X
exp
(6)
o²
n=1
Proposition.A.1 provides closed-form expression for the optimal score matching function given finite training set. In this section, we prove the close form of optimal score matching function sp(zt. t). Note that the objective of denoising score matching is given by
min EZ Za R (00(2:))) Vz, 1ogp(z;(20)11
(7)
8
Note that the score function be simplified as
V log p(z()z)) = p(z)|=0)
V Vz.P(zı(zu)
(-
2: -:--0) 200 -r(21(z))
p(z)|z0) 1
02(1)
(z0 + o [()e -20)= 20)
(8)
=21(1
((t)
Additionally, the noise sample =o)+ we have of denoising score matching as follows:
2.-2, and de o(1)
džg o(
We can obtain the empirical objective
|| N(e I)de
Cemp
Σ
s0(z,,
o(1)
N
N
nn 21 (t)
89 (g t)
N ((z 2n, (to)))
(9)
The minimization of empirical loss Lemp is a convex optimization problem. Therefore, the optimum can be obtained via first-order gradient w.r.t score function 88 (ze, t):
0
Σ N n=1
s0(2,,1) 1)
2n Ze (t)
N(z 0
2
Zn 21 S1IN En, oz
(so(Ze;)) ()
02(1)
(z, En -2: 02(1)
N (z:: 2n, (2)) 80
(z., (z),))
(10)
'The analysis is closely related to prior work (Gu et al., 2023) in image generation, where similar analysis performed in different generative models. Our work specifically addresses tabular data with mixed feature types by combining VAE with latent diffusion handle tabular data.
12

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Therefore, the optimal function be written
Σn (2(: n, 0 (2)I)
2n-2 2
(21,)
0-(6)
En=1 (z (((]))
| Zr 21112
-1
N
1
21 1
In 2
exp
exp
(11)
20²(1)
(12(1)
(2
B. Proof of Proposition. 3.1
Consider the reverse process a diffusion model defined by the score function s0(2, t) and the following backward stochastic differential equation (SDE):
dz, = (0) t)dt + (f)d)w
(12)
where wi is standard Brownian motion, and (t) are noise ratio at time instant t.
For solving this backward SDE given optimal score function sg( (z,, t), we consider the following steps:
Step 1: Euler Approximation. We use Euler approximation for backward SDE via sampling multiple time steps 0= = to < f = 7 < t2 = 2r < ... < fn = nt = T. where т is time sampling resolution and small value indicates low approximation error. Using an Euler discretization, the backward SDE can be approximated at discrete time steps ln leading to the following update rule:
Zr, 2t,, (0(() 1-t,11
((n,,))(Ln - (n41) (0) - -.-(nn. la41),
(13)
Step Update Rule Calculation.
Next, we calculate the update rule considering infinite short time resolution 7 - 0,
1-tii =00(ln+1) ((x)) =o(b+11)
lim
()(()o(()
(14)
An 1mm1-10
bn Int1
then we have
20 o((tt)))))))))))))))))))))
tt
+ (o(t)) - or (fn+1)) (tn tn+1)
(15)
For to 0, it is easy to obtain (t) 0, the generated sample in latent space 2o is giving by
2g z, + 200 (T
(16)
Step 3: The generated sample in latent space under - O. the optimal solution, we have
When the denoising score function perfectly approximates
N
N | 2e113
8(2,, (z) =
exp
Σ
exp (
In 2
(17)
2o (t)
20 20²
0² (c())
n=1
n=1
Subsequently, we consider the optimal score function under - Suppose the nearest neighbor of is 2m we have
NN, D),
z 2mll33 - || -2n113 < 0,
for n + m.
(18)
Define distribution:
NN
| 2113
exp ( こ - 2111
(= 2n) = Σ
exp (-
(19)
2 20² (1)
20
nm
n=1
13

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
where n 1,2, NN.Note Note that (T) 0 if O. It is easy calculate
7mmm Zr
|| 2r
11
lim (r 3m) x-0
lim 7-0
exp
exp
(T²(T)
20 (
11
27112
lim
cxp
7-0
20²(()
|| 1r112 112
1 lim
Σ exp (r)
(20)
20 (T)
mytm
similarly, we have, for any n' o m,
lim p- ), O. T-0
(21)
According to the above equations, the optimal score function is given by
Zm 2t
lim $5(2,,
(22)
(t
and the generated sample in latent space zo is as follows:
lim zo T-0
= lim 2, + +202(T))(Zz 1) + (T)
T-0
= lim 2, + (T) 3m - #t + 2TT=2(T) e o2(1)
T-0
2
lim Z,:
(23)
7-+
Therefore, have lim 2o 2m NN (z,, D).
T-0
To summarize, under the assumption (1) the neural network can perfectly approximate the score function s(2,, () 8 (Zt, (2) perfect SDE solver with infinite time solution (r - 0), the generated sample 20 replicates one of the training samples from the dataset D.
C. Algorithm
In this section, we provide an algorithmic illustration the proposed TabCutMix and TabCutMixPlus in Algorithms and respectively. MtbCuutix/TabCuuMiiee the hyperparameter 7n determines the augmentation ratio, i.e., the number of augmented samples over the whole number of the original training samples.
Algorithm 1 Pseudo-code of TabCutMix
Require: Training set DD, Number of samples N
1: Augmented sample set D = 0
2: for = 1 to N do
D Keep class ratio after augmentation. 4: Sample (,) and (æ yn) from class in D; Rannomly select training samples from the same class.
3: Sample class e from (I....,C) with prior class distribution;
5: Sample À ~ Unif(0, 1) and sampling binary mask M with Bernoulli distribution BemnA); Proportion of features
to exchange.
6:
+
M ② æA + (1 - M) @ DB:
Mix the features based on binary mask M. Assign the label of the new sample. & Save the augmented sample.
7:
- c
8:
DU
9:
end for
return New Training Set D U D
14

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Algorithm 2 Pseudo-code of TabCutMixPlus
Require:
Training set D, Number of samples N
1: Augmented sample set D = 0
2: Calculate correlation metrics for features: (a) Pearson correlation coefficient for numerical feature; (b) Cramér's V
based on contingency tables for categorical features; (c) ETA coefficient for numerical-categoric pairs.
3: Perform hierarchical clustering on features using correlation metrics;
>> Group features based on similarity,
4: for = 1 to N do
5: Sample class e from {I,..-, C} with prior class distribution;
D Keep class ratio after augmentation.
6: Sample (z Ma) and n. Vn) from class in D; Raandoly select two training samples from the same class.
7: for each cluster do
8: Sample À ~ Unif(O, 1) and sampling binary mask Ms with Bernoulli distribution (A);
Proportion of
features to exchange within cluster k.
9:
Tt 4- M3 @ AAA + (1 - M₁) @ BB
Mix features in cluster & based on binary mask Mg.
10:
Add Øx to x;
end for
12:
" - C
Assign the label of the new sample. Save the augmented sample.
13:
D (U.
14:
end for
15:
return New Training Set D U D
D. Experimental Details
We implement TabCutMix and all the baseline methods with PyTorch. All the methods are optimized with Adam optimizer.
D.1. Datasets
We select datasets, of datasets come from UCI Machine Learning Repository: Adult, Default, Shoppers, Magic, and Wilt. The other two Cardio and Churn Modeling. All datasets associated with classification tasks.
The statistics are shown in Table. 4. The detailed introduction for these datasets are given as follows:
Adult D The Adult Census Income dataset consists of demographic and employment-related information about individuals, derived from the 1994 U.S. Census. The dataset's primary task to predict whether an individual carns more or less than 50, 000 per year. includes features such age, education, work class, marital status, and occupation, with 48, 842 records. This dataset is widely used in binary classification tasks, especially for exploring income prediction and socio-cconomic factors.
Default Dataset 10, The Default of Credit Card Clients Dataset contains records of default payments, credit history, demographic factors, and bill statements of credit card holders in Taiwan, covering data from April 2005 to September 2005. It features 30, 000 clients and aims to predict whether a client will default on payment the following month. Key features include credit limit, past payment status, and monthly bill amounts, making it useful for credit risk modeling and financial behavior analysis.
The Online Shoppers Purchasing Intention Dataset includes detailed information about user interactions with online shopping websites, with data from 12, 330 user sessions. It records features such as the number of pages viewed, time spent on different sections of the site, and user behavior metrics. The primary task is to predict whether user's session will result in purchase. This dataset is particularly useful for studying eustomer behavior, e-commerce optimization, and purchase prediction models.
Shoppers ":
Magic 12, The Magic Gamma Telescope Dataset is designed for the classification of high-energy gamma particles collected by ground-based atmospheric Cherenkov telescope. The dataset contains 19, 019 instances and is used to distinguish between signals from gamma particles and background noise generated by hadrons. The features include
www.:iccs...eeee.. "https://archive.. ics. usi wtt.....eeddddaaasse////33//
http:////ttttteeeeeeeeeeeeeeeeee ingaaasinnnneneeeen itenti datdateset
Phttps://archivs ics uci http://ahheerr.eeoo.eeeee/aaaaa/ http://mmmiicaaaaaaaaaaaa escope.
15

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
statistical properties of the events such length, width, and energy distribution, making it useful for astronomical data analysis and high-energy particle research.
Wilt Da The Wilt dataset a high-resolution remote sensing dataset used for binary classification tasks, focusing on detecting diseased trees ('w") versus other land cover ("n"). It includes 4, 889 instances. Features include spectral and texture information derived from Quickbird imagery, such as GLCM mean texture, mean green, red, NIR values, and standard deviation of the Pan band. The dataset is imbalanced, with only 74 samples of diseased trees.
Cardio D The Cardiovaseular Disease dataset consists of 70, 000 patient records, featuring 11 attributes and binary target variable indicating the presence or absence of cardiovascular disease. The attributes are categorized into three types: objective (c.g.. age, height, weight, gender), examination (e.g., blood pressure, cholesterol, glucose), and subjective (e.g., smoking, alcohol intake, physical activity).
The Churn Modeling dataset contains data on 10, 000 customers from a bank, with the target variable indicating whether a customer has churned (closed their account) or not. The dataset includes 14 columns that represent various features such as customer demographics (c.g.. age, gender, and geography), account details (e.g.. balance, number of products, tenure), and behaviors (c.g., credit score, activity, and churn status).
Churn Modeling Dataset 15:
Table 4. Statistics of datasets, Non indicates the number of numerical columns, and Car indicates the number of categorical columns.
Dataset
#Rows
#Num
#Cat
#Train
# Vallidation
#Test
Task
Adult
48,842
9
28,943
3,618
16,281
Classification
Default
30,000
11
24,000
3,000
3,000
Classification
Shoppers
12,330
9,864
1,233
1,233
Classification
Magic
19,019
10
1
15,215
1,902
1,902
Classification
Cardio
70,000
5
7
44,800
11,200
14,000
Classification
Churn Modeling
10,000
7
5
6,400
1,600
2,000
Classification
Wilt
4,839
5
1
3,096
775
968
Classification
In Table 4, the column "# Rows" represents the number of records in each dataset, while "# Num" and "# Cat" indicate the number of numerical and categorical features (including the target feature), respectively. Each dataset is split into training, validation, and testing sets for machine learning efficiency experiments. For the Adult dataset, which has official test set, we directly use it for testing, while the training set is split into training and validation sets in ratio of 8 1. For the remaining datasets, the data is split into training, validation, and test sets with a ratio of 8:1:1, ensuring consistent splitting with fixed random seed.
D.2. Alternative Models
In this section, we present and compare the characteristics the baseline methods employed in this study.
CTGAN (Xu et al., 2019) is a generative model designed specifically for synthetic tabular data generation using GAN-based framework. CTGAN employs mode-specific normalization to effectively handle numerical columns with complex distributions, ensuring better learning of their patterns. Additionally, it incorporates conditional generation to address imbalances in categorical features by conditioning on specific class distributions, which improves the diversity and utility of the generated data.
TVAE (Xu et al., 2019) is a VAE-based approach tailored for synthetic tabular data generation. Like CTGAN, it uses mode-specific normalization for numerical features and conditional generation for categorical features, but relies on the VAE framework to model data. This approach allows TVAE to capture latent relationships in tabular datasets while addressing challenges like class imbalance and mixed-type data.
STaSy (Kim et al., 2023) a recently developed diffusion- based model designed for synthetic tabular data generation. treats one-hot encoded categorical columns as continuous features. allowing them to be processed alongside numerical columns. STaSy utilizes the VP/VE stochastic differential equations (SDEs) model the distribution of tabular data.
Bnttps: :ttppp:://zzee :/tar//ieaeeeeeee ics uci http://eee.....eee.e..eeeeaattaa
"https: http://www/k//cc com htt.:/g/le.cooatttttttte///// anova / ca ovascul Marecularraa lar-di sease- http:::s////wwwkkgl/// http:/gll..ommmmtttttttteeo www.h saaarr//chrr model ing inggreaouree-doue
16

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Additionally, the model introduces several training strategies, such self-paced learning and fine-tuning, to stabilize the training process, thereby improving both the quality and diversity of the generated data.
TabDDPM (Kotelnikov et al., 2023) follows similar framework to CoDi by applying diffusion models both numerical and categorical data. Like CoDi, it uses DDPM with Gaussian noise for numerical columns and multinomial diffusion for categorical data. However, TabDDPM simplifies the modeling process by concatenating both numerical and categorical features as inputs to a denoising function, which is implemented as a multi-layer perceptron (MLP). While CoDi incorporates more advanced techniques like inter-conditioning and contrastive learning, TabDDPM's more streamlined approach has been shown to outperform CoDi in experimental evaluations, proving that simplicity can sometimes yield better results.
TabSyn (Zhang et al., 2023a) is a SOTA approach for generating high-quality synthetic tabular data by leveraging diffusion models in unified latent space. Unlike previous methods that struggle to handle mixed data types, such as numerical and categorical features, TabSyn first transforms raw tabular data into a continuous latent space, where diffusion models with Gaussian noise can be effectively applied. To maintain the underlying relationships between columns, TabSyn uses a Variational AutoEncoder (VAE) architecture that captures both inter-column dependencies and token-level representations. The method employs an adaptive loss weighting technique to fine-tune the balance between reconstruction performance and smooth embedding generation. TabSyn's 's diffusion process is simplified with Gaussian noise that progressively reduces as the reverse
D.3. Baselines
In this section, we describe and compare the data augmentation baselines employed in this study: SMOTE, Mixup, and Independent Joint Family (LJF).
SMOTE: SMOTE (Synthetic Minority Oversampling Technique) (Chawla et al., 2002) is widely-used oversampling method designed for numerical data augmentation. It generates synthetic samples by interpolating between existing data points within the same class. While effective for numerical features, SMOTE is not designed to handle categorical features directly, which may limit its application in mixed-type tabular datasets.
Mixup: Mixup (Zhang, 2017) is data augmentation technique that creates new samples by taking a convex combination of two existing samples and their labels. While Mixup is straightforward and effective for enhancing data diversity, it assumes linear relationships between features, which might not hold true in tabular data. Additionally, Mixup can struggle with preserving the inherent relationships between numerical and categorical features.
Independent Joint Family (UF): IJF is simple augmentation method based independent feature assumption. Unlike generative models like GANs or VAEs, which are computationally expensive and often unsuitable for generating single- dimensional features, IJF estimates the parameter distribution for numerical features and uses empirical frequency distributions for categorical features. This approach assumes independence between features during augmentation, allowing for efficient and flexible sample generation. The default augmentation ratio for IJF is set to 30%, providing a balanced trade-off between data diversity and computational overhead.
D.4. Evaluation Metries
D.4.1. LOW-ORDER STATISTICS
In this part, we will introduce the details of the shape score and trend score 16 for each feature and feature pair, respectively.
The Shape Score of numerical and categorical features are determined by the KSComplement and TVComplement metrics in SDMetrics package, respectively. KSComplement compares the shapes of real and synthetic distributions using the maximum difference between their cumulative distribution function (CDFs). TVComplement based on the TVComplement, which assesses how well the categorical distributions in the real and synthetic datasets align, with smaller differences leading to higher score.
Shape Score of Numerical Features: The KSComplement is computed based on the Kolmogorov-Smimat (KS) statistic. The KS statistic quantifies the maximum distance between the Cumulative Distribution Functions (CDFs) of real and
VeWe calculate these scores based on SDMetries package, available at https: / /ocs. .dv.dvvv dww.ddmet..tt..e.. sdmetrics.
17

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
synthetic data distributions. The formula is given by:
KST sup (.(x) ( (x)x))
(24)
where (x( and (a) are the CDFs of the real distribution (z( and the synthetic distribution ps (z)). respectively. To ensure that a higher score represents higher quality,
KSComplement based on shape score KSTT. A higher shape score indicates greater similarity between the real and synthetic data distributions, resulting in higher Shape Score.
Shape Score of Categorical Features: The TVComplement is calculated derived from the Total Variation Distance (TVD). The TVD measures the difference between the probabilities of categorical values in the real and synthetic datasets. It is defined as:
TVD
Σ [R(w] S(w)),
(25)
2
WEn
where Ω represents the set of all possible categories, and R(w) and S(w) denote the real and synthetic frequencies for each category. The shape score is defined as shape score 1 TVD, D, which returns a score where higher values reflect smaller difference between real and synthetic category distributions.
In this paper, we report the average shape score across all numerical and categorical features.
The Trend Score is used to evaluate how well the synthetic data captures the relationships between column pairs in the real dataset. Different metrics are applied depending on the types of columns involved: numerical, categorical, or a combination of both.
Numerícal- Pairs. For numerical column pairs, the Pearson Correlation Coefficient is used to measure the linear correlation between the two columns. The Pearson correlation, p(x,, y), is defined as:
Cou(a,y)
Px.y
(26)
OzØy
where Coo(z, y) is the covariance, and a₂ and oy are the standard deviations of columns x and y, respectively. The trend score for numerical-numerical pair (i.e., correlation similarity) calculated as minus the average absolute difference between the real data's and synthetic data's correlation values:
Trend Score 1
2 Ex, (1 (0(x, 9)]
(27)
where n (² y) and S (xr, y) denote the Pearson correlation coefficients of the real and synthetic datasets, respectively.
Eangorical-Categorice Pairs. For categorical column pairs, the Contingency Similarity metric used. This metric measures the difference between real and synthetic contingency tables using the Total Variation Distance (TVD). The contingency score is defined as:
Contingency Score 1 +ΣΣ 2 BEB
|Ra - Sa,el.
(28)
where A and B are the sets of all possible categories in the two columns, and Ra, and Sa.0 represent the joint frequencies of category combinations and /3 for real and synthetic data, respectively. The trend score is calculated as 1 Contingency Score.
Mixed Pairs (Numerical-Cateenn Categoriia),, For column pairs involving numerical and one categorical column, the numerical column is first discretized into bins. After discretization, the contingency similarity metric is applied to evaluate the relationship between the binned numerical data and the categorical column, similar to how it is used for aategorical-cateoriiae pairs. The trend score for mixed pair is calculated as Contingency Score.
Finally, the Trend Score computed as the average of all pairwise scores (Pearson Score for numerical-numerical pairs, and Contingency Score for categorical cateoooiccl and numerical- caterooilaa pairs). This score reflects how well the synthetic data captures the relationships and trends between columns in the real dataset.
18

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
D.4.2. MACHINE LEARNING EFFICIENCY EVALUATION
We follow the experimental setting in work (Zhang al., 2023a). We split each dataset into training and testing sets. The generative models are trained using the real training data, and subsequently. synthetic dataset of equal size is generated for further experimentation.
To assess the quality of synthetic data in Machine Learning Efficiency (MLE) tasks, we evaluate the divergence in performance when models are trained on either real or synthetic data. The procedure follows these steps: First, the machine learning model is trained using real data, which is split into training and validation sets in an 8:1 ratio. The classifier or regressor trained on this data, and hyperparameters are optimized based on validation performance. Once the optimal hyperparameters are determined, the model is retrained the complete training set and evaluated using the real test data. The synthetic data undergoes the same evaluation procedure to assess its impact model performance.
The following lists the hyperparameter search space for the XGBoost classifier applied during the MLE tasks, where grid search is used to determine the best parameter configurations:
Number of estimators: {10, 50, 100}
Minimum child weight: {5, 10, 20}
Maximum tree depth: {1, 10}
Gamma: (0.0, 1.0}
The implementations of these evaluation metrics are sourced from SDMetrics!", 17, and we follow their guidelines for ensuring consistency across real and synthetic data assessments.
D.4.3. SAMPLE-LEVEL QUALITY METRICS: a- O-PRECISION AND 3-RECALL
To rigorously evaluate the quality of synthetic data, employ two complementary metrics proposed in work (Alaa et al., 2022): a- Precision and 8-Recall. These metrics offer refined approach to assessing the fidelity and diversity of synthetic data samples by focusing on their relationship with the real data distribution.
--Prrcision. The a-Precision metric quantifies the fidelity of synthetic data by measuring the probability that generated sample lies within the a-support of the real data distribution, denoted as S. The a-support includes the most representative regions of the real data, containing the highest probability mass. Therefore, high a-Precision score ensures that the synthetic samples are realistic, falling within these high-densise areas of the real data. This metric is particularly important because it distinguishes between synthetic samples that resemble real data in a typical way and those that might still be valid but are more akin to outliers. By focusing on the high-density areas, a-Precision ensures that the generated data looks both realistic and "typical" compared to real-world data. Mathematically, this is expressed as:
Pa = P(X, E SP), a E [0.1).
(29)
8-Recall. Conversely, B-Recall evaluates the coverage of synthetic data. It measures whether the synthetic data captures the entire real data distribution, particularly focusing on the 3-support of the generative model, denoted as S.g The 8-support includes all regions of the real distribution, not just the frequent or typical areas. A high 3-Recall score indicates that the synthetic data can represent even the rare or low-density parts of the real distribution. This metric crucial because it ensures that the synthetic data does not merely replicate the most common patterns but also spans the broader diversity of the real data, capturing rare edge cases. Mathematically, it is defined as:
R3 F(X, E S 10.1.
(30)
Importance of a-Precision and 8-Recall. The combination of a-Precision and 3-Recall allows for a holistic assessment of synthetic data. While a- Preccisiin ensures that the synthetic data aligns well with the most typical regions of the real data distribution (fidelity, R-Recall ensures that the synthetic data covers the full diversity of the real data (coverage). Together, these metries provide insight into both the accuracy and diversity of the synthetic data. By sweeping through values of
! httep:://docs.sdddee////ii
19

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
and 8. one can gain more dynamic understanding of how synthetic data aligns with different aspects of the real data distribution, offering comprehensive evaluation of its quality.
In summary, a-Precision ensures the generated data looks realistic and falls within typical regions of the real distribution, while 3-Recall ensures that the generated data covers the entire distribution, including rare cases. The complementary nature of these two metrics makes them essential for evaluating the fidelity and diversity of synthetic data.
1e-6
Adult
1e-5 Default
Shoppers
Magic
20
Real
Density
4
Density
Density
Density
0.010
Tab5yn
1
TabCutMix
0.005
0.000
500000
0.0
0.1
0.2
500
O
500
fnlwgt
1e6
BILL_AMT4
ExitRates
Asym
Proportion
0.4
Proportion
0.4
Proportion
Proportion
Real TabSyn
0.50
0.5
TabCutMix
0.2
0.2
0.25
0.0
Husband
Not-in-famill
Own-child
Unmarried
Wife
Other-relative
0.0
Pay_0 0
0.0
Returning Visitor
New Visitor
Other:
0.00
Class
Relationship
Visitortype
Figure 7, Visualization of synthetic data's single column distribution density v.s. the real data.
D.4.4. DISTANCE TO CLOSEST RECORD (DCR) SCORE
The Distance the Closest Record (DCR) score is a commonly used metric for assessing privacy leakage risks in synthetic data. This metric quantifies how similar synthetic sample is to records in the training set compared to those in a holdout set. By calculating the DCR score for each synthetic sample against both the training and holdout sets, we can determine whether the synthetic data poses privacy concerns. If privacy risks are present, DCR scores for the training set would tend to be significantly lower than those for the holdout set, indicating potential memorization of training data. In contrast, the absence of such risks would result in overlapping distributions DCR scores between the training and holdout sets. Moreover, a probability close to 50% that synthetic sample closer to the training set than the holdout set reflects a lack of systematic bias toward the training set, which a positive indicator for privacy preservation.
Following (Zhang et al., 2023a), employ a "synthetic holdout" evaluation protocol. The dataset is split evenly into two parts: one serves as the training set for the generative model, while the other acts as the holdout set and is excluded from training. After generating a synthetic dataset of the same size as the training and holdout sets, we calculate DCR scores for synthetic samples.
D.4.5. CLASSIFIER TWO SAMPLE TESTS (C2ST)
The Classifier -ample Test (C2ST) (Zhang et al., 2023a) is used to evaluate how well synthetic data replicates the distribution of real data. This approach involves training a binary classifier to distinguish between real and synthetic samples. If the synthetic data closely matches the distribution of the real data, the classifier should struggle to differentiate the two, resulting in test accuracy close to 50%. Conversely, if the synthetic data deviates significantly from the real data distribution, the classifier will achieve higher accuracy, indicating poor alignment. The C2ST score provides a quantitative measure of this alignment, offering insights into the quality of the synthetic data. A low C2ST score suggests that the synthetic data effectively captures the real data distribution, making it difficult for the classifier to distinguish between real and synthetic samples.
20

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
D.4.6. OUT-OOIDISTRIBUTIINN (OOD) DETECTION
TabCutMix may introduce degree of OOD (Yang et al., 2024a) issues. To investigate the potential relationship between TabCutMix and OOD, we conducted OOD detection experiments. These experiments also aimed to evaluate whether TabCutMixPlus could mitigate OOD-related challenges to some extent. We framed the OOD detection task as classification problem, treating normal samples as negative and OOD samples as positive. Since our dataset lacks explicit labels for OOD samples, we synthesized positive samples following the approach outlined in (Ulmer et al., 2020). For numerical features, we randomly selected one feature and scaled it by a factor F (where F = 100). This approach aligns with the methodology in (Azizmalayeri et al., 2023), which experimented with F' values of 10, 100, and 1000; we adopted / = 100 as a balanced choice for our experiments. For categorical features, we randomly selected a value from the existing categories of the chosen feature. This process was repeated for single feature time. We used the original training set as the negative class and the synthesized samples as the positive class. A multi-layer perceptron (MLP) was trained to classify between these two classes. Subsequently, we tested the samples generated by TabCutMix and TabCutMixPlus using the trained MLP and calculated the proportion of samples classified as OOD. This analysis provides insights into the extent of OOD issues introduced by TabCutMix and the potential of TabCusMixPlus to alleviate such issues.
5 indicates that the OOD issue introduced by TabCutMix is relatively minor across most datasets, as evidenced by low OOD ratios (e.g.. 2.06% for Adult and 0.61% for Magic) and high Fl scores (e.g., above 90% in several cases). While some datasets, such as Default and Cardio, exhibit higher OOD ratios (39.47% and 4.83%, respectively). TabCutMixPlls significantly mitigates the OOD problem, reducing the OOD ratio by substantial margins across all datasets. For instance, in the Adult dataset, the OOD ratio is reduced from 2.06% to 0.36%, while in the Default dataset, it decreases from 39.47% to 25.44%. These findings highlight the effectiveness of TabCutMixPlus in addressing potential OOD challenges while maintaining robust classification capabilities, reinforcing its utility in synthetic data augmentation workflows.
Table OOD detection of datasets.
Method
Adult
Default
Shoppers
Magic
Cardio
Churn Modeling
Wilt
Arg. Rank
Mixup (FI Score:)
92.32 0.67
71.64 +: 0.74
83.30 + 0.77
99.67 : 0.04
60.32 0.47
98.01 0.07
99.94
L,66
Mixup (Ratio%)
0.03 0.06
5.19
0.16=0.26
0.00 0.00
1,98
0.75
0.00 + 0.00
0.00 100000 0.00
L.4
SMOTE (F1 Score%)
92.24=0.79
71.55
+ 0,78
83.16:1.00
99.68 : 0.05
60.30 0.49
98.01 : 0.09
99.95 : 0.02
2.14
SMOTE (Ratio%))
0.75 1.40
28.43 1: 3.18
0.12 0.37
0.00 0.00
3.18 1: 1.83
0.00 0.00
0.001:0.00
1.57
TabCutMix (F1 Score%)
02267=0.22
71.42
+ 1.32
82.47 4 0.35
99.27 0.07
60.33 4 0.25
97.94 0.13
99.94
2.43
TabCutMix (Ratio%)
39.47
1.58 0.76
+:
4.83 1.39
0.00
0.00
3.14
TabCCuMixxPls (FI ScoreSii
92.63 0,20
71.39
+ 0.94
82,28 + 0.39
99.19 + 0.05
60.39 + 0.17
97.97 + 0.02
99.95 +: 0.03
2.57
TrbCutMixPlos (Ratio%)
0.27
25.44 + 2.81
0.70 = 0.39
+
3.88 0.19
0.00 +: 0.00
0.00
2.00
D.5. Discussion on Memorization Ratio and DCR
The DCR metric measures the closest distance of each synthetic sample to the training and holdout sets, offering insights into potential privacy risks. Synthetic samples that closely resemble training data can indicate privacy concerns. However, DCR's dependence on the holdout set limits its robustness, the results can vary with changes in the holdout set composition. This reliance underscores the need for alternative metrics that are less influenced by external data partitions.
To address this, we focus on the Memorization Ratio, which uses a distance ratio to detect overfitting by identifying synthetic samples that are disproportionately close to their nearest training neighbor compared to the second-closest. Unlike DCR, this metric is independent of the holdout set, directly assessing overfitting within the generative process. While the fixed threshold for the distance ratio is inspired by image generation literature, it provides a practical bascline for tabular data. We acknowledge that tailoring the threshold to account for tabular-specifie features (e.g., categorical and numerical distributions) could improve its accuracy and plan explore this in future work. Together, these metries provide complementary insights into privacy risks and overfitting in generative models.
D.6. Memorization Evaluation Metrics
In the context of generative modeling, memorization occurs when a model reproduces training samples too closely, rather than generating novel samples that reflect the underlying data distribution. While some degree of memorization might be acceptable or even desirable in certain scenarios (c.g., when high fidelity is required), excessive memorization can lead to overfitting, lack of diversity in generated samples, and potential privacy risks if sensitive data from the training set is replicated. Therefore, it is crucial to develop quantitative metrics to detect and measure memorization in generative models, especially for applications involving tabular data, where exact replication of training samples is particularly problematic.
21

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
To address this challenge, we propose the concept of the memorization ratio based on the relative distance ratio criterion. Let be generated sample, and let D denote the training dataset. We define the distance ratio r(x) of as:
r(x) d(x, NN1 (a, D)) d(x, NN2(2, D))
where -(:--) is distance metric in the input sample space, NN1 (a. D) is the nearest neighbor of in D, and (N D) is the second-nearest neighbor of x in DD. Intuitively, a small value of r(x:) indicates that the generated sample v is nearly identical a training sample, suggesting memorization. To formalize this notion, we follow the threshold of memorization in the image generation domain and consider a sample x to be memorized if r(x) < 3*
To quantify the extent of memorization across all generated samples, we compute the memorization ratio, defined the proportion of generated samples that satisfy r(x) < 3
Mem. Ratio
Σ I(((()) < })
1
G
DEG
where G is the of generated samples and I(-) is the indicator function.
While the memorization ratio provides a point estimate of memorization intensity at fixed threshold 3. it is also important to understand how the degree of memorization varies under different thresholds т. To this end, we propose the Memorization Area Under Curve (Mem-AUC). Mem-AUC computed as:
Mem-AUC L." Mem. Ratio(T) dr,
where Memorization RTtio(T) represents the proportion of generated samples for which r(z) < T as a function of the threshold т. Mem-AUC captures the overall memorization behavior across a continuous range of thresholds. Higher Mem-AUC values indicate stronger memorization, while lower Mem-AUC values correspond weaker memorization and better generalization.
E. More Experimental Results
E.1. Distance Ratio Distribution of TabCutMix
We analyze the distribution of the nearest-neighbor distance ratio, defined as r memorization. A more eere-concentrated ratio distribution indicates more severe memorization issue, the generated sample & is closer to real sample in training set 'D. Figure. 8 illustrates the distance ratio distribution for both the original TabSyn and TabSyn with TabCutMix, and we observe the following:
(. N) to assess the severity of
1bs.1: TabCutMix consistently shifts the distribution away from zero, indicating reduction in memorization. For example, in the Magic dataset, TabCutMix reduces the memorization ratio from 80.01% % to 52.06% by generating samples that less tightly aligned with the real data in D.
The distance ratio distributions for both TabSyn and TabSyn with TabCutMix exhibit a bipolar pattern, with higher probability mass concentrated near 0 or 1, while the probability in the middle remains low. This indicates that more generated samples are either very close to real data points (suggesting memorization) or relatively far apart (suggesting diversity). the Magic dataset, for instance, this bipolarization is prominent, with TabCutMix shifting a greater proportion of samples towards higher distance ratios, thus reducing memorization.
Obs.2:
Adult
Default
Shoppers
Magic
0.3
TabSyn (Ratie: 29.26%)
TabSyyn (Ratio: 20.11%)
TabSyn (Ratio: 27.68%)
0.75
TobSyn (Ratio: 80.01%)
CabMit (Ratio: 27.03%)
TabCutMX (Ratio: 16.86%)
0.3
TabCutMix (Ratio: 25.38%)
TabCutMix (Ratio: 52 (6.66)
Density
0.2
Density
0.2
Density
ensstty
0.50
0.2
0.1
0.1
0.25
0.1
0.0 0.0
0.5
1.0
0.0
0.0.0
0.0
0.5
1.0
0.0
0.0 0.0
0.5
1.0
0.00
0.0
0.5 Distance Ratio
1.0
Distance Ratio
Distance Ratio
Distance Ratio
Figure 8. The neireot distance ratio distributions of TabSyn with and without TabCutMix across different datascts.
22

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
E.2. Data Distribution Comparison
Figure. 7 compares the distribution of real and synthetic data, with and without TabCutMix, for both numerical and categorical features across four datasets: Adult, Default, Shoppers, and Magic. We use numerical feature and one categorical feature examples from each dataset. We observe that
Obs.1: In the numerical feature distributions, TabCutMix generally synthesizes data, similar to w/ TabCutMix, aligned with the real data's distribution. For instance, in the Magic dataset, the Asym feature shows that the synthetic data generated by TabSyn has a good alignment with real data.
Obs.2: The categorical feature distributions show similar improvement. In the Shoppers dataset, the proportion of values for the "VisitorType" feature generated by TabCutMix closely matches the real data, similar to the synthetic data generated without TabCutMix. This suggests that TabCutMix preserves the alignment between real and synthetic data for categorical features as well.
E.3. Feature Correlation Matrix Comparison
Figure. 9 presents heatmaps of the pairwise column correlations between synthetic and real data. We compare the correlation matrices of synthetic data generated by TabSyn with TabCutMix against the real data. We observe that
TabCutMix preserves the quality of data generation in terms of correlation matrices, maintaining similar patterns to the synthetic data generated by TabSyn without introducing further errors. In datasets like Default and Shoppers, TabCutMix ensures that the synthetic data retains the essential correlation structure of the real data, without significant degradation in correlation matrix accuracy.
Obs.1:
Obs.2: In the Magic dataset, while discrepancies between the synthetic and real data's correlation patterns persist, TabCutMix helps to maintain the existing data generation quality. Although it does not reduce the correlation matrix error, it ensures that the synthetic data continues to represent feature relationships similarly to TabSyn, preserving the overall structure.
Adult TabSyn
Default TabSyn
Shoppers TabSyn
Magic TabSyn
0.15
0.075
0.02
0.10
0.10
0.050
0.025
0.05
0.05
0.01
0.000
0.00
0.00
-0.00
Adult TabCutMix
Default TabCutMix
Shoppers
TabCutMix
Magic TabCutMix
0.10
0.10
0.75
0.10
0.50
0.05
00.5
0.05
0.25
0.00
0.00
0.00
0.00
Figure 9, Hcatmaps of the pair-wise column correlation of synthetic data v.s. the real data. The value represents the absolute divergence between the real and estimated correlations (the lighter, the better).
E.4. More experimental Results on Shape Score
Figure 10 visualizes the shape scores for synthetic data generated by TabSyn and TabSyn combined with TabCutMix (w/ TCM) across multiple datasets (Adult, Default, Magic, and Shoppers). Shape scores reflect how closely the distribution of individual columns in synthetic data matches the real data. This figure compares these scores for different features to assess the fidelity the generated data. We make the following observations:
Obs. TabSyn and HahSyn+TTbCCutee produce high-fidelity distributions across datasets. Across all datasets (Adult,
23

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Default, Magic, Shoppers), both TabSyn and TabSyn+TabCurMiix maintain high shape scores, suggesting that the gener- ated samples from both methods capture the real data's feature distributions effectively. For instance, in the Adult and Default datasets, shape scores are consistently close to 1.0, indicating minimal divergence between synthetic and real data distributions.
Obs. 2: Low variance in shape scores across features. One notable observation across all datasets (Adult, Default, Magic, and Shoppers) is the consistently high shape scores across features, with minimal variance. For most features, the shape scores are very close 1.0, indicating that both TabSyn and TabCutMix can replicate the real data distributions with high fidelity, regardless of feature type. The small variance in shape scores suggests that both methods generalize well across a wide range of features, from categorical to continuous, without significant degradation in performance for any particular feature.
The shape score comparison demonstrates that both TabSyn and TabCutMix generate synthetic data with high fidelity to the real data across multiple datasets.
Adult
Default
Magic
Shoppers
K5
R5
KS
KS 1
TabCutMix
KS KS
KS
KS 2
TabSyn
KS
KS
K5
Features
TV
Features
Features
KS
sture
KS 3
KS
TV
KS
4 TV 10
TV 13
TV 10
KS
TV 12
TV
KS
13
12
KS
TV 14
TV 15
TV 11
TV 17
0.900 0.925 0.350 0.975
1.000
0.92
0.34
0.96
0.98
1.00
0.8
0.9
1.0
0.85
0.30
0.95
1.00
Score
Shape Score
Shape Score
Shape Score
Figure Shape score comparison each feature synthetic data generated by TabSyn and TabSyn+TabCuuee across multiple datasets.
E.5. Case Study: Evaluating Augmented Data Quality with TabCutMix and TabCutMixPlus
To assess the quality of augmented data and identify potential issues with TabCutMix, we conducted detailed case study using the Magic dataset, as summarized in Table 6. The table presents comparison of real and generative samples produced by TabSyn with TabCutMix (denoted as TCM) and TabSyn with TabCutMixPlus (denoted as TCMP).
The results reveal that TabCutMix disrupts feature correlations, evident unrealistic relationships like the Length being smaller than Width, which contradicts the inherent structure the data. In contrast, TabCutMixPlus preserves feature coherence by clustering correlated features and swapping them within the same cluster. This approach ensures the generation of more realistic and consistent samples, demonstrating its effectiveness in maintaining data quality and utility compared TabCutMix.
Table The real and generative samples by TabSyn with TabCutMix and TabSyn with TabCutMixPlus in Magic dataset. TCM represents TabCutMix, TCMP represents PabCutMixPlus.
Samples Length
Width
Size
Cone
Conc1
Asym
M3Long
M3Trans
Alpha
Dist
class
TabSyn+TCM 24.72
32.12
3.35
0.15
0.09
150.69
-40.91
-21.80
5.11
205.42
g
TCMP 17.56
11.20
2.29
0.58
0.37
2.29
17.12
2.40
22.44
211.00
E.6. Experimental Results on More Datasets
To broaden the evaluation of TabCutMix and TabCutMixPlus, we included 3 additional datasets Churn, Cardio, and Wilt. The results are summarized in Table 7, and the following observations are made:
Obs. 1: TabCutMix and TabCutMixPlus consistently reduce the memorization ratio across all datasets compared to various
baselines. For instance, in the Churn dataset, the memorization ratio for TabSyn reduced from 25.42% 2% to 24.61% by
24

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
TabCutMix and to 24.79% by TabCutMixPlus, representing an improvement of 3.21% and 2.49% compared to the vanilla method. This highlights the effectiveness of TabCutMix and TabCutMixPlus in mitigating memorization.
Obs. 2: For data quality, TabCutMixPlus achieves superior results in metrics such as MLE, a- a-Precision, and 3-Recall, indicating its ability to maintain data fidelity and enhance synthetic data utility. For example, in the Wilt dataset, TabCut- MixPlus achieves the highest o--Precision (99.10%) and 3-Recall (48. (4%),,)) reflecting its capability to preserve structural coherence while reducing memorization. In contrast, TabCutMix shows competitive but slightly lower performance (e.g.. 78.73% a-Precision and 47.04% 3-Recall ) demonstrating that the clustering- based feature swaps in TabCutMixPlus offer additional advantages.
E.7. More experiments on Baseline LJF
To compare the effectiveness of IJF with the proposed TabCutMix and TabCutMixPlus, we evaluate memorization and data generation quality using metrics such as memorization ratio, MLE, a-Precision, 33-eccall, shape score, and trend score across four datasets. The results presented in Table 8, and we observe the following:
Obs.1: DF demonstrates moderate success in reducing memorization ratios compared to the baseline generative models without augmentation. However, In terms of data quality metries such as MLE, a-Precision, and 3-Recall, IJF performs weaker compared to TabCutMix and TabCutMixPlus in maintaining structural fidelity. For example, in the Adult dataset, the shape score for IJF (96.55%) is slightly lower than that for TabCutMix (97. 26% and TabCutMixPlus (97.50%), suggesting that [JF may not fully capture the complex relationships between features that are preserved in the proposed methods.
Obs.2: IJF shows consistent performance across datasets but lags behind TabCutMix and TabCutMixPlus in scenarios where feature dependencies play critical role. For instance, in the Shoppers dataset, TabCutMixPlus achieves the highest trend score (97.77%) indicating its superiority in generating coherent and realistic data samples.
E.8. Experimental Results on More Generative Models
The results in Table 9 demonstrate that TabCutMixPlus (TCMP) effectively reduces memorization across various generative models, including GAN-based models (e.g., CTGAN (Xu et al., 2019)) and 'AE-based models (e.g., TVAE (Xu et al., 2019)). Specifically, for CTGAN, TCMP achieves a reduction in memorization ratio by 8.03%, 2.23%, 0.52%, and 6.70% on the Default, Adult, Shoppers, and Magic datasets, respectively. Similarly, for TVAE, TCMP reduces memorization by 9.47%, 2.93%, 12.9%%, and 8.98% across the same datasets. These results indicate that TabCutMVxPlus is not limited to diffusion models but is also broadly applicable to other types of generative models. By preserving feature correlations and addressing issues introduced by traditional TabCutMix, TCMP significantly mitigates memorization while maintaining or even improving key performance metrics such as MLE, shape score, and trend score.
E.9. More Experiments on Mem-AUC
To verify the validity of using fixed threshold of 3 for measuring memorization, we computed both the Mem-AUC and the memorization ratio based on the & threshold. We then evaluated the correlation between these two metrics. The results, as depicted in Figure 11, reveal strong positive correlation across all scenarios. The high correlation between Mem-AUC and the memorization ratio supports the appropriateness of the 1/ threshold for practical applications. While Mem-AUC provides a more holistic evaluation by integrating memorization over the entire threshold range, the 1 threshold remains a valid and reliable metric for assessing memorization, balancing simplicity and effectiveness in practice. Table 10 highlights the results for different augmentation techniques, including TabCutMixPlus, TabCutMix, Mixup, and SMOTE. These findings reinforce that the & threshold remains a valid choice for simplicity and effectiveness in practice.
F. Limitation Discussion
While this work makes significant contributions to augmenting tabular data with TabCutMix and its improved version, TabCutMixPlus, several limitations remain that warrant further exploration:
Assumptions About Feature Independence. TabCutMix assumes that features can be swapped between samples independently without disrupting the data manifold. However, this assumption does not hold for datasets with strongly correlated features or complex interdependencies. For instance, in the Cardio dataset, the high feature correlation led to a relatively high OOD ratio (4.83%), highlighting limitation of the current method preserving feature relationships
25

---

Understanding and Mitigating Mcmorization in Diffusion Models for Tabular Data
Table 7. The overview performance comparison for tabular di fiffusion models on more datasets. "TCM" represents our proposed TabCut-
Mix and "TCMP" represents TabCutMixPlus. "Mem. Ratio" represents memorization ratio. "Improv" represents the improvement ratio on memorization.
Methods
Mem. Ratio 1%
Insgeox
(ly
01
Pf
Shope Scorecly""
Trend Scone/611
CISTI9
DUR(S)
STLSy
17.52 0.27
M2,92 0.30
1.30
46.07
2: 7.05
90.27 6.55
75.02 2: 4.03
4 0.92
MiLSy
T8.11 2: 0.18
0.777%
93.03 à: 0.16
91.03 a 3.57
50.19 2: 0.34
94.20 d: 1.1
96.67
1.10
79.72 6.90
50.27 1 1-22
76.88 2: 0.43
0.88% 1
92.05 +: 1.59
67.32 4: 1.04
$2.50 +: 2.18
89.78 a: 0.91
89.78 1.45
53.08 4: 3.70
51.24 2 0.74
TCM
T5.12
9.10%
91.49
92.50 3.01
35.24 2:
5.35
75.70
49.85 0.21
STMS
76.70 0.338
1.00%
92.77 0.20
97.21
1.30
1.51
96.34 0.42
76.63 6.5
48.1 0.25
TaNDOPM
77.62 0: 2.11
92.78 a: 0.25
98.41 d: 0.37
46.6 1: 115
99.77 a: 0.06
98.58
0.51
99.05 4: 0.70
50,47 a: 0.42
TINDDPM-Miupp
15.37 0.51
0.99%%
92.06 a: 0.58
92.01 4: 1.24
45.45 1.38
90.22 4: 0.53
97.26 1,00
98.33
50.92 a: 0.20
SMOTE
72.31 1.56
0.84%5 1
91.68 0.52
66.45
3.04
45.35 2:70
0.37
85.04 1.54
50,83 0.71
TADDRM--TCM
5.222
5.00%
91.59
97.92 0.38
30.51 0.70
05.07 0.06
0.11
49.73 0.35
TANDDPM: TCMP
76.22 0.39
1.81% 4
91.50 d: 0.22
96.50 4.12
36.22:245
95.08 a: 1.32
95.17 2: 3.05
95.29 6:22
49.88 a 0.74
Takkye
80.02 a 0.39
03.18 4: 0.31
99.10
a: 0.68
48.25 1 0.41
99.00 a: 0.38
99.15 0.08
99.75 1.5.4:00 0.29
50.48 a: 0.16
TakSyssMixup
78.58 0.78
1.42% J
92.63 0.45
91.68
0.20
48.50 0.17
96.70 0.10
98.41 0.34
99.68 0.43
$1.01 0.51
72.14 1.27
9.86%
92.74
63.32 0.79
89.26
0.59
56.26 2.53
0.53
TCM
62.06 8: 7.12
84.94%. J
91.77 + 0.12
96.83 s: 0.40
30.79 d: 2.92
07.63:00.65
98.09 4: 0.17
98.55 a 1.49
51.76 a: 0.49
TCMM
76.10 0.336
4.40% 4
91.01 + 0.42
98.03 +: 1.76
50.54 1.54
95.57 +1 0.57
97.26 0.27
07.58
8.35
$1.32 4: 0.63
27:01 0.90
84.00 + 2.74
92.99 99 2.7
59.42 800
87.17 + 6.64
86 5.09
48-4 10.89
50.70 2.00
21,95 3.47
T.0756 J
15.08 7,46
1.90
40.08 2:11
2.53
0.58
7.58
0.88
STAS;+SMOTE
22.50 0.87
17.20%
83.73 à: 1.37
$2.44 + 1.78
:5.55
84.51 +1 3.71
86.23 + 0.45
40.76 26 4.83
$1.90 : 0.58
22.90 + 7.42
14.06% &
62.62 + 0.77
$2.90
+ 1.10
41.34 + 2:21
67.28 + 0.68
20.15 2.18
48.53
1.61
47.25 0.43
TCM
22.95 6: 2.32
15.36%
84.01 + 2,62
56.21 4: 2.57
43.16 1.19
91,03 6: 1.00
90,30 8: 1.57
49.73 4.26
52.26 d: 1.29
ST68;-TCMP
1.06
10.09%
85.36
3.45
91.10
0.76
0.79
TaNDIIM
25.13 6- 1.00
16.11 2.57
59.10
0.96
51.05 4 0,90
94.34- 0.30
98.06 0.37
96.71 1.40
49.234 2.27
TaNDDMM-Miupp
25.00 9: 0.58
1.60% J
85,00 1: 1.49
95.04
4: 2.25
40.52 4: 1.49
03.08 6: 1.37
03.841 2:43
68.77
3.39
48.73 0.82
TANDDI--MOTE
21.57
9.4155 4
84.57 6: 2.04
$6.45 6 3.22
46.25 + 1.18
94.53 0:50
92.02 2.19
K0.12 3.95
0.23
TENDDIMM-JF
1.86% J
85.16 1.78
XM.12
9:43
43381
6:76
6-761
8.12
0.84
TCDDPM-TCMM
21,12 0.222::033
4.00% J
86.39 1: 1.82
98.51
s: 0.87
50.41 0.75
98.55 1 0.52
98.18 18 0.81
96.62 3.13
$2.98 a: 0.53
TCMD
21.00 0.32
3.05% 4
86.55 + 2.06
96.13
1.21
$1.52 +
98.15 + 0:38
97.78 0.37
9662 2.30
50,37 2.43
TaNSye
0.221
06.04 6: 7.78
59.31 4. 0.51
50:45 4: 1006
19.14 0.35
98.15 0.19
9939 6 0.05
50.80 061
TalSys+Misup
2..56::0.51
2.70% J
85.84 1,56
0.16
49.43
0.65
0.73
95.04 581
0.25
TMOST
21.57 0.46
2.195 J
85.60 + 1.97
98.63-1 0.90
64 0:40
98.25 + 0.34
75.00 + 1.50
99.08 1 0.79
50:75 1.42
Talsyo+UJP
24.53 8- 0.38
3.50% 4
N1.12 + 1.76
18.51 + 1.00
46.02 4: 1.122
94.96 s 0.54
93.51 + 0.34
K3.22
+ 2.14
52.78 A 0.45
TMM
21.81 0.17
8.215% 4
85.60 6: 2.41
99.12 4: 0.46
49.60 0.38
90.14 d 0.19
98.25 4 6.38
99.70 : 0.33
$2.15 à 0.77
TCMP
2.0956 J
19.78
0.28
0.64
0.50
08.70 0.10
49.30
STaly
25,00 0.332
79.99
99
46.46 0:87
1:24
96.37 0,80
9.66
50.10 0.21
$1.44:0.35
0.42% 1
0: 0.8
94.86 st 3.17
46.32
: 0:47
95.14 0.79
95,90 = 0.15
$2,90 + 5.11
49.79 a: 0.53
STLSy-SMOTE
22.77 2: 0.27
4.87% J
78.81 0.15
90.57 4: 3.00
45.11 1 1.24
05.27
4. 0.48
95.16 : 1,34
76.22
521
50.48 4: 0.24
21511 1.32
1
1963 091
98
1:24
40.96 0.19
1M.24 0.45
92.42 0.47
4.76
1.51
TCM
22.50
6.00%
79.71
95.11
45
2.18
95.95 1.74
4 3003
TCMP
22,51 0.48
4.78% 4
79.96 d: 0.33
94.03 a 282
46.09 a: 2.68
96.37= 1.59
96.07
= 1.06
$5.99
50,43 a. 0.74
TANDIPP
W0.24 J: 0.78
99.14 a 015
1111 0.17
19.1 d: 0.00
98.95
4 0.30
99.43
4 0.55
4MM1 a 0.36
TANDOMM-MINU
21.00:00.0
4
79.62 à 0.27
90.23
1.45
0.17
97.50 0.72
97.38 0.55
95.01 2:5
50.15 0.54
TANDDPM-SMOTE
6.21%
79.47
96.35
238
1.77411117
96.58 0.75
1.28
1669
50.73 0.19
22,00 5: 0,85
10.05%
79.37 + 0.79
9A.12 1 0.77
42.55
2 0:33
17.87 0.13
94,49 4 0.32
99.66 0.16
50.10 A: 0.45
TANDDPM- TCM
21.0G 2: 0.38
6.40% 1
79.71 1 0.58
97.82 4: 2005
48.37
4 1.11
95.66 d: 1.35
95,86
4 3.43
96.31 4 0.42
49 4 1.55
TANDOM--
23.54 à:
4.48% 4
79.32 A: 0.27
98.71 à 0.49
48.07 4 0.34
95.58 0.62
98.67
4 0.30
96.31 4 0.42
40.34 a 0.38
Tahsya
35,31
10,04
95.70
4963
07.43=
1.67
91 199
1.79
TANSyarMixup
24.66 5: 0.45
3.30% 4
79.85 :: 0.30
98.48 1: 0.78
48.15 1 0.56
98.32 0.53
96.10
1 2.38
96.44
50,69 1: 0.34
SMOTE
23.56 à: 0.58
6.77% 1
79.76 :: 0.49
97.54 4 0.82
47,47 4 0.44
97.06 2: 0.71
05.00 + 3.33
86.30 4. 3.50
48.73 2.733.10 0.36
TINSyo+LI
22,53 5: 0.50
10.97% 4
79.43 1
98.09 1.17
42.33 4
d: 0.17
0,00
0.19
50.48 0.36
TCM
22,57
9.23%
79.92 0.44
0.98
48.62
98,90 0.63
0.99
95.97
50,271:
TUMP
23.0 5: 0.42
5.00% 1
79.79 a 0.00
9N.32 + 1.36
48.00 4 U10
96.35 s 0.14
98.12
1 0.99
95.78 6 0.00
49.95 a 0.27
STaSy
98.42 5 0.24
98.74 H: 1.15
$6.68 4 5.50
42.20 1.00
62.39 e: 8.06
91.16
6. 5.11
36.64 4 6.77
52.27 4: 5.18
Mixup
97.61 0.81
0.77% 4
99:06
91.78 8.33
43.45 1
4.54
85.76 1.65
47.85 6.39
45.45 890
97.81
1.18% J
98.58 0.771
76.07 2.06
36.07
80:96 2.64
85.34 5.12
35.63 3:54
0.33
STIS
96.53 0.31
1.92% J
16.86611 à: 1.75
7944 = 6.34
35:22 +: 7.30
79.20 + 4.54
79.11 1: 4.00
37.17 4 10.11
54.7 +: 1.13
TCM
92.47 :: 5.97
6.06% 1
08,50 1: 0.64
91.10
4: 9.2
42.1 4 8.70
87.24 12.46
91.68 = 7.60
47.49 = 14.75
51.65 d: 2.00
TCMP
07.00 0.84
0.84% 1
09.33 0.31
$0.65 7:37
42.22 4 9.77
R5.32 8.57
90.94 7.43
40.05
48.19 3.50
TaNDDIM
90.32
0.73
0:47
1.51
0.35
18.33
0.54
TADD
08.16 2: 0.24
0.32% 4
99.34 1: 0.44
96.29 0.49
52.13 L.1
97.34 0.99
92.24 3.21
96.05 199
50.84 0.26
TADDPM-SMOTE
96.78 :: 0.41
1.72% J
09.22 à: 0.54
70.49 0.78
43.76 1.37
01,00 0.50
88.04 4.73
261
50.20 4: 0.28
TMNDNM-JJF
91.16 0.18
4.595% 4
98.00 5 0.66
96.51 0.6)
43.95 :
96.69 0.36
86.63 0.51
08.57 0.8
$2.46 366
TCMDPM--TM
0.117
1.33% J
99.12
97.93 1.0
48.47
97.31 1.38
95.71
96.92 1.72
48.75 218
TCMP
96.75 5 0.67
1.76% J
99.52 4: 0.37
98.55 4: 0.14
49.36 1 0.99
97.12 0.54
96.81 1: 0.63
96.76 1. 3.44
45.52 1.35
Takkye
97.07 + 0.39
99.85 + 0.07
09.83 + 0.28
47.96 1 0.64
98.55 +h 0.12
03.62 0.19
9981 1 0.07
$1.71 + 2.94
TakGyssMixup
07.02 4: 0.22
0.06% 1
99.44 à: 0.34
97.03
0.13
48.89 4:
95.51 0.06
93.38 4.30
98.85 0.14
50.68 0.42
9..81:00.00
2.89% J
99.13
77.86
1.27
41.96
00.57 0.61
3.33
1.02
2.18
TahSys+UJ
43.62 0.52
4.145% J
99:06 0.65
96.48
0.22
42.13.1 0.69
91.18 + 0.64
85.05 0.22
99.64 1 0.35
$1.66 1 0.89
95.00 :: 0.19
1.76%. J
99.45 :: 0.29
96.73 4. 0.5L
47.04 1 0.25
05.64 às 0.06
98.11 1: 0.30
99.73 4: 0.24
40.44 1: 4.08
TCNG
96.70 6 0.23
0.90% 1
99.70 :: 0.14
99.10 1: 0.23
48.14 4: 0:41
93.47 :2 0.39
98.74 : 0.07
99.06 2. 0.78
49.72. 1.02
26

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Table 8. The overview performance comparison for tabular diffusion models on IJF and our proposed methods. "TCM" represents our proposed TabCutMix and "TCMP" represents TabCusMixPlus. "Mem. Ratio" represents the memorization ratio. "Improv" represents the improvement ratio memorization.
Methods
Ratio (
Improv.
(SE)
o-Precision
Shape Score(S)
Score(S)
C2ST(%
DCR(%)
STaSy
17.57 + 0.53
76.48 + 1,18
87.78 5.20
35.94 = 5.48
90.27 2.43
89.58 = 1.35
67.68
6.89
50.30 0.36
STASy+JF
14,50 +: 1.77
16.94%
75.13 0,96
86.59 3.71
31,56
89.19 1.12
31,42 :: 0.72
49.28 1,46
$1.30 2.78
STaSy+TCM
14.51
17.44%
75.33
86.04 11.55
32.13
3.16
50.39 0.99
STAS++ICMP
13.53 2,00
11.59%
76.30 :: 0.57
90.83 1.51
32.81 1.37
91,19 0.77
92.08
2.04
50.13
2.00
50,70 1.94
TobDDPM
19.33 1 0.45
76,79 1: 0.69
98.15 1.45
4441 0.70
97,58. 0.95
91,16 -: 0.68
91.85 6.04
19.12 0.91
Defaok
TENDDPM-UF
14.02 1: 1.12
27.47%
76.19 0.87
93.82 0.68
38.59 1 0.19
96.36 0.53
28.19 1.27
95.91 1.77
49.86 1,22
TANDDFM-TCM
0.47
13.26% &
76:17
38.72
2.52
91.72
50.23 0.53
TCMD
18.00 0.24
6.88% L
76.92 + 0.17
98.26 0.25
41.92 0.52
97.37 + 0.09
91.42 1.15
95.64 0.49
49.75 0.32
TabSyn
20.11 0.03
77.00 + 0.33
98 66 0.13
46.76 +1766++000 0.50
98.96 + 0.11
96.82 1,71
98.27 1.14
$1.09 0.32
TJJ
15.82 0..2+-0.33
21.33% 4
76.53 + 0.56
66
39.11
0.30
31.84 0.31
97.64 0.80
49.58 0.67
TaNSyn-TCM
16.16%
0.34
96.16
40.69 2.46
98.02
96.51 1.42
97.65
1.82
TONS
17.60 0 0.28
12.48% :
77.17 + 0.51
97.61 0.27
44,46 0.60
99.03 0.08
96.30 - 1.48
98. 16 0:65
$1.20 0.90
STuSy
26.02 0.89
90.54 1 0.17
85.79 7.85
34.35 2,46
89.14 2.29
86.00 2.97
51.89 14.87
50.46 0.39
STaSy+JJF
2.03
20.07% 4
10.52
0.32
STOSS+TCM
20.89 - 1.33
19.71% J
00.4 + 0.30
85.39 1.61
31,24 0.97
88.33 3.63
85.39 4.09
45.49 4.78
50,92 0:39
STMS
21.45 2.60
17.59%
90.72 + 0.06
86.71 + 4,12
32,63 1.81
89.62 1.55
86.05 2.44
49.12 9.95
50.75+ 0.59
TINDPMM
31.01 0.18
0.07
94.63 1,19
$1.56 0.34
Adult
TENDDPM-UF
24.98 = 0,41
19.45%
89.96 0.52
95.3
42.57 0.30
97.45
67.80 0.97
95.68 0.49
50.47 0.27
TANDDM--TCM
27.56 = 0.19
11.16% J
91.15 + 0.06
94.97 0.06
47 1,46
98.65 0.03
97.75 0.07
85.61 16.03
50,99 0.65
TENDDPM-TCMP
26.10 2.11
15.83%
90.54 + 0.17
92.26 6.97
43,49 +: 3,74
95,10 4.27
91,50 6.53
84,76 10.12
50.68 :: 0.89
TabSyn
0.50
0.27
TabSyn-1JF
24.71 = 0.80
15.53% J
90.82 +: 0.13
98.94 0.38
41.61 0.57
97.40 =: 0.54
63.93 1.19
99.55 0.27
50.57 4: 0.43
TIbSyn-TCM
27.00 0.22
7.60% 1.
91.09 + 0.17
99.04 à 0.42
44.95 d: 0.42
99.40 =: 0.07
98.51 0.08
89.18 1.94
50.67 4: 0.11
TCMP
25.99 0.52
11.17%
90.96 +: 0.16
98.43 à 1.04
43.23 2.96
98.38 0.91
96.53 1.47
93.39
50,30 0.78
STOSy
25.51 0.32
91.26 0.23
88.02
34.58 1.84
18.18
0.29
89,10 0.53
47.85 8:48
$1.68 0.56
STa
23.71 = 0.39
7.00% L
90.28 :: 0.95
85.79 6,83
34.04 7.18
87.15
4,48
$1.55 1.20
50.70 12.74
50.29 0.20
STaSy+TCM
22,78 0.69
10.71%
90.56 1: 0.44
86 4.18
34.08 1,46
87,16 3.78
86.56 4.26
50.08 6.30
50.61 0:41
TCMP
= 1.21
13.03%
85.82
34.11
87.38
0.95
TaNDDPM
31.37 = 0.31
92.17 + 0.32
93.16 1.58
52.57 1.30
97.08 0.46
92.92 = 3.27
86.74 0.63
$1.36 0.63
Stoppes
TabDDPM LFF
26.45 = 0.61
15.67%
91.29 0.43
87.90 0.43
46.28 0.89
95.18 = 0.60
58.69 4.57
85.11 + 0.54
50.44 2.24
HANDDPM-TCM
25.56 1.17
18.51%
92.17 + 0.26
94.41
50.05 1.59
97.18 0.34
93.95 0.51
86.96 0.50
47.52 1.81
TINDDPM-TCMP
28.51 0.38
9.12% L
+
93.43
$2.30
94,79-
87.02
0.59
Ta88yn
27.68 - 0.10
91.36 + 0.66
99.20 0.29
47.79 0.77
98.54 0.19
97.83 0.10
95, 0.39
$2.50 0.44
TubSyn-IJF
0.28
7.73% 4.
91.11 1: 1.01
98.46 0.93
43.72 4.33
96.22 1.85
96.39 1.80
91.05 4.58
$1.35 1.61
TabSyn-TCM
8.30% L
91.13
99.11
0.90
0.06
97.28
49.92 1.59
TTMM
-.29 0.23
6.33% L
91.75 1: 0.17
99.24 1: 0.55
46.18 0.77
98.60 0.14
97.77 0.09
97.10 0.57
50.21 3.33
STaSy
77.52 0.27
92.92 + 0.30
91.18 1,30
46.07 1.61
88.12 7.08
90.27 + 6.53
75.02 4.03
$2.57 + 0.92
MIP
72.52 6.59
6.45%
90.00
90.54
24.11
84.58 0.49
75.70
50.79 1.14
STSSSTTM
0.29
3.10% 1
91.49
92.50
35.24
89.62
6.44
75.70
49 0.21
STaSSUTCMP
76.70 - 0.38
1.06% 4
92.77 + 0.20
97.27 1.30
40.11 + 1.63
95.37 1.51
96.34 0.42
76.63 6.85
48.41 0.22
TobDDPM
77.62 = 2.11
92.78 1 0.23
98.41 0.37
46.67 1.18
99.07 11 0.06
98.58 1 0.51
99.05 0.70
50.47 0.11
Magic
TEADDPM-UF
63.54
16.15% J.
91.81
21.05
0.78
95.90
0.86
TINDDPM-TCM
72.99 0.22
5.96% L
91.69 1 0.86
97.92 4: 0.38
32.51 0.70
98.97 0.08
99.19 0.11
97.62 à 2.44
49.73 0.35
TANDPMM-TCMP
76.22 0.39
1.81% 1
91.50 0.22
96.50 1: 4.02
36.52 2.43
98.08 :: 1.32
95.17 3.05
95.29 6.22
49.88 0.74
TakSyn
80.02 ::: 0.39
93.18 4: 0.31
99.10 1.000.0000 0.68
48.28 4:
99.00 0.28
99.15 0.08
99.75 0.29
50.48 0.16
TabSyn=IJF
28.73% 1
21.43
92.89
-: 0.67
50.05 0.76
TAsSyn-TCM
$2.06 7.12
34.94% J
91.77 0.12
96.33 10.33tt00 0.40
30.79 2.92
97.83 0.65
98.09 0.17
93.55 1.49
$1.76 0.49
TCNS
76.46
4.44% J.
91.91 +: 0.42
98.03 1,76
39.54 :t:4:::11 1.54
98.87 0.57
97.26 0.27
97.58 :t 3,36
$1.32
27

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Table overview performance comparison tabular diffusion models on more generative models. "TCM" represents our proposed TabCutMix and "TCMP" represents TabCutMixPlus, "Mem. Ratio" represents memorization ratio.
"Improv" represents the
improvement ratio on memorization.
Methodk
Mem, Ratio (8)
Improv.
MLE (537
o-Piecision((S)"
A-Recal((S51
Shape Storc(1577
Trend SSore(S
CIS1(557
IXC((5)
Defauit
CTGAN-ICMP
11.80 0.25
8.03%
70.05 0.81
71.33 0.82
17.02 0.60
85.28 0.97
73.09 0.32
63.67 = 3.07
50.14 0.60
TYAE
17:22::043
72.24 + 0.36
82.97 0.37
20.77 0,42
89.374 0.54
83.36 0.99
52.63 0.37
51.33 0,96
TVAE+TCMP
16:30+00.2
9.47%
72.75
19.52
89.33
50.07
CIGAN
21.08 + 0.62
88.64 + 0.52
78.15 3.66
26.27 + 0.67
N2.43 0.90
82.83 0,93
63.64 = 2.74
49.14 0.27
Adolt
CTGAN-TOMP
+.330 0.31
2.23%
88.63 + 0.66
76.27 0.60
25.56 0.33
81,42 0.32
82,11 1,01
60.98 1.67
50.961 0.17
79.06
48.76 0.25
TVAE-ICMP
29.88
2.93%
88.42 d: 0.26
88.02 0.72
29.78 1.42
82.76 0.92
77.92 0.66
52.52 7.24
51.35 0.29
CTGAN
19.80 +: 1,15
83.22 1.30
85.26 5.17
26.67 1,66
77.73 0.50
N6.48 0.69
67.18 5.84
48. 0.74
Shappers
CTGAN-TCMP
52.02
TVAE
86.65 1: 0.48
50.32 2.98
11,16 1.33
74.75 0.92
77.56 1.11
21.29 3.28
43.31 0.65
TVAESTCMP
20.16 2: 0.80
12.98%
87.07 + 0.74
50,69 0.44
10.54 1.68
74.86 0.36
78.55 0,48
21,78 4.78
43.21 0.69
51.53 2.07
Magic
CTGAN-TCMP
60.12 1.94
6.70%
82.21 +: 0.43
82.72: 1.19
8.52 0.76
90.70 1.40
89.07 0.16
65.99 10.68
49.37 0.31
TYAE
77.20 + 0.32
88.71 +: 0.45
92.17 +: 0.43
32,11 0.81
91.67 0.75
93.75 0.43
77.18 1.38
48.71 0.59
TabCutMixPlus
TabCutMix
Mixup
SMOTE
Memorization Ratio
80
80
3 vs Mem-AUC
Corr: 0.9977
x
Corr: 0.9426
Corr: 0.9979
x
Corr: 0.9975
60
60
60
40
40
40
40
20
20
20
20
60
80
40
60
80
40
80
Mem-AUC
Mem-AUC
Mem-AUC
Mem-AUC
Memorization Ratio
80
80
Corr: 0.9974
x
Corr: 0.9438
Corr: 0.9971
x
Corr: 0.9959
3 vs Threshold 1 3
60
60
60
40
40
40
40
20
20
20
20
25
50
25
50
25
50
25
50
75
Memorization Ratio
Memorization Ratio
Memorization Ratio
Memorization Ratio
Memorization Ratio
80
80
Corr: 0.9971
x
Corr: 0.9444
Carr: 0.9993
x
Corr: 0.9988
60
60
60
60
Threshold
40
40
40
40
20
20
20
20
25
50
25
50
25
50
75
25
50
75
Memorization Ratio
Memorization Ratio
Memorization Ratio
Memorization Ratio
Figure 11. Correlation analysis between the memorization ratio with threshold 1/3 and Mem-AUC and the memorization ratio with different thresholds.
28

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Table 10. Mem-Ratio vs. Mem-AUC
Default TahDDPM
Magic TabDDPM
Shoppers TabDDPM
Adult TabDDPM
Methods
Metries
TabSyn
STaSy
TabSyn
STaSy
TahSyn
STaSy
TahSyn
STaSy
Pab
Mem--Rtio
17.60
17.73
14.81
76.26
75,80
76.99
26.11
28.81
23.57
26.20
26.10
22.24
Mem-AUC
33.73
33.70
29.71
79.21
77.15
79.53
37.85
42.44
34,99
38.36
40,54
33,98
Mem Ratio
17.59
17.17
14.11
45.99
73.19
75.18
25.30
26.27
23.02
27.20
27.37
21.78
TabCuMMix
Mem-AUC
32.61
32.10
29.15
78.26
78.93
79.37
37.08
42.44
32.99
38.23
41.93
34.35
Mixup
Mem Ratio
19.66
18.89
18.66
78.45
78.48
77.97
28.54
28,76
26.04
28.33
30.06
23.73
Mem-AU'C
35.45
34.65
33.64
79.24
80.07
79.57
40.12
41.74
37.24
42.42
44.62
36.15
Rar-Rario
19.29
15.95
72.14
76.42
25.48
26.63
22.85
26,96
SMOTE
Mem-AUC
34.12
33.23
30.21
73.63
74.55
77.58
36.70
37.77
34.14
39.76
42.45
36.54
during augmentation.
Challenges in Complex Domains. TabCutMix struggles in sensitive domains, such as healthcare or finance, where feature interactions often carry critical domain-specifee meanings. Arbitrary feature exchanges may result in implausible or nonsensical combinations, reducing the utility of augmented data for downstream tasks. For example, relationships between features like age and medical diagnosis may be violated, leading to unrealistic augmented samples.
Classification-speeiff applicability: TabCutMix and TabCutMixPlus are specifically designed for classification tasks, where samples can be grouped based on discrete class labels. This design makes it challenging to directly apply the methods to regression tasks, where the target variable is continuous and lacks discrete boundaries for grouping. Future work could explore strategies such as pscudo-labeling, binning continuous targets, or developing regression-awe aware augmentation techniques to adapt the core ideas of TabCutMix to regression settings and expand the scope of its applicability.
Sensitivity to Outliers. The proposed mixed-distance metric, like many distance-based measures, is sensitive to outliers, particularly in numerical features, which can disproportionately affect distance calculations and distort relationships between samples. While normalization mitigates feature dominance, it does not fully address the impact of extreme values. Future work could explore robust distance metrics, such as adaptive scaling trimming, to reduce the influence of outliers and improve the reliability of distance-based approaches in tabular data modeling.
Lack of General Insights into Data-Centric Factors. While this work identifies the influence of data-centric factors, such as dataset complexity and feature interactions, on the effectiveness of TabCutMix, it does not provide a comprehensive framework for understanding addressing these factors. A deeper investigation into these data-centric elements is necessary to fully realize the potential of TabCutMix and similar augmentation techniques.
G. Future Work
While this study makes significant strides in addressing the issue of memorization in tabular data generation, several directions remain open for future exploration:
1. Theoretical Analysis of Factor Heterogeneity: A deeper theoretical investigation is valuable into how different factors, such as dataset size and feature dimensionality, influence memorization in heterogeneous ways. Specifically, understanding the nonlinear relationships between these factors and their combined effect on model memorization could provide further insights.
2. Exploring Alternative Memorization Mitigation Techniques: Beyond data augmentation, future work could explore different strategies to mitigate memorization. These could include more advanced generative model training techniques, such as regularization methods or differential privacy mechanisms that limit model overfitting to specific data points. Additionally, techniques like model pruning, weight clipping, dropout variations could be explored for their potential to reduce memorization during model training. The model architecture design by leveraging architectures like variational autoencoders (VAEs), normalizing flows, or GAN variations with modified loss functions can be developed to mitigate memorization. The prior information of the flag to the model to indicate whether it is processing real or augmented data can also integrated for advanced memorization mitigation method.
3. Evaluation Metrics for Memorization: Developing more comprehensive and practical evaluation metrics specifically tailored to detect memorization in tabular data models remains key area for future work. These metrics could better assess the trade-off between generating high-quality synthetic data and avoiding overfitting to the training data.
29

---

Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
4. Real-World Applications and Use Cases:
Applying these methods broader range of real-world cases could provide valuable feedback and improvements. Specific industries such healthcare, finance, and marketing, where tabular data is prevalent, would be ideal candidates for testing how well these approaches generalize and perform in production environments.
5. Data-Centric Investigation for Tabular Diffusion Models: This study reveals that memorization in tabular diffusion models may be predominantly driven by dataset-specifie factors such as feature complexity, sparsity, and redundancy, rather than model-specific architecture. A deeper exploration into these data-centric influences could provide valuable insights into the interplay between dataset properties and memorization behavior. Future work could focus on developing strategies quantify the impact of dataset characteristics on generative performance and proposing adaptive preprocessing. augmentation, or sampling methods tailored to diverse datasets. Such investigations would enhance the robustness and applicability of diffusion models across a wide range of tabular data scenarios.
6. Cross-Domain Generalization and Transferability Memorization Mitigation:
The generalization of memorization insights and mitigation strategies developed for tabular diffusion models to other data modalities (e.g., text, images, multimodal data) and diverse generative architectures (e.g.. VAEs, GANs, autoregressive models) are under-explored. This includes investigating shared factors (e.g., data sparsity, complexity) and domain-specifie drivers of memorization to establish unified detection frameworks and evaluating the adaptability of techniques like TabCutMix and TabCutMixPlus across models and domains. Empirical studies will assess their effectiveness reducing memorization while preserving generation quality, and adaptive variants tailored to specific architectures (e.g., sequential text models) or data characteris- tics (e.g., nigh-dimensional images) will be developed to broaden real-world applicability. Theoretical comparisons of memorization behaviors across heterogeneous data types will further clarify universal principles and context-dependent challenges.
30


Reviews:
[{"review_id": "RmZuGC9tpz", "reviewer": "Reviewer_ihym", "summary": "The paper examines memorization in diffusion models for tabular data, providing theoretical insights that demonstrate the optimal score function under the empirical distribution and show that generated data can replicate training samples. To mitigate memorization, the study introduces TabCutMix and TabCutMixPlus, which swap feature segments between training samples to reduce overfitting while preserving data quality. Experimental results confirm that these techniques effectively reduce memorization while maintaining generation quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.\n\n[**Q1**] I would suggest including (or at least referencing) important metrics such as precision and MLE in the main body of the paper rather than leaving them to the appendix.\n\n**Ans**: We have added more descriptions on evaluation metrics and the reference for these important metrics in the main text.\n\n[**Q2**] The Results in Proposition 3.1 are not new, it has been shown in e.g., [1].\n\n**Ans**: We have moved Proposition 3.1 in Appendix A and explicitly cited relevant references to prior works. Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models. \n\nOur main contribution lies in the empirical investigation of memorization phenomena within tabular diffusion models—a previously unexplored area. Moreover, our proposed novel methods, TabCutMix and TabCutMixPlus, effectively mitigate memorization issues specific to tabular data. We have clarified this distinction explicitly in the revised manuscript to better highlight our empirical findings and methodological contributions.\n\n[**Q3**] The paper's organization could be improved. Table 1 occupies too much space, pushing important details (for example, the introduction of the evaluation metrics) to the Appendix. And Figure 4 appears after Figure 5, etc.\n\n**Ans**: We have reorganized the paper to improve clarity and readability. Specifically, we have moved Magic dataset results from Table 1 to Table 8 (Appendix E.6). We have added evaluation metrics details and reference, in the main text.\n\n[**Q4**] I'm a bit confused about the motivation for using MLE (machine learning efficiency evaluation) as a metric for data fidelity.\n\n**Ans**: MLE evaluates data fidelity by measuring how well a model trained solely on synthetic data performs on real data (e.g., via AUC on a downstream task), which directly reflects the utility of the synthetic data in practical applications. While traditional fidelity metrics like shape score and trend score assess how closely the synthetic data's distribution matches that of the real data, MLE provides an **end-to-end evaluation that incorporates task-specific performance**, offering a complementary perspective on whether the generated data preserves meaningful patterns for model training.", "claims_and_evidence": "Yes, both the theoretical and empirical results are justified in the work.", "methods_and_evaluation_criteria": "Yes, the authors incorporate a variety of metrics to assess different aspects of the model. For instance, they introduce a memorization score specifically designed for tabular data, along with $\\alpha$-precision and $\\beta$-recall to evaluate data fidelity. All the chosen metrics are well-justified.\n\nI would suggest including (or at least referencing) important metrics such as $\\alpha$-precision and MLE in the main body of the paper rather than leaving them to the appendix.", "theoretical_claims": "I didn’t verify the proof in detail but reviewed the approach used to derive the theoretical results.", "experimental_designs_or_analyses": "I think the overall experimental designs make sense and the empirical results are abundant to support the paper's claims.", "relation_to_broader_scientific_literature": "Investigating the memorization phenomenon and the approaches of diffusion models is an important topic. While most of other work focus on image data, this work considers tabular data to broaden the scope of such studies, and the proposed techniques are potentially transferrable to other data formats.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n* The experiments are quite comprehensive both in terms of the different scenarios considered and the evaluation metrics used.\n* The proposed methods are both intuitive and well discussed.\n\n**Weaknesses**\n* Results in Proposition 3.1 is not new, it has been shown in e.g., [1].\n* The paper's organization could be improved. Table 1 occupies too much space, pushing important details (for example, the introduction of the evaluation metrics) to the Appendix. And Figure 4 appears after Figure 5, etc.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I'm a bit confused about the motivation for using MLE (machine learning efficiency evaluation) as a metric for data fidelity. Can the authors elaborate more here?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NWd1fgydY", "reviewer": "Reviewer_aS37", "summary": "The paper explores the issue of data memorization in tabular diffusion models, highlighting its potential privacy risks and negative impact on generalization. To explain why memorization arises in a tabular diffusion setting, the authors present a theoretical analysis that connects denoising score matching to the tendency of replicating training data. They propose two augmentation-based strategies—TabCutMix (TCM) and TabCutMixPlus (TCMP)—inspired by the CutMix approach, which are designed to mitigate memorization while preserving data utility.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.\n\n[**Q1**] I think studying memorization in tabular data might be beneficial for other domains as well.\n\n**Ans**: We agree that our study on memorization in tabular data can have broader implications for other domains. We have added the discussion in Future work (Appendix. G) in the revised paper.\n\n[**Q2**] The techniques may generate out-of-distribution samples, which can pose challenges in certain practical scenarios.\n\n**Ans**: While data augmentation may lead to out-of-distribution (OOD) samples, our method TabCutMixPlus is designed to mitigate this risk through structure-aware augmentation at the feature-group level. As shown in Table. 5 (Appendix D.4.6), TabCutMixPlus consistently yields lower OOD detection ratios (e.g., 25.44% vs. 39.47% on Default, and 0.70% vs. 1.58% on Shoppers). We highlight OOD detection experiments more clearly in the main text of the revised version.\n\n[**Q3**] Currently, TCM/TCMP focus on classification tasks and may not generalize as readily to regression settings, which limits the scope of the paper.\n\n**Ans**: TabCutMix and TabCutMixPlus rely on exchanging features between samples of the same class, which inherently limits their applicability to classification settings. As a result, extending our methods to regression tasks remains a challenge and is beyond the scope of this paper. We have acknowledged this limitation in Limitation Discussion (Appendix. F) of the revised version.\n\n[**Q4**] It would be helpful for the authors to evaluate OOD ratios for Mixup and SMOTE as well, providing a more direct baseline comparison alongside TCM/TCMP (Section D.4.6) Table. 4. \n\n**Ans**: We thank the reviewer for the suggestion. As requested, we have added OOD detection results for Mixup and SMOTE (now included in Table 5). The “Ratio%” column represents the proportion of samples detected as out-of-distribution. While Mixup shows minimal OOD risk, it performs poorly in mitigating memorization compared to our methods. In contrast, TabCutMixPlus achieves a favorable balance—substantially reducing memorization while maintaining low OOD ratios (e.g., 0.36% on Adult vs. 2.06% for TabCutMix and 0.75% for SMOTE), making it a more effective and reliable augmentation strategy for tabular diffusion models.", "claims_and_evidence": "**Memorization occurs in tabular diffusion models.**\n- The authors conduct empirical evaluations showing significant memorization across multiple diffusion-based tabular generators and datasets. They also support these findings with a theoretical rationale.\n\n**TabCutMix and TabCutMixPlus effectively reduce memorization without significantly harming synthetic data quality.**\n- The paper includes experiments demonstrating that TCM and TCMP offer an improved trade-off between memorization reduction and data utility.", "methods_and_evaluation_criteria": "- The proposed techniques (TCM/TCMP) adapt the CutMix concept from image processing to tabular data by selectively swapping or mixing feature segments between samples.\n- TabCutMixPlus further clusters correlated features before swapping to preserve coherent relationships within the data.", "theoretical_claims": "The paper provides a reasonable theoretical explanation for why memorization arises in tabular diffusion models, grounded in the denoising score-matching framework. No immediate issues are found with their theoretical arguments.", "experimental_designs_or_analyses": "- The authors run extensive experiments demonstrating that TCM and TCMP consistently yield better trade-offs between memorization reduction and overall data utility.\n- The experiments cover multiple tabular datasets and also analyze aspects such as out-of-distribution (OOD) issues arising from the proposed augmentations.", "relation_to_broader_scientific_literature": "I think studying memorization in tabular data might be beneficial for other domains as well. In particular, the proposed techniques might be extendable to other tasks.", "essential_references_not_discussed": "None identified.", "other_strengths_and_weaknesses": "**Strengths*\n- The problem of memorization is well-motivated and clearly articulated.\n- Simple yet effective augmentation strategies are proposed.\n\n**Weaknesses**\n- The techniques may generate out-of-distribution samples, which can pose challenges in certain practical scenarios.\n- Currently, TCM/TCMP focus on classification tasks and may not generalize as readily to regression settings, which limits the scope of the paper.", "other_comments_or_suggestions": "- It would be helpful for the authors to evaluate OOD ratios for Mixup and SMOTE as well, providing a more direct baseline comparison alongside TCM/TCMP (Section D.4.6).", "questions_for_authors": "- See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q36jXhfaBl", "reviewer": "Reviewer_dji2", "summary": "This paper introduces TabCutMix and TabCutMixPlus, two augmentation methods designed to reduce memorization in tabular diffusion models. The authors demonstrate that state-of-the-art diffusion models tend to memorize tabular datasets and show that their proposed augmentations mitigate this issue. They further evaluate the effectiveness of both methods across multiple datasets and diffusion models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.\n\n[**Q1**] The results in Table 1 show that, across different datasets, the memorization ratio decreases by only a few percentage points. [Figure 6. Add the ratio for different thresholds]\n\n**Ans**: We respectfully disagree that the gains on memorization ratio reduction are not significant. In Table. 1, we report the performance with balanced tradeoff in memorization ratio and data utility. Our proposed methods can achieve significant memorization ratio reduction. For example, in default dataset Tabsyn model, TCM and TCMP reduce memorization ratio by 16.16% and 12.48%, which is significant compared with mixup and smote baseline (2.65% and 6.93%). Additionally, as shown in Table 3 and Figure 6 with different augmentation ratio, TabCutMix provides a clear and tunable reduction in memorization via its augmentation ratio from 0% to 100%. For example. In Default dataset, the memorization rate drops from 20.11% to 15.34%, achieving a relative reduction of 23.7%.\n\n[**Q2**] The theoretical result in Section 3.2 is somewhat straightforward, as it directly follows from the assumed delta distribution.\n\n**Ans**: We have trimmed down the theoretical part and add more details and reference for additional experiments in appendix. Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models.\n\nWe clarify that our main contribution lies in the empirical investigation of memorization phenomena within tabular diffusion models—a previously unexplored area. Moreover, our proposed novel methods, TabCutMix and TabCutMixPlus, effectively mitigate memorization issues specific to tabular data. We have clarified this distinction explicitly in the revised manuscript to better highlight our empirical findings and methodological contributions.\n\n[**Q3**] Have you experimented with adding a flag to the model to indicate whether it is processing real or augmented data? This type of augmentation conditioning has been explored in EDM networks [1].\n\n**Ans**: We thank the reviewer for this insightful suggestion. We have not yet experimented with adding an explicit flag or conditioning signal to indicate whether a sample is real or augmented. We agree this augmentation-aware conditioning, as explored in EDM networks [1], could help the model better distinguish between real and synthetic patterns and improve both generalization and memorization control. We view this as a promising direction and will include it in the revised manuscript under Future Work discussion.\n\n[**Q4**] How do you handle discrete features for the diffusion models as the framework is mainly designed for continuous signals?\n\n**Ans**: Our proposed methods, TabCutMix and TabCutMixPlus, are model-agnostic and operate by exchanging feature values across samples within the same label. This augmentation strategy applies equally well to both discrete and continuous features, and we have successfully implemented it across various tabular diffusion models, including TabSyn and TabDDPM.\n\nWe acknowledge that diffusion models are primarily designed for continuous data, and different models adopt different strategies for handling discrete features. For example, TabSyn uses a VAE to encode mixed-type features into a continuous latent space, where diffusion is applied. These techniques enable continuous diffusion models to effectively represent and generate mixed-type tabular data. We will clarify this point in the revised manuscript to avoid potential misunderstandings.\n\n[1] Karras T, Aittala M, Aila T, Laine S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems. 2022 Dec 6;35:26565-77.", "claims_and_evidence": "The empirical results are well-supported through various experiments. However, there is a concern regarding the practical significance of the proposed augmentation methods. The results in Table 1 show that, across different datasets, the memorization ratio decreases by only a few percentage points. This raises questions about the overall impact and effectiveness of the method in reducing memorization.", "methods_and_evaluation_criteria": "The methods are evaluated using standard metrics commonly used in the field.", "theoretical_claims": "The theoretical result in Section 3.2 appears trivial. The authors assume a delta distribution for the data points and derive the ground truth score functions based on this assumption. Consequently, if the reverse diffusion process is run using these ground truth scores, the generated samples will be drawn directly from this delta distribution—essentially replicating a training sample. Under this assumption, it is evident that the sampling process would lead to 100% memorization.", "experimental_designs_or_analyses": "The experiments are well-designed and effectively demonstrate the impact of each component in the proposed methods.", "relation_to_broader_scientific_literature": "While memorization has been extensively studied in image and video diffusion models, it remains an underexplored issue in tabular data generation. Given the significance of tabular data generation across various domains, this paper is well-situated within the broader literature and studies an important gap in the field.", "essential_references_not_discussed": "All essential references are discussed in the main text.", "other_strengths_and_weaknesses": "### **Strengths**\n* The paper is well-written and easy to follow.\n* The experiments are extensive and conducted with clear purpose.\n* The authors provide ample details on various experiments, metrics, methods, and implementation details.\n\n### **Weaknesses**\n* The proposed method shows only a modest reduction in the memorization rate, raising questions about its practical significance.\n* The theoretical result in Section 3.2 is somewhat straightforward, as it directly follows from the assumed delta distribution.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1) Have you experimented with adding a flag to the model to indicate whether it is processing real or augmented data? This type of augmentation conditioning has been explored in EDM networks [1].\n\n2) How do you handle discrete features for the diffusion models as the framework is mainly designed for continuous signals? \n\n[1] Karras T, Aittala M, Aila T, Laine S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems. 2022 Dec 6;35:26565-77.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcQe2u00wg", "reviewer": "Reviewer_BdAp", "summary": "This paper investigates memorization behavior in diffusion models for tabular data generation. The authors examine how factors like dataset size, feature dimensionality, and model architecture influence memorization. They propose a data augmentation method called TabCutMix and an enhanced version, TabCutMix-Plus, that additionally mitigates out-of-distribution generations. The paper provides a theoretical explanation for why diffusion models tend to memorize training data in the tabular domain and demonstrates the effectiveness of their proposed methods through empirical evaluations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.\n\n[**Q1**] The paper's theoretical claims about memorization in diffusion models for tabular data lack novelty and are insufficiently substantiated. Propositions 3.1 and 3.2 and found them to be essentially identical to results already established in [Gu et al., 2023] for image generation.\n\n**Ans**: We acknowledge that the theoretical results presented in Proposition 3.2 build on existing foundations (e.g., [1]). Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models. As discussed in the manuscript (see the paragraph surrounding Proposition 3.2), we leverage these theoretical insights to explain why memorization can arise in the latent space of tabular data models such as TabSyn, and why it does not necessarily lead to exact duplication in practice—due to factors like imperfect score matching and the stochasticity of VAE decoding.\n\nOur primary contribution is the first empirical investigation of memorization in tabular diffusion models, a setting that has been underexplored in the literature. Furthermore, we propose two novel and effective mitigation strategies, TabCutMix and TabCutMixPlus, which address memorization in a model-agnostic and structure-aware way. We have revised the manuscript to more clearly distinguish between the reused theoretical insights and our original empirical and methodological contributions.\n\n[**Q2**] The evidence for the effectiveness of TabCutMix and TabCutMix-Plus in mitigating memorization regarding a specific distance based metric is presented, but its unclear if this metric is well-suited for the task especially given the VAE's latents are used for the final generation.\n\n**Ans**: Prior work on image generation [1,2] commonly uses L2 distance to assess similarity in the input space. Building on this and mixed-type data clustering techniques [3], we adopt a standard mixed-type distance metric suitable for both continuous and categorical features in tabular data.\n\nWe also clarify that our memorization metric is computed in the original input space, not in any model’s latent space. Although TabSyn uses a VAE, our method is model-agnostic, and the evaluation remains unaffected. We will clarify this further in the revised manuscript.\n\n[1]  Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop.\n[2] On memorization in diffusion models. arXiv:2310.02664.\n[3] An improved k-prototypes clustering algorithm for mixed numeric and categorical data. Neurocomputing 2013.\n\n[**Q3**] The method for determining memorization in tabular data is problematic. Why use 1/3 to judge if a sample belongs to be memorized.\n\n**Ans**: While we initially use 1/3 as a representative threshold to determine whether a sample is memorized, we agree that threshold selection can influence interpretation. To address this, we conducted a correlation analysis across multiple thresholds (e.g., 1/4, 1/3, and 1/2) and with Mem-AUC (which averages memorization ratios over all thresholds). As shown in Figure 11, the memorization ratio at threshold 1/3 exhibits extremely high correlation (e.g., >0.99) with both Mem-AUC and other thresholds, demonstrating that 1/3 serves as a reliable and representative choice. We have added this discussion and the supporting analysis to the revised manuscript.\n\n[**Q4**] The proposed mitigation method relies solely on data augmentation rather than addressing model architecture or training procedure issues, which limits its applicability as memorization can still occur with the augmented data.\n\n**Ans**: We respectfully disagree with the concern that our method is limited in applicability due to its focus on data augmentation. Our approach is model-agnostic and has been successfully applied across a diverse set of generative models and training procedures, including StaSy, TabDDPM, TabSyn, CTGAN, and TVAE. This demonstrates its broad compatibility and effectiveness regardless of the underlying architecture.\n\nFurthermore, we view our method as orthogonal to model design and training objectives—it can be readily integrated into more advanced generative models or used alongside other memorization control techniques. In fact, we believe it can further enhance models that already include architectural or optimization-based memorization defenses. We have included a discussion of these complementary directions in Appendix G (Future Work) of the revised manuscript.", "claims_and_evidence": "The paper's theoretical claims about memorization in diffusion models for tabular data lack novelty and are insufficiently substantiated. The authors claim theoretical contributions that appear to be direct adaptations of existing results from [Gu et al., 2023] without any modifications for the tabular domain. The claim that their analysis \"specifically addresses tabular data with mixed feature types\" is not well-supported as the core theoretical results (Propositions 3.1 and 3.2) apply to points in Euclidean space regardless of whether they represent images or tabular data's latent encodings.\n\nThe evidence for the effectiveness of TabCutMix and TabCutMix-Plus in mitigating memorization regarding a specific distance based metric is presented, but its unclear if this metric is well-suited for the task especially given the VAE's latents are used for the final generation.", "methods_and_evaluation_criteria": "The method for determining memorization in tabular data is problematic. The paper uses a mixed distance metric in the data space without adequate justification for why this specific distance and the empirical 1/3 hyperparameter align with meaningful memorization phenomena, especially regarding the utility and quality of generated data. Additionally, since the models evaluated are latent diffusion models where the final generation requires decoding through a VAE, measuring distance in data space fails to capture potential memorization effects introduced by the VAE itself. This creates a fundamental issue with the evaluation criteria that isn't addressed.\n\nThe proposed mitigation method relies solely on data augmentation rather than addressing model architecture or training procedure issues, which limits its applicability as memorization can still occur with the augmented data.", "theoretical_claims": "I checked the theoretical claims in Propositions 3.1 and 3.2 and found them to be essentially identical to results already established in [Gu et al., 2023] for image generation. The authors' statement that \"The analysis is closely related to the work of Gu et al. (2023)\" understates the similarity; in fact, the exact same analysis applies directly to tabular data without modification because both domains ultimately work with points in Euclidean space. Therefore, the claim that this represents a novel theoretical contribution specifically for tabular data is overstated.", "experimental_designs_or_analyses": "There's no ablation study examining how different memorization thresholds would affect the results or analysis of the VAE's potential contribution to memorization.", "relation_to_broader_scientific_literature": "This paper's theoretical contributions heavily overlap with those of [Gu et al., 2023], which already established similar results stated for image generation. The authors acknowledge this relationship but understate the extent of the similarity. The paper would benefit from more clearly positioning its contributions relative to existing work on memorization in generative models across domains.", "essential_references_not_discussed": "No major omissions noted, though a broader discussion of memorization metrics across different domains would strengthen the paper.", "other_strengths_and_weaknesses": "Strengths:\n- The problem of generating tabular data while avoiding memorization is important and interesting.\n- The empirical investigation of factors influencing memorization provides useful insights.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does the quality and utility of data generated using TabCutMix compare to data generated by the baseline model? Can you provide metrics beyond memorization that demonstrate the practical benefits of your approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies memorization in diffusion models for tabular data, showing that the optimal score function under the empirical distribution can lead to regeneration of training samples. To mitigate this, it proposes TabCutMix and TabCutMixPlus, which swap feature segments to reduce overfitting while preserving data quality. Experiments confirm reduced memorization without compromising generation quality.

Most reviewers find the studied problem important, the method can be widely applied, and the evaluation comprehensive.
However, the paper needs a more comprehensive literature review on related works.
Please incorporate the reviewers' feedback during the revision.
Decision: Accept (poster)