CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Authors: Aditya Gorla
Ryan Wang
Zhengtong Liu
Ulzee An
Sriram Sankararaman
Venue: ICML 2025 spotlightposter (2025)
Keywords: Tabular data imputation
Inductive bias
Context awareness
Data-driven masking
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: A novel masked autoencoding imputation method that learns from real missingness patterns and feature context

Abstract: We present CACTI, a masked autoencoding approach for imputing tabular data that leverages the structure in missingness patterns and contextual information. Our approach employs a novel median truncated copy masking training strategy that encourages the model to learn from empirical patterns of missingness while incorporating semantic relationships between features — captured by column names and text descriptions — to better represent feature dependence. These dual sources of inductive bias enable CACTIto outperform state-of-the-art methods —  an average $R^2$ gain of 7.8\% over the next best method (13.4%, 6.1%, and 5.3% under missing not at random, at random and completely at random, respectively) — across a diverse range of datasets and missingness conditions. Our results highlight the value of leveraging dataset-specific contextual information and missingness patterns to enhance imputation performance.

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Aditya Gorla 12 Ryan Wang² Zhengtong Liu
Ulzee An 13 Sriram Sankararaman 134
Abstract
ness can arise due to variety of mechanisms. Existing methods either explicitly or implicitly make simplifying assumptions about these mechanisms motivated by infer- ential tractability (see Section 2.1 and Jarrett et al. (2022)). These assumptions rarely hold in real-world settings and practitioners often lack prior knowledge on the underlying missingness mechanism. Consider medical survey where questions are hierarchically structured, with more specific inquiries contingent upon affirmative responses to general ones so that a patient is only asked about specific symptoms if they report a broader health issue. In this example, entries pertaining to more specific health issues will be missing depending on the observed values or missingness status of entries relevant to broader health status. We hypothesize that the missingness patterns in the data could potentially be leveraged to improve imputation accuracy.
We present CACTI, masked autoencoding ap- proach for imputing tabular data that leverages the structure in missingness patterns and contex- tual information. Our approach employs a novel median truncated copy masking training strategy that encourages the model to learn from empirical patterns of missingness while incorporating se- mantic relati relationships between features- captured by column names and text descriptions represent feature dependence. These dual sources of inductive bias enable CACTI to outperform sate--of-theant methods an average R² gain of 7.8% over the next best method (13.4%, 6. %, and 5.3% under missing not at random, ran- dom and completely random, respectively) across a diverse range of datasets and missing- ness conditions. Our results highlight the value of leveraging dataset-specific contextual informa- tion and missingness patterns to enhance imputa- tion performance. Code is publicly available at github.com/sirralaaSCAACT
better
Additionally, existing methods underutilize the rich con- textual information in the data. While they allow for the inclusion of fully observed covariates, they lack straight- forward mechanism effectively incorporate unstructured knowledge about the relatedness between or the context of the features being imputed. In the medical surveys exam- ple, the answer to the broader question can constrain the answers to more specific questions. We hypothesize that imputation models can use this prior information to inform their imputation.
1. Introduction
Missingness is pervasive problem in real-world tabular datasets with the potential to adversely affect downstream inferential tasks (Rubin, 1987; Schafer & Graham, 2002). While many techniques to estimate or impute missing en- tries have been proposed (see Section 2.1), missing data imputation remains a challenging problem.
Contributions In this work, we present Context Aware Copy masked Tabular Imputation (CACTI,, transformer- based architecture that leverages inductive biases from ob- served missingness patterns and textual information about features to address existing gaps in tabular data imputation. CACTI makes several novel contributions to tabular impu- tation. First, we introduce median truncated copy mask- ing (MT-CM), a novel training strategy that enables the effective application of copy masking (An al., 2023) to transformer-based Masked Autoencoders (MAE) (He et al., 2021). Unlike existing approaches which use complete data or random masks (Du et al., 2024), MT-CM uses empirical missingness patterns to guide the learning process. Our re- sults demonstrate that a naive application of copy masking to transformer based MAE architectures leads to subopti- mal performance while MT-CM addresses this gap. Second,
A primary reason underlying this challenge is that missing-
'Department of Computational Medieine, David Geffen School of Medicine, UCLA, Los Angeles, CA, USA 'Bioinformatics Interdepartmentall Program, UCLA, Los Angeles, CA, USA Deppatmment of Computer Science, UCLA, Los Angeles, CA, USA 'Department of Human Genetics, UCLA, Los Angeles, CA, USA. Correspondence to: Aditya Gorla aaddttaggrla@ucla.ddo Sriram Sankararaman <sriram @c..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
we provide theoretical motivation for MAE training with- out fully observed data which motivates the need for copy masking. Third, we leverage contextual information from feature names and descriptions as source of inductive bias. This context-aware approach enhances learning efficiency by minimizing reliance on learning features' relationships solely from limited observed data and provides direct way to incorporate unstructured information or prior knowledge. Fourth, comprehensive evaluation establishes CACTI as a state-of-the-ard tabular imputation approach across various missingness settings. Finally, both MT-CM and context awareness frameworks are simple and modular, allowing them to be used in conjunction with any deep learning frame- work beyond the tabular imputation domain.
Missingness Mechanisms Let us define selector func- tion sM, X [k=mmmm Nr that selects all the observed features in the complete data. X := SM. (X,) defines the observed part and X == S1.M. (X..) de- fines the missing part. The framework laid out by Rubin (1976) (also (Little & Rubin, 1987)) prescribes the follow- ing underlying missingness mechanisms, from the most to the least restrictive assumption: MCAR (p(M,, (X,) = p(M,), i.e. M, 1 X,, missingness is independent of the data), MAR (p(M,, Xx,)) = (M, X 9) missingness only depends on the fully observed data), and MNAR when the mechanism is neither MCAR nor MAR.
2.1. Related work
There are two main classes of tabular imputation methods: iterative and generative. Iterative methods iteratively im- pute the missing values in each feature by estimating the conditional distribution given all other features' observed data (van Buuren & Groothuis-Oudshoorr. 2011; Stekhoven & Bühlmann, 2011; Jarrett et al., 2022). While estimat- ing the conditional distribution is a simpler problem, these approaches are limited by challenges in selecting optimal conditional distributions and sometimes requiring complete observations for model fitting. In contrast, generative ap- proaches attempt to estimate a joint distribution of all the features which is a considerably harder statistical task than estimating univariate conditional probabilities (Yoon et al., 2018; Dai et al., 2021; Yoon & Sull, 2020; Mattci & Frellsen, 2019; Ipsen et al., 2021; Nazabal et al., 2020; Zhang et al., 2024a; Zheng & Charoenphakdee, 2023; Muzellec et al.,
2. Background
We begin by introducing the tabular imputation task adopt- ing notation similar to previous works (Jarrett et al., 2022: Ipsen et al., 2021; Yoon et al., 2018) to ensure consistency. The complete data for a single sample with K features, X, := (In1,-.. Fnk) E X² X1 x .... x Xx with k [K] features and n E [N] observations, is drawn i.idd from an arbitrary data generating process X,, DK-
We do not have access to the complete data but only the incomplete, observed data: X,, := (#n1..... _nk), Equa- tion (1). The incomplete data can be viewed as a corrupted version of the complete data mediated by the missingness mask Mn (m.1...., m,k) E {0, 1)x, where Fnk is observed if mnk 1 and Fa is missing (denoted as +) if
mnk 0:
2020). Many of these approaches require either complete data or restrictive assumptions on the missingness mecha- nisms (Nazabal et al., 2020; Richardson et al., 2020; Mattei & Frellsen, 2019). Other classical imputation approaches include: K-nearest neighbors, matrix completion and uncon- ditional mean substitution (Hastie et al., 2014; Hawthorne & Elliott, 2005).
Fnk=
Tnki
if mank
E & = X (+)
(I)
if m =
Across N observations, this process results in an observed
data matrix X- = (Xi;...;;,, and the associated mask M = (M;...;;,,
The imputation task can be formalized as a learning a func- tion f: X + X' resulting in an uncorrupted version of the incomplete data X,, == (Znt....,TT.k resulting in the fi- nal imputed dataset (X,, = (fnl...., Snn), Equation (2)):
Transformers Several recent works have proposed trans- former (or self-attention) based architectures to model tab- ular data (Huang et al., 2020; Arik & Pfister, 2020; Ma- jmundar et al., 2022; Yoon et al., 2020; Hollmann et al., 2025; Gardner et al., 2024). However, these approaches pri- marily focus on self-supervised learning tasks by employing a masked reconstruction task or target direct downstream prediction and do not explicitly address the imputation prob- lem. Recent works (Yin et al., 2020; Yang et al., 2024; Lin et al., 2024; An et al., 2025) have also leveraged unstruc- tured (natural language) contextual awareness to improve representation learning, pre-training efficiency and the per- formance of generative tabular models; however, these ap- proaches have not yet been effectively leveraged in tabular imputation.
Inks
mink =
Lyck
(2)
Inke if makk
Additionally, we might have access to additional informa- tion that can be leveraged to aid imputation. Specifically, we assume we have access to external information (shared across all N samples) such as the semantic context and relatedness between features which can be represented as C := (CI,..., E RCXK a C-dimensional embed- ding representation of context information for each feature.
2

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
3. CACTI
Observed Data
Masked Data
*
*
A15
Random Masking
CACTI employs an encoder-decoder Transformer architec- ture for tabular data imputation. This architecture needs to be trained on a reconstruction task. However, since the observed data is incomplete, masking strategy that in- troduces additional missingness on which the quality of reconstruction can be assessed must be devised.
Ap
My4
Hgp
Ays
Ayp
X14
Masking
Naive Copy
Fai
+
*y4
3.1. Median truncated copy masking
Previous works for tabular data imputation (Du et al., 2024) adopt the same approach used in MAEs: applying random mask on the observed portions of the incomplete data dur- ing training. Our first contribution is in replacing random masking. We extend naive copy masking (An et al., 2023) to develop median truncated copy masking (MT-CM) which leverages the missingness structure in the observed data to create masks that better reflect true missingness patterns. We hypothesize that this approach provides a useful inductive bias for the model that can be particularly effective in cases where missingness is structured (Jackson et al., 2023), e.g., consider the missingness pattern p(mas 0/mm, = 0) = where feature is missing any time j missing. While is challenging define a unified well-defined generative model for the mask, the empirical patterns of missingness provide useful information to design such a mask.
Missing Cell Values
Median Truncated Copy Masking
x
Masked Call Values
M
A
Observed Cell Values
Figure I. Median Truncated Copy Masking overview. In con- trast to random masking, where some subset of features are masked uniformly at random, copy masking recycles missing value patterns actually present in the dataset. This approach simulates realistic missingness patterns that provide a source of useful inductive bias during training. Median Truncated Copy Masking extends this strategy for MAE training by truncating the number of features available to the encoder, ensuring it has access to most me- dian number of fully observed features in each batch.
Under MT-CM (and naive copy masking), we can segregate features in each sample into three sets: the (observed but) masked values M = (k: (mnn- unmasked values 0 {k: (mak = 1) n (m 1)} and the true missing values V, (k: makk 0}. Conse- quently, we can define training strategy by minimizing reconstruction loss over the value sets M and 0
ReMasker (Du et al., 2024), ransformer-based approach for tabular imputation that builds on the MAF approach (He et al., 2021), learns to reconstruct randomly masked values based on the unmasked observed values (Figure 1). The model is highly expressive but trained under a (com- pletely) random masking strategy during training.
(m 0)}, the
A naive application of copy masking (Figure 1) to transformer-based MAE architectures, however, leads to inefficient learning due to the large variance in missingness proportions across samples (Mitra et al., 2023: Jackson al., 2023; An al., 2023) while uniform feature sizes (or se- quence lengths) within a batch are critical for efficient learn- ing with a transformer- based encoder (Krell et al., 2022). A possible approach to enforce uniformity when using naive copy masking is to replace all missing or masked features with null token. This strategy, even at low copy mask- ing rates, results significant proportion of null tokens in each batch, which provides meaningful information for learning robust latent representation. Furthermore, increasing the copy masking rate proportionally increases the fraction of null tokens that can further reduce learning efficiency and overall model performance. Empirical results confirm this trend, with higher rates of naive copy masking leading to reduced model performance (see Appendix A).
Copy masking Recent work by An et al. (2023) proposed using the missingness patterns in the observed data to create masks to train an imputation model under a reconstruction loss function. Given an observed missingness mask M, copy maksing involves shuffling the matrix row-wise to create mask M E {0, N where with probability either apply the M mask for sample or leave it unchanged. We term the resulting mask matrix as the naive copy mask M (Figure 1; See Algorithm 1 for details'). While Autocomplete (An et al., 2023) implements naive copy masking in conjunction with shallow MLP to show strong downstream performance, this approach is limited in its expressivity to learn complex relational patterns between the features.
Pem (masking ratio)
Notice that this algorithm results in sampling M with replace- ment when number of epochs >
To tackle this issue, we propose the Median Truncation Copy
3

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Input
Context Aware Input Embeddings
Encoder
Latent Representation
Context
Latent Embeddings
Decoder
Imputed
Copy observed missingness patterns
8 nl
P
8,a
IMASK]
[MASK]
[MASK]
X D2
Observed
X
Vy
Vip
C-C
IMASK]
[MAK]]
2g
IMASKI
Rat
G
00
X
X
Median Truncated Copy Mask (MT-CM)
X
Vei
Via
Ves
G
[MASK]
IMASKI
IMASKI
G.
X
Sample]
Sample
Volue embeddingg
Contaxt ambeddings
Lo.
10wl (Rakk Rna)
8 nd
Contast Information
Missing values
.Moskdd values
E. (Rat #nee?
2 ducrition
Css
3 description
[Mal
49/09/9
LCACTr(X, (X. Xx) Lo. + Lox.
Teature cincription
Peatur dideotiptionn
Language model embeddings
O, :oberrved vallas .Bii valuaes
Figure 2. CACTI model overview. CACTI samples observed missingness patterns to generate masks via Median Truncated Copy Masking (MT-CM) guide the learning. Features' context are also embedded with a language model. The MT-CM strategy masks out some portion of the observed features from sample n using observed missingness patterns from other samples (j) in the same dataset. This is followed by concatenating context information to the remaining (unmasked) features. A transformer encoder processes this data. Then the model adds context information and [MASK] tokens for the missing/masked features before being processed by the decoding transformer which reconstructs the values. CACTI optimizes reconstruction loss (CCACT)) over observed and masked features produce the final imputation estimates.
Masking (MT-CM) training strategy (Figure 1). Let NB be the number of samples in the B3th batch, and on @ be the number of observed features in the n-th sample after the application of naive copy masking in this batch. The median number of observed values within the batch is defined as medsan = (ondianno)....;0..0. BB
complete data vector undergoes a corruption process medi- ated by missingness mask which results in the partially observed data: = = M where the missingness mask process M ~ PMIX-
Under a masked autoecoding model, we aim to learn an encoder-decoder (fv, and da respectively) that minimizes the risk:
The MT-CM strategy truncates the sequence length of ob- served values for each sample ensure contains no more than oB median observed values. Formally, for each sample n in the batch, the truncated sequence length ofrune is com- puted as offune min min(on, 0g omeddia) This ensures that the proportion of null tokens in any batch is upper bounded by 50% regardless of the copy masking rate. Overall, MT- CM results the final set observed and masked features: On, [k (mak = 1) (m 1) n (k S of and M, MC (n), respectively. During training, the feature order of each sample in every batch is permuted to ensure that the first otrune features are retained observed features and is different every iteration. See Algorithm 2 for extended MT-CM details. We also empirically show that, unlike naive copy masking, our MT-CM strategy results in overall performance increasing as the copy masking rate increases (see Appendix A).
R(b, 0)
Ex.M [ (1-11MM M) doff M))
M)
Here @ denotes entrywise product.
The risk (or its finite-sample approximation) defined in Equation 3 cannot be computed since we only observe X. Instead, given the missing data X, generate mask M M Q X.M and aim to minimize the alternate risk:
RQ 0) EX. E EM X| @ M 1 (1- M')
do M M))
& M (1 1
Consider sample that is completely observed so that M
1 so that X = X. On this sample, RQ becomes:
3.1.1. 1. THEORETICAL MOTIVATION FOR COPY MASKING
Ro (o 0) Ex [M [|] @ (1 MM)
In this section, we provide a brief theoretical motivation of the need for copy masking.
do (X M)) @ (1 M)]
(5)
Assume the complete data for a single sample drawn from an arbitrary data generating distribution X i.i.d Px(x). This
Equation 5 motivates choosing Q to be the same distribution as M|X so that Rg x R. More broadly, this motivates choosing a masking distribution Q that approximates the true distribution of missing entries M For example, if the true missingness mechanism is MCAR where the
In Algorithm 1, every sample retains 2 1 feature; M is not applicd if it results in 0 remaining observed features.
4

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
probability of each feature being missing is independent and identically distributed, a random masking strategy where each entry masked independent of other features with constant probability is expected to provide an appropriate inductive bias for the imputation model.
tions (embeddings) of each column's semantic information. For each of the K columns in the data X, we process the column name and description (when available) through a language model (using default tokenizer) to obtain embed- dings Cg. Given a set of tokenized descriptions Tx, we obtain the last layer hidden state for each token and ag- gregate the information to obtain the column's semantic context C = [7 Ei=i Efbed Since language mod- els (Devlin et al., 2019; Lee et al., 2025) typically have hidden state dimensions in the range [768, 4096], we per- form a linear projection r, C - that maps each col- umn information embedding to an C-dimensional context embedding, resulting in C = (Ci,..., CC) E RCCK Transformers also require fixed sin-cosine embeddings P (P1, PP...... ,x) R to preserve positional in- formation (Dufter et al., 2021). Thus final context-aware embeddings are achieved by concatenation of the value and context E., = [U,.|C] + PP, with positional information added.
These observations drive the core rationale for copy mask- ing. Copy masking tries to approximate the true masking distribution and missingness structure by sampling from the observed missingness mask. For example, in the MCAR setting where each feature has a constant probability of be- ing missing, copy masking will reduce to random masking. On the other hand, when the missingness probability varies across features, copy masking will lead to features being masked with differential probabilities based on their empiri- cal frequencies. The use of empirical masks can also capture correlations among features in the missingness mechanism. While copy masking is still simplification and does not fully model the missingness mechanism, our empirical re- sults suggest that it enables the imputation model to attend to realistic patterns of missingness.
Different base models can be used for generating context embeddings. In this study, we use the MMM-eee-ML--- new state-of-the art text embedding model (Zhang et al., 2024b) as the default based on our empirical results com- paring the effectiveness of these models. We note that the generation of column context embeddings has a one-time, fixed cost. These embeddings can be pre-computed and reused across multiple runs for the same dataset.
3.2. Context Awareness
Our second key contribution is making the imputation back- bone context aware by incorporating prior information about the semantic information associated with each feature by using language model embedding of feature description into the value embedding vector in both the encoder and de- coder stage. We use the semantic similarity between column name and description information to make our imputation backbone context-aware, providing useful inductive bias to improve imputation performance.
3.3. Transformer Backbone
Figure 2 provides a pictorial description of the CACTI au- toencoder architecture backbone with a detailed descrip- tion deferred Appendix B. Briefly, the CACTI backbone consists of an encoder and decoder, both utilizing trans- former architectures with (residual) self-attention blocks. The encoder processes context- aware embeddings (E,) of the observed data, dropping missing or masked features af- ter applying the MT-CM strategy. The decoder combines context information embeddings and a latent representation of the MT-CM input features to estimate the uncorrupted version (X,)) of the incomplete data. The model is trained to minimize the reconstruction loss between the imputed and observed data, using unified MSE loss over the masked (M,, and fully observed values (O,)). See Algorithm 3 and Algorithm 4 for a sketch of the CACTI implementation.
Let X, : (X......, Xax) ) be the observed data for sin- gle sample in X. For the final model embedding dimension E. would like to achieve a context-aware embedding of the data sample E, (E.1...., ExK) E REXK (where E = dim (E,x))). To achieve this, we can create par- titioned embedding for each feature, which has a value component Unk and context component Cnk such that E. = (Unzi Cx)), where UnA E R C₁ E RC and E U + C. As a design choice, we set U = 0.75E and C 0.25F, prioritizing value information the primary object of relevance which warrants its overrepresentation relative to context information. We define a linear projec- tion' 4 1: X + 14 that maps each scalar feature value to a U-dimensional embedding vector representation, resulting in U,, = (UnI...., Unk) E u
4. Evaluation Results
We propose using of language models to obtain representa-
We empirically evaluate CACTI's performance against state- of-the-art methods using 10 benchmarking datasets across all three missingness scenarios. Next, we conduct a thor- ough ablation analysis to quantify the contributions of our proposed MT-CM and context awareness strategics. Finally,
'C& has no index subscript n because we assume the feature context information is shared and consistent across all samples. *We set all the true missing values any special protected value for this step.
5

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table 1. Overall benchmark results. Average performance comparison of CACTI and CMAE (CACTI without context) against existing imputation methods the train/test splits (separated by over 10 datasets. Metries (arrows indicate direction of better performance) are evaluated under MAR, MCAR, and MNAR at 30% missingness. indicates method which cannol perform out-of-samples (test split) imputation. Best metric in bold and second best underlined. Extended table with standard errors in Appendix.
R2 (t)
RMSE (L)
WD (t)
METHOD
MCAR
MAR
MNAR
MCAR
MAR
MNAR
MCAR
MAR
MNAR
CACTI (OURS)
0.46
0.47|0.47
0.46|0.46
0.66|0.64 0.67
0.68|0.67
4.35/4.46
1.97
4.45/4.57
CMAE (OurS)
0.45
0.46|0.46
0.44
0.69 0.67
0.70]0.69
4.40|4.50
1.94|2.02
4.67
REMASKER
0.44 0.44[0.44
0.40|0.40
0.68|0.67
0.69/0.71
0.73
4.62/4.72
2.46/2.52
4.79/4.93
DIFFPUTER
0.40/0.42 0.39|0.43
0.36|0.37
0.73/0.70
0.77|0.75
0.79|0.77
4.53/4.56
2.55/2.38
4.79/4.90
HYPERIMPUTE
0.41|-
0.44|-
0.39|-
0.72/-
0.73/-
0.76|-
4.26/-
2.46|-
4.30|-
MISSFOREST
0.35|0.34
0.38|0.36
0.34|0..2
0.77|0.75
0.79|0.82
0.79|0.78
6.78/6.80
3.80/3.83
7.01|7.06
NOTMIWAE
0.35|0.35
0.35|0.35
0.29|0.30
0.75|0.74
0.80)0.82
0.82/0.80
5.56/5.60
2.38/2.39
6.26/6.20
SINKHORN
0.28|- 0.29|
0.26|-
0.84|
0.89|-
0.88|-
7.02/-
3.96/-
7.51|-
ICE
0.28)0.27
0.34[0.33
0.26|0.25
0.86|0.87
0.78|0.83
0.93/0.93
4.82/5.18
2.74/2.81
5.32/5.67
AUTOCOMPLETE
0.24|0.24
0.29|0.29
0.21|0.21
0.88)0.86
0.88|0.89
0.94[0.92
10.14/10.18
5.04|5.07
10.44
МICE
0.19|0.19 0.23|0.23
0.18|0.18
1.06|
1.04]1.05
1.08|1.07
8.25|8.33
4.16/4.23
8.34|8.49
GAIN
0.19 0.18/0.22
0.17|0.18
0.91|0.86
0.95/0.93
1.01/0.96
7.73/7.34
4.44|4.10
9.53
SOFTIMPUTE
0.09|0.10 0.10 0.09/0.09
1.02|0.96
1.06
1.05|0.99
8.35/7.86
4.84|4.46
8.73
MIWAE
0.00|0.00 0.00|0.00
0.00|0.00
1.00
1.05|1.07
1.03|1.00
7.83/7.90
4.53/4.57
8.36
MEAN
0.00 0.00|0.00
0.00
0.95
1.00
0.98|0.95
11.96/12.00
6.35|6.38
12.25 12.26
est (Stekhoven Bühlmann, 2011). ICE (Royston & White, 2011) and MICE (van Buuren & Groothuis- 2011). and generative approaches: Sinkhorn (Muzellec et al., 2020), GAIN (Yoon et al.. 2018), MIWAE (Mattei & Frellsen, 2019) and notMIWAE (Ipsen et al., 2021) (an extention of MIWAE for MNAR). Lastly, we also include widely-used approaches such as Softimpute (Hastie et al., 2014) and unconditional Mean (Hawthorne Elliott, 2005). default (or recommended if avail-
HosFerest
Hyperingute-
model
DifTP cter-
ReMasker.
CACTI
0.00
6.25
0.50
0.75
1
Method
For all baselines, able) settings for all models and CACTI default settings are outlined in Appendix C.2.
Figure Top-5 methods benchmark. Violin plots display the distribution of R² metrics for each of the top five methods across all missing percentages and conditions over all datasets.
Datasets To allow for comparison with previous works (Section 2.1), we use ten real-world datasets (Kelly et al.), with details included in Appendix C.3. For each dataset, we create an 80-20 train-test split to test both in samsa and out- of of-sample imputation. The data is fully observed,
we conduct comprehensive sensitivity analysis to identify key aspects and hyperparameter configurations that signifi- cantly impact the performance and usability of our method.
simulate missingness under each of three missingness condi- tions: MCAR, MAR, and MNAR. For MCAR, each value is masked according to Bernoulli random variable with fixed mean. In MAR, a random subset features are fixed as fully observed while entries in the remaining features are masked based on a logistic model. For MNAR, we take the input features of the MAR mechanism and further mask them according to Bernoulli random variable with fixed mean. In accordance with prior work, the primary benchmarking is performed under 30% simulated missingness proportion while extended results are included for 10%, 50% and 70% simulated missingness proportions. Simulations were per- formed using the Hyperlmpute package (Jarrett al., 2022).
Baseline methods We benchmark CACTI against 13 top methods from the field. Detailed descriptions of the methods are defered to Appendix C.I. Briefly, we compare against ReMasker (Du et al., 2024) as the pri- mary masked transformer-based autoencoing method. Diff- Puter (Zhang et al., 2024a) represents the recent state-of-ther art in diffusion-basee based imputation. AutoComplete (An et al., 2023) is a naive copy masking autoencoder model developed for biomedical data. Hyperimpute (Jarrett et al., 2022) is the current best iterative hybrid machine learning approach. We also compared to leading iterative methods: Missfor-
6

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Evaluations metries We evaluate imputation perfor- mance along three metrics: Pearson's R2, root mean square error (RMSE) and Wasserstein distance (WD). We use R² as measure of imputation concordance due to its invariance to mean or scale shifts and its direct applicability across con- tinuous, binary, and ordinal features. For consistency with previous works (Zhang et al., 2024a; Jarrett et al., 2022), we report RMSE as an absolute measure of imputation accuracy and WD as a measure of alignment between the imputed and true values. Note that we perform evaluation on the original scale of each feature as opposed to the min-max transformed scale to reflect utility in real world applications. The main tables and figures present the mean of the metrics aggregated across the test split of the relevant datasets ex- cept for Table I, which reports metrics on both the train and test splits. The Appendix contains figures reporting all the per-dataset metrics with 95% confidence intervals on both train and test splits.
Table 2. Ablation analysis. Comparison of models with RM, MT-
CM, and/or CTX. v indicates model has the feature and x if not. Metrics presented represent the average model performance at 30% missingness.
MT- CM
R² (1)
RMSE (4)
MODEL RM CTX
MAR
MNAR
MAR
MNAR
RMAE
X
X
0.21
0.20
1.00
1.03
RMAE+ CTX CMAE
V
X
0.26
0.26
0.96
0.86
0.46
0.43
0.68
0.70
CACTI
X
V
0.46
0.45
0.67
0.68
and requiring < 300MB of GPU memory both of which are comparable to that of ReMasker).
4.1. Ablation Analysis
Table I summarizes the average performance of each model across the 10 datasets under 30% simulated missingness. CACTI outperforms all existing baselines across all metrics and missingness conditions. We observe an average relative improvement over the next best method (with respect It2) of 13.4%, 6.1%, and 5.3% under MNAR, MAR, and MCAR, respectively. Notably in Appendix D we also observe that across the ten datasets under all three missingness mecha- nisms, CACTI dominates all methods in at least one of the three metrics and in a majority of the datasets outperform all other methods on all metrics. In our experiments, we also include median truncated Copy Masked Auto Encoder (CMAE: CACTI without context) as additional baseline to demonstrate that CMAE alone consistently dominates ReMasker across R2 and RMSE metrics.
We aim to investigate the relative contributions of key as- pects of CACTI via series of ablation analyses.First, we assess the relative contribution of MT-CM compared to ran- dom masking (RM). Second, we evaluate the impact of context awareness (CTX) when used in conjunction with random masking alone. Third, we analyze the additional gains achieved by incorporating context awareness on top of our MT-CM training strategy. Finally, we explore the value of each of the sources of inductive bias: the observed missingness patterns or the features' context information. To do this, we construct three additional models: 1) Ran- dom Masking Auto Encoder (RMAE), 2) Random Masking Auto Encoder with ConTeXt awareness (RMAE+CTX) and 3) CMAE. The RMAE model uses the same transformer backbone as CACTI while using the same random mask- ing strategy as ReMasker, RMAE+CTX extends the RMAE model with the same context aware (CTX) embeddings used in CACTI, and CMAE is the CACTI model without the CTX embeddings. We conduct the ablation analysis over four dif- ferent datasets (see Appendix C.3) under MAR and MNAR with masking ratio fixed at 90%.
These results underscore the versatility of our approach in achieving effective imputation across diverse missingness scenarios without strong assumptions about the source of the missingness. The robust performance, particularly in the most challenging MNAR settings, highlights the advantage of leveraging inductive biases from observed data through the use of the MT-CM training strategy. Additionally, the improved accuracy of CACTI over ReMasker under MCAR, where MT-CM and random masking should be approxi- mately equivalent, highlights benefits of context awareness.
The ablation results in Table 2 first indicate that both MT- CM and context awareness are essential for achieving good performance. Next, we observe that, under MNAR, MT-CM provides a 115% gain in R² over random masking while context awareness provides a 30% gain when used with random masking. We note since this is an internal ablation, all hyperparameters were held constant. This resulted in the performance of RMAE being lower than ReMasker due to differences in their masking rates. In Table A15, con- ducted additional direct comparisons between CMAE and ReMasker on all ten datasets that demonstrate that CMAE (by replacing random masking with MT-CM) alone pro- vides a statically significant improvement in performance compared to ReMasker (t-test p<00.5).
In Appendix D, we extend our benchmarking to 10%, 50% and 70% simulated missingness proportions for all three mechanisms. Figure 3 displays the results of the top five methods for each dataset under each of the four missing- ness percentages and three mechanisms. The results show that, on average, CACTI is the most effective imputation approach across all settings. Lastly, we verify in Table A14 that the resource requirements while training CACTI are reasonable (<. 5.8 seconds per epoch on the largest dataset
7

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
the masking ratio fixed at 90%, are summarized in Table 4. These results indicate that the encoder and decoder depths have a relatively minor impact (especially in the MNAR setting) although our results tend to slightly favor deeper encoder (N₂- = 10) and a shallower decoder (Na = 4). In contrast, we observe higher sensitivity of our model with respect to the choice of embedding dimension size with highest accuracy attained at E 64. Notably, we see a significant drop-off in performance at very large embedding sizes (near 512) likely due to over-fitting.
Table Context Contributions, One sided paired T-test between CACTI and CMAE imputation Rº to evaluate the statistical signif- icance of contexts' contribution and out performance (win rate).
AvG. H² GAIN
WIN RATE
(P-VALUE x10-3
%
MISS %
MCAR MAR MNAR MCAR MAR MNAR
0.014 0.023 0.017
10
(1.96) (0.79) (0.23)
80
89
80
0.014 0.007 0.017
30
(0.35) (9.24) (0.09)
80
70
100
0.011 0.010 0.016
50
(0.65) (3.66) (0.13)
90
70
100
4.2.2. MT-CM MASKING RATE
0.010 0.012 0.015
70
(2.01) (0.72) (0.05)
89
90
100
We next investigate the impact of the choice of MT-CM masking ratio (Pem). This parameter can be interpreted as controlling the strength of the inductive bias during the learning process. A higher Pem encourages the model to place greater emphasis on the observed missingness patterns in the data, allowing the model to capture and extract addi- tional information. Figure A12 summarizes the results of this analysis over 4 different datasets (see Appendix C.3) under MAR and MNAR. Our experiments indicate that, on average, Prm 2 0.90 results in the most accurate results, with slight differences based on the missingness mechanism (Pem 0.99 for MAR and pem 0.95 for MNAR). We remark that this is a notable departure from existing random masking approaches (Du et al., 2024) which report the opti- mal choice of masking rate can differ significantly (>> 10%) based on the dataset.
Table 2 also shows that using context awareness in conjunc- tion with MT-CM leads to a nearly 5% improvement. To quantify whether context provides a statistically significant contribution, we extended our analysis to directly contrast CACTI and CMAE (CACTI without context) under all miss- ingness percentages, datasets and missingness settings (Ta- ble 3 and Figure A9). CACTI outperforms CMAE (win rate) in majority of the datasets across all settings, with respect to R². We then performed one-sided paired t-tests to demonstrate that context provides a statistically signifi- cant improvement (p<0.05) in all settings except for MAR at 30% These results confirm that context can improve imputation accuracy though its contribution can vary de- pending on the dataset and the missingness setting. Overall, while either one of our strategies could provide meaningful improvements in imputation accuracy, the use of empiri- cal missingness patterns through copy masking tends to be more useful than the contextual information.
4.2.3. CONTEXT EMBEDDING MODEL
Since our ablation analysis indicates that context awareness does provide meaningful improvement to performance, we would like to understand the sensitivity with respect to the choice of language model used to derive the contex- tual embeddings. To this end, we assess six open-source base models: BaRe-base, BERT-large (Devlin et al., 2019), DeBERTa-v3-base,, DeBERTa-V3-large (He et al., 2023), GTE-en-MLM- (Zhang al., 2024b) and NV-Embed- v2 (Lee al., 2025). These base models usually have dimension of 768 for their last layer while the large models have dimension of 1024 and NV-Embed-v2 has dimen- sion of 4096. Table 5 summarizes the results of this analysis over four different datasets (see Appendix C.3) under MAR and MNAR These results indicate that there is marginal sensitivity to the choice of embedding model with GTE-en- MLM-large leading to the highest accuracy while DeBERTa- v3-large obtains the lowest accuracy. There does not ap- pear to be a clear relation between overall performance and embedding size. This indicates that the semantic context learned by each model is more important that the size of the model. This is also supported by the fact that NV-Embed-v2 (7B parameters) consistently under performs BERT-base (1 parameters). Overall CTE-en-MLM-- or BERT-
Lastly, we explored design choices involving the loss func- tion (Table A16). Training with the loss over observed values alone (Co) yields poor imputation performance. In contrast, training on reconstruction of masked values (CM), by forcing the model to learn relationships between the ob- served and masked features, leads to significantly better performance. As expected, the combined (observed and masked value) reconstruction loss (Lo LM) consistently achieves the best performance, due to the constraint of main- taining a latent space that both preserves the relationship between observed while inferring missing features.
4.2. Model Sensitivity Analysis
4.2.1. 1. MODEL ARCHITECTURE
First, we investigate CACTI's sensitivity to three core ar- chitectural configuration choices: encoder depth (Ne), de- coder depth (Na) and overall embedding dimension size (E). The aggregated results of this analysis over four different datasets (see Appendix C.3) under MAR and MNAR, with
8

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table Model architecture sensitivity, Average performance effect (a) encoder depth, (b) decoder depth, and (c) embedding size. Metrics represent the average across four datasets at 30% missingness proportion.
(a) Encoder Depth
(h) Decoder Depth
(c) Emhedding Size
R² (t)
RMSE (1)
R (t)
RMSE (1)
R (t)
RMSE (L)
DEPTH
DEPTHI
SIZE
MAR MNAR
MAR MNAR
MAR MNAR
MAR
MNAR
MAR MNAR
MAR MNAR
4
0.46
0.44
0.68
0.69
4
0.47
0.44
0.67
0.69
32
0.46
0.42
0.67
0.70
0.46
0.44
0.68
0.69
0.46
0.44
0.67
0.69
64
0.46
0.44
0.67
0.69
8
0,45
0.44
0.68
0.69
8
0.47
0.44
0.67
0.69
128
0.40
0.43
0.73
0.70
10
0.47
0.44
0.67
0.69
10
0.44
0.44
0.69
0.69
256
0.41
0.39
0.72
0.73
12
0.46
0.44
0.68
0.69
12
0.43
0.43
0.70
0.71
512
0.38
0.35
0.85
0.97
large seems to be good default choices for generic English language tabular data. Additionally, the ratio of context dimension to total embedding dimension (CTX proportion C) directly influences the contirbution context aware- ness. Table A17 shows that CACTI fairly insensitive the choice of CTX proportion with 50% or 25% of the embeddings (E) containing context information as optimal. Finally, context embeddings from Nomain-speciff models like BioC.inicalBERR (Alsentzer et al., 2019) may help improve imputation for specialized fields like biomedicine, where features have unique contextual relations (e.g., dis- case classifications). Prior work by Lehman et al. (2023) shows these models outperform general- general-purpose purpose models on domain-specific tasks. But leave this line of inquiry for future work.
MAR
MNAA
0.22
6.70
E
0.70
1pl
20
123
Epochs
Figure 4. CACTI Training profile. Evaluated across training epochs under MAR and MNAR the letter dataset with 30% missingness. Points are mean + 95% CI.
Table Embedding model sensitivity, Average performance effect of embedding models (30% missingness proportion).
300-600 epochs and monitor overfitting validation data.
5. Conclusion
R (t)
RMSE (L)
EMBEDDING MODEL
MAR
MNAR
MAR MNAR
This work introduces a conceptual framework for leveraging information about context and the missingness patterns in the data improve tabular data imputation. We posit that the observed missingness patterns and semantic informa- tion associated with the features serve as both crucial and valuable sources of inductive bias. These hypotheses led us to develop CACTI which integrates these dual sources of bias into a transformer-basee imputation model. Our extensive benchmarking and ablation analysis demonstrate that information from each dataset's unique missingness patterns and column context significantly improves impu- tation accuracy, allowing CACTI to reach state-of-ther tftt--rrtte-rtttt performance. Our MT-CM masking strategy can be used with any masked learning model, while context awareness can be integrated into any deep learning- imputation framework, demonstrating the broad applicability of our results. These results suggest that identifying additional sources and structures of useful bias a worthwhile avenue for future tabular imputation research, particularly in fields with smaller datasets with high MNAR missingness such biomedical data.
BERT-BASE
0.47
0.45
0.67
0.69
BERT-LARGE
0.46
0.45
0.67
0.68
DEBERTA--V-BASS
0.47
0.45
0.67
0.69
DEBERTA-V3-LARE
0.45
0.43
0.68
0.70
GTE-EN-MLM MLLL-ALG-LL
0.47
0.45
0.67
0.68
NVEMBED-V2
0.46
0.44
0.68
0.69
4.2.4. TRAINING CONVERGENCE
We finally evaluate the training convergence behavior of our model in the letter dataset. The results in Figure 4 indicate that the convergence behavior differ based on the missing- ness setting. Under the more difficult MNAR imputation setting, increased training epochs results in a consistent in- crease in imputation accuracy that does not fully saturate 1500 epochs. In contrast, under the simpler MAR setting, the model quickly converges to its optimal perfor- mance around 300 epochs, with increased training causing overfitting as indicated by a reduction in test set accuracy. Given these results and assuming that we do not know the missingness regime priori, we recommend users start with

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Impact Statement
Du, T., Melis, L., and Wang, T. Remasker: Imputing tab- ular data with masked autoencoding. In The Twelfth International Conference Learning Representations, 2024. URL https: hwwpnreeeeeeeeeeeeeeeeeeeeeeeeee net forum? id=KI 9NqjWWD.
This paper presents CACTI whose goal is to advance the classical machine learning field of imputation for tabular data. There are many potential socictal consequences of work, none of which we feel must be specifically high- lighted here. This is general approach that is compatible with any generic or field specific tabular dataset. CACTI allows users to more effectively learn an imputation func- tion by leveraging the structure missingness unique to each dataset and allows for straightforward integration of the unstructured textual information about the features being imputed.
Dufter, P., Schmitt, M., and Schütze, H. Position in- formation transformers: An overview, 2021. URL https: https////xwww..///abeeeeeeee/eee
11090.
Gardner, J., Perdomo, J. C., and Schmidt, L. Large scale transfer learning for tabular data via language model- ing, 2024. URL htttt:::// /ttp://aaavv../.aa/.aaaeeeaeawaa orggab///4//55/5// 12031.
Acknowledgments
Hastie, T., Mazumder, R., Lee, J., and Zadeh, R. Matrix completion and low-rank svd via fast alternating least squares, 2014. URL https: http///axvvv./. .or.vvorr./...oo. abs/ 1410.2596.
We thank Jonathan Flint for their valuable feedback and dis- cussions throughout this project. S.S. was supported, in part, by NIH grant R35GM153406 and NSF grant CAREER 1943497. A.G. was supported, in part, by NIH grant R01MH130581 and ROIMH122569.
Hawthorne, G. and Elliott, P. Imputing cross-sectional miss- ing data: Comparison of common techniques, Australian and New Zealand Journal of Psychiatry, 39(7):583-590, Jul 2005. doi: 10. 10.10. 140.j..114-111422550.00
References
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. B. A. Publicly available clinical bert embeddings, 2019. URL https: //arxiv.org. org.///oooroooo.oooooo abs 904.0444 03323.
He, K., Chen, X., Xic, S., Li, Y., Dollár, P., and Girshick, Masked autoencoders are scalable vision learners, 2021. URL https: //tr/aaaavvvv///a///a///eaeeaaaa org/ /at1////.11.0377
An, U.. Pazokitoroudi,, A., Alvarez, M., Huang, L., Bacanu, S., Schork, A. J., Kendler, K., Pajukanta, P., Flint, J., Zaitlen, N., and et al. Deep learning-bai baasidgg-based phenotype imputation on population-scale biobank data increases genetic discoveries. Nature Genetics, 55(12):2269-2277 Nov 2023, doi: 11001///4118-000-00111111
He, P., Gao, J., and Chen, W. Debertav3: Improving deberta using electra-style pre-training with gradient- disentangled embedding sharing, 2023. URL https:
/ //arxiv.org/aat //t.//1111......
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A.,
An, U., Lee. S. A., Jeong, M., Gorla, A., Chiang, J. N., and Sankararaman, S. Dk-behrt: Teaching language models international classification of disease (ied) codes using known disease descriptions. In Proceedings The First AAAI Bridge Program on Al for Medicine and Health- care, volume 281 of Proceedings of Machine Learning Research, pp. 133-143. PMLR, 25 Feb 2025.
Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with tabular foundation model. Nature, 637(8045)3319-326 Jan 2025. doi: 10. 10.1038/s41586-024--2
Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tab- transformer: Tabular data modeling using contextual em- beddings, 2020. URL https: https://arxiv..rgg abs/ 2012. 06678.
Arik, O. and Pfister, T. Tabnet: Attentive interpretable tabular learning, 2020. URL https: /aaa/aivvvvo/ org/ abs/ abss/190 11e./190...0 07442.
Ipsen, N. B., Mattei, P.-.,, and Frellsen, not-miwae: Deep generative modelling with missing random data, 2021. URL htttttts///axiiii org/ hatgg/bs//000/ 12871.
Dai, Z., Bu, Z., and Long, Q. Multiple imputation via gen- erative adversarial network for high-ding nigh-dimensional block- wise missing value problems, 2021. URL https: //a http:////g/aa/a/////aaas
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL https:/ haapss://rrxi.... org/ arg/abs/1810) 04805.
Jackson. J., Mitra, R., Hagenbuch, N., McGough, S., and
Harbron, C. A complete characterisation of structured missingness, 2023. URL https: papps///arze //aaxiivv..//////////////////ii. arxiv.org/ Abs/2307.0266 02650.
10

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Jarrett, D., Cebere, B. C., Liu, T., Curth, A., and der Schaar, M. Hyperimpute: Generalized iterative imputa- tion with automatic model selection. In International Con- ference on Machine Learning, pp. 9916-9937. PMLR, 2022.
Richardson, T. W., Wu, W., Lin, L., Xu, B., and Bernal, E. A. Meflow: Monte carlo flow models for data imputa- tion, 2020. URL https: /arxivv..vvv..////// org/ abs 12628.
Royston, P. and White, I. R. Multiple imputation by chained equations (mice): Implementationn in stata. Journal of Statistical Software, 45(4):1-20, 2011. doi: 10. v045.i04. URL https: http::::///ww..sssssssfttttt index. hwcxx.pp//jssee
Kelly, M., Longjohn. R., and Nottingham, K. The uci ma- chine learning repository. URL https: /archive. ics. edu.
Krell, M. M., Kosec, M., Perez, S. P., and Fitzgibbon, A. Efficient sequence packing without vross-contamination: Accelerating large language models without impacting performance, 2022. URL https: /ttpxiaxvvvvoog.//..a abs/2107.020271
Rubin, D. B. Inference and missing data. Biometrika, 63(3):
581, Dec 1976. doi: 10.2307/2335739.
Rubin, D. B. Multiple imputation for nonresponse in veys. Wiley Series in Probubility and Statistics, Jun 1987.
doi: 10. 10.1002/9780470316663
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Ny-embed: Improved techniques for training llms as generalist embedding models, 2025. URL hwt.h:ttt/////ett / http//aaxxvv// www.iivvvooo org/ abw////s/55 17428.
Schafer, L.. and Graham, J. W. Missing data: our view of the state of the art. Psychological methods, 7 2:147- 77, 2002. URL https: api semanticscher org/ Carpporrruusiia I 77455077
Lehman, E., Hernandez, E.,, Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., and Alsentzer, Do we still need clinical language models?, 2023. URL attps://arxie org/ abs/ 2302. 08091.
Stekhoven, D. J. and Bühlmann, P. Missforest- parametric missing value imputation for mixed-type data. Bioinformatics, 28(1):112-118, Oct 2011. doi: 10. 10.109//bionfformaiiesssss/s//))
Lin, X., Xu, C., Yang, M., and Cheng, G. Ctsyn: A foun- dational model for cross tabular data generation, 2024.
van Buuren, S. and Groothuis-Oudshoome K. mice: Multi- variate imputation by chained equations in r. Journal of Statistical Software, 45(3):1-67, 2011. doi: 10.18637/jss. v045.i03. URL htt.pp:://ww.jjstttttttt..t mmdex.ppp/jjss/artcille//ee
URL https::p///........ //axxaxxivv/// v. orgg abs /406 04619. 9.
Little, R. J. A. and Rubin, B. Statistical analysis with missing data. Wiley, 1987.
Yang, Y., Wang, Y., Liu, G., Wu, L., and Liu, Q. Unitabe: A universal pretraining protocol for tabular foundation model data science, 2024. URL https: //arxiv. org/ abs/ 2207 09249.
Majmundar, K., K. Goyal, S., Netrapalli, P., and Jain, P. Met: Masked encoding for tabular data, 2022. URL https: //arxiv. org/ abs/ / 2206 08564.
Mattei, P.-A. and Frellsen, J. Miwae: Deep generative modelling and imputation of incomplete data, 2019. URL https: //trxva.avvvv/////////eee/aaaav org/ abs///b///00 11111111111 02633.
Yin, P., Neubig, G., tau Yih, W., and Riedel, S. Tabert: Pretraining for joint understanding of textual and tabular data, 2020. URL https: //tp///aavvv///////////////aaaav wwwpwwiiivorooooooooooooooo /agg/aba aaaa/b////000 2205. 08314.
Mitra, R., McGough, S. F., Chakraborti, T., Holmes, C., Copping, R. Hagenbuch. N., Biedermann, S., Noo- nan, J., Lehmann, B., Shenvi, A., and et al. Learn- ing from data with structured missingness. Nature Ma- chine Intelligence, 5(1):13-23, Jan 2023. doi: 10. 10310 s42256-022-00596-
Yoon, J., Jordon, J., and van der Schaar, M. GAIN: Missing data imputation using generative adversarial nets. In Dy, J. and Krause, A. (eds.). Proceedings of the 35th International Conference on Machine Learn- ing, volume 80 of Proceedings of Machine Learn- ing Research, pp. 5689- 5698. PMLR, 10-15 Jul 2018. URL hwwph:tt///ggee httpp//eceeenggsse/ee/eeee mlr. press/880/ yoon 8a html.
Muzellec, B., Josse, J., Boyer, C., and Cuturi, M. Missing data imputation using optimal transport, 2020. URL https: http///axvvvv.////e//eee/eeee..e org/ halgg//ab//// abs/2002 03860.
Yoon, J., Zhang, Y., Jordon, J., and van der Schaar, M. Vime: Extending the success of self- and semi-supervised learning to tabular domain. In Larochelle, H., Ran- zato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems,
Nazabal, A., Olmos, P. M., Ghahramani, Z., and Valera, Handling incomplete heterogeneous data using vaes, 2020. URL https: arxiv. org/ wttgg/g///b///////////////////// /1077. 03653.

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
volume 33, pp. 11033-11043, Curran Associates, Inc.,
2020. URL https::/proceeddiggg neurips... oc /
(19767330050005a555577000---2000 pdf.
Yoon, S. and Sull, S. Gamin: Generative adversarial mul- tiple imputation network for highly missing data. In 2020 IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 8453-8461, 2020. doi:
00.1009CCPR2226022200000624
Zhang, H., Fang, L., and Yu, P. S. Unleashing the po- tential of diffusion models for incomplete data impu- tation, 2024a. URL https: /tttss:/arxii...o argiv.orgi........o..ooo.o.o.oo. / 2405.22 20690.
Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., Zhang, M., Li, W., and Zhang, M. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval, 2024b. URL https: // /arxiv.ivvv.////ii//.a.aa/aaaavv org/ / 2407. 19669.
Zheng, S. and Charoenphakdee, N. Diffusion models for missing value imputation in tabular data, 2023. URL https:/ // /arxivv. org/abs/////eeeooooot..oo...esss abbs/2210 2212.17128.. 17128.
12

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
A. Copy Masking
Algorithm Naive copy masking.
1: Input: Observed mask M E {0, 1j masking ratio Pem E (0, 1)
2: a (I,..., ] > [I.
{permute indices)
3: Pú 1, if =()
E 10,1
0. otherwise
4: M PM
(Apply row-wise permutation) {init copy mask)
5: M M
6: for + N,....N .N do
7: u ~ (()
{sample from uniform} fcount features left via dot product)
8: Clic = -M
perm
9: if uu < Pom and Clean 2 1 then
M M.erm
{use copy mask]
11: end if
12: end for
13:
Output: M
Copy Masking ablation analysis We compare the performance characteristics of naive copy masking (CM) and MT-CM training strategy with respect to the masking rate (pem). To perform this analysis, we construct copy masking auto encoder architecture (CMAE) which is our CACTI model without the context aware embeddings and use either the naive CM or MT-CM training. This analysis is performed bike and obesity datasets and the average results across these two datasets under all three missingness scenarios decrease in performance across all three missingness settings Pem is increased. The higher mask probability (Pem) leads to more null tokens in each training batch which reduces training performance because the model must create meaningful latent representation (for the decoder) from positions that contain no information for the encoding layers to work with. As a best performance. Strikingly, this trend reversed by the MT-CM strategy
reported in Table A6. The first three rows of Table A6 demonstrate a consistent
result, a low Pem of about 10% producing (as seen in the last rows of Table A6) where increasing the Pem results in increased performance with best performance attained 90% masking. We also see that best performance (w.r.t R²2) under MT-CM is 6.7%, 5.8% and 2. 1 higher than naive copy masking under MNAR, MAR and MCAR, respectively.
Table Performance comparison MT-CM vs naive CM and varying masking rate for CMAE, Metrics represent the average across four datasets. Experiments were performed under three missingness scenarios at 30% missingness. Best in bold and second best underlined.
MASKING TYPE
MASKING RATE
R² (t)
RMSE (L)
MCAR
MAR
MNAR
MCAR
MAR
MNAR
10
0.378
0.396
0.358
0.644
0.620
0.643
NAIVE COPY MASKING
30
0.355
0.397
0.339
0.658
0.626
0.663
50
0.348
0.384
0.324
0.667
0.638
0.673
10
0.366
0.346
0.341
0.651
0.654
0.656
MEDIAN TRUNCATED COPY MASKING (MT-CM)
30
0.378
0.399
0.362
0.643
0.623
0.645
50
0.375
0.408
0.371
0.647
0.617
0.637
90
0.386
0.416
0.382
0.638
0.615
0.635
95
0.386
0.420
0.376
0.641
0.622
0.640
13

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Algorithm 2 Median Truncated Copy Masking (MT-CM)
I: Input: Batch of embeddings E N observed masks M (0,1) naive copy masks M
1) N)
NaxK K (from Algorithm 1)
SStep I: Calculate observed feature counts after copy masking)
2: for < 1, ... NB do
3: On -- > n,
4: end for
{Step 2: Find median observed count)
5: OB
median median (o,, ONe)) })
{Step 3: Apply median truncation with permutation)
6: for - 1 NB do
7: o f- min(On; o
(Truncate to median at most] [Random permutation of features] [Initialize observed and masked sets] (Initialize truncated embeddings)
8: Ta : (I...., K} - {1,..., K)
9: On f Mn t-
E f ]
11: count f- 0
12:
for k 1, ,,,, K do
13:
Me (k) 1 and count < of trunc then
On On U (T (k))
(Add to observed set} (Append embedding)
15:
E, E (N))...)))
16:
count < count +1
17:
else if (R)
then M, + M,, U (
18:
[Add to masked set}
19:
end if
20:
end for
if on
frunc median then O median on
trune pz- 0B
[Null token padding size}
23:
NT, + 0PEX
24:
E/ + E NT,
[Concat null token padding when needed}
25:
end if
26: end for
27: Output: EE;... F, (01..., ONa.. [M....., MIN }
14

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
B. CACTI method extended details
We now provide a detailed description on the transformer-based autonencoding backbone architecture of CACTI.
The context-aware embeddings E, are used for the subsequent steps in the transformer backbone. For the embedding En of the incomplete data X,, we define selector function DECP-MT that drops all missing or masked features based the missingness mask M median truncates each batch previously described and any remaining missing/masked cells replaced with fixed null padding. This process results in E,, SéCP-MT (E,) where E,, R and (= OB o is the number of features after median truncation. This matrix E,, is processed by the encoder that consists of a series of Ne self-attention blocks with residual connections, where the output is L,.
Encoder
Decoder : We transform the context C information embedding into decoder context embeddings using linear projection ra C - where C' C RCCK The underlying rationale is that the encoder and decoder benefit from different kinds of context information. Next, define selector function sdM that maps the latent representation to match the shape and order of the original input features. The missing/masked features are filled with fixed mask vector which then passed through linear projection become decoder value information, resulting in V, E RU*K The decoder context and value information is concated (denoted by |]), with positional encoding, to get the context-aware decoder latent representational Z. V.. C*] P. This latent representation is processed through Na layers of self-attention with residual connections. The final output is passed through 2-layer MLP g Z + X to estimate the uncorrupted version of the incomplete data X,,.
Optimization The model is trained to minimize the reconstruction loss L(X,, X..) between the imputed and the observed data. We optimize our model against loss function which is a sum of the loss over the observed value (O,) and masked (i.c., observed but hidden) values (M,)) Note that we perform a max scaling of the input data before passing the cell values the model. This allows to use unified MSE loss for all features and constrains the models search space. The model's internal output X,, is therefor logits. The final loss formulation is:
Σ 10 (Ink )
2k=1
Co.,
10
Σ
|M Rm
(# Fzccke
(6)
Lm.
[M/
L(X,, Xx)) Lo., LM
We train model through stochastic gradient descent using the AdamW optimizer with learning rate (Ir) 0.001, default decay settings (0.90, 0.95) and Cosine Annealing with warmup Ir scheduler. During inference, the model's internal output is transformed back to the original space for continuous features by inverting in the min-max scaling.
15

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Algorithm 3 CACTI Training Algorithm
1: Input: Training dataset D ((X;, M,) Context information C Masking ratio Pem: Training epochs Epochmax. Batch size B
2: Parameters: Encoder weights y!, Decoder weights 6, (encoder, decoder) Value embedding weights &e,d. (encoder,
decoder) Context embedding weights we,d, Fixed sin-cos positional embeddings P
3: for epoch + 1,.... 1p.ch. do
4: Mem < NaiveCopyMas ( [M.;... MMN.Pen)
(Apply naive copy masking, Algorithm 1}
5: for each batch B ((x,, n. Mn, M sampled from D and M do
6: (Process batch of samples]
7: for n 1, ..... B do
8:
Un < 1. (Xn)
Project observed data value embeddings) (Project context info. to context embeddings {Concat. value and context embeddings add positional embeddings}
9:
C, - Tw. (Cat)
10:
En < [U [ +P
11:
end for
E', O, M +- (MT EB). Mi:... MB), [M ... M
(Median truncated copy masking:
Algorithm 2}
13:
Lbatch 0
14:
for n ...., B do
15:
Ln <- Encode E.)
{Apply encoder on remaining context aware embeddings) Vn f- MaskRe MaskReorder (In, Mn) (match original feat. order, set missing feats. to [MASK] and project) {Project context info. to decoder context embeddings (Concat. value and context embeddings and add positional embeddings) {Decode a.k.a do imputation] {Calc. MSE loss over observed and masked values)
16:
17:
C' + (C
2, + [Va|[C
X, Decodey (nn)
20:
Ln + Lo, (Xn; Xx) fmn,, (Xn: Xn)
21:
Lbatch Lbatch Ln end for
22:
23:
Liatch +- L.archh/ B
Average loss over batch] (Gradient update]
24:
b. 0, de,ds We,d + Update Cbatch)
25:
end for
end for
27:
Output: Trained parameters vi, 0, &e,d, We,d
Algorithm 4 CACTI Inference Algorithm
1: Input: Observed data D {(,,, M,) Context information C
2: Parameters: (From Algorithm 3) Trained encoder weights v. decoder weights 0, value embedding weights de.d.
context embedding weights We,ds positional embeddings P
3: C+ Twx: (Ca)
4: C' t Twa ( (Ca)
N do 6: Un lo. (X,) +P
5: for J....,M.
7:
E,
8:
E +- FRUNC(E,,,A) M,)
[Drop all true missing feats.)
9:
L, + Encode (E,)
10:
V, +- Maske, (In)
{set missing feats. to [MASK] and project)
11:
Z, + [v,IC*++ +P
12:
X, + Decodes (Zn)
13: X, t Xn @ M, + X, @ (1 - M,,)
{set missing feats. to imputed values)
14: end for
Output: X
16

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
C. Experimental Details
C.1. Baseline methods overview
ReMasker (Du et al., 2024) applies the random masking (transformer) autoencoder framework to impute missing values. DiffPuter (Zhang et al., 2024a) is a method that leverages the Depectaatoon-axxiizzatte algorithm along with a diffusion model iteratively learn the conditional probability of missing data. AutoComplete (An et al., 2023) is naive copy masking autoencoder based model which learns reconstruct missing values. Hyperimpute (Jarrett et al., 2022) is iterative imputation fraamwork that automatically selects and configures (classical machine learning) models for column- wwie imputation. Missforest (Stekhoven & Bühlmann, 2011) iteratively trains random forest models on observed data and applies them to impute missing data. ICE (Royston & White, 2011) is a method which conditionally models and imputes missing data iteratively until convergence. MICE (van Buuren & Groothuis-Ousshoom,, 2011) is variation of ICE which utilizes Bayesian ridge regression. Softimpute (Hastie et al., 2014) uses iterative rank-restricted soft singular value decomposition to complete a matrix with missing values. Sinkhom (Muzellec et al., 2020) is a generative method utilizing optimal transport distances as a loss criterion to impute missing values. GAIN (Yoon al., 2018) 8) is aeeeerative-aveessatt network with the generator trained to impute missing values conditioned on observed values, and the discriminator trained to identify which values were imputed. MIWAE (Matei & Frellsen, 2019) is an (generative) approach which applies the importance weighted autoencoder framework and imputes missing data by optimizing the variational lower bound on log likelihood on observed data. notMIWAE (Ipsen et al., 2021) is an extension of MIWAE which incorporates prior information about the type of missingness, allowing modeling of the conditional distribution of the missingness pattern given the data, to try to effectively tackle the MNAR setting (particaully self-masking MNAR). Finally, Mean (Hawthorne & Elliott, 2005) imputes missing values with the column-wise unconditional mean.
Table A7. Default hyperparameter settings used for baseline methods.
MODEL
HYPERPARAMETERS
HYPERIMPUTE
THRESSOL = 2. IIMUTER = O. OPTIMIZER = "SIMPLE" BAICH SIZE 256. EPOCHS 1000, HINT RATE 0.9, LOSS ALPHA = 10
GAIN
ICE
MAX_ITER 500
MEAN
NONE
MICE
IMPUTATIONS 1, MAX ITER 100, TOL. 0.001
MISSFOREST
N.ESTIMATORS = 10, MAX_ITER = 500
MIWAE
N EPOCHS 500, BATCH SIZE = 256, LATENT SIZE = 1, N HIDDEN = 1, K = 20
SINKHORN
0.01, IE-3,
TORCH.OPTIM.ADAT
N_EPOCHS 500,
BATCH SIZE 256, PAIRS = 1, NOISE = 1R-2, SCALING = 0.9
SOFTIMPUTE
MAXIT = 1000, CONVERGENCE..HRESSHEE
= IE-5, MAX =,
SHRINK_IAMBI
O, CVJ.EN = 3, SANDOM_STATE = 0
REMASKER
MAX_EPOCHS 300, BATCHLSIZE 64, MASK-RATIO 0.5, EMBED 32,
DEPTH = 6. DECODER
4, NUM_HEADS 4. MLP-RATIO = 4,
ENCODER_FUNC= 'LINEAR", SEIGHT_DECAT 0.05, BASE_LR = 1E-3, MIN_LR = 1B-5,
WARMUP SORNUUPEPOCSS--- 40
DIFFPUTER
MAX_ITER 10. RATIO = 30, HID HID-DIM~ 1024. NUM_TRIALS 10, NUM-STEPS = 50
AUTOCOMPLETE
LR = 0.001, BATCH SIZE = 1024, EPOCHS = 300, MOMENTUM = 0.9, ENCODING RATIO = 1,
DEPTH = 1. COPYMASK AMOUNT 0.5, NUM TORCH. THREADS 8. SIMULATE MISSING = 0.01
NOTMIWAE
N_HIDDEN= 128, N.SAMPLES=20, BATCH_SIZE=16 S172==16.EMMEDDINGS5Z----0
MISSING. DESSING.PROCESSEEE PROCESS-SELLPAASKING KKOWN
17

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
C.2. CACTI hyperparameter configuration
Table A8. Default Parameters for CACTI
MODEL
PARAMETER
SETTING
OPTIMIZER
ADAMW
INITIAL LEARNING RATE
1E-3
SCHEDULER
STEP WARMUP COSINE ANNEALING
BETAS (GRAD MOMENTS DECAY)
(0.90, 0.95)
WARMUP EPOCHS
50
GLOBAL
GRADIENT CLIPPING THRESHOLD
5.0
TRAINING EPOCHS
300
BATCH SIZE
128
MASKING RATIO (Pem)
0.90
DEPTH (N.)
10
ENCODER
EMBEDDING WIDTH (E)
64
NUMBER OF HEADS
8
DEPTH (Na)
DECODER
EMBEDDING WIDTH (E)
64
NUMBER OF HEADS
8
MODEL
CONTEXT EMBEDDINGS
EMBEDDING SIZE (dim C.
1024
CONTEXT EMBEDDING RATIO (C)
0.25
C.3. Datasets details
To evaluate the performance of CACTI across a diverse set of data types, chose datasets that contain only continuous features, as well as datasets that contain some combination of categorical, binary, and integer features, labeled as mixed. Also, to demonstrate robustness across datasets of different feature counts and dataset sizes, we benchmark across datasets ranging from 8 features to 57 features, as well as datasets ranging from 2,111 samples to 47,621 samples.
Baseline benchmarking studies are conducted on all 10 datasets which are fully observed. The ablation and sensitivity analysis are conducted on the four following datasets: bike, default, spam and students.
Table A9. Dataset summary
FEATURE COUNT
TRAIN SPLIT SIZE
TEST SPLIT SIZE
TOTAL SIZE
NAME
FEATURE TYPE
FEAT. DESC.
California HOUSING
16,512
4,128
20,640
CONTINUOUS ONLY
YES
Magic GAMMA TELESCOPE
15,216
3,804
19,020
CONTINUOUS ONLY
YES
Spam BASE
3,680
921
4,601
CONTINUOUS ONLY
Letler RECOGNITION
16
16,000
4,000
20,000
CONTINUOUS ONLY
YES
ESTIMATION Obesity LEVELS
16
1,688
423
2,111
MIXED
YES
SEOUL Bike SHARING DEMAND
12
7,008
1,752
8,760
MIXED
NO
Default OF CREDIT CARD CLIENTS
23
24,000
6,000
30,000
MIXED
NO
YES
ADULT Income
14
38,096
9,525
47,621
MIXED
(SOMETIMES)
ONLINE Shoppers
PURCHASING INTENTION
17
9,864
2,466
12,330
MIXED
NO
PREDICT Students'
DROPOUT AND ACADEMIC SUCCESS
3,539
885
4,424
MIXED
18

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
D. Extended Benchmarking Results
This section provides additional results to demonstrate CACTI's performance against the baseline methods as measured by R², RMSE, and WD across 10 datasets, 3 missingness scenarios (MCAR, MAR, and MNAR), and missingness ratios (0.1.0.3.0.5.0.7). Furthermore, we show performance on both the train split and the test split, demonstrating performance in both in sample and out of sample scenarios.
Here we point out that method that is missing from the R2 plots (not to be confused with R² R 0) implies lack of convergence due to the loss function taking on NaN values. Notably, at missingness rates 2 30% ReMasker, DiftPuter, notMIWAE and AutoComplete show convergence difficulties for one or more datasets using their recommend parameter settings. In partieular, ReMasker failed in all MCAR and MNAR datasets at 70% simulated missingness and shoppers in MCAR and MNAR 30%. DiffPuter also fails in shoppers at 30% simulated missingness under missingness settings. notMIWAE fails on income at MCAR 30% and spam in almost all settings except for MCAR 70%. Finally, AutoComplete fails the income dataset under MNAR and MCAR at 70% simulated missingness. We made sure to re-run these failed runs at least twice to rule out random chance or a hardware issue. We report these failed runs to ensure transparency and did not adjust the recommend/default parameters to try to force these methods to not converge to NaN to ensure a fair comparison with all other methods which successfully ran and converged on all datasets.
19

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table A10. Average performance comparison with standard errors (in parenthesis) of 15 imputation methods on the train/test splits
(separated by 1) over 10 datasets at 10% missingness. Metrics (arrows indicate direction of better performance) evaluated under the MAR,
MCAR, and MNAR conditons indicates method cannot perform out-of sampl
- MISSFOREST
sempletyy imputation Best in bold and segond best underlined.
MEAN
MIWAE
SSOTTMPPTE
GAIN
BMICE
AUTOCOMPLETE
CC
INNTIORN
NOTMIWAB
IYPHRIMPUTE
PIFFPUTER
RRMASKER
CMAE
CACTI
METHOD
(0. 10.000100000
0.00000
(0. sas////a
00.0033
0 10000000000
10000 aecaaaoo
10000000
1O0 1000000000000
(0,
0 o 10.000
(0,00
(0, CoOOO
(0) 126642
(0) 1
14000
1660
100000
10.50
MCAR
10000000000000000000000000000000
36
(500
11110 80P 0 S
LIF 4000 12235
24
100000000000
(o, (oo)
112000 () 10000000
(51000)))))) (010
1500
100000000000000
773 902
190000000
2LZ 1820
(91000001))))
101100
-
1
20000
(9100000000
066 0 18877
0052200000 0 16677
MAR
R² (1)
10000000000000 oo 10000 to
0000 o
(0)
(0)
(800) 266 01066 (00000000000 (2100000)))))0
098
(S10'0)
(L100))
Chop
222 (0000)/0000000000000000
(0))
DISS
() (2000)))))000))(()))))
(900
12000
1226 0
018 (1 (5000 1 0 1
106 0 4899700 ( 411000
00
0309
0
6FF 01822 D
00000000000000000000000000000000
166 1008800
MNAR
00000000000000000
GEO
1 992
12200
((00000)
00O'T LOO'L 100000000000000000000000000000 (1 6EG
1:222 100001100
339 11 11
L 018:0100
650 119200000000
00000000000000000000000000000000
00600020000 0
139 089000008
180000
0
1106
189 Dosstto
12690
(6000000 45900/2160
MCAR
P9E4 1 0500100
(1100000000000000000000000000010 (1 (50)
1899000
13900
PE8 L1800 (9)000999000000
00 0)
(610'))
665
1955
848
10100000000
170001 960 1 1
E38'018880 02800000 0
168 188 0
2697 PILL
1161000
(1
1969
(50200)
86200
RMSE (4)
MAR
1111)
10 ())00
E76 00 1000000000001
111' 1990'1 (+500) 066 0
(9000 (900 (0000))00000000000000 (000 ()
0000000011111000 (10000000)))0000000
98 0 688000
0490000 (8
(8
(00) (0)(01) (1000000000000000000000000000000
4
=to 109000
8 282 R (7000)))))))
102 10
1
010' 10
(0000)
1
00600192200 0
(01000)
886
((610'))
118888 0
MNAR
010'110
(100 18:011 (10000
L69 S 48L molltooo 01 896'000
982
896
(0/00) Millaa
HH66000 S C
1/
3P2 01/299 8
2900
9 FEEE 8 S
2907 165
8
(26100)
MCAR
1000200
20020000000 9 B5F
485
R 1
8 020000 100000001
281 1991990
23172/2.285
9/
1000'
3.
156119
MAR
WD (L)
0861800
(0)
(801
0E6
ORO
1161 299922
222 (1l1180
18817
2E00 9
MNAR
(LFF
19970
1107
(S 1
(
(

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table AIL. Average performance comparison with standard errors (in parenthesis) of 15 imputation methods on the train/test splits
(separated by 1) over 10 datasets at 30% missingness. Metrics (arrows indicate direction of better performance) evaluated under the MAR,
MCAR, and MNAR conditinss indicates method cannot perform out-of sample
o imputation Best in bold and segond best underlined.
MEAN
MIWAE
-SOFTMPPUTE
GAIN
BMICE
AUTOCOMPLETE
CC
SINTIORN
NOTMIWAB
MISSFOREST
IYPHRIMPUTE
DIFFPUTER
BREMASKE
CMAE
CACTI
METHOD
(0. (.,00)/(o
(0. (000)/i (cO.
(0. 1o0nnyl
000000
(0, 00000000000
(0,00
00 (188888888
(0, 0.
00 O 1000
(000
()
() 007000
0.000/0
0100
00
(00000000000
I 011011180
110000
11222
(010'0) (600 (60000000000000 (200 00999900
(D)
LFF (111
MCAR
(000000000000..
(600 10000000000000000000000000000000
(010 1
1900
000)
000
(000))
00000000000000000000000000000000
00000000000000000000000000000000
00000010000000000000000000000000
06
0 () (600 0) 882 0 0F22
Lre
BEE
DEF 1000000000
98 1000000 (01010))20000
(010 (600 (222000 00
BET
(0.00000000000..
(O, 000) (o.
0.000
100000
(000
100000
1011012111 () 00000) 1EL1T
stt (000)
(600
120 L82 O
0800000
(1100))
000001000
18
101000
(210
1E200 (0100)) (010')))
0) 1995 0 CSFD
0.464
MAR
oo0 0,
Emo oo
ropou (o
o
sol 1o
tho 1/ 00000000000000000000 (600 0)
162 0
Sre 0
29
(1))
(000))
2E2
200 ()
ETP
(000)00 o oo
ooooooo (o)
(o) o
(E00 10000)e00 ()
160 oollddo
011
(0000
1652 0
966 016822
15000 00
/969 0
MNAR
BLT
( 126 0/06 0 (+100))))0000) (910')))000000 (1000'))) 01200
298 ol O ()000
152
1000
222
ZLE
(1000))))
OOL'O (800000
598 L9 296'00 80000 1 (1 10'00 (1 (1000) ()
STE 17 (000 (E (0000))))0000))))000000)00)00))0 (
(6
11100) (19000 D/
E00 500'1 00000000
ELO' I 446'0188
EDO 1 192001 (0 220 E9001 LS6'0ll
0 +S6'0 958 0
1 EEE 8 8. 120'111 1/80011 (C000) (2 (9000)))000000000)))0)0)))
16
150 ++0'1 000000000
008000
00958
(80010
00000000000000000000000000000000
1
06 428 108 0 (9000 ASL'0l
0 (200100
OLL'O MOL/
S99 111199
MCAR
698
LE7
(
(0100 (+
(1 S68 0 9
(50000) 0) ((5100)
888
CBL 000000 01866880
(410000
RSL'D
1691
(8 1
00770000 01
MAR
RMSE (4)
+56'0 (0100))))
09000000000000000000000
(00000)) (2
LSL' (F
10000
0.685 8
L86'0 0 (1 (1000)) (9009000)
(000))0000
(2 668 () L (00000//110000000000000001
18T (
(
6LT'5/8110 056'0
LLS
(900000)
19
10000000000000000000000000000000
F 10000000000
0 MO0D
+L0
169
000 1
1
2211199000 11
C16559
2.346
S01 TILEPP F 11000000000000000001000000000000
S 01
(0100)
19900
((8801)
192'5
MCAR
E40'000 (E)
0109
161
00050
ORE ISEEE
01 ELO) SEISEDN
SRE 222/222
11072 1766
-8561
3
(16000
265592
MAR
WD (L)
8 1SFF400
222
(
L. 16g
R
26F 11tte 8
(66000
1800
10000000000
MNAR
4468
2500000
199166
SE2
(SE1
18F0
098
10000000
00900
88
6899

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table A12. Average performance comparison with standard errors (in parenthesis) of 15 imputation methods on the train/test splits
(separated by 1) over 10 datasets at 50% missingness. Metrics (arrows indicate direction of better performance) evaluated under the MAR,
MCAR, and MNAR conditions indicates method cannot perform out-of sample
- imputation Best in bold and segond best underlined.
MEAN
MIWAE
SSOFTMPPUTE
GAIN
BMICE
AUTOCOMPLETE
CCE
INNTIOON
NOTMIWAB
MISSFOREST
IYPHRIMPUTE
DIFFPUTER
BREMAKKE
CMAE
CACTI
METHOD
(0. (.,000)Co
(0, 0.000/00
00000000
10000
10.00
00 10000
(0,00
222222222
1000000
000 10000000000000000000000000000000
00000000000000000000000000000000
00
(0
00000000000000000000000000000000
& 100///
000000000000
00000000000000000000000000000000
(1000
(ooo)) () (900 0) (900 10000oo0000
-1
(E00 ( (0) (200 0) (900 00000000000000000000000000000000
100000
10000
1062
OSR'O11RR (F00 (0 (200 O) (200 10000000000000000000000000000000 0) (200 (0)((80
MCAR
10000
00000000000000000000000000000000
(200 (000022000000
652 192 S61000
862 0 0. 10
(00) (o. (00) vo
00000000000000000000000000000000
(200 1ww.EEEEEE
10000000000000000000000000000000
000 0 Z0E 0
ofde
(000000000000000000 ()) 1REE (200') () (200 1o/(2/0
c100
(00000 1081100
161 161'00
(00000)) () (200 (900
(00000000000 (800 (800
882
(2
EEE 0 Z00'0
ECF 0 D.200
3000
380 0 1500
() 1L2'0880
15000000
600
$15'00000
12000
1000011 (20000000000)) (2000))
(8000 0) 000000000
MAP
22
000000
o
0800000
(010
(L00'0)
00000000000000 10 oo 10000
())
E100
(5000
OLI
000
S900 (000000000000000000000
1AST//000 4 9L8 01 (000000000001000000000000000 (8100) ( ()
1900 (0000000 ()
()) (10000000000000000000000000
0100060000
6188888/5 8 C (9 10110000000000000000000000000110 (0) 196 $19988 A ( 01 096 0l 0 (Z000 (810'0) 066 ol 1 14100lE2200
121 1000000010
(0000000000000 (2000')) 002000
111000
-1
622 00922220
6E2 02
1(5500 00
I0LZ 0
( (O000
MNAR
6100
(
-10001 (FO0
(100
902
992
595
11000000000000000000000
(0000000
(9000000000000000000000000000000
28L (220 () D.9'0 (900 (O
() 0000000000
0
(0E000) 10
$61 1916' 71111 00 (12000) 1101100000
1988'1 196'0 19600 1 16001127800
191'1 120'1 161
0L800l1E0)
E71'0'$96'0 E 0 (S100)
00 (L70 (5000) 10
01800 S6L'0 908 0000000000 0
0 218 118'O
(60000
1806
MCAR
00000000000000000000000000000000
98610
E18 908'0 0 ESR
() 0) (51000)
201 269' (0000)))888
191'T (0000 110'1 18 (
14000
EOR 6 (0100000000000000000000000000000 ()000)))) (070'))) (2 111 1SS''
111000
L66'0
966 966'0 LED'T
E28 L+8 0
196'0
508 0670
0 (8100)
128 06 (S70 (0000000000000
1833
MAR
RMSE (4)
OIL'0
0)
10000000000000000000000000000000
(
L00 1
69t 18E'R 1
ESO 16001191110
1
1
( (OOD)
(E1100)
10
10w.0000000000000000000 0
11100000000000000000000000000000
0/
0s8 (01000)
099'0
09800
MNAR
1I (E
498
058:00
2LL' 111111111111111
(1
() 9
('
019 S099'
5
8SE 1AR2E1E
() (() ())00))))))))))))))))))))))))))) (1w0000000000000000000
L228
9L0 El0'' 222
819'S
1998: R SOE 10022 ((
OSS (00011
S. 996 222222222 OSS's
IEED'S g 10000990000
1 4
4
11911111
(100000
9 ES8 2I00OL
086 16681
L
118351
100110
-22211
996
MAR
WD (L)
Lh6 196998 8 022
())
1922 (9992)
(1911166000 91
50FF-1 (10000
1
909666601 9
(
LIF (1155
(2811
2228 9
(666110)
1eg00100 + S
MNAR
ShF
9
(
41
(99
(1

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Table A13. Average performance comparison with standard errors (in parenthesis) of 15 imputation methods on the train/test splits
(separated by I) over 10 datasets at 70% missingness. Metrics (arrows indicate direction of better performance) evaluated under the MAR. samples imputation and NA indicates method failed, Best
MCAR, and MNAR conditions, indicates method cannot perform out of
underting ii
bold and second best
MEAN
MIWAE
SOFTIMPUTE
GAIN
MICE
COMPLETTE
ICE
SINKHORR
NOTMIWAE
MISSFOREST
HYPERIMPUTE
DIFFPUTER
REMASKER
CMAE
CACTI
METHOD
AUTO
00000000000000000000000000000000
(0. (0.000)/c (000)(0000)) (0. (0.0000 (0. oo.0
000000000000
00000000000000000000000000000000
0.000 0.000 000
(000))000000
(0. 10.000)60.0
0. 0001s/000
(O (002))/0001 1111
O 0000000
000000000
(0000
00000000000000000000000000000000
111000
14000200
00100
1100000
Fr22 2222 00
(100
0 2222
MCAR
19999
00000000000000000000000000000000
10000000000000000000000000000000
1L00
O #51
551
2222
0.000/00
do 01g022
000000000
(0
188 111822
00::0
(000000)(80000
0o0000o000 1220
13000
00000000000
00:000000
10000000
MAR
R² (t)
000
O (Z000 (6000
00100000000000000
16
398
2000 0J (200 (600)
383
1000000000000
100000000
ro (
(0.000
(o.
(.000 o
0000000
10010000000000000000000000000000
0/ 11100
2900 606000
1-5 PESTE
O0 12000
(0) 1200
0.227
02 11003
11900
10000
1677
MNAR
o aioo
6000 0
t00 O
0o0
10000
291
05
()))) 0) D
Ol COM000O
1
0J 100000
1000
01 S0oo
181177
024 (1666000000
1 CCF-1
(800) 01 F2610
- 99666
06600 96666660
011000
14590
098:00 19880
23000
C2600
111000
1100
DESS 18880
10
1389
MCAR
1960000
10
00)
GET'T
01)
9EF
10000001000000000000000000000000
0000011
926
0l
0888
0)
0J
2
000/66000 0)
000000
10000000000000000000000000000000
00000000000000000000000000000000
(/5600
00100000000000000001000000000000
1100000000
16666
00000000000
2288
11000000
0.746/0 721
MAR
RMSE (1)
20100
6661
C084
265
8647
110000
000000000
24
(1000
10000
10111
1911 1020-
18F CLP1
2200 1 102'1
(LO0
+96 9
1600/0
120'1
4000 (61000)
266-- +264
(YN
20044
0.839
MNAR
(6600
(1
12 d6g
11910000
1
R5e 9
10000
10/010000000
(44 00 BO000
00.82
CO 9 100000
(0,
())
01 (0)
(0)
sor L/LSS
8
335 10050100000
201 (0062)
00000000000000000000000000000000
(0000
MCAR
17000
11000
10000000000000000000000000000000
1888 4 2022 5 S9/5
1001
(PON
1955 119100
3 6
3
0460 1 1885
1658
20622
1
1,740 1.765
MAR
WD (4)
00000000000000000000000000001000
1089 T.
1
052
11000000000010000000000000011111
188
2
11
&
3.97116 600 6
7 1000/0000000
MNAR
19900
(150)
06666666
296
(2400
(0006)
(0455)
(444))
145
(3)

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
bike
califrnia
income
tragh
skasppers
spars
studerts
2 0.50
0.00
40
RMSE
30-
1662
Iriin
Teei
1e43
beill
Test
Seitt
trinn
Tarl
trein
Irain
Method
CACT
CMAR
reMaaleer
Hypedmpure
Fireeo
Sinkhom
AuthCampler
MICF
GAIN
Softmppte
MINAA
(a) MCAR at 10% missingness
24

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
bike
calbbrnia
detou
meag
obesity
shoppers
spars
stdentt
0.50
0.00
RHSE
15
WD
tmin
Iest
Trin
test
trin
train
trin
test
test
Method
CACTI
CHAE
reMesker
DiPPtter
Hypertmpote
MireFtrest
notMEWGE
Sinkhom
AutoComptere
GAIN
Softimpit
MINAAE
(b) MAR 10% missingness
bike
cellferiaa
Inc
Interr
magi
obessiy
shoppers
spars
students
2
0.50
0.25
2.5
RMSE
0.5
40
30
WW
20
10-
0-
Mast
Uast
vain
Bast
Method
Hyperimpute
AuaComoliz
Softinout
(c) MNAR at 10% missingness
Figure A5, Performance comparison of CACTI against 13 baseline methods. Experiments performed on 10 datasets, under MCAR. MAR, and MNAR, at 10% missingness. Results shown as mean + 95% CI.
25

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
0.8
9-
2 0.4
0.2
0.2
1.5
RMSE
0.0
30
10-
1651
best
Daln
105t
soin
00:1
trin
biii
1051
3aann
tist
b2in
Method
CACTI
CMAR
reMossiorr
Di8Poser
Hypedmpure
ManeForeest
Sinbhom
AutoCompler
WICF
GAIN
Settimp.te
NINOUR
MCAR at 30% missingness
califomia
maaii
cbesity
shoppers
sp4m
sladerts
0.8
70
15
tain
12st
bain
zain
tzin
Method
CACTI
CHAE
Hypertmpuse
MissForest
Sinkham
AubCoonleee
GAIN
Softinout
(b) MAR at 30% missingness
26

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
shoppors
0.5
2 3.4
02
0.0
1.5
R.8SE
3.5
20
Mstt
Mest
teot
trin
lest
bain
Dest
Dain
last
bain
best
Sain
list
Dain
Mast
Ban
trait
Method
CACTI
CHAE
reMooker
Hypertmpune
WissForest
noMMAAGE
Sinkhom
ICE
AutoComplere
MICE
GAIN
Settimptte te
NINIAE
(c) MNAR 30% missingness
Figure A6. Performance comparison of CACTI against 13 haseline methods. Experiments performed datasets, under MCAR, MAR, and MNAR, 30% missingness. Results shown mean + 95% CT.
magic
obesity
shoppers
scam
studorts
0.8
2
3.0
1.3
ROSSE
20
WD
11x
1ist
0241
vam
bast
trais
105t
bain
train
tast
N051
sain
tist
Nast
1000
train
14 Method
CACTI
CHAE
raMasker
DiePuser
Hypartmpuna
MissForest
RoMMIAAE
Sinkhom
ICE
AutaComskca
MICE
GAIN
Suftingut
NINIAE
MCAR 50% missingness
27

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
califomia
letter
Tagge
obesiiy
shoppers
spar
stadents
3.0
RMSE
15
100
MWM
tmin
lest
ta/m
test
test
tain
train
trin
1est
test
Method
CACTI
CHAE
reMesker
DitPuter
Hypertmpote
MireFt
noMMEAGE
Sinkhom
AutoComppere
GAIN
Softimpit
MINVAE
(b) MAR 50% missingness
filce
califmia
defsult
Income
lefter
muggi
obes
shoppers
spam
students
0.4
01
0.2
AMSE
WD
20
Jif
Bast
Uast
vain
last
Bain
Method
Hyperimpuse
AuaComolie
Softirout
(c) MNAR at 50% missingness
Figure A7. Performance comparison of CACTI against 13 baseline methods. Experiments performed 10 datasets under MCAR. MAR, and MNAR at 50% missingness. Results shown as mean + 95% CI.
28

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
0.3
2 3.2
0.2
0.0
RMSE
3.0
30
WD
1651
best
1050
10in
000t
72in
125t
bain
1091
3ain
bist
1051
Method
CACTI
CMAR
Hyperimpure
MisoForeett
roNNIWAF
milhcom
ICF
KatsCamplete
MICF
GAIN
Softimoute
MIMAR
Nean
MCAR 70% missingness
magic
cbesity
shoppers
sp4m
stadents
2 3.4
0.2
2.0
1.5
RMSE
0.5
15
10
u1
H8.00
tain
bain
zain
uzin
Mathod
CACT
CHAE
Hypertmpute
MissForest
AubCoonleee
GAIN
Softino
(b) MAR at 70% missingness
29

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
bite
calloma
detutt
Income
letter
obes-it
shoppers
spam
students
0.4
0.3
2
AMSE
1.3
0.5
WD
10:01
Method
Hypeltmpue
AutaComplos
(c) MNAR at 70% missingness
Figure A8. Performance comparison of CACTI against 13 baseline methods. Experiments performed 10 datasets under MCAR. MAR, and MNAR at 70% missingness. Results shown as mean + 95% CI.
Table A14. Comparison of runtime and memory statistics of CACTI with ReMasker 10 datasets. Experiments performed under MAR scenario at 30% missingness. The runtime measured for the training and inference (on train/test split) stages in seconds (s) and peak GPU memory consumed in gigabytes (GB).
PER EPOCH (S)
INFER TRAIN SPLIT
INFER TEST SPLIT (S)
PEAK GPU MEM (GB)
METHOD
DATA
(S)
OBESITY
0.42
6.15
1.52
0.16
STUDENTS
0.64
15.53
3.84
0.18
SPAM
0.71
16.36
4.11
0.26
BIKE
1.24
25.82
6.38
0.16
SHOPPERS
1.68
34.85
8.67
0.16
CACTI
MAGIC
2.39
55.37
13.70
0.16
LETTER
2.91
82.06
21.01
0.16
CALIFORNIA
2.95
59.93
14.92
0.08
DEFAULT
4.61
101.91
24.84
0.16
INCOME
5.76
165.59
42.17
0.16
OBESITY
0.28
4.28
1.07
0.14
STUDENTS
0.49
2.24
0.14
SPAM
0.58
9,30
2.34
0.15
BIKE
1.06
17.74
4.41
0.14
SHOPPERS
1.55
25,46
6.30
0.14
REMASKER
MAGIC
2.09
38.23
9.53
0.14
LETTER
2.10
40.67
10.09
0.15
CALIFORNIA
2.37
41.89
10.44
0.04
DEFAULT
3.18
61.21
15.18
0.15
INCOME
4.80
95.45
23.94
0.15
30

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
E. Extended Ablation Analysis Results
this section, we present additional ablation analysis results. First, Table A15 summarizes the results a 3-way comparison (over all 10 datasets) between CACTI, CMAE (CACTI without context) and ReMasker (the strongest MAE with random masking) to quantify the magnitude and the statistical significances of the improvements driven by MT-CM, context awareness and the combination of the two. A one-sided paired t-test is used to evaluate the statistical significance of improvement of the target method over the baseline. Next, Figure A9 shows the per-dataset difference in R2 performance between CACTI and CMAE across all datasets, missingness situations and simulated missingness percentages. We also perform additional ablation comparing the effects of calculating the loss over observed only values, masked only values and a combination of the two in Table A16.
These results are followed by the complete (per-dataset) results of all our ablation analysis, demonstrating the effects of (a) MT-CM, RM, and/or CTX and (b) loss function on model performance. For this experiment we use 4 UCI datasets (bike, default, spam, and students), each under three missingness scenarios (MCAR, MAR, and MNAR), with simulated missingness ratio of 0.3. We measure performance using R², RMSE, and WD, on both the train and test split.
Table A15, Paired t-test to evaluate statistical significance gain in performance between CACTI, CMAE (CACTI w/o context) and ReMasker.
MISSINGNESS
TARGET METHOD
BASELINE METHOD
AVG. RL GAIN
P- P-VALUE
CACTI
REMASKER
0.034
4.4E-7
ALL
CACTI
CMAE
0.013
1.1E-5
CMAE
REMASKER
0.021
4.26-5
CACTI
REMASKER
0.023
5.5E-4
MCAR
CACTI
CMAE
0.014
3.6-3
CMAE
REMASKER
0.017
8.9F-3
CACTI
REMASKER
0.025
2.3r-2
MAR
CACTI
CMAE
0.007
9.4r-2
CMAE
REMASKER
0.018
4.8E-2
CACTI
REMASKER
0.054
1.1E-4
MNAR
CACTI
CMAE
0.017
8.7E-4
CMAE
REMASKER
0.037
4.6E-4
Table A16. Loss ablations. Effeect the loss function on accuracy. Metrics represent the average across four datasets (30% missingness).
H² (t)
RMSE (1)
Loss TYPE
MAR MNAR
MAR MNAR
Lo Lu
0.46
0.46
0.68
0.67
LM
0.41
0.43
0.71
0.70
Lo
0.03
0.04
2.67
2.93
31

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Misstegness:
Missingness:
Nissingness:
Missingness:
0.06
0.04
(CAC CIAEE)
6.00
R: Change
0.04
0.00
0.22
0.11
0.00
1880
calterta
Income
Jake Canfortia
detani
income
Antter
magic
alsa
detiurt
incoome
yrue studenti
Dataset
Figure A9. Difference in 2 between CACTI and CMAE across all 10 datascts, missingness conditions and simulated missingness percentages. R change > 0 indicates CACTI better than CMAE under the respective setting.
32

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
spam
NCAA
02
0.0
0.4
MAX
0.4
02
0.0
NCAR
0.-
1.0
MAX
0.4
0.5
MCAI
3-
20-
10
4CTX
RMAF-CTX
CACT
RMAMl
Spii
Figure A10. Experiments performed across four datasets split into train/test, under MCAR, MAR, and MNAR, at 30% missingness. Metries (R2, RMSE, WD) reported 95% CI. Ablations demonstratc how model performance affected MT-CM, RM, and/or CTX.
33

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
0.4
02-
2
02
RMSE
13-
5-
Split
text
tah
Figure AIL. Experiments performed across four datasets split into train/test, under MCAR, MAR, and MNAR, at 30% missingness.
Metrics (R², RMSE, WD) are reported mean 95% Cl. Ablations demonstrate how model performance affected by the loss function.
34

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
F. Extended sensitivity analysis results
In this section, we present the present the average sensitivity analysis results for context embedding proportion (Table A17) and masking rate (Figure A12). All sensitivity analysis experiments are performed on UCI datasets (bike, default, spams, students), under missingness scenarios (MAR and MNAR), and 0.3 missingness ratio, split into train and test splits. Performance is measured using R², RMSE, and WD.
These results are followed by the complete (per-dataset) results of all our sensitivity analysis. Figure A13, Figure A14, Figure A15, Figure A16, Figure A17, and Figure A18 show the effects of independently varying MT-CM rate, encoder depth, decoder depth, embedding size, context embedding model, and context embedding proportion respectively.
Table A17. Context embedding sensitivity. Average performance effect of context (CTX) proportions (30% missing).
CTX PROPORTION
R² (t)
RMSE (4)
MAR
MNAR
MAR
MNAR
0.25
0.47
0.46
0.66
0.67
0.50
0.48
0.46
0.66
0.68
0.75
0.46
0.46
0.67
0.68
0.48
R2
0.44
0.40
Missingness Type
MAR
MNAR
7.00
6.00
WD
5.00
4.00
3.00
10
30
50
80
99
Masking Rate
Figure A12. Masking rate sensitivity, Average performance metries over range of MT-CM masking choices. Evaluated under MAR and MNAR with at 30% missingness.
35

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Bile
denut
Spam
0.6
0.-
044
0.
073
035
025
000-
1.00
MAAA
025
0.0
10
5-
10
30
52
80
30
15
Figure A13. Experiments performed across datasets split into train/test, under MAR and MNAR, 30% missingness. Metrics (It". RMSE, WD) are reported as mean + 95% CI and show model sensitivity to MT-CM.
36

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Nile
delutt
spam
Stadents
0.6
04
02-
0.0
100
0.7
050
RMS
000-
1000
075
002
10-
Spli
Figure A14. Experiments performed across four datasets split into train/test, under MAR and MNAR, at 30% missingness. Metrics (It". RMSE, WD) reported + 95% % CI and show model sensitivity encoder depth.
defult
spar
stadeets
0.-
MAi
0.0
05 004
0.9
03
3-
12-
3-
Figure A15. Experiments performed across four datasets split into train/test, under MAR and MNAR, at 30% missingness. Metrics (It2.
RMSE, WD) reported mean + 95% CI and show model sensitivity to decoder depth.
37

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Nile
delutt
spar
Stadents
04
0,4
0.
POSK
4500 004
155
13-
30-
23
128
750
312
178
236
512
258
512
170
Spli
Figure A16. Experiments performed across four datasets split into train/test, under MAR and MNAR, at 30% missingness. Metrics (It". RMSE, WD) reported + 95% % CI and show model sensitivity to embedding size.
Niva
defult
spar
stadeeis
0.6
0.04
0.2
0.0
1.0
0.5
0.50
025
8 0.00
078
050
025
BeRTuge
DedER 1ar2000d
Castii RaliBBegg
GTEEeeELMaad Mangee
AnvEmbendz
Endit Eaditi taraarge
GTE GTeeet attaane ataage
Spik
Figure A17. Experiments performed across four datasets split into train/test, under MAR and MNAR, at 30% missingness. Metrics (R²2. RMSE, WD) reported + 95% CI and show model sensitivity to context embedding model.
38

---

CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Nile
debult
spam
statents
0.4
HAV
0.0
100
075
025
1M50
002
3-
5-
0.5
0.55
0.75
Figure A18. Experiments performed across four datasets split into train/test, under MAR and MNAR, at 30% missingness. Metrics (R².
RMSE, WD) reported + 95% CI and show model sensitivity context embedding proportion.
39


Reviews:
[{"review_id": "81dC3XieCt", "reviewer": "Reviewer_PQVE", "summary": "The authors introduce CACTI (Context Aware Copy masked Tabular Imputation) for imputing missing values in tabular data. CACTI’s backbone is a Masked Autoencoder based on Transformers. It brings the following key modifications to this architecture:\n\n- Instead of randomly masking observed values as in ReMasker, it uses copy-masking: additional missing values are introduced in sample i by applying the missingness pattern of another sample j. These additional missing values will then serve to create the reconstruction loss. Copy-masking allows to exploit the missing patterns present in the dataset.\n\n- Copy-masking is further enhanced with Median Truncated Copy Masking (MT-CM), which truncates all samples in a batch to have a maximum number of observed features. This max. number is defined by the median number of observed features in a batch. This prevents from having too many missingness tokens to process.\n\n- Context-awarness is achieved by concatenating column description embeddings (obtained with GTE-en-MLM-large) with the numerical embeddings of scalar values.\n\nThis architecture is evaluated on 10 datasets, across varying missingness rates and the 3 missingness mechanisms (MCAR, MAR, MNAR). The imputation is evaluated according to the reconstruction R2, RMSE, and Wasserstein distance. CACTI is evaluated against 13 baselines and shows state-of-the-art performance across the 3 missingness mechanisms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback.\n\n## Does context help?\nOur ablation analysis focused on the *relative* contributions of median truncated copy masking (MT-CM) and context awareness with fixed architecture settings and hyperparameters (Appendix C.2). Random masking autoencoder (RMAE) and ReMasker perform nearly identically when hyperparameters match (internally verified). Performance differences primarily stem from different masking rates: 30% for ReMasker (optimal per original paper) versus 90% for CACTI/RMAE.\n\nTo address potential ambiguity, we extended our analysis of CMAE (CACTI without context but with MT-CM) from 4 to all 10 datasets (qrHN rebuttal Table. R1). CMAE consistently improves or matches ReMasker across R2, RMSE, and WD metrics, while CACTI strictly dominates CMAE in all conditions. This confirms that context can improve imputation accuracy. We also performed a comparison using one-sided paired t-tests (qrHN rebuttal Table. R2). CACTI shows statistically significant (p<0.05) improvement over CMAE for R2 (and RMSE) in most settings, reinforcing that context enhances imputation. MT-CM (via CMAE) delivers statistically significant R2 improvements over random masking (via ReMasker) across all settings. CACTI, combining both approaches, shows significant improvement across nearly all metrics and conditions. While MT-CM provides larger improvements, context contributes meaningfully, demonstrating that both MT-CM and context awareness provide useful inductive biases.\n\n## Datasets \nWe selected 10 open datasets from those used by Hyperimpute, ReMasker, and Diffputer authors for fair comparison. Our selection criteria were: 1) 6 with mixed features and 4 with continuous-only features datasets to ensure type diversity, and 2) half the datasets having feature descriptions. Table A9 will be expanded with information about which datasets have column descriptions. Our anonymous repo provides code to obtain all datasets and generate colinfo.txt files (see \"Generating UCI ML Datasets\" in README).\n\nThe 4 datasets used in ablation and sensitivity analysis were randomly chosen from the 10 used in main benchmarking. While this approach is common in prior works, we acknowledge the potential risk of inflating overall results. To maintain fairness, we used similar/same datasets and optimal parameters from previous works. To directly address this concern, we re-analyzed our main benchmarking results excluding these 4 datasets (qrHN rebuttal Table. R3). Results confirm CACTI still strictly outperforms existing state-of-the-art methods in R2 and RMSE, addressing concerns about fairness and result inflation.\n\n## Writing\nWe appreciate the reviewer highlighting these typos. While our notation was chosen for precision, we will move significant portions to the appendix, replacing them with more intuitive text descriptions and figures. ReMasker and [3] are indeed key previous works and we’ll better highlight ReMasker and move the discussion on copy masking from methods to related works. The [MASK] token is indeed fixed and identical for all features, following standard practice in MAEs and ViTs. The hyperparameters of baseline method will be specified in an appendix table. Finally, we apologize for the ambiguity - we mean row-wise shuffling (permuting rows of the observed missingness mask matrix). All these will be fixed in the final draft. \n\n## Other Comments\nRegarding the comment about including  “the mask as additional input features for the imputation model”, we included notMIWAE, which leverages this idea. It maximizes the ELBO of the joint likelihood of the observed features and missing pattern via IWVAE, this ensured we compared against methods that directly factor in missingness structure. As seen in Table 2, we outperform notMIWAE. \n\nNext, while higher copy-masking rates sometimes improve performance, keeping this as a tunable hyperparameter is justified because: 1) performance decreases at rates >95% in MNAR settings for R² and WD; 2) our conservative 90% rate maximizes average performance across sensitivity analysis datasets but isn't universally optimal; 3) practitioners often prioritize specific features based on domain needs, making flexible masking valuable; and 4) optimal masking rates vary significantly across datasets. Next, the embedding size and depths chosen here perform consistently well with reasonable resource usage (<200MB GPU RAM). Since imputation is rarely the end application, we maintain these as tunable parameters for users to optimize their specific use cases.\n\nFinally, we refer the reviewer to point 4 in the contributions section of rebuttal to reviewer eZeh, which addresses concerns about the theoretical contributions. \n\nWe hope our responses have addressed the reviewer's primary concerns and demonstrated the paper's contribution to current literature. If so, we would appreciate a reconsideration of the overall recommendation. We're happy to address any additional questions!", "claims_and_evidence": "The claims are supported by convincing evidence.", "methods_and_evaluation_criteria": "Methods and evaluation criteria make sense.", "theoretical_claims": "A “theoretical framework to motivate the need for copy masking and why random masking might not be optimal in non-MCAR settings” is provided in Appendix G. Yet it is not exposed in the main paper, and does not contain a clear proposition or theorem, so I did not consider it as a contribution.", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper provides an imputation model for tabular data. Important references for this task are cited in the introduction. A particularity of this approach is that it is meant to also be effective in MNAR, while many methods are only valid under M(C)AR (although the theoretical justification for this is not clearly stated).", "essential_references_not_discussed": "I think that ReMasker and [3], which appear to be the cornerstones of this work (respectively for using a MAE with random masking for imputation, and for using copy-masking) could be introduced with more details in the related work. \n\nFor example, the authors say: “Our first contribution is in extending an approach introduced by [3] of using “… To clearly identify contributions, it would be easier to first introduce copy-masking as a related work, and then present the proposed extension as a contribution.\n\n[3] An et al 2023, Deep learning-based phenotype imputation on population-scale biobank data increases genetic discoveries.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The improvements in the imputation performance, notably under MNAR, are large. +0.02 (MCAR) / +0.03 (MAR) / +0.06 (MNAR) R2 pts compared to the next best model ReMasker.\n\n- Many ablation studies help identify the most critical parts of a model, providing valuable insights for guiding future research.\n\n\n**Weaknesses**\n\n- I found the paper difficult to follow. The explanations rely heavily on notations, which I believe unnecessarily complicates the presentation. Some concepts require significant effort to grasp. The writing could be improved by reducing unnecessary notations and incorporating figures to clarify key ideas, notably copy-masking.\n\n- The results do not convincingly show that using the context improves imputation. We see on Table 2 that using the context on top of random masking improves the R2 from 0.20 to 0.26. However, the result is obtained over 4 datasets only. Moreover, when copy-masking is used instead of random masking, the effect of the context disappears in MAR and becomes very small in MNAR, potentially not significant. From these experiments, it is hard to tell whether using context can help imputations.\n\n- Are the 4 datasets of the ablation studies part of the 10 evaluation datasets? If yes, as it seems that CACTI’s hyperparameters were chosen based on these 4 datasets, it would mean that hyperparameters were chosen on part of the test set, and that the performances may be inflated for these datasets.", "other_comments_or_suggestions": "* On Fig 2., it may be more informative to plot the R2 relative to the mean R2 across methods for a given dataset and missingness mechanism.\n\n* The results in Table 4 on model architecture (encoder and decoder depth and embedding size) show that the smallest options considered are almost the best. It is not clear that an encoder depth of 10 provides significantly better results that a depth of 4. Exploring smaller depths and embedding size would be useful, as the smaller the model the better (given a fixed performance).\n\n* Given the results presented in Figure 3 on Masking rate sensitivity, where increasing the copy masking probability improves performances up to 99% (the maximum probability tested), it could make sense to remove this hyperparameter and apply copy-masking 100% of the time.\n\n* The default hyperparameters of competing method should be specified in Appendix to ensure reproducibility.\n\n* Is the [MASK] token the same for all features?\n\n\n**Typos:**\n\nl.11: due due\n\nl.66: to to\n\nl.134: to the to\n\nl.214: with a detail description\n\nl.239: we conduct a through ablation\n\nl.271: CACTI’s out performs\n\nl.313: of the of the top 5", "questions_for_authors": "* L. 135, the authors state: “we perform a row-wise permutation to create a mask”. Row-wise permutation would mean shuffling elements within each row. Based on other elements in the paper, notably the permutation matrix of size N by N in Algorithm 1, I think that the authors do row-shuffling (eg swapping row I with row j). What do the authors do?\n\n* CACTI outperforms ReMasker by 0.03 R2 pts in MAR (0.06 in MNAR) on average over 10 datasets, but CACTI outperforms RMAE, which uses ReMasker’s random masking, by a much larger margin (+0.25 R2 pts in MAR and MNAR) on 4 datasets. Why such difference in the size of the improvement given that both RMAE and ReMasker use random masking?  \n\n* To leverage the missingness structure, a straightforward competing approach would be to include the mask as additional input features for the imputation model. This may not be straightforward for all methods, but it is for methods such as missforest.\n\n* Do the 10 evaluation datasets include the 4 datasets used for exploring the architecture hyperparameters? \n\n* How were the evaluation datasets chosen?\n\n* What do the column description look like in the 10 evaluation datasets? Were they chosen so that these descriptors are meaningful?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lmy2xtlLcr", "reviewer": "Reviewer_qrHN", "summary": "This paper addresses the missing data imputation problem by using a transformer-based architecture that leverages the missing patterns and textual information about features as inductive biases to improve imputation accuracy. Specifically, the paper proposes a median truncated copy masking training strategy that captures the empirical missing patterns of data during training. Then, the method exploits the feature name information by extracting language model embedding of the feature name and taking it as the input of the imputation model to capture the context information. The experiments show that these two inductive biases improve the accuracy of imputation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and suggestions. We would like to highlight that all the individual per dataset/missingness condition results for all analyses performed in this paper were previously reported with 95% confidence intervals in the Appendix. We choose to exclude standard errors (SEs) in the main text to ensure visual parsimony and due to space constraints. Please see the silhouette for the main benchmarking results table with SEs reported in Table. R1 which we will include in the final version. Finally, we have a new set of results in Table. R2, which shows that CACTI demonstrates, via a paired t-test, statistically significant (p < 0.05) improvement in R2 and RMSE over all missingness conditions when compared to the next best method (ReMasker). This relative ranking improvement holds even after we exclude the 4 datasets used for the sensitivity and ablation analysis (Table. R3). This provides further evidence to show that leveraging inductive biases from dataset-specific missingness patterns and context information can help improve imputation accuracy. Finally, if the reviewer believes we have sufficiently addressed their concerns and finds this paper to be a valuable contribution to the current literature, we would appreciate their consideration in increasing their score. As always, we're happy to address any additional questions or concerns!\n\n**[Please note that we’re only reporting the top few methods since we’re limited to 5000 characters per rebuttals, final version of the paper will contain all methods when appropriate]**\n\n**Table. R1**: Main benchmarking results with standard errors in parentheses and two sets of results indicate train\\|test. \n| Method |  |R2 | | |RMSE | |\n|--------|--------|--------|--------|--------|--------|--------|\n| | MCAR | MAR | MNAR | MCAR | MAR | MNAR |\n| CACTI | 0.455\\|0.461  (0.002)\\|(0.003) | 0.469\\|0.467 (0.010)\\|(0.011) | 0.458\\|0.456 (0.003)\\|(0.004) | 0.663\\|0.640 (0.008)\\|(0.005) | 0.675\\|0.694 (0.016)\\|(0.016) | 0.683\\|0.666 (0.004)\\|(0.006) |\n| CMAE | 0.441\\|0.447 (0.002)\\|(0.002) | 0.459\\|0.460 (0.009)\\|(0.010) | 0.440\\|0.439 (0.002)\\|(0.003) | 0.673\\|0.653 (0.008)\\|(0.007) | 0.685\\|0.696 (0.017)\\|(0.016) | 0.699\\|0.691 (0.005)\\|(0.008) |\n| ReMasker | 0.437\\|0.438 (0.002)\\|(0.002) | 0.445\\|0.443 (0.010)\\|(0.010) | 0.402\\|0.402 (0.003)\\|(0.004) | 0.681\\|0.665 (0.008)\\|(0.006) | 0.691\\|0.712 (0.017)\\|(0.014) | 0.729\\|0.709 (0.005)\\|(0.006) |\n| DiffPuter | 0.400\\|0.415 (0.003)\\|(0.004) | 0.386\\|0.430 (0.010)\\|(0.012) | 0.363\\|0.372 (0.004)\\|(0.004) | 0.731\\|0.704 (0.009)\\|(0.005) | 0.770\\|0.752 (0.020)\\|(0.021) | 0.794\\|0.767 (0.024)\\|(0.024) |\n| HyperImpute | 0.406\\|0.382 (0.003)\\|(0.004) | 0.439\\|0.391 (0.010)\\|(0.011) | 0.393\\|0.347 (0.005)\\|(0.005) | 0.722\\|0.734 (0.007)\\|(0.011) | 0.727\\|0.774 (0.017)\\|(0.016) | 0.757\\|0.776 (0.006)\\|(0.007) |\n\n**Table. R2**: T-test to quantify the statistical significance of the contributions of MT-CM and context awareness (CMAE = CACTI w/o context).\n| Metric | Missingness | Target Method | Baseline Method | Diff. Est. | P Value |\n|---|---|---|---|---|---|\n|  | All | CACTI | ReMasker | 0.034 | 4.4e-7 |\n|  | All | CACTI | CMAE | 0.013 | 1.1e-5 |\n|  | All | CMAE | ReMasker | 0.021 | 4.2e-5 |\n|  | MCAR | CACTI | ReMasker | 0.023 | 5.5e-4 |\n|  | MCAR | CACTI | CMAE | 0.014 | 3.e-3 |\n|  | MCAR | CMAE | ReMasker | 0.017 | 8.9e-3 |\n| R2 | MAR | CACTI | ReMasker | 0.025 | 2.3e-2 |\n|  | MAR | CACTI | CMAE | 0.007 | 9.4e-2 |\n|  | MAR | CMAE | ReMasker | 0.018 | 4.8e-2 |\n|  | MNAR | CACTI | ReMasker | 0.054 | 1.1e-4 |\n|  | MNAR | CACTI | CMAE | 0.017 | 8.7e-4 |\n|  | MNAR | CMAE | ReMasker | 0.037 | 4.6e-4 |\n\n**Table. R3**: Results excluding the 4 datasets used for sensitivity analysis.\n| Method    |           | R2        |           |            | RMSE       |           |        |    WD       |           |\n|-----------|-----------|-----------|-----------|------------|------------|-----------|-----------|-----------|-----------|\n|           | MCAR      | MAR       | MNAR      | MCAR       | MAR        | MNAR      | MCAR      | MAR       | MNAR      |\n| CACTI     | 0.45\\|0.45 | 0.47\\|0.46 | 0.46\\|0.46 | 0.67\\|0.64  |  0.68\\|0.71 | 0.69\\|0.66 | 3.19\\|3.23 | 1.36\\|1.41 | 3.31\\|3.36 |\n| CMAE      | 0.44\\|0.44 | 0.47\\|0.46 | 0.45\\|0.44 | 0.68\\|0.66  | 0.69\\|0.70  | 0.70\\|0.68 | 3.22\\|3.27 | 1.41\\|1.45 | 3.39\\|3.44 |\n| ReMasker  | 0.44\\|0.44 | 0.45\\|0.45 | 0.42\\|0.42 | 0.68\\|0.66  | 0.69\\|0.72  | 0.72\\|0.68 | 3.24\\|3.28 | 1.69\\|1.72 | 3.33\\|3.38 |\n| DiffPuter | 0.41\\|0.43 | 0.41\\|0.46 | 0.38\\|0.39 | 0.73\\|0.69  | 0.74\\|0.73  | 0.80\\|0.76 | 3.33\\|3.32 | 1.89\\|1.53 | 3.43\\|3.41 |\n| HyperImpute | 0.40\\|0.37 | 0.43\\|0.38 | 0.39\\|0.34 | 0.74\\|0.75  | 0.73\\|0.79  | 0.77\\|0.76 | 2.75\\|3.27 | 1.46\\|1.86 | 2.76\\|3.12 |", "claims_and_evidence": "The claims in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the target problem.", "theoretical_claims": "The overall proofs in the supplementary are sound.", "experimental_designs_or_analyses": "The experimental designs is comprehensive with different data set and methods are included. Also, the hyperparameter and ablation study are included.", "relation_to_broader_scientific_literature": "The study is focused on missing data Imputation with inductive biases of missing patterns and feature name description which is applicable to any domain that has MNAR data and whose feature description conveys important information, for example, healthcare.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n The writing and organization of the paper are clear. The experiment is also comprehensive and includes many data sets and methods. Also, ablation studies and the effect of hyperparameters are also included. The paper also theoretically proves the rationale of why the proposed masking strategy is better than the naïve random masking.\n\nWeakness\n\nThe results tables could contain standard deviation to show the statistical significance.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCtfQLrYJN", "reviewer": "Reviewer_eZeh", "summary": "The authors propose a new method to impute missing data in tabular data sets. This method, named CACTI, incorporate three components : a mask autoencoding approach, a median truncated copy-masking training strategy and the use of semantic relations between features. \nThe proposed method is evaluated across 10 real-world data sets with different missingness scenario (MCAR, MAR and MNAR) and different metrics related to pointwise imputation, distributional imputation or predictive performances.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the helpful feedback and questions.\n\n## Comments Addressed\nWe've added CMAE (CACTI without context)  as an additional benchmark in the main results table (qrHN rebuttal Table. R1). We’ve also added a comparison between CACTI, CMAE to quantify the statistical significance of the improvements driven by context awareness (qrHN rebuttal Table. R2). We hope these results further clarify that context yields statistically significant (p<0.05) improvement over masked learning alone and is a useful source of inductive bias to improve imputation. We really appreciate the reviewer's attention to detail and highlighting the typos. They have now been addressed, and we apologize for the oversight.\n\nNext, our theoretical results in Appendix G show that copy masking can be *more* effective than random masking under MNAR. For MAR, consider a structured MAR setting [1] where the missing variables are dependent on some observed variable but also have additional dependencies, e.g., feature i is missing if feature j is missing. Such settings are quite common in questionnaire and other datasets. Under this setting, copy masking is still able to maintain the *structured* missingness between the variables whereas random masking completely diverges from the missingness process. The crux of our argument in Appendix G is that copy masking is better suited to modeling MAR and MNAR than random masking. For example, unlike random masking, applying copy masking to MAR results in masking only the missing features. This encourages the MAE to learn a conditional model that maximizes the probability of the masked feature given the observed features. \n\n## Contributions\nWhile MAEs with random masking (ReMasker) and MLPs with naive copy masking (AutoComplete) have both been used to tackle imputation in prior works, we believe our work makes novel, concrete contributions.\n1. MT-CM, an upgrade to copy masking, is a novel, key contribution. As highlighted by our results in Table A7 the application of naive copy masking to transformer-based MAE architecture leads to suboptimal performance and reduced learning efficiency. This is a significant gap in the field, as transformers are arguably the most popular architecture in current literature. MT-CM enables the application of copy masking effectively to such architectures and creates an upper bound on the worst-case scenario in batch-based training, i.e. the proportion of null tokens in any batch is upper bounded by 50%. Our MT-CM strategy unlocks the effective use of copy masking for any transformer-based masked learning model and can have applications beyond just tabular imputation.\n2. CACTI is the first approach (to the best of our knowledge) to effectively leverage context as a useful inductive bias and show statistically significant improvements in various imputation settings. Our results highlight the importance of investigating new inductive biases as a promising direction for future tabular imputation research, particularly in domains with MNAR missingness.\n3. Our results establish CACTI as a strong baseline for evaluating and developing future approaches in tabular imputation.\n4. Although our theoretical section (Appendix G) is not highlighted in the main text (due to space restrictions), we believe we do provide a new empirical risk framework for evaluating MAE training without fully observed data (previous works don’t consider the probabilistic nature of the missingness process and only view the objective as representation loss minimization), provide the first formal motivation for copy masking and show how random masking is suboptimal compared to copy masking in non-MCAR settings. If given the chance we would like to introduce this theoretical framework in the main text in the final draft.\n\nWe hope this addresses the reviewer's concerns and find this paper to be a valuable contribution to the current literature. We would appreciate their reconsideration of their overall recommendation. And as always we're happy to address any additional questions or concerns!\n\n[1] Jackson, J., et. al. A complete characterisation of structured missingness, 2023.", "claims_and_evidence": "The method is well described and the experimental protocol is well explained. \n\nRegarding the way masks are generated : since they are permuted across observations, I agree that the dependence between missingness components is preserved. However, if masks are MAR, that is the probability of missingness depends on the observed values, permuting masks will break this dependence. Thus, the proposed strategy seems valid only in MCAR setting, with possible dependencies between missingness components. Thus, it is not valid for any type of missingness. Please correct me if this is not right.", "methods_and_evaluation_criteria": "The experimental protocol is sound. In table 1, it would be interesting to add CACTI without the context awareness. Such a method is more comparable to classic method such as MissForest, and it would be interesting to see how they compare to each other (as I understand Table 2 does that but only on few data sets). \n\nIt could also be interesting to test the robustess to mislabeled input variables : what happens in terms of predictive performances, when the description of columns are permuted ?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental protocol is sound. Many baselines from classic imputation techniques to deep learning strategies are compared.", "relation_to_broader_scientific_literature": "The literature is well cited with respect to missing data imputation.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "This paper propose to combine different existing ingredient to impute missing data. While there is no theory, and while the techniques are not totally new, the empirical protocol is exhaustive, testing for different missing data mechanisms, different datasets, and comparing several baselines. The empirical evaluation is thorough and is interesting on its own, in my opinion.", "other_comments_or_suggestions": "Typos : \n- page 1, second column : ‘can arise due due to’\n- page 2 line 81, where $k \\in [K]$ a verb is missing here\n- page 2, line 66 : ‘a direct way to to incorporate’\n- page 3 line 134, ‘contextual awareness to the to improve’\n- page 3 line 158 since $m=1$ corresponds to an observed entry, I would say that feature $i$ is observed any time $j$ is observed\n- page 5, line 273, so can we simulate\n- page 7 line 336 just using the the feature context", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission present a missing-value imputation with a approach following the line of masked auto-encoders, but recycling missing-value patterns actually present in the data as well as exploiting the column names. The submission generated interest from the reviewers and discussion. The reviewers appreciated the gain in imputation performance.

It was noted that the readability could be improved. Should the paper be accepted, the authors should work on this.
Decision: Accept (spotlight poster)