Have Missing Data? Make It Miss More! Imputing Tabular Data with Masked Autoencoding
Authors: Tianyu Du
Luca Melis
Ting Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Imputation
Tabular Data
Masked Autoencoder

Abstract: We present ReMasker, a novel method for imputing missing values in tabular data by extending the masked autoencoding framework. In contrast to prior work, ReMasker is both {\em simple} -- besides the missing values (i.e., naturally masked), we randomly ``re-mask'' another set of values, optimize the autoencoder by reconstructing this re-masked set, and apply the trained model to predict the missing values; and {\em effective} -- with extensive evaluation on benchmark datasets, we show that ReMasker consistently outperforms state-of-the-art methods in terms of both imputation fidelity and utility under various missingness settings, while its performance advantage often increases with the ratio of missing data. We further explore theoretical justification for its effectiveness, showing that ReMasker tends to learn missingness-invariant representations of tabular data. Our findings indicate that masked modeling represents a promising direction for further research on tabular data imputation.

HAVE MISSING DATA? MAKE IT MISS MORE! IMPUT- ING TABULAR DATA WITH MASKED AUTOENCODING
Anonymous authors Paper under double-blind review
ABSTRACT
We present REMASKER, a novel method for imputing missing values in tabu- lar data by extending the masked autoencoding framework. In contrast to prior work, REMASKER is both simple besides the missing values (i.e., naturally masked), we randomly "re-mask" another set of values, optimize the autoencoder by reconstructing this re-masked set, and apply the trained model to predict the missing values; and effective - with extensive evaluation on benchmark datasets, we show that REMASKER performs par with or outperforms state-of-the-an methods in terms of both imputation fidelity and utility under various missing- ness settings, while its performance advantage often increases with the ratio of missing data. We further explore theoretical justification for its effectiveness, showing that REMASKER tends learn missingness-invarnnn tabular data. Our findings indicate that masked modeling represents a promising direction for further research on tabular data imputation. The code is available at: https: /emps://annee anonymous 4oper. science I/ remasker- 5E3B
representations of
INTRODUCTION
Missing values are ubiquitous in real-world tabular data due to various reasons during data collection, processing, storage, or transmission. It often desirable know the most likely values of missing data before performing downstream tasks (e.g., classification or synthesis). To this end, intensive research has been dedicated to developing imputation methods ("imputers")) that estimate missing values based on observed data (Yoon et al. 2019 Jarrett et al. 2022 Kyono et al. 2021 Stekhoven Buhlmann 2012, Mattei & Frellsen| 2018). Yet, imputing missing values tabular data with high hddeiity and utility remains an open problem, due to challenges including the intricate correlation across different features, the variety of missingness scenarios, and the scarce amount of available data with respect to the number of missing values.
The late-of-theear imputers can be categorized as either discriminative or generative. The dis- criminative imputers, such as MissForest (Stekhoven & Buhlmann, 2012). MICE (van Buuren & Groothuis-Oudshoom 201T), I), and MIRACLE (Kyono et al., 2021). impute missing values by model- ing their conditional distributions hindered by the requirement of specifying the proper functional forms of conditional distributions and adding the set of appropriate regularizers. The generative imputers, such as GAIN Yoon et al. 2019) MIWAE (Mattei& & Frellsen 2018). GAMIN (Yoon & Sull 20200. and HI-VAE (Nazabal et al. 2020
the basis of other values. In practice, these methods are often
estimate the joint distributions of all the features by leveraging the capacity of deep generative models and impute missing values by querying the trained models. Empirically. GAN- based methods often require a large amount of training data and suffer the difficulties of adversarial training (Goodfellow) et al. 2)14), while VAE-based methods often face the limitations of training through variational bounds (Zhao et al., 2022). Further, some of these methods either require complete data during training operate the assumptions of specific missingness patterns.
In this paper, we present REMASKER, a novel method that extends the masked autoencoding (MAE) framework (Devlin et al. 2018 He et al. 2022) to imputing missing values of tabular data. The idea of REMASKER is simple: Besides the missing values in the given dataset (i.e., naturally masked), we randomly select and "re-mask" another set of values, optimize the autoencoder with the objective of reconstructing this re- ma-madeed set, and then apply the trained autoencoder to predict the missing values. Compared with the prior work, REMASKER enjoys the following desiderata: (i) it is instantiated

---

with Transformer (Vaswani et al., 2017) as its backbone, of which the self-attention mechanism is able to capture the intricate inter-feature correlation (Huang et al., 2020); (ii) without specific assumptions about the underlying missingness mechanisms, it is applicable to various scenarios if complete data is unavailable; and (iii) as the re-masking approach naturally accounts for missing values and encourages learning high-level representations beyond low-level statistics, REMASKER works effectively even under a high ratio of missing data (e.g., 0.7).
With extensive evaluation on 12 benchmark datasets under various missingness scenarios, we show that REMASKER performs on par with or outperforms 13 popular methods in terms of both imputation fidelity and utility, while its performance advantage often increases with the ratio of missing data. We further explore the theoretical explanation for its effectivenesss We find that REMASKER encourages learning missingness-innee
representations of tabular data, which are insensitive to missing values. Our findings indicate that, besides its success in the language and vision domains, masked modeling also represents a promising direction for future research on tabular data imputation.
RELATED WORK
Here, we survey relevant literature in three categories.
Tabular data imputation. The existing imputation methods can be roughly categorized as either discriminative or generative. The discriminative methods (Stekhoven & Buhlmann 2012 van Buuren & Sroothuis-Oudshood
2011; Kyono et al.] 2021) often specify a univariable model for each feature conditional on all others and perform cyclic regression over each target variable until convergence. Recent work has also explored adaptively selecting and configuring multiple discriminative imputers (Jarettt et al. 2022). The generative methods either implicitly train imputers as generators within the GAN framework (Yoon et al. 2019 Yoon & Sull 2020) or explicitly train deep latent-variable models to approximate the joint distributions of all features (Mattei & Frellsen 2018 Nazabal et al.) 2020). There are also imputers based on representative-vant value (e.g., mcan, median, or frequent values) substitution (Hawthorne Elliou 2005). EM optimization (Garcia-Laencina
et al.
2010), matrix completion (Hastie et al.) 2015). or optimal transport (Muzellec et al. 2020).
Transformer. Transformer has emerged as a dominating design (Vaswani et al. 2017) in the language domain, in which multi-head self-attention and MLP layers are stacked to capture both short- and long-term correlations between words. Recent work has explored the use of Transformer in the vision domain by treating each image as grid of visual words (Dosovitskiy et al. (2020). For instance, it has been integrated into image generation models (Tiang et al.] [Zhang et al. 2021 Hudson & Zitnick 202I), achieving performance comparable to CNN-based models.
self- Masked autoencoding. Autoencoding is a classical method for learning representation supervised manner (Vincent et al. 2008 Pathak et al. (016): an encoder maps an input to its representation and a decoder reconstructs the original input. Meanwhile, masked modeling is originally proposed as pre-training method in the language domain: by holding out a proportion of a word sequence, it trains the model to predict the masked words (Devlin et al. 2018; Radford & Narasimhan 2018). Recent work has combined autoencoding and masked modeling tasks (Dosovitskiy et al., 2020; Bao et al. 2022). Particularly, the seminal MAE (He et al. 2022) represents the state of the art in self-supervised pre-training on the ImageNet- IK benchmark.
vision
The work is also related to that models missing data by adapting existing model architec- tures (Przewigzllkowski al. 202Tp. To our best knowledge, this represents the first work to explore the masked autoencoding method with Transformer in the task of tabular data imputation.
3 REMASKER
Next, we present REMASKER, extremely simple yet effective method for imputing missing values of tabular data. We begin by formalizing the imputation problem.
3.1 PROBLEM FORMALIZATION
Incomplete data. To model tabular data with d features, we consider a d-dimensional random variable (x1,-.., xa) X, x ... x Xd, where X; is either continuous categorical for i (1,...,d)
2

---

Fitting
Imputation
Poddied Embedding
Paddedd Embedking
Impul
mm
Recorstructed Value
Input
Embedding
m
Impuled Value
Embedkiing
Finc ooder
Decoder
Z6
Mizsing Value
Masked Value
[m] Mask Token
Figure 1: Overall framework of REMASKER. During the fitting stage, for each input, in addition to its missing values, another subset of values (re-masked values) is randomly selected and masked out. The encoder is applied the remaining values generate embedding, which padded with mask tokens and processed by the decoder re-construct the re-masked values. During imputation stage, the optimized model is applied to predict the missing values.
The observational access to x is mediated by an mask variable m - (mi,..., ma) E {0, 13d, which indicates the missing values of x, such that is accessible only if m, 1. In other words, observe x in its incomplete form x - (81..... Xd) with
X; if m, =
(1 (L......... dd)
(1)
if m,
where denotes the unobserved value.
Missingness mechanisms. Missing values occur due to various reasons. To simulate different scenarios, following the prior work (Yoon et al.) 2019 Jarrett et al.) 2022). we consider three missingness mechanisms: MCAR ("missing completely at random") ") the missingness does not depend on the data, which indicates that Vm, x, x', p(mmx) p(m[x'') MAR ("missing random"" - the missingness depends on the observed values, which indicates that Vm, x, x', if the observed values of x and x' are the same, then p(m[x) p(m xx);; and MNAR ("missing not at random")) the missingness depends on the missing values as well, which is the case if the definitions of MCAR and MAR do not hold. In general, impossible identify the missingness distribution of MNAR without domain-specific assumptions or constraints (Ma & Zhang 2021).
Imputation task. In this task, we are given an incomplete dataset D ((((0) m )
which
consists of i..d. realizations of x and m. The goal is recover the missing values of each input by generating an imputed version x A (8 ......., &a) such that
8; if m;
&
(1 (l....,,,
(2)
X
if m, =
where X; the imputed value.
3.2 DESIGN REMASKER
The REMASKER imputer extends the masked autoencoding (MAF) framework (Dosovitskiy et al. 2020 Bao et al., 2022 He et al.) (022) that reconstructs masked components based on observed components. As illustrated in Figure REMASKER comprises an encoder that maps the observed values to their representations and decoder that reconstructs the masked values from the latent representations. However, unlike conventional MAE, as the data in the imputation task is inherently incomplete (i.e., naturally masked), we employ a "re-masking" approach that explicitly accounts for this incompleteness in applying masking and reconstruction. At high level, REMASKER works in two phases: fitting- it optimizes the model with respect to the given dataset, and imputation applies the trained model to predict the missing values of the dataset.
Re-masking, the fitting phase, for each input X, in addition to its missing values, we also randomly select and mask out another subset (e.g., 25%) of X's values. Formally, letting m be x's mask,
Without ambiguity, omit the superscript in the following notations.
3

---

Algorithm 1: REMASKER
Input: D ((E m J incomplete dataset: remask: re- re-masking function; fe. de: encoder and decoder; epoch: training epochs; E: reconstruction loss
Output: {(x(0)) J==:: imputed dataset
// fitting phase
while max_epoch
reached do
foreach (x.m) E
8, , Xm/mm f- remask(X, X, m)
// remasking // encoding unmasked values
z <- (a(Xm/mm))
pad with mask tokens;
update 0, A by Vi(d)(1z)), m)
1/ minimizing reconstruction loss
// imputation phase
foreach (5.m) D do
z - (Km)
// encoding observed values
pad x with mask tokens;
10
Am <- do (
// predicting missing values
& Xm Xm:
12 return D {x):
define another mask vector m' E {0, 1] which is randomly sampled without replacement, following uniform distribution. Apparently, m and m' entail three subsets:
Imaak {i = 0 Zremask {ilm; = 1 A m² = 0} Tunmask = (ilm; = 1 A m = 1
Let 8m. X =m and 8m/m respectively be the masked, re-masked, and unmasked values. With sufficient number of re-masked values, addition to the missing values, we create a challenging task that encourages the model to learn missingnessinnvarian representations (more details in $ Note that in the imputation phase, we do not apply re-masking.
Encoder. The encoder embeds each value using an encoding function and processes the resulting embeddings through sequence of Transformer blocks. In implementation, we apply linear encoding function to each value æ: enc(#) ) = w.r + b, where and b are learnable parameters We also add positional encoding to x'' embedding to force the model to memorize x's position in the input (e.g.. k 1000028/d), where k and i respectively denote x's position in the input and the dimension of the embedding, and is the embedding width.
the k-th feature): pe(k: 2i) sin(k/
Note that the encoder is only applied to the observed values: in the fitting phase, it operates on the observed values after re-masking (i.e., the unmasked set Zunmask): in the imputation phase, it operates on the non-missing values (i.e., the union of re-masked and unmasked sets Iummaskk U Zremaskk) as illustrated Figure
Decoder. The REMASKER decoder is instantiated as a sequence of Transformer blocks followed by an MLP layer. Different from the encoder, the decoder operates on the embeddings of both observed and masked values. Following (Devlin et al., 2018 He et al. 2022), use a shared, learnable mask token as the initial embedding of each masked value. The decoder first adds positional encoding to the embeddings of all the values (observed and masked), processes the embeddings through sequence of Transformer blocks, and finally applies linear projection to map the embeddings to scalar values as the predictions. Similar to (He al. (022), we use asymmetric design with deep encoder and a shallow decoder (e.g., 8 blocks versus 4 blocks), which often suffices to re-construct the masked values. Conventional MAE focuses representation learning and uses the decoder only in the training phase. In REMASKER, the decoder is required to re-construct the missing values and thus used both fitting and imputation phases.
Reconstruction loss. Recall that the REMASKRR decoder predicts the value for each input feature. We define the reconstruction loss functions as the mean square error (MSE) between the reconstructed and original values on (i) the re-masked set Zremask and (ii) unmasked set Zunmask- We empirically experiment with different reconstruction loss functions (e.g., only the re-masked set or both re-masked and unmasked sets).
Putting everything together, Algorithm [sketches the implementation REMASKER.
We have explored other encoding functions including periodic activation function Gorishniy et al. 2022 which observes slight decrease (e.g., 0.01 RMSE) in imputation performance.
4

---

4 EVALUATION
We evaluate the empirical performance of REMASKER in various scenarios using benchmark datasets. Our experiments are designed to answer the following key questions: (7) Does REMASKER work? We compare REMASKER with a variety of state-of-the-eee imputers in terms of imputation quality. (ii) How does it work? We conduct an ablation study to assess the contribution of each component of REMAS ASKER to its performance. (iii) What is the best way of using REMASKER. ? - We explore the use of REM AKKR as standalone imputer as well one component of an ensemble imputer to understand its best practice.
Datasets. For reproducibility and comparability, similar the prior work (Yoon al. 2019 Jarrett
et al. 2022), we use 12 real-world datasets from the UCI Machine Learning repository (ua & Grafl
2017 with their characteristics deferred to Appendix SA..1
Missing mechanisms. We consider three missingness mechanisms. In MCAR, the mask vector of each input is realized following a Bernoulli random variable with a fixed mean. In MAR, with a random subset of features fixed to be observable, the remaining features are masked using logistic model. In MNAR, the input features of MAR are further masked following a Bernoulli random variable with a fixed mean. We use the Hyperlmpute platform (Jarrett et al. 2022) to simulate the above missing mechanisms.
Baselines. We compare REMASKER with 13 state-of-the-art imputation methods: Hyperlmpute (Jar- rett et al. 2022), a hybrid imputer that performs iterative imputation with automatic model selection; MIWAE (Mattei & Frellsen 2018). an autoencoder model that fits missing data by optimizing variational bound; EM
Laencina et al., 2010), an iterative imputer based on expectation- maximization optimization; GAIN (Yoon et al. 2019), generative adversarial imputation network that trains the discriminator to classify the generator 's output in an element- manner; ICE, an iterative imputer based on regularized linear regression; MICE, an ICE-like, iterative imputer based on Bayesian ridge regression; MIRACLE (Kyono et al. 2021), an iterative imputer that refines the imputation of baseline by simultaneously modeling the missingness generating mechanism; MissFor- est (Stekhoven & Buhlmann 2012). an iterative imputer based on random forests; Mean (Hawthoree Ellioti 2005). Median, and Frequent, which impute missing values using column- wise uncondi- tional mean, median, and the most frequent values, respectively; Sinkhorn (Muzellec et al. 2020), an imputer trained through the optimal transport metrics of Sinkhorn divergences; and SoftImpute (Hastie) ct al. 2015), which performs imputation through soft-thresholde singular value decomposition.
Metries. For each imputation method, we evaluate the imputation fidelity and utility by comparing its imputed data with the ground-truth data. In terms of fidelity, we mainly use two metrics: root mean square error (RMSE) to measure how the individual imputed values match the ground-truth data, and the Wasserstein distance (WD) to measure how the imputed distribution matches the ground- truth distribution. In terms of utility, we use area under the receiver operating characteristic curve (AUROC) the metric on applicable datasets (i.e., ones associated with classification tasks). In the case of multi-class classification, we use the one versus rest (OvR) setting. To be fair, we use logistic regression as the predictive model across all the cases.
4.1 OVERALL PERFORMANCE
We evaluate REMASKER and baseline imputers on the benchmark datasets under the MAR setting with 0.3 missingness ratio, with results summarized in Figure 2 Observe that REMASKER consis- tently outperforms all the basclines in terms of both fidelity (measured by RMSE and WD) and utility (measured by AUROC) across all the datasets. Recall that the benchmark datasets are collected from a variety of domains with highly varying characteristics (cf. T)) the dataset size varies from 308 to 20,060, while the number of features ranges from 7 to 57. Its superior performance across all the datasets demonstrates that REMASKER effectively models the intricate correlation among differeent features, even if the amount of available data is scarce. The only imputer with performance close to REMASKER is Hyperlmpute (Jarrtt et al. 2022). which is an ensemble method that integrates multiple imputation models and automatically selects the most fitting model for each column of the given dataset. This highlights that the modeling capacity of REMASKER 's masked autoencoder is comparable with ensemble models. In Appendix B.T conduct more comprehensive evaluation
5

---

RoMasker
Hyperimpute
GAIN
ICE
MICE
Mishfores:
Sinkhorn Softimpute
MIRACLE
Trequent
01.6
0.2
Mithaaharatt
0.0
compresiin
disbetes
abesity
spar
0.4
0.0
compression
diabetes
obesity
spain
winoo
yacht
Ideal AUROC
AUROC
0.9
0,8
abesity
spam
Figure Overall performance of REMASKER and baseline imputers 12 benchmark datasets under MAR with 0.3 missingness ratio. The results are shown as the mean standard deviation of RMSE, WD. AUROC scores (AUROC is only applicable to datasets with classification tasks). Note that REMASKER outperforms all the bascline imputers under at least one metric across all the datasets.
+ Hypertmpute
MissForest
0.7
0.3
0.7
RMSE
0.1
0.05
1000
7000
13000
20000
4
6
10
12
14
16
0.1
0.3
0.5
0.7
0.15
0.125
0.1
0.2
0.007
0.05
0.05
0.025
0.0
1000
7000
13000
20000
12
14
0.1
0.5
0.7
0.79
AURDC
0.78
0.76
0.6
0.77
0.76
13000
(al Dataset
Number
Features
(c) Missingness Ralio
Figure 3: Sensitivity analysis of REMASKER the letter dataset under the MAR setting. The results
shown terms of RMSE, WD, and AUROC, with the scores measured with respect to (a) the dataset size, (b) the number of features, and (c) the missingness ratio. The default setting is as follows: dataset size = 20,000, number of features = 16. and missingness ratio 0.3.
by simulating all three missingness scenarios (MCAR, MAR, and MNAR) with different missingness ratios. The results show that REMASKER consistently performs better across a range of settings.
4.2 SENSITIVITY ANALYSIS
To assess the factors influencing REMASKER's performance, we conduct sensitivity analysis by varying the dataset size, the number of features in the dataset, and the missingness ratio under the

---

MAR setting. Figure shows the performance of REMASKER within these experiments against the six closest competitors (Hyperlmpute, ICE. MissForest, GAIN, MIWAE, and Sinkhorn on the letter dataset. We have the following observations. (a) The performance of REMASKER improves with the size of available data, while its advantage over other imputers (with the exception of Hyperlmpute) grows with the dataset size. (b) The number of features has significant impact on the performance of REMASKER, with its advantage over other imputers increasing steadily with the number of features. This may be explained by that REMASKER relies on learning the holistic representations of inputs, while including more features contributes to better representation learning. (c) REMASKER fairly insensitive the missingness ratio. For instance, even with 0.7 missingness ratio, it achieves RMSE below 0.1, suggesting that it effectively fits sparse datasets. In Appendix $ B.2 we also conduct an evaluation on other datasets with similar observations.
4.3 ABLATION STUDY
We further conduct an ablation study of REMASKER to understand the contribution of different components to its performance using the letter dataset. The results other datasets are deferred to Appendix $ B.3
Table 1, Ablation study of REMASKER on the letter dataset. The default setting is as follows: encoder depth = 8, decoder depth 6, embedding width = 64, masking ratio = 50%, and training epochs 600.
depth
RMSE
AUROC
width
RMSE
WD
AUROC
depth
RMSE
WD
AUROC
0.0729
0.0263
0.7898
0.0902
0.0379
0.7902
0.0637
0.0239
0.7887
4 0.0636
0.0228
0,7903
32
0.0714
0.0289
0.7885
0.0625
0.0236
0.7877
6
0,0616
0,0219
0,7909
64
0,0616
0.0219
0.7909
0,0644
0.0239
0.7889
0.0611
0.0217
0.7892
128
0.0795
0.0305
0.7845
0.0616
0.0219
0.7909
10
0.0673
0.0245
0.7879
256
0.1040
0.0403
0.7868
10
0.0637
0.0227
0.7878
(a) Decoder depth
(b) Embedding width
(c) Encoder depth
Model design. The encoder and decoder of REMASKER can be flexibly designed. Here, we study the impact of three key parameters, the encoder depth (the number of Transformer blocks in the encoder), the embedding width (the dimensionality of latent representations), and the decoder depth, with results summarized Table Ta| Table [b] and Table Ic] respectively. Observe that the performance of REMASKER reaches its peak with proper model configuration (encoder depth 8, decoder depth =8, and embedding width = 64). This observation suggests that the model complexity needs to the given dataset: it needs to be sufficiently complex to effectively learn the holistic representations of inputs but not overly complex to overfit the dataset. We also compare the performance of REM ASKRR with different backbone models (i.e., Transformer, linear, and convolutional) with the number of layers and the size of each layer fixed as the default setting. As shown in Table 2 Transformer-baar
REMASKER largely outperforms the other variants, which may be explained by that the self-attention mechanism of Transformer is able to effectively capture the intricate inter-feature correlation under limited data (Huang et al. 2020).
Table 2. Performance of REMASKER with different backbone models.
lecter
foraio
backbone
RMSE
WD
AUROC
RMSE
WD
Transformer
0.0611
0.0217
0.7892
0.0663
0.0172
Linear
0.1732
0.1604
0.7821
0.1786
0.1329
Convolutional
0.1694
0.1582
0.7836
0.1715
0.1286
Table 3. Performance of REMASKER with reconstruction loss w/ w/o unmasked values (note: AUROC inapplicable to the cal forria datasct).
letter
cali california
loss
RMSE
WD
AUROC
RMSE
WD
Imaskt Tammaskk
0.0616
0.0219
0.7909
0.0663
0.0172
Imadet
0.0629
0.0237
0.7890
0,0840
0.0311
Tummask
0.2079
0.1
0.7901
0.1932
0.1906
Reconstruction loss. We define the reconstruction loss the error between the reconstructed and
original values on the re- re-masked values Imask+ and the unmasked values Iunmaste We measure the performance of REMASKER under three different settings of the construction loss: (i) Imask U
7

---

Tunmask. (it) Imask+ only, and (iit) Tunmask only the letter and cali datasets, with results shown in 3 Observe that using the reconstruction of unmasked values only is insufficient and yet including the reconstruction loss of unmasked values improves the performance, which is especially the case on the call fornia dataset. This finding is different from the vision domain in which computing the loss unmasked image patches reduces accuracy (He al., 2022). We hypothesize that this difference is explained as follows. Unlike conventional MAE, due to the naturally missing values in tabular data, relying on re-masked values provides limited supervisory signals. Moreover, while images are signals with heavy spatial redundancy (i.e., a missing patch can be recovered from its neighboring patches), tabular data tends to be highly semantic and information- dense. Thus, including the construction loss of unmasked values improves the model training.
4.4 PRACTICE OF REMASKER
Finally, we explore the optimal practice of REM REMASKER.
X10
8.0
8.0
WD
AUROC
3.5
7.0
7.9
Recordemcctio
3.0
2.5
2.0
6.0
2.2
7.8
55
3.0
1600
2.0
1600
7.7
1600
(2) Max training kpachs
(bi training Epochs
Figure 4: (a) REMASKER performance with respect to the maximum number of training epochs; (b) Convergence of REMASKER'S 's reconstruction loss. The experiments are performed on the letter dataset under MAR with 0.3 missingness ratio.
Training regime.
The ablation study above by default uses 600 training epochs. Figure (a) shows the impact of training epochs, in which vary the training epochs from 100 to 1,600 and measure the performance of REMASKER on the letter dataset. Observe that the imputation performance improves (as RMSE and WD decrease and AUROC increases) steadily with longer training and does not fully saturate even at 1,600 epochs. However, for efficient training, it is often acceptable to terminate earlier (e.g.. 600 epochs) with sufficient imputation performance. To further validate the trainability of REMASKER, with the maximum number of training epochs fixed at 600 (which affects the learning rate scheduler), we measure the reconstruction loss as a function of the training epochs. As shown in Figure (b) the loss quickly converges to plateau within about 100 epochs and steadily decreases after that, demonstrating the trainability of REMASKER.
Masking ratio. The masking ratio controls the number of re-masked values (after excluding missing values). Table 4a] shows its impact on the performance of REMASKER. Observe that the optimal ratio differs across different datasets, which may be explained by the varying number of features of different datasets (16 versus 9 in letter and california). Intuitively, a larger number of features affords higher masking ratio to balance (i) encouraging the model to learn missingness- invariant representations and (ii) having sufficient supervisory signals to facilitate the training.
Table 4. (a) REMASKER performance with respect to masking ratio: (b) REMASKER as the base imputer within Hyperlmpute. The results are evaluated on letter and cal fornia under MAR with 0.3 missingness ratio.
The
masking ratio
letter
cal i fornia
RMSE
WD
AUROC
RMSE WD
letter
california
base imputer
0.1
0.0668
0.0215
0.0789
0.0888 0.0230
RMSE
WD
AUROC
RMSE WD 0.0722 0.0134
0.3
0.0562
0.0207
0.7897
0.0654 0.0151
default REMASKER
0.0564
0.0215
0.7899
0.5
0.0554
0.0212 0.7935
0.0663
0.0172
0.0554
0.0212 0.7935
0.0702 0.0115
0.7
0.0906
0.0366 0.7878
0.1320 0.0650
(b) REMASKER as base imputer.
Masking ratio.
Standalone vs. ensemble. Besides using REMASKER as a standalone imputer, we explore its use as a base imputer within the ensemble imputation framework of Hyperlmpute, with results summarized in Table 4b It is observed that compared with the default setting (with mean substitution as the base imputer), using REMASKER as the base imputer improves the imputation performance, suggesting another effective way of operating REMASKER.
8

---

5 DISCUSSION
The empirical evaluation above shows REMASKER'S superior performance in imputing missing values of tabular data. Next, provide theoretical justification for its effectiveness. By extending the siamese form of MAE (Kng Zhang 2022), we show that REMASKER encourages learning bessinggnss--ivariant missingness- invariant representations of input data, which requires holistic understanding of the
data even in the presence of missing values.
Let fo(-) and do(+) (-) respectively be the encoder and decoder. For given imput x, mask and re-mask m', the reconstruction loss of REM ASKER training is given by (here we focus on the reconstruction of re-masked values):
((x, m, m') = [do(So(x @ m @ m') @ (1 m')
@ m - x x (1 (1-m')= m|l
where @ denotes element-wise multiplication. Let m 2 m @ m' and m - m ② (1 m') Eq 3 be simplified as: f(x, m mmm (x @ m²
m dimensionality is typically much larger than the number of features, it is possible to make the autoencoder lossless. other words, for given encoder fo(.), there exists decoder dæ (+), such that d. (Jo @ m )) @ m R x ④ m We can further re-write Eq as:
m 1 As the embedding
f(x, m .mm) Ildo m²)) o m
do (xe(x 3 m 0
s.t. J' = arg min Ex/ Id (fo(x o m o m - m
/'
We now define distance metric A 0, (z,z') (I(da (z))) ② m m= reformulated as:
Then, Eq
is
f(x, m² m-) L0.
m So( 9 m
s.t. 0' = arg min Exx ||de/ (fo(x's @ m m))) m - o
Note that optimizing Eq essentially minimizes the difference between x's representations under m' and m (with respect to the decoder). As m and m mask out different values, this formulation promotes learning representations insensitive missing values.
ClA Smilarity
Missingness Ratio
0.3
200
400
600
Training Epochs
Figure 5: CKA similarity between the representations complete and incomplete inputs (with the number of missing values controlled by the missingness ratio). tested model trained on letter under the MAR setting with 0.3 missingness ratio.
To validate the analysis above, we empirically measure the CKA similarity (Kornblith et al. 2019 between the latent representations (i.e., the output of REMASKER'S 's encoder) of complete inputs and inputs with missing values, with results shown in Figure 3 Observe that the CKA measures under different missingness ratios all steadily increase with the training length, indicating that REMA ASKER tends to learn missingness-invarian representations tabular data, which may explain for its imputation effectiveness.
6 CONCLUSION
In this paper, we conduct a pilot study exploring the masked autoencoding approach for tabular data imputation. We present REMASKER, novel imputation method that learns missingees--nnarrian representations of tabular data and effectively imputes missing values under various scenarios. With extensive evaluation benchmark datasets, we show that REMASKER outperforms sate-of--he--n methods in terms of both imputation utility and fidelity. Our findings indicate that masked tabular modeling represents a promising direction for future research tabular data imputation.

---

REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT Pre- Training of Image Transform- ers. In Proceedings of International Conference on Learning Representations (ICLR), 2022.
Jacob Devlin, Ming- Beig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NACCL), 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv e-prints, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL ics. uci. edu/ml
Pedro J. García- Larnci-Laencina, José-Luis Sancho-Gómez, and Aníbal R. Figueiras-Vidar Pattern Clas- sification with Missing Data: Review. Neural Computing and Applications, 19(2):263-282, 2010.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2014.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On Embeddings for Numerical Features in Tabular Deep Learning. ArXiv e-prints, 2022.
Trevor Hastie, Rahul Mazumder, Jason D. Lee, and Reza Zadeh. Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares. J. Mach. Learn. Res., 16(1):33673302, 2015.
G. Hawthorne and Elliott. Imputing cross-sectional missing data: comparison of common tech- niques. Aust N ZJ Psychiatry, 39(7):583-90, 2005.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. ArXiv e-prints, 2020.
Drew A. Hudson and C. Lawrence Zitnick. Generative Adversarial Transformers. In Proceedings of IEEE Conference on Machine Learning (ICML), 2021.
Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, and Mihaela der Schaar. HyperImpute: Generalized Iterative Imputation with Automatic Model Selection. In Proceedings of IEEE Conference on Machine Learning (ICML), 2022.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang, TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2021.
Xiangwen Kong and Xiangyu Zhang. Understanding Masked Image Modeling via Learning Occlusion Invariant Feature. arXiv e-prints, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In Proceedings of IEEE Conference Machine Learning (ICML), 2019.
Trent Kyono, Yao Zhang, Alexis Bellot, and Mihaela der Schaar. MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
10

---

Chao Ma and Cheng Zhang. Identifiable Generative Models for Missing Not Random Data Imputation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Piexre-Alexandre Mattei and Jes Frellsen. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data. In Proceedings of IEEE Conference on Machine Learning (ICML), 2018.
Boris Muzellec, Julie Josse, Claire Boyer, and Marco Cuturi. Missing Data Imputation Using Optimal Transport. In Proceedings of IEEE Conference on Machine Learning (ICML), 2020.
Alfredo Nazabal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. Handling Incomplete Heterogeneous Data Using VAFs. Pattern Recognition, 107:107501, 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context Encoders: Feature Learning by Inpainting. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Marein Przewigžlikowski, Marek Smieja, Lukasz Struski, and Jacek Tabor. MisConv: Convolutional Neural Networks for Missing Data. ArXiv e-prints, 2021.
Alec Radford and Karthik Narasimhan. Improving Language Understanding by Generative Pre- Training. ArXiv e-prints, 2018.
D. J. Stekhoven and P. Buhlmann. MissForest-non non non-paammeric missing value imputation for mixed- type data. Bioinformatics, 28(1):112 8, 2012.
Stef Buuren and Karin Shoothuis-Oudshooe Equations R. Journal of Statistical Software, 45(3):1-67, 2011.
mice: Multivariate Imputation by Chained
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of IEEE Conference Machine Learning (ICML), 2008
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GAIN; Missing Data Imputation using Generative Adversarial Nets. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2019.
Seongwook Yoon and Sanghoon Sull. GAMIN: Generative Adversarial Multiple Imputation Network for Highly Missing Data. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. StyleSwin: Transformer-based GAN for High-resolution Image Generation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards Deeper Understanding of Variational Autoencoding Models. In Proceedings of ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR), 2022.
A EXPERIMENTAL DETAILS
A.1 DATASETS
Table 5 summarizes the characteristics of the datasets used in our experiments.
A.2 PARAMETER SETTING
The default parameter setting of REMASKER is listed in Table 6
11

---

Table 5. Characteristics of the datasets used in the experiments.
Dataset
Dataset Size
Number of Features
Experiment Name
California Housing
20,640
california
Climate Model Simulation Crashes
540
18
climate
Concrete Compressive Strength
1,030
compression
Diabetes
442
diabetes
Estimation of Obesity Levels
2,111
17
obesity
Credit Approval
690
15
credit
Wine Quality
1,599
12
wine
Raisin
900
8
raisin
Spambase
4,601
57
spam
Bike Sharing Demand
8,760
bike
Letter Recognition
20,000
letter
Yacht Hydrodynamies
308
yacht
Table Default parameter setting of REMASKER.
model
parameter
setting
optimizer
Adam
initial learning rate
le-3
LR scheduler
cosine anncaling
global
gradient clipping threshold
5.0
training epochs
600
batch size
masking ratio
0.5
Transformer block
encoder
embedding width
64
number of heads
4
Transformer block
4
decoder
embedding width
64
number of heads
B ADDITIONAL RESULTS
B.1 OVERALL PERFORMANCE
Figure 6 and respectively show the imputation performance of REMASKER and 8 baselines on 12 benchmark datasets under the MAR, MCAR, and MNAR scenarios with the missingness ratio varying from 0.1 to 0.7. Observed that REMASKER performs on par with or outperforms almost all the baselines across a wide range of settings. Note that the MIRACLE imputer does not work on the Compressin i on dataset and the Raisin dataset under some settings, of which the results are not reported. Given that both Compression and Rai sin are relatively small datasets, one possible explanation is that MIRACLE requires a sufficient amount of data to train the model.
Why does REMASKER generalize across the settings of MAR, MCAR, and MNAR? One possible explanation is as follows. Recall that in MCAR, the mask vector of each input is realized following Bernoulli random variable with fixed mean; in MAR, with random subset of features fixed to be observable, the remaining features are masked using a logistic model; in MNAR, the input features of MAR are further masked following a Bernoulli random variable with a fixed mean. Regardless the missingness mechanism, it is rare that the values of one feature are missing across all the records. Thus, by its design, REMASKER is able to learn to re-construct feature X; conditional on other features x; (X1,... X4-1, ...1...... x.) Yet, as reflected in the imputation results, the learning to re-construct performs better under MCAR, in which the missing values are evenly distributed across different features, than MAR MNAR, in which the missing values evenly distributed.
12

---

enbutrrot
06
637
MAR-0.1
MAR-0.3
Figure 6a: Overall performance of REMASKER and 8 baselines 12 benchmark datasets under MAR scenario with 0.1 and 0.3 missingness ratio. The results shown the mean and standard deviation of RMSE, WD, AUROC scores (AUROC only applicable datasets with classification tasks).
B.2 SENSITIVITY ANALYSIS
Figure 9 shows the sensitivity analysis of REMASKER and other 6 baselines on the cal i fornia dataset under the MAR. MCAR, and MNAR settings. The observed trends are generally similar to that in Figure 3 which further demonstrates the observations we made in $ about how different factors may impact REMASKER imputation performance.
B.3 ABLATION STUDY
The ablation study of REMASKER on the california dataset is shown in Table Observed that the performance of REMASKER reaches its peak with encoder depth = 6, decoder depth 4, and embedding width = 32.
13

---

enbutrrot
06
MAR-0.5
MAR-0.7
Figure 6b: Overall performance of REMASKER and baselines 12 benchmark datasets under MAR scenario with 0.5 and 0.7 missingness ratio. The results shown the mean and standard deviation of RMSE, WD, AUROC scores (AUROC only applicable datasets with classification tasks).
B.4 TRAINING REGIME
Figure 10 shows the imputation performance of REMASKER on the ca 1 i fornia a dataset when the training length varies from 100 to 1,600 epochs. Figure II plots the convergence of reconstruction loss in REMASKER, showing trend similar to Figure b).
14

---

0.
8-
MCAR-0.1
AUTOC
MCAR-0.3
Figure 7a: Overall performance of REMASKER and baselines on 12 benchmark datasets under MCAR scenario with 0.1 and 0.3 missingness ratio. The results shown the mean and standard deviation of RMSE, WD, AUROC scores (AUROC only applicable datascts with classification tasks).
Table 7. Ablation study of REMASKER the cal fornia dataset. The default setting follows: encoder
depth 6, decoder depth embedding width 32, masking ratio 50%, and training epochs 600.
depth
RMSE
WD
width
RMSE
WD
depth
RMSE
WD
0.0783
0.0230
16
0.0678
0.0213
2
0.0886
0.0334
0.0663
0.0172
32
0.0663
0.0172
4
0.0738
0.0203
0.0821
0.0225
64
0.0974
0.0322
6
0.0663
0.0172
0.0834
0.0244
128
0.1125
0.0388
8
0.0878
0.0322
10
0.0726
0.0196
256
0.0877
0.0324
10 0.0776
0.0239
(a) Decoder depth
(b) Emhedding width
(c) Encoder depth
15

---

1000
PoOk
Partout
aindoon
0
03
14
Calonie
(c) MCAR-0.5
100
11
AJROC
(d) MCAR-0.7
Figure Overall performance REMASKER and baselines on 12 benchmark datasets under MCAR scenario with 0.5 and 0.7 missingness ratio. The results are shown as the mean and standard deviation of RMSE, WD, and AUROC scores (AUROC is only applicable to datasets with classification tasks).
16

---

1000
par
martout
zindorn
03
14
Calorie
HYM
(a) MNAR-0.1
100
0
13
(b) MNAR-0.3 -0.
Figure Overall performance REMASKER and baselines on benchmark datasets under MNAR with 0.1 and 0.3 missingness ratio. The results are shown as the mean and standard deviation of RMSE, WD, and AUROC scores (AUROC is only applicable to datasets with classification tasks).
17

---

Peloy
Partout
0
03
14
(c) MNAR-0.5
100
13
AJ0OC
(d) MNAR-0.7
Figure 8b: Overall performance REMASKER and baselines on 12 benchmark datasets under MNAR with 0.5 and 0.7 missingness ratio. The results are shown as the mean and standard deviation of RMSE, WD, and AUROC scores (AUROC is only applicable to datasets with classification tasks).
18

---

hyperimpute
gain
0.225
0.200
0.175
0.150
RN5E
0.125
0.100
0.075
0.050
0.050
0.025
0.34
0.30
0.32
0.10
0.08
0.08
0.06
0.04
0.04
0.02
0.02
0.00
1000
7000
13000
20000
01
0.3
0.5
0.7
Desorved dara size
Feature count
Missinness rate
(a) MAR
ReMaster
hyperimpute
missfores
sinchum
+-
0.20
0.38
0.225
0.36
0.200
0.20
0.34
0.15
0.32
0.30
0.100
0.20
0.18
1.0
0.18
0.16
0.14
0.12
0.6
0 0.12
0.08
0.08
0.06
0.06
1000
7000
13000
20000
0.02
0.0
0.1
0.3
0.5
0.
Observed data size
Feature count
Miss Missinnes rate
(b) MNAR
Figure 9a: Sensitivity analysis of REMASKER on the cal 1 fornia datasct under MAR and MNAR scenarios. The results shown in terms of RMSE and WD, with the scores measured with respect (a) the dataset size, (b) the number of features, and (c) the missingness ratio. The default setting is as follows: dataset size 20,000. number features = missingness ratio 0.3.
19

---

typerimpute
+-
masforest
mwae
gahh
0.20
0.250
0.250
0.225
R055
0.175
0.150
0.150
0.12
0.100
0.30
0.075
1000
7000
13000
20000
0.1
0.
0.38
0.14
0.12
0.6
2 0.12
0.10
0.08
0.4
0.08
0.06
0.04
0.02
Observed
Feature
(c) MCAR
Figure 9b: Sensitivity analysis of REMASKER the california dataset under the MCAR setting. The results are shown terms of RMSE and WD, with the scores measured with respect (a) the dataset size, (b) the number of features, and (c) the missingness ratio. The default setting is as follows: dataset size = 20,000, number of fcatures = 9, and missingness ratio = 0.3.
0.0700
0.0675
0.025
RMSE
0.000
0625
0.015
0.030
epach
epach
(a) RMSE
(b) WD
Figure REMASKER performance with respect the number of training epochs on the cal i fornii dataset. under MAR with 0.3 missingness ratio.
0.40
0.30
0.00
Loss
0.25
0.20
0.15
100
200
300
400
500
600
Epoch
Figure Convergence REMASKER'S fitting on ca fornia under MAR with 0.3 missingness ratio.
20


Reviews:
[{"review_id": "vVQ4xsJjEx", "reviewer": "Reviewer_Hron", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the imputation strategy called ReMasker for filling in missing values in tabular data. The idea is based on masking out randomly selected values and training the neural network to reconstruct them. As a neural network, the authors use Transformer, which better reflects the correlation between attributes. The evaluation was performed on typical tabular datasets and compared with SOTA imputation techniques. Additionally, the authors justify that REMASKER learns representations of tabular data, which are insensitive to missing values.", "summary_of_the_review": "To summarize, I like the paper and I think it is a valuable contribution. In particular, the application of the Transformer is a major contribution for me. The authors were able to improve the SOTA in the imputation task. However, I have a feeling the idea of re-masking appeared before. In consequence, this is a new instantiation of the previous idea. This is a borderline paper for me and I am willing to increase my score after the rebuttal.", "strength_and_weaknesses": "The idea of the proposed method is very simple, which is its great advantage. However, I have a feeling that similar ideas (masking out an additional set of values and optimizing the loss for their reconstruction) appeared in previous works, even if the authors did not sell it as a \"new imputation method\". When missing values do not appear in training, then an analogical approach was implemented in ContextEncoder (https://arxiv.org/abs/1604.07379). Since the authors deal with inpainting, the training data does not contain missing values. More importantly, re-masking approach was used as an ingredient in MisConv approach, see page 12 (paragraph \"Training of DMFA on incomplete data\") in https://arxiv.org/pdf/2110.14010.pdf. Instead of reconstruction loss, they apply negative log-likelihood since they deal with probability distributions. I think that there are more techniques with the analogical idea.\n\nNevertheless, I see some important contributions in the paper. This is the first paper where I see the application of the Transformer model in the case of missing data imputation. Even if the general strategy of re-masking was known, the authors show that the application of Transformed allows for obtaining significant improvement over SOTA. The authors could focus more on this aspect than on promoting a re-masking strategy.\n\nI appreciate the very good evaluation presented in the paper, which includes three missing scenarios, and various missing ratios. Moreover, the authors compared their method with a large number of recent imputation techniques showing very good results of ReMasker.\n\nI wonder if the authors verified what happens if we replace the Transformer model with e.g. fully connected neural network. In other words, I would like to know how much we gain by using the general strategy of re-masking with an arbitrary model and how much of the improvement is gained using the Transformer model itself.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oWkw8T6nbyg", "reviewer": "Reviewer_CAJ5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The authors proposed a simple but effective imputation framework for tabular data.\n- The proposed framework additionally introduces the missingness to the incomplete data and using the reconstruction loss on the introduced missing components and unmasked components to train.\n- The proposed method achieves comparable results in comparison to SOTA (HyperImpute).", "summary_of_the_review": "1. Complete data availability\n- Many of the imputation methods do not need complete data for missing data imputation.\n- Therefore, it would be better to tone down this contribution.\n\n2. Re-masking with MCAR setting\n- Authors introduce the re-masking with MCAR setting.\n- In other words, m' is uniformly sampled. \n- In that case, it is unclear how this kind of remasking and reconstructing those remasked components can be generalized to the MAR or MNAR settings.\n- More specifically, the model is learning to estimate the missing components with MCAR settings via remasking and reconstruction. How this learning process can be generalized to impute the missing values in MAR and MNAR settings.\n\n3. SOTA performances?\n- It seems like the performance of the proposed method is comparable with HyperImpute.\n- However, it is hard to say which one is better. \n- In that point of view, it is hard to support the claims in the abstract that \"ReMasker consistently \"outperforms\" SOTA\".\n\n4. Categorical data\n- In tabular data learning, handling categorical data is critical.\n- In this work, how to impute the categorical data? \n- What kind of embedding did the authors use for handling categorical data?\n\n5. Hyper-parameters\n- It seems like some of the hyper-parameters are important.\n- For instance, the missing ratio of remask would be a critical hyper-parameter.\n- Also, the reconstruction loss weights between remask and unmask can be another important hyper-parameter.\n- It would be good to add additional sensitivity analyses on those hyper-parameters.", "strength_and_weaknesses": "Strength:\n- The methodology is simple but the performance is quite impressive.\n- The authors provided extensive experiments to empirically verified the claims. \n- The authors utilize the transformer algorithm for tabular data imputation which seems like a proper model.\n\nWeakness:\n- It is unclear how the MCAR based remasking can be generalized to impute missing components with MAR and MNAR settings.\n- The performances are comparable with SOTA and not consistently better than SOTA. But it seems like the authors claim that it consistently outperforms SOTA.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WW0Y7a4QFU", "reviewer": "Reviewer_socG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose ReMasker, an adaptation of the masked autoencoder (MAE, for vision tasks) for tabular data imputation.  Additionally, ReMasker included transformer blocks to the autoencoder and both the encoder+decoder during training.  Missing at random (MAR) is simulated for a large number (12) of datasets, and the utility of imputation using ReMasker is evaluated for different metrics, as well as in contrast to a large number of SOTA imputation methods.  ReMasker regularly outperforms the other baseline methods.  Additional studies include the effects of the network architecture (i.e., depth and width) on overall performance, the masking ratio hyperparameter, and the combination of ReMasker with the ensemble imputer HyperImpute.", "summary_of_the_review": "Results are impressive and the use of MAE for tabular data imputation is intuitive.  However, the results are not currently reproducible and the paper could clarify more points (detailed throughout the review).  I currently lean towards accept.", "strength_and_weaknesses": "Strengths:\n-ReMasker is a great extension of MAE to tabular data imputation\n\n-ReMasker works well across different downstream performance metrics (i.e., RMSE for regression and AUAOC for classification), outperforming many widely used imputation schemes (e.g., MissForest and MICE).\n\n-The ReMasker architecture is pretty lightweight, making it a reasonable approach to large-scale tabular datasets\n\nWeaknesses:\n-Important model design choices are not explained in any depth.  E.g., \"Unlike conventional MAE, the REMASKER decoder is used in both fitting and imputation phases.\" <- Why?  Without explanations for the differences in models between MAE and ReMasker, an evaluation must be conducted to explain these decisions.\n\n-The exact model details could be better explained.  E.g., \"Note that the encoder is only applied to the observed values: in the\n fitting phase, it operates on the observed values after re-masking\" <- This statement is inconsistent with the paper; based on Figure 1,\n it is operating on missing and masked values, which are not observed.  Please clarify\n\n-More work is needed to show that ReMasker theoretically learns missing-invariant representation of tabular data (in Section 5).  The current section is unconvincing\n\n-Only MAR is considered in the main text.  A discussion of the MCAR and MNAR results included in the appendix would greatly strengthen the paper.\n\n-The ablation study is limited; it is currently an evaluation of network-architecture hyperparameters.  An actual ablation study would consider the role of transformers in ReMasker (vs without in the original MAE architecture) and the exclusion of the decoder during training (as is originally done in the MAE for vision).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O065jk4_ln", "reviewer": "Reviewer_Vv4x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a remask method which masks out some more data under the original missing data and reconstruct them. The main contribution is this remask part.", "summary_of_the_review": "The main reason I choose to reject is that the novelty of the paper is limited. ", "strength_and_weaknesses": "Strength:\n1. The authors use many datasets and include many baselines (12 datasets and make a comparison with 13 imputation methods).\n2. The authors does ablation study and has some discussion trying to explain why their method works.\n\nWeakness:\n1. The method does not seem to be novel. This is the main weakness. The \"remask\" method basically masks out some extra non-missing features and predict them. But this kind of technique is widely used in imputation method in real-world dataset when there is missingness. Researchers usually mask out extra non-missing features to create datasets for training when there is already missingness in real-world data. So I am not sure what extra novelty the paper provides.\n2. The authors only consider synthetic dataset where the authors create missingness from some pre-specified missingness patterns on data where. The authors should consider real-world dataset where there is already missingess. \n3. The discussion on learning representation invariant to missingness pattern is vague. All the methods try to learn representation invariant to missingness patterns. The authors' analysis does not provide extra intuition. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject