Dealing with missing data using attention and latent space regularization
Authors: Jahan Che Penny-Dimri
Christoph Bergmeir
Julian Smith
Venue: Submitted to ICLR 2023 (2023)
Keywords: missing data
missingness
latent space regularization
measure theory

Abstract: Most practical data science problems encounter missing data. A wide variety of solutions exist, each with strengths and weaknesses that depend upon the missingness-generating process. Here we develop a theoretical framework for training and inference using only observed variables enabling modeling of incomplete datasets without imputation. Using an information and measure-theoretic argument we construct models with latent space representations that regularize against the potential bias introduced by missing data. The theoretical properties of this approach are demonstrated empirically using a synthetic dataset. The performance of this approach is tested on 11 benchmarking datasets with missingness and 18 datasets corrupted across three missingness patterns with comparison against a state-of-the-art model and industry-standard imputation. We show that our proposed method outperforms common imputation methods and the current state-of-the-art with statistical significance.


Under review as a conference paper at ICLR 2023
DEALING WITH MISSING DATA USING ATTENTION AND LATENT SPACE REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Most practical data science problems encounter missing data. A wide variety of solutions exist, each with strengths and weaknesses that depend upon the missinggess--eeeeaatie
process. Here we develop a theoretical framework for training and inference using only observed variables enabling modeling of incom- plete datasets without imputation. Using an information and measure-thhored argument we construct models with latent space representations that regularize against the potential bias introduced by missing data. The theoretical properties of this approach are demonstrated empirically using synthetic dataset. The perfor- mance of this approach is tested on 1l benchmarking datasets with missingness and datasets corrupted across three missingness patterns with comparison against state-of- off-the-art art model and staandddd imputation. We show that our pro- posed method overcomes the weaknesses of imputation methods and outperforms the current state-of-the-art with statistical significance.
INTRODUCTION
Missing data is a common problem encountered by the data scientist. The consequences of missing data are often not straightforward and depend on the missingness generating process (Little and Rubin 2002). Choosing the best strategy to deal with missingness is critical when designing statistical or machine learning models that rely on incomplete datasets. frequent complication missingness generating process is often unknown, leading to assumptions about the missingness and potentially the introduction of bias into model training and inference (Davey and Dai 2020). The two most commonly employed strategies for dealing with missing data are to either drop data points where missing data exists or impute the values (Bertsimas et al. 2021). Both strategies can potentially introduce bias into a model if applied incorrectly (Little and Rubin 2002).
that the
The 'impute and regress' strategy has been recently critiqued the setting of machine learning predictors with the finding that an imputation method leads to consistent prediction model if that model can almost surely undo the imputation, which questions the need for sophisticated imputation strategies (Bertsimas et al. 2018, 2021). The best case scenario for imputation method is replace missing unobserved variables with values that do not corrupt the model's predictive distribution from the predictive distribution given only the observed variables. The worst case would be introducing a significant bias during training that increases the divergence of the model's predictive distribution from the true distribution. Indeed, Jeong al. (2022) proved with information-theoreti argument that there is no universally fair imputation method for different downstream learning tasks.
While practitioners continue to develop novel machine learning and statistical methods for imputation, relatively less effort has been made to create framework to reason about models that can fit incomplete data with the notable exception of decision tree models (Gavankar and Sawarkar 2015; Jeong et al. 2022). In this paper, we consider the question of designing model that trains and performs inference on only the observed variables, without imputation or data deletion.
1.1 RELATED WORK
Most recent work has focused on developing novel imputation techniques, such auto-encoders, in order to build better imputations based on the structure of the data (Abiri et al. 2019). consideration of learning and inference without imputation has been considered recently by Bertsimas et al. (2021)

---

Under review as a conference paper at ICLR 2023
who provided theoretical approach to the deficiencies of imputation in predictive modeling, drawing an important distinction between statistical inference and prediction. They build on this approach to develop adaptive hierarchical linear regression model that is capable of performing prediction in the presence of any missingness pattern. Similarly, Jeong et al. (2022) provide a method inference without imputation using decision tree approach, with fairnss--reullarized loss function. Our work differs substantially from these approaches, whereby we show that training and inference using only the observed data feasible using latent space representations and an entropy-based objective which ensures regularization against the potential bias of missingness.
1.2 SUMMARY OF CONTRIBUTIONS
1. Introduce a novel method of dealing with missingness by establishing simple framework for reasoning about model that fits and infers from the observed variables only.
2. Interpret the latent space representations in this framework with measure- and information theoretic argument.
3. Empirically validate the theoretical properties of the latent space on synthetic dataset with
a latent space attention model.
4. Demonstrate the effectiveness of this model on benchmark datasets with corrupted data and real world datasets with missingness.
INTERPRETING MISSINGNESS IN A MEASURABLE SPACE
We first describe a sample space Ω that is defined by an unknown data generating process and unknown missingness generating process. We then define three random variables:
Xa R
Y 9 + IR
Ma : 8 + 10,
Where E {I,..., D3 is the total number possibly observed variables.
We can further specify, X {X....., }D), which is the random variable X Q RD, and M {M,,..., MD}, which is the random variable M : Q + 10, A realization of X is vector representation of possibly observed variables and Y is the outcome variable of interest. M is missingness vector where a value indicates a missing value and an observed value.
If we consider the power set U1 P(X), then of missingness results a smaller o-algebra such that U = i Aj E A & j}. where define a probability space (X,U, where measure maps each
define measurable space (X,4). The impact
M = P(M). Finally, combination of possible variables in X, defined in U, to a probability whereby μ + [0, 1]. where U.
An implication of our definition of the measure, μ4, is that where no missingness exists, each subset is equally likely, and the distribution is uniform. In the presence of missingness generating process, some subsets are more likely than others and the distribution diverges from uniform.
How the probability distribution diverges from uniform can be considered with reference to the definitions of missingness originally described by Little and Rubin (2002). If Xobs = {x x E X m M A m 0} and Xmis {x : x E X A m E M m = 1} and we consider M to be defined by some unknown parameter 8, with a conditional distribution fMM]X, B3), then we can define three types of missingness:
Definition If [[M]X, 9)) = S(M13) then M. X|3 and data is considered to be missing com- pletely at random (MCAR)
Definition 2. If (XM () f[M[ B) then M 1 Xmis -obbe 3 and data is considered to be missing at random (MAR)
Definition 3. If (M)X,8) 3) + f(M(3) and X 3) + (M Xoter 3). then data is considered to be missing not random (MNAR)
2

---

Under review as a conference paper at ICLR 2023
We can extend these definitions define conditional versions of l with Maix and Medx If (UU) Puix (U)X) then missing data can be considered MCAR. If PUj (U = JUIX (U) (Xbs)s) then missing data can be considered MAR. Finally, if p(U) # PUIx(UIX) & ( (U|Xobe) then data MNAR.
From this initial description of missingness, we can consider a naïve first attempt at predicting Y from X in the presence of missingness.
USING AN ENSEMBLE TO LEARN THE POWER SET OF FEATURES
As starting point to developing proposed method, first consider simple ensemble approach. We construct a hypothetical ensemble of K models, parameterized by 0g E Q, which performs the mapping for each combination of variables our outcome space, fo, Uk Y, where Uk U, E (1....} and = [U/. If the input vector I ~ contains missing values then only the complete subsets in U with no missing values used to update the models. To perform inference, we could simply choose the output of the function with the highest cardinality set. This strategy presupposes that all input data contains information related to the outcome, and feature selection has already occurred prior to model creation. In this setting, this model definition allows us to make a prediction, ÿ. given any subset of available variables from X.
This approach is only valid for datasets with missingness defined as MCAR as there are two implicit strategies occurring. The first is simply excluding the variable with missing data from our model by defining our subsets in U which exclude that variable. This is always valid strategy as it carries no risk introducing bias into the remaining data, at the cost of losing information associated with that variable. The second implicit strategy is that in subsets that include variables with missing values, we remove all items in that subset that have a missing value. This is akin to dropping rows in a tabular dataset. By dropping missing data that is either MAR MNAR. bias is introduced into the remaining data and the resultant model that is fitted to that data will incorporate that bias. In the "ensemble of K" models deseribed above, models fitted subsets with missing data will incorporate the biases of a dropping data strategy.
4 INCORPORATING MISSINGNESS IN A LATENT SPACE TO OVERCOME BIAS
In order to overcome the deficiencies of the previous formulation we could try to introduce latent space representation for U in our model and attempt to incorporate missingness into this latent space. We redefine fe, to map each input vector U to a latent space Z. fer Up + Z. which is shared across all possible subsets and define second model, parameterized by & that maps Z to our output space, Jo: : + Y. We have now defined an "ensemble of compositions", G o fo.-
Previous work by Boudiaf et al. (2020) has derived the result that when latent space model is fit with cross-entropy objective, H(Y; Y|1), this is equivalent to maximizing the mutual information ((2;Y).
Using this result, we can reason about the representations of subsets in the latent space. Consider two subsets U2, U, U, which have identical variables, but one has an additional variable |0, - U/y 1. If both U, and U3 contain the same amount of information with respect to the outcome, then they will be clustered to the same location latent space.
To formalize this clustering effect in terms of measurable space, we can define a measure space for our dataset (D, I),, such that D {(z. y) z E y Y), = ((u,y) yt : u E U y E Y}. and I is measure based on mutual information defined as I = T(S(U)): Y), where f is general function that maps a subset, Ux E U to the latent space Z.
Mutual information is a correct basis for measure in this measurable space as it satisfies the properties of non non-tegativ,,y, mapping to zero for the empty set, and countable additivity across the subsets defined in U. The property of countable additivity relies the independence of their latent space representations of Us. This independence arises in two ways. Firstly, although we do not guarantee X; T Xy for X,, X, X there is independence across subsets of X based on the construction of U, such that U, 1 U, for U,. U, E U. The independence of the subsets Un guarantees the independence of their latent space representations. Secondly, we can notice that I is
3

---

Under review as a conference paper at ICLR 2023
really measuring the mutual information of Z Ug, and in the conditional universe of the latent space Z 1 Z
Using our measure, I, and model definition we can formally define the effect of the mutual information on the latent space.
Proposition 1. I Y) where i.j (,,.., K} and IU, Uz| = 1, then f(U,) = f(U,) - 2i0; = 2
We can broaden this equality with a geometric interpretation.
Proposition - -IUU;;Y) x (. where i.j {1. K], IU; U,}
and d is distance metric in the latent space, such as the Euclidean distance.
The effect of Proposition is that the latent space representations for each UR in our "ensemble of compositions" cluster according to the mutual information. Additionally, there is a guarantee that the model learns representations for high cardinality subsets that are regularized to the highest information but lowest cardinality subset. This obviates the need for explicit feature selection, as non-informative features are not represented in the latent space.
Remark 1. If Y is a then in the extreme case where no input features carry information with re- spect to our outcome all subsets will cluster to the latent space representation of the empty set, which mapped to the point of highest entropy in
variable parameterized by the output of Y
Bern(g(fx(U
In order to understand how the latent space representation can help with missingness, we need only consider what happens to I = I(J(U));; Y) as missingness affects Ug. If an unknown missingness generating process affects Uk. then I [(f(Uk)); Y) will decrease. This will be true whether the missingness is MCAR, MAR or MNAR. This reduction is directly represented in the latent space. When missingness present in Ug. the representation in Z is then regularized to the lowest cardinality and highest information subset.
Finally, it is interesting to note that there are many cases where missingness generating processes of the previously described patterns may also provide information related to the outcome (Li et al. 2018). This is commonly encountered, for example, in healthcare datasets where healthy people often have missing data because there was no indication to perform test. We can define this property as arising when
I( (MMa; Y)= > O. In our latent space model, I = (ff())); for a subset Us which is affected by informative missingness may actually increase.
OUR APPROACH: USING ATTENTION TO MODEL THE LATENT SPACE
There is an obvious limitation to the ensemble approach, which is poor computational scaling the number of models in the ensemble, K, increases exponentially with the dimensions of the dataset. Rather than defining a set of functions fe, for each k I,... K, we can define a general function fe. which can map each UR to the latent space, as we did in Propositions I and 2 A natural choice for such a model that can take heterogeneous length inputs is to adapt an attention based model normally used to solve sequence tasks.
Scaled dot product attention has been used to construct state-of-ther tfrte-r sequence models such as the Transformer (Vaswani et al. 2017). The Transformer maps an input sequence (#1...., .Ta) to latent space representation (z,...., 2g), from which decoder outputs a sequence of (y).... Vm). Here each x, y, E Igembed, which is the embedding dimensionality. The similarity to the aforementioned "ensemble of compositions" can begin to be appreciated at this point. To approximate the ensemble with a Transformer architecture embedding network that maps our input vector to some higher dimensional embedding space. fa R - IR embed Then we apply Transformer-style model where the output is instead = eRemmedd which is then mapped to the output space, 9s eRemeeed + ouu In this model, the output of the Transformer model z E Rembee is analogous to the latent space in the previously compositional model and the function 96 is serving the same purpose in both models.
must add two components. Firstly we add feature specific
is important to note that while the compositional ensemble explicitly trains a separate model for each subset, the Transformer must train randomly generated subsets each training step. This
4

---

Under review as a conference paper at ICLR 2023
can be achieved using dropout to stochastically create mask for the imput features applied each attention module. Concrete dropout has been shown to enable feature ranking and applying at the feature level subset sampling process means that more informative features included in the subsets more frequently (Chang et al. 2018). This choice of model design helps further regularize the output of the model toward the lowest cardinality, highest information subset.
Finally, we introduce our method as latent space attention model (LSAM), which can deal with missingness 'out-of-the-box'. In summary, this approach uses neural network embed each feature, then maps the set of available embeddings to a latent space using a Transformer-style network. It the latent space representation which regularized against missingness space representation is then mapped to an outcome space using a neural network. The pseudocode for the training procedure is demonstrated in Algorithm and training details can be found in the supplementary materials.
trains. The latent
Latent space attention model training procedure. A stochastically generated mask, drawn from a concrete distribution at each training step, ensures the model learns from different subsets of variables.
Algorithm I:
Initialize network parameters P. 1/, 0, for epoch in Epochs do for x, y in batched(X X Buxd Y E [xxoo do
mask ~ Concrete(p); / Sampling feature mask for subsetting
a(bb,));
/* Embedding network fa Rd + Rdx Remebedd
t(0, e, e,mask):
Attention network to Rdxembed - R embed
ÿ g(d,2) 2):
/x Output network do Rembed + Rout
g - V P.v.o.olly. 9)
/* Gradient of cross aross-entropy loss
P. t, 0,0 t update(g. p. p,.,0,00)
end
end
6 EXPERIMENTS
Models are built using JAX and trained using adaptive stochastic gradient optimization with early stopping (Kingma and Ba 2017; Bradbury et al. 2018; Zhuang et al. 2020).
Descriptions of hyperparameter optimization and model training can be found in the supplementary materials. Code for the model and experiments is available at httpp://amomymmouu..oooooo.ccccc Bussingnes-AAF4//EEAAA
6.1 MODEL CHARACTERISTICS
We first explore the properties of our proposed model with regard to the previously defined latent space using toy dataset. Our aim is to show the effect of both the mutual information of predictor on the latent space representations, and the effect of missingness on these representations.
6.1.1 LATENT SPACE REPRESENTATION
In order to explore the representation of input variables in the latent space, we use the classic 2- dimensional synthetic spiral dataset. This is a non-linear binary classification problem with two input variables, æ1 and x2, representing the and y axes respectively. The dimensional dataset augmented with 2 additional variables. The first, z3, is random noise from standard Gaussian carrying no information of the outcome, x3 (O, 1). The second, æ4, is the outcome variable corrupted with random noise from uniform distribution and therefore carrying some information about the outcome, z4 + e, c ~ Mj..1--
Both the LSAM and compositional ensemble model are trained the 4-dimensional dataset. Out- come measures are then bootstrapped with experiments repeated 30 times. We can obtain a statistical measure of distance between groups reported using a Student t-test and measure the p-vuuee.
5

---

Under review as a conference paper at ICLR 2023
LSAM
Ensemble
0
p-value
0
p-value
(x1)
3.69
3.95
10-10 10
0.29
4.43
10-5
{x2)
4.18
2.16 x 10
0.38
3.40 x 10
{x1, x2
5.10
4.13 x 10-14
2.74
6.78 x 10-6
X3 (noise) }
3.44
1.14 x 10-8
0.04
1.04 x 10
[x4 (signal))
5.15
3.77 x 10
4.37
6.08 x 10
Table 1: Euclidean distance between latent space representations of and the empty set for the LSAM and Ensemble based models. The p-value represents the distributional difference. The difference greatest where the signal is strongest, and least where there is only random noise.
We first explore the Euclidean distance for various subsets compared to the empty set. From Propositions and we would predict that the distance shorter when the subset is only random noise, 23, compared to carrying some information about the outcome. Table shows that this is indeed the case, with the distance between the empty set and noise being the smallest value for both the ensemble and the LSAM. The statistical difference between the bootstrapped distributions is much less for the con-informativ noise compared to subsets carrying information.
We can then explore the Euclidean distance between Uk, where u4 S {x1, 222), and uy z, where x C. {a3, 24). From Propositions and [ we predict that the distance is shorter when æ random noise, in 2=3, compared earrying information about the outcome, in x4, This is confirmed in Table where we see that adding noise changes the location the mapping in latent space less than adding informative variable across both models.
LSAM
Ensemble
+ [noise}
+[signal}
v-luuee
+(noise)
+Signal}
p-value
{x1}
0.43
4.16
3.09 x 10-12
0.16
2.41
8.24 10-5
[x2]
0.37
3.18
1.02 x 10-11
0.14
3.92
1.80
10 10-4
[x1, x2}
0.21
1.09
3.77 x 10-7
2.63
3.79
1.43 x 10-2
0
3.44
5.15
2.02 10-7
0.04
4.37
6.28
10-5
Table Euclidean distance between latent space representations up and us + either a noise variable or an informative variable for both LSAM or Ensemble based model. The location in latent space moves the most when an informative variable is added to a subset.
6.1.2 MISSINGNESS REGULARIZATION
We now explore the effect of the level of missingness on the latent space representation. In order do this we measure the Euclidean distance between Ug, where Uy C (x1,x2), and us {z4]. where the degree of missingness is increased in z4 from 0% of the data to 99% of the data. From Proposition 2 we would predict that as missingness in 24 increases, the distance between us and ua {x4} decreases. Table 3 empirically demonstrates this effect where the location in latent space converges to approximately the same location by 99% missingness.
6.3.3 MISSINGNESS, FEATURE IMPORTANCE AND CONCRETE DROPOUT
Finally, we explore the effect missingness on the concrete distribution learned during training. Specifically, we can look at how the probability of dropping I4 from subsets during training changes as the level of missingness increases from 0% to 99% of the variable. Table 4 shows that the probability of dropout increases toward 0.5, the point of maximum entropy, missingness increases to 99%. The implication of this finding is that the model assigns more importance to variables with less missing data, even when the variable with missing data is an informative variable.

---

Under review as a conference paper at ICLR 2023
uk
0%
20%
40%
60%
80%
99%
{x1}
7.84
5.93
4.62
2.88
1.48
0.07
[x2]
6.11
4.42
3.40
2.24
1.05
0.05
[xl, x2]
2.07
0.79
0.57
0.23
0.13
0.01
0
9.13
6.88
5.42
3.45
1.77
0.09
Table 3: Euclidean distance between latent space representations of un and uy informative variable (r-4) for varying levels of missingness in the informative variable (z4). The location of the subset in latent space moves less when the missingness the added informative variahle increases.
Variable
0%
20%
40%
60%
80%
99%
x1
0.34
0.33
0.32
0.33
0.33
0.34
x2
0.32
0.32
0.32
0.32
0.31
0.33
x3 (noise)
0.50
0.49
0.49
0.49
0.48
0.49
x4 (signal)
0.40
0.38
0.37
0.38
0.40
0.46
Table 4: Learned probabilities, from conerete distribution, of dropping variables during training across different levels of missingness in the informative variable (x44). The probability of dropping 24 from training approaches that of random noise (z3) as the missingness increases in that variable.
6.2 BENCHMARKING PERFORMANCE
The OpenML-CC18 classification benchmarking suite is used to assess performance in two settings (Vanschoren al. 2013).
6.2.1 DATASET SELECTION
In the first set of experiments, only select datasets with complete data availability, can controllably corrupt them with specific missingness patterns. In the second set of experiments, we select datasets with missingness, however the missingness pattern is unknown. For all experiments, we select tabular datasets with a maximum number of 250 features and rows between 1000 and 10000 for computability. For datasets with pre-existing missingness we included datasets with greater than or equal to 5% of the data missing. Our final collection includes 18 complete datasets and 11 datasets with missingness. Details on the methods used to corrupt the datasets with missingness are available in the supplementary materials.
6.2.2 COMPARATOR MODELS
In all experiments, we compare 'out-of-the-box" performance with missingness against industry standard data processing pipelines with imputation. We compare LSAM approach against high performing ensemble decision tree model, LightGBM, which is also capable of dealing with missing data natively (Ke al. 2017).
Imputation can be broadly grouped into simple imputation strategies or regression methods. Simple imputation can be the replacement of missing value by a statistic, pre-specified value, encoding the missingness as new variable. Examples include mean or mode imputation. This strategy can lead to consistent predictions in the setting of MCAR, but is not appropriate where data MAR or MNAR (Bertsimas et al. 2021). Despite the strong assumption needed, the benefit to this approach is computational efficiency and ease of implementation.
Regression strategies are often combined with multiple imputation techniques, which involves regressing multiple times with many models in order to generate a sample distribution imputed values (Rubin 1987; Buuren and Groothuis -otthioooOOdooom 201 Multiple imputation can be performed with linear regression. decision tree, or even random forest models and can provide unbiased estimates in the setting of data MCAR and MAR (Stekhoven and Buhlmann 2011) 1). Although this approach can require significant computational resources, especially in the setting of large datasets, has become the gold standard for most modeling tasks that require imputation. In experiments, compare
7

---

Under review as a conference paper at ICLR 2023
CD
8
5
2
LightGBM-Simple LightGBM-Miceffnn
LSAM
L.AM-Iterative
LightGHM
LSAM-Miceforest
LSAMeSimple
(a) Performance when data is MCAR
CD
8
7
5
2
LightGBM ISAM Simple
LightGBM Iterative
1SAM MicMMMiceforet
LightGBM Simple
MigheeeM-Miceforest
(b) Performance when data is MAR
8
7
5
2
LSAM-Simple LightGBM-Simph LSAM-Miceforest LightGBM Micefforestt
LSAM
LightGBM
BightGBM-Iterative
(c) Performance when data is MNAR
Figure I: Critical difference diagrams comparing performance for different missingness regimes, demonstrating improved performance for the LSAM without imputation. Points are labelled by the type of model as well as the imputation strategy if used. The performance metric is the change in negative log-likelihood from baseline performance with complete data. Further right in the diagram indicates better performance. A break in the solid bar underneath demonstrates statistical significance.
native model performance against 3 imputation strategies: simple imputation with mode or mean imputation based on data type, an iterative multivariate imputation strategy, and multiple imputation with random forests (MiceForest) (Pedregosa et al. 2011; Stekhoven and Buhlmann 2011) 1).
6.2.3 PERFORMANCE ON COMPLETE AND CORRUPTED REAL-WORLD DATASETS
We first report the performance on the baseline complete datasets. For the metric of accuracy, the L.SA performs better with 13/18 wins. For the metric of negative log-likelihood (NLL) the LSAM better with 12/18 wins.
To statistically compare the performance of all possible approaches we use critical difference di- agrams with Nemenyi two-tailed test (Demšar 2006). Figure shows these plots in the context of the negative log-likelihook and demonstrates that the LSAM, using out-of-the- performance, outperforms performs well as 'impute and regress' strategy when data is MCAR MNAR. Additionally, the LSAM outperforms LightGBM in most settings, with the exception data MAR. Critical difference diagrams with the metric of accuracy in the supplementary materials, and demonstrate similar findings.
6.2.4 PERFORMANCE ON REAL- WORLD DATASETS WITH MISSINGNESS
Again, we compare the results of all possible approaches using a critical difference diagram. Figure 2 shows these plots for both metrics of accuracy and negative log-likelihood. Performance measured
8

---

Under review as a conference paper at ICLR 2023
by the negative log-lik ligelike showed higher performance of the LSAM using out-of-the wh--th-teboox box performance. Performance as measured by the accuracy showed higher performance of LightGBM models. These results indicate that the LSAM model is better calibrated at the expense of accuracy.
7
5
3
1
LightGBM-micefrert LSAM-miceforest
LSAM LSAM -iteiiitee
LightGBM
LSAM-simple
LightGM--ieertive
LightGBM-simple
Figure 2: Critical difference diagram comparing performance with negative log-likelihood on datasets with unknown missingness pattern, demonstrating improved performance of the I.SAM model.
6.2.5 META-LEARNING
Finally, explore the effect of dataset characteristics when LSAM outperforms the comparators, we train a random forest model to predict an LSAM win from the known features of each dataset. Full experiment details and results are available in the supplementary materials. Datasets with higher dimensionality, a greater number of rows, and greater proportion of numeric variables tended to favor LSAM.
CONCLUSION
To the best of our knowledge, our work the first description and theoretical justification for utilizing an attention based latent space model for dealing with missingness in tabular datasets. We have shown that regularization occurs in the latent space both theoretically and empirically using a LSAM and compositional ensemble architecture. The regularization acts cluster representations in the latent space toward the lowest cardinality and highest mutually informative subset of variables. We have shown this approach to outperform imputation and LightGBM.
Our results suggest that when data is MCAR and MAR, there is less of a clear advantage of avoiding imputation. Importantly, when data is MNAR, experiments suggest potential advantage of out- of-the-box methods to avoid the bias imputation methods in this setting. This finding predicted from our theoretical derivation of latent space regularization. Performance mixed, real-world missingness patterns corroborates the finding that latent space regularization can avoid the bias in imputation methods. Interestingly, LightGBM also outcompetes imputation when data is MNAR which likely arises from gradient based decision trees allocating split with 2017). We suggest that as data
unknown, and likely
missingness the direction that minimizes the loss (Ke MNAR contains some information about the underlying value it is likely that splitting based on loss minimization allows LightGBM to incorporate information from data MNAR.
A limitation of our approach is that it bundles missingness with model design. Certain predictive tasks with known missingness that MAR or MCAR, specific model requirements, would likely benefit from an impute and regress strategy instead of our proposed method. Additionally, while we have introduced latent space regularization to compositional ensemble and iitention-based models, there is no limit to possible architectures that could achieve the same effect. Future work should further validate this latent space regularization with other model designs and explore the application of this approach in other real world dataset domains, such as imaging or time-series data.
We conclude that for predictive tasks, attention based latent space models with concrete dropout may be principled model choice in the context of missing data with unknown missingness pattern. They require no assumptions about the missingness pattern of the data and require minimal extra computational cost compared similarly parameterized model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abiri N. Linse B, Edén P. Ohlsson M. Establishing strong imputation performance of a denoising
autoencoder in a wide range of missing data problems. Neurocomputing [Internet]. 2019
Nov:365:137746 Available from: http//xx.ooi.org/... www..rrg/.....o/.....ooo 106../0......ommm22mmm wtt.u/co...09907..0 2019.07.065
Bertsimas D. Delarue A, Pauphilet J. Prediction with missing data [Internet]. 2021. Available from:
https//aarxv.oor/bb221100033313
Bertsimas D. Pawlowske C, Zhuo YD. From predictive methods to missing data imputation: An optimization approach. Journal of Machine Learning Research [Internet].
Available from:
Boudiaf M, Rony J. Ziko IM, Granger E, Pedersoli M, Piantanida P. et al. A unifying mutual information view of metric learning: Cross-entropy vs. Pairwise losses. In: Computer vision ECCV 2020 [Internet]. Springer International Publishing; 2020. p. 548 64. Available from: https://doi.org 10. 10.100//77---0000---1 58539-653393
Bradbury J, Frostig R, Hawkins P. Johnson MJ, Leary C, Maclaurin D, al. JAX: Composable
transformations of Python+NummPy programs [Internet]. 2018. Available from: http://github.com/ google/jax
Buuren S van, Ctoothuis-Oudssoor K. Mice: Multivariate imputation by chained equations in
r. Journal of Statistical Software, Articles [Internet]. 201 2011:45(3):1-67, Available from: https
Chang C-H, Rampasek L., Goldenberg A. Dropout feature ranking for deep learning models [Internet].
2018. Available from: https://arxiv.or//bbb////// /tt.b////11..0455555
Davey A, Dai T. A systematic approach to identify and evaluate missing data patterns and mechanisms multivariate educational, social, and behavioral research [Internet]. 2020. Available from:
Demšar J. Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res. 2006
Dec;7:1-30.
Gavankar S. Sawarkar S. Decision tree: Review of techniques for missing values at training, testing
and compatibility In: 2015 3rd international conference on artificial intelligence, modelling and simulation (AIMS). 2015. p. 122-6.
Jeong H, Wang H, Calmon FP. Fairness without imputation: A decision tree approach for fair prediction with missing values. Proceedings of the AAAI Conference on Artificial Intelligence [Intemett] 2022 Jun:36(9):9558 13558666.66 Available from: https://dii.or// I0. 1609/aaii. v36i. 2221188 Ke G. Meng Q, Finley T, Wang T, Chen W, Ma W, et al. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems. 2017:30:3146-8
Kingma DP, Ba J. Adam: A method for stochastic optimization [Internet]. 2017. Available from:
https://arxiv.org/ wwt..orgg/////////a//// 1412.6980
Li Wang M, Steinbach MS, Kumar V. Simon GJ. Don't do imputation: Dealing with informative
missing values EHR data analysis, In: Soon OY, Chen H. Wu X. Aggarwal C, editors.
Proceedings 9th IEEE international conference on big knowledge, ICBK 2018. Institute of
Electrical; Electronics Engineers Inc.; 2018. p. 415-22. (Proceedings 9th IEEE international conference on big knowledge, ICBK 2018).
Little RJA, Rubin DB. Statistical analysis with missing data [Internet]. Wiley; 2002. (Wiley series in probability and mathematical statistics. Probability and mathematical statistics). Available from:
http://booksgoooge htwp:k/...oooll...oooooosossooss hromm/books/ddzz haam://ooossdaaPPPaAAAAAAMA YPWAAAAMAAJ
Pedregosa F, Varoquaux G, Gramfort A, Michel V. Thirion B, Grisel O, et al. Scikit-learn Machine learning in Python. Journal of Machine Learning Research. 201 1;12:2825530.
Rubin DB. Multiple imputation for nonresponse in surveys. Wiley; 1987. p. 258.
Stekhoven DJ, Buhlmann P. MissfForest-oon-paaamttii missing value imputation for mixed-type
data. Bioinformatics [Internet]. 2011 (1):28(1):112-8 Available from: http://dx.doi.org/ 1093/
bioinformatics/br5559
Vanschoren Rijn JN van, Bischl B, Torgo L. OpenML. Networked science in machine learning. SIGKDD Ex Exploatiions [Internet]. 2013:15(2):4966)) 60. Available from: http:://oi..ccmorgg 10. 145/
1641190.200 22.22644119
Vaswani A, Shazeer N. Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In:
Guyon I, Luxburg UV, Bengio S, Wallach H. Fergus R, Vishwanathan S, et al., editors. Advances in neural information processing systems [Intemet]] Curran Associates, Inc.; 2017. Available 1c4a845aa S44444 Paper
from:
pdf
10

---

Under review as a conference paper at ICLR 2023
Zhuang J, Tang T, Ding Y. Tatikonda S, Dvorek NC, Papademetris X, et al. AdaBelief optimizer::
Adapting stepsizes by the belief in observed gradients. CoRR [Internet]. 2020;abx/2010.07466. Available from: https//axiioorr/as200007778


Reviews:
[{"review_id": "WTsiGrwmCkG", "reviewer": "Reviewer_4QuC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the supervised learning problem under missing data, and proposed a solution without the need for imputation based by utilizing an attention based latent space model for dealing with missingness in tabular datasets. The authors shows that certain regularization occurs for the latent space representation of missing data with high cardinality subsets that are regularized to the higher information. Based on this insight, this paper then proposed a training algorithm based on transformers. ", "summary_of_the_review": "I believe this paper contains some interesting ideas that could be further explored. However, given its current state (misleading factual mistakes, lack of novelty) I am not convinced to recommend acceptance to the main conference.", "strength_and_weaknesses": "Pros:\n\n- This research follows the recent findings of supervised learning under missing data, and takes an approach that does not reply on imputation models\n- I found that the information regularization perspective of latent space of missing data very interesting. I think additional works can be done based on this idea., especially on the theoretical side.\n\nCons:\n\n- Clarity could be further improved. I understand that this paper focuses on supervised learning problem under missing data (i.e., prediction) as opposed to inference (i.e., multiple imputation) problems. However, this is not absolutely clear by just reading the title and abstract. The term \"dealing with missing data\" is too broad and confusing; and the presentation of the paper could further benefit from a more pronounced narrative. \n\n- The conclusions/implications of the cited literature are not accurately presented. For example, regarding (Bertsimas et al. 2021), the authors claimed that \"This strategy (simple mean imputation and then regress) can lead to consistent predictions in the setting of MCAR, but is not appropriate where data is MAR or MNAR\" . This is very incorrect, as (Bertsimas et al. 2021) clearly proved and stated that \"mean-imputation-then-regress is asymptotically consistent\", and \"the missingness mechanism (MAR or NMAR) does not impact asymptotic Bayes optimality\". Although the authors could mention the limitation of such approach under finite data regime (which needs to support by empirical evaluations as well, which is lacking at the moment), the statement regarding consistency is clearly wrong. \n\n- Lack of novelty and a lot of similar recent advantages in literature are ignored. For example, apart from impute then regress, there are also approaches that adopt the joint generative-discriminative approaches based on latent representations, such as Ipsen et al, 2022, Ma et al., 2019,  Ipsen et a., 2021 just to name a very few, which naturally satisfies latent regularization effect that this paper proposed. These approaches are also based on the so-called set function encodings of latent space that maps $U_k$ into $z$, which is quite comparable to the proposed transformer approach. The authors claimed that \"our work is the first description and theoretical justification for utilizing an attention based latent space model for dealing with missingness in tabular datasets\", which is not entirely true. For example, in Lewis et al 2021, transformers are already used to encode $U_k$ into $z$, and the learnt $z$ can be used for any downstream tasks such as prediction, imputation and acquisition, and can be trivially extended to fully supervised setting of this paper, as argued by Ipsen et al, 2022.  \n\n- Theoretical analysis. I appreciate that the authors provided analysis on the regularization/clustering effect of latent space, however this analysis is a little pale to justify the proposed algorithm, especially considering the highly related works mentioned above. This also make the connections between Section 4 and 5 a bit far-fetched. Further theoretical analysis on consistency and identification will be appreciated.\n\nOther minor issues & questions.\n\n- In algorithm 1, it appears that the model training requires access to fully observed (at least MCAR) data, which seems quite limited. Could the authors clarify?\n\n\n\nReferences\n\n- Josse, Julie, et al. \"On the consistency of supervised learning with missing values.\" arXiv preprint arXiv:1902.06931 (2019).\n- Ipsen, Niels, Pierre-Alexandre Mattei, and Jes Frellsen. \"How to deal with missing data in supervised deep learning?.\" ICLR 2022\n- Ma, Chao, et al. \"Eddi: Efficient dynamic discovery of high-value information with partial vae.\" arXiv preprint arXiv:1809.11142 (2018).\n- Ipsen, Niels Bruun, Pierre-Alexandre Mattei, and Jes Frellsen. \"not-MIWAE: Deep generative modelling with missing not at random data.\" arXiv preprint arXiv:2006.12871 (2020).\n- Lewis, Sarah, et al. \"Accurate Imputation and Efficient Data Acquisitionwith Transformer-based VAEs.\" NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UXPtqF8Qsq", "reviewer": "Reviewer_cGGK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers introduces a framework for reasoning about model fitting and inference from observed variables when dealing with missing data. The main point seems to be the interpretation of the latent space representation using the mutual information between subsets which contain different elements of the dataset. A consequence of this interpretation is to intervene in the latent space using an attention-based representation. This makes the method to perform better than one baseline based on ensemble decision trees (LightGBM).", "summary_of_the_review": "Good direction with interesting connections between missing data models and measure theory. In particular the use of the mutual information to identify potential effects in the latent space. However, there is a general noise around the contributions that make them less clear and difficult to assess. I recommend weak rejection, but I am open to discuss/understand better what are the main contributions, the differences wrt SOTA methods and related work.", "strength_and_weaknesses": "**Strengths:** The technical formulation of the problem as a measure-theory alike method is a valid point. I also liked the perspective where one identifies the bias induced by the missing samples in the latent space. There is also a reasonable amount of empirical results, however, the number of baselines considered is somehow small in my opinion.\n\n**Weaknesses:** In general, I had a difficult time to identify the main contributions of the work, or at least to see what are the main differences with respect to previous ideas in the literature. The idea of fitting models that only use the information of observed variables under missing samples seems to not be correctly introduced wrt other previous works (i.e. MiWAE, PartialVAE or Eddi), or at least indicating what are the differences and what is new here. Additionally, the introduction of the measure theory formulation does not seem to provide a key advantage in the analysis, as in the end the authors use the mutual information between subsets for later using the attention method. From the theoretical aspect of missing data, I also feel that Le Morvan, 2021 should be cited as it is stated in the paper that there are no universally fair imputation methods (which somehow prevents the work from focusing in imputation). On the algorithmic part, it was also difficult to me to perceive what is special or at least, to understand the connection with the theoretical development. \n\nFinally, I liked the design of the synthetic experiment, but the presentation of the results is not clear or sufficiently rigorous. I really would have preferred to see the metrics tables in the main manuscript, rather than all of them in the appendix. Details about the datasets used for benchmarks, number of samples and rates of missing data is also not clearly available.\n\n\nMiWAE -- https://arxiv.org/pdf/1812.02633.pdf\n\nPartialVAE -- http://bayesiandeeplearning.org/2018/papers/75.pdf \n\nEddi -- https://arxiv.org/pdf/1809.11142.pdf\n\nLe Morvan, 2021 -- https://arxiv.org/pdf/2106.00311.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BiJ08DizM7", "reviewer": "Reviewer_Xkfr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of training models capable of making predictions while there are missing values in input data.\nThe authors proposed a method that uses the attention mechanism as it can take an arbitrary number of features. \nExperiments are conducted to investigate the learnt latent feature and the performance of the methods on various datasets. ", "summary_of_the_review": "Although the authors provide a working solution to the prediction problem with missing data, this paper doesn't give enough insight into the method. It cannot justify the method from existing approaches. So the paper is not considered as publishable as the readers might feel as unconvinced as the reviewer.\n", "strength_and_weaknesses": "(+) Making predictions with missing data is quite a challenging and undouble overlooked problem. This paper provides a working solution to address the problem with the attention mechanism.\n\n(-) Regarding the missing data problem, this paper primarily focuses on the proposed method's framework. A systematic view of related work (e.g. density-based, vae-based, etc.) is not present in the discussion or the experiments. (See details below).\n\n(-) While the proposed method acts as a working solution, this paper doesn't provide enough insights or justification of the method to either the problem or other candidate methods. The readers might not be convinced if the technique is the right choice (See details below). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7a9z0lJhZla", "reviewer": "Reviewer_ezY7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission studies an architecture to learn from data with missing values. The architecture is based on chaining two transformations with an attention module that enables modeling the set of observed values. The model learns by randomly masking the training data to ensure learning from different subsets of the variables.\n\nThe authors introduce information-theoretical considerations to justify the approach, reasonning on properties of embeddings in the presence of missing values.\n\nThe authors then empirically demonstrates their theoretical considerations and validate their approach on many datasets from openML with MCAR, MAR and MNAR simulated missingness by comparing the prediction accuracy to LightGBM.\n", "summary_of_the_review": "The ideas of the papers are interesting, but they are not very well formulated, established, and positioned.", "strength_and_weaknesses": "\nModeling prediction in the presence of missing values, with suitable architectures is an important problem.\n\nI feel that there are some implicit, or poorly-stated, assumptions behind the theoretical claims. For instance, it seems that I can create counter examples to prop 2: for a 5-dimensional feature space (X0, X1, X2, X3, X4), using Y = 1 is X0 is missing and 0 if not and f(U) = sum_U. I have then a non-zero mutual information I(U_i, Y) each time U_i contains X0, and elsewhere a zero mutual information, but this is quite independent of the value of f(U).\n\nReading the code source, I see 18 hyper-parameters for LSAM: regularization, optimization, and architectural hyperparameter (for f and g). How were they made for each experiment? In particular: what was the train-test-validation split strategy?\n\nThe fact that LSAM+iterative works as well as LSAM seems to me a bit contradictory with the spirit of LSAM, if I have understood things well. Indeed, I thought that the spirit of LSAM what to optimize jointly the representation of the missing values and the subsequent predictor.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4ni9iJyUqP", "reviewer": "Reviewer_XuHA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed a framework that treats model fitting with missing values as a latent space regularization problem using measure theoretical arguments. Specifically, the author translates the effect of missing values in the data space into the decreased mutual information between the latent variables and target variables. The author shows two propositions that mutual information is closely connected to the distances of latent variables in the latent space.\nIn the end, the author proposed an attention model as the encoder to map the observation into a latent variable with a feature embedding network. \n\nEmpirically, the author evaluates the proposed method for classification problems with a synthetic dataset and a benchmark dataset, showing improved results without imputation.", "summary_of_the_review": "My main concern about this paper is the clarity and writing quality. Since I am not sure I fully understand the paper, I cannot judge the novelty or whether the contribution is significant. ", "strength_and_weaknesses": "Training models without imputation is an interesting point of view and using a measure theoretical point of view is novel to the best of my knowledge. But, I have to admit I am not sure I fully understand this paper. The clarity is a severe issue. Using measure theory to explain the missing mechanism should be fairly mathematical including the formal definition of the term used in the paper, claims made by the paper (or some references). However, most of the claims made by the paper are described in words, which is good for intuition but not good for formal explanation. Due to this, I got confused when I read this paper and still not clear on some of the claims made by the author. \n\nFor instance, in section 4, I am not sure why using latent space regularization helps remove the bias from missing mechanisms. Where are the proofs of propositions 1 and 2? I am not sure I understand the argument for countable additivity. \n\nIn section 5 paragraph 1, you mentioned defining a general function $f_\\theta$ (since this is not has a subscript $k$, I assume it is shared for all ensembles.) but in the second paragraph, you define a feature-specific embedding network. Is the feature embedding network used to distinguish different ensembles? If so, these two are inconsistent. From algorithm 1, I assume $x$ contains some missing entries, how does the embedding network handle the missing part? In addition, why the transformer has to be trained on randomly generated subsets at each training step? Can this method be used for regression tasks, since the argument of mutual information is based on the cross-entropy loss function?\n\nIn section 2, you mentioned that without missingness, the measure is uniform. Just to double-check, the subset $u$ is the set containing variable indices (e.g. {1,3,5})?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject