RIFLE: Imputation and Robust Inference from Low Order Marginals
Authors: Sina Baharlouei
Sze-Chuan Suen
Meisam Razaviyayn
Venue: Accepted by TMLR ()

Abstract: The ubiquity of missing values in real-world datasets poses a challenge for statistical inference and can prevent similar datasets from being analyzed in the same study, precluding many existing datasets from being used for new analyses. While an extensive collection of packages and algorithms have been developed for data imputation, the overwhelming majority perform poorly if there are many missing values and low sample sizes, which are unfortunately common characteristics in empirical data. Such low-accuracy estimations adversely affect the performance of downstream statistical models. We develop a statistical inference framework for predicting the target variable in the presence of missing data without imputation. Our framework, RIFLE (Robust InFerence via Low-order moment Estimations), estimates low-order moments of the underlying data distribution with corresponding confidence intervals to learn a distributionally robust model. We specialize our framework to linear regression and normal discriminant analysis, and we provide convergence and performance guarantees. This framework can also be adapted to impute missing data. We compare RIFLE with state-of-the-art approaches (including MICE, Amelia, MissForest, KNN-imputer, MIDA, and Mean Imputer) in numerical experiments. Our experiments demonstrate that RIFLE outperforms other benchmark algorithms when the percentage of missing values is high and/or when the number of data points is relatively small. RIFLE is publicly available

Published in Transactions on Machine Learning Research (09/2023)
RIFLE: Imputation and Robust Inference from Low Order Marginals
Sina Baharlouei University of Southenn California
bahartou@kus..edu
Kelechi Ogudu University of Southern California
kogudu@usc..ed edu
Sze-chuan Suen University of Southern California
ssuen Gusc..de edu
Meisam Razaviyayn
razaviya Busc. edu
University of Southera California
Reviewed on OpenReview: /ttps://openyeeee openreviee.........mm net/ forum? id= oud7MyyKKy
Abstract
The ubiquity of missing values in real-workd datasets poses challenge for statistical infer- ence and can prevent similar datasets from being analyzed in the same study, precluding many existing dalasets from being used for new analyses. While an exlensive collection of packages and algorithms have been developed for data imputation, the overwhelming majority perform poorly if there are many missing values and low sample sizes, which are unfortunately common characteristics in empirical data. Such low-accuracy estimations ad- versely affect the performance of downstream statistical models. We develop statistical inference framework for regression and classification in the presence of missing deta without imputation. Our framework, RIFLE (Robust InFerence via Low-order moment Estima- lions), estimates low-order moments of the underlying data distribution with corresponding confidence intervals to learn a distributionally robust model. We specialize our framework to lincar regression and normal discriminant analysis, and we provide convergence and per- formance guarantees. This framework can also be adapted impute missing data. In numerical experiments, compare RIFLE to several state-of-the-art approaches (includ- ing MICE, Amelia, MissForest, KNN-imputer, MIDA, and Mean Imputer) for imputation and inference in the presence of missing values. Our experiments demonstrate that RIFLE outperforms other benchmark algorithms when the percentage of missing values is high and/or when the number data points is relatively small. RIFLE is publicly available at http:///..thhbb.oooottttattttttr
1 Introduction
Machine learning algorithms have shown promise when applied various problems, including healthcare, finance, social data analysis, image processing, and speech recognition. However, this success mainly relied the availability of large-scale, high-quality datasets, which may be scarce in many practical problems, especially in medical and health applications (Pedersen et al. 2017 Sterne et al.) 2009 Beaulieu-Jones et al. 2018). Moreover, many experiments and dalasets suller from the small sample size in such applications. Despite the availability of a small number data points in each study, an increasingly large number of datascts are publicly available. To fully and effectively utilize information on related rescarch questions from diverse datasets, information across various datasets (e.g., different questionnaires from multiple hospitals with overlapping questions) must. be combined in reliable fashion.

---

Published in Transactions on Machine Learning Research (09/2023)
X1
X3.0
X70
X
X100
y
Dataset Dataset Dataset
Missing Data
Missing Data
Missing Data
n2
Missing Data
123
Figure 1: Consider the problem of predicting the trait from feature vector "iI...,,X00)), Suppose that we have access to three data sets: The first dataset includes the measurements (X1.X2,.... ,40 y) for B1 individuals. The second dataset. collects data from another n2 individuals by measuring (X30:-.. Xso) with no measurements of the target variable in it; and the third dataset contains the measurements from the variables ( X100; y) for ng number of individuals. How one should learn the predictor ,., from these three datasets?
After integrating data from different studies, the obtained dataset can contain large blocks of missing values, as they may not share the same features (Figure D
There three general approaches for handling missing values in statistical inference (classification and regression) tasks. A Naïve method is to remove the rows containing missing entries. However, such an approach is not an option when the percentage of missingness in dataset is high. For instance, as demon- strated in Figure T the entire dataset will be discarded we eliminate the rows with at least one missing entry.
The most common methodology for handling missing values a learning task is to impute them in pre- processing stage. The general idea behind data imputation is that the missing values can be predicted using the available data entries and correlated features. Imputation algorithms cover wide range of methods, including imputing missing entries with the columns means Little & Rubin (2019 Chapter 3) (or median),
least-square and linear regression- based methods (Raghunathan et al. 2001 Kim et al. 2005 Zhang et al.
2008 Cai et al. 2006 Buuren & Stoothuis-Oudssoone
2010), matrix completion and expectation maxi- mization approaches Dempster et al. (1977); Ghahramani Jordan (1994); Honaker el. al. (2011), KNN based (Troyanskaya et al., 2201),, Tree based methods (Stekhoven & Bühlmann 2012 Xia al. 2017 and methods using different neural network structures. Appendix presents a comprehensive review of these
methods.
The imputation of data allows practitioners to run standard statistical algorithms requiring complete data. However, the prediction model's performance can be highly reliant on the accuracy of the imputer. High error rates in the prediction of missing values by the imputer can lead to the catastrophic performance of the downstream statistical methods executed on the imputed data.
Another class of methods for inference in the presence of missing values relies on robust optimization over the uncertainty sets missing entries. (Shvaasaamy et al. (2006) and et al. (2009) adopt robust optimization to learn the parameters of support vector machine model. They consider uncertainty sets for the missing entries in the datasct and solve min-max problem over those sets. The obtained classifiers are robust to the uncertainly of missing entries within the uncertainty regions. In contrast to the imputation-based approaches, the robust classification formulation does not carry the imputation error to the classification phase. However, finding appropriate intervals for cach missing entry is challenging, and it is unclear how determine the uncertainty range in many real datasets. Moreover, their proposed algorithms are limited the SVM classifier.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Mmin
HD
-
Missing
nh
min
max Bp({(z;0))
Mmnx
Mo +
Missing Data
Missiig Data
n2
Mmin S <Eplzz) max:
i ha
Cmin
Co
A
C S Ep (zz
C max
Missing Data
n3
C
Cp
Low-order Marginals
Distributionally Robust Formulation
Prediction
Figure 2: Prediction of the target variable without imputation. RIFLE estimates confidence intervals for low-order (first and second-order) marginals from the input data containing missing values. Then, it solves a distributionally robust problem over the set of all distributions whose low-order marginals are within the estimated confidence intervals.
In this paper; we propose RIFLE (Robust InFerence via Low-order moment Estimations) for the direct inference of target variable based on set of features containing missing values. The proposed framework does not require the data to be imputed in pre-processing stage. However, it can also be used pre-processing tool for imputing data. The main idea of the proposed framework is to estimate the first and second-order moments of the data and their confidence intervals by bootstrapping on the available data matrix entries. Then, RIFLE finds the optimal parameters of the statistical model for the worst-case distribution with the low-order moments (mean and variance) within the estimated confidence intervals (See Figure 2). Compared to Shivaswamy et al. (2006); ; Xu et al. (2009), we estimate uncertainty regions for the low-order marginals using the Bootstrap technique. Furthermore, our framework is not restricted to any particular machine learning model, such support vector machines (Xu et al. 2009
Contributions: Our main contributions are as follows:
We present distributionally robust optimization framework over the low-order marginals of the training data distribution for inference the presence of missing values. The proposed framework does not require data imputation as a pre-processing stage. In Section and Section we specialize the franework to ridge regression and classification models as two case studies respectively. The proposed framework provides a novel strategy for inference in the presence of missing data, especially for datasets with large proportions of missing values.
2. We provide theoretical convergence guarantees and the iteration complexity analysis of the presented algorithms for robust formulations of ridge linear regression and normal discriminant analysis. More- over, we show the consistency of the prediction under mild assumptions and analyze the asymptotic statistical properties of the solutions found by the algorithms.
While the robust inference framework is primarily designed for direct statistical inference in the presence of missing values without performing data imputation, it can also be adopted as an impu- lation tool. To demonstrate the quality of the proposed imputer, we compare its performance with several uidely-used imputation packages such as MICE (Buuren & Sebothhui-Oudshoood Amelia (Honaker et al. 2011),, Forsst (Stekhoven & Bühlmann) 2012), KNN-Imputer (Troyan- skaya et al. 2001), MIDA (Gondara & Wang 2018), GAIN (Yoon et al. 2018) on real and synthetic datasots. Generally speaking, our mothod outperforms all of the mentioned packages when the number of missing entries is large.
2010)
2 Robust Inference via Estimating Low-order Moments
RIFLE is based on distributionally robust optimization (DRO) framework over low-order marginals. As- sume that (x, 3) E Rd x R follows a joint probability distribution p*. A standard approach for predicting the target variable y given the input vector x is to find the parameter 0 that minimizes the population risk with respect to a given loss function t:
min E Exxx)>=- [((x (x.w;)) v:)]]
(1)
Since the underlying distribution of data rarely available in practice, the above problem cannot be directly solved. The most common approach for approximating (1) is to minimize the empirical risk with respect to
3

---

Published in Transactions on Machine Learning Research (09/2023)
given i..d samples (1,.y)...... (xx; Vo) drawn from the joint distribution Pt:
min 8xx,, (13, 11; 0).
The above empirical risk formulation assumes that all entries of x; and Y/ are available. Thus, to utilize the empirical risk minimization (ERM) framework in the presence of missing values, one can either remove or impute the missing data points in a pre-processing stage. Training via robust optimization is a natural alternative in the presence of missing data. Shivaswamy et al.) (2006); Xu et al. (2009) suggest the following optimization problem that minimizes the loss lunction for the worst-case scenario over the defined uncertainly sets per data points:
1 {x; 5, 36: 0),
min
max (8:EN))
where Ms represents the uncertainty region of data point i. Shivaswamy et al. (0066) obtains the uncertainty sets by assuming a known distribution on the missing entries of dalasets. The main issue in their approach is that the constraints defined on data points are totally uncorrelated, Xu et al. 2009) on the other hand defines Nr as a "box" constraint around the data point i such that they can be linearly correlated. For this specific case, they show that solving the corresponding robust optimization problem is equivalent to minimizing a regularized reformulation of the original loss function. Such an approach has several limitations: First, it can only handle few special cases (SVM loss with linearly correlated perturbations on data points). Furthermore, Xu et. al. (2009) is primarily designed for handling outliers and contaminated data. Thus, they do not offer any mechanism for the initial estimation of x; when several vector entries missing, In this work, we instcad take a distributionally robust approach by considering uncertainty on the data distribution instead of defining an uncertainly set for each data point. In particular, we aim to fit the best parameters of a statistical learning model for the worst distribution in a given uncertainty set by solving the following:
min
max PEP
p(xx))
(3)
where P an uncertainty set over the underlying distribution of data. A key observation is that defining the uncertainty set in (3) is easier and computationally more efficient than defining the uncertainty sets {Nr)== in (2). In particular, the uncertainty set P can be obtained naturally by estimating low-order moments of data distribution using only available entries. To explain this idea and to simplify the notations, let z = =x,xx)) v),
p* A E[zz,, and C é E[zz"]. While fu and C are typically not known exactly, one can estimate them (within certain confidence intervals) from the available data by simply ignoring missing entries (assumiin the missing value pattern is completely at random, e.g., MCAR). Moreover, we can estimate the confidence intervals via. bootstrapping. Particularly, we can estimate Pmin Mmaxx Cii:: and C C% from data such that Mimin S f < mmax and C C < C,ax with high probability (where the inequalities for matrices and vectors denote component-wise relations). In Appendix B we show how a boolstrapping strategy can be used to obtain the confidence intervals described above. Given these estimated confidence intervals from data, 3 can be reformulated as
min max Ep Er({(2;0)
s.t. M S Ep(zz] S mmaxx
(4)
C S Ep zzz
< C
utilize the distributionally robust optimization as (3) over the set of positive semi- definite (PSD) cones for robust inference under uncertainty. While their formulation considers &, balls for the constraints on low order moments of the data, we use Lox constraints that are computationally more nat natural in the presence of missing entries when combined with bootstrapping. Furthermore, while can be applied to general convex lossos, their method relies on the ellipsoid and the existence of oracles for performing the steps of the ellipsoid method, which is not applicable in modern high-dimensional problems. Moreover, they assume concavity in data (the existence of some oracle to return the worst-case data points) that is
Gao & Kleywegt 2017
4

---

Published in Transactions on Machine Learning Research (09/2023)
practically unavailable even in convex loss functions (including linear regression and normal discriminant analysis studied in our work).
In Section 3 we study the proposed distributionally robust framework described in 4 for the ridge linear regression. We design efficient first-order convergent algorithms to solve the problem and show how we can use the algorithms for both inference and imputation in the presence of missing values. Further, Appendix F we study the proposed distributionally robust framework for the classification problems under the normality assumption of features. In particular, show how Framework 4 robust normal discriminant analysis in the presence of missing values.
can be specialized to the
3 Robust Linear Regression in the Presence of Missing Values
Let us specialize our framework to the ridge linear regression model. In the absence of missing data, ridge regression finds optimal regressor parameter 0 by solving
min 0
X - y11 + 11012
or equivalently by solving:
min x X0 20" x 11
(5)
Thus, having the sccond-order moments of the data C= X and b X sufficient for finding the optimal solution. In other words, it sullices compute the inner product of any two column vectors a,, aj of X, and the inner product of any column a, of X with vector y. Since the matrix X and vector y are not fully observed due to the existence of missing values, one use the available data (see (24) for details) to compute the point estimators Co and by- These point estimators can be highly inaccurate, especially when the number of non-missing rows for two given columns is small. In addition, if the pattern of missing entries does not follow the MCAR assumption, the point estimators are not unbiased estimators of and b.
3.1 A Distributionally Robust Formulation of Linear Regression
As mentioned above, to solve the linear regression problem, we only need to estimate the second-order moments of the data (X"X and XTy). Thus, the distributionally robust formulation described in (4) is equivalent to the following optimization problem for the lincar regression model:
min s.t.
max C.b
CC - 2b 0 x|101
C. -A-cAs cA < C +o
(6)
bo cō < bo c
C
where the last constraint guarantecs that the covariance matrix is positive and semi-definite. We dicuss the procedure of estimating the confidence intervals (bo. Co, 8, and A) in Appendix B
3.2 RIFLE for Ridge Linear Regression
Since the objective function in (6) is convex in 0 (ridge regression) and concave in and C (linear), the min- imization and maximization sub-problems are interchangeable (Sion al. 1958). Thus, we can equivalently rewrite Problem 6
max C.b
(CC,b) b)
s.t.
C - cA < C +c
(7)
b cb < b < bo c
C 2 0,
5

---

Published in Transactions on Machine Learning Research (09/2023)
where g(b, C) m 0bn0 X||0|s. Function g be computed in closed-form given any pair of b) by setting 0 (C + - Thus, using Danskin's Theorem (Danskin) 2012), we can apply projected gradient ascent to function to find an optimal solution of 7 as described in Algorithm | At each itcration of the algorithm, we first perform one step of projected gradient ascent on matrix C and vector b; then we updale in closed-form for the obtained and b. We initialize and b using entriwise
point estimation on the available rows (see Equation 24 in Appendix B
The projection of b to the box
Algorithm RIFLE for Ridge Linear Regression in the Presence of Missing Values
1: Input: Co, bo: A, 6, T
Initialize: C Co: bo-
for i 1..... T do
Update C II
[C+ a00']
Update b II (b- 2a0)
6: Set 0 = (C + -b)
constraint by- cō < b < bo cō can be done entriwise and has the following closed-form
b;
if
bo: - cō; s b; S box + co;;
lb(b,))
bos - cō. b, < bos cō
bn: + co;
if bo: + cō; b;.
Theorem 1. Let (0, C, b) be the optimal solution of 6 (b, C) argming C |||01| and D = C C 7 + llbn- - b -bl.. Assume that for any given b and C, within the uncerlainly (constraint) sels (b, C)Il T. Then Algorithm computes an e-optimal solution of the objective function
described in 6
in O 1) iterations.
in
A
Proof. The proof is relegated to Appendix
we show how using the acceleration method of Nesterov can improve the convergence rate A technical issue of Algorithm and its accelerated version presented Appendix C is that projection of C to the intersection of box constraints and the set of positive semidefinite matrices (II + (C)) is challenging and cannot be done in closed-form. In the inplementation of Algorithmi [ we relax the problem by removing the PSD constraint on C to avoid this complexity and time-consuming singular value decomposition at cach iteration. This relaxation does not drastically change the algorithm's 's performance, as our experiments show in Section 5 A more systematic approach is to write the dual problem of the maximization problem and handle the resulting constrained minimization problem with the Alternating Direction Method of Multipliers (ADMM). The detailed procedure of such an approach can be found in Appendix D All these algorithms provably convergent to the optimal points of Problem (6). In addition to theoretical convergence, we have numerically evaluated the convergence of resulting algorithms in Appendix K Further, the proposed algorithms are consistent, as discussed in Appendix []
In Appendix
of Algorithm to (vV D(r+1)2 ex
3.3 Performance Guarantees for RIFLE
Thus far, we have discussed how efficiently solve the robust linear regression problem in the presence of missing values. A natural question in this context is the statistical performance of the obtained optimal solution in the previous section the unseen test data points. Theorem 2 answers this question from two perspectives: Assuming that the missing values are distributed completely at random, our estimators are consistent. Moreover, for the finite case, Theorem 2 part (b) states that with the proper choice of confidence intervals, with high probability, the test loss of the obtained solution is bounded by the training loss of the estimator. Note that the results regarding the performance of the robust estimator generally hold for MCAR missing pattern. However, we perform several experiments on datasets with MNAR patterns to show how
RIFLE works in practice such datasets in Section 5

---

Published in Transactions on Machine Learning Research (09/2023)
Theorem 2. Assume the data domain is bounded and that the missing pattern of the data follows MCAR. Let X be the truining data drown i.i.d. from the ground-truth distribution P with low-order momenis C and b". Further, assume that each entry of X and y is missing with probability < 1. Let (On. Cny ba) be the solution of Problem (6)
(a) Consistency of the Covariance Estimator: As the number of data points goes to infinity, the estimated loder morginals converge the ground- -ruth values, almost surely. More precisely,
lim C, Ep-[xx' a.S.,
(8)
lim b, Ep- (xy] a.s.
(9)
n 700
(b) Defining
Lerain (0,) C,, 2b, 1 Lieet (0n) = C²' C'O, - 2bb*T 0, +
where C = Px and b* = ((x.))=p' [xy] are the ground- truth second-order moments. Given
V = maxi, (ax)) (mazimum variance of pairwise feature products), with the probability of at least
dV
2>2A2((1-p) we have:
Luest (0) S Lirain( (0)
(10)
where = min(Ar}} and c is the hyper-parameter for controlling the size of the confidence intervals as presented
Proof. The proof is relegated to Appendix H
3.4 Imputation of Missing Values and Going Beyond Linear Regression
RIFLE can be used for imputing missing data. To this end, we impute different features a given dataset independently. More preciscly, impute each feature containing missing values, we consider as a target variable y and the rest of the features as the input X in our methodology. Then, we train a model to predict the feature y given X via Algorihm [ (or its ADMM version, Algorithm 7 in the appendix). Tet the obtained optimal solutions be C* b", and 0- For given missing entry, row of that missing entry are available. However, that is not usually the case in practice, as cach row can contain more than one missing entry. Therefore, one can learn a separate model for each missing pattern in the dataset. Let us clarify this point through the example in Figure [ In this example, we have three different missing patterns (one missing pattern for each dataset). For missing entries in Dataset 1, the first forty features are available. Let r, denote the vector of the first 40 features in row j. Assume that we aim to impute entry i E (41,..., 100} in row where denoted by Tjz- To this end, restrict X to the first 40 features. Moreover, consider == x; the target variable. Then, run Algorithm D on X and y to obtain the optimal C", bü, and 0 Consequently, we impute Tjz follows:
use 0 only if all other features in the
Izz ====000
We can use the same methodology for imputing missing entries in each feature for missing patterns in Dataset and Dataset While this approach reasonable for the missing pattern observed in Figure I in many practical problems, different rows can have distinct missing patterns. Thus, in the worst case, Algorithm D must be executed once for each missing entry. Such approach is computationally expensive and might be infeasible in large-scale datasets containing large amounts of missing entries. Alternatively, one can perform Algorithm ] onnly once to obtain C' and b' (considered the "worst-case/ "worst-case/pessinistt"" ppees/isssiiscc estimation of the moments). Then to impule each missing entry, C* and b" are restricted to the features available in that missing entry's 's row. Having the restricted C* and b*, the regressor 0* can be obtained closed-form (line 6 Algorithm D In this approach, we perform algorithm once and find the optimal 0* for each missing entry based on the estimated C and b*. This approach can lead to sub-optimal solutions compared to the former approach, but it is much faster and more scalable.
7

---

Published in Transactions on Machine Learning Research (09/2023)
Beyond Linear Regression: While the developed methods primarily designed for ridge linear regres- sion, one can apply non-linear transformations (kernels) to obtain models beyond linear. In Appendix we show how to extend the developed algorithms to quadratic models. The RIFLE framework applied to the quadratically transformed data is called QRIFLE.
4 Robust Classification Framework
In this section, we study the proposed framework in 4 for the classification tasks the presence of missing values. Since the target variable y y (1, (I,..., M) takes discrete values in classification tasks, we consider the uncertainty sets over the data's first- and socond-order marginals given each target value (label) separately. Therefore, the distributionally robust classification over low-order marginals can be described as:
min max Er(e(x, U, W.] P
s.t. Mmin.y S Ep(x|y] I Pmax.y Vy T y
(11)
Emin.
I Ep|xxx yy S Emax.y Ay E y
where pminn Mmax: Emin: and Emax are the estimated confidence intervals for the first and second order of the data distribution. Unlike the robust linear regression task in Section g the evaluation of the objective function in [I]] might depend on higher-order marginals (beyond second-order) due to the nonlinearity of the loss function. As a result, Problem [11 is non-convex non-coneave intractable min-max optimization problem in general. For the sake of computational traceability, restrict the distribution in the inner maxinization problem to the set of normal distributions. In the following section, we specialize (11) to the quadratic discriminant analysis as a case study. The methodology can be extended to other popular classification algorithms, such as support vector machines and multi-layer neural networks.
4.1 Robust Quadratic Discriminant Analysis
Learning logistic regression model on datasets containing missing values has been studied extensively in the literature (Fung & Wrobel 1989 Abonazel & Ibrahim 2018). Besides deleting missing values and imputation-based approaches, Fung & Wrobel (1989) models the logistic regression task in the presence of missing values as a lincar discriminant analysis problem where the underlying assumption is that the predictors follow normal distribution conditional on the labels. Mathematically speaking, they assume that the data points assigned to a specific label follow a Gaussian distribution, i.e., x|y = i (M,, E). They use the available data to estimate the parameters of each Gaussian distribution. Therefore, the paramcters of the logistic regression model can be assigned based on the estimated parameters of the Gaussian distribul ions for different classes. Similar to the linear regression case, the estimations of means and covariances are unbiased only when the data satisfies the MCAR condition. Moreover, when the nunber of data points in the dataset is small, the variance of the estimations can be very high. Thus, to train a logistic regression model that is robust to the percentage and different types of missing values, we specialize the general robust classification framework formulated in Equation (II) the logistic regression model. Instead of considering a common covariance matrix for the conditional distributions of given labels y (linear discriminant analysis), we assume a more general case where each conditional distribution has its own covariance matrix (quadratic discriminant analysis). Assume that xly ~ N(Hy Σ,) for 0, 1. We aim to find the optimal solution to the following problem:
- log (o(w" x))] PPv = 1)
min max p0. En En
Exly 1 E.)
log (1- (1-o(w7 x)) PPy = 0)
ExIg=O~NNHO En)
s.t.
Mmino < 140 < Mmaxo
(12)
pmin: < 11 < Mmax
Σ Eming < En < E Emaxo
Σ Emin,
< Σ1 < Σmax
Where (x) = 11((1 exp(-x)) is the sigmoid function.
8

---

Published in Transactions on Machine Learning Research (09/2023)
To solve Problem [1]] first, we focus the scenario when the target variable has no missing values. In this case, each data point contributes to the estimation of either (m4 Σ1) or (Ho; Eo), depending on its label. Similar to the robust linear regression case, we can apply Algorithm ! to estimate the confidence intervals for p4 E; using data points whose target variable equals (y=t). 1).
Obviously, the objective function is convex w since the logistic regression loss convex, and the expectation of loss can be seen a weighted summation, which is convex. Thus, fixing μ., the outer minimization problem can be solved with respect to using standard first-order methods such as gradient descent.
Although the robust reformulation of logistic regression stated in 12 is convex in w and concave in No and M1, the inner maxinization problem is intractable with respect to En and En We approximate Problem in the following manner:
-log Ti Ex MIBxxy=INN(M E1) - log (o(w">)) + To Tolxxp=u (po. 2o) - log (1- o oww" (1-o(w²xx)
min max EA E
s.t.
Mmino S 140 S Mmaxo
(13)
Mminn S M1 5 Mmaxi
Eg E Eoi, Σo2,... ..... M
E 11,, Σ Eik]
where = PPy = 1) and To = -P(y- = 0). To compute optimal o4o and fa,, we have:
max A1
Ex-N(H [- log (a(w"'x)) s.t. Pmin S S Mmax
(14)
Theorem 3. Let a({] be the i-th element of vector a.
The optimal solution of Problem 14
has the following
form:
Mmax [i],
if
0
Pil
(15)
1min [i],
f
w
> 0.
Note that we relaxed 12 by taking the maximization problem over finite set of E estimations. We estimate each E by bootstrapping the available data using Algorithm 4 Define fi(w) as:
fi(w) mi N(,, log (((wT)))
(16)
Similarly, we can define:
9: (w)) NoH [- log log (1- 1--oww"m x
(17)
Since the maximization problem is over finite set, we can rewrite Problem
13
as:
min max w SjE(L.....k
Si(w) + (w(w)
min
max w Pr
Eris(w)
Σ (wg)(w)
1=1
1=1
(18)
s.L.
Ei=1 11
p 2
- 1, 4 0
Since the maximum of several functions is not necessarily smooth (differentiable). we add quadratic reg- ularization term to the maximization problem, accelerating the convergence rate (Nouiehed et al. 2019) follows:
min
pifi(w)
p Σ
2
Σ
409 (w)
5
Pl...PP ........... 26
i=1
(19)
Σ
1i=1 Pi 1,
pt 2
Σ 9 1, 2

---

Published in Transactions on Machine Learning Research (09/2023)
First, we show how to solve the inner maximization problem. Note that the p; p,'s and q are independent. We show how to find optimal p:'s. Optimizing with respect to gi is similar. Since the maximization problem is a constrained quadratic program, we can write the Lagrangian function as follows:
Σ P.fi(w) - > P3
1)
Pi Ph
(20)
s.t. P:20
Having the optimal X, the above problem has closed-form solution with respect to each ps, which can be written as:
- + fi
p
26
Since Pf is a non-increasing function with respect to A, can find the optimal value of X using the following bisection algorithm. Algorithm 2 bisection idea.
demonstrates how to find an e-optimal and p 's efficiently using the
Algorithm 2 Finding the optimal À and p.'s using the bisection idea
1: Initialize: Nlow = 0, Ahih = max; f.P = Vi (1.2....,k
2: while E Ph > 6 do
Alow Ahugh
3:
Set P:- = -+
ViE (1,2.....
28
if D <
then
-i=1
6: Anigh X
7: else
8: Alow
return X, PI, P2...... Pk- Pk.
Remark 4. An alternative method for finding optimal X, and ps 's is to sort Sa values in O lol k) first, and
then finding the smallest fs such that if set X = fú, the of ps 's is bigger than 1 (let j be the index of that value). Without loss of generality, assume that fi .... fk. Then, Σ -Atf 1, which has closed-form solution with respect to X.
28
To update w, we need solve the following optimization problem:
min
f((((w) + Σ (w)),,
(21)
Similar to the standard statistical learning framework, we solve the following empirical risk minimization problem by applying the gradient descent to w on finite data sample. Define ft as follows:
f((w) m Σ log (o(wTxi))) x:))
(22)
where X1,..., x, are generated from the distribution N E1d). The empirical risk minimization problem can be written as follows:
min Σ p.f((() + Σ gg)()))
(23)
Algorithm 3 summarizes the robust linear discriminant analysis method for the case where the label of all data points is available. Theorem 5 demonstrates the convergence of gradient descent algorithm applied
to 23 in log( m) iterations to an e-optimal solution.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 3 Robust Quadratic Discriminant Analysis in the Presence of Missing Values
Input: X1; matrix of data points with labels and respectively, T', Number of iterations, a Step-size.
2: Estimate Mming and Mmax, using the available entries of Xo-
3: Estimate Mmin: and Mmax, using the available entries of X1.
4: Estimate Eol...., Eos using bootstrap estimator on the available data of Xo-
5: Estimate En....., E1s using bootstrap estimator on the available data X1.
for 1= 1....., do
Compute ui and μo by Equation (1)
Find optimal ply... Pk. and -1....:94 using Algorithm
Σ DI-IP (.()( + Eド VB:(WW)
9: w - - a
Theorem 5. Assume that M = max; fi- Gradient descent algorithm requires 0 log( M) gradient evalu-
ations for converging to an c-optimal saddle point of the optimization problem (23
In Appendix extend the methodology to the case where y contains missing entries.
5 Experiments
In this section, we evaluate RIFLE's performance on a diverse set of inference tasks in the presence of missing values. We compare RIFLE's performance to several caate-of-the-ar approaches for data imputation on synthetic and real-world datasets. The experiments are designed in a manner that the sensitivity of the model to factors such as the number of samples, data dimension, types, and proportion of missing values can be evaluated. The description of all datasets used in the experiments can be found in Appendix
5.1 Evaluation Metrics
We need access to the ground-truth values of the missing entries evaluate RIFLE and other state-of-the-an imputation approaches. Hence, we artificially mask a proportion of available data entries and predict them with different imputation methods. A method performs better than others if the predicted missing entries are closer to the ground-truth values. To measure the performance of RIFLE and the existing approaches on regression task for a given test dataset consisting of N data points, we use normalized root mean squared error (NRMSE), defined as:
NRMSE V N Ee (Ji 0:)²
EN 7)
where ys, ds. and y represent the true value the i-th data point, the predicted value of the i-th data point, and the average of true values of data points, respectively. In all experiments, generated missing entries follow either missing completely random (MCAR) a missing not at random (MNAR) pattern. A discussion the procedure of generating these patterns can be found in Appendix G
5.2 Tuning Hyper-parame parameersse of RIFLE
The hyper-parameter e in 7 controls the robustness of the model by adjusting the size of confidence intervals. This parameter is tuned by performing 50, 100), and the one with the lowest NMRSE is chosen. The default value in the implementation is e = since it consistently performs well over different experiments. Furthermore, X, the hyper-parameter for the ridge regression regularizer, (0.01, 0.1, 0.5, 2, 5, 10, 20, 50}. To tune K, the number of bootstrap samples for estimating the confidence intervals, we tried 10, 20, 50, and 100. No significant difference is observed in terms of the test performance for the above values.
voss-valid procedure over the set {0.1, 0.25, 0.5, 1, 5, 10, 20,
tuned by choosing 20% the data as the validation set from the set
11

---

Published in Transactions on Machine Learning Research (09/2023)
Furthermore, tune the hyper-parameters of the competing packages follows. For KNN-Imputer Troo- anskaya et al. 2001), we try {2, 10, 20, 50) for the number of neighbors (K) and pick the one with the highest performance. For MICE (Buuren & Groothuis- Oudshoorn 2010) and Amelia (Honaker et al. 2011). we gen- crate 5 different imputed data and pick the one with the highest performance on the test data. MissForest has multiple lyper-parameters. We keep the criterion as "MSE" since our performance evaluation measure is NRMSE. Moreover, we tune the number of iterations and number of estimations (number of trees) by checking values from {5, 10, 20) and (50, 200), respectively. We do not change the structure of the neural networks for MIDA (Gondara & Wang 2018 and GAIN Yoon et al. 2018), and the default versions are performed for imputing datasets.
5.3 RIFLE Consistency
Tn Theroem
Part (a), we demonstrated that RIFLE is consistent. Tn Figure we investigate the consistency
0.7
RIFLE
MissForest
Amella
KNN
0.5
Expectation Maximization
Normalized aalnallzeddd RMSE
0.2
0.0
100
200
400
500
50K 100K 200K 500K
1M
21
5M
Number of Training Samples (N)
Figure 3: Comparing the consistency of RIFLE, MissForest, KNN Imputer, MICE, Amelia, and Expectation Maximization methods on synthetic dataset containing 40% of missing values.
of RIFLE on synthetic datasets with different proportions of missing values. The synthetic data has 50 input features following jointly normal distribution with the mean whose entries are randomly chosen from the interval (-100, 100). Moreover, the covariance matrix equals Σ = SST where S elements are randomly picked from 1). The dimension of S is 50 x 20. The target variable is a linear function of input features added to a mean zero normal noise with a standard deviation of 0.01. As depicted in Figure 3 RIFLE requires fewer samples to recover the ground-truth parameters of the model compared to MissForest, KNN Imputer, Expectation Maximization (Dempster et al. 1977), and MICE. Amelia's performance is significantly good since the predictors have joint normal distribution and the linear underlying model. Note that by increasing the number of samples, the NRMSE of our framework converges to 0.01, which is the standard deviation of the zero-mean Gaussian noise added to each target value (the dashed line).
5.4 Data Imputation via RIFLE
As explained in Section 3 while the primary goal RIFLE is to learn a robust regression model in the presence of missing values, it can also be used as an imputation tool. We run RIFLE and several state-of- the-art approaches on five datasets from the UCI repository (Dua & Graff 2017) (Spam, Housing, Clouds, Breast Cancer, and Parkinson datasets) with different proportions of MCAR missing values (the description of the datasets be found in Appendix M. Then, compute the NMRSE of imputed entries. Table [ shows the performance of RIFLE compared to other approaches for the datasets where the proportion of missing values are relatively high (-(l-m) Rs O()). RIFLE outperforms these methods almost all cases and performs slightly better than MissForest, which uses highly non-linear model (random forest) to impute missing values.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Dataset Name
RIFLE
QRIFLE
MICE
Amelia
GAIN
MissPorest
MIDA
Spam (30%)
0.87 45.009
0.82 40.008
1,23 20.012
1.26 #0.007
0.91 40.005
0.90 20.013
0.97 20.008
0.94 0.000
Spam (50%)
40.013
4.0014
1.29 +0.018
1.33 40.024
+0.015
#0.011
0.99 +0.011
0.97 +
Spam (70%)
40.017
0.91 +0.019
1.32 +0.028
1.37 +0.032
0.97 +0.014
20.016
0.99 #0.018
Housing (30%)
1:00155
10.018
1.03 10.024
10.016
1:0015
10.018
0.93 4.0025
0.95 0.011
Housing (50%)
0.88 =0.021
0.90
1.14 40.029
1.09 40.027
=0.019
20.018
0.98 40.029
0.96 0.016
Housing (70%)
20.026
0.95 =00002
1.22 10.036
1.18 10.038
0.95
4:00227
20.024
1.02 40.007
0.017
Clonds (307%)
0.81 400018
0.79 +0.019
0.98 +000224
1.04 +20.027
0.76
+-00211
0.71
+-0.011
0.83 +:0.022
0.86 0.013
Clouds (50%)
0.84 1:00026
0.84 =0.028
1.10 1..041
1.13 1:0.46
0.82 1-0.027
0.75
1:00223
0.88 1:0033
0.89 0.018
Clouds (70%)
0.87 +-00029
0.90
+00033
1.16 +0.044
1.19 +-0.048
0.88 +-0.035
0.81 +.0.031
0.93 +-0.044
0.92 0.023
Breast Cancer (30%)
0.52 =0.023
0.51 20.027
0.74 40.031
0.81 10.032
0.58
1.0024
0.35
20.016
0.70 #0.026
0.67 = 0.014
Breasi Caucer (50%)
0.56 =0.026
0.59
=0.027
0.79 +0.029
0.85 +0.033
0.64
1:0.225
0.59
#0.022
0.76 #0.035
0.69 +: 0.022
Breast Cancer (70%)
0.59 =0.031
0.65
+0.034
0.86
+0.012
0.92 -0.014
0.70
+3.037
0.63
=0 028
0.82 +0.035
0.67 + 0.0014
Parkinson (30%)
0.57 +0.016
0.55
+1.016
0.71 +0.019
0.67 +00021
0.53
+0.015
0.54
+0.010
0.62 +0.017
0.64 + 0.011
Parkinson (50%) 0.62 +-0022
0.64
+0.025
0.77 +0.029
0.74 +-0.034
0.61
+0.022
0.65
+00.04
0.71 +-0.027
0.69 + 0.022
Packinsou (70%) 0.67 20.027
0.74 20.033
0.85 20.038
0.82 10.037
0.69
10.031
0.73
20.022
0.78 20.038
0.75 1 0.029
Table 1: Performance comparison of RIFLE, QRIFLE (Quadratie RIFLE), and state-of-the-art methods several UCI datasets. We applied to impute methods three different missing-valus value proportions for each dataset. The best impuler is highlighted with bold font, and the second-best imputer is underlined. Each experiment is done times, and the average and the standard deviation of performances are reported.
5.5 Sensitivity RIFLE to the Number of Samples and Proportion of Missing Values
In this section, we analyze the sensitivity of RIFLE and other state-of-the-art approaches to the number of samples and the proportion of missing values. In the experiment in Figure !
we create datasets containing
Spam Dataset
Parkinson Dataset
50
25
MissForest MICE
MissForest MICE
40
37
Number Wins
Number of Wins
30
29
15
26
25
13
12
22
11
10
20:
10
9
9
3
2
1
50%
60%
70%
80%
40%
70%
80%
Missing Value Propartion
Missing Value Proportion
Wave Energy Converters Dataset
Breast Cancer Dataset
60
35
MissForest MICE
MissForest
30-
50
MICE
Number Wins
Number of Wins
25
40-
36
39
20:
19
20
21
30
2.524
15-
20
11
10
10-
10
5-
4
2
0
50%
60%
70%
80%
40%
70%
80%
Missing Value Propartion
Missing Value Proportion
Figure Performance Comparison of RIFLE, MTCE, and MissForest on four UCT datasets: Parkinson, Spam, Wave Energy Converter, and Breast Cancer. For each dataset, we count the number of features that each method outperforms the others.
40%, 50%, 60%, 70%, and 80% of MCAR missing values, respectively, for four real datasets (Spam, Parkinson, Wave Energy Converter, and Breast Cancer) from UCI Repository (Dua & Graff] (017) (the description of the datasets can be found in Appendix D). Given feature in a dataset containing missing values, we say an imputer wins that feature if the imputation error in terms of NRMSE for that imputer is less than the error
13

---

Published in Transactions on Machine Learning Research (09/2023)
of the other imputers. Figure reports the number of features won by each imputer on the created datasets described above. As we observe, the number of wins for RIFLE increases as increase the proportion of missing values. This observation shows that the sensitivity of RIFLE as an imputer to the proportion of missing values is less than MissForest and MICE in general.
1.2
1.0
NMRSE
RIFLE
MissForest
MICE
Mean Imputation
0.6
KNN Imputer
MIDA
Amelia
0.4
30
Proportion of Missing Values (%)
Figure 5: Sensitivity of RIFLE, MissForest, Amelia, KNN Imputer, MIDA, and Mean Imputer to the percentage of missing values on the Drive dalaset. Increasing the percentage of missing value entries degrades the benchmarks' performance compared to RIFLE. KNN-imputer implementation cannot. be executed on datasets containing 80% (or more) missing entries. Moreover, Amelia and MIDA do not converge to a solution when the percentage missing value entrics is higher than 70%.
Vigure@dors not show how the increase the proportion of missing values changes the NRMSE of imputers. Next, we analyze the sensitivity of RIFLE and several imputers change in missing value proportions. Fixing the proportion of missing values, generate 10 random datasets containing missing values in random locations the Drive dataset (the description of datasets is available in Appendix D We impute the missing values for each dataset with RIFLE, MissForest, Mean Imputation, and MICE. Figure shows the average and the standard deviation of these 4 imputers' performances for different proportions of missing values (10% to 90%). Figure 5 depicts the sensitivity of MissForest and RIFLE to the proportion of missing values in the Drive dataset. We select 400 data points for each experiment with different proportions of missing values (from 10% to 90%) and report the average NRMSE imputed entries. Finally, in Figure G we have evaluated RIFLE and other methods on the BlogFeedback dataset (see Appendix [ containing 40% missing values. The results show that RIFLE's performance is less sensitive decreasing the number of samples.
5.6 Performance Comparison on Real Datasets
In this section, we compare the performance of RIFLE to several state-of-the-art approaches, including MICE (Buuren & Proothuis-Oudshoor 2010), Amelia (Honaker al., 2011), MissForest. (Stekhoven Bühlmann| 2012), KNN Imputer (Raghunathan et al. 2001), and MIDA (Gondara & Wang 2018). There are two primary ways to do this. One method predict continuous target variable in a dataset with linear
many missing values is first to impute the missing data with state-of-the-art package, then regression. An alternative approach is to directly learn the target variable, we discussed in Section 3
Table 2 compares the performance of mean imputation, MICE, MIDA, MissPorest, and KNN to that of RIFLE on three datasets: NHANES, Blog Feedback, and superconduuciiitty Both Blog Feedback and Superconductivity datascts contain 30% of MNAR missing values generated by Algorithm D with 10000 and 20000 training samples, respectively. The description of the NHANES data and its distribution of missing values be found in Appendix
14

---

Published in Transactions on Machine Learning Research (09/2023)
RIFLE
MissForest
1.2
MICE
Mean Imputation
KNN Imputer
NMRSE
MIDA
Amelia
0.9
200
400
600
800
1000
2000
3000
Number of Samples
Figure 6: Sensitivity of RIFLE, MissForest, MICE, Amelia, Mean Imputer, KNN Imputer, and MIDA to the number of samples for the imputations of Blog Feedback dataset containing 40% of MCAR missing values. When the number of samples limited, RIFLE outperforms other methods, and its performance very close to the non-lincar imputer MissForest for larger samples.
We perform RIFLE for 1000 iterations and the step size of 0.01 in the above experiments. At each iteration, the main operation is to find the optimal 0 for any given b and C. The average time of each method on each dataset is reported in Table 5 in Appendix L The main reason for the time efficiency of RIFLE compared to MICE, MissForest, MIDA, and KNN Imputer is that it directly predicts the target variable without imputation of all missing entries.
Efficiency of RIFLE:
Datascts
Methods
Super Conductivity
Blog Feedback
NHANES
Regression on Complete Data
0.4601
0.7432
0.6287
RIFLE
0.4873 1 0.0036
0.8326 0.0085
0.6304 + 0.0027
Mean Impuler + Regression
0.6114 + 0.0006
0.9235
0.0003
0.6329 + 0.0008
MICE Regression
0.5078 + 0.0124
0.8507
0.0325
0.6612 0.0282
EM Regression
0.5172 + 0.0162
0.8631
0.0117
0.6392 0.0122
MIDA Imputer Regression
0.5213 + 0.0274
0.8394
0.0342
0.6542 + 0.0164
MissForcst
0.4925 + 0.0073
0.8191 + 0.0083
0.6365 + 0.0094
KNN Imputer
0.5438 + 0.0193
0.8828 0.0124
0.6427 + 0.0135
Table 2: Normalized RMSE of RIFLE and several state-of-the-art Methods on Superconductivity, blog feedback, and NHANES datasets. The first two datasets contain 30% Missing Not At Random (MNAR) missing values in the training phase generated by Algorithm g Each mothod applied 5 times to each dataset, and the result reported as the average performance standard deviation of experiments terms of NRMSE.
Since MICE and MIDA cannot predict values during the test phase without data imputation, we use them in a pre-processing stage to impule the data. Then we apply the linear regression to the imputed dataset. On the other hand, RIFLE, KNN imputer, and MissForest can predict the target variable without imputing the training dataset. Table D shows that RIFLE outperforms all other statc-of-the-art approaches executed on the three mentioned datasets. In particular, RIFLE outperforms MissForest, while the underlying model RIFLE uses is simpler (linear) compared to the nonlinear random forest model utilized by Missforest.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Method
Number of Training Data Points
LDA
Robust LDA
Robust QDA
50
52.38%
3.91%
62.14% 1.78%
61.36% 1.62%
100
61.24% + 1.89%
68.46% + 1.04%
70.07% + 0.95%
200
73.49% 0.97%
73.35% + 0.67%
73.51% = 0.52%
Table 3: Sensitivity of Lincar Discriminant Analysis, Robust LDA (Common Covariance Matricos), and
Robust QDA (Different Covariance matrices for two groups) to the number of training samples.
5.6.1 Performance of RIFLE on Classification Tasks
In Section 4 we discussed how to specialize RIFLE robust normal discriminant analysis in the presence of missing values. Since the maximization problem over the second moments of the data (E) is intractable, we solved the maximization problem over a set of & covariance matrices estimated by bootstrap sampling. To investigate the effect of choosing k on the performance of the robust classifier, we train robust normal discriminant analysis models for different values of k on two training datasets (Avila and Magic) containing 40% MCAR missing values. The description the datasets can be found in Appendix [ For h = 1, there is no maximization problem, and thus, it is oquivalent to the classifier proposed in Fung & Wrobel (1989). As shown in Figure 7 increasing the number of covariance estimations generally enhances the accuracy of the classifier in the test phase. However, as shown in Theorem the required time for completing the training phase grows linearly regarding the number of covariance estimations.
17500
15000
Avila Dataset
Avila Dataset
34
12500
Test Error (%)
32
Magic Dataset
Time (Sec)
10000
Magic Dataset
30
7500
28
5000
26
2500
1
10
20
30
40
100 200 400
50
100
150
200
250
300
350
400
Number of Covariance Estimations (k)
Number of Covariance Estimations (k)
Figure 7: Effect of the number of covariance estimations on the performance (left) and run time (right) of robust LDA on Avila and Magic datasets. Increasing the number of covariance estimations (k) improves the model's accuracy the test data. However, it takes longer training time.
5.6.2 Comparison of Robust Linear Regression and Robust QDA
An alternative approach to the robust QDA presented Section 4 is to apply the robust linear regression algorithm (Section 3) and mapping the solutions each one of the classes by thresholding (positive value maps Label 1 and negative values to label -1).
Table 4 compares the performance of two classifiers three different datasets. As demonstrated in the table, when all features are continuous, quadratic discriminant analysis has a better performance. It shows the QDA model relies highly on the normality assumption, while robust linear regression handles the categorical features better than robust QDA.
Limitations and Future Directions: The proposod framework for robust regression in the presence of missing values is limited to linear models. While in Appendix we use polynomial kernels to apply non-lin linear transformations the data, such approach can potentially increase the number of missing values in the
16

---

Published in Transactions on Machine Learning Research (09/2023)
Accuracy of Mothods
Datasct
Feature Type
RIFLE
Robust QDA
MinsFoorest
MICE
KNN Inguuter
EM
Identiication
67.12%
181%
69.54%
1.97%
65.76%
1.49%
62.48%
60.37%
12.12%
Annealing
Mixed
63.41% = 2.44%
59.51% # 2.21%
64.91% + 1 1.35%
60.00% + 1.7%%
57.44% + 1.44%
69.43%
+1.2%%
Abalone
Mixed
68.41%
0.74%
63.27%
4 0.765%
69. 69.40%
0.42%
63.12% 4 0.98%
82.43% 0..8%
62.01%
40.37%
Lyuphography
1.05%
56.15%
1.21%
55,73%
57.99%
Adul
Discrete
72.2%% + 0.06%
60.36% 0.08
70.31% 0.03% 63.30% = 0.14%
60.14% 1..
00.00% +0.01%
Table 4: Accuracy of RIFLE, MICE, KNN-Imputer, Expoctation Maximization (EM), and Robust QDA on different discrete, mixed, and continuous dalasets. Robust QDA can perform better than other met when the input features are continuous, and the target variable is discrete. However, RIFLE results in higher accuracy in mixed and discrete scttings.
kernel space generated by the composition of the original features. A future direction to develop efficient algorithms for non-linear regression models such as multi-layer neural networks, decision tree regressors, gradient boosting regressors, and support vector regression models. In the case of robust classification, the methodology is extendable to any loss beyond quadratic discriminant analysis. Unlike the regression case, limitation of the proposed method for robust classification is its reliance on the Gaussianity assumption of data distribution (conditioned on each data label). A natural extension is to assume the underlying data distribution follows a mixture of Gaussian distributions.
Conclusion: In this paper, we proposed a distributionally robust optimization framework over the distri- butions with the low-order marginals within the estimated confidence intervals for inference and imputation of datasets in the presence of missing values. We developed algorithms for regression and classification with convergence guarantees. The method's performance is evaluated synthetic and real datasets with different numbers of samples, dimensions, missing value proportions, and types of missing values. In most experiments, RIFLE consistently outperforms other existing methods.
Acknowledgmentss
This work was supported by the NIH/NSF Grant 1R01LM013315-01, the NSF CAREER Award CCF- 2144985, and the AFOSR Young Investigator Program Award 229-50-22-1-0192.
17


Reviews:
[{"review_id": "7GtSNlTPQn", "reviewer": "Reviewer_opwB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nMissing data is a prevalent problem, so any methodology to address it has the potential to make a big impact. The proposed methodology is easy to understand, has only a small number of hyperparameters with clear meaning, and the paper adequately discusses ways to set such hyperparameters. \n\nThe writing is of high quality. Claims are supported by evidence. Sections/paragraphs have thesis statements that guide readers on what to take away. The transition between different sections is smooth.\n\nThe figures are very legible: the caption and the types of plots are clear in meaning, and corroborate with the text. \n\nThe proofs in the appendix are easy to verify. \n\n# Weaknesses\n\nThe section on how RIFLE can be used to impute missing values (Section 3.4) is not as strong as the rest of the paper. I believe this section will be strengthened with more exposition. The current paper already utilizes Figure 1 - would it be possible to walk through a concrete example using this Figure 1, with more (but not too many) equations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a methodology, called RIFLE, to handle missing data in supervised learning. The principle idea, Equation 4, is a variant of distributionally robust risk minimization: the uncertainty set is parametrized by mean and covariance conditions. The paper concretely instantiates the framework in Equation 4 to linear model-based hypothesis classes (Section 3 and Section 4), and suggests that the framework can be generalized beyond the linear case. Experiments indicate that the methodology performs favorably compared to existing approaches that handle missing data in supervised learning. For instance, Figure 3 shows that RIFLE has high prediction accuracy even in the presence of missing data, and Figure 4 shows that RIFLE performs accurate imputation of missing values. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The primary change I would like to see is about Section 3.4. Currently I don't understand how Algorithm 1 is used to impute missing values. \n\nSome secondary changes include:\n* Figure 3: change the x-axis to be a log scale since the values are not equally spaced in the linear scale. This is also a problem in Figure 6. \n* Section 5.2 the equation pointed to by the hyperlink (11) doesn't contain the hyperparameter c\n* Reorganize the information in Table 1. I personally find tables hard to read. Perhaps you can convert Table 1 into a boxplot. \n\nTime-permitting / future work\n* Do you have results for linear discriminant analysis? Section 4 explains the methodology, but I don't see numerical demonstrations for this methodology in the main text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "95m2Z9aFWD", "reviewer": "Reviewer_bvAX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nIt seems novel to combine the ideas of distributionally robust optimization and statistical inference for low-order moments. While confidence intervals for individual imputed values may be too wide, the sufficient statistics (i.e., low-order moments) are usually easier to estimate.\n\nWeaknesses:\n\nThe proposed framework has two major weaknesses:\n\n(1) In principle, it can only deal with data missing completely at random (MCAR), which is too restrictive. There are a wide variety of methods, including those based on inverse probability weighting and multiple imputation, that can handle data missing at random (MAR), which are more realistic.\n\n(2) The distributionally robust approach has a price to pay and inevitably incurs an efficiency loss. The theoretical guarantees developed by the paper are quite weak and not sufficient to understand the trade-off between robustness and efficiency.\n\nAlso, Theorems 1 and 2 are not mathematically rigorous and contain many incorrect or ambiguous statements. For example:\n\n(1) Theorem 1: $\\boldsymbol\\theta^*(\\mathbf{b},\\mathbf{C})$ is not defined; it is defined in Appendix H, Lemma 11. The assumption on a constant $\\tau$ is questionable in the ridge regression example when $\\mathbf{b}$ tends to infinity.\n\n(2) Theorem 2, part (a): the limiting statements (8) and (9) do not specify the mode of convergence (e.g., in probability or almost surely). Also, the statements “the estimator proposed in Algorithm 2 converges to its actual value” and “by increasing the number of samples, the size of confidence intervals goes to zero” are not mathematically rigorous.\n\n(3) Theorem 2, part (b): the definition of $L_{\\text{test}}(\\boldsymbol\\theta)$ is problematic; if it is minimized over $\\boldsymbol\\theta$, then it should not depend on $\\boldsymbol\\theta$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper develops a general framework for robust inference with missing data, by combining distributionally robust optimization and bootstrap confidence intervals. More specifically, the optimization problem is formulated with $\\ell_\\infty$ constraints on low-order moments, which are computed from the incomplete data with a bootstrap method. The paper illustrates the framework on two examples, ridge regression and quadratic discriminant analysis. The paper provides some theory and conducts performance comparisons through numerical experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) The authors should rework the theory to make it precise and understandable. Furthermore, stronger theoretical guarantees should be provided, e.g., to quantify the efficiency loss due to distributional robustness.\n\n(2) Extensions to the MAR setting should be discussed.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AKTSa8oYH9", "reviewer": "Reviewer_XaU5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper is clearly written and the approach is straightforward to understand.\n2. The algorithm has good theoretical support; convergence guarantees are provided.\n3. Experimental results show that RIFLE improves over baselines when there is more limited data.\n\n## Weaknesses\n1. I feel too much was relegated to the appendix; some discussion of the classification case in the main text and more experiments for the classification setting would strengthen the work.\n2. There is no discussion of the computational burden of RIFLE, in particular compared to the baselines, which would be of importance to practitioners.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework for prediction with missing values that combines the bootstrap to estimate data distribution characteristics and distributionally robust optimization. Explicit algorithms are given for ridge regression and discriminant analysis with convergence guarantees when the data are missing completely at random. Experiments are done on datasets (mostly from the UCI machine learning repository) with a range of sizes and dimensions. The proposed algorithm outperforms baselines when there is a greater amount of missing data and when the data may be missing not at random.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think some discussion or analysis on the computational burden of the algorithm is critical. ", "broader_impact_concerns": "The authors do not include a broader impact statement, which I think is fine since the work is largely methodological. The authors do not discuss the limitations; I feel that including them would strengthen the work, even if it's just some discussion of future work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the submission provides a strong contribution to the general framework for robust inference with missing data. The claims are supported and all the concerns are well addressed. Thus I recommend acceptance.
Decision: Accept (poster)