Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift
Authors: Liwen Ouyang
Aaron Key
Venue: ICLR 2022 Submitted (2022)
Keywords: maximum mean discrepancy
data shift
covariate shift
representation learning
missing data

Abstract: Covariate shifts are a common problem in predictive modeling on real-world problems. This paper proposes addressing the covariate shift problem by minimizing Maximum Mean Discrepancy (MMD) statistics between the training and test sets in either feature input space, feature representation space, or both. We designed three techniques that we call MMD Representation, MMD Mask, and MMD Hybrid to deal with the scenarios where only a distribution shift exists, only a missingness shift exists, or both types of shift exist, respectively. We find that integrating an MMD loss component helps models use the best features for generalization and avoid dangerous extrapolation as much as possible for each test sample. Models treated with this MMD approach show better performance, calibration, and extrapolation on the test set.

MAXIMUM MEAN DISCREPANCY FOR GENERALIZA- TION IN THE PRESENCE OF DISTRIBUTION AND MISS- INGNESS SHIFT
Anonymous authors Paper under doublc-blind review
ABSTRACT
Covariate shifts are a common problem in predictive modeling on real-world prob- lems. This paper proposes addressing the covariate shift problem by minimizing Maximum Mean Discrepancy (MMD) statistics between the training and test sets in either feature input space, feature representation space, or both. We designed three techniques that we call MMD Representation, MMD Mask, and MMD Hybrid to deal with the scenarios where only a distribution shift exists, only missingness shift exists, or both types of shift exist, respectively. We find that integrating an MMD loss component helps models use the best features for generalization and avoid dangerous extrapolation much possible for each test sample. Mod- els treated with this MMD approach show better performance, calibration, and extrapolation the test set.
INTRODUCTION
Machine learning models often work on the assumption that the training and test datasets follow the same distribution. In practice, this might not be the case, yet the difference in distribution, called dataset shift, is often ignored. This might be acceptable when the dataset shift small, but small scale shift is not always guaranteed. When it is not, it can lead to poor performance on shifted test sets. Dataset shift can be divided into three main categories: covariate shift, prior probability shift, and concept shift. Covariate shift occurs when the distribution of input features changes, prior probability shift occurs when the distribution target variables changes, and concept shift occurs when the relationship between input features and target variables changes Mluinonrro-Caneed et al., 2008). In this paper, we focus on covariate shifts.
There have been some attempts to address covariate shift in previous literature. One straightforward approach is to identify the shifted covariates and drop them from the feature set. However, this can be challenging for high-dimensional data, and one may also lose useful information in the process. A softer method is to use importance re-weighting by assigning higher weights to training samples that similar to the samples in the test dataset. The importance weights can be calculated through various techniques, including density estimation (Shimodaira. 2000; Kanamori & Shimodaira, 2000; Zadrozny, 2004; Dudík et al., 2005), kernel mean matching (Huang et al., 2007). Kullback-Leibler importance estimation (Sugiyama et al., 2008), and discriminative learning (Bickel et al., 2009). These methods work particularly well for sample selection bias, but they treat samples at row/sample level which is not flexible enough.
In this paper, we propose optimizing an auxiliary maximum mean discrepancy (MMD) loss with a mixture of kernels to treat the covariate shift problem. MMD is already widely used in unsupervised domain adaptation (UDA) (Tzeng et al., 2014; Long et al., 2015; 2017; Yan al., 2017: Chen et al., 2020). Here extend it to the more general covariate shift problem. Conceptually, can think of domain adaptation as an extreme case covariate shift as the joint distribution of input variables shifts different domain entirely.
Moreover, previous UDA works has never considered an important special case of covariate shift that involves data missingness. For example, in an internal project of carbon emission prediction in Section 4.3, we try to train model with the companies who reported carbon emission to infer carbon emission for the unreported companies. However, find that the unreported companies usually

---

have higher missing rates in various sets of features compared to the reported companies. A similar situation can happen in other use cases. For instance, various physical measurements and clinical exams need to be taken before diagnostic decision of disease can be made, but when estimating the probability of risk of the disease on a new group of people, different measurements and exams can be missing for different subgroups. Or, in financial fraud detection, the unlaheled set may also have higher feature missing rates compared to the labeled set, especially in the self-reported features. If we simply ignore the data missingness shift, the model performance may be degraded on the unlabeled set. In this paper, we propose building novel masker model to mask training samples using MMD and thereby align the missingness patterns in the training and test datasets. We find that this can help improve the test performance by preventing the model from depending on relationships that will be unavailable at test time, Also, note the MMD masker model is especially relevant when categorical features are involved, as new categories in the test set correspond to missing features in the training set, see Section 4.2 for an example.
Expanding on this, we combine missingness shift treatment and a general feature distribution shift treatment together in hybrid model, and show that the hybrid model is superior through various experiments done with both synthetic data and real data.
2 RELATED WORK
MMD was first introduced in the two-sample tests (Gretton et al., 2007). It is a non-parametric metric to measure the distance between two distributions. Therefore, to compare two distributions, one does not need to know their probability density functions, or PDFs. This is useful as the PDFs can be difficult to calculate or can even be unknown. Because of its simplicity, MMD has been widely applied to various problems, such as goodness-of of-fit testing (Jitkrittum et al., 2018), MMD GAN (Li et al., 2015; Sutherland et al., 2017; Li et al., 2018), and MMD VAE (Zhao et al., 2019).
One technique related to using MMD for covariate shift is using kernel mean matching to re-weight training samples (Huang et al., 2007). In that paper, the authors formulated quadratic problem between empirical means to find suitable weights for training samples, after which they could apply weighted ordinary least squares or support vector machines. The difference is that they tackled the problem from the importance resampling angle and used the results for more traditional machine learning methods. We incorporate MMD directly into a neural network model in one step instead of two by using the representation learning ability of neural networks, and are able to treat covariate shift in a more granular level than the row/sample level if necessary.
The most related application of MMD to our work is in UDA. Deep Domain Confusion (DDC) (Tzeng et al., 2014) is one of the carliest attempts to use MMD for domain invariant learning. Then, Deep Adaptation Networks (DAN) (Long al., 2015) were proposed match an optimal multi-kernel MMD to reduce domain discrepancy. This was followed by Joint Adaptation Networks (JAN) (Long al., 2017), which were proposed to apply joint MMD in multiple domain-specifit layers across domains. Yan et al (2017) proposed adjusting plain MMD by class-specifs auxiliary weights to account for the class weight bias across domains. Chen et al (2020) proposed matching higher-order moments to perform fine-grained domain alignment. Note that these applications all focused on domain shifts in classification problems, while we extend the application to more general distribution shift and to missingness shift that has never been considered before.
3 METHOD
3.1 PRELIMINARY: MAXIMUM MEAN DISCREPANCY
Assume we have two sets of samples X b and Y = {リ, 1 drawn from two distributions P(X) and P(Y). MMD is nar-maramt measure to compute the distance between the two sets of samples in mean embedding space. Let & be measurable and bounded kernel of a reproducing kernel Hilbert space (RKHS) H4 of functions, then the empirical estimation of MMD between the two distributions in Hj can be written
N N
M
M
N
M
(XM))(XXY) = N2
k( F.))
k(ys vy')
k m])
M
NM
E
==
1= 1'=
t= ==1
(3.1)
2

---

When the underlying kernel is characteristic (Fukumizu et al., 2008; Sriperumbudur et al., 2010b), MMD is zero if and only if P(X) = P(Y (Gretton et al., 2012a). For example, the popular Gaussian RBF kernel, k(x,x')
2²19), is characteristic kernel and was widely used in previous literature (Li et al., 2015; Sutherland et al., 2017: Long et al., 2017; Li et al., 201 8). Dziugaite al (2015) compared the RBF kernel with the rational quadratic (RQ) kernel and the Laplacian kernel and found that the RBF kernel performed best. Therefore, in this work, we also used the RBF kernel.
exp(-
What remains is how to select the bandwidth parameter a in the Gaussian RBF kernel, which can affect the hypothesis testing power of MMD measurement. There were some heuristics in previous literature (Sriperumbudur et al., 2010a; Gretton et al., 2012b), but later research (Li et al., 2015; Sutherland et al., 2017: Li et al., 2018) found that using mixture of 5 or more bandwidths performed well, so we followed this procedure as well.
3.2 MMD REPRESENTATION
Let first define the context of our predictive problem. Assume that have training and test features Xtr, Xte E RR" where m is the number of features. We also know the target value y/r for the training set, and we want to build a model predict yte for the test set. When we train the model with (Xtr, Wtr), , a covariate shift between Xtr and X.e can cause the trained model to perform poorly on the test set.
Assume we build a neural network model for the task, then similar to what has been widely used in domain adaptation, we can match the MMD statisties between embt and embie some intermediate embedding layer so that the learned embeddings have similar distributions between the training and test sets and will therefore reduce the amount of extrapolation. We call this approach MMI) Representation. See Figure for the architectural details. Note that models like DAN (Long et al., 2015) and JAN (Long et al., 2017) can be considered as special forms of MMD Representation, as they also apply MMD to multiple specific intermediate layers. The final loss we will optimize is the original task loss plus the MMD loss in the representation space:
L= Ltask + MD (ebbrr embre)
(3.2)
where À is hyperparameter to adjust the ratio between the original loss and the MMD loss. Generally, we found that setting À to match the magnitudes of the two loss terms leads to the best results. Com- pared the approach of dropping features with covariate shift before training, MMD Representation is able to retain as much information as possible from each individual feature and find common feature space that has as little shift as possible between the training and test sets.
X
Algorithm MMD Hybrid
Input: Xtrs Iers Xtes Ites Bers
MMD Mask
MMD Hybrid
Initialize sas, JF, Sp
X
emd
Per
Pred Net
for each epoch do for each batch do Ler < fas (Xrr X + Mask X by I
Feature Extractor
X
emb_.. ce
MMD Representation
Update fas min LMMD ((Xtr: Per), (Xees Ito))
Figure 1: Architecture for MMD Representation, MMD Mask and MMD Hybrid. The two blue dashed boxes contain the architectures for MMD Mask and MMD Representation respectively, and the green dashed box contains the architecture for MMD Hybrid, which combines MMD Mask and MMD Representation.
ember embre + fx(X))) Der +- fp(embrr) Update fr. fe + min Lrash (Der) +
embre)
end for
end for
3

---

3.3 MMD MASK
Despite the advantages of MMD Representation, we found that it is insufficient when missingness shift involved, as was the case for the carbon estimation problem mentioned in Section 1. In general, our MMD Mask can be effective when the missingness rates for some set of features are low or zero in the training set but much higher in the test set. The architecture of MMD Mask is shown in Figure 1. Denote fer and Ite the original missingness indicators for the training and test features X and Xte, respectively; want to learn new missingness indicator It conditioned on Xxr and fr that can minimize the MMD loss between the joint distributions P(X lte) and P(Xte, I
C L (ir), (Xte,
(3.3)
where X2 is generated by masking the original Xxr using learned fr.. The downstream model is then trained using X1 instead of Xtr- In this way, the downstream model can focus on the right features for each training sample. Importance resampling using kernel mean matching also operates in the original feature space, but it removes or downweights each sample as whole (at the row level), while we treat each sample feature by feature. Compared to MMD Representation, another advantage of MMD Mask is that it can still be applied when the downstream model cannot learn intermediate representation (c.g. a tree-based model).
3.4 MMD HYBRID
Furthermore, we can combine MMD Representation and MMD Mask together as a more compre- hensive shift treatment. We call this MMD Hybrid: see Figure for its architecture. Denote the masker model fM, the feature extractor fF, and the prediction network as fp. We use an alternating update method to train the MMD Hybrid model. That is, first update the masker model fa by minimizing the loss in Eq. (3.3) and then update the feature extractor fr and prediction network fr together by minimizing the loss in Eq. (3.2); see Algorithm 1 for details.
EXPERIMENTS
The experiments are designed to answer the following questions: (1) How do MMD Representation, MMD Mask and MMD Hybrid work with the presence of distribution shift and missingness shift in features respectively? Which MMD model performs best in which scenario? (2) Do MMD Representation, MMD Mask and MMD Hybrid work in real world datasets generally? How do they perform compared to other existing methods? intermediate representation, do any of the MMD models still work?
(3) When the downstream model cannot learn an
To answer the questions above, we tested our approaches on both synthetic data and real data. The real data is comprised of Bike Sharing dataset from the UCI Machine Learning Repository!, an IEEE-CIS Fraud Detection dataset from Kaggle², an internal project of carbon estimation, and the image dataset MNIST (LeCun al., 1998).
For all experiments except carbon estimation and MNIST, the baseline model is multilayer per- ceptron (MLP) model. MMD Representation always shares the same architecture, but also matches MMD statistics at the last hidden layer. MMD Mask outputs masks for each feature per sample; we then mask out training samples based on the learned masks, and use that to train the same downstream baseline model. MMD Hybrid combines MMD Representation and MMD Mask. For synthetic data, the Bike Sharing data and the IEEE-CIS Fraud Detection data, the baseline models take features and missing indicators inputs, and the missing values were imputed using the means from the training set. We didn't 't impute missing values for carbon estimation because it used a based model that can handle missing values inherently. For all datasets,
use MMD statistics under mixture of Gaussian RBF kernels with bandwidths of {1, 2, 4, 8, 16, 32}. The MMD Mask model generate mask probability
uses Relaxed Bernoulli distribution parametrized by temperature for each feature. The Relaxed Bernoulli is continuous approximation to Bernoulli distribution over (0, 1), similar to the Gumbel-Softmal trick (Jang et al., 2017) for categorical distributions. In all experiments, we set to 0.1 at the start of training and then gradually decrease it to 0.01. We
4

---

implemented all models in PyTorch. More details about model architectures and hyperparameters can be found in Appendix A.
For all experiments, whenever possible, we also compared our methods with kernel mean matching (KMM) (Huang et al., 2007), and UDA baselines like DAN (Long et al., 2015) and JAN (Long et al., 2017). As for KMM, we followed the procedure in their Empirical KMM optimization Section, and found the best weights for training samples by solving the quadratic problem using cvxopt package', then used these weights to reweight samples in the original loss function. Whenever DAN and JAN are used, we apply MMD or Joint MMD to the last two hidden layers as MLPs in this paper are only three to four hidden layers.
4.1 SYNTHETIC DATA
In this experiment, we used synthetic data to gain a general understanding of which MMD models work best in which situations. We con- structed a synthetic dataset as shown in Fig- ure 2. There are two input features X, and X2, both having some predictive power for the target y. The relationship between feature X1 and y is linear while the relationship be- tween X2 and y is parabolic. We injected noise a (O 0.1) into feature X, and larger-scale noise 62 (0, 0.5) into feature X2. Based on this alone, the model should prefer using feature X1 to feature X2- However, we also engineered two types of data shifts into feature X1 between the training and test datasets. We first resampled from the training dataset with replacement to get our test dataset such that both the training and test datasets had 5,000 rows. Then in the test dataset, we designed a missingness shift for X₁ in the range [-3,5, by masking it out, and a distribution shift for Xı in the range (0, 3.5] by subtracting it by e ~ (1, 0.1) see Figure 2. Therefore, the test dataset has both missingness and distribution shifts compared to the training dataset. We hope that properly designed model can learn predict using feature X₂ instead of X1 in the regions of X, affected by the shifts.
Datahotice Are
Misingooe
12
0
X
X
Figure 2: Synthetic data: plots of target y v.s. fea- ture X1 and X2 respectively. Distribution shift of X1 occurs in blue shaded area and missingness shift of X1 occurs in pink shaded area.
Table I shows the average and standard deviation of mean squared error (MSE) for each model from 10 runs on the test set. As we can see, the performances of baseline model and KMM are similar. This is expected because KMM is an importance re- re-weighting technique that downweights training samples different from test samples at row level while what we hope to see is that a method can learn to ignore X, but use X2 for those shifted training samples. On the other hand, all MMD methods improve the performance on the shifted test set to some degree. Among the methods to match the intermediate representations, DAN and JAN perform better than MMD Representation as they try to match more intermediate layers. But their performance Note that the performance of MMD Hybrid is very close to the golden model that is the true mean function used to generate this synthetic data.
not good as the MMD Hybrid model.
Figure 3 gives more details of how each model performs in different ranges of X, and X2 on the test data. We focus on the comparison among the basic baseline model, the MMD Representation model, the MMD Mask model and the MMD Hybrid Model here. The performance details for the KMM, DAN and JAN can be found in Appendix B.
In the region where data shift does not exist (areas without shades), all models learn the relationship very well and have very small residuals. In the region where missingness shift exists (areas with prink shades), since X1 is masked out in the test dataset, we focus on residuals vs. X². Although the MMD Representation model already reduced residuals significantly compared to the baseline model, it still has higher residuals in this region compared to the MMD Mask model and the MMD Hybrid model. This indicates that when there is only missingness shift, MMD Mask is sufficient as it is a natural solution to missingness shift. In the region where distribution shift exsits (areas with blue shades), we see that the bascline model is worst, followed by MMD Mask, MMD Representation and
hups://evxopt.org
5

---

Dattilbution
Missingness
Residuais w//
Baseline
MMD REPR
MMD MASK
MMD IVDRID
000
1000
100
4
-2
-2
X1
X1
X1
X
(a) residuals V.s. X1
Base
MMD REPR
MMD MASK
MMD IVVRIDD
Fooldud
000
000
00000009000
000
to000
00g
to ooo!
Rogoo
X2
X2
X2
X2
(b) residuals V.S. X2
Figure 3: Synthetic data: residuals feature Xı and X2 test set from the baseline model, MMD Representation model, MMD Mask model and MMD Hybrid model. The heatmap shows the sample frequency each small cell. The black circles and bars represent the average and standard deviation of residuals in each bucket of Xz/X2. Distribution shift of X1 occurs in blue shaded area and missingness shift of X1 occurs in pink shaded area.
Table Averages and standard deviations of performance metrics (MSE for synthetic data, RMSE for the Bike Sharing and AUC for the IEEE-CIS Fraud Detection) of 10 runs on test data for each model.
MODEL
SYNTHETIC DATA
BIKE SHARING
FRAUD DETECTION
BASELINE
17.682 + 9.911
116.2 + 3.9
86.29% 0.59%
KMM
17.666 + 9.854
116.5 5.2
N.A.
DAN
0.753 0.698
106.3 2.3
85.91% 0.79%
JAN
0.820 + 0.693
107.4 2.2
86.30% 0.54%
MMD REPR
2.303 + 1.373
107.7 + 1.7
86.44% 0.48%
MMD MASK
2.573 + 1.119
106. + 7.0
87.25% 0.19%
MMD HYBRID
0.331 + 0.201
98.7 + 3.8
87.22% + 0.32%
GOLDEN MODEL
0.180 1 N.A.
N.A.
N.A.
MMD Hybrid, in order. This indicates that MMD Mask can treat some distribution shift, but not as well as MMD Representation.
Furthermore, we checked the embeddings in the last hidden layer from the training and test phases for each model. Figure 4 shows 2-D (SNE (van der Maaten Hinton, 2008) plots embeddings from the baseline model, MMD Representation, MMD Mask and MMD Hybrid. As expected, in all MMD approaches, the learned embeddings between the training and test sets are more similar to each other than in the baseline model. Among all approaches, MMD Hybrid learned the most consistent embeddings between the training and test sets. Appendix has more embedding results for KMM, DAN and JAN, none of which is better than MMD Hybrid.

---

DAT
-20
-10
-10
-29
-10 -5 -5
-26-5 -50 23
-10
-23
50
(a) Bascline
(b) MMD Representation
(c) MMD Mask
(d) MMD Hybrid
Figure tSNE plot of embeddings from last hidden layer from each model on synthetic data.
Finally, we examined the masks generated by MMD Mask and MMD Hybrid. Figure 5a shows histograms of features X1 in the original training set, the test set, and the masked training set. We focused on X1 here because there is no shift in X2. As we can see, the masker in MMD Mask and MMD Hybrid learned to mask out the shifted range both the positive and negative sides some degree. Since the negative side has purely missingness shift, it learned better by masking out more samples for X1 E 3.3,5 0j. while masked out less samples for X1 E (0, 3.5). The masking ability is consistent with the downstream performance we observed.
4.2 BIKE SHARING AND IEEE-CIS FRAUD DETECTION
We use the Bike Sharing as the regression ex- ample and the IEEE-CIS Fraud Detection as the classification example show how our MMD models perform in general. The Bike Sharing dataset (Fanaee-T & Gama, 2013) from UCI contains the hourly and daily bike rental counts in the Capital bikeshare system in Washington, D.C., USA for 2011 1 and 2012. We focused on hourly data it provides more samples. The dataset contains features like hour, month, sea- son, workday, holiday and some weather infor- mation. We log- transsorffrr the target variable (bike rental counts) due to their long tail, and deliberately selected 201 1-03 to 2011-11 as the training data (6,567 rows) and 201 to 2012- 03 as the test data (2,917 rows) so that the fea- tures have shifts between the training and test data due to the different times of year. Also, since there are some new months in the test set (December, January and February), after ordinal feature processing, they are treated missing in the prediction stage, which can be viewed as source of missingness shift.
36-
031
(a) synthetic data
(h) bike sharing data
Figure 5: Comparison of masked training data generated by MMD Mask and MMD Hybrid with original training and test data. (a) Synthetic data: histograms for feature X; in original train, test and masked train. (b) The Bike Sharing data: fre- quency percentage of each month in original train, test and masked training data. -1 represents the new months appearing test set.
The IEEE-CIS Fraud Detection dataset is from Vesta's real-world-e wwrld-e-commeree transactions and contains a wide range of features from device type to product features. The goal is to estimate fraud probability of the unlabeled test set. We followed the champion model from Chris Deotte and Konstantin Yakovlev and used all the features except 219 Vesta engineered rich features that were determined redundant by their correlation analysis. but did not conduct further feature engineering like they did, as our goal is not to outperform the best model but use this dataset to show how our proposed MMD models can handle distribution and missingness shift in the features and outperform the baseline models. In the end, we used 212 features before one hot encoding the categorical features and adding missing indicators. Missingness shift is observed in some of the 212 features to some degree, but not severe.
Table 1 shows the averages and standard deviations of the performance metric (RMSE for the Bike Sharing and AUC for the IEEE-CI CIS Fraud Detection) test data of 10 runs for each model. KMM performance is not available for IEEE-CIS Fraud Detection as it is infeasible to solve the quadratic problem using evxopt package when feature dimension is high and sample size is large. For the Bike Sharing data, KMM does not improve the performance compared to the baseline. We think the

---

fundamental reason is that importance re-weighting techniques such KMM assume the support of test data is contained in the support of training data. However, this is not true for this test data. As we can see, all MMD approaches improved the model performance for both datasets. Because the Bike Sharing dataset contains both distribution shift and missingness shift, we see that the MMD Hybrid model works best, while DAN and JAN have similar performance compared to MMD Representation and MMD Mask. As for IEEE-CIS Fraud Detection data, we see MMD Representation style methods (MMD Representation, DAN, JAN) have comparable performance to the baseline. MMD Mask and MMD Hybrid perform better than others as they can handle missingness shift better.
We also examined if the masks learned for the month feature in the Bike Sharing dataset meet our expectation. As we can see from Figure 5b, the maskers learn to leave more of March 2011 1 data in while masking out data from other months more, which matches with our expectation, as March is the only common month in both training and test data. Note that the maskers try to match the joint distribution of all features, not just the marginal distribution of months.
The results on the two real world datasets are consistent with our findings in synthetic data. We also tested our methods on more unstructured data like MNIST to show they also work on complex image data. In the case where occlusion is correlated with features/labels on image data, MMD Mask and MMD Hybrid can work more efficiently compared to random data augmentation to learn the correct masks use on the train images. Due to space limits, we leave the experiment details and results in Appendix C.
4.3 CARBON EMISSION ESTIMATION
The experiments in this section are designed to answer question (3). Carbon Emission Estimation is an internal project that seeks to estimate companies' carbon emissions using various sets of features including environmental, social and governance (ESG) data, industry segmentation data, fundamental financial data, and country and regional data. In total, there are about 1,000 features. One challenge of the project is the missingness shift between labeled and unlabeled data. The labeled data usually has more complete feature information, while unlabeled data has many missing values in different sets of features. As a result, a model trained on the labeled data would generate poor results on the unlabeled data due to the much larger missing rates in various features. We could not take the approach of dropping all features that are sometimes missing as nearly all features in this dataset can be missing. To overcome this challenge, we propose using data augmentation where we generate masks for the labeled data to mimic the missingness patterns in unlabeled data.
In this case, the downstream predictive model is a tree-based model without representation learning ability, so only MMD Mask can be applied here. We trained a masker model to generate masks for labeled data based on all available features by matching the MMD statistics between the labeled and unlabeled data. As bascline, we also tried using a simple mask model that randomly sampled masks from unlabeled data conditioned on industry. The downstream carbon estimation model was then trained on the masked labeled data with 10 masks sampled from the mask model plus the original training data. We denote MMD Model as the downstream model trained on data masked by masks from the MMD Mask model, and Simple Model the downstream model trained data masked by masks from the simple mask model.
To evaluate the results, we designed a series of tests. First, we created a masked dataset by carrying over masks for companies from a year in which they did not report carbon emissions to following year in which they did. This is the most realistic set of masks we can get for the labeled data and we considered it the golden set. Figure 6a shows the percentage error of prediction from the MMD Model and the Simple Model on this cark-cerry set. As we can see, the MMD Model has a large performance gain over the Simple Model.
We also tested the performance of the MMD Model and Simple Model on data masked by both MMD masks and simple masks. We sampled 10 masks from the corresponding mask model and applied those to the original labeled data, then compared predictions from the downstream model trained with two different versions of masked data. Figure 6b compares the percentage error of prediction for each combination. As we can see, the MMD Model always beats the Simple Model whether the data is masked by MMD mask or the simple mask. And when the original labeled data is masked by MMD mask the performance gain is even larger. The fact that the MMD Model performs better than the Simple Model even the data masked by the simple mask is interesting. It indicates that the
8

---

00%
89%
00%
635
47%
40%
deceral 4807
dnggee rbigrao
coppesl (BIL
aboeed AX0
(a) mask carryover
(b) MMD/simple masked
(c) unmasked data
(d) masked data
Figure 6: arbon emission estimation: cumulative percentages for the percentage error of prediction. (a) MMD Model / Simple Model performance on mask-carryover set. (b) MMD Model / Simple Model performance on MMD Mask/simple mask masked data. (c) MMD Model / Simple Model performance on unmasked data when a set of feature completely missing. (d) MMD Model / Simple Model performance on masked data when a set of feature is completely missing.
missingness pattern from the simple mask is in a sense contained in the missingness pattern from MMD mask.
In addition, we tested the MMD Model and Simple Model in the specific scenarios that an entire set of features is missing, as this can be the case for certain sub-universes of companies. Figure 6c and Figure 6d plots the scenarios on unmasked data and masked data respectively. Since ESG data is much more useful for estimating carbon emissions than industry segmentation data is, the performance decreases more when ESG data is missing. However, no matter which set of features entirely missing, the MMD Model always performs better than the Simple Model. Also, if we compare the curves for unmasked data with those for masked data, we can see that most of them very similar except for the Simple Model with Segmentation data missing. The Simple Model performance on masked data is a bit worse compared to unmasked data when Segmentation data is completely missing. However, MMD Model performance masked data remains similar that unmasked data, indicating that MMD masks are more aligned with the scenarios when an entire set of features is missing.
5 CONCLUSION AND FUTURE WORK
We have demonstrated through various experiments that optimizing MMD statistics works well to build a more robust model when there are shifts between the training and test datasets. More importantly, proposed novel MMD masker model to match the joint distribution of input features and missingness indicators in input space between the training and test datasets, an approach designed specifically for treating missingness shift, which has never been considered in previous UDA works. We have shown that MMD Representation works better for distribution shifts while MMD Mask works better for missingness shifts on both synthetic data and real data. Furthermore, we combined MMD matching in the feature embedding space and in the raw input space to yield superior results.
With the current approaches introduced in this paper, the learned model is calibrated to a specific test set. We think that MMD has the potential learn a more generalized model that is robust for different sets of test data if we combine this technique with meta-learning approaches. This is out of scope for this paper and an interesting area for future work to explore. However, in scenarios where proposed MMD approaches help the model learn to use the
one has target test set to label, best features in both the input and embedding spaces and avoid dangerous extrapolation as much as possible, and result, achieve better performance the test set.
REFERENCES
K. Bickel, M. Brückner, and Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research, 10:2137-2155, 2009.
C. Chen, Z. Fu, S. Jin, Z. Cheng, X. Jin, and X. Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 2020), pp. 3422-3429, Palo Alto, CA, USA, 2020. AAAI Press.
M. Dudík, R. E. Schapire, and S. Phillips. Correcting sample selection bias in maximum entropy density estimation. Advances in Neural Information Processing Systems, 18:323-330, 2005.

---

G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Marina Meila and Yom Heskes (eds.), Proceedings of the 31st Conference Uncertainty in Artificial Intelligence (UAI 2015), pp. 258-208-022 Arlington, Virginia, USA, 2015. AUAI Press.
H. Fanaee-T and Gama, Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, pp. 1-15, 2013. ISSN 2192-6352. doi:
10.1007/113748-0113-0000.3
K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. Kernel measures of conditional dependence. In Bernhard H Schölkopf, John C Platt, and T. Hoffman (eds.), Proceedings of the 20th Neural Information Processing Systems (NIPS 2007), pp. 489-496, Red Hook, NY, USA, 2008. Curran Associates Inc.
A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the ewo-sample-problee In Bernhard H Schölkopf, John C Platt, and T. Hoffman (eds.), Proceedings of the 19th Neural Information Processing Systems (NIPS 2006), pp. 513-520, Cambridge, MA, USA, 2007. MIT Press.
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012a.
A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances in Neural Information Processing Systems, pp. 1205-1213, 2012b.
J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Scholkopf. Correcting sample selection bias by unlabeled data. In Bernhard Schölkopf, John Platt, and T. Hoffman (eds.), Proceedings of the 19th Neural Information Processing Systems (NIPS 2006), pp. 601-608, Cambridge, MA, USA, 2007. MIT Press.
E. Jang, S. Gu, and B. Poole. Categorical reparameterizzttoon with gumbel-softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), 2017. URL https: /awxvv..ivvv//////////e..//a.av org/ abs 1611. 00144.1114..
W. Jitkrittum, W. Xu, Z. Szabó, K. Fukumizu, and A. Gretton. A linear-time kernel goodness-of-fe test. In Ulrike von Luxburg, Isabelle Guyon, Samy Bengio, Hanna Wallach, and Rob Fergus (eds.), Proceedings of the st Neural Information Processing Systems (NIPS 2017), pp. 261 -270, Red Hook, NY, USA, 2018. Curran Associates Inc.
Kanamori and H. Shimodaira. Active learning algorithm using the maximum weighted log- likelihood estimator. Journal of Statistical Planning and Inference, 116:149-162, 2000.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-bard based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-2324, 1998.
C. Li, W. Chang, Y. Cheng, Y. Yang. and B. Póezos. Mmd gan: Towards deeper understanding of moment matching network. In Ulrike von Luxburg, Isabelle Guyon, Samy Bengio, Hanna Wallach, and Rob Fergus (eds.), Proceedings of the 31st Neural Information Processing Systems (NIPS 2017), pp. 2200-2210, Red Hook, NY, USA, 2018, Curran Associates Inc.
Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 1718-1727, Lille, France, 2015. JMLR.org.
M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 97-105. Lille, France, 2015. JMLR.org.
M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning (ICML 20/7), pp. 2208-2217, Sydney, Australia, 2017. PMLR.
J. Quinonero-Candee M. Sugiyama, A. Schwaighofer, and N. D. Lawrence (eds.). Dataset Shift in Machine Learning. The MIT Press, Cambridge, MA, USA and London, England, 2008
10

---

H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90:227-2444 2000.
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, and G. R. Lanckriet. Kernel choice and classifiability for rkhs embeddings of probability distributions. In Yoshua Bengio, D. Schuurmans, J D Lafferty, Christopher Kenneth Ingle Williams, and Aron Culotta (eds.), Proceedings of the 22nd Neural Information Processing Systems (NIPS 2009), pp. 1750-1758, Red Hook, NY, USA, 2010a. Curran Associates Inc.
B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. Lanckriet. Hilbert space embeddings and metries probability measures. Journal of Machine Learning Research, 11: 1517-1561, 2010b.
M. Sugiyama, S. Nakajima, H. Kashima, P. V. Bünau, and M. Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Bernhard H Schölkopf, John Platt, and T. Hoffman (eds.). Proceedings of the 20th Neural Information Processing Systems (NIPS 2007). Pp. 1433-1440, Red Hook, NY, USA, 2008. Curran Associates Inc.
J. Sutherland, H. Tung, H. Strathmann, S. De, A. Ramdas, J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. Proceedings of the 5th International Conference on Learning Representations (ICLR 20/7), 2017.
E. Tzeng. J. Hoffman, Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv::142.344 2014.
I.P.P. van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
H. Yan, Y. Ding, P. Li, Q. Wang. Y. Xu, and W. Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR 2017), pp. 2272-2281, Los Alamitos, CA, USA, 2017. IEEE Computer Society.
B. Zadrozny. Learning and evaluating classifiers under sample selection bias. In Russ Greiner and Dale Schuurmans (cds.), Proceedings of the 21st International Conference Machine Learning (ICML. 2004), pp- 114-122, New York, NY, USA, 2004. Association for Computing Machinery.
S. Zhao, J. Song, and S. Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 5885- Palo Alto, CA, USA, 2019. AAAI Press.
Z. Zhong, L.. Zheng, G. Kang, S. Li, and Y. Yang. Random erasing data augmentation. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 2020), pp. 13001-13008, Palo Alto, CA, USA, 2020. AAAI Press.

---

A MODEL ARCHITECTURES AND HYPERPARAMETERS DETAILS
For synthetic data, the baseline model an MLP with 3 hidden layers with a hidden size of 16 for each. The MMD Mask model has hidden layers with hidden sizes of {32, 32, 203. For the MMD Representation and MMD Hybrid models, we also tested different hyperparameter values E f1,5, 10} and picked the model that performed best. In all models, we used ReLU our activation function and optimized network parameters using RMSProp with learning rate of 0.01. All models were trained for 5,000 epochs with full batch.
For the Bike Sharing, the baseline model is an MLP model with 4 hidden layers with hidden size of {64, 64, 64, 64), and the MMD Mask model has 3 hidden layers with hidden sizes of (512, 512, 128), For the IEEE-CIS Fraud Detection, the bascline model is an MLP model with 4 hidden layers with hidden size of {1024, 512, 512, 256]. and the MMD Mask model has hidden layers with hidden sizes of {512, 512, 256). For both datasets, network parameters using RMSProp. For the MMD Representation and MMD Hybrid models, we also tested different hyperparameter values À E {1, 10, 100}. The labeled data is split into 3:1 as train v.s. validation. The hyperparameters are tuned based on the performance on the validation set.
used ReLU as activation function and optimized
For carbon emission estimtion, the MMD Mask model is an MLP model with 4 hidden layers with hidden size of {512, 512, 512, 512), with ReLU activation. We trained the model for 300 epochs with batch size of 5000, using RMSProp with a learning rate of 0.001.
B MORE RESULTS ON SYNTHETIC DATA
In main sections, we took scrutiny of MMD Representation, MMD Mask and MMD Hybrid in regions with different types of shift. Here we also include similar analysis for KMM, DAN and JAN. From Figure 7, we can see that KMM, DAN and JAN also have very small residuals in the region where data shift does not exist. When missingness shift or distribution shift exists, performance of all
Vill
Distribution Shift Area
NWWS Missingness Shif. Area
Residuals w/ std bar
DAN
JAN
KMM
40
000
DD
D
DD
2 Φ
$
60
-0
-6
X1
X1
residuals X1
Distribation Shift Area
N
Missingness Shill Area
Residuds mean w/ std bac
DAN
JAN
KMM
400
+ $0000
D
$ 2000o
$00
22
d
-4
0000
-8
X₂
X2
Xa
(b) residuals v.s. X
Figure 7: Synthetic data: residuals v.s. feature X, and X₂ test set from the KMM, DAN and JAN. The heatmap shows the sample frequency in each small cell. The black circles and bars represent the average and standard deviation of residuals in each bucket of X,/X,. Distribution shift of X, occurs in blue shaded area and missingness shift of X1 occurs in pink shaded area.
12

---

101
110
30
-00
70
310
-000
-76
-30
-6
36
76
100
-100
-75
-25
55
-100
-15
-50
-35
30
esild
KMM
(b) DAN
(c) JAN
Figure 8: (NNE plot of embeddings from last hidden layer from each model synthetic data.
methods deteriorates different degrees. KMM has similar performance compared to the Baseline model both missingness shift and distribution shift regions. In the region where distribution shift exists, DAN and JAN work better than MMD Mask, which is expected. DAN and JAN also work better than MMD Representation as they try to match more intermediate layers, and work similarly with MMD Hybrid. However, in the region where missingness shift exists, DAN and JAN are not as good as MMD Mask and MMD Hybrid, but similar to MMD Representation.
Further. we checked the embeddings in the last hidden layer from the training and test phases for KMM, DAN and JAN as well. Figure 8 shows 2-D tSNE plots of embeddings from each model. There is very clear separation between training and test embeddings from KMM, similarly to the baseline model. DAN and JAN both make the training and test embeddings more similar to each other, but still not as similar as those generated by our MMD Hybrid.
C MNIST EXPERIMENTS
In this section, we investigate how our MMD methods perform on image data. Our proposed methods can be useful when occlusion is correlated with features/labels on image data. For example, say you have intact images of groundhogs, squirrels and pangolins with labels in training set. But in the unlabeled set, the images groundhogs and pangolins are incomplete as those are taken when animals are partially in ground. So we need to learn appropriated masks to mask out right images in training set in order to learn a more accurate and robust classification model.
Table 2: The model architecture for the baseline model in MNIST experiments. outputs the logits of digits class.
IMAGE I MXMII
3 3, stride = 1 CONV 16 RELU
2 x 2, MAXPOOL
3 x 3, stride = 1 CONV 32 RELU
2 x 2, MAXPOOL.
To demonstrate our approaches also work on high dimensional image data, we used MNIST which is handwritten digits dataset where each image is 28 x 28 pixels. It has 60,000 training images and 10,000 test images. To demonstrate that our methods work when there a shift, particularly missingness shift, we created a patterned mask on the original test data. That is, for digits in 11,4,57 7), we mask out the last rows of pixels, and for digits in 2,, 6, 9), we mask out the last 12 columns of pixels.
We compared MMD Representation, MMD Mask, and MMD Hybrid approaches with a baseline model trained raw training data, golden model trained with the training samples masked by true masks, as well as traditional data augmentation. All models in this section adopt CNN architectures. The baseline model in MNIST experiments uses the architecture shown in Table 2. The golden model and two variants of data augmentation models also use the exactly same architecture. The MMD Representation uses the same architecture except that also matches the embeddings between
13

---

Table The model architecture for the MMD Mask MNIST experiments. It outputs the logits of mask for each pixel.
IMAGE & RMxxx
3 3, stride = CONV 16 RELU
3, stride CONV RELU
3 stride
32 RELU
3 stride = CONV 32 RELU
DENSE
4 4, stride = DECONV 32 RELU
3 3, stride = DECONV 32 RELU
4 4, stride = DECONV 16 RELU
stride DECONV 16 LINEAR +
Table 4: Mean and standard deviation of test accuracy from 10 runs for MNIST.
MODEL
Acc MEAN + STD
BASELINE
0.873 + 0.030
RANDOM ERASING VI
0.934 0.008
RANDOM ERASING V2
0.950 0.004
MMD REPR
0.959 + 0.005
MMD MASK
0.972 + 0.002
MMD HYBRID
0.972 + 0.001
GOLDEN MODEL
0.987 + 0.001
9
2
3
C
3
C11
/
/
3
6
2
8
6
7
6
3
1
7
4
/
7
1
/
c(
9
2
3
C/
(a) Original training samples
(b) Masked training samples
Figure 9: Original training samples v.s. the training samples masked by the MMD Hybrid model.
training and test set before entering the final dense layer. The MMD Mask model generates masks using a conv-decony architecture shown in Table 3. generates masks to mask original training data, then the masked training data goes through the same downstream model. The MMD Hybrid model combines MMD Representation and MMD Mask. The missing values were imputed as zero (black pixel).
For traditional data augmentation, we tested two versions of RandomErasing (Zhong et al., 2020), with one version always erasing the same area (12 x 28) of images at random place and the other erasing a random area ranging from to the same area (12 x 28) at a random place. We followed the standard procedure to split the training samples into training set of 54,000 rows and a validation set of 6,000 rows, and trained all models for 20 epochs with a batch size of 64 (except for the MMD Mask model, which trained for 600 epochs with larger batch size of 10,000). The hyperparameter
14

---

in the MMD Representation and MMD Hybrid model was set to 1. The final model in all methods was the one that gave the best validation loss.
The results summarized in Table 4. The performance reported is the average test accuracy from 10 runs for each model. Note that this bascline model can achieve around 98% test accuracy on the original test dataset, but only around 87% aceuracy the corrupted test dataset. However, MMD Representation, MMD Mask, and MMD Hybrid can all improve the performance by a large margin, 95.9%, 97.2%, and 97. 7. accuracy, respectively. Because corrupted the test by applying patterned masks, MMD Mask is a more natural treatment compared to MMD Representation. Since MMD Hybrid combines MMD Representation and MMD Masks, works as well as MMD Mask in this case. All three methods performed better than the two traditional data augmentation methods: these achieve accuracies of 93.4% 95.0%. We believe that this simply because the learned transformation is more effective and efficient compared to more random transformations. The golden model achieves 98.7% aceuracy, which is not too far away from performances from the best MMI) approaches. Figure 9 shows some samples of generated masks applied to the original training set, which are quite close to the true masks we imposed on the test set.
15


Reviews:
[{"review_id": "l3Tl6jeJj8B", "reviewer": "Reviewer_4NFR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Considering both distribution shifts and missingness is an important direction of research that can improve practical performances of modern machine learning models.\n- The paper is easy to follow and well-structured.\n- The method provides improved results on small benchmark datasets. \n\nCons:\n- Authors limit their scope in the situation that they are given all test inputs at the training phase. But this seems to be too restricted scope, considering the majority of the community works on robust machine learning models that can cope with novel distribution shifts patterns. \n- Regarding this point, I also have a concern about potential data snooping problems. Specifically, using the test inputs to regularize the feature during training seems to be analogous to using them in the semi-supervised learning loss functions. For this reason, I suggest authors compare their methods with several semi-supervised learning methods such as MixMatch and VAT. In current form, all experiments put too much favor to MMD-based models compared to other baseline.  \n- The authors did not use an approximate MMD distance, so the proposed method has limited scalability. Also, due to the implementation of MMD in the paper, the authors compare all methods using full-batch training, which is highly unlikely in practice and standard benchmark problems. For this reason, I’d recommend authors to add the baseline trained with standard training schemes involving mini-batch SGD.\n\n", "summary_of_the_paper": "This paper considers distribution shift and feature missingness problems. To solve this, the authors minimize MMD distance between training and test samples. Also, to make representation robust to novel missing patterns in test sets, the authors propose to learn making operations so that the resulting representation has small MMD distance. Authors verify their methods in synthetic dataset and small benchmark problems.\n", "summary_of_the_review": "The paper has a limited applicability to large-scale problems. Considering the modern methods and tricks has already dealt with the large-scale problems on a more diverse set of distribution shifts (such as Hendrycks 2019 ICLR), the results in the paper seem not significant. Also, the experiments can be performed more precisely if they involve more proper baselines.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hrmzLtQuzHQ", "reviewer": "Reviewer_jTXA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem, proposes an interesting approach and I enjoyed reading this work. Furthermore, it pushes the idea of thinking about the distributions of feature representations under covariate-shift and of an end-to-end learning workflow (vs. common two-step approaches in the liteature). However, I also have some concerns, which I outline below.\n\n\n# Related work on missing data:\nAt least half of the contribution of the paper pertains to supervised learning with missing data, yet no connection to existing approaches is made. Some example recent references include the following (also cf. the work of Prof. Julie Josse and statistical literature on missingness more broadly) \n\n> Smieja, M., Struski, Ł., Tabor, J., Zieliński, B., & Spurek, P. (2018, December). Processing of missing data by neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (pp. 2724-2734).\n\n> Ipsen, N., Mattei, P. A., & Frellsen, J. (2020, July). How to deal with missing data in supervised deep learning?. In ICML Workshop on the Art of Learning with Missing Values (Artemiss).\n\n> You, J., Ma, X., Ding, D. Y., Kochenderfer, M., & Leskovec, J. (2020). Handling missing data with graph representation learning. arXiv preprint arXiv:2010.16418. \n\nI think it is important to put this work into this broader context of research on missing data problems. Furthermore, in any discussion of missing data it is important to outline what missing data one anticipates (or what type of missing data the method would perform well for), is it data missing at random, is the missingness informative, etc.\n\n# Evidence for the performance of proposed approaches:\n\nAfter reading the paper, I do not know when I would try to use this approach in practice. The reason is that this work does not explain either (let's say by formal theoretical results, or heuristics), when the method is expected to perform well, and when existing approaches would be preferable (also see the point above about assumptions on missingness). Also since the only evidence provided is in terms of empirical results, I think these would need to be more comprehensive to provide insight to a reader (e.g. more ablation studies, what is the impact of learning model architectures, more settings, etc).\n\n\n# Details of method and reproducibility:\n\nWhile reading the paper, I often felt lost. I do not think I would be able to reproduce the results based on just the description in the text. I think this could be alleviated by providing more explicit detail. Some concrete questions I had:\n\n* I would have liked some more detail about the methods used to generate the masking scheme. What happens if the mask specifies that an entry that is currently missing in the training set should be unmasked? I thought for some time that maybe the assumption is that the training set has no missing data (only the test set has), but this seems not to be the case in Section 4.3. Is the assumption that there is a stochastic dominance with strictly more missingness in the test data? How do the models predict with the masked data? Is missing data treated as a \"zero\" by e.g. the linear combinations of the neural net layers?\n\n* \"Generally we found that setting $\\lambda$ to match the magnitudes of the two loss terms leads to the best results\": Could you elaborate on this step?\n\n* Algorithm 1 box: Are the update steps minimization steps? Or gradient updates? Is there a typo in the update step for $f_M$, i.e., should there be $(\\mathbf{X}'\\_{tr}, \\hat{\\mathbf{I}})$ instead of  $(\\mathbf{X}\\_{tr}, \\hat{\\mathbf{I}})$?\n\n* Simulation: Could you provide more detail on the data generation? Is the model an additive model, namely the sum of the univariate responses shown in terms of $X_1$ and $X_2$? How are $X_1$ and $X_2$ generated? I thought they would be independent, but this contradicts Figure 3, where the range in which $X_1$ is missing also corresponds to a deterioration in residuals in terms of $X_2$.\n\n", "summary_of_the_paper": "Suppose we are given training data  $(\\mathbf{X}_{tr}, \\mathbf{I}\\_{tr},  \\mathbf{Y}\\_{tr})$, where $\\mathbf{Y}\\_{tr} \\in \\mathbb R^{n\\_{tr}}$  is the response of interest, $\\mathbf{X}\\_{tr} \\mathbb R^{n\\_{tr} \\times p}$ is the feature design matrix ($p$ features, $n\\_{tr}$ samples) and  $\\mathbf{I}\\_{tr} \\in  \\\\{0,1\\\\}^{n\\_{tr} \\times p}$ is an indicator matrix that marks which features are missing in $\\mathbf{X}\\_{tr}$. In the setting studied in this paper, the researcher also has access to the test samples  $(\\mathbf{X}\\_{te}, \\mathbf{I}\\_{te})$ where $\\mathbf{X}\\_{te} \\in \\mathbb R^{n\\_{te} \\times p}$ and $\\mathbf{I}\\_{te} \\in  \\\\{0,1\\\\}^{n\\_{te} \\times p}$ are the features of the test samples and their missingness indicator. The goal is to predict the response of the test samples while accounting for the potential complications of covariate-shift and a shift in the missingness pattern between training and test set covariates.\n\nThe authors propose three approaches to this general problem. \n\n1) MMD-Representation to account for covariate-shift (ignoring missingness): here one learns a feature representation $f_F$ such that the distribution of $f_F(x\\_{tr})$ and  $f_F(x\\_{te})$ are similar, where $x\\_{tr}$ is a feature vector drawn from the training distribution and $x\\_{te}$ from the test distribution. Closeness here is measured in terms of the maximum mean discrepancy of Gretton and colleagues (2007). Then one learns a predictive model $f_P$ so that the predictions take the form $f_P(f_F(x))$.  The two models $f_P$ and $f_F$ are learned simultaneously by minimizing a weighted combination of the predictive training loss and the MMD loss between test and train featurization.\n\n2) MMD-mask to deal with varying missingness patterns: here on learns a masking model $f_M$ and applies it as $\\hat{\\mathbf{I}} = f_M(\\mathbf{X}_{tr}, \\mathbf{I}\\_{tr})$ to learn a missingness pattern that matches more closely the missingness pattern of the test dataset. Then one applies the learned missingness pattern on the training samples to get modified training samples $\\mathbf{X}'\\_{tr}$ with missingness pattern $\\hat{\\mathbf{I}}$. The masking model is learned by minimizing the maximum mean discrepancy between the distributions of  $(x'\\_{tr}, \\hat{i}\\_{tr})$ and $(x\\_{te}, i\\_{te})$. Finally one learns the predictive model based on  $(\\mathbf{X}'\\_{tr}, \\hat{\\mathbf{I}}, \\mathbf{Y}\\_{tr})$.\n\n3) MMD-hybrid: a combined approach of MMD-mask and MMD-representation to account for both shifts in the missingness pattern and covariate-shift. \n\nThese three approaches are benchmarked and compared in one synthetic dataset and two applications (Bike Sharing and IEEE-CIS Fraud Detection).\n\n\n\n\n", "summary_of_the_review": "The paper addresses an important topic and is interesting. Nevertheless, at this time I vote to reject it for the following reasons:\n\n* There is not enough intuition/theory regarding situations in which the method is expected to perform well compared to existing work, nor are the empirical results sufficiently detailed.\n\n* Related work on supervised learning with missing data is completely omitted.\n\n* The authors would need to provide additional detail to facilitate the reproducibility of the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_BVYO6QLjWa", "reviewer": "Reviewer_x6L8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n- The paper considers model adaptation under different missing patterns in the source and target domains and/or covariate shift.\n- Proposed algorithm based on MMD and intensive empirical evaluations\n\n*Weaknesses*\n- The paper lacks a formal definition of the problem and proposed solution. For instance, several works were carried out on missing data imputation under various settings (missing at random, missing completely at random...) and are not discussed in the paper. The MMD mask model is not formaly defined.\n\n- The proposed method lacks theoretical analysis on the test error bound when adapting the model.\n\n- The paper overlooks a closedly related work [1] which is more elaborated and more principled. In my opinion, compared to [1] the proposed contribution is less significant. \n\n\n*References*\n\n[1] Kirchmeyer, Matthieu, et al. \"Unsupervised domain adaptation with non-stochastic missing data.\" Data Mining and Knowledge Discovery (2021): 1-42.\n\n*After rebuttal*\n\nNo feeback was provided by the authors. Hence, the review and the paper evaluation remain the same.", "summary_of_the_paper": "This work proposes a solution dealing with model adaptation under covariate shift and/or missingness shift  by minimizing the Maximum Mean Discrepancy (MMD) between the training labeled set and unlabeled test set. The method is supported by several evaluations on synthetic data and real data.", "summary_of_the_review": "The paper proposes a methodological framework to study DA under missingness and covariate shifts. It lacks a clear formulation, a theoretical analysis and it overlooks existing methods. Hence, the contribution is not significative. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UPBU8eAmuzg", "reviewer": "Reviewer_UfTS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is easy to follow. The proposed methodology to address both missingness and distributional shift makes sense. Effectiveness of the proposed methodology has also been validated in real-world datasets.  \n\nHowever, I am still worried about the novelty of the proposed methodology. In fact, use MMD to match latent distributions is not novel at all. As authors have mentioned, similar or almost the same strategy has been used in previous literature like DAN (Long et al., 2015) and JAN (Long et al., 2017). \n\nLearnng a new mask on training data to mimic the missingness patterns in test data seems an interesting idea. However, I still feel this contribution is marginal. In practice, I only have one question: how to properly set the hyperparameters (like $\\lambda$) of both MMD mask and MMD representation in the joint loss function? \n", "summary_of_the_paper": "Authors suggest a new MMD based methodology to improve generalization in case of both distributional shift and missing data. For the missing data, authors propose to generate a new mask (or missingness indicator) on the training data to mimic the missingness patterns in the test data. The closeness of missing patterns in training and test data can be measured with MMD. ", "summary_of_the_review": "My major concern is the novelty of this work. Aligning distributions of latent representations by MMD has been extensively investigated before. It seems to me the only technical contribution is to generate a mask that mimics the missing patterns in test data. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses unsupervised domain adaptation under covariate shift and missing source and target features. Three approaches are proposed for tackling respectively covariate shift, missing data and simultaneous covariate shift and missing data. The proposed method relies on the minimization of the maximum mean discrepancy between the source and target representations in the different settings. Experiments are performed on a synthetic dataset and on two other datasets.

All the reviewers highlighted several weaknesses: lack of formal definitions and of formal analyses, lack of connection with existing approaches for handling missing data, weak reproducibility. The authors did not provide responses. Reject.
Decision: Reject