In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
Authors: Sang Michael Xie
Ananya Kumar
Robbie Jones
Fereshte Khani
Tengyu Ma
Percy Liang
Venue: ICLR 2021 Poster (2021)
Keywords: pre-training
self-training theory
robustness
out-of-distribution
unlabeled data
auxiliary information
multi-task learning theory
distribution shift

Abstract: Consider a prediction setting with few in-distribution labeled examples and many unlabeled examples both in- and out-of-distribution (OOD). The goal is to learn a model which performs well both in-distribution and OOD. In these settings, auxiliary information is often cheaply available for every input. How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt OOD error; but (ii) using auxiliary information as outputs of auxiliary pre-training tasks improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.

Published as a conference paper at ICLR 2021
IN-N-OUT: PRE-TNAINING AND SELF-TRAINING USING AUXILIARY INFORMATION FOR OUT-OF- DISTRIBUTION ROBUSTNESS
Sang Michael Xie; Ananya Kumar, Rones, Fereshte Khani, Tengyu Ma, Percy Liang Stanford University (xie, ananya, rmjones, fereshte, tengyuma, pliang ) (cs. stanford.
ABSTRACT
Consider prediction setting with few in di-drribuution labeled examples and many unlabeled examples both in- and aut-of-distrbuution (OOD). The goal is to leam a model which performs well both in in-distribution and OOD, In these settings, auxiliary information is often cheaply available for every input. How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, show that (i) using auxiliary information input features improves in-distribution error but can hurt OOD error; but (ii) using auxiliary information as outputs of auxiliary pre-training tasks improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains model with auxiliary inputs and uses it to pscudolabel all the in mn-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in in-distribution and OOD error.
INTRODUCTION
When models tested on distributions that are different from the training distribution, they typically suffer large drops in performance (Blitzer and Pereira (2077] Szegedy et al. [0044]Jr Jia and Liang 2017] AlBadawy et al.) 2018 Hendryeks et al. (019a). For example, in remote sensing, central tasks include predicting poverty, crop type, and land cover from satellite imagery for downstream humanitarian, policy, and environmental applications (Xie et al. 2016 Jean et al. 2016]] Wang et al.) 2020 Rußwurm et al. a In some developing African countries, labels are scarce due to the lack of economic resources deploy human workers conduct expensive surveys (Jean et 2016). To make accurate predictions in these countries, we must extrapolate to out-of-distribution (OOD) examples across different geographic terrains and political borders.
We consider a cuni-supervised setting with few in-distribution labeled examples and many unlabeled examples from both in- and out-of-distributie (e.g., global satellite imagery). While labels are scarce, auxiliary information is often cheaply available for every input and may provide some signal for the missing labels. Auxiliary information can come from additional data sources (e.g., climate data from other satellites) or derived from the original input (e.g., background or non-visible spectrum image channels). This auxiliary information is often discarded or not leveraged, and how to best use them is unclear. One way is to use them directly as input features (aux-inputs); another is to treat them as prediction outputs for auxiliary task (aux-outputs) in pre-training Which approach leads to better in-distribution or OOD performance?
Aux-inputs provide more features to potentially improve in-distribution performance, and one may hope that this also improves OOD performance. Indeed, previous results on standard datasets show that improvements in in-distribution accuracy correlate with improvements in OOD accuracy (Recht et al. [219 Taori al.) 2020 Xie et al. 2020 Santurkar et al., (220)) However, in this paper find that aux- aix-inputs can introduce more spurious correlations with the labels: as a result, while aux-inputs often improve in-distribution accuracy, they can worsen OOD accuracy. We give examples of this
trend on CelebA (Liu et al. 2015) and real-world satellite datasets in Sections 5.2 and 5.3
Conversely, aux-output methods such as pre-training may improve OOD performance through auxiliary supervision (Caruana 1997 Weiss et al. 2016 Hendrycks al. 201 2019a). Hendrycks et al.
"Equal contribution.

---

Published as a conference paper at ICLR 2021
Labels (9)
Unlabcled aux (Zxz)
Labels (P)
Pscudelabells (Aux-inn Nen 2gg
Bly
C
Aus-in model
Aux-our model
initialize
In-N-Out
8*
In in-diitribution unlabeled data (X,)
Data (X)
Auxiliary info (2)
Unlabeied data (Nar)
Data (X)
Figure 2: Graphical
model for our theoret- ical setting: prediction task with input x, target y, and auxiliary information z, which is related to y through the latent variable w and latent noise
Step Training (aux-in)
Step Pre-training (sux-out)
Step 3: Fine-tuning scll-training
Figure 1: sketch of the In-N-Out algorithm which consists of three steps: 1) use auxiliary information as input (Aux-in) to achieve good in-distribution performance, 2) use auxiliary information output in pre-training (Aux-out), to improve OOD performance, 3) fine-tune the pretrained model from step 2 using the labeled data and in in-distribution unlabeled data with pseudolabels generated from step to improve in- and -iitt---dditiittionn.
(2019a) show that pre-training on ImageNet can improve adversarial robustness, and Hendrycks et al. (2019b) show that auxiliary supervision tasks can improve robustness synthetic corruptions. In this paper, we find that while aux aux-outputs improve OOD accuracy, the in-distribution accuracy is worse than with aux-inpun uux-inpus. Thus, elucidate tradeoff between in- and diitr--dittiitttinn accuracy that occurs when using auxiliary information as inputs or outputs.
To theoretically study how to best use auxiliary information, we extend the multi-task linear regression setting (Du et al., (2020) Tripuraneni et al. 2020) allow for distribution shifts. We show that auxiliary information helps in-distribution error by providing useful features for predicting the target, but the re- lationship between the aux-inputs and the target can shift significantly OOD, worsening the OOD error. In contrast, the aux outpuuss model first pre- pre-trains on unlabeled data to learn lower-dimensional repre- sentation and then solves the target task in the diveeddimnsional space. We prove that the aux-outputs model improves robustness to arbitrary covariate shift compared to not using auxiliary information.
Can we do better than using auxiliary information as inputs or outputs alone? We answer affirmatively by proposing the In-N-Out algorithm to combine the benefits of auxiliary inputs and outputs (Figure I] In-N-Out first uses uux-inputs model, which has good in-distribution accuracy, pseudolabel in-distribution unlabeled data. It then pre-trains a model using aux-outputs and finally fine-tunes this model on the larger training set consisting of labeled and pseudolabeled data. We prove that -ntN-Out, which combines self fiff-rraning and pre-training. further improves both in-distribution and OOD error over the aux-outputs model.
We show empirical results on CelebA and two remote sensing tasks (land cover and cropland prediction) that parallel the theory. On all datasets, In-N-Out improves OOD accuracy and has competitive or better in-distribution accuracy over aux-inputs or aux-outputs alone and improves 1-2% in- in-distribution, 2-3% OOD over not using auxiliary information remote sensing tasks. Ablations of In-N-Out show that In-N-Out achieves similar improvements over pre-training or self-training alone (up to 5% in on-distribution, 1-2% OOD on remote sensing tasks). We also find that using OOD (rather than in-distribution) unlabeled examples for pre-training is crucial for OOD improvements.
SETUP
Let E Rd be the input (c.g., satellite image), E R be the target (e.g., crop type), and z E RT be the cheaply obtained auxiliary information either from additional sources (e.g., climate information) or derived from the original data (e.g., background).
Training data. Let Pia and Para denote the underlying distribution of (zr,g.z) triples in-distribution and cattof-diitributioo respectively. The training data consists of (i) in-distribution labeled
data {(x), y/, }=) ~ Pl- (ii) in-distribution unlabeled data {( zid ) ~ Pls and (iii)
out-of ditto---istriuttionn unlabeled data {[2.0d .2 ) Pond-
Goal and risk metrics. Our goal is to learn model from input and auxiliary information to the target. Rd x R + R. For loss function E, the in-distribution population risk of the model f is Raa(f) En.., Pi (I),,,))))) and its OOD population risk is Rosa( Er.gg.--Pad ( (x)))
2

---

Published as a conference paper at ICLR 2021
2.1 MODELS
We consider three common ways to use the auxiliary information (z) to learn a model.
The baseline minimizes the empirical risk on labeled data while ignoring the auxiliary information (accomplished by setting z O):
Baseline.
f argmin Σ e(f((x)))))))
(1)
Aux-inputs. The aux-inpus model minimizes the empirical risk on labeled data while using the auxiliary information as features:
fm arrarmmnn T1
((())))))))
The aux-outputs model leverages the auxiliary information by using it as the prediction target of an auxiliary task, in hopes that there is low-dimensional feature representation that is common to predicting both and Training the aux-outputs model consists of two steps:
Aux-outputs.
In the pre-training step,
all the unlabeled data learn shared feature representation. Let b. +R* denote feature map and Iz-out R - BR denote a mapping from feature representation the auxiliary outputs. Let lax denote the loss function for the auxiliary information. We define the empirical risk of and Gz-out as:
Rpre( (2-00-))))
Laux (rr-our (1( 1)
Laxx (gr-our (h(=our((x; (h(x:11))) (), ))22(())
(3)
Mia + Mood
The estimate of the feature map is how = argmin, groa Rpre (h.gr..un..
In the transfer step, the model uses the pre-trained feature map hear and the labeled data to learn the mapping gy-out R2* > IR from feature representation to target We define the transfer empirical risk as:
T2
Rizans (hrans( (Trutt By van)
(gg out (hea (1r)))
The estimate of the target mapping is gy-out argmin Mysn Raans (hout Jy-oaa). The final aux-outputs model is
Pout (Ix=)) By-oom (z),
(5)
Like the baseline model, the aux-outputs model ignores the auxiliary information for prediction.
3
THEORETICAL ANALYSIS OF AUX-INPUTS AND AUX-OUTPUTS MODELS
We now analyze the baseline, aux-inputs, and aux-outputs models introduced Section Our setup extends linear regression setting commonly used for analyzing multi-task problems (Du et al. 2020 Tripuraneni 2020).
Setup. See Figure for the graphical model. w Brr E R be low-dimensional latent feature (k shared between auxiliary information 2 and the target y. Let E Rm denote unobserved latent variables not captured in æ. We assume and y are linear functions of u and w:
y y= w0 ute,
(6)
2 A" ww+ C*u,
(7)
where e Pe denotes noise with mean and variance o2. As in Du et al. (0220). we assume dimension of the auxiliary information T is greater than the feature dimension k. that is T> k. and that A", B and C* have full rank (rank k:). We also assume T > m, where m is the dimension of u.
Data. Let Ps and P. denote the distribution of and u in-distribution (ID), and let P P denote the distribution and OOD. We assume and are independent, have distributions with bounded density everywhere, and have invertible covariance matrices. We assume the mean of u is in-

---

Published as a conference paper at ICLR 2021
and and unlimited access to unlabeled data both ID and OOD, a common assumption in unsupervised
We assume we have m + in-distribution labeled training examples
domain adaptation theory (Sugiyama et al. 2007 Kumar et al. 2020 Raghunathan et al.) 2020).
Loss metrics. We use the squared loss for the target and auxiliary losses: ((g.v) = (y ;) and lanx (2,,2)==1=====/1)
Models. We assume all model families (f, h, Gr-out: 9y-our) in Section 2 are linear.
Let B "rr,,C',ww Ow Pxo Pu.Pu) denote a problem setting which satisfies all the above assumptions.
3.1 AUXILIARY INPUTS HELP IN- AN-DITTRIBUTIO,, BUT CAN HURT OOD
We first show that the aux-inputs model (2 performs better than the baseline model (I) in-distribution. Intuitively, the target y depends on both the inputs z (through w) and latent variable u (Figure 2 The baseline model only uses x to predict y: thus it cannot capture the variation in due to u. On the other hand, the aux-inputs model uses x and to predict y. Since is a function of (through w)) and u, u can be recovered from x and 2 by inverting this relation. Note that u is unobserved but implicitly recovered. The aux-inputs model can then combine and x to predict y better.
oet [(OT, ) denote the (in-distribution) variance of y due to the latent variables u. The following proposition shows that if o > then with enough training examples the aux-inputs model has lower in-distribution population risk than the baseline model
Proposition 1. For all problem settings S, Pa. assuming regularity conditions (bounded v, sub-Gaussian noise e, and m., and o > for all 8> 0, there exists N such that N number of training points, with probability at least 1 8 over the training examples, the aux-inputs model improves over the baseline:
<
Although using 2 as input leads to better in in-distribution performance, we show that the aux-inputs model can perform worse than the baseline model OOD for any number of training examples. Intuitively, the aux-inputs model uses 2, which can be unreliable OOD because 2 depends on u and u can shift OOD. In more detail, the aux-inputs model learns to predict -= 2 in O in 2, where the true output y 0= ++ and 0z is approximation to the true parameter 0z2 that has some error. u and hence a can have very high variance, which would magnify (B..in (f..m-00)) T
and lead to bad predictions.
Example There exists a problem setting S. P.. such for every n, there some test distribution Pr, Pr with:
Raoo (in) E, (
3.2 PRE-TRAINING IMPROVES RISK UNDER ARBITRARY COVARIATE SHIFT
While using as inputs (aux-inputs, can worsen performance relative to the baseline, our first main result is that the aux-outputs model (which pre-trains to predict from xr, and then transfers the learned representation to predict y from 2:) outperforms the baseline model for all test distributions.
see that the mapping from inputs to auxiliary & passes through the lower dimensional features w. In the pre-training step, the aux-outputs model predicts from using a low rank linear model, and we show that this recovers the 'bottleneck' features w (up to symmetries; more formally we recover the rowspace of B*). In the transfer step, the aux-outputs model learns a lincar map from the from xr. To warm up, without distribution shift, the expected excess risk only depends on the dimension of the input, and not the conditioning. That is, the expected excess risk in linear regression exactly do²/n, //, where d is the input dimension, so the aux-outputs trivially improves over the baseline since dim(w) < dim(2), (z). In contrast, the worst case risk under distribution shift depends the conditioning of the data, which could be worse for w than x. Our proof shows that the worst case risk (over all x and u) is still better for the aux-outputs model because projecting to the low-dimensional feature representation "ztroes-out" some error directions.
Intuition. Referring to Figure
w to y. while the baseline predicts y directly
'This is not limiting because bias in can he folded into xr *Since is typically low-dimensional and a is high-dimensional (e.g., images), the aux-inputs model needs only a slightly larger number of examples hefore outperforms the haseline.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 In-N-Out
Require: in-distribution labeled data (/.. Jin ~ Pat.
in-distribution unlabeled data { ((x) ,46 )1:=1 ~ Pia.
OOD unlabeled data {(=]" : Jiil
god) Madd ~ Pond
N== 22) 2: Pre-train Iroou o how + 2 on aux- oux-ouputs from all unlabeled data {x; (2)
1: Learn (x,z)) from in-distribution labeled data ((
Ps
) j Uf( Darn
3: Return f= hom : trained on labeled and pseudolabeled data ((mr.)) R= U((a fin( 1i
Theorem 1. For all problem settings S, noise distributions P., test distributions Pa Pa and n 2 +d number of training points:
E [ Roodd(f(s))
(10)
See Appendix for for the proof.
4 IN-N-OUT: COMBINING AUXILIARY INPUTS AND OUTPUTS
We propose the In-N-Out algorithm, which combines both the aux-inputs and aux-outputs models for further complementary gains [. As reminder: (i) The aux-inputs model (æ,2 -) is good in-distribution, but bad OOD because 2 can be misleading OOD. (ii) The aux-outputs model (x +x+ is better than the baseline OOD, but worse than aux- inxuinpus in in-distribution because it doesn't 't use 2. (iii) We propose the In-N- -ut model (x-+z) y), which uses pseudolabels from aux-inputs (stronger model) in-distribution to transfer in-distribution accuracy to the aux-outpure model. The In-N-Out model does not use to make predictions since can be misleading/ / spurious OOD.
In more detail, we use the aux-inputs model (which good in-distribution) to pseudolabel in-distribution unlabeled data. The pseudolabeled data provides more effective training samples to fine-tune aux-outputs model pre-trained on predicting auxiliary information from all unlabeled data. We present the general In-N-Out algorithm in Algorithm (aani analyze in the linear multi-tase regression setting of Section the empirical risk on labeled and pseudolabeled data:
2
The In-N-Out model = g hout optimizes
9 = argmin(1 Purans (how 9)) Ra
(11)
where Ry Ra(hhansmmm))) 1 E.T (ig (xx))) .2 is the loss of self-training on pseudolabels mi from the aux-inputt model, and E [0,1] is a hyperparameter that trades off between labeled and pseudolabeled losses. In our experiments, we fine-tune and hoor together.
Theoretical setup. Because fine-tuning is difficult analyze theoretically, we analyze a slightly modified version of In-N-Out where we train an aux-inputs model to predict y given the features (x) and auxiliary information z, so the aux-inputs model Óin R² x R - R is given by In argmin, 1 ((ghouu (x)s) The population self- eeff-trainin loss on pseudolabels from the aux- aux-inputs model Ôin o hout is: Rat (hout, (in.9) = Ez, (6(g (hout (o)((())) (x)) (n) ( and we minimize the self- self-training loss: 9 = argmin, Rog (hoott At test time given input x,, the In-N-Out model predicts ((houe())) For the theory, we assume all models (Gm..gaandhoo are linear.
4.1
IN-N-OUT IMPROVES OVER RRE-TRAINING UNDER ARBITRARY COVARIATE SHIFT
We prove that In-N-Out helps on top of pre- pre-training, long as the auxiliary features give information about y relative to the noise in in-distribution that is, if o is much larger than o
To build intuition, first consider the special case where the noise x² 0 (equivalently, can be recovered from and z, we can write y as linear function of w and 2: y y= You w + z. We train an aux- aux-inputs model 9ia from w.2 to y on finite labeled data. Since there is no noise, 9tn predicts y perfectly from w.z (we learn "hu and 72). We use Òin to pseudolabel a large amount of unlabeled data, and since 9m predicts y perfectly from w,z, the pseudolabels are perfect. So here pseudolabeling gives us a much larger and correctly labeled dataset to train the In-N-Qur N-Out model on.
0). Since
The technical challenge is proving that self saff-rraning helps under arbitrary covariate shift even when the noise is non-zero (a > 0), so the aux-inputs model Oin that learn is accurate but not perfect.
5

---

Published as a conference paper at ICLR 2021
CelebA
Cooplnd
Landover
Aus nout
Baseine
000 accunacy
76
Vienalization
74-
90
91
92 distribuution fon accuracy
93
94
bisary
Vegetation, LazLon
Meseonological
Target (y)
Malotiomale?
Cooplandinnt cropland?
Land cless
ID-Syli
People
IA.
Oussale
Figure 4: Correlation (r 0.72)
OOD
People with hats
Africa
between in-distribution accuracy and OOD accuracy when adding to 15 random auxiliary inputs in CelebA
Figure 3: Summary of the datasets used our experiments.
In this case, the pseudolabels have an error which propagates to the In-N-Out model self-trained on these pseudolabels, but we want to show that the error is lower than for the aux-outputs model. The error in linear regression is proportional to the noise of the target y, which for the aux-outputs model is o² +o. We show that the In-N-Out model uses the aux- inpuuts model to reduce the dependence on the noise oi because the aux aux-inpus inputs model uses both w and to predict y. The proof reduces to showing that the max singular value for the In-N-Out error matrix is less than the min-singular value of the aux-outputs error matrix with high probability. core part of the argument min -ingular value of a random matrix (Lemmal3 3) This uses techniques from random matrix theory (see e.g., Chapter 2.7 in Tao 2012)); the high level idea is to show that with probability 1 ô each column of the random matrix has a (not too small) component orthogonal all other columns.
lower bound the
Theorem 2. In the linear setting, for all problem settings S with o > O. test distributions Pa, Z m d number of training points, and 6> O, there exists a.b > 0 such that for all noise distributions Pe. with probability at least 1 5 over the training examples and test example x' ~ Pd, the ratio of the excess risks (for all o² small enough that - bo 2
> 00) is:
Reped Rood
R
(12)
R
a bb
Here R mmigg" .h-
[[[g] (x'))./) is the min. possible (Bayes-optimal) OOD risk, Pepndar = Bu Pi (h (x') ())))'))))))) is the risk of the In-N-Out model on test example x', and Rood = Eyg g' @y ( (oou) (x))) )) is the risk of the aux-outputs model on test example x'. Note that Rend and Rood are random variables that depend on the test input x' and the training set X.
Remark 1. As a + the excess risk ratio of In-N-Out Aux-outputs goes to so the In-N-Out estimator is much better than the aux-outputs estimator.
The proof of the result is in Appendix A
EXPERIMENTS
We show on real-world datasets for land cover and cropland prediction that aux-inputs can hurt OOD performance, while aux-outputs improve OOD performance. In-N-Out improves OOD accuracy and has competitive or better in-distribution accuracy over other models on all datasets (Section Secondly, we show that the tradeoff between in-distribution and OOD performance depends on the choice of auxiliary information on CelebA and cropland prediction (Section 5.3). Finally, we show that OOD unlabeled examples are important for improving OOD robustness (Section55.4
5.1 EXPERIMENTAL SETUP
We give a summary considered datasets and setup here see Figure and Appendix Bor for details. Our datasets use auxiliary information both derived from the input (CelebA, Cropland) and from other sources (Landcover).
CelebA. In CelebA (Liu et al. 2015), the input x is a RGB image (resized to 64 x 64), the target y is binary label for gender, and the auxiliary information are (of 40) binary- attributes derived from the input (e.g., presence of makeup, beard). We designate the set of images where the celebrity is wearing a hat as OOD. We use a R8 as the backbone model architecture for all models (see Appendix B.T for details).

---

Published as a conference paper at ICLR 2021
CelebA
Cropland
Landcover
Test Acc
OOD Ace
ID Test Acc
OOD Ace
ID Acc
OOD Test Acc
Baseline
90.46 0.85
72.64
1.39
94,50 1 0.11
90.30
75.92 1 0.25
58.31 = 1.87
Aux-inputs
92.36 -: 0.29
77.4 1.33
95.24
84.15
76.58
0.44
54.78
2.01
Aux-outputs
94.0 1 0.24
77.68
1: 0.59
95.12.1.15
0.221
72.48
1 0.37
61.03 = 0.97
In-N-Out (no pretrain)
93.8 + 0.56
78.54 1.31
04.9
91.25+0.61
76.54
+ 0.23
59.19 0.98
In-N-Out
93.42 0.36
0.70
0.16
9.574
77.43
4 0.39
61.53 0.74
In-N-Out repeated ST
93.76 1 0.46
80.38
1 0.68
95,53 0.19
92.18 0.40
77.10 1 0.30
62.61 0.58
Table 1: Accuracy (%) of various models using auxiliary information as input, output, or both.
In-N-Out generally improves both in- and out-of-distribttion over aux- aux-inputs or aux aux-outputs alone. Results are averaged over trials with 90% intervals. Repeated ST refers to one round of repeated self-training on top of In-N-Out.
Cropland. Crop type or cropland prediction is an important intermediate problem for crop yield prediction (Cai et al. 2018) Johnson et al., 2016 Kussul et al.) 2017). The input x is a 50 x 50 RGB image taken by satellite, the target y is binary label that is when the image contains majority cropland. and the auxiliary information vegetation-related bands. The vegetation bands in the auxiliary information 2 is derived from the original satellite image, which contains both RGB and other frequency bands. We use the Cropland dataset from Wang et (2020), with data from the US Midwest. We designate lowa, Missouri, and Illinois as in-distribttion and Indiana and Kentucky as OOD. Following Wang et al. (2020), we use U-Net-based model (Ronneberger et al. 2015). See Appendix B.2 for details.
the center location coordinate plus 50 50
Landcover: Land cover prediction involves classifying the land cover type (e.g., "grasslands") from satellite data a location (Gislason et al. 2006; Rußwurm et al. 2020 ). The input z is a time series measured by NASA's MODIS sate llte [Vermote]|22 2013), the target y is one of 6 land cover classes, and the auxiliary information z is climate data (c.g., temperature) from ERA5, a dataset computed from various satellites and weather station data (C3S 2017). We designate non-African locations as in-distribution and Africa as OOD, We use a ID-CNN to handle the temporal structure in the MODIS data. See Appendix B.3 for details.
Data splits. We first split off the OOD data, then split the rest into training. validation, and in-distribution test (see Appendix B for details). We use a portion of the training set and OOD set as in-distribution and OOD unlabeled data respectively. The rest of the OOD set is held out test data. We run 5 trials, where we randomly re- re-generate the training/unlabeled split for each trial (keeping held-out splits fixed). We use a reduced number of labeled examples from each dataset (1 1%, 5%, 10% of labeled examples for CelebA, Cropland, and Landcover respectively), with the rest unlabeled.
Repeated self-training. In our experiments, we also consider augmenting In-N-Out models with repeated self self-training, which has fueled recent improvements in both domain adaptation and ImageNet classification (Shu al. 2018 Xie et al.) (2020). For one additional round of repeated self- celf-training. we use the In-N-Out model to pseudolabel all unlabeled data (both ID and OOD) and also initialize the weights with the In-N-Out model. Each method is trained with sarly-stopping and hyperparameters are chosen using the validation set.
5.2 MAIN RESULTS
Table comppaes the in-distribution (ID) and OOD accuracy of different methods. In all datasets, pre- training with aux-outputs improves OOD performance over the baseline, and In-N-Out (with or without repeated ST) generally improves both in- and -ittt--distribution performance over all other models.
CelebA. In CelebA, using auxiliary information either aux-inputs outputs improves both ID (2-4%) and OOD accuracy (5%). We hypothesize this is because the auxiliary information is quite robust. Figure4sshow that there is a significant correlation 0.77) between ID and OOD accuracy for 100 different sets of aux- muxx-inputs, inputs, supporting results on standard datasets (Recht et al. 2019 Xie ct al. 2020, /Snturrar et al. 2020). In-N-Out achieves the best OOD performance and comparable ID performance even though there is no tradeoff between ID and OOD accuracy.
Remote sensing. In the remote sensing datasets, aux-inputs can induce tradeoff where increasing ID accuracy hurts OOD performance. In cropland prediction, even with a small geographic shift (US Midwest), the baseline model has significant drop from ID to OOD accuracy (4%). The aux-inputs model improves ID accuracy almost 1 1% above the baseline but OOD accuracy drops 6%. land cover prediction, using climate information as aux-inputs decreases OOD accuracy by over 4% compared to the baseline. The aut-outsuts model improves OOD, but decreases IDD accuracy by 3% over the baseline.
7

---

Published as a conference paper at ICLR 2021
0.0 009-0000
Ace
76
Only in-distribution
69.73 = 0.51
57.73 = 1.58
Only ODD Both
69.92 0.41
59.28
70.07 = 0.46
59.84 = 0.98
74
Table 2: Ablation study on the use of in- distribution vs. OOD unlabeled data in pre-training models Landcover, where unla- beled sample size is standardized (much smaller than Table D Using OOD unlabeled examples are important for gains in OOD accuracy (%). Results are shown with 90% crror intervals over 5 trials.
I--29 botton accurddy
Figure 5: In-distribution vs. OOD accuracy on CelebA when sequentially adding random set of 15 auxiliary inputs one-by-one. Even if adding all 15 auxiliary inputs improves both in-distribution and OOD accuracy, some intermediate in-distribution gains can hurt OOD.
Improving in-distribution accuracy over aux-outputs. One of the main goals of the self- self-training step In-N-Out is to improve the in-distribution performance of the aux-outputs model. We compare to oracle models that use a large amount of in in-distribution labeled data to compare the gains from In-N-Out. In Landcover, the oracle model which uses 160k labeled ID examples gets 80.5% accuracy. In-N-Out uses 16k labeled examples and 150k unlabeled ID examples (with 50k unlabeled OOD examples) and improves the ID accuracy of aux-output from 72.5% to 77.4%, closing most (62%) of the gap. In Cropland, the oracle model achieves 95.6% accuracy. Here, In-N-Out closes 80% of the gap between aux out-uutputs and the oracle, improving ID accuracy from 95.1 1% to 95.5%.
Ablations with only pre-training or self-training. We analyze the individual contributions of self- training and pre-training in In-N-Out. On both cropland and land cover prediction, In-N-Out outper- forms standard self- teff-training on pseudolabels from the aux- aux-inputs model (In-N-Out without pre-training), especially on OOD performance, where In-N-Out improves by about 1% and 2% respectively. Similarly, In-N-Out improves upon pre-trate veetrrriiing (aux-outputs model) both ID and OOD for both datasets.
5.3 CHOICE OF AUXILIARY INPUTS MATTERS
We find that the choice of auxiliary inputs affects the tradeoff between ID and OOD performance significantly, and thus is important to consider for problems with distribution shift. While Figure 4 shows that auxiliary inputs tend simultaneously improve ID and OOD accuracy in CelebA, our theory suggests that in the worst case, there should be auxiliary inputs that worsen OOD accuracy. Indeed, Figure shows that when taking random set of 15 auxiliary inputs and adding them sequentially as auxiliary inputs, there are instances where an extra auxiliary input improves in-distribution but hurts OOD accuracy even if adding all 15 auxiliary inputs improves both ID and OOD accuracy. In cropland prediction, we compare using location coordinates and vegetation data as auxiliary inputs with only using vegetation data. The model with locations achieves the best ID performance, improving almost %% in-distribution over the baseline with only RGB. Without locations (only vegetation data), the ID accuracy similar to the baseline but the OOD accuracy improves by 1.5%. In this problem, location coordinates help with in-distribution interpolation, but the model fails to extrapolate to new locations.
5.4 OOD UNLABELED DATA IS IMPORTANT FOR PRE- TRAINING
We compare the role of in-distribution vs. OOD unlabeled data in pre-training. Table shows the results of using only in-distribution vs. only OOD vs. a balanced mix of unlabeled examples for pre-training on the Landcover dataset, where unlabeled sample size is standardized across the models (by reducing to the size of the smallest set, resulting in 4x less unlabeled data). Using only in-distribution unlabeled examples does not improve OOD accuracy, while having only OOD unlabeled examples does well both in nn-distribution and OOD since it also has access to the labeled in-distribution data. For the same experiment cropland prediction, the differences were not statistically significant, perhaps due to the smaller geographic shift (across states in cropland vs. continents in landcover).
RELATED WORK
Multi-task learning and weak supervision. Caruana and de Sa (2003) proposed using noisy fea- tures outxuuttpuss) multi-task output, but do not theoretically analyze this approach. Wu et al.) (2020) also study multi-task linear regression. However, their auxiliary tasks must have true parameters that are closely aligned (small cosine distance) to the target task. Similarly, weak supervision works assume access to weak labels correlated with the true label (Ratner et al. 2016 2017). In our paper,
8

---

Published as a conference paper at ICLR 2021
make no assumptions about the alignment of the auxiliary and target tasks beyond shared latent variable while also considering distribution shifts.
Transfer learning, pre-training, and self-supervision. We support empirical works that show the
success of transfer learning and pre-training in vision and NLP (Krizhessky et al. 2012 Simonyan and Zisserman 2015 Devlin et al. 2019). Theoretically, Du et al. (2020); Tripuraneni et al. (2020) study pre-training in similar linear regression setup. They show in-distribution generalization bound improvements, but do not consider OOD robustness combining with auxiliary inputs. [Hendrycks (et (20199) shows empirically that susersuperviioon can improve robustness to synthetic corruptions. We support these results by showing theoretical and empirical robustness benefits for pre-training on auxiliary information, which can be derived from the original input as in self- self-suupr
Raghunathan et al. (2020) analyze robust self-training (RST) (Carmon iet al. [219 et al. [2019) Uesato ct al. 2019)) which improves the tradeoff' between standard and adversarially robust aceuracy, in min-norm linear regression. Khani and Liang (02T) show how to use RST to make a model robust against a predefined spurious feature without losing accuracy. While related, we work in multi-tast linear regression, study pre-training, and prove robustness to arbitrary covariate shifts. |Kumar al. (020) show that repeated self self-training on gradually shifting unlabeled data can enable adaptation over time. In-N-Out is complementary and may provide better pscudolabels in each step of this method. Chen et al. (2020) show that self-training can remove spurious features for Gaussian input features in linear models, whereas our results hold for general input distributions (with density). Zoph et al.](222)))) (2020) show that self-training and pre- pre-training combine for in-distribution gains. We provide theory to support this and also show benefits for OOD robustness.
Self-training for robustness.
Domaín adaptation. Domain adaptation works account for covariate shift by using un unlabeled data from a target domain to adapt the model (Blitzer and Percira 2007 Daumé III 2007; [hu et al. 20188] Hoffman et al.] 2018) Ganin et al., (016). Often, modern domain adaptation methods (Shu et al. al. (218)) have a self-training or entropy minimization component that benefits from having a better model in the target domain to begin with. Similarly, domain adversarial methods (Ganin ct al. 2016) rely on the inductive bias of the source-only model to correctly align the source and target distributions. In-N-Our may provide a better starting point for these domain adaptation methods.
2018|Hoflman
7 DISCUSSION
Using spurious features for robustness. Counterintuitively. In-N-Out uses potentially spurious fea- tures (the auxiliary information, which helps in-distribution but hurts OOD accuracy) to improve OOD robustness. This is in contrast to works on removing spurious features from the model (Arjovsky et al.. 2019 Myas et al. 2019 Chen et al. 2020). In-N-Out promotes utilizing all available information by leveraging spurious features as useful in-distribution prediction signals rather than throwing them away.
General robustness with unlabeled data. In-N-Out is an instantiation of a widely applicable
paradigm for robustness: collect unlabeled data in all parts of the input space and learn better represen- tations from the unlabeled data before training on labeled data. This paradigm has driven large progress in few-shot generalization in vision (Hendrycks et al.) 2019a () and NL.P (Devlin et al. [2019] Brown al. 2020). In-N-Out enriches this paradigm by proposing that some features of the collected data can be used as input and output simultaneously. which results in robustness to arbitrary distribution shifts.
Leveraging metadata and unused features in applications. Many applications have inputs indexed by metadata such as location coordinates or timestamps (Christie al. 2018 Yeh et al. 20200) Ni ct al.] 2019). We can use such metadata to join (in a database sense) other auxilary data sources on this metadata for use in In-N-Out. This auxiliary information may often be overlooked or discarded, but In-N-Out provides a way to incorporate them to improve both in- and distiiiiibttiionn accuracy.
Division between input features and auxiliary information. While a standard division between in- puts and auxiliary information may exist in some domains, In-N-Out applies for any division of the input. An important further question is how to automatically choose this division under distribution shifts.
8 CONCLUSION
We show that while auxiliary information inputs improve in in-distribution and OOD on standard curated datasets, they can hurt OOD in real-world datasets. In contrast, we show that using auxiliary information as outputs by pretraining improves OOD performance. In-N-Out combines the strengths of auxiliary inputs and outputs for further improvements both in- and out-of-diitriibutinn.

---

Published as a conference paper at ICLR 2021
9
ACKNOWLEDGEEENTS
We thank Sherrie Wang and Andreas Schlueter for their help in procuring remote sensing data, Daniel Levy for his insight in simplifying the proof of Theorem Albert Gu for a key insight in proving Lemma Busin tools from random matrix theory, as well as Shyamal Buch, Pang Wei Koh, Shiori Sagawa, and anonymous reviewers for their valuable help and comments. This work was supported by an Open Philanthropy Project Award, NSF Frontier Award as part of the Center for Trustworthy Machine Learning (CTML). SMX was supported by an NDSEG Fellowship. AK was supported by a Stanford Graduate Fellowship. TM was partially supported by the Google Faculty Award, JD.com, Stanford Data Science Initiative, and the Stanford Artificial Intelligence Laboratory.
10 REPRODUCIBILITY
All code, data, and experiments are on Codal at this link.
REFERENCES
Sajjad Ahmad, Ajay Kaira, and Haroon Stephen. Estimating soil moisture using remote sensing data: machine learning approach. Advances in Water Resources, 33(1):69-800 80, 2010.
EA AlBadawy, A Saha, and MA Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys., 45, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.02897,09 2019.
John Blitzer and Fernando Pereira. Domain adaptation of natural language processing systems. University of Penmsylvania, 2007.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vor Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprini arXiv:2005. 1.t.20055..415, 2020.
C3S. ERA5: Fifth generation of ECMWF atmospheric reanalyses of the global climate, 2017.
Yaping Cai, Kaiyu Guan, Jian Peng. Shaowen Wang, Christopher Seifert, Brian Wardlow, and Zhan Li. A high-ppefformacc and in-season classification system of field-level crop types using time-series landsat data and machine learning approach. Remote Sensing of Environment, 210:74-84 84, 2018
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
Rich Caruana and Virginia R. de Sa. Benefitting from the variables that variable selection discards. Journal of Machine Learning Research (JMLR), 20003.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Computer Vision and Pattern Recognition (CVPR), 2018.
Hal Daumé III. Frustratingly easy domain adaptation. In Association for Computational Linguistics (ACL), 2007.
R S DeFries and JRG Townshend. NDVV-deeived land cover classifications at global scale. International Journal of Remote Sensing, (5(7))3367-3388, 1994.
Ruth DeFries, Matthew Hansen, and John Townshend. Global discrimination of land cover types from metrics derived from AVHRR pathfinder data. Remote Sensing of Environment, 54(3):209-222 1995.
10

---

Published as a conference paper at ICLR 2021
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-trant Tre-trrining of deep bidirectional transformers for language understanding. In Associiationn Computaiionl LingutttttsiaaLLLniiiisisss Linguisties (ACL), pages 4171-4186, 2019.
Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario March, and Victor Lempitsky. adveraradverssarill training of neural networks. Journal of Machine Learning Research (JMLR), 17, 2016.
Pall Oskar Gislason, Jon Atli Benediktsson, and Johannes R. Sveinsson. Random forests for land cover classification. Pattern Recognition Letters, 27(4):294-300 2006.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Paltern Recognition (CVPR), 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML), 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self--uper learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle consistent adversarial domain adaptation. In International Conference on Machine Learning (ICML), 2018.
Daniel Hsu, Sham M. Kakade, and Tong Zhang. Random design analysis of ridge regression. In Conference on Learning Theory (COL.7), 2012.
Andrew Iyas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv areprintarxxi: 19tp::::9900027777 2019.
Neal Jean, Marshall Burke, Michael Xie, W. Matthew Davis, David B. Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353, 2016.
Robin Jia and Perey Liang. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP), 2017.
Michael D. Johnson, William W. Hsich, Alex J. Cannon, Andrew Davidson, and Frédéric Bédard. Crop yield forecasting on the canadian prairies by remotely sensed vegetation indices and machine learning methods. Agricultural and Forest Meteorology, 218:74-84, 2016.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups dis- proportionately. In ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2021.
Serkan Kiranyaz, Onur Avei, Osama Abdeljaher, Turker Ince, Moncef Gabbouj, and Daniel Inman. 1d convolutional neural networks and applications: A survey. arXiv preprint 0rXiv:1905.035544 2019,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), pages 1097-1105, 2012.
Ananya Kumar, Tengyu Ma. and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning (ICML), (/2M.),0000.
N. Kussul, M. Lavreniuk, S. Skakun, and A. Shelestov. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 14(5): 778-782, 2017.
David J. Lary, Amir H. Alavi, Amir H. Gandomi, and Annette L. Walker. Machine learning in geosciences and remote sensing. Geoscience Frontiers, 7(1):3110, 2016.
Ainong Li, Shunlin Liang, Angsheng Wang, and Jun Qin. Estimating crop yield from multi-temporal satellite data using multivariate regression and neural network techniques. Photogrammetric Engineering & Remote Sensing, 73(10): 14:0::1119-1117 2007.

---

Published as a conference paper at ICLR 2021
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730-33 37300, 2015.
Ross Lunetta, Joseph Knight, Jayantha Ediriwickremaand John G Lyon, and L Dorsey Worthy. Land-cover change detection using multi-temppral MODIS NDVI data. Remote sensing of environment, 105(2):142-1555) 154, 2006.
Aaron E. Maxwell, Timothy A. Warner, and Fang Fang. Implementation of machine-learning classification in remote sensing: an applied review. International Journal of Remote Sensing, 39 (9):2784-2217,2013
Amir Najafi, Shin ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine- seee-ggaindd aspects. In Empirical Methods in Natural Language Processing (EMNLP), pages 188-197, 2019.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference on Machine Learning (ICML), 2020.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Very Large Data Bases (VLDB), number 3. pages 269-282, 2017.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems (NeurIPS), pages 3567-3575, 2016.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. arXiv, 2015.
Mare RuBwurm, Sherrie Wang, Marco Korner, and David Lobell. Meta-learning for few-shot land cover classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 200-201, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds; Benchmarks for subpopulation shift. arXiv, 2020.
Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T approach to unsupervised domain adaptation. International Conference on Learning Representations (ICLR), 2018.
K Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Muller. Covariate shift adaptation by impor- tance weighted cross validation. Journal of Machine Learning Research (JMLR), 8:985-10055 2007.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations (ICLR), 2014.
Terrence Tao. Topics in random matrix theory. American Mathematical Society, 2012.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint arXiv:2007.0064: 2020.
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity, arXiv, 2020.
12

---

Published as a conference paper at ICLR 2021
Jonathan Uesato, Jean- Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances Neural Information Processing Systems (NeurIPS), 2019.
E. Vermote. MOD09A1 MODIS/terra surface reflectance 8-day 1.3 global 500m SIN grid V006. 006 2015.
Sherrie Wang. William Chen, Sang Michael Xie, George Azzari, and David B. Lobell. Weakly supervised deep learning for segmentation remote sensing imagery. Remote Sensing, 12, 2020.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 3, 2016.
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference on Learning Representations (ICI.R), 2020.
Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Transfer learning from deep features for remote sensing and poverty mapping. In Association for the Advancement of Artificial Intelligence (AAAI), 2016.
Qizhe Xie. Minh-Thang Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. arXiv, 2020.
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 11, 2020.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc Le. Rethinking pre-training and self-training. arXiv, 2020.
13


Reviews:
[{"review_id": "zUFAsGtCxKt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates how to use auxiliary information to improve classification performance when few labeled examples are available. As the introduction makes clear, this is an especially important problem area for remote sensing applications, where labels are scarce for many inputs (e.g. satellite photos from countries/regions without much annotation).\n\nThe authors present three intuitively plausible baselines/ablations, two of which use auxiliary information, and explain the benefits and downsides of each. For instance, regarding the aux-inputs baseline: \"the relationship between the aux-inputs and the target can shift significantly OOD, worsening the OOD error\". These claims are later supported with theory using linear models. The theory is presented nicely, improves understanding, and is believable.\n\nTheir proposed method is very similar to the aux-out baseline/ablation. The only difference is that they fine-tune on pseudo-labeled in-distribution examples in their method. Seeing as this baseline could also be thought of as an ablation, and taking into account the improved performance of the full In-N-Out method, it is not too worrying. However, I am concerned about Remark 1 in Section 4.1, which says \"We train an aux-inputs model gˆin from w,z to y on finite labeled data—since the noise σ 2 = E[ 2 ] is small this model is very accurate.\" If In-N-Out only improves performance when aux-in is a nearly perfect generator of pseudo-labels for in-distribution data, then doesn't this imply that aux-out would learn just as much from the GT labeled in-distribution examples? How are the pseudo-labels actually helping?\n\nPros:\n- Underexplored, important problem area\n- Good clarity of writing and paper structure, including theoretical sections\n- Instructive choice of baselines/ablations\n- Empirical improvements from the proposed method\n- Not just vision datasets; they use a time series dataset as well\n- Theory in the case of linear models to improve understanding\n- The related work seems appropriate\n\nCons:\n- The proposed method is very similar to one of the baselines/ablations, and I am not certain that there is a meaningful difference between them\n- There is no comparison to previously published work that uses auxiliary information for classification. Perhaps there are no suitable baselines from prior work, but it is not clear from the paper that this is the case.\n\n======================================================\n\nUpdate after rebuttal:\n\nThe authors have addressed my concerns. Contrary to my initial understanding, the paper builds off of prior work in a methodical way, and the pseudolabeling stage of In-N-Out makes more sense now. I have raised my score from 6 to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ftk5QwCAJLi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method for leveraging auxiliary information and unlabelled data to improve out-of-distribution model performance. Theoretically, in a linear model with latent variables, they demonstrate using auxiliary data as inputs helps in-distribution test-error, but can hurt out-of-distribution error, while using auxiliary data to pretrain a \"good\" representation always improve out-of-distribution error. The proposed method uses the auxiliary data to learn an initial model, which generates psuedolabels to fine-tune the pretrained model.\n\nPros:\n- At a high level, this paper address a question of great interest to the ML community: out-of-distribution generalization.\n- The theoretical model shows, albeit in a potentially simple linear setting, that pretraining a low-dimensional shared representation generically improves out-of-distribution accuracy.  I'm not intimately familiar with all of the papers in this area, but I think this emphasis (as opposed to transfer learning) is new. This may be of interest more broadly.\n- Through experiments and a concrete example, the paper demonstrates the potential danger of using auxiliary features as input to models evaluated out of distribution.\n- The paper reports experimental numbers on two real remote sensing datasets rather than solely evaluating on synthetic data.\n\nCons:\n- Experimental results: My primary complaint with the paper is that, for the tasks considered, In-N-Out does not appear to work much better than the pretraining aux-outputs baseline. For out-of-distribution accuracy, across all of the datasets, the effect sizes are very small and the confidence intervals overlap. For in-distribution accuracy, there's only a large difference for the Landcover dataset. This makes me uncertain about the generality of the method and the potential size of the effects, though it's possible there's a more nuanced story that I'm missing. \n- Clarity: I found the description of the method confusing after several reads (section 2.1 and section 4), and the model sections are very notation heavy without necessarily providing much clarity. The graphical model, however, was very enlightening. \n\nQuestion:\n- Difference between the theoretical and experimental In-N-Out models: How come the experimental procedure differs from the one that is analyzed, e.g. fine-tuning on h_out(x)? Is the performance in practice worse? More difficult to implement? I don't mind the gap, but some explanation and, if available, associated experiments explaining this would be enlightening.\n- Generality of the theoretical model: How universal are the phenomenon captured by the linear model presented in Theorems 1 and 2? It's not obvious if the conclusions are \"representative\" or generalize beyond the ones explicitly analyzed.\n\n==============\nUpdate after rebuttal:\n\nThank you for clarifying that aux-outputs is itself a contribution and not simply a baseline for comparison. I also appreciated the additional experiment showing examples where In-N-Out can outperform aux-outputs. I'm raising my score from a 6 to a 7 accordingly.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EMlJrff8Ngw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Theoretical and empirical study on how to combine two approaches using auxiliary information for out-of-distribution samples\n\n**Quality:**\n\n_Pros._ The authors propose the method to combine two representative methods (Aux-Inputs and Aux-Outputs) to exploit auxiliary information which is usually available in real-world scenarios. Beginning with theoretical analysis on Aux-Inputs and Aux-Outputs models, they show that the proposed method, In-N-Out is effective in minimizing the risk under a distributional change in a linear regression setting. Empirical validations consistently show the preference of the proposed method highlighting the improvement in out-of-distribution (OOD) samples.\n\n**Clarity:**\n\n_Pros._ The description of the problem setting, approach, and intuition is well-written and persuasive based on their observations in the example in Introduction. The intuitions for the theoretical findings are nicely addressed.\n_Cons._ Some missing definitions (e.g., z in Alg. 1 as the output of `\\hat{h}_{out}`)  and lacking rigorousness in the text make difficulty in following (e.g., the function `\\hat{f}_{in}` takes two inputs, x and z, AND takes one input, `x^{id}_i`, in the 2nd and 4th lines in Alg. 1).  Figure 1 has ambiguities to understand which portion of the model is transferred and how to handle the change of the number of inputs (purely presumably, zero-filling as in the baseline in Sec 2.)\n\n**Originality:**\n\n_Cons._ Aside from their theoretical analysis, the combination of Aux-Inputs and Aux-Outputs is a simple model exploiting pseudo-labels (Xie et al., 2018), used for unsupervised domain adaptation tasks. In terms of novelty, the proposed method has a weak contribution. Isn't possible to borrow a sophisticated model from the works on unsupervised domain adaptation in your experiments?\n\n**What expected in rebuttal:**\n\n(1) Please explain the applicability of a sophisticated model from the literature in unsupervised domain adaptation in your experiments. (As an extension of the related work section.)\n\n(2) In Sec 3., they described, \"the aux-outputs model has better risk since w is lower dimensional than x. In particular, the in-domain risk only depends on the dimension but not on the conditioning of the data.\" However, Aux-Inputs or even a baseline (linear) model can have a lower-dimensional hidden representation in a low-rank linear model (e.g., two-layer perceptron, `f(x) := W_2 W_1 x`). So, the explanation is insufficient for the matter.\n\n(3) The proof of Proposition 1 is incorrect. Where can we find Remark 10? There is a typo missing reference after \"We use Theorem 1 in `?`.\" In Eqn. 23 and 24, if `1+cd/n = 2`, the left-most term in Eqn. 24 is `<= 2 \\sigma^2`, but not guarantee the Eqn. 24 if `\\sigma_u^2 < \\sigma^2`. To satisfy Eqn. 24, `cd/n \\sigma^2 < \\sigma_u^2` should be true.\n\n(4) In Lemma 8, the square is omitted inside of expectation in LHS of Eqn. 87. And, the dimension of R should be k x (k+m), not k x (k+T). I believe T is accidentally misplaced in this context.\n\n**Minor comments:**\n\n(5) Before Eqn. 26, a missing period right before \"For the input model ...\"\n\n(6) y' and x' instead of y and x in Eqn. 32.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of improving generalization when few annotated data is available by leveraging available auxiliary information. The authors consider the respective merits of two alternatives: using auxiliary information as complementary inputs or as additional outputs in a multi-task or transfer setting.  For linear regression, they show theoretically that the former can help improve in distribution error but may hurt OOD error, while the latter may help improve OOD error. They propose a framework for combining the two alternatives and show empirically that it does so on three different datasets. 


All the reviewers agree on the novelty, interest and impact of the method. The rebuttal clarified the reviewers’ questions. I propose an accept.

Decision: Accept (Poster)