The Value of Out-of-distribution Data
Authors: Ashwin De Silva
Rahul Ramesh
Carey Priebe
Pratik Chaudhari
Joshua T Vogelstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distribution Shift
Learning Theory

Abstract: More data is expected to help us generalize to a task. But real datasets can contain out-of-distribution (OOD) data; this can come in the form of heterogeneity such as intra-class variability but also in the form of temporal shifts or concept drifts. We demonstrate a counter-intuitive phenomenon for such problems: generalization error of the task can be a non-monotonic function of the number of OOD samples; a small number of OOD samples can improve generalization but if the number of OOD samples is beyond a threshold, then the generalization error can deteriorate. We also show that if we know which samples are OOD, then using a weighted objective between the target and OOD samples ensures that the generalization error decreases monotonically. We demonstrate and analyze this phenomenon using linear classifiers on synthetic datasets and medium-sized neural networks on vision benchmarks such as MNIST, CIFAR-10, CINIC-10, PACS, and DomainNet, and observe the effect data augmentation, hyperparameter optimization, and pre-training have on this behavior. 

Under review as a conference paper at ICLR 2023
THE VALUE OF OUT-OF-DISTRIBUTION DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
More data is expected to help us generalize to a task. But real datasets can contain out-of-distribution (OOD) data; this can come in the form of heterogeneity such intra-class variability but also in the form of temporal shifts or concept drifts. We demonstrate a counter-intuitive phenomenon for such problems: generalization error of the task can be a non-monotonic function of the number of OOD samples; a small number of OOD samples can improve generalization but if the number of OOD samples is beyond threshold, then the generalization error deteriorate. We also show that if we know which samples are OOD, then using a weighted objective between the target and OOD samples ensures that the generalization error decreases monotonically. We demonstrate and analyze this phenomenon using linear classifiers on synthetic datasets and medium-sizee neural networks on vision benchmarks such as MNIST, CIFAR-10. 10, CINIC-10, PACS, and DomainNet, and observe the effect data augmentation, hyperparameter optimization, and pre-training have this behavior.
INTRODUCTION
We procure more data to improve generalization. The central assumption behind doing so- to-that we have baked into learning theory (Vapnik, 1998)-is is that this data comes from the desired task. But this may not always be the case. Real data is often heterogeneous (Quinonero- Candella et al., 2008), this heterogeneity can arise from nuisances which are variables that do not inform the task at hand (say classification), c.g., geometric nuisances such as viewpoint, or semantic ones such as chairs of different shapes. Datasets curated at the Internet-scale Srivastava et al. (2022) may also be susceptible to erroneous annotations (resulting in label noise) (Frénay & Verleysen, 2013) or data poisoning attacks (Steinhardt et al., 2017). Such nuut-o-diistribution (OOD) data, i.e., data that does not come from our desired task, can be detrimental to the performance of the learned model. In this work, we our desired task.
aim study how OOD samples within datasets impact the generalization Our contributions are as follows.
We demonstrate a counter-intuitive phenomenon: generalization error on the target task is non-monotonic in the number of OOD samples. In other words, there exist situations when small number of OOD samples can improve the generalization error but if the number of OOD samples is beyond a threshold, then the generalization error deteriorates. This phenomenon is counter intuitive because one would expect the generalization error of the target task deteriorate or improve monotonically upon the introduction of OOD samples. Our investigation shows that the threshold is different for different tasks and different neural architectures. In Remark 2, we provide an intuitive explanation for this non- mon-monttoi behavior using the bias-variance trade-off.
We present empirical evidence for the presence of non-monotonic trends in target gen- eralization error in many popular datasets, ranging from MNIST, CIFAR-10, to PACS and DomainNet. OOD samples within a curated dataset could lead to worse generalization error on the task for which the dataset was curated. We show that when OOD samples in the dataset are unknown, using strategies such as data-augmentation. hyperparameter optimization and pre-training, are not effective eliminating the adverse impact of OOD data.
We develop an algorithmic procedure to train on the target task that is resilient to OOD data. If we know which samples within the dataset are OOD, e.g., using a two-sample test to check for changes in the distribution (Gretton et al., 2012), then we could mitigate the non-monotonic nature of the generalization error by ignoring the OOD samples. We show how one do better: using

---

Under review as a conference paper at ICLR 2023
weighted objective between the target and OOD samples, ensure that the generalization error on the target task decreases monotonically with the number of OOD samples. We empirically demonstrate the utility of this weighted objective on a variety of problems.
2 GENERALIZATION ERROR IS NON-MONOTONIC IN THE NUMBER OF OOD SAMPLES
We define a task joint distribution over the input domain and the output domain Y. We model the heterogeneity in the dataset two distributions: samples drawn from target task Pa and m samples drawn from an out-of-distribuiion (OOD) task Po. We would like to minimize the generalization error (h) = B( [h(x)) # v] on the target task. In order to do so, we may find a
hypothesis that minimizes the empirical loss
e(h)
(h(2)).,i),
(I)
using the dataset ((xy, 9.) " here E measures the mismatch between the prediction h(x)) and label ys. P: = Po. then (1) - E(h) ((n + m)-1//2 (Smola Schölkopf, 1998). But Pt * Po. then should expect that error on Pt of hypothesis obtained by minimizing the average empirical loss can be sub-optimal, especially when the number of OOD samples m > n.
2.1 AN EXAMPLE USING FISHER'S 's LINEAR DISCRIMINANT
Consider binary classification problem with one one-dimensiona. inputs in Fig. 1. Target samples drawn from a Gaussian mixture model (with means {-M, μ for the two classes) and OOD samples are drawn from a Gaussian mixture with means {-p + A, + A}; also see Appendix A. Fisher's linear discriminant (FLD) linear classifier for such binary classification problems. It computes
h(x) 1, w _
0, otherwise.
where w is a projection vector which acts as a feature extractor and c is a threshold that performs one-dimensional discrimination between the two classes. FLD assumes that the class conditional density of each class is multivariate Gaussian distribution with the same covariance structure. We provide a detailed account of FLD in Appendix A.2.
100, 2-.66-1.6
Anelyfical Expected Tergel Generiiizalio Error
[ 5,
Transitoon (5)
0.3100
Tapeet
0.8
Target-
0.3037
1.6
0.3004
00D
0OD
12
0.3031
0.3088
class
clsss
x=
m/n. 1000
Figure 1: Left: A picture of synthetic target and OOD tasks. Middle: A schematic the Gaussian mixture
model corresponding to the target task (lop) and the OOD samples (bottom). The OOD sample size (m 28) at which the target generalization error is minimized at A 1.6 is indicated at the top. Right: For n = 100, we plot the generalization error of FL.D the target task as function of the ratio of OOD and target samples m/n, for different types of OOD samples corresponding to different values of A. This plot uses the analytical expression for the generalization error in (2); Appendix A.6 for numerical simulation study. For small values of A, when the two tasks are similar to each other, the generalization error ex (h) decreases monotonically. However, beyond certain value of A, the generalization error is non-monotonic in the number of OOD samples. The optimal value of m/ n which leads to the best generalization error is a function of the relatedness between the two tasks, as governed by this example. This non-monee monotonic bchavior can explained in terms of a bias-variance tradcoff with respect to the target task: a large number of OOD samples reduces the variance but also results a bias with respect the optimal hypothesis the target task.
2

---

Under review as a conference paper at ICLR 2023
Suppose we fit FLD dataset which comprises of n target samples and OOD samples. Also, suppose we do not know which samples are OOD and believe that all the samples in the dataset come from a single target distribution. For univariate data, the FLD decision rule reduces to,
1: J. Aott/
h(x) =
0, otherwise.
Define the decision threshold to be é (fo + f1)/2. We can calculate (Appendices A.2 and A.3) an analytical expression for the generalization error of FLD on the target task:
mA- - (n m)u (n + m) +
-mA - (n + m)
e =
+
(2)
m) + m
here d is the CDF of the standard normal distribution.
1= 10.6-18
Varch
Fig. (right) shows how the generalization error (h) decreases up to some threshold of the ratio between the number of OOD samples and the number of samples from the target task m/n and then increases beyond that. This threshold is
of aF Dae en Tiverra- (B
500%
Mean Meann Squard Fime
MSES!1
8009-
different for different values of A as one can see in (2) and Fig. I (right). This behavior is surprising because one would a priori expect the generalization error to deteriorate monotonically as the number of OOD samples m increases. The fact that such a non-monotonic trend observed even for a one dimensional Gaussian mixture model and Fisher's Linear Diseriminant suggests that this may be a general phenomenon. We can capture this discussion as a theorem with our FLD example above as the proof.
009
0.00 055
0.0
1.5
20
Figure 2: Mean squared error (MSE) (Y-axis) of the decision threshold 6 of FLD (see Appendix A.3), for the same setup as that of Fig. 1, plotted against the ratio of the OOD and target samples m/n (X-axis) for A = 1.8. Squared bias and variance of the MSE are in violet and blue, respectively. This il- lustration elearly demonstrates the intu- ition behind non-monotonic target error: the MSE drops initially because of the smaller variance due to the OOD sam- ples. As more OOD samples are added, MSE increases due to the increasing bias. Non-monotonic trend in MSE of è translates to similar trend in the target generalization error (0-1 loss).
Theorem 1. There exist target and OOD tasks, P. and Po respectively, such that the generalization error on the target task of the hypothesis that minimizes the empirical risk in (1) is non-monotonic in the number of OOD samples.
Remark 2 (An intuitive explanation of non-monotonie trends in generalization error). Suppose that a learning algorithm achieves Bayes optimal error on the target task with high probability when the target sample size n exceeds N. We argue that non-monotonic trend in generalization error is likely to occur when и < N. i.e., when target generalization error is higher than the Bayes optimal error. In this case, if we add OOD samples whose empirical distribution is sufficiently close to that
Analpical Exoocted Tarpet coneraltation Enor
2 6, 16
of the target task, then this would improve generalization by
23541
reducing the variance of the learned hypothesis. But as the OOD sample size increases, the difference in the two distributions becomes apparent and this leads to bias in the choice of the hypothesis. Fig. 2 illustrates this phenomenon with regards to our FLD example in Fig. 1, by plotting the mean squared error of the decision threshold and its constituent bias and variance components. Roughly speaking, we may understand the non-monotonic trend in generalization as a phenomenon that arises due to the finite number of OOD samples (m/n in the example above). The distance between the distribution of the OOD samples and the distribution of the target task (A in the example) determines the threshold beyond which the error is monotonic. Current tools in learning theory (Smola & Schölkopf, 1998) are fundamentally about understanding generalization when the number of samples is asymptotically
22540
0.5
0.4
0.0 1.5
500
Figure 3: We can control the Bayes optimal error by adjusting μ, of the Gaussian mixture model $2.1, dis- cussed in Remark 2, when the Bayes op- timal error is large for (μ = 6, a = 16). we observe non-monotonic trends even for large number of target sam- ples (n 500). This suggests that non-monotonic trends in generalization not limited small sample sizes. they be from the target task or OOD. In future work, this non-monotonic trend in generalization error by building new maaning-thhorees tools.
we hope to formally characterize
3

---

Under review as a conference paper at ICLR 2023
RotatedCIFAR
Blurred CIFAR
Split CIFARIO
Targel: 73.
15 (WRN-10-2)
Target: Ta. OOD: oristured Ta (WRN 10-21
Targel: T1. O0D: 75
0.40
0.25
0.40
Terge: Geneeneeeennnnnn toston Drer
0.38
Target Target Ganersizaten Eiror
Ceenenazatioon Ioston Error
0.15
Target: T2 O0D:
0.42
0.20
025
0.00
Target: T.. OOD:7,
0.20
0.42
0.2
0.18
0.30
100
mih, 100
min, n=100
BureLeve (o)
Figure 4: Left: Sub-task T2 (Bird vs. Cat) from Split-CIFAR10 is the target task and images of these classes rotated by different angles 8 are the OOD task. 10-2-10-2 architecture was used to train the model. We non-monotonic curves for larger values of 8% For 60" and 135" in particular, the generalization error m/n = 20 is worse than the generalization error with a fewer OOD samples, i.c. OOD samples actively hurt generalization. See Fig. A6 (left) for a similar experiment involving SmallConv, much smaller CNN. Middle: The Split-CIFAR10 binary sub-task T4 (Frog vs. Horse) is the target task and images of these classes subjected varying levels of Gaussian blur the OOD samples. WRN- VRN-10-2 architecture
used train the model. Non-monotonic curves are observed for larger levels of blur, while for smaller levels of blur, we notice that adding more OOD data improves the generalization the target task. Right: Generalization error of two separate networks, WRN- 10-2 and SmallConv. OOD task different pairs of target-OOD tasks from Split CIFARCI All the pairs exhibit non-mone monotonic target generalization trends across both network models. See Appendices B.2 and B.3 for experimental details and Appendix B.7 for experiments on more task pairs (Figs. A4 and A5) and multiple target sample sizes (Fig. A3). Error bars indicate 95% confidence intervals (10 runs).
the target task is plotted against the number of samples from the
Even if the non non-monotonic trend occurs for relatively small values of target and OOD samples and m respectively in Fig. 1, this need not always be the case. If the number of samples required to reach Bayes optimal error in the above remark is large, then a non-monotonic trend could occur even if we have a relatively large target sample size If N is small, then the non-monotonic trend would not occur typically because then the typical target sample size would be n > N. (See Fig.
2.2 MON-MONOTONIC TRENDS FOR NEURAL NETWORKS AND REAL DATASETS
We experiment with several popular datasets including MNIST, CIFAR-10, 10, PACS, and DomainNet and 3 different network architectures: (a) small convolutional network with 0. 0.12M parameters (denoted by SmallConv), (b) wide residual network (Zagoruyko & Komodakis, 2016) of depth 10 and widening factor (WRN-10-2), and (c) a larger wide residual network of depth 16 and widening factor (6RN-16-4). See Appendix B.4 for more details.
Non-monotonic trend in generalization error can occur due to geometric and semantic nuisances. Such nuisances are very common even curated datasets (Van Horn, 2019). We constructed binary classification sub-tasks (denoted by T. for i 1,....5 from CIFAR- 10 study this aspect (see Appendix B) We consider a CIFAR-10 sub-task T2 (Bird vs. Cat) as the target and introduce rotated images by a fixed angle between 021135") as OOD samples. Fig. 4 (left) shows that the generalization error decreases monotonically for small rotations but it monooonoo for larger angles. Next, considered the sub-task T4 (Frog vs. Horse) the target task and generate OOD samples by adding Gaussian blur of varying levels to images from the same task. In Fig. 4 (middle), the generalization error on the target is a monotonically decreasing function of the number of OOD samples for low blur but increases non-monotonically for high blur.
Non-monotonic trends can occur when OOD samples are drawn from different task Large datasets can contain categories whose appearance evolves in time (e.g., a typical laptop 2022 looks very different from that of 1 1992). categories can have semantic intra-class nuisances (e.g..
4

---

Under review as a conference paper at ICLR 2023
chairs of different shapes). We use CIFAR-10 10 sub-tasks to study how such differences can lead to non-monotonic trends (see Appendix B. For 5 CIFAR-10 10 sub-tasks; each sub-task is binary classification problem with two consecutive classes: Airplane vs. Automobile, Bird vs. Cat, etc. We consider (T,, Tÿ) the (target, OOD) task pair and evaluated the trend in generalization error for all 20 distinct pairs of tasks. Fig. 4 (right) illustrates non-monotonic trends for 3 such pairs; see Appendix B for more details.
RatatedMMIST Target: MMIST B-Bitarms (Sma Klams)
PACS Target Horoo OoBi x 19RN-16-4
Domaintet
brrget Roal OOB: (WAt
Bige Groerttaaiinn Erier
0:40
0.20-
Toput Oune al aution
6:30
Faiitig Quicedran
mih. m 100
1V% a
muh ni
9
3:
45
FIRRCh
Pariiss
Figure 5: Con-monotonic trends in target generalization error on three DomainBed benchmarks. Left: Rotated MNIST classes 10 target samples/class, SmallConv), Middle: PACS (3 classes (dog, elephant, horse). 10 target samples/class, WRN-16-4), and Right: DomainNet (2 classes (bird, plane), 25 target samples/class, WRN-16-4). Error bars indicate 95% confidence intervals (10 runs).
CINIC-10 Target GEAR 000: X(WAN 10
Non-monotonic trends also occur for benchmark domain gen- eralization datasets We further investigated three widely used benchmarks in the domain generalization literature. First, we con- sider the Rotated MNIST benchmark from DomainBed (Gulrajani & Lopez-Paz, 2020). We define the 10-way classification of un-rotated MNIST images as the target task and 0-rotated MNIST images as the OOD samples. Similar to the previous rotated CIFAR- experiment, we observe non-monotone trends in target generalization for larger angles 0. Next, we consider the PACS benchmark from DomainBed which contains 4 distinct environments: photo, art, cartoon, and sketch. A 3- way classification task involving photos (real images) is defined as the target task, and we let the corresponding data from other environments be the OOD samples. Interestingly, we observe that when OOD samples consist of sketched images, then the generalization error on the real images exhibits a non-monotonic trend. We also observe similar trends in DomainNet, a benchmark that resembles PACS; see Fig. 5.
Tonqul Cfrunditaaaon Enorr
0.28-
Imagehat
0.72
0:66
0.0-
10
mah 100
Figure 6: Target task is CIFAR- 10 and OOD samples ImageNet. Although there is a distribution shift that causes the red curve to be higher error than the purple one, there no non- monotonic trend in the generaliza- tion on CIFAR- 10 due to OOD samples from ImageNet. Error bars indicate 95% confidence in- tervals (10 runs).
from
Generalization error is not always non-monotonic even when there distribution shift We considered CINIC-10 (Darlow et al., 2018), dataset which was created by combining CIFAR-10 with images selected and down-sampled from ImageNet. We train network on subset of CINIC-10 that comprises of both CIFAR-10 and ImageNet images. The target task is CIFAR-10 10 itself, so images from IetgeNet in CINIC-10 act as OOD samples. Fig. demonstrates that having more ImageNet samples in the training data improves the generalization (monotonic decrease) the target task, but at slower rate than the instance where the training data is purely comprised of target data. This phenomenon is also demonstrated in Fig. 1: for sufficiently small shifts, the target generalization error decreases as the number of OOD samples increases.
Efffec of pre-training, data aatanaangmntttion and hyperparameter optimization When we do not know which samples in the training data are OOD, we do not have a lot of options to mitigate the deterioration due the OOD samples. We could employ data augmentations, hyperparameter
5

---

Under review as a conference paper at ICLR 2023
Split CIFARIO
Split CIPARIO
Target Tq. QDD: 7. (WRN-10-22
Target: Ty. OOD: (WWNN)
Vanilla
0.375
With Opémazed Hyperpammdan
Augmentation
Pretraining
Careraliation
Toque Cunarofiationn
0.35
0.346
0.300
0.25
20
10
mih, n 1.00
mwh.mn 100
Figure 7: Left: Considering CIFAR-I0 10 subtask T2 (Bird vs Cat) as target and T3 (Ship vs Truck) as OOD, we train a WR--10-2 network with class- balanced datasets containing fixed number of target samples (n 100) and varying number (m) of OOD samples, under the following settings: (1) Vanilla without any fata-augmentttion or pre-training (darkest shade of red), (2) Data augmentations by introducing padding with random cropping and random left/right flipping (intermediate shade of red), and (3) Pre-training followed by fine-tuning (lightest shade of red). We pre-train network on 144000 class-balanced ImageNet images from CINIC- (see Appendix B. belonging to Bird and Cat classes which correspond to our hypothetical target task. Pre-training is performed for 100 epochs with a learning rate of 0.01. Next, we employ a two-step strategy of linear probing (first 50 epochs) and full-fine tuning (last 50 epochs) inspired by Kumar et al. (2022) that this fine-tuning is performed on the combined dataset of v target and m OOD samples. Even though data augmentation and pre-training followed by fine-tuning reduce the overall error, still observe the deterioration of target generalization error as the OOD sample fraction of the dataset increases. Right: For each run at each value of m, we perform hyperparameter tuning using Ray (Liaw al., 2018) over validation set including only target samples, and record the target generalization error of the model using the best set of hyperparameters. However, we still observe target generalization error degrading as the OOD samples increase. Note that hyperparameter tuning cannot be implemented in reality because we may not know the identity of the target and OOD samples. So the fact that the non-monotonic trend persists in the hypothetical instance where we know the sample identities guarantees that it will occur in practice as well. Error bars indicate 95% confidence intervals over 10 experiments.
reduced learning rate of 0.001. Note
optimization, or say pre-training followed by fine-tuning. The second option is difficult to implement for real problem because the validation data that will be used for hyperparameter optimization will itself have to be drawn from the curated dataset.
We tested whether these three techniques above improve generalization on the target task in the presence of OOD samples. We consider CIFAR-10 10 sub-task T2 (Bird vs. Cat) as the target task and Ts (Ship vs. Truck) as the OOD task and train a WRN- network under various settings. The effect of these techniques are reported in Fig. and we find that neither of them manages to curb the deterioration of target generalization error the OOD sample size of the dataset increases.
3 CAN WE EXPLOIT THE KON-MONOTONIC TREND IN THE GENERALIZATION ERROR?
Suppose knew which samples in dataset were OOD for the target task. Then we should be able to not only mitigate the pon-monotonic nature of the generalization error but also exploit it.
Theorem 3 (Paraphrased from Ben-David et al. (2010)). For two tasks Pt and Po. let ha be the minimizer of the a- te-weigtted empirical risk éa (A) = aá, (h) + (1 ()))(4) where (h) and éo are the empirical risks of Pt and P. respectively. The generalization error
a2
(1 (1-a)2
ch( s (e( ) + VV
VH log 6 + 2(1 Fdd (Ps Po),
n
with probability at least 8. Here hj argmin (h) the target error minimizer; VH is a constant proportional the VC-dimension of the hypothesis class Hf and (P,, relatedness between the tasks P; and Po-
(Pt, Po) is a notion of
In other words, if we use an appropriate value of a that makes the second and third terms on the right-hand side small, then we can mitigate the deterioration of generalization error due to OOD samples. If the OOD samples very different from those of the target task, i.e., if d(Ps ((Ps, Po) is large,

---

Under review as a conference paper at ICLR 2023
then this theorem suggests that should pick an 1. Doing effectively ignores the OOD samples and the generalization error then decreases monotonically as On 11/2).
3.1 CHOOSING OPTIMAL
If we define Vy log to be, roughly speaking, the ratio of the capacity and the distance between (P.PP.
tasks, then short calculation shows that for E [0. 1).
if 2 4p²
m
else.
This suggests that if we have hypothesis space with small VC-dimension or if the OOD samples and target samples come from very different distributions, then we should train only on the target samples to obtain optimal error. Otherwise, including the OOD samples after appropriately weighing them using a" can give better generalization error.
is not easy to estimate p because it depends upon the VC-dimension of the hypothesis class (Ben- David et al., 2010; Vedantam et al., 2021). 1). But general, we can treat a as a hyperparameter and use validation data to search for its optimal value. For our FLD example we do slightly better: can calculate the analytical expression for the generalization error for the hypothesis that minimizes the a- a-weighted empirical risk (see Appendices A.4 and A.5) and calculate a" by numerically evaluating the expression for a E [0. 1].
Fig. 8 shows that regardless of the number of OOD samples (m) and the relatedness between OOD and target tasks (A), we can obtain generalization error that is always better than that of hy pothesss trained without OOD samples. In other words, we choose a appropriately (Fig. corresponds choosing a 1/2). then do not suffer from con-monotonic generalization error on the target task.
1= 1=1,0=1
11
1,==
Anslyiical Expected onneraictoon
2.185
10
0.180
0.
Optmal
0.8
8.175
0.7
0.17
0.165
3.2 TRAINING NEURAL NETWORKS USING THE Q-WEIGHTED OBJECTIVE
min,
min,
GOD Transtation (A) 0.5
In $2.2, for a variety of computer vision datasets, we found that for some pairs of tasks, the gener- alization error is non-monotonic in the number of OOD samples. We now show that we knew rectify
Figure 8: Left: Generalization error on the target task for the Gaussian mixture model using a weighted ob- jective (Theorem 3) in FLD; see Appendix A.4. Note that unlike in Fig. 1, the generalization error monoton- ically decreases with the number of OOD samples m. Right: The optimal a that yields the smallest target generalization error a function of the number OOD samples. Note that a increases as the number of OOD samples m increases; this increase more drastic for large values of A and is more gradual for small values of Observe that a = 1/2 all values m 0. See Appendix A.6 for numerical simulation.
which samples were OOD, then this trend using appropriate value of a to weigh the samples differently. In Fig. 9, we track the test error of the target task for three cases: training is agnostic to the presence of OOD sam- ples (red), the learner knows which samples are OOD and uses an a = 1/2 the weighted risk call this "naive"), and when
to train (yellow, it uses an optimal value of using grid-search (green). Searching over improves the test error on all these 3 pairs of target-OOD tasks. See Figs. A4 A6 for more experiments with similar conclusions.
We also conducted another experiment to check if augmentation can help rectify the non monotonic trend in the generalization error, using the a-weighted objective, i.c., when we know which samples are OOD, As shown in Fig. 10, this case even naively weighing the objective (a 1/2, yellow) can rectify the non-monotonic trend, using the optimal a" (green) further improves the error. This suggests that augmentation is an effective way to mitigate non-monotonic behavior, but only if we use the a-weighted objective, which requires knowing which samples are OOD. As we discussed in Fig. if we do not know which samples are OOD, then augmentation does not help.
7

---

Under review as a conference paper at ICLR 2023
PACS
Split CIFAR10
Split CIFAR10
Trret Photo, COO, Shezt (W84-16-4)
Taiges To, 002. T. (ATIN-10-2)
Targe Po. COO: (AVIN-11-22
043
0.52
Targez T, 002 (1106-1022)
Target 72 CCO. (W11N-10-2)
0-0
8.08
Target Genee sunsralaation
Target Geneeelttttee
Openaal a
0.58
T. 000:7,
0.2
0.58
0.14
1.3
1-00
OO0 unkknown
ODD known
OGD known (Optmma)
Figure 9: Here we present three settings: minimizing the average risk over target and OOD samples is agnostic to OOD samples present (red), minimizing the sum the average loss of the target and OOD tasks which corresponds to a = 1/2 (yellow). minimizing an optimally weighted convex combination of the target and OOD empirical loss (green). The last two settings are only possible when one knows which samples are OOD, For each setting, we plot the generalization error on the target task against the number of OOD samples for (target, OOD) pairs from PACS (Lefft) and CIFAR-10 10 subtasks (Middle). Unlike in CIFAR-10 1-10 task pairs, we observe that in PACS, the target generalization error has downward trend when speculate that this could be due to the similarity between the target and OOD tasks, which causes the model to generalize the target even naive weight. Right: The optimal a² obtained via grid search for the three problems in the middle column plotted against different number of OOD samples. The value of a lies very close to but is never exactly 1. In other words, if we use the weighted ohjective in Theorem 3 then always obtain some benefit, even it is marginal when OOD samples are very different from those of the target. Error bars indicate 95% confidence intervals over 10 experiments.
0.5 (yellow line, left pancl). We
Sampling mini-batches during training For m >> n, mini-batches that sampled uniformly randomly from the dataset will be dominated by OOD samples. As a result, the gradient even if it is still unbiased, is computed using very few samples from the target task. This leads to an increase in the test error, which is partieularly noticeable with a" chosen appropriately after grid search. We therefore use biased sampling procedure where each mini-batch contains a fraction 8 samples from the target task and the remainder 1 - 3 consists of OOD samples. This parameter controls the bias and variance of the gradient of the target task (3 = ntm gives unbiased gradients with respect to the unweighted total objective and high variance with respect to the target task when n, see Appendix B.5). We found that both 3 (0.5, 0.75) improve test error.
n
Weighted objective for over-parameterized networks It has been argued previously that weighted objectives are not effective for over- peeapaaramterieeed models such as deep networks because both surrogate losses (t) and (l) are zero when the model fits the training dataset (Byrd & Lipton, 2019). It may therefore seem that the weighted objective in Theorem cannot help us mitigate the mon-monotonic nature of the generalization error; indeed the minimizer of ( + (1 a) (h) is the same for any if the minimum is exactly zero. Our experiments suggest otherwise: the value of does impact the generalization error-even for deep networks. This is perhaps because even if the cross-entropy loss is near-zero for a deep network towards the end of training, it never exactly zero
4 RELATED WORK
Distribution shift (Quinonero- Chuinonro-CCandllaee et al., 2008) and its variants such as covariate shift (Ben-David & Urner, 2012; Reddi et al., 2015), concept drift (Mohri & Muñoz Medina, 2012; Bartlett, 1992; Cavallanti et al., 2007), domain shift (Gulrajani & Lopez-Paz, 2020; Sagawa et al., 2021; Ben-David et al., 2010), sub- sub- cobp-ooulttion shift (Santurkar et al., 2020; Hu et al., 2018; Sagawa et al., 2019), data poisoning (Yang et al., 2017; Steinhardt et al., 2017), geometric and semantic nuisances (Van Horn, 2019), and flawed annotations (Frénay& & Verleysen, 2013) can lead to the presence of OOD samples in curated dataset, and thereby may yield sub-optimal generalization error on the desired task. While
8

---

Under review as a conference paper at ICLR 2023
these problems have primary been studied in the sense of out-of-domain distribution, we believe that we have identified a fundamentally different phenomenon, namely a non-monotonic trend in the generalization error due to the different numbers of OOD samples.
Internal Dataset Shift A Recent body of works (Kaplun et al., 2022; Swayamdipta et al., 2020; Siddiqui et al., 2022; Jain et al., 2022) have investigated the presence of noisy, hard-to-learn, and/or negatively influential samples in popular vision benchmarks. Existence of such OOD samples indicates that the internal dataset shift may be widespread problem in large datasets. Such circumstances may give rise to undesired non- mo--oooootocccc trends in generalization error, as we have described in our work. This coupled with the difficulty of detecting internal shifts motivates further research into tackling
this problem.
Split CIFAR10
Domain Adaptation While most works listed above provide attractive ways of adapting or being robust to various modes of shift, part of our work addresses the question: if we know which samples are OOD, then can we optimally utilize them to achieve a better generalization on the desired target task? This is related to domain adaptation (Ben-David et al., 2010; Mansour et al., 2008;
Target Gen Gen Ereor
Target: T1. OOD: (SmallConv)
0.18
0.14
0.11
0.83
Optinala* dr
Pan et al., 2010; Ganin et al., 2016; Cortes et al., 2019). A large body of work uses weighted-ERe for domain adaptation (Ben-David et al., 2010; Zhang et al., 2012; Blitzer et al., 2007; Bu et al., 2022: Hanneke & Kpotufe, 2019; Redko et al., 2017; Wang et al., 2019; Ben-David et al., 2006); this is either done to address domain shift address different distributions of tasks in a transfer or multi-task learning. This body of work is our primary motivation, except that in our case, the "source" task is actually the OOD samples.
based methods
0.81
16
n=100
000 krown (Neive))
OOD (Cplima))
Figure 10: Effect of data augmentation (padding with random cropping and random left/right flipping). Although the network trained in the setting where the OOD sample identities are unknown (red) continues per- form poorly with lots of OOD samples, even a naive weighing of the target and OOD loss (a = 1/2) is enough to provide monotoni- cally decreasing error (yellow) when the OOD sample identities are known. This suggests that data augmentation can mitigate some of the anomalies that arise from OOD data, al- though we can do better by addressing them specifically using. for instance, the weighted objective (green). Error bars indicate 95% confidence intervals over 10 experiments.
Connection with the theory of domain adaptation While generalization bounds for weighted-ERM like those of Ben-David et al. (2010) are understood to be meaningful (if not tight; see Vedantam et al. (2021)) for large sample sizes, our work identifies an unusual non mon-oonoto trend in the generalization error of the target task. First note that the calculations in Ben-David et al. (2010) can be used when we do not know the identity of the OOD samples by setting a = --mm" One can get an insight into the non-monotonic trend of the target error from Theorem 3 in Ben-David et al. (2010) because the second term
n
(1-a)²) VV log decreases with increasing m while the third term 2(1 - a)d (Pt, Po)
increases because a - O. While such a trend in the upper bound does not directly imply a non- monotonic trend in the error itself, this discussion suggests that the results from our experiments are not inconsistent with existing theory. There is discrepancy here, e.g., we notice that DaviD et al. (2010)'s 's upper bound for naively weighted empirical error (a = 1/2) does not have a non-monotonic trend (again, this is only an upper bound on the target error). A more recent paper by Bu al. (2022) presents exact characterization of the target generalization error using conditional symmetrized Kullback-Leible information between the output hypothesis and target samples given the source samples. While they do not identify non-monotonic trends in target generalization error, their tools can potentially be useful to characterize the phenomenon discovered in work.
Domain Generalization seeks to learn a predictor from multiple domains that could perform well on some unseen test domain. This unseen test domain can be thought OOD data. Since no training data is available during the training, the learner needs to make some additional assumptions; one popular assumption is to learn invariances across training and testing domains (Gulrajani & Lopez-Paz, 2020; Arjovsky et al., 2019; Sun & Saenko, 2016). We use several benchmark datasets from this literature, but the goals of this body of work and ours are very different because we are interested only in generalizing on the target task, not generalizing to the domain of the OOD samples.
9

---

Under review as conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:/90.002000 028933, 2019.
Peter L Bartlett. Learning with a slowly changing distribution. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 243-252, 1992.
Shai Ben-David and Ruth Urner. On the hardness of domain adaptation and the utility of unlaheled target samples. In International Conference on Algorithmic Learning Theory, pp. 139-153. Springer, 2012.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning. 79(1):151-175, 2010.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. Advances in neural information processing systems, 20, 2007.
Yuheng Bu, Gholamali Aminian, Laura Toni, Gregory W Wornell, and Miguel Rodrigues. Charac- terizing and understanding the generalization error of transfer learning with gibbs algorithm. In International Conference on Artificial Intelligence and Statistics, pp. 6673-8699 PMLR, 2022.
Jonathon Byrd and Zachary Lipton. What is the effect importance weighting in deep learning? In International Conference on Machine Learning, pp. 872-881, 2019.
Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Tracking the best hyperplane with a simple budget perceptron. Machine Learning, 69(2):143-167, 2007.
Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. The Journal of Machine Learning Research, 20(1):1-30, 2019.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic-10 is not imagenet or 10. arXiv preprint arXiv: 10XX::8/10000305.. 2018.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845-866) 2013.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1 21-2096-2000 2016.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Steve Hanneke and Samory Kpotufe. On the value of target data in transfer learning. Advances in Neural Information Processing Systems, 32, 2019.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2039-2037. PMLR, 2018
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. data-based perspective transfer learning, arXiv preprint arXiv:2207.05739 2022.
10

---

Under review as a conference paper at ICLR 2023
Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv preprint arXiv.2202.09931, 2022.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform ditto--distriuution. arXiv preprint arX1v:2202.10054, 2022.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp. 5542-5550, 2017.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and lon Stoica. Tune: research platform for distributed model selection and training. arXiv preprini arXiv: arXiv:/807. 8807.05/ 18, 2018.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadch. Domain adaptation with multiple sources. Advances neural information processing systems, 21, 2008.
Mehryar Mohri and Andres Muñoz Medina. New analysis and algorithm for learning with drifting distributions. International Conference on Algorithmic Learning Theory, pp. 124-138. Springer, 2012.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE transactions on neural networks, 2292):19-22111 2010.
Xingehao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 1406-141 1415, 2019.
Joaquin Cainonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. Mit Press, 2008
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance regularization for worst-case generalization. arXiv preprint 1911.v:9111.07755 2019.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark unsupervised adaptation. arXiv preprint arXiv:2 12.05090, 2021.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint 2XX::2008.04859 2020.
Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj. David Krueger. and Sara Hooker. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprint (00::2209.10115, 2022.
Alex J Smola and Bernhard Schölkopf. Learning with kernels, volume 4. 1998.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint TrXiv:2206.04615 2022.
11

---

Under review as conference paper at ICLR 2023
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450, Springer, 2016.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint MaKI::20091179955 2020.
Grant Richard Van Horn. Towards Visipedia: Combining Computer Vision and Communities of Experts. PhD thesis, California Institute of Technology, 2019.
Vladimir Vapnik. Statistical Learning Theory. John Wiley & Sons, 1998.
Ramakrishna Vedantam, David Lopez- Pazez-Paz, and David Schwab. An empirical investigation of domain generalization with empirical risk minimizers. Advances in Neural Information Processing Systems, 14:28131-221433 2021.
Boyu Wang, Jorge Mendez, Mingbo Cai, and Eric Eaton. Transfer learning via minimizing the performance gap between domains. Advances in Neural Information Processing Systems, 32, 2019.
Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. arXiv preprint 17030017 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. In International Conference on Machine Learning, pp. 3987-3995, 2017.
Chao Zhang, Lei Zhang, and Jieping Ye. Generalization bounds for domain adaptation. Advances in neural information processing systems, 25, 2012.
12

---

Under review as conference paper at ICLR 2023
A FISHER'S 's LINEAR DISCRIMINANT (FLD)
A.1 SYNTHETIC TASKS
The target task Pa and the OOD task Po are both binary classification problems with one-dimensional inputs. In both tasks, each class sampled from a univariate Gaussian distribution. The OOD task the target task translated by A. In summary, the target task has the class conditional densities,
fL. N((m,o0)
f N(+ o²);
while the OOD task distribution has the class conditional densities,
fa,0 N(I 2,
fa, N(IA + μ, o²)))
We also assume that both the target and OOD tasks have the same label distribution with equal class prior probabilities, i.c. plyt = 1) = p(yo = 1) = x 2.2 Fig. (left) depicts Pt and Po pictorially,
A.2 OOD- FISHER'S LINEAR DISCRIMINANT
In this section, derive FLD when we have samples from single task which is also applicable to
the OOD OOD-agnostic (when the identity of the OOD samples are not known) setting. Consider a binary
classification problem with D; = ((r, y)))== ~ Pt where 2; E X C gd and yı E Y = {0, 1}.
Let Ju and Tı be the conditional density and prior probability of class & (k E [0, 13) respectively. The probability that belongs to class k is
(x)
ply k x)
fo(o) (x))
and the maximum a posteriori estimate of the class label is
h(x)= argmax p(y = l x) argmax log(TT//a(t)) kE(0.1)
kc(0,1)
Fisher's s linear discriminant (FLD) assumes that each fa is multivariate Gaussian distribution with the same covariance matrix E, i.e
fx(x)) (27)d/2p2)// exp (-
- (
m2m
Under this assumption, the joint-density of (æ,8) becomes.
1(0=b)
f(z, & II
exp (- (x px) T (x - mm)
1/2
k=
Therefore, the logeliillhood [(Hu, u1, E, mo, x1) over D, is given by,
fpoo /1, F, no. m)) Σ Σ
log म. log |o]
1 ( (u)) 1 -ma) const.
2
k=0 (r.
where Dr the samples of De that belongs class Based on the likelihood function above,
we can obtain the maximum likelihood estimates Pkx Lt मt. The expression for the estimate fig
Au DDe (M.)))
1
Σ
(4)
Plugging these estimates into (3), we get,
h(xx) argmax (0.1)
log tr.
1 log |S1
2 (s mu) ;m) 1--11100 (m)
2
1 log |
+ x0 T fic - 1 --- 10
argmax kE(0.1)
log f...
2
13

---

Under review as conference paper at ICLR 2023
Therefore, M(x) 1 iff,
x -T 15-1 in 2 52 + log > x 1 i00
-00
1
5---5 + + log 40
-ター in x - 10 z
1 Hー
/1
5-1 + log Ro
log #1
( ( (mn اo)) J (É
(i1 ( )o))
log no 11
2
Hence the FLD decision rule h(x)) is
1, w x
((x)
0, otherwise
where w = 1 (Ā1 - (o) is a projection vector and = w ( = and .g = T the decision rule reduces
+ log do is threshold. When #1
h(x) 0. otherwise
A.3. DERIVING THE GENERALIZATION ERBOR THE TARGET TASK FOR SYNTHETIC TASKS WITH FLD
We would like to derive an expression for the average generalization error of the target task, when we consider the synthetic tasks described in Appendix A. For simplicity, we set the variance o² the class conditional densities of the synthetic tasks to
In the D0D-agnostic setting, the learning algorithm sees single dataset D = Dj U D. of size n + m which a combination both target and OOD samples. We can estimate using (4) to obtain
E(z,)) x E(aN))D
DE
x
veg + m/k
(x,
Tt k mg
(6)
71 mp
nē + mToo.k
71 + m
where D& the set of samples of that belongs to class k, ng = (D. and my = |Do for E {0, 1 Ft and Fo,k denote the sample means of class le in target and OOD datasets respectively. We assume that from which it follows that n2 nmg 2 and my mma T. We cannot explicitly compute F and Jo when the OOD samples are not explicitly known, because we cannot separate target samples from OOD samples in D.
Since the samples are drawn from Gaussians, their averages also follow Gaussian distributions. Hence, the threshold Authi of the hypothesis h, estimated using FLD, is random variable with a Gaussian distribution i.e., ((H,, o where
mA
B[d]
of Var(c] = m
The target error of hypothesis is
p(h(z) # y 22
2 Ponsial G) /Ponseol o
Pennsen x Po <
[1 (66 1 +6((+1)) +1]
(7)
14

---

Under review as conference paper at ICLR 2023
Using (7), the expected error on the target task cr( N ()(((((()) # v I e). is given by,
ex(h)
[1 + ( (1) +(ē+ 112+/0 m)]
1
ph
hh
On
[1 + (yya + ph - (4)
Dyyon + ph p)] Qy)dd
2 +
Ph M
+
ph
V o
V
In the last equality, we make use of the identity J_ Dce + d)o(x)dz $
(1+,2> where e and
are the PDF and CDF of the standard normal. Substituting the expressions for pha of into the above equation, we get
mA (n + m)μ ( m)(n m +
mm (n - mn m)j
(nn (n+ m) +
(8)
For synthetic tasks with o² + 1, the target generalization error can be obtained by simply replacing and A with & and A respectively in (8).
A.4 OOD-AWARE WEIGHTED FISHER'S LINEAR DISCRIMINANT
We consider a target dataset D, = [(24, ==) and an OOD dataset D. = ((1g v:) which are samples from the synthetic tasks from Appendix A.1. This setting differs from Appendix A.3 since we know whether each sample from D; U D. is OOD or not. This difference allows us consider log-likelihood function that weights the target and OOD samples differently, i.e. we consider
f10.P1. og.
Σ
logoy
(ar ma) 2
a)
E (P.))ED.,.
log "
(x- me))))
1) 2o
const.. (9)
(A.))DDAA
is weight that controls the contribution of the OOD samples in the log-likeddhod function. Under the above log-likelihan likelihhodd, the maximum likelihood estimate for μą is
L E(an)eD, I + (1 a) E(aa)ED..
(10)
a + (1 11--jy DD.
We can make use of the above dn to get weighted FLD decision rule using (5).
A.5 DERIVING THE GENERALIZATION ERROR OF THE TarGet TasK FOR SYNTHETIC TASKS wITH WEIGHTED FLD
We consider the synthetic tasks in Appendix A.I with o² notation from Appendix A.3:
We re-write üy from (10) using
noT + m(1 a.)o..k
Ag no m(1 a))
We can explicitly compute Ft, and Fo,k in the OOD-aware setting since we can separate target samples from OOD samples. For the synthetic dataset, the threshold é fat of the hypothesis ha follows normal distribution A ((Haa: oia) where
m(1 - a)
Pha E[Ca]
na + m01 a.)
2² (1- 2m
2. Var[e.] (an + a)
Similar to the Appendix A.3, we derive analytical expression for the expected target risk of the weighted FLD, which is
1 e+ (ha
Pho
+ Pha
(11)
2
V o2
15

---

Under review as a conference paper at ICLR 2023
A.6
ADDITIONAL EXPERIMENTS USING FLD
Exgacted Tagus Generatatoo Entr
(OCD unirown)
4- 5, pm 10 (C00 urloow))
03120
Enotttd Target Gerenaaatttn Ertt
223120
05007
222007
82091-
2.004
0.021
2.001
mun.n 100
mL/n 100
(a)
(b)
Optiaa
Arwlyfa Expacted Cenerilzation
Enggriall Target Caneeal Cannediaattan Eroor
0.105
0.580-
0.180
0.775
0.175
0.105
(c)
(d)
Figure A1: The FLD generalization error (Y-axis) on the target task is plotted against the ratio of OOD samples target samples (X-axis). Figures (a) and (c) plotted using the analytical expressions in (8) and (11) respectively while figures (b) and (d) are the corresponding plots from Monte-carlo simulations. The Monte-carlo simulations agree with the plots from the analytical expression, which validates its correctness. (a) and (b): The figure is identical to Fig. and considers synthetic tasks with 100, = 5 and a = 10 in the OOD-agnostic setting. While small number of OOD samples improves generalization
the target task, lots of samples increase the generalization error on the target task. (c) and (d): The figures consider synthetic tasks with 4, consider the weighted FLD trained with optimal a", then the average generalization error monotonically decreases with more OOD samples. Shaded regions indicate 95% confidence intervals over the Monte-Carlo replicates.
μ 1 a the OOD-awan aware setting.
B EXPERIMENTS WITH NEURAL NETWORKS
B.I DATASETS
We experiment images from CIFAR-10. CINIC-10 (Darlow al., 2018) and several datasets from the Bem benchmark (Gulrajani & Lopez-Paz, 2020): Rotated MNIST (Ghifary et al., 2015), PACS (Li et al., 2017), and DomainNet (Peng et al., 2019). We construct sub-tasks from these datasets as explained below.
CIFAR-10 We use of tasks from Split-CIFAR10 (Zenke et al., 2017) which are five binary classification sub-tasks constructed by grouping consecutive labels of CIFAR-10. The 5 tasks airplane vs. automobile (T1), bird vs. cat (T2), deer vs. dog (T3), frog vs. horse (T1) and ship vs truck (Tb). All the images are of size (3, 32).
CINIC-10 This dataset combines CIFAR- 10 with downsampled images from ImageNet. It contains images of size (3, 32, 32) across 10 classes (same classes as CIFAR- 10). As there are two sources of the images within this dataset, it is natural candidate for studying distribution shift. The construction of the dataset motivates us to consider two tasks from CINIC-10: (1) Task with only CIFAR images, and (2) Task with only ImageNet images.
Rotated MNIST This dataset is constructed from MNIST by rotating the images (which are of size 28, 28). All MNIST images rotated by angle 0º are considered to belong to the same
16

---

Under review as a conference paper at ICLR 2023
task. Hence, consider the family of tasks which is characterized by 10-way way classification of hand-written digit images rotated 0º. By varying 0, we can obtain number of different tasks.
PACS PACS contains images of size (3, 224, 244) with classes present across 4 domains fart, cartoons, photos, sketches]. In experiments, we consider only 3 classes ((Dog, Elephant, Horse)) out of the 7 and consider the 3-way classification of images from given domain as task. Therefore, we can have total of 4 distinct tasks from PACS.
DomainNet Similar PACS, this dataset contains images of size (3, 224, 244) from domains clipart, infograph, painting, quickdraw, real, sketches} across 345 classes. In our experiments, we consider only 2 classes, ((Bird, Plane)) and consider the binary classification images from a given domain as task. As result, we can have a total of distinct tasks from PACS.
B.2 FORMING TARGET AND OOD TASKS
We consider types of setups to study the impact of OOD data:
OOD data arising due to geometric intra-class nuisances We study the effect of intra-class nuisances using classification task as the target task and transformed versions of the same task as different OOD tasks. In this regard, we consider the following experimental setups.
1. Rotated MNIST: unrotated-domain task as target and 0 rotated-domain task as OOD: We consider the 10-way classification (see Appendix B. 1) of unrotated images as the target task and that of the 0 rotated images the OOD samples. We can have different OOD tasks by selecting different values for 0.
2. Rotated CIFAR-10: T2 as target and rotated T2 as OOD: task from Split-CIFAR 10 as the target task. We then rotate the images of T2 by an angle 8º counter-clockwise around their centers to form a new task denoted by 0-12, which we consider as the OOD task. Different OOD tasks can be obtained by selecting different values for
We choose the bird vs. cat (Tp)
3. Blurred CIFAR-10: T4 target and blurred T4 as OOD: We choose the Frog vs. Horse (T4) task from Split-CIFARII 10 as the target task. We then add Gaussian blur with standard deviation a to the images of T4 to form a new task denoted by o-T2, which we consider as the OOD task. By setting distinct values for o, we can have different OOD tasks.
OOD data arising due to category shifts and concept drifts We study this aspect using two different target and OOD classification problems as described below.
1. Aplit-CIFARIVE T. as Target and T3 as OOD: We choose a pair of distinct tasks from the 5 binary classification tasks of Split-CIFAR1O 10 and consider one the target task and the other as the OOD task. We perform experiments for all pairs of tasks (20 in total) in Split-CIFAR10.
2. PACS: Photo-domain task as target and X-domain task OOD: Out of the four 3-way classification tasks from PACS described in Appendix B.1, we select the photo-domain task as the target task and consider one of the remaining 3 domain tasks (for instance, the sketch-domain task) as the OOD task.
3. DomainNet: Real-domain task target and X-domain task OOD: Out of the six binary classification tasks from DomainNet described in Appendix 1, we consider the real-domain task as the target task and select one of the remaining 5 domain tasks (for
instance, the painting-domain domain task) the OOD task.
4. CINIC-10: CIFAR task as target and ImageNet task as OOD: Here we simply select the 10-way classification of CIFAR images as the target task and that of ImageNet as the OOD task.
B.3 EXPERIMENTAL DETAILS
In the above experiments, for each random seed, we randomly select fixed sample of size from the target task. Next, we select samples from the OOD task of varying sizes m such that each progressive sample subset of the next sample. The samples from both target and OOD tasks preserve the ratio of the classes. For rotated MNIST, rotated CIFAR-10, 10, and blurred CIFAR-10. when selecting
17

---

Under review as a conference paper at ICLR 2023
multiple sets of OOD samples, the OOD images that correspond to the selected target images disregarded. For PACS and DomainNet, the images are downsampled to (3, 64, 64) during training. For both the OOD-agnostic (OOD unknown) and OOD-aware (OOD known) settings, each m-value, we construct a combined dataset containing the sized target set and m sized OOD set. We use a CNN (see Appendix B.4) for experiments the both of these settings. We experiment with a fixed to 0.5 (naive OOD-aware model) and with the optimal a". We average the runs over random seeds and evaluate
test set comprised of only target samples.
In the optimal OOD-aware setting, we use a grid-search to find the optimal a for each value of m. We use an adaptive equally-spaced a search set of size 10 such that it ranges from dprev 1.0 (excluding L.0) where aprev is the optimal value of corresponding to the previous value of m. We use this search space since we expect a² to be an increasing function of m.
B.4 NEURAL ARCHITECTURES AND TRAINING
We primarily use 3 different network architectures in our experiments: (a) small convolutional network with 0.12M parameters (denoted by SmallConv), (b) a wide residual network (Zagoruyko & Komodakis, 2016) of depth 10 and widening factor (WRN-10-2), and (c) larger wide residual network of depth 16 and widening factor 4 (WRN-16-4) SmallConv comprises of 3 convolution layers (kernel size and 80 filters) interleaved with max-pooling, ReLU, batch-norm layers, with fully-connected classifier layer in our experiments.
Table Al provides summary of network architectures used in the experiments described carlier. All the networks are trained using stochastic gradient descent (SGD) with Nesterov's 's momentum and cosine-annealed learning rate. The hyperparameters used for the training are, learning rate of 0.01, and weight-decay of 10-5, All the images normalized have mean 0.5 and standard deviation 0.25. In the O0D-agnostic setting, we use sampling without replacement to construct the mini-batches. In the OOD-aware settings (both naive and optimal), we construct mini-batches with a fixed ratio of target and OOD samples. See Appendix B.5 and Fig. A2 for more details.
Experiment
Network(s)
# classes
Image Size
Mini-Batch Size
Rotated MNIST
SmallConv
10
100
(1,28,28)
128
Rotated CIFAR-10
SmallConv, WRN-10-2
2
100
(3,32332)
128
Blurred CIFAR-I0
WRN-10-2
2
100
(3,32,32)
128
Split-CHi CAII-CIFARIO
SmallConv, WRN-10-2
2
100
(3,32,32)
128
PACS
WRN-16-4
3
30
(3,64,64)
16
DomainNet
WRN-16-4
2
50
(3,64,64)
16
CINIC-10
WRN-10-2
10
100
(3,32,32)
128
Table A1: Summary of network architectures used in the experiments
B.5 CONSTRUCTION MINI-BATCHES
Consider a mini-batch Vo, E of size B. Let the randomly chosen mini-batch contains B, target samples and B. OOD samples (B Bt + Bo). Let ê B, and ( denote the average mini-batch surrogate losses for the B, target samples and B, OOD samples respectively.
In the OOD-aware (when know which samples computed explicitly for each mini-batch resulting in the mini-batch gradient
OOD) setting, é,, (h) and (h) can be
(h) = @V& + (1 (h).
(12)
If we were to sample without replacement, expect the fraction of the target samples in every n-m on average. However, if m >>> n, we run into couple of issues. First, we observe that most mini-batches have no target samples, making it impossible to compute (h) Next, even if the mini-batch does have some target samples, there very few of them, resulting in high variance in the estimate (h)
mini-batch to approximately equal n
Hence, we find it beneficial to consider alternative sampling schemes for the mini-batch. Inde- pendent of the values of n and m, we use a sampler which ensures that every mini-batch has fixed fraction of target samples, which denote by B. For example if the mini-batch size B is 20 and if
18

---

Under review as conference paper at ICLR 2023
0.5, then every Mani-batch has 10 target samples and 10 OOD samples regardless of and Note that this sampling biases the gradient, but results in reduced variance estimates, In practice, we observe improved test errors when we set / to either 0.5 or 0.75.
B.6
COMPARING EFFECT USING CONVENTIONAL CUSTOM BATCHES
Target Generalization Error
OOD known (Naive)
OOD known (Optimal)
0.19
Conteettional Batches
Custom Batches - 0.5)
0.18
Cuslom Batches ( 0.75)
0.16
Conventional Batches
0.5)
0.15
Cusiom Betches (8 0.75)
12
12
m/n, 100
m/n, 100
Figure A2: The test error of neural network (SmallConv) the target task (Y-axis) is plotted against the number of samples from the OOD task (X-axis) for the target-OOD task pair of Ti and Tş- One set of curves (lightest shade of green and yellow) considers mini-batches which constructed using sampling without replacement; This is the conventional strategy used in supervised learning. The other curves consider 3 = 0.5 (intermediate shades of orange and green) and 0.75 (darkest shade of red and green). All plots in the OOD- aware setting. Left: If we consider a = 0.5, then the choice of 3 has little effect on the generalization error. Right: However, we use the a" to weight OOD target losses, then generalization error depends on the the choice of f with 0.75 having the lowest test error.
B.7 ADDITIONAL EXPERIMENTS WITH NEURAL NETWORKS
Target: Tz. OOD:
0.42
n= 50
100
Target Generalization Error
0.40
200
0.38
0.36
0.34
0.32
200
400
600
800
1000
the target task (Y-axis) against the number of samples (X-axis) from the OOD task across three different target sample sizes, = 50, 100 and 200 for the target- OOD task pair T2 and from Split-CIFAR CIFARIO. non-monotonic trends in generalization error are present in all the three cases. The trend is less apparent for n = 50 since the number of samples is small resulting in a large variance. Error bars indicate confidence intervals (10 runs).
Figure A3: We plot the generalization
19

---

Under review as conference paper at ICLR 2023
Split CIFAR10 (SmallConv)
OOD Tasks
Task 1
Task
Task
Task
Task
0.38
0.28
0.36
0.20
Task
0.16
0.16
0.16
0.14
0.37
0.35
0.38
Target Generalization Error
Task
0.31
0.30
0.51
0.31
Target Tasks
0.23
0.22
0.31
Task
10
10
min, n- 100
0OD unknown
OOD known (Nalve)
OOD known (Oplimal)
(a)
OOD Tasks
Task
Task
Task
Task
Task
1.0
1.0
Task
Optimal Alpha
Target Tasks
1.0
1.0
1.0
1.0
Task
0.9
0.9
1.0
1.0
1.0
0.9
0.9
0.9
1.0
1.0
10
min, 100
(b)
Figure A4: (a) We plot the test error of SmallConv on the target task (Y-axis) against the ratio of number of samples from the OOD from
the number of samples on the target task (X-axis), for target-OOD task pairs A neural net trained with loss weighted by a is able to leverage OOD data to improve networks ability to generalize on target task. Shaded regions indicate 95% confidence intervals over 10 experiments. (b) The optimal ex" (Y-axis) is plotted against the number of OOD samples (X-axis) for the optimally weighted OOD-aware setting. As we increase number OOD samples, we see that o. increases. This allows us to halance the variance from few target samples and the bias from using OOD samples from different disitribution.
20

---

Under review as conference paper at ICLR 2023
Split CIFARIO (WRN-10-2)
OOD Tasks
Task 1
Task
Task
Task 4
Task
0.32
0.37
0.36
0.25
Task
18
0.17
0.17
0.18
0.44
0.43
0.46
0.4
Target Generalization Error
Task 2
0.30
0.51
0.50
0.31
Target Tasks
0.43
0.34
0.41
044
Task 3
0.25
0.55
0.28
0.25
0.46
0.43
0.42
0.40
Task 4
0.19
0:18
0.19
0.20
0.32
Task 5
0.16
0:18
0.18
0.17
20
min,
0DD uninoram
OOD known (Naive)
ODD known (Optmal)
(a)
OOD Tasks
Task 1
Task 2
Task
Task 4
Task
1.00
1,00
1.00
1.00
Task
1.00
1.00
1.00
1.00
Task 2
Optimal Alpha
Target Tasks
1.00
1.00
1.00
1.00
Task
0.90
0.90
0.50
0.90
1.00
1.00
1.00 -1
1.00 4
Task
0.90
0.90
0.90
0.90
1.00
1.00
1.00
1.00
Task
0.90
0.90
0.90
0.50
10
20
10
20
20
10
20 10 20
m// n-100
(b)
Figure A5: (a) We plot the test of WRN-10-2 the target task (Y-axis) against the ratio of number of samples from the OOD task to the number of samples on the target task (X-axis), for all target-OOD task pairs from Split- COFiR-CIFARIO. neural net trained with loss weighted by a is able leverage OOD data to improve the networks ability to generalize the target task. Shaded regions indicate 95% confidence intervals over 10 experiments. (b) The optimal a" (Y-axis) plotted against the number OOD samples (X-axis) optimally weighted This allows us to halance variance from target samples and bias from using OOD samples from different disitribution.
setting. As we increase the number of OOD samples, we see that a increases.
21

---

Under review as conference paper at ICLR 2023
Split
Target: Tq.
T3 rotated by (SmaliCanv)
Target: T,, (SmallConv)
Target: Ti.
0.20
1.00
0.32
Targel Genersizalion Error
Targel Generaiizatin Ereor
0.14
0.90
Target: T,, OOD: T
Target: T OOD: 7
0.28
Target: T,. T.
Target: Tq.
0.38
1.00
0.20
6
80
0.16
0.31
0.58
15
30
45
10
15
20
15
m/n, 1-100
mim,, n-100
mtn 1-100
(Naive)
known
Figure A6: Left: A binary classification problem (Bird vs. Cat) is the target task and images of these classes rotated by different angles 8 are the OOD task. We see non monotonic curves for larger values 8", For 135 in particular, the generalization error m/n = 50 is worse than the generalization samples, i.e. OOD samples actively hurt generalization. Middle: Generalization error on the target task is plotted against the number of samples from the OOD task for different pairs of target-OOD tasks constructed from CIFAR-10 10 for three settings: OOD-agnostic ERM where we minimize the total average risk over both tasks (rod), objective which minimizes the sum of the average loss of the target and OOD tasks which corresponds to = 1/2 (OOD-awart yellow) and an objective which minimizes optimally weighted convex combination target and OOD empirical loss (green). Right: optimal obtained grid search the three problems in the middle column plotted against different number of OOD samples. Note that the appropriate value very close to but is never exactly 1. other words the OOD samples always benefit we the weighted objective in Theorem 3, even if this benefit is marginal in cases when OOD samples are very different from those the target.
with OOD
22


Reviews:
[{"review_id": "dOMG4-T2npS", "reviewer": "Reviewer_Gur8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that generalization error on the target task is non-monotonic in the number of OOD samples. The authors empirically evaluate this phenomenon on MNIST, CIFAR-10, PACS, and DomainNet. Finally, the authors show how to ensure that the generalization error does decrease monotonically with the number of OOD samples. ", "summary_of_the_review": "Please see weaknesses. I think the paper is interesting but not quite ready for acceptance.", "strength_and_weaknesses": "Strengths:\n- The paper shows and brings insight into the phenomenon that generalization error doesn’t decrease monotonically with the number of OOD samples. \n- The writing is quite clear overall, and the paper is well-structured.\nWeaknesses:\n- Not clear why intuitively generalization error should decrease monotonically with the number of OOD samples. In particular, if the out-of-distribution task is far from the target task, it’s not clear why have those samples would improve generalization on the target task. Additionally, if there are a lot more OOD samples than target task samples, increasing the number of OOD samples intuitively may not improve the generalization on the target task. Can the authors provide a cite for the claim that the monotonicity is intuitive? If not, would suggest changing the framing to deemphasize how intuitive this is. \n- In order to mitigate the non-monotonic nature of the generalization error, the paper requires knowing which samples in the dataset are OOD. In addition, the proposed method is just using a weighed objective, which has been proposed before.\nSmall nits:\n- Not sure if there’s a point to stating Theorem 1.\n- Italicized sentence at the end of page 7 seems important in making a point but doesn’t make sense. Can the authors clarify?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xg3cQ_sTXPj", "reviewer": "Reviewer_Z5vh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper begins by providing a toy example on how OOD samples could reduce generalization error. Then they show that this phenomena can also occur in some real dataset when the training data is extremely small. Finally, under the assumption of knowing the exact target and OOD distribution, they can come up with an optimal weighting to reduce the effect of OOD samples.", "summary_of_the_review": "The paper does provide interesting examples on how OOD examples might effect the model. However, none of the experiment setup/assumption operates in a condition that commonly seem tasks operates in, which limits the scope of the paper.", "strength_and_weaknesses": "Strength:\n- The writing is clear, the analysis seems rigorous.\n\nWeakness: \n- This phenomenon only seems to appear in extremely low data regime, which makes this phenomenon not really counter-intuitive. In a low data regime when the feature space is large, there are generally not much guarantees, which makes the existence of Theorem 1 not that surprising. Being only able to operate in such low data regime also limits the scope of the paper.\n- The choice of n, m and m/n seems quite arbitrary in each figure.\n  - Also I think it should be more reasonable to keep m+n fixed instead of keeping n fixed and increase m. In many of their figures, m/n > 1, which is not common to have more OOD samples than in-distribution samples.\n- Only generalization error is shown. Without also looking into the training/testing accuracy, it is hard to grasp what exactly is happening. One can have a constant classifier, it would always have a generalization error close to 0. In other words, generalization error itself cannot reflect how well the model is learning.\n- The developed algorithmic procedure for reducing the affect of OOD samples is also not very useful as normally we don't even have access to P_o and P_t.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ViDt61c5wvL", "reviewer": "Reviewer_w9jQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how OOD samples within datasets impact the generalization error on the desired task and observe that the generalization error of the task can be a non-monotonic function of the number of OOD samples. This work also develops an algorithmic procedure to train on the target task that is resilient to OOD data.", "summary_of_the_review": "This paper presents interesting observations that are useful for applications. The extensive experiments and ablation studies are good but there are a few weak issues that need to be clarified.", "strength_and_weaknesses": "Strengths:\n\nS1. The main observation that generalization error on the target task is non-monotonic in the number of OOD samples is counter-intuitive and interesting.\n\nS2. The structure of this paper is well-written and easy to follow. The motivation is clear.\n\nS3. The extensive experimental results on numerous datasets look good.\n\nWeaknesses:\n\nW1: It is unclear how to select the threshold of OOD samples, after which the generation error can deteriorate.\n\nW2: How to quantify the OOD samples that can improve generalization.\n\nW3: It would be interesting if the authors can conduct experiments on large vision benchmarks such as ImageNet-O, and ImageNet-R.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3VYag52LDf", "reviewer": "Reviewer_nSTb", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I informed the area chair that I cannot review this paper.", "summary_of_the_review": "I informed the area chair that I cannot review this paper.", "strength_and_weaknesses": "I informed the area chair that I cannot review this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject