Identifying and Benchmarking Natural Out-of-Context Prediction Problems
Authors: David Madras
Richard Zemel
Venue: NeurIPS 2021 Poster (2021)
Keywords: robustness
invariance
spurious correlations
benchmark
out-of-context
domain generalization

Abstract: Deep learning systems frequently fail at out-of-context (OOC) prediction, the problem of making reliable predictions on uncommon or unusual inputs or subgroups of the training distribution. To this end, a number of benchmarks for measuring OOC performance have been recently introduced.  In this work, we introduce a framework unifying the literature on OOC performance measurement, and demonstrate how rich auxiliary information can be leveraged to identify candidate sets of OOC examples in existing datasets.  We present NOOCh: a suite of naturally-occurring "challenge sets", and show how varying notions of context can be used to probe specific OOC failure modes. Experimentally, we explore the tradeoffs between various learning approaches on these challenge sets and demonstrate how the choices made in designing OOC benchmarks can yield varying conclusions.

Identifying and Benchmarking Natural Out-of-Context Prediction Problems
David Madras University of Toronto
Richard Zemel University of Toronto Vector Institute
Vector Institute madras&cs toronto.... edu
Columbia University zemel@cs. toronto.co edu
Abstract
Deep learning systems frequently fail out-of-context (OOC) prediction, the prob- lem of making reliable predictions on uncommon or unusual inputs subgroups of the training distribution. To this end, number benchmarks for measuring OOC performance have been recently introduced. In this work, we introduce a framework unifying the literature on OOC performance measurement, and demon- strate how rich auxiliary information can be leveraged to identify candidate sets OOC examples existing datasets. We present NOOCHE a suite naturally- occurring "challenge sets", and show how varying notions of context can be used to probe specific OOC failure modes. Experimentally. we explore the tradeoff's between various learning approaches these challenge sets and demonstrate how the choices made in designing OOC benchmarks can yield varying conclusions.
Introduction
People often find context useful for prediction, both for improving accuracy and processing efficiency 10. However, deep learning systems frequently over-rely on context cues [88]
19 38 which
lead poor performance out-of-context (OOC) examples, when contextual information is misleading. By OOC examples, we mean inputs which are uncommon or unusual with respect to the training distribution; these can be thought of sampled from under-represented subgroups, or low (non-zero) density regions, of the training distribution. In safety-critical situations, this can be problematic; as such, it important have reliable methods for measuring how well model can perform OOC. Furthermore, given the rise of larger models and datasets [31], there is need for scalable approaches OOC evaluation even manual evaluation of "corner cases" by domain experts may (always) be the gold standard.
key pre-requisite task evaluating OOC performance is identifying which examples should be considered OOC. This identification task a challenging one in and of itself: a natural image, Therefore, any evaluation
"context" be varied, complex and high-dimensional (6, 47 59 (%)
method intending to measure model's OOC performance must (implicitly) select a specific notion of "OOC performance". Indeed, since deep learning yields underspecified models [14), is plausible that different choices may yield different measurements. Common approaches include generating semi--yynheed data to simulate the effect of a shift in some salient feature 32
or using some
auxiliary information to guide choices about what a reasonable OOC set should be 28 33
In this work, develop a conceptual framework for identifying sets of OOC examples in existing datasets. We show how our framework unifies and generalizes prior literature on OOC performance measurement, and allows to utilize complex, structured annotated data define various notions of "OOC performance". We demonstrate this framework's effectiveness through uncovering two O0C "challenge sets" (30] within existing benchmark, each corresponding differing notions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of context. We show how our framework enables scalable and targeted measurement of models' OOC performance through clarifying the relationship between the concept of "OOC performance" and its implementation, allowing for clearer insight on current approaches as well as opportunities for improvement. Our contributions are as follows:
We present NOOCH Naturally-Occuurinn Out-of-context Challenge sets), a suite of "chal- lenge sets" for evaluating performance on naturally-arising OOC problems, available at attps://giit thub. homm/mmdras////// dmadras/noochi
We develop conceptual framework for automatically identifying OOC challenge sets from existing data by leveraging known underlying structure;
We contrast two instantiations of this framework using two notions of "context", defining concepts of "hard positives" and "hard negatives" in the OOC setting; and
We quantitatively analyze the performance of several methods from the robust learning literature on these challenge sets, exploring the tradeoff's inherent in different approaches to OOC performance measurement; and qualitatively demonstrate how rich notions of context can yield rich investigation of OOC errors.
2 Measuring "Out-of-Context Performance"
Intuitively, model which has good "OOC performance" should be able to maintain good performance under unusual or perturbed contextual conditions. We distinguish this from the out-of-distribution (OOD) problem (27]. which is usually concerned with inputs from a different domain than the training set. Rather, the OOC prediction problem is more similar subgroup robustness distributional shift, where a model must perform well on uncommon input regions at training time. However, even after drawing this distinction, the notion is ill-defined: "context" may refer to concepts as varied as object relationships (59), image backgrounds [63], experimental settings 44 or world models 6 Furthermore, even fixing notion of context, the criterion for what should make something "out-of-context" (OOC) is still unclear. For instance, Peters et al. [47] focus on previously unobserved contexts (i.e. environments), whereas Xiao al. [63] are concerned with unusual contexts given the class of interest (i.e. perturbations to image background).
Clearly, defining benchmark to measure a method's OOC performance requires number of design choices, which has enabled a recent proliferation of OOC benchmarks. We note in particular, one of the key choices is around the usage of auxiliary information. Across the literature OOC performance measurement, there are a plethora of approaches to defining OOC criteria using some type of auxiliary information C. For the purposes of algorithm designers, C may be assumed be available at training, validation, and/or test time, or not at all - however, at the the time of benchmark design, is available on sufficiently large portion of the collected dataset guide the designers' choices about what a suitable OOC criterion should be, Examining the current literature on mcasuring OOC performance, we identify the following as unifying framework:
1. Identify some existing auxiliary information C, a variable which takes some value on many (or all) examples and specifies some underlying structure in the data.
2. Select notion of "OOC" (e.g. "images with misleading backgrounds are OOC", "examples from unfamiliar time periods are OOC") and define an "OOC criterion" by choosing a binary function of C.
3. Restrict the test set to those examples where 6 = 1. Optionally, also restrict the training set to those examples where o O.
We show in Table I how a range of prior literature leverages auxiliary information define OOC criteria. We can think of C' as providing benchmark designers with some type of "inductive bias" around what should be considered OOC for given benchmark. The above framework implies that there is diversity of OOC criteria which can be defined over any dataset, and this class is as broad as the class of functions ó which can be defined over the available auxiliary information. In the rest of the paper, we take advantage of the flexibility of this framework to give two examples of such an approach. We show that by leveraging more complex annotated structure, we can create multiple OOC benchmarks from an existing dataset using multiple criteria for what should be considered "OOC". We trace out how the choices made in designing these criteria correspond to different notions of context, and demonstrate experimentally that these yield varying measurements of OOC performance.

---

Dataset
Auxiliary Information C'
00C function
Waterbirds [54
if background is water (binary)
C
WWildCam2220-Widd 33
camera trap ID (categorical)
(1... (1.....4455) 245}
FMoW-Wilds 33
time stamp (ordinal)
Ctime 2 2013
Imagenet-A 28
max NLL ensemble (continuous)
C2 log (0.15)
Breeds [55
subclass (categorical)
C target subset
Table 1: Examples of OOC benchmarks from the literature under our framework. The right-most
column lists the condition under which 1. The OOC function in 28 has several additional
filtering steps, some heavily manual.
Nodio
Hard Pouitre Oge
Pard/rotthe buffberds
HOVE Norgaten Fooes, hal
Har Neginie Harthwand
Using the co-occuurennceeettaciiiillll (CE) criterion, examples of (0.05, 0.1)-hard positives (top row) and negatives (bottom row) for the classes (L to R): kite, sports_ball, surfboard.
Figure 1:
Finding Naturally-Occurring OOC Problems
We now demonstrate concretely how rich auxiliary information can be used to study the way that context shifts arise naturally within an existing computer vision benchmark, and provide two criteria for OOC performance that can be computed from these annotations. Throughout, we consider the binary prediction task of determining object presence, a problem where relationships between various objects naturally provide helpful context given image X, is an object of class Y present or not?
Background: COCO and COCO-Stuff. The Microsoft Common Objects in COntext dataset (COCO) 36) is a computer vision dataset consisting of images natural scenes. Each image is annotated with instance labels and segmentations for every "thing" in the image, well several captions describing the content of the scene. Images usually contain multiple items and as such usually have multiple labels. However, for the purposes of investigating OOC prediction, many relevant objects are not labelled in COCO; for instance, background objects such as "sky" or "grass" not COCO classes. Fortunately, the COCO-Stur dataset (7) provides labels and segmentations for all of the "stuff" in the images from COCO; thing is an object with specified size and shape, whereas stuff has no "defined spatial extent" (16). Having both thing and stuff labels is essential for understanding model behaviour on OOC examples, since it is exactly these "stuff" classes which often (but not always) provide important context cues. Taken together, the thing and stuff annotations yield a rich sandbox for queries about the role of context in prediction. For our purposes, COCO-Stuff contains 171 binary tasks for determining object presence (81 thing classes and 90 stuff classes).
3.1 Automatically Identifying 0OC Examples: Hard Positives and Negatives
We develop two contrasting notions of "OOC": 1. the presence/abseene of frequently co-occurring, easily extractible objects; and 2. an unusual "gist" of a scene. We define these notions below, presenting two criteria for identifying naturally -cccurlly OOC prediction problems within the

---

f1
hard positive (CF)
(b) hard positive (Gist)
(c) hard negative (CE) (d) hard negative (Gist)
Figure 2: To contrast the CE and Gist criteria, we show samples from the airplane (L) and bowl (R) tasks.
existing COCO(-Stuff) dataset, and discussing how we can use annotations as proxies to define an OOC indicator 6. We identify two types of OOC examples: hard positives, where the class is present despite an unusual context, and hard negatives, where the class is not present, despite a usual context.
3.1.1 Defining Context Using Co-Occurrences and Extractibility
For an object class Y, context cues often come in the form of another object class C' that has two properties (cf. (88) First, C and co-occur frequently [5]. Second, C is more extractible than Y easier to detect. If C' were less extractible than Y it would not be useful cue for detecting Y. as model could detect Y directly.
We can utilize these properties to create candidate context cues C for a class of interest Y. Given segmentations, we can use an object's size within image proxy for extractibility (larger objects tend to be more extractible). Let the Area operator take the sum of the areas of all segmentations of instances of that object, or return 0 if the object is not present. Then, to estimate from the training set how important of a context variable C' is, we can compute A(C, Y) = B[Area(C))
Area(Y
1]. When ACC, Y) is larger, this means that when Y is present, C usually also present, and on average. takes up more of the image than Y does. When A(C, Y) > a, we say that C' is an a-strong context cue for Y (or just a-context for brevity). We find that many of the contexts identified using this method for large enough a are intuitive. Some examples of (label, 0.05-context) pairs are: (car, road), (bowl, diningg table), (cow, grass).
Using this notion of context, we can then define hard positive and hard negative examples. We make the simplifying noisy-or assumption: that each context cue provides evidence for Y, so that the presence of any cue supports Y being present, while the absence of all provides evidence against Y's presence. Given some image, we can define an (a, 33)-hadd positive or negative. If Y 1, and for all a-context cues C, we have Area(C) < / in this image (and there is at least one a-context cue), then the example is an (a, 9)rhard positive. Alternatively, if Y O, and there exists some a-context variable C' such that Area(C)= > 8. then the example is an (a, -ar- negative. We will call this method the co-occurrencc/exttraiiiiti
(CE) criterion (see Fig for examples). Throughout, we use 0.05, B = unless otherwise noted: these parameters were chosen since they approximately equalize P((X, Y) (a, hard Y y) across y = 0, 1. See Appendix B for more details.
3.1.2 Defining Context Using Gist
We now turn to broader notion of context, that of the "gist" of scene [59, or its overall semantic content. This is something that humans can recognize easily (35). but goes beyond object frequency and extractibility. When an object present scene whose gist is very different from the scenes the object was present in at training time, this may make prediction difficult.
We describe our method for estimating gist shift, which we call the gist criterion. We use caption annotation data in COCO (cach image has 5 caption sentences), making the assumption that informa- tion in a caption captures the gist of scene. We then take an SBERT embedding [48] of each image caption for an image, and average these embeddings to get a single embedding for that image. Then, for a given image and some target label Y, we find the cosine similarity between that image's embedding and the average embedding across all training images with Y = 1. If this similarity is below some threshold т, and Y for the test image, it is a r-hard positive; if this similarity is above and Y = 0 for the test image, it is r-hard negative. Note, we do not look at distance to Y examples; we assume that captions for Y = 0 images may have little in common, whereas the mean caption for Y = 1 is prototypical description of Y "in context". Throughout, we set

---

Ner
00 8.1 05
96
We find that hard positives/negaaives induce higher average loss for both criteria. Each point is task in COCO-Stuff: the x- and y-axis values show the average loss achieved by an ERM model on all positives (negatives) and the average loss on hard positíves (negatives). From L to R: CE criterion (positives), CE criterion (negatives), gist criterion (positives), gist criterion (negatives). The diagonal line represents where the hard example losses match marginal losses.
Figure 3:
the threshold for each task so that the number of hard positives and negatives is the same as the CE criterion for that task, to facilitate comparisons. See Figure [ examples of hard positive and negatives chosen by this criterion in comparison to the CE criterion, and Appendix B for more details.
3.2 NOOCH-CE and NOOCH-Gist: Selecting Challenge Sets
We train binary classifiers minimize average NLL on each of the 171 classes in COCO-Stuff. We find that for nearly all tasks, the hard positives and negatives defined by our methods incur higher average loss than positive and negative examples respectively (Fig. 3). for both the CE and Gist criteria. This provides some evidence that our criteria are, in fact, identifying examples which are more difficult to classify correctly.
To select candidate OOC tasks for our challenge sets, we select the 12 tasks with the largest difference between average NLL hard examples (by the CE criterion) and average NLL on all examples. We call these tasks the NOOCH NNauurally-Ocuurrin We then identify two groups of challenge sets: NOOCH-CE, which consists of the hard positive and negative examples on each of the 12 tasks in NOOCH as identified by the CE criterion; and NOOCH-Gist, which is the analogous set for the gist criterion. These tasks are: car, bowl, boat, fire-hydrant, airplane, cow, backpack, cup, surfboard, tie, sports-ball kite. This gives 24 total challenge sets on which to evaluate an ML model's OOC performance.
Out-of-context Challenge) suite (Table (5)
Contrasting the Two Criteria (Fig. 2). The left two images in Fig. 2 are hard positives on the airplane task. On the far left, we see the inside of an airplane: this is selected as hard positive by the CE criteria because there is no sky visible. On the second left, the sky is visible but the overall scene is unusual; this is selected as hard positive by the Gist criteria. The right two images are hard negatives on the bowl task. On the second right, we see large dinner with many plates: this is selected as hard negative by the CE criteria because there is prominent dining table but no bowls. On the far right, we see paper plate on desk (not a dining table): this is selected as a hard negative by the Gist criteria since there is no bowl but it is similar to images where you might expect a bowl.
Evaluating Robustness Approaches on NOOCH
We now turn to evaluating various approaches on the NOOCH benchmarks. We focus on four categories which provide useful contrast between different approaches to OOC prediction: expected risk minimization, label-based adjustments, environment-based methods, and adaptive methods.
We are given dataset {I; y/ ) with inputs and target labels respectively, and possibly some side information {c as well (e.g. environment variables). If side information available, we assume it is available at training time but not necessarily test time. We aim to learn a function f: W. We assume to the the example-wise cross-entropy loss function.
Notation.
Expected Risk Minimization. Expected risk minimization (ERM) the standard paradigm for training ML. models; Guirajani and Lopez-Paz (20) show it can be difficult to beat on domain generalization problems. ERM, we minimize the mean loss l on the training set: LERM (f) = En (())..)))
5

---

Label-Based Adjustments. Let w, P(Y yr) in the training set. For fairer comparison with environment- based methods below, we add a tuning parameter a to control the degree of adjustment. = represents the standard versions of these loss functions, but we found other values a E [0, 2 to be useful. We weight the loss for each example proportional to ( (label reweighting Reweight), or sample each point with probability proportional to (label undersampling US).
4.1 Environment-Basn Based Methods
One common setting is where the auxiliary information e; for each example is categorical variable, representing an environment. This type of grouping procedure can be used separately from its original causal context [47]: we consider it to represent some informative partition our data. Here, E {1... CJ is an integer, is the number of examples in environment c, and the average loss for an environment is: (f( 1 Ei= (r)())) () = =c.
Group DRO. Group Distributionally Robust Optimization (GDRO) 54 aims to minimize the loss on the worst of C' partitionns The group adjustment with hyperparameter K ensures greater focus on max
smaller groups: LGDRO(S) cE cE(I...C) { (e(()) +
K
vnc
IRM. Invariant risk minimization (IRM) (2), uses a gradient penalty on the output of f. with w a constant multiplier on the output of f, and hyperparameter X. The intuition is somewhat involved, but motivation is to learn representation such that the same predictive classifier optimal across ing/undersampling above, but with w; P(C c.) (Reweight-Enys, US-Envs).
environments: LIRM(f) E fc(f) + V| f)ll-
Environment Reweighting and Undersampling. These are equivalent to label reweight-
4.2 Adaptive Methods
We also consider loss functions which, rather than using side information to specify which examples are OOC, focus dynamically on the hardest examples at each step. In conditional variance-at-risk optimization (CVaR) [49], we aim minimize the loss over a worst-case distribution over training examples. For some E (0, 1), CVaR(p) is defined as the average loss of the p-percent worst-loss examples. focal loss 337. we dynamically upweight high loss examples using parameter y 2 With binary let q(z, y) = yf(x) + (1 g) f())). Then, at Y 0 focal loss reduces to cross enoss-y as increases, focuses more on the examples which have higher loss already: Focal (f) = Di - -(1 q y))² 7 log(g(æ;,) g(z,,)))
Related Work
range of datasets have been proposed for the purposes of benchmarking OOC prediction. Sev- eral focus on realistic OOC prediction: Koh et al. (33) contains problems from across range of applications: Hendrycks al. (8 select examples which perform worst on an ensemble of models; Barbu et al. [4] focuses object recognition and shows objects varied by a range of attributes; Choi et al. [8] isolates 26 images of objects in unusual contexts from a larger dataset. Our work functions well as complement to any of these datasets; we believe it is novel due to our ideas for scalably identifying challenge sets from annotated data, as well as its delincation of hard positives and hard negatives. The notion of "challenge sets" (30], "stress tests" [41]. or "contrast sets" [1]] from the
NLP literature is an inspiration for our work as well. A range of primarily semi-synthetic datasets
include those that center around: image corruption 26 39]; object hierarchy [55]: synthetic shifts in
background [54 63
color
332 group attribute
1 or purely synthetic data 3
Several works have discussed explicit examples where deep models failed to perform OOC prediction practice. Oakden- Bakden-Rayner al. [44 and Winkler et [6]] discuss the risk of this occurring the
medical domain, and Shetty et al. [7 in the autonomous driving domain. Other works have detailed the challenge of OOC prediction for deep models, using frames of "shorteuts" [19) "simplicity" [56).
"extractibility" [38]. texture biases in CNNs [18), or the challenge of out-of-place objects [50].

---

Task
ERM
CVaR
Focal
Reweight
US
Reweight (Envs)
US (Envs)
GDRO
IRM
0.769
0.787
0.759
0.773
0.773
0.886
0.846
0.891
0.868
bowl
0.749
0.781
0.734
0.751
0.759
0.864
0.814
0.865
0.828
boat
0.869
0.888
0.823
0.866
0.877
0.954
0,923
0.945
0.925
fire-hydrant
0,913
0.933
0.908
0.927
0.913
0,933
0,920
0.946
0.942
airplane
0.986
0,984
0.983
0,986
0.985
0.991
0,986
0.991
0.986
cow
0.935
0,937
0,932
0,939
0.938
0,963
0.948
0.963
0,943
backpack
0.812
0.806
0.809
0.816
0.812
0,813
0.816
0.871
0.844
cup
0.870
0.863
0.867
0.876
0.873
0,879
0.875
0.825
0.869
surlboard
0.939
0,947
0,933
0,940
0.939
0,960
0.940
0.960
0.957
0.742
0.752
0.728
0.760
0.763
0.756
0.761
0.806
0.775
sports-ball
0.867
0.869
0.871
0.869
0.868
0,911
0.890
0.911
0.894
kite
0.932
0.932
0.937
0,940
0,928
0.949
0.936
0.960
0.950
Average
0.865
0.873
0.857
0.870
0,869
0.905
0.888
0.911
0.899
Table 2: AUC hard test examples for all 12 NOOCH-CE stress tests, after hyperparameter
selection. Bold numbers have overlapping standard deviations with the highest observed mean's s.
Task
FRM
CVaR
Focal
Reweight
US
Reweight (Envs)
US (Fnvs)
GDRO
IRM
car
0.766
0.785
0.763
0.768
0.773
0.845
0.823
0.863
0.832
bowl
0.642
0.686
0.635
0.656
0.678
0.692
0.698
0.704
0.670
boat
0.826
0.855
0.771
0.823
0.823
0.897
0.868
0.884
0.872
fire-hydrant
0.840
0.865
0.822
0.849
0.841
0.834
0.836
0.823
0.845
airplane
0.977
0.979
0.975
0.978
0,976
0.978
0.977
0.981
0.977
cow
0.901
0.912
0.877
0.903
0,906
0.908
0.908
0.911
0.896
backpack
0.716
0.717
0.725
0.731
0.749
0.729
0.736
0.732
0.750
cup
0.733
0.738
0.727
0.735
0.742
0.759
0.771
0.742
0.755
surfboard
0.913
0.920
0.900
0,912
0.914
0.909
0.912
0.882
0.894
tic
0.822
0,816
0.813
0,829
0.824
0,829
0.831
0.835
0.840
sports-ball
0.830
0.832
0.828
0,830
0.822
0.866
0.859
0.880
0.855
kite
0.942
0.939
0.945
0.947
0.936
0.947
0.939
0.947
0.943
Average
0.826
0.837
0.815
0.830
0.832
0.849
0.847
0.849
0.844
Table 3:
AUC on hard test examples for all 12 NOOCH-Gist stress tests, after hyperparameter
selection. Bold numbers have overlapping standard deviations with the highest observed mean's 's
Metrie
ERM
CVaR
Focal
Reweight
Reweight (Envs)
US (Envs)
GDRO
Worst-Group Error
0.4
0.354
0,402
0.361
0.336
0.233
0.146
0.108
0.127
Worst-Group NLL
1.03
0.657
0.722
0.884
0.883
0.535
0.417
0.306
0.346
AUC-Hard
0.691
0.703
0.506
0.657
0.701
0.744
0.778
0,929
0.788
Table 4: AUC on "hard" test examples on the Waterbirds dataset. Hard examples are the union two
groups: land birds on water backgrounds and water birds on land backgrounds. Worst-group error or NLL takes the worse the two; AUC-Hard calculates the AUC across the union.
range of work outside deep learning considers the OOC prediction problem from a different direction, focusing on how to improve prediction by taking context into account 9 25
Other work looks the idea of using latent variable to represent scene gist [45] 62]. number newer methods not discussed elsewhere in the paper also aim to solve the OOC problem, including those
40
65
that involve side information [29] 34 58 (44, those that involve side information through causal
underpinnings 24 [22. and those that ignore side information altogether [12 13
6 Experiments
In this section, we compare and contrast the various measurements of OOC performance yielded by NOOCH, along with the semi-synthetic Waterbirds dataset [54) For all experiments we use
ResNet-50 (23], finetuned from ImageNNtt-peerraine features [53]. See App. B C further experimental details. For the environment- based methods, we follow Sagawa et al. [34] and create 4 environments: 1 for each element of the cross-product of the label and its highest-a context class. Many of the robust baselines from Sec. come with a hyperparameter which aims to trade off between average performance and OOC performance; we choose the hyperparameter which minimizes the maximum loss of hard positives and hard negatives the validation set.
7

---

6.1 Quantitative Analysis
6.1.1 Main Results: Different Criteria Yield Different Evaluations
In Tables 2 and 3 we show AUC (area under the ROC curve) on hard examples for each method focus and each of the 12 NOOCH tasks individually. We choose AUC as metric since it is robust to label imbalance, and tasks in NOOCH contain 1-10% positive examples.
54 semi- synthetic dataset which generated by pasting images of birds on top of either either land or water backgrounds; the goal is to classify land birds from water birds. At training time, land birds mostly shown on land (and water birds on water), but at test time we hope to perform well, regardless of background. The information regarding background is made available through environments: the four environments are land/land, land/water, etc. For Waterbirds, we show both AUC-Hard (our metric) and worst-group eror//NLL (the metrics from Sagawa et al. (4
For comparison, we show in Table 4 an analogous table of results for Waterbirds
Environments are More Useful on Simpler Benchmarks. We note that the three benchmarks yield varying conclusions about the methods in question; in particular, the relative performance the best mnvironment-basee methods vary greatly between the benchmarks. We find that environment- based methods perform better on the benchmarks where the context shift's structure (i.e. the form of σ) is better specified by the environments. In Table 4 we see there is a very large gap in performance between the best environment- based methods (particularly GDRO) and the methods that do not use environments. For instance, GDRO and IRM improve over ERM by about 0.3 in worst-group error (the metric of choice in Sagawa et al. S4 ; and GDRO improves over all other methods by about 0.14 in AUC on hard examples. This difference is order of magnitude greater than observed either NOOCH benchmark, suggesting that semi-synthetic datasets (such as Waterbirds), may overestimate the performance of current environment-basee based methods, and possibly GDRO in partieular.
We also see this when comparing NOOCU-CE to NOOCUI-Gst: the environment- baaed methods are also the ones whose performance falls off the most from NOOCU-CE to NOOCN-Gist. This could be because NOOCIICEE a simpler benchmark, whose notion of OOC more well specified by the given environments. The contrast between 2, Table 4 documents the usefulness of having benchmarks for robustness across a range of complexity, and motivates the creation of benchmarks such as NOOCH.
Gist Shift is Difficult. We further note that performance on NOOCII-Gist is generally worse than on NOOCH-CE, This suggests that the more complex notion context embodied by the gist yields a more difficult OOC task. In some ways, would expect models to find these more holistic shifts harder, as these gist shifts go well beyond "shortcuts" [9]]
Access to Auxiliary Information More Important than Algorithm. Overall, environment- based methods perform the best on all three benchmarks - we expect this to be the case, since these methods are given access to structure which is relevant to the OOC task at hand. This is mostly clearly indicated in the improvement between rather than labels. In fact, we find that on the more complex NOOCH benchmarks, reweighting examples by environment performs similarly to more specialized methods such as GDRO/IRM, given an equivalent amount of hyperparameter tuning. This suggests that current environment- algorithms have significant room to improve when it comes to more complex OOC benchmarks.
methods when using environments
6.1.2 Secondary Observations
In Figures 4 5 6 we show AUC, mean negative log-likelihood (NLL), and expected calibration error (ECE) 21 respectively, averaged across the 12 tasks. We show results for classification error in App. For AUC and ECE, display results for hard and easy examples separately (where an "casy" example is defined as one which is not "hard", ie. not a hard positive or hard negative). For NLL, we further break out the results on hard examples into hard positives and hard negatives. For all, we show NOOCH-CE (L) and NOOCH-Gist (R).
Tradeoffss Exist Between Harder and Easier Examples. We note two tradeoff's across AUC, and NLL results: models that are better hard examples tend to be worse easy examples (and vice
8

---

Had
Hadl
ERM
3.50
1.30
0:00
Reweight
Hinwaght
US
255
US
3.50
N
11:0
Rowaln [Ero]
(Eres)
0:00
Romight [Eno]
US
CDRO
Figure 4: AUC (area under the ROC curve) achieved on hard and easy examples. Higher is better.
Hay Sega
Hod Negatinn
14
OGB
055
CWR
Rewwight
Foul
FRM
125
133
1.3
028-
005
CVuR
Howeight
Foca
3
0.1+
Rovcialt (Emaa
1.30
R
03
(Eres)
3.75
01B
US (Eres)
GARD
AARD
D46
IMM
3.35
3.51
Figure 5: Negative log-likelihood (NLL) achieved on hard positive, hard negative, and easy examples.
Facy
6.12
FRM
395
1.200
0.33
Rowiigh
Hrowight
3.113
US
8:09
Beoeight [Env]]
Brusgge [Uiwe]
8.54
E01
US
1.50
E
Figure 6: Expected Calibration Error (ECE) achieved on hard and easy examples. Lower is better.
versa), and models that are better on hard positives tend to be worse on hard negatives. As we expect, ERM is perfroms better on easy examples and hard negatives: since the dataset imbalanced, negatives are the majority class and the hard negatives are the easier "hard" examples.
Adaptive Methods Find Different Tradeoffs.
We find that CVaR and focal loss find interesting tradcoff's between ERM and environment-basee methods. CVaR performs comparably to ERM on both hard and easy examples by AUC; and by ECE is by far the worst of any method. How- ever, CVaR'ss NLL on hard positives is between ERM and the environment- based methods'. Fo- cal loss, on the other hand, performs similarly to ERM by AUC on both hard and easy examples, but it performs similarly to the environment- based methods on hard positives and negatives by NLL, providing strong tradcoff between overall and OOC performance.
Figure 7: Test examples where GDRO most improves over ERM (L) hard positives and (R) hard negatives from the NOOCH-CE car category. Titles show NLL for each method on that image.
Two Surprising Calibration Results. We found that there was not the same tradeoff with calibration as there was with the other metrics. In particular, ERM and undersampling had the best calibration on both hard and easy examples, suggesting path forward to avoid tradeoff' be- tween OOC and average performance by thinking more about calibration. Secondly, we found the environment-basee methods were not better calibrated, even on hard examples.

---

6.2 Qualitative Analysis
show the images with the largest gaps in NLL on the car task among hard positives and hard negatives (in NOOCH-CE). The hard positive and hard negative where GDRO most overperformed are both images we might expect, where the object label and context do not match: a living room with a tiny toy car on the shelf, and normal street scene that happens have any cars. See App. D for analysis on where GDRO underperforms ERM.
Where Environment-bast Learning Improves Over ERM. In Fig.
Contrasting NOOCH-CE and NOOCH-Gist. In Fig. 8 we use examples of hard positives from the cow task to contrast the CE and gist criteria. On the left, we show a hard positive from NOOCH-CE, with cows standing on pavement: this is hard positive in NOOCH-CE since there is no grass; GDRO and IRM outperform ERM on this image. However, it is not a hard positive in NOOCH-Gist, since cows are the central focus of the image. On the right, show a hard positive from NOOCH-Gist, where the image focuses on a giraffe, but there are several white cows in the background standing on grassy field. This is not a hard positive in NOOCH-CE, due to the large field, but it is a hard positive in NOOCH-Gist, since the giraffe is the focus of the image. ERM outperforms GDRO and IRM on this image the anvironment-based objectives do not encourage as strong performance where the context (grass) and object (cow) align.
7 Discussion
Limitations. The idea of computa- Hor (Car), CIM canno: 144, 18M 2.6 tionally identifying OOC examples is necessarily limited: by their nature, OOC examples are exceptions to rules, and there will always be OOC exam- ples which can only be discovered through qualitative examination. Fur- ther, the task chosen for our bench- mark was chosen for simplicity of analysis in research context rather than real world significance - out- putting segmentations or many-way classification may be more applicable to most applications. We hope that the impact of our work is to enable better evaluation of model performance on atypical or under-represented examples, both through usage of these challenge sets and ones inspired by the ideas presented here. However, the usage of bench- marks can have downsides where standardized performance metrics are prioritized over fundamental advances in modelling which not captured in those metrics. In various applications of interest, results may need to be reported across number of different metrics since this gives a clearer picture of model performance AUC may not always be (or the most) relevant metric.
Hed Rss (CF) FR 52, CORO 1PM-111
Figure 8: Examples of hard positives from NOOCH-CE (L.) and NOOCH-Gis (R) for the cow task. Titles show NLLL for several methods that image.
Conclusion & Looking Forward. this work, we study OOC evaluation as its own field, drawing attention to the range of evaluation schemes which can be used and the ways in which they may differ. We demonstrate that using auxiliary structure in the data can be a useful means of defining context, and that this structure can be used in rich ways to identify various faces of the OOC problem. Through this exploration, we find that methods which take advantage of auxiliary information may be more generously evaluated by OOC benchmarks which more are cleanly defined by that information.
In closing, we would like to highlight the idea systematizing OOC evaluation, whether this is through automatic discovery of OOC examples through annotations or some other means. This not only enables the scalable creation of challenge sets, but allows for faster generation and exploration of new hypotheses about the type of context shifts that models struggle possibly analogous to automatic test generation in the debugging literature [T] 42 We hope that methods of this type can be applied elsewhere to better understand the challenges of OOC prediction.
10

---

Acknowledgments and Disclosure of Funding
Thanks to Marc-Etienne Brunet, Eleni Triantafilliou and Elliot Creager for their helpful thoughts on the manuscript, as well as to four anonymous reviewers for useful feedback. David Madras was supported by an NSERC Alexander Graham Bell Canada Graduate Scholarship-Doctorall (CGS- D). Resources used in preparing this research were provided, in part, by the Province of Ontario, Government of Canada through CIFAR, and companies sponsoring the Vector Institute vectorinstitute.. ai/#partners)
Www
References
[1] R. Adragna, E. Creager, D. Madras, and R. Zemel. Fairness and robustness in invariant learning: case study in toxicity classification. arXiv preprint arXiv:201 1,06485, 2020.
[2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 02893, 2019.
[3] B. Aubin, A. Stowik, M. Arjovsky, L. Bottou, and D. Lopez-Paz. Linear unit-tests for invariance discovery. arXiv preprint arXiv:2102. 10867, 2021.
[4] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32:9453-9440 2019.
[5] I. Biederman, R. J. Mezzanotte, and C. Rabinowitz. Scene perception:
Detecting and judging objects undergoing relational violations. Cognilive psychology. 14(2):143-177, 1982.
[6] A. Bobick and C. Pinhanez. Using approximate models source of contextual information for vision processing. In Proc. of the ICCV, volume 95, pages 13-21. Citeseer, 1995.
[7] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209- 1218, 2018.
[8] M. J. Choi, A. Torralba, and A. S. Willsky. A tree-based context model for object recognition. IEEE transactions pattern analysis and machine intelligence, 34(2):240-22) 252, 2011.
[9] M. J. Choi, A. Torralba, and A. S. Willsky. Context models and out-of-context objects. Pattern Recognition Letters, 33(7):853-862, 2012.
[10] M. M. Chun. Contextual cueing of visual attention. Trends in cognitive sciences, 4(5):170-178 2000.
[11] D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton. The combinatorial design approach to automatic test generation. IEEE software, 13(5):83-88, 1996.
[12] E. Creager, J.-H. Jacobsen, and R. Zemel. Environment inference for invariant learning. In ICML Workshop on Uncertainty and Robustness, 2020.
[13] N. Dagaev, B. D. Roads, X. Luo, D. N. Barry, K. R. Patil, and B. C. Love. A loo-good-to-ber be-tree prior reduce shortcut reliance. arXiv preprint arXiv:2102. 06-00000066622. 2021.
[14] A. D' Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Fisenstein, M. D. Hoffman, al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint 100X::20/11033992 2020.
[15] J. Duchi and H. Namkoong. Variance-based regularization with convex objectives. arXiv preprint 1111:1111, 2016.
[16] D. A. Forsyth, J. Malik, M. M. Fleck, H. Greenspan, T. Leung, S. Belongie, C. Carson, and C. Bregler. Finding pictures of objects large collections of images. International workshop on object representation in computer vision, pages 335-360. Springer, 1996.

---

[17] M. Gardner, Y. Artzi, V. Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1307-1323, 2020.
[18] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1eti:121121111 2018.
[19] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11)665-673, 2020.
[20] I. Guirajani and D. Lopez-Paz. In search of lost domain generalization. arXiv preprint 0rXiv:2007.01434, 2020.
[21] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[22] R. Hamon, H. Junklewitz, and I. Sanchez. Robustness and explainability of artificial intelligence. Publications Office of the European Union, 2020.
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[24] C. Heinze-Dene and N. Meinshausen. Conditional variance penalties and domain shift robust- ness. Machine Learning, 110(2):303- -38,, 2021.
[25] G. Heitz and D. Koller. Learning spatial context: Using stuff to find things. In European conference on computer vision, pages 30-43, Springer, 2008
[26] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv: /2261, 2019.
[27] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv: 1610.02 136, 2016.
[28] D. Hendrycks, K. Zhao. S. Basart, Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[29] W. Hu, G. Niu, Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pages 2029-2037. PMLR, 2018
[30] P. Isabelle, C. Cherry, and G. Foster. A challenge set approach to evaluating machine translation. arXiv preprint arXiv: 104400743,,
[31] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 008361 2020.
[32] B. Kim, H. Kim, K. Kim, S. Kim, and Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9012-9020, 2019.
[33] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips. I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:20/2.07421, 2020.
[34] D. Krueger, E. Caballero, J. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. L. Priol, and A. Courville. -ut-of-distriaattio generalization via risk extrapolation (rex). arXiv preprint arXiv:2003.00688, 2020.
12

---

[35] A. M. Larson, T. E. Freeman, R. V. Ringer, and L. C. Loschky. The spatiotemporal dynam- ics of scene gist recognition. Journal of Experimental Psychology: Human Perception and Performance, 40(2):471, 2014.
[36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[37] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[38] C. Lovering, R. Jha, Linzen, and Pavlick. Predicting inductive biases of pre-trained models, 2021.
[39] C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel. Benchmarking robustness object detection: Autonomous driving when winter coming. arXiv preprint arXiv:/907.070: 1907.07484 2019.
[40] K. Murphy, A. Torralba, W. Freeman, et al. Using the forest to see the trees: graphical model relating features, objects and scenes, Advances in neural information processing systems, 16: 1499-1506, 2003.
[41] A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig. Stress test evaluation for natural language inference. arXiv preprint arXiv:/806.00060 2018.
[42] C. Nebut, F. Fleurey,
Traon, and J.-M. Jezequel. Automatic test generation: use case driven approach. IEEE Transactions on Software Engineering, (433):400-555 2006.
[43] A. Noack, I. Ahem, D. Dou, and B. Li. An empirical study on the relation between network interpretability and adversarial robustness. SN Computer Science, 2(1):1-13, 2021.
Oakden- [44] L. meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151-159, 2020.
Dunnmon, G. Carneiro, and C. Ré. Hidden stratification causes clinically
[45] A. Oliva and A. Torralba. Building the gist of a scene: The role of global image features in recognition. Progress in brain research, 155:23-36, 2006.
[46] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[47] J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series (Statistical Methodology), pages 947-1012, 2016.
[48] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019.
[49] R. T. Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distributions. Journal of banking & finance, 1443-114444
[50] A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant in the room. arXiv preprint Xrxiv: I 03305, 2018
[51] A. Ross and F. Doshi-Veles, Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[52] D. Rothenhäusler, N. Meinshausen, Bühlmann, and J. Peters. Anchor regression: heteroge- neous data meets causality. arXiv preprint 1801.009,,,, 2018.
[53] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1155)):21-2222
13

---

[54] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: TtV.:/911.07750
2019.
[55] S. Santurkar, D. Tsipras, and A. Madry. Breeds: Benchmarks for subpopulation shift, arXiv preprint arXiv:2008.
0488, 04859, 2020.
[56] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pitfalls of simplicity bias neural networks. arXiv preprint arXiv:2006.07710 2020.
[57] R. Shetty, B. Schiele, and M. Fritz. Not using the car to see the sidewalk--uantifying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $218-8226 2019.
[58] M. Srivastava, T. Hashimoto, and P. Liang. Robustness spurious correlations via human annotations. In International Conference on Machine Learning, pages 9109-9119. PMLR, 2020.
[59] A. Torralba. Contextual priming for object detection. International Internattionaloural of computer vision, 53(2):169-191) 2003.
[60] E. Triantafillou, H. Larochelle, Zemel, and V. Dumoulin. Learning a universal template for few-shot dataset generalization. arXiv preprint arXiv:2105.07000 2021.
[61] J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmannn Welnnnnooe,llff, L. Thomas, A. Lallas, A. Blum, W. Stolz, et al. Association between surgical skin markings in dermo- scopic images and diagnostic performance of deep learning convolutional neural network for melanoma recognition. JAMA dermatology, 155(10): 13350) 2019.
[62] K. Wu, E. Wu, and G. Kreiman. Learning scene gist with convolutional neural networks to improve object recognition. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pages 1-6. IEEE, 2018.
[63] K. Xiao, L. Engstrom, A. Ilyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.099: 2020.
[64] S. M. Xie, A. Kumar, R. Jones, F. Khani, T. Ma, and P. Liang. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. arXiv preprint arXiv:2012.045500 2020.
[65] B. Yao and L. Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 17-24, IEEE, 2010.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss how our work
complements existing benchmarks; see the Supplemental Material (App. A for
continuation of discussion around limitations of our evaluations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
14

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material included in the supplemental material: App. B
URL)? [Yes] These
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they
were chosen)? [Yes] These are included in the supplemental material: App. B and C
(c) Did you report error bars (c.g., with respect to the random seed after running exper-
iments multiple times)? [Yes] These reported in the supplemental material (C)
where not shown in the main body.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We were not actively tracking amount of compute throughout the process; we used an internal cluster and discuss some this information in the supplemental material (App. C
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] COCO and COCO- Stuff, ResNets
(b) Did you mention the license the assets? [Yes] This is included in the supplementary
material, App. B C
(c) Did you include any new assets either the supplemental material or as URL.? (Yes) There will be code and data splits included in the supplemental material or URL.
(d) Did you discuss whether and how consent using/curating? [Yes] The data is already publicly available for research use.
obtained from people whose data you'ne rr
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] The data is already publicly available for research use.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "daBgMenZNG0", "reviewer": "Reviewer_h5rh", "summary": "This work presents two novel approaches automatically identify out-of-context instances for image classification tasks, based either on co-occurences with other object classes or the overall gist of a scene. The authors show that these two criteria are useful at automatically detecting challenging OOC instances and evaluate the generalization ability of a number of robust learning methods on \"hard\" and \"easy\" splits of the dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Evaluating and improving model performance on out-of-context settings is an important task machine learning research, and this work shows a feasible way to automatically detect naturally occurring OOC examples in the MS-COCO dataset based on information available from MS-COCO, COCO-Stuff, and a (pre-trained) SBERT model. The originality of this work lies in the manner the sources of external information are leverage to automatically define a standard context for and object class. Based on this established standard context hard-negatives and hard-positives could be clearly defined based on their relationships with the standard context. \n\nThe central components to the proposed approaches are presented clearly in this work, however the takeaways from this research are less clear. The results indicate the researchers have successfully identified harder and easier examples from the datasets of interests, but do not contrast these results with other methods of determining hard-negative/hard-positives. The proposed approaches also seem highly specialized to the COCO dataset and it's unclear how these approaches could be effectively applied to other vision datasets, this also impacts the significance of the findings.  There are several tests of how various robust learning approaches fare on the proposed benchmarks, which has interesting results for calibration and generalization performance, however a comparison of the observed benchmark with other OOC benchmark test would further demonstrate the significance of the proposed automatic detection of OOC instances.\n\n--- clarity in presentation of the graphs could be improved by ensuring that paired graphs share the same y-axis. ---\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have effective communicated the limitations of their work, however they have not discussed potential societal impacts.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not believe there are ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ctKOHiW5DDF", "reviewer": "Reviewer_wXHX", "summary": "This paper presents a method to find hard, out-of-context examples within a training set by using existing bounding box annotations.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles a very important question of understanding model failures, particularly on hard, out of distribution, or atypical examples. \n\nThe authors propose a novel definition for identifying out of context examples by using existing bounding box annotations for an image, and categorizing the relationships between various objects across the dataset. The method is unique, and qualitative results are interesting.\n\nThe authors also perform a thorough benchmarking analysis of the performance of ERM as well as other robust-learning alternatives, and analyze the results using their out of context prediction definition.\n\nWriting exposition is very clear.\n\nComments:\n- Can this method only be used if a dataset has complete bounding box annotations? That seems very costly for most use cases.\n- What is the intuition behind using -CE or -Gist? A greater discussion of the practical tradeoffs & settings where each is preferrable would be appreciated.\n\nThe method is very clear, and perhaps may provide inspiration for designing out-of-context definitions in other settings.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impact are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "acxAtsR0Qp_", "reviewer": "Reviewer_LYwb", "summary": "The authors propose an approach to identify \"naturally\" occurring out of context prediction problems within an existing dataset, focusing specifically on COCO. Using their found images, they report robustness results from a range of models and propose a future challenge task.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: to my knowledge, this is the first generalizable approach to automatically finding existing OOC images from datasets. Existing work seems well-cited.\n\nQuality: the submission seems technically sound, and the claims seem supported by the author's evaluations. I think the proposed approach seems interesting and insightful, and appreciate the depth of the discussion the authors provided about its nuances and limitations. I would have liked to see the authors discuss in more detail the practical matter of creating challenge benchmarks from these images that they discover. Is there a standardized way that results should be reported?\n\nClarity: this submission is well-written and organized. The motivation is clearly discussed.\n\nSignificance: This work seems of strong importance to the neurips community. That said, one potential area for improvement would be for the authors to better establish the significance of this work. Why is it so important to identify natural OOC examples versus synthetic ones? Can we better improve models by having real examples like this? If so, how?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One potential societal impact to consider: OOC images might be more likely to inherently represent minority populations or viewpoints. Are there potential consequences from proposing a method to actively surface these?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "BIhvSp0uHti", "reviewer": "Reviewer_25HS", "summary": "The paper suggests that it is possible to extract examples from existing computer vision datasets that represent “out-of-context” (OOC) examples, i.e. examples that differ in contextual information from that in most of the images (per class), either through the absence of commonly co-occuring objects (CE criterion) or unusual scene-based deviation (gist criterion). \n\nIn MS-COCO, using instance-labels and segmentation-mask-areas, images are identified that are likely to exhibit potential object co-occurence based confusion (if a pair of objects co-occur at a high rate, and the co-occuring object tends to occupy a larger spatial area). If the caption-embeddings for an image differ beyond a threshold from the average embedding for the class of interest, then the image is said to differ in a “gist shift” sense. Hard positives are such context-shifted examples, and hard negatives are examples where the common context appears but not the object of interest. 12 classes are identified that seem to suffer the most based on differences in NLL with trained binary classifiers.\n\nAn empirical comparison is performed across existing OOD generalization methods for binary classification tasks on the identified subsets, showing mixed results across all subsets and evaluation metrics.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Since the paper’s primary contribution is to introduce a new OOD generalization dataset in the presence of many existing ones (some of which are identified by the paper), it is necessary to provide a compelling motivation. The paper states that the proposed sets/tasks are “a closer analogue to the OOC problems that arise in the standard model development cycle” and therefore “might more scalably provide insights into the challenges faced by current approaches”. \n\n - This motivation needs clearer unpacking, because it is not clear to me that a set of binary classification tasks extracted from a much larger multiway classification dataset with heuristics such as using “an object’s size within an image as a proxy for extractibility“ is any more a closer analogue to problems that “naturally arise” than alternatives such as splitting full-scale datasets by environments, as done for WILDS [1], or curating a collection of examples that trained classifiers fail to perform well on, as done for Imagenet in Imagenet-A [2].\n\n - The method for creating stress-tests presented in the paper would not be applicable generally for any computer vision dataset in its “model development cycle”, since one would require costly per-object annotations (segmentation masks) and image captions, which don’t exist for most computer vision datasets currently.\n\n - The binary classification splits might be a way to whittle down to the 12 chosen classes, but typically people perform multi-way classification with MS-COCO. Perhaps keeping the challenge-task and evaluations multi-way might be more aligned with the paper’s stated goal of developing a “naturally-arising” OOC problem.\n\nMinor:\n\nL201: “compliment” —> “complement”\n\nL278” “Fig. 11” —> “Fig. 8”\n\nL327: “what we” —> “that we”\n\nOn the whole, I think the paper could be improved with stronger motivations for the proposed test sets and tasks, and demonstrations of wider applicability of the method used to develop such OOC stress-tests when it is not always within budget to acquire annotations as detailed as in MS-COCO.\n\n[1] WILDS: A Benchmark of in-the-Wild Distribution Shifts, 2020\n\n[2] Natural adversarial examples, 2019\n\n--------------\nI have read the rebuttal, and as of now stand by my rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact have been adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-review: In the context of image classification, the authors propose a method for constructing challenge sets of natural out-of-context examples from bounding-box annotations. They apply the method to COCO to construct a suite of challenge test sets, and evaluate their new benchmark task against other algorithms. All reviewers agreed on technical soundness. Most reviewers (h5rh, LYwb, wXHX) agreed that a benchmark of natural out-of-context examples is a valuable contribution. Reviewer 25HS questions the motivation given that other natural challenge sets exist, but in their rebuttal the authors differentiate by focusing on context shifts specifically. Reviewers 25HS and h5rh argue that the method is too specific to COCO (e.g. due to reliance on bounding boxes), but after rebuttal, h5rh doesn't think this detracts significantly and I tend to agree: the methodology seems secondary to the result, i.e. a potentially-useful new benchmark. On balance, I recommend acceptance.
Decision: Accept (Poster)