Identifying and Benchmarking Natural Out-of-Context Prediction Problems

1 Introduction . People often find context useful for prediction , both for improving accuracy and processing efficiency [ 10 ] . However , deep learning systems frequently over-rely on context cues [ 18 , 19 , 38 ] , which can lead to poor performance on out-of-context ( OOC ) examples , when contextual information is misleading . By OOC examples , we mean inputs which are uncommon or unusual with respect to the training distribution ; these can be thought of as sampled from under-represented subgroups , or low ( non-zero ) density regions , of the training distribution . In safety-critical situations , this can be problematic ; as such , it is important to have reliable methods for measuring how well a model can perform OOC . Furthermore , given the rise of larger models and datasets [ 31 ] , there is a need for scalable approaches to OOC evaluation — even if manual evaluation of “ corner cases ” by domain experts may ( always ) be the gold standard . A key pre-requisite task to evaluating OOC performance is identifying which examples should be considered OOC . This identification task is a challenging one in and of itself : in a natural image , “ context ” can be varied , complex and high-dimensional [ 6 , 47 , 59 , 63 ] . Therefore , any evaluation method intending to measure a model ’ s OOC performance must ( implicitly ) select a specific notion of “ OOC performance ” . Indeed , since deep learning yields underspecified models [ 14 ] , it is plausible that different choices may yield different measurements . Common approaches include generating semi-synthetic data to simulate the effect of a shift in some salient feature [ 32 , 54 , 63 ] , or using some auxiliary information to guide choices about what a reasonable OOC set should be [ 28 , 33 ] . In this work , we develop a conceptual framework for identifying sets of OOC examples in existing datasets . We show how our framework unifies and generalizes prior literature on OOC performance measurement , and allows us to utilize more complex , structured annotated data to define various notions of “ OOC performance ” . We demonstrate this framework ’ s effectiveness through uncovering two OOC “ challenge sets ” [ 30 ] within an existing benchmark , each corresponding to differing notions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of context . We show how our framework enables scalable and targeted measurement of models ’ OOC performance through clarifying the relationship between the concept of “ OOC performance ” and its implementation , allowing for clearer insight on current approaches as well as opportunities for improvement . Our contributions are as follows : • We present NOOCH ( Naturally-Occurring Out-of-context Challenge sets ) , a suite of “ challenge sets ” for evaluating performance on naturally-arising OOC problems , available at https : //github.com/dmadras/nooch ; • We develop a conceptual framework for automatically identifying OOC challenge sets from existing data by leveraging known underlying structure ; • We contrast two instantiations of this framework using two notions of “ context ” , defining concepts of “ hard positives ” and “ hard negatives ” in the OOC setting ; and • We quantitatively analyze the performance of several methods from the robust learning literature on these challenge sets , exploring the tradeoffs inherent in different approaches to OOC performance measurement ; and qualitatively demonstrate how rich notions of context can yield rich investigation of OOC errors . 2 Measuring “ Out-of-Context Performance ” . Intuitively , a model which has good “ OOC performance ” should be able to maintain good performance under unusual or perturbed contextual conditions . We distinguish this from the out-of-distribution ( OOD ) problem [ 27 ] , which is usually concerned with inputs from a different domain than the training set . Rather , the OOC prediction problem is more similar to subgroup robustness or distributional shift , where a model must perform well on uncommon input regions at training time . However , even after drawing this distinction , the notion is ill-defined : “ context ” may refer to concepts as varied as object relationships [ 59 ] , image backgrounds [ 63 ] , experimental settings [ 47 ] , or world models [ 6 ] . Furthermore , even fixing a notion of context , the criterion for what should make something “ out-of-context ” ( OOC ) is still unclear . For instance , Peters et al . [ 47 ] focus on previously unobserved contexts ( i.e . environments ) , whereas Xiao et al . [ 63 ] are concerned with unusual contexts given the class of interest ( i.e . perturbations to image background ) . Clearly , defining a benchmark to measure a method ’ s OOC performance requires a number of design choices , which has enabled a recent proliferation of OOC benchmarks . We note in particular , one of the key choices is around the usage of auxiliary information . Across the literature on OOC performance measurement , there are a plethora of approaches to defining OOC criteria using some type of auxiliary information C. For the purposes of algorithm designers , C may be assumed to be available at training , validation , and/or test time , or not at all — however , at the the time of benchmark design , it is available on a sufficiently large portion of the collected dataset to guide the designers ’ choices about what a suitable OOC criterion should be . Examining the current literature on measuring OOC performance , we identify the following as a unifying framework : 1 . Identify some existing auxiliary information C , a variable which takes some value on many ( or all ) examples and specifies some underlying structure in the data . 2 . Select a notion of “ OOC ” ( e.g . “ images with misleading backgrounds are OOC ” , “ examples from unfamiliar time periods are OOC ” ) and define an “ OOC criterion ” by choosing a binary function φ of C. 3 . Restrict the test set to those examples where φ = 1 . Optionally , also restrict the training set to those examples where φ = 0 . We show in Table 1 how a range of prior literature leverages auxiliary information to define OOC criteria . We can think of C as providing benchmark designers with some type of “ inductive bias ” around what should be considered OOC for a given benchmark . The above framework implies that there is a diversity of OOC criteria which can be defined over any dataset , and this class is as broad as the class of functions φ which can be defined over the available auxiliary information . In the rest of the paper , we take advantage of the flexibility of this framework to give two examples of such an approach . We show that by leveraging more complex annotated structure , we can create multiple OOC benchmarks from an existing dataset using multiple criteria for what should be considered “ OOC ” . We trace out how the choices made in designing these criteria correspond to different notions of context , and demonstrate experimentally that these yield varying measurements of OOC performance . 3 Finding Naturally-Occurring OOC Problems . We now demonstrate concretely how rich auxiliary information can be used to study the way that context shifts arise naturally within an existing computer vision benchmark , and provide two criteria for OOC performance that can be computed from these annotations . Throughout , we consider the binary prediction task of determining object presence , a problem where relationships between various objects naturally provide helpful context — given an image X , is an object of class Y present or not ? Background : COCO and COCO-Stuff . The Microsoft Common Objects in COntext dataset ( COCO ) [ 36 ] is a computer vision dataset consisting of images of natural scenes . Each image is annotated with instance labels and segmentations for every “ thing ” in the image , as well as several captions describing the content of the scene . Images usually contain multiple items and as such usually have multiple labels . However , for the purposes of investigating OOC prediction , many relevant objects are not labelled in COCO ; for instance , background objects such as “ sky ” or “ grass ” are not COCO classes . Fortunately , the COCO-Stuff dataset [ 7 ] provides labels and segmentations for all of the “ stuff ” in the images from COCO ; a thing is an object with a specified size and shape , whereas stuff has no “ defined spatial extent ” [ 16 ] . Having both thing and stuff labels is essential for understanding model behaviour on OOC examples , since it is exactly these “ stuff ” classes which often ( but not always ) provide important context cues . Taken together , the thing and stuff annotations yield a rich sandbox for queries about the role of context in prediction . For our purposes , COCO-Stuff contains 171 binary tasks for determining object presence ( 81 thing classes and 90 stuff classes ) . 3.1 Automatically Identifying OOC Examples : Hard Positives and Negatives . We develop two contrasting notions of “ OOC ” : 1. the presence/absence of frequently co-occurring , easily extractible objects ; and 2. an unusual “ gist ” of a scene . We define these notions below , presenting two criteria for identifying naturally-occurring OOC prediction problems within the existing COCO ( -Stuff ) dataset , and discussing how we can use annotations as proxies to define an OOC indicator φ . We identify two types of OOC examples : hard positives , where the class is present despite an unusual context , and hard negatives , where the class is not present , despite a usual context . 3.1.1 Defining Context Using Co-Occurrences and Extractibility . For an object class Y , context cues often come in the form of another object class C that has two properties ( cf . [ 38 ] ) . First , C and Y co-occur frequently [ 5 ] . Second , C is more extractible than Y — easier to detect . If C were less extractible than Y it would not be a useful cue for detecting Y , as a model could detect Y directly . We can utilize these properties to create candidate context cues C for a class of interest Y . Given segmentations , we can use an object ’ s size within an image as a proxy for extractibility ( larger objects tend to be more extractible ) . Let the Area operator take the sum of the areas of all segmentations of instances of that object , or return 0 if the object is not present . Then , to estimate from the training set how important of a context variable C is , we can compute A ( C , Y ) = E [ Area ( C ) −Area ( Y ) |Y = 1 ] . When A ( C , Y ) is larger , this means that when Y is present , C is usually also present , and on average , takes up more of the image than Y does . When A ( C , Y ) > α , we say that C is an α-strong context cue for Y ( or just α-context for brevity ) . We find that many of the contexts identified using this method for large enough α are intuitive . Some examples of ( label , 0.05-context ) pairs are : ( car , road ) , ( bowl , dining_table ) , ( cow , grass ) . Using this notion of context , we can then define hard positive and hard negative examples . We make the simplifying noisy-or assumption : that each context cue provides evidence for Y , so that the presence of any cue supports Y being present , while the absence of all provides evidence against Y ’ s presence . Given some image , we can define an ( α , β ) -hard positive or negative . If Y = 1 , and for all α-context cues C , we have Area ( C ) < β in this image ( and there is at least one α-context cue ) , then the example is an ( α , β ) -hard positive . Alternatively , if Y = 0 , and there exists some α-context variable C such that Area ( C ) > β , then the example is an ( α , β ) -hard negative . We will call this method the co-occurrence/extractibility ( CE ) criterion ( see Fig 1 for examples ) . Throughout , we use α = 0.05 , β = 1 unless otherwise noted ; these parameters were chosen since they approximately equalize P [ ( X , Y ) is ( α , β ) -hard |Y = y ] across y = 0 , 1 . See Appendix B for more details .

Summary: The paper suggests that it is possible to extract examples from existing computer vision datasets that represent “out-of-context” (OOC) examples, i.e. examples that differ in contextual information from that in most of the images (per class), either through the absence of commonly co-occuring objects (CE criterion) or unusual scene-based deviation (gist criterion).   In MS-COCO, using instance-labels and segmentation-mask-areas, images are identified that are likely to exhibit potential object co-occurence based confusion (if a pair of objects co-occur at a high rate, and the co-occuring object tends to occupy a larger spatial area). If the caption-embeddings for an image differ beyond a threshold from the average embedding for the class of interest, then the image is said to differ in a “gist shift” sense. Hard positives are such context-shifted examples, and hard negatives are examples where the common context appears but not the object of interest. 12 classes are identified that seem to suffer the most based on differences in NLL with trained binary classifiers.  An empirical comparison is performed across existing OOD generalization methods for binary classification tasks on the identified subsets, showing mixed results across all subsets and evaluation metrics.