Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a method to find hard, out-of-context examples within a training set by using existing bounding box annotations.
main_review: This paper tackles a very important question of understanding model failures, particularly on hard, out of distribution, or atypical examples. 

The authors propose a novel definition for identifying out of context examples by using existing bounding box annotations for an image, and categorizing the relationships between various objects across the dataset. The method is unique, and qualitative results are interesting.

The authors also perform a thorough benchmarking analysis of the performance of ERM as well as other robust-learning alternatives, and analyze the results using their out of context prediction definition.

Writing exposition is very clear.

Comments:
- Can this method only be used if a dataset has complete bounding box annotations? That seems very costly for most use cases.
- What is the intuition behind using -CE or -Gist? A greater discussion of the practical tradeoffs & settings where each is preferrable would be appreciated.

The method is very clear, and perhaps may provide inspiration for designing out-of-context definitions in other settings.
limitations_and_societal_impact: Limitations and social impact are adequately discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose an approach to identify "naturally" occurring out of context prediction problems within an existing dataset, focusing specifically on COCO. Using their found images, they report robustness results from a range of models and propose a future challenge task.
main_review: Originality: to my knowledge, this is the first generalizable approach to automatically finding existing OOC images from datasets. Existing work seems well-cited.

Quality: the submission seems technically sound, and the claims seem supported by the author's evaluations. I think the proposed approach seems interesting and insightful, and appreciate the depth of the discussion the authors provided about its nuances and limitations. I would have liked to see the authors discuss in more detail the practical matter of creating challenge benchmarks from these images that they discover. Is there a standardized way that results should be reported?

Clarity: this submission is well-written and organized. The motivation is clearly discussed.

Significance: This work seems of strong importance to the neurips community. That said, one potential area for improvement would be for the authors to better establish the significance of this work. Why is it so important to identify natural OOC examples versus synthetic ones? Can we better improve models by having real examples like this? If so, how?
limitations_and_societal_impact: One potential societal impact to consider: OOC images might be more likely to inherently represent minority populations or viewpoints. Are there potential consequences from proposing a method to actively surface these?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper suggests that it is possible to extract examples from existing computer vision datasets that represent “out-of-context” (OOC) examples, i.e. examples that differ in contextual information from that in most of the images (per class), either through the absence of commonly co-occuring objects (CE criterion) or unusual scene-based deviation (gist criterion). 

In MS-COCO, using instance-labels and segmentation-mask-areas, images are identified that are likely to exhibit potential object co-occurence based confusion (if a pair of objects co-occur at a high rate, and the co-occuring object tends to occupy a larger spatial area). If the caption-embeddings for an image differ beyond a threshold from the average embedding for the class of interest, then the image is said to differ in a “gist shift” sense. Hard positives are such context-shifted examples, and hard negatives are examples where the common context appears but not the object of interest. 12 classes are identified that seem to suffer the most based on differences in NLL with trained binary classifiers.

An empirical comparison is performed across existing OOD generalization methods for binary classification tasks on the identified subsets, showing mixed results across all subsets and evaluation metrics.
main_review: Since the paper’s primary contribution is to introduce a new OOD generalization dataset in the presence of many existing ones (some of which are identified by the paper), it is necessary to provide a compelling motivation. The paper states that the proposed sets/tasks are “a closer analogue to the OOC problems that arise in the standard model development cycle” and therefore “might more scalably provide insights into the challenges faced by current approaches”. 

 - This motivation needs clearer unpacking, because it is not clear to me that a set of binary classification tasks extracted from a much larger multiway classification dataset with heuristics such as using “an object’s size within an image as a proxy for extractibility“ is any more a closer analogue to problems that “naturally arise” than alternatives such as splitting full-scale datasets by environments, as done for WILDS [1], or curating a collection of examples that trained classifiers fail to perform well on, as done for Imagenet in Imagenet-A [2].

 - The method for creating stress-tests presented in the paper would not be applicable generally for any computer vision dataset in its “model development cycle”, since one would require costly per-object annotations (segmentation masks) and image captions, which don’t exist for most computer vision datasets currently.

 - The binary classification splits might be a way to whittle down to the 12 chosen classes, but typically people perform multi-way classification with MS-COCO. Perhaps keeping the challenge-task and evaluations multi-way might be more aligned with the paper’s stated goal of developing a “naturally-arising” OOC problem.

Minor:

L201: “compliment” —> “complement”

L278” “Fig. 11” —> “Fig. 8”

L327: “what we” —> “that we”

On the whole, I think the paper could be improved with stronger motivations for the proposed test sets and tasks, and demonstrations of wider applicability of the method used to develop such OOC stress-tests when it is not always within budget to acquire annotations as detailed as in MS-COCO.

[1] WILDS: A Benchmark of in-the-Wild Distribution Shifts, 2020

[2] Natural adversarial examples, 2019

--------------
I have read the rebuttal, and as of now stand by my rating.

limitations_and_societal_impact: Limitations and societal impact have been adequately discussed.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work presents two novel approaches automatically identify out-of-context instances for image classification tasks, based either on co-occurences with other object classes or the overall gist of a scene. The authors show that these two criteria are useful at automatically detecting challenging OOC instances and evaluate the generalization ability of a number of robust learning methods on "hard" and "easy" splits of the dataset.
main_review: Evaluating and improving model performance on out-of-context settings is an important task machine learning research, and this work shows a feasible way to automatically detect naturally occurring OOC examples in the MS-COCO dataset based on information available from MS-COCO, COCO-Stuff, and a (pre-trained) SBERT model. The originality of this work lies in the manner the sources of external information are leverage to automatically define a standard context for and object class. Based on this established standard context hard-negatives and hard-positives could be clearly defined based on their relationships with the standard context. 

The central components to the proposed approaches are presented clearly in this work, however the takeaways from this research are less clear. The results indicate the researchers have successfully identified harder and easier examples from the datasets of interests, but do not contrast these results with other methods of determining hard-negative/hard-positives. The proposed approaches also seem highly specialized to the COCO dataset and it's unclear how these approaches could be effectively applied to other vision datasets, this also impacts the significance of the findings.  There are several tests of how various robust learning approaches fare on the proposed benchmarks, which has interesting results for calibration and generalization performance, however a comparison of the observed benchmark with other OOC benchmark test would further demonstrate the significance of the proposed automatic detection of OOC instances.

--- clarity in presentation of the graphs could be improved by ensuring that paired graphs share the same y-axis. ---

limitations_and_societal_impact: The authors have effective communicated the limitations of their work, however they have not discussed potential societal impacts.
review_title: null

Meta-review: Meta-review: In the context of image classification, the authors propose a method for constructing challenge sets of natural out-of-context examples from bounding-box annotations. They apply the method to COCO to construct a suite of challenge test sets, and evaluate their new benchmark task against other algorithms. All reviewers agreed on technical soundness. Most reviewers (h5rh, LYwb, wXHX) agreed that a benchmark of natural out-of-context examples is a valuable contribution. Reviewer 25HS questions the motivation given that other natural challenge sets exist, but in their rebuttal the authors differentiate by focusing on context shifts specifically. Reviewers 25HS and h5rh argue that the method is too specific to COCO (e.g. due to reliance on bounding boxes), but after rebuttal, h5rh doesn't think this detracts significantly and I tend to agree: the methodology seems secondary to the result, i.e. a potentially-useful new benchmark. On balance, I recommend acceptance.
Decision: Accept (Poster)