Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a new few-shot learning framework COSOC that can focus more on foreground objects at both pre-training and evaluation stage. COSOC is a two-stage algorithm. At the pretraining stage, a clustering-based object seeker (COS) module is used to force the pretraining model to focus on found foreground objects by a fusion sampling strategy. At the evaluation stage, a shared object concentrator (SOC) module is used to seek for shared contents and filter out background, and then the foreground of testing images is matched with the recognized foreground objects of each class. Experiments on two benchmarks are conducted to verify the effectiveness of the presented method.
main_review: The strengths of this work are as follows.

+This paper considers that background information is easy to be used as a shortcut to distinguish images and constructs a small dataset D_new with proper experiments to analyze the adverse effect of background information in few-shot learning.

+This paper presents a new few-shot learning framework COSOC.

+The results seem good.

However, several major concerns should be addressed.

-More details should be provided for the training of contrastive learning models (Exampler used in this paper, a modified version of MoCo). It raises a major concern on the baseline. How the model is initialized? How many training epochs are used? The training is commonly expensive, and how much time is used? Does the training follow the original paper?

-As shown by the results, the presented method achieves more performance gain in the 1-shot setting. But it is really in doubt that the presented method can locate the foreground by using V randomly crops (no clustering is used in 1-shot setting, as stated in the paper). This is a major problem. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors conducted an emperical analysis and revealed that image background serves as a source of shortcut knowledge which is harmful for few-shot learning task. To alleviate this, they proposed a two-stage method, COSOC, that employs contrastive learning to draw the model's attention to image foreground during both pre-training and evaluation stage. Given the assumption that foreground objects from different images of the same class share more similar patterns as background, in the pre-training stage, a clustering is performed on randomly cropped patches to identify crops that best represent the foreground objects in the image. Then, fusion sampling is adopted, i.e. these crops are used to replace original image with a certain probability based on its foreground score during pre-training. In the evaluation stage, mean features maximally representing the shared contents among training images from the same class is obtained. Given this, the foregrounds in testing images can be located via matching.
main_review: Strengths
- The overall writing is clear and easy to follow. 
- Although the idea of removing background to avoid learning shortcut knowledge is intuitive, its realization is however non-trivial. The authors provide a feasible solution to this, where its effectiveness is well proven by extensive experiments.

Questions
- The fusion strategy selects either the original image or randomly cropped patch with a certain probability. Why not using the top K crops discovered in step 4 in L202 to replace the randomly cropped patch? This also better simulates the training setting in Sec 3.1.
- In the evaluation stage, instead of computing features $\omega$ that minimizes the sum of pair-wise distances between crops from different images within the same class, why not performing clustering on the training images to extract reliable crops representing the foregrounds similar to what's being done in the pre-training stage for matching?
- A simple yet possibly strong baseline could be running unsupervised saliency detection to extract the most salient region in an image, followed by cropping to obtain patches without background. This also eliminates the need of having large number of crops to avoid missing the foreground. The authors could consider comparing with this.
limitations_and_societal_impact: Please refer to "Main Review" section above
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper dives into the importance of learned “shortcuts” based on background similarly across the training samples in few-shot learning scenarios, at the detriment of generalization to images with a “category gap” - a background that does not match what was seen in training. They explore the effect of background on recognition and propose two simple algorithms based on contrastively-learned features (one method at train time, one at test) to find foreground vs background and learn more generalizable representations with little data. At training time they use clustering over contrastive-pretrained features extracted from a set of crops from images across the training data for each class to find crops that are highly likely to contain foreground and use those crops instead of the image (with some probability, based on the “foreground score” of the crop). At test time a representative feature per-class is generated using the training data, and simple feature matching is used to determine the class.
main_review: This paper was an enjoyable read - It was well laid out and clearly written. The problem formulation was justified with the analysis in section 3.1, the methods were simple, intuitive, and explained clearly and the results outperformed the methods they compared against. I like that the methods were so simple (feature matching!) and relied heavily on available and powerful contrastively-learned feature representations. There is possibly further room to improve by extending these simple approaches in the future.

Here’s an earlier paper that seems quite relevant, it points out performance drops when moving to novel image backgrounds: https://arxiv.org/abs/1807.04975 

This method has a strong inter-class visual similarity assumption, are there categories which might break this? Did you see any consistent failure modes of the method? Possibly worth mentioning in discussion.

In line 109: “crop each image manually” - how exactly? Do you take the largest bounding box inside the object or make a per-pixel segmentation mask?

limitations_and_societal_impact: Beyond the known and problematic biases in the underlying datasets they use, I do not see potential for negative societal impact from this work
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposed a few-short learning (FSL) method by excluding the negative impact of background noise common in the existing FSL framework. A few separate stages are employed here to achieve the goal: 1) Clustering-based Object Seeker: a pretraining stage to pretrain the network and hence the feature space of the foreground objects; 2) Shared Object Concentrator (SOC): get the representation of the foreground of each class (which is then used in test time to determine the foreground of a test image). The method showed good empirical results on selected datasets. 
main_review: The originality of the work is moderate - Specific steps of training and extracting foreground features are integrated into the existing FSL framework. The paper is well-written in general except for a few issues mentioned below. The significance of the work though is work debating and a few major questions are listed here. 

The key issue of the work resides in the test time of the method, i.e. using multiple crops to aggregate the scores of each class. I am sure this step can improve the final performance, however it is dangerously close to the common trick of test time augmentation to boost the model performance in practice. Simply put, for other few-shot methods, if we simply do multiple crops and averaging the scores as the final output, it is very likely that most of them, if not all, would see some performance improvement. If we further do a clever weighted sum, e.g. assign a weight based on the similarity of crops to the images of a given class in the training data, the performance may get further improved. In other words, the setup here is not persuasive enough to me that the improvement is mainly caused by rectifying the short learning of background - which is the main conclusion of the paper, although I’d be very much inclined to believe so in theory. 

Secondly, the reason why the miniImageNet is not directly used in the comparison is worth a detailed explanation. Most of the SOTA methods report performance on miniImageNet directly, yet here a subset of miniImageNet is deliberately constructed for comparison. Why add this additional step into the experiment preparation? 

Additionally, a large portion of the work tried to construct an effective foreground detector and tried to persuade the audience that the proposed method: random cropping + contrastive learning + clustering solved the problem well. It could easily trigger the curiosity of an audience that why not try some existing saliency detector and combine it with the few-shot framework? Does the proposed foreground detector work better than the existing saliency detectors? If so, is it because of better foreground detection results or because of the inherent pretraining step? A further discussion would be helpful for the audience to understand better the value of this work. 


A couple of relatively minor issues that worth consideration: 

1. In the Step 2 of 4.2, the meaning of \alpha^{j-1} and \beta^{n-1} is unclear. Specifically, the meaning of the superscript here could be further clarified especially if alpha and beta are all fixed in the paper? 


2. Considering the numerous separate steps in the entire workflow, it would be helpful to show some failure cases and try to attribute the cause to the failure of individual steps. 

limitations_and_societal_impact: Limitations have been discussed in the work. No negative societal impact as far I can think of.
review_title: null

Meta-review: The authors provide a simple but effective technique for reducing bias towards background pixels, in a few-shot setting. They provided extensive data and explanations in their rebuttal. Though two of the reviewers remained unconvinced, the other two reviewers were very supportive. However, I believe that the simplicity of the method and it's clear effectiveness, along with the extensive empirical studies, makes it a good candidate to be accepted to Neurips 2021. The authors should (as they have themselves promised) revise the paper to incorporate the good comments from the reviewers.
Decision: Accept (Poster)