Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an interesting approach that expands the recent success of self-supervised pre-training with *instance discrimination* to supervised pre-training. The key insight is to have each class being represented not just by a single weight vector, but in a non-parametric fashion via KNN lookup from a MoCo memory bank. The paper avoids to compare the results for direct, supervised learning, but rather focuses on transfer learning to downstream tasks -- to me this has some empirical novelty. The transfer learning experiments are done extensively, including many settings for quantitative comparisons and qualitative visualizations. 
main_review: + As hinted in the summary, the biggest realization in the paper is to focus on such methods for *pre-training*, and not just for training the current task. So it has some empirical novelty.
+ The paper did a good job in empirically studying the transferred representations. E.g., it covers linear/full fine-tuning cases, different percentage of labels for some datasets, ablation studies, hyper-parameter searches, generalization to more backbones, parametric vs. non-parametric classification, visualizations, quantitative measures to justify the claims made in the introduction about intra- vs. inter- class distances. A lot of such studies make the paper rich and strong in terms of experimentation.
+ Besides some minor typos (please get it proof-read by others), the paper is well written and well organized. The illustrations, visualizations, and stories are also very clear.

- Idea-wise, I do not think the approach of using non-parametric classification for modern visual tasks is novel. E.g.:
Wu, Zhirong, Alexei A. Efros, and Stella X. Yu. "Improving generalization via scalable neighborhood component analysis." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
This is a follow-up work of memory bank paper that applies it to supervised classification (and discovery of sub-classes) which this paper is not citing.
- Related: I feel comparing against CE or SupCon is a bit weak. I am sure in the literature there are already quite a few works (like the one above) that explore non-parametric classification. Their transfer learning ability may also be very good.
- I am not sure the paper's approach is really "leave-one-out". It is using a memory bank to store the examples, so there is no explicit operation to leave the current example of interest out. So to me the title is not so appropriate.
- Regarding gradient explosion: I am not sure I understand it, especially why having an extreme-value filtering strategy can be effective here.
- I understand the focus is on transfer learning, but at least it is still good to present the supervised learning results on ImageNet here. 

* How does the method perform without l2 normalization? I assume compared to normal cross entropy learning, one could still do multi-positive cross-entropy by having multiple prototypes per-class. An important difference here is it follows SSL methods and used InfoNCE loss with l2 normalization. So I am wondering how important it is in the pipeline? Is it fine to remove it?
* A related question is for studying the temperature in the InfoNCE loss, how important is this?
summary_of_the_review: Overall I am on the acceptance side of the paper. While non-parametric classification has ben studied in the literature, the paper presents an interesting angle of checking its effectiveness for down-stream transfer. The proposed approach is not only quite effective according to the paper's experiments, but also giving some insight about why the popular SSL methods based on instance discrimination has good transfer abilities. Some revision of the paper is surely required (on top of proof reading), but to me it is above the bar of acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a leave-one-out kNN supervised method for pre-training on large-scale datasets.  The paper claims such a method will benefit downstream tasks by NOT enforcing each class, whether visually similar or not, to cluster together. Instead, each sample just needs to be close to the nearest K neighbors.  In this way, the model will not be overfitted to pre-train datasets. To implement this method, the paper uses a loss of weighted soft kNN loss with cosine distance and softmax normalization. To solve the large search space and feature update problem of kNN, the paper uses ideas from MoCo, utilizing a memory queue as a feature bank to store features of previous N samples and First-in-first-out to update the memory queue. To solve the convergence issue, the paper uses ideas from SimCLR and utilizes MLP instead of the linear layer to project features into a more compact space. 

The proposed is tested on a standard fine-tuning setting, using ImageNet as the up-stream pre-training dataset and multiple fine-grained datasets as down-stream fine-tuning datasets. The proposed method is compared with both the supervised and self-supervised methods,
 
main_review: Overall, I do not think this paper convinces me the proposed method can actually improve the performance of downstream tasks. Intuitively, I think the motivation of the paper makes sense, we might want to relax the class definition and allow multiple clusters inside each class based on visual similarity, however, I do not think the proposed method achieves that:

1. My first concern is about experimental results. Looking at Table 2, all the numbers from baseline are significantly lower than what's in the original paper. For example, in Table 3 of the BYOL paper (Grill et al.), BYOL reaches 88.1 on Aircraft, Standard CE reaches 86.0, SimCLR reaches 87.6/88.1, but in Table 2 of this paper, BYOL is only 65.56, CE is 69.73 and SimCLR is 67.27. I found all numbers of baseline are lower than what's claimed in the original paper. Similarly, the baseline numbers of SupCon are also lower here than in the paper (Table 4 in Khosla et al.). Note that all baseline numbers are obtained on ResNet-50.  My experience in fine-tuning these datasets with standard CE is more aligned with what is in the BYOL and SupCon paper. I suspect the authors might not use the optimal hyperparameters. In my opinion, without reasonable baseline results, it is hard to conclude that one method is better than others.

2. My second concern is about the explanation. The intuition of the paper is that we need to relax the cluster of each class. In Table 6, the proposed method actually has higher intra-class similarity. In my understanding, this means a tighter cluster of each class, which contradicts what the paper claims. It also has high inter-class similarity. The paper claims clearer boundaries. I think higher inter-class similarity usually means fuzzier boundaries, maybe the authors can explain further on this claim. This might be due to the confusion of using 'distance' in Table 6's title and 'similarity' in the explanation. Also, I think to support the claim, intra-class variance might be a better measurement

3. Last but not least, the paper uses a lot of ideas from recent self-supervised papers, such as memory queue from MoCo and MLP from SimCLR, and adds kNN loss on top of that. I am fine with utilizing the ideas that work, as long as the original idea in this paper actually shows performance improvements. However, based on 1), I cannot say the proposed loss is actually better.
summary_of_the_review: See in the Main Review

--- Post-rebuttal
Overall, I am still on the fence about the paper. However I am not against accepting it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes LOOK, a new supervised pre-training method which can maintain intra-class semantic differences for better transferability to downstream tasks. Specifically, LOOK proposes to use a KNN classifier instead of the simple linear classifier for the pre-training task. Experiments show that LOOK outperforms previous supervised and unsupervised pre-training methods.
main_review: Strengths:
1. This paper is well organized and easy to understand.
2. The idea of using a stronger classifier (eg. KNN classifier) for preserving intra-class semantic differences is reasonable and shows impressive performance.
3. The experiments and analysis are sufficient.

Weaknesses:
1. Current experiments only present results on downstream classification tasks. It’s not clear how the proposed method will perform on other tasks like detection or segmentation. 
2. It shows in Table 6 that compared to previous works, higher intra-class distance and lower inter-class distance will bring better transferability. However, it’s not clear how actually intra-class and inter-class variance influence transferability. It would be better to have more analysis on this issue so that the advantages of the proposed method can be better illustrated. 

summary_of_the_review: This paper is generally well motivated and well written. The experiments have sufficiently validated the effectiveness of the proposed method on downstream classification tasks. My minor concern is the performance on other downstream tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new supervised method for visual pretraining by leveraging human labels. To avoid the intra-class instances to collapse into a single vector, the method implicitly encourages sub-clusters to occur within a category by a leave-one-out k-nearest neighbor classification loss. Compared with contrastive learning and several supervised baselines, the approach shows improved performance on several downstream tasks on car/flower/pets classifications, etc.
main_review: The paper starts with a good observation that ImageNet categories often exhibit even finer categorical structures, such a static helmet in display and a helmet in action. These labels may be beneficial to the upstream pretraining task, however, it is potentially harmful for the downstream applications. Inspired by the observation, the paper seeks to find sub-clusters within a close neighborhood of each training sample. In this way, not all samples belong to the same category would collapse to a single point, thus allowing for sub-clusters.

While being pleased with the motivations, I have several concerns regarding to the details for the proposed approach. 

1) I assume the data is randomly shuffled for each epoch. The paper uses a memory queue of 65536 samples in 1000 classes. Since ImageNet classes are well balanced, this means each class contains 65.5 samples in the queue on average. The paper uses a value of K=100 for k nearest neighbors, I wondering how many of the 65.5 samples fall in and out of the k=100 nearest neighbors. If all positives fall in the k nearest neighbors, there will not be sub-clusters to be emerged. Moreover, in what cases should a negative sample fall closer to the query sample, than some intra-class positives?

2) Following the previous comment, the ImageNet dataset contains 1300 image instances per category, which is a lot larger than 65.5 examples in the queue. I am wondering to what extent could 65.5/1300 (approximately 5%) amount of data represent a meaningful class. This is also related to how many sub-clusters usually occur within a class?

3) I appreciate the downstream evaluations for a set of classification tasks. It would be great to include the base performance on ImageNet with the proposed LOOK learning methods. Also, It would be also valuable to include the results on detection/segmentation transfer on VOC and MSCOCO. How is the performance when compared with contrastive or supervised baselines?

4) The leave-one-out nearest neighbor classification has been studied long ago, "Neighbourhood Components Analysis https://www.cs.toronto.edu/~hinton/absps/nca.pdf". The paper should be cited and discussed thoroughly. 
summary_of_the_review: I am concerned about what actually sub-clusters are emerged from the LOOK model. Based on the current algorithm description, I believe that any positive examples even if they do not belong to the current sub-cluster would be sampled and drawn closer to the query sample, and hence no sub-cluster could be found. I guess lots of nearest neighbor visualizations would help answer this question.

---- Post rebuttal ----
I am satisfied by the additional analysis provided by the authors. I raised my rating to accept this paper.
review_title: null

Meta-review: The paper introduces a simple yet effective technique for supervised pre-training based on kNN lookup from a MoCo memory queue . Initially, the reviewers raised concerns about limited novelty with respect to neighborhood component analysis, baseline results lower than the original papers, and several other questions such as how many positive samples fall in and out of kNN. The author response was strong, adequately addressing the reviewer’s comments with additional experiments and clarifications. After the discussion period, three reviewers recommended borderline acceptance. One reviewer maintained score 5, suggesting a more exhaustive search for hyper-parameters, but indicated he/she was on the fence and would be ok if the paper is accepted. The AC considers the response of the authors regarding hyper-parameter search (and the small gap from other reported results) is reasonable, and agrees with the majority that the paper passes the acceptance bar of ICLR.
Decision: Accept (Poster)