Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a dense contrastive representation learning framework for visual feature learning in a self-supervised manner. The proposed framework considers a local patch query strategy for local contrasting together with a global contrasting. The method achieves superior performances on multiple challenging benchmarks for dense predictions. 


strength_and_weaknesses: Pros:
+ the idea of combing local and global contrastive learning is interesting, and also important for dense prediction tasks, as local and global contexts provide different semantic information of the image. 
+ the results are strong. 

Cons:
- As shown in Table 7, the local patch query introduces a lot of extra memory. It seems that the method sacrifices the memory overhead to gain performance. How about a larger-batch size? Would the method still be able to train with a reasonable memory consumption?
- The performance on ImageNet-1k seems to be minor compared to DINO and iBOT (only 0.3%). But from the comparison with state-of-the-arts on the dense prediction tasks, the gain is more clear. Could the authors explain the performance differences?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the novelty is moderate. 
summary_of_the_review: The idea of performing local and global contrasting seems to be interesting, but on the other hand, this would also bring extra computational overhead, especially when the number of image crops is high. The authors need to further clarify their advantage on this point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents ADCLR, a method for contrastive and self supervised representation learning. The main target task is dense prediction i.e. semantic segmentation and object detection, however the framework fits the classification task as well.

Within the general framework of DINO Caron et al. (2021)  authors state their contributions to be (i) cross-views are used for contrasting learning, (ii) the unidirectional cross attention module, and (iii) achieving SOTA.

strength_and_weaknesses: The first natural application is image classification, so ADCLR is benchmarked for that task. Common benchmarks in the field include (i) k-NN and (ii) top-K linear accuracy on imagenet and (iii) comparison against a strong supervised classifier on smaller datasets. The selection of methods (CNN and ViT) and datasets is comprehensive and shows  SOTA improvement.

However, the central application is dense prediction i.e. semantic segmentation and object detection, so ADCLR is also benchmarked for those tasks.

clarity,_quality,_novelty_and_reproducibility: There are two typos, on section 3.2 (i) the the same objects and (ii) tiken -> token. Otherwise the paper is very well written. 
summary_of_the_review: My recommendation is to accept this submission, the paper is sound and the results do support the claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning dense contrastive representation based on vision transformer backbone. To achieve this goal, the authors propose a clean, simple and effective framework. In detail, they perform the cross-attention between query patches and raw patches. Then the updated query patch features are positive pairs. Despite the local contrastive loss, they also add global contrastive loss. The proposed method outperforms the previous methods.
strength_and_weaknesses: Strengh
1. The framework is clean, simple, and effective.
2. This proposed method does not introduce too many extra parameters.
3. The method outperforms the previous methods.

Weakness
1. This paper claims that [CLS] token may extract mismatched information. However, the query patch token may also has this problem, if the patch does not exist in one augmented view. 
2. How to select the query patches? Previous works like SoCo use selective search to select good patches. Selective search may also be used in this paper.

Other questions
1. In Table 7, are the local views used as raw patches or query patches?

clarity,_quality,_novelty_and_reproducibility: In general the quality of this work is good. The idea is clean. The experiments are solid. 
The proposed method has some novelty in the transformer based patch-level contrastive learning.
summary_of_the_review: I like the idea that perform attention between query patches and raw patches. Then contrast the query patch features.
I also like the analyses related to the collapse.
There are some details are not doing well, such as selecting the query patches.
In summary, I prefer to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is about unsupervised learning of image classification. It is focused on patch-level contrastive by an attention mechanism. Experiments on ImageNet-1k, MSCOCO, ADE-20k verify the effectiveness. 
strength_and_weaknesses: --Strength

The paper introduces ADCLR, a novel patch-level contrastive learning method, improves the quality of dense representation of unsupervised learning. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction, and opens a new direction for future research.
The improvement is marginal comparing to iBOT on ImageNet. 

--Weakness

Figure 1 does not describe the method well, e.g., what do we get after the augmentation image goes through Transformer Encoder? Reconstructed image or deep features?

In page 4, "...By feeding the whole sequence of two views xA and xB to..."  I cannot figure out how to get xA and XB.  It seems z is input, and goes through of attention layer, but where does x comes from? what is two views?
clarity,_quality,_novelty_and_reproducibility: Some presentation is not clear.
summary_of_the_review: The idea and experiments seem good, but the description of the method is not self-contain and clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a dense contrastive representation learning framework for visual feature learning in a self-supervised manner. The proposed method combines local and global contrasting and achieves strong performance on multiple challenging benchmarks for dense predictions. The main strengths of the method are its clean, simple and effective framework and its ability to outperform previous methods. One weakness is that the local patch query strategy used in the method may introduce some extra memory and computation. The performance on ImageNet-1k is relatively minor compared to other benchmarks, such as DINO and iBOT. The authors could provide more explanation for this difference in performance. Overall, the paper is well-written and the results support the claims made by the authors.

justification_for_why_not_higher_score: There are some concerns expressed by the review, and the idea behind the work is not groundbreaking so it may not be up to the oral or spotlight level.
justification_for_why_not_lower_score: All reviewers support for acceptance so it should not be rejected.
Decision: Accept: poster