Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This is a pioneering work considering SuperPixel-like idea and point cloud modeling for 2D general visual backbone.
The main contributions are:
- Proposing a model called CoC to explore the possibility of point-cloud-clustering modeling for 2D image processing, and initially verifiing its effectiveness.
- Providing fresh insights into the design of general vision model, which are not only well suited to cope with dense prediction tasks (detection and segmentation), but also facilitate the combination with 3D point cloud data in order to promise a very general cross-modal vision model.
strength_and_weaknesses: **[strength]**

- This is a very insightful work towards general visual backbone design. The fusion of SuperPixel idea as well as PointNet structure is natural and elegant. I believe this work reveals the potential of point cloud modeling for 2D image processing, which is certainly a great contribution to the community and is worth exploring.
- I would say that CoC is modeled in a way that fits well with the nature of visual signal processing, as it has the inductive biases that are as useful as those of convolutional networks (such as locality and multi-scale).
- The interpretability is also as good as convolution (CoC embodies spatial selectivity, while convolution exemplifies the idea of pattern matching). Besides, CoC acts like the deformable convolution and is clearly more flexible and general than ordinary convolution that operates on regular grids.


**[weakness]**
- **Potential representation weakness**. As an architecture similar to ConvNet and PointNet, CoC could be significantly weaker than Transformers in terms of global modeling capabilities (e.g., interacting two spatially distant objects). What are the authors' insights on this problem?
- **Related work**. I find the first paragraph in Related Work is way more relevant to CoC than the latter two. It would be much appreciated if the authors could devote more space to the first paragraph, e.g., introduce [R1,R2,R3,R4] in more detail and discuss them with this paper in terms of method philosophy.
- **Figure clarity**. I recommend using more contrasting colors when visualizing. Currently these two figures (Fig. 4 and 6) do not illustrate the clustering effect of CoC models very well.
- **Font formats**. The spacing between characters in the headings is too small, making them look cramped. Also, there seems to be too much use of bolding or italics in the body text. I would suggest using a more concise format.

------------

**[open questions]**

I have more questions as follows and would like to discuss them further with the authors:

- From Fig. 5(b), the CoC block has similarities with DAT [R5]. Can the authors further analyze or compare them?
- I don't think it's necessarily a disadvantage that the methods [R3, R4] can only be used for specific tasks like segmentation; in other words, a point-based (or SuperPixel-based) modeling might be inherently more useful for dense prediction tasks (detection, segmentation, etc.) than classification. The authors mentioned that they expected to see more work that seamlessly integrates CoC and DETR in the future. This is reasonable, but I still expect the authors to come up with some prototypes of CoC for dense prediction :).

----------
[R1] Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In ECCV, 2018.

[R2] Zixuan Huang and Yin Li. Interpretable and accurate fine-grained recognition via region grouping. In CVPR, 2020.

[R3] Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. Superpixel segmentation with fully convolutional networks. In CVPR, 2020.

[R4] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022a.

[R5] Xia, Zhuofan, et al. Vision transformer with deformable attention. In CVPR, 2022.

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written and of high quality. The authors are very thorough in providing reproduction details (source code, model checkpoints, etc.).
I think the paper is highly innovative. Its idea of point cloud modeling on 2D images to instantiate generic vision models is groundbreaking. It has a high future value for our field moreover. See "contribution" and "strength" parts above.
summary_of_the_review: This work is innovative, insightful, written with ample detail, and highly reproducible.
I believe it deserves to be published and discussed in ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new view of images that considers each image as a set of points (the pixels) and uses a clustering algorithm to extract the features from it. The goal is to investigate the way to utilize this new form of visual representations and evaluation the performance that could be achieved. To this end, the paper introduces a novel backbone network that includes the proposed Context Clusters and evaluation this model on several vision tasks as well as a point cloud data application.
strength_and_weaknesses: Strengths:

- To the best of the reviewer’s knowledge, the topic of considering an image as a set of points and extracting features from it for vision tasks is original and very interesting.
- The proposed method that uses the clustering algorithm as the basic build block is novel and of significance to the community.
- The evaluation plan of the paper is comprehensive. It provides experiments on standard vision tasks like image classification and object detection/segmentation and applications for point cloud inputs like object classification.
- The evaluation results show that the method provides improvements on various tasks over the CNN and ViT baselines  (though not outperforming the state-of-the-art approach).

Weaknesses:

- By using the region partition mechanism, the set of points is no longer unorganized but becomes structured based on their locality. Additional experiments are required to clarify the role of the region partition.
    - Before applying the context clusters operation, the region partition operation, which is similar to the shifting windows in Swin [1], is introduced to reduce the computational cost. The authors seem to imply that the region partition trades off performance for speed. However, the locality introduced by the region partition could also bring useful inductive bias for the encoder. Therefore, additional experiments are required to answer the following questions:
        1. If the region partition operation is removed in the clustering process, could the model achieve similar or better performance? What would the clustering map be like in this case?
        2. It would be nice to introduce Swin as one baseline to investigate this problem.

[1] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. *arXiv*. https://doi.org/10.48550/arXiv.2103.14030
clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and easy to follow. The authors also provide additional explanations of some model designs in the appendix which are much appreciated.
- Both the topic and the proposed method are both original.
- The general architecture is reproducible based on the model description, additional hyper-parameters are required to reproduce the experimental results.
summary_of_the_review: This paper introduces a new form of image representation that considers each image as a set of points and proposes a clustering-based architecture for feature extraction. Both the idea of “image as set of points” and the proposed architecture are interesting and novel. The experiment result also shows that the method achieves comparable performance to ConvNets and ViTs. A small concern is that the role of the region partition mechanism is unclear since good performance could actually be attributed to this design.

---

## Post-rebuttal updates:
I appreciate the detailed feedback and the extra experiments. It is very interesting to see that: 

1. Removing the partition operation actually improves the performance. This demonstrates the efficiency of the original design.
2. The clustering behavior reveals superpixel-based grouping. This is anticipated, but demonstrating that it is actually working in this way in the paper is much appreciated.
3. Slightly worse than Swin in performance. The answers to my questions and the performance difference are also acceptable for a novel model design.

Since most of my concerns have been addressed, I decided to raise the score to 8. This is a good paper and worth acceptance
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present an alternative approach to image content representation called context clusters. These are compared as alternatives to convolutional networks and vision transformer approaches. 

The work is well presented and well written. They argue for their approach in a scientific manner and set it in place for future use in other vision techniques and applications. 
strength_and_weaknesses: The paper is well written (although requires an editorial read)  and the experiments provided are detailed. 

The methodology could be more clearly presented though. It should be immediately understandable but the reader has to re-read to follow properly. As a case, I would like to be explicitly told how the context clusters are believed to capture the content better. 

The citations are incorrectly used throughout. 
The use of italics and bold in paragraphs seems unprofessional and I would prefer it not used. 

Why are some figures bold in Table 1? It is not clear what this means.

The authors state "See appendix for more experiments." It would read well if a short explanation of what is in the appendix is provided here.  
clarity,_quality,_novelty_and_reproducibility: The data and code are cited - so the work is technically reproducible but would not be immediately straightforward I guess? 

The clarity of the methodology could be improved for ease of following. 
summary_of_the_review: I enjoyed this paper. The authors write with enthusiasm and back up their claim that context clusters are worth further investigation through scientific means.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper proposes a different formulation for vision tasks, i.e. considering image as a set of unorganized points instead of organized rectangular shape or patches sequence. Each point has raw feature and positional information and they are grouped with clustering algorithm. It shows comparable or sometimes better result as CNN or transformer. The code is also released. 
strength_and_weaknesses: Strength:
+ The idea is very interesting and this new architecture could be an alternative architecture for CNN, ViT. It provides a new perspective on how the information of an image is represented.
+ The idea should be of great interest for the representation learning and computer vision community.
+ The computation is a hard problem if using clustering on 224x224 image, while the authors use a smart way to address it, similar to Swin Transformer. 
+ The performance is impressive as compared with recent popular architectures.
+ The propose method also generalizes well on downstream tasks like object detection and segmentation.
+ The paper is well organized and the writing is good.

Weakness:
- The paper claims to bridge the gap between representation of image and point cloud. It seems to work well on PointNet. But it is kind of hard to understand. The authors use fixed centers (Fig.5) to save computation, how does this work if the discrete pixels are sparsely sampled from one image? It might be helpful if the authors can provide more explanation about this. It would also be interesting to see the performance of dynamic centers? For example, maybe use only 2 iteration for center updates?
- The proposed method may have good interpretation ability, but not better than CNN or ViT. The figure 4 might need more explanation since the framework is very different from other architectures. The method seems to cluster similar contexts in early stage, but it is not easy for reader to get this insight from figure 4. 
clarity,_quality,_novelty_and_reproducibility: This paper proposes a interesting idea and the architecture is novel for representation of image. The writing is good with high-quality figures and tables. The code is provided for reproducibility.
summary_of_the_review: To summary, I think this paper is very interesting and should be discussed at the main conference. It might be a good alternative architecture for vision tasks. The results are impressive and well-presented. I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper proposes a new visual representation learning pipeline called Context Clusters (CoC). The main idea is to consider the image as a set of points (similar to a pointcloud) and to use a clustering algorithm to form local context clusters instead of treating the image as a regular grid of patches. Point features in a cluster are aggregated using an adaptive average pooling and then "dispatched" back to the constituent points via a fully-connected layer. The architecture further considers multi-head dispatching like in self-attention networks. Experimental results are given on a variety of recognition tasks, including ImageNet-1K object recognition, 3D pointcloud classification on ScnObjectNN, object detection and instance segmentation on MS-COCO, and semantic segmentation on ADE20K.

# Strengths

+ **Novelty**: The proposed approach is novel, naturally interpretable, and shows great potential as a hierarchical model for image representation based on deformable convolutions instead of fixed-patch convolutions.

+ **Experiments**: The breadth and depth of the experimental evaluation provides very strong evidence of the efficacy of the Context Cluster approach. The considered CoC models outperform similarly-sized convolutional and Transformer-based architectures on multiple tasks and have comparable computational costs. The simple ablation study in Table 2 supports the claim that each contribution of the CoC architecture (positional coding, context clustering, and multi-head dispatch) all contribute to the performance.

+ **Interpretability and Reproducibility**: Context Clusters are as interpretable as convolutions, and the experimental results seem reproducible based on the technical details given in the paper (code is also provided). The supplementary appendices provide more detailed and complementary results and analyses.

+ **Clarity**: The paper is generally well-written and the main contributions are motivated from first principles throughout (see Weaknesses below for some qualifications).

# Weaknesses

+ **Clarity**: The paper needs some editorial revision for language and some elements of the technical development. Some reviewers point to difficult passages (some of these were addressed in rebuttal), and to difficulty in interpreting some figures -- especially those intended to support the interpretability of the model. Finally, reviewers single out several typographical inconsistencies that interfere with readability.

+ **Comparison with Other Architectures**: Reviewers pointed to the Swin Transformer as a self-attention model with similar motivations (the authors provided a comparison in rebuttal). A more measured comparison with models having similar goals is in order. The novelty of the proposed approach is not at all in question at this point, so a careful consideration of alternative approaches seems appropriate.


# Summary

The reviewers are overwhelmingly positive in their evaluation of the paper. Their concerns were addressed through a lively back-and-forth with the authors during the discussion phase, which has significantly clarified the novelty and effectiveness of the CoC architecture. What the authors propose is a novel approach to deep, hierarchical image representation that adds something interesting to the discussion on deep representation learning for visual recognition.

That said, there is definitely room for improvement. At many points in the paper the authors interject somewhat vacuous assertions about the novelty of the approach, for example restating many times that it is distinct from self-attention, which becomes a bit gratuitous. More effort could be spent meaningfully contrasting and comparing alternative architectures.

justification_for_why_not_lower_score: The paper proposes a very novel and interesting approach to deep, hierarchical image representation. It is not a variant of existing themes, and the manuscript is well-written with excellent motivations for its main contributions throughout. The experimental evaluation considers a broad variety of recognition problems (in images and pointclouds), providing convincing empirical evidence of the usefulness of CoC as a generically applicable backbone architecture.
Decision: Accept: notable-top-5%