Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper delves deep into ordering and grouping methods for causal image modeling. Moving on from iGPT, which used quantized RGB tokens for its causal prediction, this paper lets the models predict the pixel values directly as MAE does. When it comes to ordering, the paper showed that random ordering could yield much better performance compared to conventional raster ordering. And for the grouping (segmentation) method, two approaches were square segments and blob segments, and both performed better than no segment baseline. Further, the paper proposed a learnable skip connection module connecting the transformer encoder and decoder layers.
strength_and_weaknesses: - (C1) There have not been many papers studying ordering or grouping methods in autoregressive methodologies, I would like to give a high score to find a way to do it well.
- (C2) I believe DINO [1] is not a contrastive learning method but instead bootstrapping or self-distillation-like method (e.g., BYOL [2]). We can find another stream of works, such as SwAV [3] and DeepCluster [4,5], which are not contrastive learning, masked image modeling, or bootstrapping (these methods are better classified as a clustering method). Thus, coarsely dividing SSL methods into contrastive and predictive categories is not appropriate.
- (C3) CIFAR10 and CIFAR100 experiments in table 7 are rather low-resolution pre-training than low-data pre-training. Although it is a worthy experiment, I think the authors can do an eye-to-eye low-data pre-training by limiting the number of examples for the ImageNet-1K dataset.
- (C4) I am confused about whether or not the learnable skip connection module is used for the experiments in section 5. Plus, is it okay for the learned matrix W to be trained without any constraints (e.g., \sum_{k=1}^{L_{enc}} W_{l, k} = 1)? If so, will the constrained version worsen the performance?
- (C5)
  - Typo: (Section 5) Suppl. Sec. D.3 -> B.1.
  - Typo: (Section 5) Supplemental Appendix B → D

[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2021.
[2] Grill, Jean-Bastien, et al. "Bootstrap your own latent-a new approach to self-supervised learning." *Advances in neural information processing systems* 33 (2020): 21271-21284.
[3] Caron, Mathilde, et al. "Unsupervised learning of visual features by contrasting cluster assignments." *Advances in Neural Information Processing Systems* 33 (2020): 9912-9924.
[4] Caron, Mathilde, et al. "Unsupervised pre-training of image features on non-curated data." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2019.
[5] Caron, Mathilde, et al. "Deep clustering for unsupervised learning of visual features." *Proceedings of the European conference on computer vision (ECCV)*. 2018.
clarity,_quality,_novelty_and_reproducibility: - (Clarity) The paper is clearly written and easy to follow.
- (Quality) The experiments were well organized and clearly addressed the research questions brought throughout the paper.
- (Novelty) I consider this model to be novel enough as it delivers new ways of grouping tokens and ordering the prediction sequence.
- (Reproducibility) Source code is not provided, but implementation details and the pseudo-code in the appendix provide reproducibility.
summary_of_the_review: It would be nice if some of my questions (see C2-C4) were answered, but I want to give this paper a high recommendation because its strengths far outweigh its weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an interesting method continuing the previous auto-regressive self-supervised learning schema. Unlike the existing iGPT which operates on pixel-leve in a fixed order, RandSAC proposes to regress the image tokens in sequential (across grouped image segments) and parallel fashions (within segment). Surprisingly, with this simple method, the exhibited empirical results are optimistic and much efficient compared with iGPT. 
strength_and_weaknesses: Strength:
1. Great paper writing with clear and easy understanding motivation.
2. Good empirical results and comprehensive ablations.


clarity,_quality,_novelty_and_reproducibility: The presented method is novel. I do not have comments regards the reproducibility as no source-codes are provided.
summary_of_the_review: The presented RandSAC builds upon the previous auto-regressive self-supervised visual representation fashion (i.e., iGPT). Compared with existing work, the proposed grouping and regressive fashion is simple but effective. The proposed hierachical grouping strategy brings obvious goodness: 1. allows various length learning on visual segments. 2. It largel improves the efficiency of pre-training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on autoregressive visual pre-training. The authors propose to spatially group tokens into a larger-resolution unit called segment, and then arrange them into a random order. After that, the authors sequentially predict the segments using transformer architecture with skip connection. The cifar dataset is used as the main benchmark to show the effectiveness of the proposed components.
strength_and_weaknesses: Issues about the autoregressive mechanism.
+ In NLP, the autoregressive pre-training paradigm might have more benefit for generative downstream tasks. The reviewer suggests the authors to elaborate more on this point and compare with the autoencoding counterpart. The reviewer believes that such discussion will make the paper standing out.
+ Currently, the authors simply demonstrate the effectiveness of this mechanism by showing the performance on downstream classification. The reviewer want to have more comprehensive impressions of the pretraining-finetuning gap of this mechanism. Could the authors conduct more downstream tasks such as detection and segmentation?
+ Since this paper focuses on the pre-training, the reviewer believes that using cifar dataset as the main example through the paper is somehow not convincing. As it is widely recognized that the scaling effect is very common in terms of the pre-training data, parameters, etc. As shown in the paper, the benefit of the proposed components become unsignificant when the data scale is increasing. To this end, the reviewer encourages the authors to use ImageNet as the main example and try to verify on models with larger capacities, such as the standard variant of MAE (i.e., ViT-Large).
***
Questions about the framework.
+ The reviewer finds that the decoder is heavier than that of MAE, since the dimensions of its hidden embeddings are enlarged to match with the encoder. Could the authors clarify how much the extra overhead this framework brings when compared with MAE?
+ Could the authors explain why the proposed methods are strongly aligned with the autoregressive modeling? The reviewer finds that the some of them, such as grouping the tokens as a segment or the skip connection in the transformer, can be applied to autoencoding modeling as well. The reviewer suggests the authors to show their strong connections with the autoregressive mechanism.
***
Other issues needed to be addressed.
+ In section 3, why the authors divide the tokenization strategy of BEiT and MAE into two categories. In fact, they share the same implementation to patchify images.
+ For table 7, there are also many contrastive learning based methods used on cifar. The authors can include them for comparison.
+ Current MAE can be regarded as predicting all the masked tokens sequentially at only one time. One the other hand, MAE can also be adjusted that reconstructing a partial ratio of masked tokens at every time sequentially. From the finetuning performance, improvement over MAE cannot be seen. Could the authors show the superiority of the proposed method against MAE?
clarity,_quality,_novelty_and_reproducibility: + Although the authors show their diverse vocabulary, however, this paper is not very easy to follow for the reviewer.
+ The content is not well organized. The reviewer suggests the authors to deliver the information in a way that how the proposed techniques are critical for the autoregressive mechanism. Currently, the proposed methods are not strongly aligned with autoregressive manner. Some of them can be also applied with autoencoding. Overall, the messages in this paper are somehow messy.
+ Since there are many details of the proposed framework, the reviewer is not sure whether it can be fully reproduced.

summary_of_the_review: + Currently, the experimental results do not strongly prove the potential of the proposed framework in visual pre-training. The reviewer encourages the authors to conduct experiments with ImageNet and larger model since the scale effect is important for pre-training.
+ The authors should try to finetune their model on other downstream tasks beyond the same-dataset classification, in order to verify the gap between the proposed autoregressive pre-training and downstream transfer.
+ The authors should improve the writing and show how the proposed methods are strongly aligned with the autoregressive mechanism.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a generative method for self-supervised learning on images which divides images into tokens, then models tokens autoregressively. All reviewers appreciated the novelty of the approach, and were generally satisfied with the empirical results. Reviewer pLid raised many relevant issues in their initial review, including more discussion, better comparison with MAE, and more downstream tasks. The authors submitted an extensive response including new experimental results which were successful in mostly addressing these concerns. In the end all reviewers recommended accepting the paper. The authors are encouraged to take the suggestions of Review pLid into account when preparing the camera-ready version of the paper.
justification_for_why_not_higher_score: The results are interesting, but not a dramatic improvement on prior work.
justification_for_why_not_lower_score: All reviewers recommended accepting the paper.
Decision: Accept: poster