Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a novel contextual autoencoder framework for SSL. The proposed CAE approach introduced a cross-attention module to learn the latent space knowledge between the visible and masked patches. The paper had a comprehensive analysis between the proposed approach and the existing SSL approaches which gave some insight to the reader to understand the core contributions.  

strength_and_weaknesses: Strength
1. The paper is well written and easy to follow. The comprehensive analysis between the proposed approach and the existing SSL works add a valuable insight into its contributions.
2. The evaluation confirms the proposed approach can outperform the SOTA SSL approach on various backbone and downstream tasks.

Weakness
1. The novelty of this paper is limited. The proposed CAE is an improvement work over MAE and BEIT by further exploiting the knowledge between the masked and visible patches in the latent space. There is no sufficient evidence that the proposed cross-attention module can further improve representation performance learned by CAE.
2. The proposed CAE approach used a random block-wise masking approach. It is necessary to add an ablation study about the performance between various masking approaches, e.g. random patch masking. Moreover, the masking ratio is set to 0.5 by default. What would the performance be for other masking ratios? An ablation study would help clarify that.
3. Some technical details are missed in the paper. E.g. how to decode \bar{Y}_m from \bar{Z}_m in figure 7a is not clear. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written with most of the details clearly presented.

The paper’s quality is good in general but missed some technical details.

The paper is an improvement of BEIT and MAE. The novelty is limited as the comments above.

summary_of_the_review: The paper is an improvement work over BEIT and MAE with some minor novelties. The proposed CAE algorithm achieved the SOTA performance on various backbone and downstream tasks. The paper can be improved by clarifying the technical details and adding ablation studies.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents context autoencoder (CAE), which follows BEiT and uses an additional latent contextual regressor to make predictions for the masked patches with cross attention over the visible patches. By doing that we can encourage the masked patches' representation predicted from visible patches are aligned with the one from the encoder and then benefits the learning and downstream transferring. Experiments on downstream tasks also show that it can achieve better performance compared with the baseline BEiT and MAE.
strength_and_weaknesses: The paper is well-written. The motivation of this paper is clear and easy to follow.

The proposed attentive probing is interesting and well-motivated, however, it seems that even with this metric the CAE trained with 800 epochs using ViT-B cannot surpass the contrastive learning-based model MoCo-v3 according to Table 1.

Performance wise, the downstream task results on COCO look superior, especially when using ViT-B, but when moving to ViT-L the gain seems diminished (from ~2 points to ~0.5 points compared with MAE on 1600ep).

The technical part seems to be somewhat incremental, how important/critical the alignment is? Seems like this issue only happens when making predictions in the representation space. For example, MAE without the alignment loss will not show meaningless output like Figure 3. The role of the cross-attention based latent contextual regressor is also not clear, one could also use the output from the MAE decoder's first layer to compute the alignment loss.
clarity,_quality,_novelty_and_reproducibility: In the paragraph of `Relation to BEiT and MAE.`, the authors mentioned that `In MAE (He et al., 2022), the so-called
decoder may play a partial role for representation learning as the representations of the visible
patches are also updated in the MAE decoder`, which is somewhat inaccurate and makes me confused as the "updated representation" for visible patches is neither supervised with the reconstruction loss nor used for downstream tasks. Whether using extra layers to process representations for visible patches should not be a key difference and the cross attention used in this paper also has MLP for the key/value (which are the visible patches feature).

Figure 3 is somewhat hard to follow without referring to the caption, might be better to add some illustration about how the output is generated.

Misc: I believe the citation of LARS is incorrect.
summary_of_the_review: I think this paper shows some interesting findings and good performance, although the technical contribution is somewhat limited from my perspective. Thus my rating is 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a method for self-supervised representation learning based on masked image modeling. Unlike previous approaches, the proposed method decouple representation learning part and pretext task completion part where the learning signal comes from the reconstruction in the encoded representation space rather than in image space. Alignment constraint is introduced encouraging the predicted representations to be lied in the encoded one. The experiments are conducted on three different downstream tasks; semantic segmentation, object detection, and instance segmentation, surpassing the previous approaches.
strength_and_weaknesses: ** Strengths
- Comparisons to the concurrent methods with analysis and discussion
- Extensive experimental results achieving state-of-the-art performances on various downstream tasks
- Clear implementation details
- Using pretrained tokenizer for masking strategy is interesting

** Weaknesses
- The idea of reconstructing just the features of the masked patches is also used in iBOT
- The reliance on a pre-trained tokenizer could be cumbersome. While being with 250M images as extra data, there are not many insights and improvements compared to previous works, e.g. competitive performances compared to iBOT and MAE even they don't require using such pre-trained tokenizers
- 
clarity,_quality,_novelty_and_reproducibility: Please see Strength And Weaknesses section above.
summary_of_the_review: As the weaknesses outweigh the strengths, I lean towards 5: marginally below the acceptance threshold at this point.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to decouple the encoder and decoder space of MAE, thus improving the representation quality. Similar to contrastive learning, a Latent contextual regressor is further used to align the predicted masked patch with the encoded masked patch. Experiments on downstream object detection and semantic segmentation show the effectiveness of the proposed method.
strength_and_weaknesses: Strength:  
Good results on downstream transferring learning, especially for semantic segmentation.  
Clear idea, decoupling the space of encoder and decoder explicitly may benefit the learning of representation.  

Weaknesses:  
(1) In fact, the “Context” in the Title does not correspond well with the interpretation in the main manuscript. MAE, BEiT and other Masked Image Modeling approaches can also be understood as including context information. I am not sure what specific meaning the context in the paper refers to.  
(2) Several details and designs are not clear:  
-- Why is the masking rate set to 50%? (Section 3.2) Why not 75% or other values?  
-- How to balance the two loss functions? (Eq (1))  
-- How to choose the number of layers in the Latent Contextual Regressor and Decoder? The number of layers may be related to the performance.    

(3) Questions about the experiments:  
-- In fact, the finetuning results in Table 1 are only slightly higher than MAE (0.3%).  
-- How about removing the Decoder? That’s only containing the latent contextual regressor. In addition, in Table 2, why not use the finetuning or linear probe metrics?  
-- In Table 3, according to the MAE paper, the detection performance of MAE with ViT-B is 50.3%, not 48.4%. In addition, MAE+ViTDet [1] has already achieved higher results (ViT-B: 51.6% and ViT-L: 56.7%).  
[1] Yanghao Li, et al. Exploring Plain Vision Transformer Backbones for Object Detection. ECCV2022.  

clarity,_quality,_novelty_and_reproducibility: Overall, the proposed method may be of some inspiration to the community. However, some details and experiments need to be further investigated.  
Lack of limitations (e.g. training costs analysis).  

summary_of_the_review: I am inclined to accept the paper if the author can dispel my concerns well.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the Contextual Autoencoder (CAE) for self-supervised learning. It builds on MAE and BEiT to perform masked image modeling with Vision Transformers (ViTs). The primary technical novelty compared to these prior works is that CAE attempts to predict the latent representations of the masked patches using a latent content regressor module in addition to predicting the full image representation from the decoder; this essentially amounts to training MAE with dVAE targets, and an additional alignment loss on latent representations in the decoder.

Initial reviews for this paper were borderline. Reviewers found the paper well-written and easy to follow, understood the key difference between CAE and prior work, and agreed that CAE can outperform prior SSL methods in some contexts. Reviewer sentiment was slightly positive overall, but no reviewers argued strongly in favor of accepting the paper. Reviewers asked for many clarifications and extra experiments which the authors supplied in their responses, including results with different mask ratios (KNoM and cbX1), clarifications on Figure 7a (KNoM), results on using the MAE decoder’s first layer representations to compute the alignment loss (cbX1), experiments removing the decoder (cbX1), linear and finetuning results for the ablation study (cbX1), and a discussion of training costs (cbX1). While the reviewers (aside from cbX1) did not meaningfully engage with these extensive author responses, in the AC’s view the additional results and explanations supplied by the authors were successful in addressing the many of the specific issues raised by reviewers.

However, in the ACs view there is one key issue that was not fully addressed by the author's responses: novelty and significance with respect to MAE and BEiT. A common theme among the reviewers is that the proposed method is a small variation on MAE without significant technical novelty (KNoM: “The novelty of this paper is limited.”; TXRx: “the technical contribution is somewhat limited”; b4Df: “there are not many insights and improvements compared to previous works”). The proposed method is not exactly the same as any presented in prior work, but in light of MAE, BEiT, and iBOT there are no “big ideas” or “big insights” that are significantly different from the body of prior work in this area.

In the ACs view, minor technical novelty is not necessarily a reason to reject the paper; simple changes to existing methods can be of benefit to the community if those changes yield significant experimental results. While the authors have performed an extensive and admirable set of ablations and experiments, taken as a whole the experiments do not demonstrate that CAE has a definitive and consistent advantage over prior methods. On ImageNet (Table 1), CAE vs MAE gives just +0.3 for finetuning on both ViT-B and ViT-L; while CAE gives significant gains over MAE for linear and attention probing, it is outperformed by MoCo-v3 and iBOT under those evaluation protocols. For semantic segmentation (Table 3), CAE gives +0.1 vs iBOT for ViT-B, and +1.0 vs MAE for ViT-L. For object detection, CAE vs MAE gives larger improvements on both detection (+1.8 AP) and instance segmentation (+1.6 AP) on ViT-B, but these advantages shrink on ViT-L (+0.6 and +0.5). As a whole, these results show CAE as having some advantages over prior methods, but other than COCO detection with ViT-B and ADE20K segmentation with ViT-L, CAE does not offer large performance improvements over prior methods; it also uses slightly more resources (time and memory) vs MAE for training.

The AC met with Reviewer KNoM to discuss this paper, who largely agreed with the points above. Overall, the experimental results are not significant enough to offset the relatively minor technical novelty presented by the paper and so the paper is not suitable for publication at this time.
justification_for_why_not_higher_score: The paper offers limited technical novelty over prior methods (MAE, BEiT, iBOT) and also gives relatively minor performance gains over these methods.
Decision: Reject