Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a theoretical analysis to understand why mask reconstruction pre-training (MRP) works well in downstream tasks. It's found MRP can capture all discriminative semantics of each class in the pre-training dataset and thus utilize them to help downstream tasks. It provides solid proof, straightforward visualization and experiment result.
strength_and_weaknesses: Strengths:
1. The problem of why MRP works so well and is much better than supervised learning is quite interesting and important for the community. This paper is a pioneer in studying how semantic features are learned under the above problem.
2. The theoretical analysis covers both multi-view data and single-view data. It also covers both MAE-like methods, Teacher-Student MRP methods, and supervised learning methods. It also discusses different downstream tasks. 

Weaknesses:
1. In this paper, it assumes the encoder in the student network is just a two-layer convolution + Smoothed ReLU network. I understand that this setting is easier to analyze. But in real-world applications, people usually use deeper models. And we know that some conclusions found in a simple model don't necessarily generalize to deeper models. So I wonder if it's possible to generalize the analysis to deeper models?
2. I am a bit confused about the definition of whether one semantic is captured by the kernel. In 4.1, If I understand correctly, when the correlation score is increased to `1/polylog(k)`, it means matched. Why would the threshold be set as that?
3. The experiments mainly cover visualization and accuracy. But those two proxies cannot indicate whether all discriminative semantics are captured in MAE/MRP. Is there any way we can quantitatively measure how many semantics in each class are captured by kernels?
4. As you mentioned in section 4.2, Transformer is not discussed because of its correlated manipulations and highly nonlinear attention. Is that possible that we remove the nonlinear part and make it analyzable?
5. Some typos such as `The pretraining dataset Z has` in Assumption1. 
clarity,_quality,_novelty_and_reproducibility: The paper writing is clear. The theoretical proof is novel and inspirational to the community.
summary_of_the_review: Overall, this paper targets an intriguing problem and provides solid proof. I'd lean to accept it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the mask-reconstruction pretraining (MRP) approach and its ability to generalize well for downstream tasks. They show that under certain dataset and task assumptions, MRP can perform well for downstream classification tasks whereas supervised learning cannot. 
strength_and_weaknesses: Strengths
- The problem is clearly an important one to study, and hasn't been adequately addressed by the existing literature. 
- The use of the multi-view data assumption from  (Allen-Zhu & Li, 2020) is clever. 
- The empirical results match the theoretical findings, from multi-view data assumption to test performance on downstream tasks. 

Weaknesses
- The assumption that pre-training and downstream datasets follow the same distribution is in some ways limited. While the authors discuss how this framework is useful, it would be nice to see more analysis of "transfer learning", since this is much of the appeal of pre-training approaches. 
- Dataset assumptions are fairly strict, though the authors reason about how these assumptions are justified. 
clarity,_quality,_novelty_and_reproducibility: Clarity

There are parts where the work is not clear and has typos. I've listed two glaring examples here, but I'm sure there are more throughout the text that should be re-checked. 

Top of page 8: "in the pretraiing"
The use of "semantic" as a singular noun is somewhat strange. I haven't seen this before and the authors should consider a different word choice. 

Quality and Novelty

While some data assumptions come from a previous work, the problem setting is unique and the analysis is valuable.  
summary_of_the_review: The theory is both correct, novel, and matches the empirical results of MRP, which is an important problem to study. While there are important results, the dataset assumptions cast some doubt on how generally applicable the results are.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a theoretical analysis that explains the benefit of MRP in SSL. Utilizing the multi-view data distribution and certain assumptions about the model architecture, the paper shows that the encoder trained with MRP can capture *all* the discriminative semantics of each semantic class in the pretraining dataset. This gives an advantage to the SSL+SL compared the SL only model since in SL training the model only captures *some* semantics due to lottery ticket hypothesis.
strength_and_weaknesses: Strength:

1) The paper provides a strong theoretical analysis, supporting the benefit of MRP for SSL.

2) It is really hard to pick a minimal set of assumptions about the data and model that are just sufficient to prove the advantage of MRP in SSL. I like how the paper mathematically defines multi/sing-view distributions and picks a simple network architecture and manages to prove the benefit of MRP over SL.


Weaknesses:

1) There are several limiting assumptions in the multi-view data distribution and network architectures that do not apply in practice. However, I understand that these restrictions are required for a sound theoretical analysis. 

2) The theoretical analysis in the paper does not result in a new practical approach, instead it supports the success of the existing methods in a limited setting.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the definitions, assumptions, and theorems are cleanly stated and make sense to the reader. Note that I did not read the proofs and can not speak of their correctness.
summary_of_the_review: Recent works have shown practically that supervised fine-tuning of the encoder learned via MRP remarkably surpasses the SL training from scratch but I am not aware of any theoretical work that supports the benefit of MRP. The paper provides new theoretical justification for the success of MRP in practice. Overall, the theoretical contributions of the paper are significant and novel.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is overall a quite interesting paper. Based on an abstract teacher-student framework with simple network architecture, the authors explore how a masked reconstruction pre-training architecture performs feature learning and becomes beneficial for downstream tasks by modeling "semantics" as attribute vectors associated with different annotation labels. With the above analysis, the authors further conduct a MRP pre-training with ResNet and demonstrate more superior performance compared with supervised pre-training. 


strength_and_weaknesses: - Strength:
  - The authors provide a very detailed theoretical analysis based on the proposed teacher-student MRP framework.
  - The idea of modeling "semantics" as several attribute vectors associated with classes is interesting.
  - After theoretical analysis, the authors further conduct implementations to support their analysis.

- Weakness:
  - In the MRP architecture proposed in Sec. 3.2, are there any requirements of the teacher model parameters, like they should be the EMA of the parameters of the student model?
  - The analysis heavily bases on the assumption that the "semantics" of the downstream dataset should be a subset of the pre-training dataset. Does it suggest that the downstream dataset should have similar distribution with the pre-training one, which is actually a very strong and unrealistic assumption? If not, what is the difference?
  - The authors claim that the MRP pre-trained models would contain all the semantics of the downstream tasks, each at least corresponding to one kernel. Does it suggest to do transfer learning, if I can find the specific wining ticket of a given downstream task, pruning can actually replace fully fine-tuning?
  - I wonder whether the MRP pre-trained models would be worse when the pre-training dataset does not contain all the semantics in the downstream datasets compared with supervised pre-training, since the latter replies on partial semantics matching while MRP might response to more semantics according to Fig. 3.
- Could you further explain what the role of masking \epsilon is, since it seems not to appear in your analysis in Sec. 4?


clarity,_quality,_novelty_and_reproducibility: Check the weakness part for more details.
summary_of_the_review: This is overall an interesting paper solid theoretical analysis, which will provide good insights for future researches and algorithm development.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper provides a theoretical prespective for understanding mask-reconstruction pretraining techniques that are popular for self-supervised computer vision pre-training. I am an empiricist and I won't pretend to fully understand the theoretical details of the paper, but the reviewers universally agreed that this paper was technically sound, novel, and an advance in our understanding of self-supervised learning, and I defer to their judgment.
justification_for_why_not_higher_score: This paper was quite dense and difficult for me (a proxy for a general audience) to understand. I do not think it will have sufficiently broad accessibility or appeal to the rest of the community to deserve a spotlight or oral.
justification_for_why_not_lower_score: The reviewers universally agreed that this paper met the bar for ICLR and was technically sound.
Decision: Accept: poster