Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduce a simple mechanism to allow the training of MIM model tacking the reconstruction difficulty into account. It also introduce deep supervision that can help the diversify the feature representation across different layers. Good performance are reported to justfiy its methods. 
strength_and_weaknesses: Strength:
- The intuition that reconstructing patches has different levels of difficulty makes a lot of sense. The proposed method is also simple, and the required modifications on top of the MAE framework are quite minimal. 
- The experimental results are solid. 
- The analysis of locality(eg. Fig 4) is interesting and inspiring. 

Weakness:
- The author claims that deep supervision introduces diversification and mentions the independence of beta_i. However, the Fig 5 shows that the beta_i are highly correlated and mostly the same. The difference in beta_i as shown in zoom-in figure in Fig 5 right plot seems too brittle to support the hypothesis that lower layer focus on more simple patches and deeper layers focus on more difficult patches. Also, why is the beta curve for block 12 missing from Fig 5?
- More discussion of how the initial beta is chosen is needed.  The paper initialize beta to -0.5 and the during the whole training beta varies from -0.6 to -0.425, very close to the initial value. What will happen if you initialize beta differently? 

 
Other comments

clarity,_quality,_novelty_and_reproducibility: This paper proposes a very simple modification to the MIM framework. The intuition behind the approach is clear, and the experimental results are adequate. This paper is well-written in general and it should be not difficult to reproduce the results.


summary_of_the_review: The presented approach is simple yet effective as shown in their benchmark results. The analysis is in general good. Thus I am leaning to accept. 

Post rebuttal:
I decided to down-rate the score after the reviewer discussion. Although the core idea of weighting patch reconstruction is inspiring, the improvement from dynamic loss is comparably marginal compared to the deep supervision, which makes the current paper writing less proper as the majority discussion is focused on dynamic loss. Given the current reviewers opinions, I recommend the authors to rephrase the paper writing or provide stronger results to validate dynamic loss and submit to another venue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new MIM framework that dynamically focuses on patch reconstructions based on the degree of difficulty (i.e., the nearby visible patches) during pre-training. Besides, this paper proposes to self-distill intermediate features from the momentum encoder. Experiments show that it outperforms previous self-supervised methods on ImageNet for image classification and on ADE20K for semantic segmentation.
strength_and_weaknesses: Pros:

- Interesting exploration to improving MIM by providing more local priors.
- State-of-the-art performance among tokenizer-free methods. 

Cons:

- The motivation of this paper is not convincing. The reconstruction difficulty of patches are not necessarily correlated with distance from visible patches. For example, it may depend on how much information is included in the patch. The background patches will still be easy to reconstruct following regular patterns even if it is far away from visible patches, but the foreground patches (e.g., nose of a dog) are hard to guess and require stronger semantic reasoning.

- The empirical improvement is too marginal. This is no clear advantage of the proposed method compared to SdAE(Chen et al., 2022c) according to the experiments. It has the same accuracy as SdAE for 100 epochs and only outperforms SdAE by 0.3% for 800 epochs.

- This paper is not well-written. The introduction is too redundant and messy. Figure 1 is hard to follow. The caption is not self-contained.
clarity,_quality,_novelty_and_reproducibility: As mentioned above in Cons, this paper is not well-written and hard to follow. The basic idea of providing more local priors to MIM is interesting, but the design of the method is not satisfactory.
summary_of_the_review: Overall, I think this paper is a borderline paper and slightly below the acceptance threshold. I would like the authors to provide more justifications to their motivation to further judge this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes to introduce deep supervision into masked image modelling. Technically, there are two loss functions: the first one is to reconstruct pixel densities using tokens produced by intermediate transformer encoders; the second one seems to be enforcing the transformer encoder outputs to mimick those of corresponding outputs of an EMA version of the encoder (although I am not very sure about the details due to clarity issues). The primary selling point is to use a transformed (by equation 2) distance map that weights both losses so that patches with different distances to visible ones get different weights. This technique is considered 'dynamic' by the authors, in the tile, as the concentration parameter beta is learned in a layer-wise manner. Evaluations are done with downstreaming ImageNet fine-tuning and linear classification and an ADE20k fine-tuning experiments, reporting marginally improved performance. Codes are not provided or promised.

[Post-rebuttal]
The method is conceptually novel and has some value. 
The significance is left for experts on large-scale pre-training to judge.
The major issue is unclear presentation. Although I can guess how to implement it from these vague descriptions, people who are not familiar with topics involved may have some difficulty in understanding this. This is sub-bar for an ICLR paper, in terms of clarity.
After several rounds of communication, the authors correct some factual errors (1-M v.s. M), and typos. They also tune-down some unjustified interpretations that may be misleading for the community. Remaining issues include:

(1) The behavior of the regularization term in Equation.6 lacks clarity. The authors give an intuitive example in the response, which is not included into the paper. If included, this raises the clarity of this part above the bar. But I still recommend an analysis, as mentioned before: 'The impact of this regularization needs more (theoretical and experimental) analysis' in the first round and 'How gradients propagate to beta, through h() is not clear to me and this needs an analysis' in the second round. This is not a difficult thing, which only needs some gradient analysis and I don't understand why authors insist it is not necessary.
(2) The training loop. After some clarification, I know that the implementation is aligned with my guessing. But the paper still lacks clarity. Including a training loop (like Algorithm 1 in the MoCo paper) is not difficult and I dont' understand why authors insist it is not necessary.

Although it is disappointing that the authors still ignore my sincere suggestions on improving the clarity for a larger audience that are not familiar with these techniques, I have raised the recommendation to BA.
strength_and_weaknesses: Strengths:
+ I am familiar with and like deep supervision and distance transforms. They are techniques specific to computer vision due to their links to hierarchical representations and spatial affinity. This idea (if better developed) is something that I would like to cite and follow. Introducing deep supervision into masked image modeling is novel to my knowledge. Using distance transforms to weight training is a novel add-on too and makes sense in this masked image modelling setting.
+ Although the method is unclear in several places, I can somewhat figure out how to implement it as the modules are standard ones.
+ Although lacking in significance, the experimental results are conclusively better than baseline.
+ The attempt to reveal the underlying mechanism is appreciated (fig.3 and 4), although I am not fully convinced.

Weaknesses:
- I think the major issue is clarity:
- The notations make me confused. In equation 1, we use x_{1-M} to reconstruct x_M. But in equation 2 and 3, it seems that the paradigm changes? For equation 2, I believe we should reconstruct masked-out patches x_M right? Correct me if I am wrong. For equation 3, I think it is possible to distill the features of masked-out patches or remaining patches. But since the equation does not contain decoder I guess the input should be remaining patches x_{1-M} ? Having that said, if the distillation is done with only the encoder, how is the 'distillation decoder' in Fig.1 used?
- Reading section 3.3 makes me lost. There is no formal description of the EMA procedure or the training loop. And what's worse, the figure, the loss and the notation seem to contradict each other. The sentence 'Deep dynamic supervision is shared with raw pixel regression' brings much more confusion. What is 'shared with'? According to Fig.1 I guess the deep dynamic supervision term refers to the whole training objective?
- I think the beta regularization is the key to the success. According to my own experience, training this kind of parameters is not easy. The impact of this regularization needs more (theoretical and experimental) analysis. Since beta can be either negative or positive, regularizing its L1-norm is not valid. So I can see the authors use h for regularization, but the behavior is not readily clear to me.

- Although I think the method is novel from several points of view, I have to point out several missing references. In the 'Deep Supervision' related works section, the authors claim that making deep supervision work for modern architecture is not easy. I agree and I would like to point out that papers like [A][B], which are the referred 'Contrastive Deep Supervision' paper's baselines, have studied the problem. 

[A] Deeply-supervised knowledge synergy, CVPR 2019
[B] Be your own teacher: Improve the performance of convolutional neural networks via self distillation, ICCV 2019

- My last major concern is about significance:
- Personally, I only care about whether these pre-trained models can help my group develop better down-streaming robotic scene understanding algorithms. I feel excited about CLIP, Swin or MDETR, but the performance improvement of this one seems too marginal. I don't think this is a strong enough pre-training paper for ICLR. Maybe other colleagues with hands-on expertise on large-scale pre-training can better assess the significance.

I have other comments:
- I doubt he central claim of 'locality inductive bias in deeper layers'. I am not convinced why we we should have local bias in late stages. Meanwhile, I cannot understand why the deep dynamic supervision design allows this to happen.
- 'gradually reconstructed vicinity' I am not convinced. There is no technical modules that allow gradual reconstruction.
- 'this paper shows another potential use of deep supervision to make the network more diverse' This is too vague and I am not convinced by this 'diverse' claim.
- The online calculation of distance transforms of random masks brings computation overhead, which should be analyzed.
- The better convergence claim should be supported by curves instead of two datapoints.

Several capical issues:
- The Experimental results
- Dynamic Loss
- The Reconstruction targets
clarity,_quality,_novelty_and_reproducibility: They have been discussed in the last box.
summary_of_the_review: The idea is fine, but the current version lacks in clarity. As for significance, I am not fascinated and others with hands-on experience in pre-training may have a better judgement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper designs a deep dynamic supervision mechanism that can be migrated into existing MIM methods.
It proposes to dynamically focus on patch reconstructions with different degrees of difficulty at different pretraining phases and depths of the model. 
Further experiments demonstrate the effectiveness of proposed method.
strength_and_weaknesses: Strength
+ The performance is good
+ Detailed experiments demonstrate the effectiveness of each component 

Weakness
+ The idea seems to be incremental. Add deep supervision is straight-forward and should be expected to work.
+ It lacks motivation on using both of distillation decoder and regression decoder. It seems minor improvement when combing them. What is the experiment variance? 
+ Using extra tokenizer is not necessary a bad thing. Will proposed method be further improved when use extra tokenizer?
+ What is the improvement on down-stream fine-tuning when compared to MAE?
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. The experiments are detailed and well conducted.
summary_of_the_review: Overall, this paper is valid. However, the idea seems to be incremental. This prevents me from further increasing the score. 

After reading the rebuttal, I still feel that this paper lacks enough novelty and the main components proposed yeild minor improvements.  

Hence, I downgrade my rating and would not recommend to accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two practical regularization techniques for MAE training and the related quantitative results verify their efficacy. One is to exert different weights to the reconstruction loss of different patches based on how difficult it is to reconstruct these masked ones with visible ones. Another is to employ a momentum encoder to regularize the learned intermediate features. Experiments show these two designs are effective and complementary. They can accelerate training and further improve baseline performance, evaluated on two tasks, e.g., classification and segmentation.
strength_and_weaknesses: Strengths:
1) It gives a strong motivation for the proposed two designs. Applying spatially variant reconstruction losses to different masked patches based on their distances from the existing ones is a sound technique. The momentum encoder is also a typical method to improve representation learning.
2) It conducts extensive and solid experiments. Classification and semantic segmentation are conducted along with several compared methods. Ablations are also sufficient.

Weaknesses:
1) Though dynamic loss is complementary to deep supervision (in Table 3), the sole improvement brought by dynamic loss is limited (only 0.2 compared to MAE) and the major benefits are provided by deep supervision. This experimental result weakens the key role of dynamic loss while the loss is stressed with the most pages. Undoubtedly, dynamic loss can accelerate training convergence, but it is expected to give better improvement if it is the most crucial part of this paper. Besides, such a dynamic loss design is a routine in inpainting literatures [1-4].
> [1] Pathak, Deepak, et al. Context encoders: Feature learning by inpainting. CVPR. 2016.
>
> [2] Yeh, Raymond, et al. Semantic image inpainting with perceptual and contextual losses. arXiv preprint arXiv:1607.07539 2.3. 2016.
>
> [3] Yu, Jiahui, et al. Generative image inpainting with contextual attention. CVPR. 2018.
>
> [4] Wang, Yi, et al. Image inpainting via generative multi-column convolutional neural networks. NeurIPS. 2018.

2) The analysis of the used momentum encoder is unsatisfying from my perspective. If we treat the masked input as a view of the input, then MAE can be considered as maximizing the similarity between the positive pair in contrastive learning. To this end, the usage of momentum encoder can be discussed in depth.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and easy to follow. I suppose the analysis of deep supervision could be improved for a better understanding about the role of the momentum encoder.

About the novelty, as described in Strength And Weaknesses, the dynamic loss and momentum encoder are intensively studied in the inpainting and contrastive learning areas respectively. Their introduction and combination to MAE lead to interesting performance improvement with simple designs, which is inspiring.

The method description is clear and concise, and the related techniques are widely implemented in different topics. Thus, I believe this work is easily reproducible.
summary_of_the_review: This paper gives two simple designs, a dynamic loss and deep supervision, to improve the current MAE training and downstream performance. These two designs are widely researched in inpainting and contrastive learning topics, and they perform quite well on MAE, especially on classification and semantic segmentation. The motivations are well-described and well-supported by the experiments. I believe it gives good practices for the development of MAE.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Five out of six reviewers were negative about the paper, and the only positive reviewer did not find the rebuttal convincing. Hence, the decision is **not** to recommend acceptance. 
justification_for_why_not_higher_score: Five out of six reviewers were negative about the paper, and the only positive reviewer did not find the rebuttal convincing. 
justification_for_why_not_lower_score: Lowest already
Decision: Reject