Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a method to accelerate the training of MAE by reducing the number of tokens to reconstruct. Specifically, the authors use a teacher-student training pipeline where the teacher operates on mask tokens to progressively identify those redundant tokens among them, finally providing a compact token set as the reconstruction target of the student network. A simple ranking method is proposed for scoring the importance of tokens by calculating the similarity of each token with the mean value.
strength_and_weaknesses: Pros:
1. The proposed method is simple and easy to follow. 
2. Experimental results demonstrate the method can significantly improve the training efficiency of MAE in terms of both throughput and memory cost, while still retaining the performance.

Cons
1. The motivation for improving efficiency by reducing reconstructed tokens is questionable. The authors claim that the decoder of MAE takes up most training costs, which however only holds when the decoder is deep. Actually, as pointed out in the original MAE paper, applying a lightweight decoder would not significantly affect its performance. In this case, the decoder gets efficient so that it is unnecessary to use a small set of mask tokens. Further, in another SSL method SIMMIM, the decoder is even more lightweight than in MAE, rendering the applicability of PCAE limited.

2. Several things are lacking in the analysis. More work needs to be done for verification.
- In Sec. 3.1.1, the authors illustrate the diversity of tokens. How is this diversity related to the representation power of the model? Why does PCAE has similar top1 results on Imagenet1k as  MAE even though its tokens are more diversified?
- If other methods, like using saliency to select tokens, can achieve comparable results with the proposed method.
- It is unclear how to utilize the average of abandoned tokens in downstream task.
- The sensitivity of token dropping ratio on the performance of downstream tasks.

3. Overall, the performance of PCAE is marginally above that of MAE as shown in Table 1 and Table 2, suggesting the learned representations in this way have little to no improvement over the baseline.
clarity,_quality,_novelty_and_reproducibility: Refer to my above comments.
summary_of_the_review: Refer to my above comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Self-supervised learning helps learn effective representation that is often useful in downstream tasks. Reconstruction of masked patches from visible ones is a crucial step in learning effective representation. However, reconstruction of the entire image seems redundant given most parts of a natural image are highly correlated. This leads to issues in computational and storage space complexity. This paper proposes PCAE to reconstruct some portions of the natural image by progressively dropping non-relevant patches based on mean similarity score. It uses the vision transformer to measure the leakage of information from discarded tokens to retained ones. The efficacy of the proposed strategy is demonstrated on large-scale pre-training datasets and several downstream tasks. 

strength_and_weaknesses: **Main Review**

**Major Strengths**

The paper proposes a strategy to remove redundant masks from self-supervised training that is demonstrated to be quite effective in several tasks including object detection and segmentation. 

It either accelerates throughput or improves memory performance in the reported experiments.

The experiments are comprehensive and help visualize the step-wise discarding method. 

The ablation study is quite useful in choosing drop case, drop ratio, and drop stage which are three crucial hyper-parameters of the proposed method PCAE.


**Weaknesses**

Similarity measure in pixel space using mean statistics is fundamentally not so efficient, especially for images (Ref Equation 3). This is because natural images often lie on a low-dimensional manifold where a smaller distance in pixel space may not help identify images containing similar features. Similarity in feature space might be helpful, but not so much when mean is used rather than a learned metric as done in related works (Sec 2).

I am confused about how to choose the size of the patches. If I choose a large patch size, it may have high frequency components and a small patch size may have low frequency components. The algorithm may not consider patches to be redundant if there are too many high frequency components in it. On the other hand, there might be too many redundant patches if the size is small.
Also, how to choose the cut-off frequency to separate high and low frequency components while discarding redundant patches? What is the intuition behind choosing a threshold?

What is the optimal stopping point while discarding redundant patches and what is the criterion for choosing this optimal stopping point?

In Section 3.4, what is the intuition behind concatenating the average of dropped tokens with the retained ones? Is it just another heuristic? My main concern is that the patches are dropped based upon their similarity with the mean, which means the average of these dropped patches must be close to the mean. Then, what additional information does it provide when we append it to the retained ones?

In Appendix E, compared to RD, PCAE offers an improvement from 81.1 to 82.3, which does not seem significant when their computational complexity is considered. It is relatively easier to randomly drop patches than the proposed strategy as long as the performance drop is not so much. 

clarity,_quality,_novelty_and_reproducibility: There are some minor typos in the current draft, such as inconsistent use of parentheses for inline citations in Section 2. 
 
The source code is not provided to help with the reproducibility. 

I think the novelty of the paper is limited, refer to the main review. 

summary_of_the_review: I do believe that the paper has some interesting ideas. However, there are so many statements as listed in the main review that seem adhoc. The reason or intuition for the proposed ideas is not well-motivated. Although the empirical results look good in terms of throughput and memory requirement, the methodology could be communicated more clearly. 

After discussion with the authors, I am convinced that the paper has some major strengths that will benefit the community. Assuming that the authors will make the necessary changes in the final version, I am raising my score to borderline accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an improvement for MAE pre-training, that instead of reconstructing all the patches, it attempts to reconstruct only a certain number of diverse/important patches. The selection for diverse/important patches is done through a momentum encoder -- it will discard patches that are closest to the mean (as it is empirically corresponding to the low-frequency, background patches), and only use the remaining patches for MAE decoder to predict. Such an idea is shown to perform well on ViT-B, with both faster training and (slightly) better results than MAE. The idea of progressively discarding patches is also attempted during the fine-tuning stage as well.
strength_and_weaknesses: Strengths:
- The goal of reducing the number of reconstructed patches and the idea of progressively discarding patches are interesting and new to me. Regardless of how significant the progress is made in terms of accuracy or training efficiency, the exploration is also revealing quite interesting properties of the original MAE.
- The paper is relatively well-written, and the illustrations are professional and clean.
- I particularly like that the paper is highly focused on the goal of reducing the number of reconstructed patches, and does extensive evaluations on the removal strategies. This gives a more complete picture of the proposed idea.

Weaknesses:
- My biggest concern is about the computation advantage over MAE when the encoder gets larger. Right now the analysis (especially on speed) is completely dependent on ViT-B -- this is where MAE has the least advantage due to the use of a fixed-sized decoder and the size ratio between the encoder and the decoder in this case is the smallest. I believe the speed gain may not be so healthy if one uses a larger encoder, as the momentum encoder requires an extra forward pass through the encoder.
- While I think the paper is relatively well-written, some of the paragraphs in the abstract and in the introductions are not so welcoming for readers. For example, before introducing that the method contains a momentum encoder, it is already talking about doing the removal using the momentum encoder.
- I think using the "crops" to compare methods in Table 1 is a bit misleading. Right now PCAE indeed only uses 1 crop from each image in each iteration, but it does *2* forward passes through the encoders (one base, one momentum), at least one plus a partial one. The computation complexity is not reflected in the number of crops, but rather in the number of forward passes and backward passes.

Questions:
- What if we just have another pre-trained model (potentially self-supervised) just do the judge about which patches are more important/salient? The model can even be small. The model can even be non-neural network based. For example, one can just compute a saliency map, and remove patches based on the average saliency map within each patch? Maybe the computation can be further saved without even using a momentum encoder?
clarity,_quality,_novelty_and_reproducibility: Clarity: As I mentioned above, the paper is relatively clear in writing, with some minor issues. I think they are not major if one continues reading and does multiple passes, but would be much more friendly to readers who are confused that MAE does not have momentum encoders.

Novelty: I think the exploration on reducing the computation complexity with progressive downsampling is interesting and new. I have not seen it before. 

Reproducibility: It is a concern here. The paper did not mention about the code/model release, and did not reveal important hyper-parameters (e.g., the momentum coefficient, the learning rate, batch size etc.). I don't think readers have enough information to easily reproduce the results in the paper.
summary_of_the_review: Overall I am leaning toward acceptance. The paper has the merit that explores further reducing the computation complexity by discarding redundant patches along the way, which no one has done before if I remember correctly. The writing and illustrations are relatively clear. The results are showing some improvement over the baseline. However, I do have concerns about the scalability of the approach (to larger models), and the reproducibility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper improves the efficiency of MAE training by exploiting the redundancy in reconstructed tokens. They propose to use the similarity to mean token as an emprical criterior to prune the tokens, which leads to savings in computation. They achieve 1.5x-1.9x speed up compared with MAE, while maintain similar performance.
strength_and_weaknesses: Strength:
- The overall idea of improving the efficiency of MAE training by reducing the redundancy in reconstruction is reasonable. 
- Large amount of experiments and ablations(drop ratio, drop location etc) to validate the claim. 

Weaknesses:
- The proposed emperical criterior of 'similarity to mean token' is not compared with any other strategy. The author mention the potential to use matrix decomposition but no experimental results are shown. 
- How are the token discarded in pretraining phase is not very clear. Is there also a warmup strategy for token discarding in pre-training? 
- There is no comparison to other approach that dynamically discard token. Since the presented method can also be applied to supervised task, there should be room for comparison?

Other comments:
- The table number are incorrect. For example, you said "As the training may be influenced by many factors, we show the throughput comparison between PCAE and MAE in Table 4.2". But I guess you really mean Table 3.
- In the second paragraph of section 3.2, consider use a different symbol for the redundant token identification function? As D is already used in the definition of x_i.
clarity,_quality,_novelty_and_reproducibility: Exploiting the redundancy in tokens to accelerate training is not novel, but the paper apply such idea to the self-supervised regime. A large amounts of experiments are conducted to support the claim. The paper is generally easy to follow. 
summary_of_the_review: The method this paper presented is empirical but a large amounts of experiments are conducted to validate the effectiveness. The results of introducing 1.5x-1.9x speed up over MAE is somewhat significant afaik. 

Post rebuttal:
I appreciate the response and additional exp results from the author. The exp results are solid but the relatively weak novelty keeps me from giving a higher rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to design a more efficient MIM method by taking visual redundancy into account. The paper identifies the redundant tokens and progressively reduces the number of tokens for the reconstruction target. Competitive results are achieved while accelerating MAE for 1.9 times.
strength_and_weaknesses: ### Strengths
1. **Motivation**: It is an important task to reduce the computation overhead in MIM pre-training. And the introduced method seems practical in the real world.
2. **Writing**: The paper is generally well-written and easy to follow.
3. **Method design**: The introduced method for redundant token identification is easy to implement and makes sense.
4. **Results**: Competitive results on ViT-Base are shown.

### Weaknesses
1. **Motivation**: Though I get the overall point of speeding up MIM pretraining, I am confused by some of the declarations. (1) In the first paragraph of the introduction, the authors pointed out that the efficiency bottleneck lies on the decoder, which is limited to the encoder-decoder architectures like MAE while neglecting the other ones like BEiT and SimMIM (unify encoder and decoder). Moreover, the number of layers in the decoder should influence the accuracy and efficiency of PCAE, which should be analyzed. (2) In the second paragraph of the introduction, the authors claimed that useful information is discarded when naively removing part of the reconstruction patches. The claim is ungrounded and the authors should provide some evidence for it. Intuitively, useful information should be learned in the unmasked tokens and have nothing to do with removing part of the masked tokens.
2. **Method design**: Though the introduced method for redundant token identification (removing the ones close to mean tokens) is intuitive and indeed helps training. The authors are recommended to analyze other alternative options for it. 
3. **Results**: (1) The experiments on more architectures, e.g., ViT-S/L, are necessary for verifying the generalization ability of the introduced method. Especially since the method depends on the drop ratio and drop layers, which might "overfit" on the architecture and make it hard to deploy on other models. (2) As the method can save time, it would be interesting to see if better results can be achieved by training with more data or more epochs while the overall time is still less than other MIM methods. (3) In the appendix, the authors introduce to adapt PCAE on top of SimMIM but did not provide accuracies. Will PCAE perform well on SimMIM?
clarity,_quality,_novelty_and_reproducibility: + **Clarity**: The paper is easy to follow despite that some claims of the motivation need clarification (see Weaknesses).
+ **Quality**: The paper addresses a practical problem in an intuitive way, but the experimental results seem insufficient to fully validate the method.
+ **Novelty**: The method is new to the community of MIM. But it should be better to discuss some works on manipulating tokens in ViT architectures, e.g., "Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer" (CVPR'22).
+ **Reproducibility**: I did not find the details of implementation, e.g., hyper-parameters. 
summary_of_the_review: Overall, I am intrigued by the motivation for this work and believe it is important for practical applications. However, there are still some issues, see Weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to reduce the pre-training overhead of MAE by decreasing the number of tokens to be reconstructed. Specifically, the authors follow CAE to design a momentum encoder to operate on masked tokens and gradually discard some tokens (that are closer to the average tokens) at some layers. In this way, the overhead of decoding is reduced, since fewer tokens are needed to be predicted. This paper claims that this manner can speed up the pre-training without compromising performance on different downstream tasks.
strength_and_weaknesses: Strength:

1.  This paper addresses the patch redundancy problem in MIN, which was largely overlooked in previous literature.

2. The paper is well-written and easy to follow. 

Weaknesses:

1. It seems unclear to me where speed-up comes from. It seems to me that speed-up mainly comes from reducing the decoding overhead by decreasing the number of tokens to be reconstructed. However, the decoder of the original MAE has been kind of lightweight. Even more, following papers like SimMIM, CAE, etc, have shown that a one-layer decoder can achieve nearly the same performance.  In this regard, do the authors still believe their proposed methods are practical and significant compared to just continue reducing the capacity of the decoder, e.g., a one-layer decoder?

2. There are some issues with the formulations. 
- Equation (1) is not rigorously presented: (i) The dimension of w and p is not aligned. One is “a” shared token while the other one is “a sequence of” tokens; (ii) The authors claim that the M function is a similarity function. But one of its inputs are the embeddings of masked tokens and the other input is a scalar k. What does their similarity mean? (3) So, what is the objective this equation wants to maximize? 

- Around Equation (2), please clarify the NxD. Does N mean the number of remaining tokens at every layer? If so, then a fixed N cannot be used since the authors change it at some selected layers.

- Equation (3) has a minor error. It should be summing over the j.

clarity,_quality,_novelty_and_reproducibility: The writing part of this paper is good and easy to follow. However, the formulation of the framework is not rigorously presented, which has been pointed out in the previous box.
summary_of_the_review: This paper proposes to speed up the decoding of MAE by reducing the number of tokens to be predicted, which is interesting. However, It seems that the current technique can only be applied in MAE since it mainly reduces the decoding part. For other methods such as BEiT, CAE and SimMIM have more lightweight decoding, it seems not that suitable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Six experts reviewed the paper. Four reviewers recommended it "marginally above the acceptance threshold" with high confidence, one reviewer labeled it as a good paper, and only reviewer B38C placed it "marginally below the acceptance threshold". Though reviewer B38C raised questions about the paper's experimental results and analyses, AC found the authors' rebuttal convincing. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make the necessary changes to the paper to the best of their ability following the reviewers' comments. 
justification_for_why_not_higher_score: The paper's experimental results and analyses could be improved.
justification_for_why_not_lower_score: The overall quality is good, and five out of six reviewers liked the paper.
Decision: Accept: poster