Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unsupervised method for improving already trained self-supervised models. The method, TEC, consists in reconstructing the masked patch representations of an image. The target, i.e. the feature representation of the dropped patches, are obtained from a pre-trained SSL model.
strength_and_weaknesses: **Strengths**

- Extensive comparisons with sota and thorough presentation of the related works.

- The experimental results and absolute numbers reported in this paper are strong. This paper advances the state of the art in self-supervised representation learning.

- In my opinion, the strongest result is in Table 7d, where the paper shows that MAE-300 epochs + TEC-100 epochs (i.e. 400 epochs)is competitive with MAE-1600 epochs pre-training. This convinces me that the method can speed up SSL pre-training.

- Many ablations are presented.


**Weaknesses**

- Unconvincing “sustainability” ability of the paper.
This work starts from the observation that even though SSL is progressing in computer vision, models are not really used in practice for downstream tasks. The paper argues that this is because (i) pre-training SSL costs are too high and (ii) new and better SSL methods are appearing very frequently, preventing the field to stabilize on a framework.
However this argumentation is flawed for the following reasons.
(i) Models with extremely high pre-training budgets are being adopted in practice (GPTs, CLIP, models trained on JFT), meaning that expensive pretraining costs do no prevent the pretrained models to be used subsequently.
(ii) The fact that new SSL methods arrive might just be a signal that SSL in vision is not fully mature yet and that further progress needs to be made to clearly beat supervised learning (which is still the most used pre-training in vision).

- Table 6 is the weakest part of this paper. It shows that TEC does not achieve “sustainability” as advertised in Figure 1.
Unlike the sustainability property advertised by the paper in Figure 1, Table 6 shows that a TEC model cannot be a based model for a new TEC-training round (marginal improvement of +0.1%).

- Table 1 should include comparison with strong supervised baseline like DeiT-3. In addition, this is very misleading in Table 1 not to count the base model epochs into the total number of epochs. In other words, when training TEC-MAE for 300 epochs, the column should be 1900 epochs (1600 + 300) instead of 300.

- Linear evaluation is 10 points below sota (iBOT for example). I find it weird that authors don’t report sota numbers in this Table 4.

- The method is complex with many added components (input and encoder adapters, use of self-attention maps for feature selection). Complexity might not go towards sustainability.

- This is not really a new SSL method but rather a feature distillation or enhancement one. Should compare with this literature (in last paragraph of related work) more and start from the same backbone as theirs for fair comparison.

clarity,_quality,_novelty_and_reproducibility: **Clarity**: the paper and figures is difficult to read and the contributions are not well stated. Also, I find that presenting the method in the scope of sustainability is rather unconvincing. The paper would benefit from some re-writing to emphasize more the fact that it is a kind of “self-supervised distillation method” (falling under the same umbrella as methods in the last paragraph of the related work section).

**Novelty**: The different components of the method are not novel: reconstructing dropped patches based with feature representations from another model as target (BeiT, MaskFeat), leveraging the salient attention maps given by a ViT (LeoPart), distilling from a pre-trained SSL model (clusterFit, maskfeat).
However we can argue that the novelty lies into their combination.

**Reproducibility**: As a practitioner, I would argue that the paper does not give enough implementation details for allowing the reproduction of the reported results (for example there is no implementation details about data augmentation nor about the downstream tasks). It wouldn't be a problem if the paper mentions that code will be made publicly available but there is no such mention in the current version.

summary_of_the_review: For the reasons stated above (weaknesses section + problem of reproducibility), I lean towards rejection of the paper.

That being said, I still think the paper is promising and some reported results are strong. I will be willing to upgrade my initial assessment based on the authors’ rebuttal.

In any case (acceptance or rejection), I definitely think the paper should be re-written to tone down the “sustainability” aspect and present the contribution through the scope of “distillation” or “feature enhancing”.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies a relatively new problem in self-supervised learning, which is called sustainable self-supervised learning. Based on a pretrained SSL model (or partially trained with certain epochs), the authors want to further improve it with the proposed target-enhanced conditional mask-reconstruction. The authors conduct experiments on benchmark datasets and shows the effectiveness of the proposed "fine-tuning" strategy.
strength_and_weaknesses: Strength: 

1. The problem setting is new or interesting. It should find applications in different areas, such as medical images. 

2. The proposed solution is reasonable and achieves good performance.

Weaknesses:

1. It seems that the authors should further clarify/motivate the proposed setting. Why do we need "sustainable" SSL. In the current introduction, the authors mention that researchers may have limited computational budgets to XXXXX, while it seems that the proposed solution cannot reduce the computation budget if I want to adopt a pretrained SSL model to downstream tasks. For my understanding, we still need to use a large ViT model and need to train many epochs. In other aspects, the previous SSL knowledge distillation can reduce the computational budget by re-training a small model. 

2. For the imagenet fine-tuning experiments, it is hard for me to understand the motivation of this new problem setting. This experiment looks like we want to continue "pre-trained" an SSL model with large-scale datasets. In this case, why do we not adopt a "well-trained" SSL model (MAE trained with 1600 epochs)? As it is only pretrained once, it can be done by a giant company and release to others for downstream tasks. In a word, the authors should reconsider the evaluation part to support the argument for adopting this new problem setting.

3. It seems that the authors should compare with SSL knowledge distillation works. I understand the problem settings are different, while the method design should be similar. In some sense, the proposed method is also an advanced knowledge distillation method. 

4. For Table 1, it seems that the authors should clearly mention how the previous results () are obtained. For example, from released public models or re-implemented models? And how the base SSL model is acquired for the proposed framework. Did the authors re-train the base MAE model, or did the authors adopt the publicly released MAE model?

5. The technical novelty may be incremental, as the key technical components are focused on designing new pretested tasks.
clarity,_quality,_novelty_and_reproducibility: In general, this paper is well-written, and the logistics are clear. The proposed new problem setting seems promising, but it needs more experiments and clear motivation. The proposed method is incremental, as it only involves designing new pretext tasks.
summary_of_the_review: In summary, this paper studies an interesting paper and should have an impact on the computer vision field, while more clarification of motivation and experimental evaluation is needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new reusable self-supervised learning framework by distilling knowledge from existing pretrained SSL models. Specifically, authors first introduce patch-relation enhanced targets to encourage the new model to learn semantic-relation knowledge and then introduce a conditional adapter that adaptively adjusts new model prediction to align with the target of each base model. Experiments using MAE and iBOT show the effectiveness of the proposed framework on ImageNet dataset.
strength_and_weaknesses: **Strengths**:

- The paper is easy to read, and very well written with clear presentation of the figures/tables.
- The overall problem of reusing pretrained SSL models is interesting and the solution proposed to solve the problem has clearly demonstrated advantage on ImageNet dataset. 

**Weaknesses**:

- The main technical contribution of the paper is actually on distilling knowledge from an existing pretrained model to another new model. This is in fact very much similar to reversed KD presented in "Revisiting knowledge distillation via label smoothing regularization" (transferring knowledge from a small model to a large model) but there is no discussion about this in the paper. I don't fully agree with the authors that knowledge distillation only focuses on transferring knowledge from strong and large models to compact new models. Authors should clearly discuss about the similarity between the proposed framework and (reversed) KD.

- Is there any restriction on the model size across base and new model? Since KD is agnostic to the model size, authors should perform experiments by transferring knowledge from a smaller SSL model for training a larger model, e.g., use of ViT-Small for training ViT-Base with MAE.

- In many practical scenarios, multiple pretrained models are usually available that can be leveraged for training a new model? How this framework can be used to exploit multiple models with same and/or different architectures? Authors should perform experiments to verify this using multiple base models.

- Can the proposed framework exploit SSL models trained using instance discrimination, e.g., SimCLR for training a new ViT MAE model? What about the performance in these scenarios?

- Sustainability usually refers to savings in terms of training cost which is measured by FLOPs or GPU wall time. While the comparison includes number of epochs, it does not reflect real savings. I would encourage authors to include Accuracy vs FLOPs and Accuracy vs Wall time to demonstrate how use of pretrained SSL model can help in training efficiency.  See this paper: Knowledge Inheritance for Pre-trained Language Models for more details which uses a smaller pretrained model to accelerate training of a larger language model or bert2BERT: Towards Reusable Pretrained Language Models which solves a similar problem in NLP.

- The proposed method uses a bunch tricks on top of the vanilla knowledge transfer strategy without analyzing other alternatives. What is the use of adapters at the input level and encoder level is not clear? Can authors qualitatively demonstrate how this helps in learning better features in the new model?

- What about the number of trainable parameters? What about the increase in training time due to the increase in number of parameters? A more thorough analysis on this should be included in the paper.

-  Besides empirical performance, how does semantic attention help learning diverse complementary features? Can authors perform visualization experiments to verify this?

- What about the additional computation incurred by passing all the images through the base model? An analysis on how much data we need to pass through the base model for an effective knowledge transfer should also be included in the paper.

- Did authors try considering this knowledge transfer as an additional regularizer on top of the standard MAE loss for the new model? What happens if we train the new model using a combination of the standard MAE loss that does depend on the base model and another loss that actually matches the prediction with the base model using the proposed framework?
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is good and well structured. However, the technical novelty of the paper is very limited. It simply combines different existing well known approaches such as adapters at input level, encoder level, distillation at feature level and attention level etc. The only contribution I see in the paper is the combination of these techniques for developing a technique for reusing pretrained SSL model.
summary_of_the_review: The experiments are limited and not convincing in the current version of the paper. Due to the limited novelty and lack of convincing experiments, my initial recommendation is to reject the paper (below the acceptance threshold).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel method to train an SSL model by distilling knowledge from an existing SSL model (base model). By leveraging the base model, the method can help to learn a stronger SSL model in a cost-friendly way. Specifically, they propose two complementary reconstruction targets that enable knowledge distillation, 1)the patch-dim normalized features and 2) patch attention maps with rich semantics. They also have a conditional adapter module that adapts the new model's prediction to that of the base model. Exhaustive experimentation shows that their approach outperforms existing approaches on varied vision tasks and at a fraction of computation time.
strength_and_weaknesses: 1. The proposed method to distill knowledge from an existing SSL model to build a stronger new model is novel and addresses an important issue of training/improving SSL models in resource-constraint situations.
2. Exhaustive experimentation shows that their approach outperforms existing approaches on varied vision tasks and at a fraction of computation time.
3. The contribution of different parts of the model like patch-norm features, selective attention maps and others are well demonstrated by the ablation experiments performed by the authors.

Weakness/Suggestions:
1. It would be nice to see the computation comparisons in terms of FLOPs.
2. I would recommend modifying the schematic diagram, especially the Attention Selection and Conditional Adapter part so that it is more intuitive from the schematic diagram itself. Currently, just from the schematic diagram, it is hard to understand these modules of the method.
3. Maybe in the appendix, it would be nice to visualize some more reconstruction targets (i.e. the patch-dim normalized features and selected attention maps) so that it helps to develop an intuition for the readers for what knowledge the new model is trying to distill.
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel. Some aspects of the method like selective attention maps and conditional adapters can be explained with more clarity. An average deep learning researcher who is not super familiar with the SSL field would find it different to understand. In terms of reproducibility, I didn't find the code that enables the reproducibility of the results. Authors are encouraged to share the code in the supplementary material. In terms of quality, the experiments and ablation studies are exhaustive and well-performed. 
summary_of_the_review: Overall, based on the strengths, weaknesses, and other factors like clarity, novelty, and reproducibility, I place this submission marginally above the acceptance threshold (6).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a recipe for obtaining a stronger self-supervised model by exploiting and building upon an off-the-shelf SSL backbone. Using a form of MIM, the masked patches are represented using the pre-trained model. This is in line with some of the recent work where multimodal features are used in a similar fashion. The paper presents relatively strong results, especially when showing that 300 epochs of MAE pretraining + 100 epochs of TEC surpasses the 1600 epochs of pure MAE. 
However, the proposed positioning about sustainability and motivation could be improved. Moreover, Table 6 shows that several rounds of TEC training do not improve results further, making the point of the paper weaker. Also, the difference between the proposed method and knowledge distillation is unclear. The paper has received borderline ratings, with a lot of constructive and actionable feedback. I encourage the authors to take this feedback into account, modify the paper’s positioning and submitting to another venue. For the time being, I recommend rejection.
justification_for_why_not_higher_score: The positioning of the paper should be improved. The proposed method allows to improve a pre-trained SSL backbone by exploiting it as a target predictor, and hence should be compared with distillation approaches. Some of the results are interesting, but they are not in axis with the claims about sustainability.
Decision: Reject