Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning

Authors: Amin Karimi Monsefi, Mengxi Zhou, Nastaran Karimi Monsefi, Ser-Nam Lim, Wei-Lun Chao, Rajiv Ramnath
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning, Frequency domain analysis, Knowledge distillation, Vision Self-Supervised Learning, Pre-training

Abstract:
We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. Prior work in this direction masks out pre-defined frequencies in the input image and employs a reconstruction loss to pre-train the model. While achieving promising results, such an implementation has two fundamental limitations as identified in our paper. First, using pre-defined frequencies overlooks the variability of image frequency responses. Second, pre-trained with frequency-filtered images, the resulting model needs relatively more data to adapt to naturally looking images during fine-tuning. To address these drawbacks, we propose FOurier transform compression with seLf-Knowledge distillation (FOLK), integrating two dedicated ideas. First, inspired by image compression, we adaptively select the masked-out frequencies based on image frequency responses, creating more suitable SSL tasks for pre-training. Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks. Our experimental results demonstrate the effectiveness of FOLK in achieving competitive performance to many state-of-the-art SSL methods across various downstream tasks, including image classification, few-shot learning, and semantic segmentation.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730672807042

thread_type:
official_review

<thread_object>
creator: reviewer_6en5

<content>
summary:
The authors identify two fundamental limitations in the existing masked frequency modeling (MFM) paradigm: 1) constant filters overlook the variability of image frequency responses, and 2) no access to naturally looking images during pre-training requires more data to adapt to downstream tasks during fine-tuning. To address 1), the authors adaptively select the masked-out frequencies based on image frequency responses. To address 2), the authors employ a student-teacher framework via self-distillation. Experimental results on image classification, few-shot learning, and semantic segmentation demonstrate the effectiveness of the proposed method compared to the MFM baseline.

strengths:
-	The proposed method is well motivated. The authors motivate the method by identifying two key limitations in MFM, and propose two interesting solutions to address these drawbacks.
-	The paper is generally well-written and easy to follow.
-	The authors provide a comprehensive analysis based on their method. The experiments are extensive and the results are promising, especially for few-shot settings.

weaknesses:
-	The idea of using adaptive filers is interesting. However, the fitters still rely on some pre-defined thresholds, e.g., [0.005, 0.01, 0.05]. In practice, the authors may also need to tune these hyper-parameters to achieve the optimal performance for different datasets.
-	For CNN, according to Table 4 in Sec. B.2, the proposed method does not lead to further gains compared with MFM when it comes to full fine-tuning, which makes me concerned about its effectiveness for CNN architectures. Could the authors provide the justification on this?
-	For efficiency analysis, the authors only provide a comparison on GPU memory usage (Table 12, Sec. B.6). A comparison on training time with previous methods is also preferred.

questions:
See the questions mentioned above. Overall, I think it is an interesting paper with extensive experiments and analysis, which could provide some new insights for the community. Thus, I am leaning to accept this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698228747

thread_type:
official_review

<thread_object>
creator: reviewer_dqrd

<content>
summary:
The paper introduces a self-supervised learning (SSL) approach named FOLK, which stands for FOurier transform compression with seLf-Knowledge distillation. The method aims to address the limitations of previous frequency-based pre-training approaches by adaptively selecting frequencies for masking based on unique image responses. The dual-branch framework leverages both filtered and original images during pre-training, which is claimed to minimize the adaptation requirements for natural-looking images in downstream tasks. The experimental results demonstrate the effectiveness of FOLK, showing competitive performance in various downstream tasks such as image classification, few-shot learning, and semantic segmentation.

strengths:
The paper presents a new method that combines frequency-based masking with self-knowledge distillation, addressing known limitations in the field of SSL for computer vision tasks. The paper provides extensive experimental results that demonstrate FOLK's effectiveness across a range of tasks and benchmarks, showing improvements over existing state-of-the-art methods.

weaknesses:
The author proposed two limitations in the introduction, but the experiments did not directly discuss how to address these limitations. Simply showing performance improvements (e.g., image classification tasks) is not enough to support the author's claims.

questions:
1. The paper primarily focuses on the Com and RCom filters. It would be beneficial to see a comparison with other filtering techniques to establish the robustness and generalizability of the FOLK framework. Could the authors experiment with additional filtering methods, such as Gabor filters or wavelet transforms, and report on their effectiveness?
2. The related work section mentions that MFM has been applied to low-level vision tasks. Since FOLK builds upon MFM, it would be valuable to include a comparison of FOLK's performance on such tasks. Could the authors add experiments that benchmark FOLK against existing methods on low-level vision tasks to provide a more comprehensive evaluation?
3. While the supplementary material shows some results on robustness, a more detailed analysis would be appreciated. Could the authors provide additional benchmarks that specifically measure the robustness of the FOLK framework against various types of image degradations and noise?
4. The paper integrates knowledge distillation into the FOLK framework, but ablation studies regarding the contribution of this component are missing. Could the authors conduct ablation studies to isolate the impact of the knowledge distillation component on the overall performance? This would help readers understand the significance of this technique in the context of the proposed framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707733683

thread_type:
official_review

<thread_object>
creator: reviewer_nbws

<content>
summary:
This paper introduces Fourier transform compression with self-knowledge distillation (FOLK), a frequency-based self-supervised learning (SSL) method designed to improve pre-training efficiency. FOLK addresses the limitations by adaptively selecting masked frequencies based on image frequency responses and employing a two-branch framework for knowledge distillation. Experimental results show that FOLK achieves competitive performance across various SSL tasks.

strengths:
1. The framework is applicable and straightforward to understand.
2. The proposed method improves the learning of the student model and facilitates a more efficient training process.
3. The paper presents experiments across multiple datasets and various vision tasks, demonstrating the effectiveness of the proposed method.

weaknesses:
1. The dual-stream and frequency-domain masking approaches applied in the article are relatively common schemes. Could the authors elaborate further on the motivation of the proposed method?
2. More analysis and experiments are required on the framework design and cost computation, please see the questions.

questions:
1. Two views (u and v) of the input image are processed through the informed filtering process in the proposed FOLK framework. What is the optimal method for selecting views to enhance the model performance?
2. How can the complexity of the FOLK be reduced to enhance the framework's accessibility? Could you analyze the computational costs of the various methods evaluated across different models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732899381

thread_type:
official_review

<thread_object>
creator: reviewer_tjxa

<content>
summary:
This paper proposes a frequency-based SSL method to learn visual representations from unlabeled images, which significantly improves the training performance compared with existing works. In particular, the authors built upon the MFM method and identified two key limitations: 1) pre-defined frequency masking filters that ignore the intrinsic structure in individual images; 2) model pre-trained with frequency-filtered images leads more data to adapt to natural images in downstream model fine-tuning. In response, two specific new designs (a. masked frequency modeling with Com and RCom filters; b. multi-task self-supervision with self-knowledge distillation) are proposed to target these two problems. Their reported experimental studies have shown the effectiveness of their designs.

strengths:
**Originality**. The paper investigated two fundamental limitations in the MFM work and proposed two novel designs to address these limitations.  The presentation clearly shows what are the novel elements.

**Quality**.  The paper shows a successful way to perform masking in the frequency domain for unlabeled training images. Additionally, the authors provided a proper self-knowledge distillation framework to deal with the negative effect of training with frequency-masked images.

**Clarity**.  The submission neatly shows all the experiments that were carried out and the description of the underlying method is clear.

**Significance and Relevance**.  The topic is very interesting and important. Considering the growing demand for learning effective representations from unlabeled data, this paper pushed the boundary of SSL.

weaknesses:
**Training Cost**. Given that the proposed method employs a two-branch framework for model training, will it bring additional training costs compared with the original MFM?

**Masking Filters**.  What are the exact formulations of Com and RCom masking? or pseudo code to construct Com and RCom might be helpful.

**Data Augmentations**. In generating two views, u and v, distinct transformations (random cropping, color jittering, etc.) are conducted. It seems no ablation studies are provided for analyzing the effect on the consecutive image frequency masking and final model training.

questions:
See above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732083568827

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their tremendous efforts in the paper review process. We are glad that the reviewers found our proposed method “novel” (tJxA) and “well motivated” (6EN5), our experiments “extensive” (6EN5, DqRd), and our paper “well-written” (6EN5) and “straightforward to understand” (nBwS). We appreciate all the review comments and have tried our best to address most, if not all, of the concerns.

According to the reviewers' suggestions, we have now made modifications to the manuscript to better clarify our method. **Modifications are highlighted in red texts in the updated manuscript for easy identification. All line numbers mentioned in our responses are according to the modified version of the paper.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732207097551

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you for taking the time to review our paper. We appreciate the insightful questions and feedback provided in your initial review. We have carefully considered your comments and have addressed each point in our rebuttal. However, we are more than willing to engage in further discussion if you have any further questions, require clarification on any aspect of our work, or wish to suggest additional improvements.

Thank you again for your valuable feedback, and we look forward to hearing from you. Best regards, Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a frequency-based self-supervised learning method that improves pre-training efficacy by addressing limitations in masked frequency modeling (MFM). The proposed approach adaptively masks image frequencies and employs self-knowledge distillation. Reviewers appreciated the novelty, motivation, and extensive experiments, particularly highlighting improvements in few-shot learning. Concerns were raised about training cost, hyperparameter tuning, and the dual-branch complexity, but the authors’ clarifications and modifications addressed most of these. Given the overall positive feedback, the AC recommends acceptance.