Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning
Authors: Amin Karimi Monsefi
Mengxi Zhou
Nastaran Karimi Monsefi
Ser-Nam Lim
Wei-Lun Chao
Rajiv Ramnath
Venue: ICLR 2025 Poster (2025)
Keywords: Self-supervised learning
Frequency domain analysis
Knowledge distillation
Vision Self-Supervised Learning
Pre-training
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce FOLK, a novel SSL method that adaptively masks image frequencies and uses knowledge distillation, enhancing pre-training and reducing downstream data needs.

Abstract: We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. Prior work in this direction masks out pre-defined frequencies in the input image and employs a reconstruction loss to pre-train the model. While achieving promising results, such an implementation has two fundamental limitations as identified in our paper. First, using pre-defined frequencies overlooks the variability of image frequency responses. Second, pre-trained with frequency-filtered images, the resulting model needs relatively more data to adapt to naturally looking images during fine-tuning. To address these drawbacks, we propose FOurier transform compression with seLf-Knowledge distillation (FOLK), integrating two dedicated ideas. First, inspired by image compression, we adaptively select the masked-out frequencies based on image frequency responses, creating more suitable SSL tasks for pre-training. Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks. Our experimental results demonstrate the effectiveness of FOLK in achieving competitive performance to many state-of-the-art SSL methods across various downstream tasks, including image classification, few-shot learning, and semantic segmentation.

Published as a conference paper at ICLR 2025
FREQUENCY-GUIDED MASKING FOR ENHANCED VI- SION SELF-SUPERVISED LEARNING
Amin Karimi Monsefi Mengxi Zhou?, Nastaran Karimi Monsefi'
Ser-Nam Lim², Wei-Lun Chao², Rajiv Ramnath
(karimt [karimimonseff chac 209, ramnath. 8osu edu, sernam@ucf edd The Ohio State University, 'Hamedan University of Technology. +University of Central Florida
ABSTRACT
We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. Prior work in this direc- tion masks out pre-defined frequencies in the input image and employs a recon- struction loss pre-train the model. While achieving promising results, such an implementation has two fundamental limitations as identified in our paper. First, using pre-defined frequencies overlooks the variability of image frequency re- sponses. Second, pre-trained with frequency-flteree images, the resulting model needs relatively more data to adapt to naturally looking images during fine-tuning. To address these drawbacks, we propose FOurier transform compression with Soo-KKnoleedge distillation (FOLK), integrating two dedicated ideas. First, in- spired by image compression, adaptively select the masked-out frequencies based on image frequency responses, creating more suitable SSL tasks for pre- training. Second, we employ two-branch framework empowered by knowl- edge distillation, enabling the model to take both the filtered and original im- ages input, largely reducing the burden of downstream tasks. Our experi- mental results demonstrate the effectiveness of FOLK in achieving competitive performance to many ofate-of-the-art SSL methods across various downstream tasks, including image classification, few-shot learning, and semantic segmenta- tion.
INTRODUCTION
In recent years, Self- Seff-Supervised Learning (SSL) has gained considerable interest in the context of visual pre-training. This interest stems from its prominent capability of extracting meaningful visual representations from the vast expanse of readily available, unlabeled images without the need for costly manual labeling (Ben-Shaul et al., 2024 [Su et al., 2024) Almalki Latecki 2024). Key to this advancement is several pre- tre-ttraning methods established with different pretext tasks, including multi-view contrastive learning (Oord et al.] 2018 Chen et al.] 2020bb Tian et al.) |020bb] He et al.
2020,,
Masked Image Modeling (MIM) (Bao al. 2021 He al. (022aa] Xie et 2022 Monsefi
et al.
2024a| Oquab et al., 2024), Masked Frequency Modeling (MFM) (Xie et al. 2023] Liu et al.
2023
Zheng et al. 2024). and self-supervised Knowledge Distillation (KD) Kakogcorgiou
al,
Zhou et al., 2022] Chen et al. (2020c) Chen & He [02]] Caron et al. 2021). In the recent popular approach of Masked Image Modeling (MIM), key strategy involves masking portions of an image and then tasking models with either reconstructing these hidden sections or generating feature representations for them (Bao et al.] [02T]] Xie al., [022] He et al.) 2002a Yi et al., 2023). Through this process, the model is encouraged to learn robust feature representations that capture the underlying structure between unmasked and masked image parts, thereby enhancing its
2022
understanding of image semantics.
Instead of masking in the spatial domain (MIM), Masked Frequency Modeling (MFM) (Xie et al., 2023) introduced self-supervised approach that masks frequency components of the input image. Since high-level semantics and low-level details of an image can be separated into different fre- quency components (Oppenheim & Lim 1981 Piotrowski & Campbell 1982; Navard & Yilmaz 2024]. the frequency domain offers a more convenient avenue for revealing underlying image pat- terns. The pretext task in MFM is predict the masked frequencies from the requency-filteree

---

Published as a conference paper at ICLR 2025
(b) Low-pass Filtered
(c) Low-pass Result
(d) High-pass Filtered
(e) High-pass Result
(a) Original
Image
(1) Com Filtered
(g) Com Result
(h) RCom Filtered
(i) RCom Result
Figure 1: Detailed visualizations outlining our proposed masking approach compared to the low/high-pass filters used in the MFM method (Xie et al.] 2023). Figures d, h here show the retained frequencies after applying the corresponding filter (zoom-in views the shifted spec- trum for better comparison). Figures c, e, g, show the restored image from the retained frequencies. More examples can be found in Appendix D. All images used in this paper from ImageNet (Deng et al. 2009).
has two main advantages: first, can help avoid issues encountered when analyzing raw pixel values in the spatial domain, such as spatial redundancy (Wang et al., 2020
image (see Section 3.1
Chen al. 2023); ; second, unlike the patch-based masking used in MIM which often restricts the model to Vision Transformer (ViT), MFM is suitable for both VIT and Convolutional Neural Network (CNN)-basee based models.
However, though MFM (Xie et 2023) has demonstrated promising results, it has two fundamental limitations. Firstly, MFM uses constant filters, controlled by a pre-defined hyperparameter radius, for masking in the frequency spectrum. This disregards the intrinsic structure specific to individual images, which leads to a less challenging task for reconstruction (see Figure TB and Figure [d] Secondly, in the pretext stage, MFM only shows frequency-maske images to the model without mechanism to properly expose the raw information the original images. This potentially restricts the pre- pre-trained model's understanding of normal image distribution which further hampers MFM's training efficiency and model effectiveness, especially making it unsuitable in a few-shot learning scenario (see Section 4.2.2).
Our motivation is to achieve more effective vision pre-training through MFM, by addressing its aforementioned limitations. To this end, we propose novel framework that integrates FOurier transform compression with seLf-Knowledge distillation, termed as FOLK. Similar to MFM and dissimilar MIM, FOLK applies masking to image frequency responses and embraces both ViT and CNN architectures. Furthermore, it resolves MFM's 's problems from two main perspectives.
Firstly, instead of adopting constant filters. we seek an improved masking scheme that considers each image's 's unique frequency responses, to improve the pre- pre-training efficiency and model effectiveness. Inspired by the attention-based masking in MIM approaches such as AttMask (Kakogeorgiou al., 2022), where the most critical parts of the image are hidden from the student model to create more challenging pretext task, FOLK utilizes the Fourier transform compression (Pratt et al. 1969) concept to mask the most critical parts in image frequency responses. Two types of filters, the Com filter and its counterpart, the RCom filter, are designed to retain (or remove) the highest coefficient values within the frequency spectrum (see Fig. [ In contrast to the constant filters used in MFM, these Com and RCom filters adaptively mask the frequencies that carry the essence (or finer details) of each individual image, creating greater variations across training samples. Hence,
2

---

Published as a conference paper at ICLR 2025
more challenging pretext task is presented to the model, enforcing its understanding of macro and micro visual cues uniquely held by each image.
Secondly, we question how to properly expose natural image information to the model during pre- training to enhance fine-tuning efficiency. To this end, FOLK incorporates a knowledge distillation strategy using self-supervised teacher-student design. With the original image fed to the teacher model, and the frequency-masee masked image fed to the student, the student model learns not only to reconstruct the masked frequencies (as what MFM does), but also to reconstruct the original image's representation (generated by the teacher model) from the frequency-masker view of the same image. This multi-task teacher-student approach allows for model perception on both masked and original image realms, hence enhancing training stability and the pre-trained model's efficacy when applied to downstream tasks, as demonstrated in our experimental findings (Section 4.2
To summarize, our contributions are threefold:
We introduce novel masking technique in masked frequency modeling with Com and RCom filters, which presents a meaningful and considerably more challenging pretext task for efficient SSL..
We propose the FOLK framework, an innovative multi-task self-supervision methodology with self self-knowledge distillation to allow for model perception on both frequency- images and original images in the pre-training stage.
Through extensive experimentation, we demonstrate the efficacy of FOLK. Our findings indicate that FOLK performs par better than many offtt-- MIM and MFM techniques in various downstream tasks, including image classification, few-shot learning, and semantic segmentation.
2 RELATED WORK
2.1 SELF-SUPERVISED LEARNING
Self-supervised Learning (SSL) methods have been developed to exploit large-scale unlabeled data for learning discriminative representations, which can then benefit variety of downstream tasks
Chong et al., 2023). Early SSL approaches rely on several pretext tasks, such as rotation prediction
Gidaris et al. 2018), . jigsaw puzzle (Noroozi & Favaro (016), and colorization (Zhang al. 2016
A branch of more recent studies follows a contrastive SSL paradigm (Chen et 2l.[ c Ci et al.
2022 Chen et al. 2020d Tian et al., 2020b Perera al. 2024). SimCLR (Chen et al. 2020b e
Monseti et al. [2024b] Zhou et al.) 2024) considers two augmentations of given image as positives and the augmentations of all other images the batch as negatives, on top of which a contrastive
loss is utilized for model learning. MOCO (He et al. 2020 Chen et al.) 2020d) maintains a dynamic dictionary of encoded representations with a coomentum-updated encoder to generate consistent em- beddings for contrastive learning. Instead of relying negative samples and large training batches, BYOL Grill et al., 2020) adopts a teacher-student framework that enforces consistency between representations of two augmented views of the same image generated by the two models. More recently, Correlational Image Modeling (CIM) (Li et al. 2023) operates by predicting correlation maps between randomly cropped image regions (exemplars) from a given image (context). It em- ploys a bootstrap learning architecture with online and target encoders and a simple cross-attention mechanism to process the exemplars and context.
2.2 MASKED IMAGE MODELING
Masked Image Modeling (MIM) is an exciting approach to self-supervised visual learning. The central concept is to rebuild or reconstruct the hidden parts of images or to predict general char-
acteristics like the image's s category (Bao et al. 2021 Zhou et al. 2022, Chen et al. 2020al Xie et al. 2022 Wei et al. 2022). It draws inspiration from the success of masked language modeling
in natural language processing (Devlin et al. 2018 Liu et al., 2019). adapting the concept to the visual domain. Specifically, BETT (Bao et al.
2021 utilizes discrete pre-trained Variational Au- toEncoder (VAE) to create discrete tokens for mmge patches, then tasks the model with predicting such discrete tokens of masked patches in images. The iBOT method (Zhou et al. 2022) offers
3

---

Published as a conference paper at ICLR 2025
improvements BEiT by adopting teacher-student self-distillation strategy where the teacher model simultaneously serves as an online tokenizer, instead of using a pre-trained discrete VAE. MAE (He et al., 2022a) takes more aggressive masking strategy, with typically around 75% of patches being masked, and recovers the missing pixels using an autoencoder. To further investigate which parts of an image should be masked for efficient self-supervisory. AttMask (Kakogeorgiou et al. 2022) proposes the utilization of an attention map generated by a teacher model, and masks the highly attended patches from the image for the student model learning.
2.3 DISTILLATION-BASED MODELING
Knowledge Distillation (KD) in general endeavors to transfer knowledge from complex, teacher
model to simpler, student counterpart (Buciluă et 2006
Hinton al. 2013). This is often
achieved by aligning the network logits (Hinton et al. 2015
Zhou et al. 2021) or intermediate
representations (Romero et af., 2014). as well as designated statistics between teach and student models (Tian et al. 2020a Ahn et al. 2019; He et al., 2022b Chen et al. 2021). In the self- supervised domain, SEED Fang et al., 2021 innovates by training a student encoder to mirror the similarity score distribution inferred by larger, pre-trained teacher across spectrum of instances. The EMA teacher, adopted by numerous SSL methodologies (Grill et al. 2020 He et al. 2020 Zhou et al. 2022; Caron al.) (202T), leverages the benefits of knowledge distillation to foster stabilized training and improved model efficacy. Our method extends the single model approach used by MFM
Xie al. 2023) teacher-student distillation strategy for robust visual representation learning.
3 METHOD
3.1 PRELIMINARY AND BACKGROUND
In the domain of supe-suue learning for visual models, MFM (Xie et al. 2023) introduces a novel approach that diverges from traditional spatial domain masking strategies. By leveraging the frequency domain, which encapsulates both high-frequency details and low-frequency elements, MFM bases its learning process on the masking of frequency components and the prediction of the masked frequencies. More specifically, given single-channel image E HxW the frequency representation is obtained via 2D Fast Fourier transform (FFT) F(x):
H-IW- Σ
F(x)(u, Σ
22 ( #7),
where æ(h, w)) represents the pixel value the spatial coordinate (h, w) the image, while F(x)(u, v) denotes the complex frequency value at the coordinate (u, v) on the spectrum. Here, is Euler's number, and is the imaginary unit.
To mask some frequencies from the spectrum and task a model with the reconstruction of these missing frequencies, frequency-masee image ă is first obtained by:
ã FFF(
where M E {0, Hx is mask, with 0 denoting corresponding frequencies being masked and
1 denoting frequencies being retained. F-1 denotes the inverse Fourier transform operation, and
O signifies element-wise multiplication. The learning objective of MFM, designed to minimize the discrepancy between the reconstructed and the original frequency components, can then be written
CMPM = [1 -(go())) o (1 1MM]22
(3)
For RGB images, the procedure is applied each channel independently.
4

---

Published as a conference paper at ICLR 2025
Informed Filtering
M
F(a) 0 M
F())
Com
IFFT
FFT
M
F(x) o M
IFFT
RCom
Random Selection
The frequency-basee filtering process with our proposed informed filters, Com and RCom. A portion of frequencies with the highest magnitude will be determined to create the masks. The process randomly returns the resulting image from the Com or RCom filter.
Figure 2:
where F(x) is the frequency spectrum of the original image, (z) is the reconstructed spectrum using neural network model ge. parameterized by 0. And - M indicates that only the masked areas of the frequency spectrum are considered for loss.
3.2 FOLK FRAMEWORK
Before introducing our proposed method, we start by re-emphasizing the MFM method (Xie et al., 2023) limitations. Firstly, notice that, in Eq. B] the MFM's loss depends on the filter M applied to the spectrum. However, the low/high-pass filters used in MFM are simple, using circular area with a fixed radius (see Fig. [].. This can lessen the difficulty of the frequency reconstruction task hence hampering the model learning. Another limitation of MFM is that the model only sees frequency- masked images the pre-training stage, expressed by go(ā) in Eq. 3 As a result, the pre-trained model may be relatively unfamiliar with natural images and requires more data during fine-tuning to adapt effectively (see Section 4.2.2p.
To overcome these limitations and achieve effective masked frequency modeling, propose the FOLK framework. Our key ideas involve the creation informed frequency- based filters, Com and RCom, well as self-distillation strategy based teacher-student design.
3.2. INFORMED FILTERS
Successful vision pre-training largely depends on the suitable and challenging enough pretext task presented to the model. Demonstrated by AttMask (Kakogeorgiou al.] (022). masking the most- attended patches in images creates a more effective training scheme than random masking for MIM approaches. However, for MFM methods, this gap still exists as only constant masking/filters have been explored (Xie et al. 2023), which presumably presents less challenging task in the pre- training.
To bridge this gap, we introduce two types of filters, Com and RCom, for informed masking. Inspired by Fourier image compression techniques (Pratt et al. 1969). where the most significant frequencies (those with the highest magnitudes) that carry the bulk of an image's visual informa- tion are preserved while other frequencies are discarded for efficient storage or bandwidth usage, our Com filters selectively retain these significant frequencies and discard the rest. This approach highlights the main semantics of an image for the model and necessitates the reconstruction of the less significant frequencies that correspond finer details, such as edges. Conversely, RCom filters remove the significant frequencies and require their reconstruction from the finer details preserved by the less significant frequencies. By applying both filter types during pre-training, the model is effectively trained to comprehend both macro and micro visual cues, thereby enhancing its general- izability and effectiveness in downstream tasks.
The generation of Com and RCom filters illustrated in Fig. 2 with the pseudocode available in Appendix A.4 The input image is first converted grayscale to ensure the creation of a single
5

---

Published as a conference paper at ICLR 2025
Student
Patch
hg
FFT
* MFM
Informed Filtering
ū,
U
Input
go
hg
CCS
Two
Views
EMA
L dis
Teacher
Stop Gradient
h
9Φ
U, V
CLS
Figure 3: The Proposed FOLK Framework. Two views (u and v) the input image are processed through the informed filtering process, introduced in Figure 2 This yields two trequency-masker views (ū and ū) which serve as the input to the student model. The student model is tasked with reconstructing the missing frequencies from the masked view (or ū), as well as reconstructing the feature representation of the other original view e (or u), generated by the teacher model, using the masked view (or 8). The student model go and the teacher model 9s, with their corresponding heads he and hg. have the same architecture but different parameters, except that the student has an additional MFM head hg. Only the student model (with its both heads) is updated through back- propagation, while the teacher parameters are periodically updated with an exponential moving average (EMA) of the corresponding student parameters.
common filter for all three RGB channels, as generating and applying filters separately to each chan- nel can result in unnatural and corrupted visual information. The image converted a frequency spectrum using 2D FFT, after which the frequency components with the highest magnitudes are iden- tified based on threshold uniformly sampled from a predefined set of values ([0.005, 0.01, 0.05] in our experiments, detailed in Appendix Section B.5.I). The informed filters are then created to retain (Com) or mask (Rcom) these frequencies. The two filters are randomly selected with an equal possibility (i.e. 50%) and applied to the spectrum (effects of different selection probabilities are provided in Appendix Section [B5.2.. By applying inverse FFT to the filtered spectrum, we image which will serve
restore a frequency-maskee important to note that Com and RCom filters are uniquely generated based on individual im- ages (examples are provided in Appendix D). which introduces more significant variation in training samples and presents a more challenging training task compared to using constant filters. In addition to comparing approach with the low/high- low/high-passed filters used in MFM (Xie et al. 2023), also conducted an ablation study using a set of random frequency filters to demonstrate the effectiveness of our proposed informed filters (refer Appendix B.5.4).
input the model during pre-training.
3.2.2 MAKING BACKBONE FAMILIAR WITH NATURAL IMAGES
To further improve the training efficiency and model robustness in masked frequency modeling, we incorporate a self-distillation design (Grill al., 2020 Caron al.) 2021 Tarvainen & Valpola 2017) in our proposed approach. The original MFM (Xie et al. 2023) method only tasked the model with the reconstruction of missing frequencies from the rrequency-maskee an approach potentially overlooked the data distribution of the original image space, as the model only sees frequency-masked images in pre-training, resulting higher data demands to adapt to naturally looking images during fine-tuning. To resolve this issue, we propose to properly inject the original image information into the training process via a self-distillation technique (see Section 2.3 for more information about self-di aef-distillation).. Here, present FOLK framework, detailed in Fig. 3 Note that FOLK does not require additional training stages for pre- preeprocsssing such as the offline tokenizer employed by BETT (Bao et al. 2021)
view of the image. Such

---

Published as a conference paper at ICLR 2025
FOLK starts by generating two views, u, and from an imput image following the same data augmentations adopted by DINO (Caron et al. (021). After applying 2D FFT to the view u (or v), the Com and RCom filters are uniquely generated according to this view (detailed in Section 3.2.1 and Fig. 2) One filter is then randomly selected and applied to the frequency spectrum, and the retained frequency components are processed through inverse FFT to restore a frequency-masked view ū (or ū) and predicts against two targets: reconstruction of the missing frequencies discarded by the filter, and reconstruction of the feature representation of the other original view u (or u) generated by the teacher model.
view u (or ü), see Equation 2 The student model receives this frequency-
Two different heads are appended to the student model, each facilitating one of the prediction tasks. The MFM head he serves to reconstruct the missing frequencies, with a single linear layer imple- mentation following the original design proposed in MFM (Xie et al. 2023). The student head he (and teacher head ho), aiming at the reconstruction of the feature representation of the other orig- inal view, adopts a three-layer multi-layer perceptron (MLP) design. Moreover, the student head (and teacher head) is followed by scaled softmax function to transform the outputs into probability distributions for the computation of a distillation loss (described below). More specifically,
Pz(z)(0- exp(fa(x)() (1) Σ cxp(fo(r)(M/T)))
(4)
where fo(x) he (1)) K is the output dimension of ht and T, > O is a temperature parameter. A similar formula holds for the teacher counterpart with Pt and T.. To sharpen the output distribu- tions passsuuaalllllll for the teacher we employ scaled softmax function alongside the centering technique introduced in Caron et al. (202T). This combination helps prevent model collapse and reduces the dependency on large batch sizes during training. Detailed descriptions of the heads provided in Appendix A.33
The intended revealing of unmasked original image information is effectively achieved through FOLK's teacher-student design. During pre-training, the teacher model sees naturally looking im- ages, which better align with those encountered during the fine-tuning stage, thereby enhancing fine-tuning efficiency, particularly in few-shot learning scenarios. On the other hand, the student model only observes masked views but is guided by the teacher model using the following distilla- tion loss. Both the student and teacher models go and Ja. with their heads he and hg, share the same architecture and initialization but have distinet parameters during training. Only the student model Ge and its both heads he and he are updated by the loss back-propase proek-rpppggtionn while the teacher pa- rameters periodically updated with an exponential moving average (EMA) of the corresponding student parameters.
A distillation loss is employed to enforce the less-informed student model to emulate the more knowledgeabl teacher model who perceives the original views. For a single input image æ, this loss can be written as:
Cais = - (P((w) log (P)(5) + P.(v) log (P)).
(5)
where u, are two different views of x, and ū, ō are the corresponding frequency-masked views. P. and P. are the student and teacher output probability distributions introduced in Eq. The EMA update to the teacher is then ruled by + \ø + +1 Λ)e, ensuring a gradual integration of knowledge over time.
Note that, same as MFM (Xic et al. 2023), FOLK features compatibility with both ViT and CNN- based architectures. Illustrated Fig. 3] when using a ViT-based model as the student (and teacher), the patch tokens out of the final encoder layer are fed into the MFM head, whereas the class token [CLS] is directed to the student (and teacher) head. Conversely, when using a CNN-based model, the framework utilizes the final feature maps out of the CNN encoder the input for the MFM head. An average-pout pooled feature map from the original feature maps will serve as the input to the student (and teacher) head. We provide experiments and results using CNN model (i.e. ResNet-50 (He et al.) 2016)) in Appendix B.2, Hence, FOLK facilitates a consistent approach across different architectural paradigms.
7

---

Published as a conference paper at ICLR 2025
3.2.3 COMPREHENSIVE LOss CALCULATION
Finally, comprehensive loss can be derived by integrating the two primary loss components:
Lus = a Lais + LMPM-
(6)
where a hyperparameter controls the weights between two loss terms, which is experiments, unless stated otherwise. Note that for a single input image, LMFM takes an average terms for the different views. This comprehensive loss facilitates the simultaneous model learning on the two tasks introduced above, the masked frequencies reconstruction (through (MFM) and original image feature reconstruction with self-distillation (through Lai)). Furthermore, an ablation study for different choices of the hyperparameter is provided in Section B.5.3
in our
3.2.4 COMPARISONS WITH OTHER SELF-DISTILLATION BASED METHODS
Though FOLK deliberately leverages self-distillation to overcome the limitations MFM - par- ticularly its inability to expose the model to natural images it is also important to clarify FOLK's distinctions from other self-distillation-aaee
methods such as DINO (Caron et al., 2021). AttMask (Kakogeorgiou et al. 2022), and iBOT (Zhou et al. 2022). In contrast to the spatial domain aug- mentation paradigms adopted by these methods (e.g. random cropping and masking on the original images), FOLK performs augmentation in the frequency domain by masking with adaptive filters and presenting rrequency-masker images to the student model to enhance its visual understandings. Note that the input the teacher model is only spatially augmented (i.e. random cropping), thereby preserving the model's perception of natural images during pretraining. Moreover, FOLK incor- porates combination of both reconstructive and comparative losses, presenting rational yet more challenging pretraining tasks compared to approaches that rely solely on comparative losses, such as DINO and iBOT. Hence, we do not consider FOLK close replication of previously presented methods. Please refer to Appendix Section C for more detailed comparison.
4 EXPERIMENTS
In this section, we detail the experimental setup and evaluate proposed FOLK framework classification tasks using both full fine-tuning and few-shot learning approaches. Additional experi- ments, including semantic segmentation downstream task and ablation studies, are provided in Appendix B for further insights and comprehensive results.
4.1 SETUP
In this study, we employ three well-established model architectures the foundation for our experi-
ments: the ViT-Small (ViT-S/16), ViT-Base VVTTBB/1yDDoooviiskii et al. 2021 and the ResNet-50
(He al. 2016) (For ResNet-50 results see Appendix Section [B.2). These models are chosen for their proven effectiveness and versatility, showing the power of our model with different types of
model architecture. We adopt the ImageNet- 1 training dataset (Deng al.; 2009) without labels for pre-training our self-supervised learning.
Our model's s performance is evaluated across two critical areas: image classification and semantic segmentation. For image classification, we continue to leverage the ImageNet-1K IK dataset (Deng) ctt al., 2009) to assess the generalizability and effectiveness of the learned features. contrast, for semantic segmentation, we utilize the ADE20K dataset (Zhou al. 2017), standard benchmark in scene parsing and segmentation tasks. This bifurcated approach to evaluation ensures a thorough analysis of the models' capabilities in varied contexts. Our computational infrastructure supports these extensive experiments, consisting of four nodes, each which has four NVIDIA A100 80GB GPUs, in total 16 GPUs. Please see Appendix for implementation details.
8

---

Published as a conference paper at ICLR 2025
Method
Ref
Data
Epoch
Token
ViT-S
ViT-B
Scratch
Touvron al.
2021
79.9
81.8
MAE
(He et al. 2022a)
INet-IK
300
80.6
82,9
iBOT
Zhou et al.
2022
INet-1K
1600
81.1
84.0
INet-1K
BETT
(Bao et al.
2021)
300
81.3
82.9
+ DALLLE
AttMask
Kakogeorgiou et al.
2022
IN
300
81.3
N/A
MoCo V3
Ct al. (0222)
INet-1K
600
81.4
83.2
DINO
Caron al.
202
INet-1K
1600
81.5
82.8
MFM
Xie et al.
2023)
INet-1K 1K
300
81.6
83.1
MFM
Xie et
2023)
IN
300
81.2
82.9
MFM
Xie et al.
2023
NNet-IK
300
81.4
83.2
R/Com"
FOLK
Ours
INet-K
300
81.6
83,4
FOLK
Ours
INet-IK
800
82.1
84.0
Table 1: Top-1 results of fine-tuning self-supervised approaches utilizing ViT-S/16 and ViT-B/16 as encoders for INet-1K (ImageNet-1K)) All recorded data were resized to images of size 224 x 224. Data means the pre-training dataset, and token means methods that need a masked token. "Our reproduced results with MFM official code through pre-training phase of 300 epochs followed by 200 epochs of full fine-tume toninntnng.g...nn Also, MFM R/Com means using the MFM approach with our proposed filters instead of its original low/high-pass filters. Reported in AttMask paper with 300 pre-training epochs. iBOT ViT-B is pre-trained on 1600 epochs based on their report.
4.2
EXPERIMENTAL ANALYSIS
4.2.1 IMAGE CLASSIFICATION
In this section, we focus on the fine-tuning capabilities of different vision pre-training techniques using the ViT-S/16 and VYT-B/16 encoders on the ImageNet-Ie 1K dataset. The motivation behind this analysis stems from the need to understand how different SSL strategies, which range from tradi- tional methods to novel approaches like FOLK method introduced here, perform under uniform testing conditions with the well-established ImageNet benchmark dataset (Deng et al. 2009).
Table presents the top-1 accuracy results of various SSL approaches fine-tuned on ImageNet- IK using both ViT-S/16 and ViT-B/16 encoders. Methods utilizing masked token strategies, such as MAE, BEiT, iBOT, and AttMask, consistently outperform the baseline, achieving up to 81.3% with VIT-S and 84.0% with ViT-B with huge number of pre-training epochs. These improvements highlight the effectiveness of masked token approaches in enhancing model performance, particu- larly when extensive pre-training epochs additional data (e.g.. DALL-E (Ramesh al. 2021 for BEiT) are employed.
However, our proposed FOLK method demonstrates superior performance without relying kens or external datasets. With only 300 epochs of pre-training, FOLK achieves 81.6% accuracy with ViT-S and 83.4% with ViT-B, matching or, surpassing other methods that require more epochs or additional data. Extending the pre-training to 800 epochs, FOLK further improves the ViT-S and ViT-B accuracy 82.1% and 84.0%, surpassing all other methods in the VYT-S and VIT-B category under similar conditions or lower number of pre-training epochs. These results showcase FOLK's ability to enhance learning efficiency and model efficacy by integrating learning from dual and original external data sources.
the complexities of token masking or the need for
4.2.2 2 FEW SHOT LEARNING
Our few-shot learning experiment's 's motivation lies in demonstrating the robustness and efficiency of FOLK framework compared to other pre-training methodologies, especially in scenarios char- acterized by limited data availability. In this context, we particularly challenge the efficacy of the FOLK model against the MFM approach and others under the premise that showing original im-

---

Published as a conference paper at ICLR 2025
ages (solving the second weakness of MFM by applying KD) significantly enhances performance on few-shot learning tasks.
In this experiment, aim highlight FOLK's superior adaptability and efficiency by fine-tuning pre-trained models using only 10% of the ImageNet- IK dataset over 200 epochs. This setup allows us eritically assess the influence of learning rate adjustments performance under sparse data conditions. We explore variations in base learning rates and warm-up periods using a cosine learning
rate strategy for optimization (Gotmare et al. 2019).
BLR = 2e-4
BLR 2e-4
BLR 2e-3
Method
AVG
MAX
Epoch
WUp
WUp 100
WUp 5
iBOT AttMask
64.0
71.1
2.0
45.7
71.1
800
69.8
71.0
31.3
57.4
71.0
300
MFM
57.7
58.5
41.9
52.7
58.5
300
MFM + Com/RCom
66.3
63.9
59.9
63.4
66.3
300
FOLK
71.2
68.1
62.2
67.2
71.2
300
FOLK
76.3
73.5
63.9
71.2
76.3
800
FOLK!
77.4
74.9
66.4
72.9
77.4
800
Table 2: Results of few-shot learning that fine-tunes the pre-trained ViT-S with different approaches for 200 epoch with 10% of labeled data from ImageNet-Ik, BLR: Base Learning Rate, means the peak value of the learning rate, and WUp: Warm Up, refers to the initial epochs during which the learning rate increases from 0 to the predefined BLR. After reaching the BLR, the learning rate then decreases according to a cosine function, from the BLR back down O. AVG: average. MAX: Maximum. 1000 fine-tune epochs.
The results in Table@higghlgee FOLK consistently outperforms methods like iBOT, AttMask, MFM, and MFM with Com/RCom filters across different base learning rates (BLR) and warm-up (WUp) settings. For instance, with BLR of 2 x 10-4 and no warm-up (WUp=0), FOLK achieves top-1 accuracy of 71.2%. surpassing iBOT's 14.0% AutMask's 's 69.8%, MFM's 57.7%, and MFM with Com/RCom's 's 66.3%. When the pre-training epochs are increased 800, FOLK's performance further improves, reaching 76.3% accuracy under the same BLR and WUp settings, and even achiev- ing 77.4% when fine-tuned for 1000 epochs (denoted FOLK!). The average accuracy across the three learning rate settings for FOLK with 800 pre-training epochs is 71.2%, significantly higher than iBOT's 45.7% with the same number of pre-training epochs. Notably, FOLK maintains ro- bust performance even when the learning rate is less optimal, such achieving 62.2% accuracy with a higher BLR x 10-3 and a short warm-up period (WUp=5), where other methods like iBOT drop drastically to 2.0% accuracy. These results demonstrate that FOLK not only excels in leveraging limited labeled data but also exhibits resilience to variations in fine-tuning hyperparam- eters, underscoring its effectiveness and adaptability compared to other state-of-the -hrtt-rrt methods in few-shot learning scenarios.
CONCLUSION
In this paper, we introduce the FOLK framework, novel SSL method that addresses the limitations of previous frequency-based pre-training approaches. By integrating Fourier transform compres- sion with self-knowledge distillation, FOLK adaptively selects frequencies for masking based on unique image responses, which allows the model to focus on more distinctive image features in the frequency domain, thereby enhancing the pre-training efficiency and model efficacy. Moreover, our dual-branch framework, which leverages both filtered and original images in pre-training, minimizes the adaptation requirements for natural-looking images in downstream tasks. Our experimental re- sults demonstrate the effectiveness of FOLK, achieving competitive performance compared to many leading SSL methods. Notably, FOLK excels tasks such as image classification, few-shot learn- ing, and semantic segmentation, all while requiring fewer pre-traiee pre-training epochs.
ACKNOWLEDGMENTTS
This project was made possible. part, by support from the National Science Foundation grant number OAC-2018627 and from the Ohio Supercomputer Center.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D I Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9163-9171, 2019.
Amani Almalki and Longin Jan Latecki. Self-supervised learning with masked autoencoders for teeth segmentation from intra-oral 3d scans. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 7820-7830, 2024.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0821 2021.
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineer- ing self-supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 335-541, 2006.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9999 2924,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang. Zhe Wang, Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7028-7036, 2021.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference machine learning, pp. 1691 1703. PMLR, 2020a.
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji Cf-vit: A general coarse-to- coossetto-fin method for vision transformer. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pp. 7042-7052, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong sumi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-2221 22255, 2020c.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020d.
Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng. Masked spectrogram prediction for self-supervised audio pre-training. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yuanzheng Ci, Chen Lin, Lei Bai, and Wanli Ouyang. Fast-moco: Boost momentum-batt con- trastive learning with combinatorial patches. In European Conference on Computer Vision, pp- 290-306. Springer, 2022.
Kevin Clark, Minh- Than-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2025
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. The Eleventh International Conference on Learning Representations, 2023.
Zhiyuan Fang, Jianfeng Wang. Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: sulf-supe distillation for visual representation. In International Conference on Learning Representations, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look deep learning heuristics: Learning rate restarts, warmup and distillation. In International Conference on Learning Representations, 2019.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Berardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21281 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022a.
Ruifei He, Shuyang Sun, Jihan Yang, Song Bai, and Xiaojuan Qi. Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transfcrability. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9161-9171, 2022b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rx3v 2015.
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What hide from your students: Attention-guided masked image modeling. In Europaan Conference Computer Vision, pp. 300-318. Springer, 2022.
J-K Kamarainen, Ville Kyrki, and Heikki Kalviainen. Invariance properties of gabor bilter-based deatures-overviee and applications. IEEE Transactions on image processing, 15(5):1088-1099 2006.
12

---

Published as a conference paper at ICLR 2025
Wei Li, Jiahao Xie, and Chen Change Loy. Correlational image modeling for self- self-supervised vi- sual pre-training In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15105-15115, 2023.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Ran Liu, Ellen L Zippi. Hadi Pouransari, Chris Sandino, Jingping Nie, Hanlin Goh, Erdrin Azemi, and Ali Moin. Frequency-awan masked autoencoders for multimodal pretraining on biosignals. arXiv preprint arXiv:2309.0592 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetrlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Amin Karimi Monsefi, Payam Karisani, Mengxi Zhou, Stacey Choi, Nathan Doble, Heng Ji, Srini- vasan Parthasarathy, and Rajiy Ramnath. Masked logonet: Fast and accurate 3d image analysis for medical domain. arXiv preprint 24Xx::2422006990 2024a.
Amin Karimi Monsefi, Kishore Prakash Sailaja, Ali Alilooee, Ser-Nam Lim, and Rajiv Ramnath. Detailelip: Detail-oriented clip for fine-grained tasks. arXiv preprint arXiv:2409.0663 06809, 2024b.
Pouyan Navard and Alper Yilmaz. A probabilistic- based drift correction module for visual inertial slams. arXiv preprint arXiv:2404.1014p 2024.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 107.0:1807 2018.
Alan V Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5):529 541, 1981.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Lear- ing robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Shehan Perera, Pouyan Navard, and Alper Yilmaz. Segformer3d: an efficient transformer for 3d medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4981-4988, 2024.
Leon N Piotrowski and Fergus W Campbell. A demonstration of the visual importance and flexibility of spatial- mratia-frequeecc amplitude and phase. Perception, 11(3):337-346, 1982.
William K Pratt, Julius Kane, and Harry C Andrews. Hadamard transform image coding. Proceed- ings of the IEEE, 57(1):58-66 1969.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 4rXiv:1412.6550, 2014.
Qing Su, Anton Netchaev, Hai Li, and Shihao Ji. Fisl: Feature-level self-supervised learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-average con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing sys- tems, 03:6827-68833 6839, 2020b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang. and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification. Advances in Neural Information Processing Systems, 33:2432-2444, 2020.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure timm. In NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
Tete Xiao, Yingcheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pp- 418 434, 2018.
Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Masked frequency modeling for sulf-supervised visual pre-training. The Eleventh International Con- ference Learning Representations, 2023.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In The Eleventh International Conference on Learning Representations, 2023.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF international conference on computer vision, pp- 6023-60333 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 649-666. Springer, 2016.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
Tianyi Zheng, Bo Li, Shuang Wu, Ben Wan, Guodong Mu, Shice Liu. Shouhong Ding, and Jia Wang. Mfae: Masked frequency autoencoders for domain generalization face anti-spoofing. IEEE Transactions Information Forensics and Security, 2024.
14

---

Published as a conference paper at ICLR 2025
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. Interna- tional Conference on Learning Representations, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022.
Mengxi Zhou, Yue Zhang, Amin Karimi Monsefi, Stacey S Choi, Nathan Doble, Srinivasan Parthasarathy, and Rajiv Ramnath. Reducing manual labeling requirements and improved reti- nal ganglion cell identification in 3d ao-oct volumes using semi-supervised learning. Biomedical Optics Express, 1588):440044555 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
The following appendices provide in-depth information to supplement our main findings and methodologies presented in the paper. These appendices aim to enrich the reader's understand- ing by providing comprehensive insights and evidence supporting the robustness and versatility of our proposed method.
Appendix Implementation details like pre-training, fine-tuning process, and projection heads architecture.
Additional experimental results and analysis like semantic segmentation, few- shot learning, image classification for CNN-based model, and ablation study.
Appendix
Appendix Distinctiveness of FOLK with other SSL based model.
Appendix
Visual prediction results.
A IMPLEMENTATION DETAILS
Throughout our experimental investigations, we adopted the methodologies prescribed by the iBOT Zhou et al. 2022 and MFM (Xic et al. 2023) frameworks. In mulli-GPUU training cireumstance, the learning rate adjustment is vital for optimizing the model's learning efficiency. The formulation used to compute the scaled learning rate is delineated below:
W WorldSize
SealedLR LRaseLR BalchSize.
(1)
512
In this context, LR signifies the optimal peak learning rate identified for the model's training process. The term BatchSize refers to the number of training examples processed simultaneously by each GPU. Lastly, WorldSize denotes the total count of GPUs employed for parallel computa- tion. The coefficient 512 in the denominator is normalization factor, ensuring the scaled learning rate maintains appropriate magnitude relative the hardware configuration. We used the PyTorch Library Paszke et al. 2019) for our code development.
A.1 PRE-TRAIN STAGE
Our pre- pre-training procedures largely align with those outlined in the BEiT (Bao et al., 2021) study, albeit with a few modifications. Specifically, our pre-training regime for the models incorporates simple yet effective data augmentation techniques, including random resized cropping a resolution of 224 x 224 pixels and image flipping.
We employ the AdamW optimizer (Loshchilov & Hutter 2019), with pre-training duration set to 300 or 800 epochs, a batch size of 2048, 128 per GPU, and a peak learning rate of 1.2 x 10-3 Additional parameters include cosine decay learning rate schedule, 20 warmup epochs, and specific setting for optimizer momentum (81, 82 = 0.9, 0.95) (Chen et al. 2020a) with a weight decay of 0.05. Also, we used value of 3.0 for gradient clipping to prevent the exploding gradient problem.
A.2 FINE-TUNE STAGE
For the fine-tuning stage, we tried to keep most of the configuration of our FOLK the same as MFM (ie et 2023) for fair comparison.
A.2.1 CLASSIFICATION TASK
We ran 200 epochs for fine-tuning the pre-trained model (i.e. ViT-S/16 or ViT-B/16) on ImageNet- IK for image classification, employing the AdamW optimizer across all configurations with weight decay of 0.05 and the optimizer momentum 3, 3, 0.9. 0.999. Moreover, the approach includes cosine decay learning rate schedule (Li & Arora 2020). with layer-wise learning rate
16

---

Published as a conference paper at ICLR 2025
decay equal to 0.8 (Bao et al., 2021: Clark al. 2020). We also utilized advanced augmenta- tion techniques such as Mixup Zhang et al. 2018) and Cutmix (Yun et al. 2019), as well as label smoothing and random augmentation to further improve model robustness and generalization capa- bility (Szegedy et al. 2016 Cubuk et al. 2020). The batch size is maintained at 2048, with a peak learning rate set at
10
In contrast, the fine-tuning settings for the ResNet-50 model (He et al., 2016) generally follow the configurations suggested by (Wightman al. 2021), with modifications to adopt the AdamW op- timizer as recommended by (Fang et al. (023). ). This includes binary cross-entropy loss function the learning rate scheduler. The weight decay is set to 0.02, and the batch size is set to 2048 to optimize performance. For ResNet-50, the fine-tuning epochs are specifically set to 300, with distinet configurations for repeated and random augmenta- tion, indicating tailored approach to maximize the model's i efficacy on the ImageNet- K challenge.
(Zhag & Sabuncu 2018) and adjustments
A.2.2 SEMANTIC SEGMENTATION TASK
We followed the pipeline demonstrated by the iBot (Zhou et al., 2022 paper for fine-tuning the pre-trained model for semantic segmentation using the ADE20K dataset (Zhou et al. (017). More specifically, we combined a pre-trained ViT-S/16 encoder with a UPerNet decoder (Xiao et al.) 2018). The ViT-S/16 encoder extracts detailed features from images, while the UPerNet decoder specializes in semantic segmentation, translating these features into precise pixel-level classifica- tions. This process employed the A optimizer and fine-tuned for 160K iterations.
A.3 PROJECTION HEAD
FOLK has three projection headers: one for frequency reconstruction (MFM Head, ha) and two for the student (ha) and teacher (ht) header, see 3 We used a single linear layer for the frequency reconstruction head, similar to that in the MFM method (Xie et al., 2023). However, when it came to the distillation heads (student and teacher heads), comprised a 3-layer multi-layer perceptron (MLP) with a hidden dimensionality of 2048. All layers were followed by a GELU activation except the final layer. We refrained from applying batch normalization (BN), VIT architectures, unlike standard CNNs, typically eschew BN by default. Also, the output dimension adopts a dimensionality of 8192.
A.4 PSEUDOCODE OF INFORMED FILTERS
Algorithm presents the pseudocode for our proposed Com and RCom filters (denoted as M in Eq- and Eq. Here, S is the set of predefined values (00.005, 0.01, 0.05] in experiments) from which a threshold is uniformly sampled to filter frequency components with the highest magnitudes. The parameter com prob represents the probability of applying the Com filter, set 50% in experiments; accordingly, the RCom filter is applied with probability (1 com_prob). Ablation studies evaluating different choices of and com-prob are provided in Appendix Sections B.5.1 and Section B.5.2 respectively.
Algorithm 1 Com and RCom Filter Generation
1: procedure if image is colored then Convert image to grayscale.
S, com prob)
2:
3:
4: end if
5: spectrum apply 2D FFT to image and retrieve the amplitude
6:
compression_rate ~ Uni (om(())(S) threshold +- the ((1 compression__rate)
7:
100)th percentile of spectrum 8: Com filter + (spectrum > threshold)
9:
RCom filter < Com filter
10: Return Com-filter with a possibility of com,proh, and RCom_filter with (1
prob.
11: end procedure
17

---

Published as a conference paper at ICLR 2025
B EXTRA EXPERIMENTS
B.1 SEMANTIC SEGMENTATION
Semantic segmentation is typical downstream task in the vision domain, where a classification needs to be performed on each pixel individually. We evaluated FOLK and compared it with several alternative SSL approaches on this task, using the ADE20K dataset (Zhou at. 2017) and incorpo- rating task layer from UPerNet as described by (Xiao et al. 2018) to the SSL pre- pre-trained encoder. The whole model was fine-tuned over 160k iterations, handling images at a resolution of 512 x 512, following the methodology established by iBOT (Zhou et al. 2022)
Method
mloU
Supervised Learning" (Zhou et al. 2022)
44.5
iBOT (Zhou al. 2022)
45.4
AutMask Kakogeorgiou et al.
2022
45.3
MFM (Xie et al. 2023)
44.9
MFM + Com/RCom (Xie et al. 2023)
45.1
FOLK
45.3
FOLK*
45.5
Table 3: The full fine-tuning ViT-S/16 model for semantic segmentation task with ADE20K dataset. Supervised Learning result taken from iBOT paper. We produced MFM results with their official code. FOLK was pre-trained with 300 and 800 epochs.
Results this fine-tuning effort for the semantic segmentation task on the ADE20K dataset are shown in Table 3 It presents comparative analysis of different methodologies: Supervised Learn- ing, iBOT, iBOT with Attention Mask (AttMask), MFM, and our FOLK model at different pre- training epochs (300 and 800 epochs). Notably, the FOLK model with 800 epochs of pre-training achieves the highest mloU at 45.5, slightly surpassing the iBOT's 45.4 mloU, This indicates suc- cessful adaptation of the FOLK methodology, showing not only an improvement over the MFM results but also demonstrating that extended pre-training can lead to marginal yet significant perfor- mance gains.
B.2 IMAGE CLASSIFICATION CNN BASE MODAL.
One limitation of many research studies, such iBOT Zhou et al. 2022) and AttMask (Kakoge- orgiou et al. 2022), is that they are only compatible with a specific type model architecture (c.g. ViTs), but not with others (e.g. CNNs). Our approach, like MFM (Xie al. 2023). does not have this limitation and works with a wide range of encoder models.
presen4s the ResNet-50 (He 2016) (a CNN-based model) performance under the FOLK framework. The same FOLK pre-training strategies that have been applied to ViT's were seamlessly adopted here for CNNs. The only necessary modification adopt CNNs to alter the inputs to the heads: replacing patch tokens from ViTs with reshaped feature maps from CNNs, and replacing the [CL.S] token from ViTs with the average-pookl (and reshaped) feature map from CNNs. Our approach has demonstrated equal or superior performance to other methods with fewer epochs.
In the experiments above, we observed that FOLK did not lead to significant gains over MFM when using CNNNbasee models in the standard many-shot (full-data) setting. However, upon further inves- tigation, we discovered that FOLK offers considerable advantages over MFM few-shot learning scenarios, even with CNN architectures. We conducted additional experiments using a ResNet-50 (He et al. 2016) model as the backbone for both FOLK and MFM, and evaluated their performance on ImageNet-I Ik with limited labeled data (1% and 10% subsets). The experimental setup mirrored that our ViT-based models to ensure a fair comparison.
As shown in Table 5 FOLK significantly outperforms MFM in the few-shot setting when using CNN backbone. Specifically, FOLK achieves a top-1 accuracy of 71.1% with 10% labeled data and 46.2% with 1% labeled data, compared to MFM's 53.9% and 22.7%. respectively. These results highlight FOLK's robustness and effectiveness in scenarios with limited labeled data, making it a superior choice for applications where labeled data is scarce.
18

---

Published as a conference paper at ICLR 2025
Method
Ref
Epoch
Top-1 Acc
SimSiam
Chen & He
202
400
79.1
MoCo v2
Chen et al.
2020d
400
79.6
SimCLR
Chen
2020b
800
79.9
BYOL
(Grill et al.
20200
400
80.0
SWAV
Caron al.
20220)
600
80.1
MFM
XIe ct al.
2023)
300
80.1
MFM + Com/RCom
(Xie et al.
2023
300
80.1
FOLK
Ours
300
80.1
Table The top-1 1 full fine-tuning accuracy on ImageNet- 1K for self-supervised models that utilize ResNet-50 the encoder. This information compares our methods against others, with the results of these comparative methods being sourced from (Xie et al.) 2023) and (Fang et al. 2023). The highest model performance highlighted in bold, with the second highest being underscored.
Model
Ref
Epoch
10%
1%
MFM
Xie et al.
2023)
300
53.9
22.7
MFM + Com/RCom
Xie et 2023)
300
58.6
31.0
FOLK
Ours
300
71.1
46.2
Table 5: Few-Shot for ResNet-50 with 300 epochs pre-training.
The performance gains can be attributed to FOLK's self-distillation mechanism, which allows the student model to learn from the teacher model's representations of unmasked images. This mech- anism provides additional guidance during pre-training, enabling the student model to better un- derstand the underlying data distribution, even when the amount of labeled data for fine-tuning limited. In contrast, MFM's reliance solely on reconstructing masked frequencies without exposure to unmasked images during pre-training may limit effectiveness low-data regimes.
In the full-data setting, the performance gap between FOLK and MFM using CNNs is less pro- nounced. This observation aligns with prior findings that CNNs, due to their inherent inductive biases and localized receptive fields, may not benefit as much from global context as VIT archi- tectures do. Nonetheless, the substantial improvements observed few-shot settings demonstrate FOLK'ss potential to enhance CNN-based models, especially when labeled data is limited.
These findings underscore the importance of evaluating self-supervised learning methods across dif- ferent architectures and data availability scenarios. By effectively incorporating the original image information through self-distillation, FOLK enhances the pre-training process for CNNs, leading to better generalization in downstream tasks with scarce labeled data.
B.3 ROBUSTNESS TO IMAGE NOISE AND ARTIFACTS
While our FOLK framework leverages the Fourier transform to create rrequency-masker augmen- tations, concerns may arise regarding the potential sensitivity of the Fourier transform to image artifacts and noise. It is well-known that the Fourier transform can amplify the impact of noise due its global nature (Oppenheim & Lim 1981). However, our method uses the Fourier transform primarily to generate diverse and challenging masked inputs, and the actual model training occurs in the spatial domain.
To assess the robustness of our model's learned representations of image noise and artifacts, we con- ducted an experiment evaluating the impact of common types image degradation on the model's performance in reconstructing masked frequencies. We randomly selected 100 images from the ImageNet- 1k validation set and applied following types of degradation each image:
Salp-and-Pepper Noise: Randomly replaces pixels with maximum minimum intensity values, simulating impulsive noise.
Gaussian Noise: Adds zero-mean Gaussian noise with a standard deviation of a -50 to simulate sensor noise.
19

---

Published as a conference paper at ICLR 2025
Brightness Variation: Adjust the brightness by random factor within +50% to simulate different lighting conditions.
Contrast Variation: Adjust the contrast by a random factor within +550% to mimic varia- tions in image quality.
For each original and degraded image, we applied the Com or RCom filters described in Sec- tion 3.2. to create frequency-masker versions. We then used our pre-trained FOLK model to predict the masked frequencies and calculated the MFM loss as defined Equation
Noise Type
Without Noise 0.179
Salt-and-Pepper 0.181
Gaussian
Brightness
Contrast 0.189
MFM Loss
0.182
0.186
Table 6: MFM loss values for original and degraded images. The slight increase loss for noisy images indicates minimal sensitivity noise. All the results of this table collected by ViT-S/16.
Table 6 presents the average MFM loss values for the original images and images subjected to var- ious types of degradations. The results indicate that these degradations cause only a slight increase in the MFM appr-xaproximaaell 0.002 to 0.01 higher than the loss for the original images. This marginal difference suggests that common image degradations do not significantly compromise our model's s ability reconstruct masked frequencies.
Several factors contribute to the robustness of our model to image artifacts and noise:
Spatial Domain Training: Although
the Fourier transform for masking, the input to the model remains the spatial domain. This approach mitigates the potential amplifi- cation of noise in the frequency domain.
Randomized Masking Parameters: The of randomly sampled thresholds for filter creation (as discussed in Section B.2.T) exposes the model to wide range of frequency masking scenarios. This diversity encourages the model focus on significant frequencies while reducing sensitivity to noise.
Our training pipeline includes standard data augmentations such as random cropping and flipping. These augmentations help the model learn robust features that generalize well to variations in the input data.
Data Augmentation:
The minimal impact of noise on the MFM loss indicates that our model effectively learns to recon- struct masked frequencies even when the input images contain artifacts. This robustness is crucial for practical applications where images may be degraded due sensor imperfections or environ- mental conditions.
B.4 FEW SHOT LEARNING
In addition to the primary few-shot learning results discussed in Section 4.2.2 Table 7 presents extended evaluation few-shot learning performance using smaller set of labeled data. Various pre-trained models were fine-tuned using only 1% of the ImageNet-1K dataset over 1000 epochs. This setup facilitates a detailed comparison of each model's ability to adapt to new data with min- imal examples, highlighting crucial aspect of model robustness and versatility. Furthermore, the evaluation considers three different settings for the base learning rate (LR) and warm-up periods, which are crucial hyperparameters in training deep learning models, especially under few-shot sce- narios. The different configuratio configurations aim to assess each model's 's robustness across varying learning rate adaptation conditions.
The performance data presented in Table 7 underscores the robustness of the FOLK method across various learning rates and warm-up settings, evidenced by its superior average performance of 42.8%. This consistency indicates FOLK's inherent stability and adaptability in few-shot learning scenarios, distinguishing from other models. Unlike iBOT and MFM, which exhibit fluctuating accuracies with changes in learning rates and warm-up periods, FOLK maintains high level of performance. This suggests that FOLK is less sensitive to hyperparameter adjustments, thus requir- ing less fine-tuning to achieve good results. This characteristic is particularly significant in practical
20

---

Published as a conference paper at ICLR 2025
BLR 2e-4
BLR 2e-4
BLR 2e-3
Method
AVG
MAX
Epoch
WUp 0
WUp 100
WUp 5
iBOT
33.2
59.0
1.4
31.2
59.0
800
AttMask
50.4
59.1
3.2
37.6
59.1
300
MFM
26.9
31.6
6.3
21.6
31.6
300
MFM Com/RCom
42.5
44.5
10.7
32.6
44.5
300
FOLK
51.7
56.1
20.5
42.8
56.1
300
FOLK
58.9
64.2
29.3
50.8
64.2
800
Table 7: Results of few-shot learning by fine-tuning pre-trained models for 1000 epochs on 1% of labeled ImageNet- Ik. All models were sourced directly from their respective original repositories. All the results of this table value the learning rate, and Wup: Warm Up. refers to the initial epochs during which the learning rate increases from 0 to the predefined BLR. After reaching the BLR, the learning rate then decreases according to a cosine function from the BLR back down to O. AVG: average. MAX: Maximum.
collected by VIT-S/16. BLR: Base Learning Rate, means the peak
applications where extensive parameter tuning is inapplicable. By effectively integrating dual in- puts filtered and original Images-FOL.K enhances feature extraction and generalization capabili- ties, resulting in more reliable performance across various settings. This robustness, combined with reduced dependency on precise parameter tuning, positions FOLK as attractive option for tasks demanding high accuracy with minimal labeled data and limited pre-processing.
B.5
ABLATION STUDY
B.5.1 RATIONALE BEHIND THRESHOLD SELECTION
Our primary goal in selecting multiple threshold values was introduce diversity in the frequency- masked input images during pre-training. By randomly sampling thresholds from a set (e.g.. [0.005, 0.01, 0.05)), we ensure that the model is exposed to variety of masking levels, which helps learn more robust and generalized representations. The thresholds correspond to the proportion of frequency components retained masked; for instance, threshold of 0.005 means retain (or mask) the top 0.5% of the highest magnitude frequencies. By varying this threshold, we create different levels of difficulty for the reconstruction task, encouraging the model to learn both coarse and fine-grane fine-grained features.
Threshold(s)
CIFAR-10
CIFAR-100
ImageNet-Ik
.05
98.8
90.2
81.1
99.0
90.7
81.4
.005
99.0
90.6
81.3
.005,.01.005
99.1
90.9
81.6
0002,.01,007
99.1
90.9
81.6
Table 8: Evaluation of Com and RCom filters across benchmarks using varied threshold proba- bilities. Optimal results are achieved with thresholds [0.005, 0.01, 0.05], proving the method's effectiveness across diverse datasets. ViT-S/16 collects all the results of this table.
While the initial selection of these thresholds might seem arbitrary, our experiments indicate that the exact values are not highly sensitive parameters. To demonstrate this, we conducted additional experiments using different sets of thresholds and evaluated their impact on downstream task perfor- mance. We compared the original set of thresholds [0.005, 0.01, 0.05) with an alternative set (0.002, 0.01, 0.07]. The results, presented in Table g show that both sets yield similar performance across various datasets (CIFAR-10, CIFAR-100 100, and ImageNet-Ike and both outperform using a single threshold value.
These results suggest that the method's effectiveness robust to the specific choice of thresholds long they sufficiently separated produce diverse frequency-masker formance gains from using multiple thresholds indicate that the model benefits from the increased variability in the masking patterns. In practice, select set of thresholds that produce visu-
images. The per-
21

---

Published as a conference paper at ICLR 2025
ally distinct trequency-maske masked images without extensive tuning. We found that thresholds leading to the retention or masking of approximately 0.2% to 7% of the highest magnitude frequencies work well. Visual inspection of the resulting images can help in determining appropriate thresholds, as shown in Figures [3 14 and 15 in the Appendix. By observing the images produced with different thresholds, practitioners taining important image structures and introducing sufficient masking to challenge the model during pre-training. The choice of multiple thresholds enhances the diversity of the training data without requiring extensive hyperparameter tuning. Our empirical results demonstrate that as long as the thresholds are spread out to produce varying levels of frequency masking, the model performance remains robust. This approach avoids the need for fine-grained tuning of threshold values and sim- plifies the practical application of our method.
ensure that the selected thresholds provide good balance between
In summary, while we initially selected the thresholds with the intention of creating diverse and challenging pre-training tasks, our experiments indicate that the exact values are not critical. The key is to have multiple thresholds that lead to different masking levels, and our findings show that the method is effective across various datasets with minimal sensitivity to these values.
B.5.2 EFFECT OF FILTER SELECTION PROBABILITY
To assess the impact of different selection probabilities for the Com and RCom filters in our FOLK framework, we conducted an ablation study by varying the probability of selecting each filter during pre-training. While our default setting randomly selects between the two filters with equal probabil- ity (50%), this experiment aims to determine whether this choice optimal and how sensitive our method is to this hyperparameter.
Com Probability
RCom Probability
Accuracy
0.1
0.9
81.3
0.3
0.7
81.4
0.5
0.5
81.6
0.7
0.3
81.4
0.9
0.1
81.4
Table 9: The impact of different selection probabilities for the Com and RCom filters on model accu- racy, ViT-S model, 300 epochs of pre-training, and 200 epochs of fine-tuning on the ImageNet- 1k dataset are adopted in these experiments.
The top-1 accuracy results on the ImageNet-1k 1k validation set are presented in Table 9 We observe that selecting the filters with equal probability (Pcom 0.5) yields the highest accuracy of 81.6%. Slight deviations from this equal probability result in marginal decreases in performance, with ac- curacies ranging from 81.3% to 81.4%. The performance differences are within 0.3% of the highest accuracy, indicating that our method is not highly sensitive to the exact selection probability of the filters.
These results suggest that while equal selection probability slightly outperforms other settings, the FOLK framework is robust to variations in the selection probability of the Com and RCom filters. The performance does not significantly degrade when the selection probabilities are skewed towards one filter over the other. This indicates that both filters contribute positively to the learning process, and the model benefits from the diversity introduced by both types of frequency masking.
B.5.3 WEIGHT OPTIMIZATION
We explore the optimal weight values for Lists introduced in Eq. 6 Table 10 provides an insight into how variations in the parameter influence the Top Accuracy of a ViT-S/16 model employing the FOLK methodology. As a is adjusted from 4 down to 0.05, a clear trend is observed where the model's accuracy improves notably when is reduced from 4 1, peaking at accuracy of 81.6% at a = 1. This suggests that a lower a enhances the model's performance, potentially indicating an optimal configuration of the loss function Lua at this point.
Further adjustments of a beyond this optimal point (decreasing it to 0.1 and 0.05) result in a slight decrease in accuracy to 81 34%, indicating plateau. This stability around lower values implies
22

---

Published as a conference paper at ICLR 2025
that while 1 is optimal, the model's performance does not degrade significantly with minor deviations from this value. The findings suggest that a critically influences the learning dynamics or loss function weighting, making its precise tuning essential for achieving the best performance from the ViT-S/16 model in the FOLK framework.
Param
a = 4
a = 3
a =
2
0,1
a 0.05
Acc
80.7
80.8
81.2
81.6
81.4
81.4
Table 10: Effect of values in Lax top 1 accuracy for ViT-S/16 model using FOLK methodol- ogy.
B.5.4 DIFFERENT FILTERS
Another part of our ablation study demonstrates the advantages of selective masking over random masking when applied to the frequency spectrum during model pre-training. The Com and RCom filters play crucial role in optimizing self-supervvsed learning by leveraging the principles of image compression. The Com filter focuses on major visual elements by preserving significant frequen- cies, thus compressing the image and emphasizing crucial features. Conversely, the RCom filter retains less dominant frequencies to highlight finer details and textures, enhancing the model's sen- sitivity subtle visual cues. This approach ensures that models are trained on both comprehensive and detailed representations, fostering adaptability and improved performance across diverse appli- cations.
Additionally, our masking approach generates unique masks according to each image's frequency responses, thereby accounting for each image's 's distinctive features and semanties (see examples in the Tables 13 14 and 15). This contrasts sharply with random masking. By targeting essential frequencies, selective masking ensures that the model adapts to recognize and prioritize these key signals, resulting in more robust and effective pre-training. Our findings indicate that this method significantly enhances the model's generalization capabilities and overall accuracy, confirming the efficacy of selective masking in the frequuncy domain for developing advanced predictive models.
Original
Com
Gabor
Token Mask
Circle Mask
Figure 4: Visual comparison between Com/ RCom filters employed by FOLK and other random masking techniques.
Figure illustrates the implementation of three supplementary random filtering techniques on the input frequencies and the effect of those filters compared with our Com filter. The Gabor filter that we directly applied in the frequency domain, (Kamarainen et al., 20066. In addition this, Token Mask is utilized, drawing inspiration from the Sim.MMM approach (Xie et al. 2022), where random square regions are masked in the frequency domain to mimic missing information. Furthermore, we introduce the Circle Mask strategy, which involves randomly masking circular areas at various distances from the center of the frequency spectrum. One the significant advan- tages of our introduced filter (Com/RCom) is that uses actual image information for masking, while other filters just randomly or constantly mask (like MFM) a portion the frequency area without considering the structure and information of the image.
23

---

Published as a conference paper at ICLR 2025
Filters Ace
Gabor
Token Mask
Circle Mask
Com/RCom
80.1
80.3
80.4
81.6
Table 11: Image classification results of utilizing different filters for masking in the FOLK framework. All the results of this table is collected by ViT-S/16.
*MFM Com/Rcom" "MFM" Loss Over Epochs
Com/Rcom
MFM
0.28
0.24
LoSs
0.22
0.20
0.18
20
80
Epochs
Figure 5: The loss progression of MFM using the original constant filters (denoted "MFM") and MFM using the proposed Com and Rcom filters (denoted as "MFM + Com/Rcom") for the first 100 epochs during pre-training.
Table illustrates the impact of various filtering techniques used for masking within the FOLK framework model accuracy. The methods compared include Gabor filters, Token Mask, and Cir- cle Mask filters. A clear pattern emerges from the data, with the Com/RCom filters significantly outperforming the other techniques, achieving the highest accuracy at 81.6%. This indicates perior efficacy Com/RCom filters capturing and utilizing relevant image features for model training. The other filters-Gabor, Token, and Circle-als also show competitive accuracies, but they are notably less effecctie than Com/RCom, with accuracies hovering around the 80% mark. This suggests that the design and application of the Com/RCom filters are better aligned with image intrinsic information, enhancing the model's ability to generalize from the training effectively.
B.5.5 EFFECT OF INFORMED FILTERS ON PRE-TRAINING
To further illustrate the impact of the proposed Com and RCom filters on the pre-training process, we present Fig. 5 comparison of the loss progression of MFM when utilizing the original con- stant filters versus the Com/RCom filters. Importantly, both experiments were conducted without the self-distillation design FOLK, thereby isolating the analysis to the effect of the filters alone. As depicted in the figure, using the proposed Com and Reom filters results in higher MFM recon- struction loss (Eq. 3) throughout the pre-training process. This indicates that the model finds it more challenging predict the masked frequencies produced by the Com/ RC//RCCm filters, suggesting that our filters are more effective in encouraging the model to learn richer representations by presenting more demanding reconstruction tasks.
24

---

Published as a conference paper at ICLR 2025
B.6 EFFICIENCY ANALYSIS
Our proposed enhancement involves augmenting the MFM model framework to accommodate dual inputs: both the original and the filtered images. By processing both types of inputs, the model gains a more comprehensive understanding of the data, which is particularly advantageous in scenarios requiring robust feature recognition, such as few-shot learning tasks. This approach aims to optimize the model's performance by leveraging the distinct characteristics captured in the filtered versus original images.
Methods
FOLK
MFM
iBOT
AttMask
MemGPU (GB)
19.8
10.1
21.9
39.3*
Few-Shot Accuracy (%)
67.2
52.7
45.7
57.4
Classification Accuracy (%)
81.6
81.4
81.1
81.3
Table 12: Comparison of GPU memory usage and model accuracy between FOLK and other meth- ods. All methods are based on a ViT-S backbone. MemGPU is GPU memory with a batch size of 128. Few-Shot Accuracy is averaged from Table Classification Accuracy comes from Table [ in the main manuscript. "We could not run AttMask's s official code with batch 128 with A100 80G and received memory error. Hence, ran it with batch 64.
To assess the effectiveness of our enhanced model, provide memory usage the GPU, and overall accuracy. Table summarizes performance metries of various self-supervised learning methods, highlighting the impact of integrating dual poputs-original and filtered images-on on model efficacy, particularly within the FOLK framework. Memory usage is crucial consideration, as only GPUs with high capacity AttMask which consume more memory while our model requires less than 20GB of memory for the same batch size. The increased memory usage in models like FOLK and iBOT correlates with the advanced processing capabilities necessary for handling dual inputs. However, the evident gains in learning accuracy justify the resource allocation, making a worthwhile trade-off for applications where high precision and robust feature recognition are critical, such as few-shot learning scenarios. FOLK achieves the highest few-shot learning accuracy at 67. 67.2% and tops classification accuracy at 81.6%. This performance indicates that the model's ability to effectively utilize both filtered and original images significantly enhances its learning capabilities, particularly tasks requiring robust feature recognition.
the model. This is noteworthy when compared to iBOT and
C COMPARISON WITH DINO AND DISTINCTIVENESS OF FOLK
While proposed FOLK framework and DINO (Caron al. 2021) both employ self-distillation techniques, it is crucial to clarify the key differences between the two methods and highlight the unique contributions of FOLK.
C.I USE OF SELE-DISTILLATION
Both FOLK and DINO utilize knowledge distillation in a self-supervised learning context involving student-teacher architecture where the student model learns from the teacher model's output. How- ever, knowledge distillation is common technique in self-supervised learning (SSL) and has been adopted in several works such as BYOL (Grill et al. 2020), iBOT (Zhou et al. 2022), and AttMask (Kakogooriiou et al. 2022). Therefore, using self-distillation alone does not define method's identity or make it equivalent to DINO.
C.2 ADDRESSING DIFFERENT PROBLEMS
FOLK integrates delf-distillation address specific limitation in Masked Frequency Modeling (MFM) (Xie et al. 2023). Namely, the model lacks looking at the natural view of images and un- masked images during pre-training This limitation can hinder the model's performance, especially in few-shot learning scenarios where labeled data is scarce. By incorporating self-distillation. FOLK
AttMass cannot with batch size of 128 A 100 80GB GPU, we it with batch size of 64.
25

---

Published as a conference paper at ICLR 2025
allows the student model, which only sees frequeney-maske images, to lear from the teacher model's 's representations unmasked images, thereby enhancing its ability to generalize to natural images.
C.3 FREQUENCY-BASED MASKING AND FILTERS
A key distinction of FOLK is the introduction of informed trequency-basee filters (Com and RCom) for masking in the frequency domain. These filters are designed adaptively mask signifi- cant frequencies based on each image's unique frequency components, creating more challenging pretext task and enhancing the model's ability to learn both macro and miero visual cues. This approach fundamentally differs from DINO, which operates in the spatial domain.
C.4 COMPUTATIONAL EFFICIENCY
FOLK is designed to be computationally efficient compared to DINO. Notably, FOLK requires fewer pre-training epochs to achieve competitive or superior performance (see Table instancce FOLK achieves strong results with 300 or 800 pre-training epochs, whereas DINO typically requires 800 1600 epochs reach even lower performance levels (Caron al. 202T..
Additionally, FOLK has lower GPU memory requirements. Using batch size of 128, FOLK's GPU usage approximately 19.8 GB (see B.6 while DINO's s GPU usage is around 30.8 GB (based on reported usage in (Caron et al. 2021) with adjusted batch sizes). This efficiency is partly due to FOLK not requiring additional local views images during training, unlike DINO.
C.5 METHOOOLLOGICAL DIFFERENCES
While both methods self seff-disiillation FOLK and DINO differ in their training methodologies:
Input Views: FOLK employs frequency-masked views generated using Com and RCom filters, whereas DINO uses multiple spatial views, including global and local crops. FOLK does not need any local view.
Loss Functions: FOLK combines reconstruction loss (MFM loss) for masked frequency prediction with a distillation loss, enabling the model to learn both from reconstructing missing frequencies and from the teacher's representations. DINO relies solely the distillation loss.
Exposure to Original Images: FOLK, the teacher model is exposed to unmasked im- ages, and the student learns to reconstruct the teacher's representations from frequency- masked images. This design helps the student model adapt to natural images despite not seeing them directly during training.
D PREDICTION VISUALIZATION
This section presents visualizations of series of images from the ImageNet- 1K dataset, processed by our newly proposed techniques, namely the Com and RCom filters. Tables 13, 14 and 15 col- lectively demonstrate the impact our proposed approach. Distinct from the MFM method, which employs static and conventional low/high-pass filters, Com and RC'om filters are dynamic and tai- lored to each image. This adaptability allows the filters to change in response to an image's unique concept and structural characteristics, offering more nuanced and effective processing method. In addition, Tables 14 and 15]also show the reconstructed missing frequencies for each image, gen- erated by the FOLK pre-trained model. The clear alignments between the model predicted and the ground-truth masked frequencies provide further evidence for successful pre-training of FOLK and, therefore the improved model efficacy.
26

---

Published as a conference paper at ICLR 2025
Original Image
Frequency
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
The effect of applying Com and RCom filters to different images, and the predicted missing frequencies the FOLK pre-trained model. Rate means the rate of compression (between 0 and 1).
Table 13:
27

---

Published as a conference paper at ICLR 2025
Original Image
Frequency
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
Table 14: The effect of applying Com and RCom filters to different images, and the predicted missing frequencies the FOLK pre-trained model. Rate means the rate of compression (between 0 and 1).
28

---

Published as a conference paper at ICLR 2025
Original Image
Frequency
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
Rate
Com Masked
Input
Predicted
RCom Masked
Input
Predicted
.05
.01
.005
The effect of applying Com and RCom filters to different images, and the predicted missing frequencies the FOLK pre-trained model. Rate means the rate of compression (between 0 and 1).
Table 15:
29


Reviews:
[{"review_id": "ORM7g1jqGV", "reviewer": "Reviewer_tJxA", "summary": "This paper proposes a frequency-based SSL method to learn visual representations from unlabeled images, which significantly improves the training performance compared with existing works. In particular, the authors built upon the MFM method and identified two key limitations: 1) pre-defined frequency masking filters that ignore the intrinsic structure in individual images; 2) model pre-trained with frequency-filtered images leads more data to adapt to natural images in downstream model fine-tuning. In response, two specific new designs (a. masked frequency modeling with Com and RCom filters; b. multi-task self-supervision with self-knowledge distillation) are proposed to target these two problems. Their reported experimental studies have shown the effectiveness of their designs.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**. The paper investigated two fundamental limitations in the MFM work and proposed two novel designs to address these limitations.  The presentation clearly shows what are the novel elements.\n\n**Quality**.  The paper shows a successful way to perform masking in the frequency domain for unlabeled training images. Additionally, the authors provided a proper self-knowledge distillation framework to deal with the negative effect of training with frequency-masked images.\n\n**Clarity**.  The submission neatly shows all the experiments that were carried out and the description of the underlying method is clear.\n\n**Significance and Relevance**.  The topic is very interesting and important. Considering the growing demand for learning effective representations from unlabeled data, this paper pushed the boundary of SSL.", "weaknesses": "**Training Cost**. Given that the proposed method employs a two-branch framework for model training, will it bring additional training costs compared with the original MFM?\n\n**Masking Filters**.  What are the exact formulations of Com and RCom masking? or pseudo code to construct Com and RCom might be helpful.\n\n**Data Augmentations**. In generating two views, u and v, distinct transformations (random cropping, color jittering, etc.) are conducted. It seems no ablation studies are provided for analyzing the effect on the consecutive image frequency masking and final model training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w2L8HmXAAO", "reviewer": "Reviewer_nBwS", "summary": "This paper introduces Fourier transform compression with self-knowledge distillation (FOLK), a frequency-based self-supervised learning (SSL) method designed to improve pre-training efficiency. FOLK addresses the limitations by adaptively selecting masked frequencies based on image frequency responses and employing a two-branch framework for knowledge distillation. Experimental results show that FOLK achieves competitive performance across various SSL tasks.", "questions": "1. Two views (u and v) of the input image are processed through the informed filtering process in the proposed FOLK framework. What is the optimal method for selecting views to enhance the model performance?\n2. How can the complexity of the FOLK be reduced to enhance the framework's accessibility? Could you analyze the computational costs of the various methods evaluated across different models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The framework is applicable and straightforward to understand.\n2. The proposed method improves the learning of the student model and facilitates a more efficient training process.\n3. The paper presents experiments across multiple datasets and various vision tasks, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The dual-stream and frequency-domain masking approaches applied in the article are relatively common schemes. Could the authors elaborate further on the motivation of the proposed method?\n2. More analysis and experiments are required on the framework design and cost computation, please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9kQZHtYKn", "reviewer": "Reviewer_DqRd", "summary": "The paper introduces a self-supervised learning (SSL) approach named FOLK, which stands for FOurier transform compression with seLf-Knowledge distillation. The method aims to address the limitations of previous frequency-based pre-training approaches by adaptively selecting frequencies for masking based on unique image responses. The dual-branch framework leverages both filtered and original images during pre-training, which is claimed to minimize the adaptation requirements for natural-looking images in downstream tasks. The experimental results demonstrate the effectiveness of FOLK, showing competitive performance in various downstream tasks such as image classification, few-shot learning, and semantic segmentation.", "questions": "1. The paper primarily focuses on the Com and RCom filters. It would be beneficial to see a comparison with other filtering techniques to establish the robustness and generalizability of the FOLK framework. Could the authors experiment with additional filtering methods, such as Gabor filters or wavelet transforms, and report on their effectiveness?\n2. The related work section mentions that MFM has been applied to low-level vision tasks. Since FOLK builds upon MFM, it would be valuable to include a comparison of FOLK's performance on such tasks. Could the authors add experiments that benchmark FOLK against existing methods on low-level vision tasks to provide a more comprehensive evaluation?\n3. While the supplementary material shows some results on robustness, a more detailed analysis would be appreciated. Could the authors provide additional benchmarks that specifically measure the robustness of the FOLK framework against various types of image degradations and noise?\n4. The paper integrates knowledge distillation into the FOLK framework, but ablation studies regarding the contribution of this component are missing. Could the authors conduct ablation studies to isolate the impact of the knowledge distillation component on the overall performance? This would help readers understand the significance of this technique in the context of the proposed framework.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents a new method that combines frequency-based masking with self-knowledge distillation, addressing known limitations in the field of SSL for computer vision tasks. The paper provides extensive experimental results that demonstrate FOLK's effectiveness across a range of tasks and benchmarks, showing improvements over existing state-of-the-art methods.", "weaknesses": "The author proposed two limitations in the introduction, but the experiments did not directly discuss how to address these limitations. Simply showing performance improvements (e.g., image classification tasks) is not enough to support the author's claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XIMUAML81D", "reviewer": "Reviewer_6EN5", "summary": "The authors identify two fundamental limitations in the existing masked frequency modeling (MFM) paradigm: 1) constant filters overlook the variability of image frequency responses, and 2) no access to naturally looking images during pre-training requires more data to adapt to downstream tasks during fine-tuning. To address 1), the authors adaptively select the masked-out frequencies based on image frequency responses. To address 2), the authors employ a student-teacher framework via self-distillation. Experimental results on image classification, few-shot learning, and semantic segmentation demonstrate the effectiveness of the proposed method compared to the MFM baseline.", "questions": "See the questions mentioned above. Overall, I think it is an interesting paper with extensive experiments and analysis, which could provide some new insights for the community. Thus, I am leaning to accept this paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed method is well motivated. The authors motivate the method by identifying two key limitations in MFM, and propose two interesting solutions to address these drawbacks.\n-\tThe paper is generally well-written and easy to follow.\n-\tThe authors provide a comprehensive analysis based on their method. The experiments are extensive and the results are promising, especially for few-shot settings.", "weaknesses": "-\tThe idea of using adaptive filers is interesting. However, the fitters still rely on some pre-defined thresholds, e.g., [0.005, 0.01, 0.05]. In practice, the authors may also need to tune these hyper-parameters to achieve the optimal performance for different datasets.\n-\tFor CNN, according to Table 4 in Sec. B.2, the proposed method does not lead to further gains compared with MFM when it comes to full fine-tuning, which makes me concerned about its effectiveness for CNN architectures. Could the authors provide the justification on this?\n-\tFor efficiency analysis, the authors only provide a comparison on GPU memory usage (Table 12, Sec. B.6). A comparison on training time with previous methods is also preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)