Masked Frequency Modeling for Self-Supervised Visual Pre-Training
Authors: Jiahao Xie
Wei Li
Xiaohang Zhan
Ziwei Liu
Yew-Soon Ong
Chen Change Loy
Venue: ICLR 2023 poster (2023)
Keywords: unsupervised learning
self-supervised learning
representation learning
masked frequency modeling

Abstract: We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models. Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain. Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy. Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token. Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches. Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach. Project page: https://www.mmlab-ntu.com/project/mfm/index.html.

Published as a conference paper at ICLR 2023
MASKED FREQUENCY MODELING FOR SELF-SUPERVISED VISUAL PRE-TRAINING
Jiahao X Wei , Xiaohang Zhan', Ziwei Liu Yew Soon Ong2.1 , Chen Change Loy SS-Lab, NTU 2SCE, NTU *CUHK *A*STAR, Singapore {jiahac003, wei.1, ziwei. .iu, asysons, celoy)entu. edu. sg x xlachangzenneotlook.ccom
ABSTRACT
We present Masked Frequency Modeling (MFM), unified frequency-domain- based approach for self-supervised pre-training of visual models. Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain. Specifically, MFM first masks out portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due the heavy spatial redundancy. Our findings sug- gest that with the right configuration of mask-aand-a and-predict strategy, both the struc- tural information within high-frequency components and the low-level statistics among oow-frqueecyy counterparts are useful in learning good representations. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non- Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token. Experimental re- sults image classification and semantic segmentation, well several ro- bustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches. Further- more, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from unified frequency perspec- tive and reveal their intriguing relations with our MFM approach. Project page: www.eiiwmiiiiiiiiiiemmimmmm .htll
https:// /ww. mmlab ab ntu. com/project/mmm/iiie
INTRODUCTION
Following the success of Masked Language Modeling (MLM) such as BERT (Devlin et al.) 2019 in natural language processing (NLP), Masked Image Modeling (MIM) (Bao et al. 2022 He et al.
2022
Wei et al.) 2022 Xie et al. 2022 has shown promising performance in self-supervised pre- training of visual models. Both MLM and MIM follow common corrupt-ane cattt---nd--didddid paradigm randomly masking a portion of input data and then learning to predict the missing parts. This simple
recipe enables modern Transformer-ban
deep architectures (Vaswani et al. 2017 Dosovitskiy al. 2020) to learn generalizable representations from ubiquitous unlabeled text or image data.
By default, current MIM methods such as BEiT (Bao et al.) 2022). MAE (He et al. 2022 and SimMIM (Xie et al., 2022) perform masking in the spatial domain by excluding image patches randomly, a strategy inspired by MIM that performs masking words (Figure [a-b)). However, unlike human- generated language that is succinet and highly semantic, raw pixel values the spatial domain are of low information density. To cope with heavy spatial redundancy in images, MAE (He) ct al. 2022) shows that one would need to mask very high proportion (e.g., 75%) to encourage the learning of meaningful features.
Beyond masking image patches, which is a particular way of corruption, in this paper, we are in- terested in investigating the effectiveness of other corruption strategies for self sule-super repre- sentation learning. We first explore the corruption recipes commonly applied in low-level image processing tasks, including image super-resolution (SR), deblurring and denoising. As shown in

---

Published as a conference paper at ICLR 2023
A white cat runs A cat
zuna
on green grass mesk on a green
mask
covered with small flowera. small
covered with
(a) MLM
(b) MIM
Image
image
FFT
A
IFFT
Trquuonc domain
traquancy domain
high-pass
Low-level image processing
MFM
Figure 1: Comparison of masking recipes in Masked Language Modeling (MLM), Masked Image Modeling (MIM), low-level image processing and Masked Frequency Modeling (MFM). Note the differences of masked information among MIM, low-level image processing and MFM.
Figure cc) the downsampling, blur, and noise operations can degrade the exemplar image effec- tively in the spatial domain, thus potentially serving as useful corrpption strategies. However, the corruption induced in the spatial domain prevents us from analyzing what specific information is corrupted and needs be reconstructed. To better understand these low-level corruptions, we shift our attention from the spatial image domain to the frequency domain.
In the frequency domain, one could observe underlying patterns of an image not conveniently visible from raw pixel values. For example, the downsampling and blur operations dominantly remove the figh-frequency image details, while adding noises tends to corrupt the full frequency spectrum of an image globally (Figure[[e))
Driven by this observation, we present simple and effective masking strategy in the frequency domain for seff-supervised visual representation learning, dubbed as Masked Frequency Modeling (MFM). Specifically, we first perform Fast Fourier Transform (FFT) to convert each input image into its frequency representation, i..,, frequency spectrum. We then mask a portion of frequencies on the frequency spectrum using a low-/high-pass filter. With inverse FFT (iFFT), we finally take the corrupted image with some of the frequencies attenuated as input. Our encoder is quite flexible as no mask tokens are inserted. Thus, MFM can embrace both the vision Transformer (ViT) (Doso- vitskiy al. 2020) and convolutional neural network (CNN) (LeCun al.) 1989) families. Our decoder a lightweight linear layer that reconstructs the masked frequency values on the frequency spectrum via frequency loss. As shown in Figure d)), image with low high frequencies attenuated would reveal entirely different patterns: the low-frequency components usually contain object smooth structure such as colors and styles, while the high-frequency counterparts largely depict the object outline or silhouette structure. Such unique properties of the frequency domain make appealing for reducing information redundancy, thus creating a nontrivial and meaningful self-sum self-supervisory task.
Our contributions are summarized follows:
1) We propose new masked frequency modeling task to pre-train visual encoders in self- supervised manner. Our MFM agnostic to the architectures, and we demonstrate the flexibility of applying MFM for both VIT and CNN families.
2) We contribute the first study of low-level corruption tasks for self-supervised learning (SSL) in frequency domain. We investigate the effectiveness of corruption strategies commonly adopted low-level image processing tasks (i.e., SR, deblurring and denoising) for SSL. from unified frequency perspective and reveal that the representation learning capability of these corruption tasks actually depends on the architectures: they can achieve comparable and even better results than their supervised counterpart on ViT, but gains observed on CNN.
3) Extensive experiments show that our MFM can achieve competitive performance among existing MIM approaches downstream tasks, such image classification and semantic segmentation, while not using mask tokens or other more complex designs. Further analysis on several robustness benchmarks also exhibits more appealing robustness of the studied corruption tasks than MIM.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Masked language modeling and its auto-regressive variants, such as BERT (Devlin et al. 2019) and GPT (Radford et al.; 2018 2019 Brown el al., 2020), have achieved great success pre- training large-scale language models in the NLP community. These approaches perform masking on the human-genn -gnneratd language by holding out random words and then predicting the missing content. This simple mask-word recipe has shown excellent ability in pre-training generalizable representations for broad NLP applications.
Masked image modeling leverages images corrupted by masking to learn useful representations.
Pioncered with stacked autoencoders (Vincent et al. 2010)) and context encoders (Pathak et al. 2016
using CNNs, recent approaches (Bao et al. 2022 He et al. 2022 Xie et al. 2022 Wei et al.
2022
Chen et al. 2022) follow the mask-word strategy in NLP to randomly mask image patches in the spatial domain using the vision Transformers (Dosovitskiy et al. 2020 Liu et al. 2021). Along with this mask-patch strategy, different types of prediction targets have been studied, including discrete tokens (Bao et al., 2022 Dong et al. 202T), raw pixels (He al. 2022 Xie et al.) 2022), and hand- crafted features (Wei et al. 2022). Besides, iGPT (Chen et al. (2220a) takes low-resolution image sequence as imput and predicts missing pixels in an auto-regressive manner. Several methods (Zhou iet al.] 22022 El-Nouby et al. [222]) also integrate MIM into contrastive- Siamese frameworks. Our work differs from previous approaches that we perform masking in the frequency domain, which relies on none of the following: (i) extra data (Bao et al., 2022] Dong et al.] 202 Fang
ct al.
2022), (ii) extra model (Zhou et al., 2022 El-Nouby et al. |021 Fang et al. 2022 | Shi et al.,
2022
Chen et al. 2022), or (iii) mask token (Bao et al. 2022 He et al. 2022 Xie et al. 2022 Wei 2022 Chen et al. 2022). CIM (Fang et al. 2022) also does not use mask token. However,
et al.
introducing auxiliary generator to corrupt the input images adds nontrivial pre- pre-training overhead. In contrast, our grruuency-domii--ee with negligible computational cost.
corruption strategy can achieve comparable performance
Self-supervised learning mainly focuses on designing effective pretext tasks for pre-training (Do-
eesch et al.] [013 Wang & Gupta 2015 Noroozi & Favaro 2016
Larsson et al. [2016] /Zhang al.
2016 2017c; Noroozi et al. [017] [oojaooos & Joulin, 2017 Pathak et al.J [007]] 7| Gidaris et al.
2018 : Contrastive learning Wu et al.
2018: He al. [020]] Misra & Maaten (2200) Chen al.
20206 c Grill et al. 2020: Chen & He 2021
Chen et al. [202]] Caron et al. (221) has dominated
the field over the past few years. Unlike the mask-and mask-and-predicr pretext task, contrastive learning
typically uses a Siamese framework and greatly relies on data augmentation.
Low-level image processing tasks, such as image super-resolution (Dong et al. 2015). deblur- ring (Zhang et al.] 2022) and denoising (Zhang et al., 2017b), focus on restoring the high-fidelity image from its corrupted input. The corrupted images are usually generated with degradation trans- formations, which consist of downsampling, blur, noise and JPEG compression. Recent promising results of MIM motivate us to investigate the effectiveness of these corruption operations in the context of representation learning.
Frequency domain analysis has been widely adopted in many computer vision tasks, such as image generation (Jiang et al. 2021). domain adaptation (Xu et al. 2021). and image super- resolution (Pang et al. 2020). Early studies (Oppenheim et al. 1979] Oppenheim & Lim 1981
Piotrowski & Campbell 1982 Hansen & Hess, (007) have revealed that in the frequency domain,
the phase component largely captures high-level semantics of the original signals, while the am- plitude component mainly retains low-level statisties. As such, underlying image patterns can be more conveniently observed in the frequency representation, compared with the raw pixel values in the spatial domain. Motivated by the intriguing properties of the Fourier domain, we propose novel mask-frequency recipe and conduct the first study w.r.t. masked information modeling in the frequency domain for image data.
APPROACH
Our masked frequency modeling (MFM) is a simple yet effective self-supervised pre-training ap- proach, which masks out portion of image frequency components and predicts the missing fre- quencies on the frequency spectrum. Figure[]show the overview of our approach. The framework consists of four components: masking strategy, encoder, decoder, and reconstruction target. We first.
3

---

Published as a conference paper at ICLR 2023
FFT
IFFT
FFT
encoder (VIT, CNN, etc.)
linear
head
mask frequencies
predict
Figure 2: Overview of our MFM pre-training pipeline. We convert each input image into fre- quency domain via FFT and mask a portion of frequencies on the frequency spectrum via low-pass (top) high-pass (bottom) filter. After iFFT, the low-lhigh-pass filtered spatial images are then ran- domly fed to the encoder (e.g., ViT, CNN), with a lightweight one-layer head to predict the masked frequency values on the frequency spectrum via frequency loss. The red circle denotes the selected mask radius, and the dice icon refers to the random sampling process of low-/high-pass filters, fol- lowing a Bernoulli distribution.
detail each component of MFM in Section 3.1 and then discuss the relation of our approach with
low-level image processing tasks in Section 3.2
3.1 MASKED FREQUENCY MODELING
Preliminary: Frequency representation of images. Given single channel image E #xw we can obtain the corresponding frequency representation via 2D Discrete Fourier Transform F (z):
H-1 W F (x) (1,
Σ
Σ x (h, w) 12a WW) W)
(1)
h=
===
where x (h, w) is the real pixel value at the coordinate of (h, w) on the spatial image, F (a:) (u, v) is the complex frequency value at the coordinate of (u, v) on the frequency spectrum, e and are Eu- ler's number and the imaginary unit, respectively. Accordingly. F-1 (xx) defines the inverse Fourier transform that maps spectral signals back into original image space. Both the Fourier transform and its inverse can be calculated efficiently using the FFT algorithm (Nussbaumer) 1981).
Masking strategy. We define mask M 1) 1) whose value is determined by thresh- olding function that separates the low and high frequency components from JF (z) according to a hyper-parameter, Lee, radius
1, if d ((u,)) (ch, <r M (u, v)
0, otherwise
where (Ch: Cwr) denotes the center of the image, d ((. denotes a certain distance criterion. Here, the Fuclidean distance, i.e., circle mask as default. Note that the mask shape is not solely restricted to a circle one, and we study the effects of different mask shapes in the experiment section.
With the predefined mask M, we can easily obtain the decomposed low-pass filtered image xy and the high-pass filtered counterpart Ty as follows:
x = = (F (x) ④ M), I (F)
(x)
M)),
where 1 is the all-ones matrix, O is the Hadamard product between matrices. These filtered images are then randomly selected with a Bernoulli distribution and fed to an encoder as inpu
MFM encoder. The architecture of encoder is quite flexible since do not insert any mask tokens on the corrupted non-overlapping patch embeddings as in MIM (Bao et al.) 2022 He et al. 2022 Xie et al. 2022. Wei et al. (0022). Therefore, our MFM can be applied on both ViT and CNN architectures without any special designs. In this paper, we mainly use a standard VIT (Dosovitskiy
et al. 2020) as our encoder for direct comparison with MIM methods. Specifically, we first divide
For RGB images, the procedure is operated on each channel independently. Despite performing masking in the frequency domain, still take the converted spatial images input such that model would not suffer input domain gap between pre-training and fine-tuning.

---

Published as a conference paper at ICLR 2023
filtered spatial image into regular non-overlapping patches. Then, the encoder embeds the patches by linear projection with added positional embeddings. The combined embeddings are then processed via a series of ceff-attention-based Transformer blocks /Vaswani et al. 2017). We also consider typical CNN architecture, i.e. ResNet-50 (He et al. 2016). to demonstrate the versatility of MFM. To this end, we simply send the filtered spatial image to the CNN encoder input.
MFM decoder. The decoder accomplishes the frequency reconstruction task. It can be of arbi- trary form as long as its input is compatible with the encoder's output. Here, we simply adopt lightweight linear layer as our decoder for efficiency, after which we perform FFT to convert each output image into the frequency domain for frequency reconstruction. The effect of different de- coders is further studied in Appendix A
Reconstruction target. Our MFM reconstructs the input by predicting the missing frequency values on the frequency spectrum. To faithfully recover the frequency values, we should define a frequency distance metric that considers both amplitude and phase as loss function. Regarding each fre- quency value F (I)) (u, v) as a wwo-dimensional Euclidean vector f. one can easily derive that the magnitude of the vector corresponds to the amplitude while the angle corresponds the phase. In- spired by Jiang et al. (2021), we thus define the frequency distance ' (:.)) as the distance between the reconstructed vector Sr and the original vector So at each spectrum coordinate (u, v):
D (T..To)) = 1 - [ Frr (x) (u Fo (a)) (u,
(4)
= ((R, (x) (u, u) R. o)) + (I, (u, To (x) (u, (u))))))?
where R(x) and [(x) are the real and imaginary part of F(x), respectively, y is an exponent to control the sharpness of the distance function and is set to by default. For each image, the final loss function, i.e., the average frequency distance of all spectrum positions can thus be written as:
H-1W-1
== D
(.) Fo (z,
(5)
HW
In practice, we compute the loss only on the masked area of the frequency spectrum instead of the full spectrum as the latter tends to decrease the accuracy according our experiments.
3.2 RELATION WITH LOW-LEVEL IMAGE PROCESSING TASKS
The notion of recovering masked frequency components in MFM is reminiscent to the objectives in low-level image processing tasks, such as image super-resolution (SR), deblurring and denoising. In these tasks, model takes degraded image as input, and the aim is to restore the missing compo- nents. Different degradations corrupt different components in the frequency domain. As discussed in Section 1 for the image SR and deblurring tasks, most of the high-frequency components are removed while the low-frequency counterparts are retained; for the image denoising task, both low- and high-free freeuereqeeeees are significantly altered.
By analyzing the frequency spectrum of these tasks, we can observe how different frequencies of an image contribute visual representation learning. thus gaining better insights on designing more effective learning objectives. Compared with these tasks, MFM provides a more general and unified frequency perspective to perform these low-level corruptions while being conceptually simpler: we directly remove certain frequencies the frequency spectrum via a low-/high-par iments show that MFM can achieve better performance than these tasks for representation learning. We will comprehensively study these tasks and show more details in the experiment section.
filter. Our exper-
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We use the vanilla ViT-Small (ViT-S/16), ViT-Base (VIT-B/16) and ResNet-50 models as the back- bones in our study. We perform self-supervised pre-training the ImageNet-IK (Deng et al.) 2009) training set without labels. For VřT, our pre-training setting generally follows BETT (Bao al. 2022). while we only use random resized cropping (224 x 224 resolution) and flipping as data
5

---

Published as a conference paper at ICLR 2023
Table 1: Ablations for MFM with ViT-B/16 on ImageNet-IK. All models are based on 300-epoch pre-training, and we report top-1 fine-tuning accuracy. Unless specified, the default settings are: the mask type is random (i.e., random sampling of low-/high-pant paaigg-psssss filters), the mask radius is 16, the mask shape is circle, the sampling ratio for low-pass filters is 50% (i.e., 50% for low-pass filters and 50% for high-pass counterparts), the reconstruction target is masked frequencies on the spectrum, and the loss function is frequency loss with 7 1. Default entry is marked in gray.
(a) Mask type. Random sampling
(b) Mask radius. Using a fixed ra- dius enough.
(c) Mask shape. circle mask is more accurate.
of both filters works the best.
Mask type
Top-1 acc (%)
Mask radius
Top-1 acc (%)
Mask shape
Top-1 acc (%)
none
76.5
8
82.8
cirele
83.1
16
83.1
square
82,9
low-pass
82.4
24
82.7
rhombus
82.8
high-pass
82.3
32
82.6
random
83.1
[8, 24]
83.0
(d) Sampling ratio. Sampling low-thigh-pass filters with an equal probability is effective.
(e) Reconstruction target. Predict- (f) Loss function. A frequency
ing only the masked frequencies yields better performance.
loss works better than spatial loss.
Loss
Top-1 1 acc (%)
Sampling ratio
Top-I acc (%)
freq. (7 1)
83.1
82.5
Reconstruction target
Top-1 1 acc (%)
freq. (7 = 2)
82.5
83.1
masked spectrum full spectrum
83.1
82.3
0.7
82.7
82.4
f2
82.2
augmentation, with dropout and stochastic depth not applied. We also do not use relative position or layer scaling. After pre-training, conduct supervised end-to-end finettuning ImageNet- K image classification and ADE20K (Zhou et al. (207)) semantic segmentation to evaluate the quality of learned representations, following BETT (Bao et al.] 2022). For ResNet-50, we adopt the same pre-training configuration as that in ViT without further parameter tuning. We provide the detailed pre-training and fine-tuning recipes in Appendix G
4.2 MAIN PROPERTIES
We start by ablating our MFM using ViT-B/16 as the default backbone. All experiments are con- ducted with 300-epoch pre-training and 100-epoch fine-tuning on the ImageNet-1K dataset unless otherwise specified. Several intriguing properties observed.
Masking strategy. We first study different masking strategies on the frequency spectrum. We consider two kinds of filters: low-pass filter (i.e., mask high frequencies), and high-pass filter (i.e., mask low frequencies). As shown Table Ia| masking and predicting either high frequencies ("low- pass" entry) or low frequencies ("high-pass' entry) perform significantly better than simply encoding and reconstructing the original image ("none" entry). This indicates that both high-frequency and low-frequency components
useful in representation learning, where the former largely depicts the object structure information such as outline or silhouette and the latter usually captures low-level statistics such colors and styles. A random variant, i.e., randomly selecting one filter from both low-pass and high-pass filters ("random" entry), benefits from all lens of frequencies, thus further improving the performance.
Mask radius. Table Ib] studies the effect of mask radius, which controls the difficulty of our task. larger radius leaves more frequencies for low-pass filter while removes more frequencies for high-pass filter. MFM works the best with moderate difficulty. Using fixed radius (e.g.. 16) performs slightly better than a random one, i.e., the radius is uniformly sampled within a range (e.g.. [8, 24).
Mask shape. We study three centrosymmetric mask shapes Table Ic focus on different masking directions. Take low-pass filter as example, square shape removes more frequencies in the horizontal and vertical direction, while a rhombus one removes more the diagonal direction. The results demonstrate that circle mask shape that pays equal attention to each direction on the frequency spectrum performs the best. We hypothesize that the effect of different mask shapes is largely correlated with the category statisties of pre-training datasets.
Different mask shapes

---

Published as a conference paper at ICLR 2023
Table Comparison of SR, deblurring, denoising and MFM tasks with ViT-B/16 on ImageNet- IK. All models are pre-trained for 300 epochs, and evaluated with top-1 fine-tuning accuracy. Cor- rupted image samples from ImageNet-Il 1K training set with different degradation levels are visual- ized in both image and frequency domain. The studied hyper-parameter that controls the difficulty of degradation for each task is (a) downsampling scale factor, (b) Gaussian blur sigma, (c) Gaussian noise sigma, and (d) mask radius, respectively. More examples are provided in Appendix H Zoom in for best view.
X8
x16
Parameter
Top-1 acc (9)
x2
82.1
x4
82.2
SR
82.4
x16
82.1
orggnnl (b)
79.7
81.2
(b) Deblur
81.7
25
75
100
81.5
82.4
50
82.6
Denoise
82.7
100
82.6
8
82.8
(0)
(d) MFM
24
82.7
32
82.6
Sampling ratio. Table TablleTdaallann Tdlatts different sampling ratios for low-/high-pase filters. Here, the sam- pling ratio denotes the probability of sampling low-pass filter, following Bernoulli distribution. The results show that simply sampling both filters with an equal probability works the best.
Reconstruction target. Table Ie compares two reconstruction targets: 1) predicting only the masked frequencies on the frequency spectrum as in our default setting, and 2) recovering both the masked and unmasked frequencies the frequency spectrum. Predicting the masked spectrum performs better than reconstructing the full spectrum by a clear margin (83.1 % vs. 82.4%). This suggests that predicting the invisible signals which is in accordance with the observation in recent MIM approaches.
favourable task in representation learning.
Loss function. Table T studies the design of loss functions. frequency loss (freq.) performs better than a spatial loss (E,, f2), with = 1 working the best. It makes sense as directly predicting the missing frequencies in the frequency domain better aligns our MFM task.
4.3 DIAGNOSIS LOW-LEVEL IMAGE PROCESSING TASKS
In this subsection, study the representation learning capability of low-level image processing tasks from a unified frequency perspective. We examine three representative tasks: image super- resolution (SR), deblurring, and denoising.
Setup. To ensure a direct comparison, we adopt the same pre-training and fine-tuning hyper- parameters MFM and only alter the types of image degradation during pre-training. Specifically, for the SR task, we first use its standard data pre-processing, i.e., bicubic downsampling, to down- sample the input images by a scale factor. We then upsample them back to the original input size, i.e., 224 x 224. For the deblurring task, we consider the commonly-used isotropic Gaussian filter and uniformly select the blur kernel size from {7,9, 9, 11, 13, 15, 17, 19, 21} as suggested in Wang et al. (021I. For the denoising task, we employ the typical Gaussian noise. The intensity of both deblur- ring and denoising tasks is controlled by the standard deviation (i.e., sigma value) of the Gaussian distribution. For all tasks, the reconstruction target is the original image but in the frequency domain the same frequency loss as MFM.
Observations. Table 2 shows the results with different levels of degradation for each task. We first notice that the optimal degradation level each task the context of representation learning is much heavier than its original task setting. For instance, standard SR task usually has down- sampling factor within x4, while we show that a much heavier x8 setting works the best. With right configuration of the task difficulty, all these tasks achieve comparable or even better per- formance than their supervised counterpart (e.g., 81.8% in Touvron et al.) (2021a)), indicating that these low-level tasks
or less helpful in representation learning. In addition, observe that

---

Published as a conference paper at ICLR 2023
representation learning benefits from all lens of frequencies. This be verified by the superior performance of denoising over SR and deblurring. As visualized in the frequency spectrum of the example image, denoising tends to intensify all frequencies of the spectrum, while SR and deblur- ring only removes high-frequency components. Thus, the performance of denoising is much closer to MFM, both utilize the full frequency spectrum. Attenuating and intensifying frequencies on the spectrum are essentially two different ways of performing corruption in the frequency domain. We believe other corruption types may also work well and leave this exploration for future work.
4.4
COMPARISON WITH PREVIOUS METHODS
4.4.1 IMAGE CLASSIFICATION
Table 3: ImageNet- top-1 fine-tumie ainettnnng accuracy of self-supervised models using ViT-S/16 and VTT-B/16 as the encoder. DINO and MoCo v3 use extra momentum encoder. BEÎT requires extra 250M DALL-E data (Ramesh et al., 202T) to pre-train dVAE. BEiT and MAE also use mask tokens (inserted either in the encoder or the decoder). All entries are image size of 224 x 224. We use the actual processed images/views to measure the effective pre-training epochs (Zhou et al. 2022). Scratch indicates the supervised baseline in Touvron et al. 2021a). I: doubled attention heads. our reproduced results with official code.
Method
Pre-train data
Extra model
Mask token
Epochs
ViT-S
ViT-B
Scratch
Touvron et al.
2021a
79.9
81.8
MoCo v3 Chen et 2021
IN-IK IN-IK
momentum ViT
81.41
83.2
DINO Caron et al.
2021
momentum VIT
1600
81.5
82.8
BETT
2022
IN-IK+
dVAE
300
81.3
82.9
MAE He et al.
2022
IN-IK
v
300
80.67
82,91
SR
IN-IK
300
80.8
82.4
Deblur
IN-IK
300
79.4
81.7
Denoise
IN-IK
300
81.1
82.7
MFM
IN-IK
300
81.6
83.1
Table 4: ImageNet- K top-1 fine-tuning accuracy of self-supervised models using ResNet-50 as the encoder. Table is split to three sub-tables for better placement. Results for other methods taken from Fang et al. )002) as we adopt the same fine-tuning recipe. i modified ResNet-50 architecture.
(a) Training- from scratch basclines.
(b) Fine- Fine-tuning for 100 epochs.
(c) Fine-tuning for 300 epochs.
Method
Epochs
Top-1 acc (%)
Method
Epochs
Top-1 acc (%)
Method
Epochs
Top-1 1 acc (%i
Originalge
75.3
RSB A3
78.1
RSB A2
79.8
Py Torchho
76.1
SimSiam
400
79.1
FixRes120
DeiTa0o
78.4
77.0
SR
Dehlur
300
300
78.0
77.9
MoCo v2
SimCLR
400
800
79.9
79.6
Denoise
300
77.5
ReaNct-RS
78.8
BYOL
400
80.0
a0MS
79.5
MFM
300
78.5
SWAV
600
80.1
MFM
300
80.1
VIT. In Table we compare the ImageNet-1K end-to-end fine-tuning results of self-supervised ViT- S/16 and ViT-B/16 models. We fine-tune ViT-S/16 for 200 epochs, and ViT-B/16 for 100 epochs. Other self-supervised models use the same or longer fine-tuning schedule. Compared with other representative self-supervised learners, our MFM can achieve comparable performance with fewer pre-training epochs while using none of the following: (i) extra data, (ii) extra model, (iii) mask token. This demonstrates the great potential of masked frequency modeling.
ResNet-50. We demonstrate that MFM can also pre-train a high-capacity ResNet-50 model. We simply adopt the same pre-training settings as VIT. During fine-tuning, we generally follow the ad- vanced vanilla ResNet "training from scratch" recipe in RSB (Wightman et al. 2021 except that we use the AdamW optimizer (Loshchilov Hutter 2017) following Fang et al. (2022). Table shows the results. Different from VIT, we observe performance degeneration of level image process- ing tasks like SR. deblurring and denoising compared with the RSB training faining-from--ae from--rrrate baseline (Table 4b). We hypothesize this discrepancy is due the architectural difference between ViT and CNN. Compared with ViT, the convolution operation in CNN tends to be more effective cap- turing high-frequency components. Thus, encouraging CNN model reconstruct high-frequency
8

---

Published as a conference paper at ICLR 2023
components of images brings no benefits to the performance. Instead, learning nigh-frequenc, infor- mation can compensate for the ability of ViT models in capturing the feeh-freeunccy components. In contrast, MFM outperforms its supervised counterparts in both ViT and CNN architectures leverages both low- and high-frequency components. Even under a demanding training procedure, e.g., fine-tuning for 300 epochs (Table Ac). MFM can still improve the supervised RSB A2 base- line by 0.3% and surpass several representative contrastive-basee self-supervised learning methods.
Table 5: ADE20K semantic segmen- tation (mloU) of ViT-B/16 models.
4.4.2 SEMANTIC SEGMENTATION
We evaluate MFM and low-level image processing tasks on the ADE20K semantic segmentation benchmark. We use UperNet (Xiao al. 2018 and adopt the same setup fol- lowing BETT (Bao et al. 2022). All models are fine-tuned for 160K iterations with an input resolution of 512 x 512, As shown in Table 5 our corruption-base -baeed models can achieve competitive performance compared with other representa- tive self- celf-supervised learners that are usually more expensive to compute.
Method
mloU
Supervised Touvron al.
2021a
45.3
MoCo CChen
2021
47.2
DINO Caron dr
46.8
BETT Bao al. 2022
47.7
MAE HC et a. 2024
48.1
SR
48.5
Deblur
47.0
Denoise
47,6
MFM
48.6
Table 6: Robustness evaluation six robustness benchmarks. We report top-1 accuracy of ViT- B/16 (leff) and ResNet-50 (right) models except for IN-C that uses the mean corruption error (mCE). The original ImageNet top-1 fine-tuning results are also appended for reference. The best results are in bold, and the second best results are underlined.
Method
Robustneas benehmarks
Orig.
Method
Rohusiness benchmarks
Orig.
FGSM
PGD
IN-C (1)
IN-A
IN-R
IN-SK
FGSM
PGD
IN-C
IN-A
IN-R
IN-SK
Scratch
46.3
21.2
48.5
28.1
44.7
32.0
81.8
Scratch
20.2
3.4
77.0
6.6
36.0
25.0
7%.1
MAE
38.9
11.2
52.3
31,5
48.3
33,8
82.9
SimMIM
16.8
2.1
77.0
5.7
34.9
24.2
77.7
SR
46.1
21.5
46.3
29.1
49.2
35,5
82.4
SR
17.2
1.9
73.6
6.5
35.8
25.4
77.9
Deblur
42.5
17.2
49.2
25.3
46.9
33.2
81.7
Deblur
17.2
2.0
74.8
8.2
37.2
26.5
78.0
Denoise
47.6
24.3
47.8
30.7
48.4
34.8
82.7
Denoise
15.8
1.8
78.0
7.2
35.6
24.7
77.5
MFM
47.7
24.4
47.5
32.7
48.6
34.8
83.1
MFM
18.5
2.3
74.2
9.0
36.9
26.7
78.5
4.5 ROBUSTNESS EVALUATION
We evaluate the robustness of our models on series of benchmarks in three aspects: (i) adversarial robustness, (ii) common corruption robustness, and (iii) out-of-distribution robustness. For (i), we study the adversarial examples generated by white-box attackers (e.g., FGSM (Goodfellow et al. 2014) and PGD (aadry et al. (2017)) on ImageNet-IK validation set as well as natural adversarial examples on IeageNet-A (Hendrycks al. 2021b); for (ii), evaluate ImageNet-C (Hendrycks) & Dietterich 2019) that includes 15 types of algorithmically generated corruptions with five levels of severity; for (ii),, we test on ImageNet-R (Hendrycks al., (202 and ImageNet-Sketch (Wang et al. 2019) that contain images with naturally occurring distribution shifts. We evaluate the same models fine-tuned on original ImageNet- IK (ViT-B/16 in Table abbleBamd and ResNet-50 Table 4bb) with- out any specialized fine-tuning on the different validation sets. As shown in Table 6 we can con- clude three observations: 1) Transformer-baat
models (e.g., ViT) are more robust than the CNN counterparts (e.g., ResNet-50). 2) Corruption-base based tasks (e.g., SR, Deblur, Denoise and MFM) are generally more robust than the MIM task (e.g. MAE and SimMIM). 3) MFM achieves the best trade-off between standard performance and robustness (the robustness of MFM always ranks within the top two, while the standard accuracy is the best).
CONCLUSION
In this work, we have studied the effectiveness of low-level image processing tasks for visual rep- resentation learning from a new frequency perspective and introduced a unified, flexible and robust self-supervised visual pre-training framework to perform image corruptions in the frequency do- main. We show that without relying on mask tokens or more complex designs (e.g., discrete visual tokens), simple mask free-freyunncy strategy achieve competitive performance for both VIT and CNN. We hope our unique frequency perspective can motivate the community to rethink the role of low-level tasks for unsupervised representation learnin.-

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The proposed method learns statistics of the training dataset and may reflect the biases in the data. Debiased measures thus have to be taken. The method may be deployed with large-scale models and data, causing negative impacts on the environment.
REPRODUCIBILITY STATEMENT
We provide detailed typer-parameter specifications for experiments in the main text (Section 4 and the supplementary material (Appendix G to ensure reproducibility. Code and models will be released at https:/ /www. mm ab ntu. cwm/pro jetm///mo ine////dexxx..///// html to facilitate future research.
ACKNOWLEDGEMENTS
This work is supported by NTU NAP, MOE AcRF Tier 2 (MOE-T2EP20120-000) MOE- (2EP20221-0012) and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well cash and in-kind contribution from the industry partner(s).
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, arXiv preprint arXiv:2204.0714 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555. 2022.
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR,, 2022.
Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. Multigrain: unified image embedding for classes and instances. arXiv preprint arXiv:1902.055000 2019.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In ICML, 2017.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang Context autoencoder for self-supervised representa- tion learning. arXiv preprint arXiv:2202.03000 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0420) 2020c.
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self ceff-supervised vision transformers. In ICCV, 2021.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated
data augmentation with reduced search space. In CVPRW, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wet Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2015.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transform- ers. arXiv preprini arXiv:211 112111.222 1.Niv:2/11.127100 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave, Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint 1rX1v:211..10740, 2021.
Yuxin Fang, Li Dong, Hangho Bao, Xinggang Wang, and Furu Wei. Comrupted image modeling for self-supervised visual pre-training. arXiv preprint (xXx:2202.00382. 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICIR, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020.
Bruce C Hansen and Robert Hess. Structural sparseness and spatial phase alignment in natural scenes. JOSA A, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 12261, 2019,
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ui--f-diittibution generalization. In ICCV, 2021a.

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. arXiv preprint arXiv:1901.0933. 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image recon- struction and synthesis. In ICCV, 2021.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In ECCV, 2016.
Yann LeCun, Berhhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989.
Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for sup-rvupereieed visual pre-training. arXiv preprint 082202 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint urXiv:1706 (00.003306000833 2017.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algo- rithms, pp. 80-111. Springer, 1981.
A Oppenheim, Jac Lim, Gary Kopec, and SC Pohlig. Phase in speech and pictures. In ICASSP, 1979.
Alan V Oppenheim and Jae S Lim. The importance of phase in signals. Proc. IEEE, 1981.
Yingxue Pang, Xin Li, Xin Jin, Yaojun Wu, Jianzhao Liu, Sen Liu, and Zhibo Chen. Fan: frequency aggregation network for real image super-resolution. In ECCV, 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. CVPR, 2017.
Leon N Piotrowski and Fergus W Campbell. A demonstration of the visual importance and flexibility of
amplitude and phase. Perception, 1982.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate-tare transfer learning with pretrained models for natural language understanding: When and why does work? In ACL, 2020.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-tratee pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot texl-to-image generation. In ICML, 2021.
Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. Adversarial masking for self-supervised learning. ICMI., 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention. In
ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitore-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. JMLR, 2010.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurlPS, 2019.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data, In ICCVW, 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuuee Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint NNNv:2110000470: 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier- based framework for domain generalization. CVPR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710000410 2017a.
13

---

Published as a conference paper at ICLR 2023
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning deep enn for image denoising. TIP. 2017b.
Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Bjom Stenger, Ming-Hsuan Yang, and Hongdong Li. Deep image deblurring: survey. arXiv preprint arXiv:2201.10700 2022.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017c.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
14

---

Published as a conference paper at ICLR 2023
A MORE ABLATIONS
Table 7: More Ablations for MFM on ImageNet-IK. All models are pre-trained for 300 epochs, and evaluated with top-1 fine-tuning accuracy. Default entry (the same settings in Table [ of the main text) is marked in gray.
(a) Decoder depth. A simple linear layer forms the best with lower training costs (Transformer blocks have hidden size of 384 with heads).
(b) Masking in different domains. Frequency mask- ing more flexible and unified option for different architectures.
Decoder
Blocks
Top-1 acc (%)
Arch.
Task
Top- acc (%)
linear
83.1
MIM
82.8
ViT-B/16
1
83.0
MFM
83.1
Transformer
2
83.1
MIM
77.7
blocks
4
83.1
ResNet-50
MFM
78.5
8
83.1
Decoder depth. Table 7 ablates the effect of different decoders. MFM can work the best with a simple linear layer while benefiting from lower training costs compared with deeper decoder.
Frequency masking vs. spatial masking. Considering that different design choices in existing masked image modeling (MIM) methods could affect the performance significantly, to eliminate the interference of other factors, we directly replace our reeueecy-doomain-baee domain-based random patch masking for fair comparison. The results are shown in Table 7b Applying MIM to ResNet-50 leads to inferior performance even than the supervised baseline (i.e., 78.1% in RSB A3 (Wightman al.] (02T)). In contrast, MFM outperforms the supervised RSB baseline as well as the MIM counterpart regardless of architectures, demonstrating that frequency masking is indeed more flexible and unified option for different architectures.
masking with spatial-
B TRAINING TIME COMPARISON
We measure the pre-trate pre-training time per epoch in Table 8 Note that MoCo v3 (Chen al. 2021) and DINO (Caron et al. 2021) need to switch two global views and have four and 14 forward passes in total, respectively. BEiT (Bao et al. 2022), MAE (He et al. 2022) and MFM are 1-view methods without switching. MFM is relatively efficient compared with other MIM methods (e.g.. BEiT) except for MAE. However, taking only visible patches as input breaks the regular 2D structure of images, which makes MAE only applicable to VIT. In contrast, MFM is agnostic to architectures and can be flexibly applied for both ViT and CNN families. Considering the flexibility and universality of MFM, slightly increasing time over MAE is acceptable.
Table 8: Training time comparison. The time is measured on the same 8-GPU machine with the same batch size using ViT-B/16, counted in relative to our approach. I; BEiT requires an additional stage to pre-train dVAE, which is not included.
Method
Setup
Time per epoch
MoCo v3
2-view, 4-pass
1.84
DINO
(2+10)-vien view, 14-pass
2.04x
BEiT
1-view, 2-pass
1,53x1 1
MAE
1-view. 1-pass
0.82x
MFM
1-view, 1-pass
1.00
C COMBINATION OF LOW-LEVEL IMAGE PROCESSING TASKS
In Table of the main text, we have shown that randomly sampling low-/high-pass filters to predict both high and low frequencies benefits MFM make full use of the frequency spectrum, thus
15

---

Published as a conference paper at ICLR 2023
leading better performance. Here, further study the effect of combining low-level image processing tasks, i... SR, deblurring and denoising. Table 9 reports the ImageNet- IK top-1 fine- tuning accuracy of ViT-B/16 models. We find that combining these low-level corruptions does not bring similar benefits as MFM and even degrades the performance.
Table 9: Combination of SR, deblurring, denoising tasks using ViT-B/16 ImageNet-1K, All models are pre-trained for 300 epochs, and evaluated with top-1 fine-tuning accuracy.
Task
Top-1 acc (%)
Individual task:
SR
82.4
Deblur
81.7
Denoise
82.7
Integrated task:
SR+Denoise
82.2
Deblur+Denoise
82.5
D FURTHER DISCUSSION
Mask tokens. In MIM methods, mask tokens are learnable patch embeddings inserted in the position where the input tokens are masked out. They are highly coupled with the Transformer architecture and not directly applicable to CNNs. Introducing special mask tokens in any intermediate stage of CNN is infeasible, as the intrinsic dense-sliding-wnddon
paradigm in convolution layers brings information leakage between visual features in previous layers (Fang et al. (022). Thus, the large CNN family cannot directly benefit from this pre-training scheme like Transformers. In contrast, MFM performs masking in the frequency domain without relying mask tokens. Thus, MFM is agnostic to the architectures and can flexibly applied for broader ViT and CNN families.
Table 10: System-level comparison with Siamese- hybrid MIM methods (e.g., iBOT Zhou et al. 2022) and data2vec (Baevski et al.) 2022) using ViT-B/16 on ImageNet-1K K. For a fair com- parison, re-implement iBOT without multi-crop augmentation (but keeping the two global views) and data2vec (Bacvski et al. (022)) without additional losses of intermediate Transformer layers us- ing their official code. Training costs are counted in relative our approach. Note that MFM is agnostic to architectures while these hybrid methods are not.
Method
Pre-text task
#Views
#Epochs
Top-1 acc (%)
Training costs
iBOT
MIM+CL
2
300
82.0
2.14x
data2vec
MIM+CL
300
83.0
1.60
MFM
MFM
300
83.1
1.00x
Comparison with hybrid MIM methods. A line of recent research (Zhou et al. 2022 Baevski al. [022 Assran al. 2022) combines MIM with contrastive learning (CL) into a Siamese frame- work and achieves better performance than a single task. Apart from adopting a Siamese network,
additional techniques used in these works include multi-crop augmentation Zhou al.
(202
Assran et al. 2022) and multiple losses of intermediate Transformer layers in Baevski et al. without which their performance will be degraded significantly as shown in [O] take iBOT
2022
and data2vec as examples here since most of these works are concurrent to ours). Therefore, to
eliminate the interference of other design factors, we mainly compare with pure MIM methods in
study. More vancce techniques used in these works may also be incorporated into MFM to further improve the performance, which is beyond the focus of this work.
Concurrent work. A concurrent work (Liu al. 2022) also involves self-supervised learning in the frequency domain. Our work differs from theirs in the following aspects: 1) aims at improving upon existing MIM approaches by additionally designing computing losses in both spatial and frequency domain. In contrast, we aim at exploring an alter- native frequency corruptio strategy beyond MIM. Our method does not rely any existing MIM
complex frequency decoder and
16

---

Published as a conference paper at ICLR 2023
approaches and show that MFM can also work well independently. 2) It is based on MAE (He et al., 2022) and still performs spatial masking with mask tokens. Thus, it is still not applicable to masking strategy is agnostic to architectures. 3) As
CNNs, whereas opposed to Liu et al. (2022) that mainly targets at improving MIM performance, we comprehen- sively study the effectiveness of low-level image processing tasks for representation learning from unified frequency perspective and provide rather different insights that other low-level tasks beyond MIM can also work well.
fiequency-domain-
Limitations. Our study has several limitations: 1) We mainly focus on the architectural flexibility and universality of MFM, while leaving the scaling behaviour under-explored. 2) We mainly eval- uate the quality of learned representations classification and ADE20K semantic segmentation, following Bao et al. (2022). The transferability
representative benchmarks, i.e., ImageNet- IK image
on more downstream tasks can be studied in future research. 3) Despite the intriguing properties of the Fourier domain, the redundancy in frequencies may still exist. More advanced information
suppression strategies can be further explored. We believe that our MFM can also complement contrastive learning and MIM approaches to further improve the performance. We leave these ex- plorations for future work.
Future work. In this paper, we have shown that MFM is simple, unified and flexible self- supervised pretext task for various architectures. Compared with MIM, MFM can embrace broader architectures (e.g., VIT, CNN, etc.) and has more appealing robustness. Some possible future re- search directions may include: 1) More self-supervised learning works in the frequency domain with different modalities (e.g., image, video, audio, etc.). 2) Combine MFM with existing con- trastive learning and MIM paradigms to further improve the performance. 3) Apply MFM for model robustness analysis and calibration. 4) The idea of MFM may also be used in low-level image reconstruction and synthesis tasks.
E PSEUDOCODE
Algorithm Pseudocode of MFM in a PyTorch-like style.
E; backbone encoder (e.g., vit, cnn) linear prediction head mask: frequency mask 1ox-/high-pass filters sampled with a Bernoulli
-1- (m is defined in Bg. (2), p is the probabil lit of sampling a low-pass filter m) gamma: exponent
distribution, i.e., mask Bernoulli pesnoolli(pp
control the sharpness the frequency distance
for (x, mask) in loader: # load minibatch with samples
x = axg # "andom view, NxCxilxl
f convert spatial domain into frequency donain
freg ff:2 # 2D F?T
x free ffrshi (x freq, dim= (-2, -1)) # shift low frequency to the center frequencies
x_reeq x_freq mask mask portion
x_freq ifftshift (x_ freq, dim-(-2, -11) 4 restore the original frequency order it convert frequency domain back
spatial domain
x_corrupted ff real # 2D iFFT (only keep the real part) x_prediated fxx_oorrppted; x_
predicted view, NxCxixM
loss txegeeney x, gamma) $ frequency loss
# only compute the frequency less loss (lass (1 mask)) .sum() / (1 mask).
masked
# update model loss.backe backwardd (f)
def FrequencyLoas (x, y, gamma)
x s_feeq y_freq fft2 (x), fft2ly) 2D FFT
# shift low frequency to the center
x_freq fftshift f_eer dim=l-2, -1])
y f_freq fitshiftly freq, dim= -11)
stack the
and imaginary parts along the last dinension
x_ stack [x rreq x freq. -1)
y f_freq stack [l__feg..eear F compute the Frequency distance
e_freq.imagl
d = (x_freq y_freq) return (dl..., 0] dl..., 11) (0.5 gamma)
17

---

Published as a conference paper at ICLR 2023
Figure 3: Example frequency spectrums of soaaldddmai--baseee random patch masking from ImageNet-IK training set. The masking ratio is 75%. Performing patch-wise masking in the spatial domain incurs grid-wise artifacts the frequency spectrum.
F FREQUENCY SPECTRUM OF MIM
Figure 3 visualizes some example frequency spectrums of spatial-domain-basee random patch mask- ing used in MIM. Performing patch-wise masking the spatial domain incurs grid- wwie artifacts on the frequency spectrum, preventing further meaningful observations.
G IMPLEMENTATION DETAILS
Pre-training. Table summarizes the pre-training settings for vanilla VIT and ResNet-50 models. All experiments are conducted 16 V100 32G GPUs for VIT models and 8 V100 32G GPUs for ResNet-50. The configurations are shared by different architectures, without specialized tuning. This demonstrates that MFM is general across architectures.
Fine-tuning, Table 12 and Table [3] summarize the fine-tuning settings for vanilla ViT and ResNet- 50 models, respectively. The configurations for ViT are shared across models, except that smaller models are fine-tuned longer. The configurations for ResNet-50 basically follow Wightman et al. 2021 except that we adopt the AdamW optimizer following FFngiinnggngggggog et (2022).
Semantic segmentation on ADE20K. We use UperNet (Xiao et al. 2018) following the configu- rations BETT (Bao et al. 2022). Specifically, we use AdamW as the optimizer and fine-tune for 160K iterations with batch size of 16. We search the learning rate for all the results in Table 5 the main text. The input resolution is 512 x 512, and we use single-scale inference. As suggested in BEiT (Bao et al. 2022), initialize all segmentation models using model weights after super- vised fine-tuning on ImageNet-IK, following the common practice BERT (Devlin et al. 2019) fine-tuning in NLP (Pruksachatkun al. 2020).
Table 11: Pre-training settings for vanilla ViT-S/16, VIT-B/16 and ResNet-50 models on ImageNet-IK. Note that we adopt the same pre-training configurations across different architec- tures without further parameter tuning.
Configuration
Value
Optimizer
AdamW Loshchilov Hutter 2017)
Pre-training epochs
300
Peak learning rate
1.2c-3
Batch size
2048
Weight decay
0.05
Optimizer momentum
B1, = 0.9, 0.95 (Chen et al. 2020a
Learning rate schedule
Cosine decay
Warmup epochs
20
Gradient clipping
3.0
Dropout (Srivastava et al. 2014)
Stochastic depth (Huang et al. 2016
X
LayerScale (Touvron et al. 20216
X
Data augmentation
RandomResizedCrop
Pos. emb. Transformer layers Patch size
1-D absolute pos. emb. Dosovitskiy et al. 2020
Pre- training resolution
224
18

---

Published as a conference paper at ICLR 2023
Table 12: Fine-tuning settings for vanilla VIT-S/16 and ViT-B/16 ImageNet-1K. We fine-tune VIT-S/16 for 200 epochs, and ViT-B/16 for 100 epochs. All other hyper-parameters are the same.
Configuration
Value
Optimizer
AdamW Loshchilov Hutter 2017
Fine-tuning epochs
200 (S), (B)
Peak learning rate
8e-3
Layer-wise learning rate decay [ao al. 2022
0.8 (Clar et al. 2020 2048
Batch size
Weight decay
0.05
Optimizer momentum
B1, = 0.9, 0.999
Learning rate schedule
Cosine decay
Warmup epochs
5
Loss function
eots-yy loss
Gradient clipping
X
Dropout (Sriaastava al.
2014
X
Stochastic depth (Huang et al. 2016
0.1
Mixup (Zhang
7a)
0.8
Cutmix Yun al. 2019)
1.0
Label smoothung (Szegedy et al.
2016
0.1
Random augmentation (Cubuk et al
2020
9/0.5
Patch size
16
Fine-tuning resolution
224
Test resolution
224
Table 13: Fine-tuning settings for vanilla ResNet-50 on ImageNet-1K. The hyper-parameters
generally follow Wightman al. (202T), except that adopt the AdamW optimizer following Fang et al. (2022).
Configuration
100 epoch
300 epoch FT
Optimizer
AdamW Loshchilov & Hutter 2017)
Peak learning rate
120-3
Layer-wise lcarning rate decay |ao et al. 2022
X
Batch size
2048
Weight decay
0.02
Learning rate schedule
Cosine decay
Warmup epochs
Loss function
Binary cross-entropy loss
Gradient clipping
Dropout Srivastava
2014
Stochastic depth (Huang el
2016
Mixup /hhng et al
201 7a)
0.1
Cutmix Yun et al.
2019)
1.0
Label smoothing (Szegedy al. 2016
0.1
Repeated augmentation (Berman et
2019
Hoffer et al. 2019)
Random augmentation CCbbu el al.
2020
6 0.5 0.5
7 0.5
Fine-tuning resolution
160
224
Test resolution
224
Test crop ratio
0.95
H VISUALIZATION
We provide more qualitative results of corrupted images Figure 4 following Table 2 in the main text, as well as recovered images using unseen ImageNet- IK (Figure 5) and COCO (Figure 6) vali- dation images.
19

---

Published as a conference paper at ICLR 2023
X16
Figure More visualizations of corrupted image samples from ImageNet- training set follow- ing Table in the main text. We visualize both images and their frequency spectrums with different degradation levels. (a) SR. (b) Deblur (kernel size 21). (c) Denoise, (d) MFM. Each task achieves its best performance with moderate degradation intensity. See Section 4.3 in the main text for more discussion. Zoom for best view.
20

---

Published as a conference paper at ICLR 2023
SR
Deblur
Denoise
MFM
ouiput
Input
output
output
Imput
output
imput
output
Figure 5: Example results of recovered images ImageNet-IK IK validation set for SR, deblurring, denoising and MFM tasks. We visualize both images and their frequency spectrums. We use the best pre-trained model of each task in Table 2 of the main text for visualization, iee., the downsampling scale factor is x8 for SR, the Gaussian blur sigma is for Deblur, the Gaussian noise sigma is 75 for Denoise, and the mask radius is 16 for MFM'. Compared with SR. Deblur and Denoise, MFM can utilize both high-frequency and low- cow-frequency information for prediction. Zoom in for best view.
"A MFM only predicts the masked area of the frequency spectrum, we overlay the output with the visible frequency spectrum for better visual quality.
21

---

Published as a conference paper at ICLR 2023
SR
Deblur
Danoise
MFM
original
imput
output
imput
outpat
input
output
imput
oulput
imput
outuu
Figure 6: Example results of recovered images COCO validation for SR, deblurring, de- noising and MFM tasks, using the models pre-trained on ImageNet- (the same model weights as in Figure 5) We visualize both images and their frequency spectrums. Zoom for best view.
22


Reviews:
[{"review_id": "xp5P3AwVWPJ", "reviewer": "Reviewer_Lxfd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the possibility of masked frequency modeling for representation learning. It covers several research topics centered on MFM: a method that does MFM; connecting previous pre-training methods to MFM (e.g., super resolution); architecture change (from ViT to ConvNets) as an advantage of MFM over masked image modeling (the more popular methods for self-supervised pre-training); more tasks and benchmarks beyond classification with MFM pre-training. The results are not state-of-the-art per-se, but the complete assessment of MFM is interesting and of value.", "summary_of_the_review": "Overall I think this paper has already reached its maturity. It has a simple, basic method for MFM, and has explored a lot centered around this simple methods (I especially appreciate the connection to other pre-training methods, and experiments on ResNet). Although the method has not advanced state-of-the-art, I would still vote for acceptance given the added knowledge and value from this work to the research community.", "strength_and_weaknesses": "Strengths:\n- The exploration of a simple masked frequency modeling algorithm is interesting and valuable to the community, given the recent success of masked image modeling methods for representation learning.\n- The paper's exploration is quite complete, covering a simple basic method for MFM, MFM in the context of previous methods, different architectures, different tasks and benchmarks.\n- The paper is also relatively well-written and well-polished.\n\nWeaknesses:\n- In an ideal world, the newly proposed approach should advance the state-of-the-art in a meaningful way. Unfortunately, this work is still yet to reach that level of performance.\n- It would also be great if bigger models are trained to show the power of pre-training (if the computation permits).\n- With MAE-style design, MIM is made very efficient that it does not need to compute for all the patches in the encoder. Right now MFM still looks at the full-sequence. This is a potential downside in speed.\n- (minor) At page 6 last paragraph, the paper states an hypothesis about the mask shape being largely correlated with the category statistics of pre-training datasets. Is there evidence for this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "uus60RdG_WZ", "reviewer": "Reviewer_Un1w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an empirical study of different masking-noise strategies for an image auto-encoder model. In contrast to the recent masked auto-encoded, authors propose to mask the inputs in the frequency domain by randomly dropping high or low frequency components. Their approach learns a representation in a self-supervised manner by reconstructing the missing component in the frequency domain.\n\nAuthors demonstrates the soundness of their approach on ImageNet finetuning, ADE20K segmentation task and various robustness tasks where they show comparable or better results compared to the previous work.\n", "summary_of_the_review": "The paper proposes an empirical study of the masking noise in auto-encoder which seems novel and is an interesting study. However, the practical impact of the study is somewhat limited. It is unclear to me why one practitioner should prefer masking in the frequency domain compared to the patches masking scheme, as the later provides good scalability or can leads to strong results as Data2Vec or MSN demonstrate.\n\n=== After reading rebuttal.\n\nThank you for your rebuttal. While I appreciate the additional ablation experiments that compare MFM with MIM and different predictor depths, my main concern regarding the paper is not fully addressed.  It is unclear to me why one practitioner should prefer masking in the frequency domain compared to other patch masking scheme,", "strength_and_weaknesses": "Strengths:\n- To my knowledge, the proposed masking strategy is novel and sound.\n- Authors evaluate their representation on various downstream tasks requiring different level of abstraction (classification, segmentation).\n- Authors propose an ablation of the masking design choice.\n\nWeaknesses:\n- One of the main advantage of the masked auto-encoder is its scalability as the encoder does not need to process the masked patches. The current MFM proposal requires to process the full-images and therefore comes with higher computational cost. It is not clear to me if this approach is viable to train larger model such as VIT.L or VIT.H.\n- Some baselines are missing in the experimental study. Data2Vec (Baevski et al., 2022) for instance, is a masked image modeling approach which achieves 84.2 top-1 on imagenet finetuning with a VIT.B. MSN (Assran et al., 2022) is another approach combining mask image modeling and siamese networks which exhibits good robustness results (see table 14 in their paper). \nGiven that those approaches using patches masking obtain good results, it is unclear to me why one should prefer the use of mask frequency modeling.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "baAGA2Y3lsy", "reviewer": "Reviewer_34gh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised representation learning method by applying high-pass or low-pass filters on the image and restoring the missing frequency components. The experiments also include results for self-supervised learning via other common image restoration methods such as denoising, super-resolution and deblurring. ", "summary_of_the_review": "This paper presents some interesting empirical results about using different image corruptions for self-supervised learning. However, the proposed method is currently not principled enough to be a significant technical contribution. I would recommend incorporating frequency analysis in a more general form beyond low-pass and high-pass filtering.", "strength_and_weaknesses": "Strengths:\n\n1. The paper provides empirical evidence on the possibility of using various low-level image processing methods for self-supervised learning, including high-pass or low-pass filtering, denoising, super-resolution and deblurring, which will be a useful contribution to the community.\n\n2. The proposed method could achieve comparable performance for ViT-S, ViT-B, Resnet50 and improved robustness for ViT-B.\n\n3. The paper is generally well-written and easy to follow. \n\nWeaknesses:\n\n1. The major concern is that the methodology contribution is straightforward:\n- The masked frequency modeling proposed is straightforward and limited to low-pass and high-pass filtering, which are basically blurring and sharpening. To be a principled method, it would be better to include more general forms of frequency manipulations, which would correspond to modeling more complex periodical patterns in the spatial domain.\n- Although the authors attempt to provide a unified perspective of the method by relating to denoising, super-resolution and deblurring (Section 3.2), such perspective is very natural to have since most image processing methods in the spatial domain would have a corresponding interpretation in the frequency domain. \n\n2. The authors claim that no extra model (e.g. momentum teacher) and no mask token are advantages of MFM over other self-supervised methods. However, these differences are more about implementation choices rather than principled advantages. \n\n3. Although the experiments show promising results, they lack further analysis and insights:\n- Why is no gain observed for CNNs? \n- Why could masked frequency modeling achieve better robustness than MAE?\n- etc.\n\nMinor issues:\n- In Figure 2, what does the FFT after the linear head mean? This seems to be contradicting the text.\n- It is unclear how the linear head would output the frequency map of the same size as the original image, especially for CNNs encoder.\n- In Eq.4, the lowercase symbols f_r and f_o seem to be redundant and unnecessary.\n- In Table 1(f), are the l1 and l2 losses in spatial domain? This is not clearly stated.\n- In Table 6, the robustness is not compared to other self-supervised methods for ResNet.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PjxVWIPzstq", "reviewer": "Reviewer_Nxgg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus masking frequencies in the input image for self-supervised pretraining rather than masking out sections of the spatial domain. The insight is that the masked frequencies will carry more information about the patterns in the underlying image as compared to spatial patches.", "summary_of_the_review": "Overall, I think the idea is interesting and novel. It is also intuitive that the frequency domain may capture a wider range of information that is covered by multiple corruptions. However, I would like to see some of the above mentioned clarity issues resolved.\n\nAdditionally, it is not clear to me why someone would choose to use this strategy. The bottom line results are somewhat ambiguous, and there are several recent SSL baselines which *could* be added that achieve higher performance.  Therefore I think it could be better to include more information analyzing the benefits of using MFM and possible future research directions which may be able to make further use of this work.", "strength_and_weaknesses": "## Strengths\n\n- The method is agnostic to architectures\n\n- It is interesting that denoising is closer to MFM and the effect of noise also intensifies all frequencies of the spectrum. Would it be possible to compare with masking as a corruption as well to see what that does to the overall frequency of the image?\n\n## Weaknesses\n\n- Figure 1 is hard to interpret. I can see that there are differences between MFM and the applied corruptions, but the frequency domain has no axis labels and it is difficult to gain intuition behind why the frequency domain plots look different from the corruptions, and what significance this implies.\n\n- Figure 2, which frequencies in the \"mask frequencies\" brackets in the figure are low and high frequencies? Or are they frequencies and amplitudes? What is the red circle in the first part of the figure? Is that the selected radius which is specified in equation 2?\n\n- After equation 3, it says that the filtered images are \"fed to and encoder as input, following a Bernoulli distribution.\" This statement is not clear, what does it mean?\n\n- After equation 3, the authors state that the model still takes the spatial images as input as well. It would be good to state this earlier in the paper. Until this point, I was under the impression that the model would only receive the frequency domain as input. \n\n- Given the confusion from the previous points, it would be nice to see a better diagram of the entire model architecture in a figure (maybe revamping figure 2) to allow the reader to visualize what the entire model flow looks like. It is not clear to me when and where the frequency inputs come into the model as inputs. For example, figure 2 looks like like the frequencies are input into the encoder, but the MFM Encoder section says nothing about this.\n\n- Did you ever consider using difference unit circle norms as shapes for ablation c (cirlce = 2 norm, square = infinity norm, etc)? The reason I ask this is because the frequency domain charts seem to show some interesting designs which would be captured by unit cirlce norms < 1 which would form a star-like shape.\n\n- Building on the point above, what is the shape of the rhombus in this context. Would this be equivalent to the 1-norm unit circle?\n\n- page 8: \"representation learning benefits from all lens of frequencies...\" What does this mean?\n\n- mask tokens are not used, which means that pretraining should be much more expensive than other MIM approahces, right? Because most MIM models produce a shorter sequence length to the transformer. Can you provide a breakdown of the pre-training per iteration computational cost in terms of wall-clock time between MFM and some baselines?\n\n## Minor\n\nSection 2 MIM section: \"Besides, iGPT...\" IT is not clear why besides is used here, besides what?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster