Masked Siamese ConvNets: Towards an Effective Masking Strategy for General-purpose Siamese Networks 
Authors: Li Jing
Jiachen Zhu
Yann LeCun
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
siamese networks
masking
convNets

Abstract: Siamese Networks are a popular self-supervised learning framework that learns useful representation without human supervision by encouraging representations to be invariant to distortions. Existing methods heavily rely on hand-crafted augmentations, which are not easily adapted to new domains. To explore a general-purpose or domain-agnostic siamese network, we investigate using masking as augmentations in siamese networks. Recently, masking for siamese networks has only been shown useful with transformer architectures, e.g. MSN and data2vec. In this work, we identify the underlying problems of masking for siamese networks with arbitrary backbones, including ConvNets. We propose an effective and general-purpose masking strategy and demonstrate its effectiveness on various siamese network frameworks. Our method generally improves siamese networks' performances in the few-shot image classification, and object detection tasks.

Under review as a conference paper at ICLR 2023
MASKED SIAMESE CONVNETS: TOWARDS AN EFFEC- TIVE MASKING STRATEGY FOR GENERAL-PURPOSE SIAMESE NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Siamese Networks are a popular self-supervised learning framework that learns useful representation without human supervision by encouraging representations be invariant to distortions. Existing methods heavily rely on hand-crafted augmen- tations, which are not easy to adapt to new domains. To explore a aggosiic siamese network, we investigate using masking as augmentations in siamese net- works. Recently, masking for siamese networks has only been shown useful with transformer architectures, e.g. MSN (Assran et al., 2022] and data2vec (Baevski et al. 2022). In this work, we identify the underlying problems of masking for stamese networks with arbitrary backbones, including ConvNets. We propose effective masking strategy and demonstrate its effectiveness on various siamese network frameworks. Our method generally improves siamese networks' perfor- mances in the few-shot image classification, and object detection tasks.
INTRODUCTION
self-supervised learning aims learn useful representations from scalable unlabeled data without relying on human annotation. It has widely used in natural language processing (Devlin et al. 2019 Zhang et al.] 2022 Brown et al., 2020), speech recognition (van den Oord et al. [208]] Hsu al.
1021:Schneider et al. 2019 Baevski et al. 2020) and other domains (Rives et al. 2021 Rong et al.
2020).
Recently, self-supervised visual representation learning has also become an active research
First introduced by Bromley et al. (1993), the siamese network (Chen et al. 2020a b He et al. 2020
Chen et al. 20220 2021; Caron al. 2020, Grill et al. 2020 Chen & Hel 2020; Wang
et al. 2022 Zbontar et al., 2021 Bardes et al., 2021 is one promising approach among many self- supervised leanning approaches and outperforms supervised counterparts in many visual benchmarks. encourages the encoder to be invariant to human-designed augmentations, capturing only the essential features. Practically, in the vision domain, the siamese network methods rely domain- specific augmentations. such as cropping, color jittering and Gaussian blur, which do not transfer well to other domains. Therefore, is desired have a general augmentation approach for siamese networks that require minimal domain knowledge and can generalize.
Among various augmentations, masking the input one of the simplest and effective choices, which has been demonstrated to be useful for language (Devlin et al. 2019) and speech (Hsu et al. (2021). However, not until the recent success of vision transformers (ViTs) (Dosovitskiy al.) [021] Touvron
et al. 2021) can vision models leverage masking as a general augmentation. Self-supervised learning
with masked inputs has demonstrated more scalable properties when combined with ViTs (He et al..
2021 Bao al. 2021 Zhou al. 2021 Baevski et al. 2022). Unfortunately, siamese networks with
naive masking do not work well with arbitrary architecture, e.g., ConvNets (He et al. 2016 Liu et al.
2022
We identify the underlying issues behind masked siamese networks with ConvNets. We argue that ConvNets do not have mechanism encode null information. In addition, masking will introduce parasitic edges. We propose a preprocessing procedure to solve these problems. We present several general-purpose purpose designs that allow siamese networks to benefit from masked inputs. Experiments

---

Under review as a conference paper at ICLR 2023
show that siamese networks with ConvNets backbone can benefit from masked inputs with masking strategy.
We summarize contributions below:
We discuss the role augmentations in siamese networks and explore a ceneral-purposs. augmentation approach of masking. We identify the underlying problem of masking for siamese with ConvNets backbones.
We propose preprocessing step to overcome the problem behind masked siamese networks with ConvNets. We present series masking designs which allow masking to benefit siamese network with ConvNets backbones.
We propose Masked Siamese ConvNets (MSCN), effective masking strategy for general- purpose siamese networks. Our method can be applied to various siamese network frame- works, and it demonstrates competitive performances on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.
RELATED WORKS
2.1 STAMESE NETWORKS
Self- celf-supervised visual representation learning has become an active research area since they have shown superior performances over supervised counterparts recent years. One promising approach
is to learn useful representations by encouraging them to be invariant to augmentations, known as
siamese networks or joint-embedding methods (Misra & van der Maaten 2020; Chen et al.. 2020a b
He et al., 20200 Chen et al. 20220 2021 Caron et al.]]222000 20200] Grill et al. [0200] Chen & He 2020
Wang et al. 2022 Zbontar et al. 2021; Bardes et al. 2021). These methods use different mechanisms
to prevent collapse, and they all rely carefully designed augmentations such as random resized cropping, color jittering, grayscale and Gaussian blur. These augmentations prevent the encoder from only using trivial features. Empirically, siamese networks with these standard augmentation settings usually work well with arbitrary architectures, including ResNets (He et al. 2016) and ViT's (Dosovitskiy et al. 2021). Their representations are label-efficient (Assran al. 2021 20022. more robust (Hendrycks et al. 2019), and have improved fairness (Goyal et al. 2002). Siamese networks have been demonstrated benefit from scalable data (Goyal al. 2021a).
In addition,
2.2 REPRESENTATION LEARNING WITH MASKED INPUTS
Masking the input is one of the simplest methods to corrupt the information input and could be applied to a wide range of data types. It has been mostly used in two scenarios.
The first is for denoising autoencoder frameworks (Vincent al. 2010
2008). Motivated by its
success in NLP (Devlin et al. 2019; Brown et al. (020) with transformers Vaswani et al., 2017). various visual representation learning methods (He al. 202 Bao al. 2021 Zhou 202
using ViT's have also shown benefit from masked inputs. These methods have proven to be promising purpose supervised learning approach.
The second is for siamese networks. Siamese networks can benefit from masked inputs (Baevski jtt al. 2022 Assran et al. 202T) where masking serves as an extra augmentation. These methods are able to learn more transterable representations and have the benefit of being label-efficient, These works are limited to ViT architectures. However, no previous work has shown that the masking approach can work equally well with arbitrary ConvNets.
3 AUGMENTATIONS FOR SIAMESE NETWORKS
In this section, we discuss the role of augmentations siamese networks and outline several design principles which will be used to guide masking strategy.
2

---

Under review as a conference paper at ICLR 2023
3.1 PRELIMINARIES
Siamese networks for visual representation learning start with creating, two random views X1 and X2 from the same imput x. with two different sets random augmentation transformations T and To to each view. These methods then train an encoder fa(-) and projector (o(-) to mínimize Lp = o (xx network loss function. To simplify the notation, we will ignore the projector (o(-) in all of our following discussions, since we could always consider the projector as part of the encoder.
known as the positive term of the siamese
However, if the encoder fe(-) is trained with the positive term alone, it f- will quickly converge to a collapsed solution that produces constant representation for all inputs. Preventing collapse can be solved by various frameworks, including contrastive loss Chen et al. (0200a;; He et al. (2020), redundancy reduction /bbontar et al. ) Bardes et al. (2021). clustering Caron et al. (2020) and distillation Grill et al. (2020): Chen & He (2020): Caron et al. (202T). These methods explicitly or implicitly prevent Eg.or or fe (x) fe (T (x' [ becoming too small for x' coming from different images. This is known as the negative term the siamese networks loss function. Although the negative term design is an important topic of siamese network methods, is relatively independent of augmentation choice.
In this work, focus the positive term analyize the role of augmentations Tg. Data aug- mentation plays an important role in guiding encoders fo(.) to learn useful representations of the input that they can be used for downstream tasks. With weak augmentation, encoder based on simple input statistics such that (x)) gTas (x)) is small for all ø and 6', then the encoder f only needs capture those superficial features to minimize the positive of the loss function. With aggressive augmentations, it becomes difficult to find a encoder such that (Tg (x)) aTar (x)) is small for all é and o' due the mutual information between two augmented views is destroyed. Then the encoders fo (-) are impossible to capture that information which is potentially useful for downstream tasks.
could find
Furthermore, other than assessing the augmentations based on how aggressive it is, Huang et al. (202T) shows that the generalization ability of contrastive self-supervised learning depends on the concentration of augmented data within the same latent class. Intuitively, it means that increase the performance downstream tasks that require the semantic information of the data, the augmentations applied to input data should be able to increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.
3.2 AUGMENTATION DESIGNING PRINCIPLE
Following the discussion in 3.1 we present three design principles of data augmentations for siamese networks:
1. Prevent easy solutions such that encoder fe(-) only captures certain superficial features;
2. Keep as much mutual information between the two views as possible;
3. Increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.
PROBLEMS IN MASKED SIAMESE NETWORKS WITH CONVNETS
Siamese networks using masking as an extra augmentation have demonstrated competitive perfor- mances (Assran et al. 2022, Baevski et al. 2022) with ViT backbones. Naively replacing ViTs with ConvNets results in significantly worse performances. Here, we identify its underlying problems.
See Figure 0
Masking As Missing Information The main reason masking as augmentation performs poorly on siamese networks with ConvNets is that masking represents missing information, and there is good choice assigned values for continuous input. Figure Ib shows the color histograms for the original input and its masked version. Naively assigning noise trainable value will
3

---

Under review as a conference paper at ICLR 2023
(a) input images
(b) histogram
(c) learned convolution kernels
Figure I: Underlying problems in masked siamese networks with ConvNets. Each pair a subfigure corresoonds to the original imput and the masked version. (a) visualization of the input image and the masked version; (b) the color histogram was distorted with masks. The values close to the assigned mask value represent missing information and could confuse the network; (c) visualization of the encoder's first convolutional layer kernels after pretraining.
confuse the network. For ViTs, this problem is automatically avoided by the self-attention mechanism that does not attend the masked area. Hence, it is essential for the network to identify masking as missing information with proper preprocessing.
Masking Introduces Parasitic Edges The convolutional kernels well known for their edge detection behaviors (LeCun et al.) 1998). Applying masks creates large number of parasitic edges in the images. The feature maps generated by edge-detecting kernels are drastically distorted; hence, these kernels are suppressed during training by siamese networks. More severally, these parasitic edges will remain the intermediate feature maps and affect all the convolutional layers. In Figure Ic we visualize the encoder's ss first convolutional layer kernels pretrained with standard augmentation or masked input. Due to the parasitic edges, many kernels collapsed to trivial blank features. ViTs dodge this problem by simply matching the patch boundaries to the mask.
METHODS
5.1 PREPROCESSING
To solve the problems caused by masking in siamese networks with ConvNets backbones discussed we propose to apply a high-pass filter during the preprocessing stage. See Figure 2 in Section 4
High-pass filter allows zeros represent null information With high-pass filter, the values in the input image now represent null information instead a regular pixel. Therefore, by applying an extra mask with zero value perfectly fit the value distribution and results minimal information distortion change to the regular pixels.
High-pass filter elimiates parastic edges- We observe that the masking edges the high-pass filtered image becomes invisible. See Figure 2 Empirical results also verify that the ConvNets encoder is able to learn useful edge detection kernels.
Figure Preprocessing: high-pass filter able solve the problem caused by masks on ConvNets. The high-pass filter shifts the pixel value distribution so that zeros represent null information. These zeros mean that the original information can be interpolated from surrounding pixels. addition, the mask edges the image become invisible after applying the high-pass filter.
We quantitatively show that a high-pass filter extra preprocessing step significantly improves siamese networks' performance with masked inputs. We start with applying two random grid masks with grid size 32 the same random crop with fixed 30% masking ratio. We do not apply any other
4

---

Under review as a conference paper at ICLR 2023
Augrinnation
(a) noisy mask
(b) independent
(c) focal mask
(d) asymmetry
(e) multicrops
Figure 3: Masking Strategy: We present five effective design components that allow siamese networks to benefit from masked input. These design components are general general-purpose purpose and can work with various input formats.
augmentations, such as random resized crops color distortions. This mask-only setting achieves a non-trivial 21.0% linear probe accuracy on ImageNet. By adding high-pass filter, the accuracy improves 30.2%. We empirically find that a smaller high-pass filter kernel a 5 is optimal, see Appendix B
One may suspect that high-pass filter will eliminate low-frequency information and results in learning worse representations. We empirically show that high-pass filter will cause negligible change in terms of performance. We conduct supervised training experiments with without high-pass filter and evaluate their image classification accuracy. A ResNet-50 with a high-pass filter reaches 76.2% ImageNet supervised Top-1 accuracy, while the one without a high-pass filter gets 76.0%. These two numbers are within an error range.
5.2 MASKING STRATEGY
Besides standard augmentations, adding masks could prevent the encoder fo(-) from learning some superficial features, as discussed in Sec 5 but at the same time, masking might also introduce new superficial features that confuse the encoder. We propose several masking designs to prevent such behaviors.
Noisy Mask Inspired by masked language modeling Devlin et al. (2019) where masked tokens were replaced by random tokens with certain probability, we find that adding Gaussian noise to the masked area is beneficial and reduces the siamese networks' reliance on augmentations such as color jittering. A noisy mask prevents the network from easily overfitting to the color histogram the entire feature, which similar to the role of applying color-jittering on the unmasked area (Chen al. 2020a).
Channel-Indepeedden Mask In a discrete space, it is natural to mask the entire patch or token. However, for continuous input with ConvNets backbone, it is essential for the network to attend to not only spatial correlation but local channel-wise dependency. In standard siamese networks, this is done by handcrafting augmentations such color-jittering and grayscale. Here, we propose add such functionality for masks by introducing a Channcl-indepeneee spatial-wist masking, where we apply the same mask on three color channels, we generate three random masks and apply them to each color channel separately. Thus, this design is agnostic to input format as it blurs the difference between spatial and channel dimensions. We suspect that this design can further improve high-din seth-dimmnsionall input such as RGBD images or videos.
mask. In addition to standard
Combining Random Masks and Focal Masks siamese networks with ViTs, AsTs,Assran Assran al. (2022) show that mixture of random grid masks and focal masks can significantly boost the performance of the siamese network. This is also similar to (Bao et al. 202T) where larger blockwise masks are combined with randomly smaller ones. Compared to random grid mask, focal mask decreases the probability of similar augmented views of images from the different latent classes. Intuitively, the larger patch is, the smaller the probability that images from the different latent classes could both generate the patch. Therefore, the siamese network will be encouraged to learn semantic-level features. In our design, we randomly apply focal masks distributed grid masks for each view.
Asymmetric Augmentation Wang et al. (2022) highlighted the importance of asymmetry for siamese networks in augmentations various siamese network frameworks. Here, find that
5

---

Under review as a conference paper at ICLR 2023
embeddings
t~ T
fo
Z
input
preprocessing
masking
encoder
L(Z,z)
t'~ Tx
fe Z'
Figure 4: Masked Siamese ConvNets (MSCN) framework. MSCN first generates multiple views from the preprocessed image using a series of standard augmentations. Then it applies random masks on each view using our masking strategy. ConvNet encoder computes the representations of these masked views. Then it applies joint-embedding loss is applied to pairs of these representation vectors.
masking asymmetrically between the two branches in the siamese networks improves learned rep- resentations. This has also been used in masked siamese networks with VITs (Assran et al., (002) that one branch always takes unmasked view. Empirically, branches essential for good representations, especially for frameworks that do not use momentum encoders (He al., 2020; Grill et al.) 2020). In our design, we will apply masking alongside other augmentations on both branches with different magnitudes.
find that applying masking both
Multicrops Besides the main views used in siamese networks, Multicrops (Caron al. 2020 feed additional views for each image and apply siamese networks between some pairs of them. Multicrops have been proven beneficial in various frameworks. Instead of relying on small resolution views used in previous methods (Caron et al. 2020; Assran et al. 2022), we simply apply multiple views with the same resolution and apply siamese network loss on every pair of them. Specifically, we propose to use two extra masked views. Thus, our strategy does not assume input format.
5.3 MASKED SIAMESE CONVNETS FRAMEWORK
The overall Masked Siamese ConvNets (MSCN) architecture is shown in Figure 4 This framework can be a simple add-on to various siamese network frameworks with arbitrary backbone architecture. During preprocessing, the high-pass filter is conducted after normalization. During the augmentation stage on each branch, the masking applied after other augmentations.
RESULTS
In this section, we evaluate the representations obtained by Masked Siamese ConvNets pretrained on the ImageNet- KK dataset (Russakovsky et al. 2015) with ResNet-50 backbone (He et al.) (0166). We train MSCN with different jmbelden losses, namely SimCLR and BYOL. The network pretrained for 800/1000 epochs, including 10 epochs of warm-up, and uses a cosine learning rate schedule. We use the LARS optimizer (You et al., 2017) with a batch size 4096. All the hyperpa- rameters, including learning rate, closely follow the original SimCLR and B BYOL implementation so that we could have a fair comparison. For each model, the pretraining is distributed across 32 V100 GPUs and takes approximately 180 hours. We list all the pretraining and evaluation details in supplementary material.
6.1 IMAGE CLASSIHICATION
We first evaluate the representations on the ImageNet-IK IK dataset using linear probe and semi- supervised classification. We compare MSCN with baselines in Table

---

Under review as a conference paper at ICLR 2023
For linear probe, train linear classifier 100% of the labels with frozen weights. MSCN improves SimCLR but performs slightly worse with the BYOL baseline. We suspect BYOL may require different asymmetric masking setting due to its asymmetric nature of the framework.
For semi-supervised classification, we finetune the network using 1% of the labels. MSCN demon- strates superior performances. The advantage of masked siamese networks in few-shot image classification tasks has also been observed by Assran et al. (2022) using ViTs.
We compare the effect of masking ConvNets and ViTs in Table 2 backbone demonstrates similar behaviors to MSN with a ViT backbone.
MSCN with ConvNet
Table Image Classification on ImageNet-IK. We first evaluate the MSCN representations with ImageNet- Ik classification. For linear evaluation, we freeze the backbone and train linear classifier with all the labels. For low-shot eemi-supervised learning, we finetune the network with 1% of the labels. MSCN demonstrates improvement over baselines on both SimCLR and B BOL frameworks on both settings.
Method
Linear Evaluation
Finetuning
Label fraction
100%
1%
SimCLR (Chen et al.
2020a)
69.5
48.3
+MSCN
71.5 (+2.0)
54.0 (+5.7)
BYOL (Grill et al. +MSCN
2020
74.3
53.2
74.5 (+0.2)
55.2 (+2.0)
Table 2: Effect of Masking on ConvNets and ViTs. We compare the effect masking on ConvNets and ViTs ImageNet- IK with linear probe. MSCN with ConvNet backbone demonstrates similar behaviors to MSN with a ViT backbone.
Method
Architecture
Parameters
Use Mask
Accuracy
Supervised (Touvron et al.
2021
ViT-S
22M
79.9
DINO (Caron et al. 202T)
VIT-S
22M
X
78.3
MSN
Assran al. 2022)
VIT-S
22M
76.9
Supervised (He et al.
2016)
ResNet-50
24M
76.5
BYOL (Grill al. 2020
ResNet-50
24M
74.3
MSCN (ours)
ResNet-50
24M
74.5
6.2 TRANSFER LEARNING
We then evaluate the representations by transferring the network to other downstream tasks. We report the transferred image classification results on iNaturalist 2018 (Horn et al. 2018) dataset and Places-205 (Zhou et al.] 2014) dataset in Table 3 In Table 4 we report the object detection and
instance segmentation performance on VOC07+12 Everingham et al. 2009) and COCO datasets (Lin
et al. 2014).
Comparing MSCN with other methods in Table We observe similar results as Table I that MSCN improves SimCLR but performs worse with BYOL. We still suspect that the worse performance of BYL could come from the non-optimal masking hyperparameters. Since most of the objects in Places-205 have larger scale than the objects in iNaturalist 2018, the non-optimal masking hyperparameters cost more harm on the smaller scale objects than on larger-scale objects.
For the object detection and instance segmentation tasks, MSCN demonstrates superior performances over previous siamese network frameworks on VOC07+12 detection task and performs comparably to the state-of-the-a oortt--rtt representation learning methods on COCO dataset.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Image classification transfer learning with ResNet-50 pretrained on ImageNet-IK. We follow the standard evaluation protocol that trains the linear classifiers on fixed features with the same hyperparameters other methods except for the learning rate.
Method
Place-205
iNat18
SimCLR Chen et al. (020a)
52.5
37.2
+MSCN
53.8 (+1.3)
38.2 (+1.0)
BYOL Grill et al. 2020)
54.0
47.6
+MSCN
54.5 (+0.5)
45.7 (-1.9)
Table 4: Object detection and instance segmentation transfer learning with a ResNet-50 pretrained on ImageNet- All VOCO7+12 results using Faster R-CNN (Ren al. 2013) with C4 backbone variant (Wu et al. 2019) finetuned 24K iterations. All COCO results using Mask R-CNN (He et al.) 2017) with C4 backbone variant (Wu et al. (2019) finetuned using the Ix schedule. Methods with an asterisk are reproduced by Chen & He (2020). 0.3 within the best are underlined
Method
VOC07+12 det
COCO det
COCO instance seg
APall
AP50
AP75
Apbb
Apbb
APbb
Apmk
Apmk 50
APmik
Supervised
53.5
81.3
58.8
38.2
58.2
41.2
33.3
54.7
35.2
MoCo v2
57.4
82.5
64.0
39.3
58.9
42.5
34.4
55.8
36.5
SWAV
56.1
82.6
62.7
38.4
58.6
41.3
33.8
55.2
35.9
SimSiam
57
82.4
63.7
39.2
59.3
42.1
34.4
56.0
36.7
Barlow Twins
56.8
82.6
63.4
39.2
59.0
42.5
34.3
56.0
36.5
SimCLR
55.5
81.8
61.4
37.9
57.7
40.9
33.3
54.6
35.3
BYOL*
55.3
81.4
61.1
37.9
57.8
40.9
33.2
54.3
35.0
MSCN
57.5
83.0
64.4
39.1
59.1
42.1
34.2
55.7
36.4
6.3
ABLATION STUDY
We conduct ablation experiments to gain insights into our masking design strategy. By default, pretrain MSCN for 100 epochs with the SimCI. framework. We measure the performance by linear probe accuracy on ImageNet-1K.
We first explore the optimal masking ratio in Figure 5a A small masking ratio of 15% is optimal for ResNet-50 backbone. This matches the observation in (Assran et al. (022)) that smaller networks (e.g. ViT-S) prefer smaller masking ratio. We also observe that the accuracy is relatively stable against the masking ratio up to 50% with our masking strategy.
Masking Ratio
Masking Grid Size Mask grid size is an important hyperparameter that controls the balance between local and global features in the input, and determine what the siamese network will learn. masked siamese networks with ViTs, the masking grid size is fixed, and it is always boundaries. However, an optimal masking grid size can vary. In Figure 5bb we show that the siamese networks benefit from more appropriate mask grid size. We observe a large grid size of 32 is optimal for our current masking strategy with ConvNets backbone.
match the patch
Focal Mask Probability We explore the optimal focal mask probability in Figure 5c focal masks with random grid masks leads to significant improvements. remains an interesting open direction on how to optimally mix local features with global ones.
Combining
Independent Mask Probability We explore the optimal independent mask probability in Table 5d Independent mask with high probability results in significant accuracy boost.
DISCUSSION AND FUTURE DIRECTIONS
General-purpose Siamese Networks Human desnandd augmentations leveraging domain knowl- edge essential for siamese networks to learn useful representations. Similar to masked siamese
8

---

Under review as a conference paper at ICLR 2023
ImageNet uccumey (%)
68
67.4
67
66
67.3
64
ImageN of
67.2
63
67.1
10
30
50
70
90
7
14
28 32
56
masking ratio (%)
grid size
(a) masking ratio
(b) grid size
67.7
ImageNer sccuracy
67.6
Nntggeet accumey
67.5
67.4
67.4
67.3
67.3
67.2
67.2
67.1
67.0
67.1
local mask probability (%)
independent mask probability (%)
(c) focal mask probability
(d) independent mask probability
Figure 5: Ablation Study. (a) a masking ratio of 15% is optimal for masked siamese ConvNets; (b) masked siamese ConvNets prefer large masking grid size of 32: (c) focal mask probability of 40% is optimal; (d) independent mask with high probability benefits siamese networks.
networks with ViTs (Assran et al. 2022). our approach still combines standard augmentations with masking by proposing masking as an extra augmentation. However, desirable find lonaaii--kowledgee-ee augmentation strategy so that this approach can be applied to more general domains or outtof-distribution scenarios. And it reasonable to believe that the importance of domain-knowledge-speciff augmentations will diminish with increasing amount of pretraining data.
Masked ConvNets for Masked Image Models Besides siamese networks, masked image modeling based on denoising autoencoders (Vincent et al. 2008 2010) is another promising approach for visual representation learning. These methods (Bao et al. 2022 He et al. 2021) have demonstrated impressive performances in visual representation learning. Unfortunately. masked autoencoders also fail to work with ConvNets caused by similar problems mentioned in Sec 4 for siamese networks. Even though the mask here serves a different purpose, we suspect that our design in Masked Siamese ConvNets may also apply masked image modeling with ConvNets. We hope the discovery in this paper may shed light on general self-supervised learning and reduce the requirement for inductive bias of different architectures.
8 CONCLUSION
This work explores whether masking can be applied as an general-purpose augmentation to siamese networks with arbitrary backbones, including ConvNets. We first present the problems introduced by use masking as augmentation. We then propose an effective masking strategy and demonstrate its effectiveness on various siamese network frameworks. Our method performs competitively on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.
REPRODUCIBILITY STATEMENT
The pretraining code can be found in the supplementary material. We also provide detailed implementation setup for pre- pre-training and downstream experiments in Appendix[A] After publication, will provide pretrained checkpoints and open-source the code public repo

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by pon-parametrically predicting view assignments with support samples. arXiv preprint arxiv:2 1122214444.3333.. 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, and Nicolas Ballas. Masked siamese networks for label- efficient learning. arXiv preprint arxiv:2204.07144 2022.
Alexci Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for selef-up learning of speech representations. arXiv preprint arxiv:2006 11477, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arxiv:2202.0333 03555, 2022.
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arxiv:2106.0822 2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ArXiv, 0os/2106.082555 2022.
Adrien Bardes, J. Ponce, and Y. LeCun. Vicreg:
inwariancc-covvriinnne regularization for self-supervised learning. arXiv preprint 1xxiv22005.04906, 2021.
Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In Int. J. Pattern Recognit. Artif. Intell., 1993.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vone Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arxiv:2005. 14165, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arxiv:2006.0988) 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint Srhi:2104414294 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arxiv:2006. 10029, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arxiv:2011.10666 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9620- 9629, 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arxiv:2003.04220 20220.
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprini arxiv:2010. 1999,, 2021.
Mark Everingham, Luc Van Gool, Christopher K. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-335 2009.
Priya Goyal, Dhruv Kumar Mahajan, Abhinav Kumar Gupta, and Ishan Misra. Scaling and bench- marking self-super visual representation learning. 20/9 /EEE/CVF International Conference on Computer Vision (ICCV), pp. 6390-6399, 2019.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Mangalore Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features in the wild. arXiv preprint arxiv:2103.01956 202 la.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu. Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Ishan Misra.
Vissl. mittps://ge thub. cww. facebookccoooooooa rew.eaachhkeeeereeeceacaa viss]
2021b.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models more robust and fair when pretrained uncurated images without supervision. arXiv preprint arxiv:2202.08.8 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint 2rxxv:20006.77333 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Patlern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2966 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9735-9735, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. arXiv preprint vexiv:21/1.06377 2021.
Dan Hendrycks. Mantas Mazeika, Saurav Kadavath, and Dawn Xiaodong Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8769-8778. 2018.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self- supervised learning. arXiv preprint
2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proc. IEFF, 86:2278-299 23244, 1998.

---

Under review as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.0354. 2022.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6706-6716 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences of the United States of America, 118, 2021.
Yu Rong, Yatao Bian, Tingyang Xu, Wei yang Xie, Ying Wei, Wen bing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. arXiv preprint arXiv: 2007.02833 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115: 211-252, 2015.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In INTERSPEECH, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /00.03/887.03448, 2018.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint 1706.077......... 2017.
Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre- Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML '08, 2008
Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371-3408, 2010.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. arXiv preprint arxiv:2204.006.1 2022.
Yuxin Wu, Alexander Kirilloy, Francisco Massa, Wan- Yen Lo. and Ross Girshick. Detectron2. 2019.
hwweeeeewo.eeeeeeeeeeeeeeeeeeeee sttps://qitte com http:///..haabccccaaaaaaaeeeeeee
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arxiv: 1708,03888 03888, 2017.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arxiv:2103 103.03230, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arxiv:2205.0/000 2022.
12

---

Under review as a conference paper at ICLR 2023
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NIPS, 2014.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Loddon Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arxiv:2111.07882. 2021.
13

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAIL
A.1 PRETRAINING
We closely follow the original setting (Chen et al. 2020a> for our MSCN (w/ SimCLR) pretraining and original setting in (Grill al. 2020 for our MSCN (w/ BYOL) pretraining.
Augmentation For both methods, we use the same augmentation methods. Each augmented view is generated from a random set of augmentations from the same input image. We apply series of standard augmentations for each view, including random cropping, resizing to 224x224, random horizontal flipping, random color-jittering, randomly converting to grayscale, and random Gaussian blur. These augmentations are applied symmetrically on two branches.
Architecture For MSCN (w/ SimCLR), the encoder is ResNet-50 network without the final classification layer followed by projector. The projector is a two-layer MLP with input dimension 2048, hidden dimension 2048, and output dimension 256. The projector has ReLU between the two layers and batch normalization after every layer. This 256-dimensional embedding is fed to the infoNCE loss. We use temperature 0.2 for the infoNCE loss.
For MSCN (w/ BYOL), the online encoder is ResNet-50 network without the final classification layer. The online projector is two-layer MLP with input dimension 2048, hidden dimension 4096, and output dimension 256. The predictor is two-layer MLP with input dimension 256, hidden dimension 4096, and output dimension 256. The projector and the predictor have ReLU between the two layers and batch normalization after every layer except the final linear layer. The target encoder and projector are the exponential moving average of the online encoder and projector, with an initial momentum 0.996 with cosine decay schedule 1.0 during the pretraining.
Optimization We follow the training protocol (Zbontar et al. 2021 For MSCN (w/ SimCLR), use LARS optimizer and base learning rate 4.8 using cosine learning rate decay schedule. We pretrain the model for 800 epochs with 10 epochs warm-up with batch size 4096.
For MSCN (w/ BYOL), rate decay schedule. We pretrain the model for 1000 epochs with 10 epochs warm-up with batch size 4096.
a LARS optimizer and base learning rate 3.2 using cosine learning
A.2 LINEAR PROBE ON IMAGENET
We closely follow the setting used (Zbontar al., 2021 for our linear probe evaluation. The linear classifier is trained for 100 epochs with base learning rate of 1.0 and cosine learning rate schedule. We minimize the cross-entropy loss with the SGD optimizer with momentum and weight decay of 10-6, We use batch size of 256.
At training time, we use random resized crops to 224x224, followed by random horizontal flip and a High-pass filter. At test time, we resize the image to 256x256 and center-crop it to size of 224x224 and followed by a High-pass filter.
A.3 FINETUNING
We closely follow the setting used in (Zhontar et al. 2021) for our finetuning evaluation. We finetune the ResNet-50 encoder for 20 epochs with learning rate of 0.002 and the classifier with a base learning rate 0.5. The learning rate is multiplied by factor of 0.2 after the 12th and 16th epoch. We minimize the cross-entropy loss with the SGD optimizer with momentum and do not use weight decay. We use batch size of 256. The image augmentations are the same as in the linear evaluation setting.
A.4 OBJECT DETECTION
We use the detectron2 library (Wu al. 2019) for training the detection models. All the configuration files are from the VISSL library (Goyal et al. 1b). which are closely follow the evaluation settings from (He et al. 2020). The backbone ResNet50 network for Faster R-CNN (Ren et al., 2015) and Mask R-CNN He et al. 2017) is initialized using our pretrained model.
14

---

Under review as a conference paper at ICLR 2023
VOC07+12 We the VOC07+12 (Everingham et al. 2009 trainval set of 16K images for training a Faster R-CNN C-4 backbone for 24K iterations using a batch size 16. The initial learning rate for the model is 0.085 which is reduced by a factor of 10 after 18K and 22K iterations. We use linear warmup (Goyal et al. 2019) with a slope of 0.333 for 1000 iterations.
COCO We train Mask R-CNN C-4 backbone the COCO (Lin et al. 2014) 2017 train split for 90K iterations using batch size of 16 The initial learning rate for the model is 0.05 which reduced by factor of 10 after 60K and SOK iterations. We linear warmup (Goyal al. 2019) with slope of 0.333 for 1000 iterations. We report results on the 2017 val split.
B ADDITIONAL ABLATION STUDY
We conduct additional ablation experiments to gain insights into our masking design strategy. By default, we pretrain MSCN with SimCLR loss for 100 epochs. We measure the performance by linear probe accuracy on ImageNet-1K:
High-pass Sigma- We explore the optimal high-pass filter a in Figure 6 In addition to the varying for pretraining, we also update the high-pase -pas filter a for the transformation during evaluation. In practice, we prefer a small because there less computational overhead.
67.5
ImageNet accuracy
67.1
66.9
66.7
5
7
9
high-pass sigma (pixel)
Figure 6: Impact of high-pass filter parameter a during pretraining on ImageNet-IK 1K linear probe accuracy.
View Sharing In our masking strategy, we apply the standard augmentations to generate multiple views and then randomly apply masks these views. One alternative apply random masks on the same augmented view. Figure shows that applying masks on the same view results in significantly worse representations.
As discussed our design principle, the masks are used to prevent superficial solutions based on masked areas. still important to apply different augmentations to the original image prevent the superficial solutions based on unmasked areas.
Table 5: Impact of view-sharing during pretraining ImageNet- 1K linear probe accuracy (in %). A view is generated with standard augmentations, including RandomResizedCrro ColorJitter, HorizontalFlip. GaussianBlur, and Grayscale. Our standard approach applies masks to two different views. Here, we find that applying masks on shared view results in significantly worse performance.
Augmentation strategy
Accuracy
Same view random mask Different views + random mask
29,1
65.6
C ADDITIONAL VISUALIZATION
15

---

Under review as a conference paper at ICLR 2023
Figure 7: Color histograms of (left) high-pass filtered image and (right) high-passed image with mask.
Figure 8: Visualization of convolutional kernels trained with MSCN
16


Reviews:
[{"review_id": "VoTScmkUCF", "reviewer": "Reviewer_29aC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a pre-processing method to enable the masking strategy available for siamese networks with ConvNet backbone under self-supervised settings. It is motivated by performing random masking would cause missing information and parasitic edge issues. Also, systematic masking strategies are studied, including noisy mask, channel-independent mask (RGB), focal mask, and asymmetric augmentation. Extensive experiments on various tasks are conducted, including semi-supervised classification in IN-1K, transferred image classification results on iNaturalist and Places-205, and object detection & instance segmentation on VOC & COCO. The results show consistent improvements in combining the proposed processing method, high-pass filter, and masking strategies.", "summary_of_the_review": "The reviewer feels that all the empirical studies presented in this paper will likely benefit the relevant community. However, the proposed masking strategies still heavily involve human priors. Also, the reviewer is unsure if the improvements come from introducing the preprocessing method or the hybrid masking strategies. ", "strength_and_weaknesses": "This paper makes the masking strategies available for convolutional-based siamese networks for SSL. The color histogram and convolutional visualization observation about the masking augmentation causing missing information and parasitic edges are interesting. The systematic studying of masking strategies can offer a way for other researchers to follow. Overall the paper is very easy to follow.\n\nThe experiments are conducted thoroughly, including various tasks (semi-supervised classification, transferred image classification, object detection, and instance segmentation) and datasets (IN-1K, iNaturalist, Places-205, VOC, and COCO). Results show the proposed method helps the standard SSL methods achieve higher performance. The related ablation studies are included to prove the effectiveness of different masking strategies. \n\nThe proposed preprocessing method may reduce the information in the image, which will cause the performance drop. In the current manuscript, all the comparisons include both the preprocessing methods and the comprehensive masking strategies; the reviewer is not clear if the preprocessing method brings improvements or masking strategies bring improvements. It would be better to test SimCLR, SimCLR + MSCN, BYOL, and BYOL + MSCN performance with simple/standard masking strategies. \n\nHow does the proposed processing method change the color histogram and learned convolution kernels, as shown in Figure 1? Since the proposed method is motivated by those observations, it would be more self-contained to include the updates once introducing the new methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U6SdcLNhnRa", "reviewer": "Reviewer_Kw3b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a general masking strategy for visual representation learning with siamese neural network architectures. The approach is applicable to CNN-based or ViT-based neural networks.\nIt consists in pre-processing the input with a high-pass filter and then doing different augmentations that mask out parts of the input images with a certain probability, inspired from the natural language processing domain.\nExperimental results have been applied to different baseline models, notably to SimCLR and BYOL, and show an improved performance using the pre-trained models for image classification on ImageNet, and transfer lerning for classification, detection and instance segmentation on other datasets.\n", "summary_of_the_review": "The presented method is interesting and relatively novel although largely inspired from other papers using masking. The results show the benefit of the proposed approach for self-supervised representation learning with general siamese neural network architectures. But the presented results lack some detail and rigor.\n\n", "strength_and_weaknesses": "Strengths:\n- Good experimental results\n- Genericity of the approach\n\nWeaknesses:\n- The description of the approach is unclear at some places\n- Some of the results are not clearly presented\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HoSkU599r5", "reviewer": "Reviewer_nzgE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nSummary:\nThis paper explores solving the masking augmentation problem in convolutional neural networks. This topic is very important. The authors' method is also very clever. However, unfortunately, its performance is not satisfactory. Moreover, it does not solve the more important problem of masking image modeling. Therefore, this article is a borderline article, leaning towards being rejected. If the authors can answer my important questions, I will consider raising the rating. If this article can solve the problem of masking image modeling, I will give it a high score.\n", "summary_of_the_review": "\nSee \"Summary Of The Paper.\" This article is a borderline article, leaning towards being rejected. If the authors can answer my important questions, I will consider raising the rating. If this article can solve the problem of masking image modeling, I will give it a high score.\n", "strength_and_weaknesses": "\n(Positive) The subject studied in this paper is important. Exploring the role of masking as augmentation in self-supervised learning of convolutional networks is encouraged.\n\n(Positive) After analysis, the authors believe that convolutional networks cannot correctly ignore the null information of masked areas through the attention mechanism like VIT. This is very reasonable.\n\n\n(Positive) The authors argue that masking brings artificial edges, which will hurt the learning of convolutional neural networks. This is very reasonable.\n\n\n(Negative) When the authors discuss Siamese networks, their formulation ignores the commonly used and very important projectors in SSL algorithms. Please add a projector to the formulation.\n\n(Positive) Regarding the principles that augmentation needs to adhere to in self-supervised learning, the author's conclusion is correct: Preventing easy solutions is very important; the augmentations are hoped to be class-aware.\n\n\n(Positive) The authors use high-pass filtering to suppress the null information in the masked area, which is very clever.\n\n\n(Positive) The authors use high-pass filtering to suppress artificial edges, which is also very clever.\n\n\n(Negative) However, in the masking strategy, the author uses a noisy mask. I think this contradicts the motivation of the authors. We all know that noise masks, especially Gaussian noise, bring a lot of high-frequency signals. High-pass filtering preserves these signals. The authors claim that this kind of noisy mask is useful, isn't that slapping one's own face?\n\n\n(Positive) The Channel-Independent Mask proposed by the authors is a bit interesting.\n\n(Negative) The caption \"Few-shot Semi-supervised Learning on ImageNet-1K\" in Table 1 is misleading. Also included here is an evaluation of the very important linear probing in SSL. Please modify this caption.\n\n\n(Negative) The proposed method has obvious gains on SimCLR. But the gain is not obvious on BYOL.\n\n(Negative) The application of the proposed method in ViT even leads to performance degradation, which is very worrying.\n\n(Negative) In the Object detection and instance segmentation tasks, the authors are missing one of the most important comparison objects, which is the baseline on which the authors' method is based. I'm guessing it could be BYOL or SimCLR. Given that the authors' method doesn't gain much, I'm worried that the authors' method won't outperform this baseline.\n\n\n(Negative) The authors have repeatedly emphasized that masking is the most general augmentation. This is worrying. This expression is too subjective.\n\n\n(Negative) In fact, the biggest challenge of Masking augmentation in convolutional networks is masked image modeling. Unfortunately, after reading the whole article, I found that the authors' method does not solve this problem. This is quite disappointing for readers.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject