Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling
Authors: Keyu Tian
Yi Jiang
qishuai diao
Chen Lin
Liwei Wang
Zehuan Yuan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Self-Supervised Learning
Masked Autoencoding
Masked Pre-training
Masked Modeling
Convolutional Neural Networks

Abstract: We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, randomly masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet’s hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method, called Sparse masKed modeling (SparK), is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). The improvements on object detection and instance segmentation are more significant (up to +3.5%), validating the strong transferability of features learned. We also find SparK’s favorable scaling behavior by observing more gains on larger networks. All of these findings support the promising future of generative pre-training on convnets. Both codes and pre-trained models have been released at https://github.com/keyu-tian/SparK.

Published as a conference paper at ICLR 2023
DESIGNING BERT FOR CONVOLUTIONAL NETWORKS: SPARSE AND HIERARCHICAL MASKED MODELING
Keyu Tian 1..3. Chen Lin',
Yi Jiang" Liwei Wang"*,
Qishuai Diao?
Zehuan Yuan
'Center for Data Science, Peking University PPazhou Lab (Huangpu)
"Bytedance Inc
*University of Oxford
keyutian edu..u en, thiangyi.enji diaoqi shooishuaillc @ chen. lin@eng. ox. ax.ac.ub uk, wanglwepku.edu cn, yuazzehaanbbytddkcccccom
ABSTRACT
We identify and overcome two key obstacles in extending the success of BtRT-style pre-training, or masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, randomly masked input images; (i) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), treat unmasked pixels sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop hierarchical decoder reconstruct images from multi-scale encoded features. Our method, called Sparse masKed modeling (SparK), is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-bart
masked modeling by similarly large margins (around + +.0%). The improvements on object detection and instance segmentation are more significant (up to +3.5%), validating the strong transferability of features learned. We also find SparK's s favorable scaling behavior by observing more gains larger networks. All of these findings support the promising future of generative pre-training on convnets. Both codes and pre-trained models have been released at https: /ggthtb....... com/ keyu- tian Spark.
INTRODUCTION
The pretrain-finetune paradigm in natural language processing (NLP), exemplified by BERT and GPT (Devlin et al., 2018; Clark et al., 2020; Radford et al., 2019; Brown et al., 2020), is remarkably effective and thus long envied by our vision community. It is the emerging masked image modeling (Bao et al., 2021; He et al., 2021; Xie et al., 2021; Chen et al., 2022) initially extends the success of BERT from language transformers to vision transformers (ViTs). A bold move that increases the mask ratio to a staggering level (60-75%) is largely credited with this success (He et al., 2021; Xie ViTs (Dosovitskiy et al., 2020;
al., 2021). ). As result, the field of visual self-supervised learning Liu et al., 2021) 1) has now shifted from contrastive learning (Grill et al., 2020; Chen et al., 2021; Caron al., 2021) to BERT-style masked modeling or fusion of the two (Zhou al., 2021).
Despite this progress, extending the success of BERT pre-training from transformers to convolutional networks (convnets) remains desirable but unrealized goal. Early pioneering work (Pathak et al., 2016) predated BERT but performed much worse than supervised pre-training. Although there have been efforts over the past year to port BERT convnets, they ultimately compromise by proposing a non-convolutional model (Gao et al., 2022) or non-masked modeling (Fang et al., 2022). One might therefore wonder: what exactly is impeding the application of BERT to convnets?
We try to conclude that in essence, the difficulty is rooted in the fundamental differences in data processing between language and vision (Bateman, 2014; Cheng et al., 2022). While typical NLP models like recurrent networks or transformers process text variable-length sequence of words (well-defined semantic units), convnets have to recognize objects of different sizes from raw pixels ("units"" different scales). This large disparity rises two challenges: (i) Removing the information
"Corresponding authors: jiangyi.enjoy @ bytedance.comm wanglw @pku.edu..n

---

Published as a conference paper at ICLR 2023
Transformer
Sparse CNN
CNN
Encoding process:
piael Intersity
Data Distribution
Before After
probabblty
Madling:
(4) directly dropping
[B] sero-outing
(4) sparsely dropping
(4) raw input
Figure 1: Diffeeent masking strategies with pixel intensity histograms plotted before (in gray) and after (blue) masking. (b) is a straightforward idca to apply masked modeling to convnets, which results in a distribution shift. (a) illustrates MAE (He et al., 2021) that has no such side effect thanks to the transformer's is ability to process variable-length input. We propose (c) adapt convnets irregular masked input without distribution shift.
of masked "words" is difficult for convnets. In ViTs, input image is divided into non-overlapping patches. Simply dropping masked patches or replacing them with mask tokens can remove the information. This ease relies transformer being able to handle irzegular (variable-lengthh and non- overlapping patches, thus cannot be achieved on convnets as they not only operate on regular grids, but also perform sliding window with overlapping. One may zero-out all masked pixels and feed this "mosaic" into convnet, but this would lead to significant distribution shift (in figure 1) and other issues (discussed further in section 3.1 and figure 3). thus cannot be an ideal solution. (ii) Single-scale algorithms are inadequate for learning multi-scale (hierarchical) features. Multi-scale structures have been gold standard in computer vision, which allows visual processing systems like SIFT descriptors (Lowe, 1999; Bay et al., 2006) and pyramid networks (He et al., 2015; Lin et al., 2017) to handle variations in object scale. In contrast, the masked modeling approach from NLP operates in a single-scale manner. Applying it directly convnets will miss the advantage of model hierarchy.
In this work, we clear the hurdles above and make BERT suitable for convnet by proposing Sparse masKed modeling with hierarchy (SparK). We first randomly mask an image in a patch-wise manner. Observing the sparse nature of point clouds coincides with these unmasked patches, we treat them as flatten point cloud and use sparse convolution for encoding. This enables convnets handle irregular masked images. For decoding and reconstruction, the sparse features are filled with mask embeddings and fed into a multi-scale decoder, leveraging the hierarchical structure of convnets.
SparK is a general method that does not limit the specific encoder to be pre-trained. We test it with two representative convnet famlies: classical ResNets (He al., 2016) and modern Xon (Liu et al., 2022). All models benefit from SparK. with more gains on larger models that demonstrates its favorable scaling ability. On standard downstream tasks (classification, object detection and instance segmentation), convnet-based SparK outperforms both (i) state-of-the-art contrastive learning and (ii) transformer-bant based masked modeling by similarly large margins (around +1.0%). The improvements over COCO baselines are more significant than those on ImageNet (up to +3.5%), indicating the representations learned by SparK are highly transferable. To summarize, SparK provides:
The first pre-training method in the style of BERT that can be directly applied to any convnets without backbone modifications, overcoming their inability handle irregular masked inputs.
The insights into the design of generative pre-training for convnets, e.g., the first of sparse convolution for masked image modeling and a hierarchical design for BERT-style pre-training.
A leap in convnet's performance across downstream tasks with gains of up to 3.5 points, showing the promise of extending the success of transformer's pretrain-finetung paradigm to convnets.
The recent surge of interest in vision transformers (Liu et al., 2021; He et al., 2021) has shifted the focus away from convnets in the computer vision community. However, convnets embody the core principles of many classical vision processing systems, such as seale- and translation-equivrrianee (Lowe, 1999; Csurka et al., 2004). These networks
locality, weight-sharing and serdware-friendinnee continue to be indispensable in addressing a variety of challenging and structural real-world tasks beyond classification (Jaderberg et al., 2015; Liu et al., 2017: 2022). We hope SparK's 's inspiring performance will prompt us to revisit convnets as generic backbones for computer vision community, and motivate more future arts in exploiting their potential through generative pre-training.
2

---

Published as a conference paper at ICLR 2023
2
RELATED WORK
2.1 HIERARCHICAL VISUAL PROCESSING SYSTEMS
Hierarchical structure is acknowledged as gold standard for visual representation systems. Many fundamental handcrafted feature descriptors (Lowe, 1999; Bay et al., 2006; Rublee et al., 2011) extract multi-scale visual representations via scale-space extremum on feature pyramid (say, octave). The crux behind this hierarchical design is to extract scale-invariant (or equivariant) features, thus, allows the system to cope with varying object sizes (scales). Widely used in visual tasks (Felzenszwalb al., 2008; Yang et al., 2009). these descriptors also motivate the design principles of convolutional networks (He et al., 2016; Tan & Le, 2019; Liu et al., 2022). Some recent arts also elaborately design hierarchical modules that allow the information aggregation at different granularities better tackle detection and segmentation tasks using convnets (Long et al., 2015; Liu et al., 2016; Lin et al., 2017).
2.2 RECENT PROGRESS ON VISUAL SELF-SUPERVISE LEARNING
Recently, the contrastive learning formulates self-supervise learning an instance classification task (Van den Oord et al., 2018; He et al., 2020; Chen et al., 2020a). Efforts have been made (Grill al., 2020; Caron et al., 2020; Chen & He, 2021) to overcome the core issue of mode collapse. More advanced methods are developed since then (Tian et al., 2020; Zbontar et al., 2021; Li et al., 2023; Chen et al., 2021), and this line of work had dominated the area of visual unsupervised learning until masked generative pre-training along with the vision transformer architecture came into view.
Masked image modeling, inspired by the recent success of masked language modeling in natural language processing (NLP) (Devlin et al., 2018; Liu et al., 2019), has attracted growing interest for visual pre-training The pioneering work (Bao et al., 2021) pre-trains vision transformers by learning to predict token indices of masked patches. He et al. (2021) takes advantage of transformer's 's ability to handle variable-length inputs and implements an efficient and scalable method. Both He al. (2021) and Xie et al. (2021) regress raw RGBs to simplify the pre-training, while Wei et al. (2022) and et al. (2022) selects HOG (Dalal Triggs, 2005) or frequencies as targets due to their rich semantics structures. Gao et al. (2022) designs transformer with a heavier patchifier to perform masked modeling. Zhang et al. (2022) have verified this idea in 3D computer vision. So far, enormous studies have successfully verified the efficacy of these algorithms 2021; Chen et al., 2022). However, on the other hand, their methodology is almost the same as that in NLP (Devlin et al., 2018; Liu et al., 2019), and is therefore difficult to be used for hierarchical convolutional models- on convnets, contrastive learning still remains state-of-the-art.
vision transformers (Zhou et al.,
2.3 SPARSE CONVOLUTION FOR VISUAL REPRESENTATION
Convolution is widely used in 2D computer vision (Dalal & Triggs, 2005; He et al., 2016), which typically performs sliding window on regular grids (pixels). When facing with 3D point clouds, this operator quickly becomes unaffordable due to the cubic increasing number of grids (voxels). Considering point clouds are highly sparse and irregular, one can skip all empty voxels for speed. This motivates the sparse convolution (sparseconv) (Liu et al., 2015), which is heavily used in modern convnets for 3D visual tasks (Zhou & Tuzel, 2018; Sindagi et al., 2019). Minkowski Engine (Choy al., 2019) is one of the most common sparseconv frameworks. Some prior arts (Verelst & Tuytelaars, 2020) also tried to introduce sparseconv for faster 2D visual understanding. And in this work, we have observed the similarity between 3D point clouds and 2D masked images in BERT-style pre- tre-training. We thus use sparseconv, for the first time, with the purpose of "facilitating the adaptation of convnet BERT masked modeling". rather than of "speeding up the computation convolution"..
APPROACH
Illustrated in figure 2, our SparK framework aims to pre-train convolutional network encoder via hierarchical masked image modeling - masking a portion of image and learning to recover it. We are going to detail SparK by introducing a sparse masking strategy (section 1), hierarchical encoder- decoder architecture (section 3.2), and the optimization target of SparK pre-training (section 3.3).
3

---

Published as a conference paper at ICLR 2023
Densify
Sparsely Gathering
Reconstruction
Hierarchical Encoding
Hierarchical Decoding
Figure 2: Sparse masked modeling with hierarchy. To adapt convolution to irregular masked input, visible patches gathered into sparse image and encoded by sparse convolution. To pre-train hierarchical encoder, we employ UNet-style architecture to decode multi-scale sparse feature maps, where all empty positions are filled with mask embedding. This "densifying" is necessary to reconstruct a dense image. Only the regression loss on masked patches will be optimized. After pre- pre-training, only the encoder is used for downstream tasks.
3.1 SPARSELY GATHERING UNMASKED PATCHES
We start by the patch-wise masking strategy widely used in masked image modeling, An image is divided into several non-overlapping square patches, each of which will then be masked independently with given probability called mask ratio. The key to masked image modeling algorithm is how to eliminate the pixel information from these masked patches.
masked modeling can easily eliminate the information by directly removing masked patches or replacing them with a mask token. This ease relies on the fact that vision transformers are born to handle irregular (variable-length) input and operate on non- non-overlapping image patches. Since convnets cannot do this, new approaches have to be sought. A straightforward idea is to set all masked pixels to zero and feed this image to convnet. This, however, has three evident shortcomings: (i) the computation on masked regions is redundant; (ii) it would disturb the data distribution of pixel values, illustrated in figure 1; (iii) the patterns on mask maps will vanish after applying several convolutions to this zero-out masked image. We examine problem (iii) in figure 3, where we also give our solution. Note that this problem is particularly acute when using modern deep convnets due to the large number of successive convolutional blocks.
Previous transformer-basee
To overcome the problems, propose to sparsely gather all unmasked patches into sparse image, and then use sparse convolutions' to encode it. This strategy: (i) ensures no information is leaked; (ii) can be applied directly to any convnet without backbone modifications; (iii) is efficient as sparse convolution computes only at visible places; (iv) solves the aforementioned issues of "pixel distribution shift" and "mask pattern vanishing". As shown in figure 3, sparse convolution will skip all masked positions on sparse feature maps, and only computes at unmasked points. This helps to prevent the shape of the mask pattern from changing with convolution, thus ensures consistent masking effect and ratio throughout all convolution layers. Another fact is that when fine-tuning, all sparse convolutional layers can be naturally reduced to ordinary dense ones. This is true because dense images are actually the special cases of sparse images that have no "holes".
3.2 HIERARCHICAL ENCODING AND DECODING
By "hierarchical" encoding, we mean the encoder will generate a set of feature maps with different resolutions, namely different scales. Taking a ResNet-style model for example, it typically contains stages each with series of convolutional blocks and downsampling module. The feature resolution
downsampled by factor of 2 after every stage. For an image shaped as H x W, a 5e
produces feature maps at 4 scales with resolutions of T T. g S,, S2, S3, and S4 be these sparse features, respectively. They will be used to decode.
H W H W H
5
1E
W x T6 and H
32 x
W
32 Let
By "sparse convolution". we mean the submanifold sporse convolution that computes only when the kernel center covers a non-empty element. Please refer to Graham & van der Maaten (2017) for more details.
4

---

Published as a conference paper at ICLR 2023
unmasked masked
Raw Mask (binary)
After Ordinary Conv.
Or
After Sparse Conv.
for
ordinory comy. computtes of and
sporse conv. only computes and skips
both
centered
fewer ofter coms.)
pattern shrinks and vanishes due to the erosion offect of ardinory comu.
pattern remains the some thonks sporse cony.
Figure 3: Using sparse convolution to address "mask pattern vanishing" issue. Three mask examples are shown. As in left, when computing ordinary "dense" convolution centered at a zero (masked) position, the result would be non-zero if the filter covers any non-zero (unmasked) points. Repeating this convolution will erode masked regions (zero positions) and dilate unmasked ones, leading the vanishing problem. We sparse convolution overcome this undesired property by skipping all masked positions keeping the mask pattern.
Overall, the decoder follows the design of UNet (Ronneberger al., 2015). We use relatively light decoder that contains three successive blocks (Ba, B2, } with upsampling layers. Before reconstructing dense image, it is necessary fill in all the empty positions sparse feature maps. This is called "densifying" Taking the smallest sparse feature S₁ as example, all empty positions (inactive sites) on S4 are filled with mask embedding [M4] get a dense feature S/. projection layer @4 is applied then, in case encoder and decoder have different network widths:
D4 (S))
So Da is the input of decoder's first block B3. has the same resolution of S4 with W Similarly, W и x
32
we can get D3, D2, and D2 (with shapes of 16
H x via: 8
16
D, = B,( (S))
(ViE (3, 2,
(2)
Note that four different mask embeddings [M4-1] and projection layers Φ4-1 are required: they belong to different scales, and may have different network widths. The final output of decoder D1-
3.3 OPTIMIZATION TARGET AND TRANSFERRING TO DOWNSTREAM
To reconstruct an image from D1, head module h is needed, which should include two more upsampling layers to reach the original resolution of input H x W. As for the reconstruction target, choose per-patch normalized pixels targets with an -o-soss, and calculate errors only masked positions. These designs have been proven to facilitate models to learn more informative features in He et al. (2021), and also verified by the ablation study later in section 4.5.
After pre-training, we discard the decoder and only use the encoder for downstream tasks. When fine-tuning, the pre-trained sparse encoder can be directly generalized to dense images without any tuning, due to the fact that dense input is special case of the sparse, where every position is active.
4 EMPIRICAL RESULTS
4.1 IMPLEMENTATION DETAILS
Components. SparK can use any convolutional network as the encoder, without any special design of the backbone architecture. We implement SparK with two of the most representative convnet families: classical ResNets (He et al., 2016) and modern ConvNeXts (Liu al., 2022). One can easily test SparK on other convolutional architectures as well. As for the mask embeddings [M4-1]; J, we implement them random- initialliiiied learnalle feature vectors. For decoding,
lightweight UNet decoder. See Appendix A for its detailed structure. Positional embeddings are not used since convnet already encodes the spatial information. We also test this in the ablation study (section 4.5).
5

---

Published as a conference paper at ICLR 2023
Simple implementation of pre-training For simplicity the same hyperparameters used for all architectures (ResNets, ConvNeXts) and model sizes, even though tuning each may improve our fine-tuning performance at face value. All models pre-trained with 1.28 million unlabeled images from ImageNet- K (Deng et al., 2009) training set for 1600 epochs. Only the minimal augmentation is required (random cropping and horizontal flipping). We use the same mask patch size (32) and ratio (60%) as in SimMIM (Xie et al., 2021). We train with a LAMB optimizer (You et al., 2019), batch size of 4096, and a rosine-annealing learning rate with peak value 0.0002 batchsize/255
Fine-tuning, We use the official implementations of ResNet (Wightman et al., 2021), MoCoV2 (Chen et al., 2020b). and ConvNeXt (Liu et al., 2022) to fine-tune, See Appendix and D for recipes.
4.2 IMAGENET EVALUATION
Performance comparison with self-supervised transformers. We first validate SparK ImtgeNet with the pure convolutional model ConvNeXt (Liu et al., 2022). Smaller models (ViT, Swin, ConvNeXt)-S and the bigger ones (VTT, Swin, ConvNeXt)-B compared separately. By comparing the results vertically in table I, one can find the convolutional models, with SparK pre-training, overwhelmingly outperform transformer-basee though SparK neither employs external models (DALL-E dVAE (Ramesh et al., 2021)), nor profits from advanced (MIM+CL) pre-training. This is somewhat surprising since transformers are well- known data-hungry models with much less inductive bias than convnets, and therefore are considered to benefit more from large-scale self-super training. The result here conveys a new message: convnets may have much more potential than expected, and their capability in visual representation may not be inferior to that of transformers. The key may depend on how to use powerful pre-training algorithms (e.g., SparK masked modeling) to turn this potential into capability.
pre-training methods by large margins (+00772.7),
Efficiency. Similar to MAE (He et al., 2021), SparK has the advantage of encoding efficiency, more images in forward pass. For instance, DINO and iBOT by default (Caron et al., 2021; Zhou et al., 2021) use multi-crop with global crops of 224 224 and 10 locals of 96 96, leading + 10 (96/224) 2 ss 3.8 times the cost of single image encoding. In contrast, SparK requires only 40% of the theoretical overhead thanks to the sparsity of masked imput: 60% of patches are masked, and sparse convolution only processes the rest, In practice, we found a sparse ResNet-50 can save -23% memory footprint (26.4 GB vs. 34.5 GB for single batch size of 128). This allows to train on a 32GB Tesla V100, which otherwise impossible for non-sparse pre-training. The efficiency also helps SparK scale up more easily.
especially compared to contrastive learning that encodes
Table I: Comparing SparK and self-supervised transformers on ImageNet. All methods pre-train on ImageNet-IK an fine-ture with the resolution of 224. Top-1 I validation accuracy is reported, the best results in bold. "Extra model" indicates whether DALL-E's dVAE (trained on 250 million extra data) is used in pre-training. Entries with are quoted from Zhou et al. (2021). $ our reproduction using the official codes.
PT
Enc.
Extra
Small backbone
Base backbone Arch.
Pre-training method
task
cost
model
Arch.
Acc.
Acc.
Vision Transformer Backbone
MoCov3 (Chen al., 2021)
CL
5.0x
ViT-S 81.4
ViT-B
83.2
DINO (Caron et al., 2021)
CL
9.5x
ViT-S
82.0
ViT-B
82.8
BEiT (Bao et al., 2021)
MIM
2.5x
VIT-S
81.4
VIT-B
83,2
CIM (Fang et al., 2022)
MIM
2.5x
ViT-S
81.6
ViT-B
83.3
CAE (Chen et al., 2022)
MIM
2.5x
/
ViT-S
81.8
VIT-B
83.6
MAE (He et al., 2021)
MIM
0.6x
VIT-S
81.5
VIT-B 83.6
SimMIM (Xie et al., 2021)
MIM
2.5x
ViT-S
81.7
VIT-B 83.8
iBOT (Zhou al., 2021)
MIM+CL
9.5x
ViT-S
82.3
VIT-B 84.0
SimMIM (Xie et al., 2021)
MIM
2.5x
Swin-S
83.4
Swin-B
84.0
Convolutional Backbone
SparK (ours)
MIM
1x
ConvX-S 84.1
ConvX-B
84.8
2-Fffeective epoch" takes into account the total amount of images processed in pre-training. For instance, typical contrastive learning encodes images per forward pass, the effective epoch is twice the literal value.

---

Published as conference paper at ICLR 2023
Table 2: Comparing convnet- SparK with transformer-based self-supervised learning on down- stream tasks. On ImageNet, the same tine-tuning resolution of 224 is used. On COCO, Mask R- R-CNN with FPN is equally applied. All methods follow 3 COCO schedule (36 epochs). while MAE fine-tunes longer (50 epochs). Average precisions of detection box (AP*)) and segmentation mask (AP")) va12017 reported. reproduced using the official codes, since Liu al. (2022) only runs models with Cascade Mask R-CNN.
Eff.
Cls.
Det.
Seg.
Pre-training method
Arch.
epoch
Acc.
Aptb
Apt
Apmk
Apmk
MoCov3 (Chen et al., 2021)
VIT-B
1600
83.2
47.9
42.7
BEiT (Bao et al., 2021)
VIT-B
800
83.2
49.8
44.4
Supervised (He et al., 2021)
ViT-B
300
82.3
47.9
42.9
MAE (He et al., 2021)
ViT-B
1600
83.6
50.3
44.9
improvements over baseline
+1.3
+2.4
+2.0
Supervised (Liu al., 2021)
Swin-B Swin-B
300
83.5
48.5
53.2
43.2
46.7
SimMIM (Xie al., 2021)
800
84.0
50.4
55.5
44.4
47.9
improvements over baseline
+0.5
+1.9
+2.3
+1.2
+1.2
Supervised (Liu et al., 2022)
ConvX-B
300
83.8
47.7
52.6
43.2
46.6
Spark (ours)
ConvX-B
1600
84.8
51.2
56.1
45.1
48.9
improvements over baseline
+1.0
+3.5
+3.5
+1.9
+2.3
4.3
TRANSFERRING DOWNSTREAM TASKS
Previous results on ImageNet classification have exposed the potential of SparK pre-training. In this part we further evaluate the representation quality on fundamental downstream tasks, including object detection and instance segmentation COCO (Lin al., 2014). These tasks challenging, serving as professional feature evaluators because they place higher demands than classification: models need predict not only what, but also where the objects (instances) are. Here, we consider two different settings: comparison with self-supervised vision transformers, and then with convolutional networks. In all COCO experiments, do not use advanced techniques such multi-scale testing, large-scale jittering augmentation and soft-NMS. For more details on line-tuning see Appendix C and D.
Performance vs. self-supervised transformers. Table compares the fine-tuning results on three downstream tasks: classification (Cls.), object detection (Det.), and instance segmentation (Seg.). Among self-supervised methods, SparK the best performer and the only one that pre-trains convnet. Even when compared to the strongest SimMIM (Xie et al., 2021) with swin-transformer SparK still yields superior results by +0.8, +0.8, +0.7 on three tasks respectively. It is particularly worth noting that without pre- te--training, ConvNeXt-B and Swin-B perform similarly. This indicates that the gains are indeed due to our SparK pre-training rather than the backbone difference.
Overall, it can also be seen that approach exhibits the highest improvements baselines in table 2 (up to +.5%%). All these observations are consistent with those in section 4.2 and once again validate that the BERT-style pre-training convolutional networks is promising.
supervised
Table 3: ResNet-50 results downstream tasks. SparK is compared to state-of-ther the-art art contrastive learning algorithms. For ImageNet, the training recipe from Wightman al. (2021) (300-epoch fine- senitunnng with 224 resolution) used. For COCO, Mask R-CNN ResNet50- equally fine-tuned for 24 epochs or 2x), with average precision on va12017 reported. SparK highlighted as the only generative method.
Pre-train
Eff.
Cls.
1x Schedule Aphh
2x Schedule Aphh
Pre-training (on ResNet-50)
task
epoch
(Acc.)
Apmk
Apmk
Supervised
79.8
38.9
35.4
41.3
37.3
SimSiam (Chen & He, 2021)
Contrastive
800
79.1
MoCo (He et al., 2020)
Contrastive
800
38.5
35.1
40.8
36.9
MoCov2 (Chen et al., 2020b)
Contrastive
1600
79.8
40.4
36.4
41.7
37.6
SimCLR (Chen et al., 2020a)
Contrastive
4000
80.0
InfoMin (Tian et al., 2020)
Contrastive
800
40.6
36.7
42.5
38.4
BYOL (Grill al., 2020)
Contrastive
1600)
80.0
40.4
37.2
42.3
38.3
SwAV (Caron et al., 2020)
Contrastive
1200
80.1
42.3
38.2
SparK (ours)
Generative
1600
80.6
41.6
37.7
43.4
39.4

---

Published as a conference paper at ICLR 2023
Performance self-supervised convnets. We then compare Spark to state-of-the-ar convolutional contrastive learning methods. In table 3, all contrastive methods are basically on par with supervised pre-training. While SparK, the first generative pre- training method for hierarchical convnets, performs significantly better than them across all downstream tasks by +0.5-1.2 points. In particular, Spark does not rely sophisticated augmentations which have proven to be essential for contrastive learning (Chen et al., 2020a; Tian et al., 2020). We attribute these superior results to the fact that generative pre-training (SparK) can inherently provide more supervisory signals than discriminative methods: it optimizes reconstruction loss, a form of regression loss, which is considered to be more dense and localized than contrastive learning's instance classification loss.
F Feature transferability. An intriguing phenomenon in table 2 and 3 is that the improvements over supervised baselines are more significant on COCO tasks than on ImageNet (+-3.5 for ConvNet and +2.7 for ResNet). Notice there are several key differences between these two datasets: (i) the image resolution of COCO is much higher than that of ImageNet; (ii) most images in ImageNet are object-centric, while COCO images usually contain multiple disorganized objects. This domain gap poses a challenge for transfer learning, and SparK is demonstrated able to face it. This shows SparK can learn highly transferable features through the BERT-style generative pre-training.
4.4 SCALING SPARK
We gradually scale up the model size or training resolution and test SparK's performance. Results are reported in table 4, where we quote the accuracy of supervised baselines from Wightman et al. (2021) (the latest ResNet baselines) and Liu et al. (2022) (ConvNeXt) as "Baseline Acc.". As shown in the last column table 4, one can observe that with our SparK pre-training, models except ResNet- 50 achieve performance
par with their non-pretrained versions of larger sizes. Such qualitative leap indicates SparK can push a convnet to the "next level" in terms representation capability. Comparing the results horizontally, SparK improves all supervised baselines by large margins of +0.8-1.7, verifying such a self-supervised learning can make better use of model capacity than supervised pre-training in this evaluation. Overall, the results demonstrate favorable scaling ability of Spark as larger models benefit more. The steady gains across classical and modern architectures also make believe SparK can boost many other state-of-the-ar convolutional networks like VAN (Guo et al., 2022), RepLKNet (Ding et al., 2022), and Internlmage (Wang et al., 2022).
Table 4: Scaling SparK with model size and training resolution. ImageNet top-1 accuracy is reported. Absolute improvements over baselines are listed as A. The last column indicates whether SparK's 's performance with smaller model (e.g., 84.1 of ConvNeXt-S) reaches the baseline of larger one (e.g., 83.8 of ConvNeXt-B),
#Para. (M)
FLOPs
Baseline Acc.
Spark Acc.
Reach the next level
Architecture
Reso.
A
(G)
Classical Architecture
ResNet-50
224
25.6
4.1
79.8
80.6
+0.8
X
ResNet-101
224
44.5
7.9
81.3
82.2
+0.9
/
ResNet-152
224
60.2
11.6
81.8
82.7
+0.9
/
ResNet-200
224
64.7
15.1
82.1
83.1
+1.0
Modern Architecture
ConvNeXttSmalll
224
50.0
8.7
83.1
84.1
+1.0
/
ConyNeXt-Bass
224
89.0
15.4
83.8
84.8
+1.0
/
ConwNeXt-Large
224
198
34.4
84.3
85.4
+1.1
ConyNeXt-Large
384
198
101
84.3
86.0
+1.7
4.5 ABLATION STUDY
In this study, we gradually ablate the components in SparK framework and check the corresponding performance respectively. ImageNet fine-tuning results of cach SparK's variants are listed in table 5.
Core designs. We first remove the two most important designs in SparK: sparse masking strategy and hierarchical architecture. By replacing our sparse strategy with the zero-outing discussed in section 3.1, we observe a noticeable performance degradation in row of table 5 that almost reaches the supervised baseline. This suggests the issues raised by zero-outing (like data distribution shift in figure and mask pattern vanish in figure 3) can lead to ineffective pre-training. We then remove the hierarchical design (row 4), which results single-scale masked modeling that is commonly used
8

---

Published as conference paper at ICLR 2023
Table 5: The ablation study on the importance of each components in SparK. Experiments are based on ConvNeXt-Smie Small, with ImageNet validation accuracy reported. Our default setting is in row 2. Difterences highlighted blue. "APE": absolute positional embedding: "std.": standard deviation of four experiments.
Method
Masking
Hierarchy
APE
Loss
Epoch
Acc.
std.
Not pretrained
83.1
-1.0
SparK (ours)
sparse
X
masked only
84.1
0.0
0.07
zero-outing
zero-outing
/
X
masked only
1600
83.2
-0,9
0.06
w/o hierarchy
sparse
X
masked only
1600
83.6
-0.5
0.04
w/ APE
sparse
masked only
1600
83.9
-0.2
0.10
w/ more loss
sparse
X all
1600
83.3
-0.8
0.12
pre-train less
sparse
X masked only
800
83.7
-0.4
0.05
for transformers (Devlin al., 2018; Bao al., 2021; He al., 2021; Xie al., 2021). It only the features at the end of encoder to reconstruct. This modification shown to impair the fine-tuning performance well. In sum, both sparse strategy and hierarchy design play key roles in SparK.
Other components. addition, we find adding absolute positional embeddings (row 5) is practically useless for learning convolutional representations. We also observe calculating loss values only on masked patches gives higher accuracy (row 6), which is consistent with He et al. (2021). Finally and reasonably, our SparK benefits from longer pre-training as verified in row 7.
4.6 VISUALIZATION
We visualize some reconstruction results to check how the model performs pre-training. From figure 4 we can see that the model is able to make different but plausible predictions on masked regions (e,g., ., in the 2-nd column). the 4-th and 6-th columns, the model can almost reconstruct the round shape of red fruits from the very small portion of exposed edges. The clear texture in the 3-rd column also shows the model can capture the visual signals with medium or high frequencies.
Masked Input
Prediction capico!
Input
Pure Prediction
Figure 4: Reconstruction examples by a pre-trained ConvNeXt-Base with a mask ratio of 60%, Images are randomly selected from ImageNet validation set. Several interesting regions are highlighted.
5 CONCLUSION
The field of natural language processing (NLP) has witnessed the rise and proliferation of masked modeling in NLP transformers. More recently, there have been efforts to extend this paradigm to vision transformers, although its application convnets has proven problematic. This has spurred us to investigate the fundamental differences between language and image processing, and motivated us propose solution: SparK. SparK involves treating unmasked patches as sparse voxels, and encoding them using sparse convolution. Additionally, we employ hierarchical decoder to fully leverage the benefits convnet's hierarchy. SparK enables masked modeling be applied effectively to any convnet, and results in substantial performance increase on downstream tasks. Our research showcases the potential of BERT-style pre- pre-training on convnets and is an initial step towards its future implementation. We hope our findings will inspire further exploration of generative pre-train pre-training convnets, and facilitate the adoption the pretrain- finetune paradigm in computer vision community.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint vrXiv:2/06.08254 2021.
John Bateman. Text and image: critical introduction the visual/verbal divide. Routledge, 2014.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pp. 404-40 417. Springer, 2006.
Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supersuud vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1650-9666 2021.
Kai Chen. Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: /ett://90000700 07155, 2019.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607, PMLR, 2020a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03022 2022.
Xinlei Chen and Kaiming Hc. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0 04297, 2020b.
Xinlei Chen, Saining Xic, and Kaiming Hc. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang. and Yuexian Zou. M3ST: mix at three levels for speech translation. CoRR, abs/2212.03657, 2022. doi: 21..5500a2Xi.22222.066666 URL https: //cci /cci /rrg 0 48550/ a i v. 221 2, 03657.
Christopher Choy, Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3075-3084, 2019.
Kevin Clark, Minh- Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Minh-Thang
Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. Workshop on statistical learning computer vision, ECCV, volume pp. 1-2. Prague, 2004.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 113-123, 2019.
Navnect Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR '05) volume 1, pp. 886-893. Iccc, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
Xianhan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-19775, 2022.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2010 11929, 2020.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self- supervised visual pre-training. arXiv preprint arXiv:2202.00 033822 2022.
Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, deformable part model. In 2008 IEEE conference on computer vision and pattern recognition, pp. 1-8. Icse, 2008
Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. Convmac: Masked convolution meets masked autocncoders, arXiv preprint arXiv:2005.03800 2022.
Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks, arXiv preprint 1rXiv:1706.01 3077, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar. et al. Bootstrap your own latent-a new approach to self-superviset lcarning. Advances in neural information processing systems, 33: 21271-21284, 2020.
Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202. 09741, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 37(9); 1916-1916, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arX.v:2111.06377, 2021.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Hongxiang Li, Meng Cao, Xuxin Cheng. Zhihong Zhu, Yaowei Li, and Yuexian Zou. Generating templated caption for video grounding. CoRR, abs/2301 abs/2301.009977, 2023, doi: 10.48550/arxxv.2331..00 055997.. URL
https: / /cc.iiiiia// wrg/10........ /00. 48550/ 2301. 05997.
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, Baigui Sun, Hao Li, Xuansong Xie, Stan Li, et al. Archiiecture--gnosticc masked image modeling-from vit back to cnn. arXiv preprint arXiv:2205. 13944 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, 2014.
Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 806-814, 2015.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
11

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 11692, 2019.
Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, and Xiang Bai. Richer convolutional features for edge detection. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3000-3009, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012 10022, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for 2020s. arXiv preprint urXiv:2001.03545 2022.
Jonathan Long, Evan Shelhamer, Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431 -4440,3404040 2015.
David Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp. 1150-1157. Ieec, 1999.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders:
Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 188);9 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp- 8821-8831. PMLR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Fthan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orh: An efficient altemnttve to sift or surf. In 2011 International conference on computer vision, pp. 2564-2571. Iece, 2011.
Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. In 2019 International Conference on Robotics and Automation (ICRA), pp. 7276 7282. IEEE, 2019.
Mingxing Tan and Quoc V L... Efffciinttet: Rethinking model scaling for convolutional neural networks. arXiv preprins arXiv: 1946, 2019.
Keyu Tian, Chen Lin, Ser Nam Lim, Wanli Ouyang, Puneet Dokania, and Philip Torr. continuous mapping for augmentation design. Advances in Neural Information Processing Systems, 34: 34:13732-13743, 2021.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 2320-2329. 2020.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.05777 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuaa Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 14668 14678, 2022.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:21 10.00446, 2021.
12

---

Published as a conference paper at ICLR 2023
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. httpa:
//github. com/ fac.bookresearchh htw.rrrssscarreeettttteeee.. detectren2, 2019.
Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: 2021.
simple framework for masked image modeling. arXiv preprint arXiv:2111.
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conference on computer vision and pattern recognition, pp- 1794 1794-1801. JEEE, 2009.
Yang You. Jing Sashank Reddi, Jonathan Hseu, Sanjiv Kumar. Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert 76 minutes. arXiv preprint arXiv: 190,,0092, 2019.
Jure Zbontar, Li Jing. Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 1810022220. PMLR, 2021.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang. Bin Zhao, Dong Wang. Yu Qiao, and Hongsheng Li. Foint-m2ac. scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint arXiv:2205. 14401, 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arX0v:2111.07832, 2021.
Yin Zhou and Oncel Tuzel. VoxeInet: End-to-end end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4490 4499, 2018.
13

---

Published as a conference paper at ICLR 2023
A DETAILS: DECODER ARCHITECTURE
SparK a general method that does not limit the specific encoder to be pre-trained. other words, the definition of the encoder all up the user (e.g., standard ResNet-50). In the implementation presented in section 4.1, , the only undefined component is the decoder. We thus give its PyTorch implementation follows. our experiments, the same decoder of Decode (768, 32) is used equally for all encoders, including different ResNets and ConvNeXts.
import math import torch.nn nn as nn
class UNetBlock22xmn.. :odule)):
def iniit__ (self, cin, cout):
super(O._init___0 O
self.b mn
n.. Sequential(
nn. Conv2d(cin,, cin, kernel size=3, kernel_size=3, stride=1, padding=1, bias=False),
Bo.Baatchhorm2d(cin)) nn. (m.RelU((impllac-TTru))
nn. (cnnndd cout, kernel_size=3, stride=l, padding=1, bias=False), nn. mm.Batchhlorm/d(cot)))
def forward(self, x): return self.b b(x)
class DecoderConv (nn. Moduull
def _init__ cin, cout):
super (opper(._iiii____ init__
self. up nn. conn/raassssseedciin cin, kernel_size stride=2, padding=1, bias=True)
self.com = UNerBlocc22xcin, cout)
def (srward x): up() return self. (x)v
class LightDecoder (an.Deeecderrrn.MMMoull.. Module):
def sinit___self,, decoder_ fea_dimm ratto))
super init__ self. dem_ decoder_ fia_dim
round (mathh 1og2 (upsample >rssample_raatoo)
channels [self. fea_ dim // ** for in range(n 1)] dec = nn. ModuleList([
DecoderConv (cin, cout) for (cin, cout) in zip(chanmess:--1), chamnels[1:])
1)
self. self.proj = nn. Conv22 3, kiznel, stride=1, bias=True)
def forward(self, to_dec):
for i, in enumerate(selffddc))
if
and to dec[]]] is not None:
x = x [ec]]c self.dec[ii((
return self.prof
14

---

Published as a conference paper at ICLR 2023
B ADDITIONAL RESULTS: LINEAR EVALUATION
We report the small-sizer models' linear evaluation performance in table 6. In this evaluation protocol, the pre-trained backbone model frozen and only a linear projection head would be fine-tuned. This protocol is all the rage in contrastive learning (Chen al., 2020a; He al., 2020; Chen & He, 2021; Caron et al., 2021), which can probe the linear separability of deep representations, and has been quite popular computer vision due the richness of image data augmentations compared to other modalities (Cubuk et al., 2019; Tian et al., 2021; Cheng et al., 2022). Note that MoCoV3 (Chen et al., 2021) is the only contrastive learning method table 6, which aims learn global representation, and is therefore more suitable than non-contrastivv methods on tasks like linear evaluation. SparK shows its decent performance compared to other non-contrastive methods.
Table Linear evaluation results. Numbers other work are directly quoted form Chen al. (2022).
Method
BEiT
CAE
SparK
MoCoV3
Contrastive
X
X
X
Accuracy (%)
15.7
51.8
54.7
73.1
C DETAILS: IMAGENET FINE-TUNING
We refer to the latest open-source ResNet baseline of Wightman et al. (2021) to fine-tune ResNets. For Con Liu et al. (2022), we simply use their official implementation. Since the original configurations in Wightman al. (2021); Liu et al. (2022) based supervised training from scrach, we adjust some hyperparameters for doing fine-tuning. Details are given in table and table 8.
Table 7: ImageNet fine-tuning recipe ResNets, referring Wightman al. (2021).
Configuration
Value
Configuration
Value
Image resolution
224
Epochs
300
Test image crop
0.95
Batch size
2048
Optimizer
LAMB
Learning rate
8e-3
Scheduler
Consine
Weight decay
0.02
Repeated aug.
Dropout
x
Rand aug.
7 7/0.5
Stoch. depth
Gradient clip.
X
BCE loss
Mixup alpha
0.1
Label smoothing
0.1
Cutmix alpha
1.0
EMA
(0.99,0.999)
Table ImageNet fine-tuning recipe for ConvNeXts, referring to Liu (2022).
Configuration
Value
Configuration
Value
Image resolution
224
Epochs
200
Test image crop
0.95
Batch size
2048
Optimizer
AdamW Consine
Learning rate
3.2e-3
Scheduler
Weight decay
0.01
Repeated aug.
Dropout
Rand aug.
9/0.5/inc1
Stoch. depth
/
Gradient clip.
X
BCE loss
Mixup alpha
0.8
Label smoothing
0.1
Cutmix alpha
1.0
EMA
(0.99,0.999)
15

---

Published as a conference paper at ICLR 2023
D DETAILS: COCO FINE-TUNING
On COCO, we use the official implementations MoCoV2 (Chen et al., 2020b) and ConvNeXt (Liu al., 2022) to evaluate ResNets and ConvNeXts. These implementations are based Detectron2 (Wu et al., 2019) and MMDetection (Chen et al., 2019) respectively. Following the convention, we do use advanced techniques like multi-scale testing, large-scale jittering augmentation, or soft-NMS, in all our COCO experiments for faimess. Details are in table and table 10.
Table 9: COCO fine-tuning configuration for ResNets, referring the standard implementation of MoCoV2 (Chen et al., 2020b), Mask R-CNN with FPN is used. 2 = A for the so-called "Ax" schedule. For instance, a fine-tuning schedule means a 24-epoch training with 0.1-epoch warm-up.
Configuration
Value
Configuration
Value
Image resize
(384, 600)
Normalization mean
[123.7, 116.3, 103.5] [58.4, 57.1, 57.4]
Multi-scale testing
X
Normalization std
Large-scale jittering aug
x
Optimizer
AdamW
Soft-NMS
X
Weight decay
0.0001
Epochs
12x
Learning rate (LR)
2e-4
Warm-up epochs
0.05x
LR layer decay
0.65
I.R scheduled epochs
[9r., 11x]
LR scheduled ratio
0.2
Table 10: COCO configuration for ConvNeXts, referring the standard implementation of ConyNeXt
(Liu al., 2022). Following the convention self-supervised learning, Mask R-CNN with FPN is used.
Configuration
Value
Configuration
Value
Image resize
(1333, 800)
Normalization mean
[123.7, 116.3, 103.5] [58.4, 57.1, 57.4]
Multi-scale testing
X
Normalization std
Large-scale jittering aug
Optimizer
AdamW
Soft-NMS
Weight decay
0.05
Epochs
36
Learning rate (LR)
2e-4
Warm-up epochs
0.15
LR layer decay
0.65
LR scheduled epochs
[27, 33]
LR scheduled ratio
0.2
16


Reviews:
[{"review_id": "N8EgfEmEMH", "reviewer": "Reviewer_71p6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a masked modeling method for convolutional vision backbones. Instead of simply zeroing out the masked regions in the input images, the method exploits the submanifold sparse convolution to handle irregular masked input. The method is evaluated on multiple popular vision tasks and backbone models.", "summary_of_the_review": "The method presented in this paper is interesting and may be useful in future research and applications. However, there are still many important issues that require further clarification and discussion including missing comparisons, baselines and experimental details. I would upgrade my rating if the above-mentioned issues are addressed.", "strength_and_weaknesses": "Strengths:\n\n- Transferring the successful experiences of MIM from Transformers to convolutional models is an important and potentially impactful topic. This paper exhibits some promising results of applying MIM to ConvNeXt and ResNet.\n\n- The idea of using the submanifold sparse convolution to handle irregular masked input is well-motivated and new. The hierarchical encoding and decoding design also look interesting.\n\nWeaknesses:\n\n- The motivation shown in Figure 1 may be straightforward but a bit misleading. I think the main advantage of the proposed method based on the submanifold sparse convolution is that the mask patterns are kept in all intermediate features. If we use the \"vanilla sparseconv\" mentioned in Sec 2.3, the model will also be difficult to generalize to raw input even if the input data distributions are close. \n\n- There are several missing references that should be discussed. [r1] is a widely used sparseconv framework in 3D understanding. [r2] is an existing method that uses sparseconv to accelerate 2D image recognition. \n\n[r1] 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks, CVPR 2019\n\n[r2] Dynamic convolutions: Exploiting spatial sparsity for faster inference, CVPR 2020.\n\n- Many important implementation details are missing. Sec. 4.1 introduces the details of the proposed pre-training method. However, there is no detail about the fine-tuning process for ImageNet and downstream tasks, which is very important information to compare different pre-training methods and reproduce the results. For example, MAE and SimMIM report the results with 50/100-epoch finetuning. Can SparK models achieve competitive results on ImageNet with fewer finetuning epochs than standard training (300 epochs)? Are extra tricks used during fine-tuning?  \n\n- The discussions in Sec 4.2 emphasize the encoding efficiency of the proposed method. From my experience, sparseconv is usually much slower than standard convolutions with similar theoretical complexity since the standard convolution is well-supported and highly optimized on GPUs. I guess 60% masked patches may not lead to significantly higher training speed than models without masked input in practice. The encoding cost shown in Table 1 is very ambiguous. I think it would be better to report the wall-clock time of the whole system and training process like the MAE paper to clearly compare different methods.\n\n- The comparisons of small backbones are unfair.  ConvNeXt-S have doubled computational costs/parameters compared to ViT-S/Swin-T.\n\n- Why are the detection frameworks used for small and base backbones in Table 2 different? Besides, it would be better to report the performance of the baseline supervised ConvNeXt models in both settings.\n\n- According to Table 4, it seems ConvNeXt-L cannot outperform MAE (ViT-L). It would be better to add more discussions and analyses of this. \n\nMinor issues:\n\n- How about the results if we zero out features of the masked regions (before all 7x7 convs) or replace them with the mask embeddings instead of zeroing out input? Considering sparseconv requires extra libraries/extensions and might not be easy to use, this implementation can be a competitive baseline.\n\n- The formats of citations are different in some sentences. For example, in Section 2.2, there are both \"Bao et al. (2021)\" and \"(Devlin et al., 2018)\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8JZcPDGDL_", "reviewer": "Reviewer_Zhkz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on the pretext task learning a subfield of self-supervised learning. It follows the track of predicting the original image given random non-overlapped masks. To make it work on non-transformer architectures, the authors propose to use a hierarchical and sparse convolution network to pre-train the network with the reconstruction loss on the masked region. To show the advantages of the paper, the authors provide extensive comparisons with other baseline methods on several downstream tasks. The authors also conduct the ablation study to show the effectiveness empirically. ", "summary_of_the_review": "Overall, I do appreciate the messages that this paper sends to us. It reinforces the claim of ConvNext that convolution networks can be better than transformers. However, the proposed framework seems very engineering oritented.", "strength_and_weaknesses": "Pro:\n1. This paper solves the patch mask on the ConvNet by using a \"sparse\" convolution operation. At the same time, the authors also incorporate the U-Net-like design, which passes the encoding mask to the decoding part to preserve the mask information.\n\n2. This paper provides strong performance across all the tasks and architecture in a self-supervised learning setting.\n\n\nCons:\n\n1. The authors seem to overclaim several two important contributions. \n\na. Removing information of masked parts is difficult for ConvNet. This has been identified by submanifold sparse convolution, which aims to preserve the structure of the unmasked regions. Therefore, it should not be a difficulty anymore.\n\nb. The author claims in the experiment that their work proves that the convnet is not inferior to the transformers. This seems wrong to me. I do think ConvNext has already shown that.\n\n2. In Fig.1, since the second row is the histogram, I can understand that using zeros as masks could increase the number of zeros, but I do not understand how the sparsely dropping does not decrease the frequency of other pixels (Fig. 1c).\n\n3. Table 5 shows that even without the hierarchical encoding, ConvNext outperforms the transform architectures. Indeed, the hierarchical encoding works, but not impressive. The performance gained mostly due to ConvNext itself.\n \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5M79QcIfCc", "reviewer": "Reviewer_tYbJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method of pre-training convolutional neural networks via masking. They mask parts of the input image, take a convolutional network as encoder, and add a U-net style decoder, with which they predict the parts which were masked out. After pre-training, the decoder is removed, and the encoder is fine-tuned for downstream tasks, similar to recent Transformer-based approaches.\n\nThe approach is experimentally validated on ImageNet classification and COCO object detection tasks and shows improvements over other contrastive and masked pre-training methods with models of similar size in terms of the number of parameters and computational complexity.", "summary_of_the_review": "I suggest accept, seems like a good paper. It would be beneficial to have linear probing results in addition to fine-tuning.", "strength_and_weaknesses": "Strengths:\n- well written and easy to follow\n- strong results on using networks pretrained with the proposed method for object detection, which could be attributed to the tested object detection architecture, Mask R-CNN, designed for convolutional networks.\n- detailed ablation study. Would be interesting to see if object detection sees improvements when using zero-out masking mode.\n- the provided PyTorch code is useful to have better understanding of the approach\n\nWeaknesses\n- the authors do not show results on large backbones with 86+ classification accuracy. It is not clear if the approach does not scale to larger backbones, or the authors were limited by computational resources. Including such results would be very useful for future work.\n- no statistical significance evaluation, which makes validating the claims difficult, especially in the ablation study. What is the variance in the experiment?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%