Waxing-and-Waning: a Generic Similarity-based Framework for Efficient Self-Supervised Learning
Authors: Sheng Li
Chao Wu
Ao Li
Yanzhi Wang
Xulong Tang
Geng Yuan
Venue: ICLR 2024 poster (2024)
Keywords: Self-supervised learning
efficient training
image similarity
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Deep Neural Networks (DNNs), essential for diverse applications such as visual recognition and eldercare, often require a large amount of labeled data for training, making widespread deployment of DNNs a challenging task. Self-supervised learning (SSL) emerges as a promising approach, which leverages inherent patterns within data through diverse augmentations to train models without explicit labels. However, while SSL has shown notable advancements in accuracy, its high computation costs remain a daunting impediment, particularly for resource-constrained platforms. To address this problem, we introduce SimWnW, a similarity-based efficient self-supervised learning framework. By strategically removing less important regions in augmented images and feature maps, SimWnW not only reduces computation costs but also eliminates irrelevant features that might slow down the learning process, thereby accelerating model convergence. The experimental results show that SimWnW effectively reduces the amount of computation costs in self-supervised model training without compromising accuracy. Specifically, SimWnW yields up to 54\% and 51\% computation savings in training from scratch and transfer learning tasks, respectively.

Published as a conference paper at ICLR 2024
A GENERIC SIMILARITY- BASED FRAMEWORK FOR EFFICIENT SELF- SUPERVISED LEARNING
WAXING-AND- WANING:
Sheng Li', Chao Wu', Ao Li", Yanzhi Wang", Xulong Tang', Geng Yuan²
University of Pittsburgh University of Georgia Northeastern University University of Arizona {sh1188, xul onggonnggggggpttteeed
geng. maggyyaan@ug..ee aol soli18arizona.
fcha. wu, yanz. www.://eoorrrttttttrm..coo
ABSTRACT
Deep Neural Networks (DNNss, essential for diverse applications such as visual recognition and eldercare, often require a large amount of labeled data for train- ing, making widespread deployment of DNNs challenging task. Self-super learning (SSL) emerges promising approach, which leverages inherent pat- terns within data through diverse augmentations to train models without explicit labels. However, while SSL has shown notable advancements in accuracy, its high computation costs remain daunting impediment, particularly for resource- constrained platforms. To address this problem, introduce SIMWNW, similarity- based efficient sulf-suue learning framework. By strategically re- moving less important regions in augmented images and feature maps, SIMWNW not only reduces computation costs but also eliminates irrelevant features that might slow down the learning process, thereby accelerating model convergence. The experimental results show that SIMWNW effectively reduces the amount of computation costs self-supervised model training without compromising accu- racy. Specifically, SIMWNW yields up to 54% and 51% computation savings in training from scratch and transfer learning tasks, respectively.
INTRODUCTION
The conventional supervised learning paradigm that underpins their training process relies heavily on large amounts of labeled data (Li & Feng 2019: at. 2019; Hou et al. (022). To address these limitations, researchers increasingly turn to Self-Supervised Learning (SSL) as a promising alterna-
tive and have made significant progress over the last few years (Bachman et al. 2019 Chen et al.) 2020a Zheng et al., 2021] Xu et al. 2022: Cao et al. 2023 Li al.
SSL intends to learn the inherent representations that invariant under different distortions by maximizing the similarity of representations obtained from training samples using different augmentation methods (Misra &
Maaten 2020 Reed al. 202 Zbontar al. 202I).
While recent work has come long way in terms of improving the accuracy of SSL He et al. 2020 Ren al. 2022 Zhu et al., 2023), the exorbitant training cost associated with SSL methods still a critical challenge of SSL. For example, SSL designs like BYOL (Grill et al. 2020) demand 23 times the computation costs, attributed
8.8x more required training iterations and 2.6x higher computation cost iteration (Wen & Li 2021) compared to the supervised learning counterparts. This will inevitably impede the actual deployment of SSL. in real-world applications, especially for resource-limited limited platforms such edge devices.
Although a few recent works have begun to acknowledge the importance of the efficiency of SSL, this area is still severely neglected by the community. Recent work (Addepalli et al. 2022) intro- duces a rotation prediction task along with the original training process to improve the convergence speed of SSL. And Meng et al. (022) introduces contrastive dual gating mechanism with extra conditional paths to learn the dynamic gating policy and reduce the computation FLOPs. The ex- isting methods often require increased complexity on top of the original SSL. paradigm improve

---

Published as a conference paper at ICLR 2024
efficiency. So, it is natural to raise a question: Is there more general and effective method that can significantly improve the training efficiency of SSL?
Considering the training paradigm of the SSL that leverages different data augmentations on two branches with a siamese encoder model used, results in a unique property of SSL compared to the conventional supervised learning methods (Tian et al. 2020 Chen et al. 2020a). That is, the augmented input images and feature maps on the two branches inherently have a certain degree of similarity. This is a natural opportunity that could be potentially used for computation saving or simplifying. However, it is insufficient to lead us directly to a simple solution. It is not clear whether similar and dissimilar regions of augmented imput images and feature maps in the two branches are equivalently crucial for SSL and whether the similarity remains invariant in low-level features and high-level features. And how can we effectively utilize the similarity to improve training efficiency?
Motivated by these questions, we make a comprehensive exploration of the impact of similar regions on SSL. accuracy. And we explore two types of methods (i.e., reuse and remove) to exploit the similarities for computation-savins of augmented input images and each layer's activations can significantly reduce the computation and speed up SSL. To mitigate the region shrinking problem caused by convolution layers, we propose a strategy to effectively and efficiently identify and expand high high-similarity regions to ensure decent overall computation saving. This strategy can be considered a waxing- and waning process. Putting it all together, we propose our SIMWNW, generic and efficient SSL framework that can significantly reduce training costs and improve the convergence speed of SSL.
We find that eliminating the computation on similar regions
Our NNWWWW framework is generic and can be easily applied to different SSL training methods for training cost-saving. We evaluate our framework in both training from scratch and transfer learning tasks and validate the effectiveness and generalizability of SIMWNW. Specifically, in training from scratch tasks. compared to representative SSL works, SIMWNW W provides significant computation savings, peaking at 54% and averaging at 40%, and without sacrificing accuracy. In transfer learning tasks, SIM WNW shows a notable reduction in computation costs. peaking 51% and averaging at 48%, without accuracy loss. We also compare SIMWNW to efficient SSL approaches. In both training from scratch and transfer learning tasks, SIMWNW consistently outperforms SOTA works, achieving an average computational cost reduction of 18% and 14%, respectively.
2 BACKGROUND AND RELATED WORK
2.1 REPRESENTATIV SELF-SUPERVISED LEARNING FRAMEWORKS
SimCLR (Chen al. 2020a) stands out as a pivotal framework in contrastive self-supervised learn- ing. It operates by maximizing the similarity between representations of differently augmented versions of the same image and concurrently reducing the similarity with representations of other images in the batch. These representations, derived from encoders, capture the essential features of the images. Specifically, for any given image, its representations z; and zy are treated as a positive pair. The contrastive loss is:
LSINCLR - log exp(z: ZJ/T)
(1)
k= exp(2: 22/T)
Here, F denotes a temperature parameter that regulates the sharpness of the distribution, Z₁ is the representation of other images (negative samples), and N is the batch size.
MoCo (Momentum Contrast) (He et al., 2020) extends the idea of contrastive learning by incorporat-
ing a momentum-based encoder and dynamic dictionary of data samples. This dynamic dictionary facilitates rich source of negative samples, enhancing the contrastive task. The loss employed in MoCo is akin SimCLR, though enriched by the abundance of negatives from the dictionary. To stabilize learning and enhance the quality of the dictionary representations, MoCo updates its en- coder's 's parameters using a momentum mechanism. Given two sets of encoder parameters, fos and Boww, and the momentum coefficient m, the update rule is:
Oold + m&old + (1 - m))mew
BYOL (Bootstrap Your Own Latent) Grill et al. 2020) and SimSiam (Chen & He 2021 stand out for their innovative emphasis aligning representations from distinct augmentations of single
2

---

Published as a conference paper at ICLR 2024
image, eliminating the need for negative samples commonly used in other contrastive learning meth- ods. Within BYOL and SimSiam, the concepts of online and target branches are crucial. The online branch processes one augmented view of image through its neural network encoder, generating a representation the feature space. Simultaneously, the target branch processes another distinct augmented view of the same image. The primary objective is align the representation from the online branch to mirror closely that of the target branch. Crucially, back prok-propagttion occurs in the online branch, updating the network weights. This design ensures the alignment of representations between the two branches, aiming to maximize their similarity within the feature space.
2.2 EFFICIENT SELF-SUPERVISED LEARNING.
The recent work (Addepalli et al., 2022, argues that key reason for the slow convergence of self- supervised learning attributed to the presence of noise in the training objective, and proposes to add the rotation prediction as noise free auxiliary training objective to accelerate the convergence speed. Another work (Koçyigit et al., 2023) increases the training efficiency of self-supervised training by accelerating model convergence through novel learning rate schedule and an input image resolution schedule. It also proposes a new augmentation method to enhance the quality of augmented images to further accelerate model convergence. Compared to these methods, WIMWNW focus on exploring and exploiting the similarity inside of input images and intermediate feature maps to save computation costs. Our SIMWNW is compatible with these methods as well.
MOTIVATION AND EXPLORATION
In SSL, the siamese encoder models are usually used in the two branches, and each branch processes an augmented input derived from the same source image. So, an inherent similarity exists between the augmented images processed in the two branches. Based on this unique property of SSL, we have an assumption that regions with high similarity between the images in the two branches might have limited contribution to the self-suue model training. If this true, we have the potential to reduce computation costs. In this section, first investigate the importance of regions with varying degrees of similarity the two augmented images from the same source image. Following this, explore two strategies reuse and remove reduce computation in regions perceived to be less important. We further illustrate the advantages and drawbacks associated with each method.
3.1 IMPORTANCE OF DIFFERENT REGIONS ON AUGMENTED IMAGES
To verify our assumption, we conduct a preliminary study focusing on the importance of similar versus dissimilar regions in augmented images during training. Using the SimSiam framework as our baseline (Figure I). each image is transformed into two distinct views (i.c., augmented images), serving inputs of two branches in the SimSiam framework. In our study, preprocess the two augmented images before sending them to be processed by the model. We first divide the augmented image of the online branch into blocks of the same size. Each block undergoes block matching process (exhaustive search) to locate its most similar counterpart in the paired augmented image of the target branch. Here, the similarity between blocks quantified using the PSNR (peak signal- to-noise ratio) (Hore & Ziou 2010 An example of block matching is shown in Figure Ip. For the
Similarity
Augmented image In Onlline Branch
Augmented image Target Branch
Remove
Enooder
Reuse
View
View
Block Matching
Target Branch
Branch
Image
(a)
Figure 1: (a) SSL framework. (b) Examples of block matching, reuse, and remove.
3

---

Published as a conference paper at ICLR 2024
Ancuracy (%)
40
60
44.81
46.22
41.29
48.97
50.82
40
50.87
42.23
45.09
49.06
50.85
Accuracy
20
20
Baseline (SimSlam)
Similar Blocks (25%)
Similar Blocks (50%)
Similar Blocks (78%)
Dissimilar Blocks (25%)
Dissimilar Blocks (50%)
Dissimilar Blocks (75%)
Baceline (SimSiam)
Similar Blooks (25%)
Similar Blocks (50%))
Similar Blocks (75%)
Dissimilar Blocks (25%)
Dissimilar Blocks (50%)
Diasimilar Blocks (15%)
(a) Removed a portion of the similar/dissimilar block (b) Reuse the computation results of a portion of the pairs within the two augmented images. The notation similar/dissimilar blocks in the online branch by using (x%) represents that 2% of the most similar/dissimilar the convolution results of their corresponding blocks block pairs retained, implying that (1 x)% of the in the target branch. The (x%) indicates that 2% of block pairs have been removed.
the most similar/dissimilar blocks are not "reused".
Figure 2: Preliminary study. The encoder is ResNet50, which is pre-trained on the ImageNet dataset. Then the ResNer50 trained for 100 epochs using Stanford Cars dataset. The accuracy results are obtained by the linear evaluation method.
image block marked with yellow boxes in the online branch, the most similar image block found in the target branch is also marked with yellow boxes.
Once the search process done, we create many block pairs in the augmented images of two branches. We search for the most similar counterpart in the target branch for each image block in the online branch, trying to find the corresponding block pair that is semantically related. We then sort the block pairs by their similarity (their similarity already calculated during the search process) and remove either the most similar or dissimilar pairs. In this case, if we remove block pairs with high similarity, by removing block pairs with low similarity, we can assess the impact of similar regions on training. The modified augmented images then continue through the standard SimSiam processing pipeline.
evaluate the effect of dissimilar regions training. Conversely,
In this experiment, we use ResNet50 (He et al.) (016) as the encoder and use Stanford Cars dataset (Krause et al. (013) as an example. The augmented images are resized dimensions of 224x224, and we opt for a block size of 30x30, drawing from recommendations in image block matching studies (Wei& & Wu 2013 Shin et al., 2008). We will discuss the impact of block size later in Section 5.2 Figure 2a presents the experimental results. Notably, when retaining an equivalent percentage of block pairs, prioritizing the retention of dissimilar ones yields better accuracy. The reason behind this is that the dissimilar regions provide difficulty for learning, compelling the model to learn meaningful representations (Tian al., 2020). In addition, when the appropriate number of block pairs retained (i.e., dissimilar blocks (75%) in Figure 2a), there is accuracy drop. These results highlight the importance of the dissimilar regions in augmented images for self-supervised learning, affirming assumption and motivating to reduce computation similar regions.
3.2 REUSE VS. REMOVE
By leveraging the similarity in SSL, there are two possible ways that we can use to reduce the computation: reuse and remove (depicted in Figure/Ip). For the reuse method, we can replace some blocks on the online branch with corresponding similar blocks on the target branch. In this way, can directly reuse the computation results (i.e., the corresponding blocks on the output feature map of convolutional layers) from the target branch as an approximation of the online branch result, avoiding computing the convolution output of the similar blocks on two branches separately. For the remove method, we directly eliminate specific regions of images, causing the convolution operation to skip these regions.
To compare the effects of reuse and remove, we conduct an experiment following the setup described in Section 3.1 and applied the reuse operation. The results are presented in Figure 2b Comparing Figures 2a and 2b we observe that at high reuse/remove ratio (i.e., retain 25% blocks), the reuse operation delivers higher accuracy as it keeps more semantic information. However, with a moderate level of block retention (50% and 75%), both operations yield comparable accuracies. Notably, when the majority of blocks (i.c., 75% % dissimilar blocks) are retained, either the remove or reuse method will not degrade accuracy.
4

---

Published as a conference paper at ICLR 2024
identlyy simiaattin
Ramore
(a) idontilying Similaritiss on Augmented Images
protossing
Prossed outer giniier paims
Check surrundding pioess
weriker
regiora
Inpud Fraeuuee Mapa of
Ouatut Maps of
Region (Pixels)
Similer Region (Pizwla)
Identifying Similarities Feature
Figure 3: Overview of SIMWNW framework. The similar image blocks are marked with yellow boxes and matched with lettered notations. The removed regions (i.c., pixels) are shown as white boxes. In steps & and @ the region with color surrounding the white boxes is the region where we check for similarity. The region inside the dashed boxes after step represents the previously removed pixels prior step @ The white region outside the dashed boxes denotes newly identified and removed similar regions from the feature map.
Yet, when it comes to computation savings, reuse operation falls far behind remove due to three rea- sons: i) Reuse operation benefits only one branch. The corresponding augmented images in the other branch still undergo the regular model processing, essentially meaning that computation savings are halved. ii) The benefits of the reuse operation don't translate well during back propaaaapogg,.nnn This because SSL frameworks like MoCo, BYOL, and SimSiam perform pack-propagation exclusively in one branch. This singular branch proppaaaann renders computation reuse non-viable. iii) Furthermore, frameworks like MoCo and BYOL models with identical architectures but varying parameters across the two branches, eliminating the possibility of computation reuse even during forward propagation. In contrast, the remove operation fully eliminates the convolution computa- tion in the removed region. In a nutshell, when appropriately strategized, the remove operation is more promising for computational efficiency.
FRAMEWORK DESIGN
In this section, propose generic similarity-based efficient self-supervised learning framework SIMWNW to reduce the computation cost. As shown Figure 3 we first launch a block-matching process identify (①) similar blocks on the two augmented images and remove (@) them for computation-saving However, one challenge is that the removed region in the augmented images tends to progressively diminish and eventually vanish layer by layer due the inherent properties of convolution. The consequence is that the potential for computation savings gradually decreases as the image is processed deeper into the network layers. address this, SIMWNW not only removes similar block pairs from the input augmented images but also identifies and removes similar regions the intermediate feature maps (i.e., activation), which are steps to @ Figure
4.1 IDENTIFYING SIMILARITIES ON AUGMENTED IMAGES
The first step in NW is identify the similarities in the two augmented images from the same original image. We divide one of the two augmented images into blocks, and for each block, we search for similar block in the other paired augmented image. The similarity between image blocks is quantified using the well-established lightweight PSNR metric, which is calculated as:
MAX MSE
Similarity 10 logio
where MAX, is the maximum possible pixel value of the image, and MSE is the Mean Squared Error between the two compared images. In Section 3 we apply unoptimized exhaustive search method, which traverses the entire image using the same block size. However, the exhaustive search not desirable for training acceleration due to the computation overhead introduced. Moreover,
5

---

Published as a conference paper at ICLR 2024
such a search could possibly overlook the image's semantic information, leading to the removal of two blocks that appear similar but carry different semantic meanings. As shown in Figure blocks and c' look similar due to the lighting conditions, but they semantically diverge. Removing such blocks could potentially hurt the image's semantic information, decreasing model performance.
To circumvent this, our strategy involves searching regions in the paired augmented image that share semantic links. Instead of scanning the entire image, our focus narrows down to a specific region surrounding block's counterpart in the paired augmented image. This approach is based on the belief that semantically related blocks in two augmented images are highly possible to come from the same region the original image. This approach not only reduces computation overhead but also ensures searching semantically related blocks, safeguarding the image's s overall semantic essence.
Pinpointing the semantically related corresponding re- gions necessitates tracking pixel position shifts dur- ing augmentations such as random crops or horizontal flips. It's worth noting that our search is not confined strictly to the direct counterpart block. We search for an expanded region (empirically set to four times the block size). This expansion ensures a balance of com- putational efficiency and precision. Furthermore, our search is not a full sweep of this expanded region. We apply the diamond search algorithm (Zhu & Ma 2000) for efficient block-matching.
b
Figure 4: example of using exhaustive search to match similar blocks. The similar image blocks are marked with yellow and red boxes and matched with lettered nota- tions. Note the regions within the two dot- ted green boxes not similar.
Given the widespread use of random crop augmenta- tion in SSL, there will be instances where one image's blocks lack a direct match within its counterpart aug- mented images. Here, by default, we scan the nearest region in the paired image, even if the exact counterpart is cropped out. At first blush, this might seem at odds with our semantically aware strat- egy. Yet, in our practice, this method proves adept at discarding voluminous, semantically sparse backgrounds. Take, for instance, the blocks and a' or b and // in Figure 4 Even though they aren't direct matches, removing them doesn't I detract from the image's semantic information. Conversely, regions brimming with vital details, like intricate objects, remain largely untouched by this method due to their inherent dissimilarity. A case in point is the relationship between blocks d and d' in Figure 4 Despite block d' being closest to where the counterpart block would be located, they will not be considered similar.
4.2 IDENTIFYING SIMILARITIES ON FEATURE MAPS
4.2.1
CHALLENGES: SHRINKING REMOVED REGION SIZE
Identifying and removing similar regions in augmented images offers a pathway to saving computation costs. Yet, this strategy has its limitations: the size of the re- moved region typically shrinks and eventually disap- pears as the images proceed through network layers. As result, the computation saving is progressively reduced as the images proceed deeper layer by layer. This is due to the intrinsic behavior of convolution op- erations. If a convolution kernel traverses across both removed and normal regions in the augmented images and feature maps, the convolution operations can not be skipped. The convolution operations can be skipped only when the kernel is entirely within the removed re- gion. For a clearer picture, consider a scenario depicted in Figure 5 After undergoing convolution with a 3x3 kernel, the removed region shrinks from 5x5 to 3x3, ef- fectively "waning" its outermost pixel boundary. This waning phenomenon becomes inevitable when the kernel size is larger than Ix1, which is often the case with CNNs that often use 3x3 or larger kernels. Some networks do not even contain 1x1 kemels, making this issue more pronounced.
Before Convolution: Removed Region (5x5)
After Convolution: Removed Region (3x3)
Convolution Kernel
Normal Pixel "Eroded" Pixel
Removed Pixe!
An example to show how the removed region shrinks after convolution operation. The kernel size here is 3x3 and the stride is 1.
Figure

---

Published as a conference paper at ICLR 2024
This directs our attention to strategizing ways sustain the size of the removed region. Under- standing that various layers in a neural network cater to extracting features at differing levels is key: the front layers are responsible for low-level feature extraction (c.g., detecting edges). whereas the deeper layers extract high-level features (Yosinski et al. 2014 Zeiler & Fergus) 2014 Chen et al.) 2023). This layered hierarchy suggests that images move deeper in the model layers, the emer- gence of similar regions on the feature maps may become more evident as similarities in high-level features on the feature maps gradually emerge.
4.2.2 EXPANDING THE SIZE OF REMOVED REGION
Building on this insight, we propose to identify and remove similar regions in the feature maps of various model layers. Specifically, before the convolution operation the feature maps, we would scrutinize the surrounding regions around the removed regions within a pair of feature maps. We advocate for concentric circle approach to identify similarities. That is, we begin by evaluating the similarity of the immediate circle of pixels around the removed region (step @ in Figure 33 If the immediate circles of pixels in two paired feature maps are similar, we proceed to the next pair of outer circles (step @ in Figure B}. This iterative examination continues until the similarity for a certain circle pair falls below the similarity threshold. In contrast to bock-matchinn in augmented images, identifying similarities in feature maps is direct one-to-one similarity check without searching process, For a visual illustration, see the feature maps after step in Figure 3 The white regions outside the dashed box in the feature maps represent regions newly identified as similar and consequently removed.
The reason we focus on regions surrounding the removed region arises from the presumption that neighboring regions might exhibit higher similarities. For instance, referring to the upper-left convo- lution kernel in Figure although its computation is not skipped and its output is "croded" pixel, it is crucial to recognize that the source of this eroded pixel contains a portion of the removed (sim- ilar) region. Moreover, given that changes in images are usually relatively flat (Huang et al. 2000), it is reasonable to assume the regions around the removed regions are more likely to be similar.
To further refine this process, advocate for channel-wise approach to determine similarities within the feature map, followed by tailored removal procedure for each channel based on its distinct similarity results. Given that feature maps can have plethora of channels (sometimes even in the hundreds), and the spatial distribution of similar regions can vary significantly among channels, a holistic approach encompassing all channels would not be efficient.
EVALUATION
We evaluate the proposed SIM WNW framework in both training from scratch and transfer learning tasks. In the training from scratch task, we use three representative datasets CIFAR-10, CIFAR- 100 (Krizhevsky & Hinton| 2009) and ImageNet (Deng et al.) 2009). In the transfer learning task, the encoder is pre-trained on the ImageNet dataset and then trained on three datasets Stanford Cars, FGVC Aircraft (Maji al. 2013), and Caltech-UCSD Birds (CUB) (Wah et al.) (011). The batch size in all the experiments is set to 256. For all datasets except CIFAR, we employ ResNet50 as the encoder. For the CIFAR dataset, we use ResNet18 as the encoder. We apply SIMWNW in two representative SSL frameworks for evaluation: SimSiam and SimCLR.
In our experiments, the similarity threshold is set to 20 to consider two image blocks are similar. We select this threshold based on the common practice in wireless image transmission, where a PSNR value of 20 between compressed image and its original version is generally deemed to be of accept- able similarity for many applications (Muralikrishna et al. 2013 Minallah et al. 202T). However, this similarity threshold can be modified to adapt to the unique needs of specific applications. The overhead introduced by similarity calculation is included in the results and we present detailed analysis of overhead Appendix C When we do the block-matching for augmented images, we use block size of 30x30 for 224x224 images and scale down to 5x5 block size for 32x32 images. When the images cannot be evenly divided, we accommodate overlapping blocks at the edges. We follow the linear evaluation protocol (Goyal et al. 2019) in our experiments, where the pre-trained model remains static, and only the appended linear classification layer undergoes fine-tuning. All the results in this paper are the average of runs using different random seeds.
7

---

Published as a conference paper at ICLR 2024
Table 1: Results of training from scratch experiments. The encoder for ImageNet is ResNet50 and the encoder for CIFAR is ResNet18. The model is trained for 800 epochs.
ImageNet
CIFAR-I0 10
CIFAR-100
Method
Training Training
Training Training Time
Training Training
Acc.
Acc.
Acc.
FLOOPs
Time
FLOPs
FLOPs
Time
SimCLR
66.39
100%
100%
90.16
100%
100%
57.34
100%
100%
SimCIR + Back Razor
66.30
85%
89.82
64%
57.60
69%
SimCLR (same FI.OPs as Ours)
64.76
80%
86.07
48%
50.46
52%
SimCLR WWM (Ours)
66.25
80%
89%
90.10
48%
68%
57.69
52%
73%
SimSiam
71.12
100%
100%
90.80
100%
100%
57.21
100%
100%
SimSiam Back Razor
71.04
84%
91.05
57.87
70%
SimSiam (same FLOPs as Ours)
69.10
81 %
87.28
46%
52.18
53%
SimSiam + SIMWNW (Ours)
71.28
81%
90%
91.17
46%
65%
57.94
53%
71%
Table 2: Results of transfer learning experiments. The encoder is ResNet50, which is pre-trained on ImageNet and then trained on the three datasets for 100 epochs using self-supervised learning methods. After that, we apply linear evaluation protocol get the accuracy results.
Stanford Cars
FGCV Aircraft
CUB-200
Method
Training Training Acc.
Training Training Acc.
Training Training
Acc.
FLOPSs
Time
FLOPs
Time
FLOPs
Time
SimCLR
46.12
100%
100%
48.43
100%
100%
35.78
100%
100%
SimCLR Back Razor
46.10
62%
48.19
60%
36.27
59%
SimCLR (same FL.OPs as Ours)
43.03
53%
46.74
51%
33.07
55%
SimCLR + SIMWNW W (Ours)
46.38
53%
79%
48.26
51%
75%
36.15
55%
84%
SimSiam
50.87
100%
100%
51.82
100%
100%
38.40
100%
100%
SimSiam Back Razor
50.83
65%
51.68
61 %
38.07
57%
SimSiam (same FLOPs Ours)
46.22
50%
48.17
55%
36.80
49%
SimSiam + SIMWNW (Ours)
50.95
50%
78%
51.76
55%
73%
38.22
49%
74%
5.1 MAIN RESULTS
Training from scratch. Table I shows the accuracy and training FLOPs of training from scratch experiments. We apply our proposed SIMWNW framework two SSL frameworks SimCLR and SimSiam for evaluation. We compare our framework with a recognized activation gradient pruning approach Back Razor (Jiang et al. 2022). which prunes the activation gradient with lower mag- nitudes to reduce computation costs. In our experiments, we modulate the pruning ratio of Back Razor to ensure similar accuracy to SIMWNW. As shown in the table, compared to the SimCLR and SimSiam baselines, SIMWNW provides impressive training FLOPs savings, peaking at 54% and averaging at 40%. Importantly, this computational advantage does not come at the expense of accuracy. In comparison with Back Razor, NIMWNW reduces training FLOPs by up to 25%, with an average savings of 18%.
Transfer learning. In addition to the training from scratch tasks, further evaluate the perfor- mance of SIMWNW on transfer learning tasks. The experimental results are shown in Table As one can observe, compared SimCLR and SimSiam baselines, SIMWNW showcases a commend- able reduction in computation costs without compromising accuracy, peaking at 51 % and averaging 48%. When compared to Back Razor, NIMWNW manages to reduce computation costs by up to 23%, with an average saving of 14%.
With the savings in training FLOPs, SIMWNW is able to reduce the training time by 24% on aver- age. We also present the accuracy results when SimCLR and SimSiam basclines consume the same training FLOOPs with our proposed SIMWNW in Tables D and 2 and we can see that SIMWNW con- sistently provides significantly higher accuracy (3.5% on average) compared to them. The com- mendable performance of SIMWNW can be attributed to its astute focus on the intrinsic features of augmented images. By eliminating regions of less significance, it not only reduces computation costs but also removes unnecessary noise or irrelevant features that might mislead or slow down the learning process (Wang et al. 2022 Singh et al. 2016). Therefore, it allows the model to learn fea- ture extraction capabilities faster. In other words, the model can converge in fewer epochs, offering further computation savings.
8

---

Published as a conference paper at ICLR 2024
Figure@deepcct example of model con-
SimCLR
SimGLR-SimWWwW
SimSlam
Simmiin-Simmhnii
vergence training from scratch experi- ment on the CIFAR-10 dataset. It's evident that applying our WIM framework effectively accelerates the convergence, re- quiring more than 30% fewer epochs than the SimCLR and SimSiam baselines. The breakdown of computation savings, de- rived from skipping convolution compu- tation for removed regions, and ii) accel- erating model convergence, are detailed in Appendix [ how the computation savings translate into training time reduction in Appendix F On the other hand, Back Razor keeps dense activation during forward propagation, which means could miss opportunities for computation saving.
Accurasy 1h6
90
90
Accursey
Saving
Saving
70
70
2.5
3.8
1.2
2,4
3.6
Training Hours
Training Hours
(a)
Figure 6: The convergence of ResNet18 on CIFAR-10
dataset of training from scratch experiments.
We also present a detailed discussion on
We also conduct other experiments to evaluate SIMWNW, including comparison and compatibility to some efficient self-supervised learning frameworks (Appendices A.I on vision transformers (Appendix A.4), and transferability experiments (Appendix A.5
A.2 and A.3 experiments
11.0
Accuracy
Training Cost
100
Training Cost (Kg)
5.0
Accuracy
Training Cost
100
Training Coost (K6)
Accuracy Drop
80
Accuracy Drop
3.0
5.0
60
2.0
40
-1.
20
Similarity Threshold
Block Size
(b)
Figure 7: (a) Effect of different similarity thresholds. (b) Effect of different block sizes. All the
results are obtained from the transfer learning experiment on the Stanford Cars dataset.
5.2 SENSITIVITY STUDY
We investigate the effect of similarity thresholds. Figure shows accuracy drop and training FLOPs (compared to SimCLR and SimSiam baselines) when the similarity threshold varies. As expected, higher thresholds indicate fewer similar block pairs being removed, thus leading to lower accuracy drop but higher training computation cost. When the similarity threshold reaches a sufficiently high value (e.g.. 20), the accuracy essentially remains stable. However, for applications that are not particularly sensitive to accuracy, opting for aggressive threshold can more computation.
We also study the impact of the block size when we conduct block-matching on augmented images. As shown in Figure 7p, larger block size results in lower accuracy drop but higher training costs. The reason behind this is that a larger block captures more features and details. As the two augmented images have different transformations applied, the aggregated difference block will be more pronounced than in a smaller block. Therefore, a larger block size indicates fewer blocks considered similar. It is worth noting that the optimal block size is tied to the dimension of input images. Users have the flexibility to tailor the block size to suit specific applications. Some other dlock-matchin details and analyses are presented in Appendix
larger
CONCLUSION
In this work, we propose SIMWNW, similarity- efficient self-supervised learning frame- work. By removing the less important similar regions in augmented input images and feature maps, SIMWNW manages to skip unnecessary computations. Moreover, removing the less important re- gions can also remove irrelevant features that might slow down the learning process. Therefore, SIMWNW is also able to improve the model convergence speed, which can further reduce the com- putation costs. Our extensive experiments demonstrate that SIM WWW can achieve significant train- ing computation cost savings without compromising accuracy.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
The authors would like to thank the anonymous reviewers for their constructive feedback and sug- gestions. This work supported in part by NSF grants #2011146, #2154973, #2312157, #2312158, and #2052528.
REFERENCES
Sravanti Addepalli, Kaushal Bhogale, Priyam Dey, and R Venkatesh Babu. Towards efficient and effective self-supervised learning of visual representations. In European Conference on Computer Vision, pp. 523-538. Springer, 2022.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Yun-Hao Cao and Jianxin Wu. Rethinking self-supervised learning: Small is beautiful. arXiv preprint arXiv:2103.13:: 2021.
Yun-Hao Cao, Peiqin Sun, and Shuchang Zhou. Three guidelines you should know for universally slimmable self- supe-super learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15742-15751, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Yixiong Chen, Alan Yuille, and Zongwei Zhou. Which layer is learning faster? a systematic ex- ploration of layer- wise convergence rate for deep neural networks. In The Eleventh International Conference on Learning Representations, 2023. TRRUhttps:///peneeett www/:wwweeeeooooe.eo
[ddw1MDF11jF88
Yuanzheng Ci, Chen Lin, Lei Bai, and Wanli Ouyang. Fast-moco: Boost momentum-based con- trastive learning with combinatorial patches. In European Conference on Computer Vision, pp. 290-306. Springer, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning, In Proceedings of the ieee/cvf International Conference on computer vision, pp. 6391-6400, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 23:21271-21288 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369. IEEE, 2010.
Mingzheng Hou, Ziliang Feng, Haobo Wang, Zhiwei Shen, and Sheng Li. An adaptive regression based single-image super-resolution. Multimedia Tools and Applications, 81(20):8831-28248, 2022.
Jinggang Huang, Ann B Lee, and David Mumford. Statistics of range images. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume I, pp. 524-331. IEEE, 2000.
Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, and Zhangyang Wang. Back razor: Memory-efficiett transfer learning by self-sparsified backpropagation. Advances Neural Information Processing Systems, 35:29248-29266 2022.
Mustafa Taha Koçyigit, Timothy M Hospedales, and Hakan Bilen. Accelerating self-supervised learning via efficient training strategies. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5654-5664, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- tne-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Sheng Li and Hanxin Feng. Eeg signal classification method based on feature priority analysis and cnn. In 2019 international conference on communications, information system and computer engineering (CISCE), pp. 403-406 IEEE, 2019.
Sheng Li, Zanhan Ding, and Hongly Chen. A neural network- based teaching style analysis model. In 2019 11th International Conference Intelligent Human-Machine Systems and Cybernetics (IHMSC), volume 2, pp. 154-157. IEFE, 2019.
Sheng Li, Geng Yuan, Yue Dai, Youtao Zhang, Yanzhi Wang, and Xulong Tang. Smartfrz: An efficient training framework using attention-based layer freezing. In The Eleventh International Conference on Learning Representations, 2022.
Sheng Li, Geng Yuan, Yawen Wu, Yue Dai, Chao Wu, Alex K Jones, Jingtong Hu, Yanzhi Wang, and Xulong Tang. Edgeol: Efficient in-situ online learning edge devices. arXiv preprint 2rXiv:2401. 16694, 2024.
Haoyu Ma, Chengming Zhang, Xiaolong Ma, Geng Yuan, Wenkai Zhang, Shiwei Liu, Tianlong Chen, Dingwen Tao, Yanzhi Wang, Zhangyang Wang, et al. Hrbp: hardware-friendil regrouping towards block-based pruning for sparse enn training. In Conference on Parsimony and Learning, pp. 282-301. PMLR, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Jian Meng, Li Yang. Jinwoo Shin, Deliang Fan, and Jae-sun Seo. Contrastive dual gating: Learn- ing sparse features with contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12257-12265, 2022.
Nasru Minallah, Khadem Ullah, Jaroslav Frnda, Laiq Hasan, and Jan Nedoma. On the performance of video resolution, motion and dynamism in transmission using near-capacity transceiver for wireless communication. Entropy, 23(5):562, 2021.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of irvaexttinvariant representa- tions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6707-6717, 2020.
11

---

Published as a conference paper at ICLR 2024
SN Muralikrishna, Meghana Ajith, and KB Ajitha Shenoy. Msb based new hybrid image compres- sion technique for wireless transmission. In Advances in Computing and Information Technology: Proceedings of the Second International Conference Advances in Computing and Information Technology (ACTTY) July 13-15, 2012, Chennai, Volia-Voluue 2. pp. 749-756. Springer, 2013.
Colorado J Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Selfaugment: Automatic augmentation policies for self-super learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2674-2683 2021.
Sucheng Ren, Huiyu Wang. Zhengqi Gao, Shengfeng He, Alan Yuille, Yuyin Zhou, and Cihang Xie. A simple data mixing prior for improving self-supervised learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2022.
Dong-Hak Shin, Byung-Gook Lee, and Joon-Jae Lee. Occlusion removal method of partially oc- cluded 3d object using sub-image block matching in computational integral imaging. Optics Express, (6(21):16294-16682 2008
Amanpreet Singh, Narina Thakur, and Aakanksha Sharma. review of supervised machine learning algorithms. In 2016 3rd international conference computing for sustainable global develop- ment (INDIACom), pp. 1310-1315. leee, 2016.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing sys- fems, 33:6827-6839 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Peng Wang, Bing Xue, Jing Liang, and Mengjie Zhang. Differential evolution- based feature selec- tion: A niching-based multiobjective approach. IEEE Transactions on Evolutionary Computation, 27(2):296 310, 2022.
Yangjie Wei and ChengDong Wu. Modeling of nano piezoelectric actuator based on block matching algorithm with optimal block size. Science China Technological Sciences, 56:2649-2657, 2013.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pp. 11112-11122. PMLR, 2021.
Haohang Xu, Xiaopeng Zhang, Hao Li, Lingxi Xie, Wenrui Dai, Hongkai Xiong, and Qi Tian. Seed the views: Hierarchical semantic alignment for contrastive representation learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(3):3753-3761 2022.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, al. Mest: Accurate and fast memory-econmmic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-2088 2021.
Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of generic framework for sparse training. Advances in Neural Information Processing Systems, 35: 35:19061-19074 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
12

---

Published as a conference paper at ICLR 2024
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. Advances in Neural Information Processing Systems, 34:2543-2555 2021.
Shan Zhu and Kai-Kuang Ma. A diamond search algorithm for fast block-matching motion estimation. IEEE transactions on Image Processing, 92):287-290, 2000.
Wentao Zhu, Jingya Liu, and Yufang Huang. Hussl: Hard negative-based self- learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4777-4786, 2023.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXTENDED EXPERIMENTAL RESULTS
A.1 COMPARISON WITH SOTA EFFICIENT SUPERVIIE LEARNING METHODS
In this section, we compare our NIMWNW with two statc-of-the-an efficient self-supervised learn- ing methods (Addepalli al., 2022 Kocyigit al., 2023). Addepalli al. (2022) identifies the noise present in the training objective as a primary factor contributing to the slow convergence in self-supervised learning. To mitigate this, they introduce rotation prediction as an additional, noise- free training objective, aimed at expediting the model convergence. On the other hand, [Koyyihtt et al. (2023) proposes strategy to accelerate model convergence through a combination an in- novative learning rate schedule and input image resolution schedule. They also introduce new image augmentation technique, designed to improve the quality of augmented images, thereby fur- ther accelerating the convergence of the model. For these two SOTA methods, modulate their training epochs to ensure a similar accuracy to SIMWNW for a fair comparison. The experimental result is shown in Table 3 We can observe that SIMWNW consistently provides higher training time reduction with similar model accuracy. The reason behind this is that some computation re- duction operations applied by these two methods cannot effectively translate into time reduction. For example, using low-resolution images to reduce the training computation might degrade model performance, resulting the need for more training epochs.
Table 3: Comparison with SOTA efficient SSL methods.
ImageNet
CIFAR-10
Method
Accuracy
Training Time
Accuracy
Training Time
SimCLR
66.39
100%
90.16
100%
SimCLR
Addepalli et al.
2022
66.21
91%
90.04
80%
SimCLR
Koçyigit et
2023
66.30
96%
90.12
78%
SimCLR
SIMWNW (Ours)
66.25
89%
90.10
68%
SimSiam
71.12
100%
90.80
100%
SimSiam
Addepalli et al
2022
71.20
94%
91.19
75%
SimSiam
Kogyight
2023
71.19
95%
91.11
81%
SimSiam
SIM (Ours)
71.28
90%
91.17
65%
A.2 COMPATIBILITY OF SIMWNW
In this section, we further evaluate the compatibility of WwW with two recent efficient self- supervised learning frameworks Fast-MoCo (Ci al. 2022) and S3L (Cao & Wu (022I). These two works are complementary SIMWNW Specifically, Fast-MoCo accelerates the training process by adding more positive pairs to regulate the training process, thereby accelerating convergence. S31. accelerates the training process by using smaller-resolution images and partial backbone network. On the other hand, SIMWNW accelerates the SSL in a different dimension, which removes less important image blocks during training.
We integrate our proposed method SIMWNW into these two frameworks. We follow the setting in their paper and use the MoCo framework (Chen al.) (020b) as the baseline for S3L and MoCo v3 as the baseline for Fast-MoCo. Specifically, Fast-MoCSS divides the input image in the online branch into four patches and then combines their four output embeddings form six new embeddings, each of which involves two patches. In this case, the number of positive pairs is six times as normal training. Thus, it can get more supervision signals in each iteration and thus achieves promising performance with fewer iterations. For S3L, we follow their original setting for the ImageNet experiment in their paper, which uses 52x52 input images to train the model for 800 epochs and then uses 224x224 input images to train the model for 200 epochs. As shown in Table 4 applying SIMWNW NNW to the Fast-MoCo and S3L can further reduce the training cost by 23% and 30% without accuracy loss, respectively, demonstrating the compatibility of SIM NWW.
14

---

Published as a conference paper at ICLR 2024
Table Compatibility of SIMWNWWith SOTA efficient SSL framework. The encoder is ResNet50 and the dataset is ImageNet.
Method
Accuracy
Training Time
MoCo S3L (MoCo v2 based)
71.12
100%
69.96
65%
S3L + SIMWNW
70.06
46%
MoCo v3
72.28
100%
Fast-MoCo (MoCo v3 based)
72.46
30%
Fast-MoCo + SIMWNW
72.30
23%
A.3 COMPARISON WITH SPARSE SELF-SUPERVISED LEARNING APPROACH
There are also some other popular efficient learning methods such as layer freezing (uuan et al. 2022 Li et al. 2022) and sparse training (Yuan et al. 2021 Ma et al. 2024). For comprehensive evaluation, we compare our proposed SIM WNW framework with a sparse training approach tailored for self-supervised learning: Contrastive Dual Gating (CDG) (Meng et al. 2022). CDG exploits spa- tial redundancy by using a spatial gating function and skips those less important pixels in the feature maps. To ensure fair comparison, modulate the prune ratio of CDG to achieve accuracy that is comparable to our SIMWNW framework. Table 5 shows the experimental results. Under similar accuracy, SIMWNW outperforms CDG by saving 12% computation costs, demonstrating the superiority of our proposed SIMWNW framework.
Table Comparison with sparse training method CDG (Meng et al. 2022). Accuracies are obtained by linear evaluation. The encoder is ResNet1 8 with 800 training epochs.
CIFAR-10
CIFAR-10 100
Method
Accuracy
Training FLOPs
Accuracy
Training FLOPs
SimCLR
90. 16
100%
57.34
100%
SimCLR + CDG
90.01
56%
58% 57.50
SimCLR SIMWNW
90.10
48%
57.69
52%
A.4 EXPERIMENTS ON VISION TRANSFORMERS
In this section, we apply the proposed method WIMWWW to vision transformers. There is no re- moved region shrinking problem in vision transformers, which indeed makes easier to apply SIMWNW VIT. We also do not need to consider the block size since VIT naturally divides the image into many tokens, typically the size of 16x16. Therefore, removing similar blocks can be directly achieved by removing similar input tokens, resulting in reduced input sequence length. We use a well-recognized self sele-supe vision transformer learning framework DINO (Caron et al. 202I) our baseline. As shown in Table 6 WW significantly reduces the training cost by 38% without accuracy loss compared to the DINO haseline. We also compare our approach to an ef- ficient vision transformer training framework EVIT. For fair comparison, we apply EVIT to DINO and let it consume the same FLOPs as our approach. As shown in Table 6 SIMWNW achieves 0.82% higher accuracy than EViT with the same training computation.
Table 6: Evaluation on Vision Transformers. The encoder is DeiT-S and the dataset is ImtgeNet.
Method
Accuracy
Training FLOPs
Training Time
DINO
58.95
100%
100% 57%
DINO EVIT
58.01
72%
DINO SIMWNW
58.83
57%
72%
15

---

Published as a conference paper at ICLR 2024
A.5 TRANSFERABILITY
For a more comprehensive comparison, we conduct an experiment that directly fine-tunes the pre- trained ResNet50 model the downstream task without self-supervised learning on them. In this experiment, SIMWNW is applied to SimCLR and SimSiam baselines during the self-supervised pre-training on ImageNet. Then we directly train the classifier of the pre-trained ResNet50 using three downstream datasets. As shown in Table 7 applying SIMW SIMWNW during the pre-training stage will not hurt the accuracy when directly transferring the pre-trained model to downstream datasets.
Table 7: Accuracy results of transferability experiment.
Method
Stanford Cars
FGCV Aircraft
CUB-200
SimCLR
38.49
40.46
30.10
SimCLR SIMWNW (Ours)
39.90
41.28
30.19
SimSiam
39.78
38.37
32.89
SimSiam SIMWNW (Ours)
40.46
38.92
33.66
B BLOCK MATCHING DETAILS
Search region. During the block-matching process, for each block, when searching for the most similar block in the paired augmented image. the search region narrows down to specific region surrounding the block's counterpart. The reason why we search for spe- cific region is to handle the effects of image scaling and aspect ratio changes. When we perform data augmentation, RandomResizedCrop is a commonly used method, specifically transforma Random ResizedCropo(224 scale = (s1, s2), ratio = (r1, r2)). This method oper- ates as follows. Initially, randomly selects scale ratio between s and s2, and the image scaled up or down according to the randomly picked ratio. Additionally, it will randomly pick an aspect ratio between rl and r2, and the aspect ratio of the image is accordingly modified. After the scaling and aspect ratio adjustment, the image is then cropped and resized, this case, 224x224 pixels.
When we apply the augmentation to the original image twice to get two augmented images for two branches, the randomly picked scale ratio and aspect ratio are likely to be different for the two augmented images. Therefore, the corresponding regions in the two augmented images, which come from the same region in the original image, be different in size and shape. Therefore, it is hard to directly pair the exact matches and we need to search for a neighborhood region of the counterpart block. In practice, the image scaling ratio is generally not more than twice, the length and width of the search region are set to twice the length and width of the block size to ensure that the most similar blocks are included. For example, the search region is 60x60 for 30x30 block. We also perform a sensitivity study on the search region size and the results in Table &how that WWW is robust different sizes of the search region.
Block removal ratio. Table P presents the block removal ratio with the default similarity threshold 20 in our experiments. In practice, the block removal ratio varies across datasets. This is because different datasets have different image complexity and different resolutions, so the number of similar blocks that can be found under the same threshold varies.
Table Accuracy when using different search region sizes. The block size is 30x30 and the results are obtained from transfer learning experiments on the Stanford Cars dataset. The base framework SimSiam.
Size of Search Region
45x45
60x60
75x75
Accuracy
50.72
50.95
50.98
C OVERHEAD ANALYSIS
The computational overhead in our framework primarily arises from the calculation of PSNR, which employed for similarity checks between pair of images and feature maps. However, the silver lin-
16

---

Published as a conference paper at ICLR 2024
Table Image block removal ratio.
Dataset
ImageNet
CIFAR-10
CIFAR-100
Stanford Cars FGCV Aireraft
CUB-200
Removal Ratio|
20%
30%
30%
33%
31%
35%
ing that PSNR computation is relatively lightweight and hence does not introduce a large amount of computation overhead. The formula for PSNR is derived from the Mean Squared Error (MSE) between the two images:
M-IN-1 Σ 1=0
MSE
EI ((, -((,,))
(4)
MN
3=0
where 7 and K represents two images (blocks). Using the MSE, the PSNR is calculated as PSNR 10- log (MAX/MMSE) where MAX, is the maximum possible pixel value of the image.
Considering calculating PSNR for two 30 30 image blocks, the calculation requires 30 x 30 900 subtractions to determine differences, 900 multiplications to square these differences, and 899 additions to sum the squared differences. While the PSNR computation does encompass additional operations, such as logarithmic operation, their quantity is minimal and they hold negligible numer- ical impact on computation cost. Overall, the computation amounts to approximately 2,700 FLOPS.
Although the cost for calculating PSNR once is small, we also apply further methods to reduce the overhead by reducing the number of PSNR calculations. For block-matching in the paired aug- mented images, each block's 's search space is limited to the semantically related, constrained region in its counterpart image, leading space, the efficient search method diamond search is invoked, enhancing efficiency. Note, when checking the similarity between paired feature maps, the process direct one-to-one match with- out a searching process (the detail of identifying similarities in feature maps is in Section 4.2).
substantial reduction in the search space. Within this limited
In contrast, performing convolution on two 30 x 30 images with 3 filter, leading to an output space of 28x28 (due to boundary effects), has much higher computation cost. Each pixel in the image necessitates 9C' multiplications and 8C additions, where C' represents the number of output channels. For the entire 30x30 block, the required FLOPs are given by:
(5) Suppose the number output channels = 64, the convolution operation demands 852,992 FLOPs for a single image. Consequently, for the paired augmented image (blocks) in two branches, the computation reaches 1.7 MFLOPs.
FLOPs 28 x 28 x (9C' + 8C)
To summarize, convolution operation on the image blocks requires computational resources that are several orders of magnitude higher than what's required for PSNR calculation on the image blocks. Morcover, our design limits the search space of each block, thereby ensuring that the PSNR not calculated too many times. As a result, the overhead introduced is minimal when compared to the gains in computation savings. In experiments, the computation overhead of PSNR is less than 1% and the overhead is already included in the reported results.
D COMPUTATION SVAINGS BREAKDOWN
In this section, we break down the computation savings achieved by SIMWNW framework. Through strategically removing regions of less importance from augmented images and feature maps, SIMWNW achieves computational efficiency in two ways: i) skipping convolution computation for the removed regions, and ii) improving the model convergence speed. Table 10 and Table [1] present the breakdown of computation savings in training from scratch and transfer learning tasks, respectively.
E ANALYSIS ON FLOPS REDUCTION
The FLOPs reduction of our SIMWNW mainly comes from two aspects. For the first aspect, our SIMWNW can improve the model convergency speed, indicating fewer number of training itera-
17

---

Published as a conference paper at ICLR 2024
Table 10: Breakdown of Computation Savings in training from scratch experiments. "Skipped Convolution" represents the savings achieved by skipping convolution computation for the removed region. "Epochs Reduction" indicates the savings in training epochs derived from improved model convergence speed.
ImageNet Skipped Convolution
CIFAR-I0
CIFAR-100
Method
Overall Savings
Epochs Reduction
Overall Savings
Skipped Convolation
Epochs Reduction
Overall Savings
Skipped Convolution
Epochs
Reduction
SimCLR + WIMWKW
20%
12%
11%
52%
29%
32%
48%
29%
27%
SimSiam SIMWNW
Table 11: Breakdown of Computation Savings in transfer learning experiments. "Skipped Convolu- tion" represents the savings achieved by skipping convolution computation for the removed region. "Epochs Reduction" indicates the savings in training epochs derived from improved model conver- gence speed.
Stanford Cars
FGCV Aircrali
CUB 200
Method
Overall Savings
Skipped Convolution
Ppochs Reduction
Overall Savings
Skipped Convolution
Epochs Reduction
Overall Savings
Stipped Convolution
Ppachs Reduction
SimCI SIMWWW SimSiam SIMWNW
47%
33%
32% 25%
25%
45%
35% 34%
17%
50%
36%
22%
45%
27%
51%
26%
tions/epochs required achieve target accuracy. This can directly lead to FLOPs reduction and training time savings without any dedicated sparse computation support. Specifically. SIMWNW re- moves the less important regions, resulting in removing irrelevant features that slow down the learn- ing process, thereby improving model convergence speed.
The second aspect of FLOPs reduction is achieved by removing similar blocks. For the ViT-based models, removing similar blocks can be directly achieved by removing similar input tokens, resulting in reduced input sequence length. This also directly achieve acceleration, while does not require any dedicated sparse computation support.
For the case of using CNN models, SIMWNW indeed requires some support for sparse computa- tion. This is similar problem faced by the designs in other fields, such as sparse training or weight pruning. This usually can be solved in different ways. For general- purpose devices such as GPUs or mobile devices, SIMWNW can be supported by using sparse computation libraries and compiler optimizations. For FPGA platforms, the convolution kernels need be divided into tiles and com- puted separately. So, the tiling size used in FPGAs can be aligned with the block size used in the SIMWNW. In this way, we can easily skip the computation clock cycle for the corresponding block, leading to direct time-saving. It is worth mentioning that, in our SIMWNW, we remove the entire similar blocks during the computation, which creates a coarse-grained sparsity. Compared to the unstructured irregular sparsity which is usually used in sparse training or weight pruning works, the coarse- coarse-grained sparsity created in our SIM WNW is much more friendly for sparse computation acceleration both general-purponn devices and FPGA platforms.
18


Reviews:
[{"review_id": "IRNjctjFRK", "reviewer": "Reviewer_vumm", "summary": "The authors propose a method for improving the efficiency of SSL methods by discarding features in augmented images and feature maps that are deemed less important, saving computation and reducing the risk of slowing the learning process by providing irrelevant features. The authors propose to remove blocks from pairs of augmented images that share high semantic similarity, in order to prevent unnecessary processing of irrelevant information such as image backgrounds. To this end, they provide a method for semantic matching of block pairs in images, their removal, and the treatment of the resulting feature maps throughout the network. Authors show results for training from scratch and transfer learning compared to a number of other SSL methods, in most cases showing barely degraded performance - or even improved performance - at a significantly reduced computational cost.", "questions": "Could you give a little more explanation for figure 1. In my opinion, the first two paragraphs of 3.1 read a bit confusingly. What is the distinction between the block matching described in the first paragraph and the similarity calculation after the creation of block pairs in the second paragraph? Aren’t they overlapping?\n\nHow does computational complexity of the block-matching factor into the overall training complexity? I.e. do the FLOPs listed in tables 1 and 2 contain the overhead for your method? I think this should definitely be taken into account.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide a sensible method for improving computational efficiency of SSL methods, one of their main challenges currently. The authors are very thorough in motivating and describing their method, using illustrative examples throughout the paper. Experimental results are impressive, the proposed method shows good performance in its ability to reduce computational cost while retaining model performance.  A very sound paper overall, with good experimental design. Given that the authors spend some time sculpting the manuscript to improve its readability for the rebuttal, I think it represents an interesting and valuable addition to the CVPR proceedings.", "weaknesses": "Overall readability of the paper could be improved, I’m having a bit of a hard time understanding some of the specifics of the approach as outlined in 3.1 and 3.2. Specifically, the block matching as outlined in paragraphs 1 and 2 under 3.1 seem to overlap; from my understanding you first search for most similar block pairs (paragraph 1) after which you calculate similarity for all block pairs (paragraph 2)? Why not calculate similarity for all block pairs directly?\n\nUnder 4.1, you indicate that, for a given pair of original and augmented image, you divide the first into blocks and loop for a similar block in the paired image. However, instead of performing an exhaustive search over all possible blocks in the augmented image, you narrow the search to “a specific region surrounding a block’s counterpart in the paired augmented image” to ensure semantic consistency. Where does this block’s counterpart come from? Is it simply the same augmentation applied to the block in the original image, i.e. the location of the original block under a flip? In this case, why would the same block in the augmented image not be the most similar block? Semantically, their content is identical is it not? Could you give an intuition as to why you would want to pair image blocks in the same region in the online and target images but not simply pair exact matches under augmentation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EyI4OL18ga", "reviewer": "Reviewer_Aoaq", "summary": "The authors aim to improve the training efficiency of self-supervised learning (SSL) and they propose a similarity-based SSL framework called SIMWNW. SIMWNW removes less important regions (remove most similar regions in two views) in augmented images and feature maps and saves the training cost. Experimental results show that SIMWNW reduces the amount of computation costs in SSL.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow.\n2. The authors analyze the importance of different regions on augmented images by removing and reusing similar blocks for the two branches.\n3. The authors show that the removed region will shrink after convolution operation and they propose to expand the size of removed region in the feature map.\n4. Experimental results show that the proposed method can achieve comparable accuracy using fewer training FLOPs.", "weaknesses": "1. Compared with the training FLOPs, the actual time used for training is more important, and the authors did not report it. How much the proposed method can reduce the training time is what we are concerned about. Steps such as matching in the method cannot actually be reflected intuitively through FLOPs.\n2. In Table1 and Table2, the authors should list the accuracy of the baseline methods using the same training overhead. For example, how much lower will simclr be than the proposed method when using 80% overhead?\n3. Do the training FLOPs in Table2 refer to pre-training or downstream fine-tuning? If it is the former, why is it different from Table1？If it is the latter, how is the proposed method used in single-branch supervised learning?\n4. From Figure 6, I cannot see the obvious advantages of the proposed method. I suggest the author change the horizontal axis to training hours.\n5. Some related works [1], [2].\n\n[1] Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches. ECCV2022.\n\n[2] Rethinking Self-Supervised Learning: Small is Beautiful. arXiv 2103.13559.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NEKQKZ6ap", "reviewer": "Reviewer_5JXS", "summary": "This paper proposes to enhance the efficiency of self-supervised learning (SSL). Based on contrastive SSL methods, such as SimCLR and SimSiam, this paper proposes to reuse and remove the similar regions so as to save computation. To achieve this, this paper first identifies the similarities between regions. However, directly operating on regions would face the region shrinking problem caused by convolution layers, this paper proposes to expand the size of removed region. Compute savings in FLOPs in observed in ImageNet benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Self-supervised learning is computation expensive, this paper proposes to reduce the pretraining cost while preserving the accuracy, which is important topic for the community. \n\n+ The idea of reusing and replacing similar regions is intuitive. Also I am not sure if there are other similar works proposing similar ideas, it is good to see these simple yet effective training techniques.", "weaknesses": "- This paper claims that the proposed method is efficient regrading the FLOPs. However, reduced FLOPs may not directly lead to time saving given that the proposed method requires dedicated sparse computation of convolutional kernel. It is important to report the real run time saving to claim efficiency.\n\n- In the title, authors claim the proposed method is generic. It is worth to apply SimWnW to self-supervised vision transformers as well. Moreover, the reuse and replace strategies are expected to be applicable to ViTs since there would be no region shrinking problem in ViTs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SIGSqs7RbQ", "reviewer": "Reviewer_wGaZ", "summary": "This paper proposes an efficient SSL approach called SimWnW. Through studying the impact of similar and dissimilar image regions on SSL performance, the authors find that similar regions are less important and removing them in augmented images (and in feature maps) can significantly reduce the computation cost and improve model convergence. To remove similar regions, the authors propose a new method under the ResNet/ConvNet settings. Specifically, a waxing-and-waning process is proposed for region removal while mitigating the region shrinking problem in convolutional layers. Experiments show that SimWnW can reduce the computation cost of SSL without compromising accuracy -- SimWnW yields up to 54% and 51% computation savings in training from scratch and transfer learning tasks, respectively.", "questions": "Key question around the portion of block removal:\n- Intuitively, comparing similar blocks won't generate too much useful signal for SSL. This is validated by Fig. 2 where the performance of \"Similar Blocks (x\\%)\" is consistently worse than \"Dissimilar Blocks (x\\%)\". On the other hand, comparing dissimilar blocks (after removing similar ones), despite being more useful, has a key hyper-parameter of the removing portion (1-x)\\% which can significantly affect the learning quality. Specifically, if we remove too much, comparing those top dissimilar blocks either makes learning too hard or the dissimilar blocks may not even be semantically related (which hurts SSL quality). If we gradually increase x\\%, the retained blocks would include both dissimilar and relatively similar blocks, which makes the learning signals more balanced for SSL.\n- Fig. 2 shows that SSL performance peaks at \"Dissimilar Blocks (75\\%)\". What's the actually used x\\% after region removal in SimWnW? If it's 75\\% or higher, then it shouldn't lead to that much of computation saving. Fig. 7(a) shows some hint about x in terms of similarity threshold. 1) When the default threshold is set to 20, what's the corresponding x\\%? 2) With the default similarity threshold 20, the SSL performance remains about the same but the training cost is increasing. So again, the computation saving is still concerning. Any comments?\n- One side question, why the compute saving on ImageNet is much smaller than CIFAR 10/100? This suggests the amount of removed blocks from high-resolution ImageNet images is smaller than that of low-resolution CIFAR images, given the same similarity threshold (if that's how it works). Any intuitions about why this is the case?\n\nOther minor questions:\n- To find similar blocks, what's the neighborhood size for searching? Does it depend on augmentation parameters? - since how we crop/rotate/flip images will impact the block locations a lot.\n- For \"block matching\" in pixel space, is PSNR an accurate enough metric? What if the found correspondence is wrong and how well can SimWnW tolerate such errors?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper offers a comprehensive exploration of the impact of similar/dissimilar regions on SSL accuracy, which lays a good foundation for a region removal-based method to improve SSL efficiency.\n- Strong results in efficiency boost are achieved for two representative SSL frameworks.\n- Decent analysis is provided for region removal-related hyper-parameters like similarity threshold and block size.", "weaknesses": "- The key hyper-parameter of block removing portion is unspecified, and convincing explanations are missing (see questions below).\n- The comparisons with recent related works seem insufficient, e.g. (Addepalli et al., 2022) and (Koc¸yigit et al., 2023).\n- The proposed waxing-and-waning method is customed too much to ConvNets. It seems hard to translate to transformers and hence transformer-based SOTA SSL methods (this makes the paper title a bit overclaim).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)