Synchronized Contrastive Pruning for Efficient Self-Supervised Learning
Authors: Jian Meng
Li Yang
Deliang Fan
Jinwoo Shin
Jae-sun Seo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Sparse Training
Self-supervised Learning
Neural Network Pruning

Abstract: Various self-supervised learning (SSL) methods have demonstrated strong capability in learning visual representations from unlabeled data. However, the current state-of-the-art (SoTA) SSL methods largely rely on heavy encoders to achieve comparable performance as the supervised learning counterpart. Despite the well-learned visual representations, the large-sized encoders impede the energy efficient computation, especially for resource-constrained edge computing. Prior works have utilized the sparsity-induced asymmetry to enhance the contrastive learning of dense models, but the generality between asymmetric sparsity and self-supervised learning has not been fully discovered. Furthermore, transferring the supervised sparse learning to SSL is also largely under-explored. To address the research gap in prior works, this paper investigates the correlation between in-training sparsity and SSL. In particular, we propose a novel sparse SSL algorithm, embracing the benefits of contrastiveness while exploiting high sparsity during SSL training. The proposed framework is evaluated comprehensively  with various granularities of sparsity, including element-wise sparsity, GPU-friendly N:M structured fine-grained sparsity, and hardware-specific structured sparsity. Extensive experiments across multiple datasets are performed, where the proposed method shows superior performance against the SoTA sparse learning algorithms with various SSL frameworks. Furthermore, the training speedup aided by the proposed method is evaluated with an actual DNN training accelerator model. 

Under review as a conference paper at ICLR 2023
SYNCHRONIZED PRUNING FOR EFFICIENT CONTRASTIVE SELF-SUPERVISER LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Various self-supervised learning (SSL) methods have demonstrated strong capa- bility in learning visual representations from unlabeled data. However, the current sate-oo-thh-asd (SoTA) SSL methods largely rely on heavy encoders to achieve comparable performance the supervised learning counterpart. Despite the well-learned visual representations, the large-sized encoders impede the energy- efficient computation, especially for resource-constrained edge computing. Prior works have utilized the sparsity-inducee -induced asymmetry to enhance the contrastive learning of dense models, but the generality between asymmetric sparsity and self-supervised learning has not been fully discovered. Furthermore, transferring the supervised sparse learning SSL is also largely under-explored. To address the research gap in prior works, this paper investigates the correlation between in- training sparsity and SSL. In particular, we propose novel sparse SSL algorithm, embracing the benefits of contrastiveness while exploiting high sparsity during SSL training. The proposed framework is evaluated comprehensively with vari- ous granularities of sparsity, including element- sparsity, GPU-friendly N:M structured fine-gras grained sparsity, and hardware-specifie structured sparsity. Ex- tensive experiments across multiple datasets are performed, where the proposed method shows superior performance against the SoTA sparse learning algorithms with various SSL frameworks. Furthermore, the training speedup aided by the proposed method is evaluated with an actual DNN training accelerator model.
INTRODUCTION
The early empirical success of deep learning was primarily driven by supervised learning with mas- sive labeled data, e.g.. ImageNet (Krizhevsky et al., 2012). To overcome the labeling bottleneck of deep learning, learning visual representations without label-intensive datasets has been widely investigated (Chen et al., 2020a; He et al., 2020; Grill et al., 2020: Zbontar et al., 2021). The re- cent self-supervised learning (SSL) methods have shown great success and achieved comparable performance to the supervised learning counterpart. The common property of various SSL designs is utilizing different augmentations from the original images to generate contrastiveness, which re- quires duplicated encoding with wide and deep models (Meng et al., 2022). The magnified training effort and extensive resource consumption make the SSL-trained encoder infeasible for on-device computing (c.g., mobile devices). The contradiction between label-free learning and extraordinary computation cost limits further applications of SSL, also necessitating efficient sparse training tech- niques for self-supe learning.
For supervised learning, sparsification (a.k.a. pruning) has been widely explored, aiming to reduce computation and memory costs by removing unimportant parameters during training or fine-tuning. Conventional supervised pruning explores weight sparsity based on a pre-trained model followed by additional fine-tuning to recover the accuracy (Han et al., 2016). For self-supervised learning, recent work (Chen et al., 2021) also sparsified pre-trained dense SSL model for the downstream tasks with element-wise pruning. In addition to the fine-grained sparsity, MCP (Pan et al., 2022) exploited the filter-wise sparsity on the MoCo-SSL (He et al., 2020) model. Both of these sparse SSL works (Chen et al., 2021; Pan et al., 2022) exploit sparsity based on the pre- pre-trained dense model. However, compared to supervised learning, obtaining the pre-trained model via SSL requires a significant amount of additional training effort (~200 epochs vs. 1,000 epochs). Therefore, exploring post-training sparsity via fine-tuning is not ideal solution for efficient SSL..

---

Under review as a conference paper at ICLR 2023
Applying SoTA Prune-and-regrow to SSL
Self-Damaging SSL
This work
Loss
Loss
Loss
Consistent
Unstable Contrastiveness
Consistent Contrastiveness
Contrastiveness
High Sparsity
EMA/
Regrow-
EMA COPY
EMA Copy
Synchronized Spars. Gup: A
Dense Online Encoder
Sparse Offline Encoder
Sparsity Online Encoder
Sparse Offline Encoder
Sparsity Onlline
Sparse Encoder
Encoder
SSL Friendly !
SSL Unfriendly
@ HW Compatibilit
SSL Friendly
Low Compatibility
High HW Compatibility
Unproved Generality
High Sparsity
Hlgh Sparsity
(a)
(b)
(c)
Figure 1: (a) Applying self-damaging scheme (Jiang et al., 2021) to SSL. (b) Applying prune-and- regrow scheme (Liu et al., 2021) to SSL.. (c) Proposed contrastiveness-awaa aware sparse training.
On the other hand, sparsifying the model during supervised training (Dettmers & Zettlemoyer, 2019; Evci et al., 2020) has emerged as promising technique to elevate training efficiency while obtaining sparse model. To accurately localize the unimportant parameters, prior works investigated various types of important metrics, including gradient-basee pruning (Dettmers & Zettlemoyer, 2019) and the "prune-regrow" scheme (Liu et al., 2021). Compared to the post-training sparsification methods, in-training sparsification for supervised training has achieved memory/computation reduction as well as speedup of the training process. However, exploiting in in-training sparsity for SSL models that are trained from scratch is still largely under-cxplored.
More recently, SDCLR (Jiang et al., 2021) proposed the sparsified "self-damaging" encoder, which generates the "sparse-dense" SSL by exploiting fixed high sparsity on one contrastive path (e.g., off- line encoder), while keeping the counterpart dense (e.g.. online encoder), as shown in Figure (a). Such "sparse-dense" SSL. architecture helps to enhance contrastive learning, leading to improved performance with the non-salient and imbalanced samples. Nevertheless, it mainly focuses on the performance enhancement of the SSI.-trained dense model (i.e., SimCL. (Chen et al., 2020a)), and whether such a "sparse-dense" asymmetric learning scheme works in other SSL methods re- mains unclear. On the other hand, the compatibility of the existing SoTA sparse training techniques (Dettmers & Zettlemoyer (2019); Evci et al. (2020); Liu et al. (2021)) is also ambiguous to SSL. As shown in Figure 1(b), they require to frequently prune and regrow the model architecture during SSL training. while SDCLR maintains fixed "sparse-dense" architecture during the entire training process. The under-explored sparse contrastiveness and expensive self self-supervised learning inspire us to investigate the following question: How to efficiently sparsify the model during self- supervised training with the awareness of contrastiveness?
To address this question, we present Synchronized Contrastive Pruning (SyneCP), novel sparse training algorithm designed for self-supervised learning. To maximize the energy efficiency of SSL training. SyncCP exploits in n--training sparsity in both encoders with high compatibility of SSL. The main contributions of this work are:
We first discover the limitations of the sparsity-induced asymmetric SSL in SDCLR (Jiang et al., 2021) and show that the sparsity-induced "sparse-dense" asymmetric architecture is not universally applicable for various SSL schemes.
We demonstrate the incompatibility of the SoTA "prune-and-regrow" sparse training method for SSL. Specifically, we formalize the iterative architectural changes caused by applying grune-andd-egrown to SSL, named as architecture oscillation, and observe that frequently updating the pruning candidates lead larger architecture oscillation, which further hinders the performance of self-supervised learning.
We present SyncCP, new sparse training algorithm designed for self-supervised learn- ing. SyncCP gradually exploits high in-training sparsity in both encoders with contrastive
2

---

Under review as a conference paper at ICLR 2023
sparsification, maximizing the training efficiency without hurting the contrastiveness of SSL.
synchronization and
SyneCP is general sparse training method for SSL which is compatible with various granularities of sparsity, including element-wise pruning, N:M sparsity, and structured sparsity designed for custom hardware accelerator.
We validated the proposed method against previous SoTA sparsification algorithms on CIFAR-IO, CIFAR-100, and ImageNet datasets. Across various SSL frameworks, SyneCP consistently achieves SoTA accuracy in all experiments.
2
RELATED WORKS
2.1
CONTRASTIVE SELF-SUPERVISED LEARNING
Self-supervised learning recently has gained popularity due its ability to learn visual representa- tion without labor-intensive labeling. Specifically, pioneering research works (He et al., 2020; Chen al., 2020a) utilize the contrastive learning scheme (Hadsell al., 2006) that aims group the correlated positive samples while repelling the mismatched negative samples (Oord et al., 8). The performance of the contrastive learning- based approaches largely depends on the contrastive- ness between the positive and negative samples, which requires large-sized batches to support. As indicated by SimCLR (Chen et al., 2020a), the performance of SSL. is sensitive to the training batch size, and the inflated batch size elevates training cost. MoCo (He et al., 2020; Chen et al., 2020b) alleviates such issue with queue-based learning and momentum encoder, where the extensive queue- held negative samples provide proficient contrastiveness, and the slow-moving average momentum encoder derives consistent negative pairs. BYOL (Grill et al., 2020) simplifies and outperforms the prior works by only learning positive samples, while the online latent features projected by additional predictor qo:
online prediction = go(oo(Fo(X)))
offline target (Xe))(
(2)
Where f and g represent the encoder and projector of online (0) and offline (E) paths with augmented input X and X', respectively. The predictor do generates an alternative view of the projected positive samples, and the offline momentum encoder provides consistent encoding for contrastive learning. Overall, salient and consistent contrastiveness is essential to contrastive self-supervised learning.
2.2 SPARSE TRAINING
DNN sparsification has been widely investigated under the supervised learning domain, which can be generally categorized based on the starting point of sparsification. Early works mainly focus on post-training sparsification (Han et al., 2016; Evei et al., 2020; Jayakumar et al., 2020), which removes the weights from the pre-trained model and then recovers the accuracy with subsequent fine-tuning. Other works exploit weight sparsity prior to the training process (Wang et al., 2019; Lee et al., 2018), and the resultant model is trained with the sparsified architecture.
In contrast to post-training or pre-training sparsification, exploiting sparsity during training gener- ates the compressed model with one-time training from scratch, eliminating the requirement of pre-trained model or extensive searching process. With the full observability the training pro- cess, the magnitude of the gradient can be used evaluate the model reflection with the exploited sparsity. Motivated by this, GraNet (Liu et al., 2021) utilizes the "prune-and-regrow" technique to periodically remove the unimportant non-zero weights from the sparse model and then regrow cer- tain pruming candidates back. Given the targeted sparsity s; and total prune ratio rt at iteration f, unimportant weights w are removed based on the Top-K K magnitude scores:
w = TToKK(ww),r) rt)
(3)
Subsequently, the sensitive weights are re- re-grown back based on the reflection of gradient g":
w
Since the gradient 9; indicates the instant model sensitivity at iteration t, the optimal sparse model architecture can be varied between two adjacent pruning steps.
3

---

Under review as a conference paper at ICLR 2023
2.3 CONTRASTIVE LEARNING WITH SPARSITY-INDUCE ASYMMETRY
As introduced in Section 2.1, salient and consistent contrastiveness is essential for contrastive SSL, where the contrastiveness can be constructed via negative samples or the auxiliary predictors (Grill et al., 2020). Inspired by (Hooker et al., 2019), SDCLR (Jiang et al., 2021) amplifies the con- trastiveness by pruning one encoder of SimCLR (Chen et al., 2020a) while keep the identical twin dense. Such "sparsity-induce formance of the dense model on the long-tailed data samples. However, SDCLR (Jiang et al., 2021) is not designed for model compression or efficiency improvements. Furthermore, the generality of such sparsity-induced asymmetry remains under-explored in other SSL frameworks.
asymmetry" clevates the performance of SSL with the improved per-
CHALLENGE OF SPARSE SELF-SUPERVISED LEARNING
3.1 LIMITATIONS OF SPARSITY-INDUCHE ASYMMETRY
has been shown in SDCLR (Jiang et al., 2021) that the sparsity-induced "sparse-dense" asym- metry is beneficial to contrastive SSL. SDCLR (Jiang al., 2021) is specifically built upon the SimCLR (Chen et al., 2020a) framework with shared encoders, where the pruned architectures have the dense twin in the mirrored contrastive encoder. However, the generality of sparsity- -nnducd asymmetry remains unproved in other SSL methods, which motivates us to investigate the question:
Question 1: For contrastive self-supervised learning with non-identical encoders, will the sparsity- induced asymmetric encoders still result in elevated performance for contrastive learning?
To answer the above question, use MoCo-Va V2 (Chen et al., 2020b) and follow the procedure of SDCLR (Jiang et al., 2021) to generate a highly-sparse online encoder prior to the training process. Given the online and offline (momentum) encoder and with weights Wa and Wg, we have:
online output = go gf (Ms We)))
(5)
offline output 95 (X'))
(6)
The online encoder mask Ma produces sparse online encoder with initialized element-wise spar- sity (Han et al., 2016) at 90%, while the offline encoder is updated by exponential moving aver- age (EMA). The gradient-based update of the online encoder keep recovering the performance drop caused by the high sparsity mask. Following the setup of SDCLR (Jiang et al., 2021), the sparsity is periodically updated at the beginning of each epoch. Table summarizes the linear evaluation accuracy on the CIFAR-10 dataset with different static online sparsity values. As opposed to the performance of SimCLR in (Jiang et al., 2021), directly applying the high sparsity- perturba- tion to MoCo-V2 (Chen et al., 2020b) is challenging, and leads to considerable performance degra- dation. Reversing the sparsity between online and offline encoder also shows the similar results, as presented in Appendix D.
Table I:
Largely degraded performance of MoCo-V2 (Chen et al., 2020b) with self-damaging SSL (Jiang et al., 2021) on CIFAR-10 dataset.
R8
Dense Model Ace. = 92.09%
Encoder
Online 90%
Momentum
Online
Momentum
Fixed Sparsity
0%
50%
0%
Linear Eval. Ace (%)
88.72 87.68 (-4.31) 92.10 (+0.01%) 92.07 (-0.02%)
Summarizing these empirical results, our main observation is:
Observation 1: Compared to the online encoder, the EMA-updated momentum encoder has the delayed architecture, which makes it unqualified be the "compptitorr" as SDCLR (Jiang et al., 2021). The directly-applied high sparsity overshoots the asymmetric learning, leading to degraded self- learning.
3.2 FREQUENT ARCHITECTURE CHANGING HINDERS SELF-SUPERVISER LEARNING
As depicted in Eq. 4, the "prune-and and--e--odrrero""" scheme such as GraNet (Liu et al., 2021) uses instant gradient magnitude to indicate the model sensitivity after magnitude pruning, removing the unimpor- tant and insensitive weights while gradually achieving high sparsity. Observation demonstrates the incompatibility of the directly-applics high sparsity in SSL., then the following question raises:
4

---

Under review as a conference paper at ICLR 2023
Table 2: Sparse training with "prune-and-reeroon scheme on BYO (Grill et al., 2020).
BYOL (Grill et al., 2020)
CIFAR-IU 10 Acc (%)
Re
Dense Model Acc. 92.42%
Online Encoder Sparsity
0%-+80%
0%-+90%
Online Linear Eval. Ace (%)
91.20++0.02
90.10+0
Momentum Encoder Sparsity
-50%
0%-~
Momentum Linear Eval Acc. (%)
91.31=0.07
90.09=0.04
100
Architacture Osciltation (%)
8C1
Lo
Changing
80
Asymmetry!
L12
SC3
20
Online Encoder Sparn,
Momentum Encoder Spars.
10
30
50
200
400
608
800
1980
Prune Stops
Training Epochs
(a)
(b)
Figure 2: (a) Layer-wise oscillation at different steps of pruning. "SC"
stands for the shortcut
connection of ResNet-18 model. (b) Gradually-inereased sparsity of GraNet (Liu et al., 2021) leads to inconsistent asymmetry.
If we apply the gradually-increase sparsity for both encoders, will the "prune-and- regrow" scheme also be feasible for self-supervised learning?
Question 2:
To address Question 2, we use the SoTA GraNet (Liu et al., 2021) as the example algorithm to ex- ploit in ttainiiinn sparsity on both encoders of BOL (Grill et al., 2020), where the regrowing process is only applied to the online encoder. Starting with the dense models, we gradually prune the online and offline encoders 90% and 60% sparsity in an element-wise fashion with periodically-updan sparsity. For sparse SSL training, the results of such gently-increased sparsity scheme reported in Table outperforms those by (Jiang et al., 2021) (Table 1) by significant margin. However, the state-of-the-arr supervised pruning algorithm still incurs 2.3% linear evaluation accuracy degrada- tion with SSL on the CIFAR- 10 dataset.
Compared to the self-damaging SSL with fixed sparsity (Jiang et al., 2021), the grune-and-regrow method keeps swapping the pruning candidates to minimize the model sensitivity. oscillating the en- coder architecture during training. We quantify such architecture oscillation by XORing the masks generated from magnitude pruning MMP and gradient-based regrow Mg:
Geor MMP Mg {0,
Under the same sparsity ratio, the number of "I"s in Geor indicates the amount of architecture oscillation caused by the gradient- basedd regrow. During the early stage of training. almost all the magnitude pruning candidates are replaced by the regrowing process, as shown in Figure 2(a). The high degree of architecture oscillation implies drastic changes in the sparse model architecture. In the meantime, gradually sparsifying two encoders with different target sparsity further destroys the consistency of self-supervised learning, as shown in Figure 2(b). As a result, we have the following observation for Question 2:
Observation 2: Sparsifying the model with frequently changing architecture hinders the contrastive- ness and consistency of supervised learning and leads to degraded encoder performance.
As shown in Observation and Observation 2, high sparsity-induced asymmetry is not directly- applicable to sparse SSL, while the consistency requirements of SSL negates the plausibility of gradual sparsity increment. The dilemma between self-supervised learning and sparse training de- rives the following challenge:
How can we efficiently sparsify the model during self-supervised training while maximizing the benefits of the sparsity-idduced asymmetry?
5

---

Under review as a conference paper at ICLR 2023
4 METHOD
To address the above challenge, we propose Synchronized Contrastive Pruning (SyncCP), which successfully alleviates the contradiction between the needs of high sparsity and the requirements of consistency in self-supervised learning.
4.1 SYNCHRONIZED SPARSIFICATION (SYNCS)
The rationale behind the -enduty-idduuedd asymmetric SSL is that the perturbation generated by the pruned encoder elevates the difference between contrastive features. As indicated by Observation and Table I, the high sparsity- -inuucd asymmetry is not universally applicable, but the SSL can be rewarded from the asymmetry incurred by lower sparsity (e.g., 50%), where the SSL-trained sparse and dense encoders exhibit negligible accuracy degradation compared to the baseline. Motivated by this, we propose the Synchronized Sparsification (SyncS) technique to exploit sparsity in both contrastive encoders. Given the online and offline (momentum) encoder and with weights Wa and We. the in-training sparsification can be expressed as:
online output go(fo(X (Mg Wo)))
(8)
offline output 9 (fç('' (Mę W()))
(9)
Where Mg and Mc represent the online and offline (momentum) sparse masks with sparsity 59 and sg. The proposed SyncS scheme gradually exploits the sparsity in both encoders while maintaining
a consistent sparsity gap As between them during SSL training. At each pruning step t,, we have:
s/ = 5/ + (o6 *1)(1 t- to n
(10)
2/ o + (st o s5) nAt
t- to y²
(11)
s.t 1s8 - 2/y = D.. for t {to. to + At, + nāt]
(12)
The exponent controls the speed of sparsity increment, we adopt the sparsity schedule of Eq. 10-12 from (Liu et al., 2021) to minimize the impact of the parameter tuning. The synchronized sparsity increment with the constraints of A. prevents the exceeding asymmetry between contrastive en- coders while minimizing the distortion caused by the changing sparsity. In practice, A. is treated tunable parameter which impacts the final sparsity of both online and offline encoders. To guarantee the consistency of the contrastive sparsity, both Sp and s& are initialized by Erdos Renyi Kernel (ERK) (Evci et al., 2020), and with respect to A., we evaluate the impact of A, in Appendix.
4.2 CONTRASTIVE SPARSIFICATION INDICATOR (CSI)
Achieving high sparsity requires gentle sparsity increment, but as presented in Observation 2, the inconsistent architecture difference deteriorates the contrastiveness of SSL. On the other hand, the popular EMA-based update (He et al., 2020) allows the momentum encoder to generate consis- tent latent representation, but the lagged architecture makes the momentum encoder become an unqualified "competitor" to the online encoder, which violates the findings of (Jiang et al., 2021). To address such conflict, we propose the Contrastive Sparsification Indicator (CSI), a simple-yet- effective method that automatically selects the starting point of sparsity increment based on the learning progress of SSI.. During the SSI. training, CST first generates the pseudo pruning decisions of both encoders based on element-wise magnitude pruning with target sparsity sy and s. se.
M 1 [eo] E TopK( (wol: $6)3
(13)
M = 1 |w/| E (lpK(
(14)
Where 1 represents the indicator function, and the resultant pscudo masks of M, and M; will not be applied to the weights. Subsequently, CSI XORs the pseudo pruning masks to generate G (Eq. 15), and the percentage "1"s in G is equivalent to the architecture inconsistency I (Eq. 16), where |G] represents the total number of element in G. Instead of using cosine similarity, the bit-wise XOR can be easily implemented on hardware quantify the architecture difference during training.
G = M2 + M
(15)
I= Σ
(16)

---

Under review as a conference paper at ICLR 2023
80
Online Encoder
80
Online Encodar
70-
Momentum Encoder
CSI: Arch. Difference
0.6
CSt Arch Difference
70-
CBi:
Spars.
Synchronized Sparsification (Proposed)
00.6
Sparsity (%)
60-
0.6
60
0.5
50-
Sparaity
80-
Inconsistent asymmetry!
40-
40-
CSI Checkpoint (Proposed)
30-
CSI Checkpoint (Proposed)
CSR
20
0.3
20
3.
0.3
200
400
600
800
1000
0
200
400
600
800
1000
BYOL Training Epoch
BYOL Training Epoch
(8)
(b)
Figure 3: Sparse BYOL training process (a) without SyncS and (b) with SyneS.
Table 3: Performance comparison of BYOL CIFAR-10 dataset with/without SyncS.
Method
Online Encoder Spars.
Momentum Encoder Spars.
Online Linear Eval. Acc. (%)
CSI SyneS
50% 50%-80%
20%-+5%
92.24%
CSI Only
20%-20% (Fixed)
91. 91.64%
Given the sparsity gap A, defined by SyncS, CSI activates the sparsity increment when I equals to A., and this moment is defined as the CSI checkpoint. In other words, when the architecture difference between online and offline encoders is mainly caused by the sparsity difference, it is the optimal moment to start exploiting the in-training sparsity with the gradually-increased sparsity. With the ability automatically select the starting point of sparsity increment, the proposed CSI method automatically sparsifies the model with the full awareness of the SSL process. For the SSL framework with shared encoder (Zbontar al., 2021), the architecture inconsistency I is computed based on the sparse architecture of two consecutive epochs, and the sparsity increment is activated when I less than a pre-defined threshold (e.g.. = 0.1).
Figure shows the sparsification scheme with and without SyneS. As summarized in Table 3, hold- ing the sparse momentum architecture after the CSI checkpoint interrupts the consistency between the online and momentum encoders. Although the momentum encoder retains the low sparsity at 20%, the absence of consistent asymmetry from synchronized contrastive pruning (SCP) causes the degraded model performance.
On top the proposed SyneS and CSI schemes, we adopt the prune-and-regrow scheme (Liu et al., 2021) with modifications exploit sparsity during SSL training. To further alleviate the contrastive- ness oscillation caused by changing sparsity, we slowly average the gradient magnitude by exponen- tial moving average (EMA) with gentle momentum, instead of using the instant score. The detailed pseudo code of the proposed algorithm is summarized in Appendix C.
5 EXPERIMENTAL RESULTS
In this section, we validate the proposed sparse training scheme and compare it with the current SoTA sparse training schemes. Unlike the work by (Jiang et al., 2021), the proposed scheme ex- ploits in-training sparsity in both contrastive paths (encoders) and aims to achieve energy-efficient supf-super learning. The proposed method is applied multiple mainstream SSL frameworks, including EMA-based methods (Chen et al., 2020b; Grill et al., 2020) and SSL with shared en- coder (Zbontar al., 2021). The linear evaluation accuracy and training cost reduction are reported for multiple datasets, including CIFAR-10, CIFAR- 100, and ImageNet-2012. Furthermore, this work exploits in- training sparsity with various sparsity granularities, including element wise spar- sity, N:M sparsity (Zhou et al., 2020), and structural sparsity for custom hardware accelerator.
CIFAR-10 and CIFAR-100 Table 4 summarizes the linear evaluation accuracy of the proposed method CIFAR-10 10 and CIFAR- 100 datasets with element-wise sparsity. We use ResNet-18 (Ix) as the backbone and train the model from scratch by 1,000 epochs. Following the typical high spar- sity results reported with supervised learning, we report the model performance with 80% and 90% target sparsity. To sparsify both encoders during SSL training, we initialize the sparsity of online and offline (momentum) encoders 30% and 0%, where the A, is set to 30%. The initialized sparse
7

---

Under review as a conference paper at ICLR 2023
Table 4: Linear evaluation comparison on CIFAR-10/100 datasets with clement-wise sparsity.
Dataset
CIFAR-10 Ace (%)
CIFAR-100 Ace (%)
Encoder
ResNet-18 (Ix)
R8 (Ix)
wiem Sparsity
0%
80%
906
0%
80%
90%
This work
92.09
91.77
91.31 +0.04
67.72
67.56 0.04
66.78+00007
McCo-V2
GraNet MoCo (Liu et al., 202
92.09
90.66:0.07
90.05:10.08
67.72
67.17=0.05
64.92:0.06
(Chen al., 2020b)
SD-MoCa (Jiang 2021)
92.09
90,261:0005
87.68:10.06
67.72
65.04=0.04
61.33:0.005
This work
92.42
92.26-H.06
92.03-H0.05
68.80
68.69-0.06
67.73+-0.44
BYOL (Grill et al., 2020)
GraNet-BYe BYOL.
(Liu et 2021)
92.42
91,20:10.0
90.13:10.03
68.80
67.17=0.05
63.85:00.08
SD-BYOL (Jing et al., 2021)
92.42
90.33++0.07
87.38+0.04
68.80
66.13=0.08
62.20+0.10
This work
91.74
91.67:00099
90.84:10007
68.62
68.75=0.13
68.48:00112
Barlow Twins
BraaNe-BBrrlow
(Zbontar et al.,, 2021)
(Liu et al., 2021)
91.23:10.03
90.44.10.12
68.40-00.10
68.15100.4
SD-Barlow
UJang et al., 2021)
91.74
90.09:10.03
88.41=0.07
68.62
66.42=0.07
61.77:0004
Table 5: ImageNet-2012 accuracy and training cost comparison with SoTA works on ResNet-50
with BYOL (Grill al., 2020).
ImageNet
Top-1 Accuracy
FLOPS (Training)
FLOPS (Inference)
Top-1 Accuracy (%)
FLOPS (Training)
FLOPS (Inference)
Baseline
68.12
18
(1xx)09(11)
68.12
7.94 (1x)
8.
Element-wise Sparsity
80%
906
Model Sizee (MB)
20.5
10.2
This work
67,02
0.62x
0.33
65.67
0,56x
0.22x
(Gril et al., 2020)
GenNei-RYOL. (Liu al., 2121)
65.45
0.20x
SD-BYOL
(Jiag all,, 2021)
63.02
0.68
0.34 x
60.55
0.63x
0,21
The encoders trained from scratch the ImageNer dataset with 300 epochs. The FP32 dense RetNet-50 model requires 102MB storage.
encoders reduce the overall memory footprint throughout the entire training process. We rigor- ously transfer the SoTA GraNet Liu et al. (2021) to SSL based on its open-sourced implementation, the proposed method outperforms GraNet-SSL with 1.26% and 1.86% accuracy improvements on CIFAR-10 and CIFAR-100 datasets, respectively. In all experiments, we report the average accuracy with its variation in 3 runs.
In addition to element- wise sparsity, the recent Nvidia Ampere architecture is equipped with the Sparse Tensor Cores to accelerate the inference computation on GPU with N:M structured fine- grained sparsity (Zhou et al., 2020), where the N dense elements remain within M-sized group. Powered by the open-sourced Nvidia-ASP library, SyneCP sparsifies BYOL training (Grill et al., 2020) by targeting 100% N:M sparse groups in online encoders. Starting from scratch, the per- centage of the N:M sparse groups is initialized as 30% and 0% in online and momentum encoders with =,00.% After the CSI checkpoint, the percentage of N:M groups gradually increases. Ap- pendix A describes the detailed pruning algorithm of N:M sparsification. Table 6 summarizes linear evaluation accuracy and inference time reduction on the CIFAR-10 10 and CIFAR- 100 datasets. The resultant model achieves up to 2.08x inference acceleration with minimum accuracy degradation. The inference time measured on an Nvidia 3090 GPU with FP32 data precision.
ImageNet-2012 Since the BYOL (Grill et al., 2020) learning scheme achieves the best perfor- mance with CIFAR datasets, we further evaluate the proposed method with ResNet-50 on ImageNet based on the BYOL framework (Grill et al., 2020). Following the typical high sparsity results re- ported in Table 4, we report the model performance with 80% and 90% element-wise sparsity. The data augmentation setup is adopted from the open-sourced library (Costa et al., 2022). Starting from scratch, the model is trained by 300 epochs, where both online and momentum encoders are ini- tialized by ERK with A, 30%. While we believe more fine- -naiggd hyperparameter tuning and extended training efforts could lead to better accuracy, we choose the above scheme for simplicity and reproducibility. Table 5 shows the comparison of linear evaluation accuracy on ImageNet-2012
8

---

Under review as a conference paper at ICLR 2023
Table 6: Linear evaluation accuracy comparison on CIFAR-10/100 datasets with N:M structured fine-grained spansity.
Datasets
CIEAR-10 Ace (%)
CIFAR-I00 Ace (%)
Encoder
ResNet-18 (1x)
R8 (1x)
N:M Sparse Pattern
2:4
1:4
2:4
1:4
MoCo-V2
91.99:10077
91.53:+0.04
67.58+0.05
67.11+00005
[Chen al., 2020h)
BYOL (Grill et al,, 2020)
92.61++0.05
91.83++0.02
68.69:0.02
68.09+00.07
Barlow Twins
91.6810.04
90.97:0.03
68.26100077
68.19:0.06
(Zbontar ct al., 2021)
Inference time reduction (8)
1.40x
2.08x
1.40x
2.08x
FLOPs (Dense 5.56c+8)
0.50
0.25
0.50x
0.25
Weight Memory (MB)
22.4
11.2
22.4
11.2
The FP32 dense ResNet-18(1x) model requires 44.8MB storage and takes 1.27 seconds per
10K testset images during inference.
dataset. Compared to the -ala-dnn scheme (Jiang et al., 2021) and GraNet (Liu et al., 2021), the proposed algorithm achieves the same highly-sparse network with 4.72% and 1.21% Top-1 in- ference accuracy improvements, respectively. GraNet exploits in in-training sparsity throughout the entire training process, but the inconsistent contrastiveness hampers the model performance. On the other hand, the dense encoder limits the efficiency of the self-damaging scheme (Jiang et al., 2021) scheme, and the static high sparsity degrades the model performance. It has been shown that SSL- trained encoders strong visual learners (Grill al., 2020; Ericsson et al., 2021). Appendix A summarizes the performance of the proposed algorithm by fine-tuning the ImageNet-trained sparse encoders CIFAR-10 10 and CIFAR-100 1-00 datasets. With only 300 epochs of sparse SSL training, the resultant sparse encoder outperforms the SoTA supervised sparse training algorithms.
Table 7: Hardware training acceleration of the proposed structured SyncCP on CIFAR-10 datasaet.
Top-1 Accuracy (%)
Training Speed-up
Top-1 Accuracy (%)
Training Speed-up
BYOL.<ResNet-18
Dense Baseline
92.42
1x
92.42
1x
Target Structured Sparsity
80%
90%
BYOL
(Grill et al., 2020)
This work
92.16
1.74
91.77
1.91
Hardware-based Structured Pruning The hardware practicality of element-wise sparsification often limited by the irregularity of fine-grained sparsity and index requirement. To that end, we employ structured sparsity based on group- EMA scores towards achieving actual hardware training acceleration. The encoders are initialized by ERK with 30% and 0% sparse groups while keeping A, = 30%. The structured sparsity starts to gradually increase after the CSI checkpoint. We adopt the training accelerator specifications from (Venkataramanaiah et al., 2022) and choose K, (# of output channels) Ci (# of input channels) 8x8 as the sparse group size. Table 7 evalautes the training speedup of B SOL (Grill et al., 2020) aided by the structured sparse training. The proposed algorithm achieves up 1. 1.91 training acceleration with minimal accuracy degradation.
CONCLUSION
In this paper, we propose a novel sparse training algorithm designed for sulf-supervised learn- ing (SSL). As one of the first studies in this area, we first point out the imperfections of the sparsity- induced asymmetric self-supervised learning, as well as the incompatibility of the supervised sparse training algorithm in SSL. Based on the well-knit conclusions, we propose a contrastiveness-aww sparse training algorithm, consisting of synchronized contrastive pruning (SCP) and contrastive spar- sification indicator (CSI). The proposed method outperforms the SoTA sparse training algorithm on both CIFAR and 1mageNet-2012 datasets with various mainstream SSL frameworks. We also demonstrate the actual training and inference hardware acceleration with structured sparsity and N:M structured fine- ine-gnained pattern.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre- training in Computer Vision Models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16306-16316, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning (ICML), pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive learning. arXiv preprint w422:20033.4297, 2020b.
Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci, solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. Journal of Machine Learning Research, 23(56);1-6, 2022.
Tim Dettmers and Luke Zettlemoyer. Sparse Networks From Scratch: Faster Training without losing performance. arXiv preprint arXiv:1907.048400 2019.
Linus Ericsson. Henry Gouk, and Timothy M Hospedales. How well do self--upprvised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5414-5423, 2021.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICMI.), 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 33:21271 11221227-22284 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1735 1742, 2006.
Song Han, Huizi Mao, and William J Dally. Deep Compression: Compressing Deep Neural Net- works with Pruning. Trained Quantization and Huffman Coding. International Conference Learning Representations (ICLR), 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and 2020.
Pattern Recognition (CVPR), pp.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do Com- pressed Deep Neural Networks Forget? arXiv preprint 1rXiv:1911.05248 2019.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top- always sparse training. Advances Neural Information Processing Systems (NeurIPS), 33: 2020.
Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging Contrastive Learning. In International Conference Machine Learning (ICML), pp. 4927-4939 2021.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting Self-supervised Visual Represen- tation Learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019.
Simon Kornblith. Jonathon Shlens, and Quoe V Le. Do Better ImageNet Models Transfer Better? In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2661-2671, 2019.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. In International Conference on Learning Representations (ICLR), 2018.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse Training via Boosting Pruning Plasticity with Neuroregeneration. Advances Neural Information Processing Sys- tems (NeurIPS), 2021.
Jian Meng, Li Yang, Jinwoo Shin, Deliang Fan, and Jae-sun Seo. Contrastive Dual Gating: Learning Sparse Features With Contrastive Learning. In IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 12257-12265, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding. arXiv preprint arXiv:1807.03748, 2018.
Siyuan Pan, Yiming Qin, Tingyao Li, Xiaoshuang Li, and Liang Hou. Momentum Contrastive Pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 064472656,, 2022.
Shreyas K. Venkataramanaiah, Jian Meng, Han-Sok Suh, Injune Yeo, Jyotishman Saikia, Sai Kiran Cherupally, Yichi Zhang, Zhiru Zhang, and Jac sun Sco. A 28mm 8-bit Floating-Point Tensor Core based CNN Training Processor with Dynamic Activation Weirattion/Weight Sparsification. In IEEE European Solid-State Circuits Conference (ESSCIRC), 2022.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. In International Conference on Learning Representations (ICLR), 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv preprint arXiv: 011.77088.88888 2017.
Jure Zbontar, Li Jing Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-supervised Learning via Redundancy Reduction. In International Conference on Machine Learning (ICML), pp. 12310-12320, 2021.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. In International Conference Learning Representations (ICLR), 2020.

---

Under review as a conference paper at ICLR 2023
A DOWNSTREAM TASKS PERFORMANCE WITH THE PRE-TRAINED SPARSE ENCODER
The pre-trained sparse encoder can be used for downstream tasks. We verified the performance sparse ImageNet-trained model in Table 5 with the downstream tasks. The following table summa- rizes the transfer learning performance by fine-tuning the ImageNet-trained sparse BYOL encoder on CIFAR-10 and CIFAR-100 datasets.
Table 8: Transfer learning performance of the ImageNet-traing trainne B BOL encoder on CIFAR-10 and CIFAR- datasets.
Dataset
CIFAR-10 Ace (%)
CIFAR-100 Acc. (%
Encoder
ResNet-50 (1x)
ResNet-50 (1x)
Element-wise Sparsity
0%
90%
0%
90%
SNIP
94.75
92.65
78.23
73.14
(Lee et al., 2018)
GraSP (Wang al., 2019)
94.75
92.47
78.23
73.28
Supervised Training
RigL (Evci et al., 2020)
94.75
94.45
78.23
76.50
GraNet (Liu et al., 2021)
94.75
94.64
78.23
77.89
Transfer learning from BYOL-trained model
This work
96.09
95.72
80.13
79.48
The pre-trained sparse encoder are consistently preserved during the downstream fine-tuning. As shown in the table above, the strong visual representation learners are obtained by training the backbone model with self-supervised learning on ImageNet dataset. Sparsified by the proposed algorithm on ImageNet with 300 epochs during pre-traie pre-training, fine-tuning the SSL-trained backbone model leads to 1.08% and 1.59% accuracy improvements compared to the recent SoTA supervised pruning algorithms.
B THE IMPACT OF As
We evaluate the impact of A, with different sparsification schemes CIFAR-10 dataset with BYOL SSL framework. With the target sparsity = 90%, the following table summarizes the accuracy and training cost reduction with different A, values associated with different initial and final density.
Table 9: The linear evaluation accuracy and the training FLOPs reduction with sparsity gap A..
ELOPs (Training)
A,
Online Encoder Spars.
Offline Encoder Spars.
Online Linear Eval Aec.
50%
50% - 90%
0%
40%
91.68
0.60x
40%
50% > 90%
10% + 50%
91.41
0.58x
30%
50% + 90%
20% + 60%
91.27
0.53
40%
40%
> 90%
0%
50%
91.79
0.57
30%
30% + 90%
0% - 60%
92.02
0.56
Given the fixed target sparsity = 90%, is easy to tell that the higher A, leads to denser momentum encoder and less computation reduction. Meanwhile, sparsifies the momentum encoder at the be- ginning training oottootimal. Furthermore, less initial sparsity and smaller A, value (4th row) achieves the best tradeoff between computation reduction and model performance.
12

---

Under review as a conference paper at ICLR 2023
C
APPENDIX C
C.1 PSEUDO CODE OF SYNCCP WITH ELEMENT- WISE SPARSITY
Algorithm 1: Synchornized Contrastive Pruning (SyneCP)
Initialize Sparse online encoder fo. Sparse offline encoder fç. EMA updater, Momentum 7. SyneS density gap A,, CSI threshold 7 (Default=A.).
Initial sparsity sg. se, such that |s
sg A.
Target sparsity sg. &g. such that 1 5c A Initial mask My. M
Pruner udpate frequency + while < Total Iterations do
Draw augmented data (X,');
Forward pass: online encoding = fo(Ms 0, X);
Forward pass: offline encoding = fe fM &, X'):
Update Exponential Moving Average (EMA) gradient score based on Eq. 17;
End Epoch then
Get pseudo masks Mğ and M based on magnitude pruning; Compute layer-wise G and I based on Eq. 15 and Eq. 16; if A, then Prune=True
end
end
if % = then
if Prune=True then
Update sparsity s/., 2 based on Eq. 10 and Eq. 11;
Maintain the SyneS constraint Ag;
Inside fo and fe. prune sg. and s elements with least magnitude score;
Prune extra r1 elements of the unpruned elements, then regrow ry elements with hights E.A-gradient score;
Update My. M based on Eq. and Eq. 4;
else end
end
C.2 EMA- EMA-BASED BASED PRUNE AND REGROW
As aforementioned, the findings of Observation 2 implies the incompatibility of the instant gradient and magnitude score. Together with the proposed SyneS and CSI methods, weight importance is measured by the magniude score, while the sensitivity the model quantified by the gently averaged gradient magnitude with EMA:
lgl
(17)
Table 10 summarizes the linear evaluation accuracy of ResNet-18 trained by BYOL (Grill et al., 2020). We initialize og and of 40% and 10%, where the A, is set to 30%, the EMA momentum is set to 0.1 for gentle gradient score averaging.
Metric
Prune
Regrow
EMA /
Online Linear Eval. Acc. (%)
This work
91.88
frune-and-regrow
X
91.52
Magnitude Pruning
x
X
90.99
Table 10: Peformance comparison between different sparsification metrics.
13

---

Under review as a conference paper at ICLR 2023
C.3 PSEUDO CODE SYNCCP WITH N:M SPARSITY
Algorithm 2: Synchornized Contrastive Pruning (SyncCP) with N:M Sparsity
Initialize Sparse online encoder Ja. Sparse offline encoder Je. EMA updater, Momentum SyneS density gap As. CSI threshold (Default=A.). Group size M, Number of dense element per group N. Initial percentage pg of N:M groups in fe. Initial percentage p of N:M groups in fei Such that |/ P = As;
Target percentage p = 100%, P = Pb As:
Initial mask M My.
Pruner udpate frequency 4
while Total Iterations do
Draw augmented data (X, X'):
Forward pass: online encoding = [M, 0 X):
Forward pass: offline encoding fe [M, E.X")
Update Exponential Moving Average (EMA) weight gradient score based on Eq. 17; if End Epoch then
Get pseudo masks Mž and M2 based on magnitude pruning:
Compute layer-wise G and I based Eq. 15 and Eq. 16;
if A, then Prune=True
end end
if % + = 0 then
if Prune=True then
Update sparsity ph. p based on Eq. 10 and Eq. 11;
Maintain the SyneS constraint ph. p ph A.;
Inside So and fg, pick ph. and pt M-sized groups with least sum magnitude score;
Inside each group, prune the N-M elements with smallest magnitude score;
Update Mt, M based on Figure 4;
else
end
end
Input:
Ingort:
on p group-arioe weight
Thashhod value gthre calculated bead on p and all the group- vide EMA veight grodicnt soures.
weight magiltode
Output: praning
Syane weggl makk with
agroup numol)/M
the
Outpot: Binnry weight
weight vaighs abs parmute 2,311) restaps (agroup,
tocol surber of groups agzoup gota. nias10/M
zont
asgnéficoss elesents
group
reshape the ess gradiest score of curresi isger
torch arguort (wuight, diari)it, (ix106-3)
gema gosa 2eemute(0,2,3,1 roslapeiagroopp 80
initialsae maak
w.b turch, mma abape)
anly regros the vedght groups shat sireedy operce asue: torch, cupfo, dia 1) eidx note egtself (b.flooxt)
fose-grssee garstfccation for all groupe
w.b v.D. ocotser, (diart, index, valle-0)
# pick the least ficust groupa baxed ox grin wnight. mun (stne1)
Jwn the gradiest of the elesenso incide the aparoified groups groigrp groigrp-e:dx(: Mone] 8:08 torct, sunggradgrp, diar1)
glndex gets gettare)
recover the aigns/Scant
sort the responoe
[tndee, :]
y. idx torch sorsigron flastent). descending-True)
regros the group
cush back reshape Eveight shape)
weight
tidxxitotaal_rggee (oooott_orgoott)) 1.0
reshape the bisery each back (permstation is reguired ie pesetsce) reabapeeeeiight.... shaps)
(b)
Figure 4: Group-wise (a) prune and (b) regrow algorithm based EMA gradient score. SyneCP sparsifies M - N elements inside each group, while keep p' N:M groups inside f.
14

---

end Under review as a conference paper at ICLR 2023
C.4 PSEUDO CODE OF SYNCCP WITH STRUCTURED SPARSITY
Algorithm 3: Synchornized Contrastive Pruning (SyncCP) with Structured Sparsity
Initialize Sparse online encoder So. Sparse offline encoder Je. EMA updater, Momentum SyneS density gap A., CSI threshold 7 (Default=A.).
Group size K; (# of output channels) x C₁ (# of input channels) g x 9.
Initial percentage p of sparse groups in fe, Initial percentage pe of sparse groups in fe:
Such that pp P = م
Initial structured sparsity sg. s such that 1s sel A
Target structured sparsity sn. s such that 16 o6 A Initial mask MI. M
Pruner udpate frequency + while / < Total Iterations do
Draw augmented data X'):
Forward pass: online encoding = So,M, 0, X);
Forward pass: offline encoding [Mg E, X'):
Update Exponential Moving Average (EMA) gradient score based on Eq. 17:
End Epech then
Get pseudo masks Mž and M based on magnitude pruning:
Compute layer-wise and I based on Eq. and Eq. 16;
if A, then
Prune=True
end end
if % + = 0 then
if Prune=True then
Update structured sparsity s/j, st based on Eq. 10 and Eq. 11;
Maintain the SyneS constraint s/. st = s/ - A.:
Inside fo and fe. pick sy and st groups with least sum of magnitude score; Outside the sparsified groups of fa. prune ró more groups with least sum of magnitude score;
Among the sparsified groups fo. regrow the rá groups back with highest sum of EMA gradient score;
Update Mý. M;:
end end
D SPARSIFYING ONLINE OR OFFLINE ENCODERS
We mirror the experiment of Table 1 by exploiting high sparsity in the momentum encoder while keeping the online encoder dense:
Table 11: Largely degraded performance of MoCo-V2 with self-damaging SSL on CIFAR-10 dataset.
ResNet-18
Dense Model Acc. %
Encoder
Online
Momentum
Online
Momentum
Fixed Sparsity
90%
0%
50%
0%
Linear Eval. Ace (%)
88.72
(-3.41%%
87.68 (-4.31%)
92.10
(+0.01%)
92.07
(-0.02%)
Fixed Sparsity
0%
90%
0%
50%
Linear Eval. Acc (%)
88.44 (-3.65%)
87.52 (-4.57%)
15

---

Under review as a conference paper at ICLR 2023
With the proposed CSI + SyncS sparsification method, we empirically observe that exploiting high sparsity in online model leads to better performance with gradient-bar based model update. Table 12 summarizes the comparison results of BYOL on CIFAR- 10 dataset with the proposed algorithm.
Table 12: Performance comparison of exploiting higher sparsity in online momentum encoders.
Method
Online Encoder Spars.
Momentum Encoder Spars.
Online Linear Eval. Acc. (%)
CSI SyneS
0% +50%
+80%
91.88%
CST SyneS
30%-+800 -00%
-%0+
92.24%
Overall, sparsifying the online encoder leads to the optimal performance.
E DETAILED EXPERIMENTAL SETUP OF SYNCCP
E.1 LINEAR EVALUATION PROTOCOL
As in (Kolesnikov al., 2019; Kornblith al., 2019; Chen et al., 2020a), we use the standard linear evaluation protocol on CIFAR-101 10000-000100 and ImageNet-2012 datasets, which training linear classifier on top of the frozen SSL-trained encoder. During linear evaluation, we apply spatial augmentation and random flips. The linear classifier is optimized by SGD with cross-entropy loss.
E.2 CIFAR-10/100 EXPERIMENTS
The training hyper- parer-parameters of the compared individual sparse training works are same for CIFAR- 10 and CIFAR-100. We provide the detailed training setup of different self-supervised learn- ing frameworks as follow:
MoCo-V2 The ResNet-18 (x) encoder trained by MoCo-V2 (Chen al., 2020b) from scratch by 1,000 epochs with SGD optimizer and 256 batch size. The learning rate is set to 0.3 with Co- sine learning rate decay and 10 epochs warmup. The detailed data augmentation is summarized in Table 13.
Parameter
X
X
Random crop size
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment probability
0.4
0.4
Contrast adjustment probability
0.4
0.4
Saturation adjustment probability
0.2
0.2
Hue adjustment prohability
0.1
0.1
Gaussian blurring probability
0.0
0.0
Solarization probability
0.0
0.0
Table 13: Detailed image augmentation settings for MoCo-V2 (Chen et al., 2020b) on CIFAR- 10/100.
The ResNet-18 (x) encoder is trained by BYOL (Grill et al., 2020) from scratch by 1,000 epochs with LARS-SGD optimizer (You et al., 2017). The predictor is constructed with 4096 hidden features and 256 output dimension. We use 256 batch size along with 1.0 learning rate. The Cosine learning rate scheduler is used with 10 epochs warmup training. The detailed data augmentation is summarized in Table 14.
BYOL
16

---

Under review as a conference paper at ICLR 2023
Parameter
X
X²
Random crop size
32
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment prohability
0.4
0.4
Contrast adjustment probability
0.4
0,4
Saturation adjustment probability
0.2
0.2
Hue adjustment probability
0.1
0.1
Gaussian blurring probability
0.0
0.0
Solarization probability
0.0
0.2
Table 14: Detailed image augmentation settings for BYOL (Chen et al., 2020b) CIFAR-10/100.
Barlow Twins The ResNet-18 18 (x) encoder is trained by Barlow Twins (Zbontar et al., 2021) from scratch by 1,000 epochs with LARS- SGD optimizer (You et al., 2017). We use 256 batch size along with 0.3 learning rate and le 4 weight decay. The Cosine learning rate scheduler is used with 10 epochs warmup training. The detailed data augmentation summarized in Table 15.
Parameter
X
X'
Random crop size
32
32
32
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment probability
Contrast adjustment probability
0.4
0.4
Satuation adjustment probability
0.2
0.2
Hue udjustment probability
0.1
Gaussian blurring probability
0.0
0.0
Solarization probability
0.0
0.2
Table 15: Detailed image augmentation settings for Barlow Twins (Zhontar et al., 2021) CIFAR- 10/100.
E.3 IMAGENET EXPERIMENTS
Starting from scratch, the proposed SyneCP algorithm exploits in-training sparsity with the BYOL framework (Grill et al., 2020) on ImageNet-20122 dataset. The ResNet-50 encoder is trained by LARS-SGD (You et al., 2017) with 0.45 learning rate and a momentum of 0.9. We uses 0.1 for the for EMA-averaged gradient score. We 128 batch size along with 1e-6 weight decay. The detailed image augmentations are summarized in Table 16.
Parameter
X
X'
Random crop size
224
224
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
Brightness adjustment probability
0.4
Contrast adjustment probubility
0.4
0.4
Saturation adjustment probability
0.2
Hue adjustment probability
0.1
Gaussiun blurring probubility
1.0
0.1
Solarization probability
0.0
0.2
Table 16: Detailed image augmentation settings for BYOL (Grill et al., 2020) ImageNet.
17

---

Under review as a conference paper at ICLR 2023
F COMPUTATION REDUCTION WITH DIFFERENT SPARSITY VALUES
Table 17: ImageNet-2012 linear evaluation accuracy and training cost comparison with different sparsity on ResNet-50 with BYOL (Grill et al., 2020).
Dataset
ImageNet-2012
Top-1 Accuracy (%)
FLOPs
FLOPs
Sparsity
(Training)
(Inference)
68.12
7.94++18 (Ix)
8.18e+09 (1x)
50%
68.31
0.78
0.68
80%
67.02
0.62
0.33
90%
65.67
0.56x
0.22
18


Reviews:
[{"review_id": "vDSZOEycQyx", "reviewer": "Reviewer_5j3N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper enables model pruning to accelearate contrastive self-supervised training. Compared to the exsited method, this paper tries to sparsify online decoder and offline decoder simultaneously. To stabilize the sparse training. The authors propose Contrastive Sparsification Indicator (CSI) to guide the model pruning.\n\nTo evaluate the effectiveness of SyncCP, the authors conduct experiments on various classification datasets with unstructured sparsity, N:M sparsity, and structured pruning.", "summary_of_the_review": "Na", "strength_and_weaknesses": "Strength:\n\nThe paper is well written.\n\nThe experiments including various different sparse types (unstructured, N:M, structured) are persuasive.\n\nThe performance (accuracy and efficiency) improvement is significant compared to the existing method [1].\n\n\n\n\nWeaknesses:\n\nThe self-supervised learning method usually trains on large-scale datasets, but this paper doesn't show any training GPU hours improvement on large-scale datasets. I feel confused about the motivation.\n\nThe NVidia GPUs only support 2:4 sparsity, can you explain the detail implementation of inference gain about 1:4 sparsity in Table5.  \n\n\nThe BYOL and MOCO-v2 are not SOTA SSL methods, the recent SOTA methods are more convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jHkPFIGkGYP", "reviewer": "Reviewer_inij", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new self-supervised learning framework (SSL), which aims to transfer the supervised sparse learning to SSL and reduce the required computational resources during the pretraining stage. Specifically, this paper investigates the correlation between training sparsity and SSL, which embraces the benefits of contrastiveness while exploiting high sparsity during SSL training. Experiments results show that this work surpasses the prior work on multiple datasets.", "summary_of_the_review": "See Strength and Weaknesses above", "strength_and_weaknesses": "**Strength** \n1. The overall idea is novel and interesting.\n2. The theoretical analysis of this work is promising.\n\n**Weaknesses** \n\nI am worried about the experimental results of this paper., although the performance of this work surpasses the prior work.\n\n1. Obviously, in table 6, the linear probing performance is significantly lower when the training flops are reduced to 0.64× or 0.58× (64.89 vs. 66,16 and 63.76 vs. 66.16). I am wondering, in the case of reducing how many flops, this work can get the on-pair performance with the default setting,\n\n2. Self-supervised learning aims to transfer the learned representations or whole network parameters into various downstream tasks. However, I do not see any transfer learning experiments in this paper. Could you provide more transfer learning experiments, for example, linear evaluation and fine-tuning in fine-grained classification tasks, semi-supervised learning, and object detection/segmentation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aSdrdlxyHK", "reviewer": "Reviewer_NxVm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper explores the problem of inducing sparsity during training for self-supervised learning.\n- The authors discuss challenges with applying existing sparse training techniques to self-supervised learning: sparsity-induced asymmetric non-identical encoders and sparsifying the model with frequent architectural changes both lead to degraded performance. \n- The authors then present SyncCP, which sparsifies both encoders in an SSL setup while ensuring that the two encoders maintain a consistent gap between their sparsity levels and that their architectural difference is mainly due to this difference in sparsity levels rather than the disjointness of which weights to prune. The sparsity levels start gradually increasing once the architectural difference criterion is met.\n- Experiments demonstrate improvements over other sparse training techniques and training speedups.", "summary_of_the_review": "In general, I like this paper. It describes a problem worth solving and presents a method backed by a discussion of the challenges it aims to fix. However, I have some concerns with the current state of the paper (see Weaknesses and Clarity), which I believe need to be addressed before it can be ready for publication.", "strength_and_weaknesses": "Strengths:\n- The paper walks you through the challenges of inducing sparsity during SSL training and then presents a method that addresses those challenges. I quite like the overall presentation.\n- The proposed method is novel and well-motivated.\n\nWeaknesses:\n- The results for baseline BYOL on ImageNet with ResNet-50 (66.16) are very low compared to BYOL's reported numbers (74.3). Why?\n- The presented method is only for momentum encoder-based SSL, which is not clear upfront. For instance, there is an introductory discussion about SimCLR and SDCLR, but the method is not actually applicable to SimCLR. Although the authors talk about generality to \"other SSL methods,\" it is not clear which classes of SSL methods are included or excluded.\n- Section 3 provides little context on the modeling decisions being made, such as why the online network is the one to be sparsified or why the online and offline encoders are pruned to different sparsity levels.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Agc4e11Fi_", "reviewer": "Reviewer_EBsm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For self-supervised learning with contrastive loss, they proposed Synchronized Contrastive Pruning (SyncCP) that have two encoders (online and offline/momentum) with different sparsity, but the difference in sparsity is maintained at $\\Delta_s$.  They also propose Contrastive Sparsification Index (CSI) that determines when to start sparsification.  CSI calculates the difference in the pruning masks (M) as inconsistency (I).  When inconsistency reaches the sparsity gap $\\Delta_s$, CSI activates the sparsity increment. \n\nThey compare their proposed approach with two existing sparsification approaches using 3 SSL approaches (MoCo, BYOL, Barlow Twins) and 2 datasets (CIFAR 10 and 100).   With Imagenet, they only use BYOL in SSL.   Generally, the results indicate that the proposed approach compares favorably.\n\n", "summary_of_the_review": "The authors propose SyncCP to synchronize sparsity in asymmetric encoders in SSL.   SnycCP constraint the difference in sparsity to $\\Delta_s$.   Empirical results indicate SyncCP compares favorably with 2 other sparsification approaches.  Additional analysis on $\\Delta_s$ and downstream tasks can benefit the paper.", "strength_and_weaknesses": "Strengths:\n\n1.  They compared SyncCP with two existing sparsification approaches on 3 datasets.  Generally, the results indicate that SyncCP compares favorably.\n\n2.  Motivation on the disadvantage of sparsification inconsistency in asymmetric encoders in SSL.\n\nWeaknesses:\n\n1.  Some parts of the approach could be explained further (see next section)\n\n2.  An analysis of how $\\Delta_s$ affect performance could be studied.\n\n3.  Additional downstream tasks would help illustrate the quality of learned representations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "61svxBvi92P", "reviewer": "Reviewer_19nX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new sparse SSL approach by exploring the correlation between in-training sparsity and SSL. This paper investigates the challenges of the sparsity-induced asymmetric SSL (a.k.a prune-and-regrow) and proposes synchronized contrastive pruning approach. The proposed approach is adaptive to various granularities of sparsity.\n", "summary_of_the_review": "The paper presented a novel synchronized contrastive pruning approach in sparse SSL. As its main contribution, the work did not evaluate the inference efficiency which is hard to justify the effectiveness of its sparsity. The paper can be further improved by introducing the core idea of its baseline to make the paper easy to follow.\n", "strength_and_weaknesses": "Strength\n1. The paper presented a detailed analysis of the existing sparse self-supervised learning approaches.\n2. The proposed approach is novel in solving the sparsity in SSL and also improve the model performance.\n\nWeakness\n1.The paper did not give a sufficient introduction of its baseline, SDCLR. It is hard to follow the ideas without that background.\n2. The paper did not evaluate the inference complexity, e.g. inference FLOPs, latency, memory cost, etc. Without that, it is hard to justify the effectiveness of its sparsity.\n3. The proposed approach is contrastive learning focused. If the paper is scoped to the contrastive SSL, the title needs to be confined. Otherwise, the paper needs to further evaluate its performance on MIM SSL frameworks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject