Blockwise self-supervised learning with Barlow Twins
Authors: Shoaib Ahmed Siddiqui
David Krueger
Yann LeCun
Stephane Deny
Venue: Submitted to ICLR 2023 (2023)
Keywords: Blockwise training
self-supervised learning
local learning

Abstract: Current state-of-the-art deep networks are all powered by backpropagation. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. Notably, we show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48\%, only 1.1\% below the accuracy of an end-to-end pretrained network (71.57\% accuracy). We perform extensive experiments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience.

Under review as a conference paper at ICLR 2023
BLOCKWISE SELF-SUPERVISER LEARNING WITH BARLOW TWINS
Anonymous authors Paper under double-blind review
ABSTRACT
Current wate-of-the--ee art deep networks are all powered by backpropagation. In this paper, we explore alternatives to full backpropagation in the form of block- wise learning rules, leveraging the latest developments in self-superviser learning. Notably, we show that a blockwise pretraining procedure consisting of training in- dependently the 4 main blocks of layers of ResNet-50 with Barlow Twins loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: linear probe trained on top of our blockwise pretrained model ob- tains a top-1 classification accuracy of 70.48%. %, only 1.1% below the accuracy end-to-end pretrained network (71.57% accuracy). We perform extensive periments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules large networks, with implications ranging from hardware design to neuroscience.
INTRODUCTION
One of the primary ingredients behind the success of deep learning is backpropagation. It remains an open question whether comparable recognition performance can be achieved with local learning rules. Previous attempts in the context of supervised learning and unsupervised learning have only been successful on small datasets like MNIST (Salakhutdinov and Hinton, 2009; Löwe al., 2019); Ahmad et al. 2020; Ernoult et al., 2022 Lee et al., 2015) or large datasets but small networks like VGG-I1 (Belilovsky al. 2019) (67.6% top-1 accuracy on ImageNet).
Being able to train models with local learning rules at scale is useful for a multitude of reasons. By locally training different parts of the network, one could optimize learning of very large networks while limiting the memory footprint during training. This approach was recently illustrated at scale in the domain of video prediction, using a stack of VAEs trained sequentially in greedy fashion (Wu et al. 2021). From neuroscientific standpoint, it is interesting to explore the viability of alternative learning rules to backpropagation, as it is debated whether the brain performs backprop- agation (mostly considered implausible) (Lillicrap et al., 2020), approximations of hackpropagation (Lilficrap et al., 2016), or relies instead on local learning rules (Halvagal and Zenke, [0222; Clark al., 2021; Mlling et al. 2021). Finally, local learning rules could unlock the possibility for adaptive computations as each part of the network is trained to solve subtask in isolation, naturally tuning different parts solve different tasks (Yin al. 2022, Baldock et al. (02I) offering interesting en- ergy and speed trade-offs depending on the complexity of the input. There has been some evidence that suggests that the brain also uses computational paths that depends on the complexity of the task (Sepaar and Metzler ( (1971)).
Self-supervised learning has proven very successful as an approach to pretrain deep networks. A simple linear classifier trained on top of the last layer can yield an accuracy close to the best su- pervised learning methods (Chen et al.) 2020a b, He et al. 2020, Chen et al. 2220cc; Caron et al., 2021a b; Zbontar et al. 2021; Bardes et al. 2022; Dwibedi et al., 2021), and even exceeds perfor- mance of supervised learning when finetuned (He et al. 2016). Self-supervised learning objectives typically have two terms that reflect the desired properties of the learned representations: the first term ensures invariance to distortions that do not affect the label of an image, and the second term ensures that the representation is informative about its input (Zbontar et al. 2021). Such loss func-

---

Under review as a conference paper at ICLR 2023
Forward propagation path
Backpropaçation patn
Dependence within layer
No dependence within layer
Biological Plausibiltt
Large-Scale Datacet Performence
Demonstraled
Impul
Label
W4
Bg
Wg
Wg
Wg
End-lo-end backpropagation [Ha al. 2016)
Foodback alignment [Lilicrap et al. 2016)
[Nokland 2016]
2021]
LoCe
[Halvagal & Zenke 2022]
[iling et al. 2021)
[Belkovsky et 2019]
Doang
[Lowe 2019]
Figure 1: We rank blockwise/local learning methods of the literature according their biological plausibility, and indicate their best demonstrated performance on large-scale datasets i.e. ImageNet (intensity of the blue rectangle below each model family). Our method is situated at a unique trade- off between biological plausibility and performance on large-scale datasets, by being (1) par performance with intertwined blockwise training (Xiong et al., 2020) and supervised broad- casted learning (Nøkland, 2016; Clark et al., 2021; |Belilossky et al., 2019) while being (2) more biologically plausible than these alternatives. Methods represented with magenta color used sim- (Halvagal and Zenke, 2022
ilar methodology performance on small datasets.
Löwe et al. 2019), but only demonstrated
tions may better reflect the goal of intermediate layers compared to supervised learning rules which only try to extract labels, without any consideration of preserving input information for the next layers.
In this paper, we revisit the possibility of using local learning rules replacement for backpropa- gation using recent self-supervised learning method, Barlow Twins. As natural transition towards purely local learning rules, we resort to blockwise training where we limit the length of the back- propagation path to single blocks of the network"" We applied a self-supervised learning loss locally different blocks of a ResNet-50 trained on ImageNet and made the following contributions:
We show that network trained with a blockwise self-supervised learning rule can reach performance on with the same network trained end-to-end, as long as the network is not broken into too many blocks.
We find that training blocks simultaneously (as opposed sequentially) is essential to achieving this level of performance, suggesting that important learning interactions take place between blocks through the feedforward path.
We find that methods which increase downstream performance of lower blocks on the clas- sification task, such as supervised blockwise training, tend to decrease performance of the overall network, suggesting that building invariant representations prematurely in early blocks may be harmful for learning higher-level features latter blocks.
We compare different spatial and feature pooling strategies for the intermediate block out- puts (which are fed to the intermediate loss functions via projector networks), and find that expanding the feature- dimerriioiimiiiiittty of the block's output is key the successful training of the network.
We evaluated a variety of strategies to customize the training procedure at the different blocks (e.g., tuning the trade-off parameter of the objective function as function of the block, feeding different image distortions to different blocks, routing samples to different blocks depending on their difficulty) and found that none of these approaches added substantial gain to performance. However, despite negative initial results, we consider these attempts to be promising avenues to be explored further future work. We exhaustively describe these negative results in Appendix A
We consider all layers with the spatial resolution belong the block.
2

---

Under review as a conference paper at ICLR 2023
Sequential Blockwise
Trained Frozen
Simuitaneuus Blackwise Training
Training
Backpropagation path
Pro
Block
Block
Block
Block 1
Block
Block3
Figure 2: Overview of our blockwise local learning approach. Each of blocks of layers of ResNet-50 is trained independently using self-supervised learning rule and a local backpropa- gation path. We apply this procedure in two main settings: (left) sequential training, where each block, starting from the first block, is independently trained and frozen, before the next block is trained; (right) simultaneous blockwise training, where all the blocks are trained simultaneously us- ing 'stop-grad' operation to limit the backpropagation paths to a single block. The yellow gear symbol refers to the combination of the pooling layer, projection head and loss function used in self-supervised learning.
RELATED WORK
This section provides a comparison between existing local learning paradigms in the literature and our method (Fig. We compare these methods from two perspectives: biological plausibility and performance on large-scale datasets, such as ImageNet (Deng al. 2009).
End-to-end backpropagation (He et al., 2016) is the best performing paradigm on large-scale datasets, but it is also considered the least biologically plausible (although not impossible, see (Lil- licrap et al. 2020)), because it requires long-range backpropagation path with symmetric weights to the feed-forward path.
Feedback alignment (Lillicrap et al. 2016) alleviates the symmetry constraint by making the back- propagation path to be random and independent from the feed-forward path. However, this method is not explicitly focused on reducing the length of the backpropagation paths.
Nøkland (2016); Clark et al. (202T); Belilovsky et al. (2019) propose three different methods to avoid long-range backpropagation paths by directly broadcasting the labels to all layers. Therefore, these methods are more biologically plausible than the previously described methods. However all these methods are supervised and thus rely large number of labeled inputs, which itself is not biologically plausible.
LoCo (Xiong et al., 2020) applies backpropagation separately to different blocks of a ResNet-50. This method does not rely on an abundance of labels as it uses a self-supervised objective, and performs well large-scale datasets (69.5% top-1 accuracy on ImageNet with linear probe). However, LoCo introduces a coupling between subsequent blocks by applying backpropagation to intertwined pairs of successive blocks. Therefore, this method still conceptually relies on a full backpropagation path.
Our method and the closely related works (Halvagal and Zenke, 2022; Löwe al. 2019) remove this coupling between subsequent blocks by training each block independently using some variant of recently proposed self supervised learning rules, namely SimCLR (Chen et al. 2020a), Barlow Twins (Zbontar et al. 2021 ) and VicReg (Bardes et al. (022). The main difference between our method and (Halvagal and Zenke, 2022; Löwe et al. 2019) that we demonstrate the effectiveness of our method on a large-scale dataset i.c. ImageNet. In terms of biological plausibility, our method still relies on backpropagation within blocks, and also relies on interactions between neurons within layers due to the exact form of the loss function. These remaining dependencies make method only partially local, and thus not fully biologically plausible.
Finally, Illing et al. (2021) removes all learning dependencies within and across layers and proposes a truly local learning rule, but this method has not been demonstrated to perform well on large-scale datasets date.
3

---

Under review as a conference paper at ICLR 2023
(a)
(c)
256
W
Conv-based
1x1 čone (2048 chanpils)
Expansion (CbE) GSP
Global Spatial Pooling (GSP)
(b)
Average Pooling
2048
2048
Local Spatial Pooling (LSP)
Figure 3: Overview of the pooling strategies employed in this work. The first one (a) the simple Global Spatial Pooling (GSP) which just computes global average activation over the entire feature volume. The second one (b) the Local Spatial Pooling (LSP) where we divide the feature volume into small spatial bins, compute the averages in these small bins, and concatenate these outputs to compute the final feature vector. The third one (c) a (Ixl) Conv-based Expansion (ChE), followed by global spatial pooling. This last pooling strategy provides the best performance in our case.
METHODS
We consider a ResNet-50 backbone and mostly adapt the Barlow Twins (Zbontar et al. 2021) code- base for our experiments. We implemented SimCLR loss function (Chen al.) 2020aa) within the Barlow Twins codebase, but tweaked the official VicReg implementatior for VicReg experi- ments (Bardes et al. 2022).
3.1 BLOCKWISE TRAINING
We consider a logical separation of the network into 4 blocks where we limit the length of the backpropagation path each block. The ResNet-50 architecture comprised of 5 different feature spatial resolutions, followed by global average pooling and a linear layer for final classification. The first block of spatial resolution only comprised this layer within the first block. We then train each of these 4 blocks separately with self-supervised learning objective, using stop- doopggradient to ensure that the loss function from one block does not affect the learned parameters of other blocks (see Fig. 2 We consider two different settings: (i) sequential blockwise training where we train the blocks in a purely sequential way, i.e. one after the other, and (ii) simultaneous blockwise training where train all the blocks simultaneously. We always report
single stride-2 layer. Therefore, we integrated
top-1 1 accuracy on ImageNet by training a linear classifier on top of the frozen representations the output of the different blocks.
3.2 BARLOW TWINS LOSS
Most the experiments in the paper rely on the Barlow Twins objective function (Zbontar et al., 2021), which we briefly describe here. The input image is first passed through the deep network, and then passed to a two-layer projector network in order to obtain the representations to be fed the self-supervised learning I We compute these representations for two different image views (each image being transformed by distinct set of augmentations) بĄ &b ((V(x;)) where
https://git tiu.....ubb... http:////ececccooeeeeectoooooooe
Using an additional projection head instead of learning directly on the backbone features has been found be more effective and useful (Chen al., 2020a)
4

---

Under review as a conference paper at ICLR 2023
[8] Simutaneeus Blockwise Training tw/ 1x1 CbE GSP Noisej
Simultaneuus Blockwise Training Lw/)
70
[D] Sequential Blackwise Training (W LSP)
Sequential Blockwise Training (Supervised LSP)
Sequential Blockwise Training w/ Untrained Previous Blocks (wl LSP)
60
Accuracy
40-
71.57%
70 4..48%%%
68. 38.84%
67.12%
60
65 6..66..
Accura racy (%)
51, 79%
30
10.
[A]
[6] [C]
[D]
[E]
[F]
Model Prefix (see legend)
Number of blocks
Figure 4: Overview our key results. (left) Top-1 accuracy of a linear probe trained on top of our pretrained network on ImageNet for different pretraining procedures. Our best blockwise-trained model [B] is almost on par with full backpropagation [A] (only 1.1% performance gap). (right) Accuracy ImageNet
function of the depth of the network. Since our blockwise models trained for each block in isolation, we can include different number blocks when computing the linear probe accuracy. We visualize the accuracy
include more blocks into the network.
V represents the ResNet-50 encoder, and represents the projection head. We then define cross- correlation matrix over these embeddings:
A B En
Cjj A
(1)
E.
26.1
V E
2b.
where indexes batch samples and i.ý index the vector dimension of the networks' outputs.
The Barlow Twins objective function attempts to maximize the on-diagonal terms and minimize the off-diagonal terms using the following loss function:
LBT Σ C1.
ΣΣ
JF
invariance
redundancy redluctio term
where the invariance term (on-diagonal elements should be close to ensures that the extracted fea- tures are invariant to the applied augmentations, while the redundancy reduction term (off- loff-diagonal elements should close 0) ensures that the different units (i.e., neurons) forming the final image representation capture distinct attributes of the input image. The redundancy reduction term thus encourages the representation to richly informative about the input.
Despite most of our investigation being restricted to Barlow Twins, our findings generalize to other self-supervised learning (SSL) losses such as VicReg (Bardes et al. 2022) or SimCLR (Chen et al. 2020a) shown in the result section.
3.3 POOLING SCHEMES
The original Barlow Twins projector receives an input representation dimension 2048. However, initial blocks of ResNet-50 have smaller number of channels. For example, the first block

---

Under review as a conference paper at ICLR 2023
contains only 256 channels instead of 2048. Therefore, explored different possible ways of pooling these features spatially in order to construct the feature vector which is fed to the projector.
Fig. illustrates the different pooling strategies evaluated in this study, including (1) the trivial Global Pooling (GSP) strategy which requires changing the size of the projector's input layer, (2) Local Spatial Pooling (LSP) strategy where we divide the grid into spatial bins, average pool within those bins locally, and concatenate the outputs to ensure that the dimensionality of projector is 2048, (3) Conv-based Expansion (CbE) pooling strategy, which first expands the number of features to 2048 using the defined convolutional filter size (defaults to 1 x 1 convolution), and then performs global spatial pooling. Finally, we evaluate alternatives to the last variant, where we keep an expansion layer, but instead of global spatial pooling, we apply L2 or square-root pooling.
3.4 TRAINING
We follow the ablation hyperparameter settings from Barlow Twins with 300 epochs of train- ing Zbontar et al. (2021). All these results are computed for the model based on 300 epochs of training on ResNet-50 using LARS optimizer. Batch size set to be 2048. We use cosine learning rate decay with an initial learning rate of 0.2. The projector is a three-layered MLP with 8192 hidden as well output units. For supervised training, we use the same training protocol as Barlow Twins, but instead of using output of size 8192 from the projector, and applying Barlow Twins loss on top, we use the same 8192 hidden units for the projector, but 1000 output units corresponding to the scores of the 1000 classes of ImageNet. We then apply simple cross-entropy loss to the model for classification. The difference in terms of architecture between a regular ResNet-50 and this model is that the classification head is lightweight in ResNet-50, with direct mapping 1000 classes, while we use the SSL projector for classification. We also use Barlow Twins augmentations for training, which are more diverse compared regular transformations used train supervised models.
4 MATCHING BACKPROPAGATION WITH BLOCKWISE TRAINING
A blockwise training procedure with self- supervised objective leads to accuracies com- parable with an end-to-end trained network. We trained simultaneously the 4 blocks (see Ima- geNet, using the Barlow Twins loss function at the end of each block. Before being sent to the projectors, the outputs of each block is fed to an expansion-basse pooling layer (see Section 3.3
Barlow Twins annd-to-End Training GSP)
Simultaneous Blockwise Barlow Twins (w/ 1x1 CbE GSP)
SimCLR (End to End Training w/ GSP)
Simultaneous Blockwise SimCLR (w/ 1x1 CbE + GSP)
vicReg (End-to-End Training w/ GSP)
Section 3 for details) of a ResNet-50
simultaneous Blackowie VicReg (w/ 1x1 ChE GSP)
70
for description of these pooling layers). We also add noise to the feature maps of each module with standard deviation of markably, we find that despite each block being trained independently (aside from the forward propagation interactions), our model achieves competitive performance against an end-to-end trained model: we see gap of only ~ 1.1% in performance (Fig. 4 This is surprising, well as non-trivial since we usually expect a large drop in performance when interactions between blocks are restricted to only forward dynam- ies on large-sees scale datasets (Illing et al., 2021; Xiong et al., 2020).
Accuracy (%)
0.25. Re-
30
20
i
Number blocks
Figure 5: Imppac of the SSL objective func- tion used. Our approach (simultaneous blockwise training with conv-based expansion pooling of the block outputs) almost matches end-to-ent training performance for all SSL methods tested.
Robustness against natural image degrada- tion (ImageNet-CC. We evaluate the robust-
ness of our best model (simultaneous blockwise training w/ 1x1 CbE GSP + Noise) against natural image degradation using ImageNet-C (Hendrycks and Dictterich, 2019) benchmark. The results are presented in Appendix C In summary, our results show that our blockwise training protocol de-

---

Under review as a conference paper at ICLR 2023
grades robustness in comparison to the end-to-end trained model. This reduction in robustness can be in part attributed to the lower top-1 accuracy of our blockwise trained model.
Extending to other SSL methods: SimCLR VicReg. We tested blockwise paradigm using alternative self-supervised learning rules i.c. VicReg and SimCLR. We visualize these results in 5 The results obtain both VicReg and SimCLR are consistent with our results on Barlow Twins: we only observe slight loss in accuracy as we move from end-to-end trained networks to expansion-be bbasdd pooling with simultaneous blockwise training, indicating that our blockwise learning paradigm is robust to the specific choice of the self-supervised learning rule used.
5 UNDERSTANDING THE CONTRIBUTION OF DIFFERENT COMPONENTS
We next analyze and discuss the contribution of the different components of our method to develop in-depth understanding of its success. For simplicity, we avoid noise addition to the layers unless we analyze its impact explicitly. Furthermore, we fix the block output pooling strategy to be local spatial pooling unless specified otherwise. We list some further ablations in Appendix B
Importance of using self-supervised learning rule opposed to supervised learning rule. We first compare our blockwise model trained with SSL loss against a supervised blockwise train- ing procedure, where we include the full projection head in the supervised case, but instead of applying the Barlow Twins loss, we project to the 1000 classes of ImageNet, and use the class labels for supervising the model at each of these intermediate blocks. We observe that the performance on the classification task of the initial blocks is significantly higher in the supervised learning case than in the SSL case, mak- ing the initial block very task-specific, while the latter blocks see reverse trend. This observation points an potential advantage of SSL over supervised learning for successful blockwise learning, as SSL rules preserves information about the input in initial blocks for further processing by latter blocks, whereas supervised learning objectives already discard information in inital blocks which could have been used by blocks stacked on top of them.
train each of these blocks using supervised learning. For fair comparison,
Importance of simultaneously training all the blocks. We compare the scheme where we simultaneous optimize the blocks to a se- quential optimization scheme. In the sequen- tial optimization scheme, we trained each of the blocks in a sequential manner where we freeze the parameters of the previous block (including batch-norm statistics i.e. mean and standard de- viation) before starting to train the next block. We see from Fig. 4 that sequential training re- sults in modest drop in performance with the
Barlow Twins (Endito-End Training w/ GSP)
Simultaneous Blockwise Training (LSP)
Simultaneous Blackwise Training (GSP)
Simultaneous Blockwise Training (Ix1 CbE GSP)
Simultaneous Blockwise Training (1x1 CbE L2)
Simultaneous Blockwise Training (1x1 CbE Square-Root)
Accuracy/95 (%%
model accuracy dropping from 68. 8..8%% (simul- tancous) to 67. 67.12% (sequential). This decrease in performance can be attributed to two pos- sible reasons: (1) during simultaneous train- ing, the learning trajectory in latter blocks is affected by the learning trajectory co-occurring in earlier blocks and this results in an overall better learning trajectory for the model, or (2) the batch-norm statistics could be interacting in subtle ways during simultaneous training which are beneficial. We tested the second hypothe- sis by adapting the batch-norm means and vari- ances in early blocks during the training of all the subsequent blocks i.e. by keeping the batch- norm parameters always in training mode. This change makes negligible impact to the end results, indicating that the it is the optimization trajectory taken by these blocks together during training that is beneficial for downstream performance.
50
40.
30
1
Number of blocks
Figure 6: Comparison of different pooling strate- gies. Alternative pooling strategies show better performance on earlier blocks, but worse perfor- mance at the final block.
7

---

Under review as a conference paper at ICLR 2023
Importance of our custom global average pooling strategy. Fig. Glighhighht the results that obtain when comparing the different pooling strategies presented Fig. We see that conv-based expansion w/ global spatial pooling yields the best final accuracy. Local spatial pooling and global spatial pooling obtains similar performance at the final block, which is slightly inferior to the per- formance obtained by conv-hased expansion w/ global spatial pooling. Conv-based expansion w/ L2-pooling significantly improves the performance of the initial blocks, but deteriorates the perfor- mance of the final blocks accordingly (this trend is similar to the supervised blockwise training). This could be attributed to the fact that 12-pooling normalizes the feature maps using the L2 norm, which overly emphasizes the highest activations. Square-root pooling attempts the opposite, where it tries to squeeze the large values, while amplifying small values. However, the overall trend for Square-rool pooling remains similar to the trend observed for 1.2pooling despite being the opposite, with significant degradation in performance at the final block in contrast to conv-based expansion w/ global spatial pooling.
Impact of filter size conv-based expansion pooling Our conv-based expansion pooling strat- egy uses a filter size of 1 x 1. We explore the impact of expansion pooling layer size in Fig. [] The figure shows that increasing the filter size of the expansion pooling layer significantly harms the per- formance of the system, suggesting that aggregating information over larger neighborhoods (higher effective receptive field) increases the level of supervision provided at the initial layers, harming the performance at the final blocks.
Comparison against a trivial baseline of un- trained previous blocks. We next exclude the possibility that the overall accuracy of our blockwise model could be trivially matched by a network of equivalent depth with untrained previous blocks (such random feature models have been used the past with some suc- cess when only training the final layers of the model (Huang et al., (2004)). In order to test this, compare the accuracy of our model against a model where the previous blocks are untrained, initialized randomly and frozen. We only train the top block in this case while keep- ing the (untrained) blocks below fixed. We find that the gap in accuracy between our layer-wise model and the random feature model widens as we stack more blocks without training the ini- tial blocks as visualized in Fig.
---- Barlow Twins (End-to-End Training w/ GSP)
Simultanecus Blodkwise Training (w/ LSP)
Combined Simultaneaus Blackwise Training (B1 82 w/ LSF)
Sequential Blockwise Training (81 trained end-to-end -w/ LSP)
70
65-
Accuracy (8))
60-
55-
45
1
2 Mumber blocks
Impact of first block on performance Cor- rectly supervising the first block is particularly challenging all subsequent blocks depend on it. We highlight the impact of training the first block in different settings in Fig. 7 We observe that if the first block is trained in an end-to- end setting, and then plugged into the block- wise training protocol, this largely bridges the gap with end-to-end Barlow Twins (71.57% for the end-to-end variant vs. 70.07% for the vari- ant where the first block is trained end-to-end, while the rest of the blocks are trained in a sequential manner). Alternatively, if we combine the first two blocks (B1 and B2), and then train them using the same simultancous blockwise training protocol, this achieves performance on par with using an end-to-end trained first block. This highlights that the main challenge of our blockwise train- ing procedure resides in finding a way to correctly train the first block. We explore different ways to customize the learning process for the first block in order to improve the performance of the whole network in Appendix A However, none of these strategies produced the expected gains in performance.
Figure 7: Evaluating the impact of (1) training the first block separately using an end-to-end training procedure, and of (2) merging the first and sec- ond blocks during training. Both modifications greatly reduce the performance gap with end-to- end training, indicating that finding a correct train- ing procedure for the the first block would be crit- ical to completely match end-to-end training per- formance.
8

---

Under review as a conference paper at ICLR 2023
Impact of noise addition Self-supervision in the early blocks may make the blocks too task- specific, reducing overall performance. Therefore, we experiment with the use of noise injected in the activations during training. We pick particular and add Gaussian noise with zero mean and the chosen standard deviation to the activations of the model before the beginning of every block. Injecting independent Gaussian noise to each of the feature values with μ = 0 and boost in performance of ~ 0.5%. Using a = 0.5 results in lower gains due to high levels of noise (70.22% with = 0.5 vs. 70.48% with = 0.25). We further experimented with different schemes for noise injection such as injecting the same noise to all the features at particular spatial location, and found independent noise for each of the features to work the best. We present the results with noise injection in Fig. Without the addition of noise, our expansion pooling strategy achieves an accuracy of 70.15%.
0.25 gave
DISCUSSION & CONCLUSION
In this paper, we showed that by adapting recent superviedd learning rules to a blockwise train- ing paradigm, we were able to produce a blockwise training procedure that is comparable in per- formance to full backpropagation on a large-scale dataset. Although there no practical advantage of our method at this point, we helieve that our findings are promising from neuroscientific stand- point, as they propose a viable alternative to backpropagation for learning at scale. Moreover, such blockwise training procedures combined with an adapted neuromorphic hardware could eventually speed-up learning and make it more meegg-eefficient by eliminating the need for information trans- portation through the usual backpropagation path.
Limitations and Future Work We are using blocks which do not interact through a backpropa- gation path. However, each of these block from ResNet-50 (He et al., 2016) is composed of large number of layers as well as two-layer projector network throughout which the backpropagation algorithm is still used. In order to completely get rid of backpropagation, one would need to find a learning rule functioning at layerwise granularity. Unfortunately, our experiments in Fig. 7 showing that merging the two first blocks into one increases the final accuracy of the network indicate that this gap in performance widens as we progressively subdivide the networks into more blocks. Consequently, our current approach would not be successful a purely local learning paradigm.
Moreover, Barlow Twins' loss function computes a cross-correlation matrix across all features of given layer, which implicitly adds a interaction term between all the neurons of this particular layer. In a purely local learning setup. interactions should be limited to local neighborhoods. Illing et a (2021) proposed such a local learning rule, but they have not demonstrated them to scale to large-scale datasets such as ImageNet.
We find that although there is no backpropagation of information from upper blocks lower blocks, the feed-forward path still conveys some information that couples the training of the blocks, and that this coupling important, as we show that simultaneous training better than sequential training from 1.72% points on ImageNet. This means that we cannot yet use our approach to reduce the memory footprint of training by training the blocks sequentially. In future work, one could try to emulate the training of previous blocks by adding specific noise to the previously frozen blocks in the ferd-forward path to the block being trained, effectively emulating sinultaneous training.
Future work could also be directed towards finding ways to adapt the augmentations difficulty to the layer being trained. But the parameter space to explore is large, and our attempts at sorting sample difficulty and routing samples with simple distortions to early blocks has not worked so far (see Appendix A for details). Masked Auto-Encoders (MAE) are an interesting recent class of self-supervised methods based on Vision Transformer architecture (Dosovitskiy et al. 2020) where the model learns via simple masking of tokens/patches (He et al., 2022). We postulate that our blockwise training framework could be particular well-suited such a masking scheme, given that one can adapt the level of difficulty for the augmentations fed to different blocks very naturally via the fraction of patches masked.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmad, N., van Gerven, M. A. and Ambrogioni L. (2020). Gait-prop: biologically plausible learning rule derived from backpropagation of error. NeurIPS.
Baldock, R., Maennel, H., Neyshabur, (2021). Deep Learning Through Lens Example Difficulty. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S., and Vaughan, J. W. editors, Advances in Neural Information Processing Systems, volume 34, pages 10876-10889. Curran Associates, Inc. [
Bardes, A., Ponce, J., and LeCun, Y. (2022). VICReg: Self- Learning. Number: Sttl:2105.04990 MxXiv:2105,04900 [cs]. 3 4
Invariiane--Cvvvriane Regularization for
Belilovsky, E., Eickenberg, M., and Oyallon. E. (2019). Greedy layerwise learning can scale to ImageNet. In
ICML. 22 E
Caron, M., Misra, I,, Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. (2021a). Unsupervised Learning of
Visual Features by Contrasting Cluster Assignments. Number: arXiv:2006. 09882 arXiv:2006.09882 [es]
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021b). Emerging
Properties in Self-Supervised Vision Transformers. Number: arXiv:2104. 14294 arXiv:2104. 1ett:1144..4494 [cs]
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020a). A simple framework for contrastive learning of
visual representations. In International conference machine learning, pages 1597-16077 PMLR.
Chen, T., Korbblith, S., Swersky, K., Norouzi, M., and Hinton, G. (20206). Big Self- Supervieed Models Strong Semi-Superrised Leamerr. Number: arXiv:2006. 10029 arXiv:2006. 10029 [es, stat].
Chen. X.. Fan, H., Girshick, R.. and He, K. (2020c). Improved Baselines with Momentum Contrastive Learning. Number: arXiv:2003.04297 arXiv:2003.00 0429770700000000 [es].
Clark, D., Abbott, L. F., and Chung, S. (2021). Credit Assignment Through Broadcasting Global Error Vector. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S., and Vaughan, J. W., editors, Advances Neural Information Processing Systems, volume 34, pages 10053 10066. Curran Associates, Inc. 12
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009). Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. (2020). An image is worth 16x 16 words: Transformers for image recognition 20t::2000.111222
scale. arXiv preprint
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman, A. (2021). With Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations. Number: 1Xiv:2104.1454:
arXiv:2104. 14548 [es]. D
Emoult, M., Normandin, F., Moudgil, A., Spinney, S., Belilovsky, E., Rish, I., Richards, B., and Bengio, Y. (2022). Towards scaling difference target propagation by learning backprop targets, arXiv.
Halvagal, M. and Zenke, F. (2022). The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Pages: 2022.03. 12..00..7.4447111
Section: New Results. |23
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16000 16009, 9
He. K. Fan, H., Wu. Y., Xie, S., and Girshick, R. (2020). Momentum Contrast for Unsupervised Visual Representation Learning. Number: arXiv:1911.0572. 1XXv:1911.00722 [cs].
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Hendrycks, D. and Dictterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprini ar0i:: /2261.
16
Huang, G. Zhu, Q.- and Siew, C.-K. (2004). Extreme learning machine: a new learning scheme of
feedforward neural networks. In 2004 IEEF International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), volume 2, pages 985-990 vol.2. g
10

---

Under review as a conference paper at ICLR 2023
Ilnin, B., Ventura, J., Bellec, G., and Gerstner. W. (2021). Local plasticity rules learn deep representations using supervisdd contrastive predictions. In Ranzato, M., Beygelzimer, A., Dauphin, Liang, S., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 30365 30379. Curran Associates, Inc. 33
Loe, D.-.., Zhang, S., Fischer, A., and Bengio, Y. (2015). Difference target propagation. ECML PKDD.
Lillicrap, T. Cownden, D., Tweed, B., and Akerman, C. J. (2016). Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276. Number: Publisher: Nature Publishing Group.
Lillicrap, T. P., Santoro, A., Marris, L.,, Akerman, J. and Hinton, G. (2020). Backpropagation and the brain. Nature Reviews Neuroscience, (1(6):335-3) -366-646.6. Number: Publisher: Nature Publishing Group. I
Löwe. S., O'Conoo, P., and Veeling, B. (2019). Putting an end to and-to-end: Gradient-isolated learning of representations. arXiv preprint ar5.iv:905..112200
223
Nøkland, A. (2016). Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29.
Salakhutdinov, R. and Hinton, G. (2009). Deep boltzmann machines. In AISTATS.
Shepard, R. N. Metzler, (1971). Mental rotation three-dimensional objects. Science, 171(3972):701- 703. Place: US Publisher: American Assn for the Advancement of Science.
Wu, B., Nair, S., Martin-Martin, R., Fei-Fei, L., Finn, C. (2021). Greedy hierarchical variational autoen- coders for large-scale video prediction. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2318 2328.
Xiong, Y., Ren, M., and Urtasun, R. (2020). Loco: Local contrastive representation learning. arXiv preprint arXiv:2008. 01342. 230 6
Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., and Molchamov, P. (2022). A-VIT: Adaptive Tokens for Efficient Vision Transformer. Number: arXiv:21 1XXi:22122007658 [es]. !
Zbontar, J., Jing, L., Misra, L., LeCun, Y., and Deny, S. (2021). Barlow twins: Self-supervised learning via 15
redundancy reduction. arXiv preprint arXiv:2103.032330

---

Under review as a conference paper at ICLR 2023
A CUSTOMIZING THE BLOCKWISE TRAINING PROCESS
We tried to customize the training procedure to our blockwise paradigm by progressively increasing the complexity of the task for each block. Although none of these schemes improved accuracy of the final model, we hope that laying out these schemes and discussing them will aid future research by providing insights on directions worth exploring further.
A.1 BLOCKWISE ADAPTIVE LEARNING STRATEGIES
Analyzing the limitations of the first block. We visualize the distribution of the on-diagonal and off-diagonal terms from the Barlow Twins loss in the form of violin plot in Fig- X Since the end- to-end trained model has no projection head, we always plot these terms based on the features from the ResNet-50 backbone, without any projection head and using global average pooling. We see that supervised learning provides a steady increase in the correlation of the on-diagonal terms with increasing number blocks, while simultaneously reducing the off-diagonal correlation. However, when looking at blockwise trained models, we see that the initial block is much better at modeling the on-diagonal correlation, while is worse at modeling the on-diagonal correlations in the last block. This again relates back to our observation that providing strong supervision at the initial blocks can worsen the performance of the overall network. We further explore reasons for this mismatch and how they can be catered in the subsequent sections via block combinations well tuning of the learning rate or lambda values. It is apparent from Fig. 8 that the first block of block-wise trained model significantly different than the end-to-end trained model. Therefore, as sanity check, we combine the first-two blocks of the network. This ensures that the first block is not strictly super- vised, and is mainly guided by the performance of the second block. We visualize these results in Fig. 7 We find that this configuration helps in closing the gap between the supervised variant and our block-wise variant where the simultaneous blockwise training achieves 68. 84% while the com- bination of the first two blocks achieves 70.14% top-1 linear probe accuracy. This provides evidence for our initial hypothesis that strong supervision at the initial blocks is significantly detrimental to the downstream performance.
Matching the Invariance Characteristics of End-to-End Model Looking the difference in distribution between end-to-end and blockwise trained models, it seems likely that minimizing this shift will improve the performance of the model. For this purpose, evaluate matching the invari- ance term with that of the end-to-end trained model (redundancy reduction mismatch is already low). One important detail to note is that the difference between the invariance and redundancy reduction terms reported in Fig. are computed based on the application of global spatial pooling to the fea- tures of the network without the projection head (as there no projection head for the end-to-end trained model). Therefore, we use slightly higher values for the target invariance, changing it from 1.0 for all the blocks to 0.6, 0.75, 0.9, and 1.0 for block- block-2, block-3 and block-4 respectively. However, we found that this fails to improve performance (results instead in an insignificant drop in performance from 70.15% for our model with СbE GSP to 69.90% with the new invariance targets). [SAS-TODO: Add figure reference]. We also visualize the feature difference (similar to Fig. [ but for the model trained with adapted invariance targets). It shows that the chosen target was higher than the desired. Therefore, different schedule for targets can potentially outperform the current method in practice.
Lambda tuning does not help This experiment was based on blockwise training. In order to re- duce the computational burden, we trained the rest of the blocks end-to-end by only adjusting the Lambda value for the first block. One recurring theme that we observed throughout our previous ex- periments was the inferior results when trying to closely supervise the initial layers of the model. We find that supervising these initial layers makes them task specific, reducing their overall utility. In order to circumvent this issue, we tune the lambda value which provides a trade-off between obtain- ing invariance to the applied augmentations We assume that enforcing invariance at these initial layers can be premature, however, enforcing diversity in the extracted features by minimizing their correlation can still be beneficial. The Barlow Twins implementation used a very small value of x = 0.0051. Therefore, we experimented with larger lambda values. These results visualized in Fig. 9 We see negligible impact of chang-
minimizing correlations between different features.
12

---

Under review as a conference paper at ICLR 2023
(a) Blockwise (w/ LSP)
(b) Simultaneous Blockwise
(c) Simultaneous Blockwise
(w/ LSP)
(w/ CbE GSP)
1.0
2.0
1.4
0.8
0.8
0.8
oN- mndilggnal co conelation
8.6
0.4
and to-and
5.3 soiend
kndito end
0.0
0.0
0.0
0.4
0.4
oef Pl---laguaallcoo cornaation
0.2
0.2
0.7
0.0
0.0
Rocovas Endto-and
-0.
Smrutrnnus Slodwiis LS7
Senuitamooa Eloccoton Ind 2e1 CBS CDP)
Block Indix
Indix
Block Index
Figure 8: Comparison between ON-diagonal and OFF-diagonal terms of the cross-correlation ma- trix between features (without projection head or expansion layers and using global average pooling) in comparison with the end-to-end Barlow Twins model. Conv-based pooling improves the align- ment on the OFF-diagonal terms, while only improving alignment the ON-diagonal terms the last block.
70
68
Accuracy (%)
66
64
62
60
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Lambda
Figure 9: Comparison of different Lambda values used to train the model. The default value of Lambda for Barlow Twins is À = 0.0051.
ing the lambda value. However, the value of lambda gets significantly larger, this reduces the performance of the model significantly.
Adaptive augmentations Considering that tuning the lambda value made negligible contribu- tion to the model performance, we tuned the model augmentations such that easier augmentations are applied at the initial blocks and the difficulty of the augmentations increases gradually with higher
13

---

Under review as a conference paper at ICLR 2023
Barlow Twins (End-to-End Training w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Adaptive Loss (Train all below - w/ LSP)
Adaptive Loss (weight=0.5- w/ LSP)
Adaptive Loss (weight=0.75- LSP)
Augmentation Tuning LSP)
70
60
Accuracy (%)
50
40
30
Number blocks
Figure 10: Impact of using adaptive augmentations.
blocks. This is beneficial initial blocks can easily learn invariance against color jitters. How- ever, learning invariance against large random crops can be very difficult due to limited receptive field. Since the dataset is changing, we experiment with this protocol in strictly sequential setting (where we train the blocks sequentially instead of training them simultaneously). For this purpose, defined successive augmentation list, where only applied color jitter at the first block. We additionally applied small random crops at the second block. For the third and fourth block, we ap- plied the full-set of Barlow Twins augmentations. We visualize the results from this experiment in Fig. [0] We see that adaptive augmentations makes the performance significantly worse However, might be due to some missing component that we were unable properly tune. We consider this to be an interesting and open problem for further research, and can result in non-trivial gains.
Example difficulty-basse routing The previous setting with adaptive augmentations can intro- duce disparity due to differences in the patches seen during training of the initial blocks vs. training of the later blocks. Therefore, is not truly ideal. We experimented with more principled approach of routing the examples to the correct block based on their difficulty by leveraging their training dy- namics i.e. routing examples with the smallest loss to the initial blocks while routing examples with the highest loss to the highest blocks. However, it is still important to train all the other blocks aside from the primary block an example routed towards. Therefore, we tested different schemes in this regard. The first scheme is [Train all below] setting where an example when routed particular block, also trains all the blocks below. All the other settings where we specify the weight refers to the setting where train all the other blocks (not only the blocks below), but with reduced weight specified using the weight mentioned the legend. We visualize the results with this routing mechanism in Fig. [0] Similarly the adaptive augmentations, found this routing reduce the performance of the model significantly. However, we also consider this to be a very interesting avenue be explored further in future work.
B FURTHER ABLATIONS
We performed some trivial ablations by increasing the size of the projector, changing the learning rate of the linear classifier, even employing group convolutions.
14

---

Under review as a conference paper at ICLR 2023
Barlow Twiins (End-to-End Training w/ LSP)
Simultaneous Blockwise Training (1x1 CbE + GSP)
Simultaneous Blockwise Training (7x7 CbE + GSP)
70
60
Accuracy (%)
50
30
2
3
Number blocks
Figure 11: Evaluating changing the filter size of our expansion pooling layer.
Barlow Twins (End-to-End Training w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Simultaneous Blockwise Training (Large Projector w/ LSP)
Simultaneous Blockwise Training (1x1 CbE GSP)
Simultaneous Blockwise Training (1x1 Group CbE + GSP)
70
60
Accuracy (%)
50
40
30
Number blocks
Figure 12: Impact of large projector size group convolution performance.
Increasing projector size does not help In all prior experiments, we fixed the projector size to be 8192 in accordance with the original Barlow Twins implementation (Zbontar al., 2021). However, given that local learning is significantly different than end-to-end learning, we experimented with large projection head. We visualize the impact of large projector size in Fig. 12 where we double the hidden dimensionality of the projection head to 16384. It is interesting to note that increasing the size the projector increases the performance of the initial blocks, but degrades performance at the final block. These results consistent with the observations made for end-to-en and-to-end Barlow Twins where they found negligible impact of increasing the size of the projection head beyond the limit of 8192 (Zbontar et al. 2021).
15

---

We find minimal impact of Under review as a conference paper at ICLR 2023
Twins (End-to-End Training-
Simultaneous Blockwise Training (w/ LSP) [LR: 0.1]
Simultaneous B Blockwise Training (w/ LSP) 0.3)
Simuitaneous Blockwise Training (w/ LSP) [LR: 1.0)
Combined Simultaneous Blockwise Training (MI+ MM1+M2) (w/ LSP) [LR: 0.1]
Combined Simultaneous Blockwise Training
[UR: 0.3]
Combined Simullaneous Blockwise Training (MI+M2) (w/ LSP) [LR: 1.0)
70
65-
Accuracy (%)
55
50
Number of blocks
Figure 13: Comparison of different learning rate settings. The default learning rate of Barlow Twins is 0.3.
Tuning the learning rate for the linear classifier does not help. The default learning rate for the linear classification head is set to be 0.3 in the original Barlow Twins evaluation (Zbontar et al., 2021). This was discovered after grid-search over three different values of learning rate. Therefore, in order to identify if different learning rate if more suitable in our case, we did small search over the same learning rate settings. These results are summarized in Fig. T3 changing the learning rate, with only deteriorating performance.
Usage of group convolutions Since the expansion-based pooling with x 1 convolution was the most impressive in terms of results, we experimented with extending this to group convolution in order to avoid performing this convolution operation over large spatial grid which a memory intensive operation. We visualize these results Fig. 12] Unfortunately, using group convolutions results in minor loss in performance in contrast to using the conventional x I convolution. However, this drop in performance is negligible considering real-world applications.
C ROBUSTNESS ON IMAGENET-C
In order to probe our model's robustness to out-of-distribution images, we use ImageNet-C bench- mark Hendrycks and Dietterich (2019) which was constructed by injecting different kinds of syn- thetic noise into the ImageNet test set. Each noise type further segmented into five different levels of noise, ranging from very mild level of noise to extreme level of noise. The noise types are selected such as to cover noise models found in the real-world.
We compared the robustness of our best model (simultancous blockwise training w/ 1x1 CbE + GSP + Noise) and the end-to-end trained Barlow Twins model on ImageNet-C. We report the error rate as well as the standard deviation computed over all five corruption levels for each corruption type. The results are visualized in Fig. 14 The mean error rate (averaged over all corruption types) for the end-to-end trained model is 62.81%, while the mean error rate for best model is 65.68%. This reduction in robustness can be partly attributed to the lower clean accuracy (71,57% vs. 70.48%). However, these results indicate that blockwise training as operationalized in case is not particu- larly useful for robustness against (natural) image degradations.
16

---

Under review as a conference paper at ICLR 2023
100
End-to-end
Simultaenous Blockwise (1x1 Noise
80
Error rate (%)
20
Train
Validation
Zoom Blur
Motion Blur
Glass Blur
Defocus Blur
Gaussian Noise
Impulsa Noise
Shot Noise
Contrast
Elastic Transform
Jpeg Compression
Pixelate
Frost
Brightness
Fog
Snow
Spatter
Saturate
Gaussian Blur
Speckle Noise
Corruption type
Figure 14: Comparison between rates of our best model (simultaneous blockwise training w/ Ixl CbE + GSP + Noise) and the end-to-end trained Barlow Twins model. The figure shows that blockwise trained model exhibits lower robustness ImageNet-C distortions compared to the end-to-end trained model, with an average error rate of 65.68% vs. 62.81 % of the end-to-end trained model.
17


Reviews:
[{"review_id": "td4mwkuk-_", "reviewer": "Reviewer_Y92E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes a strategy to train the network in a blockwise manner. Specifically, this strategy explores two local learning methods for blockwise BP training based on the basic blocks in the ResNet-50 model, i.e. simultaneous blockwise training and sequential blockwise training. In addition, the paper also explores some design details, such as pooling methods, parameter settings, and augmentation methods. At last, the manuscript presents experiments and ablations of the proposed strategies.", "summary_of_the_review": "The blockwise training strategy that uses a stop-gradient operation between blocks is not novel. This approach reduces the performance of the Barlow Twins method and the robustness of the model. It also does not achieve the purpose of limiting the memory and improving the adaptive capability of the model.", "strength_and_weaknesses": "Strength\nThe paper explores two strategies for blockwise training based on the Barlow Twins method. The authors perform more detailed experiments and ablation of the proposed methods. Overall, the paper describes the proposed method in detail and validates the method with experiments, and the paper is clearly written and well organized.\nWeaknesses\nMotivated by the fact that local learning can limit memory when training the network and the adaptive nature of each individual block, the paper extends local learning to the ResNet-50 to handle large datasets. However, it seems that the results of the paper do not demonstrate the benefits of doing so. The detailed weaknesses are as follows:\n1)The method proposed in the paper essentially differs very little from the traditional BP method. The main contribution of the paper is adding the stop gradient operation between blocks, which appears to be less innovative.\n2)The local learning strategy is not superior to the BP optimization method. In addition, the model is more sensitive to each block after the model is blocked, especially the first block. More additional corrections are needed to improve the performance and robustness of the model, although still lower than BP's method.\n3)Experimental results show that simultaneous blockwise training is better than sequential blockwise training. But the simultaneous blockwise training strategy cannot limit memory. \n4)The blockwise training strategy relies on a special network structure like the block structure of the ResNet-50 model.\n5)There are some writing errors in the paper, such as \"informative informative\" on page 5 and \"performance\" on page 1, which lacks a title.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RYRbfI4Mw8I", "reviewer": "Reviewer_Voxk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a blockwise learning strategy based on the Barlow Twins framework as an alternative to replacing standard backpropagation that is commonly used in existing SSL algorithms. Compared to related works, their work addresses the challenge of large-scale datasets. In addition, they compared several spatial and feature pooling strategies for building intermediate features and evaluated a number of strategies to customize the training procedure for blockwise training. Through their numerical studies, they demonstrate that blockwise training can achieve competitive performance compared to the original framework.", "summary_of_the_review": "Please see my comments in Strength And Weaknesses.", "strength_and_weaknesses": "Strength \n- This paper proposed an alternative approach to training SSL models that can be used for training large neural networks with a limited memory budget.\n- They evaluated a variety of strategies to customize the training procedure at the different blocks and the proposed strategies are also transferable to other SSL frameworks.\n- They conducted various ablation studies to investigate individual component of their framework, such as comparing different feature pooling strategies for the intermediate block output.\n\nWeaknesses\n- The objective formulation for blockwise training is somewhat unclear to me. How do the authors define the loss in sequential or simultaneous blockwise training?  Similarly, what is the formulation for other frameworks, such as SimCLR?\n- It lacks experiment details. For example, what training parameters, such as epoch and batch size, are associated with the results in section 4? How do authors conduct supervised tasks, e.g., loss and model change?\n- It would be interesting to extend the current results to different model architectures. e.g., different CNN/transformer-based models, and include more results on different SSL frameworks, such as SimCLR, and MoCo.\n- The authors mentioned one of the main advantages of this learning strategy is its ability to handle larger neural networks, it would be interesting to include more comparisons by varying the model size to support this claim. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "10iUdLNEoC", "reviewer": "Reviewer_JnWG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes multiple tricks to push the performance and of the self-supervised Barlow Twins step in a block-wise setting. Most prominently, it is reported than a block-wise training of a ResNet architecture with Barlow twins leads to a high top-1 accuracy of a down-stream classifier (70+% which is one percent away from end-to-end training). To reach this performance a new method to extract a low-dimensional vector from the ResNet feature map and a noise injection strategy are described.", "summary_of_the_review": "Most of the results are very clearly written. It is very valuable to see some tricks which make Barlow twins competitive in this setting. This paper would be probably more impactful or important if the experiments were targeted to support one strong result with a central story line rather than a list of somewhat unrelated tricks.", "strength_and_weaknesses": "The biggest result of this paper is certainly to demonstrate how competitive can be the Barlow Twins in this block-wise setting. Although this had been demonstrated for the CPC loss by Lowe et al, this result had not been reproduced with other losses on a dataset as hard as ImageNet as far as I know. This alone is likely to be useful intermediate step torwards back-prop-free self-supervised learning method which perform well on ImageNet.\n\nAlthough it certainly took a lot of work to make it work, I do not see big conceptual innovation but this is not always necessary to make a good paper. Let's recap why I think some results were already published by other authors:\n- Barlow twins was already shown to work very well on ImageNet in the original paper,\n- Lowe et al. 2019 demonstrated than block-wise CPC worked well,\n- Lowe et al. 2019 inspected some training variants including simultaneously or sequential trainings of the blocks,\n- The noise addition is likely to have an effect similar to drop out. It is unfortunate that we do not have a comparison between the suggested noise model against regular dropout.\n\nThe combination of feature expansion and pooling is probably the most creative innovation here. It is indeed expected that some kind of reshaping and post-processing is necessary to transform the feature map into a low-dimensional latent variable that is relevant for the loss function. Inspecting which variant is best is certainly useful for future work.\n\nI find it very curious that the network with lower top1 accuracy at early layers perform well for later layers. This curiosity is also put forward in the paper as an interesting side result. This might indeed hide some important effect which explain the performance difference. A deeper analysis of this might make the paper stronger, why does it happen with the block-wise barlow twins with the specific pooling strategy? Is that an effect of the pooling loss only or the loss function too?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject