Blockwise Self-Supervised Learning at Scale
Authors: Shoaib Siddiqui
David Krueger
Yann LeCun
Stephane Deny
Venue: Accepted by TMLR ()

Abstract: Current state-of-the-art deep networks are all powered by backpropagation. However, long backpropagation paths as found in end-to-end training are biologically implausible, as well as inefficient in terms of energy consumption. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. We show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins' loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48\%, only 1.1\% below the accuracy of an end-to-end pretrained network (71.57\% accuracy). We perform extensive experiments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience.

Published in Transactions on Machine Learning Research (01/2024)
Blockwise Self-Superviser Learning at Scale
Shoaib Ahmed Siddiqui University of Cambridge, UK
msas30comm.com uk.
David Krueger
dsk:30@com.cco uk
University of Cambridge, UK
Yann LeCun
yann com
Meta-FAl FATR, NY, USA New York University, NY, USA
Stéphanc Deny
stephane dengy@ualto.
Aolto University, Espoo, Finland
Reviewed on OpenReview: https: // openreview. net/ forum? id= N2m618:11P
Abstract
Current stale-of-the-art deep nelworks are all powered by backpropagation. However, long backpropagation paths as found in end-to-end training are biologically implausible, as well as inefficient in terms of energy consumption. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. We show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins' loss function al each block performs almost as well as end-to-end backpropagation on Ima- geNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48%, only 1.1% below the accuracy of an end-to-end pretrained network (71.57% accuracy). We perform extensive experiments understand the impact of different components within supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience. Code to reproduce our experiments is available at: httpaa/..a...aaaaaiaaaee//llalaa b
method and explore variety of adaptations of self-
1 Introduction
One of the main components behind the success of deep learning is backpropagation. It remains an open question whether comparable recognition performance can be achieved with local learning rules, question that dates back to the carly days of deep Icarning when Hinton al. (2006); Bengio al. (2006) proposod greedy layerwise training algorithm for deep belief networks (DBN). Recent atlempts in the context of supervised learning and unsupervised learning have only been successful
small datasets like MNIST
(Salakhutdinov & Hinton 2009 Löwe et al.] 2019 Ahmad et al. 2020 Ernoult. et al. 2022 Lee et al.
2015) or large datasets but small networks like VGG-11 (Belilovsky et al. 2019 (67.6% top-1 accuracy on ImageNet).
Being able to train models with local learning rulcs at scale useful for multitude of reasons. By locally
training different parts of the network, one could optimize learning of very large networks while limiting the memory footprint during training. This approach was recently illustrated at. scale in the domain of

---

Published in Transactions on Machine Learning Research (01/2024)
Forward propagation path
Backpropagation path
Dependence within layer
No dependence within layer
Biological Plausibility
Demonstrated Large-Scale Dalaset Performance
Input
Label
Bg
W4
Bg
W3
B2
W2
B1
W
x
End-to-and backpropagation [He et al. 2015]
Feedback alignment [Lillicrap et al. 2016)
[Bellkovsky al. 2019]
[Clark al. 2021]
[Nekland 2016]
[Hinton 2022]
[Xiong et al. 2020]
LoCo
(Halvagal 2022]
[Lówe al. 2019]
Ours
[Journe et al. 2022]
[Tang al. 2022
[iling et al. 2021]
Figure 1: We rank blockwise/looe llocal learning methods of the literature according to their biological plausibility (from left to right), and indicate their demonstrated ability to scale to large-scale datasets (e.g., ImageNet) by the intensity of the blue rectangle below each model family, Our method is situated at a unique trade-off between biological plausibility and performance on large-scale datasets, by being on par in performance with intertwined blockwise training (Xiong et al., 2020) and supervised broadcasted learning (Nokland 2016
Clark et al.] 2021 Belilovsky et al. 2019) while being more biologically plausible than these altornatives.
hods represented with magenta color used similar methodology ours (Halvagal & Zenke 2022 Löwe
ct al. 2019 but have only been demonstrated to work on small datasets.
video prediction, using a stack of VAEs trained sequentially in a greedy fashion (Wu et al. 2021 From a ncuroscientific standpoint, is interesting to explore the viability of alternative learning rules back- propagation, as it is debated whether the brain performs backpropagation (mostly considered implausible) [Lilicrap et al., 2, approximations of backpropagation (Lillierap et al., 2016), or relies instead local learning rules (Hllagal & Zenke 2022 Clark et al. 2021 Illing et al. 2021). Finally, local learning rules could unlock the possibility for adaptive computations, as cach part of the network is trained to solve a subtask in isolation, naturally tuning different parts to solve different lasks ( et. al.) 2022 Baldock el al. 2021), offering interesting energy and speed trade-offs depending on the complexity the input. There is evidence that the brain also uses computational paths that depend on the complexity of the task (c.g..
Shepard Metzler (1971)).
Self-supervised learning has proven very successful as approach to pretrain deep networks. A simple linear classifier trained on top of the last layer can yield accuracy close to the best-supervised learning methods
(Chen et al. 2020a b; He et al. 2020 Chen et al. 2020c Caron et al. 2021a b Zbontar et al. 2021 Bardes
(et al. 2022 the desired properties the learned representations: the first term ensures invariance to distortions that do not affect the label of an image, and the second term ensures that the representation informative about its input (Zbontar et al. 2021). Such loss functions may better reflect the goal of intermediate layers compared to supervised learning rules which only try to extract labels, without any consideration of preserving input information for the next layers. In this paper, revisit the possibility of using local learning rules replacement for backpropagation on a large scale dataset. As natural transition towards purely local learning rules, we resort to blockwise training where we limit the length of the backpropagation path to single blocks the network]] We apply Barlow Twins, a recently proposed self-supervised learning loss, locally different blocks of a ResNet-50 trained on ImageNet and make the following contributions:
Dwibedi et al. 2021 Self-supervised learning objectives typically have two terms that reflect
I We consider layers with the same spatial resolution belong the same block.
2

---

Published in Transactions on Machine Learning Research (01/2024)
Sequential Blockwise Training
Trained and Frozen
Simultaneous Blockwise Training
Training
Backpropagation path
Pool
Proj
Loss
Block
Block 2
Block
Block 3
Block
Block
Block
Block
Figure Overview of our blockwise local learning approach. Each of the 4 blocks of layers of a ResNet-50 trained independently using a self-supervieed learning rule and a local backpropagation path. We apply this procedure in two main settings: (left) sequential training, where each block, starting from the first block, is independently trained and frozen before the next block is trained; (right) simultaneous blockwise training, where all the blocks are trained simultaneously using 'stop-grad' operation to limit the backpropagation paths to a single block. The yellow gear symbol refers to the combination of the pooling layer, projection head, and loss function used in self-supervised learning.
We show that ResNet-50 trained with our blockwise self-supervised learning method can reach performance on par with the same network trained end-to-end with backpropagation, as long as the network not broken into too many blocks.
We find that training blocks simultaneously (as opposed to sequentially) is essential to achieving this level of performance, suggesting that important learning interactions take place between blocks through the feedforward path during training.
We find that methods that increase downstream performance of lower blocks on the classification task, such supervised blockwise training, tend to decrease performance of the overall network, suggesting that building invariant representations prematurely in early blocks is harmful for learning higher-level features in latter blocks.
We compare different spatial and feature pooling strategies for the intermediate block outputs (which are fed to the intermediate loss functions via projector networks), and find that expanding the feature-dimmnssionllity of the block's output is key to the successful training of the network.
We evaluate variety of strategics to customize the training procedure at the different blocks (c.g.. luning the trade-off parameter of the objective function as a function of the block, feeding different image distortions to different blocks, routing samples to different blocks depending on their diffi- culty), and find that none of these approaches addod substantial gain porformance. However, despite negative initial results, we consider these attempts be promising avenues to be explored further in future work. We exhaustively describe these negative results in Appendix
2
Related Work
This section provides comparison between the existing local learning paradigms in the literature and our method as summarized in Fig. []] We compare these methods from two perspectives: biological plausibility and demonstrated performance on large-scale datasets, such as ImageNet. (Deng el al. 2009
End-to-end backpropagation (He al. 2016 the best-performing paradigm on largo-scale datascts, but it also considered the least biologically plausible (although not impossible, see (Lillicrap et al., (2020)) because it requires long-range backpropagation path with symmetric weights to the feed-forward path.
3

---

Published in Transactions on Machine Learning Research (01/2024)
(a)
(c)
256
Conv-based
Global Spalial Pooling (GSP)
1x1 Çoný (2048 channels)
Expansion (CbE) GSP
(b)
Average
Pooling
2048
2048
Local Spatial Pooling (LSP)
Figure 3: Overview of the pooling strategies employed in this work. The first one (a) is the simple Global Spatial Pooling (GSP) which simply computes global average activation over the entire feature volume. The second one (b) is the Local Spatial Pooling (LSP) where we divide the feature volume into small spatial bins, compute the averages in these small bins, and concatenate these outputs to compute the final feature vector of size 2048. The third one (c) is a (1x1) Conv-based Expansion (CbE), followed by global spatial pooling. This last pooling strategy provides the best performance for our method.
Feedback aligument (Lillierap et al. 2016) path to be random and independent from the forwrd path. However, this method is not explicitly focused on reducing the length of the backpropagation path.
alleviates the symmetry constraint by making the backpropagation
Nokland (2016) Clark et al. 2021] |5 Belilovsky et al. (2019); Hinton 2022 are four different methods that avoid long-rane range backpropagation paths by directly broadcasting the labels to all layers. Therefore, these methods are more biologically plausible than backpropagation. However, all these methods are supervised and thus rely on a large number of labeled inputs, which itself is not biologically plausible.
LoCo (Xiong et al. 2020) applies backpropagation separately to different blocks of a ResNet-50. This method does not rely on an abundance of labels as it uses self-supervised objective, and performs well on large-scale datasets (69.5% top-1 accuracy ImageNet with linear probe). However, LoCo introduces coupling between subsequent blocks, by applying backpropagation to intertwined pairs of successive blocks. Therefore, this method still conceptually relies on a full backpropagation path.
Our method and the closely related methods (Halvagal & Zenke 2022 Löwe et al.
2019 avoid any coupling between subsequent. blocks by training each block independently, each using a variant. of self-supervised
lcarning, namely SimCLR (Chen et al.) 2020a). Barlow Twins (Zbontar ct al., 2021 and VicReg (Bardes let al., 2022). The main difference between our method and (Halvagal & Zenke 2022 Löwe et al. 2019 is that we demonstrate the effectiveness of our method biological plausibility, our method still relies on backpropagation within blocks and also relies on interactions between neurons within layers due to the exact form of the loss function. These remaining dependencies make our method only partially local, and thus not fully biologically plausible.
large-scale dataset i.e. ImageNet. In termss of
Finally, methods such as (Iling et al. 2021 Tang et al. 2022 Journé et al.) 2022) avoid all learning depen- dencies within and across layers and propose truly local learning rules, but these methods have only partially been demonstrated to perform well on large-scale datasets to date.
4

---

Published in Transactions on Machine Learning Research (01/2024)
3 Methods
We use a ResNet-50 network and adapt the Barlow Twins Zbontar et al., 2021 codebase to blockwise training paradigm.
3.1 Blockwise Training
We divide the ResNet-50 network into 4 blocks and limit the length of the backpropagation path to each of these blocks. The ResNet-50 architecture is comprised of 5 blocks of different feature spatial resolutions, followed by a global average pooling operation and a linear layer for final classification. The first. block of spatial resolution is only comprised of a single stride-2 layer. For simplicity, we integrate this layer within the first training block, obtaining training blocks. We train each of the blocks separately with self-supervised learning objective, using stop-gradient to ensure that the loss function from one block does not affect the learned parameters of other blocks (see Fig. 2]. We consider two different settings: (i) sequential blockwise training where we train the blocks in purely sequential way, iee. one after the other, and (ii) simultaneous blockwise training where we train all the blocks simultaneously. We always report top-1 accuracy on ImageNet by training a linear classifier on top of the frozen representations at the output of the diffcrent blocks.
3.2 The Barlow Twins loss
Most the experiments in the paper rely on the Barlow Twins objective function (Zbontar et al. 2021 which we briefly descríbe here. The input image is first passod through the deep network and then passod to two-layer projector network in order to obtain the embeddings to be fed to the self-supervised learning los We compute these embeddings for two different image views (each image being transformed by distinct set of augmentations) 2; (<((z))) where V represents the ResNet-50 encoder, and & represents the projection head. We then define cross-correlation matrix over these embeddings:
A Es 66,
B
Cjj A
(1)
B
bb.
where b indexes batch samples and i.j index the vector dimension of the networks' outputs.
The Barlow Twins objective function attempts to maximize the on-diagonal terms of the cross-correlation matrix and minimize the off-diagonal terms using the following loss function:
CBT & (1-
ΣΣ<r C2
(2)
invariance term
redundancy reduction term
where the invariance term, by ensuring that on-diagonal elements are close to 1, ensures that the extracted features are invariant to the applied augmentations, while the redundancy reduction term, by ensuring that off-diagonal elements are close to O, ensures that the different units (i.e., neurons) forming the final image rep- resentation capture distinct attributes of the input image. The redundancy reduction term thus encourages the representation to be richly informative about the input.
Despite most of our investigation being restricted to Barlow Twins, our findings generalize to other self- supervised learning (SST.) losses such as VicReg (Bardes et al.] 2022) or SimCLR (Chen et al. 2020a) as shown in the result section. We implemented SimCLR loss function (Chen al. 2020a within the
hhtpps: /tat:s://ithuue httbbbtttttttttt/ggg//ett/bbbb hwtm///cebbokreaar/////aaoootota
Using additional projection head learning directly on the backbone features has heen found effective and useful CChnn al. 2020a).
5

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins codebase and directly adapted the official VieReg implementation for our experiments with VieReg (Bardes et al. 2022
3.3 Pooling Schemes
The original Barlow Twins projector reccives an input of dimension 2048. However, the initial blocks of a ResNet-50 have a smaller number of channels. For example, the first block contains only 256 channels instead of 2048. Therefore, we explored different possible ways of pooling these features spatially in order to construct the feature vector that is fed to the projector.
Fig. 3 illustrates the different pooling strategies evaluated in this study, comprised of (1) the Global Spatial Pooling (GSP) strategy, which consists in applying simple average spatial pooling operation to the block's output layer, such that the projector's input dimensionality equals the channel dimensionality of the block's output layer (e.g., 256 for the first block), (2) a Local Spatial Pooling (LSP) strategy, where we divide the block's output layer into spatial bins, spatially pool within those bins locally, and then concatenate the outputs of these pooling operations to ensure that the input dimensionality to the projector is 2048, (3) a Conv-based Expansion (CbE) pooling strategy, which first expands the number of features to 2048 using the defined convolutional filter size (defaults to x convolution), and then performs global spatial pooling, Finally, we evaluate alternatives to the last variant, where we keep expansion layer, but instead of global spatial pooling, apply L2 or square-root pooling.
3.4 Training
We follow closely the original Barlow Twins training procedure (Zbontar et al. 2021 All our results are for ResNet-50 trained for 300 epochs, using the LARS optimizer with batch size of 2048. We use cosine learning rate decay with an initial learning rate of 0.2. The projector is three-layered MLP with 8192 hidden units and 8192 output units. In the supervised training case, we use the same training procedure as for the self-supervised case, except that we restrict the output layer of the projector to have 1000 output units, corresponding to the 1000 classes of ImageNet, and apply simple coos-ey loss to the output of the projector for classification. For consistency, we also use Barlow Twins' data augmentations in the supervised case. We provide the PyTorch pscudocode for blockwise training of models in Appendix B
4 Matching Backpropagation Performance with Blockwise Training
A blockwise training procedure with a self-supervised objective leads to accuracies comparable with an end-to-end trained network. We trained simultaneously the 4 blocks (see Section 3 for details) of a ResNet-50 on ImageNet, using the Barlow Twins loss function at the end of each block. Before being sent to the projector, the output each block is fed to an expansion-based pooling layer (see Section 3.3 for a description of these pooling layers). We also add noise to the feature maps each module with a standard deviation of 0.25. Remarkably, we find that despite each block being trained independently (aside from the forward propagation interactions), our model achieves competitive performance against an end-to-end trained model: we see gap of only 1.1% in performance as highlighted in Fig. 4 This is non-trivial result, as one usually expects a large drop in performance when interactions between blocks are restricted to
only forward dynamics on large-scale datasets (Illing et al. 2021 Xiong et al., 2020). Results on the smaller CIFAR-10 datasct are consistent with these results, as presented in Appendix D
Extending to other SSL methods: SimCLR / VicReg. We tested our blockwisc paradigm using alternative self-supervised learning rules i.e. VicReg and SimCLR. We visualize these results in Fig, 5 The results we obtain on both VicReg and SimCLR are consistent with our results on Barlow Twins: we only observe a slight loss in accuracy as we move from end-to-end trained networks to expansion- based pooling with simultancous blockwise training, indicating that our blockwise learning paradigm is robust to the specific choice of the self-supervised learning rule used.
6

---

Published in Transactions on Machine Learning Research (01/2024)
Training w/ GSP)
Simultaneous Blockwise Training (w/ 1x1 CbE GSP Noise)
[C] Simultaneous Blockwise Training (w/ LSP)
[D] Sequential Blockwise Training (w/ LSP)
Sequential Training (Supervised
Sequential Blockwise Training w/ Untrained Previous Blocks (w/ LSP)
60
50
70:
Accuracy (%)
71.57%
70. 48%
68 68.844 8..84%
67. 12%
65 23.64%
60-
Accuracy (%)
51. 11.79%
50
20
10
30
[A]
[B] [C]
[D]
Model Prefix (see legend)
2
3
Number blocks
Figure Overview of our key results. (left) Top-1 accuracy a linear probe trained on top of our pretrained network on ImageNet for different pretraining procedures. Our best blockwise-trained model [B] is almost on par with full backpropagation [A] (only 1.1% performance gap). (right) Accuracy ImageNet function of the depth of the network. Since our blockwise models are trained for each block in isolation, we can include a different number of blocks when computing the linear probe accuracy. We visualize the accuracy as we include more blocks into the network.
Robustness against natural image degradation (ImageNet-C). We evaluate the robustness of our best model (simultaneous blockwise training w/ 1x1 CbE + GSP + Noise) against natural image degradation using the ImageNet-C benchmark (Hendrycks & Dietterich 2019 The results are presented in Appendix C In summary, our results show that our blockwise training protocol degrades robustness comparison to the end-to-end trained model. This reduction in robustness can be in part. altributed to the lower top-1 accuracy of our blockwise trained model.
5 Understanding the Contribution of Different Components
We now analyze and discuss the contribution of the different components of our method to build in-depth understanding of its success. For simplicity, we avoid noise addition to the layers unless analyze its impact explicitly. Furthermore, we fix the block output pooling strategy to be local spatial pooling unless specified otherwise. We list some further ablations in Appendix B
Importance using a self-supervised learning rule as opposed to a supervised learning rule. We first compare our blockwise model trained with an SSL loss against a supervised blockwise training procedure, where we train each of these blocks using supervised learning. For fair comparison, we include the full projection head in the supervised case, but instead of applying the Barlow Twins loss, project to the 1000 classes of ImageNet and use the class labels for supervising the model at each of these intermediate blocks. We observe that the performance on the classification task of the initial blocks is significantly higher in the supervised learning case than in the SSL case, making the initial block very lask-specific, while the latter blocks see a reverse trend. This observation points to an advantage of SSL over supervised learning for successful blockwisc learning: whereas SSL rulcs preserve information about the input in initial blocks for further processing by later blocks, supervised learning objectives already discard information the initial blocks which could have been used by blocks stacked on top of them.
7

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training - ww/ GSP)
Simultaneous Blockwise Barlow Twins (w/ 1x1 CbE + GSP)
SimCLR (End-to-End Training w/ GSP)
itanous Blockwise SimCLR (w/ 1x1 CbE GSP)
VicReg (End-to-End Training - GSP)
Simultaneous Blockwise VicReg (w/ 1x1 CbE GSP)
70
60
Accuracy (%)
50-
40
30
20
1
3
4
Number of blocks
Figure 5: Impact of the SST. objective function used. Our approach (simultaneous blockwise training with conv-based expansion pooling of the block outputs) almost matches end-to-end training performance for all SSL methods lested.
Importance of simultancously training all the blocks. We compare the scheme where we simulta- neously optimize the blocks to a sequential optimization scheme. In the sequential optimization scheme, we train each of the blocks a sequential manner where we freeze the parameters of the previous block (including batch-nore norm statistics i.e. mean and standard deviation) before starting to train the next block. We see from Fig. f that sequential training results in modest drop in performance with the model accuracy dropping from 68.84% (simultaneous) to 67.12% (sequential F This decrease in performance tributed to two possible reasons: (1) during simultaneous training, the learning trajectory in latter blocks is affected by the learning trajectory co-occurring in carlier blocks and this results in an overall better lcarning trajectory for the model, or (2) the batch-norm statistics could be interacting in subtle ways during simul- tancous training which is beneficial. We tested the second hypothesis by adapting the batch-norm means and variances in early blocks during the training of all the subsequent blocks i.e. by keeping the batch-norm parameters always in training mode. This change makes a negligible impact on the end results, indicating that it is the optimization trajectory taken by these blocks together during training that is beneficial for downstream performance.
be at-
Importance of our custom global average pooling strategy. Fig. G highlights the results that we obtain when comparing the different pooling strategies presented Fig. 3 We see that conv-based ex- pansion w/ global spatial pooling yields the best final accuracy. Local spatial pooling and global spatial pooling strategies obtain similar performances at the final block, which is slightly inferior to the perfor- mance obtained by conv-based expansion w/ global spatial pooling, Conv-based expansion w/ L2-pooling significantly improves the performance of the initial blocks, but deteriorates the performance of the final blocks accordingly (this trend is similar to the supervised blockwise training case). This could be attributed to the fact that I.2-pooling normalizes the feature maps using the L2 norm, which overly emphasizes the highest activations. Square-root pooling attempts the opposite, where it tries to squeeze the large values while amplifying small values. However, the overall trend for Square-root pooling remains similar to the
Note that the training procedure for both sequential blockwise and simultaneous blockwise training equivalent when only the first. block trained.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training - w/ GSP)
Sinultaneous Blockwise Training (LSP)
Simultaneous Blockwise Training (GSP)
Simultaneous Blockwise Training (1x1 CbE + GSP)
Simultaneous Blockwise Training (1xl CbE L2)
Simultaneous Blockwise Training (1x1 CbE + Square-Root)
70
60
Accuracy (%)
50
40
30
1
4
Number of blacks
Figure 6: Comparison of different pooling strategies. Alternative pooling strategies show better performance on earlier blocks, but worse performance at the final block.
trend observed for 1.2-pooling despite being the opposite, with a significant degradation in performance at the final block in contrast to conv-based expansion w/ global spatial pooling.
Impact of filter size conv-based expansion pooling. Our conv-based expansion pooling strategy uses filter size of 1 x 1. We explore the impact of the size of the expansion pooling layer in Fig. 12 We show that increasing the filter size of the expansion pooling layer significantly harms the performance of the model. A possible explanation is that aggregating information over larger neighborhoods (i.e. higher effective receptive field) increases the level of supervision provided at the initial layers, harming the performance at the final blocks.
Comparison against a trivial baseline of untrained previous blocks. We next exclude the possibility that the overall accuracy of our blockwise model could be trivially matched by network of equivalent depth with untrained previous blocks (such random feature models have been used the past with some success when only training the final layers of the model (Huang el al. 20044). In order to test this, we compare the accuracy of our model against a model where the previous blocks are untrained, initialized randomly, and frozen. We only train the top block in this case while keeping the (untrained) blocks below fixed. We find that the gap in accuracy between our layer-wise model and the random feature model widens stack more blocks without training the initial blocks, as visualized in Fig. 4
Impact of the first block on performance. Correctly training the first. block can be supposed to be particularly critical, as all subsequent blocks depend on it. We indeed demonstrate the sensitivity of the model to the exact training procedure of this first block in Fig, 7 by training the first block in end-to-end setting, and then plugging it into the blockwise training protocol, we largely bridge the gap with end-to-end Barlow Twins (71.57% for the end-to-cnd variant vs. 70.07% for the variant where the first block is trained end-Lo-end, while the rest of the blocks are trained in a sequential manner vs. 67.12% for the baseline model trained sequentially). Moreover, if we combine the first two blocks (B1 and B2), and then train them using the same simultancous blockwise training protocol, this achieves performance on par with using an end-to- end trained first block. This highlights that the main challenge our blockwise training procedure resides in finding way to correctly train the first. block. We explore different ways to customize the learning process

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow (End-to End Training w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Combined Simultaneous Blockwise Training (B1+82 w/ LSP)
Sequential Blockwise Training (81 trained end-to-end - w/ LSP)
Accuracy (96)
60-
55-
2
Number of blacks
Figure 7: Evaluating the impact of (1) training the first block separately using an end-to-end training procedure, and of (2) merging the first and second blocks during training. Both modifications greatly reduce the performance gap with end-to-end training, indicating that finding a correct training procodure for the first block would be critical to completely match end-to-end training performance.
for the first block order to improve the performance of the whole network in Appendix However, none of these strategies produced the expected gains in performance.
Impact of noise addition. Self-supervision in the early blocks may make the blocks too task-specific, reducing overall performance. Therefore, we experiment with the use of noise injected in the activations during training. We pick particular and add Gaussian noise with zero mean and the chosen standard deviation to the activations of the model before the beginning of every block. Injecting independent Gaussian noise each of the feature values with = and a 0.25 gave boost performance of 0.5%. Using a 0.5 results lower gains due to high levels of noise (70.22% with 0.5 vs. 70.48% with = 0.25). We further experimented with different schemes for noise injection such as injecting the same noise to all the features at particular spatial location, and found independent noise for each of the features to work best. We present the results with noise injection in Fig. Without the addition of noise, our expansion pooling strategy achieves an accuracy of 70.15%.
6 Discussion & Conclusion
In this study, we showed that by adapting recent self-supervised learning rules to a blockwise training paradigm, we were able to produce a blockwise training procedure that comparable in performance to end-to-end backpropagation on large-scale dataset. Although there no practical advantage of our method at this point, we believe that our findings are promising from neuroscientifie standpoint, as they propose viable alternative to end-to-end backpropagation for learning at scale. Moreover, such blockwise training procedures combined with adapted neuromorphic hardware could eventually speed up learning and make it more cnergy-efficicnt by climinating the need for information transportation through the usual backpropa- gation path.
Limitations We are using blocks that do not interact through backpropagation path. However, each of these blocks from ResNet-50 (He al.) 2016)) is composed of a large number of layers well as two-layer projector network throughout which the backpropagation algorithm is still used. In order to completely
10

---

Published in Transactions on Machine Learning Research (01/2024)
get rid of backpropagation, would need to find learning rule functioning at a layerwise gramularity. Unfortunately, our experiments in Fig. 7 showing that merging the two first blocks into one increases the final accuracy of the network indicate that this gap in performance widens as we progressively subdivide the networks into more blocks. Consequently, our current approach would not be successful in a purely local learning paradigm.
Moreover, Barlow Twins' loss function computes a cross-correlation matrix across all features of given layer, which implicitly adds an interaction term between all the neurons of this particular layer. In purely local learning setup, interactions should be limited to local neighborhoods, Illing et al. (2021) proposed such local learning rule, but they have not demonstrated it to scale to large-scale datascts such as ImageNet.
We find that although there is no backpropagation of information from upper blocks to lower blocks, the feed-forward path still conveys some information that couples the training of the blocks and that this coupling is important, as we show that simultancous training is better than sequential training from 1.72% points on ImageNet. This means that we cannot yet use our approach training the blocks sequentially. In future work, one could try to emulate the training of previous blocks by adding specific noise to the previously frozen blocks in the ford-forward path to the block being trained, effectively emulating simultancous training.
reduce the memory footprint of training by
Future work could also be directed toward finding ways to adapt the augmentations" difficulty to the layer being trained. However, the parameter space to explore is large, and our attempts at sorting sample difficulty and routing samples with simple distortions to early blocks have not worked so far (see Appendix A for details). Masked Auto-Encoderr (MAF) are interesting recent class of self-supervised methods based on the Vision Transformer architecture (Dosovitskiy et al. 2020 where the model Iearns via simple masking of tokens/ patches (He et al. 2022). We postulate that our blockwise training framework could be particularly well-suited to such a masking scheme, given that one can adapt the level of difficulty for the augmentations fed to different blocks very naturally via the fraction of patches masked.
References
Nasir Ahmad, Marcel A. J. van Gerven, and Luca Ambrogioni. Gait-prop: A biologically plausible learning rule derived from backpropagation of error. NeurIPS, 2020.
Robert Baldock, Hartmut Maennel, and Bchnam Neyshabur. Deep Learning Through the Lens of Example Difficulty. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 10876 10889. Curran Associates, Inc., 2021. URL 2afb5/aac22cceeee144/444440000 Paper.
Adricn Bardes, Jean Ponce, and Yann LeCun. VICReg: Maraance-nvaariacceCCooraaaa Regulariza-
tion for Self-Supervised Learning, January 2022. URL http://araiv.org//////000.... Number: arXiv:2105.04906 arXiv:2105.04900 [cs].
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to Ima- geNet. In ICML, 2019.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL
http://eed.nneeeer.ree.oc//ppppp fttppperrrilleppppprrreee pdf
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised Learning of Visual Features by Contrasting Cluster Assignments, January 2021a. URL http://arxiv www./ogg//////eeeeoo/o//e..eeeee 098822 Number: arXiv:2006.0958 arXiv:2006.0988) [cs].
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers, May 2021b. URL http:://arxi: Number: arXiv:2104.1429. arXiv:2104.14294 [es].
org/abs/2104. 14294
11

---

Published in Transactions on Machine Learning Research (01/2024)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMT.R, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Supervised Learners, October 2020b. URL htllhttp::/// http://arxivy http:///aax..vvorgaaa//00000000 10029 Number: MEXiv:2006.1002 arXiv:2006.10022 [cs, stat].
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Con-
trastive Learning, March 2020c. URL http://arxiv.. org. www.oogg////..oooooooooooooooeos aataa/bb///...00/ 04297 Number: arXiv:2003.04997
urXiv:2003.0429) [cs).
David Clark, I. F Abbott, and Sueyeon Chung. Credit Assignment Through Broadcasting Global Error Vector. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 10053 10066. Curran Associates, Inc., 2021. URL https://procceedings. httprrrrp..ccpppppe///0//111111.
552281t222abbee74445555552222222 .dff
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fci-Fci. Imagenct: A largo-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1..0..111110 11929, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations, October 2021. URL http///xxvi.orgab/////aa/ 14548 Number: arXiv:2104.1444 14548 arXiv:2104.14548 [es].
Maxence Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. arXiv, 2022.
Manu S. Halvagal and Friedemann Zenke. The combination of Hebbian and predictive plasticity learns invariant object representations deep sensory networks, March 2022. URT https/ttss:///// http:::://wwwboooxxvv.. biorxiv. org/ content/10.1101/2002.
17.4771211 Pages: 1922.03.17.444712 Section: New Results.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pallern recognilion, pp. 770 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsuper- vised Visual Representation Learning, March 2020. URL http::/aarii..rrgggee httg//bb/1911.... 05722 Number:
arXiv:1911.05722 arXiv:1911.05722 [cs].
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Xr preprint arXiv:1908. Stt:1993.12900 12261, 2019.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXin preprint arXiv:2212. 13345, 2022.
Geoffrey E Hinton, Simon Osindero, and Yeo- Chyy Teh. A fast learning algorithm for deep belief nets. Neural computation, 127711555, 2006.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Guang-Bi Bin Huang, Qin-Yu Zhu, and Chee- Chee-Kheong Siew. Extreme learning machine: new learning scheme of feedforward neural networks. In 2004 TEEE International Joinl. Conference on Newral Networks (IEEE Cat. No.04CH37541), volume 2, pp. 985 990 vol.2, 2004. doi: (AL.10//LICN.2004......
Bernd Mlling, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Tocal plasticity rules can learn deep representations using self-supervised contrastive prodictions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, vol- wmne 34, pp. 30365 30379. Curran Associates, Inc., 2021. URL Paper. pdff
Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timolcon Moraitis. without feedback. arXiv preprint Xiiu:2209.11880 2022.
Hebbian deep learning
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. ECML
PKDD, 2015.
Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic leedback weights support error backpropagation for deep learning. Nature Communicationn,, 7(1):13276, Novem- doi: 10. 10.1038/ncome (038//ccomm11326. URL
ber 2016. ISSN 2041-1723.
ncomms 13276
Number: Publisher: Nature Publishing Group.
Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):333 336-346, June 2020. ISSN 1471-0048. doi: 10.
0t09/441553-020-027775 URL httpss//www www.wwwwww........t..te http://www..aatuue..coo/rrtccces htam///rtticceees111882000000000 Publisher: Nature Publishing Group.
Number:
Sindy Löwe, Peter O'Connor, and Bastiaan S Veeling, Putting an end to end-to-end: Gradient-isolated learning of representations. arXic preprint 1rXi::1905.1/986 2019.
Arild Nøkland. Direct feedback aligmment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. AISTATS, 2009.
Roger N. Shepard and Jacqueline Metzler. Mental rotation of tiree-simensional objects. Science, 171(3972): 701-703, 1971. ISSN 1095-9203. doi: 10.1126/scicncc..71. 701. Place: US Publisher: American Assn for the Advancement of Science.
Mufeng Tang, Yibo Yang, and Yali Amit. Biologically plausible training mechanisms for self-supervised learning in deep networks. Frontiers in Computational Neuroscience, 16, 2022.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale vidco prediction. In Proceedings of the IEFE/CW Conference Computer Vision and Pattern Recognition (CVPR), pp. 2318 2328, June 2021.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive represcntation learning. arXiv preprint trXiv:2008.013.1 2020.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-VIT: Adaptive Tokens for Efficient Vision Transformer, March 2022. URL UBLLatpp//aaxiiv.rr/////222212 11X60:2112.07658 arXiv:2112.07655 [cs).
Number:
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. Xrxiv preprint arXio:2103. 032330, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2024)
(a)
(b)
(c)
Blockwise (w/ LSP)
Simultaneous Blockwise
Simultaneous Blockwise
(w/ LSP)
(w/ CbE + GSP)
ON-diagonal wataaggnnallceeeeeiee conelation
ON-diaganal Ww.iaaginal.oongiiiii. conglition
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
Siesknbe End and
Sinu Bl Blcckbbe (57)
Semilaneaus Bicknbe Let CLe CSP End vnd
Erd end
0.0
0.0
0.0
Block Index
Block Indiex
Block Index
DFF-dagooaa corrlation
OFF-daggaral cardiation
orF-dlegonal correation
-0.2
-0.
blesknos End aTC
-0.4
Senu Lamoaas drocknse IN 39
-0.4
Sandiaan Blocknie let LNd, dUe USM
Crd end
3 Block Index
Block Index
Block Index
Figure 8: Comparison between ON-diagonal and OFF-diagonal terms of the cross-correlation matrix be-
tween features (without projection head or expansion layers and using global average pooling) in comparison with the end-to-end Barlow Twins model. Conv-based pooling improves the alignment on the OFF-diagonal terms, while only improving the alignment of the ON-diagonal terms at the last block.
A Customizing the Blockwise Training Process
We tried to customize the training procedure to our blockwise paradigm by progressively increasing the complexity of the task for each block. Although none of these schemes improved the accuracy of the final model, we hope that laying out these schemes and discussing them will aid future research by providing insights on directions worth exploring further.
A.1 Blockwise Adaptive Learning Strategies
Analyzing the limitations of the first block. We visualize the distribution of the on-diagonal and off-diagonal terms from the Barlow Twins loss in the form of a violin plot in Fig. M Since the end-to-end trained model has no projection head, we always plot these terms based on the features from the ResNet-50 backbone, without any projection head, and using global average pooling. We see that supervised learning provides steady increase in the correlation of the on-diagonal terms with an increasing number of blocks, while simultancously reducing the off-diagonal correlation. However, when looking at blockwisc trained models, we see that the initial block is much better al modeling the on-diagonal correlation, while is worse at modeling the on-diagonal correlations in the last block. This again relates back to our observation that providing strong supervision at the initial blocks can worsen the performance of the overall network. We further explore reasons for this mismatch and how they can be catered in the subsequent sections via block combinations well as tuning of the learning rate lambda values. It is apparent from Fig. that the first
14

---

Published in Transactions on Machine Learning Research (01/2024)
(a)
(b)
(c)
Blockwise (w/ LSP)
Simultaneous Blockwise (w/ CbE + GSP)
Simultaneous Blockwise (w/ CbE + GSP)
Fixed invariance target
Adjusted invariance target
1.0
1.0
0.8
0.0
D1-dingnna comelktiio
ON-ciagonal corglation
oN oigglagnall comelation
0.6
0.6
0.4
0.4
0.4
0.2
Sérr Kancus Stockwiic for 2nd dee CS7
Semu Runceus Blockwtsc twr Led Cle (CSP-117
Ind
0.0
0.0
Block Index
Block Indlex
Block Index
0.4
0.4
0.4
OFF orF-diagenal correlation
0.2
OFF- Pi-ggiagaaal corsiation
0.2
OFF- diagonall corretation
0.2
0.0
0.0
0.0
0.2
-0.2
-0.4
Sledknns And
-0.4
Konceus dtockwiie 9.099
-0.4
Kanceus Blockoiss (0CP
3
Block Index
Black Index
Figure 9: Comparison between ON-diagonal and OFF-diagonal terms of the cross-correlation matrix be- tween features (without projection head or expansion layers and using global average pooling) in comparison with the end-to-end Barlow Twins model and adjusted invariance target model. Conv-based pooling im- proves the aligument of the OFF-diagonal terms, while only improving the aligument of the ON-diagonal terms the last block. Variable invariance fails to improve the aligmment for the on-diagonal terms in comparison to conv-based pooling while making it worse for the off-diagonal terms. This indicates that the chosen invariance targets are not ideal for this setup, and hence, a different set of targets might be more suitable.
block of the blockwise trained model is significantly different than the end-to-end trained model. Therefore, sanity check, we combine the first two blocks of the network. This ensures that the first block is not strictly supervised, and is mainly guided by the performance of the second block. We visualize these results in Fig. 7 We find that this configuration helps in closing the gap between the supervised variant and our blick-wisc variant where the simultancous blockwise training achicves 68.84% while the combination of the first two blocks achieves 70. 14% top-1 linear probe accuracy. This provides evidence for our initial hypothesis that strong supervision at the initial blocks is significantly detrimental to the downstream performance.
Matching the Invariance Characteristics of End-to-End Model Looking at the difference in dis- tribution between end-to-end and blockwise trained models, it seems likely that minimizing this shift will improve the performance of the model. For this purpose, we evaluate matching the invariance term with that of the end-to-end trained model (redundancy reduction mismatch is already low). One important detail to note that the difference between the invariance and rodundancy reduction terms reported in Fig. g is computed based on the application of global spatial pooling to the features of the network without the projection head (as there is no projection head for the end-to-end trained model). Therefore, we use slightly
15

---

Published in Transactions on Machine Learning Research (01/2024)
70
Accuracy (%)
66
64
60
0.00
0.02
0.04
0.06
0.08
0.i0
0.12
0.14
Lambda
Comparison of different lambda values used train the model. The default value of À for Barlow Twins is À 0.0051. We see that changes in À values result in a negligible change in the model performance, except when the Ả value becomes significantly large, where it has significantly detrimental effect on performance.
Figure 10:
higher values for the target invariance, changing it from 1.0 for all the blocks to 0.6, 0.75, 0.9, and 1.0 for block-1, block-2, block-3, and block-4 respectively. However, we found that this fails to improve performance (results instead in an insignificant drop in performance from 70.15% for our model with CbE GSP 69.90% with the new invariance targets). We also visualize the feature difference (similar to Fig. 9 but for the model trained with adapted invariance largets). It shows that the chosen larget was higher than the one desired. Therefore, different schedule for targets can potentially outperform the current. method in practice.
Lambda tuning does not help This experiment was based on blockwise training. In order to reduce the computational burden, we trained the rest of the blocks end-to-end by only adjusting the lambda value for the first block. One recurring theme that we obscrved throughout our previous experiments was the inferior results when trying to closely supervise the initial layers of the model. We find that supervising these initial layers makes them task-specific, reducing their overall utility. In order to circumvent this issue, we tune the lambda value which provides trade-off between obtaining invariance to the applied augmentations minimizing correlations between different features. We assume that enforcing invariance at these initial layers can be premature, however, enforcing diversity in the extracted features by minimizing their correlation can still be beneficial. The Barlow Twins implementation used very small value of X = 0.0051. Therefore, we experimented with larger lambda values. These results are visualized in Fig. I0 We see a negligible impact of changing the lambda value. However, as the value of lambda gets significantly larger, this reduces the performance of the model significantly.
Adaptive augmentations Considering that tuning the lambda value made a negligible contribution to the model performance, we tuned the model augmentations such that casier augmentations are applied at the initial blocks and the difficulty of the augmentations increases gradually with higher blocks. This is beneficial initial blocks easily learn invariance against color jitters. However, learning invariance against large random crops be very difficult due to the limited receptive field. Since the dataset is changing, we experiment with this protocol in a strictly sequential setting (where we train the blocks sequentially instcad of training them sinullaneously). For this purpose, we defined a successive augmentation list, where we only applied color jitter at the first block. We additionally applied small random crops at the second block. For the third and fourth blocks, we applied the full set of Barlow Twins augmentations. We visualize the results from this experiment in Fig. I1 We see that adaptive augmentations result in significant drop performance. However, might be due to some missing components that unable to properly tune.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Adaptive Loss (Train all below w/ LSP)
Adaptive Loss (weight=0.5
LSP)
Adaptive Loss (weight=0.75 w/ LSP)
Augmentation Tuning (w/ LSP)
60
Accuracy (%)
50
40
30
Number af blocks
Figure Evaluation of performance with adaptive augmentations or loss functions. Our formulation of adaptive augmentations degrades performance in contrast to regular training where all the blocks are trained using identical augmentations. Similarly, our formulation of the adaptive loss function also degrades performance.
We consider this to be interesting and open problem for further rescarch and result in non-trivial gains.
Example difficultybasedd routing The previous setting with adaptive augmentations can introduce disparity due to differences in the patches seen during the training of the initial blocks vs. training of the later blocks. Therefore, we experimented with a more principled approach of routing the examples to the correct block based on their difficulty by leveraging their training dynamics i.c. routing examples with the smallest loss the initial blocks while routing examples with the highest loss to the highest blocks. However, it is still important to train all the other blocks aside from the primary block example is routed towards. For this reason, we tested multiple schemes. The first scheme is [TRAIN ALL BELOW] setting where example when routed to particular block, also trains all the blocks below. All the other settings where we specify the weight refers to the setting where we train all the other blocks (not only the blocks below), but with a reduced weight specified using the weight mentioned in the legend. We visualize the results with this routing mechanism in Fig. [] Similar to the case with adaptive augmentations, we found this routing reduce the performance of the model significantly. However, we also consider this to be a very interesting avenue to be explored further in future work.
B Further Ablations
We performed some trivial ablations by increasing the size the projector, changing the learning rate of the lincar classifier, or even employing group convolutions. However, all these changes had a negligible effect on the model performance.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training w/ LSP)
Simultaneous Blockwise Training (1xl CbE GSP)
Simultaneous Blockwise Training (7x7 CbE GSP)
60
Accuracy (%)
50
40
30
1
2
3
4
Number of blocks
Comparison of performance with changes in filter size for conv-based expansion. Our default setting for conv-based expansion followed by global average pooling uses a filter size of 1 x 1. Increasing the size of the filter to 7 improves performance at the initial blocks, but degrades performance towards the end.
Figure 12:
Increasing projector size does not help. In all the prior experiments, we fixed the projector size to 8192 in accordance with the original Barlow Twins implementation (Zbontar et al.) 2021). However, given that local learning is significantly different than end-to-end learning, experimented with large projection head. We visualize the impact of large projector size in Fig. 13 where double the hidden dimensionality of the projection head to 16384. It is interesting to note that increasing the size of the projector increases the performance of the initial blocks but degrades performance at the final block. These results are consistent with the observations made for end-to-end Barlow Twins, where they found a negligible impact of increasing the size of the projection head beyond the limit of 8192 (Zbontar et al. 2021
Tuning the learning rate for the linear classifier does not help. The default learning rate for the lincar classification head is set to be 0.3 in the original Barlow Twins evaluation (Zoontar et al. 2021). This was discovered after grid search over three different values of learning rate. Therefore, in order to identify if a different learning rate is more suitable in our case, we did a small search over the same learning rate settings. These results are summarized in Fig. 14 We found minimal impact changing the learning rate, with only deteriorating performance.
Using group convolutions degrades performance. Since the expansion-based pooling with 1 x 1 convolution was the most impressive in terms of results, we experimented with exlending this to group convolution in order to avoid performing this convolution operation over a large spatial grid which is a memory-intensee operation. We visualize these results in Fig. 13 Unfortunately, using group convolutions results in minor loss in performance in contrast to using the conventional convolution. However, this drop in performance negligible considering real-world applications.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training - w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Simultaneous Blockwise Training (Large Projector - w/ LSP)
Simultaneous Blockwise Training (1x1 CbE + GSP)
Simultaneous Blockwise Training (1x1 Group CЫE GSP)
70
60
Accuracy (%)
50
40
30
Number of blocks
Figure 13: Changing the size of the projection head or using group convolution has a negligible impact on performance.
Barlow Twins (End-to-En) End Training w/ GSP)
Simultaneous Blockwise Training (w/ LSP)
Simultaneous Blockwise Training (w/ LSP) [LR: 0.3]
Simultaneous Blockwise Training (w/ LSP) [LR: 1.0]
Combined Simultaneous Blockwise Training (M1+M2)
[LR: 0.1]
Combined Simultaneous Blockwise Training (M1+M2) (w/ LSP) [LR: 0.3]
Combined Simultaneous Blockwise Training (M1+M2) (w/ LSP) [LR: 1.0]
70
65
Accuracy (%)
60
55
50
45
Number of blocks
Figure 14: Comparison of different learning rate settings. The default learning rate for Barlow Twins is 0.3. Changes in learning rate have negligible impact on final model performance.
19

---

Published in Transactions on Machine Learning Research (01/2024)
100
End-to-end
Simultaenous Blockwise (1x1 CbE+GSP+Noise)
80
Error rate (%)
60
40
20
0
Train
Validation
Zoom Blur
Motion Blur
Glass Blur
Defocus Blur
Gaussian Noise
Impulse Noise
Shot Noise
Contrast
Elastic Transform
Jpeg Compression
Pixelate
Frost
Brightness
Fog
Snow
Spatter
Saturate
Gaussian Blur
Speckle Noise
Corruption type
Figure 15: Comparison between error rates of our best model (simultancous blockwise training w/ 1x1 CbE + GSP + Noise) and the end-to-end trained Barlow Twins model. The figure shows that the blockwise trained model exhibits lower robustness on ImageNet-C distortions as compared to the end-to-end trained model, with an average error rate of 65.68% vs. 62.81% of the end-to-end trained model.
C Robustness on ImageNet-C
In order to probe model's robustness to out-of-distribution images, mark (Hendrycks & Dietterich, 2019) which was constructed by injecting different kinds of synthetic noise into the ImageNet test set. Each noise type is further segmented into five different levels of noise, ranging from very mild levels of noise to extreme levels of noise. The noise types are selected to cover noise models found in the real world.
ImageNet-C bench-
We compared the robustness of our best model (simultancous blockwisc training w/ 1xl CbE + GSP + Noise) and the end-to-end trained Barlow Twins model ImageNet-C. We report the error rate well the standard deviation computed over all five corruption levels for each corruption type. The results are visualized in Fig. 15 The mean error rate (averaged over all corruption types) for the end-to-end trained model is 62.81%. while the mean error rate for our best model is 65.68%. This reduction in robustness can be partly altributed to the lower clean accuracy 77.55%% vs. 70.48%). However, these results indicate that blockwise training as operationalized in our case is not particularly useful for robustness against (natural) image degradations.
Generalization CIFAR-10 10
To showcase the generalizability of our main results to smaller dataset, we trained the ResNet-50 model CIFAR-10 from scratch. In order to reuse augmentation pipeline and model architecture which is
20

---

Published in Transactions on Machine Learning Research (01/2024)
Barlow Twins (End-to-End Training)
Simultaneous Blockwise Barlow Twins (Expansion Global Pool)
78
76
74
Accuracy (%)
72
70
68
66
3
Number blocks
Figure 16: Demonstration of the generalization of our main results to the smaller CIFAR-10 dataset. Although CIFAR-10 is relatively small for self-supervised learning to be effective, these results indicate that our main findings generalize to other datasots.
designed assuming 224 x 224 images, we upscale the images in CIFAR-10 to 224 x 224. The model is trained for 1000 epochs with a smaller batch size of 128 and a learning rate of 0.01, keeping all other settings intact.
Fig. 16 presents comparison between end-to-end training and our best method on ImageNet ie., expansion pooling- based sequential blockwise training with filter size of with no additional noise added the feature maps. The figure indicates that our results do generalize to other datasets, where the performance of our blockwise approach nearly matches the end-to-end performance. Note that the end-to-end SSL training falls short of the fully-supervised numbers reported on CIFAR-10 for ResNet-50 He et al. (2016) due to the smaller dataset size, use of larger image size as well as the use of Image also interesting to note that the performance of the model saturated after the third block, indicating the excess capacity the model for this smaller dataset.
angmentations. It is
E Torch Pseudocode for Blockwise Training
We present the PyTorch pseudocode for our blockwise training scheme Algorithm 1 training scheme uses PyTorch's 'detach' function (which detaches the tensor from the computational graph) and returns one loss value for cach of the 4 blocks of the network. We then backpropagate through cach of these 4 loss values individually by calling the "backward' function.
Our blockwise
21

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm 1: PyTTrch pscudocode for our blockwise training schome.
Class BlockRessNed ("rrggs **kwargs)
Function _.jinit__ssel)):
/* All ResNet initializations
Function forward Tensor):
/* first layer of the network
+ selimaxpoolsel..reetteeefe (seee/eeee.bl(/effffoo///)))))))
output_liss + [
xl < self.layer
output (ost.appeendtorccffllllllae (sel. conv block 1xx1), 1))
x2 self.layer2(x11
output_list.nn
(sclf poo.f.ooolllsl.o....oooooooooooo _onv__ooovlllee __x22_2(x22________ 1))
x3 + self.layer3 (xeacaach))))))
output _tp.tt_ssttppddlll_eeee (torocndddooohhhhaooee (torch.flatter cecittennsssf.... (ww.lffooolll...coooooooooooooos pool__oovy conv _ block_3(x3)) 1))
x4 + self. (x3.detach()
output_liste listtappend (torch. flatten (self, ploo_pooooooee cony block block_4(x4),, 1))
return output list
return
** Training code
loss salussalucsseee t mxdel(x)
for I in loss_values values do
l.backward()
// loss values, one for each block
end
22


Reviews:
[{"review_id": "2ytQ1IF8Wn", "reviewer": "Reviewer_84cX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The high-level justification for the method, from the perspective of optimization efficiency, makes sense, which motivates the work.\n- The related works discussed appear sufficient\n- The experimental protocol description appears fairly reproducible\n- The paper is well-structured and easy to read\n- The method figures are well-designed and helpful\n- The results figures are clear\n\n# Weaknesses\n- Abstract does not motivate why the proposed blockwise pretraining procedure is desirable. The only metrics mentioned are ImageNet accuracy, which is a performance degradation. Is the method faster? Cheaper? More amenable to hardware restrictions? By what metrics? I'm hoping to learn more by reading the main text, but a better abstract would be self-contained.\n- The experiments do not include a metric that measures the purported benefit(s) of the approach relative to end-to-end training (e.g. energy consumption, memory footprint, etc.). Thus, there is no evidence for why the proposed approach would be preferred to end-to-end training, which achieves superior accuracy. Including measurements of these statistics for each approach would improve the paper. \n- Page 2: \".... as long as the network is not broken into too many blocks\" where is this result presented? I couldn't find a figure or table where there was significant performance degradation after the addition of \"too many\" blocks. Ideally the paper would include a reference to the result in the contribution description and make the contribution wording more precise (e.g. \"after <X> many blocks\").\n- The paper organizes optimization approaches along a 'biological plausiblity' spectrum, but there is no self-contained justification for this ordering.\n- The experiments are limited to a single base model architecture (ResNet-50)\n- The experiments are limited to a single dataset\n- The experiments appear to be limited to a single seed per configuration, which does not attempt to isolate possible variance due to initialization and optimization randomness\n- Fig. 4 (right). It's not clear what the difference between the Sequential and Simultaneous training methods is when there is only 1 block (Fig. 4 (right), at x=1). Aren't the methods equivalent in this case? The Fig. and the main text should make this clear", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates different optimization strategies for ResNet-50-based architectures on ImageNet self-supervised objectives. The strategies investigated focus on 'local' optimization updates, for purported gains in optimization efficiency. Experiments investigate the classification accuracy as a function of the number of ResNet-blocks used, and vary the architecture and optimization. These experiments illustrate that 'simultaneous' blockwise training nearly matches full gradient-based optimization, and outperforms 'sequential' optimization, given certain restrictions on the block architecture (spatial and feature pooling strategies). The paper also presents negative results on other blockwise optimization strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the weaknesses would strengthen the work. In my opinion, the main weaknesses are the absence of evidence that highlights in which cases the proposed method would be preferred, as well as the fact that the results are limited to single dataset and architecture. Despite these weaknesses, the claims are accurately scoped.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5FyPTVRzLn", "reviewer": "Reviewer_8yqr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is well-written and easy to understand. \n- Section 5 is particularly important since it helps indicate the relative importance of each of the components of the framework here. For example, the authors assess the impact of noise, the first block, and the SSL training objective. It is particularly interesting to see the effect of a supervised loss vs and self-supervised one. It seems that doing SSL training helps also improve the representations learned by later blocks.\n- The paper addresses an important problem of seeking local learning alternatives to backprop.\n- The paper demonstrates that they are able to scale the approach to ImageNet models.\n\n## Weaknesses\n- **Biological Plausibility**: Figure 1 is interesting, but I am not sure how the authors decided whether an approach is biologically plausible or not. As far as I can tell, evidence for against the biological plausibility of backprop is still unclear, so I am not sure how the authors can get to Figure 1. It seems like the assumption here is that standard backprop is completely not biologically plausible.\n- **Design Decisions**: It is unclear to me how the authors picked to partition Resnet-50 into 4 blocks. What is the inspiration for that?\n- **Barlow Loss**: The formulation here, as presented, seems excessively tied to the Barlow twins loss. Eventhough their experiments suggest that other SSL losses can also be effective here. If that is the case, what is the point for tailoring this method to the Barlow loss?\n- **Is this approach really an alternative to backprop?**: I am not sure we can conclude here that the approach here is an alternative to backprop since you still run a local backprop within each block. The authors mention this in the later parts of the paper.\n- **Section 3.1**: I assume this section is meant to be the main meat of the proposed method, but it is too high level for me to really understand what is going on exactly. Can the authors make it clear what the blockwise training does there in detail? Specifically, it would be help to know: given a forward pass, what exactly happens at each block during the local backprop process. I assume, here you backprop within each module as usual (but stopping flow to any parameters outside of that block), but it is not clear if something else happens. \n- **Focus on ImageNet and ResNet-50**: The authors tailor most of their findings and analysis to ResNet-50 on image. I'd imagine switching to a different model would be tricky, but it seems like the authors can easily replicate this on a different dataset like celeb-a or another image dataset to test that their findings generalize beyond ImageNet. It will also be interesting for the authors to discuss how this approach can be translated to other architectures, specifically, how should I partition the model? How do you handle the shortcut connections between blocks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a strategy to optimize a Resnet-50 model using local learning rules (i.e. no backprop). The specific strategy here uses a block-wise approach and relies on barlow twins ssl approach. The barlow twins approach operates on the embeddings of a batch distorted in two ways to produce two embeddings for the same batch of data. A cross-correlation matrix is then computed for these two embeddings, and  the loss function has an invariance term, and a redundancy reduction term on the correlation matrix. The resnet-50 model is divided into 4 blocks, and a local version of backprop is applied and propagated only within each block. The primary dataset considered here is ImageNet. The paper shows that this approach scales to imagenet, and remains well performing. Extensive ablation experiments also demonstrate the relative importance of each component of the pipeline here.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses listed above. Of those weaknesses, here are the ones I would say they authors should address:\n - The discussion in Section 3.1 needs to be expanded,\n-  Explain design decisions,\n-  Explain biological plausibility, and \n- If possible, test a smaller dataset to show that this method extends to that setting, and explain how the local learning approach here can be applied to say a densenet.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ko1puCYSxI", "reviewer": "Reviewer_iqhT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a novel approach to training deep networks, offering an alternative to the traditional backpropagation method.\n\n2. The results achieved, especially the close performance to end-to-end pretrained networks, are commendable and highlight the potential of the blockwise self-supervised learning approach.\n\n3. The study provides a comprehensive understanding of the impact of different components within the method, contributing to the broader knowledge of local learning rules.\n\nWeaknesses/Questions:\n\n1. How scalable is this approach to even larger networks or more complex architectures? Would the results be consistent across different architectures or datasets beyond ResNet-50 and ImageNet?\n\n2. The biological plausibility of the method is still a topic of debate, as the method relies on backpropagation within blocks and interactions between neurons due to the specific form of the loss function.\n\n3. How does the blockwise approach impact the computational efficiency and training time compared to traditional methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper titled delves into the exploration of alternatives to the conventional backpropagation method, focusing on blockwise learning rules that harness the advancements in self-supervised learning. The authors introduce a blockwise pretraining procedure, where they independently train the four primary blocks of layers of a ResNet-50 using the Barlow Twins' loss function for each block. The results are impressive: a linear probe trained on this blockwise pretrained model achieves a top-1 classification accuracy of 70.48% on ImageNet, which is only 1.1% below the accuracy of a network pretrained end-to-end (71.57% accuracy). This approach is distinct from other methods as it restricts the backpropagation path to individual blocks, and the study demonstrates its effectiveness on large-scale datasets like ImageNet.\n\nThe paper also highlights the significance of simultaneous training of blocks, as opposed to sequential training. This is crucial for achieving optimal performance, suggesting that vital learning interactions occur between blocks via the forward propagation path during training. The research indicates that methods enhancing the downstream performance of lower blocks on classification tasks might decrease the overall network's performance. This suggests that building invariant representations too early in the initial blocks could be detrimental to learning higher-level features in subsequent blocks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "/", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers were convinced that the experiments support the claims made in the paper: the approach can train vision models in a blockwise fashion without too large of an accuracy degradation. However, reviewers also shared concern about whether this demonstration is significant. Some of the reviewers hoped to see a well defined metric that would benefit from blockwise training, whether in terms of compute efficiency, energy efficiency, or the compute footprint. 

Given the TMLR guidelines, I do believe that some individuals in TMLR's audience, especially those working on "biologically plausible" learning or those who are working on alternatives to backprop, will find results in this paper interesting. The evidence is clear that authors can get within ~1% of baseline ResNet-50 trained on ImageNet without using full backprop.
Decision: Accept (poster)