Scaling Supervised Local Learning with Augmented Auxiliary Networks
Authors: Chenxiang Ma
Jibin Wu
Chenyang Si
KC Tan
Venue: ICLR 2024 poster (2024)
Keywords: Local Learning
Biologically Motivated Learning
Deep Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose an augmented local learning method that can scale up to tens of local layers with comparable accuracy to end-to-end training and significantly reduced GPU memory.

Abstract: Deep neural networks are typically trained using global error signals that backpropagate (BP) end-to-end, which is not only biologically implausible but also suffers from the update locking problem and requires huge memory consumption. Local learning, which updates each layer independently with a gradient-isolated auxiliary network, offers a promising alternative to address the above problems. However, existing local learning methods are confronted with a large accuracy gap with the BP counterpart, particularly for large-scale networks. This is due to the weak coupling between local layers and their subsequent network layers, as there is no gradient communication across layers. To tackle this issue, we put forward an augmented local learning method, dubbed AugLocal. AugLocal constructs each hidden layer’s auxiliary network by uniformly selecting a small subset of layers from its subsequent network layers to enhance their synergy. We also propose to linearly reduce the depth of auxiliary networks as the hidden layer goes deeper, ensuring sufficient network capacity while reducing the computational cost of auxiliary networks. Our extensive experiments on four image classification datasets (i.e., CIFAR-10, SVHN, STL-10, and ImageNet) demonstrate that AugLocal can effectively scale up to tens of local layers with a comparable accuracy to BP-trained networks while reducing GPU memory usage by around 40%. The proposed AugLocal method, therefore, opens up a myriad of opportunities for training high-performance deep neural networks on resource-constrained platforms. Code is available at \url{https://github.com/ChenxiangMA/AugLocal}.

Published as a conference paper at ICLR 2024
SCALING SUPERVISED LOCAL LEARNING WITH AUGMENTED AUXILIARY NETWORKS
Chenxiang Ma', Jibin Wu': Chenyang SI, Kay Chen Tan' 'Te Hong Kong Polytechnic University, Hong Kong SAR, China Nanyang Technological University, Singapore
ABSTRACT
Deep neural networks are typically trained using global error signals that back- propagate (BP) end-to-end, which is not only biologically implausible but also suffers from the update locking problem and requires huge memory consumption. Local learning, which updates each layer independently with gradient-isolated auxiliary network, offers a promising alternative to address the above problems. However, existing local learning methods are confronted with large accuracy gap with the BP counterpart, particularly for large-scale networks. This is due the weak coupling between local layers and their subsequent network layers, as there is no gradient communication across layers. To tackle this issue, we put forward an augmented local learning method, dubbed AugLocal. AugLocal constructs each hidden layer's s auxiliary network by uniformly selecting a small subset of layers from its subsequent network layers enhance their synergy. We also propose to linearly reduce the depth of auxiliary networks as the hidden layer goes deeper, ensuring sufficient network capacity while reducing the computational cost of auxiliary networks. Our extensive experiments on four image classification datasets (i.e.. CIFAR-10, 10, SVHN, STL-10, and ImageNet) demonstrate that Au- gLocal can effectively scale up to tens of local layers with comparable accuracy to BP-trained networks while reducing GPU memory usage by around 40%. The proposed Auglocal method, therefore, opens up myriad of opportunities for training high- nigh-performance deep neural networks on resource platforms. Code is available at https:: hittps://githute http://ac....eeeaaaaaaa/aa/agoga
INTRODUCTION
Artificial neural networks (ANNs) have achieved remarkable performance in pattern recognition tasks by increasing their depth (Krizhessky al., 2012 LeCun et al., 2013 He et al. 2016 Huang et al. 2017). However, these deep ANNs are trained end-to-end with the backpropagation a algorithm (BP) (Rumelhart et al. 1985), which faces several limitations. One critical criticism of BP is its biological implausibility (Crick) 1989. Lillicrap et al. 2020), as it relies a global objective optimized by backpropagating error signals across layers. This stands in contrast to biological neural networks that predominantly learn based on local signals (Hebb) 1949 Caporale & Dan 2008 Bengio et al., 2015). Moreover, layer-by-layer error backpropagation introduces the update locking problem (Jaeeberg et al. (017). where hidden layer parameters cannot be updated until both forward and backward computations are completed, hindering efficient parallelization of the training process.
Local learning (Bengio al. 2006 Mostafa et al. 2018 Belilovsky et al. 2019 Nøkland &
Bidnes 2019 [ling et al. 2022 Wang et al.) 2021) has emerged as a promising alternative for training deep neural networks. Unlike BP, local learning rules train each layer independently using a gradient-isolated auxiliary network with local objectives, avoiding backpropagating eиror signals from the output layer. Consequently, local learning can alleviate the update locking problem as each layer updates its parameters independently. allowing for efficient parallelization of the training process (Belilovsky et al.) [020 Laskin al. [2200] Gomez et al., (022).. Additionally, local learning does not require storing intermediate network states, as required by BP, leading to much lower
memory consumption (Löwe et al. 2019; Wang et al. 2021 Wu et al. 2021)
"Corresponding author; Stthi.ww@@ollu..ccm

---

Published as a conference paper at ICLR 2024
However, local learning suffers from lower accuracy than BP, especially when applied to large-scale networks with numerous independently optimized layers (Belilovsky et al. 2020; Wang et 2021] Siddiqui et al. 2023). This is primarily because hidden layers only learn representations that suit their local objectives rather than benefiting the subsequent layers, since there is a lack of feedback interaction between layers. Previous efforts in supervised
100
holbin
DGL
Inteiro
Auglocalouun)
That Acc.
80-
local learning have focused on designing local losses to provide better local guidance (Nøkland & Eidnes 2019), such as adding reconstruction loss to preserve input information in InfoPro (Wang) jet al. 2021). However, these methods still fall short BP in terms of accuracy, especially for very deep networks (see Figure D
In this paper, we address the scalability issue of supervised lo- cal learning methods by strengthening the synergy between local layers and their subsequent layers. To this end, we propose an augmented local learning rule, namely AugLocal, which builds each local layer's auxiliary network using a uniformly sampled small subset of its subsequent layers. To reduce the additional computational cost of auxiliary networks, we further propose a pyramidal structure that linearly decreases the depth of auxiliary networks as the local layer ap- proaches the output. This is motivated by the fact that top layers have fewer subsequent layers than bottom ones. The proposed method has been extensively evaluated on image classification datasets (i.e., CIFAR-10 (Krizhevsky et al. 2009), SVHN (Netzer et al. 2011). STL-10 (Coates et al. 2011]. and ImageNet (Deng et al. (2009), using varying depths of commonly used network architectures. Our key contributions are threefold:
Figure 1: Comparison of super- vised local learning rules and BP on CIFAR-10 dataset. ResNet- 32 architecture, with 16 local layers, has been used in this ex- periment.
We propose the first principled rule for constructing auxiliary networks that promote the synergy among local layers and their subsequent layers during supervised local learning. Our method, AugLocal, provides the first scalable solution for large-scale networks with up to 55 independently trained layers.
We thoroughly validate the effectiveness of AugLocal on a number of benchmarks, where significantly outperforms existing supervised local learning rules and achieves comparable accuracies BP while reducing GPU memory usage by around 40%. Our ablation studies further verify that the pyramidal structure can reduce approximately 45% of auxiliary networks' FLOPs while retaining similar levels of accuracy. Furthermore, we demonstrate the generalizability of our approach across various convolutional architectures
We provide an in-depth analysis of the hidden representations learned by local learning and BP methods. Our results reveal that AugLocal can effectively learn hidden representations similar to BP that approach the linear separability of BP, providing compelling explanation on the efficacy of the proposed method.
2 BACKGROUND: SUPERVISED LOCAL LEARNING
Given a deep neural network Fe : X + y, the forward propagation of an input sample E X' can be represented by
Fe(x) = go f" pl-l o (x)
(1)
where o stands for function composition, and g(-) represents the final classifier. the number of layers excluding the classifier. (-) denotes the operational function at layer l, 1 < l < L. Parameters of all trainable layers denoted by O !
Instead of backpropagating global error signals like BP, supervised local learning employs layer-wise loss functions update the parameters of each hidden layer (see Figure 2)) Specifically, for any hidden layer f, an auxiliary network G6 H - y maps the hidden representation h E H to the output space. The computation of the layer E can be represented by h' = f (sg(h&-1) where sg stands for the goo-graaient operator that prevents gradients from being backpropagated between adjacent layers. During training, local loss function L LgG (ht), y) is computed based the
2

---

auxiliary network output G (h") and the label y. and the parameters of both the layer i and its auxiliary network are updated solely based on the gradients derived from the local loss. Note that the Published as a conference paper at ICLR 2024
lassifier
C Classifier
Classifier
t
Layer8
Layer8
Layer8
Laver?
Layer?
Auxiliary
Layer?
Network
* Auxiliary
Layer3
Layer3
Nensork
Layer3
1 v
Layer2
Layer2
Auxilisry
Layer2
Layerl
Layerl
Auxiliaay
Laverl
Natwork
End-to-end
(b) Supervised Local Learning
(c) AugLocal
Backpropagation
Figure 2: Comparison of (a) end-to-end backpropagation (BP), (b) supervised local learning, and (c) our proposed AugLocal method. Unlike BP, supervised local learning trains each hidden layer with gradient-isolated auxiliary network. AugL.cal constructs the auxiliary networks by uniformly selecting given number of layers from each hidden layer's subsequent layers. Additionally, the depth of auxiliary networks linearly decreases as the hidden layer approaches the final classifier. Black and red arrows represent forward and gradient propagation during training.
last hidden layer I. of the primary network (i.e., the network we are interested in) does not require any auxiliary network as it is connected directly to the output classifier. The parameters of layer and the classifier updated jointly according to the global loss L(F(x), y). It is worth mentioning that the auxiliary networks are discarded during inference, and the primary network operates exactly the same as BP-trained ones.
Supervised local loss Supervised local learning rules typically adopt the same local loss function as the output layer, such as the cross-entropy (CE) loss used for classification tasks (Belilovsky et al. 2019: 2020). Recent works have proposed incorporating additional terms into the local objective function. For instance, PredSim (Nøkland & Eidnes 2019) proposes to minimize an additional similarity matching loss to maximize inter-class distance and minimize intra-class distance between pairs. InfoPro (Wang et al.] (202)) shows that earlier layers may discard task-relevant information after local learning, which adversely affects the network's final performance. Therefore, they introduce reconstruction loss to each hidden layer, coupled with the CE loss, to enforce hidden layers to preserve useful input information.
Auxiliary networks Existing supervised local learning works use handcrafted auxiliary networks. For example, DGL (Belilovsky al., 2020) utilizes MLP-SR-aux, which incorporates spatial pooling and three 1 xx 1 convolutional layers, followed by average pooling (AP) and a 3-layer fully-connected (FC) network. PredSim (Nøkland & Bidnes 2019) uses two separate auxiliary networks FC layer with AP for the CE loss and a convolutional layer for similarity matching loss. Similarly, InfoPro (Wang et al. 202T) adopts two auxiliary networks comprising single convolutional layer followed by two FC layers for the CE loss and two convolutional layers with up- sampling operations for reconstruction loss. Despite their promising results under relaxed local learning settings, there is a lack of principled guidelines for designing auxiliary networks that can promote synergy between the local layers and their subsequent layers, as well as can generalize across different networks.
3 SUPERVISED LOCAL LEARNING WITH AUGMENTED AUXILIARY NETWORKS
3.1 PROBLEM FORMULATION
In supervised local learning, each layer is updated independently with its own local loss function, which can alleviate the update locking problem (Jaderberg et al., 2017 Belilovsky et al., (020). How- ever, this problem will become exacerbated when the depth of auxiliary networks grows. Therefore,
3

---

Published as a conference paper at ICLR 2024
impose maximum depth d (i.e., the number of trainable layers) auxiliary networks to control the degree of update locking. Note that a smaller value of d can enhance training efficiency, as the theoretical training time ratio between AugLocal and BP is approximately L+I (refer to Appendix(A.1 for detailed analysis). Additionally, we impose constraint on the computational cost of auxilarry networks to ensure efficient training. With the target of jointly minimizing the output and local losses, the supervised local learning can be formulated as:
min B.
((F(a))))
Σ
( (sg(h""))))
t=1
C I d, 8= 1=1.... L-1, -ー- FLOPs(G (h')) S n.
s.t.
(2)
where M ( ..... $ $1-1] is the set of parameters for all auxiliary networks, 1 denotes the number of trainable layers of the 4t auxiliary net. d and y represent the given maximum depth limit and maximum floating point operations (FLOPs) for auxiliary nets, respectively.
3.2 AUGMENTED AUXILIARY NETWORKS
Following supervised local learning, each hidden layer is updated based on its gradient-isolated auxiliary network. without any feedback about the global loss from top layers. As result, the optimization of bottom layers tends to prioritize short-term gains which, however, may overlook some essential features that can lead to better performance for subsequent layers. This has been seen in early works (Wang et al. 2021) and our feature representation analysis presented in Section 443
To mitigate the problem of sight-sightedness and improve the performance of final outputs, it is desirable to learn similar hidden representations as those developed by BP. To this end, we introduce structure prior to the local auxiliary networks. In particular, we propose constructing the auxiliary network of a hidden layer by uniformly selecting a given small number of layers from its subsequent layers, as illustrated in Figure[[c). Our intuition is that by leveraging the structures of subsequent layers from the primary network, we can capture the essential transformations of the subsequent layers, which could guide the hidden layer to generate a more meaningful feature representation that resembles the BP- one, and ultimately benefit the final output.
Specifically, for hidden layer f, we select layers from the primary network with layer indices d depth of the 8th auxiliary network. Then, construct the auxiliary network using these selected layers, which can be denoted as g g e f o f o ....o f (h²). Here, we use the same classifier g()) as used in the primary network, which typically consists of a global average pooling layer followed by fully-connected layer. Note that the selection of the auxiliary network takes place prior to training, and only the architectural structures of the selected layers are employed in the construction of the auxiliary network. When the dimensions of two adjacent layers in the auxiliary network do not match, we modify the input channel number of the later layer to match the output channel number of the preceding one. Furthermore, we perform downsampling when the channel number of the later layer is doubled or increased even further.
{ = e 14--",1, 1,...,, - 1, where [-] is the nearest integer function. d' denotes the given
3.3 PYRAMIDAT. DEPTH FOR AUXILIARY NETWORKS
Since each hidden layer is coupled with an auxiliary network, which will introduce substantial computational costs during the training process. To alleviate this challenge, we propose a method for reducing the computational burden associated with auxiliary networks by shrinking the depth of auxiliary networks progressively. We refer to this approach as pyramidal depth for auxiliary networks, which linearly reduces the depth of auxiliary networks as the hidden layer index increases. Our approach is motivated by the fact that top layers in the primary network have fewer subsequent layers than the lower ones. Specifically, we obtain the depth d' of the gth auxiliary network as per:
= [
d
dmin -
1L-6+)), t= 1,
This ensures that d' starts at a predefined maximum depth and asymptotically approaches a minimum depth dmin with the layer l going deeper. The decay rate is determined by the factor r.
4

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of supervised local learning methods and BP on image classification datasets. The averaged test accuracies and standard deviations are reported from three independent trials. L denotes the number of independently trained layers (residual blocks in ResNets). represents the depth of auxiliary networks. For example, d = 2 indicates that each auxiliary network contains one hidden layer in addition to the linear classifier.
Network
Method
CIFAR-I0 10
SVHN
STL-10
BP
93.73.10.04
97.01
80.80.10.17
PredSim Nøkland & Eidnes 2019
79.29=#0.37
92.02.+0.35
70.67+0..99
InfoPro (Wang et al.
(02T)
87.26=0.16
93.30-10.73
70.85+0.144
DGL
Belilovsky et al. 2020
86.3011.112
95.14:10.09
73.13::1008
ResNet-32 (L = 16)
AugLocal (d 2)
91.12:10.24
95.8611.004
78.58:10.66
Augl.ocal (d = 3)
92.26++0.20
96.43++0.02
79.79+00.27
AugLoccl (d
93.08-+0.10
96.79-10.08
80.65
Auglocal (d 5)
93.38:10.11
96.87110.33
80.73:10.18
AugLocal (d
93.47=00.09
96.85=00.08
80.95++0.55
BP
94.6110.118
97.10.10.05
80.4111.74
PredSim Nøkland & Eidnes
2019
74.95++0.36
89.90-00.76
68.9144048
InfoPro (Wang et al.
202T)
86.95+0.46
92.26++0.59
70.61+0.50
DGL
Belilovsky et al. 2020
85.6910.32
95.12:10.06
72.27:0.51
ResNet-110 (L = 55)
AugLocal (d 2)
90.98=0.05
95.92-10.10
78.29:10.37
Augloccl (d = 3)
92.62.40.22
96.45-1..08
79.30+10.26
AugLocal (d 1)
93.22-10.17
96.74.10.11
80.77
AugLocal (d = 5)
93.75+00.00
96.85+0..55
80.20
AugLocal (d
93.96=00.15
96.96+00011
80.40+0.17
Table 2: Comparison of different learning methods on VGG19 with 16 independently trained layers.
BP
PredSim
DGL.
Augl Local 2) Augl.ocal (d AugL.ocal (d 4)
CIFAR- 0316 86.4910.30 92.422242 0.00
93.16:10.28
93.46=0.19
93.69.10.05
SVHN 96.79=0.08 94.07 96.63
96.4810.07
96.65=0.04
96.64=00.06
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We evaluate the performance of our method on four datasets: CIFAR-10 (Krizhevsky al. 2009)
SVHN (Netzer et al.) 2011). STL-10 (Coates et al. 2011). and ImageNet (Deng et al. 2009).
Experiments are primarily based ResNets (He al. 2016) and VGGs (Simonyan & Zisserman
2014) with varying depths. We compare AugLocal with BP and three state-of-the-art supervised
local learning methods, including DGL (Belilovsky et al. 2020), PredSim (Nokland Eidnes 2019). and InfoPro (Wang et al. (02T). For a fair comparison, these bascline methods are re-implemented in PyTorch using consistent training settings (see details Appendix A.2). We follow the original configurations as stated in their respective papers for the auxiliary networks of DGL, PredSim, and InfoPro. The structures and computational costs these auxiliary networks are detailed in Appendix A.3 In addition, local learning rules evaluated with networks being divided into independently trained local layers, which is the minimal indivisible unit. For example, local layers residual blocks in ResNets and convolutional layers in VGGs. We adopt the simultaneous training scheme for local learning rules, which sequentially triggers the training process of each local layer with every mini-batch of training data (Belilovsky et al. 2020, Nøkland & Hidnes 2019 Wang et al. 202T). For Augl we begin with the maximum depth limit of d = 2, whereby each auxiliary network containing one hidden layer in addition to the linear classifier. Considering the varying availability of computational resources in different scenarios, we progressively increase the depth limit in our experiments, reporting the results for each value. We adopt the cross-entes crosspyntropy loss as the local loss function of AugLocal and the decay rate r = 0.5 in experiments.
4.2 RESULTS ON IMAGE CLASSIFICATION DATASETS
Results on various image classification benchmarks We begin our evaluation by comparing AugLoocal against baseline methods CIFAR-10, SVHN and STL-10 datasets. Here, we ResNet-
5

---

Published as a conference paper at ICLR 2024
Table 3: Results the validation set of ImageNet.
Table 4: Comparison of GPU memory usage, measured by those best-perforning models on each network and dataset with batch size of 1024. 'GC' refers to gradient
checkpointing (Chen et al. (016).
Network
Method
Top-1 Acc. Top-5 Acc.
Dataset
Network
Method GPU Memory (GB)
BP
71.59
90.37
VGG13
ResNet-32
BP
3.15
DGL
67.32
87.81
(L = 10)
(L= = 16) AugLocal
1.67 (L 47.0 %)
AugLocal
70.92
90.13
CIFAR-10
BP
9.27
34
BP 74.28
91.76
ResNet-110
GC
3.03 (4 67.3%)
(L 55) AugLocal
(L 17)
AugLocal
73.95
91.70
1.72 () 81.5%)
ResNet-34 BP
42.95
ResNet-101
BP
77.34
93.71
ImageNet (L 17) AugLocal
29.04 (L 32.4%)
(L= =4) AugLocal
76.70
93.29
ResNet-101
BP
157.12
(L = 34) AugLocal 97.65 (L 37.9 %)
32 and ResNet- 10 that have 16 and 55 local layers, respectively. As shown in Table ! all three supervised local learning rules exhibit significantly lower accuracy compared to BP, and this accuracy gap increases the number of layers increases. In contrast, our AugLocal outperforms all local learning methods with only a single layer and a linear classifier in its auxiliary networks (d = 2). This observation indicates auxiliary network design strategy is more effective than those handcrafted designs as deeper auxiliary networks are used in DGL (Belilovsky et al.) 2020) and InfoPro (Wang et al. 2021). As expected, the accuracy of AugLocal improves further with deeper auxiliary networks. This is because the local learning can better approximate the global learning when more layers of the primary network are selected into the construction of auxiliary networks. Notably, AugLocal achieves comparable accuracies to BP using auxiliary networks with no more than six layers on both network architectures.
Moreover, we also conduct experiments with VGG19 architecture, which consists of 16 independently trained layers. As shown in Table 2 AugLocal consistently outperforms local learning baselines and achieves comparable or better accuracy to BP. This reaffirms the effectiveness of AugLocal in handling deeper networks and ability to approximate the learning achieved by BP.
We further evaluate the generalization ability of AugLocal on three popular convolutional networks, including MobileNet (Sandler et al.) 2018), EfficientNet (Tan Le 2019), and RegNet (Radosavovic et al., 2020). These architectures differ significantly from VGGs and ResNets, both in terms of their overall structures and the types of building blocks. Our results in Appendix A.4 demonstrate that AugLocal consistently obtains comparable accuracy to BP, regardless of the network structure, highlighting the potential of AugLocal to generalize across different network architectures.
Results on ImageNet We validate the effectiveness of AugLocal on ImageNet (Deng et al. 2009) using three networks with varying depths. Specifically, adopt VGG13 (Simonyan & Zisserman 2014), ResNet-34 (He et al., 22166), and ResNet-101 (He et al. 2016), which contain 10, 17, and 34 local layers. It is worth noting that the majority of existing supervised local learning rules have not been extensively evaluated on ImageNet using the challenging layer-wise local learning setting.
As shown in Table our AugLocal achieves comparable top-1 accuracy BP on VGG13 (d 3), which significantly outperforms DGL (Belilovsky et al., 2020) by 3.6%. In the case of deeper architectures, such as ResNet-34 (d 5) and 10s (d 4), AugLocal consistently approaches the same level of accuracies as that of BP. These results demonstrate the effectiveness of AugLoocal in scaling up to large networks with tens of independently trained layers on the challenging ImageNet.
4.3 REPRESENTATION SIMILARITY ANALYSIS
We have shown that AugLocal can achieve comparable accuracy to BP across different networks. However, critical question remains: how does AugLocal achieve such high performance? Does AugLocal develop identical hidden representations as BP or different ones? We provide an in-depth analysis by studying representation similarity and linear separability between AogLocal and BP.

---

Published as a conference paper at ICLR 2024
Representation Similarity ResNet-32 32 on CIFAR-10
Representation Similarity of ResNet-110 1ateeett-110oo CIFAR-10
1.0
0.9
0.9
CKA Similerity
Similaitty
0.8
0.8
0.7
0.7
0.6
SPLd sedt. Sapcio
oraij 5.26
0.5
(Pge Aes
ocakd-iyy 0.67
Pressin, Arg. tas
Augiocalizedd di 2327
Prods Prg CSK 2.50
Auglocandmii O5A: 0.79
retees Aug cel 5.76
Aug ocalts 63, Cil
ints Cwp. 0.314
PAd crand 41 OKt 0.815
osL. Aeg 2.71
0.4
DCL day CoR 0882
2
10
12
14
16
5
10
20
25
30
35
40
45
50
55
Layer
Layer
Figure 3: Comparison of mnyer-wise representation similarity. We utilize centered kernel align-
ment (CKA) (Kormblith et al. 2019) to measure the layer-wise similarity representations between BP and other local learning rules. To provide fair baseline for BP, we measure the similarity between two networks trained with different random seeds. The average CKA similarity scores for different learning rules are provided in the legend.
Linear Separability of ResNet32 on CIFAR-10
Linear Separability of ResNett110 110 on CIFAR-10
90
80
70
Test 70-
60
Augioraldd
50-
Aud 1d
Frubim
Asglocatdee)
Prediim Intiio
(e-))
inforo haglocaldd
DGL
40
DCL
14
10
15
25 Layer Index
30
45
50
Layer
Figure Comparison of wise linear separability across different learning rules.
Representation similarity We use centered kernel alignment (CKA) (Kornblith et al. 2019) to quantitatively analyze the representation similarity between BP and local learning rules. Specifically, we calculate the CKA similarity of each local layer, and then we compute an average CKA score based all local layers.
Figure 3 presents the results of ResNet-32 (He et al. 2016) and ResNet-110 (He et al.) 2016). We ob- serve monotonic increase in similarity scores the depth of AugLocal auxiliary networks increases. This suggests that hidden layers trained with AugLocal generate representations that progressively resemble those generated by BP the depth of auxiliary networks increases. This phenomenon provides compelling explanation for AugLocal's 's ability to achieve comparable accuracies to BP. We notice that earlier layers achieve lower similarity scores. This expected because the features of earlier layers are less specific and highly redundant as compared to top layers. However, despite the lower similarity scores, the earlier layers still play significant role in providing meaningful features that contribute to the overall performance of the network. This is evident from the high representation similarity observed between middle and final output layers.
Linear probing We further employ the linear probing technique compare the hidden layer linear separability of different learning methods. Specifically, we freeze the pre-trained parameters of the primary network and train additional linear classifiers attached to each hidden layer. As shown in Figure our results reveal that most local learning baselines, when compared to BP, achieve higher accuracies earlier layers perform significantly worse at middle and output layers. This indicates that these local learning rules tend to be short-sighted, with early features predominantly optimized for their local objectives rather than supporting the subsequent layers (Wang et al., (202I). In contrast, AugLocal exhibits poorer linear separability in the early layers, suggesting the learned features are more general. However, a significant improvement in linear separability is observed in middle and output layers, closely following the pattern achieved by BP. This trend is more obvious the depth of auxiliary nets increases. These results indicate that AugLocal can alleviate the -ight--ight problem of other local learning methods, providing more useful features for top layers.
7

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of different auxiliary networks construction strategies with ResNet- 10 on CIFAR-10. Unif., Seq., and Repe. represent the uniform, sequential, and repetitive strategies. CIx x and C3x3 denote constructing auxiliary networks using I Xx] I and 3 convolutions, respectively, Note that the last two convolutional networks are designed to have comparable FLOPs to AugLocal.
(d = 2) (d = 3) (d= 4) (d=3) 5) (d= 6) (d= (d=8) 8) (d=9) 9)
Unif. 90.98:10.05
92.62.10.22 93.22.10.17 93.75:10.20
05 94.03.10.13 94.01 94.30.10.17
Seq. 82.19-10.47 85.34.10.18 86.88.10.23
88.52.10.06
89.49-0.12 90.44.10.17
91.15-0.17
91.8510.04
Repe,
82.24=2..25 85. 16. 86.47:0.009
87.64+0.20 88. 88.59=0.26 89. 39.30++.200
89.85=0.11 90.79+40.99
CIxI 80.6110.34 87.3410.14 88.4010.51 89.29110009 88.82-00.99 89.55100.08 89.16-0.05 89.05.0.19
C3x3 83.03+00.24 85.82+0.08 87 87.84 89. 19.0000020 90.01 +0.05 90.73+0.11
91.11 91.50++0.12
45
de3
Ue2
4.6
d+?
FLOPS Reductin
Accuracy (65)
93
35
92
DMO
25
0.6
0.7
0.8
0.9
1.0
0.6
0.9 Decay Factor r Pyramidal Depth
0.7
0.8
1.0
Decay Factor o Pyramisal Depth
Figure 5: Influence of pyramidal depth on accuracy and computational efficiency. The FLOPs reduction is computed as the relative difference between with and without pyramidal depth. Refer to Table IT for specific FLOPs values. Results are obtained using ResNet- 10 on CIFAR-10.
4.4 MEMORY EFFICIENCY ANALYSIS
Local learning has the potential to improve memory efficiency compared to BP since intermediate variables such activations not necessarily kept in memory anymore after local updates (Nøkland & Eidnes (019). We provide empirical evidence of the memory efficiency on CIFAR-10 and ImageNet with ResNets. We report the minimally required memory that aggregated across all GPUs for each task in Table[4 Results show that AugLocal can significantly reduce memory footprint compared to BP and gradient checkpoint (Chen al. 2016) while maintaining comparable accuracies.
4.5 ABLATION STUDIES
We perform ablation studies to examine the effects of different auxiliary nets construction strategies and the pyramidal depth on CIFAR-IO. Here, we choose ResNet-110 due to the large number of local layers it possesses, making it a suitable candidate for evaluation. Additionally, we increase the maximum depth limit of auxiliary networks to nine for more comprehensive analysis.
Comparison of different auxiliary network construction strategies In AugLocal, we propose to build auxiliary networks by uniformly selecting a given number of layers from its subsequent layers in the primary network. To examine the effectiveness of this strategy, we compare it with another two: sequential (Seq.) and repetitive (Repe.) selection. Seq. uses the consecutive subsequent layers of a given hidden layer as its auxiliary network, while Repe. constructs the auxiliary network by repeating the same primary hidden layer the desired depth. Results in Table uniform selection adopted in AugLocal consistently outperforms the others regarding accuracy. This highlights the effectiveness of AugLocal's 's auxiliary network construction approach improving local learning. Note that the accuracy difference for d = 2 can be mainly attributed to the channel number difference. Both Seq. and Repe. tend to choose layers closer to the primary network layer, resulting in smaller channel numbers compared to the uniform one, leading to decreased accuracy.
that the
To further demonstrate the role our augmented auxiliary networks, we compare with handcrafted ones under comparable FLOPs. Specifically, we build auxiliary networks using commonly adopted 1x1 and 3x3 convolution layers, maintaining the same network depth while increasing the channel numbers. The results presented in Table 5 demonstrate significant performance advantage of AugLocal over these primitive network construction methods, effectively validating the importance of augmented auxiliary networks. More ablation studies provided in Appendix A.6
8

---

Published as a conference paper at ICLR 2024
Influence of pyramidal depth We empirically investigate the impact of pyramidal depth on model accuracy and computational efficiency. Specifically, with duin = 2 and d varies from 5 to 9, we increase the decay factor of pyramidal depth from 0.6 to 1.0 with increments of 0.1. We then record both accuracy and FLOPs reduction. The FLOPS reduction is computed as the relative difference between the FLOPs of auxiliary networks with and without applying pyramidal depth. Figure 5 shows that increasing the decay factor leads to greater FLOPs reduction at the cost of slightly reduced accuracy. For example, by using the pyramidal depth method, we achieve a 45% reduction in FLOPs with graceful accuracy degradation of only around 0.5%. This suggests the pyramidal depth method can effectively improve computational efficiency while maintaining high classification accuracy.
RELATED WORKS
Local learning was initially proposed as a pretraining method to improve global BP learning (Bengio ct al. 2006 Hinton et al. 2006), which has soon gained attentions as an alternative overcome limitations of BP. In addition to the supervised local learning rules discussed previously, there exist some other approaches. For instance, (Pyeon et al. 2021) proposes differentiable search algorithm to decouple network blocks for block-wisc learning and to select handcrafted auxiliary networks for each block. Some works employ sulf-super contrastive losses in their local learning rules (Ilingg et al. (022)) such as Contrastive Predictive Coding in GIM (Löwe et al. 2019), SimCLR R in LoCo (Xiong al. 2020), and Barlow Twins in (Siddiqui et al. (023). LoCo additionally introduces coupling between adjacent blocks. However, most of these approaches are block-wise with only few independent blocks, and it remains unclear whether they can effectively scale up
to tens of local layers. A very recent work SoftHebb (Journé et al.) 2023) explores local hebbian learning in soft winner-take-all all networks, but its performance on ImageNet remains limited.
Alternative learning rules to BP have gained a growing interest in recent years (Lillicrap et al. 2020). Some efforts are devoted to addressing biologically unrealistic aspects of BP, such as the
weight transport problem (Crick 1989). by using distinct feedback connections (Lillicrap et al.
2016 Akrout et al. 2019), broadcasting global errors (Nø kland 2016 [Clark et al. 2021). random |016|Cllar
target projection (Frenkel et al., 2021), and activation sharing Woo et al.) 2021 p. Alternatively,
target propagation (Le Cun 1986; Bengio 2014 Lee et al. 2015b) trains a distinct set of backward
connections by propagating targets and using local reconstruction objectives. [Baydin et al. (2022); ;
(2023); Fournier et al. (2023) propose forward gradient learning to eliminate BP completely. To enable parallel training of layers, decoupled parallel BP (Huo et al. 2018b) and feature replay (Huo) eet al. 2018a) are proposed to update parameters with delayed gradients or activations. Decoupled neural interface (Jaderberg et al. 2017) uses synthetic gradients to tackle the update locking problem. These methods differ fundamentally from our method as they rely on global objectives. Most of them face challenges when scaling up to solve large-scale problems like ImageNet (Bartunov et al. 2018).
et et al. {
As a widely used regularization technique, stochastic depth (Huang et al. 2016) randomly drops set of layers for each mini-batch effectively mitigating the effect of layer coupling, In contrast, AugLocal uniformly samples some layers from the primary network to construct the auxiliary network, which happens before the training starts. These auxiliary networks are specifically used to optimize their respective hidden layers and do not have any direct impact on other layers in the primary network. [Lee] cct al. (2015a) and Szegedy et al. (2015) utilize auxiliary networks to provide additional supervision for intermediate features in the primary network. However, the training of these auxiliary networks still affects their preceding layers as gradients are not decoupled across layers as in local learning.
6 CONCLUSION
We proposed a novel supervised local learning method named Auglocaa that constructs auxiliary networks to enable hidden layers be aware of their downstream layers during training. We evaluated AugLocal on various widely used large-scale network architectures and datasets including ImageNet. Our experimental results demonstrate that AugLocal consistently outperforms other supervised local learning methods by a large margin and achieves accuracy comparable to end-to-end BP. It is worth noting that dedicated parallel implementation required to take full advantage of the training efficiency of large-scale local learning rules, and we leave it as future work. In addition, to alleviate the update locking problem in auxiliary networks, one potential solution could involve gradually freezing some auxiliary layers during training, which will be investigated as part of our future work.
9

---

Annu. Rev. Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Research Grants Council of the Hong Kong SAR (Grant No. Po1yU11211521, PolyU15218622, PolyU15215623, and PolyU252164231, The Hong Kong Poly- technic University (Project IDs: PO039734, P0035379, P0043563, and P0046094), and the National Natural Science Foundation of China (Grant No. U21A20512, and 62306259).
REFERENCES
Mohamed Akrout, Collin Wilson, Peter Humphreys. Timothy Lillicrap. and Douglas B Tweed. Deep learning without weight transport. H. Wallach, Larochelle, A. Beygelzimer. F. d'Alché- Buc, Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivtted deep learning algorithms and architectures. Advances in Neural Information Processing Systems, 31, 2018.
Atilim Güney Baydin, Barak A Pcarlmutter, Don Syme, Frank Wood, and Philip Torr. Gradicnts without backpropagation. arXiv preprint arXi::2202.08855 085887 2022.
Fugene Belilovsky, Michael Fickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In International Conference on Machine Learning. pp. 583-593, PMLR, 2019.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In International Conference on Machine Learning, pp. 736-745. PMLR, 2020.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. 2014.
arXiv preprint
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer wise training of deep networks. In B. Schölkopf. J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Yoshua Bengio, Dong-Hyun Lee, Jorg Borncchein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning. arXiv preprint arXiv:1502.04133 2015.
Natalia Caporale and Yang Dan. Spike timing-dependent plasticity: a hebbian learning rule. Neurosci., 31:25-46, 2008.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1eXtv:1604.00 1604.06174, 2016.
David Clark, L F Abbott, and Sueycon Chung. Credit assignment through broadcasting global error vector. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 10053-10066. Curran Associates, Inc., 2021.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, pp. 215-223, 2011.
Francis Crick. The recent excitement about neural networks. Nature, 337:129-132, 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 Ieee, 2009.
Louis Fournier. Stéphane Rivaud, Eugene Belilovsky, Michael Eickenberg. and Edouard Oyallon. Can forward gradient match backpropagation? In International Conference on Machine Learning, 2023.
Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in Neuroscience, 15:629892, 2021.
Aidan N. Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, and Yarin Gal. Interlocking backpropagation: Improving depthwise model-paralleiinn Journal of Machine Learning Research, 23(171): 1-28, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Donald Olding Hebb. The organization of behavior: neuropsycholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7::1527-1554, 2006.
Gao Huang. Yu Sun. Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt IV 14, pp. 646-661. Springer, 2016.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700- 4708, 2017.
Zhouyuan Huo, Bin Gu, and Heng Huang. Training neural networks using features replay. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018a. 8a.
Zhouyuan Huo, Bin Gu, qian Yang. and Heng Huang. Decoupled parallel backpropagation with convergence guarantee. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2098-2106. PMLR, 10-15 Jul 2018b.
Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 30365-30377
Curran Associates, Inc., 2021.
Max Jaderberg, Wojeiech Marian Czamecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Muchine Learning Research, pp. 1627-1635, PMLR, 06-11 Aug 2017.
Adrien Journé, Hector Garcia Rodriguez. Qinghai Guo, and Timoleon Moraits. Hebbian deep learning without feedback. In The Eleventh International Conference on Learning Reprexentations, 2023.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529. PMLR, 09-15 Jun 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of Citeseer, 2009.
features from tiny images. Technical report,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates. Inc., 2012.
Michael Laskin, Luke Metz, Seth Nabarro, Mark Saroufim, Badreddine Noune, Carlo Luschi, Jascha Sohl- Dickstein, and Pieter Abbeel. Parallel training of deep networks with local updates. arXiv preprint arxiv:2012.03887, 2020.
Yann Le Cun. Learning process in asymmetric threshold network. In Disordered Systems and Biological Organization, pp. 233-240. Springer, 1986.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553)) 444, 2015.
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher. Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Artificial Intelligence and Statistics, pp. 562 570. PMLR, 2015a.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part 15, pp. 498-515. Springer, 2015b.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276, 2016.
11

---

Published as a conference paper at ICLR 2024
Timothy P Lillicrra, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):35 346, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- Eeoon-CCV// 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2019.
Sindy Löwe, Peter O' Connor, and Bastiaan Vecling. Putting an end to end-to-end Gradient-isolated learning of representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buee E. Fox, and R. Garnett (eds.),
Advances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenherghs. Deep supervised learning using local errors. Frontiers in Neuroscience, 12:608, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Arild Nø kland. Direct feedback alignment provides learning in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International Conference on Machine Learning, pp.
PMLR, 2019.
Myeongjang Pyeon, Jihwan Moon, Tacyoung Hahn, and Gunhee Kim. (SEDONA): Search for decoupled neural networks toward greedy block-wist learning. In International Conference Learning Representations, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju. Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. (0428-10436) 2020.
Mengye Ren, Simon Korbblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In The Eleventh International Conference on Learning Representations, 2023.
David Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginow, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4510-4520, 2018.
Shoaib Ahmed Siddiqui, David Krueger, Yann LeCun, and Stephane Deny. Blockwise self-supervised learning with barlow twins, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 3Xxx:1409.155 1556, 2014.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Compuder Vision and Pattern Recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105 6114. PMLR, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-super deep learning results. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. In International Conference on Learning Representations (ICLR), 2021.
Sunghyeon Woo, Jeongwoo Park, Jiwoo Hong, and Dongsuk Jeon. Activation sharing with asymmetric paths solves weight transport problem without bidirectional connection, Advonces in Neural Information Processing Systems, 34:29697-22709, 2021.
12

---

Published as a conference paper at ICLR 2024
Bohan Wu, Suraj Nair, Roberto Martin-Martin Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2318-2328, 2021.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11142-11153.
Curran Associates, Inc., 2020.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 THEORETICAL ANALYSIS ON THE PARALLELIZATION OF AUGLOCAL
To analyze the parallelization, we compare the training time of AugLocal with that of BP. For simplicity, we assume the forward time ł and backward time to of each layer be the same. We denote the maximum depth of the auxiliary networks as d and the depth of the primary network as L L+ 1, consistent with the notations our paper. N denotes the number of training iterations.
For BP training, the time to train N iterations can be calculated as (L + 1)(t/ + fs).V.
In AugLocal, we define local layer as a hidden layer along with its associated auxiliary networks. The training time for any local layer per iteration can be represented as (e ((-1)ty+ (d+ (tty)()+f))) By parallelizing the training of these local layers once their inputs are available, the time of (d+ can be shared among all local layers. Furthermore, starting from the second iteration, the forward pass of the (E - 1 hidden layer can be parallelized with the backward pass of the 4 auxiliary network. Based on these considerations, the training time of AugLocal for N iterations can be calculated as L + (d + (), + NN, which is approximated to (d + 1)(t/ + N after omitting the constant term.
T+1 Consequently, the ratio of the training time between AugLocal and BP is approximately d+1 This
suggests that as the maximum depth of the auxiliary network decreases, AugLocal demonstrates higher parallelization and faster training speed compared to BP. It is worth noting that to achieve the theoretical training speed-up, up, we need the customized parallel implementation that we consider as future work.
A.2 IMPLEMENTATION DETAILS
Our experiments are based on four widely used benchmark datasets (i.e., CIFAR-10 (Krizhevsky et al.,
2009), SVHN (Netzer et al. 2011). I). STL-10 (Coates et al. 2011), and ImageNet (Deng et al. (009)).
We compare our proposed AogLocal method with the end-to-end backpropagation (BP) (Rumel-
hart et al. 1985) algorithm and three state-of-the-art supervised local learning methods, inc Including DGL (Belilovsky al. 2020). PredSim (Nøkland & Eidnes 2019), and InfoPro (Wang et al., 2021).
We re-implement all of these methods in PyTorch using their official implementations| We utilize
consistent training configurations across all learning methods. All experiments are conducted on a
machine equipped with 10x NVIDIA RTX3090.
Datasets CIFAR- 10 (Krizhevsky et al. 2009) dataset consists of 60K 32 x 32 colored images that are categorized into 10 classes with 50K images for training and 10K images for test. We use the standard data augmentation (He et al.) 2016: Nøkland & Eidnes, 2019, Wang et al. (021) in the training set, where 4 pixels are padded on each side of samples followed by 32 32 crop and random horizontal flip. SVHN (Netzer et al.) 2011) dataset contains 32 x 32 digit images, each with a naturalistic background RGB format, The standard split of 73,257 images for training and 26,032 images for test is adopted. Following Tarvainen & Valpola (017)) ; Wang et al. (222). augment training samples by padding 2 pixels on each side of images followed by a 32 X 32 crop. ST (Coates et al. 2011) provides 5K labeled images for training and 8K labeled images for test. The size of each image is 96 x 96. Data augmentation is performed by 4 x 4 random translation followed by random horizontal flip (Wang al. 2021). ImageNet (Deng al.) 2009) is 1. 1,000-elass dataset with 1.2 million images for training and 50,000 images for validation. Following He et al. (016) Huang et al. (2017): |Wang et al. (202T) 224 224 random crop followed by random horizontal flip is adopted for training samples, and 224 x 224 resize and a central crop are applied for test samples.
Training setups For CIFAR-10. SVHN, and STL-10 experiments using ResNet-32 (He al. 2016), ResNet-110 10 (He et al. 2016), and VGG19 (Simonyan & Zisserman 2014), we use the SGD optimizer with Nesterov momentum of 0.9 and the L2 weight decay factor of 1e-4. We adopt batch size of 1024 on CIFAR- and SVHN and a batch size of 128 on STL10. We train the networks
InfoPro: htte://tttuu.ccooooottttttee-ree
PredSim:
htt h/tp:/gttta www wttp:/cc.mmokaannoommmm http:p////ttttoo..om/a/ao///////
loss, and DGI. ht https::/githhbb. wwwp://......eegeneeeeeoo
DGL.
14

---

Published as a conference paper at ICLR 2024
Table 6: Auxiliary networks in local learning methods for the three stages of ResNet- 110. '/" is used to separate two auxiliary networks for two local losses in both PredSim and InfoPro. The former network is used for the cross-entropy loss, and the latter one serves another loss function.
Stage
PredSim
InfoPro
DGL
AugLocal (d = 2)
AP-I6CI-16CI-16II AP-64FC-644C-1100
OP-10FC/16CS
32C3-AP-128FC-10RC// 12033/0
64R-AP-10FC
AP-32CI-320-3221-
AP-1OFC/32C3 (IC-AAP-22FFC-10C//22333
64R-AP-10FC
RP-128FC-128FC-10
AP-64CT-64C]-64CC AP-256FC-256FC-
3
AP-10FC/ 64C3 1ww.-AP-1FFFFFF//// 12C3-3C3
64R-AP-10FC
Table 7: Comparison of computational costs including FLOPs, GPU memory and computational overhead among PredSim, DGL,, InfoPro and our proposed AugLocal method as well as BP and gradient checkpoint (Chen et al., 2016) on CIFAR-10 using the ResNet- 110 architecture.
BP
Gradient Checkpoint
PredSim
DGL
InfoPro
HLOPs (G)
0.25
0.25
0.25
0.26
0.34
GPU Memory (GB)
9.27
3.03
1.54
1.61
3.98
Computational Overhead
34.1%
65.9%
76.2%
292.3%
-Wall-elock Time)
Acc.
94.61
94.61
+0.18
74.95+00.36
85.69+0.32
86.95=0.46
AugLocul (d 2)
AugLocul (d 3)
AugLocul (d 4) AugLocal (d 5) AugLocal (d = 6)
FLOPs (G) GPU Memory (GB)
0.63
0.69
0.80
0.98
1.13
1.71
1.62
1.70
1.71
1.72
Computational Overhead
87.2%
115.9%
135.6%
180.8%
214.6%
(Wall-clock Time) Acc.
90.98110.05
92.62:10.22
93.22:10.17
93.75::0.20
93.96:10.15
for 400 epochs, setting the initial learning rate 0.8 for CIFAR- 10/SVHN and 0.1 for STL-10, with the cosine annealing scheduler (Loshchilov & Hutter 2019). For ImageNet experiments, we train VGG13 (Simonyan & Zisserman 2014) with initial learning rate of 0.1 for 90 epochs, and train ResNet-34 (He et al. 2016) and ResNet- 101 (He et al. 2016) with initial learning rates of 0.4 and 0.2 for 200 epochs, respectively. We set batch sizes of VGGT3, ResNer-34, and ResNet- 101 256, 1024, and 512, respectively. We keep other training configurations consistent with the ones on CIFAR-10, It worth noting that, to reduce the computational costs of auxiliary networks, we change the number of hidden neurons in each auxiliary network's 's classifier from 4096 to 512 on VGG13.
A.3 COMPARISON OF COMPUTATIONAL COSTS AMONG LOCAL LEARNING METHODS
Auxiliary networks We keep the original configurations as stated in their respective papers for the auxiliary networks of DGL, PredSim, and InfoPro in our experiments. The auxiliary networks in these local learning baselines and AngLocal (d = 2) are provided in Table For clarity, show the auxiliary nets based on the three stages of R 10, each stage with the same number of output channels. We use the following notations: R denotes residual block, C is convolutional layer, AP signifies average pooling, and FC indicates a fully-connected layer. CI and C3 refer to x and 3x3 convolutional kernel sizes, respectively. The value preceding C, R, and FC denotes the number of output channels.
Computational costs We compare the computational costs including FLOPs, GPU memory, and wall-clock time among BP, PredSim, DGL, InfoPro and our proposed AugLocal method. Note that the wall-clock time across all local learning methods is measured specifically under the sequential implementation setting, where each hidden layer trained sequentially after receiving a batch of samples. Table demonstrats that AugLocal achieves significantly higher accuracy at slightly higher computational costs than the other methods using the ResNet- 110 architecture on CIFAR-10 10. The accuracy of AugLocal can be further improved by employing deeper auxiliary networks and larger computational costs. Additionally, we further compare AugLocal with gradient checkpointing (Chen (et al.) 2016). Our results in Table 7 demonstrate that AugLocal can achieve much lower GPU memory footprint than gradient checkpoint, albeit accompanied by a moderate increase in wall-clock time. It is worth noting that the actual memory overhead of Locao does not follow perfect linear
15

---

Published as a conference paper at ICLR 2024
growth due to the PyTorch backward implementation has been optimized for e2e BP training. In our future work, we will explore more efficient CUDA implementation to address this issue.
A.4 GENERALIZATION TO DIFFERENT CONVNETS
To evaluate the generalization ability of AugLocal across different convolutional networks (Con- vNets), we conduct experiments on three popular ConvNets: MobileNet (Sandler et al.) 2018). EfficientNet (Tan & Le 2019), and RegNet (Radosavovic al. 2020).
Network architectures MobileNetV2 (Sandler et al., 2018) is a lightweight architecture and com- prises two types of building blocks. One is the inverted bottleneck residual block with a stride of 1, and another is the variant with a stride of 2 for downsizing. Each block contains 3 convolutional layers, including two point-wise convolution and one depth- wise convolution. EfficientNetB00 (Tan &
Le 2019) employs compound scaling strategy to jointly scale network's 's depth, width, and resolution, offering superior performance in terms efficiency. The building block of EfficientNetBO the inverted residual block with an additional squeeze and excitation (SE) layer. RegNerX_ DegNeeXX400MFF (Ra dosavovic et al. (020) is derived from a -iwddmmnniooal network design space consisting of simple and regular networks. The standard residual bottleneck blocks with group convolution are adopted as its building blocks, each of which comprises 1 x 1 convolution, followed by a 3 x 3 group convolution and final x convolution.
Training setups As the minimal indivisible units, the building blocks in the three architectures are their local layers, which independently trained with local learning rules. For Augl.ocal, the down- sampling operation in auxiliary networks is performed by changing the stride of the corresponding auxiliary layer to 2. Other training configrations are the same the previous ones on CIFAR-IO.
Our experimental results in Table & demonstrate that AugLocal consistently obtains comparable accuracy BP, regardless of the network structure. highlighting the potential of AugLocal to generalize across different network architectures.
Table 8: Performances of AugLocal on different ConvNets. CIFAR-10.
The experiments are conducted on
BP
AugLocal (d = 3)
(d = 4)
(d 5) 5)
(d = 6)
MobileNetV2 =L= = 19)
94.8910.15
92.16.10.24
93.94.10.23 94.43.1..04
94.5210.08
EfficientNetBO (L 17)
93.52+0.15
92.70+0.11
92.84=0.15 93.03+0. 16 93.134:008
RegNetX_400MFF (L = 23) 95.70=+0.12
94.42+0.11
94.72=0.01 94.96+0.12 12 95.09++..10
A.5 COMPARISON OF LOCAL LEARNING METHODS WITH COMPARABLE FLOPS
This experiment compares AugLocal other local learning methods with comparable FLOPs. We scale up the auxiliary networks of DGL by using 3x3 3 convolutions with the same network depth and multiplier scale up the channel numbers of the convolutional layers ensure similar FLOPs as AugLocal. We further implement PredSim and InfoPro. which incorporate additional local losses and auxiliary networks, resulting higher FLOPs than DGL. The ResNet- 110 architecture on the CIFAR-10 10 dataset is adopted in this experiment. Our results in Table Table coonssslcon demonstrate that LogLocal outperforms the other methods with similar FLOPSs, reaffirming the effectiveness of our approach in constructing auxiliary networks for improved performance in supervised local learning.
A.6 ABLATION STUDY OF AUGLOCAL'S AUXILIARY NETWORKS
We conduct ablation experiments to investigate the impact of altering the auxiliary architecture on AugLocal's performance. In this ablation study, focus AugLocal with depth (d) of 6. which equals the depth of DGL Beliioossy et al. (2020). We use the ResNet-110 architecture that has three stages, each having the same number of output channels. To align with DGL, which uses the same auxiliary networks for layers in the same stage, we gradually modify AugLocal's 's auxiliary architectures to match those of DGL.
Initially, we replace the auxiliary networks in the second stage with a repetitive selection strategy combined with downsampling while keeping the other two stages unchanged. This modification
16

---

Published as a conference paper at ICLR 2024
Table Comparison of local learning methods with comparable FLOPs on ResNet-110.
Method
(d = 2)
(d = 3)
(d -0=00 4)
(d (d=00 5)
(d = 6)
AugLocal
0.09
92.62:+:0.2
93.22++0.17
93.75++0.20
93.96:10.15
DGL
83.03:10.24
85.82:10.08
87.8410.46
89.19:10220
90.01
PredSim
72.06:1:0.63
80.16:10.47
86.00.10.56
88.27:00.72
88.34.0..4
InfoPro
83.71=0.20
89.14
90.75:0:22
91.455:0.05
92.1010.177
results in 1.21% 1% accuracy drop. Subsequently, remove the downsampling operation, leading to a further accuracy drop of 1.02%. Based on these modified auxiliary architectures, we replace the first stage layers with the repetitive auxiliary networks and downsampling, resulting in accuracy degradation of around 1.5%. Finally, we replace all residual blocks in the auxiliary networks with 3 x 3 convolutional layers while maintaining the same number of channels. This change significantly affects the accuracy. resulting in a drop to 88. 88.28%. It is worth noting that DGL further adopts convolutional x 1 layers and fully-connected layers, which achieve a baseline accuracy of 85.69%
These ablation experiments clearly demonstrate the important role of each component in the auxiliary networks of AugLocal. The results highlight the effectiveness of our approach in constructing auxiliary networks for improved performance in local learning.
We provide the details of auxiliary networks in the ablation experiment in Table 10 For consis- tency, we adopt the same notations as A.3 with the addition of s2 representing a stride of 2 for downsampling.
Table 10: Results of the ablation study for AugLocal by gradually modifying auxiliary networks to match those of the baseline. Method
Acc.
Stuge
Stage
Stage 3
AugLocal (dl 6)
93.96:10.15
Uniform Selection
Uniform Selection
Uniform Selection
Replace with repe. and
32Rs2-32R-32R-- 32R-32R-AP-IOFF
downsampling (ds.) in Stage 2
92.75-+0009
Uniform Selection
Unitorm Selection
32R-32R-32R-
Replace with repe. in Stage 2
91.73:10.11
Uniform Selection
Uniform Selection
32R-32R
11--32R-A--1EE 10FC
Replace with repe. and ds. both Stuge and 2
16Rs2- 16R-16R-
32Rs2-32R-322
91.40=00088
Uniform Selection
16R-I6R-AP-1OCC
32R-12R-AP-10FC
Replace with repe. and ds. in Stage 1 and with repe. in 2 2
16Rs2- 16R-16R-
32R-32R-32R,
89.92-10.37
Uniform Selection
16R-16R-AM IP-I
32R-32R-AP 1OFFC
Replace with convlutional layers and dowasampling in all stages
AP-16C3-16C3-Jo6
AP-32C3-32C3-32C3
AP-64C3-64C3-64C3
88.28:1024
16C3-16C3-AP-10FC
-32C3-32C-AA-110
64C3-64C3-AP-100
AP: 16C1-1661-166
AP-32C1-32C1
32C1-3201-
1P64CC1-44C1-66
DGL.
85.69+0.32
AP-64FC-64FC-10FF
AP-129FC-128FC-10RG
AP-156CC-26FCC-1OFC
A.7 RESULTS OF AUGLOCAL ON DOWNSTREAM TASKS
To evaluate the generalization ability of AugLocal on downstream tasks, we conduct experiments on the challenging COCO dataset (Lin et al. 2014) for object detection and instance segmentation. Following the common practice, we use the pre-trained ResNet-34 on ImageNet as a backbone and integrate it with the Mask R-CNN detector (He et al.] (017). To ensure fair comparisons, we maintain consistent training configurations for both Aogoocal and BP. Specifically. we utilize the AdamW optimizer, 1 x training schedule consisting of 12 epochs and a batch size of 16. The results in Table [2] show that AugLocal consistently achieves comparable performance to BP across all average precision (AP) metrics, suggesting that AugLocal can effectively generalize pre-trained models for downstream tasks.
A.8 CONVERGENCE SPEED
In this experiment, we aim to investigate the convergence speed of our proposed AugLocal method by comparing it with BP and other local learning rules. To this end, visualize the learning curves of these methods on ResNet-32 and ResNet-110, As shown in Figure AugLocal achieves a faster decrease in the network output loss as compared to other local learning rules. Moreover, as the
17

---

Published as a conference paper at ICLR 2024
Table 11: Influence of pyramidal depth on computational efficiency. This complements Figure with explicit values of FLOPs (G). The FLOPs for BP is 0.25G.
T
0.9
0.8
T
0.7
T= 0.6
= 5
0.79
0.83
0.87
0.89
0.93
d 6
0.90
0.95
0.99
1.03
1.09
d = 7
0.99
1.06
1.11
1.17
1.22
8
1.09
1.16
1.22
1.29
1.36
d = 9
1.18
1.26
1.35
1.41
1.49
Table 12: Results AugLocal on the COCO object detection and instance segmentation benchmarks.
Method
AP
AP
APA
Apm
APM
AP75
BP
36.3
56.4
39.4
33.8
53.9
36.1
AugLocal
36.2
56.0
39.1
33.4
53.2
35.7
Table 13: Performance of AugLocal with the InfoPro loss (Wang et al. 2021 on ResNet-110. The results of AugLocal with the cross-entropy (CE) loss are provided baseline.
Loss
(d 2)
(d=3) (d= (d= 5)
(d=
(d 7)
(d (d=8) 8) (d = 9)
CE 90.98+00.05 23.2222222.22.2...1. 93, 0..55.55.20000 93.96 94.03
94.01 94. 04. InfoPro 91.5910.111 92.75 93.71 94.1 94.00.....000000000..0 94.02.10.09 17.1..13 94.29.00.07 94.08
depth of auxiliary networks increases, the convergence speed of AugLocal improves and gradually approaches the one of BP. This finding reconfirms the efficacy of our AugLocal method in optimizing networks to achieve high performance.
Layer in ResNet.32
Top Layer 110
10a
10a
10P
10-1
Los5
Lo5
10-1
10
Auglocal
AugLocal
Peediim
Augincal (d-()
Pred
Arpt (d-4)
inoPro
Auguocal (d-5)
10
InfoPro
Aug (d=S)
10
DGL
Augiocal (d-b)
DGL
Auglocal (d=a)
Auglocal (d=2)
Augloral (d=2)
50
100
150
200 Epochs
250
300
350
400
50
100
150
200 Epochs
300
350
400
Figure 6: Learning curves of different learning rules ResNet-32 and ResNet-110 110. The y-axis is in log scale.
A.9 SYNERGY BETWEEN AUGLOCAL AND INFOPRO
Our proposed AugLocal method is orthogonal to existing supervised local learning works (Nøkland & Eidnes] 2019 Wang et al. 202I) that propose advanced local loss functions. In this part, investigate the potential benefits of combining Augl.ocal with InfoPro (Wagg et al. 2021). Specifi- cally, each hidden layer additionally incorporates reconstruction loss with auxiliary network. Following (Wang et al.) 2021). we adopt the auxiliary network containing two convolutions and up-sampling operations. is worth noting that the original augmented auxiliary network with the cross-entropy loss in each hidden layer keeps unchanged.
The results Table 13 demonstrate that incorporating the additional reconstruction loss can lead to accuracy improvements in most cases. This suggests that AugLocal can generalize and synergize with advanced local objectives to improve performance further.
18


Reviews:
[{"review_id": "r75rlYuaFi", "reviewer": "Reviewer_BcQr", "summary": "This paper presents AugLocal, a local learning method for neural networks that alleviates the backward locking problem of traditional back-prop (end-to-end) training. Another benefit of local learning is the reduced peak GPU memory utilization, as activation storage can be reused between different local layers or blocks. In AugLocal, the size of the auxiliary network attached to each local layer is controlled by how far the layer is from the output layer. This facilitates the design of auxiliary networks. The idea behind AugLocal is that gradients obtained from auxiliary networks \"emulate\" gradients from subsequent layers (in traditional back-prop), hence the intuition presented in this work that earlier layers require deeper auxiliary networks. The authors benchmark AugLocal against different local learning methods, as well as back-prop, on various CNNs and different datasets.", "questions": "- In the ImageNet experiments, the DGL numbers correspond to a much waker end-to-end baseline (66.6 vs 71.59 in this paper), thus making DGL seem weaker than it actually is. The authors already reproduced the DGL numbers (as well as other methods) for the other datasets to ensure a fair comparison. Why haven't the authors done the same for ImageNet?\n- What would the GPU memory savings look like with varying batch sizes, does it require a large BS of 1024 to show significant improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work tackles an important problem: local learning in neural networks. \n- The paper is well-written, and ideas are clearly presented.\n- The proposed idea is simple, yet effective. The idea of using gradually smaller auxiliary networks in local learning seems novel to me\n- The experimental results, despite focusing on CNNs, are decently thorough, and back the authors claims.", "weaknesses": "- In Table 1, the authors can provide a more complete picture by including some complexity measure (similar to what is reported in Table 2 in Appendix). What is the wall-clock time of all of these methods? I suspect that AugLocal will incur some significant overhead due to having deeper auxiliary networks.\n- The theoretical speedup of (d+1)/(L+1) is not very informative, as it assumes an ideal setting where all local modules can be run in parallel, which is not trivial to implement. The point of local learning is save GPU memory, which I suspect comes at the expense of extra time. It is also worth comparing AugLocal with back-prop with gradient checkpointing.\n\n\nMinor comments:\n- eq(2) is a bit misleading, it implies that only the auxiliary network parameters $\\Phi^l$ impact the local losses, and the final loss is a function of all network parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AEB3IdxSze", "reviewer": "Reviewer_SLqf", "summary": "This paper studies the supervised local learning problem. It proposes a new rule to construct auxiliary networks for each local layer/block: using a subset of the following layers and the classifier as its auxiliary net, thus closing the gap between local learning and end-to-end learning. Experiments are conducted on CIFAR-10, SVHN, STL-10 and ImageNet. The experimental results show that constructing the auxiliary networks in such a way can effectively close the gap between local learning and bp-based end-to-end learning with tens of local layers/blocks while keeping the memory footprint low.", "questions": "1. It is not clear when the authors say: \"when more layers of the primary network are selected into the auxiliary networks\". Does that mean the weights are shared between layers in the primary network and the auxiliary network? Or just the initialization is shared, or just the architecture is shared?\n\n2. Based on 1, My guess is that other layers are not shared based on the last paragraph of Sec 3.2 and the last sentence in A.2 in the supplementary. In this case, it is not clear why Unif., Seq., Repe. can show a large difference even for d = 2 in Table 5. Does that mean downsampling in the auxiliary network is critical? If the authors want to claim the importance of architectural bias, I believe the missing results are C1×1/C3x3 with downsampling, which should not be considered architectural bias from my perspective. The authors should also show C3x3 performs equally well with the VGG network. The authors should also ablate to what circumstances, when we alter the auxiliary architecture presented in Table 1 in the supplementary, the AugLocal performance degraded and performed similarly to other baselines, simply attributing the performance gain to architecture bias is not convincing as the learning is free-form if the auxiliary network weights are not shared with the primary network.\n\n3. Table 4 shows weird results; how did the authors get the 157.12 GB GPU memory results as there is no single GPU that can handle it AFAIK? The ResNet-110 result looks like an outlier. Can the authors elaborate on why it shows significantly better memory saving while the architecture is similar to other cases (ResNet-32/34/101)?\n\n4. It would be great to show FLOPs comparison between local learning and end-to-end learning instead of just showing FLOPs reduction in Figure 5. Also, gamma is introduced in eq2 but never used in the following text, making it confusing about how it guides the rule design.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The overall idea is straightforward and intuitive. \n2. The performance looks impressive.\n3. The representation similarity looks interesting to me.", "weaknesses": "I have several questions regarding the experimental results and design details in the Question section. Unfortunately, given the current version of the paper, it is not clear why the proposed method works well, what plays the critical role (downsampling in the auxiliary network? shared classifier?) and what is the overhead of the proposed updating rule. I'd be happy to raise my rating if the authors could address those questions during the discussion period, for now, my rating would be 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qeG9aoijg", "reviewer": "Reviewer_6ww4", "summary": "Unlike traditional local learning techniques, AugLocal does not create a new auxiliary network. Instead, it is constructed by selecting a few layers from the subsequent layers of the hidden layers. With this approach, AugLocal was able to achieve higher accuracy compared to traditional local learning, and it is comparable to backpropagation.", "questions": "Despite the aforementioned shortcomings, I find the idea of using the subsequent layers of the hidden layer as an auxiliary network to be very innovative. As a result, I have awarded a score of 6. However, I believe addressing the following questions could further elevate the quality of the paper.\n\n1. Comparisons in same FLOPs\n : I acknowledge that it might be challenging to compare with AugLocal at the same FLOPs since the method of determining the auxiliary network in DGL and InfoPro is already fixed. Nevertheless, if there is a comparison result with AugLocal at the same FLOPs, it would enhance the quality of the paper if it is possible.\n\n2. Refer the update locking issue as limitation\n: As mentioned above, due to the nature of the algorithm, the current layer cannot be trained with the next data until the subsequent layer of the hidden layer is trained (update locking). It would be good to mention this limitations in the paper's conclusion or Appendix and suggest directions for future work. If possible, briefly mentioning a possible solution would also be beneficial.\n\n3. Recent related works\n: The author discusses algorithms like FA (2016), DFA(2016), and Weight-Mirror (2019) in section 5 to address the weight transport issue. However, there are more recent algorithms that have proposed. Please add and compare AugLocal with DRTP (2021) and ASAP (2021), which are recent biologically plausible learning rules to solve weight transport problem, to section 5.  \n\n(1) Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks, 2021\n\n(2) Activation Sharing with Asymmetric Paths Solves Weight Transport Problem without Bidirectional Connection, 2021\n\n4. (minor) Perform another tasks\n : It would be even better if results for simple models like GCN2 or BERT-Base could be added.\n\nIf requests 1-3 are perfectly addressed, I am willing to raise the score to 7-8 points.\n\n**# After the discussion, all my concerns have been resolved, and I will raise the score to 8.**", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to understand.\n\n2. Unlike traditional methods, the concept of utilizing existing hidden layers to construct auxiliary networks, thereby learning representations related to the global loss, is novel.\n\n3. The representation similarity analysis convincingly shows that AugLocal learns in a manner more akin to backpropagation compared to traditional local learning.\n\n4. Not only did it achieve higher accuracy compared to traditional local learning, but it also saved GPU memory.", "weaknesses": "1. Fair comparison to previous works\n: The author presents comparison results between AugLocal, DGL, and InfoPro. However, looking at Table 2 in the Appendix, AugLocal has higher FLOPs than the other methods. Typically, networks with a larger number of FLOPs tend to achieve higher accuracy. Therefore, it seems appropriate to compare the accuracy among AugLocal, DGL, and InfoPro with the same FLOPs, that is, auxiliary networks with equivalent FLOPs.\n\n2. Do not solve update locking perfectly\n: AugLocal utilizes the subsequent layers of the hidden layers. As a result, this method has to wait for the subsequent layers to be trained for training the current layer with the next data (update locking). In contrast, traditional local learning methods like DGL and InfoPro completely solve this update locking issue, enabling asynchronous training.\n\n3. (Minor) Another tasks like GNNs or NLP\n: The author demonstrated the efficacy of AugLocal in image classification tasks using models like ResNet, EfficientNet, and MobileNetv2 for the sake of generality. If AugLocal also performs well in graph tasks or NLP, it would further attest to its generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wcIriW3P27", "reviewer": "Reviewer_ptT3", "summary": "This paper attempts to use local learning which update each layer of the neural networks in an isolated way to reduce the huge memory consumption of classical contrastive learning. As mentioned by the authors, such kind of update neglects the dependence of different layers of neural networks which leads to significant performance drop. To alleviate this problem, the authors propose to select a small subset of layers and train them together during local learning. For different layer, the algorithm takes different number of layers to construct the subset. To validate the performance of the proposed method, the authors conduct extensive experiments on several widely used datasets such as CIFAR-10, SVHN, STL-10 and ImageNet. The results illustrate that the performance can be decent with reduction of 40% GPU memory.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The intuition of the proposed method is reasonable, taking several layers to do local learning should help enhance the performance of local learning via importing information from other layers. \nThe detailed method is direct, select layers by some step from the current layer to the output layer. \nThe performance on several small-scale dataset is decent, deep networks such as ResNet-110 can be trained using the proposed method with small performance gap compared with BP. The GPU memory reduction is very significant. \nThe authors also validate the performance on different CNN network structures. \nOn several CNN network structures as well as several small-scale dataset, the proposed method outperform the counterparts by a significant margin.\nThe authors also analyze the method with extensive ablation studies.", "weaknesses": "The performance gap between AugLocal and BP seems to be enlarged on ImageNet. The authors also do not show the experiments on vision transformer which is also a kind of widely used structure in computer vision. \n\nCould the authors present the performance comparison when using the model trained using BP and Auglocal for downstream task fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)