Cutting Long Gradient Flows: Decoupling End-to-End Backpropagation Based on Supervised Contrastive Learning
Authors: Cheng-Kai Wang
Hung-Hsuan Chen
Venue: Submitted to ICLR 2023 (2023)

Abstract: End-to-end backpropagation (BP) is the foundation of current deep learning technology. Unfortunately, as a network becomes deeper, BP becomes inefficient for various reasons. This paper proposes a new methodology for decoupling BP to transform a long gradient flow into multiple short ones in order to address the optimization issues caused by long gradient flows. We report thorough experiments conducted to illustrate the effectiveness of our model compared with BP and associated learning (AL), a state-of-the-art methodology for backpropagation decoupling. We will release the source code for the experiments after acceptance.


Under review as a conference paper at ICLR 2023
CUTTING LONG GRADIENT FLOWS: DECOUPLING END-TO-END BACKPROPAGATION BASED ON SUPER- VISED CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
End-to-end backpropagation (BP) is the foundation of current deep learning tech- nology. Unfortunately, as a network becomes deeper, BP becomes inefficient for various reasons. This paper proposes a new methodology for decoupling BP to transform a long gradient flow into multiple short ones in order to address the op- timization issues caused by long gradient flows. We report thorough experiments conducted to illustrate the effectiveness of our model compared with BP, Early Exit, and associated learning (AL), state-of-the-ar methodology for backpropa- gation decoupling. We release the experimental code for reproducibility.
INTRODUCTION
Current deep learning technology largely depends backpropagation and gradient-based learn- ing methods (c.g., gradient descent) for model training. Meanwhile, many successful applications rely extremely deep neural networks; for example, Transformer contains at least 12 layers (most have several sublayers) (Vaswani et al. 2017), BERT has 12 to 24 layers (most also have several sublayers) (Devlin al. 2018), and GoogLeNet has 22 layers (many layers Inception modules containing several sublayers) (Szegedy et al. 2015). However, training a deep network based on backpropagation is inefficient for many reasons. First, a long gradient flow may suffer from gradient vanishing or explosion (Hochreiter 1998). Second, a long gradient flow may lead to unstable gradi- ents in the early layers (the layers close the input layer) (Nielsen| 2015). Third, backpropagation results in backward locking, meaning that the gradient of a network parameter can be computed only when all other gradients on which it depends have been computed (Jaderberg et al., 2017). These issues may become severe bottlenecks, especially when network is deep. To train deep networks more efficiently, researchers have developed various strategies, such as batch normalization, gra- dient clipping, new activation functions (e.g.. ReLU and leaky ReLU), new network architectures (e.g., LSTM (Hochreiter & Schmidhuber 1997)), and many more.
Since long gradient flow is root cause of the above issues, a possible way to eliminate these issues is to shorten the length of the gradient flow, for example, by cutting a network into multiple components and assigning local objective to each component. In this way, long gradient flow can be divided into multiple shorter pieces, which should, at least partially, address the issues of vanishing/exploding gradients, unstable gradients in early layers, and backward locking. Perhaps the most straightforward approach for assigning a local objective to a component is by adding a local auxiliary classifier that outputs a predicted I and updates the local parameters based on the difference between ý and the ground-truth target We call this strategy "Early Exit" in this paper because each such auxiliary classifier can be regarded as an exit of the neural network. The concept of Early Exit has been used in many previous studies, c.g., Mostala et al. (2018); Teerapittayanon al. (2016); (Szeedy et al. (0155). However, most of these studies have used Early Exit for other purposes, e.g., creating multpple prediction paths or helping to obtain gradients for the parameters in the early layers. Consequently, these studies have not investigated the separation of end-to-end backpropagation (BP) into multiple pieces, and the associated gradient flows are still long. In addi- tion, even if Early Exit is used to isolate the gradient flow, as shown in (Mostafa et al. 2018). the test accuracies are lower than those of models trained via BP. There other methods of cutting long gradient flows (Jaderberg et al. 2017 (Czarnecki et al. 2017 Löwe et al.) 2019; (W et al. 2022 Kao & Chen 2021). However, most of these methods have been applied only simple networks, and

---

Under review as a conference paper at ICLR 2023
Supervised
Contrastive loss
contrastive
loss
2
g
9
11
X2
Figure 1: An illustration of contrastive learning and supervised contrastive learning.
their test accuracies still unsatisfactory likely because the local objectives may not align with the global objective; details will be discussed in Section 4
methodology, Delog- which decouples the long gradient flow of deep neural network by leveraging supervised contrastive learning (SCL). In this design, the forward path transforms an input into the corresponding prediction ý as usual. However, the gradient flow on the backward path is blocked between different components. Instead of using global objective, we assign a local objective to each component and force each gradient flow remain within one component. We present experiments conducted on multiple open datasets and compare the result with those of models trained via BP, Early Exit, and associated learning (AL) (Wu et al., [022] Kao & Chen 202T), state-of- of-the-art methodology for BP decoupling that yields results comparable to those BP. We find that Delog-SCI. outperforms AI. in terms test accuracy most cases with fewer parameters. Additionally, since our method has a more straightforward network architecture than AL, our method could be a favorable alternative to AL.
This paper proposes
The rest of the paper is organized as follows. In Section 2 we introduce Delog-SCL. and its proper- ties. Section 3 presents comparison Delog-SCL with BP, Early Exit, and AL in terms of their test accuracies and numbers of parameters. We also report the results of analyses on certain prop- erties of Delog-SCL in the same section. Section reviews previous works on BP decoupling and presents comparison of these works with our model. We conclude contribution in Section
2 METHODOLOOGY
2.1 PRELIMINARIES: CONTRASTIVE LEARNING AND SUPERVISED CONTRASTIVE LEARNING
Contrastive learning (CL) is a self-supervised technique for learning visual representations of im- ages. Referring to the left of Figure 1 given an image æ, CL involves generating different views (i.e., 22 and x2 in Figure via the same family of data augmentations T. The generated views (æ) and 22) are further transformed via an encoder function f and a projection head g to minimize the contrastive loss between the output vectors (i.e., 21 and x2). After training, the projection head g is disregarded, and only the encoder f is used to generate the visual representations of images (Chen) jet al. 2020). In other words, given an anchor image æ. CL. relies on regarding its augmented images as positive instances and all other images negative instances and considering that positive pairs should be close after encoding and projection.
SCL refers to the extension of CL from self-supervised setting to fully supervised setting. There- fore, the training data for SCL consist of not only the training images themselves but also the classes of those images. Referring to the right of Figure [ given anchor image x of class the positive instances include the other images of class c in the same batch, whereas all other images in the same batch are regarded as negative instances (Khosla et al. 2020).
2

---

Under review as a conference paper at ICLR 2023
f2
f3
x(
i
g
Lour
y
drour
dcout
08/
08
f
Figure 2: An example neural network with hidden layers. The black arrows correspond to forward path, the red arrows correspond to the backward path, and the green box denotes the com- parison of the distance between two incoming variables g(() and y!!)
Component f
Component
Component 3 fa
Component f+
x(0
n
900
J.L.
O.2
d.s
acoUr
a0,
3.10
30
0.2
aB
a
00
L
00 81
3 82
00 Bh
!
22
z
y
i
L2
0C3
rour
a.f
d8g 81
91
0.2
92
a.g
a8
00p
00
00
08
9
r3
f3
44
Figure 3: An example of a decoupled neural network based on supervised contrastive learning. The black arrows correspond the forward path, the red arrows correspond to the backward path, and the green box denotes the comparison of the distance between two incoming variables.
2.2 DECOUPLING END-TO-END BACKPROPAGATION VIA SUPERVISED CONTRASTIVE LEARNING
This section presents Delog-SCL which leverages the supervised contrastive loss to split one long gradient flow in deep neural network into multiple shorter ones.
Let us first consider standard neural network with hidden layers as an example. As shown in Figure
(1) az(i) refers to an input image i, and the function ft (E = 1,....4) transforms r into
r (under the assumptions that (6) = =r and the predicted class g( r ())) Depending on the network architecture, the functions fe could be various well-known neural network layers, such as fully connected layers, convolutional layers, pooling layers, or residual blocks. The objective LOUT
determined by the task type. For example, we are addressing a classification task, we could use
the cross-entropy loss between the predicted class g(t) and the ground-trune class y() as the objective COUT, We use backpropagation to obtain OcoUT /00, for each layer E, where 0,, represents the optimization
parameters of function ft. Once the gradients are obtained, we can use gradient- strategies, e.g., gradient descent, to update the parameter values. Given neural network with H hidden layers, it can be seen that the longest gradient flow is constructed as a product of // local gradients. For example, to obtain aLouT" /Co in network with 3 hidden layers (as shown in Figure?) 2). we need the following:
OLoUT
acouT
Og
Or
(8)
dr
dr
x
x
(1)
00
0ÿ((1)
dr
0r
dr
5
09
h
3

---

Under review as a conference paper at ICLR 2023
The number of terms of this product grows linearly with the depth of the network. Therefore, as networks become deeper, their long gradient flows cause several optimization issues, as discussed in [
We use Figure to illustrate our strategy of cutting a long gradient flow into several local gradients for a neural network with 3 hidden layers. Let (i) (i.e., a(+)) and r/ (i.e., xl1) be two image views in the same batch (ro)) and ( may or may not be augmented images, i.e., views, from the same image). We use fi to transform each of them, obtaining r(²) and r((), and further use the function 91 to convert them into z)" and x1 respectively. The functions fi and 91 can be regarded as the encoding function and the projection head, respectively, in CL. (refer Figure [ We repeat the same process for each hidden layer & to form the corresponding component L. If (+ and x!() are two different views of the same image or if y()) (the class of (1)) is equal to y (the class of ())),, then we should ensure that zi is close z for all É. Otherwise, we should increase the distance between 2
(1) and z(²). In the last layer, we compute the distance between j(t) and y as
the loss COUT. Eventually, we define the local supervised contrastive loss L for batch B in layer as shown Equation 2
(0) (p) =((/TT 28 zt
exp
C3C
Σ WEB
log
P(i)
ΣBEB
T(j + 1) exp (0 2(177)
VpEP())
where = 1, 112,..., N represents batch of multiview images, P()) is the set of all positive samples for image i, 7 is hyperparameter, and I(j # {0, 1} is an indicator function that returns 1 if j * i and otherwise.
Ultimately, the global objective function is an accumulation of the local supervised contrastive losses and the losses in the output layer, as defined below:
L
Σ
C.
SC
COUT
1=1
where H is the number of hidden layers and LOUT is the ith loss the output layer (refer to Figure
The computation of L.5 and the pseudo code of Delog-SCI for a 3-layer vanilla ConvNet given in Algorithm[aan and Algorithm 2 in Appendix A.5
2.3 FORWARD PATII BACKWARD PATII, AND INFERENCE FUNCTION
For a regular neural network (e.g., Figure 2). the forward path and the inference function are identi- cal, and the backward path is simply obtained by inverting the direction of the forward path. How- ever, the situation is more complicated in our case because we divide the global objective into several local ones. Consequently, we have multiple short forward paths, multiple short backward paths, and one inference path. Thus, the inference path and the forward paths are no longer identical in Delog- SCL.
During training, each component é has its own forward and backward paths. Taking Figure as an example, the forward path of component transforms each r into r! via the local encoding
(8)
function fe and further transforms each r into x via the local projection head ge. On the backward path, each hidden layer computes BCSC/00, and OC$C /80,, based on the chain rule and updates the parameters by means of gradient- based optimization strategies. We block the gradient flow between each component] As result, each gradient flow remains within component and is therefore short. Equation 4 and Equation show these local gradient flows.
'The gradient flow can be blocked by using Tensor Tensor.detach in PyTorch or wfw.ffttopp........oooooo atop_gradieett 'Ten seensorFlee ow.
4

---

Under review as a conference paper at ICLR 2023
x dzt
(4)
08
00g.
acs
Oc
0z
dr
x
x
(5)
j
dz (8)
dr 00,;
Eventually, even we construct a deep neural network, the cost of computing each ag /00,, and each aLSc/ /00g, remains constant. Additionally, the gradient flow in the output layer is also short: we simply compute 0Cgut /00 fx+1 (where H is the number of hidden layers). This design alleviates various issues caused by long gradient flows.
In the inference (prediction) phase, we need the functions fe but not 9r, shown by Equation 6
go fH+1 o fH .... o fe fi((1)).
where o is the function composition operator (H = 33 for the example illustrated in Figure
Although our proposed method (e.g.. Figure involves more parameters than standard neural network structure (e.g., Figure 2 during trainin, they have the same number of parameters during inference because both of them only the functions fr- Therefore, they have the same hypothesis space. The parameters that participate in the inference phase (denoted by 0 ft) are called the effective parameters, and the parameters used during training but not during inference (denoted by 0g.) called the affiliated parameters.
2.4 PROPERTIES
Table 1: An illustration of the training process pipeline
4
t2
f3
44
to
to
B1
Task Task Task
Task 3
B₂
Task 2
Task 3 Task 2
B3
Task I
Task 3
B4
Task
Task
Task
In this section, we discuss three properties of our proposed model - short gradient flows, a flexible structure, and the ability to perform parallel (pipelined) training.
training a regular neural network with BP requires a gradient flow of length O(H). In contrast, the length of each gradient flow in our model independent of the number of layers: the length is always a constant. Therefore, the various optimization issues resulting from long gradient flows, as discussed in Section [ are eliminated (or least alleviated).
As discussed Section
The network structure is more flexible and perhaps easier to understand than that of associated learning (AL), state-of-the-art methodology for decoupling BP in terms of test accuracy (Kao & Chen 202T; Wu al. 2022). Specifically, AL involves projecting the features æ and the target y into the same latent space for each layer é. Although this design yields excellent test accuracies that are comparable to those of BP-trained models (Wu et al.) 2022). it has at least two unconventional and perhaps mysterious characteristics. First, AL involves projecting a one-hot-encoodd target variable into latent vector i and then transforming t back into Interestingly, the length of t is sometimes greater than the number of classes. This process corresponds to building an autoencoder whose bottleneck layer is larger than the input and output layers. Although this unconventional approach works surprisingly well in practice (Wu et al. [022]] Kao & Chen, 2021). the fundamental reasons for this still unclear. Second, when converting neural network into its AL form, sometimes need to create extra fully connected layers. In contrast, our design is more natural because need neither the autoencoder nor the extra fully connected layers.
5

---

Under review as a conference paper at ICLR 2023
Finally, since each component has its own local objective, we can parallelize the training procedure by means of pipelining. We use the network illustrated in Figure 3 as an example. Let Task denote the entire forward and backward process in layer É; then, we can illustrate the pipelining process as shown Table T Specifically, in the first time unit (1, component 1 uses the first batch (B1) to perform Task 1. At 12, component 2 performs Task 2 based on B1, and component 1 continues to performing Task based on the second batch (B2). Starting at 13. all three components can perform forward propagation, backward propagation, and parameter updating simultaneously. However, we have shown here only that parallelization by means of pipelining is feasible; implementation of the pipeline mechanism is left for future work.
3 EXPERIMENTS
We compare Delog-SCI. with three baselines using different neural networks on different datasets. The baseline models include BP, the Early Exit mechanism introduced Section [ and AL, a state- of-the-art method for BP decoupling in terms of the test accuracy. We test three neural networks: vanilla convolutional neural network (vanilla ConvNet), the VGG network (Simonyan & Zisserman, 2014), and the residual network (ResNet) (He et al., 2016). The experimental datasets include CIFAR-10 (consists of 50, 000 color training images and 10, 000 test images; each image belongs to of 10 classes). CIFAR-100 100 (consists of 50, 000 color training images and 10, 000 test images; each image belongs to 1 of 100 classes), and T'ny-ImageNet (consists of 100, 000 color training images, 10, 000 validation images, and 10, 000 test images; each image belongs of 200 classes).
3.1 ACCURACY COMPARISON
Table 2: A comparison of the test accuracies (mean + standard deviation) of different methodologies when using different neural network architectures on CIFAR-10. We highlight the winner among the non-BP methodologies in bold face. We mark a methodology with a f symbol if the test accuracy of this methodology is higher than that of BP.
Vanilla ConvNet
VGG
ResNet
BP Early Exit
86.85 + 0.57
93.02 = 0.03
93.95 - 0.11
83.16 16,16-20 + 0.33
91.28 + 0.15
89.63 + 0.34
AL Delog-SCL
86.98 + 0.24
93.22 + 0.12 f
91.33 = 0.09
86.98 0.33
93.42 + 0.11 f
92.78 + 0.11
Table 3: A comparison of the test accuracies of different methodologies when using different neural network architectures on CIFAR-100. We follow the same notations used in Table
Vanilla ConvNet
VGG
ResNet
BP
58.68 + 0.13
72.58 + 0.39
73.59 + 0.11
Early Exit
50.64 + 0.44
71.11 + 0.95
64.48 + 0.41
AL
53.06 + 0.15
72.43 0.27
67.53 + 0.32
Delog-SCL
59.63 + 0.37
73.14 + 0.30 f
70.41 + 0.27
Table 4: A comparison of the test accuracies of different methodologies when using different neural network architectures Tiny-ImageNer. We follow the same notations used in Table 2
VGG
ResNet
BP
48.30 + 0.14
49.71 + 0.18
Early Exit
46 + 0.18
40 + 0.34
AL
49.06 + 0.14 t
44.83 + 0.15
Delog-SCL
48.95 + 0.17 f
46.87 + 0.26

---

Under review as a conference paper at ICLR 2023
Table shows the test accuracies of the various methods the CIFAR-I0 dataset. The simple Early Exit mechanism can be used to learn the relationship between an image and its corresponding class. However, the test accuracies of Early Exit are much worse than those of BP. Both AL and our proposed Delog-SCI, yield better test accuracies than BP based on the Vanilla ConyNet and VGG architectures. However, when ResNet is used, BP yields the highest test aceuracy. If compare only the methods that involve BP decomposition, Delog- SCL performs the best among them.
We also tested BP, Early Exit, AL, and Delog-SCL on CIFAR-I00. are similar to those on CIFAR-10: Delog-SCL performs better than both AL and BP based on Vanilla ConvNet and VGG, whereas Delog-SCL performs worse than BP when ResNet used. These results are also consistent with those reported in (Kao & Chen 202T; Wu et al., (022).
The results, as shown in Table 3
Table 4 gives the results obtained on Tiny-ImageNet, When VGG is used, both AL and Delog-SCL outperform BP. However, for ResNet, BP performs much better.
Delog-SCL is stable in training, as can be shown by Figure in the Appendix.
3.1.1 DISCUSSION ON ACCURACY COMPARISON
When BP is used, all parameters are updated to minimize a global objective - the residual between the prediction 9 and the target y. On the other hand, methods to decouple end-to-end backpropa- gation, such as Delog-SCL and AL, are composed of many local objectives, which may differ from the global objective. Therefore, is surprising that Delog-SCL and AL outperform BP for some network structures. The authors of AL proposed several conjectures to explain this remarkable re- sult. First, projecting the feature vector æ and the target y into the same latent space may be helpful. Second, the autoencoder may implicitly perform some feature extraction and regularization. Third, overparameterizztion may be helpful for optimization (Arora et al. 2018 Chen & Chen 2020). However, the first and second conjectures only apply to AL but not to Delog- but Delog-SCL still yields better accuracies than BP and AL in vanilla ConvNet and VGG. Therefore, the above conjectures may not fully explain the success of Delog-SCL. Further investigation will be needed to uncover the fundamental reasons.
As for ResNet, its authors state that the main effect of residual is not about promoting gradient flows (He et al. 2016). Instead, ResNet performs better than vanilla ConvNet because the latent representations H, and He+1 deep neighboring layers l and + 1 are likely similar. Regular nonlinear transformations may be difficult to approximate an (almost) identical mapping from He H6t+ However, the residual connection sets He+1 be f(H) + He. Even if f( is nonlincar function, solver is easier to make He+1 = He by making f(H)) x O. The property that H s He likely true when a network is deep. However, when using Delog- or AL, each local network is short, so DeLog-SCL. and AL are unlikely to take advantage of the residual connections. As result, optimizing a ResNet by BP usually gives better results than by BP-decoupling methods, such as DeLog-SCL and AL.
3.2 NUMBER OF EFFECTIVE PARAMETERS
Table A comparison of the numbers of parameters used during training and testing with different methodologies (using CI as example)
VGG
ResNet
Training
Testing
Training
Testing
BP
11.9M
11.9M
11.2M
11.2M
AL DeLog-SCL
284.9M .M
13.9M
325.1M 1M
14.4M
70.
11.9M
72.4M
11.2M
Table 5 shows the numbers of parameters required during training and inference for VGG and ResNet (using CIFAR-10 an example).
For BP, the training and testing stages involve the same set of parameters. In contrast, the training process of AL requires additional bridge functions and encoding functions, which are not used
7

---

Under review as a conference paper at ICLR 2023
Test accuracy (%)
60
Muitiple local gradient flows
30
Single gradient flow
10
10A
Epoch
Figure 4: Training Delog-SCI. on CIFAR-10 using multiple short gradient flows vs using single long gradient flow.
during testing. So, the number of training parameters is much larger than the number of parameters required for BP. During testing, the extra fully connected layers in AL also require more parameters testing parameters.
than are needed in BP. Finally, Delog-SCL, and BP have the same number However, during training, Delog-SCL needs the projection heads ge for each component, the number of required parameters during training is more than for BP but much fewer than for AL.
The difference of the parameter counts may also be reflected on the training and inference speed. We show the practical training and testing time of different methods in Table 7 in the Appendix.
3.3 MULTIPLE SHORT GRADIENT FLOSS ACCELERATE LEARNING
This section shows that dividing long gradient flow into multiple short ones accelerates learning.
Referring to Figure our proposed Delog-SCL. uses local supervised contrastive loss to create short local gradient flow C We compare the standard Delog-SCL. with a modification where only a single long gradient flow used. the compared baseline, we enable the global objective LoUT pass through the entire network and remove all local supervised contrastive losses LSC.
The results are shown in Figure We label the original Delog-SCL as "multiple short gradient flows" and the modification with single long gradient flow as "single gradient flow". Using multiple short gradient flows accelerates the learning speed, especially in the first 100 epochs.
3.4 THE EFFECT OF BATCH SIZE AND PROJECTION HEAD
We also experimented with how the batch size and the type of projection head influence learning. The experimental results show that larger batch size improves the learning quality, which is
sistent with previous studies (Chen et al. 2020) Henaff 2020) Bachman et al. 2019). As for the projection head, using nonlinear function benefits the representation quality of layers before it. The result also matches the experiments conducted in Chen et al. (2020).
The experimental details of batch size and projection heads are presented in Section A.3 and Sec- tion A.4 in Appendix.
4 RELATED WORK
Studies alternatives to BP mostly aim to address optimization and performance issues, such as gradient vanishing/explosionn and training costs. We review some of these works that have particu- larly focused the creation of local objectives and local gradient flows.
8

---

Under review as a conference paper at ICLR 2023
Table 6: A comparison of the properties of BP, AL., and Delog-SCI.
Number of affiliated parameters
Structure flexibility
Length of gradient flows
Allows pipelined
training
BP AL Delog-SCI.
0 Many Few
High Medium Medium
Long
False
Short
True
Short
True
The first type of BP alternative is target propagation (Lee et al.) 2015 Meulemans et al., 2020 Manchev Spratling 2020; Bengio 2014). which assigns a local target for each layer via feedback (inverse) mapping. Such methodology can alleviate the problem of vanishing/cxplodie since each gradient flow is short. However, the parameters are still updated in layerwise fashion, so it could be challenging to learn the parameters in different layers simultaneously.
gradients
Methods of the second type model BP as a constrained optimization problem. in which the output of one layer is forced to equal the input to the next layer (Gotmare et al., 2018 Marra et al., 2020). Such a design shortens the gradient flows and enables parallel parameter updates. However, the experimental results show that the test accuracies are lower than that of standard BP.
Methods of the third type determine the local objectives through transformations of the target. A representative method of this type is AL Wu et al. (002);; Kao & Chen (202I), which transforms both the feature vector æ and the target y into the same set of latent spaces. To the best of our knowledge, AL is the only existing method that network architectures and yield test accuracies that are comparable to those obtained with BP.
achieve BP decoupling for wide range of
Our proposed Delog- SCL is motivated by both AL and Greedy InfoMax (GIM) (Löwe al. 2019), which uses the contrastive loss as each local objective. However, GIM targets self-supervised learn- ing tasks, whereas Delog-SCI. can handle supervised learning tasks because Delog-SCL. uses the supervised contrastive loss in the hidden layers and the distance between the predicted and observed targets in the output layer.
Since only Delog-SCL and AL yield test accuracies comparable to those of BP, in Table 6 we further compare the properties of these three methods. First, BP requires no affiliated parameters because all parameters collaborate to reduce the global loss. BP can be applied to almost all kinds of neural networks. However, its gradient flow long (especially when the network is deep), and it is challenging to achieve pipelined training with this method. AL requires transforming both the feature vector æ and the target y alongside each other. As a result, AL usually requires additional fully connected layers, resulting in a large number of affiliated parameters and less structural flexi- bility. However, each gradient flow in AL short, and parameters in different layers can be updated simultaneously via pipelined training. Finally, because Delog-SCL. requires computing the super- vised contrastive loss in each hidden layer, this method also needs additional affiliated parameters (although much fewer than AL) during training. The introduction of the supervised contrastive loss also adds complexity in the network design. The advantages of Delog-SCI. are similar to those of AL: the gradient flows are short, and parallel parameter updating is possible (via pipelining).
CONCLUSION
This paper presents Delog-SCI., new methodology for decoupling the components of the BP pro- cess in a neural network. Delog-SCL may address various optimization issues (e.g., vanishing/ex- ploding gradients and unstable gradients in the early layers) resulting from the long gradient flows in deep neural networks. We report experiments conducted show that Delog-SCL's predictive power is comparable to (and frequently better than) that of either BP or AL, which is state-of the-ar art al- ternative BP. Delog-SCL is more flexible than AL because Delog-SCL does not require additional fully connected layers, whereas AL usually does. Therefore, Delog-SCL. is a natural substitute for AL and could be promising alternative BP.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning. pp. 244 453. PMLR, 2018.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint 1o7iv.1407.7906, 2014.
Pu Chen and Hung- Hsuan Chen. Accelerating matrix factorization by overparameterization.. In International Conference on Deep Learning Theory and Applications, pp. 89-97, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Wojeiech Marian Czarnecki, Grzegorz Swirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In International Conference on Machine Learning, pp. 904-912. PMLR, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Akhilesh Gotmare, Valentin Thomas, Johanni Brea, and Martin Jaggi. Decoupling backpropaga- tion using constrained optimization methods. In Credit assignment in Deep Learning and Deep Reinforcement Learning at NeurlPS Workshop, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning. pp. 4182-4192. PMLR, 2020.
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Jourmal of Uncertainty, Fuzziness and Knowledge-Band -Basd Systemx, 6(02): 107-116, 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Inter- national conference on machine learning, pp. 1627-1635. PMLR, 2017.
Yu-Wei Kao and Hung- Haang Chen. Associated learning: Decomposing end-to-end backpropaga- tion based on autoencoders and target propagation. Neural Computation, 33(1):174-193, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 13:18661-18673 2020.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pp. 498-515. Springer, 2015.
Sindy Löwe, Peter O' O'Connor, and Bastiaan Veeling. Putting end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems, 32, 2019.
Nikolay Manchev and Michael W Spratling. Target propagation recurrent neural networks. J.
Mach. Learn. Res., 11(7):1-33, 2020.
10

---

Under review as a conference paper at ICLR 2023
Giuseppe Marra, Matteo Tiezzi, Stefano Melacci, Alessandro Betti, Marco Maggini, and Marco Gori. Local propagation in constraint- based neural networks. In 2020 International Joint Confer- on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin Grewe. A theoretical framework for target propagation. Advances in Neural Information Pro- cessing Systems, 33:20024-20033 2020.
Jovana Mitrovic, Brian McWilliams, and Mélanie Rey. Less can be more in contrastive learning. In "I Can't i Believe It's s Not Better!" at NeurlPS Workshops, Virtual, volume 137 of Proceedings of Machine Learning Research, pp. 70-75. PMLR, 2020.
Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:608, 2018.
Michael A Nielsen. Neural networks and deep learning, volume 25. Determination press San Francisco, CA, USA, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409:114 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Surat Teerapittayanon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469, IEEE, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Dennis YH Wu, Dinan Lin, Vincent Chen, and Hung- Chen. Associated learning: an alter- native to end-to-end backpropagation that works on cnn, rnn, and transformer. In International Conference on Learning Representations, 2022.
A APPENDIX
A.1 TEST ACCURACY VS EPOCH
Figure 5 compares DeLog-SCL and BP on vanilla CNN in terms of their dynamics of test accuracy when the epoch increases. First, the test accuracy of Delog-SCI, improves stably. Second, the DeLog-SCL outperforms BP after approximately 100 epochs. BP is better than Delog-SCL, at the beginning, likely because all the parameters in BP are updated to minimize a global objective. On the contrary, most of the parameters in Delog-SCL are updated to fit local objective functions, which usually have no direct access to the target variable.
are
Different hyperparameter settings may lead to slightly different curves. However, most of them follow a similar pattern. Experiments on other datasets (CIFAR-10 and tiny-ImageNet) for the VGG network structure also show similar trends.
A.2 A COMPARISON OF TRAINING AND TEST TIME OF BP, AL, AND DELOG-SCL
Table 7 compares Delog-SCL, BP, and AL in terms of their practical training and testing seconds per epoch on ConvNet, VGG, and ResNet using the CIFAR-10 dataset. The empirical training and testing speed of Delog-SCL. and BP are extremely close. However, AL is apparently slower than the other two.
The experiments tested the Container Compute Service with NVIDIA Tesla V100 GPU.
11

---

Under review as a conference paper at ICLR 2023
60
40
accuracy
30
20
10
Delog-SCL
BP
100
200
300
400
500
Epoch
Figure 5: Epoch vs test accuracy for BP and Delog- SCL on CIFAR- 100 using vanilla ConvNet.
Table 7: A comparison of the training and testing seconds (mean + standard deviation) per epoch with different methodologies (using CIFAR- as an example)
ConvNet
VGG
ResNet
Training
Testing
Training
Testing
Training
Testing
Delog-SCI. BP
16.62 0.71
1.58 + 0.01
51.63 + 0.09
2.67 + 0.01
42.14 0.54
2.34 + 0.01
15.74 + 0.33
1.59 + 0.02
50.05 + 0.06
2.67 + 0.01
42.05 + 0.55
2.35 + 0.02
AL
22.81 + 0.56
1.82 + 0.02
71.40 + 0.81
3.19 0.02
56.62 + 1.29
2.39 + 0.01
A.3 A LARGER BATCH SIZE IMPROVES THE LEARNING QUALITY
We also tested how the batch size influences the test accuracy. shown in Table X performing DeLog-SCL training using a large batch size is helpful, and the improvement on VGG is more evident than other networks. This finding is consistent with the results reported in previous studies, e.g., Chen al. (2020): Henaff (2020): Bachman et al. (2019). in which the authors noted that because larger batch tends to include more negative pairs (as shown Equation 2}. the model has access more information that be used distinguish positive pairs from negative pairs. Although other studies, e.g.. Mitrovic et al. (2020), have shown that the number of negative pairs may not be critical to the improvement of the test accuracy, most studies tend to agree that larger batch size leads to better results.
Table The test accuracies of Delog-SCC SCL when using different batch sizes on CIFAR-10.
Batch Size
VGG
ResNet
32
92.67 0.10
92.54 + 0.14
128
93.11 0.16
92.53 + 0.11
1024
93.42 0.11
92.78 = 0.11
12

---

Under review as a conference paper at ICLR 2023
Table 9: The test accuracies when using different projection heads CIFAR-10.
Type of projection head
VGG
ResNet
Identity Linear
79.2
84.2
90.4
90.2
MI.P
93.0
92.4
A.4 A NONLINEAR PROJECTION HEAD BENEFITS THE REPRESENTATION QUALITY OF THE LAYER BEFORE IT
This section presents the influence of different projection heads. Table 9 compares the accuracies of VGG and ResNet on CIFAR-10 when different types of projection heads are used: identity mapping (i.e., 96
r (i) w r (2) + wo), and the default mapping based on a multilayer perceptron (MLP). The results are similar to those reported in (Chen) al. 2020):: the MLP mapping shows 2.6% improvement linear projection, which outper- forms identity projection by over 10%.
r2 (2)
r( linear mapping (i.e., 96
Using MLP as the projection head beneficial likely because the information loss induced by the contrastive loss is more severe when simple projection head is used (Chen et al. 2020). In particular, since projection head ge (refer Figure 3 and Figure [ maximizes the agreement between augmented images, 9e may remove information relevant to image rotation, flipping, and other data augmentation operations, which could be useful for downstream tasks. When a simple
projection head ge is used, the information contained in T
(0) which (4) will be similar to that in z
means that r invariant to data augmentation. On the other hand, when complex projection (7) head such MLP is used, the information in r may be very different from that in =/() As
a result, even if zi loses information relevant to data augmentation, re may still preserve this information.
A.S
PSEUDO CODE
Here provide a PyTTrchh pseudocode for the creation of the local supervised contrastive losses (Algorithm[] M and Delog-SCL (Algorithm
13

---

Under review as a conference paper at ICLR 2023
Algorithm 1: PyTorch-like pseudocode for *
import torch
import torch. class SupConLoss inn.Module)-
def _init_ (self, dim)
super._init__0)
self.linear =nn. Sequentaa (nn. Linear (dim, 512), , nn ReLU()
nn.Linear 1024))
self.temperature =0.1
def (sel, label):
x self.linear
= mm..uncttoonal.nnmalllaee
label -label.view 1)
bsz = Label.
mask =torch.es label. 1. .flott ()
anchor_mask -to (torch.ones____1
(mask), torch
aaange (bsz)) 1), 0)
logits #torch.divvttorch.rr (x, x.T), self. telferatrrrtt
deno storch.exp - -anchor_mast
prob loggit log (deno. sum (eepdin-Tzue))
loss (anchori_maee mask *prob) sum (1) /mask. sum ()
return loss view.viewife (1, bsz) m)
Algorithm 2: PyTorch-like pseudocode for Delog-SCI.
import terch
import torch.nn as nn A 3-layer vanilla ConvNet example for Delog-SCL class LNN_DelogSCI (nn.Module))
def inin __iii_inell_t (self, dim)
iiper.__nit___ CNNs =[]
losses =[]
channels = 13, 128, 256, 512]
self.shape =32
for i in range (3):
append (nn. Sequential (nn. Con.Co (channels [i],
Channels[i+1] padding=1), Re )))
losses.appent (SupContass ( (self. shape chapeena
i+1]))
self.CNN (CX.etuleeList(CNNN)
self.loss =nn MndMMuudlllliitttoosssss
self.fc emt.squuentaal((1atttenn)) nn. Linear (self.
self.
10))
self.ce =nn, Crm.crrrssfftrryl.ssss
def forward (self, x, label):
loss =0
for i in range (3):
# .detach () prevents the computation graph from propagating gradients to the next layer = self. [i] (x. detach)) ())
if self.training
loss +=self.loss [i] (x, label)
y felf.fc(x. fcl.
if self.training
loss #=self. (y, label)
return loss
return y
14


Reviews:
[{"review_id": "g6uTxpPnrQ", "reviewer": "Reviewer_tuDH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposed a supervised contrastive learning (SCL)-based training method for deep neural networks in which the gradients only flow locally in each layer, solving some of the problems associated with long gradient flows in backpropagation. The authors also benchmarked its performance against early exit, associated learning (AL) and backpropagation (BP) for training some standard feedforward network architectures for classification tasks. The results showed that the proposed method outperforms early exit, AL on most tasks while BP is still superior for ResNet. \n\nIn addition, the author also did some analysis on the effects of batch size on the final test performance and found larger batch size generally works better. The author also found that a nonlinear projection head performs better than identity or linear heads. \n", "summary_of_the_review": "The authors proposed a supervised contrastive learning inspired approach to train deep neural networks without long gradient flows. The proposed method has several characteristics such as 1) it only requires local gradient computation and parameter updates 2) it enables parallel training via pipelining 3) it requires fewer additional parameters than similar methods in training phase and same number of parameters as BP in inference phase 4) the test accuracy are comparable to BP. The authors also studied the effects of batch size, non-linear projection head. However, a thorough study and comparison of the learning dynamics are missing. ", "strength_and_weaknesses": "This work has several strengths:\n\n1. The proposed method is able to cut the long gradient flow in training deep neural networks.\n\n2. The proposed method decouples the parameter updating across layers, making parallel training possible, though it needs follow-up work to validate this claim.\n\n3. The proposed method requires fewer parameter than similar method such as AL in training phase, and has the same effective parameter in inference phase. \n\n4. The test accuracy on simpler tasks and simpler architectures are better than AL and comparable to BP.\n\nWeaknesses\n\n1. One important aspect of learning procedures are the learning dynamics, which is not thoroughly studied. What are the learning curves like for the methods compared? How much longer or shorter does the proposed method converge compared to AL and BP? Is the training stable? Do we need to use different learning steps for each layer? How does the parameters change during training and use this to validate whether it actually addresses some problems with long gradient flows, such as the exploding/vanishing gradient problem for long gradient flows?\n\nThe research community would also benefit from some potential follow-up work such as the pipelining implementation and its possible extension to training other architectures such as RNNs. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d-nJ59CPJT", "reviewer": "Reviewer_UdqE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As neural networks get deeper, end-to-end backpropagation requires computing the gradients along longer paths, thus making gradient computations more prone to numerical instabilities and vanishing/exploding gradients. This paper proposes a new approach which divides the network into different submodules that are independently trained. Specifically, inner modules are trained using supervised contrastive learning, while the last layer is trained as usual. This enables the computation of short gradients, as well as the asynchronous training of modules, which beats the current state-of-the-art method and obtains comparatively results with end-to-end backpropagation.", "summary_of_the_review": "While the proposed approach is novel in the supervised setting, I have serious concerns of its novelty with respect to Greedy InfoMax. While this would not directly imply a rejection, the deferred extra work and the lack of proper experiments to fully understand the benefits and hurdles of the proposed approach in the specific setting it is used for, makes me lean towards rejection.", "strength_and_weaknesses": "**Strengths:**\n- S1. The approach is clear and simple, and provides results at times better than back-propagation, which I find remarkable.\n- S2. The proposed method is a clear improvement over the competing method, beating it in every aspect as far as I can tell.\n- S3. The ablation study is nice and agrees with existing knowledge in contrastive learning.\n\n**Weaknesses:**\n- W1. The motivation is rather weak. While the asynchronous training is interesting, having long gradients does not seem as much of a problem, as it can be overcome with architectural changes (which is clearly shown by the performance difference of the existing approaches in the ResNet experiments).\n- W2. Despite contrastive learning being quite intense in computations, there are no mentions to memory and time consumption.\n- W3. Needed work is left as future work. Specifically, pipelining as well as understanding where comes the improvement in performance come from.\n- W4. Despite talking all the paper about gradients, there is not a single experiment exploring the gradients and showing that there was actually a problem in the chosen experimental setups.\n\n**Comments:**\n- C1. While parallelism is trivialized in the proposed approach, pipelining is also possible in traditional methods, as demonstrated by companies such as [OpenAI](https://openai.com/blog/techniques-for-training-large-neural-networks/).\n- C2. Doesn't the fact that Early Exit perform well in CIFAR-100 show that it is not an appropriate experimental setup?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7LHy-rFGmt", "reviewer": "Reviewer_XPBU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a novel method that decouples back propagation gradient computation in artificial neural networks. The authors identify issues that result from long gradient flows in back propagation, and propose a method that tackles some issues, which include vanishing/exploding gradients, and update locking. In short, the proposed method cuts long gradient flows into several local gradients with a ‘read-out’ at each layer and a corresponding contrastive loss.", "summary_of_the_review": "The work is an interesting alternative to the backpropagation algorithm as well as other existing decoupled gradient computation algorithms. However, the biggest shortfall of this manuscript is the lack of clarity on its contributions. The manuscript lacks a problem statement.", "strength_and_weaknesses": "Strengths\n-----------\n\nDelog-SCL provides competitive results against Backpropagation and is parameter efficient in comparison to related work (AL). Naturally, as a consequence of gradient decoupling, Delog-SCL circumvents the gradient update locking problem found in Backpropagation.\n\nThe algorithm is promising, lots of insights can be teased from this method and it is a fair addition to the literature on decoupled gradient optimization in artificial neural networks.\n\nWeaknesses\n----------------\nThere is a lack of clarity on the novel contributions of the proposed method, Delog-SCL. There is no clear consensus on the prevailing issue that Delog-SCL aims to tackle, a host of issues have been presented, some of which some are unrelated, in particular the Vanishing/Exploding gradient problem. The issue of vanishing and exploding gradients is prematurely explored, for example there is little mention of Delog-SCL being extended to recurrent nets.\n\nSome of the issues highlighted are naturally tackled by contemporary backprop decoupling methods, there was some justification that Delog-SCL is more flexible and simpler than AL, but this was not raised in light of a critical issue (beyond efficient parallelization).\n\nSimilarly, the claim that Delog-SCL tackles the issue of unstable gradients in the early layers was mentioned in the introduction and conclusion with little to no explanation on the reasons behind the instability and how this relates to Delog-SCL. This was not explored in the experiments nor methodology section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0F4O2ivSh0", "reviewer": "Reviewer_Xd2U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "UPDATE: FOLLOWING THE AUTHOR'S RESPONSE, I CHANGE MY SCORE FROM 5 to 6. \n\nThe authors describe a method to reduce the problems associated with gradients in settings where backpropagation happens through many layers. In the context of supervised contrastive learning, the authors propose to compute contrastive losses not only at the output layer, but also at intermediate layers. This allows the authors to segment the neural network into blocks that each have their own losses, and can therefore be trained nearly independently (of course, layers closer to the output still must adapt to changes in the closer-to-input layers during training). The authors compare their method with the Associated Learning method, and find that their method performs about as well, but is easier to work with. The authors also compare with vanilla back propagation, and find that their method (as well as the associated learning baseline) outperform back propagation on a number of tasks and network layer types (but not on residual networks). ", "summary_of_the_review": "UPDATE: FOLLOWING THE AUTHOR'S RESPONSE, I CHANGE MY SCORE FROM 5 to 6. \n\nFor now, I mark this submission \"5: marginally below the acceptance threshold\". \n\nThe main idea presented in the paper is interesting and useful. However, before the paper can be accepted, the authors should improve the reproducibility of the paper by releasing code and describing experiments in more detail in an appendix. The authors should also investigate whether skip connections could achieve the same that their method does. ", "strength_and_weaknesses": "# Strengths:\n\n- The paper addresses a question that remains relevant: how do we best minimize losses in deep neural networks? How do we prevent gradients from vanishing? \n\n- The paper is very clear and well-written, and easy to read. \n\n# Weaknesses:\n\n- The authors did not include the code with their submission, and mentioned no intent to do so. There is also no appendix with experimental details such as layer sizes and learning rates. Reproducing the experiments would be practically impossible. \n\n- Besides the methods described in the submission, skip connections (e.g. in highway networks https://arxiv.org/pdf/1505.00387.pdf or residual networks) are an effective tool to reduce vanishing gradient problems. The authors should highlight this in their literature review, and ideally also perform experiments to check if adding skip connections would remove the need for the method described in this paper. It is interesting that the authors find that their method (as well as the AL benchmark method) do not outperform backpropagation when ResNets are used. This suggests that skip connections may remove the need for methods such as the one proposed in this paper, as well as the AL baseline. It would be an interesting experiment to check how e.g. VGG with added skip connections performs with and without the proposed Delog-SCL method. Skip connections have also been used in LSTMs, e.g. in https://aclanthology.org/C16-1020.pdf and https://ojs.aaai.org/index.php/AAAI/article/view/4613/4491 . \n\n- The authors write “We omitted Early Exit because this method performs much worse on more complicated datasets”. In my opinion, it would be better to include the Early Exist performance numbers, and not need the explanation of why Early Exit was not included. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject