End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
Authors: Keitaro Sakamoto
Issei Sato
Venue: Accepted by TMLR ()

Abstract: End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been proposed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap.
In this paper, we reconsider why E2E training demonstrates a superior performance through a comparison with layer-wise training, which shares fundamental learning principles and architectures with E2E training, with the granularity of loss evaluation being the only difference. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate representations based on the Hilbert-Schmidt independence criterion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation. Furthermore, we show that this layer-role differentiation leads to the final representation following the information bottleneck principle. Our work not only provides the advantages of E2E training in terms of information propagation and the information bottleneck but also suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.

Published in Transactions on Machine Learning Research (05/2024)
End-to-End Training Induces Information Bottleneck
through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
Keitaro Sakamoto
sukakei-199982 ecc. w-w...ookoo..ooooooooooooooooo..
The University of Tokyo
Issci Sato The University of Tokyo
sato wwc.g...ccccco eettoogg.ec..u-tookge.
Reviewed on OpenReview: /ttps://openreeet net/ forum? id= 03wmRh2SfT
Abstract
End-to-end (E2E) training, optimizing the entire model through error backpropagation, fun- damentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been pro- posed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap.
In this paper, reconsider why E2E training demonstrates superior performance through comparison with layer-wise training, which shares fundamental learning principles and architectures with E2E training, with the granularity of loss evaluation being the only dif- ference. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate represen- tations based on the Hilbert-SShmidt independence critcrion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different informa- tion dynamics across layers, in addition to efficient information propagation. Furthermore, show that this layer-role differentiation leads to the final representation following the in- formation bottleneck principle. Our work not only provides the advantages of E2E training in terms of information propagation and the information bottleneck but also suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.
1 Introduction
Backpropagation, which has been fundamentally supporting the recent advancements machine learning, faces challenges in computational problems related memory usage and parallel computing. Moreover, from biological plausibility perspective, discrepancies with actual brain functions have been pointed out (Crick, 1989 Baldi et al. 2017). To mitigate these concerns, various learning algorithms and architectures have been proposed; however, there is still a performance gap between these alternatives and E2E training, which prevents practical use. A deeper underslanding of the relationships between the E2E training method and non-E2E methods is significant, as it helps us set a direction for future research on learning methods without backpropagation. This will also lead recvaluation of the advantages of E2E training. Although the recent success of large language models (LLMs) such as GPT-4 remarkable, the human brain has connections on the order of 100 trillion, whereas GPT-4 has roughly 1 trillion parameters (Stiles & Jernigan, 2010 OpenAT

---

Published in Transactions on Machine Learning Research (05/2024)
E2E
Layer-wise
100
Linear Probe Accuracy
90
69
70
Test
Test
60
Tiann
Tiain
50
40
10
20
32
10
10
22
10
Layers
Layers
Figure Linear probing accuracies of ResNet50 trained on CIFAR10 dataset. Linear separability for test data and training data are presented. Left:: E2E training with cross-entropy loss. Right: Layer-wise training with cross-entropy loss. The model was trained with an auxiliary linear classifier for each block.
2023 This parameter efficiency prompts us to reconsider the biological plausibility in machine learning technologies, turning attention to the efficiency of current E2E training via backpropagation (Hinton 2023
To explore the properties of E2E training, we adopt layer-wise training for comparison, which shares the fundamental learning principles and architecture but differs in setting errors locally. As first step, we confirm the performance gap between E2E and various layer-wise training methods and demonstrate that the performance of layer-wise training saturates with increasing depth, as illustrated in figure [ with the finding of Wang et al. (2020), who showed that the greedy optimization in layer-wise training leads to the loss of input information required for classification at the carly layers. This observation, coupled with the fact that E2E training can cooperatively learn each layer through backpropagation, motivates the information-theeortiii analysis of the trained model. Specifically, we analyze the mutual dependence among the model input, class labels, and intermediate representations, following the information bottleneck work (Tishby et al., 2000 Tishby & Zaslavsky 2015 We investigate with normalized HSIC (Fukumizu et al, 2007), which is used to analyze the neural network in several existing studies, to circumvent the difficulty in estimating mutual information (Ma et al., 2020 Wang & Isola, 2020 Wang et al. 2023b Our analysis centered around HSIC dynamics reveals the two following findings:
This correlates
1. The E2F-trained model obtains the middle representations with a higher correlation with labels, despite having lower class separability. We present & toy example, independently of the HSIC analysis, where layer-wise training causes the degeneration in mutual information. These analyses imply that F2E training is advantageous in information propagation.
F2E training compresses middle representations maintaining a high HSIC value the final layer for relatively deep models. In contrast, layer-wise training shows uniform compression every layer uniform increase with the method to keep input information as in Wang et al. (2020) The layer-wise training shows the same HSIC behavior at each layer, whereas E2E training assign different roles of information compression among layers. We refer to this property as layer-role differentiation. The term "differentiation" " refers to the process through which different layers develop distinct roles or functions and different from the mathematical operation of differentiation in calculus.
Additionally, our HSIC analysis shows the information bottleneck behavior in the final layer only for E2E training. It shows the benefit. of E2E training in terms of information bottleneck and suggests a connection between the middle layer compression in E2E training brought by layer-role differentiation and the acquisition of information bottleneck representation at the final layer. Finally, as another advantage of intermediate layer compression, we derive the connection with Frosst et al. (2019). claiming that the class entanglement in the middle layers leads to better representation in the final layer. Since both F2E and layer-wise training
2

---

Published in Transactions on Machine Learning Research (05/2024)
shares the same backbone architecture, note that our analysis is not about the benefits of deepening the layers but is about the advantages of training in the E2E manner. Our code is available on GitHub
2 Difference between E2E and Layer-wise Training
We first provide a clear definition and problem setting for E2E training and layer-wise training, Subsequently, layer-wisc training mothods, which have some variations depending on the type of local loss and architectures, are organized and compared. We show that while there are differences performance among these methods, the persisting performance gap remains compared to E2E training.
2.1 Problem Setting
First, let us clarify the definitions of End-to-end (F2F) training and layer-wise training. F2E training refers to a learning method where the objective function for the entire model is sct, and the weights gradient updates are propagated throughout the entire model using backpropagation. In contrast, layer-wise training is learning approach to set an objective function for each layer. The influence of the local objective function is confined to the specific layer alone, and gradient information does not propagate to the preceding layers. In layer-wise training, each layer is trained simultaneously, and the model's prediction is made by using only the last, output. We consider the local module size to be one layer; however, in cases where straight forward layer-wise modularization is difficult, such as a skip-comnection the ResNet blocks, we handle the natural minimum module that includes one skip-connection (sce section B.1 for training details).
Next, we organize the problem settings and define the notations in the paper. We consider a multi-class classification problem with dataset {-, ) consisting of inputs x; E X and the class label yi E D. Specifically, we consider the domain X' = R R y {1, (1. (l....,c), where e is the number of classes, and use æ to express the vector imput explicitly. The dataset is sampled from the underlying distribution pxy, and we denote the input random variables and label random variables as X and Y.
We have L layers JI. and stack them to form a single neural network f. The dimensions of the I-th layer's representation are denoted as di, i.e. I-th layer is fi : Rd411 + Rd., Note that the intermediate reprosentations are allowed to be more than one-dimensional, but they are vectorized this notation. The I-th layer's representation for input is denoted as zt, meaning input is indexed, the [-th representation for a; is written as Z1,-- We simply use to denote the middle representation without specifying the layer explicitly. To clear the optimized weights, let us denote the weights of each layer ft by 8,, which are denoted as collectively. The objective loss for E2E learning is
Si o... fxr). In cases where the
m LE2E = Σ (ff(x)),)..
(1)
where é is the loss function that takes the model outputs and the true label. The cross-cntropy loss is widely used for multi-class classification. In contrast, the loss to optimize for layer-wise training is
I. LLW( C.)))
1.
Σ
(1(Stpprra(aa...)..n))) m
(2)
where StopGrad is the operator to stop gradient propagation, such as "Tensor.detach(() in the PyTorch implementation, which enables layer-wise training. Here, the loss of the 1-th layer is denoted as b; however, note that we use the same loss among the layers in this study. When calculating this local loss, various layer- wise training methods adopt auxiliary networks for each layer. While the algorithm that directly optimizes the representation space does not need these networks, in the case of classification loss such as cross-entropy, the loss is evaluated after projecting onto dimensional space. We denote the auxiliary network used in the 1-th layer by gi di + ęd which can be linear layer, multi-layer perceptron (MLP), identity mapping in the case of no auxiliary networks. Additionally, the output of the auxiliary networks is denoted as h ... hr. The problem settings of E2E training and layer-wise training are sunmarized in figure 2
https://gitthu: httptttuuu.ccmcheeeeeeeeeeee htt.://iaaasssttt//aaa....
3

---

Published in Transactions on Machine Learning Research (05/2024)
Forward
Backward
Input
Label
Loss L
X
L(Z,,))
L(z,,)) L(x,,)
M(z,,))
M(h,,)
1(h2,))
L(hs,))
L(z,,))
[hg
f
f2
f4
fz
f3
fi
f2
Jf
f3
Jf
fx
Z1
22
Zz
Z1
Z4
Z1
Z2
Zz
Z+
E2E Training
Layer-wise Training
Layer-wise Training
with Auxiliary Networks
Figure 2: Comparison between E2E training and layer-wise training. The models have four layers, and each layer is denoted as f2, fa, f4. Left: E2E training evaluates the loss only after the last representation 21. The gradient updates are back-propagated to the preceding layers. Middle: Layer-wise training calculates the loss for each layer's 's outputs and updates weights. Right: Layer-wise training with the auxiliary networks.
2.2 Performance Gap
In this section, we empirically explore the performance gap between E2E training and several methods of layer-wise training. Layer-wise training has several variations based on the type of locally defined loss function and the auxiliary networks' types and sizes. We begin by organizing these methods and elucidating their performance differences compared to E2E learning methods.
In the layer-wise training methods, locally used loss falls into two primary types based on the method of providing the label information y. The first optimizes the classification loss each layer by forwarding input only. The second one forwards both the input x and its corresponding input, called the prototype, representing the class label information of x, and optimizes losses based on spatial relationships in each layer's representation. For the former case,
cross-entropy loss (CE), while for the latter, supervised contrastive loss (SupCon) (Kosla et al. 2020) and similarity matching loss (Sim) (Nokland & Eidnes 2019) are adopted for the case where random images are used as prototypes, and signal propagation (SP) (Kohan let al. 2023 is used for the case where the class itself is projected onto the input space and used as a prototype. Please refer to section B.2 for the details of these methods.
The comparison of these methods' accuracies for CIFAR10 is presented in Table [ and for the CIFAR100 dataset, they detailed in Table 4 in the appendíx. We used three networks with different sizes: VGG11, ResNet18, and ResNet50. Here, please note that the layer-wise training without auxiliary networks corre- sponds to the row of the O-layer. The experimental details are also deferred to section F.I These results indicate two significant observations. First, the performance of layer-wise training was worse when directly optimizing middle layers without auxiliary networks, and was greatly enhanced with the existence or the size them. However, there remains an accuracy gap compared to E2E training across all loss functions. When the network had already achieved relatively high accuracy with an auxiliary network with one layer, adding second layer did not yield significant. improvements, and the model was not competitive with the one trained in the E2E way. Second, in E2E training, there was substantial improvement in accuracy by increasing the depth of networks in the order of VGG11, ResNet18, and ResNet50. Conversely, the amount of accuracy improvement was limited in the case of layer-wise training. As in the size of auxiliary networks, increasing the number of layers in the backbone model did not lead to a significant accuracy improvement.
2.3 Linear Separability
Then, what properties of the trained models contribute to such performance differences? A key distinction between E2E and layer-wise training lies in how the behavior of the intermediale layers is specified. In E2E training, this behavior is indirectly specified, whereas, in layer-wise training, directly controlled through the optimization of local losses. For instance, when optimizing classification errors each layer using lincar auxiliary classifiers, the training aims to enhance the linear separability of intermediate representations. Contrasting this with the F2E training provides insight into the property differences. Figure T compares
4

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Test accuracies of E2E training and layer-wise training methods trained on CIFAR1O dataset. The "Auxiliary" column shows the size of auxiliary networks g for layer-wise training. For clarity, the highest accuracy among the different sizes of auxiliary networks indicated in bold. For CE loss, the middle reprosentations converted to space of class numbers' dimensions; thercfore, it does not have O-layer case.
E2E
Layer-wise
Model
Auxiliary
CE
SupCon
CE
SupCon
Sim
SP (Hard)
SP (Soft)
O-layer 1-layer 2-layer
60.6
72.4
82.4
81.5
VGG11
91.5
90.4
89.3
83.7
89.2
88.4
89.1
89.0
O-layer 1-layer 2-layer
74.0
87.8
58.0
45.9
ResNet18
93.9
95.7
89.3
88.1
89.4
89.9
92.5
91.6
O-layer
73.2
84.7
51.7
19.5
ResNet55
I-layer
94.7
96.3
89.9
88.7
91.2
2-layer
90.4
92.0
91.3
the linear separability measured through linear probing (Alain & Bengio, 2016) at each layer. In the E2E case, the linear separability gradually improves, whereas, in layer-wise training, it abruptly improves in the carly layers and then saturates the same level as depicted the red region. Both approaches ultimately achieve perfect separation the training data, but E2E demonstrates better linear separability the test data. Interestingly, in layer-wise training, stacking layers does not improve the separability of the final layer. This is consistent with previous experiments that show that the increase in the model size did not lead to as much accuracy improvement as observed in the E2E training. This holds true for similarity- local loss functions, and the results for the SupCon case are presented in section F3
We further investigated why the layer-wise training does not benefit from stacking layers. The layer-wise trained model was retrained in the F2E manner after the layer where the linear separability starts to saturate, i.e., the border between blue and red regions in figure D Our aim here was to observe test accuracy improvement, which would po- tentially indicate whether valuable input information remains in these intermodiate layers that layer-wise training does not fully utilizo. As indicated in Table 2 the outcomes of this experiment under the same setting as figure
Table 2:
Retrain results of
ResNet50, CIFAR10. The same trained models as figure
are used.
E2E
LW
Retrain
Test acc
94.8
89.9
90.8
T demonstrated only marginal accuracy improvement
through E2E retraining. This suggests that useful input information for classifying is lost even in the early layers due to localized optimization, reconfirming the information collapse hypothesis suggested by (Wang et al. 2020). In the following sections, we explore the information propagation within the trained model, elucidating the role of E2E training.
3 Information Bottleneck for Analyzing Deep Neural Network
The comparison between E2E and layer-wise training in the previous section motivates us to evaluate the information flow between layers. We focus on the concept of information bottleneck to deal with the change of mutual information within the trained model. In this section, let us introduce the related concepts and normalized HSIC as an alternative to mutual information.
3.1 Information Bottleneck
Information Bottleneck (IB) Tishby et al. 2000) refers the compression of the input X obtain the representation with the help of the random variable Y behind X, in this case, the label information. It uses mutual information among these random variables to compress X while trying to retrain much
5

---

Published in Transactions on Machine Learning Research (05/2024)
information about Y contained in X as possible during the compression process. In recent years, IB has
been used to analyze the dynamics of deep learning (Tishby & Zaslavsky 2015 Saxe et al. 2018). TB aims
to extract the task-relevant information from X as much as possible, by minimizing
[(;; Z) Z),Y;
(3)
where B > 0 is the trade-off parameter and / denotes the mutual information (see section D for the definition). The task-irrelevant information in X is squeezed by minimizing this Lagrangian objective, and the random variable Z can be regarded a compression of X to solve the task. In this framework, the dynamics of the final representation in deep learning are explained by learning the compressed representation Z that minimizes the IB objective. From the IB perspective, multilayered neural networks can be viewed as a sequence of data processing steps. The imput X generated from the label Y behind X and transformed into the representations Z1, Zz.... as passes through each layer. This process can be represented by the ZL From the data processing inequality,
Markov chain Y + X Z1
(4) Intuitively, the compressed representation Z cannot have more information about the label Y than the original imput X. The network aims to transform X without losing much information from the given upper bound I(Y; X), extracting useful features from X and enhancing the class separability in the final layer. Morcover, considcring the reverse data processing incquality, holds that
X(Y;)) 2 (Y: 2 ... 2 I((Y;z))
= ((X:
(5)
for any middle representations Z. The IB hypothesis states that in the final layer, there is an initial phase where [(;; Z) increases along with [(X; Z), followed by a phase that reduces [(X; Z) to compress task- irrelevant information. This implics that while increasing the lower bound in equation network training subsequently reduces [(X; to minimize the gap with the lower bound.
5 raises I(X; Z), the
Analyzing mutual information within the deep neural network is an intriguing research area. However, there are two major challenges: 1) the potential for I(X; Z) to become infinite, and 2) the difficulty of estimation due to the high dimensionality. For more details, please refer to section D.2 in the appendix.
3.2 HSIC an alternative to Mutual Information
Considering the difficulty in estimating mutual information, we use the Hilbert-Schhidt independence cri- terion (HSTC) (Gretton et al. 2005) because it also allows the capture of non-linear relationships of random variables in & non-parametric way. The use of HSIC is supported by the fact that it has a relationship with a variant of mutual information and the previous studies (see related work, section 6
The HSIC statistical measure of the dependence between two random variables. Suppose we have two random variables X, Y on probability spaces (X, Px), (J', Py), and their corresponding reproducing kernel Hilbert spaces (RKHSs) are Hx, Hy,, with measurable positive definite kernels Kı xXx + R. Ka x) R. The mean μx is an element of Hn, such that (ux, mean element on Hr: He, @ He, respectively. The cross-covariance operator CxY Hex + HHe is defined by Cxy == pxy - pxpy. Note that pxy - -Xxpy is an element of Hx, & Hxz, and il can be regarded as a linear operator from Hr, to Hay defined as (f,Cxy g)H,, = (uxY - - .fnrf.g))) fg) @Hag for any f Ha,s 9 E Hxz
Ex (X)) for all E Hu, Let my, denote the
The Hilbert Schmidt norm of the operator C: Haz + H., is defined as ||C|| EEeg(dn Cus)H,,, where {or) and {vi} are the orthogonal basis of Ha, and Hug, respectively. HSIC is defined by the Hilbert Schmidt norm of the cross-covariance operator,
HSIC Y) = ||Cxy
(6)
Since this equals to ||xx /x Ii,,, M7nng the value of the HSIC can be computed specifically as follows:
HSIC(X, Exyxy (x,'') YD) 2E 2Exr [R (x, X/
+Exx (X, (Y,Y'),
(7)
6

---

Published in Transactions on Machine Learning Research (05/2024)
where (X', Y') is independent from (x, Y) and follows the identical distribution.
Given finite samples {I1,, yI)...., bmn} following the joint distribution Pxy, one can give an empiri- cal estimator of HSIC as (r)K/HKHH)/((
1)², where 1, K2, H E Rm Kiē = Ki (T,, æy), K2g = K2 K2( 8y), Hij = 6G -1/m. In this work, we use the normalized HSIC (nHSIC), which is a normalized version of the HSTC. It. equals the chi-squared divergence between Pxy and Pr, which is called the chi-squared mutual information (see section D.3 The nHSIC is defined by the Hilbert Schmidt norm of the normalized
The empirical estimator for nHSIC has consistency (Fukumizu et al. 2007); however, the unbiased estimate is not calculated analytically. Regarding the choice of kernels and the estimate for nHSIC, we followed existing information bottleneck analyses that use nHSIC (Ma et al. 2020 Wang et al. 2021 2023b). RBF kernels were used for the input variables X and the intermediate variables Z to capture mon-lineat relationships. As for the class categorical variable Y, lincar kernel was applicd because it is discrete random variable (sec section F.1 for details).
cross covariane operator, nHSIC(X. Y) = x
4 HSIC Dynamics of E2E and Layer-wise Training
As observed in section 2 the input information required to solve the task might be lost in the early layers in the layer-wise training case. If I(X; 2) is not significantly increased in the initial layers, then according to equation 5] I(Y; Z) does not sufficiently improve either. Consequently, I(Y; Z) presumably fails to become large in the subsequent layers and eventually in the last. layer. This is not desirable behavior in terms of the information flow within the whole model. Additionally, it is uncertain whether an independent training of each layer induces a compression phase that reduces [(X; Z) while preserving I() in the final layer, which is suggested by the IB hypothesis.
Under these motivations, we aim to analyze the difference in the dynamics of I(X; Z) and I(Y; Z), i..., the information plane, between E2E training and layer-wise training. We first experiment on the smaller settings where the difference starts to appear and explain the mechanism behind the observation. We then move to the same scttings as section 2 in the subsequent sections. As mentioned in the previous section, while our discussion is centered around the mutual information, we experiment with nHSIC(X, Z) and nHSIC(Y, Z) to show the information plane.
4.1 HSIC Plane Dynamics for LeNet5 Setting
Firstly, we compared the dynamics of nHSIC Z) and nHSIC(Y, Z) between E2E training and layer-wise training using a simple model. Figure illustrates the results for the LeNet5 model (LeCun et al.) [998] for 100 epochs, and the training progression is depicted using a color bar and arrows. The LcNetō is composed of five layers, with the initial two convolutional layers and the latter three linear layers, and for the layer-wise training, linear auxiliary networks are adopted for each layer.
Figure 3a shows the results for the MNIST dataset, where it is clear that there is not so much difference in the HSIC plane dynamics between E2E training and layer-wise wise training. Furthermore, the actual test accuracy of the trained models showed little difference as in the caption of figure 3al This outcome suggests that for the MNIST problem setting, where the local optimization already works reasonably well, even the greedy optimization in each layer could learn representation Z that has a high correlation with X and Y. Next, Figure 3b) shows the results for the CIFAR10 dataset, which is the larger setting. In this setup, there is noticcable gap in the test accuracy between E2E and layer-wise training. Morcover, an intriguing difference was observed in the HSIC dynamics. In E2E training, both nHSIC(X, Z) and nHSIC(Y, Z) show a consistent increase in all layers as seen in the MNIST setting. However, in the first layer of layer- wise training, the HSIC values increase at first and subsequently turn to decrease. While this tendency was not observed in the later layers, the layer-wist wise training did not obtain representations with HSIC values as high as those in F2F learning.
7

---

Published in Transactions on Machine Learning Research (05/2024)
Layer 1
Layer 2
Layer
Layer
50-
Layer
8.70
Liyerweie
Layerwise
Layorwit
8:50
35-
80
epoch
8.25
30
9.0
8.00
73
opoch
1.7
74
1.5
1.2
20
20
10
nH5ICIX:Z)
(HSIC(X:Z)
nHSIC(X:Z)
nHSIC(X:Z)
(a) Lenet5 models trained on MNIST dataset. Test accuracy 100 epoch: 98.9% (E2F), 98.9% (Tayer-wise).
Layer
Layer
Layer
Layer
tzt
a--
90
80
epoch
35
80
-00
epoch
Epoch
3.
2.5
2.0
20
20
2.0
100
300
120
360
300
120
90
10
nHSICIX:Z)
nHSICIXZZ)
nHSICOX:2)
nHSIC(X:2))
(b) Lenet5 models trained CIFAR1O dataset. Test accuracy 100 epoch: 62.5% (E2E), 59.2% (Layer-wise).
Figure 3: HSIC plane dynamics of LeNet5 model. The color gradation shows the progress of training, i.e., the number of epochs. Inverted triangles with blue-yellow-based colormap denote layer-wise training, whereas circles with red-based colormap show E2E training.
4.2 Example of Greedy Optimization Failure
If the HSIC value achieved in the final layer accounts for the performance discrepancy between E2E and layer-wise training, understanding how this difference arises could shed light on the shortcomings of layer-wise training and further highlight the strength of E2E training. We observed that the HSIC value for layer-wisc training is lower even in the initial layer and turns into an intriguing decrease. Considering that each layer is provided the class information explicitly the layer-wiss wise training, such an HSIC behavior is not necessarily intuitive. this section, we demonstrate how such a phenomenon might arise using a toy example. When we focus on the initial representation Z1, there is no difference in the incoming inputs between E2E and layer-wise training. The distinction lies in the distance to the loss evaluation: in layer-wise training, it occurs after one linear layer, while in E2E training, it happens after the subsequent backbone model.
We first see how the optimization of cross-entropy loss is related to mutual information with label Y. From the non-nogativity of KT. divergence and the Markov chain Y + X + Z, we obtain the lower bound of the mutual information as follows (see section for detail):
I(Y: Z) Ep(y,)) [pp()))) [log ( 2)]] H(Y),
where g(ylz) is an arbitrary variational distribution and H(Y ) is the entropy of the label distribution. The bound is tight when g(ylz) = p(ylz) holds. The p(z[z) is determined by the backbone model up to the layer, and in the layer-wise training, the variational distribution ((ylz) modeled by the auxiliary classifier. Optimizing the cross-entropy loss calculated from its output equivalent to maximizing the lower bound of equation The layer-wise training, which can only train the final layer of p(z|æ) and linear classifier q could not significantly maximize the lower bound due to the insufficient capacity of the model to represent the relationship between X and Y. Therefore, learning representations with a high I(Y; 2) value becomes challenging. Furthermore, as will be discussed next, the local optimization in this scenario could lead to inappropriate learning of the backbone model for the later layers.
Let us consider the toy data model in figure 4 as an example of p(y]x) having non-linear relationships. Note that in the E2E training setting, which allows non-lineal transforms, perfect class separation possible based the length of the radius We consider binary classification setup where R2 follows distribution
8

---

Published in Transactions on Machine Learning Research (05/2024)
Original Data
Reprezentation Z
Layer Linear Head
Layer 1, MLP Head
Liyervaite
Liyenwite
MHSIC/WZZ
epoch
epoch
60
-00
Epoch
40
300
110
130
312
100
130
120
330
nHSICC(X:
nHSICIX:Z)
Figure 4: Toy data model. Left: Original data. Class 1: r = 3.0 and von-Mises(1)) (....). Class -1: r = 1.0 and 0 ~ von-Mises(r, 2.0). A small noise is added
Figure 5: HSIC plane dynamics of the initial layer of Lenet5 models trained on CIFARIO dataset. The size of auxiliary networks different; the left shows the results for the linear head (same as figure 3b). and the right shows the results for the two-layer MLP head.
the data. Right: Representa- tion 1 when W = - (v19 ) The x-axis is set to the direction of auxiliary classifier v.
illustrated in figure 4 The representation of the first layer Z1 is determined as: 21 = f1(x) (()=)==0 Wa, where W E 2R is constrained by || S C; c is a constant. Additionally, the auxiliary classifier is lincar as g1 (21) = v z1, where ||v||2 = 1, and the class is predicted based on the output sign. In the layer-wise training, the binary cross-entropy is optimized based on the output of gi. Adding the constraint term on W, the loss function for the first layer is follows:
(w v) = 1 Σ log (1 + exp (-2y,0 T Wx:)) + W/[W
(9)
where À > 0 is the hyperparameter. The gradients for and W are
dL Ov
dL OW
WaW,
x y,o., + 2XW.
(10)
The data symmetry implies that Eneg== x; points in the positive direction of the x-axis, while Em. x points in the negative direction of the x-axis. Consequently, E, y,æ, aligns with the positive direction of the x-axis. As result, based on the gradient update equation TO the second column of W, W1,2 and W2.,, converges to zero due to norm regularization. Figure 4 illustrates the process of rank collapse during training. In this figure, the basis is changed so that the auxiliary classifier v aligns with the positive direction of the x-axis. The first column of W points in the same direction as in this figure, which follows from the gradient update equation 10 for small À intuitively, but the alignment is not necessarily required for this discussion. The imporlant point here is the rank collapse of W, leading to representations that more closely adhere to the x-axis. After the collapse, when observing points near the origin on the x-axis, we cannot accurately distinguish their corresponding labels original data points. This situation results in a reduction in I(Y; Z) and I(X; Z) because the uncertainty in Y and X arises after observing the realization of Z.
On the basis of this analysis, we hypothesize the following two steps could occur in the increasing and decreasing phases of HSIC in figure 3b} First, in the early phase of training, the representation is trained to be correlated with the label. Next, as training progresses, a decrease in mutual information similar to that in the toy data example occurs. In actual computing, decline happens when representation becomes indistinguishable in terms of the computational precision of the floating point even before a complete collapse of representation. Although this analysis provides insights only into linear auxiliary head case, there exists performance gap between E2E and layer-wise wise training even with two-layer MLP seen in section [ Figure 5 illustrates the HSIC dynamics of the first layer when using two-layer MLP instcad of lincar auxiliary classifier, under the same LeNet5 and CIFAR10 setup as in the previous section. This change effectively resolves the decrease in the HSIC, leading to a similar increasing trend as in E2E training. However, the fact that E2E training achieves better test accuracy in the setting of table [ suggests that the two-layer MLP might be insufficient as model for q[g|z), potentially due to its small size or not being a convolutional network, leading to such a collapse of input information with greedy optimization.
9

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
Layer
8.3
Lapenwss
85
Leye
epoch
0.50
80
80
epoch
0.1
epoch
82-
8.3
PHS
7.8
epoch
1.75
10.b0bog pinp
Epoch
7.2
20
-20
139
7.0
6.15
170
580.0
382.5
181-3
370.0
320
220
140
nHSICIX:Z)
nHSIC((:Z)
nHSICIX:2)
NHSIC|X:Z)
Figure HSIC plane dynamics of ResNet18 model trained on CIFAR1O. Left: Middle layers. Right: Output layers. The color gradation shows the progress of training. Inverted triangles with blue-yellow- based colormap denote layer-wise training, whereas circles with a red-based colormap show E2E training.
4.3 HSIC Plane Dynamics for ResNet Setting
Thus far, we have investigated relatively small models. In this section, we present the HSIC dynamics of ResNet. Figure o shows the results for the CIFAR10 dataset. While ResNet18 consists of nine middle representations from the first. to the penultimate layer, the middle layers and output layers are shown in the figure. Please refer to figure 15 in the appendix for all layers' results. In the case of LeNet5 settings, both E2E and the layer-wise training generally showed an increase in HSIC values during training, while in the ResNet setting, both approaches demonstrated a decrease of nHSIC(X, Z) and nHSIC(Y, Z) in the middle layers of the network. However, they do not have similar HSIC plane dynamics for every layer. In the output-side layers, E2E training showed a slight compression phase of nHSIC(X, Z) while maintaining a high nHSIC(Y, Z), aligning with the compression phase suggested by the IB theory. In contrast, the layer- approach, both nHSIC(X, Z) and nHSIC((Y Z) decreased as in the intermediate layers. Note that the value of nHSIC((Y Z) itself does not necessarily imply linear separability; therefore, the actual test accuracy did not deteriorate during this HSIC decreasing phase. On the basis of the findings, the advantages of E2E learning over layer-wise training can be attributed to the following two main factors. The results of other modols are presented in section F.4 and similar behavior was obscrved. In particular, the IB behavior in the final layer was more distinct in ResNet50 and VGG11 with batch normalization.
Efficient learning of Z with high HSIC. E2E training can acquire the representation with high HSIC at the very early phase of training. This suggests that learning with prop-propngttion is more efficient from the perspective of HSIC than directly providing class information to each layer, as in the layer-wise training.
Layer-role differentiation. While both E2E and layer-wise approaches exhibit the HSIC decrease phase in the middle layers, there is a difference in the mechanism behind this phenomennn. In the layer-wise training, due to repeating similar optimization at each layer, even the output layers showed the same HSIC behaviors. However, in the E2E training, the reduction in HSIC occurs in the middle layers while maintaining high nHSIC(Y, Z) in the output layers. E2E can compress the middle layers while guaranteeing a certain level of Zy) from equation 4. In contrast, layer-wise training lacks knowledge of the later layers during optimization, and each layer's optimization limits the upper bound of [(Y; Zt). These observations imply that the advantages of being E2E, i.c., enabling interaction among layers, is having different information- theoretical roles among layers and acquiring information bottleneck representation at the last layer.
4.4 Controlling HSIC in Layer-wise Training
The obscrvations in the preceding sections have indicated that the layer-wise training can cause a reduction phase in HSIC through greedy optimization at each layer, potentially hindering the information propagation to subsequent layers. In this section, we explore the possibility that enhancing nHSIC(X, Z) during local optimization enables layer-wise training to achieve a model performance comparable to E2E training. Wang et al. (2020) incorporated the reconstruction error, which appears in the variational lower bound of [(X; Z), into the local error to mitigate the reduction of T(X;Z) due to local optimization. Although we differ in
10

---

Published in Transactions on Machine Learning Research (05/2024)
Layer 4
Layer 5
Layer
Layer 9
8.3
Layerwile
Layer wice (inoc)
X
Layor wise (inesc)
Layerstte
Layer wise (nesic)
Layer-tte
0.2
Layer wise tretic
Layerwitee
5.-
(HSIC(F22)
IHSIC
epoch
epoch
8.1-
7.5
Epoch
40
40
8.0
0.2
7.3
7.0
319 120
100
nHSICIX:Z)
nHSIC((::)
nHSIC((:Z)
nHSIC(x.Z)
Figure 7: HSIC plane dynamics for layer-wise training with HSIC augmenting term; À 0.05. ResNet18 is trained on CIFAR1O, which the same setting as figure G Left: Middle layers. Right: Output layers. The color gradation shows the progress of training. Inverted triangles with blue-ycllow-based colormap denote layer-wise training in the original setting as baseline, whereas cross marks with a red-yellow-based colormap show training with HSIC augmenting term.
the training of increasing nHSIC(X, Z), our motivations align with it. This study investigates whether such operations fundamentally address the issues in layer-wise training.
We explore the model performance trained with the local loss with an HSIC augmenting term. Let X E Rdxd be (Zi,..., Em)) and Zi E IR mxd be the I-th layer's output. We adopted the following loss in each layer:
Lusic, -
(11)
where À > is a positive hyperparameter to define the regularization strength. In practice, computing the empirical nHSTC for the entire training data requires ((m²) memory consumption, so the estimated values within the mini-batch are used for training.
Figure shows the results for layer-wise training with the HSIC augmenting term when À set 0.05. While the original layer-wise training exhibited an HSIC decreasing phase, both nHSIC(X, 2) and nHSIC(Y, Z) increased throughout the training without decreasing. Although these HSIC plane dynamics were expected, the actual test accuracy turned out to be worse. Since identity mapping, for example, does not learn useful representations but preserves information, prioritizing information preservation might. have hindered the learning of a useful representation here. The results with smallor À are provided in section F.4 but we did not observe any performance improvement from the original setting. Our findings show that 1) the initial HSIC values remained unchanged from the original setting, failing to achieve values as high those with E2E training, and 2) nHSIC values were uniformly improved in every layer without the middle layers' compression and the information bottleneck behavior in the final layer. The learning dynamics are essentially unchanged from the original layer-wise setting and fail to achieve cooperative interaction among the layers.
5 Compression in Middle Layers from Geometric Perspective
E2E training has an information compression in the middle layers, exhibiting behavior distinet from that of the layers on the output side. However, the benefits of this intermediate compression in F2E training remain subject of discussion. One reason would be that, based on the previous observations, the compression in the middle layers leads to the information bottleneck behavior in the final layer: a better representation from an information--heerrtiic perspective. In this section, we discuss the advantages of middle compression from a geometric standpoint. Frosst et al. (2019) stated that entangling representations of different classes in the hidden layers contribute to the final classification performance. Although mutual information does not specify the geometric arrangement of the representation space (Tschannen et al.) 2020), we have the following result for the value of HSIC(Z, Y) and soft nearest neighbor loss in Frosst. al. (2019)
Theorem 1 (Informal version of Theorem 4. Suppose the representation z is bounded as || S M for some constant M > 0. If the RBF kernel k(0, w) exp (-1|p- w0ll/(200)) and the linear kernel 1(0, w) TTw
11

---

Published in Transactions on Machine Learning Research (05/2024)
are used for Z and Y respectively, then we have
2M r²
soft nearest neighbor loss logc- cexp
HSIC(Y, 2).
(12)
This theorem implies that reducing HSIC(Y, Z) leads to an increase in the lower bound of soft nearest neighbor loss, leading to a higher performance of the final layer according to Frosst et al. (2019). Please
refer to section E.2 in the appendix for proofs and more detailed interpretation.
6 Related Work
Information and HSIC Bottleneck Information bottleneck (Tishby et al. 2000 Tishby & Zaslavsky
2015
Saxe ct al.)
2018) is one of the concopts for dissecting deep learning. It has been studied in the
context of the ana ysis the learning dynamics through the information plane (Goldfeld 2019 Geiger
2021
Lorenzen et al. 2022 Adilova al. 2023). tte relationship between generalization of neural network
(Achille & Soatto 2018 Chelombiev al. 2019 Kawaguchi al.
2023), , and the objective of self-supervised
reprosentation learning (Sridharan & Kakade 2008 Federici et al. 2020 The HSIC bottleneck (Ma et al.
2020 Pogodin& & Latham 2020: Wang et. al., 2021: Jian et al. 2022 Guo et al. 2023 Wang el al. 2023b) is
inspired by the information bottleneck but uses HSIC instead of mutual information because of the simplicity of estimation. In our research, we focus on information plane analysis rather than designing loss functions based on HSIC bottleneck; in this sense, we are influenced by the plots presented in Wang et al. 2021
Layer-wise Analysis While there are prior studies on the differing roles of the layers and compression of representation (Darlow & Storkey 2020 Peer et al., 2022 Zhang et al. 2022 Chen et al. 2023 Masarczyk et al.) 2024), to the best of our knowledge, there has not been any research leveraging the characteristic of laser-wise training, which allows specifying roles for each layer, to reconsider the advantages of E2E training. One study conducted in parallel with ours (Wang et al., 2023a) shares a perspective in analyzing representation learning layer by layer. However, work differs in that we put emphasis the information- thcoretic interactions between layers for E2E training, while they analyze the geometric representations of classes using deep linear networks.
7 Discussion
7.1
Forward-Forward Algorithm
One of the merits of layer-wise training the possibility of adopting divide-and-conquer approach, which enables us to specify the role of each layer specifically, leading to more interpretable models and parallel efficient training. However, as of this moment, layer-wise training imposes the same objective function layers, making the role of each layer uniform and negating the importance of stacking layers. We focus on the forward-forward algorithm (Hinton 2022) as potential solution to this challenge and investigate its possibility in section C Unlike the divide-and-conquer approach, it trains layers in sequential way and forces each layer to learn different features from preceding layers while adopting the same objective function among the layers. This process encourages distinct roles for each layer. While there is a problem in terms of diminishing class information as the layers deepen (see section Ch, it presents an intriguing approach.
7.2 Limitation and Future Direction
In this study, we observed the information compression in the middle layers in E2E training and discussed its benefits from the information bottleneck perspective and the geometric perspective. However, it does not explain why this compression occurs. Furthermore, we do not investigate the merits of backpropagation. The layer-role differentiation obscrved E2E training could potentially be obtained by solving the credit assignment problem for each neuron via backpropagation. However, even with biologically plausible methods, if they can address the credit assignment problem appropriately and achieve interaction between layers, there
12

---

Published in Transactions on Machine Learning Research (05/2024)
is potential to achieve the advantages in E2E training observed in our work. This paper provides criteria that backpropagation alternatives should meet, and exploring this potential is left for future work.
8 Conclusion
This study explored the advantages of E2E training through the comparison with layer-wise training. Our information-thcorrtiic analysis using HSIC shows that E2E training propagates efficiently imput information and assigns diverse roles to the layers, leading to the information bottleneck representation at the final layer. The information compression of intermediate layers observed in this study could offer an intriguing perspective for understanding neural network behavior in the context of the information bottleneck principle. We leave the investigation to future work of the information compression mechanisms of the middle layers and the influence of layer interaction on the last representation dynamics.
References
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 111:199447 1980, 2018.
Linara Adilova, Bernhard C Geiger, and Asja Fischer. Information plane analysis for dropout neu- ral networks. In The Eleventh International Conference on Learning Representations, 2023. URL
https://openrevieew. http:///wwww..eeeo.ooooooooeaaaa
Guillaunc Alain and Yoshua Bengio. Understanding intermediate layers using linear classificr probes. arXiv preprint arXiv:1610.0164, 2016.
Rana Ali Amjad and Bernhard Geiger. Learning representations for neural net based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, (2(9):222-222333 2019.
Pierre Baldi, Peter Sadowski, and Zhiqin Lu. Learning in the machine: the symmetrics of the deep learning channel. Neural Networks, 95, 09 2017. doi: 2.001//j.neunet.221..000
David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assossing the scalability of bioloically--ootiateed deep learning algorithms and architectures. Advances in neural information processing systems, 31, 2018.
Atilm Güneş Baydin, Barak A Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation, arXiv preprint arXiv:2202.06587 2022.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning scale to ima- genet. In International conference machine learning, pp. 583 593. PMLR, 2019.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. International Conference on Machine Learning, pp. 736-745. PMLR, 2020.
Yoshua Bengio. How aulo-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906, 2014.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, 2006.
Mario Beraha, Alberto Maria Metelli, Mattco Papini, Andrea Tirinzoni, and Marcello Restelli. Feature selection via mutual information: New theoretical insights. In 2019 international joint conference on neural networks (IJCNN), pp. 1-9. IEFE, 2019.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2299146,11 2020.
Ivan Chelombiev, Conor Houghton, and Cian O'Donnell. Adaptive estimators show information compression in deep neural networks. In International Conference on Learning Representations, 2019. URL https:
//openrevies http://www....e.ooorri/ddddeeeee
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublincar memory cost. arXiv preprint arXiv:1604.0617, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMT.R, 2020.
Yixiong Chen, Alan Yuille, and Zongwei Zhou. Which layer learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks. In The Eleventh International Conference Learning Representations, 2023. URL http:p/////ggeeeeeeeeeeee/rrrrrr wttp://oorrmmmdics.//00111..010.
Francis Crick. The recent excitement about neural networks. Nature, 337(6203)::29 132, 1989.
Luke Nicholas Darlow and Amos Storkey. What information does resnet compress? arXiv preprint +rXiv:2003.06255 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth Transformers for image recognition scale. arXiv preprint arXiv:2010.1192) 2020.
16x16 words:
Shiyu Duan and Jose C Principe. Training deep architectures without end-to-end backpropagation: A survey on the provably optimal methods. IEEE Computational Intelligence Magazine, 17(4):39 51, 2022.
Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. Normalized mutual information feature selection. IEEE Transactions neural networks, 20(2):189 201, 2009.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akala. Learning robust represen- tations via multi-view information bottleneck. In International Conference on Learning Representations, 2020. URL httpps://pperreveeee.eeee/eeeeee
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In International conference on machine learning, pp. 2012 2020. PMLR, 2019.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. Advances in neural information processing systems, 20, 2007.
Kenji Fukumizu, Francis R Bach, and Michael I Jordan. Kernel dimension reduction in regression. 2009.
Bernhard C Geiger. On information plane analyses of neural network classifiers a review. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review, 70(3):419-433) 2002.
Ziv Goldfeld. Estimating information flow in deep neural networks. In International Conference on Machine Learning, 2019.
Aidan N Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, and Yarin Gal. Interlocking backpropagation: Improving depthwise model-parallclism. The Journal of Machine Learning Rescarch, 23 (1):7714-7741, 2022.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 112.v21412.6572, 2014.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring statistical depen- dence with hilbert-schmidc norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient back- propagation through time. Advances in neural information processing systems, 29, 2016.
Song Guo, Lei Zhang, Xiawu Zheng, Yan Wang, Yuchao Li, Fei Chao, Chenglin Wu, Shengchuan Zhang, and Rongrong Ji. Automatic network pruning via hilbert-schmidtt independence criterion lasso under information bottleneck principle. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 17458 17469, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton. The haward-forward algorithm: Some
preliminary investigations. arXiv preprint
arXiv:2212.13345, 2022.
Geoffrey Hinton. S3 E9 Geoff Hinton, the "Godfather of AI", quits Google warn of Al risks (Host: Pieter Abbeel), 2023. URL https::/www.youtuuiii wot//vataalv=
Geoffrey E Hinton, Simon Osindero, and Yee Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527 1554, 2006.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoic-Marchildonn Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06677 2018.
Furong Huang, Jordan Ash, John Langford, and Robert Schapire. Learning deep resnet blocks sequentially using boosting theory. In International Conference on Machine Learning, pp. 2058 2067. PMLR, 2018.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lec, Jiquan Ngiam, Quoe V Le, Yonghui Wu, el al. Gpipe: Efficient training of giant neural nel using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Tong Jian, Zifeng Wang, Yanzhi Wang, Jennifer Dy, and Stratis loannidis. Pruning adversarially robust neural networks without adversarial examples. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 993 998. IEFE, 2022.
Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without foedback. In The Eleventh International Conference on Learning Representations, 2023. URL
https://openrevier net/forummiddd@gdtit http:///pprreeiiie.eeee/o//////a _Rjj
Yu- Wei Kao and Hung-Hsun Hsuan Chen. Associated learning: Decomposing end-to-end backpropagation based on autoencoders and target propagation. Neural Computation, 33(1):174-198, 2021.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? arXiv preprint arXiv:2305.1 18887, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot,
Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 18:18661-18677 2020.
Diederik P Kingma and Jimmy Ba. Adam: A mothod for stochastic optimization. arxiv preprint arXiv:1412.69%% 2014.
Adam Kohan, Edward A Rietman, and Hava T Siegelmann. Signal propagation: The framework for learning and inference in forward pass. TEEE Transactions Neural Networks and Learning Systems, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 7Xiv:1886.0880 2018.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint
arXiv:1404.5097, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Mandar Kulkarni and Shirish Karande. Layer-wise training of deep networks using kernel similarity. arXiv preprint arXiv:1703.07115, 2017.
Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyamn. A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation. Advances in Neural Information Processing Systems, 32, 2019.
Yann LeCun, Bernhard Boser, John $ Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwrilten xip code recognition. Neural computation, 1(4):541-553 5511, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proccodings of the IEFE, 86(11):2278-25) 2324, 1998.
Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases: Europcan Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part 15, pp. 498-515. Springer, 2015.
Yazhe Li, Roman Pogodin, Danica J Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems, 14:15543-15555 2021.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370-403, 2021.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:1411.0241 2014.
Stephan Sloth Lorenzen, Christian Tgel, and Mads Nielsen. Information bottleneck: Exact analysis of (quantized) neural networks. In International Conference on Learning Representations, 2022. URL https://openreviiee http://opperrrrioooonootttrrrrr 7
ott. KF9DZQQrUOw
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting an end end-to-end: Gsadient-isolated learning of representations. Advances in neural information processing systems, 32, 2019.
Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back- propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5085-5092 2020.
Enrique S. Marquez, Jonathon S. Hare, and Mahesan Niranjan. Deep cascade learning. IEEE Transactions on Noural Notworks and Learning Systems, 29(11):5475-5483 2018. doi: 10.1109 C000//NNSS..22. 2018 2805098.
Wojeiech Masarczyk, Mateusz Ostaszowski, Ehsan Imani, Razvan Pascanu, Piotr Miloš, and Tomasz Trzcin- ski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1765 1773, 2017.
Timolcon Moraitis, Dmitry Toichkin, Adrien Journé, Yansong Chua, and Qinghai Guo. Softhebb: Bayesian inference in unsupervised hebbian soft tinnerltako-all networks. Neuromorphic Computing and Engineering, 2(4):044017, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Alan Mosca and George D Magoulas. Deep incremental boosting. arXiv preprint arXiv:1708.037000 2017.
Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:608, 2018.
Arild Nøkland. Direct leedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International conference machine learning, pp. 4839-4855 PMLR, 2019.
OpenAL. Gpt-4 technical report, 2023.
David Peer, Bart Keulen, Sebastian Stabinger, Justus Piater, and Antonio Rodriguez-sanchez. Improving the trainability of deep neural networks through layerwisc batch-cntropy regularization. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL http:////ppprreeeeeeeeeeeee/rrre
LJoh15DnZf
Roman Pogodin and Peter Latham. Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks. Advances in Neural Information Processing Systems, 33:7296-7300 2020.
David E Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, 323(6068)::33 536, 1986.
Ruslan Salakhutdinov and Geolf Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In Artificial intelligence and statistics, pp. 412-419. PMLR, 2007.
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018. URL
Shoaib Ahmed Siddiqui, David Krueger, Yann LeCun, and Stéphane Deny. Blockwise self-supervised learning al scale. arXiv preprimt arXiv:2302.01647, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556. 2014.
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-viers view learning. 2008
Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, characteristic kernels
and rkhs embedding of measures. Journal of Machine Learning Research, 12((0))2389-22 24100, 2011. URL http://jmie httpp://jmr..rgg// hwwpla/rorg/ppppppeeee ott/ppppers////2//rrrrruubbbbbaa
1a. html
Joan Stiles and Terry L Jernigan. The basics of brain development. Neuropsychology review, 20(4):327 348, 2010.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual informa- tion maximization for representation learning. In International Conference on Learning Representations, 2020. URL http::////gaaeeeeeeeeeeee//////p netr/
17

---

Published in Transactions on Machine Learning Research (05/2024)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
Fan Wang, Hao Tian, Haoyi Xiong, Hua Wu, Jie Fu, Yang Cao, Yu Kang, and Haifeng Wang. Evolving docomposed plasticity rules for information-bootlenddee meta-learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:
Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzan,, Wei Hu, and Qing Qu. Understand- ing deep representation learning via layerwise feature compression and discrimination. arXiv preprint arXiv:2311.02960 2023a.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: alternative to end-to-end training. In International Conference on Learning Representations, 2020.
Zifeng Wang, Tong Jian, Aria Masoomi, Stratis Ioannidis, and Jennifer Dy. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. Advances in Neural Information Processing Systems, 34:586 597, 2021.
Zifeng Wang, Zheng Zhan, Yifan Gong, Yucai Shao, Stratis loannidis, Yanzhi Wang, and Jennifer Dy. Dualhsic: Hsic bsttebootleeeck and alignment for continual learning. arXiv preprint arXiv:2305.00380 2023b.
Dennis YH Wu, Dinan Lin, Vincent Chen, and Hung-Hsuan Chen. Associated learning: an alternative to end-to-end backpropagation that works on enn, rnn, and transformer. In International Conference on Learning Representations, 2021.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 33:11142 11153, 2020.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? The Journal of Machine Learning Rescarch,
2022.
18

---

Published in Transactions on Machine Learning Research (05/2024)
A
Related Work
In the main text, related work for information bottleneck and HSIC-based analysis is presented due to space constraints. Here, we introduce related studies addressing the drawbacks of F22 training and investigating layer-wise training.
Problems in backpropagation. Backpropagation technique (Rumelhart al. 1986 LeCun et al., 1989]
is foundation of the training methods of current machine learning. While this mechanism has under- pinned recent successes in machine learning, there are still challenges in the following three perspectives: computational problem, biological problem, and interpretability problem (Duan & Principe 2022).
First, it has computational problems with memory usage and the difficulty of parallel computing. The outputs of intermediate layers are necessary for computing the derivative of the earlier layers; therefore, they should be put on the memory besides the parameter information of the neural network. This can oppress memory usage limitations and lead to the need to compromise the network size and the quality of training data. To alleviate this problem, a check-pointing technique that recomputes the update from chockpoints in the backward step (Chen et al., 2016 Gruslys et al. 2016 Kusumoto et al. 2019) and network compression techniques, such as pruning and quantization, are proposed (Krishnamoorthi 2018 Blalock et all 2020; Liang et al.] 2021); however, they are still the trade-off between memory usage and training time or test accuracy. In addition, there is an update locking problem, which prevents updating the weights until the forward and backward path of the rownstreem modules is completely completed, preventing the model parallelism [Krizhevsky 2014 Huang et al. 2019 Gomez et al. 2022).
Second, training with backpropagation is questioned from biologically plausible perspective. Biological learning rules are considered to be local in terms of both space and time, and there is gap from the backpropagation because it satisfies neither of them (Crick) 1989 Baldi et al. 2017]. Specifically, if the actual brain adopts the backpropagation-like learning style, we observe various gaps, for example, forward and backward propagation must alternate, the error must be propagated throughout the entire model, the update path must be symmetric because the same weights are used in backward propagation as in forward pass, and continuous values must be represented at the binary value synapses. A Hebbian neural network is
one of the techniques to handle the problem of global learning rules (Moraitis et al. 2022 Wang et al. 2022:
Journé et al. 2023). Hebbian- like learning methods enable local learning by increasing the weight bet ween presynaptic neurons and postsynaptic neurons when both are activated. Target propagation (Bengio 2014
2015 Bartunov et al. 2018) trains each block to regress the target assigned for each block. The key idea of target propagation is backpropagating the target with the approximated inverse of the block instead of backpropagating error with the chain rule of the gradient. This inverted target is generated with the auxiliary networks; therefore, target propagation can alleviate the symmetric weight problem of backpropagation and the problem of gradient instability. A different approach that solves the problem of weight, symmetry is feedback alignment (FA) (Lillicrap et al. 2014), which uses the fixed random weight matrix to transport the updated information to the earlier layers. FA alleviated the difficulty of weight symmetry but still has the backward locking problem that the parameter weights cannot be updated until the backward pass has been completed. Direct foedback alignment (DFA) (Nokland 2016) passos the gradient information directly to each layer from the output layer so that the weight update can be completed simultaneously.
Lee el al.
Finally, training models with backpropagation have a problem of interpretability. The trained network with E2E learning is black box and difficult to understand its behavior, for example, unexpected performance deterioration is observed in certain situations and the trained model is difficult to be in under control GGoodfelow et al. 2014 Moosavi-Dezfooll et al.] 2017). In particular, the intermediate layer's behavior is controlled only through the outputs of downstream layers, and these behaviors are difficult to understand and specify. This fact is one of the reasons that makes understanding the benefits of backpropagation difficult.
Kayer-wise training. Layer-wise training was originally studied in the context of getting good initial- ization to train multi-layer neural networks successfully (Hinton et al., 2006 Bengio et al.] (006). With advancements in the model architecture and the normalization techniques, layer- wise training has gained attention as one of the training methods to address the aforementioned problems with backpropagation. The
19

---

Published in Transactions on Machine Learning Research (05/2024)
in con- trast, the similarity-based loss is used (Kulkarni & Karande, 2017 Nøkland & Fidnes 2019 Siddiqui et al. 2023). The essential problem of layer-wise training is that the previous layer cannot use the information of the layer behind it at all. Under the sctting that backpropagation within a local block is allowed, Xiong et al. (2020) has lackled this problem so that the information of the subsequent layer can gradually flow to the previous layer by training by shifting one layer at a time, using two pairs of layers as one block. Gomez et al. (0022) extends this idea of "overlapping local updates" in a general way, proposing a learning method that balances the high parallelism of layer-wise training with the high prediction accuracy of E2E learning.
classification-basse loss is adopted in each layer (Mostafa et al. 2018 Belilovsky et al. 2019
2020
Signal propagation (Kohan et al. 2023 has a shared characteristic with similarity-base layer-wise training in that it projects the input and class information into the same representation space and compares them in this space. Associated learning (AL.) (Kao & Chen 2021 Wu et al., 2021] is similar to signal propagation in that it forwards inputs and class labels in the same direction and compares them. The difference with signal propagation is that AL does not share the parameters of the networks used for forwarding inputs and class labels. Recently, the Forward-Forward algorithm has been proposed by Hinton and attracted attention Hinton 2022p. This learning method shares idcas with signal propagation and AL, but it can enforce the subsequent layers to learn the different features from the previous one. We investigated this possibility in section in the appendix.
Another direction of this research focuses on the similarity between layer-wise training and ensemble learning (Mosca & Magoulas, 2017 Huang et al.) 2018). Huang et al. (0018) proposed the incremental learning methods of ResNet (He et al. 2016) based on research that ResNet can be interpreted as an ensemble of many shallow networks of different lengths (Veit et al. 2016). They also gave a theoretical analysis of this method based on generalization bound for boosting algorithms.
Wang et al. (2000) adds reconstruction error term to the local objective function from the information- theoretic perspective. They hypothesized that layer-wise training aggressively discards the input-related information, which leads to the performance drop from the E2F learning. Our study has been inspired by their work, yet differs in that we employ the information-theoreticc tools for analyzing middle layers from the perspective of E2E training, rather than proposing a layer-wisc method. Additionally, while their work asserted the effect of increasing T(X; Z) in the layer-wise training, we argue that there still exists a performance gap compared with E2E training in terms of the diversity of roles across layers (see section 4.4
B Further Descriptions for Layer-wise Training
B.1 Details of Training
Size of local module. The distinction between the E2E training and layer-wise training allows the con- tinuous middle choice by changing the range granularity of gradient propagation. In various recent studies local loss training, the entire model is divided into multiple blocks and performs backpropagation within those blocks, instead of strictly decomposed up to the layer level. This approach, which could be termed "block-wise training", can balance the trade-offs between the high accuracy of E2E training and the compu- tational efficiency of layer-wise training, such as memory consumption and parallel computational efficiency. This paper, however, docs not aim to propose a practical training method; instead, it primarily focuses on analyzing the reason why E2E training achieves higher accuracy compared to layer-wise training, without considering block-wise training.
Sequential vs simultaneous training. In the layer- wise training, there are two major training methods: one scquentially trains layers, adding new layer on top of fully trained previous modules (Belilovsky et al. 2019 Huang et al.) 2018 Marquez el al., 2018);; and the other method, for the given model architecture, simultaneously trains each layer based on the local loss of each module in a single forward pass. The soquential training can dynamically decide the model depth depending on the current performance of the validation set; however, it takes more time and achieves lower performance than when each layer is trained simultaneously (Löwe et al. 2019 Wang et al. 2020: Siddiqui et al. 2023). They show that the later modules
20

---

Published in Transactions on Machine Learning Research (05/2024)
have fallen into overfitting, and this be attributed to the regularizing effect from the noisy input from not fully trained earlier modules in the simultaneous training. Additionally, the main focus of this study lies in its comparison with E2E learning; therefore, we primarily consider the setting of simultaneous training, which forward inputs throughout the entire model as E2E training does. As for the sequential layer-wise training, refer to section F.4.3 the appendix for experimental results.
B.2 Liser-wise Training Methods
We conducted experiments in section 2.2 to confirm the performance gap between E2E training and the various layer-wise training methods. In this section, we introduce the loss functions and methods used in the experiments. These methods correspond to the names listed in the table and table 4
Cross-Entrope Loss (CE). The most naive approach is to apply the same cross-entrope loss (CE) as those used in E2E lcarning. In this case, the class probabilitics are calculated using a linear or multi-layer projection head 9i from the intermediate representations, so the local loss is calculated as
exp sxphi
(.)
1 (y: k) log Σj=1 exp exp h1,s3
(13)
i=1
Am
where 1 is an indicator function. Let us remind that the his E R. is the output of I-th layer's auxiliary network for input 24, and it is calculated as hi, = (gi o fi) (StopGrad((z-1..).
Supervised Contrastive Loss (SupCon). As case of optimizing the similarity- base loss at each layer, supervised contrastive (SupCon) loss (Khosla et al. 2020 is used in experiment. The intermediate reprosentations of an input pair are brought closer together if they belong to the same class and moved apart if they come from different classes. Before training, the data augmentation is performed to assert there is at least one sample of the same class in the batch. Let N be the size of the mini- batch; the new mini-bated of size 2NV is created with augmented samples T2--1, Z2z from the original æ; In addition, let P(t) C [2.]] be the set of indices of samples that belong to the same class as Ii, except for itself. The loss is as follows:
exp [1 hTTpppp/TT
Limi (0))
Σ
Σ log PEP(()
(14)
2N
P(1)
EMER2NIM exp (ITTT
where 7 is hyperparameter for softmax temperature, and Li,mini denotes the loss for the mini-batch. In this case, the label information is not explicitly forwarded, but the class information is given through the other training imput belonging to the same class.
Similarity Matching Loss (Sim). We use the similarity matching loss from Nøkland & Eidnes (2019) as similarity-based local loss. While they showed that combining the similarity- loss and classification- based loss yields layer-wise trained model with better performance, since the purpose of this study is the analysis of the properties of the trained model, only the similarity-based one is considered. Given outputs after the auxiliary networks H; = (hi,,,.... h..m) E mmxd and the one-hot encoded label information Y (y...... T the loss calculated as:
(()))) sim (Hi) sim (Yi) 1
(15)
where sim(Hi) measures the similarities between each rows of Hy. Specifically, each element of sim( Hi) is
hT
sim (H:). h 1
(16)
where hys is mean-centered h1s We caleulate sim(Y)) in the similar manner.
While both this similarity matching loss and supervised contrastive loss are the similarity- based local loss, one major difference is that the auxiliary network g composed of one or more convolution layers instead of linear or MLP (Nokland & Eidnes, 2019). The output is obtained by taking the standard deviation for each feature map after the convolutional auxiliary network and concatenating them.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Signal Propagation (SP). Signal propagation is one of the layer-wise training methods without relying on the auxiliary networks (Kohan et al. 2023). It trains directly each layer so that the similarities of its outputs would be close to the similarities among the corresponding class labels determined dynamically by the structure of the network. To forward the class information y with the same network structure as the input x; an auxiliary network is used to map y to the same inpul space as a: This network is required only at the network input, which is different from the auxiliary networks other layer- wise methods, so the parameter size becomes small.
We denote by S (st.1....,, S1.m) E R= the class information of each example forwarded until the I-th layer, which called signal. Given the intermediate representations Z = (zt,1,..., = R.xx, we train fi so that the similarities between the elements of S and Z is close. That is, each layer is trained so that the layer's outputs would be in the spacial relationships based on the similarity of the class information forwarded up to that layer. The similarities among the representations Zi are calculated by dot product and row-wise softmax fonction, like the attention matrix calculation in the transformer (aaswani et al. 2017 The loss for signal propagation is specifically calculated as follows:
m
m
((,)))
(- (sim, (S:=1)) log (sim; (Zi)).s)
(17)
m
where
2,2 T
sim: (Z)) = -Row WiseSSofmaax
(18)
di
There can be two types of the sim, settings. In the original implementation (Kohan et al. 2023 the top several, e.g. six, examples with the highest similarity are set to one, and the rest are set to zero for each example. Another naive choice is to apply row-wise softmax normalization similar to sim.. The former is denoted as "SP (hard)" and the latter as "SP (soft)". This loss can also be thought of as optimizing the Kullback-Teibler divergence between the empirical distributionn of class information's 's similarity and the output similarity for each example.
C Forward-Forward Algorithm
In this section, we investigate the Porward-Forwant algorithm, which is of the layer-wise training meth- ods, proposed by Hinton (022)). In this algorithm, the input æ is combined with label information y using techniques like one-hot encoding. This combined input is passed through the network, where each layer is trained to distinguish whether the imput pair (æ, y) is correct label pair or not. Specifically, when the input pair (x. y) is correctly labeled, the L2 norm of the layer's output is enlarged; conversely, it is incorrectly labeled, the norm is diminished.
The key point of this method is the normalization of the outputs at each layer. Each layer trains the similarity itself of the input pair through its output norm and normalizes the output's L2 norm before passing it to the subsequent layer. This normalization aligns with the norms of the representations; therefore, in the training of the subsequent layers, the features learned in the previous layer cannot be utilized. As a result, stacking layers takes on explicit significance in extracting complex information about the input pair similarity. In the case of wise training methods based on the spacial relationship in the representation space, even a layer has meaningless parameter, e.g. an identity mapping, the performance does not deteriorate as the representation from the previous layer can be directly reused. However, the forward-forward algorithm constantly imposes a new role for feature extraction on each layer, potentially enabling the diversity of each layer's role, similar to the advantage of E2E training discussed in the main text. This is why this algorithm is specifically addressed in this section.
An inherent issue arises in the original setting, where passing through more convolutional layers leads to the loss of class information. This because one-hot encoded information can be easily lost by the convolutional operation.
To address this, we explored two modifications.
22

---

Published in Transactions on Machine Learning Research (05/2024)
9
9
+
X
y
One-hot encoded
9
9
y
Prototype encoded
Figure 8: Possible methods for embedding label information in the forward-forward algorithm. Top: The label one-hot encoded to the top-left of images. Bottom: The actual image that represents the label, ie. prototype, combined with the input.
Table 3: Test accuracy of the original forward-forward algorithm and modified version. For MNIST training, toy MLP model that has four layers is used, while toy model with two convolutional layers and two linear layers is used for CIFARIO.
MNIST
CIFAR10
Original
96.4
9.4
Ours
98.0
48.7
1. Use class prototypes instead of onc-hot encoding to preserve the label information through con- volution. Figure 8 illustrates this change. These prototypes could be fixed images for each class, randomly selected images from each class, or the average images within each class.
Change the location of label information mixing. In the original setting, the class information is only mixed before the entire model. We revised this approach to mix class information just before the currently trained layer and forward and y separately through the model to mitigate the loss of class information.
TableBshows ssows the test accuracy for the MNIST dataset and CIFARIO dataset when using the original forward- forward algorithm and our modified version. The test accuracy is improved from the original algorithm, and particularly for CIFAR10, the issue that the model cannot be trained with the forward-forward algorithm has been resolved. Although this modification has led to improved accuracy to some extent, there is still performance gap compared to other existing layer-wise training meth Moreover, despite expecting that the forward-forward algorithm could extract more complex features as the layers are stacked, the accuracy deteriorates, rather than improves. These results suggest that despite these modifications to prevent loss of the class information, the information about inputs or class labels is being lost or inadequately propagated as the network deepens.
Mutual Information and HSIC
D.1 Mutual Information
The concept of assessing the amount of information one random variable by observing the other one is mutual information. The mutual information between random variables X and Y is defined as,
T(X;Y)= / px log pxy(x, v) drdy,
px px (x
23

---

Published in Transactions on Machine Learning Research (05/2024)
where Pxy is the density functions of the joint distribution, and Px and py are the density functions of the marginal distributions. It equals the KT, divergence between joint distribution and the product of marginal distributions. In the field of machine learning, mutual information has been widely employed in contexts
such as representation learning and feature selection (Estévez et al., 2009 Beraha et al. 2019: Hjelm et al.
2018
Tian et al., )020).
D.2 Difficulty in Neural Network Analysis based Mutual Information
Analyzing mutual information within the deep neural network is an intriguing research subject; however, it faces two challenges, as stated below.
The first one is that mutual information can be infinite under certain formulations, which makes the com- parison of mutual information meaningless. We naturally consider that the input X and intermediate representations Z are continuous random variables in a real vector space. However, if Z is determined by X as Z = f(X) in a deterministic way, and if the entropy of Z is finite, the conditional density distribution turns into a Dirac delta function, leading infinite mutual information. [Amjad & Geiger (2019) demon- strates that I(X; Z) becomes infinite under certain assumptions about the activation function. One of the methods to avoid this problem is adding noise to the middle representation, but it does not align with the actual behavior of the network. Note that the actual Z is discrete variable because of the precision of the floatigg-oint, but estimating the probability of falling into all bins is intractable.
The second problem is the difficulty of estimating mutual information. Measuring mutual information is an intrinsically difficult problem due to the high dimensionality and density ratio. The initial attempt analyze the DNN from the IB perspective used the binning- based estimator for measuring mutual infor- mation; however, it does not scale to high-dimensional settings and the TB phenomenon was not observed for the unbounded ReLU activations, which is commonly used in the machine learning implementations. Other approaches to estimating mutual information are non-parametric KDE estimator and a k-NN-basod estimator. Using these algorithms still suffers from the high dimensionality in the network representations.
D.3 HSIC and Chi-Squ Mutual Information
To our knowledge, there no clear relationship between mutual information and HSIC In terms of the independence of two random variables X, Y, I(X;Y) = 0 if and only X and Y are independent. The O, and other direction holds when the product kernel kl is characteristic (Fukumizu et al.) 2007 Sriperumbudur et al. 2011), i.e. the probability distribution the input space corresponds onc-to-onc with the mcan in the RKHS. This property does not hold with a lincar and polynomial kernel, however, the Gaussian RBF kernel used in this study is characteristic.
independence of X and Y implies HSIC Y)
Moreover, besides independence, this characteristic kernel gives a further interpretation of HSIC values. Fukumizu et al.] (2007) demonstrated that the normalized HSIC value equals the Xi-divergence between the joint distribution and the product of the marginal distributions, which is a variant of mutual information. Let us review the definition of the xe-divergence and chi-squared mutual information.
Definition (xe-divergence). Let Px and Qx be the distributions of random variable X, with the probability density function px (x) and qx (x), respectively. The X2 xi-divergence between these two distributions is
xe(Px11xx== = / (px(ee)) 1) 2
qx (z)dz.
(19)
(x(x))
Definition 2 (chi-squared mutual information). Let X and Y be the two random variables. Tf their joint distribution is Pxy and the marginal distributions are Px and Py, with the probability density function pxy (a, v), px (r), and pr (v), the chi-squared mutuul information between these two random variables is defined by
-
Ix (x;)) /
pxy(a,u) (xx(x)p(()) -1) px (t)dddd
(20)
24

---

Published in Transactions on Machine Learning Research (05/2024)
Mutual information is defined the KL divergence between the joint distribution and the marginal distribu- tions, and the used divergence is different from the chi-squared mutual information. Both of these distances for measures belong to the class known as f-divergence. These values cannot be directly computed from the other value, but the following incquality relationship holds.
Theorem 2 Gibbs & Su 2002). Theorem 5). The KL divergence and X2 xe-divergence salisfy
(L(PxIIQx) & log [1 + x2(Px(1Qx)).
(21)
From this inequality, we can also obtain the relationship for mutual information:
I(X - log [1 + Ixa (x; (x,
(22)
We introduced chi-squared mutual information and discussed its relationship with the standard mutual information. Next, we give the result. that normalized HSIC equals this chi-squared mutual information.
Theorem 3 (Fukunizuu et al. (2007), Theorem 4). Let X and Y be the two random variables on X and y, and let H.. and Hus be the associated RKHSs with the kernels K1 X x X - R and R2 y x y - R, respectively. Assume that there erists probability density functions pxy, Px, and py for the joint distribution and the marginal distributions. If (Mx, @ Hmr)) + +R is dense in L22 @ Py), and the Hilbert-Schmidtt norm of the operator xxx exists, then we have
nHSIC(X,Y) =|ccx Cxy 1 (X;())
(23)
According to Proposition 5 in Fukumizu el al. condition part of this theorem, where (Hx, & H..)) +R dense in L2 @ Py), are equivalent. Therefore, if the product kernel kl is characteristic, then nHSIC equals the chi-squared mutual information. This theorem justifics the use of normalized HSIC to analyze the information flow within the neural network model.
2009 the characteristic property of the kernel and the
When considering continuous random variables for input X and representation Z, we can use the RBF kernel and this theorem. In contrast, since the label Y is discrete random variable, which does not have density function, it necessitates re-definition of the mutual information. While the relationship between nHSIC and chi-squared mutual information in this setting is not explicit, it can be verified that the condition similar to that in this theorem is satisfied when a linear kernel applied to label Y.
Proposition 1. Let Y be a random variable that takes values on the one-hot encoded classes y fei..... ect, and H. is an associated RKHS with a linear kernel K. on R x R. For the two probability distributions P and Q, if (E(((()) = =Y)[((Y) for any f E H, then P = Q.
Proof. Use the condition for e; E H = (w + o aa/e RR), leading to P(Y e;) = Eple, YJ = Bole, YJ Q(Y e;). The same holds for e....., Ca, and we get P = Q
E Further Details
E.1 Lower Bound of Mutual Information
In the main text, we brielly mentioned the relationship between the lower bound of mutual information and the optimization of cross-entropy loss in equation S Here, we discuss this relationship in a bit more detail.
The non-negativity of KL divergence yields the following lower bound on mutual information (Barber &
Agakov 2004).
I(Z 2 Ep(v.>) [(rl)) HYY),
(24)
where the equality holds when (()22) equals to p(u|z).
25

---

Published in Transactions on Machine Learning Research (05/2024)
The initial term becomes
Bp(v.s) [log g[vlz)] = dydzp(y, =) log (1]2)
(25)
dydzdapp(y) a ()) log (vl==)
(26)
I dydapply, a:) dzp(z|a) s(ylz)
(27)
p() [Ep(( [log [[]l=>]]]]
(28)
In equation 26 we use ply, 2|x) = p(y)=))((2))) derived from the Markov chain Y Y X
Z. This takes similar form the negalive cross-entropy loss. In fact, the population form of cross-entropy loss in
equation 13 is as follows:
exp fy
LCE = -Ep(v.e) log
(29)
exp
where ÿ is the output of the classifier, in this case, the auxiliary classifier's output h. We are now considering the case where is determined by æ; therefore, can be expressed z f(x), where I is the backbone model up to the target layer. At this time, Ep(z|=) [log g[y[=)] log (()f(()) holds in equation 28 If
we represent the variational distribution by the output of the auxiliary classifier g and softmax function, consequently, minimizing the cross-entropy loss
this value corresponds to the cross entropy-loss equation 29
implies maximizing the lower bound in equation
E.2 Soft Nearest Neighbor Loss and HSIC
In this section, we highlight the relationship between the HSIC values of the intermediate representation and the soft nearest neighbor loss (Salakhutdinov & Hinton 2007 Frosst et al. 2019 We consider the advantages of the middle layer's compression in E2E training through the connection with soft nearest neighbor loss.
Note that while nHSIC used in the information plane plot, the discussion here based HSIC.
Definition 3 (soft nearest neighbor loss, original form Frosst et al. (001)). The soft nearest neighbor loss al lemperalure T, for a batch of samples (x, y) is
[ - 2z 11 T
Σ exp
C. (x,N,) =
Σ log
(30)
2z
iEl.....b
exp
kel.....b
where x can be either the row input its representation in some hidden layer.
Frosst et al. 2019) analyze the structure of middle representations through this value. They argue that maximizing the entanglement of representations, i.e. maximizing soft nearest neighbor loss, is bene- ficial for the discrimination the last layer. In the main text, we observed the compression phase of HSIC value the middle layers even with the F2E, training. Although redueing nHSIC(X, Z) in the middle layers is undorstandable because leads to the decrcase in the final layer's nHSIC(X, Z), reducing nHSIC((, Z) is intriguing as it reduces the necessary information for final prediction. The main theorem in this section sug- gests that minimizing the value of HSIC(Y, Z) contributes to increasing soft nearest neighbor loss; therefore, based on Frosst et al. (2019), it promotes the entanglement of representation and leads to the final better performance.
In this section, we consider the population version soft nearest neighbor loss and the intermediate rep- resentation for x; we use z. We sample n instances from the conditional distribution of the class instead of counting the examples that belong the same class for a fixed batch. The dataset is assumed to be class-balanced in this analysis. We define the soft nearest neighbor loss follows.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Definition 4 (soft nearest neighbor loss). For class-bolances dataset with classes, the soft nearest neighbor loss at temperature at T is defined as:
Σ
2
2 2
exp
T
Lan ---
E
log
(31)
2 - 2 12
(xf Ni Rid p(*lv)
exp
T
64.d (xT)=) p(x)
Please note that since the examples from the numerator do not appear in the denominator summation, the loss is not necessarily non-negative.
When the number of positive samples n goes to infinity, and consequently the total number of data en also goes to infinity, such that the inside of the logarithin converges to the expected value, the following result is obtained regarding the soft nearest neighbor loss and HSIC.
Theorem 4 (Formal version of Thoorem D Suppose the representation is bounded as ||z||2 S M for some constant. M 0. The RBF kernel k(v, w) exp (-1le- w (212/(20²) and the linear kernel I(v, w) v w are used for Z and Y respectively. As goes to infinity, the left-hand side of the following equation converges, and we have
2M HSTC(Y, 2).
lim lan (n, 20²) 2 log cexp n-
(32)
This result might raise concerns about providing a vacuous bound of the loss. However, we can observe the following three facts: 1) lan is not necessarily non-negative as mentioned earlier, 2) the loss value becomes loge c even in the poor representation where all vectors collapse single point, and 3) HSIC(Y, Z) remains at most 1/c, as indicated by the following lemma 1 These facts allow the meaningful bound by appropriately selecting the value of σ. In the nHSIC analysis in this study, a is chosen so that o² would be proportional to the dimensions of the representation space, resulting in the small values of HSIC itself.
We start with the following two lemmas before presenting the proof.
Lemma 1. Suppose that the label y is one-hot encoded to y {0, 13,, and the linear kernel I(0, w) TTw is used for Y. Then, the exact form of HSIC(Y, is
HSIC(Y, Z) =) (Eppor(z,2)) (6(z22)) - Ep(z)p(=)) [k(z, x'))
(33)
where (2 Eÿ=1 ((v)p(=))))) (=)))(())))))))))))
This is supervised setting of Theorem A.1 in Li et al. 2021
Proof. From equation
7
HSIC(Y,2) Ep(u.=)p(w' 1k(,, x 23), w/) p(v.)= [Ept=))(((, =1(= [1 (o,z')l y')[
Ep(t))())) [k(2, (2) Bpprip (1 v²)).
(34)
Since the linear kernel 1 is applied to the one-hot vector y.
l(A.v') {
(リ=v) 1 v²). (otherwise)
(35)
27

---

Published in Transactions on Machine Learning Research (05/2024)
Using this, the first term of equation 34
is
1
e
E Epu.ss)pe k2)
( (1(2, - y')
(36)
c2
y
y'=1
(p(=ss)(()))))) [k(2,2))
(37)
c²
E
ppo [k(z,
(38)
In the last line, we used p(y) - 1/c,ye E [I, and the definition of Pros. The second term written as
Ep(..)) [pp())) (k(z))/k2,,,,, z 212,) = (( y v = Ep(v.2) (p(e)]lK(z 2=]
(39)
Ep(()) (R 2') ==]]
[K
(40)
pp(z)p(=() 16(2,21))
(41)
Finally, the third term is
Ep( 2'),22))BBx 2 PM (1,, = (kz,)
[k(z,
(42)
Combining these yields the conclusion of the lemma.
Lemma 2. Let s,t be the real number satisfying r S S 1, where r > 0 is the constant. We have 1 logt log S (t 8).
(43)
Proof. If < t holds, from the mean value theorem, there exists in (s, such that
1
logt log s (t -88) 8).
(44)
c
Since r < we get logt logs < (t--s), s). In the case of t = both sides of the inequality are equal to zero, thus proving the lemma's statement.
Next, we prove the theorem using These lemmas.
Proof of theorem
We first show the limit of the loft-hand hand side exists.
k(2, x
(n,
E (v.x)=((y.x)
log
im
(45)
cn
(xf Vi Aiu p(*)a)
on
k(z, 2x
en
1a, yen aid p(z)
llgge c
log Σ k(2z
+
E
log
Σ ((
(46)
(v.x)~p(y.z))
(1..))
(x yn p(*)))
(x, )en s.i.d
p(x)
Regarding the inside of the expected value in the second term, using the strong law of large numbers and the continuous mapping thcorem from the continuity of the log function, we obtain the following:
lim log
k(2, x1²)
log
E (xmpp(z()
[k(2,2))) a.s.
(47)
i=1
28

---

Published in Transactions on Machine Learning Research (05/2024)
From this convergence of almost everywhere and the fact that [log( Σi=1 k(z,z*))) is bounded for any = ..., which is derived from the assumption ||z||2 I M for any 2, the dominated convergence theorem yields
1 ri
lim
E
log
k(z,
lim log n-000
Σ 21,
(48)
100000
(())))(())))))))
n
(xt) Rim Gnd p(z1x)
(x 1i-1 p(zl))
E (u.*)~((a)=)
log
E * (pp()
[M(2, (((2,22)]
(49)
Applying similar reasoning to the third term of equation
46
lim to 1-000
log
1 cn
Σ (
cn
E log (u:2)~-(((x)
E (--x)(>)
[b(2,
(50)
(y..)>-(y.=> (0, )
Lud p(*)
Taking the limit of both side of equation 46 leads
(BB [M(2,22)) (1222221) + log
lim enn (n, 20 = log c
E log
(((2,22))
(51)
n=000
p(v.z)
((=1)))
p(u.z) (p(z))
In the remainder, proceed to get the lower bound related to HSIC.
lim (n, 20²) log -0000
dzp(z1u) log
(fdz'(t/llot
+ dzy(2) (s (dd'D(=')((E,)))
(52)
log c log (Eppoo(z.2)) [k(z, 2'1l + dzp(z) log (J de'pp (Jd''(z/))bzzx
(53)
log c - /zppz) [oog (ppo (6(2,2')) log
(54)
Equation 53 follows from Jensen's incquality.
We apply the lemma 2 to lower-bound the difference in logs. As the inside of the exponential in kernel k is non-negative, it is less than equal to 1. Given the assumption the norm of the intermediate representation, ||=||e < M, the value of kernel becomes greater than equal to exp(-(2M22/o3)) Since we have exp(-(2M' //o s k(z,21 x²) I 1 for any 2, x', we can further tower-bound it using lemma as follows:
lim len 20²) 2 log c / dxp(z) exp
22M² o²
(Eroote..e) 1(zzz2)) / de' de'p(=")k(zzz))
(55)
n-+0c
2M 0²
= log exp
(Epror [k(z, z')] Ep(z)p(=) 16(2,21))
(56)
Using the result of lemma
we conclude
(2M lim Cyn (n, 202) 2 log cexp n=0
HSIC(Y, 2).
(57)
o²
F Appendix for Experiments
F.1 Details on Experimental Setup
F.1.1
Performance Gap Experiments
Here we describe the experimental setup for the experiments in section 2.2 and section 2.3 the performance gap between F2F training and layer-wise training.
demonstrating
29

---

Published in Transactions on Machine Learning Research (05/2024)
We conducted experiments on CIFAR10 and CIFAR100 Krizhevsky et al. 2009 Both CIFAR10 and CIFAR100 datasets consist of 50000 training images and 10000 test. images, but the number of label types differs between 10 and 100. For backbone models, VGG (Simonyan & Zisserman 2014) and ResNet (He et al. 2016) are used in the experiments. The VGG network is a combination of convolutional layers, batch-norm layers, and max-pooling layers, allowing for training layer-wisely for each convolutional layer. In contrast, ResNet characterized by the skip-connections. The smallest unit containing a single skip connection, such as two convolution layers for ResNet18 and three convolution layers for ResNet50, is trained as one module in layer-wise training. The middle layer's representation could have more dimensions than that of the last layer of the backhone network, and it would have computational difficulty to directly flatten the middle representation and calculate similarity on it or pass it to the linear or MLP auxiliary network. To handle this, we applied appropriate average pooling so that the output dimension size would be 2048 before performing the calculation. We trained the model for 400 epochs with the Adam optimizer (Kingma & Bal 2014);; the learning rate was selected from т E {0.0005, 0.001), and the learning rate schoduling was performed to decay the learning rate by 0.2 at [200, 300, 350, epochs. For SupCon loss, the model was trained for 1000 epochs with a learning rate 7 0.0005 due to the long time to convergence, and the learning rate was decayed at [700, 800, 900] by 0.2. Furthermore, the hyperparameter for weight decay was chosen from 17 E {0.0, 0.0005). It prevents overfitting to the training data, and it was especially effective when using auxiliary networks with multiple layers.
F.1.2 Normalized HSIC Experiments
In the nHSIC experiments in scction 4 RBF kernels are employed for input X and intermediate representation
Z, while a linear kernel is used for label Y. The RBF kernel, k(v, = exp (-1le
| -
w
w//// (2)2 has a value
of 5 following the existing rescarch (Ma al. 2020 Wang et al. 2021 Jian et al. 2022 Wang al. 2023b
Additionally, we use an empirical estimator for nHSIC as Tr K H(KKK ((KH' em LLH + em m
where é is a small constant; set 16-5 for this experiment. Although the nHSIC estimator demonstrates consistency (Fukumizu et al. 2007], this study derives the estimator by averaging values based mini- batches due to computational limitations. Since the estimates calculated from the mini-batch are basically biased estimator, we avoid direct comparison of nHSIC values among layers and instead focus on the dynamics of nHSIC values. Please note that the training dynamics of nHSIC values itself is the comparison of nHSIC values in the same layer.
F.2 Comparison between E2E and Liyer-wise for CIFAR100
Table | shows the results for E2E training and layer-wise training with the CIFAR100 dataset. Similar to the case of CIFAR1O, the performance of E2E training improves as the model size increases, whereas the performance enhancements of layer-wise training are limitod. In the case of directly optimizing the intermediate layers, as in the row of O-layer, it was challenging to train effectively on the CIFAR100 dataset.
F.3 Linear Separability for SupCon Loss
Figure compares the linear separability between E2E and layer-wise training; the SupCon loss adopted for both training methods. When the cross-entropy loss is used as in the main text, the linear separability of E2E training gradually improves, whereas the scparability starts to saturate at the relatively carly layers. The difference from using the cross-entropy loss is that in layer-wise training, the linear separability at the final layer does not reach 100% for the training data, resulting in a smaller gap between the separability of training and test data. However, the separability for the training data does not improve even if more layers are stacked in the layer-wise training.
F.4
Additional HSIC Dynamics Experiments
F.4.1
Other Loss Settings for E2E Training
The purpose of this section is observe HSIC dynamics using loss functions other than cross-entropy loss used in the paper and to discuss the effect of loss functions. Please note that it does not take into account
30

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Test accuracies of E2E training and layer-wise training methods trained on CIEAR100 dataset. The "Auxiliary" column shows the size of auxiliary networks g for layer-wise training. For clarity, the highest accuracy among the different sizes of auxiliary networks indicated in bold. For CE loss, the middle reprosentations converted to the space of class number's dimensions; therefore, it does not have O-layer case.
E2E
Layer-wise
Model
Auxiliary
CE
SupCon
CE
SupCon
Sim
SP (Hard)
SP (Soft)
O-layer
39.0
1.7
36.5
41.7
VGG11
1-layer 2-layer
68.9
64.2
60.3
55.3
51.9
57.6
62.2
50.9
O-layer 1-layer l-yerrr
48.8
47.0
14.4
1.1
ResNet18
73.5
75.2
64.9
62.4
53.0
62.3
69.8
54.4
O-layer
52.5
21.0
5.2
1.2
ResNet50
1-layer
76.3
78.1
67.5
65.6
58.3
2-layer
65.8
70.2
60.6
E2E
Layer-wise
Linear Probe Accuracy
80-
70-
Test
Test
40-
Train
Train
Layers
Layers
Figure 9: Linear probing accuracies of ResNet50 trained on CIFAR10 dataset. Linear separability for test data and training data are presented. Left: E2E training with SupCon loss. Right: Layer-wise training with SupCon loss. The model was trained with an auxiliary linear classifier for each block.
the comparison between E2E training and layer-wise training discussed in other sections. As loss other than cross-entropy, we experimented with the supervised contrastive loss and the self-supervised loss used in SimCLR (Chen et al. 2020),, which maximizes the InfoNCE lower bound of mutual information among different views. Figure 10 shows that both of these losses lead to a uniform reduction nHSIC(X; Z) and nHSIC(Y: Z) without information bottleneck behavior. The supervised contrastive loss obtained the representations with higher information content with the label Y because of the supervision. In contrast, SimCLR--style loss deals with the mutual information among the representations of different views, there is interpretation provided for the mutual information with labels inputs in cross-entropy.
F.4.2 E2E training with Forward Gradient
We explored the forward gradient method (Baydin et al.) 2022 as a backpropagation-ffrc method, other than layer-wise training. Forward gradient updates weights without using backpropagation but optimizes a single loss function for the entire model; therefore, is E2E training, unlike layer-wise wise training. Figure [[] compares backpropagation and forward gradient for the LeNet5 model, like figure 3 The forward gradient method showed slow convergence, and the test accuracy for CIFAR10 did not improve sufficiently. The improvement. in nHSIC values at each layer is also gradual, particularly the information amount. with input
31

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
Layer
Sup5n
apcas
Supion
Saptio
30
6.1
10-
40
13
and ar
15
40
6.0
68
20
20
110
120
140
190
183
170
80
130
140
12.
nHSICIX:2)
n-SICIX::1
nHSIC(X:2)
nHSIC(K:2)
Figure 10: HSIC plane dynamics E2E training with SimCtR-ttyle loss and supervised contrastive loss. ResNet18 model was trained on CIFAR10 dataset. The color gradation shows the progress of training. Circles with pavy-yellow-basee colormap denote SimCLR-sttye loss, whereas left triangles with a purple- yellow-based colormap show supervised contrastive loss.
Layer
Layer
Layer
Layer
0.2
30
-00
(1
0.0
2.3
Seckprs
Beccang
Recigroo
Backpres
20
(HSICIXZZ)
aHSIC(X:Z)
nHSIC(X:Z]
nHSIC/XiZ)
(a) LeNet5 models trained on MNIST dataset. Test accuracy at 100 epoch: 99.0% (E2E with backpropagation)
82.4% (E2E with forward gradient).
Layer
Layer
Layer
Layer
semaad crackerz
sonaeed
loward trudin:
sonwar uradiert
30
80
12
3
spach
60
40
DHSICIX.Z)
(H5ICIX:2)
nHSICIX:21
(b) LeNet5 models trained on CIFAR10 dataset. Test accuracy at 100 epoch: 62.2% (F2F) with backpropagation) 26.8% (E2E with forward gradient).
HSIC plane dynamics of E2E training with backpropagation and forward gradient method for LeNet5 model. The color gradation shows the progress of training. Inverted triangles with a blue-yellow- based colormap denote forward gradient, whereas circles with a red-based colormap show backpropagation.
Figure 11:
X improved slowly. The important point here is that in the same setling of LeNeu5 CIFAR10 as in figure 3 we did not observe early layer information degradation as seen in the layer-wise training model. While the forward gradient method is a backpropagation-fie technique but an E2E training method, unlike layer-wise training, the interaction among the layers is possible through forward automatic differentiation.
F.4.3 Sequential Layer-wise Training
In this section, we experimented with the sequential layer-wise training mentioned in section B.1 The sequential layer-wise training exhibits lower test accuracy than the simultaneous setting as mentioned in the previous section. When ResNet18 trained on the CIFAR10 datasct using CE loss, table shows that test accuracies of the simultaneous setting are 89.3 and 89.9 for linear and MLP auxiliary classifiers, respectively. In contrast, these test accuracies dropped to 86.6 and 86.2 in the case of sequential training. Here each layer
32

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
Layer 9
DF
DF
RF
Inyrraiie
Inyersine
150
40
12
1.00
40
1.0
311
1.7
1.5
13
36-
62
7-4
10
20
140
145 150 196
170
-30
140
301
190
(HSIC(X:2)
n-SICIX:21
rHSIC(X,2)
nHSICIX
(a) Results for sequential layer-wise training and E2E training. Inverted triangles with blue-yellow-based colormap denote sequential ayee-wise training, whereas circles with a red- based colormap show E2E training.
Layer
Layer 4
Layer 7
Layer
Layeratie
10
Lase "eke
Layer wis
Lavernesee
Layereese
Layerwete
Layer
40
30
30
362
23 Agp
122
6-
6.2
10
-10
125
150
135
140
545
345
150
155
300
370 179
150
140
150
382
170
100
110
139
155
340
150
160
nHSICIX:2)
[HSIC(X:2]
DHSICIX:Z)
aHSICIX:2)
(b) Results for sequential layer-wise training with HSIC augmenting term. Inverted triangles with blue-yellow-baset colormap denote sequential layer-wise training, whereas cross marks with red-yellow-ber colormap show training with HSIC augmenting term (X 0.005). Test accuracy at 50 epoch: 85.6% (original), 85.1% (nHSIC).
Figure 12: HSIC plane dynamics of sequential layer-wise training. ResNet18 is trained on the CIFAR10 dataset for 50 epochs per layer, resulting in 500 epochs to train the entire model because ResNet18 consists of 10 layer-wise trainable modules. The color gradation shows the progress of training.
is trained for 100 epochs, meaning that the entire model is trained for 1000 epochs given that ResNet18 has 10 layer-wise trainable modules.
On the other hand, it is the same between sequential and simultaneous layer-wise training that nHSIC behavior uniform among the layers, and there no conemmttion-bottleneee behavior in the final layer (see figure 12h. In the case of nHSIC regularization, the improvement in information content between X and Z, as well Y and Z, because of the nHSIC augmenting term was more significant compared to simultancous training in section 4.4 This can be attributed to the fact that the sequential layer-wise training exactly corresponds to the greedy layer-wise optimization, which was discussed in section 4.2 and the nHSIC regularization can prevent the severe corruption of the input information. However, there was no improvement in accuracy, which was the same as the simultaneous layer-wise training.
F.4.4 Model Settings
Toy-MLP. We discussed the CNN model LeNet5 in section 4.1 while here, we experimented with MLP. Figure 13 shows the HSIC dynamics for five-layer MLP with the hidden size of 512. In the case of MNIST, there is no difference in Lest accuracy between E2E training and layer-wise training, similar to LeNet5. However, E2E training learns the inputs and labels slowly, which could be attributed to the distance of the backpropagation path from the loss evaluation. the case CIFARIO, layer-wise training exhibited slower improvement in nHSIC values as layers progressed like the LeNet5 case; however, there no information compression in the initial layer observed in LeNet5. In MT.P models, please note that training the first layer in base manner with linear auxiliary classifier is equivalent to E2E training of two-layer MLP model.
Vision Transformer. Figure 14 shows the HSIC dynamics for vision transformer (VIT) Dosovitskiy et al. 2020) with six transformer blocks, the patch size of 4, and the hidden dimension of 512. The local
33

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer 3
Layer
DF
50
8.6
IMISICYY:2)
3-4
60
5.2
0.7
3.
0.5
20
20
(HSIC(X:2)
nHSICIX:Z)
nHSICIX:Z)
nHSICIXZZ)
(a) Toy models trained on MNIST dataset. Test accuracy at 100 epoch: 98.2% (E2E), 98.1% (Layer-wise).
Layer 1
Layer
Layer
Layer
inpersate
Lxperaaiee
aypn-hh wite
30
spoct
2.0
320
200
140
250
170
380
160
160
60
302
1.0
nHSICIX-Z)
(HSICCX:2)
nHSICC(:
nHSICIX:ZI
(b) MLP models trained on CIFAR10 dataset. Test accuracy 100 epoch: 58.7% (E2E), 57.8% (Tayer-wise).
Figure 13: HSIC plane dynamics of Toy-MLP model. The color gradation shows the progress of training, i.c., the number of epochs. Inverted triangles with blue-yellow-basee colormap denote layer-wise training. whereas circles with red-based colormap show E2E training.
Layer
Layer
Layer 3
160
100
100
180
nH5IC(X:2)
mHSICLX:))
nHSICIX:Z)
Layer
Layer
Layer
2000
200m
125
125
75
75
300g
25
25
120
nHSIC(X:2)
MHSIC(X2))
(HSICIX:2)
Figure 14: HSIC plane dynamics of VIT model trained CIFAR10 dataset. The color gradation shows the progress of training. For complex behavior of layer-wise training, the number of epochs and dynamics are annotated in the output layer. Inverted triangles with blue-yellow-basee colormap denote layer-wise training, whereas circles with a red-based colormap show E2E training.
block-wist wise training results in information compression and leads to worse test accuracy, as with the results presented the main text. Conversely, E2E training shows no information compression both middle and output representations, meaning no IB behavior. The test accuracy of trained ViT not sufficiently high for our problem setting, and further validation in larger-scale settings will be necessary to conclude.
34

---

Published in Transactions on Machine Learning Research (05/2024)
Layer 1
Layer 2
Layer
L2L Layer whb
Layerwise
Jaeruise
nHSiC
5.50
8.55
8.20
6.6
9.15
6.1
nHS C(X:Z)
nHSIG(X:Z)
nHSIC(X:Z)
Layer 4
Layer
Layer
6.3
E2t
E2E
0.3
Agloy
8.4
nHSIC(Y:2)
0.3
0.2
12.53
6.0-
0.1
7.8
81-
150
180
167
164
300
140
170
160.0 167.5
165.0
167. 170. 172.5 175.0
-30
-25
160
165
nHS C(X:Z)
nHSIC(X:Z)
nH5IC((:Z)
Layer 7
Layer 8
Layer
F??
Layor wisc
0.4
Layer wise
8.50
.ayer
80
n#SIC(Y.Z)
6.2
7.15
13802/RA00 gangg
Epoch
2000 1000
0.0
40
40
20
20
nH5 C(X:Z)
nHSIC(X:Z)
nHSIC(X:Z)
Figure 15: HSIC plane dynamics ResNet18 model trained on CIFAR1O dataset. The color gradation shows the progress of training. Inverted triangles with a blue-yellow-based colormap denote layer-wise training, whereas circles with red-based colormap show E2E training.
ResNet18. The HSIC dynamics for all layers of ResNet18 are illustrated in figure 15 As observed in figure ( while layer-wise training showed compression all layers, the E2E training compressed the in- termediate layers while maintaining nYSIC(Y, Z) the layer in the output side. This HSIC preservation was seen in the last two layers, which corresponds the last module group when classifying components of ResNet18 by the size of output channels.
ResNet50. The HSIC dynamics for all layers of ResNet50 are presented in figure 16 ResNet50 has 17 intermediate representations, and the layer-wise training demonstrates compression across layers. In the E2E training, the HSIC reduction in the middle layers less observed than in the ResNet18 setting, and it primarily occurs in the early layers. Additionally, the final block not only preserves nHSIC(Y, Z) but also increases it while reducing nHSIC(XX showing behavior closer to the IB hypothesis.
The results for VGG11 are shown in the figure 17 preciscly trains a single convolutional layer, whereas block with one skip-connection the ResNet sotting, The overall behaviors are similar to those of ResNet. The E2E training demonstrates the compression in the intermediate layers while exhibiting behavior suggested by IB hypothesis in the output layers. The layer-wise
VGG11.
this setting, the layer-wise training
35

---

Published in Transactions on Machine Learning Research (05/2024)
training shows little compression of HSIC values in the output layers, differing from ResNet, but generally presents the same low nHSIC values as in the ResNet case.
VGG11 without batch normalization. We made an intriguing observation regarding the compression in the middle layers for E2F training. In figure 18 while layer-wise training includes batch normalization layers and the same setting as before, for the E2E trained model, we excluded the batch normalization layers. This configuration without batch normalization is often used in the VGG network. Interestingly, we observed compression in the nHSIC value for the intermediate and output layers. The final test accuracy higher than that of layer-wise training but lower than when batch normalization was included. This suggests the potential role of batch normalization in compressing intermediate representations. There is a possibility that the E2E training could cooperatively optimize the scales of batch normalization layers across layers. Further exploration is future work.
ResNet18 with HSIC Augmenting Term. The figure 19 shows the results layer-wise training with an additional term aimed increasing nHSIC values. While this modification can prevent compression HSIC dynamics, the obtained test accuracy deteriorated. The smaller À values result in the nHSIC behavior more similar to the original layer-wise training. Please note that used batch size of 256 here instead of the batch size of 128 in the original training due to the effect of nHSIC estimation. As discussed in the main text, the behavior of HSIC dynamics fundamentally remains similar to that of bayer-wisc training, revealing a gap from the E2E training in terms of both the HSIC dynamics and the final performance.
36

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
Layer 4
EIC
1IC
0.72
C22
Layeraan
layer mar
wyyerwen
Layen saie
82-
0.55
0.1
6.00
0.0
5.4
9.2
150
157.5
125.0
310.0
177
134
170
180
nHSIC(XiZ)
nHSIC(X:Z)
nHSiC(XiZ)
nHSIC(X;2)
Layer
Layer
Layer
Layer 8
Layerwite
Lapenanse
5.70
Rysrame
Layerante
3.1
0.55
0.65
0.3
8.50
0.60
8.45
8.2
0.52
8.1
0.41
nHSIC(X:Z)
nHSIC(X:2)
nHSIC(XiZ)
nHSICIX:Z)
Layer
Layer 10
Layer 11
Layer 12
06
Layenwrce
jayervitee
Layoreate
8.1
NHSICAV:2Z
0.4
J0pp
0.2
0.2
0.0
8.0
157
360
105
135
160
103
270
155
160
165
170
175
150
nHSIC(X:Z)
nH5IC(X:Z)
nHSiC(XiZ)
nHSIG(X:Z)
Layer 13
Layer 14
Layer 15
Layer 16
0.0
5C
8.03
8.6
7.50
Agghtigehe pah
X.00
0.2
2200000
6.57
6.07
150
160
170
160
100
150
160
170
185
90
150
110
725
150
105
11.0
125
136
nH5IC(X:Z)
nH5IC(X:2)
nHS/C(X:Z)
nHSIC(X:Z)
Layer 17
Layerwise
(H:)C():2)
60
-60
Epoch
0:75
Di
40
555
20
525
90
100
130
130
150
aHSIC(X.Z)
Figure 16: HSTC plane dynamics of ResNet50 model trained on CIFAR10 dataset. The color gradation shows the progress of training. Inverted triangles with a blue-yellow-based colormap denote layer-wise training, whereas circles with red-based colormap show E2E training.
37

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
E2E
Est
0.6
67
5.4
5.4
N+SICIY:2)
5.-
5.2
5.0
6.0
125.0
1275
130
132.5 115.0 227.5 240.0
200
105
130
115
120
225
130
95
100
102
11.
215
120
nHSIC(X:Z)
nHSIC((:Z)
nHSIC(X:2)
Layer
Layer
Layer 5
7.00
Layer
Layer
Loyer
6.75
(H5IC(Y:Z)
5.50
0.25
8.00
3.00
100
105
123
125
50
230
120
60
nHSIC(X:Z)
nHSIC(X:Z))
nHSIC(K:Z)
Layer
Layer
Layer 9
E2t
EZE Layer wise
Layer u
nHSIC(Y.2)
Laperwice
nHSIC(X:Z)
nHSICIX:
nHSIC(X:Z)
Layer
E2E
IHSIC(V:2))
-60
Epoch
40
0
0
(HSIC(X:))
Figure 17: HSIC plane dynamics of VGG11 model trained on CIFAR10 dataset. The color gradation shows the progress of training. Inverted triangles with a blue-yellow-based colormap denote layer-wise training, whereas circles with a red-based colormap show E2E training.
38

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer 2
Layer 3
F2t
F2t
E2t
SICIY))
2
D0
123
120
340
20
60
DO
300
120
20
40
60
B0
100
120
140
nHSIC(X:Z)
nHSiC(X;ZZ)
nHS ((X:Z)
Layer 4
Layer 5
Layer 6
Layer
Layer
Layer
WHSICAY:
30000
25
225
130
25
1.0
135
100
150
373
nHSIC(X:Z)
nHSiC(XZZ)
nHS H(X:Z)
Layer 7
Layer 8
Layer 9
D2 Layer wise
E2E Layer mise
E2t
wiservise
Glate
4
nHSICIX:Z)
nHSIC(X:Z)
nH5 C(X:Z)
Layer 10
121
Layensazce
DHSICLY.Z)
Epoch
40
0
0
nHSICIX:Z)
Figure 18: HSIC plane dynamics of VGG11 model trained on CIFAR10 dataset. In this figure, the model trained the E2E manner does not contain the batch normalization layers. The color gradation shows the progress of training. Inverted triangles with a blue-yellow-basee colormap denote layer-wise training, whereas circles with a red-based colormap show E2E training.
39

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Layer
Layer
Layer
(ritai))
50
90
1.
High
0.3
Epochh
0.2
aHSIC(X:Z)
aHSIC(XZ))
(a) A 0.05. Test accuracy at 100 epoch: 86.6% (original), 83.1% (nHSIC)
Layer
Layer
Layer 7
Layer 9
lust aux Layerwss 122)
Leycreoz Laparnetar (32)
Lape mis
Loyat mac Lacytratie
Laganenn
Tohlllipp .tar Vay
1.0
60
50
6.4
144
103
100
200
100
150
L00
1/0
1:41
nHSIC(X.Z)
nHSIC(X:Z]
nHSIC(X:2)
nHSICDX:2)
(b) A = 0.01. Test accuracy 100 epoch: 86.6% (original), 85.3% (nHSIC)
Layer
Layer
Layer
9
72
whtpgppp 6 Prppaac
so (ineic)
Layer vise lesicl
Light dentict
50
80
MUSICIY:
Saltier Juge
7.6
Spoch
nHSICIX(Z)
aHSICIXIZ)
(HSIC(X:Z)
nHSICIX.Z)
(c) A = 0.005. Test accuracy at 100 epoch: 86.6% (original), 86, 86.2% (nHSIC)
Layer
Layer
Layer
Layer 9
Lijstsie
Lase mebe
Laye rise
Lavee wise
Layer
Layer
80
233
MISICIIV:))
60
-00
Epoch
0.2
3.2
6.4
7.5
7.0
-0
20
130
340
325
125
302
222
160
301
170
150
560
355
110
:20
150
340
DHSICIX:Z1
RHSIC(X2))
nHSICUI:ZI
nHSIC(X:2]
(d) 0.001. Test accuracy at 100 epoch: 86.6% (original), 86.1% (nHSTC)
Figure 19: HSIC plane dynamics for layer-wise training with HSIC augmenting term for different À values. 18 is trained on CIFAR10. The results of four layers are sunmarized in this figure. The color gradation shows the progress of training. Inverted triangles with a blue-yellow-basee colormap denote layer- wise training in the original setting as baseline, whereas the cross marks with a red-yellow-basse based colormap show training with HSIC augmenting term.
40


Reviews:
[{"review_id": "7JQyoHGydJ", "reviewer": "Reviewer_tneY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe idea of explaining the representations of different neural network layers with HSIC is insightful. \n\nWeakness:\n\n-  The writing is unclear. It is hard to understand the argument of the paper. What does the paper prove? Why they are important? For example, the introduction discussed \" the human brain has connections on the order of 100 trillion, whereas GPT-4 has roughly 1 trillion parameters\". Why is this relevant to this paper?\n\n- Comparing E2E training with layer-wise training is not well-motivated. How does it help to understand the information bottleneck by E2E training? How does the local optimization phenomenon observed in layer-wise training support the IB theory of E2E training? \n\n- Instead of showing the HSIC of different epochs, it will be more straightforward to visualize those of different layers in the same plot. In the paper the x and y scales of different layers are not shared.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzed the difference between end-to-end training and layerwise training from the information bottleneck perspective. It mainly compared normalized HSIC - nHSIC(X,Z) and nHSIC(Y,Z) of representations z at different layers. The paper found end-to-end training is able to maintain high HSIC in the final layer.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the questions in the weakness.", "broader_impact_concerns": "No broader concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IxleRHUjMu", "reviewer": "Reviewer_CGUu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper reveals an interesting analysis of the information flow in the networks, and leads to insights that are novel to the best of my knowledge. \n* The use of HSIC is judicious, and provides a criterion that can be useful for further studies in developing backpropagation free approaches. \n* The paper is well structured, and clearly written, and the discussion is of relevance to the community. \n\nWeaknesses: \n* The end-to-end alternative studied here is really simple and lacks some clarity (more details below). It would be interesting to consider other backpropagation alternatives and study their impact. An example that comes to mind is the \"forward gradient\" formulation (see e.g. Gradients without Backpropagation, Baydin et al. 2022). \n* It seems that more empirical validation is in order. For example, concerning the analysis in Figure 1, it would be interesting to study the impacts of varying data (complexity of the input, number of classes,...) and models (containing convolutions, skip connections, attention mechnisms, etc...).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an analysis of end-to-end training of neural networks as compared to layer wise training. In particular, the authors consider the information flow in the networks and analyse it through the Hilbert-Schmidt independence criterion (HSIC) dynamics in different layers in networks trained either end-to-end or layer-wise. \n\nBeyond confirming the performance gap in different models, this analysis revealed 2 main observations: With end-to-end training, middle representations are more entangled but exhibit higher correlation with the labels. Moreover, with end-to-end training, the HSIC dynamics are different across layers, and the layers play different roles in information compression, while layer-wise training shows a more uniform behavior. \n\nThe authors additionally proposed to use HSIC to regularise layer-wise training to increase information preservation across layers, but this led to worse test performance. The advantages of end-to-end training are further discussed from an information bottleneck and from a geometric perspective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From the first point under weaknesses: \n* Can the authors clarify the layer-wise training procedure? Is it done jointly for all layers using the objective (2), or is it done sequentially as in e.g. \"Greedy Layerwise Learning Can Scale to ImageNet\" Belilovsky et al. 2019? If it's the former, is it evaluated using only the last layer output or by pooling outputs from intermediary layers too? Either way, it would be interesting to compare these two alternatives and their impact on the performance gap, and with the HSIC regularization.\n* Can the authors analyse other backpropagation free baselines? \n\nFrom the second point under weaknesses: \n* Can the authors show how their observations vary when varying data and models?", "broader_impact_concerns": "The authors discuss the limitations of their work. Given the theoretical nature of the work, this discussion is interesting and sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jhZL8Pbcj9", "reviewer": "Reviewer_cn4C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strength**\n- The paper is overall organized, facilitating a smooth reading experience. Additionally, the illustrative figures and experimental visualizations simplifies the understanding of the analysis. \n\n- The mathematical analysis are technically sound. It effectively demonstrates how E2E training promotes better information flow and differentiation of layer roles, leading to superior model performance.\n\n- This comparative analysis offers a fresh perspective on neural network training dynamics, contributing valuable insights to the field of deep learning research.\n\n### **Weakness**\n- Although the paper leverages the analysis in the lens of information theory, specifically of mutual information, it mostly discuss the observation in terms of information compression with model trained with cross-entropy. How would the model behave when using mutual-information-oriented loss, e.g. the one proposed in Hjelm, R. Devon, et al. 2018? \n\n- The paper only demonstrates the results on convolutional model architectures for its conclusions, this could limit the universality of its findings. It could be beneficial to valid the analysis on recent transformer architecture, pure MLP architecture, etc.\n\n- Considering the models studied in the paper are limited, and some recently popular architectures such as ViT, Swin transformer, ConvNext are not studied. The paper might not fully explore the practical implications of its findings, particularly how they could be applied in real-world scenarios or influence the development of future deep learning models.\n\n\n### References\n\n- Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the efficacy of end-to-end (E2E) training versus layer-wise training in the learning of deep neural networks. It looks into how E2E training excels in information propagation and assigns diverse roles to layers, leading to an efficient model representation. Through information-theoretic analysis and HSIC metrics, the study demonstrates that E2E training promotes better information flow and layer-role differentiation. It suggests that these advantages contribute to the observed performance benefits of E2E training, offering a new perspective and insight on neural network training dynamics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part. Specifically, including the analysis with the transformer architecture could improve the scope of the paper.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite some concerns from the reviewers, they are unanimous in deciding that the paper meets both the claims and evidence and audience criteria.

For my part, I feel as though the paper does a good job of not overselling what it is.  The paper is a primarily experimental investigation and some associated commentary about those experiments.  TMLR is meant to be a home for such papers.
Decision: Accept (poster)