Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
Authors: Yulin Wang
Zanlin Ni
Shiji Song
Le Yang
Gao Huang
Venue: ICLR 2021 Poster (2021)
Keywords: Locally supervised training
Deep learning

Abstract: Due to the need to store the intermediate activations for back-propagation, end-to-end (E2E) training of deep networks usually suffers from high GPUs memory footprint. This paper aims to address this problem by revisiting the locally supervised learning, where a network is split into gradient-isolated modules and trained with local supervision. We experimentally show that simply training local modules with E2E loss tends to collapse task-relevant information at early layers, and hence hurts the performance of the full model. To avoid this issue, we propose an information propagation (InfoPro) loss, which encourages local modules to preserve as much useful information as possible, while progressively discard task-irrelevant information. As InfoPro loss is difficult to compute in its original form, we derive a feasible upper bound as a surrogate optimization objective, yielding a simple but effective algorithm. In fact, we show that the proposed method boils down to minimizing the combination of a reconstruction loss and a normal cross-entropy/contrastive term. Extensive empirical results on five datasets (i.e., CIFAR, SVHN, STL-10, ImageNet and Cityscapes) validate that InfoPro is capable of achieving competitive performance with less than 40% memory footprint compared to E2E training, while allowing using training data with higher-resolution or larger batch sizes under the same GPU memory constraint. Our method also enables training local modules asynchronously for potential training acceleration.

Published as a conference paper at ICLR 2021
REVISITING LOCALLY SUPERVISED LEARNING: AN ALTERNATIVE TO END-TO-END TRAINING
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang & Gao Huang"
Department of Automation, BNRist, Tsinghua University, Beijing, China, (wang-y119, nz117, yangle15 5 @mail 1s. ts nighu edu. {shijis, gaohuang (aanaanggttsinghuue edu.cn
ABSTRACT
Due to the need to store the intermediate activations for back- pocckpropagation, end-to- end (E2E) training of deep networks usually suffers from high GPUs memory foot- print. This paper aims address this problem by revisiting the locally supervised learning, where a network is split into gradient-isolated modules and trained with local supervision. We experimentally show that simply training local modules with E2E loss tends to collapse task-relevant information at early layers, and hence hurts the performance of the full model. To avoid this issue, we propose an information propagation (InfoPro) loss, which encourages local modules to preserve as much useful information as possible, while progressively discard task-irrelevant informa- tion. As InfoPro loss is difficult to compute in its original form, we derive feasible upper bound as a surrogate optimization objective, yielding a simple but effective algorithm. In fact, we show that the proposed method boils down to minimizing the combination
reconstruction loss and a normal cross-entropy/coolrraiieee term. Extensive empirical results on five datasets (i.e.. CIFAR, SVHN, STL-10, ImageNet and Cityscapes) validate that InfoPro capable of achieving competitive performance with less than 40% memory footprint compared to E2E training, while allowing using training data with higher-resolution or larger batch sizes under the same GPU memory constraint. Our method also enables training local modules asynchronously for potential training acceleration. Code is available at: waeer--anng/// wang/Innforoo-ytorre
attps://githus
INTRODUCTION
End-to-end (E2F) back-propaggttio has become standard paradigm train deep networks 2012 Simonyan & Zisserman| 2014; Szegedy et al. 2015 He et al.) 2016 Huang et al. 2019). Typically, training loss is computed the final layer, and then the gradients are propagated backward layer-by-layer to update the weights. Although being effective, this procedure may suffer from memory and computation inefficiencies. First, the entire computational graph as well as the activations of most, if not all, layers need to be stored, resulting in intensive memory consump- tion. The GPU memory constraint is usually a bottleneck that inhibits the training of state-of-the-ar models with high-resolution inputs and sufficient batch sizes, which arises in many realistic scenarios, such as 2D/3D semantic segmentation/objec detection in autonomous driving, tissue segmentation in medical imaging and object recognition from remote sensing data. Most existing works address this issue via the gradient checkpointing technique (Chen et al. 2016) or the reversible architccture design (Gomez et al. 2017), while they both come at the cost of significantly increased computation. Second, E2E training is a sequential process that impedes model parallelization (Belilovsky et al.) 2020
(Krizhevsky et al.
Löwe et al. 2019). as earlier layers need to wait for their successors for error signals.
As an alternative to E2E training, the locally supervised learning paradigm (Hinton et al. 2006 Bengio al., 2007 Nokland & Eidnes 2019 Belilovsky al.) 2019 2020) by design enjoys higher memory efficiency and allows for model parallelization. In specific, it divides deep network into several gradient-isolates modules and trains them separately under local supervision (see Figure [
(b)). Since back-propagation is performed only within local modules, one does not need to store all
"Corresponding author.

---

Published as a conference paper at ICLR 2021
Regubee Sinika
Comv Layer
Freward
Book Propapation
Thak Incelsnnt Information
Tink Relesoi Information
Full Networ
Iapurar
Intemaaiat:
baskpeedd Lce
End-to--nd Supervised Leariin
balaseed
Find-ro-end Supervised carning
Greedy Supervised Leaniing
Modale 1
Module
(C = Rid--te-ond Less
Minimito
Informationn
Progugation
(C (uues))
Locally Supervised Learing
Compatison of Three Training Approsches
Figure 1: (a) and (h) illustrate the paradigms of end-to-end (F2F) learning and locally supervised learning (K 2). "End-to-end Loss" refers the standard loss function used by E2E training, e-g., softmax cross-entropy loss for classification, etc., while denotes loss function used train local modules. (c) compares three training approaches in terms of the information captured by features. Greedy supervised learning (greedy SL) tends to collapse some of task-relevant information with the beginning module. leading to inferior final performance. The proposed information propagation (InfoPro) loss, however, alleviates this problem by encouraging local modules propagate forward all the information from inputs, while maximally discard task-irrelevant information.
intermediate activations at the time. Consequently, the memory footprint during training is reduced without involving significant computational overhead. Moreover, by removing the demands for obtaining error signals from later layers, different local modules can potentially be trained in parallel. This approach is also considered more biologically plausible, given that brains are highly modular and predominantly learn from local signals (Crick 1989 Dan Poo 2004, Bengio et al.) 2015). However, major drawback of local learning g is that they usually lead to inferior performance compared to E2E training (Mostafa et al. 2018 Belilovsky al. 2019 2020).
In this paper, we revisit locally supervised training and analyse its drawbacks from the information- theoretic perspective. We find that directly adopting an E2E loss function (i.e., cross-entropy) train local modules produces more discriminative intermediate features at earlier layers, while it collapses task-relevant information from the inputs and leads to inferior final performance. In other words, local learning tends to be short sighted, and learns features that only benefit local modules, while ignoring the demands the rest layers. Once task-relevant information is washed out carlier modules, later layers cannot take full advantage of their capacity to learn more powerful representations.
Based on the above observations, we hypothesize that a less greedy training procedure that preserves more information about the inputs might be rescue for locally supervised training. Therefore, propose a less greedy information propagation (InfoPro) loss that aims to encourage local modules to propagate forward much information from the inputs as possible, while progressively abandon task-irrelevant parts (formulated by an additional random variable named nuisance), as shown in Figure (c). The proposed method differentiates itself from existing algorithms (Nokland & Eidnes 2019 Belilovsky et al. 2019 2020) on that it allows intermediate features to retain a certain amount of information which may hurt the short-term performance, but can potentially be leveraged by later modules. In practice, as the InfoPro loss is difficult to estimate in its exact form, we derive a tractable upper bound, leading to surrogate losses, e.g., cross-entropy loss and contrastive loss.
Empirically, we show that InfoPro loss effectively prevents collapsing meslvreleannt information at local modules, and yields favorable results on five widely used benchmarks (i.e., CIFAR, SVHN, STL-100 ImageNet and Cityscapes). For instance, it achieves comparable accuracy as E2E training using 40% or less GPU memory, while allows using a 50% larger batch size or a 50% larger input resolution with the same memory constraints. Additionally, our method enables training different local modules asynchronously (even in parallel).
2 WHY LOCALLY SUPERVISED LEARNING UNDERPERFORME E2E TRAINING?
We start by considering local learning setting where deep network is split into multiple successively stacked modules, each with the same depth. The inputs are fed forward in an ordinary way, while the gradients are produced at the end of every module and nack-propagated until reaching an earlier module. To generate supervision signals, straightforward solution is to train all the local modules as
2

---

Published as a conference paper at ICLR 2021
Liver Sepuraiiity intemzatate beatern des (troe)
Abrtaal Inkeraatioo beteors Fottro Expstex
Steteal Infernation betons Frat Latel
0.50
W EU42
7.00-
Edieeato
6.64
16
Figure 2: The linear separability (left, measured by test errors), mutual information with the input (middle), and mutual information with the lahel y (righr) of the intermediatc fcatures from different layers when the greedy supervised learning (greedy SL) algorithm adopted with K local modules. The ends local modules are marked using larger markers with black edges. The experiments are conducted on CIFAR-10 with a ResNet-32.
independent networks, e.g. in classification tasks, attaching classifier to each module, and computing the local classification loss such as cross-entropy. However, such a greedy version of the standard supervised learning (greedy SL) algorithm leads to inferior performance of the whole network. For instance, in Table [ we present the test error of a ResNet-32 (He et al.) 2016) CIFAR-I0 (Krizhevsky et al. (2009)
Table I: Test errors of Res using greedy SL on CIFAR-10. The network is divided into K successive local modules. Each module is trained separately with the softmax cross-entropy loss by appending a global-pool layer followed by a conneccd layer (see Appendix for details).
"K = 1" refers to end-to-end (E2E) training.
K =
K
2
K
4
K
8
K = 16
Test Error 7.37%
10.30%
16.07%
21.
24.59%
when it is greedily trained with K modules. One can observe a severe degradation (even more than 15%) with K growing larger. Plausible is inherent for local learning and how to alleviate this problem. In this section, we investigate the performance degradation issue of the greedy local training from information-theoretis perspective, laying the basis for the proposed algorithm.
this phenomenon seems, it remains unclear whether it
Linear separability of intermediate features. In the common case that greedy SL operates directly on the features output by internal layers, a natural intuition is to investigate how these locally learned features differ from their E2E learned counterparts in task-relevant behaviors. To this end, we fix the networks in Table [ and train a linear classifier using the features from each layer. The test errors of these classifiers are presented in the left plot of Figure [2, where the horizontal axis denotes the indices of layers. The plot shows an intriguing trend: greedy SI. contributes to dramatically more discriminative features with the first one (or few) local module, but is only able to slightly improve the performance with all the consequent modules. In contrast, the E2E learned network progressively boosts the linear separability of features throughout the whole network with even more significant effects in the later layers, surpassing greedy SL eventually. This raises interesting question: why does the full network achieve inferior performance in greedy SL compared to the E2E counterpart, even though the former is based more discriminative earlier features? This observation appears incompatible with prior works like deeply-supervisee nets (Lee et al. 2015a).
Information in features. Since we use the same training configuration for both greedy SI. and E2E learning, we conjecture that the answer to the above question might lie in the differences of features apart from merely separability. To test that, we look into the information captured by the intermediate features. In specific, given intermediate feature h corresponding to the input data æ and the label y (all of them are treated as random variables), we use the mutual information / (h, x) and / [(,, y) to measure the amount of all retained information and rele-relevant information in h, respectively. As these metrics cannot be directly computed, we estimate the former by training a decoder with binary cross-entropy loss to reconstruct æ from h. For the latter, we train CNN using h as inputs to correctly classify x, and estimate 7 T(h, with its performance. Details are deferred to Appendix G
The estimates of T(h, .) and I(h. y) at different layers are shown in the middle and right plots of Figure 2] We note that in E2E learned networks, 1(h, v) remains unchanged when the features pass through all the layers, while [(h, æ) reduces gradually, revealing that the models progressively discard insk-irreeevant information. However, greedily trained networks collapse both I [hh, x)) and [(h, y) in their first few modules. We attribute this to the short sighted optimization objective of earlier modules, which have relatively small capacity compared with full networks and are not capable of extracting and leveraging all the task-relevant information in æ, the E2E learned networks
3

---

Published as a conference paper at ICLR 2021
do. As consequence, later modules, even though introducing additional parameters and increased capacity, lack necessary information about the target y to construct more discriminative features.
Information collapse hypothesis. The above observations suggest that greedy SL induces local modules to collapse some of the task-relevant information that may be useless for short-term per- formance. However, the information is useful for the full model. In addition, we postulate that, although E2E training is incapable of extracting all task-relevant information at earlier layers as well, alleviates this phenomenon by allowing a larger amount of task-irrelevant information be kept, even though it may not be ideal for short-term performance. More empirical validation of our hypothesis is provided in Appendix A
3 INFORMATION PROPAGATION (INFOPRO) LOss
In this section, we propose an information propagation (InfoPro) loss to address the issue of infor- mation collapse in locally supervised training. The key idea is to enforce local modules to retain as much information about the input as possible, while progressively discard task iesk-iirelevant parts. As is difficult to estimate InfoPro loss in its exact form, we derive easy-to-compute upper bound as the surrogate loss, and analyze its tightness.
3.1 LEARNING TO DISCARD USELESS INFORMATION
Nuisance. We first model the task-irrelevant information in the input data æ by introducing the concept of nuisance. A nuisance is defined as an arbitrary random variable that affects x but provides no helpful information for the task of interest (Achille & Soatto 2018). Take recognizing car in the wild for example. The random variables determining the weather and the background are both nuisances. Formally, given nuisance r, we have [r,, a:) > 0 and (r, y) = O, where y is the label. Without loss of generality, we suppose that y. r, x and h form the Markov chain (y.r) - + h, namely p(h xz, y, r) = p(hl As a consequence, for the intermediate feature from any layer, we obviously have [r,, x) 2 ht) Nevertheless, we postulate that max, (r; h) 0. This assumption is mild since it does not hold only when h strictly contains no mask-irreeevant information.
Information Propagation (InfoPro) Loss. Thus far, have been ready to introduce the proposed InfoPro loss. Instead of overly emphasizing on learning highly discriminative features at local modules, also pay attention preventing collapsing useful information in the feed-forward process. simple solution to achieve this maximizing the mutual information 7 7(h, x:). Ideally, if there is no information loss, all useful information will be retained. However, it goes another extreme case where the local modules do not learn any task-relevant feature, and is obviously dispensable. By contrast, in E2E training, intermediate layers progressively discard useless (task- irrelevant) information as well as shown above. Therefore, to model both effects simultaneously, we propose the following combined loss function:
LintoPro (h) al-1(h, x) + 31 h)], a, B 2 0, s.t. r"
argmnax ((raa))0, T(ra)=
[(r, h),
(I)
where the nuisance , is formulated to capture as much task-irrelevant information in h as possible, and the coefficient 3 controls the amount of information that is propagated forward (first term) and task-irrelevant information that is discarded (second term). Notably, we assume that the final module is always trained using the normal E2E loss (e.g., softmax cross-entropy loss for classification) weighted by the constant 1, such that a is essential to balance the intermediate loss and the final one. In addition, (hnropto(h) is used to train the local module outputting h. whose inputs are not required to be æc The module may stack over another local module trained with the same form of (I)) but (possibly) different a and B.
Our method differs from existing works (Nøkland & Eidnes 2019 Belilovsky et al. 2019 2020)
in that is a non-greedy approach. The major effect of minimizing LintoPro (h) can be described as maximally discarding the task-irrelevant information under the goal of retaining as much information of the input as possible. Obtaining high short-term performance not necessarily required. As we explicitly facilitate information propagation, we refer to Lintorro (h) as the InfoPro loss.
3.2 UPPER BOUND OF LinfoPro
The objective function in [ is difficult to be directly optimized, since it is usually intractable to estimate r*, which is equivalent to disentangling all task-irrelevant information from intermediate
4

---

Published as a conference paper at ICLR 2021
features. Therefore, derive casy-to-compute upper bound of LintoPro surrogate loss. Our with the proof in Appendix B
result summarized in Proposition [
Proposition Suppose that the Markov chain (y, - x - h holds. Then an upper bound of LintoPm is given by
LinfoPro I <h) A2d (h, v) LintoPro:
where \ aa11 3). 3), A2 a33.
For simplicity, we integrate and 3 into two mutually independent hyper-parametert X1 and X2- Although we do not explicitly restrict A 2 O, we find in experiments that the performance of networks is significantly degraded with X1 - 0+ (see Figure 4 or say, 3 + 1 where models tend to reach local minima by trivially minimizing I(r", h)) Eq. I Thus, we assume Å1, A2 2 O.
With Proposition] 1 we can optimize the upper bound LintoPro an approximation, circumventing dealing with the intractable term [(r", h)) in LintoPro- To ensure that the approximation is accurate, the gap between the two should be reasonably small. Beloo we present analysis of the tightness of LintoPro in Proposition 2 (proof given in Appendix C). We also empirically check it in Appendix H Proposition
2 provides useful tool to examine the discrepancy between LinfoPro and its upper bound.
Proposition 2. Given that r" = argmax,, I(r,))>,, I(ry)= J(r, h) and that is deterministic function with respect a, the gap LinfoPro LintoPro upper bounded by
S A2 [((x,)) 1(h,
3.3 MUTUAL INFORMATION ESTIMATION
In the following, we describe the specific techniques we use to obtain the mutual information I(h, and T(,, y) in LintoPro Both of them are estimated using small auxiliary networks. However, we note that the involved additional computational costs are minimal or even negligible (see Tables 3
Estimating [h, Z). Assume that R(x h) denotes the expected error for reconstructing æ from h. It has been widely known that R(x(h) follows I(h, x) (( (x)h) H(x) (-)))((())))))))
where (x) denotes the marginal entropy æ. as constant (Vincent et al. 2008 Rifai et al.
2012 Kingma & Welling 2013) Makhzani et al., 2015 Hjelm et al. Therefore, we estimate I[h, x by training a decoder parameterized by w to obtain the minimal reconstruction loss, namely 1 [h, x) max (xx Rw [.)]. In practice,
the binary cross-entropy loss for R (x ht).
Estimating 1(h, v). We propose two ways to estimate [(, Since I [(h, y) = H(y) H(y h)) = -( ((
P(y h)]. straightforward approach is to train an auxiliary clas- sifier Zep (v/ h)) with parameters Tp approximate p(y) h). such that we have I(h, v) max { (2))
Ex, - only if qv (y/h)))))))) hh)=((y(h)) (according to the Gibbs' inequality). Finally, we estimate the expectation on h using the samples {xx,, h4, ) namely I I(h, y) R max { (H (v) + + -log 4v (y h.)]].
uu (b)/) Note that this approximate equation will become an equation if and
Consequently, q (y/ /t) can be trained in a regular classification fashion with the cross-ent cross-entropy loss.
In addition, motivated by recent advances in contrastive representation learning (Chen al. 2020 Khosla et al., 2020 He et al.] 2020), we formulate a contrastive style loss function Leontrats and prove in Appendix D that minimizing Leonrnst is equivalent to maximizing lower bound of [(h. v). Empirical results indicate that adopting Lcontrast may lead to better performance if a large batch size is available. In specific, considering mini-batch of intermediate features (h1,...; hw) corresponding to the labels {B1..... }. Coootrast is given by:
exp(2,xx 2z/T)
1 = fh:))). (4)
Leontrast
Σ E
1 9.= log
Enz 1
1itk exp(2] 2x/T)
Herein, LA E {0, 11 returns 1 only when A is true, T > 0 is a pre-defined hyper-paramette temper- ature, and f is projection head parameterized by Φ that maps the feature h, representation vector 2; (this design follows Chen et al. (2020); Khosla et al. (2020))
We defer the details on the network architecture of w, zb and ø to Appendix Briefly, on CIFAR, SVHN and STL-10, w is a two layer decoder with up- up-sampled inputs (if not otherwise noted), with ub and of sharing the same architecture consisting of single convolutional
Implementation details.
5

---

Published as a conference paper at ICLR 2021
layer followed by two fully-connected layers. On ImageNet and Cityscapes, auxiliary nets, but they are very small compared with the primary network. Empirically, we find that these simple architectures are capable of achieving competitive performance consistently. Moreover, in implementation, we train w, vp and & collaboratively with the main network. Formally, let denote the parameters of the local module be trained, and then our optimization objective
relatively larger
1 Σ N
minimize X Ru (z(h))+2 0.w,
llog gep (uil;)) or minimize R (2: ht) 22 Contiaatt (5) 0,w.4
which corresponds to using the cross-entropy and contrast loss to estimate [h,, y), respectively. Such an approximation is acceptable as we do not need to acquire the exact approximation of mutual information, and empirically it performs well in various experimental settings.
4 EXPERIMENTS
Setups. Our experiments are based on five widely used datasets (i.e., CIFAR-10 (Krizhevsky et al.
2009), SVHN (Netzer et al. 2011), STL-10 (Coates al., 2011), ImageNet (Deng et al. 2009) and Cityscapes (Cordts et al., 2016) and two popular network architectures (i.e., ResNet (He et al.) 2016) and DenseNet (Huang et al. 2019 with varying depth. We split each network into K local modules with the same (or approximately the same) number layers, where the first K-1 modules trained using LintoPm: and the last module is trained using the standard E2E loss, as aforementioned. Due to spatial limitation, details on data pre- pre-prcesssnng training configurations and local module splitting are deferred to Appendix F The hyper-parameters X and X2 selected from {0, 1.100.1,.... 1,2, 2, 5, 10, 20}. Notably, to avoid involving too many tunable large (e.g., K 16), assume that A and X2 change linearly from 1st to (K -1) local module, and thus we merely tune X and A2 for these two modules. We always use T= 0.07 in Leontras--
when K
Two training modes are considered: (I) simultaneous training, where the proc-prrppggtton process of all local modules sequentially triggered with every mini-batch of training data; and (2) asyn- chronous training, where local modules are isolatedly learned given cached outputs from completely trained earlier modules. Both modes enjoy high memory efficiency since only the activations within single module require be stored time. The second mode removes the dependence of local modules on their predecessors, enabling the fully decoupled training of network components. The experiments using asynchronous training are referred as "Asy-InfoPro", while all other results are based on simultancous training.
4.1 MAIN RESULTS
Comparisons with other local learning methods. We first compare the proposed InfoPro method with three recently pro- posed algorithms, decoupled greedy learn-
0.00
14
1150
Tet broes (8)
(.30
ing (Belilovsky al.) 2020) (DGL), Boost-
0.55
ResNet (Huang et al. 201 8a) and deep in-
cremental boosting (Mosca & Magoulas 2017) (DIB) in Figure 3] Our method yields the lowest test errors with all val- ues of K. Notably, DGL can be viewed as special case of InfoPro where Å₁ =0. Hence, we use the same architecture of auxiliary networks as us in DGL for fair comparison. In addition, we present the estimates of mutual information between intermediate features and labels in the right plot of Figure 3 One observe that DGL suffers from a severe collapse of task-relevant information at early modules, since it optimizes local modules greedily for merely short-term performance. By contrast, our method effectively alleviates this problem, retaining a larger amount of task-relevant information within intermediate features.
16P%
DALL (Bikki A.A.
140 m1 2017
0.52
10
K-2 K-4 K-8 1-10
Loger Inder
Figure 3: Comparisons of InfoPro and state-of-the-art local learning methods terms of the test errors at the final layer (left) and the task-relevant information capture by intermediate features, I(h, y) (right). Results of ResNet-32 on CIFAR-10 reported. We use the contrastive loss in EidaPro
Results on various image classification benchmarks presented in Table 2 We also report the result of DGL (Belilovsky et al.] 2020) in our implementation. can be observed that InfoPro outperforms greedy SI. by large margins consistently across different networks, especially when K is

---

Published as a conference paper at ICLR 2021
Table 2: Performance of different networks with varying numbers of local modules. The averaged test errors and standard deviations 5 independent trials are reported. InfoPro (Softmax/Contrast) refers to two approaches to estimating [(,, y). The results of Asy-InfoPro is obtain by asynchronous training. while others are based on simultancous training. Greedy SI adopts deeper networks have the same computational costs as InfoPro
Diinser
Network
Method
K =
K =
K 16
Greedy SL
10.30 0.20%
16,07 0.48%
21.19 = 0.52%
24.59
0.83%
ResNet-32
DGL (Belilovsky et al. 2020
8.69 =: 0.12%
11,48 : 0.20%
14.17 = 0.28%
16.31 :: 0,36/
0.10%)
IntoPro (Sofimax)
8.13 +: 0.23%
8.64 + 0.25%
11.40 0.18%
14:23 +: 0.42%
InfoPro (Contrast)
7,76 +: 0.12%
8,58 4: 0.17%
11.13 + 0.19%
12.75 a: 0.11%
Greedy St.
8.21 # 0.24%
13.16 + 0.28%
15.61 = 0.57%
18.92 :: 1.27%
Gready SL*
8.00 + 0.11%
12.47 = 0.17%
14.58 = 0.36%
17.35 +: 0,31%
ResNet-110 (E2E: 6.50 0.34%)
DGLL (Belloossky et al. 2020
7.20 + 0.28%
10.50 = 0.115
12:46 = 0.37%
13.80 d: 0.15%
CIBAR-1O
IntoPro (Softmax)
7.01 +: 0.34%
7.96 d: 0.06%
9:40 0.27%
10.78 d: 0.28%
Asy-InfoPro (Contrast)
7.34 - 0.31%
8.39 0.15%
InfaPro (Contrast)
6.4 d: %
7,30 :: 0.14%
0.40%
9.90 ::: 0.19%
Gready SI.
5.10 :: 0.05%
: 0.21%
0.31%
10:41 :: 0.42%
DenseNet-BC BC-10000- (E2E: 4.61 +: 0.08%)
DGL Bellioovsky et al. 2020
4.86 d: 0.15%
0.04%
7.67 0.16%
InfaPro (Soltmax)
4.79 +: 0.07%
6.44 0.11%
7.47 0.21%
InfoPro (Contrast)
4.74 0.04%
5.24 +: 0.25%
5,86 0.18%
6.92 0.16%
Greedl SL.
3.71 +: 0.16%
5.39 0.22%
5.75 :: 0.10%
6.37 0.42%
ResNet-110 (E2E: 3.07 0.23%)
DGLL Relilovsky et al. 2020
361 + 0.16%
4:97
5.35 0.13%
5.55 0.34%
SVHN
IntaPro (Sofemax)
3.41 1 0.08%
3.72 :: 0.03%
4.67 0.07%
5.14 :: 0.08%
InfaPro (Contrasl)
3,15 + 0.03%
3,28 + 0.11%
3,62 0.11%
3,91 0.16%
Greedy SL
25.56 1.37%
27.97 # 0.75%
29.07 = 0,76%
30.38 #: 0.39%
ResNet-110 110 (E2E: 22.27 + L611%)
DGL. Relilovsky et al. 2020
24.96 + 1.18%
26.77 = 0.645
27.33 0.245
27.73 #: 0.188%
STL-10
IntaPro (Solimaas)
21,02 + 0.51%
21.28 +: 0.27%
23.60 + 0.49%
26.05 =: 0.71% %
InfaPro (Contrast))
20.99 1: 0.64%
22,73 +: 0.40%6
25.15 0.52%
26.27 d: 0.48%
Table 3: Trade-off between GPU memory footprint during training and test errors. Results of training ResNet- 110 on a single Nvidia Titan Xp GPU are reported. 'GC' refers gradient checkpointing (Chen al. 2016).
CIFAR 10 (butch size 1024)
STL-10 10 (butch sice = 128)
Mathods
Computational Overhead (Thoeetical / Wall Time)
Computational Overhead (Thoonticall / Wall Time)
Test
Menory Cost
Test Frror
Memory Cast
EIE Training
6.50 # 0.34%
9:40 GB
22:27 + 161%
10.77 GB
GC Chen et al 2016
6,50 :: 0.34%
3,91 GB (158. 496)
32.8% 27.55%
22.27 1 161%
4.50 GB (L 58.2%)
32.8% 727.0%
Intolro
0.13%
5.38
(1 (J42.88) (26)
1.3% /1.1%
0.57%
GB (L (2.99.9%)
1.7%
IntoPro",
6.74
4,22
(1/55.16))
3,3% 7.55
21.00
7.00% GB 33.9%)
InfoPro", K
6.93 0,20%
3.52 GB (1622.6%)
5.9% / 13,45
21.22 + 0.72%
4,08 GB (1 62. 1%%
5.9% 111.4%
Table 4: Single crop error rates (%) on the validation set of ImageNet. We use Tesla V100 GPUs for training.
Memory
Computational Overhead
Models
Methods
Batch Size
Top-1 Error
Top-5 error
(per GPU)
(Theonetical/ Wall Time)
E2R Training
1024
22.03%
5.93%
19.71
ResNei-101
IntoPrd", K
1024
21.85% 9
5.89%
12,06 GB (8)
5.7% / 11.7%
ResNet-152
EZE Training
1024
21.60%
5.92%
26.29 GB
InfoPro", K 1024
21.45% %
5.84%
15.53 GB (L-40.9%)
3.9% 8
E2E Training
$12
20.64%
5.40%
19.22 GB
ResNeXi-101, 3d
IntoPro", , K
512
20.35%
5.28%
11.55 GB (L (0.9)99))
2.7% / 5.66
large. For example, on CIFAR-IO, ResNet-32 + InfoPro achieves a test error of 12.75% with K = 16, surpassing greedy SL by 11.84%. For ResNet- 110, we note that our method performs on par with E2E training with K = 2, while degrading the performance by up to 0.8% with K 4. Moreover, InfoPro is shown to compare favorably against DGL under most settings.
In addition, given that our method introduces auxiliary networks, we enlarge network depth for greedy SL to match the computational cost of InfoPro, named as greedy SI.+. However, this only slightly ameliorates the performance since the problem of information collapse still exists. Another interesting phenomenon is that InfoPro (Contrast) outperforms InfoPro (Softmax) on CIFAR-10 and SVHN, yet fails to do so on STL-10. We attribute this to the larger batch size we use on the former two datasets and the proper value of the temperature T. A detailed analysis given in Appendix H
Asynchronous and parallel training. The results of asynchronous training are presented in Table as -Asy-InfoPro and it appears to slightly hurt the performance. Asy-InfoPro differentiates itself from InfoPro on that it adopts the cached outputs from completely trained earlier modules as the inputs of later modules. Therefore, the degradation of performance might be ascribed lacking regularizing effects from the noisy outputs of earlier modules during training (Löwe et al.) 2019). However, Asy-InooPro is still considerably better than both greedy SL and DGL, approaching E2E
7

---

Published as a conference paper at ICLR 2021
Table 5: Results of semantic segmentation Cityscapes. Nvidia GeForce RTX 3090 GPUs are used for training. 'SS' refers to the single-scale inference. 'MS' and 'Flip' denote employing the average prediction of multi-scale ((0.5, 1.75)) and left-right flipped inputs during inference. We also present the results reported by the original paper in the "original" row. DGL refers to decoupled greedy learning (Belilovsky et al. 2020).
Training
Training
Batch Size
mloU
Memory Cost
Compatational Overhead
Model
Crop Size
Algorithms
Iterations
SS
MS
MS-Fiip
(per GPL)
(Therrtical Wall Time)
E2E (original)
40k
769x760
77.82%
79.00%
79.30%
E2F: (ours)
40k
512x1024
79.1
79.81%
80.025:
19,43G
DeepLab-V3
IGLL
40k
512x1024
78.15%
79. 40%
79. 19.56%
R111
InfoPro",
40k
512x1024
79.37%
80.53%
80.54%
12.0IGB (L 38.2%)
6.4% /2.2%
(wl syncHN)
E2E (ours)
60k
512x1024
79.32%
79.95%
80.074%
19,43GB
InfoPro", =
40k
12
512x1021
79.99%
81.09%
81.20%
16.62GB (L (1.5)
6.1% / 12.1%
InfoPro K =
40k
640x1280
80.25%
81.33%
81.42%
17,00GB (L (2.5%)
10.3% / 42.9%
training. Besides, we note that asynchronous training can be easily extended to training different local modules in parallel by dynam- ically caching the outputs of carlier modules. To this end, we preliminarily test training two local modules parallelly
1 locol modale
2"d local moddlle
10-
18:04
9:93
10:04
13:23
10:33
10-
13:72
11.00
10:38
2.50
10:40
(cnoliickon kot 1/
8.42
9:63
5.75
19.57
13.49
8.66
8.56
130
2 GPUs when K === 2, using the same experimental proto- cols as Huo al. (011 8b) (train ResNett-110 10 with a batch size of 128 on CIFAR-10) and their public code. Our method gives a 1.5x speedup over the standard parallel paradigm of E2E training (the PataParallel toolkit in Pytorch). Note that parallel training has the same performance as simultancous training (i.e., "InfoPro" in Table 2 since their training processes are identical except for the parallelism.
9.0
18:97
10.55
0.34
0.48
45
(cnndlicion of ((b.
Figure 4: Sensitivity tests. The CIFAR-10 test errors of Re trained using InfoPro (K = 4) are reported. We vary λ and A2 for 1st and 3rd local modules respectively, with all other modules unchanged. We do not consider A = X2 =0, where we ohviously have Lintoro = Lintomo = 0.
Reducing GPUs memory requirements. Here we split the network into local modules to ensure each module consumes similar amount of GPU memory during training. Note that this is different from splitting the model into modules with the same number of layers. We denote the results in this setting by InfoPro", and the trade-off between GPU memory consumption and test errors is presented in Table 3 where we report the minimally required GPU memory to run the training algorithm. The contrastive and softmax loss are used in InfoPro" on CIFAR-10 and STL-100, respectively. One can observe that our method significantly improves the memory efficiency of CNNs. For instance, on STL-10, InfoPro* (K 4) outperforms the E2E baseline by 1.05% with 37.9% of the GPUs memory requirements. The computational overhead is presented in both the theoretical results and the practical wall time. Due to implementation issues, we find that the latter is slightly larger than the former for InfoPro". Compared to the gradient checkpointing technique (Chen et al. 2016), our method achieves competitive performance with significantly reduced computational and time cost.
Results on ImageNet are reported in Table The softmax loss is used in InfoPro" since the batch size is relatively small. The proposed method reduces the memory cost by 40%, and achieves slightly better performance. Notably, our method enables training these large networks using 16 GB GPUs.
Results of semantic segmentation on Cityseapes are presented Table 5 We report the mean Intersection over Union (mIoU) of all classes on the validation set. The softmax loss is used in InfoPro". The details of zb and w are presented in Appendix E Our method boosts the performance of the DeepLab-V V3 (Chen et al.) 2017) network and allows training the model with 50% larger batch sizes (2 - per GPU) under the same memory constraints. This contributes to more accurate statisties for batch normalization, which is practical requirement for tasks with high resolution inputs. In addition, InfoPro" enables using larger crop sizes (512x1024 + 640x1280) during training without enlarging GPUs memory footprint, which significantly improves the mloU. Note that this does not increase the training or inference cost.
4.2 HYPER-PAR PARAMETER SENSITIVITY AND ABLATION STUDY
The coefficient A1 and Ag. To study how X and A affect the performance, we change them for the 1st and 3rd local modules of ResNet-32 trained using InfoPro (Contrast), K = 4. with the results shown in Figure[4] We find that the earlier module benefits from small X2 to propagate more information forward, while larger X2 helps the later module to boost the final accuracy. This is
8

---

Published as a conference paper at ICLR 2021
compatible with previous works showing that removing earlier layers in ResNets has minimal impact on performance (Veit et al. 2016).
Table 6: Ablation studies. Test errors of ResNet-32 on CIFAR-10 are reported.
Ablation study. For ablation, test directly removing the decoder w or replacing the contrastive head &f by the linear classifier used in greedy SI, as shown in Table
E
==
K =
10.30 1. 0.20%
21.19 1. 0.52%
8.90 +: 0.17%
15.82 1: 0.34%
8.49 + 0.16%
14.13 + 0.22%
5 RELATED WORK
7.76 :: 0.12%
11.13 0.19%
Greedy training of deep networks is first proposed to learn unsupervised deep generative models, or to obtain an appropriate initialization for E2E supervised training (Hinton al.. 2006; [engio ctt al. 2007). However, later works reveal that this initialization is indeed dispensable once proper
networks architectures are adopted, e.g., introducing batch normalization layers (Toffe & Szegedy
2013). skip connections (He et al., 2016) or dense connections (Huang et al.] 2019) Some other
works (Kulkarni & Karande 2017 Malach & Shalev-Shwartz/ 2018 Marquez et al. 2018 Huang
cet al., 2011 (2) attempt to learn deep models in a layer-wise fashion. For example, BoostResNet
(Huang et al. 2018a) trains different residual blocks in ResNet (He al. 2016) sequentially with a
boosting algorithm. Deep Cascade Learning (Marquez et al. 2018) extends the cascade correlation algorithm (Fahlman & Lebiere| 1990) deep learning. aiming at improving the training efficiency. However, these approaches mainly focus on theoretical analysis and are usually validated with limited experimental results on small datasets. More recently, several works have pointed out the inefficiencies of back-propagation and revisited this problem (Nokland & Eidnes, 2019 Belilovsky itt al., 2019; 2020). These works adopt a similar local learning setting to us, while they mostly optimize local modules with greedy short-term objective, and hence suffer from the information collapse issue we discuss in this paper. In contrast, our method trains local modules by minimizing the non-greedy InfoPro loss.
Alternatives of back- paeck-roppggtioon have been widely studied in recent years. Some biologically- motivated algorithms including target propagation (Lee et al., 2015h) Bartunov et al. 2018) and feedback alignment (Lillicrap et al.] 2014] Nokland (0166) avoid back-propagation by directly propagating backward optimal activations or error signals with auxiliary networks. Decoupled Neural Interfaces (DNI) (Jaderberg et al. 2017) learn auxiliary networks to produce synthetic gradients. In addition, optimization methods like Alternating Direction Method of Multipliers (ADMM) split the ead-to-end optimization into sub-problems using auxiliary variables (Taylor et al.. Phoromooooansk.c jet al. 2018). Decoupled Parallel fack-propagation (Huo et al. 2018b) and Features Replay (Huo et al. 2018a) update parameters with previous gradients instead of current ones, and show its convergence theoretically, enabling training network modules in parallel. Nevertheless, these methods are fundamentally different from as they train local modules by explicitly implicitly optimizing the global objective, while we merely consider optimizing local objectives.
Information-theoree analysis in deep learning has received increasingly more attention in the past few years. Shwartz-Ziv Ziv & Tishby (2017) and Saxe et al. (209)) study the information bottleneck (IB) principle (Tishby al.) (2000) explain the training dynamics of deep networks. Achille & Soatto| (2018) decompose the cross-entropy loss and propose a novel IB for weights. There are also efforts towards fulfilling efficient training with IB (Alemi al. 2016). In the context of unsupervised learning, a number of methods have been proposed based on mutual information maximization (Oord al. 2018 Tian et al. 2020 Hjelm et al. 2019). SimCLR (Chen et al.) 2020) and MoCo (He et al. 2020) propose to maximize the mutual in information of different views from the same input with the contrastive loss. This paper analyzes the drawbacks of greedy local supervision and propose the InfoPro loss from the information- -hoemation--hhoretiic perspective as well. In addition, our method can also be implemented as the combination of contrastive term and a reconstruction loss.
CONCLUSION
This work studied locally supervised deep learning from the wetmmmattioo-thhorette perspective. We demonstrated that training local modules greedily results in collapsing task-relevant information at earlier layers, degrading the final performance. To address this issue, we proposed an information propagation (InfoPro) loss that encourages local modules to preserve more information about the input, while progressively discard task irrelevat information. Extensive experiments validated that InfoPro significantly reduced GPUs memory footprint during training without sacrificing accuracy. It also enabled model parallclization in an asynchronous fashion. InfoPro may open new avenues for developing more efficient and biologically plausible deep learning algorithms.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work is supported in part by the National Science and Technology Major Project of the Ministry of Science and Technology of China under Grants 2018AAA0100701, the National Natural Science Foundation of China under Grants 61906106 and 62022048, the Institute for Guo Qiang of Tsinghua University and Beijing Academy of Artificial Intelligence.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1:11447-198) 2018.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint wr1iv:1612.00410, 2016.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E. Hinton, and Timothy Lilli- crap. Assessing the scalability of biologically-motivaee deep learning algorithms and architectures. In NeurlPS, pp. 9368-9378, 2018.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In ICMI., pp. 583-593. PMLR, 2019.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In ICML, 2020.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Yoshua Bengio, Dong- Hyun Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning. arXiv preprint arXiv:1502.04156 2015.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang. Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 07155, 2019.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint urXiv:106.003877 2017.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604.6664 74, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Anna Choromanska, ECENYU Tandon, Sadhana Kumaravel, Ronny Luss, Irina Rish, Brian Kings- bury, Ravi Tejwani, and Djallel Bouneffouf. Beyond backprop: Alternating minimization with co-activation memory. stat, 1050:24, 2018.
Adam Coates, Andrew Ng. and Honglak Lee, An analysis single-layer networks in unsupervised feature learning. In AISTA7S, pp. 215-223, 2011.
MMSegmentation Contributors. Mmsegmentation, an open source semantic segmentation toolbox. htttps://githbbb.cm wtt.c/oo//pppoeee mm lab wwt.aaaammeeeeeettttii.o 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3222 2016.
Francis Crick. The recent excitement about neural networks. Nature, (37(6203) 1989.
Yang Dan and Mu-ming Poo. Spike timing-dependent plasticity of neural circuits. Neuron, 44(1): 23-30, 2004.
10

---

Published as a conference paper at ICLR 2021
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In ICML, pp. (188-255, 2009.
Scott E Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In NeurIPS, pp. 524-532, 1990.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In NeurIPS, pp. 2214-2224, 2017
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Furong Huang, Jordan Ash, John Langford, and Robert Schapire. Learning deep resnet blocks sequentially using boosting theory. In ICML, pp. 2058-2067, 2018a.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018b. URL
net forum? aatp:ummm-d-akaaaaaaaaaaaaaa ImxAb
Gao Huang, Zhuang Liu, Geoff Pleiss, Laurens Van Der Maaten, and Kilian Weinberger. Convo- lutional networks with dense connectivity. IEEE transactions on pattern analysis and machine intelligence, 2019.
Zhouyuan Huo, Bin Gu, and Heng Huang. Training neural networks using features replay. In NeurIPS, pp. 6659-6668, 2018a.
Zhouyuan Huo, Bin Gu, Qian Yang, and Heng Huang. Decoupled parallel backpropagation with convergence guarantee. In ICML, 2018b.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167, 2015.
Max Jaderberg, Wojciech Marian Czamecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In ICML, pp. 1627-1635. PMLR, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint orXiv:2004.11362 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14122 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114. 2013.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurlPS, pp. 1097-1105, 2012.
Mandar Kulkamn and Shirish Karande. Layer-wise training of deep networks using kernel similarity. arXiv preprint arXiv: /703.071 2017.
11

---

Published as a conference paper at ICLR 2021
Chen- Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In AISTATS, pp. 562-570, 2015a.
Dong- Hyyu Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pp. 498-515 Springer, 2015b.
Timothy P LLiliccrpp Daniel Cownden, Douglas B Tweed. and Colin J Akerman. Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:1411.02444 2014
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2014. URL http:://axxii org/ abs///b////// 1405 0312
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, pp. 2117-2125, 2017.
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. In NeurlPS, pp. 3039-3051, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders, arXiv preprint arXiv:1571.05644 2015.
Eran Malach and Shai Shalev-Shwartz. A provably correct algorithm for deep learning that actually works. arXiv preprint / 2018.
Enrique S Marquez, Jonathon S Hare, and Mahesan Niranjan. Deep cascade learning. IEEE transactions on neural networks and learning systems, 29(11):5475-5485 2018.
Alan Mosca and George D Magoulas. Deep incremental boosting, arXiv preprint arXiv:1708.0341 2017.
Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:608, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. NeurIPS, pp. 1037-1045, 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. arXiv preprint arXiv:1901.066 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.037488 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object
detection with region proposal networks. In NeurlPS, pp. 91-99, 2015.
Salah Rifai, Yoshua Bengio, Aaron Courville, Pascal Vincent, and Mehdi Mirza. Disentangling factors of variation for facial expression recognition. In ECCV, pp. 808-822. Springer, 2012.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, (019(12):12402)) 2019.
Ravid Shwartz- Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 11X3:17000000100 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
12

---

Published as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, pp. 1195-1204, 2017.
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Training neural networks without gradients: scalable admm approach. ICML, pp. 2722-2731, 2016.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physiccs/0004057, 2000.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In NeurIPS, pp. 550-558, 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096- 1196-1103, 2008
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. In NeurIPS, pp. 12635-12644, 2019.
Yulin Wang, Jiayi Guo. Shiji Song, and Gao Huang. Meta-semi: A meta-lcarning approach for semi-supervised learning. arXiv preprint wwXiv:2007.23394 2020a.
Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: dynamic approach to reducing spatial redundancy in image classification. In NeurlPS, 2020b.
Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In CVPR, pp. 2369-2378, 2020.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
A A TOY EXAMPLE
To further validate the proposed information collapse hypothesis, we visualize the "information flow" within deep networks using toy example. First, we establish a MNIST-STLI0 dataset via placing MNIST digits on a certain position (randomly selected from 64 can- didates) of a background image from STL-10. Then, three specific tasks can be defieed on MNIST-STL10, namely classifying digits, backgrounds and positions of numbers. We refer to the labels of them as y1, 1/2 and ya. respectively, as illustrated by Figure
Input 2c
Labels
11: Background
12: Digit
1/s: Position of Digit
Figure 5: Illustration of the
MNIST-STI.10 dataset.
We train ResNet-32 networks for the three tasks with greedy SL (K=4) and end-to-end training (K 1K) The estimates of mutual information [(,, y1), [h, 1/2) and [h,, y/3) are shown in Figure 6 with the same estimating approach Figure (details in Appendix G). Note that when one label (take yı for example) is adopted for training, the information related to other labels (1 (h, y2) and
I [h, 9s)) is task-irrelevant. From the plots, can clearly observe that end-to-end training retains all task-relevant information throughout the feed-forward process, while greedy SL usually yields less informative intermediate representations in terms of the task of interest. This phenomenon confirms the proposed information collapse hypothesis empirically. In addition, we postulate that the end-to-end learned early layers prevent collapsing task-relevant information by being allowed keep larger amount of task- iireeirtel information, which, however. may lead to the inferior classification performance of intermediate features, and thus cannot be achieved by greedy SL.
Aistral Inseraation shateme Eabil
Mttnal Arcomaatio betwenn stattew Labed tu
Metual Inderaaiion strree Hout Label
0.51
3,00
6.50
0.20-
201
0.91
2
8.77
0.3
R-t
037
0.8
0.3
miaadd wak
mread sak
0.76
K-
[takedd with
[tabeed g1
0.30
0.70
12
16
4.00
12
Lawe Todes
Laser
Figure 6: The estimates of mutual information between the intermediate features h and the three labels of MNIST-STLIO (see: Figure 5). i.e. yn (left, background), y2 (middle, digit) and ys (right, position of digit). Models are trained using greedy SL (K = 1, 4) supervised by one of the three labels, and the results are shown with respect to layer indices. "K= = 1" refers to end-to-end training.
B PROOF OF PROPOSITION
Proposition 1. Suppose that the Markov chain (y,r) + x - h holds. Then an upper bound of Lintaro given by
Lintomm S - - A2T(h, v) - LintoPm:
(6)
where A1 ax11 3), A2 = a33.
Proof. Note that, LintoPro is given by
LintaPro (h) aa-I(hl, x)) + 31(r", h]], a. 0,820, 0, s.t. r" argmnax
hrr
(7)
1(,,)>0 1(ra)=0
Due to the Markov chain (y,r) - + h, we have 1(h, (u, S T(h, Given that
I(h, (v,r") Hhh) H(h r*)
= =IH(h) - [[]]]*"]= + [H(h)r*) - H(lly,r")) =I(hrr*) + I(h,glr*)
(8)
we have
I(h,r") < T(h, x) T(h,ylr").
(9)
14

---

Published as a conference paper at ICLR 2021
By the definition of nuisance, note that r" and y mutually independent, and thus obtain
[h,, ylr") = H(yrr") ryyl,rrr)
H(()) rv)h,rr)
(10)
2 11 (1y)/ = ((.,v).
Combining Eqs. (9) and (10) we have
r(h, r-) - [(,, x)) ( y)-
(11)
Finally, Proposition proved by combining Eq. (7) and Inequality
C PROOF OF PROPOSITION 2
We first introduce a Lemma proven by Achille & Soatto 20188)
Lemma 1. Given a joint distribution p(æ, y), where y is discrete random variable, we can always find a random variable independent of such that x r), for some deterministic function f. Proposition 2. Given that r" = argmax,, T(ræ)>00 [(ra)=0 [(r, h) and that is deterministic function with respect the gap LintoPro LintoPro is upper bounded by
E A2 (xx y) I(h, y)]
(12)
Proof. Let f be the random variable in Lemma [ and then, since find deterministic function that maps x to y, we have
1(h, x)) I(h, (v, I(h, (h, v/r).
(13)
Assume = [[(h,x) - I(h, y)1 I(h, r"), in terms that r" = argmax, 1(,,))>0, ((r:)= [rr, h). we obtain
5 $ [L(h, x) I(h, y)1 I(h, F)
I[h, F) + I(h, y) I(h.y) v) I(h,
1(h, y/F) - (hh
(14)
=1(h, [(x, + 1(x v) I(h, y)
(y/F)) - -H h) - Hy) + Hy) + I(x, y) - I(h, v).
Since y and i are mutually independent, namely Hy) = Hy)), we have
< H()) (vlf, (v) H [z) [xx v) - 1 (h, = (y)xx) - Hw 1(x,)) (h, y) < (y]a)) [(x,v) T(h, v).
(15)
When considering as a deterministic function with regards to æ, we obtain Hy]æ) = 0, and therefore
I<x y) - I(h,
(16)
Given that LinfoPro LinfoPro
= >2C, we have
< Ao [1(x, y) I(h, v)1.
(17)
for which we have proven Proposition 2
D WHY MINIMIZING THE CONTRASTIVE LOSS MAXIMIZES THE LOWER BOUND OF TASK- RELEVANT INFORMATION?
In this section, we show that minimizing the proposed contrastive loss, namely
L log exp(2; 2,/T)
Cocntrast Σij 1/
2; = fo(h)),
(18)
Ek-1 Ligh exp(2, xxx/T//))
actually maximizes an lower bound task-relevant information [(h, 3). We start by considering simplified but equivalent situation. Suppose that have query sample zt, together with set
15

---

Published as a conference paper at ICLR 2021
x= ........ 2N) definitely, and other negative samples are randomly sampled, namely X {22) U Xaeg Then the expectation of Leontass can be written as
consisting of N samples with one positive sample 2P from the same class as zt
exp(=t =P/T)
E coontros] Egt,x
log Ex= exp(xx 2,/T)
(19)
Eq. (19) can be viewed as a categorical cross-entropy loss of recognizing the positive sample zP correctly. Hence, define the optimal probability of this classification problem pro X), which denotes the true probability of 2; being the positive sample. Assuming that the label of =P is
the positive and negative samples can be viewed as being sampled from the true distributions p 0(z =1))
(z,[X] X) can be derived as and p(=), respectively. As consequence, P (z,
p(zz (v) II 1izi p(z))
pla. lv) p(*.)
p (z (x)
(20)
Σn P(zj v) T1 p(zi)
p(=,l) p(z,)
which indicates that optimal value for exp(=t 2P 2P/T) is p(="ly) p(an)
Therefore, by assuming that st
uniformly sampled from all classes, we have
r==" y)
conntras] N
comtrast = Eyyx
log
p(z²)
(21)
N Σ
p( (8/)
j=1 p(a;)
p(= v)
Ey,x
log
p(*²)
(22)
p(z*ly) p(29)
>EEEx... Ex
p
p(2,)
p(zp)
p(z)(u)
Eyyx X
log
1
(23)
p(zply)
p(2,)
2, x,EX
1 p(29)
p(z,ly)
Ey.x
log
(N 1)B2z
(24)
p(zply)
p(z,)
p(z0) (N 11
By.
log
(25)
p(>ply)
p(zP) (N - 11]
2 Ey =0
log
(26)
((2Ply)
= = w) + log(N 1) - 1(h, v) + log(N 1).
(27)
In the above, Inequality 24 follows from Oord al. (2018). which quickly becomes more accurate
when N increases. Inequality (27) follows from the data processing inequality (Shwartz-Ziv &
Tishby 2017)). Finally, we have EeConnta llog(N 1) [hh, y). and thus minimizing Coontrast
under the stochastic gradient descent framework maximizes a lower bound of [lh, y).
E ARCHITECTURE OF AUXILIARY NETWORKS
Here, we introduce the network architectures of w, ab and & we use in our experiments. Note that, w is decoder that aims to reconstruct the input images from deep features, while tb and Φ share the same architecture except for the last layer. The architectures used on CIFAR, SVHN and STL-10 are shown in Table 7 and Table 8 Architectures on ImageNet are shown in Table and Table 10 The architecture of &े for the semantic segmentation experiments on Cityscapes is shown in Table II where we use the same decoder w as on ImageNet (except for the size of feature maps). An empirical study on the size and architecture of auxiliary nets is presented in Appendix
F DETAILS OF EXPERIMENTS
Datasets. (1) The CIFAR-10 (rizhevsky et al. 2009) dataset consists of 60,000 32x32 colored images of 10 classes, 50,000 for training and 10,000 for test. We normalize the images with channel
16

---

Published as a conference paper at ICLR 2021
Table 7: Architecture of the decoder CIFAR, SVHN and STL-10.
Input: 32x32/ 16x / 8x8 feature maps (96x96 48 488 24 on STL-10)
Bilinear Interpolation to 32x (96x96 on STL-10)
3x3 3 conv., stride=1, paddig=, output channels= BatchNorm-RRLUU
3x3 conv., stride=1. padding=1, output channels=3, Sigmoid
Table Architecture of ub and d on CIFAR, SVHN and STL-10.
Input: 32x32/ / 16x 16 8x8 feature maps (96x96 48 x48 24 on STL-
32x (96x96) input features: 3x3 conv., stride=2, padding=1, output channels=32, BatchNorm++RLLe RRLL 16x 16 (48x48) x48) input features: 3x3 conv., stride=2, padding=1, output channels=64, BatchNorm+ ReLU
padding=1, 8x8 (24x24) input features: 3x3 cony., stw. output channels=64, Rath
Global average pooling
Fully connected 32 64->128, ReLU
Fully connected 128-+10 for 2b or 128-128 for Φ
Table Architecture of the decoder w on ImageNet.
Input: 28 feature maps
1xl conv., stride=1, padding=0), output channels= 128, BatchNorm-RRlle
Bilincar Interpolation 56x55
3x3 conv., stride=1, padding=1, channels=32, chanme ReLL
Bilinear Interpolation to 1 112x112
3x3 conv., stride=1, padding=1, output channels=12, chapprcccsc RetchNorm-Rel.U
Bilinear Interpolation to 224x224
3x3 conv,, stride=1, padding=1, output channels=3, Sigmoid
Table 10: Architecture of sb on ImageNet.
Input: feature maps
Ix] conv., stride=1, padding=0, output channels=128. BatchNorm RRLL
3x3 conv,, stridc=2, padding=1, output channels=256, BatchNorm ReLU
3x3 conv., stride=2, padding=1, output channels=512, BatchNorm- ReLU
1x I conv,, stride=1, padding=0, output channels=2088, BatchNorm- ReLU
Global average pooling
Fully connected 2048-+100
Table 11: Architecture of &b on Cityscapes.
Input: 64x 128 feature maps, 1024 channels
3x3 conv., stride=1, padding,, output channels=512, BatchNorm -eNNtNoo-----UUUUUU- Dropout, p=0.1
xx 1 conv., stride=1, padding=0, output channels=19
means and standard deviations for pre-processing. Then data augmentation is performed by 4x4
random translation followed by random horizontal flip (He et al. 2016 Huang et al. 2019). (2)
SVHN (Netzer et al., (011) consists of 32x32 colored images of digits. 73.257 images for training
and 26,032 images for test are provided. Following Tarvainen & Valpola 2017): Wang et al. (2020a)
we perform random 2x2 translation augment the training set. (3) STL-I0 (Coates et al. 2011
contains 5,000 training examples divided into 10 predefined folds with 1000 examples each, and 8,000 images for test. We use all the labeled images for training and test the performance on the provided test set. Data augmentation is performed by 4x4 random translation followed by random horizontal flip. (4) ImageNet is 1,000-class dataset from ILSVRC2012 (Deng et al., 2009), with 1.2 million images for training and 50,000 images for validation. We adopt the same data augmentation and pre- pre-processing configurations as Huang et al. (2019 2018b): Wang et al. 2019 2020b): Yang et al. (2020). (5) Cityscapes dataset (Cordts al. (016) contains 5,000 1024x x 2048 level finely annotated images (2,97 (2,975/500/1. 1,555 for training, validation and testing) and 20,000 coarsely annotated images from 50 different cities. Each pixel of the image is categorized among 19 classes.
17

---

Published as a conference paper at ICLR 2021
Following Chen et al. (2017), we conduct experiments the finely annotated dataset and report the performance on the validation set. The training images are augmented by randomly scaling (fromm 0.5 to 2.0) followed by randomly cropping high-resolution patches (512x x 1024 or 640x 1280). At test time, we simply feed the whole 1024x2048 images into the model.
Networks and training hyper-parameters. Our experiments on CIAFR-10, SVHN and STL-10 based on three popular networks, namely ResNet-32/110 (He et al.) 2016) and DenseNet-BC-100-1 (Hang et al. 2019). The networks are trained using a SGD optimizer with Nesterov momentum of 0.9 for 160 epochs. The L2 weight decay ratio is set to le-4. For ResNets, the batch size is set to 1024 and 128 for CIFAR-10/SVHI and STL-10, associated with an initial learning rate of 0.8 and 0.1, respectively. For DenseNets, we use a batch size of 256 and an initial learning rate of 0.2. The cosine learning rate annealing adopted. Note that, the results of greedy supervised learning presented in Table 1 follows exactly the same experimental configurations stated here. On ImageNet, we train ResNet-101 10T and ResNet- 152 with batch size of 1024 and an initial learning rate of 0.4. For ResNeXt-101 32x8d. 8d, we use a batch size of 512 and an initial learning rate of 0.2. The number of training epochs is set to 90. Other hyper-parameters are the same as CIAFR-10 10. For the DeepLab-V3 model used in semantic segmentation, (Contributors] 2020) (with ResNet- 101 and synthetic batch normalization), except for using an initial learning rate of 0.015 when setting the batch size to 12 or using 640 1280 cropped patches.
follow the training configurations of MMSegmentattin
Local module splitting. Since ResNets consist of a cascade of residual blocks, which naturally should not be further divided into smaller parts, view each residual block minimal indivisible unit, or say, a basic layer (distinguished from a single convolutional layer). Particularly, the first convolutional layer of the network is individually viewed as basic layer. As a consequence, ResNet- 32 has 16 basic layers, and ResNet- 10 has 55 basic layers. If ResNet-32 is split into K = 8 local modules, then each module will have 2 basic layers. In the cases where the number of basic layers is not divisible by K, we assign one less basic layer to earlier modules. For example, if ResNet- 10 split into K = modules, the corresponding numbers of basic layers will be {13, 14, 14, 14). If ResNet- 10 is split into K = 16 modules, the corresponding numbers of basic layers will be {3} x 9 modules + {4] x 7 modules. For BCC, similarly, we view each dense layer (the composite function BN-ReLU-1 x 1 CComv-BN-ReLU-- xConvv as a basic layer following their paper (Huang et al. 2019). The first convolutional layer and the transition layers are viewed as individual basic layers. The splitting criteria is the same as ResNets. Notably, for InfoPro", the networks divided to make sure that each local module has the same memory consumption during training, and hence the aforementioned splitting criteria based on the same numbers of basic layers is not applicable, as we discussed in Section
G DETAILS OF MUTUAL INFORMATION ESTIMATION
In this section, we describe the details on obtaining the estimates of I (h, and I(h,y) y) we present
in Figures 2 and 6
As we have discussed in Section 3.3 the expected reconstruction error R(x x h)) follows 1(h, x) =
(xx) (x h) > H (x) ) R(
x: h Vincent et al. 2008 Rifai et al. 2012 Kingma & Welling
2013 Makhzani et al. 2015| Hjelm et al., 2019). Therefore, similar to Section 3.3 we estimate
Th, x) by training decoder parameterized by to obtain the minimal reconstruction loss, namely [hh, a)) ~ max, (x)) ) - (x h.). Note that, ideally, this bound can be arbitrarily tight provided that w has sufficient capacity. In specific, we use the same network architecture as Table 7 and train it for 10 epochs to minimize the averaged binary cross-entropy reconstruction loss of each pixel. An adam (Kingma& & Ba 2014) optimizer with default hyper- parameters (1r=0.001, betas=(0.9, 0.999), eps=1e-08. weight decay=0) is adopted. Naive as this procedure might seems, for one thing, we find that it is sufficient to reconstruct the input images well given enough information, and meanwhile distinguish different values of I(h, æ)) via the quality of reconstructed images, as shown in Figure 7 For another, we are primarily concerned with the comparisons of [h,, x)) between end-to-end training and various cases of greedy supervised learning rather than obtaining the exact values of [h, x). The same training process is applied to all the intermediate features h, and hence the comparisons are fair. Finally, since H (x) is constant, for the case of understanding, we simply present AverageBinaryCroo
Lnsropy (x) h) as the estimates of I (h, x),, equivalent to adding the constant 1 H ((x) to the real estimates of T (h,
18

---

Published as a conference paper at ICLR 2021
Input Images
Layer
Layer
Layer
Layer
Layer
End-to-end Training
Greedy SL (K-4)
InfoPro (K-4)
Figure 7: Visualization of the reconstruction results obtained from the decoder w.
Pojeet
Table 12: Test errors of ResNet- 110 trained by InfoPro (Contrast) on CIFAR-1Ó, with different ar- chitecture of Φ. "MLP" refers to the multi- layer perceptron.
Sak Cadrett,
ta (hanae) Kid
335
30
25
31
90-
93-
45
30--1
0.-
Auchitectue of
R=2
R-8
8.5
05
0.55
1.35
01
135
05
0.55
1.25
2-MLP
0.25
WSlt Distilet
1 Liner
634
005%
9.50 0.768
2MMPP
6.42
Figure Test errors of ResNet- I 10 trained by InfoPro (Contrast) on CIFAR-10. with varying width of the decoder w (leff) and the projection head & (right).
Coar a 3-MLP
6.40
0.0999
5,79 0.39%
M/P
6.07 0.1%
For (h, v). as we have discussed in Section 3.3 as well, since I [(h y) = (t)h) H)) ll p(y] we train an auxiliary classifier qu (y/l h) with parameters +b approximate p)y] such that we have T(h. y) Re max (H() x -E -log gr (v:(k)))) Here we simply adopt the test accuracy of 9 (y/h) as the estimate I(h, y). which is highly correlated to the value of - 1 generation performance that a classifier based on h is able to reach. Notably, we use a ResNet-32 as TΦ- For the inputs of lzp, we up-sample h to 32 x 32 and map h to 16 channels at the first layer. All training hyper- parammeareeters of the ResNet-32 are the same as Appendix F
qu q (v.)h))] (or say, the cross-entropy loss). This can be viewed as the highest
H MORE RESULTS
Size and architecture of auxiliary nets. Here we investigate how the auxiliary nets (i.e., w, sb and Φ) influence the performance of our method. Since vp and & share the same architecture, we study the decoder and the projection head & for example. We start by scaling their width with certain factor (which equals for our original design), and show the results in Figure g It can be observed that using larger w and & both improve the performance, but the effects are less significant. In addition, their sizes can be shrunk by up to 2 times without severely degrading the accuracy. We also test other architectures for & Table 12, where "Conv" and "Linear" refer to convolutional and linear layers, respectively. We find that involving at least one conv layer and a following MLP instead of simple linear layer are both important designs. Besides, although adding more conv layers further boosts the performance, we observe that this comes at considerably increased computational overhead.
Temperature T. In Figure 9 we change the temperature 7 for InfoPro (Contrast). Our method is robust when < 0.1. However, we find the training tends to be unstable when < 0.005.
Batch size. For a comprehensive comparison of InfoPro (Softmax) and InfoPro (Contrast), we vary the batch sizes for the SGD optimizer, and present the results in Table 13 It is shown that training models with small aaii-bachess for sufficient epochs is beneficial for InfoPro (Softmax), but produces limited positive effects InfoPro (Contrast). We also note that using small mini-batches
19

---

Published as a conference paper at ICLR 2021
Emmry Viize Raapemitm
Table 13: Performance of InfoPro (Contrast/Softmax) with varying batch sizes. Two settings are considered: training models with the same number of iterations (40/80/160 epochs for batch $ize=25655121122 and epochs (160 epochs). All other training papereras (i.c., the learning rate schedule, weight decay, etc.) remained unchanged. Test errors of ResNet- 10 on CIFAR- 10 are reported.
lakibu (Custar)) Rod
(Cotrall)
0000
2.00
D00
Triining Epochs
80
160
Treeprntion
Size
256
Figure 9: Performance of InfoPro (Con- trast) with varying temperature r. Test emorr of ResNet- 110 CIFAR-10. 10. K 8
InfoPro (Sofunax)
8,88 1. 0.36%
7.20 0.21%
7.01
0.34%
6.14 0.11%
5.95
0.19%
InfoPro (Contrast)
9.96 1: 0.29%
7.49 :: 0.35%
6.42 1: 0.08%
6.16 0.15%
6.19 1: 0.20%
InfaPro (Sotusak)
(Sofuuus)
11.22
0.10%
9.42 + 0.86%
9:41 0.27%
H.37 + 11.3%
8.04 0.29%
InfoPro (Contrast)
13.22 0.57%
0.13%
0.40%
1.18%
1.4%%
Table 14:
Performance of InfoPro with the VGG network (Simonyan Zisserman 2014). The averaged
errors and standard deviations of independent trials are reported. IntoPro (Sottma//Conrras refers to two approaches to estimating [h,, v).
Dataset
Network
Method
K
K
Greedy SL
8.88 + 0.11%
10.04 + 0.31%
VGG 11 (w/ BatchNorm) (E2E: 8,10 = 0.14%)
DGL
Belilovsky
2020
8.25 4: 0.12%
8.46 0.29%
CIFAR-10
InfoPro (Softmax)
8.30 + 0.10%
8.36 = 0.16%
InfoPro (Contrast)
8.18 + 0.15%
8.23 0.19%
Table 15: Object detection results on COCO [Li al. 2014]. initialize the hackbone of Faster-RCNN-FPN (Ren et al. 2015) using ResNet-101 trained by E2E training and InfoPro", K = 2 on ImageNet. The COCO style box average precision (AP) metric is adopted, where AP5o and AP7s denote AP over 50% and 75% IoU thresholds, mAP takes the average value of AP over different thresholds (50%-95%) and mAPs, mAPM and mAP, denote mAP for objects at different scales. All results are presented in percentages (%). The better results
are bold-faced.
Method
Backbone
mAP
APso
AP75
mAPs
MAPM
mAPL
ResNet-101 (E28 training)
40.0
60.7
43.7
22.7
44.2
51.4
Faste-RRNNN-FNN Ren at.
2015
ResNet-101 (InfoPro", K 2)
40.2
61.1
44.0
23.4
44.4
52.1
usually prolongs the practical training time as they cannot make full use of even a single GPU (Nvidia Titan Xp). Besides, when considering a short schedule with the same number of updates (iterations), adopting small mini-batcher significantly hurts the performance of InfoPro (Contrast).
This observation is consistent with previous works (Chen et al. 2020 He et 2020 Khosla et al.. 2020).
Empirical study on the tightness of the upper bound. Here, we empirically study the gap be- tween of the upper bound LinfsPro and Lintorro Since the gap bounded by e S A2 [[x y) I(, y)). we sim- ply need to check the gap between [(x, y) and T(,, To this end, we consider training a ResNet- 1 10 on CIFAR-I0 using InfoPro (Con- trast) with K = 2, and estimate the mutual infor- mation [h, y) between the outputs of the first local module and the label. In addition, to obtain a comparable estimate of I [xx y). we assume that in end-to-end learned networks, the intermediate features retain all the task-relevant information within the original inputs (which is empirically observed in this paper). Hence, we can estimate [(x y) by training the network in an end-to-end fashion, and estimating the mutual information I I(h, y) in the same position as the end of the first module. The comparisons between the estimates of I [(x, y) and I(h, y) are presented in Figure I0 One can observe that the gap shrinks gradually during the training process.
Tightness of the Upper Bound Conar
0.94
0.88
LinfoPro LintoPro is upper
0.55
0.62
0.79
0.76
0.73
0.70
0.67
7/e. s)
0.64
7.A. x)
0161
60
80
120
100
Training Emoch
Figure 10: The estimates of I [(x 9) and [(h, v).
Results with VGG are presented Table 14 Obviously, similar observations to Table 2 can be obtained. InfoPro achieves competitive performance with E2E training, while outperforming DGL.
Transferability. To verify the transferability of the models trained by the proposed method, we initialize the backbone of Faster-RCNN (Ren et al. 2015) using ResNet- 101 trained by E2E training
20

---

Published as a conference paper at ICLR 2021
and InfoPro", K = on ImageNet, and train it for the MS COCO (Lin et al. 2014) object detection task. The training process adopts the default configuration of MMDetection (Chen et al. (019)) with feature pyramid networks (FPN) (Lin et al.) 2017). The results reported in Table [5] It can be
observed that the locally learned backbone using InfoPro" outperforms its E2E counterpart in terms of average precision (AP). This result is consistent with the accuracy ImageNet.
ADDITIONAL DISCUSSIONS AND FUTURE WORK
Applying InfoPro to regression tasks. Although this paper mainly focuses implementing InfoPro in the context of classification based tasks (i.e., image classification and semantic segmentation), the formulation of LinfoPro and LintoPro general and flexible. As long the mutual information I(h, x)) and [h.,)) can be estimated, InfoPro is able to be used in more tasks. In vision tasks, I [(h, x) can usually estimated with decoder, we introduced Section 3.3 while for estimating T(h, u), the technique we discussed may be easily extended to the regression tasks (e.g., depth estimation, bounding box regression in object detection). For example, consider a target value E [0 corresponding to the sample and the hidden representation h, It can be viewed as a Bernoulli distribution where P(y = 1) yı, such that we have [(h, y) s m N N == yilog Uu (v Ih,) (1 y1 Jog (o = 0[h,)]}. As result, the auxiliary network Th yy h)) can be trained with the binary cross-entropy loss. This might also be approximated by the mean-square loss as it has the same minima the binary cross-entropy loss. In the future, will
focus on applying InfoPro to more complex tasks, such as 2D/3D detection, instance segmentation and video recognition.
21


Reviews:
[{"review_id": "oxUDvHcbTOb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzed the reason why locally supervised training led to performance degradation. And based on the analysis, the author proposed the information propagation loss (can be understood as the combination of a classification loss and a reconstruction loss) aiming to prevent information collapse. Equipped with the proposed method, 40% memory footprint can be reduced demonstrated by their experiments, which is surprising. \n\nStrength \n(1) Deep neural networks need heavy memory cost, especially for object detection and segmentation task. Reducing memory cost in the training can enlarge the batch size in the training, thus speed up the training process. This is a very promising direction.   \n(2) The paper is well motivated by analyzing traditional locally supervised training. Moreover, the analyze of information in features is also informative and reasonable. \n(3) The experimental results are good. The proposed method can achieve comparable performance with 40% less memory footprint. \n(4) The paper is clearly and is easy to understand.\n(5) The proposed method is simple but efficient. And it can be used in different tasks, like classification and semantic segmentation.\n\nWeaknesses \n(1) This is seldom comparisons with other methods in this paper. Only comparisons with the baseline are provided.\n(2) Is the computational overhead is measured by training time or theorical computation cost? The comparison of real training time is more informative.\n(3) Comparisons with other methods for reducing the memory consumption, e.g., [a] should also be added since they share similar aim. [a] Chen et al. Training Deep Nets with Sublinear Memory Cost.\n(4) Recent contrastive learning works show that hyper-parameter τ in contrastive loss plays a very important role, like MOCO. Though τ = 0.07 is fit for MOCO, the setting here is different from MOCO. In this paper, the contrastive loss makes use of label information. If the ablation study for training hyper-parameter τ in the contrastive loss function is provided, it will be good. \n(5) In the experiments, the contrastive loss outperforms cross-entropy loss. And the authors attribute this phenomenon to using larger batch size for contrastive loss. However, I think the strong regularization on intra-class representation may be the key factor. Anyhow, ablation study of different batch sizes for contrastive loss should be added here.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "awlOJvlPzF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe paper analyzes the pitfalls of locally supervised learning from the point of view of information propagation and proposes a new auxiliary loss that can facilitate locally supervised learning. The proposed loss, \"infopro loss\", is then relaxed to a tractable upper bound, which is then used instead. To implement the loss, mutual information is approximated with a decoder, as well as a classifier. The authors further introduce now contrastive learning fits in the framework as a lower bound maximization process regarding mutual information. The experimental results on standard datasets demonstrate the efficacy of the proposed method.\n\nI have enjoyed reading the paper quite a lot, and therefore recommend accepting the paper. Still, I have some reservations that I would love the authors to clarify via the rebuttal.\n\n# Strength\n\nThe paper is well written. There are some issues (which I detail in the weaknesses section), but most are very clear and easy to follow. There are some parts that are repetitive, but it does allow readers to scheme through without missing the important point.\n\nThe part that I like most about the paper is how proposition 1 and appendix D is presented. They are theoretically well-motivated and gladly seems to work, despite the relaxation. I personally think appendix D deserves more attention in the main text, but this is my personal preference.\n\nThe results clearly show that the method improves over simple greedy locally supervised learning, as well as other attempts at this problem. \n\n# Weaknesses\n\n## Section 3.1, regarding equation (1)\n\nI am not 100% sure on this, but shouldn't the third last sentence of Section 3.1 read \"...under the goal of retaining as much information of the input as possible\"? I am not sure this is actually a constraint, and the I(h,x) term is not explicitly task-relevant information.\n\n## Section 3.3, estimating I(h,y)\n\nIt is not clear to me how I(h,y) is finally approximated. Shouldn't p(y|h) disappear after approximation? In the provided approximation it still exists. If p(y|h) is somehow directly used, isn't q not needed at all?\n\n## Section 3.3 final equation\n\nIn my opinion, even when it is not referred to in text, equations should have numbers so that future readers can refer to it.\n\n## Regarding Asy-InfoPro\n\nIt is somewhat unclear whether the dynamic caching was used at the end. Are the experiments in Table 2 with dynamic caching? From my current understanding, it does not seem to be the case, which leads to my second issue.\n\nThis is assuming that the results are without dynamic caching as this seems most logical. The explanation in the \"Asynchronous and parallel training\" paragraph was not obvious to me during the first read. The second sentence could be rephrased and split so that it becomes clear that the distinction between the two modes is that transient feature maps are seen/unseen and that this has a regularizing effect.  This then brings up the important question, whether the dynamic caching version suffers from the same fate---it should not. Having this experimental verification would greatly strengthen the observations in this paper.\n\n## Softmax vs Contrastive\n\nI am curious as to why contrastive works better. Could it be a coincidence of better hyperparameter tuning? Because the softmax version is a direct estimate on mutual information, whereas the contrastive one optimizes the lower bound.  It would be nice if this was discussed in more detail (I do understand that there is already very little space though!)\n\n## Computational overhead\n\nIs the computational overhead including the memory transfer cost? Are the results the actual physical measures observed via monitoring the resource usage? Or are they theoretical? I might have missed it, but this is not very clear to me.\n\n## Typos and grammar errors\n\nThere are quite a few grammatical mistakes throughout the paper. For example, at the beginning of Section 2, it is more natural to write \"we start by considering\" than \"we start with considering\". In the italic question in the second paragraph of Section 2, \"..., even the former...\" should be \"..., even though the former...\". While I generally did not find these errors to be critical, I would suggest a thorough proofread.\n\nI also found a typo in Appendix B. The last sentence should refer to equation (10) not (9).\n\n## Early stopping, and the choice of the number of epochs\n\nThe training process in this paper does not utilize early stopping. While this is somewhat mitigated by the fact that multiple runs are performed, this is in fact another source of overfitting to the dataset and is strictly speaking tuning hyperparameters on the test set. This is a practice that should be avoided.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8yIna2y6Ri6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Briefing:\nThis paper proposes a new infoPro loss for locally supervised training that alleviates the problem from greedy supervised learning, which collapsing task-relevant Information at the beginning of the layers.\nThe infoPro loss requires an auxiliary network to infer the Information I(h,y) (For ImageNet, the paper used ASPP).\n\nStrong points:\n\nAnalysis of the phenomenon when applying the greedy supervised learning (GSL): \nThe two observations from the authors seem natural.\n\n(1): GSL underperforms E2E training (little trivial for the reviewer)\n\n(2): Each separated layer trained by GSL captures more discriminative features.\n\nHowever, the information-based explanation of why the task-relevant info at the early stage of the network is essential (information collapse hypothesis) seems new and worth considering for the reviewer. \n\nInfoPro-loss:\nTheoretical analysis of the loss seems credible to the reviewer.\n\nExperiments:\nMutual Information at each layer index looked interesting.\n\nInterestingly, the proposed method outperforms E2E training in the ImageNet classification task.\n\nHyper-parameter does not seem to be sensitive.\n\nWeek point:\n\nUsing the additional network might not be fair when strictly comparing the performance. But practically, when training the large network, the size of the auxiliary network seems ok.\n\nComments:\n\n(1) It is not strongly required. Is it available to apply the method to the detection task?\n\n(2) Adding the experiments for other widely used network s.a. VGG, would be meaningful.\n\n(3) Can we use LR-ASPP in MobileNet v3 instead of ASPP? or Can we use a heavier Decoder for better performance?\n\n(4) How come when we set K=4 and set a larger batch size? We do not need to waste the remaining memory.\n\n(5) Training the network larger than ResNet-152 from the proposed method would be impressive.\n\n(6) Can we replace the imageNet pre-trained (by E2E) backbone with that trained by the proposed method? Experiments or discussion verifying the transferability of the trained network would be required.\n\n \nRating: Nice paper, Accept \n\n(1) But the reviewer wants a discussion with the author about the comments mentioned above\n\n(2) Literature checks from the other reviewers may affect the rating.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8JoPOQ4wVhK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a strategy for training feed-forward networks in a more memory-efficient manner by employing local as opposed to end-to-end supervision. End-to-end/global (E2E) supervision as the dominant paradigm in training deep networks considers a loss function at the very end of the network for backpropagation of the resulting gradients, whereas local supervision injects supervisory signals (such as the same E2E objective, e.g. classification) at intermediate layers in the network. The benefit of such intermediate supervision is the ability to train larger networks in smaller chunks piece by piece, where each individual training is more memory efficient due to reduced need to store activations (and weights and biases) in GPU memory. As a drawback, however, it had been shown earlier that such local training is less optimal than global training in terms of the achievable generalization performance. The authors propose a new training strategy that aims at combining the memory efficiency of local supervision and piecewise training with the error performance of global training. Considering a given intermediate layer, the paper motivates to maximize the mutual information between the activations in this layer and the input signal to retain relevant information, while minimizing the mutual information of the activations and a nuisance variable, where the nuisance is defined as having no mutual information with the target variable (e.g. the classification prediction). The authors argue that this local supervision allows to train the features at the intermediate layer such that they carry relevant information from the input to the target variable without resorting to direct supervision with the target variable. Direct computation of the nuisance variable is infeasible and the authors propose a bounded approximation. \nEmpirical results are discussed on five common vision datasets and two CNN architectures in relation to the existing state of the art.\n\n### Strengths\n**[S1]** The paper is largely written well and addresses a relevant problem. Contributions and claims are laid out well.\n\n**[S2]** The method has potentially multiple benefits: (a) more memory efficient training without loss of performance, (b) speedups for asynchronous training, (c) use of larger batch size or larger models.\n\n**[S3]** The method is motivated reasonably well by the argument of minimizing the loss of mutual information with the input variable while minimizing the mutual information of nuisance variables with respect to the target quantity.\n\n**[S4]** The inclusion of Greedy SL+ to account for the additional degrees of freedom of the proxy networks in InfoPro is a good attention to detail in the empirical evaluation.\n\n**[S5]** The paper looks at a reasonable set of tasks and studies the performance on relevant data, for instance Imagenet and Cityscapes semantic segmentation (but consider [W1] below)\n\n### Weaknesses:\n**[W1]** Baselines of empirical comparisons: With the exception of Fig 3, the empirical results are compared to only a comparatively simple baseline (Greedy SL/SL+), but not the same state of the art mentioned in section 4.1 (DGL, DIB, BoostResNet). This seems a rather odd omission, particularly as Fig 3/Section 4.1 elaborate on alternative methods (DGL, DIB, BoostResNet). It would support the paper's case to either include the results of those in table 2 or elaborate on their absence.\n\n**[W2]** Fig 3: The scale of the y-axis is chosen in a way that a casual reader may visually misinterpret the absolute difference in error of the shown methods. For instance, at first glance DGL at K=2 seems twice as bad as Infopro, while in actuality is only about 14% worse (error from 7.76% to ~8.8%)\n\n**[W3]** A methodical concern is the choice of $\\phi$ and $\\psi$ and the limited discussion around their choice (section 3.3, App. E): What is the sensitivity of the optimization with respect to their size and structure, what happens if I make them larger or smaller, how small can I make them? How would they look like for objectives other than classification?\n\n### Further comments:\n**[C1]** It would be insightful to consider what the implications for networks with recurrent structures would be.\n\n**[C2]** It took a while to infer in section 3.3. that $\\mathcal{R}$ is the reconstruction of the input data. A short sentence there may help future readers go through more smoothly.\n\nI feel that I have learned something from the paper. It discusses the contributions, motivations and method reasonably thorough. My concerns are largely around the empirical substantiation of the claims, see [W1].", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper brings new knowledge in the field of locally supervised learning, and as such it should be accepted.  The authors should keep all reviewers' comments into account when preparing their camera ready version.
Decision: Accept (Poster)