Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing
Authors: Peng Ye
Shengji Tang
Baopu Li
Tao Chen
Wanli Ouyang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Residual Networks
Network Loafing
Stimulative Training

Abstract: Residual networks have shown great success and become indispensable in today’s deep models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training strategy to strengthen the performance of residual networks. As residual networks can be viewed as ensembles of relatively shallow networks (i.e., unraveled view) in prior works, we also start from such view and consider that the final performance of a residual network is co-determined by a group of sub-networks. Inspired by the social loafing problem of social psychology, we find that residual networks invariably suffer from similar problem, where sub-networks in a residual network are prone to exert less effort when working as part of the group compared to working alone. We define this previously overlooked problem as network loafing. As social loafing will ultimately cause the low individual productivity and the reduced overall performance, network loafing will also hinder the performance of a given residual network and its sub-networks. Referring to the solutions of social psychology, we propose stimulative training, which randomly samples a residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the given residual network, to act as extra supervision for sub-networks and make the overall goal consistent. Comprehensive empirical results and theoretical analyses verify that stimulative training can well handle the loafing problem, and improve the performance of a residual network by improving the performance of its sub-networks. The code is available at https://github.com/Sunshine-Ye/NIPS22-ST. 

Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing
Peng Ye't Shengji Tang Baopu Li Tao Chen'r, Wanli Ouyang² School of Information Science and Technology, Fudan University, Oracle Health and AI, USA, 'hee University of Sydney, SenseTTime Computer Vision Group, Australia, and Shanghai Al Lab yepeng208fudan.edde edu.c
Abstract
Residual networks have shown great success and become indispensable in today's deep models. In this work, we aim to re-investigate the training process of residual networks from novel social psychology perspective of loafing, and further pro- pose new training strategy to strengthen the performance of residual networks. As residual networks can be viewed as ensembles of relatively shallow networks (i.e., unraveled view) in prior works, we also start from such view and consider that the final performance of residual network is co-determined by a group of sub-networks. Inspired by the social loafing problem of social psychology, we find that residual networks invariably suffer from similar problem, where sub- networks in residual network are prone to exert less effort when working part the group compared to working alone. We define this previously overlooked problem network loafing. As social loafing will ultimately cause the low in- dividual productivity and the reduced overall performance, network loafing will also hinder the performance of given residual network and its sub-networks. Referring to the solutions of social psychology, we propose stimulative training, which randomly samples residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the given residual network, to act as extra supervision for sub-netwe nebbrrs and make the overall goal consistent. Compre- hensive empirical results and theoretical analyses verify that stimulative training can well handle the loafing problem, and improve the performance of residual network by improving the performance of its sub-networks. The code is available at htttp://github.com/uusiiii Set.oo////Suuiiiiee//i/2/2---/
Introduction
Since ResNet wins the first place at the ILSVRC-2015 competition, -unfee-but-effective residual connections are applied in various deep networks, such as CNN, MLP, and transformer. To explore the secrets behind the success of residual networks. numerous studies have been proposed. He et. al [1] exploit the residual structure to avoid the performance degradation of deep networks. Further, He al (2) consider that such a structure can transfer any low level features high level layers in forward propagation and directly transmit the gradients from deep to shallow layers in backward propagation. Balduzzi et. al [ find that residual networks can alleviate the shattered gradients problem that gradients resemble white noise. In addition, Veit et. al [] experimentally verify that residual networks can be seen as a collection unraveled view. Following this view, Sun et. al [5] further attribute the success of residual networks to shallow sub-networks, which may correspond to the low-degree term when regarding the neural network polynomial function. Since the unraveled view is supported both experimentally and
numerous networks of different lengths, namely
*Corresponding Author (eetchen @ fuuan.edu.cc).. 'Equal Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

90
80
70
++ Restlet20
Topl Accuracy
60
ResNet32
50
ResNet44
ReaNet55
40
ResNet110
Resteet1202
20
0
ResMet20
ResNet32
ResNet44 ResNet56 ResNet110 ResNet1202
Different Networkss from ResNetx
Figure 1: Different residual networks invariably suffer from the problem of network loafing, and deeper residual network tends to have more serious loafing problem. All these networks trained on CIFARIO dataset. The horizontal axis means the sampled different sub-networks from ResNetx.
theoretically, we further investigate some interesting mechanism behind residual networks based on this inspiring work. Specifically, we treat residual network the ensemble of relatively shallow sub-networks and consider its final performance to be co-determined by group of sub-networks.
In social psychology, working in group is always tricky thing. Compared with performing tasks alone, group members tend to make less efforts when working as part of group, which is defined as the social loafing problem 66 7. Moreover, social psychology researches find that increasing the group size may aggravate social loafing for the decrease of individual visibility S 9 Inspired by these, we find that the ensemble-like networks formed by residual connections also have a similar problem and behavior. As shown in Fig. I, we can see that, different residual networks invariably suffer from the loafing problem, that is, the sub-networks working in a given residual network are prone to exert degraded performances than these sub-net networks working individually. For example, a sub-networt ResNet32 in the ResNet56 only has 46.80% Topl accuracy, much lower than ResNet32 trained individually with accuracy of 92.63%. Moreover, the loafing problem of deeper residual networks is inclined be more severe than that of shallower ones, that is, the same sub-network in deeper residual networks constantly presents inferior performance than that in shallower residual networks. For example, ResNet20 within ResNet32 has 55. 55.28% Topl accuracy, while ResNet20 within ResNet56 (deeper than ResNet 32) only has a 27% Topl accuracy. Such problems have not define this previously overlooked problem as network loafing 2 As social psychology researches show that social loafing will ultimately cause low productivity of each individual and the collective [1O, [T] we consider that network loafing may also hinder the performance of given residual network and all of its sub-networks.
been addressed in the literature so far as know. Hereafter,
In social psychology, there groups: 1) establishing individual accountability by increasing the individual supervision and 2) making tasks cooperative by setting up the overall goal B Inspired by this, propose novel training strategy for improving residual networks, namely stimulative training. In details, for each mini-batch during stimulative training, besides the main loss of the given residual network in conventional training, we will randomly sample a residual sub-network (individual supervision) and calculate the KI-ddieereence loss between the sampled sub-network and the given residual network (consistent overall goal). This simple yet effective training strategy can relieve the loafing problem of residual networks, by strengthening the individual supervision of sub-networks and making the goals of residual sub-networks and the given residual network more consistent.
commonly used solutions for preventing social loafing within
Comprehensive empirical analyses verify that stimulative training can solve the loafing problem of residual networks effectively and efficiently, thus improve both the performance of a given residual network and all of its residual sub-networks by large margin. Furthermore, we theoretically show the connection of the proposed stimulative training strategy and the improved performance of a given residual network and all of its residual sub-networks. Besides, experiments on various benchmark datasets using various residual networks demonstrate the effectiveness of the proposed training strategy. The contributions of our work can summarized as the following:
2 Network loafing is just a loose analogy to describe behavior in neural networks that has strong connection with biology.

---

We understand residual networks from social psychology perspective, and find that different residual networks invariably suffer from the problem of network loafing.
We improve residual networks from social psychology perspective, and propose simple- but-effective stimulative training strategy to improve the performance of the given residual network and all of its sub-networks.
Comprehensive empirical and theoretical analysis verify that stimulative training can well solve the loafing problem of residual networks.
2 Related Works
2.1 Unraveled View
As one pioneer work to investigate residual networks, 4 experimentally shows that residual networks can be seen as collection of numerous networks of different length, namely unraveled view. Subsequently, [5] follows this view and further attributes the success of residual networks to shallow sub-networks first. Besides, [5] considers the neural network as a polynomial function and corresponds shallow sub-network to low-degree term to explain the working mechanism of residual networks. Similar to [4] and (5), this paper also investigates residual networks from the unraveled view. Differently, inspired by social psychology,
further reveal the loafing problem of residual networks under the unraveled view. Besides, we propose a novel stimulative training method to relieve this problem and further unleash the potential of residual networks.
2.2 Knowledge Distillation
As classical method, knowledge distillation 13 transfers the knowledge from a teacher network to student network via approximating the logits 22 14 15] or features 166 17
[8] 19 output. To avoid the huge cost of training high performance teacher, some works abandon the naive teacher- student framework, like mutual distillation (20) making group of students learn from each other
online, and self distillation [21] transferring knowledge from deep layers to shallow layers. Generally, all these distillation methods need to introduce additional networks or structures, and employ fixed studdet-t pairs. As a comparison, our method does not require any additional network or structure, and the student network is randomly sampled sub-network of a network. Besides, our method is essentially designed to address the loafing problem of residual networks, which is different from knowledge distillation that aims to obtain compact network with acceptable accuracy.
2.3 One-shot NAS
One-shot NAS is an important branch of neural architecture search (NAS) 22 23 24 25 26 27 Along this direction, (28) trains an once-for-ale (OFA) network with progressive shrinking and knowledge distillation to support kinds of architcctural settings. Following this work, BigNAS [29] introduces several technologies to train a high-quality single-stage model, whose child models can be directly deployed without extra retraining or post-processing steps. Both OFA and BigNAS aim at simultaneously training and searching various networks with different resolutions, depths, widths and operations. Differently, the proposed method aims at improving given residual network, thus can be seamlessly applied to the searched model of NAS. As OFA and BigNAS are not designed to solve the loafing problem, their sampling space and supervision signal are also different with the proposed method. More importantly, the social sishial-sychooog--ine -nnspiiredddidddddeeeee problem of network loafing may explain why OFA and BigNAS work.
2.4 Stochastic Depth
As regularization technique, Stochastic Depth [30] randomly disables the convolution layers of residual blocks, to reduce training time and test error substantially. In Stochastic Depth, the reduction in test error is attributed to strengthening gradients of earlier layers and the implicit ensemble of numerous sub-networks of different depths. In fact, its improved performance can be also interpreted as relieving the network loafing problem defined this work. The theoretical analysis in this work can be also applied to Stochastic Depth. Besides, for better solving the loafing problem, our method samples sub-networks with ordered depth, and uses an additional -lv-dieeeeecce loss to provide more achievable target and make the output of given network and its sub-networks more consistent.
3

---

network loafing
60%
Building blook
(65%6)
Skip cxmmazzioon
70%
(75%)
fua
Moin luse
Residal modile
77.3%
(a) Common training scheme suffers from severe network loafing problem
K1.
loe.
fu
Aut
(65%)
80%
Aux
(75%)
Maiin
81.07%
(b) Stimulative training scheme relieve network loafing and improve the performance Figure 2: Illustration of common and stimulative training schemes. Stimulative training can relieve the network loafing problem, and improve the performance of a given residual network (from 77.3% to 81.07%) and all of its sub-networks (e.g., from 60% to 79%).. 65% and 75% denote the individual performance of each sub-network.
Stimulative Training
3.1 Motivation
Social loafing is social psychology phenomenon that individuals lower their productivity when working in group. Based on the novel perspective that residual network behaves like an ensemble network 4 find various residual networks invariably exhibit loafing-like behaviors shown in Fig. and Fig. 4 which we define as network loafing. As social loafing is "a kind of social disease' that will harm the individual and collective productivity [10), we consider network loafing may also hinder the performance of residual network and its sub-networks. To alleviate network loafing, it is intuitive learn from social psychology. There are two common methods for solving social-loafing problem in sociology, namely, establishing individual accountability (i.e., increasing the individual supervision) and making tasks cooperative (i.e., setting up the overall goal) R 9 In order to increase individual supervision, we sample sub-networks in the whole network and provide extra supervision to train each sub-network sufficiently. For the overall goal, we adopt KL divergence loss to constrain the output of sub-networks not far from that of the whole network, which aims at reducing the performance gap and driving the sampled sub-networks to develop cooperatively.
3.2 Training Algorithm
In this subsection, we briefly illustrate the working scheme of the proposed stimulative training strategy, and show its difference from the common training method. As shown in Fig. Fig.2 common training only focuses on optimization of main network, thus suffers from severe network loafing, that is, sub-networks lower their performance when working in an ensemble. For example, shown in Fig. aa) sub-networks within the residual network only have an accuracy of 60% and 70%, much lower than their individual accuracy of 65% and 75%. As a comparison, stimulative training optimizes the main network and meanwhile uses it to provide extra supervision
a sampled sub-network at each training iteration, thus well handles the loafing problem. In the test procedure, our method can adopt the main network or any sub-network as the inference model, thus requiring the same or lower memory and inference cost compared with a given residual network.
Formally, for a given residual network to be optimized, define the main network as D,, and the sub-network as D.. All the sub-networks share weights with the main network, and make up the sampling space
[D.ID, = T rD,m)), where is sampling operator, usually random sampling. In the training process, we randomly sample sub-network at each iteration to ensure the whole sampling space can fully explored.
make the training more efficient and effective, we define new sampling space obeying ordered residual sampling rule to be discussed in Section 3.3 Denoting 0pm and Dp as weights of the main network and the sampled network respectively, æ as the mini- batch training sample and y its label, Z as the output of network, the total loss of stimulative

---

Main Network Stage
Outputt
Samsling
Stochastle Residual Sampling
Main Network Tep 1:75.34%
Main Network Tupl: 530%%%%33%
(M
Chaset (tue) (tee))
3. Ordered Residual Sampling Main Network Top1:81077%
Keep reideal
strucura
sampling
easke training
OC O(n)
Coafora to acanal differential equation
Oapel
Seea
Figure 3: Illustration of ordered residual unraveling rule. Compared with two other sampling
strategies, ordered residual sampling can facilitate training and reduce the sampling space to get better main network performance.
training is computed as
Limt
()). ()), D2 E θ
(1)
main supervision
direction supervision
where the CE and KL are the standard cross entropy loss and KL divergence loss. À is the balancing coefficient, and CE loss denotes the main supervision for the main network. K loss provides an extra supervision for the sampled sub- netone,wor,, and guarantees all sub- neb-networks to be optimized in the same direction with the main network, which can be considered as setting a common goal for the "ensemble network". We the standard stochastic gradient descent for updating the model, denoted ac.t. where 7) is the learning rate. Thanks to the weight-sharing between as 0z!! DD.. 10 sub-networks and the main network, we only need update 0D.,. once for each iteration to optimize both the 8.., and 0D.. The pseudo code is shown in Appendix C.1.
3.3 Ordered Residual Sampling
Network loafing is based the novel view that residual networks can be seen the ensemble of numerous different sub-networks It is intuitive to adopt raw unraveled view to design sampling space, namely raw sampling. However, there exist two problems in raw sampling. First, introduces a mass of single branch sub-networks which are hard to optimize, as shown in the left sub-figure of Fig. 3 Second, with the growth of network depth, the size of sampling space increases exponentially. For residual stage of n blocks, there are 2" sub-networks, which constitutes a too large space to explore. Under limited computation resources, it difficult provide sufficient supervision each sub-network, causing insufficient stimulative training.
Therefore, in order facilitate stimulative training, we re-design two different sampling ways: stochastic residual sampling and ordered residual sampling. Fig. shows a three-block residual network simple illustration for three sampling ways. For sampling. we obey the raw definition of 4. that means each sub-network is composed of stacked convolution layers. For stochastic residual sampling, we keep the basic structure of residual blocks and skip some blocks randomly. For ordered residual sampling, we also keep the residual structure but skip blocks orderly, (i.c.. always skip the last several blocks). We implement these sampling methods for MobileNetV3 [3] on CIFAR- 1000. Experimental results show that ordered residual sampling performs better than other two methods (by 5.77% and 0.58% respectively). To explain the superiority of ordered residual sampling, we should pay attention to keeping residual structure and ordered sampling. For keeping residual structure, it can drive the network training process easier and improve the final performance [T] For ordered sampling, it can noticeably reduce the size of sampling space (from ((2 to O(n)), making it possible to train each sub-networf sufficiently. Besides, all deep networks with residual connections can be approximated by neural differential equations (32), which treat discrete network layers as
5

---

Topa Acurary 1991
18
Top3 Arcunny
62-
T0p
POTTuII Thinin Connnc Thaning
8
Thaniing
Semaiste Tramnin
Common Training (CT)
Conncce Thaninn Xm
30
30-
130
120
1.60
160
180
200
220
100
120
1.00
160
160
200
200
150
120
160
162
180
260
220
PlopstM))
PlopstM)
Plops0M)
(a) sub-networks
(b) 48 sub-networks
(c) 144 sub-networks
Figure 4: Stimulative training using 6, 48, and 144 sub-networks. With more sub-networks involved, the network loafing problem could be relieved better, and the performance the residual network and all of its sub-networks could be improved more significantly. We train MobileNetV3 with different schemes on CIFAR 100 dataset.
80
Max Top1 Aee (S)
Moan of Topt Acc Oh)
73
20
70
78-
Std
50
144
40
144
Number Sammpieed Subnetworks
Number Sampled Subnetworks
Number of Sampled Subnetworks
Maximum Value
Mean Value
(c) Standard Deviation
Figure 5: (a) Maximum, (b) mean value and (c) standard deviation when training MobileNerV3 with different strategies on CIFARI00 dataset. Sampling 0 sub-networks denotes the common training strategy. Stimulative training with more sub-networks can provide higher maximum, mean and lower standard deviation.
continuous and regard the layer output the result of adding integral to the layer input. From this perspective, ordered sampling will maintain the continuity of residual networks. Because these outstanding features, we will adopt ordered residual sampling in the following experiments.
Empirical Analysis
In this section, we conduct extensive experiments analyze and compare the proposed stimulative training and the common training strategy in detail. We first illustrate the loafing problem of residual networks with different training strategies. We further demonstrate statistical characteristics of the performance of the trained residual network and all of its sub-networks. Then, we show the performance drop when destructing trained residual networks at test time, including deleting individual layers, deleting more layers and permuting layers. Besides, we also record the KL distance between given residual network and all of its sub-networks in the whole training process. For the convenience of analysis, we always use NAS-searehed model MobileNetV3 [31] as the residual network and train on CIFAR 100, unless otherwise specified. The detailed experimental settings can be found in Appendix C.2.
4.1 Experiment 1: Checking the loafing problem of residual networks with different training strategies
First. we find that the NAS-scarched residual network MobileNetV3 still suffers from the loafing problem. As shown in Fig. 4 when applying the common training strategy to MobileNetV3, the sub-networks from MobileNetV3 (blue solid circle) have much lower accuracies than the same sub-networks individually trained (green five-pointed star), and the performance gap increases when the size of the sub-network decreases. Second, we show that the proposed stimulative training strategy can well handle the loafing problem. As shown in Fig. 4c)) when applying the stimulative training strategy to MobileNetV3, the sub-networks from MobileNetV3 (red five-pointed star) have even better performance than the same sub-networks individually trained (green five-pointed star). More interestingly, as shown Fig. 4(aa even when we apply stimulative training with 6 sub-networks (red five-pointed star) to MobileNetV3. V3, the performance of all the sub-networks (red solid circle) can

---

be also greatly improved compared to that of common training (blue solid circle). Besides, As shown in Fig. [ with more sub-networks involved in stimulative training, the network loafing problem could be relieved better, and the performance of given residual network and all of its sub-networks could be improved more significantly. More similar results of different datasets and residual networks can be found in Appendix A.
4.2 Experiment 2: Collecting statistical characteristics of the performance of all residual sub-networks
To demonstrate that stimulative training can improve the performance of a given residual network and all of its sub-networks, we collect various statistical characteristics of the performance of all residual sub-networks after the stimulative training and common training. As shown in Fig. [ MobileNetV3 with common training (i.e., sample sub-networks) has the lowest maximum value, mean value, and the highest standard deviation, which is mainly caused by the network loafing problem. As contrast, MobileNetV3 with stimulative training (i.e., sample 144 sub-networks) has the highest maximum value, mean value, and the lowest standard deviation. Moreover, stimulative training with more sub-networks can usually provide higher maximum value, mean value and lower standard deviation. All these results indicate that stimulative training improve the performance of all residual sub-networks effectively and efficiently.
see that, MobileNetV3 using
4.3 Experiment 3: Destructing residual networks at test time
In this experiment, we verify that stimulative training can provide stronger robustness in resisting various network destruction operations including deleting individual layers, deleting more layers and permuting layers than common training.
Deleting individual layers. We first show the performance drop when deleting individual layers from trained MobileNetV3 test time. As shown in Fig. [(a] when applying common training to MobileNetV3, the performance drop is non-neglig when some layers are deleted (especially for the last several layers of each stage), and the trained MobileNetV3 sensitive the index of deleted layer. As a comparison, when applying stimulative training to MobileNetV3, no matter which layer is deleted, the performance drop of stimulative trained model always lower than that of the common trained model, and the performance drop is always smaller than or equal to 1 1%. Such results verify that stimulative training can reduce the dependence between sub-networks significantly.
Deleting more layers. We then show the performance drop when deleting more layers from trained MobileNetV3 at test time. Note that, deleting more layers may come a variety of combinations, thus we show the statistical box plot of performance drops for all combinations. As shown in Fig. 6(b) when applying common training to MobileNetV3, the medium value, minimum value and maximum value of performance drop for all combinations keeps continuously rising as the number of deleted layers increases. And the difference in performance drop for different combinations when removing the same number of layers can be very large. For example, when deleting layers of common trained MobileNetV3, the Top1 accuracy drop for different combinations can vary from less than 5% to higher than 35%. Differently, when applying stimulative training, matter how many layers deleted, the values of performance drop for all combinations always keep very low (i.e., no more than 3%), indicating that stimulative training can bring strong robustness in resisting network destruction.
Permuting layers. We further show the performance drop when permuting layers in trained MobileNetV3 at test time. Note that, only the layers in the same stage can be permuted since the resolution and width of different stages are varying, and we show the detailed experimental settings in Appendix C.3. As shown in Fig. [(C) when applying common training to MobileNetV3, the medium value, minimum value and maximum value of performance drop for all combinations increases with the rising number of permuted layers, and the variance of performance drop for different combinations when permuting the same number of layers may be also very large. As comparison, the values of performance drop for all permuting combinations always keep very low (i.e., less than 1 1%), when stimulative training is applied. This further verifies the superiority of stimulative training.
7

---

Somulaive Tharing Acc $1.77%
Conmor Tronsng Acc
Top3 Accurcry Droptee
50
Cammoon Tromnng Toxt Acc 17395
Ton! Accuracy Dropp(()
40
15
Top1 Accursc
12
t 8 3
N to
Layer
Number LfLapers Deleted
Permutting Layers
(a) deleting one
(b) deleting more
(c) permuting
Figure 6: Topl accuracy drop when (a) deleting layer, (b) deleting more layers and (c) permuting
layers from MobileNetV3 with stimulative/common training on CIFAR 100 dataset. Stimulative training can provide stronger robustness in resisting various network destruction operations than common training.
4.4 Experiment 4: Recording the KL divergence between residual network and all of its sub-networks when training
To study the consistency between the output of given residual network and all of its sub-networks, record their KL distance in the whole training process. As shown in Fig- when applying com- training to MobileNet the variance, maxi- mum and medium value of KI. distance between a given residual network and all of its sub-networks continuously increases with rising training epoch. For comparison, when stimulative training ap- plied to MobileNetV3, all values of KL distance between MobileNetV3 and of its sub-networks keep very small (i.e., no more than 0.25) in the whole training process. Such results imply that the proposed stimulative training can maintain the consistency between the output of a given residual network and all of its sub-networks.
1.75
Stimulebove Training
KL Locs of All Subnetwork
1.50
Common Training
1.25
1.00
0.50
0.25
Epoch
Figure 7: The distribution of KL distance between MobileNetV3 and all of its sub- networks in the whole training process. We train MobileNetV3 on CIFAR100 with stimu- lative and common schemes.
5 Theoretical Analysis
Why the performance of sub-networks can improved. Based on Eq. our method can be treated as optimizing CE((((D,,, y) and KL(Z((D... 2 ((0... x)) at the same time. According to the convergence of stochastic gradient descent, after enough number of iterations, both losses can be bounded by constant, which is formulated
Analysis 1:
(E(Z((Dm..)),)))
Σ yi log <1
P T
KL((((D,,;
2(0,, 2(),,, = Eo
Σ
log
62
(3)
P
where is the number of categories. The 6 and 62 are the tiny constants for corresponding losses. θ is the set of all sub-networks. The pd and p² are the classification probability of main network and sub-network, respectively. Further, we can prove that the absolute difference of cross entropy of sub-networks and main network can be bounded by constant (detailed proof can be found in Appendix B), which can be formulated as
62 log N ICFE(2(0D... x). Ee [CE(Z(OD,,
+
where
N + is small constant after optimization. Eq. demonstrates that the performance e-11 gap between sub-networks and the main network is constrained. Thus, as the rising of the main network performance, sub-networks performance will be improved step by step.

---

Table 1: The main network and all sub-networks performance (%) of stimulative training (ST)
compared with common training (CT), on various residual networks and datasets.
method
MBV3_CIC
MBV3 _1000
Res50 C100
Res50 Img
Res101 Img
CT(main)
95.72
77.39
76.53
76.1
77.37
ST(miin)
96.88(+1.16)
81.07(+3.68)
81.06(+4.43)
77.22(+1.12)
78.58(+1.21)
(alll
78.13=14.29
55.26+13.37
34.03+18.92
39.46+16.82
35.78+18.84
ST(all)
96.21+0.43
80.01
79.66*1.77
73.3+2.89
75.52+3.6
Analysis 2: Why the performance of given residual networks can be improved. Neural network training is in fact form of polynomial regression, and any neural network with activation functions can roughly correspond to a fitted polynomial [33]. For simplifying notations, suppose that both input x and output y in neural network have only one element. Then, we can consider the training of a neural network as the fitting of a polynomial function.
y Fx) = + C₁ + c2ax +.... Cn +...
(5) represent the polynomial coefficients. Based on training samples, we can
where CI2 C22 ..... establish the polynomial (neural network), which denotes the mapping relationship between inputs and ouiputs. For a test sample there always exists a point Eo (e.g., a training sample) close enough to z. which can obtain an accurate output via the trained model. According to Taylor expansion, F F(z) at z0 can be computed as
('(xx) y = F(x) F(zo) F" [10o (x
U. +
11 (xt xo) Fin) (mo) (I xn)" " + Rx)
(6)
x0)²
2!
n!
where Rn (2)) denotes the higher degree infinitesimal of (æ zn)". According to Eq. 5 and Eq. 6 it is obvious that different terms of the fitted polynomial have different effects, and the low-degree terms tend to have greater effects on the predicted performance. Based on this point, (5] further figures out that shallow sub-networks residual networks roughly correspond to low-degree polynomials consisting of low-degree terms, while deep sub-networks are opposite. Following this theory, the proposed stimulative training can not only strengthen the learning of different items of the polynomial (neural network), but also pay more attention to the learning of low-degree items. As result, the final performance given residual networks can be improved by stimulative training.
Verification on Various Datasets and Networks
CIFAR implementation details. CIFAR 34 is a classical image classification dataset consisting of 50,000 training images and 10,000 testing images. includes CIFAR-100 in 100 categories and CIFAR-10 in 10 categories. For MobileNetV3 and ResNet50, the data augmentations follow [35], we use SGD optimizer and train the model for 500 epochs with a batch size of 64. The initial learning rate is 0.05 with cosine decay schedule. The weight decay is 3 x 10 and momentum is 0.9.
ImageNet implementation details. We implement our method on large-scale ImageNet 36 dataset containing 1.2 million training images and 50,000 validation images from 1,000 categories. For ResNet families, we use standard data augmentations for ImageNet, as done in [37 38 39]. We utilize SGD optimizer to train the model for 100 epochs with a batch size of 512, and the learning rate is 0.2 with cosine decay schedule. The weight decay x 10-4 and momentum 0.9.
Main network and sub-networks performance. We further verify the effectiveness of stimulative training (ST) various networks and datasets. [ablee[[shows its comparison with common training (CT). Besides the Top1 accuracy of main network, we also report the the mean value and standard deviation of Topl accuracy for all sub-networks. It obvious that stimulative training can noticeably improve the performance of various networks on various datasets compared to common training (e.g.. by 4.43% for ResNet50 on CIFAR- and 1.21% for ResNet101 on ImageNet). Moreover, the mean performance of all sub-networks is dramatically improved and standard deviation greatly drops. For example, stimulative training for MobileNetV3 on CIFAR has a mean performance of 80.01 % (much higher than common training of 55.26%) and a standard deviation of 0.59% (much lower than 13.37%). Such results validate that stimulative training can well alleviate the network loafing problem, and improve the performance of the given residual network and all of its sub-networks. Experimental results on various models and datasets verify the generalization of stimulative training, advocate utilizing general technology to train residual networks.

---

7 Conclusions
The paper understands and improves residual networks from a social psychology perspective of loafing. Based on the novel view that residual network behaves like an ensemble network, we find that various residual networks invariably exhibit loafing-like behaviors that are consistent with the social loafing problem of social psychology. We define this previously overlooked problem as network loafing. As the loafing problem hinders the productivity of each individual and the whole collective, we learn from social psychology and propose stimulative training strategy to solve the network loafing problem. Comprehensive empirical analyses show that stimulative training can improve the performance of given residual network and all of its sub-networks, and provide strong robustness in resisting various network destruction operations. Furthermore, we theoretically show why the performance of a given residual network and its sub-networks can be improved. Experiments on various datasets and residual networks demonstrate the effectiveness of the proposed method.
8 Limitations
The proposed method suffers from about 1.4 times of computation cost of the original model training to get better performance and robustness. As the first research of network loafing problem, the proposed method is a positive pioneer-like exploration. We believe designing a more efficient method to solve the network loafing problem worthy research direction in the future.
Residual structure is widely applied in numerous different types of models including DenseNet and transformer. It will be of vital value to study whether the loafing problem exists in these models and explore the proper method to solve this problem. Since most of existing works overlook the loafing problem, it may also be feasible way to apply the proposed method in pretrained models. What's more, the proposed method adopts main network logits as supervision to alleviate loafing and analogously, the proposed method may be applied in seff-supervised learning.
We believe that taking full advantage of splendid achievements interdisciplinary research can help promote the development of deep learning. We hope that this paper can provide new perspective to inspire more researchers to comprehend and improve deep neural networks from other fields such as social psychology.
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. U1909207 and 62071127), and Shanghai Municipal Science and Technology Major Project (No. NN.22021HH22200000) Wanli Ouyang is supported by the Australian Research Council Grant DP200103223, Australian Medical Research Future Fund MRFA100008S, CRC-P Smart Material Recovery Facility (SMRF) Curby Soft Plastics, and CRC-P ARIA bionic visual-spatial prosthesis for the blind.
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-645, Springer, 2016.
[3] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what is the question? In International Conference Machine Learning, pages 342-350. PMLR, 2017.
[4] Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
[5] Tongfeng Sun, Shifei Ding, and Lili Guo. Low-degree term first in resnet, variants and the whole neural network family. Neural Networks, 2022.
10

---

[6] Alan G Ingham, George Levinger, James Graves, and Vaughn Peckham. The ringelmann effect: Studies of group size and group performance. Journal of experimental social psychology. 10(4):371-384, 1974.
[7] Richard E Petry, Stephen G Harkins, Kipling Williams, and Bibb Latane. The effects of group size on cognitive effort and evaluation. Personality and Social Psychology Bulletin, 3(4):579-582, 1977.
[8] Kipling D Williams and Steven J Karau. Social loafing and social compensation: The effects of expectations of co-worker performance. Journal of personality and social psychology. 61(4):570, 1991.
[9] Jennifer M George. Asymmetrical effects of rewards and punishments: The case of social loafing. Journal of Occupational and Organizational Psychology, 68(4):327-338, 1995.
[10] Bibb Latané, Kipling Williams, and Stephen Harkins. Many hands make light the work: The causes and consequences of social loafing. Journal of personality and social psychology. 37(6):822, 1979.
[11] Ashley Simms and Tommy Nichols. Social loafing: review of the literature. Journal of Management Policy and Practice, 15(1):58, 2014.
[12] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint /503:025000023300 2(7), 2015.
[13] Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint (xXiv:2206.06067, 2022.
[14] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances neural information processing systems, 31, 2018.
[15] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198, 2020.
[16] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 4rXiv:1412.65500, 2014.
[17] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: (tfl::/777.12200 01219, 2017.
[18] Bycongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779-3787, 2019.
[19] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 675-684, 2018.
[20] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320 4328, 2018
[21] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[22] Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: Beta- decay regularization for differentiable architecture search. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10874-10883, 2022.
11

---

[23] Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun. Junjie Yan, and Wanli Ouyang. Bn-nas: Neural architecture search with batch normalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 307-316, 2021.
[24] Peng Ye, Baopu Li, Tao Chen, Jiayuan Fan. Zhen Mei, Chen Lin, Chongyan Zuo, Qinghua Chi, and Wanli Ouyang. Efficient voint-dimensional search with solution space regularization for real-time semantic segmentation. International Journal of Computer Vision, pages 1-21, 2022.
[25] Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 13836-13845, 2020.
[26] Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. 3-darts: Beta-decay regularization for differentiable architecture search. arXiv preprint arXiv:2203.0 016666,,,000,,,,,, 2022.
[27] Bicheng Guo, Shibo He, Tao Chen, Jiming Chen, and Peng Ye. Neural architecture ranker. arXiv preprint arXiv:2201.1272. 2022.
[28] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment. arXiv preprint arXiv: 0979,,
[29] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In European Conference on Computer Vision, pages 702-717. Springer, 2020.
[30] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European conference on computer vision, pages 646-661. Springer, 2016.
[31] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[32] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[33] Xi Cheng, Bohdan Khomtchouk, Norman Matloff, and Pete Mohanty. Polynomial regression as alternative to neural nets. arXiv preprint arXiv: (ale:8806.00000 806.06850, 2018
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Manas Sahni, Shreya Varshini, Alind Khare, and Alexey Tumanov. Compofa: Compound once-for-all networks for faster multi-platform deployment. arXiv preprint arXiv:2104. 12642, 2021.
[36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[37] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1-9, 2015.
[38] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[39] Golnaz Ghiasi, Tsung-Yi Lin, and Quoe V Le. Dropblock: A regularization method for convolutional networks. Advances in neural information processing systems, 31, 2018
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "cMq4ExYU8mH", "reviewer": "Reviewer_gf27", "summary": "The authors argue that subnetworks in ResNet tend to perform sub-optimally due to a \"social loafing effect\". That is, since no single subnetwork is accountable for the final performance, they all tend to rely on the aggregate group performance rather than achieving competitive individual performance. In order to alleviate this problem, the authors propose including an additional loss term that encourages subnetworks to match the predictions of the group.  In experiments the authors show that not only their loss solves the loafing problem but also improves the overall final performance. An additional theoretical analysis shows that improving subnetworks is tied to improving overall network performance. ", "questions": "* Does loafing happen in stochastic depth networks? In fact, it would be nice to stochastic depth networks as a baseline in all the figures.\n* Can your method be applied to pre-trained neural networks? \n* What if instead of the KL you used the same cross-entropy loss as in the output? That might reduce the computational cost since you would be able to compute the subnetwork loss without the logits of the main network.\n* It would be interesting if you checked whether DenseNet also suffers from loafing.\n* If the block dropping strategy is the same as in stochastic depth networks, you could just cite them and use the space to introduce changes suggested by the reviewers.\n* A main limitation is the additional computational cost, could you comment on it? Another option is to find a better strategy that does not increase it (e.g. keeping a buffer of previous logits so that you do not have to execute the main network every time).\n\n**Minor**\n* L48: The loafing problem.. inclines to be more severe -> is inclined to be\n* L142: sub-networks.. only has -> only have\n* L157: balanced coefficient -> balancing coefficient\n* L161: To open and close quotes (\") in latex use `` blabla '', this way they display properly in the text\n* L34: unraveled view -> the unraveled view", "limitations": "There is no limitations section, I encourage the authors to add one discussing issues such as the computational cost or the applicability of their method beyond resnets (such as transformers).", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall Review\n===========\nOverall, the authors present an interesting phenomenon in residual networks in an ingenious way (social loafing), while making use of that knowledge to improve such architecture. On the other hand, the experimental evaluation leaves many questions open (see below). In its current state, I think this work does not meet NeurIPS standards but I encourage the authors to continue working on it and to submit a rebuttal.\n\nStrengths\n=======\n* The authors succeed at showing that \"social loafing\" does happen in residual networks and that fixing it results in a performance improvement.\n* The experimental design succeeds at convincing the reader that there's loafing and that it can be fixed.\n* The authors include a theoretical analysis and reproducibility details in the Appendix.\n\nWeaknesses\n=========\n* Reading the submission generates a series of questions that are not answered by this work. For instance, does loafing happen in stochastic depth networks? In fact, it would be nice to stochastic depth networks as a baseline in all the figures. Can your method be applied to pre-trained neural networks? What if instead of the KL you used the same cross-entropy loss as in the output? \n* The authors of DenseNet [A] state that this kind of neural network is less redundant than resnet since each layer is aware of what the other layers are learning. It would be interesting if you checked whether DenseNet also suffers from loafing.\n* The block dropping strategy is the same as in stochastic depth networks, thus I would devote less space to it in the paper and state that you use the same strategy.\n* While stochastic depth networks reduce the training computation time, this work increases it. \n\n[A] Huang, Gao, et al. \"Densely connected convolutional networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. \n\nDetailed comments\n===============\nOriginality\n------------\nI found original the connection with social loafing. The method to solve it is a combination of multiple existing ideas but I think that it can be considered novel given the context in which it is introduced.\n\nQuality\n--------\nThe overall quality is good but the experimental quality could be improved. For example, you could explore if loafing happens or if it is solved with other methods such as stochastic depth or densenet.\n\nClarity\n-------\nThe text is clear and easy to read. There are some minor typos.\n\nSignificance\n--------------\nGiven the performance improvement introduced by the proposed method, it could be adopted by future work. A major drawback is the increase in computational cost due to having to compute two losses every time. \n\nReproducibility\n-----------------\nThe authors provide details to reproduce their work, however they do not provide the final code.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SDIpuKHalhD", "reviewer": "Reviewer_g3Zh", "summary": "This paper explores the use of a training method for residual networks called \"stimulative training\". This method is inspired by the psychological phenomenon of \"social loafing\" where members of a group give less effort when in a group compared to when not in a group. The paper hypothesizes a similar \"network loafing\" phenomenon for neural networks where subnetworks in a larger network would perform suboptimally compared to when a similar network is trained by itself. The paper devises a training strategy called stimulative training that samples random subnetworks and minimizes a kl divergence loss between the output of the sampled subnetwork and the full network. The paper performs a few experiments to show that \n- The loafing problem exists with traditional training strategies and is eliminated/reduced with their method\n- Their method improves the robustness of the network to deletion/permutation of layers\n- Improves the overall performance of the network for CIFAR 100 and Imagenet datasets.", "questions": "- When you mention sampling x subnetworks (eg sampling 144 subnetworks), is that on each step? If so, what is the difference in training time between the two methods?\n- In section 3.3, when discussing the ordered residual sampling, the justification given for why ordered residual sampling is better than raw sampling is that because of the size of the space, raw sampling isn't able to train all the subnetworks enough. Does that mean that if you trained the method with raw sampling for longer, performance would continue improving until it beat the ordered residual sampling? Is the learning curve for raw sampling converged, or is it still improving?\n- Is the sampling method uniform sampling?\n- Did you try experiments with MSE instead of KL divergence for the stimulative training? Was there a difference?", "limitations": "Other than showing reproducibility, I'd mostly want to see a comparison to/discussion of a method like self-distillation, which is not limited by the type of architecture, which this method is. Assuming the network loafing phenomenon exists with other architectures as well, why shouldn't we use something like self-distillation instead of this method, since it's more general? Is there a big improvement in the ease of use or performance with resnets when using this method? Also, I am curious as to how much longer this method takes compared to conventional training.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Connecting the training of neural networks to the social loafing phenomenon was an interesting point. Showing that simply getting each individual subnetwork more supervision to a common goal leads to an improvement in performance is also an interesting result, as I could imagine that it could lead to worse generalization on test set (something like the group think phenomenon).\n- I think the types of experiments selected was good. They covered a range of questions about this phenomenon.\n\nWeaknesses:\n- As far as I can tell, the results are based on only a single run of data. To draw the conclusions that the paper wants us to, I think they need to show that the results are replicable. For example, even if we say network loafing exists, it's unclear if we can conclude that it is a problem with a single run of data. Table 1 shows the results comparing the final accuracy of conventional training with stimulative training, and the improvement is small. To conclude that it is significant, the paper needs to show that it is replicable.\n- The method seems fairly similar to self distillation. In self distillation, they have some added machinery to be able to use the method on architectures where the output of each layer might not be the same size. In this paper, because the architectures are limited to residual networks, that machinery is not needed. Still, it seems to be a special case of self distillation for residual networks (although the paper does examine the specific phenomenon of network loafing). It would be interesting to see something like self-distillation as a baseline.\n- For Figure 4, please use more distinctive markers. It's difficult to tell the difference between markers of the same color.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LWZb2AqhXOf", "reviewer": "Reviewer_yqSA", "summary": "This paper finds an interesting problem with the residual networks, i.e., the network loafing problem. They find that the sub-networks will perform worse than those that are individually trained. Inspired by social psychology, they propose a stimulative training strategy, which randomly samples a residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the residual network, to provide additional supervision to the sub-network. The proposed strategy is validated by a comprehensive theoretical and empirical analysis.", "questions": "1. Can you explain why different sampling schemes make such a big difference (~5% in accuracy, figure 3)? It seems that in order for the main network to perform well, the subnetworks need to contain residual links inside. It will be great to see the explanation to this or see a comparison with normal training vs random sampling.\n2. How do you measure the performance of subnetworks? Will you retrain the last logits layer? If you do not retain the last logits layer, then the performance drop is understandable because the range/distribution of the input data to the last logits layer has changed. Even if the last logits layer is not retrained, I would expect some scaling/correction when the residual connection is removed (removing the residual connection will somewhat downscale the signal). I think the performance of a subnetwork should be evaluated in a fair way and we need to include the necessary corrections. Because what we would like to know is how well the subnetwork itself is to perform classification and we should minimize the impact from other factors (e.g., the scaling effects).\n3. I am curious about how does the proposed stimulative training strategy compared with providing supervision(the class label) directly to each layer? I am wondering if the authors directly compute CE loss between the labels and intermediate output instead of the KL loss, how the results would be like? I would like to know if the supervision to intermediate layers are the key to improve the performance and how effective the KL loss is.\n", "limitations": "See the questions above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: I like the idea to view the residual network from a social psychological perspective. This perspective is creative. Following the \"unraveled view\", they see the residual network as an ensemble of sub-networks and study the performance of each sub-network. They discover a novel problem called \"network loafing\", which has analogous to the problem in social psychology where an individual contributes less when in a group.\nQuality: \nIn general, the paper is technically sound. \nClarity: Overall, the paper is well written and easy to understand. The motivation and the contributions are clear, and the whole paper is organized well.\nSome minor issues with the writing: Some experimental details are missing, e.g., how do you measure the performance of subnetworks? Will you retrain the last logits layer? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6yf1Enz5fLS", "reviewer": "Reviewer_6eTg", "summary": "The paper proposes a view on residual network training from a social psychology perspective. An analogy is made where individual sub-networks of a ResNet are compared to people that are assigned a group project. Social loafing is the phenomenon that work is often not equally distributed in such groups. The paper argues that sub-networks do not perform equally on the classification task in a similar fashion and proposes two mechanisms (random sampling of blocks, and KL divergence between intermediate representations) as a remedy.", "questions": "The questions mainly follow the weaknesses above.\n1. Is the grounding of the work in social psychology fundamentally important to the presented approach? \n\n2. How can the objective of learning redundant residual blocks be combined with the understanding that CNNs extract feature hierarchies?\n\n3. How does the method differ in performance to previous methods with similar ideas?\n\n", "limitations": "Limitations and societal impact are not discussed in the paper. Especially because of the framing as inspired by social psychology, it would be very important to discuss both aspects in the paper. \n", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe results show that the proposed mechanisms do improve the performance of the overall network and also individual sub-networks.\n\nThe writing of the paper is clear and the goal, method and results are presented in an easily understandable format. \n\nThe paper contains many results of training networks with error bars which has high computational load.\n\nWeaknesses\n\nFraming: The grounding of the work in social psychology is extremely far fetched. ResBlocks are compared to people in a complex social group setting. The paper itself compares training a network to polynomial regression which has nothing to do with the complex social dynamics in a work group. There are no insights gained from this analogy and comparing a couple of ResBlocks to people contributes to false AI hype and other ethical issues. I strongly advocate for removing this framing from the paper.\n\nLearning redundant blocks: A large portion of the analysis is aimed at the performance of individual sub networks, permuting and deleting residual blocks and the KL divergence between intermediate representations. In all these experiments, the proposed training strategy shows better results than conventional training. This is not very surprising as the additional objective of adding a KL divergence term between representations and sampling sub-networks trains the model exactly for these tasks. Moreover, both objectives essentially train the individual block to be redundant. (This is another breakdown point for the loafing analogy where the optimal group work outcome is not achieved when every member does the same job) It is questionable if this is a desired behavior as CNNs are commonly understood to learn a hierarchical structure of representations. If blocks can be reshuffled or deleted without affecting the performance much, this structure is lost. It is possible that this effect is less important on small images such as CIFAR100. \n\nComparisons: The method is only compared to conventional training. Similar approaches have been explored that add auxiliary losses, distill representation (student teacher, KD, [12-19]) or drop residual blocks([22]) and it would be important to understand their effect on the network in comparison to the proposed approach. This would also make the comparison of dropping and reshuffling blocks more interesting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept