The Regularizing Effect of Different Output Layer Designs in Deep Neural Networks
Authors: Benjamin Bergner
Christoph Lippert
Venue: NeurIPS 2021 Submitted (2021)
Keywords: regularization
overfitting
output layer
architecture
deep learning
neural networks

Abstract: Deep neural networks are prone to overfitting, especially on small datasets. Common regularizers such as dropout or dropconnect reduce overfitting, but are complex and prone to hyperparameter choices, thus prolonging development cycles in practice. In this paper, we propose simple but effective design changes to the output layer - namely randomization, sparsity, activation scaling, and ensembling - that lead to improved regularization. These designs are motivated by experiments showing that standard fully-connected output layers tend to rely on individual input neurons, which in turn do not cover the variance of the data. We call these two related phenomena neuron dependency and expressivity, propose different ways to measure them, and optimize the presented output layers for them. In our experiments, we compare these layer types for image classification and semantic segmentation across architectures, datasets, and application settings. We report significantly and consistently improved performance of up to 10% points in accuracy over standard output layers while reducing the number of trainable parameters by up to 90%. It is demonstrated that neither training of output layers is required, nor are output layers themselves crucial components of deep networks.

The Regularizing Effect of Different Output Layer Designs in Deep Neural Networks
Anonymous Author(s)
Affiliation
Address email
Abstract
Deep neural networks are prone to overfitting, especially on small datasets. Com- regularizers such dropout dropconnect reduce overfitting, but complex and prone to hyperparameter choices, thus prolonging development cycles in prac- tice. In this paper, propose simple but effective design changes the output layer namely randomization, sparsity, activation scaling, and ensembling that lead to improved regularization. These designs motivated by experiments showing that standard fully-connected output layers tend to rely on individual input neurons, which in turn do not two related phenomena neuron dependency and expressivity, propose different ways measure them, and optimize the presented output layers for them. In experiments, we compare these layer types for image classification and semantic segmentation across architectures, datasets, and application settings. We report sig- nificantly and consistently improved performance of up to 10% points in accuracy over standard output layers while reducing the number of trainable parameters by up to 90%. It is demonstrated that neither training of output layers is required, nor are output layers themselves crucial components of deep networks.
3
5
the variance of the data. We call these
11
12
13
14
15
16
Introduction
17
18 Neural networks are powerful feature extractors that have become the standard approach for a myriad
19 of tasks. New architectures are continuously introduced and set records on benchmark datasets
20 (e.g. [54 15 44 ). These networks differ in layer composition, depth/width or use specific concepts [15] or self-attention 34.. With growing capacity, their performance on However, model complexity is also associated with overfitting, especially for small datasets where fine details of the training data are easily memorized [52] 2 53
21 such as residual connections
large datasets tends to increase
23
24
Rather than defining another, possibly more complex architecture, we analyze what often remains unconsidered: the output layer. image classification, networks usually end with fully-connected (fc) layer that combines extracted features for the final output [3 15
25
26
As we will show, this layer is prone to overfitting since high dependencies on individual, possibly memorized features can arise. The same neurons are subsequently not able to generalize across examples. We call these two related phenomena neuron dependency and expressivity and illustrate simplified example Fig. 2
27
28
29
30 Both problems can be improved by simple but effective changes to the output layer that require only few lines of code and achieve better generalization (i.c., better results on the test [25] see e.g.
31
32 Fig. D. Those changes rely on four principles: activation scaling, fixed randomization, sparsity and of their capability reduce 34 dependencies and/or increase expressivity. Then, the connection to network performance is shown
33
in-layer ensembling (see Fig. [.. This work analyzes all layers
35 through a comprehensive empirical study across datasets, architectures and application settings in image classification and segmentation. Furthermore, we investigate how stronger regularization
36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

90
Witeined
3.5
80
Wrandom
3.0
70
Wascole
2.5
test accuracy
60
Waparss
Wilo1
2.0
Wensemble
1.5
40
1.0
30
20
0.5
10
0
50
100
150
200
0
50
100
150
200
epoch
epoch
Figure 1: Effect of different output layer designs on cross-entropy loss (left) and accuracy (right) in a ResNet-50 for the STL-10 dataset. Best viewed in color.
Model 1 Training input
Model
Model 2 Unseen input
Unseen input
A1
O1
y
01
y
A1
01
y
A2
02
A2
02
A2
02
a)
b)
C)
Figure 2: Schematic of neuron dependencyleeppressiityy in fe output layers. The left side of each subfigure represents penultimate layer activations (A1-2), the right shows output neurons for each class (01-2). Filled/blank circles indicate high/low activation, up-/downward facing arrows signal positive/negative weights. Higher activations of 01 lead to correct predictions this example. Model depends neuron A1 be activated to give high prediction scores 01. This is the training instance in a). If Model 1 is applied to an unseen input pattern of same class in b), higher scores erroneously given to 02 since A1 remains inactive and A2 slightly favors 02. Model fails to generalize as it depends on AI, which is not expressive enough to cover the variance of the target class. Instead, Model shown in e) exhibits neurons with low dependency and high expressivity, where Al generalizes to unseen patterns, while the activation of A2 can be regarded as backup. Note that this example is simplified and educational. See Sect. 3.3 for measurements.
for
37 can be induced by applying the identified principles to other parts of a network while reducing the 38 computational footprint. In contrast common practice, we find neither training of output layers to be necessary, nor that output layers are crucial components of deep networks. In summary, our contributions are:
40
41
Introducing neuron dependency and expressivity two factors contributing to overfitting and proposing ways to measure these factors
42
43
Showing improved regularization of 5 different output layer designs up to 10% in absolute accuracy compared to standard fe layers and other common regularizers
44
45
Empirical results showing that the proposed layers have improved dependency and expressiv- ity, computational efficiency, wide applicability to both small and large datasets, extensibility to other parts of the network, and robustness in the choice hyperparameters
46
47
2
Related Work
49 Regularization in deep learning is approached various ways. Widely used methods are, e.g..
normalization 3), weight decay [32,, data and adversarial augmentation 40 1. early stopping [
51 boosting (38), multitask learning (6), dropout 42), dropconnect (50), and Gaussian noise layers [10]
To the best of our knowledge, this is the first work that evaluates regularization with respect to

---

78 We consider neural networks consisting of an encoder fene X + a followed by an output layer 79 fout : a - 9. In this paper, the encoder is a CNN, taking as input an image X E RCCHHW with 80 C. W and H being input channels, width and height, respectively; and transforming it to feature 53 different output layer designs. Similar to dropoot/droopooneet,, output layers can be categorized as
54
affecting the architecture according to the regularization taxonomy described in (25] Unlike other regularizers, our methods are either hyperparameter-frie free or robust to their choice and can be applied
55
56
to any deep net, including pre-trained ones that are less affected by overfitting (see Sect. [.4 and 5.7).
57
Related to fixed randomization are the output layers used in [66 39 which show comparable performance to trained layers. One can also preallocate oulput layer weights with defined struc-
58
ture 31 16]. Besides output layers, weight fixing is for example applied to the first layer in the 59
Extreme Learning Machine [8]. or to different weight dimensions in [36.. In contrast, we omit haad-crafiee weights, show improved regularization and relate to neuron dependencies. Further, we show that fixing or scaling the last conv block next to the output layer has a strong regularizing effect.
60
61
62
63
Sparsity is common in deep learning, e.g. the ReL.U activation [3] or a L1 penalty term in the loss Sparsity has also been applied to the channels of Convolutional Neural Networks
function 46
65
(CNNs) g 29 Others induce sparsity by pruning connections before training under the lottery
86
ticket hypothesis [ 30), with the goal of reducing the number of parameters while not sacrificing performance [27
87
45 51]. Different to them, we show that (extreme) sparsity is not merely useful to
68
improve computational efficiency, but to improve performance when applied to the output layer.
69
The Network in Network (NIN) (8 and All-CNN [41] both use global average pooling (GAP) followed by softmax, which replaces the fc output layer with identify transform simplify the network. This is further analyzed in B3] We show its connection to neuron dependeenc/exxpresvvvyy
71
72 and achieve comparable or better performance on various datasets. Further, we observe that previous
73 works do not leverage the full capacity of the last layer in modern networks, which enables the construction of computationally efficient in-layer ensembles that further boost performance in small
75 and large datasets.
3 Neuron dependency and expressivity
76
77
3.1 Setting and notation
81 vector a E R R Commonly in CNNs, 2D representations resulting from the final conv layer aggregated by GAP (28) where N corresponds to the number of pooled conv channels. The 83 output layer transforms the embedding to output & E IR holding the probabilities of K classes.
84 The output layer is parameterized by weight matrix W E N and is commonly initialized as Wrandom ~ 20-22(1 (o((-vii ((VVVVVI/, / I//N) (26). Both ğ and target y are used to compute the cross-entropy
85
loss E = - Σ y: log(ūt). We use the terms features/channels/nodes or neurons interchangeably meaning activations
86
87
When required, we refer individual instances with a superscript, e.g. y ()) E D, with D being a dataset. Corresponding subsets are denoted as D Dtrain and Drest-
86
89
3.2
Concepts
90
During training, CNNs learn set of visual patterns that are combined for a classification decision. However, if patterns remain undetected, e.g. due to noise in the image or inherent but unseen variance in the data, their activation values can become small and thus reduce the output values for the target
91
93 class. When a network is overfitting, learns malignant image-specific patterns by heart (52). Such a network may depend on the activation of individual nodes, which in turn fail to generalize to patterns 95 that are salient to a class. We call these two related phenomena neuron dependency and expressivity.
94
97
Neuron dependency: How much does a model depend on a single neuron? In a network with high neuron dependencies, output scores and thus performance drop significantly when certain neurons remain inactive. In contrast, a network with low neuron dependencies distributes activations acToss many neurons, so that single inactive node does not have much influence the classification. Neuron expressivity: How much class-specific variance does a neuron cover? Neurons with low expressivity focus on unimportant details that do not characterize the properties of class. In contrast,
99
100
101
102
a neuron with high expressivity generalizes by activating to various patterns pertinent to a given class.
103
An example of neuron deperndencylexpressvvvty for simplified fc output layer is illustrated in Fig. 2
3

---

0.4
0.30
0.14
reduction
output probability
Wirahned
Wrandom
probabliity density
0.25
a=
1.0
reduction output probability
0.12
0.10
0.20
a 3.0
0.08
0.2
0.15
0.06
0.1
0.10
0.04
0.05
0.02
Wacaledd
0.0
0.00
0.00
20
40
60
80
100
-10
0
10
20
% training set
logits
Figure 3: The effect of dataset size and activation scale on neuron dependency in a ResNet-50 trained on subsets of CIFAR-100, evaluated on the test set. Left: Small training lead high neuron dependencies. Center: Scaling activations results in larger absolute logits. Right: Larger scales lead higher dependencies. Best viewed in color.
80
80
Wtrained
70-
75
Wrendom
accuracy (%)
60
70
Wisalldd
50-
Waperse
eccuracy
65
W
40
60
Wansemale
30
55
20
10
50
45
50
100
150
200
2
4
10 12 14 16 18 20 used features per class
8
cut features per class
Figure 4: Neuron dependency (left) and expressivity (right) in a ResNet-50 with 2048 penultimate layer channels trained on CIFAR-100 for different output layer designs, showing the change in accuracy on the test set. Best viewed in color.
105
3.3
Measuring dependency and expressivity
106
We introduce two ways measuring dependencylexpprssiiit:: instance- instance-based and class-based. The former is used to determine the dependency the most important node for the predicted class given instance. Importance scores for node n and output class & computed with AaadiietOActivatioo [4]
108
109
a global attribution method where we leverage the partial derivative of the softmax values:
On Dap
(1)
110
Instance-based dependency is then measured as avg. reduction in output probabilities when ablating
111
the most important feature w.r.t. the output class of any instance. This illustrated for various 112 training set sizes of CIFAR- 100 (23 in Fig. B (left). With less data, fc output layers tend to depend
113 more on single nodes. This is in contrast class- based measures, which enable quantifying both dependennyllexpressvity and use various features jointly. Importances are determined for each class and over test instances:
115
[Danel
OUk
dal (4)
116 Class-based dependency then measured as drop accuracy when ablating a given number of most
117 important neurons per class. Measuring expressivity reverses this the most important neurons per
118 class are retained, all others are ablated. This illustrated for both dependenncylexpressvityy
Fig. 4
119 We see that standard (i.e. trained) fc output layers tend to depend on single channels to achieve high performance, but these very channels hold only limited class information.
120
4
Output Layer Types
We describe several simple output layer variants that require minimal changes standard networks, decrease neuron dependency and/or increase neuron expressivity. All types are illustrated in Fig.
123

---

Witriined
Wrasiled
Wreandew
Willld
Figure 5: A visual comparison of various output layer types. Red/blue represent variable/fixed.
124
4.1 Standard output layers
The ubiquitous approach to compute class scores is to learn the parameters of a weight matrix W Wtrained, s.t. ý = g (a (attrained) with (s (-) being softmax. Each feature is considered the computation of each class score. As shown in Sect. trained fe output layers can lead to high neuron dependencies, where the deletion of a single neuron might cause significant loss in performance, and low neuron expressivity, where multiple features are required for adequate predictions.
126
128
4.2
Scaled output layers
The reduction of an activation, e.g. due to changing light conditions, has large influence the output scores. This is simulated in Fig. 3 (center) by multiplying features during training with a scalar 0, that y asas (cra W W Note that the variances of the output logit distributions increase with a, resulting in larger differences (or smaller entropies) after softmax normalization.
132
133
134
This results in greater dependencies of the model individual neurons, shown in Fig. 3 (right). However, if a is chosen small, the activations of individual neurons become insufficient for class discrimination with high confidence. The model is therefore forced to learn multiple class-specific features for each instance, which increases the expressivity of the neurons and also reduces their dependencies to some extent, as shown in Fig. If not specified otherwise, we use a 0.1.
136
137
138
139
4.3
Random fixed layers
This setting uses Wrondom during training/inference, and its classification performance was first analyzed in [16]. The encoder learns to extract patterns that adjust to predetermined weights. Unlike activation scaling, the parameters are bounded and fixed to a small value range. For any class, the chosen uniform initialization is expected to assign similar weight values to multiple neurons, making them learn similar features. We suppose that the enforced similarity reduces dependeney shown in Fig. 3 and 4 (both left), while small initialization values increase expressivity as in Sect. 4.2 shown (right).
141
143
145
147
148
4.4
Sparse fixed layers
149
In sparse output layers, class nodes use predetermined sets of channels, some of which might of cutting indices I, is randomly sampled for each class k. where sparsity is determined by the proportion of class-specific connections to cut, so that I [gN] with 0 < q < 1. Then, starting from fixed random initialization in Sect, 4.3 weights connecting to a given class are ablated so that: W
be shared across classes. First,
151
152
0 V E Zz. Hyperparameter g trades off dependdencylexpressivity Larger values induce more sparsity, leading to greater dependencies to the remaining nodes, but forcing them activate across instances, making them expressive. We set 0.9 the experiments to show that high sparsity benefits generalization.
153
154
155
156
157
4.5
1-to-1 correspondence layers
158
The most extreme type of sparsity in an output layer is one with a single connection between feature and class. these connections correspond to an identity transform, the activations of the penultimate layer are equivalent to the class logits in practice, the output layer can hence be omitted.
159
160
161
This was analyzed [33 28 and showed comparable results to standard output layer. Formally, we have y asM (a.Wlto) with RRIX and Wilol E RK*K where Wioll = diag(1, .....:1) 1). In this layer, both the model's 's dependency on individual neurons as well as each neuron's 's expressivity maximal. If single neuron is ablated, the output logits for the class this neuron is connected to is
163
5

---

185
reduced to zero. However, individual neurons learn to cover the whole variance of a given class in the training set, which is one conjecture for their performance. Note that as mentioned in [33. we have the constraint N- = K, which might be restrictive for small networks and large numbers of classes.
186
167
188
4.6
Ensemble layers
169
Is there way to optimize for both low neuron dependency and high expressivity? Of the approaches discussed, 1-to-1 I correspondence layers have the highest expressivity. Starting from this layer, a simple approach reduce neuron dependency is to use the capacity of the penultimate layer and create multiple heads h = 1... H with N = K H, each head being a 1-to-1 correspondence layer. Each head's 's output effectively computed as g T (aaa)) with a² being the activation part of head h. As in Sect 4.2) we introduce a scalar a, which controls the magnitude of feature activations. For consistency, we denote this approach as Whkaads The loss is computed for each head and averaged: " Σm= ((gh, y). Similarly, logits are averaged over heads for inference. Due to the induced redundancy, the performance only drops considerably after removing class-related neurons from all heads. In our experiments, we set H to its maximum given any setting (architecturcc/dttast))) Note that hyperparameter a in ensemble layers is the only one which is tuned to individual settings.
170
171
172
173
SSM
174
175
176
178
Experiments
180
We aim to show that the presented output layers from Sect. outperform standard output layers and common regularization methods in various settings. Details about training, compute resources, code, datasets as well as additional experiments on depeendencylexpressvviy are included in the appendix.
182
184
5.1
Small-scale and fine-grained classification
185
All layer types are first applied to small-scale and/or fine-grained classification, both of which are challenging and require regularization. Datasets include STL-10 (500 img/class) (9] CUB-200 (~~30 img/class) 499. Cars- 196 (~40 img/class) Pp and 101 (750 img/class) [5]. Table I shows
186
187
188
results for the two popular backbones ResNet-50 15 and DenseNet- 169 [77.. exchanging the output layer accordingly. In 53/56 settings, we see improved results over standard layers. Of these, 48 and 36 are significant with < 0.1 and p < 0.001, respectively. Although there is no clear best method, it is worth noting that sparse and ensemble layers as enhancements of both random and 1-to-1 layers are significantly better (p < 0.001) in 7/8 settings, respectively. As expected, smaller performance differences are exhibited in Food-101, which requiring less regularization. Among the worse settings, only I is significant (p < 0.1) for Food-101 since it involves strong regularization to multiple layers. These regularizers are discussed seperately in Sect. 5.5
189
190
191
192
193
considerably larger dataset, thus
195
197
5.2
Large-scale classification and transfer learning
198
Machine learning models are subject to the bias-variance tradeoff [12. in which induced biases of the presented output layers might be too strong to fit the training data. We therefore want to shed light on how these layers behave in large-scale and transfer learning settings, where overfitting is less problematic. Datasets include CIFAR- 100 (C100, 5000 img/class) (23), ImageNet (IN, 1200 img/class) from ILSVRC2012 B7 reported on the validation set, as well as CUB/Cars/Food with models being pre-trained on IN. Table 2 shows the results for the ResNer-50 backbone. In C100, we see consistent improvements with at least p < 0.1. On the other datasets, results are mostly comparable corroborating widespread applicbiiityyy is worth mentioning that Witol and Wensemble perform consistently better, and Wensemble significantly (p- < 0.1) in multiple cases. With growing dataset sizes, both layers expose strong constraint on the class neurons to fit an increasing number of examples. We believe this to be responsible for progressively separating the signal from the noise, leading to better generalization. On the other hand, neuron dependency is reduced in larger datasets (see Fig. leftt) diminishing the effect of Waeole and Wrandom Moreover, W Wrandom and sparase can be affected by predetermined feature-clasy weights that do not have match features learned during pre-trainn vre-training, which might require larger adjustments to the weights of the last cony layer.
199
200
201
202
203
204
205
206
208
210
212

---

Table 1: Classification accuracy for different output layer designs in small-scale and fine-grained classification without pre-training- Exponent repeats describe probability values (*: p < 0.1, **: < 0.01, ***; : p 0.001) indicating statistical significance based on a one-tailed normal approximation interval test comparing accuracy of the proposed layer designs to baseline fe layer (Wtrained). Symbols and denote better/worse performance than baseline, respectively. Bold denotes best performance.
STL-10
CUB-200
Cars-196
Food- mood-101
Wirrined (baseline)
81.36
57.18
81.20
83.70
Wseated
83.33*
63.46
87.07**
85.46***
ResNet- 50
Wseated block
86.42**
66. "6.74***
87.53
85.03**
Wrandom
86.08**
60.91 "*
83.02*
84.20
Wrandom block
86.59**
67.21 ******
84.07
84.04
Waparse
87.23* **x
66.27***
85.47**
85.45
Wiltol
84.78*
58.56
80.51
$4.41*
Wensemble
87.94***
62.98
85.76***
85.36
Wttaiine (baseline)
81.88
55.33
80.82
84.31
DenseNet- 169
Wscaled
86.53**
63.31**
85.85
85.05*
Wseated block
85.89***
65.57
85.35
85.44**
Wrandom
86.11 ***
61.24***
83.52
84.90*
Wrandom block
86. 86.64* *
65.99 ***
82.93*
83.251
W Wsparse
86.58**
85.79**
84.63
Wittl
86.06* **
55.37
83.75
84.11
Wensemble
87.00***
64.15
85.09
84.91#
Table 2: Classification results for different output layer designs in large-scale image recognition and transfer learning. denotes fine-tuning from ImageNet. See Table for other symbols.
C100
IN-topl
IN-top5
CUB-200 +00+
Cars-196+
Food-1011
W Wttained
77.75
76.36
93.12
80.91
91.73
87.32
W sscaled
79.65*
76.08
92.84
78.681
90.91
87.21
W Wrandom
78.91*
76.08
93.15
80.89
91.72
87.29
W sparse
79.46*
75.32ft
92.36ftt
80.38
92.07
87.31
W lfal
79.07*
76.53
93.32
81.79
91.87
87.31
Wensemble 80.38
76.62
93.46*
82.22*
92.77*
87.76
5.3
Use Case: Medical imaging
214
Output layer design is critical in fields such medical imaging, which presents special challenges to regularization: Datasets tend to be small, imbalanced, abnormalities might fill only a few pixels of the image, and appearances between classes are often similar. In addition, transfer learning with IN weights is either inaccessible due to architectural differences (e.g. image segmentation, 3D Magnetic Resonance Imaging) or less effective due large domain differences. This is first illustrated on the
215
216
217
218
(20). with the goal of detecting diabetic retinopathy severities in retinal fundus images. We use the public training dataset to train multi-class classifier and perform 5-fold cross- cross-validation. Table shows the results. We consistently get better performance with regularization and reduce the gap to pre-trained network. Furthermore, additional experiment in the appendix indicates that the standard output layer is biased towards the prevalent class, which is inherently remedied through randomization.
APTOS Kaggle challenge dataset (3662 images, 193-1805 img/class)
220
222
224
225
We provide further evidence that the proposed layer designs positively affect tasks other than classification. We learn U-Net (35) for binary semantic slice-based segmentation of Computed
226
227
Tomography scans of livers comparing a standard Ixl conv output layer with 64 parameters to both
fixed randomized and an ensemble layer. Due to the limited number of parameters, we omit Wesale and Waparse here. Different to classification, the output of U-Net itself can be interpreted as 1-to-1 l layer. One can still build an ensemble by treating each output channel as a head. Both Wrandom and W Wensemble (H 10) are then applied to the CHAOS 21 and SLIVER [47 datasets. For CHAOS.
228
230
7

---

Table 3: Quadratic weighted kappa and accuracy (with significance) for differ- ent output layers in ResNet-50 for the APTOS dataset. + denotes fine-tur tuning from IN. See Table for other symbols.
Table 4: Jaccard coefficients in segmentation
CHAOS
SLIVER
Wfrained 0.77
0.83
Wrandom 0.80
0.85
Wensemble
0.78
0.86
Kappa
Acc.
W Wtrained
0.816
77.44
Table 5: Regularization comparison
W Wscaled
0.818
78.38
W Wrandom
0.848
79.32*
STL
CUB
CUB+
Cars
W Wsparse
0.840
79.60*
Dropout 422
82.73
63.20
80.26
83.98
W W
0.856
80.08*
Dropconn. 500
86.15
61.48
80.41
85.06
W Wensemble
0.866
80.78**
Add. Noise
10
82.51
52.74
80.91
76.77
W Vtrained+
0.909
85.02
Wivained
81.36
57.18
80.91
81.20
W Weparse+
0.910
85.17
W Wsparse
87.23
66.27
80.38
85.47
W Wensemble+
0.912
85.56
Wensemble 87.94
62.98
82.22
85.76
232
we train on 15 randomly selected patients (2155 slices) and evaluate on the remaining 5 (719 slices). We then test for generalization by training on all 20 patients from CHAOS and evaluating on the external SLIVER dataset consisting of 20 patients (4159 slices). Table shows improved results in
233
235
both settings.
5.4
Other regularization techniques
237
Table compares our most competitive methods to other popular regularizers when applied to standard output layer. Nodes/connections in dropoout/ropcooneec are both removed with 0.7, and the noise layer adds a Gaussian with μ = and = 0.1 before applying the layer. In all cases, our variants perform better. Whereas noise does not benefit training here, dropout/-connect is supporting regularization. However, both of the latter methods come with two main disadvantages. First, they add complexity by changing states in each iteration and having different behavior during training and inference. Second, hyperparameter tuning is necessary, while our layers are either hyperparameter-free stable them. See the ablation study in Sect. 5.7 for evidence.
239
241
243
5.5 Beyond output layers block scaling and randomization
Activation scaling and randomization are techniques applicable to any layer and increase regulariza- tion further. This is demonstrated for ResNet-50 and DenseNet-166 in Table Both architectures consist of multiple blocks, each holding groups of conv layer, batch normalization (BN) and activa- tion function. For W7rondom block, all layers of the last block and the output layer are kept in their initialized state during training. Similarly, in Wasaled block, activations of all layer groups in the last block are scaled during training. In ResNet-50, block scaling outperforms output layer scaling in 3/4 datasets by up 3% points, and block randomization increases performance in 3/4 datasets by up to 6% points compared to output layer randomization. In DenseNet-169, block scaling outperforms out- put layer scaling in 2/4 datasets by up 2% points, and block randomization increases performance in 2/4 datasets by up to 4% points compared to output layer randomization. Only in Food- 101 and DenseNet, block randomization performs significantly worse than baseline because regularization too strong leading to underfitting (train loss = 0.47 compared to 0.02 in Wrandom).
247
249
251
252
253
255
256
257
258
5.6
Computational efficiency
259
The design of the head of deep CNNs has a great impact computational efficiency. Standard output layers alone can contain a large amount of parameters, as CNNs typically hold more channels as they get deeper and the number of classes can become large. In ImageNet and a ResNet-50, for example, the output layer alone generates over 2 million parameters, which are saved Witel and Wensemble This problem compounds when using multiple fc layers. In a VGG-16, for instance, 3 fc layers are employed after the last conv layer. Table shoows, omitting all fe layers saves up to 90% in parameters, a considerable amount of memory, and time for a forward/backwar pass while
260
262
264

---

Table 6: Computational efficiency comparison in CUB-200 highlighting that the number of trainable parameters can often be reduced while accuracy is improved. + denotes fine-tuning from ImageNet.
Architecture
#Params in M.
Mem.[GB]
GFLOPS
Time [ms/it.]
Accuracy
VGG16 Witained+
135.1
7.5
31.1
114
78.68
VGG16 Willo++
13.5
5.9
30.4
106
79.27
VGG16 Weneemble-
14.7
5.9
30.8
106
81.15
Res50 W Wtrriied
23.9
5.1
8.2
71
57.18
Res50 Wrandom block
8.5
5.0
8.2
67
67.21 RXx
STL-10
CUB-200
90
85-
85
880-
80-
60
Wylel
default (K=10)
Σ 70
60-
275 775
WWitaaned
accuracy
75
default
accurae aay
75
2 20-
70-
default-
40-
655
65-
30
W W/haned
60-
60-
default (K=200)
55
1.0
0.5
0.1
60
0.75
0.90
1.00
55-
25
so
100
150
200
0.0 0.2 0.4
0.6
0.8
scale (c)
sparsity (e)
heads (H)
dropout rate
Figure 6: Ablation study showing stability and consistency of our output layer designs
increasing accuracy. If a ResNet-50 is used, randomization of the last conv block next the output 267
layer yields savings of about 65% in trainable parameters while increasing accuracy by 10% points.
268
5.7
Ablation study
Note that Wrandom and Witol are hyperparameter--ree compared to other regularizers such as dropout/-connec, thus saving the cost of tuning them. Although other layer variants possess hyperpa- rameters, we show in F oo the ResNet backbone that they are stable (no large jumps in vicinity) and consistent (tend to monotonicity w.r.t. performance). In Wesparse, the maximum accuracy for both datasets is at 0.99 (20 nodes per class) and drops only slightly for 0.995. Similarly, downsealing in Wscale improves performance
269
271
273
274
small cost if the optimum is not hit. Also, more heads in W Wensemble tend to increase performance. What is the result of adding more heads than given by the constraint N K H? If N KH, which is the case for CUB-200 and H 10, we add an additional 1xl conv layer, BN and ReLU with K H nodes to adjust for the missing channels. Although this leads to a considerable increase in parameters (N K H for the conv layer), helps with generalization, contradicting the common belief that overparameterization
275
276
277
278
279
leads to overfitting [48] In contrast, dropout is not stable or consistent. With a dropout rate of 0.9, the network fails train in both datasets. Furthermore, the optimum for CUB lies at 0.8, the same hyperparameter choice in
280
281
282
STL would result in worse performance than baseline.
6
Conclusion
283
284
In this work, we introduced neuron dependency and expressivity as factors contributing to overfitting. Then, different output layers were defined to optimize both and showed improved regularization in various settings while being efficient and robust to hyperparameters. Although these layers are simple, they have high practical relevance due to the importance of regularization and the ubiquity of output layers in deep nets. In addition to their application, they may also be useful as primitives in future (automatically created) architectures. Although improving regularization, we note that optimizing for neuron depeeddeniessexppessiiii does not solve overfitting. For example, an unknown or noisy instance may result in reduced activations in the majority of nodes in the penultimate layer. Finally, we speculate that overfitting may not just be a function of the number of parameters in the encoder. Instead, it might be more important how the extracted features are combined in the output layer.
285
286
287
288
290
252

---

References
294
295
[1] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04344 2017.
297
[2]
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pages 233-242. PMLR, 2017.
296
299
300
301
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 1rXiv:1607.0644 2016.
302
303
[4] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
305
306
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 - mining discriminative components with random forests. In European Conference Computer Vision, 2014.
308
[6]
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
309
[7] Rich Caruana. Steve Lawrence, and Lee Giles.
Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, pages 402-408, 2001.
310
311
312
[8] Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov. The power of sparsity in convolu- tional neural networks. arXiv preprint arXiv: 2017.
313
99] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223 JMLR Workshop and Conference Proceedings, 2011.
315
317
[10]
Terrance Vries and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint 1rXi::1702.05538, 2017.
318
319
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 03635, 2018.
320
321
[12 Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
322
[13] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings, 2011.
324
325
[14] Moritz Hardt and Tengyu Ma. 161iv:1611.00 1.0231,, 2016.
Identity matters in deep learning. arXiv preprint
327
328
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
329
330
331
[16] Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. arXiv preprint 1rXivv1801.000 0404000 2018.
332
333
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
334
335
336
[18] Guang-Bin Bin Huang, Qin- Yu Zhu, and Chee-K Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1-3):489-51 501, 2006.
337
[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International conference on machine learning, pages 448-456. PMLR, 2015.
339
10

---

341
[20]
Kaggle. Aptos 2019 blindness detection, 2019. URL hatpp:://www.kaee www www.wkgggl..co/..ooo c/ pptos201s-biindness-eeettte
342
343
[21]
A Emre Kavur, N Sinem Gezer, Mustafa Banı, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Due Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, et al. Chaos challenge- combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis, 69:101950, 2021.
344
345
348
347
[22]
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
348
349
350
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
351
2009.
[24]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:7
353
354
1 1105, 2012.
[25] Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprint 1rXiv:1710.10658 2017.
356
357
[26]
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.
[27] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.023440, 2018.
360
361
[28]
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint 1312 2013.
362
383
[29] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 806-814, 2015.
384
365
388
[30] Eran Malach, Gilad Ychudai, Shai Shalev-Sehwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning. pages 1.82-6691. PMLR, 2020.
387
388
389
[3]] Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo. Fix your features: Stationary and maximally discriminative embeddings using regular polytope (fixed classifier) networks. arXiv preprint arXiv: 10441, 2019.
370
371
372
[32] David C Plaut, Steven J Nowlan, and Geoffrey E Hinton. Experiments on learning by back propagation. 1986.
373
[33] Zhongchao Qian. Deep Convolutional Networks without Learning the Classifier Layer. PhD thesis, Rochester Institute of Technology, 2020.
375
376
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint ar wwXiv:1906.099000
378
2019.
379
[35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
381
382
[36] Amir Rosenfeld and John Tsotsos. Intriguing properties of randomly weighted networks: Generalizing while learning next nothing. In 2019 16th Conference on Computer and Robot Vision (CRV), pages 9-16. IEEE, 2019.
384
385
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-555 2015. doi: 10.10077/11123-015-000
387

---

[38] Holger Schwenk and Yoshua Bengio. Boosting neural networks. Neural computation. 12(8):
390
1869-1887, 2000.
391
[39] Gabi Shalev, Gal-Lev Shalev, and Joseph Keshet. Redesigning the classification layer by randomizing the class representation vectors. arXiv preprini vrXiv:2011.08704 2020.
[40] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
394
385
[1]] Jost Tobias Springenherg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
398
397
[42] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1 ): ) 1999-1999,, 2014.
398
389
400
[43]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9,
402
2015.
[44] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-61 14. PMLR, 2019.
405
408
[45] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.00 05467, 2020.
407
408
[46] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
410
411
[47] Bram Van Ginneken, Tobias Heimann, and Martin Styner. 3d segmentation in the clinic: A grand challenge. In MICCAI Workshop on 3D Segmentation in the Clinic: A Grand Challenge. volume 1, pages 7-15, 2007
413
414
[48] Vapnik Vladimir and V Vapnik. Statistical learning theory. Xu JH and Zhang XG. translation. Beijing: Publishing House of Electronics Industry, 2004, 1998.
416
[49] C. Wah, S. Branson, Welinder, P. Perona, and S. Belongie. The Caltech-UCSH Birds-200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology. 2011.
417
[50] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066. PMLR, 2013.
419
[51] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020.
422
429
[52] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. 2016.
424
425
[53] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterrizatoo.. arXiv preprint arXiv: 04698, 2019.
428
427
12

---

Checklist
428
429
1. For all authors...
430
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Sect. and the appendix.
431
432
(b) Did you describe the limitations of your work? [Yes] See Sect. 4.5
(N K con- straint). Sect. 5.2 (some layers are not optimal for large-scale datasets and fine-tuning), Sect. [.3 (strong regularization may lead to underfitting), and Sect. c (neuron depen- dencylexpressivity are factors of overfitting, but this does not constitute all factors).
433
434
435
436
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Focus on architecture and technical, no particular application affecting society is targeted. Note that medical ML applications should be thoroughly evaluated, e.g., for bias and generalizability, before being used in practice.
437
438
439
440
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
441
442
2. If you are including theoretical results...
443
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Only empirical results are included.
444
445
(b) Did you include complete proofs of all theoretical results? [N/A] Only empirical results included.
446
447
3. If you ran experiments...
448
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Code implementing the shown layer types and dataset descriptions are given in the supple- mental material/appendix.
450
452
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] We report the corresponding splits if not defined by the datasets themselves. Hyperparameters and information about the training are partly provided in the main text (e.g. architecture type, layer hyperparameters) and are detailed further in the appendix.
454
456
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We indicate statistical significance for our classification results.
458
459
460
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We have not tracked CO2 due to missing awareness but some details about the setup are provided in the appendix. We plan to track CO2 in subsequent works.
462
464
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
465
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sect. S This is done after first mentioning the corresponding datasets in the main text.
467
(b) Did you mention the license of the assets? [Yes] We mention asset licenses in the
468
appendix.
469
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide code as supplemental material.
470
471
(d) Did you discuss whether and how consent was obtained from people whose data you' using/curating? [N/A] The datasets in Sect. [53 5.3 are open source from past challenges and contain de-identified data to the best of our knowledge. Further details on data collection are provided in the corresponding references.
472
473
474
475
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [N/A] All datasets are commonly used for benchmarks and/or do not contain obviously offensive content. However, ImageNet likely contains images showing persons.
478
477
478
5. If you used crowdsourcing conducted research with human subjects...
13

---

480
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No crowdsoureing or research with human subjects has been conducted.
481
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] See above.
484
485
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] See above.
486
14


Reviews:
[{"review_id": "TMY1A8mka8f", "reviewer": "Reviewer_oqDg", "summary": "This paper designs and studies regularization effects for different types of output layers on training deep neural networks, which is very useful in practice. They propose five output layer designs and conduct various experiments including a case study. The results show that their proposed designs can achieve improvements of up to 10% on evaluation datasets. Overall, this paper is well-written and easy to follow. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The authors propose two important factors, neuron dependency and expressivity, for  measuring the regularization effect on different output layer designs. \n2) The authors conduct various experiments and a case study to validate their designs and the results show up to 10% improvements in accuracy for validation datasets. \n\nCons:\n1) In table 1, I wonder why random fixed layers can achieve such high performance in accuracy, even better than baseline. Is there any explanation? \n2) It is not very clear to me why Gradient⊙Activation can measure the dependency. The authors should give a brief discussion/explanation here. \n3) The novelty of this paper is somewhat limited. The proposed output designs seem very similar to existing approaches, e.g., w_random - random project, w_sparse - dropout.\n4)  What is the difference between w_spare and dropout regularization? What is the difference between w_ensemble and simultaneously training multiple classifiers?\n5) Figure 2 can be improved as it is a little bit hard to tell the difference by color. I recommend using different groups of colors or different types of line styles. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This has been discussed in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RuQE-SOwYAW", "reviewer": "Reviewer_V64q", "summary": "This experimental work studies various sparse output-layer types for transfer-learning. Authors report interesting gains through usage of certain learned/fixed output layers. Although I found the experiments interesting and believe that they would spark more research on the topic; the story and results needs a little bit more work.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths \n- The correlation between neuron saliency and results are very interesting and novel as far as I am aware.\n- Usage of sparse output layers for transfer also novel as far as I know. \n\n# Limitations/Questions/Recommendations\n0. It seems like authors use weight decay value of 0.0001 (seems to be same for backbone and output layer). This choice affects the performance of dense output layers greatly. Therefore needs to be searched (possibly only for the output layer). It would be great to have a curve showing the performance of dense baseline output layer with different regularisation coefficients on x-axis.\n1. I think the \"scaling head\" corresponds to using a temperature parameter in softmax (https://en.wikipedia.org/wiki/Softmax_function). Assuming that it is case, I think this connection must be clear and this variant should be studied under \"loss type\" used, as this is one parameter one can pick during finetuning (for all different architectures).\n2. Figure 2 didn't really helped me to understand the terms introduced neuron dependency/expressivity better. The formulas presented later did though. I recommend authors to use the definition earlier. More importantly I think authors should use existing terms instead of introducing new ones. \"neuron dependency\" means \"saliency\" (as cited in text). Similarly I think neuron expressivity measure can be captured with 'per-class saliency' or \"class-saliency\". I think the introduction can be shorten significantly with these changes helping reading to focus on main results. \n3. Can the ensemble connections be represented by a single sparse layer? It would be nice to discuss connections between random sparse layers, 1-to-1 layers and ensemble layers. To me the later 2 seems to be a specific case of a random sparse layer. Furthermore it is not clear why authors choose to report values for 90% for the sparse layers, despite the fact higher sparsities seem to give better results. \n\n## Minor Points \n- It would be better if References to accuracy gain (i.e. 10%) in abstract and intro has more detail in them like up and downstream datasets.\n- Having 3 input neurons in Figure-5 for 1-to-1 and ensemble layers can make them easier to understand.\n- Personally I find the p-scores confusing compared to confidence intervals or variance; which are used more commonly in ML (from what I read/see). Is there a reason this is preferred. If so, I think it would be nice to have a sentence on this in the main text.\n\n## After Rebuttal\nI like to thank authors for their response, especially for spending the time to create the weight decay ablation. It is helpful to see and I recommend authors to use the values found in the next version of the paper, even if the conclusions are not changing. Another recommendation is to have an l-1 penalized dense output layer baseline. \n\nI decided to keep my score as it is, which is a \"weak-accept\". I think the paper would become a clear accept with further experiments (i.e. more backbones/down-stream tasks like transformers and NLP) and ablation studies (like different levels of sparsity). \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors talk about the limitations at the end of the paper, which I appreciate. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "FQp_yp0i9y", "reviewer": "Reviewer_MAam", "summary": "Usually a fully connected layer is used as the final layer of an image classification network. To reduce the overfitting of this layer, this paper studies the use of activation scaling, fixed randomization, sparsity and in-layer ensembling as a regularization technique.\nThis paper also introduces neuron dependency and expressivity as two factors contributing to overfitting and proposes ways to measure these factors.\n", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- There are no evaluation results on the popular classification benchmarks such as ImageNet, CIFAR10 or CIFAR-100. Without results on these benchmarks it’s hard to know if the proposed method gives additional benefit in comparison to the current state-of-the-art regularization methods. Specially, it is important that the paper includes results on the large scale dataset of ImageNet as the results may change as we have more training data.\n- The current-state-of-the-art results on the provided benchmarks is much higher than the reported results in the paper. It would be helpful if authors provide results on top of stronger networks.\n- By adding more regularization to the previous layers of the network, we may need less regularization for the output layer. I think authors should also provide analysis for their proposed method when they apply it on top of other common regularization and also data-augmentation methods.\n- The writing of the paper is not clear and needs improvement.\n- Table 5 provides a comparison to the dropout and shows that ensemble method which builds multiple heads on top of the network has a better performance in comparison to the dropout. Are the proposed ensemble method and dropout orthogonal and can we combine them together?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion on potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have nay concern related to ethical issues.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "DvfChqb7osv", "reviewer": "Reviewer_kZyP", "summary": "This paper investigates the regularizing effect of different output layer designs in deep neural networks. The authors propose several different design choices of the output layers, which are all very simple but effective as the authors demonstrated in the experiments. They also propose two terms named neural dependency and expressivity between the neurons and classes. With these simple designs of the different output layers, the experiments on multiple classification tasks and resnet/densenet CNN models improve the performance for a large margin, which helps avoid the overfitting problem. \n\nThe main contributions:\n1. Propose simple yet effective output layer designs for deep neural networks.\n2. Introduce neural dependency and expressivity factors to measure the overfitting problem.\n3. Experiments on different classification tasks are expressive. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the overfitting problem and the regularization methods for deep neural networks. Different from previous work, the authors work on the output layer and specifically design different output layers for regularization. They introduce five different simple methods of the output layers. Easy to follow and interesting analysis, the introduced two factors are also some evidence for their analysis. The experiments are also express to see the improvements. The paper is also very clear to follow. \nHowever, there are also some questions and concerns:\n1. The alpha in the scaled output layers is designed to smooth the output distributions as I understand, though the authors connect to the dependencies. Therefore, it is expected alpha has huge effects on the model training, can you plot the curves of the performance change? Btw, the choice of alpha also depends on the different datasets since the data (class) distribution of each task should be different. It is expected to see the comparisons.\n2. Similarly, the choice of q is also interesting, could the authors also study the effect of q? Too many zero connections hurt the performance badly. For sparse fixed layers, could you give a clear comparison between this to the dropout and dropconnect method? Strong relationships, is sparse version a static version of dropout?\n3. Could you give some clear explanations or visualizations about the 1to1 layer? What's the most important impact of this design? At least, this does not change the output distributions.\n4. One major weakness is that the method is only compared with the basic resnet and densenet models, and also the basic training, without comparison to advanced methods/models. This significantly reduces the importance since this paper tries to sell the output design, which should be applied together with other stronger models and regularization methods to see the importance and effectiveness. \n5. Final minor one, In table 2, IN-top1 is only 76.36% for baseline. However, it seems that the Resnet-50 can achieve about 77.15% as shown in https://paperswithcode.com/sota/image-classification-on-imagenet. Is there some difference?\n\nOverall, I like this paper and am pleased to improve the score with answers. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work experiments with various strategies for improving addressing overfitting and regularization of the output layer. In particular, the authors explore ideas such as activation scaling, fixed randomization, sparsification, and ensembling as regularization techniques.This paper also introduces two methods, neuron dependency and expressivity, for measuring the degree of overfitting. The authors tested their ideas out on various image classification tasks including STL-10, CUB-200, Cars-196 and Food-101.

The reviewers were interested in the results and the techniques introduced for measuring overfitting. However, the reviewers also identified several concerns about the thoroughness of the experiments, and encouraged the authors to perform additional experiments and ablations to add more evidence for the viability of the methods. Additionally, no reviewer stepped forward to strongly advocate for the acceptance of this work. For all of these reasons, this paper will not be accepted at NeurIPS, but the authors are heavily encouraged to perform additional experiments and submit this to future venues.

Decision: Reject