The Regularizing Effect of Different Output Layer Designs in Deep Neural Networks

Deep neural networks are prone to overfitting , especially on small datasets . Com-1 mon regularizers such as dropout or dropconnect reduce overfitting , but are complex2 and prone to hyperparameter choices , thus prolonging development cycles in prac-3 tice . In this paper , we propose simple but effective design changes to the output4 layer - namely randomization , sparsity , activation scaling , and ensembling - that5 lead to improved regularization . These designs are motivated by experiments6 showing that standard fully-connected output layers tend to rely on individual7 input neurons , which in turn do not cover the variance of the data . We call these8 two related phenomena neuron dependency and expressivity , propose different9 ways to measure them , and optimize the presented output layers for them . In our10 experiments , we compare these layer types for image classification and semantic11 segmentation across architectures , datasets , and application settings . We report sig-12 nificantly and consistently improved performance of up to 10 % points in accuracy13 over standard output layers while reducing the number of trainable parameters by14 up to 90 % . It is demonstrated that neither training of output layers is required , nor15 are output layers themselves crucial components of deep networks.16 1 Introduction17 Neural networks are powerful feature extractors that have become the standard approach for a myriad18 of tasks . New architectures are continuously introduced and set records on benchmark datasets19 ( e.g . [ 24 , 15 , 44 ] ) . These networks differ in layer composition , depth/width or use specific concepts20 such as residual connections [ 15 ] or self-attention [ 34 ] . With growing capacity , their performance on21 large datasets tends to increase [ 44 ] . However , model complexity is also associated with overfitting,22 especially for small datasets where fine details of the training data are easily memorized [ 52 , 2 , 53 ] .23 Rather than defining another , possibly more complex architecture , we analyze what often remains24 unconsidered : the output layer . In image classification , networks usually end with a fully-connected25 ( fc ) layer that combines extracted features for the final output [ 43 , 15 , 17 , 44 ] . As we will show , this26 layer is prone to overfitting since high dependencies on individual , possibly memorized features can27 arise . The same neurons are subsequently not able to generalize across examples . We call these two28 related phenomena neuron dependency and expressivity and illustrate a simplified example in Fig . 2.29 Both problems can be improved by simple but effective changes to the output layer that require only30 few lines of code and achieve better generalization ( i.e. , better results on the test set [ 25 ] , see e.g.31 Fig . 1 ) . Those changes rely on four principles : activation scaling , fixed randomization , sparsity and32 in-layer ensembling ( see Fig . 5 ) . This work analyzes all layers in terms of their capability to reduce33 dependencies and/or increase expressivity . Then , the connection to network performance is shown34 through a comprehensive empirical study across datasets , architectures and application settings in35 image classification and segmentation . Furthermore , we investigate how stronger regularization36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . can be induced by applying the identified principles to other parts of a network while reducing the37 computational footprint . In contrast to common practice , we find neither training of output layers38 to be necessary , nor that output layers are crucial components of deep networks . In summary , our39 contributions are:40 • Introducing neuron dependency and expressivity as two factors contributing to overfitting41 and proposing ways to measure these factors42 • Showing improved regularization of 5 different output layer designs up to 10 % in absolute43 accuracy compared to standard fc layers and other common regularizers44 • Empirical results showing that the proposed layers have improved dependency and expressiv-45 ity , computational efficiency , wide applicability to both small and large datasets , extensibility46 to other parts of the network , and robustness in the choice of hyperparameters47 2 Related Work48 Regularization in deep learning is approached in various ways . Widely used methods are , e.g.,49 normalization [ 19 , 3 ] , weight decay [ 32 ] , data and adversarial augmentation [ 40 , 1 ] , early stopping [ 7 ] ,50 boosting [ 38 ] , multitask learning [ 6 ] , dropout [ 42 ] , dropconnect [ 50 ] , and Gaussian noise layers [ 10 ] .51 To the best of our knowledge , this is the first work that evaluates regularization with respect to52 different output layer designs . Similar to dropout/dropconnect , output layers can be categorized as53 affecting the architecture according to the regularization taxonomy described in [ 25 ] . Unlike other54 regularizers , our methods are either hyperparameter-free or robust to their choice and can be applied55 to any deep net , including pre-trained ones that are less affected by overfitting ( see Sect . 5.4 and 5.7 ) .56 Related to fixed randomization are the output layers used in [ 16 , 39 , 14 ] , which show comparable57 performance to trained layers . One can also preallocate output layer weights with a defined struc-58 ture [ 31 , 16 ] . Besides output layers , weight fixing is for example applied to the first layer in the59 Extreme Learning Machine [ 18 ] , or to different weight dimensions in [ 36 ] . In contrast , we omit60 hand-crafted weights , show improved regularization and relate to neuron dependencies . Further , we61 show that fixing or scaling the last conv block next to the output layer has a strong regularizing effect.62 Sparsity is common in deep learning , e.g . the ReLU activation [ 13 ] or a L1 penalty term in the loss63 function [ 46 ] . Sparsity has also been applied to the channels of Convolutional Neural Networks64 ( CNNs ) [ 8 , 29 ] . Others induce sparsity by pruning connections before training under the lottery65 ticket hypothesis [ 11 , 30 ] , with the goal of reducing the number of parameters while not sacrificing66 performance [ 27 , 45 , 51 ] . Different to them , we show that ( extreme ) sparsity is not merely useful to67 improve computational efficiency , but to improve performance when applied to the output layer.68 The Network in Network ( NIN ) [ 28 ] and All-CNN [ 41 ] both use global average pooling ( GAP ) 69 followed by softmax , which replaces the fc output layer with an identify transform to simplify the70 network . This is further analyzed in [ 33 ] . We show its connection to neuron dependency/expressivity71 and achieve comparable or better performance on various datasets . Further , we observe that previous72 works do not leverage the full capacity of the last layer in modern networks , which enables the73 construction of computationally efficient in-layer ensembles that further boost performance in small74 and large datasets.75 3 Neuron dependency and expressivity76 3.1 Setting and notation77 We consider neural networks consisting of an encoder fenc : X → a followed by an output layer78 fout : a → ŷ . In this paper , the encoder is a CNN , taking as input an image X ∈ RC×H×W with79 C , W and H being input channels , width and height , respectively ; and transforming it to a feature80 vector a ∈ R1×N . Commonly in CNNs , 2D representations resulting from the final conv layer81 are aggregated by GAP [ 28 ] where N corresponds to the number of pooled conv channels . The82 output layer transforms the embedding to output ŷ ∈ RK holding the probabilities of K classes.83 The output layer is parameterized by a weight matrix W ∈ RN×K , and is commonly initialized as84 W random ∼ U ( − √ 1/N , √ 1/N ) [ 26 ] . Both ŷ and target y are used to compute the cross-entropy85 loss ` = − ∑K i yi log ( ŷi ) . We use the terms features/channels/nodes or neurons interchangeably86 meaning activations a . When required , we refer to individual instances with a superscript , e.g.87 ( X ( i ) , y ( i ) ) ∈ D , with D being a dataset . Corresponding subsets are denoted as Dtrain and Dtest.88 3.2 Concepts89 During training , CNNs learn a set of visual patterns that are combined for a classification decision.90 However , if patterns remain undetected , e.g . due to noise in the image or inherent but unseen variance91 in the data , their activation values can become small and thus reduce the output values for the target92 class . When a network is overfitting , it learns malignant image-specific patterns by heart [ 52 ] . Such a93 network may depend on the activation of individual nodes , which in turn fail to generalize to patterns94 that are salient to a class . We call these two related phenomena neuron dependency and expressivity.95 96 Neuron dependency : How much does a model depend on a single neuron ? In a network97 with high neuron dependencies , output scores and thus performance drop significantly when certain98 neurons remain inactive . In contrast , a network with low neuron dependencies distributes activations99 across many neurons , so that a single inactive node does not have much influence on the classification.100 Neuron expressivity : How much class-specific variance does a neuron cover ? Neurons with low101 expressivity focus on unimportant details that do not characterize the properties of a class . In contrast,102 a neuron with high expressivity generalizes by activating to various patterns pertinent to a given class.103 An example of neuron dependency/expressivity for a simplified fc output layer is illustrated in Fig . 2.104 3.3 Measuring dependency and expressivity105 We introduce two ways of measuring dependency/expressivity : instance-based and class-based . The106 former is used to determine the dependency on the most important node for the predicted class given an107 instance . Importance scores for node n and output class k̂ are computed with Gradient Activation [ 4 ] ,108 a global attribution method where we leverage the partial derivative of the softmax values:109 an ∂ŷk̂ ∂an . ( 1 ) Instance-based dependency is then measured as avg . reduction in output probabilities when ablating110 the most important feature w.r.t . the output class of any instance . This is illustrated for various111 training set sizes of CIFAR-100 [ 23 ] in Fig . 3 ( left ) . With less data , fc output layers tend to depend112 more on single nodes . This is in contrast to class-based measures , which enable quantifying both113 dependency/expressivity and use various features jointly . Importances are determined for each class114 k and over all test instances:115 |Dtest|∑ i=1 a ( i ) n ∂ŷ ( i ) k ∂a ( i ) n . ( 2 ) Class-based dependency is then measured as drop in accuracy when ablating a given number of most116 important neurons per class . Measuring expressivity reverses this - the most important neurons per117 class are retained , all others are ablated . This is illustrated for both dependency/expressivity in Fig . 4.118 We see that standard ( i.e . trained ) fc output layers tend to depend on single channels to achieve high119 performance , but these very channels hold only limited class information.120 4 Output Layer Types121 We describe several simple output layer variants that require minimal changes to standard networks,122 decrease neuron dependency and/or increase neuron expressivity . All types are illustrated in Fig . 5.123 4.1 Standard output layers124 The ubiquitous approach to compute class scores is to learn the parameters of a weight matrix125 W trained , s.t . ŷ = σSM ( aW trained ) with σSM ( · ) being softmax . Each feature is considered in the126 computation of each class score . As shown in Sect . 3 , trained fc output layers can lead to high neuron127 dependencies , where the deletion of a single neuron might cause significant loss in performance , and128 low neuron expressivity , where multiple features are required for adequate predictions.129 4.2 Scaled output layers130 The reduction of an activation , e.g . due to changing light conditions , has a large influence on the131 output scores . This is simulated in Fig . 3 ( center ) by multiplying features during training with a132 scalar α > 0 , so that ŷ = σSM ( αaW scaled ) . Note that the variances of the output logit distributions133 increase with α , resulting in larger differences ( or smaller entropies ) after softmax normalization.134 This results in greater dependencies of the model on individual neurons , as shown in Fig . 3 ( right ) .135 However , if α is chosen small , the activations of individual neurons become insufficient for class136 discrimination with high confidence . The model is therefore forced to learn multiple class-specific137 features for each instance , which increases the expressivity of the neurons and also reduces their138 dependencies to some extent , as shown in Fig . 4 . If not specified otherwise , we use α = 0.1.139 4.3 Random fixed layers140 This setting uses W random during training/inference , and its classification performance was first141 analyzed in [ 16 ] . The encoder learns to extract patterns that adjust to predetermined weights . Unlike142 activation scaling , the parameters are bounded and fixed to a small value range . For any class , the143 chosen uniform initialization is expected to assign similar weight values to multiple neurons , making144 them learn similar features . We suppose that the enforced similarity reduces dependency shown in145 Fig . 3 and 4 ( both left ) , while small initialization values increase expressivity as in Sect . 4.2 , shown146 in Fig . 4 ( right ) .147 4.4 Sparse fixed layers148 In sparse output layers , class nodes use predetermined sets of channels , some of which might149 be shared across classes . First , a set of cutting indices Ik is randomly sampled for each class150 k , where sparsity is determined by the proportion q of class-specific connections to cut , so that151 |Ik| = bqNcwith 0 < q < 1 . Then , starting from a fixed random initialization as in Sect . 4.3 , weights152 connecting to a given class are ablated so that : W sparsei , k = 0 ∀ i ∈ Ik . Hyperparameter q trades153 off dependency/expressivity . Larger values induce more sparsity , leading to greater dependencies to154 the remaining nodes , but forcing them to activate across instances , making them expressive . We set155 q = 0.9 in the experiments to show that high sparsity benefits generalization.156 4.5 1-to-1 correspondence layers157 The most extreme type of sparsity in an output layer is one with a single connection between a158 feature and a class . If these connections correspond to an identity transform , the activations of the159 penultimate layer are equivalent to the class logits - in practice , the output layer can hence be omitted.160 This was analyzed in [ 33 , 28 ] and showed comparable results to a standard output layer . Formally , we161 have ŷ = σSM ( aW 1to1 ) with a ∈ R1×K and W 1to1 ∈ RK×K , where W 1to1 = diag ( 1 , 1 , . . . , 1 ) .162 In this layer , both the model ’ s dependency on individual neurons as well as each neuron ’ s expressivity163 are maximal . If a single neuron is ablated , the output logits for the class this neuron is connected to is164 reduced to zero . However , individual neurons learn to cover the whole variance of a given class in the165 training set , which is one conjecture for their performance . Note that as mentioned in [ 33 ] , we have166 the constraint N = K , which might be restrictive for small networks and large numbers of classes.167 4.6 Ensemble layers168 Is there a way to optimize for both low neuron dependency and high expressivity ? Of the approaches169 discussed , 1-to-1 correspondence layers have the highest expressivity . Starting from this layer , a170 simple approach to reduce neuron dependency is to use the capacity of the penultimate layer and171 create multiple heads h = 1 . . . H with N = KH , each head being a 1-to-1 correspondence layer.172 Each head ’ s output is effectively computed as ŷh = σSM ( αah ) with ah being the activation part of173 head h. As in Sect 4.2 , we introduce a scalar α , which controls the magnitude of feature activations.174 For consistency , we denote this approach as W heads . The loss is computed for each head and175 averaged : 1H ∑H h=1 ` ( ŷ h , y ) . Similarly , logits are averaged over heads for inference . Due to the176 induced redundancy , the performance only drops considerably after removing class-related neurons177 from all heads . In our experiments , we set H to its maximum given any setting ( architecture/dataset ) .178 Note that hyperparameter α in ensemble layers is the only one which is tuned to individual settings.179 5 Experiments180 We aim to show that the presented output layers from Sect . 4 outperform standard output layers and181 common regularization methods in various settings . Details about training , compute resources , code,182 datasets as well as additional experiments on dependency/expressivity are included in the appendix.183 5.1 Small-scale and fine-grained classification184 All layer types are first applied to small-scale and/or fine-grained classification , both of which are185 challenging and require regularization . Datasets include STL-10 ( 500 img/class ) [ 9 ] , CUB-200 ( ∼30186 img/class ) [ 49 ] , Cars-196 ( ∼40 img/class ) [ 22 ] and Food-101 ( 750 img/class ) [ 5 ] . Table 1 shows187 results for the two popular backbones ResNet-50 [ 15 ] and DenseNet-169 [ 17 ] , exchanging the output188 layer accordingly . In 53/56 settings , we see improved results over standard layers . Of these , 48189 and 36 are significant with p < 0.1 and p < 0.001 , respectively . Although there is no clear best190 method , it is worth noting that sparse and ensemble layers as enhancements of both random and191 1-to-1 layers are significantly better ( p < 0.001 ) in 7/8 settings , respectively . As expected , smaller192 performance differences are exhibited in Food-101 , which is a considerably larger dataset , thus193 requiring less regularization . Among the worse settings , only 1 is significant ( p < 0.1 ) for Food-101194 since it involves strong regularization to multiple layers . These regularizers are discussed seperately195 in Sect . 5.5.196 5.2 Large-scale classification and transfer learning197 Machine learning models are subject to the bias-variance tradeoff [ 12 ] , in which induced biases of198 the presented output layers might be too strong to fit the training data . We therefore want to shed199 light on how these layers behave in large-scale and transfer learning settings , where overfitting is200 less problematic . Datasets include CIFAR-100 ( C100 , 5000 img/class ) [ 23 ] , ImageNet ( IN , ∼1200201 img/class ) from ILSVRC2012 [ 37 ] reported on the validation set , as well as CUB/Cars/Food with202 models being pre-trained on IN . Table 2 shows the results for the ResNet-50 backbone . In C100,203 we see consistent improvements with at least p < 0.1 . On the other datasets , results are mostly204 comparable corroborating widespread applicability . It is worth mentioning thatW 1to1 andW ensemble205 perform consistently better , and W ensemble significantly ( p < 0.1 ) in multiple cases . With growing206 dataset sizes , both layers expose a strong constraint on the class neurons to fit an increasing number207 of examples . We believe this to be responsible for progressively separating the signal from the noise,208 leading to better generalization . On the other hand , neuron dependency is reduced in larger datasets209 ( see Fig . 3 left ) diminishing the effect of W scale and W random . Moreover , W random and W sparse210 can be affected by predetermined feature-class weights that do not have to match features learned211 during pre-training , which might require larger adjustments to the weights of the last conv layer.212 5.3 Use Case : Medical imaging213 Output layer design is critical in fields such as medical imaging , which presents special challenges to214 regularization : Datasets tend to be small , imbalanced , abnormalities might fill only a few pixels of215 the image , and appearances between classes are often similar . In addition , transfer learning with IN216 weights is either inaccessible due to architectural differences ( e.g . image segmentation , 3D Magnetic217 Resonance Imaging ) or less effective due to large domain differences . This is first illustrated on the218 APTOS Kaggle challenge dataset ( 3662 images , 193-1805 img/class ) [ 20 ] , with the goal of detecting219 diabetic retinopathy severities in retinal fundus images . We use the public training dataset to train a220 multi-class classifier and perform 5-fold cross-validation . Table 3 shows the results . We consistently221 get better performance with regularization and reduce the gap to a pre-trained network . Furthermore,222 an additional experiment in the appendix indicates that the standard output layer is biased towards the223 prevalent class , which is inherently remedied through randomization.224 We provide further evidence that the proposed layer designs positively affect tasks other than225 classification . We learn a U-Net [ 35 ] for binary semantic slice-based segmentation of Computed226 Tomography scans of livers comparing a standard 1x1 conv output layer with 64 parameters to both a227 fixed randomized and an ensemble layer . Due to the limited number of parameters , we omit W scale228 andW sparse here . Different to classification , the output of a U-Net itself can be interpreted as a 1-to-1229 layer . One can still build an ensemble by treating each output channel as a head . Both W random and230 W ensemble ( H = 10 ) are then applied to the CHAOS [ 21 ] and SLIVER [ 47 ] datasets . For CHAOS,231 Table 3 : Quadratic weighted kappa and accuracy ( with significance ) for different output layers in ResNet-50 for the APTOS dataset . + denotes fine-tuning from IN . See Table 1 for other symbols . Kappa Acc . W trained 0.816 77.44 we train on 15 randomly selected patients ( 2155 slices ) and evaluate on the remaining 5 ( 719 slices ) .232 We then test for generalization by training on all 20 patients from CHAOS and evaluating on the233 external SLIVER dataset consisting of 20 patients ( 4159 slices ) . Table 4 shows improved results in234 both settings.235 5.4 Other regularization techniques236 Table 5 compares our most competitive methods to other popular regularizers when applied to a237 standard output layer . Nodes/connections in dropout/dropconnect are both removed with p = 0.7,238 and the noise layer adds a Gaussian with µ = 0 and σ = 0.1 before applying the fc layer . In all239 cases , our variants perform better . Whereas noise does not benefit training here , dropout/-connect is240 supporting regularization . However , both of the latter methods come with two main disadvantages.241 First , they add complexity by changing states in each iteration and having different behavior during242 training and inference . Second , hyperparameter tuning is necessary , while our layers are either243 hyperparameter-free or stable to them . See the ablation study in Sect . 5.7 for evidence.244 5.5 Beyond output layers - block scaling and randomization245 Activation scaling and randomization are techniques applicable to any layer and increase regulariza-246 tion further . This is demonstrated for ResNet-50 and DenseNet-169 in Table 1 . Both architectures247 consist of multiple blocks , each holding groups of conv layer , batch normalization ( BN ) and activa-248 tion function . For W random block , all layers of the last block and the output layer are kept in their249 initialized state during training . Similarly , in W scaled block , activations of all layer groups in the last250 block are scaled during training . In ResNet-50 , block scaling outperforms output layer scaling in 3/4251 datasets by up to 3 % points , and block randomization increases performance in 3/4 datasets by up to252 6 % points compared to output layer randomization . In DenseNet-169 , block scaling outperforms out-253 put layer scaling in 2/4 datasets by up to 2 % points , and block randomization increases performance254 in 2/4 datasets by up to 4 % points compared to output layer randomization . Only in Food-101 and255 DenseNet , block randomization performs significantly worse than baseline because regularization is256 too strong leading to underfitting ( train loss = 0.47 compared to 0.02 in W random ) .257 5.6 Computational efficiency258 The design of the head of deep CNNs has a great impact on computational efficiency . Standard output259 layers alone can contain a large amount of parameters , as CNNs typically hold more channels as260 they get deeper and the number of classes can become large . In ImageNet and a ResNet-50 , for261 example , the output layer alone generates over 2 million parameters , which are saved in W 1to1 and262 W ensemble . This problem compounds when using multiple fc layers . In a VGG-16 , for instance , 3 fc263 layers are employed after the last conv layer . As Table 6 shows , omitting all fc layers saves up to264 90 % in parameters , a considerable amount of memory , and time for a forward/backward pass while265 increasing accuracy . If a ResNet-50 is used , randomization of the last conv block next to the output266 layer yields savings of about 65 % in trainable parameters while increasing accuracy by 10 % points.267 5.7 Ablation study268 Note that W random and W 1to1 are hyperparameter-free compared to other regularizers such as269 dropout/-connect , thus saving the cost of tuning them . Although other layer variants possess hyperpa-270 rameters , we show in Fig . 6 for the ResNet backbone that they are stable ( no large jumps in vicinity ) 271 and consistent ( tend to monotonicity w.r.t . performance ) . In W sparse , the maximum accuracy for272 both datasets is at q = 0.99 ( 20 nodes per class ) and drops only slightly for q = 0.995 . Similarly,273 downscaling in W scale improves performance at a small cost if the optimum is not hit . Also , more274 heads in W ensemble tend to increase performance . What is the result of adding more heads than275 given by the constraint N = KH ? If N < KH , which is the case for CUB-200 and H > 10 , we276 add an additional 1x1 conv layer , BN and ReLU with KH nodes to adjust for the missing channels.277 Although this leads to a considerable increase in parameters ( NKH for the conv layer ) , it helps with278 generalization , contradicting the common belief that overparameterization leads to overfitting [ 48 ] .279 In contrast , dropout is not stable or consistent . With a dropout rate of 0.9 , the network fails to train280 in both datasets . Furthermore , the optimum for CUB lies at 0.8 , the same hyperparameter choice in281 STL would result in worse performance than baseline.282 6 Conclusion283 In this work , we introduced neuron dependency and expressivity as factors contributing to overfitting.284 Then , different output layers were defined to optimize both and showed improved regularization in285 various settings while being efficient and robust to hyperparameters . Although these layers are simple,286 they have high practical relevance due to the importance of regularization and the ubiquity of output287 layers in deep nets . In addition to their application , they may also be useful as primitives in future288 ( automatically created ) architectures . Although improving regularization , we note that optimizing289 for neuron dependencies/expressivity does not solve overfitting . For example , an unknown or noisy290 instance may result in reduced activations in the majority of nodes in the penultimate layer . Finally,291 we speculate that overfitting may not just be a function of the number of parameters in the encoder.292 Instead , it might be more important how the extracted features are combined in the output layer.293 References294 [ 1 ] Antreas Antoniou , Amos Storkey , and Harrison Edwards . Data augmentation generative295 adversarial networks . arXiv preprint arXiv:1711.04340 , 2017.296 [ 2 ] Devansh Arpit , Stanisław Jastrzębski , Nicolas Ballas , David Krueger , Emmanuel Bengio,297 Maxinder S Kanwal , Tegan Maharaj , Asja Fischer , Aaron Courville , Yoshua Bengio , et al.298 A closer look at memorization in deep networks . In International Conference on Machine299 Learning , pages 233–242 . PMLR , 2017.300 [ 3 ] Jimmy Lei Ba , Jamie Ryan Kiros , and Geoffrey E Hinton . Layer normalization . arXiv preprint301 arXiv:1607.06450 , 2016.302 [ 4 ] David Baehrens , Timon Schroeter , Stefan Harmeling , Motoaki Kawanabe , Katja Hansen , and303 Klaus-Robert Müller . How to explain individual classification decisions . The Journal of304 Machine Learning Research , 11:1803–1831 , 2010.305 [ 5 ] Lukas Bossard , Matthieu Guillaumin , and Luc Van Gool . Food-101 – mining discriminative306 components with random forests . In European Conference on Computer Vision , 2014.307 [ 6 ] Rich Caruana . Multitask learning . Machine learning , 28 ( 1 ) :41–75 , 1997.308 [ 7 ] Rich Caruana , Steve Lawrence , and Lee Giles . Overfitting in neural nets : Backpropagation,309 conjugate gradient , and early stopping . Advances in neural information processing systems,310 pages 402–408 , 2001.311 [ 8 ] Soravit Changpinyo , Mark Sandler , and Andrey Zhmoginov . The power of sparsity in convolu-312 tional neural networks . arXiv preprint arXiv:1702.06257 , 2017.313 [ 9 ] Adam Coates , Andrew Ng , and Honglak Lee . An analysis of single-layer networks in unsuper-314 vised feature learning . In Proceedings of the fourteenth international conference on artificial315 intelligence and statistics , pages 215–223 . JMLR Workshop and Conference Proceedings , 2011.316 [ 10 ] Terrance DeVries and Graham W Taylor . Dataset augmentation in feature space . arXiv preprint317 arXiv:1702.05538 , 2017.318 [ 11 ] Jonathan Frankle and Michael Carbin . The lottery ticket hypothesis : Finding sparse , trainable319 neural networks . arXiv preprint arXiv:1803.03635 , 2018.320 [ 12 ] Stuart Geman , Elie Bienenstock , and René Doursat . Neural networks and the bias/variance321 dilemma . Neural computation , 4 ( 1 ) :1–58 , 1992.322 [ 13 ] Xavier Glorot , Antoine Bordes , and Yoshua Bengio . Deep sparse rectifier neural networks . In323 Proceedings of the fourteenth international conference on artificial intelligence and statistics,324 pages 315–323 . JMLR Workshop and Conference Proceedings , 2011.325 [ 14 ] Moritz Hardt and Tengyu Ma . Identity matters in deep learning . arXiv preprint326 arXiv:1611.04231 , 2016.327 [ 15 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image328 recognition . In Proceedings of the IEEE conference on computer vision and pattern recognition,329 pages 770–778 , 2016.330 [ 16 ] Elad Hoffer , Itay Hubara , and Daniel Soudry . Fix your classifier : the marginal value of training331 the last weight layer . arXiv preprint arXiv:1801.04540 , 2018.332 [ 17 ] Gao Huang , Zhuang Liu , Laurens Van Der Maaten , and Kilian Q Weinberger . Densely connected333 convolutional networks . In Proceedings of the IEEE conference on computer vision and pattern334 recognition , pages 4700–4708 , 2017.335 [ 18 ] Guang-Bin Huang , Qin-Yu Zhu , and Chee-Kheong Siew . Extreme learning machine : theory336 and applications . Neurocomputing , 70 ( 1-3 ) :489–501 , 2006.337 [ 19 ] Sergey Ioffe and Christian Szegedy . Batch normalization : Accelerating deep network training338 by reducing internal covariate shift . In International conference on machine learning , pages339 448–456 . PMLR , 2015.340 [ 20 ] Kaggle . Aptos 2019 blindness detection , 2019 . URL https : //www.kaggle.com/c/341 aptos2019-blindness-detection.342 [ 21 ] A Emre Kavur , N Sinem Gezer , Mustafa Barış , Sinem Aslan , Pierre-Henri Conze , Vladimir343 Groza , Duc Duy Pham , Soumick Chatterjee , Philipp Ernst , Savaş Özkan , et al . Chaos challenge-344 combined ( ct-mr ) healthy abdominal organ segmentation . Medical Image Analysis , 69:101950,345 2021.346 [ 22 ] Jonathan Krause , Michael Stark , Jia Deng , and Li Fei-Fei . 3d object representations for347 fine-grained categorization . In 4th International IEEE Workshop on 3D Representation and348 Recognition ( 3dRR-13 ) , Sydney , Australia , 2013.349 [ 23 ] Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images.350 2009.351 [ 24 ] Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . Imagenet classification with deep352 convolutional neural networks . Advances in neural information processing systems , 25:1097–353 1105 , 2012.354 [ 25 ] Jan Kukačka , Vladimir Golkov , and Daniel Cremers . Regularization for deep learning : A355 taxonomy . arXiv preprint arXiv:1710.10686 , 2017.356 [ 26 ] Yann A LeCun , Léon Bottou , Genevieve B Orr , and Klaus-Robert Müller . Efficient backprop.357 In Neural networks : Tricks of the trade , pages 9–48 . Springer , 2012.358 [ 27 ] Namhoon Lee , Thalaiyasingam Ajanthan , and Philip HS Torr . Snip : Single-shot network359 pruning based on connection sensitivity . arXiv preprint arXiv:1810.02340 , 2018.360 [ 28 ] Min Lin , Qiang Chen , and Shuicheng Yan . Network in network . arXiv preprint arXiv:1312.4400,361 2013.362 [ 29 ] Baoyuan Liu , Min Wang , Hassan Foroosh , Marshall Tappen , and Marianna Pensky . Sparse363 convolutional neural networks . In Proceedings of the IEEE conference on computer vision and364 pattern recognition , pages 806–814 , 2015.365 [ 30 ] Eran Malach , Gilad Yehudai , Shai Shalev-Schwartz , and Ohad Shamir . Proving the lottery366 ticket hypothesis : Pruning is all you need . In International Conference on Machine Learning,367 pages 6682–6691 . PMLR , 2020.368 [ 31 ] Federico Pernici , Matteo Bruni , Claudio Baecchi , and Alberto Del Bimbo . Fix your features:369 Stationary and maximally discriminative embeddings using regular polytope ( fixed classifier ) 370 networks . arXiv preprint arXiv:1902.10441 , 2019.371 [ 32 ] David C Plaut , Steven J Nowlan , and Geoffrey E Hinton . Experiments on learning by back372 propagation . 1986.373 [ 33 ] Zhongchao Qian . Deep Convolutional Networks without Learning the Classifier Layer . PhD374 thesis , Rochester Institute of Technology , 2020.375 [ 34 ] Prajit Ramachandran , Niki Parmar , Ashish Vaswani , Irwan Bello , Anselm Levskaya , and376 Jonathon Shlens . Stand-alone self-attention in vision models . arXiv preprint arXiv:1906.05909,377 2019.378 [ 35 ] Olaf Ronneberger , Philipp Fischer , and Thomas Brox . U-net : Convolutional networks for379 biomedical image segmentation . In International Conference on Medical image computing and380 computer-assisted intervention , pages 234–241 . Springer , 2015.381 [ 36 ] Amir Rosenfeld and John K Tsotsos . Intriguing properties of randomly weighted networks:382 Generalizing while learning next to nothing . In 2019 16th Conference on Computer and Robot383 Vision ( CRV ) , pages 9–16 . IEEE , 2019.384 [ 37 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng385 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei.386 ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision387 ( IJCV ) , 115 ( 3 ) :211–252 , 2015. doi : 10.1007/s11263-015-0816-y.388 [ 38 ] Holger Schwenk and Yoshua Bengio . Boosting neural networks . Neural computation , 12 ( 8 ) :389 1869–1887 , 2000.390 [ 39 ] Gabi Shalev , Gal-Lev Shalev , and Joseph Keshet . Redesigning the classification layer by391 randomizing the class representation vectors . arXiv preprint arXiv:2011.08704 , 2020.392 [ 40 ] Connor Shorten and Taghi M Khoshgoftaar . A survey on image data augmentation for deep393 learning . Journal of Big Data , 6 ( 1 ) :1–48 , 2019.394 [ 41 ] Jost Tobias Springenberg , Alexey Dosovitskiy , Thomas Brox , and Martin Riedmiller . Striving395 for simplicity : The all convolutional net . arXiv preprint arXiv:1412.6806 , 2014.396 [ 42 ] Nitish Srivastava , Geoffrey Hinton , Alex Krizhevsky , Ilya Sutskever , and Ruslan Salakhutdinov.397 Dropout : a simple way to prevent neural networks from overfitting . The journal of machine398 learning research , 15 ( 1 ) :1929–1958 , 2014.399 [ 43 ] Christian Szegedy , Wei Liu , Yangqing Jia , Pierre Sermanet , Scott Reed , Dragomir Anguelov,400 Dumitru Erhan , Vincent Vanhoucke , and Andrew Rabinovich . Going deeper with convolutions.401 In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1–9,402 2015.403 [ 44 ] Mingxing Tan and Quoc Le . Efficientnet : Rethinking model scaling for convolutional neural404 networks . In International Conference on Machine Learning , pages 6105–6114 . PMLR , 2019.405 [ 45 ] Hidenori Tanaka , Daniel Kunin , Daniel LK Yamins , and Surya Ganguli . Pruning neural networks406 without any data by iteratively conserving synaptic flow . arXiv preprint arXiv:2006.05467,407 2020.408 [ 46 ] Robert Tibshirani . Regression shrinkage and selection via the lasso . Journal of the Royal409 Statistical Society : Series B ( Methodological ) , 58 ( 1 ) :267–288 , 1996.410 [ 47 ] Bram Van Ginneken , Tobias Heimann , and Martin Styner . 3d segmentation in the clinic : A411 grand challenge . In MICCAI Workshop on 3D Segmentation in the Clinic : A Grand Challenge,412 volume 1 , pages 7–15 , 2007.413 [ 48 ] N Vapnik Vladimir and V Vapnik . Statistical learning theory . Xu JH and Zhang XG . translation.414 Beijing : Publishing House of Electronics Industry , 2O04 , 1998.415 [ 49 ] C. Wah , S. Branson , P. Welinder , P. Perona , and S. Belongie . The Caltech-UCSD Birds-200-2011416 Dataset . Technical Report CNS-TR-2011-001 , California Institute of Technology , 2011.417 [ 50 ] Li Wan , Matthew Zeiler , Sixin Zhang , Yann Le Cun , and Rob Fergus . Regularization of418 neural networks using dropconnect . In International conference on machine learning , pages419 1058–1066 . PMLR , 2013.420 [ 51 ] Chaoqi Wang , Guodong Zhang , and Roger Grosse . Picking winning tickets before training by421 preserving gradient flow . arXiv preprint arXiv:2002.07376 , 2020.422 [ 52 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding423 deep learning requires rethinking generalization . arXiv preprint arXiv:1611.03530 , 2016.424 [ 53 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Michael C Mozer , and Yoram Singer . Identity425 crisis : Memorization and generalization under extreme overparameterization . arXiv preprint426 arXiv:1902.04698 , 2019.427 Checklist428 1 . For all authors ... 429 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s430 contributions and scope ? [ Yes ] See Sect . 3 , 5 and the appendix.431 ( b ) Did you describe the limitations of your work ? [ Yes ] See Sect . 4.5 ( N = K con-432 straint ) , Sect . 5.2 ( some layers are not optimal for large-scale datasets and fine-tuning ) ,433 Sect . 5.5 ( strong regularization may lead to underfitting ) , and Sect . 6 ( neuron depen-434 dency/expressivity are factors of overfitting , but this does not constitute all factors ) .435 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] Focus436 is on architecture and technical , no particular application affecting society is targeted.437 Note that medical ML applications should be thoroughly evaluated , e.g. , for bias and438 generalizability , before being used in practice.439 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to440 them ? [ Yes ] 441 2 . If you are including theoretical results ... 442 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] Only empirical443 results are included.444 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] Only empirical results445 are included.446 3 . If you ran experiments ... 447 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-448 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] Code449 implementing the shown layer types and dataset descriptions are given in the supple-450 mental material/appendix.451 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they452 were chosen ) ? [ Yes ] We report the corresponding splits if not defined by the datasets453 themselves . Hyperparameters and information about the training are partly provided in454 the main text ( e.g . architecture type , layer hyperparameters ) and are detailed further in455 the appendix.456 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-457 ments multiple times ) ? [ Yes ] We indicate statistical significance for our classification458 results.459 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type460 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] We have not tracked CO2 due to461 missing awareness but some details about the setup are provided in the appendix . We462 plan to track CO2 in subsequent works.463 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 464 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See Sect . 5 . This is465 done after first mentioning the corresponding datasets in the main text.466 ( b ) Did you mention the license of the assets ? [ Yes ] We mention asset licenses in the467 appendix.468 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 469 We provide code as supplemental material.470 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re471 using/curating ? [ N/A ] The datasets in Sect . 5.3 are open source from past challenges472 and contain de-identified data to the best of our knowledge . Further details on data473 collection are provided in the corresponding references.474 ( e ) Did you discuss whether the data you are using/curating contains personally identi-475 fiable information or offensive content ? [ N/A ] All datasets are commonly used for476 benchmarks and/or do not contain obviously offensive content . However , ImageNet477 likely contains images showing persons.478 5 . If you used crowdsourcing or conducted research with human subjects ... 479 ( a ) Did you include the full text of instructions given to participants and screenshots,480 if applicable ? [ N/A ] No crowdsourcing or research with human subjects has been481 conducted.482 ( b ) Did you describe any potential participant risks , with links to Institutional Review483 Board ( IRB ) approvals , if applicable ? [ N/A ] See above.484 ( c ) Did you include the estimated hourly wage paid to participants and the total amount485 spent on participant compensation ? [ N/A ] See above.486

Summary: This paper designs and studies regularization effects for different types of output layers on training deep neural networks, which is very useful in practice. They propose five output layer designs and conduct various experiments including a case study. The results show that their proposed designs can achieve improvements of up to 10% on evaluation datasets. Overall, this paper is well-written and easy to follow.