The Regularizing Effect of Different Output Layer Designs in Deep Neural Networks

Deep neural networks are prone to overfitting , especially on small datasets . Com-1 mon regularizers such as dropout or dropconnect reduce overfitting , but are complex2 and prone to hyperparameter choices , thus prolonging development cycles in prac-3 tice . In this paper , we propose simple but effective design changes to the output4 layer - namely randomization , sparsity , activation scaling , and ensembling - that5 lead to improved regularization . These designs are motivated by experiments6 showing that standard fully-connected output layers tend to rely on individual7 input neurons , which in turn do not cover the variance of the data . We call these8 two related phenomena neuron dependency and expressivity , propose different9 ways to measure them , and optimize the presented output layers for them . In our10 experiments , we compare these layer types for image classification and semantic11 segmentation across architectures , datasets , and application settings . We report sig-12 nificantly and consistently improved performance of up to 10 % points in accuracy13 over standard output layers while reducing the number of trainable parameters by14 up to 90 % . It is demonstrated that neither training of output layers is required , nor15 are output layers themselves crucial components of deep networks.16 1 Introduction17 Neural networks are powerful feature extractors that have become the standard approach for a myriad18 of tasks . New architectures are continuously introduced and set records on benchmark datasets19 ( e.g . [ 24 , 15 , 44 ] ) . These networks differ in layer composition , depth/width or use specific concepts20 such as residual connections [ 15 ] or self-attention [ 34 ] . With growing capacity , their performance on21 large datasets tends to increase [ 44 ] . However , model complexity is also associated with overfitting,22 especially for small datasets where fine details of the training data are easily memorized [ 52 , 2 , 53 ] .23 Rather than defining another , possibly more complex architecture , we analyze what often remains24 unconsidered : the output layer . In image classification , networks usually end with a fully-connected25 ( fc ) layer that combines extracted features for the final output [ 43 , 15 , 17 , 44 ] . As we will show , this26 layer is prone to overfitting since high dependencies on individual , possibly memorized features can27 arise . The same neurons are subsequently not able to generalize across examples . We call these two28 related phenomena neuron dependency and expressivity and illustrate a simplified example in Fig . 2.29 Both problems can be improved by simple but effective changes to the output layer that require only30 few lines of code and achieve better generalization ( i.e. , better results on the test set [ 25 ] , see e.g.31 Fig . 1 ) . Those changes rely on four principles : activation scaling , fixed randomization , sparsity and32 in-layer ensembling ( see Fig . 5 ) . This work analyzes all layers in terms of their capability to reduce33 dependencies and/or increase expressivity . Then , the connection to network performance is shown34 through a comprehensive empirical study across datasets , architectures and application settings in35 image classification and segmentation . Furthermore , we investigate how stronger regularization36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . can be induced by applying the identified principles to other parts of a network while reducing the37 computational footprint . In contrast to common practice , we find neither training of output layers38 to be necessary , nor that output layers are crucial components of deep networks . In summary , our39 contributions are:40 • Introducing neuron dependency and expressivity as two factors contributing to overfitting41 and proposing ways to measure these factors42 • Showing improved regularization of 5 different output layer designs up to 10 % in absolute43 accuracy compared to standard fc layers and other common regularizers44 • Empirical results showing that the proposed layers have improved dependency and expressiv-45 ity , computational efficiency , wide applicability to both small and large datasets , extensibility46 to other parts of the network , and robustness in the choice of hyperparameters47 2 Related Work48 Regularization in deep learning is approached in various ways . Widely used methods are , e.g.,49 normalization [ 19 , 3 ] , weight decay [ 32 ] , data and adversarial augmentation [ 40 , 1 ] , early stopping [ 7 ] ,50 boosting [ 38 ] , multitask learning [ 6 ] , dropout [ 42 ] , dropconnect [ 50 ] , and Gaussian noise layers [ 10 ] .51 To the best of our knowledge , this is the first work that evaluates regularization with respect to52 different output layer designs . Similar to dropout/dropconnect , output layers can be categorized as53 affecting the architecture according to the regularization taxonomy described in [ 25 ] . Unlike other54 regularizers , our methods are either hyperparameter-free or robust to their choice and can be applied55 to any deep net , including pre-trained ones that are less affected by overfitting ( see Sect . 5.4 and 5.7 ) .56 Related to fixed randomization are the output layers used in [ 16 , 39 , 14 ] , which show comparable57 performance to trained layers . One can also preallocate output layer weights with a defined struc-58 ture [ 31 , 16 ] . Besides output layers , weight fixing is for example applied to the first layer in the59 Extreme Learning Machine [ 18 ] , or to different weight dimensions in [ 36 ] . In contrast , we omit60 hand-crafted weights , show improved regularization and relate to neuron dependencies . Further , we61 show that fixing or scaling the last conv block next to the output layer has a strong regularizing effect.62 Sparsity is common in deep learning , e.g . the ReLU activation [ 13 ] or a L1 penalty term in the loss63 function [ 46 ] . Sparsity has also been applied to the channels of Convolutional Neural Networks64 ( CNNs ) [ 8 , 29 ] . Others induce sparsity by pruning connections before training under the lottery65 ticket hypothesis [ 11 , 30 ] , with the goal of reducing the number of parameters while not sacrificing66 performance [ 27 , 45 , 51 ] . Different to them , we show that ( extreme ) sparsity is not merely useful to67 improve computational efficiency , but to improve performance when applied to the output layer.68 The Network in Network ( NIN ) [ 28 ] and All-CNN [ 41 ] both use global average pooling ( GAP ) 69 followed by softmax , which replaces the fc output layer with an identify transform to simplify the70 network . This is further analyzed in [ 33 ] . We show its connection to neuron dependency/expressivity71 and achieve comparable or better performance on various datasets . Further , we observe that previous72 works do not leverage the full capacity of the last layer in modern networks , which enables the73 construction of computationally efficient in-layer ensembles that further boost performance in small74 and large datasets.75 3 Neuron dependency and expressivity76 3.1 Setting and notation77 We consider neural networks consisting of an encoder fenc : X → a followed by an output layer78 fout : a → ŷ . In this paper , the encoder is a CNN , taking as input an image X ∈ RC×H×W with79 C , W and H being input channels , width and height , respectively ; and transforming it to a feature80 vector a ∈ R1×N . Commonly in CNNs , 2D representations resulting from the final conv layer81 are aggregated by GAP [ 28 ] where N corresponds to the number of pooled conv channels . The82 output layer transforms the embedding to output ŷ ∈ RK holding the probabilities of K classes.83 The output layer is parameterized by a weight matrix W ∈ RN×K , and is commonly initialized as84 W random ∼ U ( − √ 1/N , √ 1/N ) [ 26 ] . Both ŷ and target y are used to compute the cross-entropy85 loss ` = − ∑K i yi log ( ŷi ) . We use the terms features/channels/nodes or neurons interchangeably86 meaning activations a . When required , we refer to individual instances with a superscript , e.g.87 ( X ( i ) , y ( i ) ) ∈ D , with D being a dataset . Corresponding subsets are denoted as Dtrain and Dtest.88 3.2 Concepts89 During training , CNNs learn a set of visual patterns that are combined for a classification decision.90 However , if patterns remain undetected , e.g . due to noise in the image or inherent but unseen variance91 in the data , their activation values can become small and thus reduce the output values for the target92 class . When a network is overfitting , it learns malignant image-specific patterns by heart [ 52 ] . Such a93 network may depend on the activation of individual nodes , which in turn fail to generalize to patterns94 that are salient to a class . We call these two related phenomena neuron dependency and expressivity.95 96 Neuron dependency : How much does a model depend on a single neuron ? In a network97 with high neuron dependencies , output scores and thus performance drop significantly when certain98 neurons remain inactive . In contrast , a network with low neuron dependencies distributes activations99 across many neurons , so that a single inactive node does not have much influence on the classification.100 Neuron expressivity : How much class-specific variance does a neuron cover ? Neurons with low101 expressivity focus on unimportant details that do not characterize the properties of a class . In contrast,102 a neuron with high expressivity generalizes by activating to various patterns pertinent to a given class.103 An example of neuron dependency/expressivity for a simplified fc output layer is illustrated in Fig . 2.104 3.3 Measuring dependency and expressivity105 We introduce two ways of measuring dependency/expressivity : instance-based and class-based . The106 former is used to determine the dependency on the most important node for the predicted class given an107 instance . Importance scores for node n and output class k̂ are computed with Gradient Activation [ 4 ] ,108 a global attribution method where we leverage the partial derivative of the softmax values:109 an ∂ŷk̂ ∂an . ( 1 ) Instance-based dependency is then measured as avg . reduction in output probabilities when ablating110 the most important feature w.r.t . the output class of any instance . This is illustrated for various111 training set sizes of CIFAR-100 [ 23 ] in Fig . 3 ( left ) . With less data , fc output layers tend to depend112 more on single nodes . This is in contrast to class-based measures , which enable quantifying both113 dependency/expressivity and use various features jointly . Importances are determined for each class114 k and over all test instances:115 |Dtest|∑ i=1 a ( i ) n ∂ŷ ( i ) k ∂a ( i ) n . ( 2 ) Class-based dependency is then measured as drop in accuracy when ablating a given number of most116 important neurons per class . Measuring expressivity reverses this - the most important neurons per117 class are retained , all others are ablated . This is illustrated for both dependency/expressivity in Fig . 4.118 We see that standard ( i.e . trained ) fc output layers tend to depend on single channels to achieve high119 performance , but these very channels hold only limited class information.120 4 Output Layer Types121 We describe several simple output layer variants that require minimal changes to standard networks,122 decrease neuron dependency and/or increase neuron expressivity . All types are illustrated in Fig . 5.123 4.1 Standard output layers124 The ubiquitous approach to compute class scores is to learn the parameters of a weight matrix125 W trained , s.t . ŷ = σSM ( aW trained ) with σSM ( · ) being softmax . Each feature is considered in the126 computation of each class score . As shown in Sect . 3 , trained fc output layers can lead to high neuron127 dependencies , where the deletion of a single neuron might cause significant loss in performance , and128 low neuron expressivity , where multiple features are required for adequate predictions.129 4.2 Scaled output layers130 The reduction of an activation , e.g . due to changing light conditions , has a large influence on the131 output scores . This is simulated in Fig . 3 ( center ) by multiplying features during training with a132 scalar α > 0 , so that ŷ = σSM ( αaW scaled ) . Note that the variances of the output logit distributions133 increase with α , resulting in larger differences ( or smaller entropies ) after softmax normalization.134 This results in greater dependencies of the model on individual neurons , as shown in Fig . 3 ( right ) .135 However , if α is chosen small , the activations of individual neurons become insufficient for class136 discrimination with high confidence . The model is therefore forced to learn multiple class-specific137 features for each instance , which increases the expressivity of the neurons and also reduces their138 dependencies to some extent , as shown in Fig . 4 . If not specified otherwise , we use α = 0.1.139 4.3 Random fixed layers140 This setting uses W random during training/inference , and its classification performance was first141 analyzed in [ 16 ] . The encoder learns to extract patterns that adjust to predetermined weights . Unlike142 activation scaling , the parameters are bounded and fixed to a small value range . For any class , the143 chosen uniform initialization is expected to assign similar weight values to multiple neurons , making144 them learn similar features . We suppose that the enforced similarity reduces dependency shown in145 Fig . 3 and 4 ( both left ) , while small initialization values increase expressivity as in Sect . 4.2 , shown146 in Fig . 4 ( right ) .147 4.4 Sparse fixed layers148 In sparse output layers , class nodes use predetermined sets of channels , some of which might149 be shared across classes . First , a set of cutting indices Ik is randomly sampled for each class150 k , where sparsity is determined by the proportion q of class-specific connections to cut , so that151 |Ik| = bqNcwith 0 < q < 1 . Then , starting from a fixed random initialization as in Sect . 4.3 , weights152 connecting to a given class are ablated so that : W sparsei , k = 0 ∀ i ∈ Ik . Hyperparameter q trades153 off dependency/expressivity . Larger values induce more sparsity , leading to greater dependencies to154 the remaining nodes , but forcing them to activate across instances , making them expressive . We set155 q = 0.9 in the experiments to show that high sparsity benefits generalization.156 4.5 1-to-1 correspondence layers157 The most extreme type of sparsity in an output layer is one with a single connection between a158 feature and a class . If these connections correspond to an identity transform , the activations of the159 penultimate layer are equivalent to the class logits - in practice , the output layer can hence be omitted.160 This was analyzed in [ 33 , 28 ] and showed comparable results to a standard output layer . Formally , we161 have ŷ = σSM ( aW 1to1 ) with a ∈ R1×K and W 1to1 ∈ RK×K , where W 1to1 = diag ( 1 , 1 , . . . , 1 ) .162 In this layer , both the model ’ s dependency on individual neurons as well as each neuron ’ s expressivity163 are maximal . If a single neuron is ablated , the output logits for the class this neuron is connected to is164 reduced to zero . However , individual neurons learn to cover the whole variance of a given class in the165 training set , which is one conjecture for their performance . Note that as mentioned in [ 33 ] , we have166 the constraint N = K , which might be restrictive for small networks and large numbers of classes.167 4.6 Ensemble layers168 Is there a way to optimize for both low neuron dependency and high expressivity ? Of the approaches169 discussed , 1-to-1 correspondence layers have the highest expressivity . Starting from this layer , a170 simple approach to reduce neuron dependency is to use the capacity of the penultimate layer and171 create multiple heads h = 1 . . . H with N = KH , each head being a 1-to-1 correspondence layer.172 Each head ’ s output is effectively computed as ŷh = σSM ( αah ) with ah being the activation part of173 head h. As in Sect 4.2 , we introduce a scalar α , which controls the magnitude of feature activations.174 For consistency , we denote this approach as W heads . The loss is computed for each head and175 averaged : 1H ∑H h=1 ` ( ŷ h , y ) . Similarly , logits are averaged over heads for inference . Due to the176 induced redundancy , the performance only drops considerably after removing class-related neurons177 from all heads . In our experiments , we set H to its maximum given any setting ( architecture/dataset ) .178 Note that hyperparameter α in ensemble layers is the only one which is tuned to individual settings.179 5 Experiments180 We aim to show that the presented output layers from Sect . 4 outperform standard output layers and181 common regularization methods in various settings . Details about training , compute resources , code,182 datasets as well as additional experiments on dependency/expressivity are included in the appendix.183 5.1 Small-scale and fine-grained classification184 All layer types are first applied to small-scale and/or fine-grained classification , both of which are185 challenging and require regularization . Datasets include STL-10 ( 500 img/class ) [ 9 ] , CUB-200 ( ∼30186 img/class ) [ 49 ] , Cars-196 ( ∼40 img/class ) [ 22 ] and Food-101 ( 750 img/class ) [ 5 ] . Table 1 shows187 results for the two popular backbones ResNet-50 [ 15 ] and DenseNet-169 [ 17 ] , exchanging the output188 layer accordingly . In 53/56 settings , we see improved results over standard layers . Of these , 48189 and 36 are significant with p < 0.1 and p < 0.001 , respectively . Although there is no clear best190 method , it is worth noting that sparse and ensemble layers as enhancements of both random and191 1-to-1 layers are significantly better ( p < 0.001 ) in 7/8 settings , respectively . As expected , smaller192 performance differences are exhibited in Food-101 , which is a considerably larger dataset , thus193 requiring less regularization . Among the worse settings , only 1 is significant ( p < 0.1 ) for Food-101194 since it involves strong regularization to multiple layers . These regularizers are discussed seperately195 in Sect . 5.5.196 5.2 Large-scale classification and transfer learning197 Machine learning models are subject to the bias-variance tradeoff [ 12 ] , in which induced biases of198 the presented output layers might be too strong to fit the training data . We therefore want to shed199 light on how these layers behave in large-scale and transfer learning settings , where overfitting is200 less problematic . Datasets include CIFAR-100 ( C100 , 5000 img/class ) [ 23 ] , ImageNet ( IN , ∼1200201 img/class ) from ILSVRC2012 [ 37 ] reported on the validation set , as well as CUB/Cars/Food with202 models being pre-trained on IN . Table 2 shows the results for the ResNet-50 backbone . In C100,203 we see consistent improvements with at least p < 0.1 . On the other datasets , results are mostly204 comparable corroborating widespread applicability . It is worth mentioning thatW 1to1 andW ensemble205 perform consistently better , and W ensemble significantly ( p < 0.1 ) in multiple cases . With growing206 dataset sizes , both layers expose a strong constraint on the class neurons to fit an increasing number207 of examples . We believe this to be responsible for progressively separating the signal from the noise,208 leading to better generalization . On the other hand , neuron dependency is reduced in larger datasets209 ( see Fig . 3 left ) diminishing the effect of W scale and W random . Moreover , W random and W sparse210 can be affected by predetermined feature-class weights that do not have to match features learned211 during pre-training , which might require larger adjustments to the weights of the last conv layer.212 5.3 Use Case : Medical imaging213 Output layer design is critical in fields such as medical imaging , which presents special challenges to214 regularization : Datasets tend to be small , imbalanced , abnormalities might fill only a few pixels of215 the image , and appearances between classes are often similar . In addition , transfer learning with IN216 weights is either inaccessible due to architectural differences ( e.g . image segmentation , 3D Magnetic217 Resonance Imaging ) or less effective due to large domain differences . This is first illustrated on the218 APTOS Kaggle challenge dataset ( 3662 images , 193-1805 img/class ) [ 20 ] , with the goal of detecting219 diabetic retinopathy severities in retinal fundus images . We use the public training dataset to train a220 multi-class classifier and perform 5-fold cross-validation . Table 3 shows the results . We consistently221 get better performance with regularization and reduce the gap to a pre-trained network . Furthermore,222 an additional experiment in the appendix indicates that the standard output layer is biased towards the223 prevalent class , which is inherently remedied through randomization.224 We provide further evidence that the proposed layer designs positively affect tasks other than225 classification . We learn a U-Net [ 35 ] for binary semantic slice-based segmentation of Computed226 Tomography scans of livers comparing a standard 1x1 conv output layer with 64 parameters to both a227 fixed randomized and an ensemble layer . Due to the limited number of parameters , we omit W scale228 andW sparse here . Different to classification , the output of a U-Net itself can be interpreted as a 1-to-1229 layer . One can still build an ensemble by treating each output channel as a head . Both W random and230 W ensemble ( H = 10 ) are then applied to the CHAOS [ 21 ] and SLIVER [ 47 ] datasets . For CHAOS,231 Table 3 : Quadratic weighted kappa and accuracy ( with significance ) for different output layers in ResNet-50 for the APTOS dataset . + denotes fine-tuning from IN . See Table 1 for other symbols . Kappa Acc . W trained 0.816 77.44 we train on 15 randomly selected patients ( 2155 slices ) and evaluate on the remaining 5 ( 719 slices ) .232 We then test for generalization by training on all 20 patients from CHAOS and evaluating on the233 external SLIVER dataset consisting of 20 patients ( 4159 slices ) . Table 4 shows improved results in234 both settings.235 5.4 Other regularization techniques236 Table 5 compares our most competitive methods to other popular regularizers when applied to a237 standard output layer . Nodes/connections in dropout/dropconnect are both removed with p = 0.7,238 and the noise layer adds a Gaussian with µ = 0 and σ = 0.1 before applying the fc layer . In all239 cases , our variants perform better . Whereas noise does not benefit training here , dropout/-connect is240 supporting regularization . However , both of the latter methods come with two main disadvantages.241 First , they add complexity by changing states in each iteration and having different behavior during242 training and inference . Second , hyperparameter tuning is necessary , while our layers are either243 hyperparameter-free or stable to them . See the ablation study in Sect . 5.7 for evidence.244 5.5 Beyond output layers - block scaling and randomization245 Activation scaling and randomization are techniques applicable to any layer and increase regulariza-246 tion further . This is demonstrated for ResNet-50 and DenseNet-169 in Table 1 . Both architectures247 consist of multiple blocks , each holding groups of conv layer , batch normalization ( BN ) and activa-248 tion function . For W random block , all layers of the last block and the output layer are kept in their249 initialized state during training . Similarly , in W scaled block , activations of all layer groups in the last250 block are scaled during training . In ResNet-50 , block scaling outperforms output layer scaling in 3/4251 datasets by up to 3 % points , and block randomization increases performance in 3/4 datasets by up to252 6 % points compared to output layer randomization . In DenseNet-169 , block scaling outperforms out-253 put layer scaling in 2/4 datasets by up to 2 % points , and block randomization increases performance254 in 2/4 datasets by up to 4 % points compared to output layer randomization . Only in Food-101 and255 DenseNet , block randomization performs significantly worse than baseline because regularization is256 too strong leading to underfitting ( train loss = 0.47 compared to 0.02 in W random ) .257 5.6 Computational efficiency258 The design of the head of deep CNNs has a great impact on computational efficiency . Standard output259 layers alone can contain a large amount of parameters , as CNNs typically hold more channels as260 they get deeper and the number of classes can become large . In ImageNet and a ResNet-50 , for261 example , the output layer alone generates over 2 million parameters , which are saved in W 1to1 and262 W ensemble . This problem compounds when using multiple fc layers . In a VGG-16 , for instance , 3 fc263 layers are employed after the last conv layer . As Table 6 shows , omitting all fc layers saves up to264 90 % in parameters , a considerable amount of memory , and time for a forward/backward pass while265 increasing accuracy . If a ResNet-50 is used , randomization of the last conv block next to the output266 layer yields savings of about 65 % in trainable parameters while increasing accuracy by 10 % points.267 5.7 Ablation study268 Note that W random and W 1to1 are hyperparameter-free compared to other regularizers such as269 dropout/-connect , thus saving the cost of tuning them . Although other layer variants possess hyperpa-270 rameters , we show in Fig . 6 for the ResNet backbone that they are stable ( no large jumps in vicinity ) 271 and consistent ( tend to monotonicity w.r.t . performance ) . In W sparse , the maximum accuracy for272 both datasets is at q = 0.99 ( 20 nodes per class ) and drops only slightly for q = 0.995 . Similarly,273 downscaling in W scale improves performance at a small cost if the optimum is not hit . Also , more274 heads in W ensemble tend to increase performance . What is the result of adding more heads than275 given by the constraint N = KH ? If N < KH , which is the case for CUB-200 and H > 10 , we276 add an additional 1x1 conv layer , BN and ReLU with KH nodes to adjust for the missing channels.277 Although this leads to a considerable increase in parameters ( NKH for the conv layer ) , it helps with278 generalization , contradicting the common belief that overparameterization leads to overfitting [ 48 ] .279 In contrast , dropout is not stable or consistent . With a dropout rate of 0.9 , the network fails to train280 in both datasets . Furthermore , the optimum for CUB lies at 0.8 , the same hyperparameter choice in281 STL would result in worse performance than baseline.282 6 Conclusion283 In this work , we introduced neuron dependency and expressivity as factors contributing to overfitting.284 Then , different output layers were defined to optimize both and showed improved regularization in285 various settings while being efficient and robust to hyperparameters . Although these layers are simple,286 they have high practical relevance due to the importance of regularization and the ubiquity of output287 layers in deep nets . In addition to their application , they may also be useful as primitives in future288 ( automatically created ) architectures . Although improving regularization , we note that optimizing289 for neuron dependencies/expressivity does not solve overfitting . For example , an unknown or noisy290 instance may result in reduced activations in the majority of nodes in the penultimate layer . Finally,291 we speculate that overfitting may not just be a function of the number of parameters in the encoder.292 Instead , it might be more important how the extracted features are combined in the output layer.293 References294 [ 1 ] Antreas Antoniou , Amos Storkey , and Harrison Edwards . Data augmentation generative295 adversarial networks . arXiv preprint arXiv:1711.04340 , 2017.296 [ 2 ] Devansh Arpit , Stanisław Jastrzębski , Nicolas Ballas , David Krueger , Emmanuel Bengio,297 Maxinder S Kanwal , Tegan Maharaj , Asja Fischer , Aaron Courville , Yoshua Bengio , et al.298 A closer look at memorization in deep networks . In International Conference on Machine299 Learning , pages 233–242 . PMLR , 2017.300 [ 3 ] Jimmy Lei Ba , Jamie Ryan Kiros , and Geoffrey E Hinton . Layer normalization . arXiv preprint301 arXiv:1607.06450 , 2016.302 [ 4 ] David Baehrens , Timon Schroeter , Stefan Harmeling , Motoaki Kawanabe , Katja Hansen , and303 Klaus-Robert Müller . How to explain individual classification decisions . The Journal of304 Machine Learning Research , 11:1803–1831 , 2010.305 [ 5 ] Lukas Bossard , Matthieu Guillaumin , and Luc Van Gool . Food-101 – mining discriminative306 components with random forests . In European Conference on Computer Vision , 2014.307 [ 6 ] Rich Caruana . Multitask learning . Machine learning , 28 ( 1 ) :41–75 , 1997.308 [ 7 ] Rich Caruana , Steve Lawrence , and Lee Giles . Overfitting in neural nets : Backpropagation,309 conjugate gradient , and early stopping . Advances in neural information processing systems,310 pages 402–408 , 2001.311 [ 8 ] Soravit Changpinyo , Mark Sandler , and Andrey Zhmoginov . The power of sparsity in convolu-312 tional neural networks . arXiv preprint arXiv:1702.06257 , 2017.313 [ 9 ] Adam Coates , Andrew Ng , and Honglak Lee . An analysis of single-layer networks in unsuper-314 vised feature learning . In Proceedings of the fourteenth international conference on artificial315 intelligence and statistics , pages 215–223 . JMLR Workshop and Conference Proceedings , 2011.316 [ 10 ] Terrance DeVries and Graham W Taylor . Dataset augmentation in feature space . arXiv preprint317 arXiv:1702.05538 , 2017.318 [ 11 ] Jonathan Frankle and Michael Carbin . The lottery ticket hypothesis : Finding sparse , trainable319 neural networks . arXiv preprint arXiv:1803.03635 , 2018.320 [ 12 ] Stuart Geman , Elie Bienenstock , and René Doursat . Neural networks and the bias/variance321 dilemma . Neural computation , 4 ( 1 ) :1–58 , 1992.322 [ 13 ] Xavier Glorot , Antoine Bordes , and Yoshua Bengio . Deep sparse rectifier neural networks . In323 Proceedings of the fourteenth international conference on artificial intelligence and statistics,324 pages 315–323 . JMLR Workshop and Conference Proceedings , 2011.325 [ 14 ] Moritz Hardt and Tengyu Ma . Identity matters in deep learning . arXiv preprint326 arXiv:1611.04231 , 2016.327 [ 15 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image328 recognition . In Proceedings of the IEEE conference on computer vision and pattern recognition,329 pages 770–778 , 2016.330 [ 16 ] Elad Hoffer , Itay Hubara , and Daniel Soudry . Fix your classifier : the marginal value of training331 the last weight layer . arXiv preprint arXiv:1801.04540 , 2018.332 [ 17 ] Gao Huang , Zhuang Liu , Laurens Van Der Maaten , and Kilian Q Weinberger . Densely connected333 convolutional networks . In Proceedings of the IEEE conference on computer vision and pattern334 recognition , pages 4700–4708 , 2017.335 [ 18 ] Guang-Bin Huang , Qin-Yu Zhu , and Chee-Kheong Siew . Extreme learning machine : theory336 and applications . Neurocomputing , 70 ( 1-3 ) :489–501 , 2006.337 [ 19 ] Sergey Ioffe and Christian Szegedy . Batch normalization : Accelerating deep network training338 by reducing internal covariate shift . In International conference on machine learning , pages339 448–456 . PMLR , 2015.340 [ 20 ] Kaggle . Aptos 2019 blindness detection , 2019 . URL https : //www.kaggle.com/c/341 aptos2019-blindness-detection.342 [ 21 ] A Emre Kavur , N Sinem Gezer , Mustafa Barış , Sinem Aslan , Pierre-Henri Conze , Vladimir343 Groza , Duc Duy Pham , Soumick Chatterjee , Philipp Ernst , Savaş Özkan , et al . Chaos challenge-344 combined ( ct-mr ) healthy abdominal organ segmentation . Medical Image Analysis , 69:101950,345 2021.346 [ 22 ] Jonathan Krause , Michael Stark , Jia Deng , and Li Fei-Fei . 3d object representations for347 fine-grained categorization . In 4th International IEEE Workshop on 3D Representation and348 Recognition ( 3dRR-13 ) , Sydney , Australia , 2013.349 [ 23 ] Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images.350 2009.351 [ 24 ] Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . Imagenet classification with deep352 convolutional neural networks . Advances in neural information processing systems , 25:1097–353 1105 , 2012.354 [ 25 ] Jan Kukačka , Vladimir Golkov , and Daniel Cremers . Regularization for deep learning : A355 taxonomy . arXiv preprint arXiv:1710.10686 , 2017.356 [ 26 ] Yann A LeCun , Léon Bottou , Genevieve B Orr , and Klaus-Robert Müller . Efficient backprop.357 In Neural networks : Tricks of the trade , pages 9–48 . Springer , 2012.358 [ 27 ] Namhoon Lee , Thalaiyasingam Ajanthan , and Philip HS Torr . Snip : Single-shot network359 pruning based on connection sensitivity . arXiv preprint arXiv:1810.02340 , 2018.360 [ 28 ] Min Lin , Qiang Chen , and Shuicheng Yan . Network in network . arXiv preprint arXiv:1312.4400,361 2013.362 [ 29 ] Baoyuan Liu , Min Wang , Hassan Foroosh , Marshall Tappen , and Marianna Pensky . Sparse363 convolutional neural networks . In Proceedings of the IEEE conference on computer vision and364 pattern recognition , pages 806–814 , 2015.365 [ 30 ] Eran Malach , Gilad Yehudai , Shai Shalev-Schwartz , and Ohad Shamir . Proving the lottery366 ticket hypothesis : Pruning is all you need . In International Conference on Machine Learning,367 pages 6682–6691 . PMLR , 2020.368 [ 31 ] Federico Pernici , Matteo Bruni , Claudio Baecchi , and Alberto Del Bimbo . Fix your features:369 Stationary and maximally discriminative embeddings using regular polytope ( fixed classifier ) 370 networks . arXiv preprint arXiv:1902.10441 , 2019.371 [ 32 ] David C Plaut , Steven J Nowlan , and Geoffrey E Hinton . Experiments on learning by back372 propagation . 1986.373 [ 33 ] Zhongchao Qian . Deep Convolutional Networks without Learning the Classifier Layer . PhD374 thesis , Rochester Institute of Technology , 2020.375 [ 34 ] Prajit Ramachandran , Niki Parmar , Ashish Vaswani , Irwan Bello , Anselm Levskaya , and376 Jonathon Shlens . Stand-alone self-attention in vision models . arXiv preprint arXiv:1906.05909,377 2019.378 [ 35 ] Olaf Ronneberger , Philipp Fischer , and Thomas Brox . U-net : Convolutional networks for379 biomedical image segmentation . In International Conference on Medical image computing and380 computer-assisted intervention , pages 234–241 . Springer , 2015.381 [ 36 ] Amir Rosenfeld and John K Tsotsos . Intriguing properties of randomly weighted networks:382 Generalizing while learning next to nothing . In 2019 16th Conference on Computer and Robot383 Vision ( CRV ) , pages 9–16 . IEEE , 2019.384 [ 37 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng385 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei.386 ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision387 ( IJCV ) , 115 ( 3 ) :211–252 , 2015. doi : 10.1007/s11263-015-0816-y.388 [ 38 ] Holger Schwenk and Yoshua Bengio . Boosting neural networks . Neural computation , 12 ( 8 ) :389 1869–1887 , 2000.390 [ 39 ] Gabi Shalev , Gal-Lev Shalev , and Joseph Keshet . Redesigning the classification layer by391 randomizing the class representation vectors . arXiv preprint arXiv:2011.08704 , 2020.392 [ 40 ] Connor Shorten and Taghi M Khoshgoftaar . A survey on image data augmentation for deep393 learning . Journal of Big Data , 6 ( 1 ) :1–48 , 2019.394 [ 41 ] Jost Tobias Springenberg , Alexey Dosovitskiy , Thomas Brox , and Martin Riedmiller . Striving395 for simplicity : The all convolutional net . arXiv preprint arXiv:1412.6806 , 2014.396 [ 42 ] Nitish Srivastava , Geoffrey Hinton , Alex Krizhevsky , Ilya Sutskever , and Ruslan Salakhutdinov.397 Dropout : a simple way to prevent neural networks from overfitting . The journal of machine398 learning research , 15 ( 1 ) :1929–1958 , 2014.399 [ 43 ] Christian Szegedy , Wei Liu , Yangqing Jia , Pierre Sermanet , Scott Reed , Dragomir Anguelov,400 Dumitru Erhan , Vincent Vanhoucke , and Andrew Rabinovich . Going deeper with convolutions.401 In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1–9,402 2015.403 [ 44 ] Mingxing Tan and Quoc Le . Efficientnet : Rethinking model scaling for convolutional neural404 networks . In International Conference on Machine Learning , pages 6105–6114 . PMLR , 2019.405 [ 45 ] Hidenori Tanaka , Daniel Kunin , Daniel LK Yamins , and Surya Ganguli . Pruning neural networks406 without any data by iteratively conserving synaptic flow . arXiv preprint arXiv:2006.05467,407 2020.408 [ 46 ] Robert Tibshirani . Regression shrinkage and selection via the lasso . Journal of the Royal409 Statistical Society : Series B ( Methodological ) , 58 ( 1 ) :267–288 , 1996.410 [ 47 ] Bram Van Ginneken , Tobias Heimann , and Martin Styner . 3d segmentation in the clinic : A411 grand challenge . In MICCAI Workshop on 3D Segmentation in the Clinic : A Grand Challenge,412 volume 1 , pages 7–15 , 2007.413 [ 48 ] N Vapnik Vladimir and V Vapnik . Statistical learning theory . Xu JH and Zhang XG . translation.414 Beijing : Publishing House of Electronics Industry , 2O04 , 1998.415 [ 49 ] C. Wah , S. Branson , P. Welinder , P. Perona , and S. Belongie . The Caltech-UCSD Birds-200-2011416 Dataset . Technical Report CNS-TR-2011-001 , California Institute of Technology , 2011.417 [ 50 ] Li Wan , Matthew Zeiler , Sixin Zhang , Yann Le Cun , and Rob Fergus . Regularization of418 neural networks using dropconnect . In International conference on machine learning , pages419 1058–1066 . PMLR , 2013.420 [ 51 ] Chaoqi Wang , Guodong Zhang , and Roger Grosse . Picking winning tickets before training by421 preserving gradient flow . arXiv preprint arXiv:2002.07376 , 2020.422 [ 52 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding423 deep learning requires rethinking generalization . arXiv preprint arXiv:1611.03530 , 2016.424 [ 53 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Michael C Mozer , and Yoram Singer . Identity425 crisis : Memorization and generalization under extreme overparameterization . arXiv preprint426 arXiv:1902.04698 , 2019.427 Checklist428 1 . For all authors ... 429 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s430 contributions and scope ? [ Yes ] See Sect . 3 , 5 and the appendix.431 ( b ) Did you describe the limitations of your work ? [ Yes ] See Sect . 4.5 ( N = K con-432 straint ) , Sect . 5.2 ( some layers are not optimal for large-scale datasets and fine-tuning ) ,433 Sect . 5.5 ( strong regularization may lead to underfitting ) , and Sect . 6 ( neuron depen-434 dency/expressivity are factors of overfitting , but this does not constitute all factors ) .435 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] Focus436 is on architecture and technical , no particular application affecting society is targeted.437 Note that medical ML applications should be thoroughly evaluated , e.g. , for bias and438 generalizability , before being used in practice.439 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to440 them ? [ Yes ] 441 2 . If you are including theoretical results ... 442 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] Only empirical443 results are included.444 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] Only empirical results445 are included.446 3 . If you ran experiments ... 447 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-448 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] Code449 implementing the shown layer types and dataset descriptions are given in the supple-450 mental material/appendix.451 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they452 were chosen ) ? [ Yes ] We report the corresponding splits if not defined by the datasets453 themselves . Hyperparameters and information about the training are partly provided in454 the main text ( e.g . architecture type , layer hyperparameters ) and are detailed further in455 the appendix.456 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-457 ments multiple times ) ? [ Yes ] We indicate statistical significance for our classification458 results.459 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type460 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] We have not tracked CO2 due to461 missing awareness but some details about the setup are provided in the appendix . We462 plan to track CO2 in subsequent works.463 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 464 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See Sect . 5 . This is465 done after first mentioning the corresponding datasets in the main text.466 ( b ) Did you mention the license of the assets ? [ Yes ] We mention asset licenses in the467 appendix.468 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 469 We provide code as supplemental material.470 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re471 using/curating ? [ N/A ] The datasets in Sect . 5.3 are open source from past challenges472 and contain de-identified data to the best of our knowledge . Further details on data473 collection are provided in the corresponding references.474 ( e ) Did you discuss whether the data you are using/curating contains personally identi-475 fiable information or offensive content ? [ N/A ] All datasets are commonly used for476 benchmarks and/or do not contain obviously offensive content . However , ImageNet477 likely contains images showing persons.478 5 . If you used crowdsourcing or conducted research with human subjects ... 479 ( a ) Did you include the full text of instructions given to participants and screenshots,480 if applicable ? [ N/A ] No crowdsourcing or research with human subjects has been481 conducted.482 ( b ) Did you describe any potential participant risks , with links to Institutional Review483 Board ( IRB ) approvals , if applicable ? [ N/A ] See above.484 ( c ) Did you include the estimated hourly wage paid to participants and the total amount485 spent on participant compensation ? [ N/A ] See above.486

Summary: This experimental work studies various sparse output-layer types for transfer-learning. Authors report interesting gains through usage of certain learned/fixed output layers. Although I found the experiments interesting and believe that they would spark more research on the topic; the story and results needs a little bit more work.