Superseding Model Scaling by Penalizing Dead Units and Points with Separation Constraints

1 INTRODUCTION . The success of Deep Neural Networks ( DNN for short ) is linked to its ability to learn abstract representations from input data in a hierarchical fashion ( LeCun et al . ( 2006 ) ; Ramachandran et al . ( 2017 ) ) . However , the concepts of depth and width in networks are often used as instrumental elements to address different DNN pathologies during the learning process . Examples of these are : vanishing gradient ( Hochreiter ( 1991 ; 2001 ) ) , exploding gradient ( Pascanu et al . ( 2013 ) ) , dead units ( Maas ( 2013 ) ; Douglas & Yu ( 2018 ) ; Guerraoui et al . ( 2017 ) ) , or the degradation problem ( He et al . ( 2015b ) ) . In order to address the former issues , we find many methods and techniques that we can roughly classify in two families : data manipulation and architectural modifications . The most commonly used data manipulation technique is data normalization on the output of the layers , for example using batch normalization ( Ioffe & Szegedy ( 2015 ) ) . Examples of architectural modifications include the use of additional connections , as done in ResNets ( He et al . ( 2015b ) ) or DenseNets ( Huang et al . ( 2016 ) ) ; unit augmentation , as in leaky-ReLU ( Maas ( 2013 ) ) , PReLU ( He et al . ( 2015a ) ) , C-ReLU ( Shang et al . ( 2016 ) ) , or linked neurons ( Riera & Pujol ( 2017 ) ) ; or the increment of layer ’ s width with depth ( Zagoruyko & Komodakis ( 2016 ) ; Szegedy et al . ( 2014 ) ) . This last widely used approach effectively increases the the size of the network requiring larger computational power . This has led to several works ( Hasanpour et al . ( 2018 ) ; Tan & Le ( 2019 ) ) that address it by offering heuristics on how to scale the network . Recent research ( Liu et al . ( 2018 ) ) suggests that this inverted pyramidal architecture is not optimal . Besides of the width scaling strategy , the concept of dead units1 is of particular interest to this work . A dead neuron is defined as the neuron with a constant or zero output for all training data points . This effectively renders this unit ineffective during the learning process . In ( Lu et al. , 2019 ) it is shown that as the depth increases the probability of finding dead neurons also increases , to the point 1In this work we use unit and neuron indistinctly . that the entire network can be dead even at initialization . Additionally , as expected , as the width of a layer increases , the probability of having dead neurons decreases . In this article we characterize another pathology , the dead point . The dead point is a dual concept to the dead neuron . A dead point corresponds to a data point that does not reach the output of the network . This is , the activation is zero for all the units in a given layer . As a result , this data point will have no influence in the training process . As in the case of dead units , both condition are not recoverable by back-propagation . This virtually reduces the size of the training set . In order to solve the former issues , we propose and present a geometrical optimization constraint that is added to the loss function . As a result of adding this constraint we ensure that all neurons are active and no points are dead . The rationale behind the geometrical constraint is to control that all units/neurons and data points are alive by constraining pre-activation values in such a way that the hyperplanes associated to the non-linearity of each ReLU neuron separates , at least , one data point from the rest of the dataset . This has the additional effect of enabling the learning process to propagate all information through all the network avoiding the instrumental need of using additional connections or inverted pyramidal architectures . We test our proposal in a series of controlled experiments to showcase the effect of applying the proposed Separation Constraints . We find that we can arbitrarily increase the network depth using the same constant width when compared to standard feedforward network and Batch Normalization Ioffe & Szegedy ( 2015 ) networks even when the width of the layer is extremely small ( 4 or 8 neurons ) . We additionally provide evidences that using this same approach we can initialize network parameters to zero and still achieve reasonable performance . These promising results gives insight on learning dynamics and suggest potential lines for future checks and research . The article is organized as follows : Section 2 introduce the characterization of dead neurons and dead points , Section 3 introduces the two geometrical constraints that ensure that neurons and data points are alive , Section 4 describes the experiments and result , and , finally , Section 5 concludes the paper and suggests future lines of research . 2 CHARACTERIZING DEAD NEURONS AND DEAD POINTS . A standard feed-forward ReLUDNN ( LeCun et al. , 2015 ) F can be formally written as a multivalued real function , F ( x ) , that is created by composing a collection of D vector layer functions ` : Rnk−1 → Rnk . Layer k is defined as the sum of a collection of scalar functions ( units ) : ` k ( x ) = nk∑ j=1 ukj ( x ) êj ( 1 ) that affinely depend on a weight vector wkj ∈ Rnk and a bias parameter bkj ∈ R. When using rectified linear units this value is truncated on negative values : ukj ( x ) = max ( 0 , w k j · x+ bkj ) . ( 2 ) Considering the hyperplane defined by wkj · x + bkj = 0 , each unit defines a partition of the space Rnk in two sets : the upper part of unit ukj and the lower part of ukj : upper ( ukj ) = { x : wkj · x+ bkj > 0 } lower ( ukj ) = { x : wkj · x+ bkj ≤ 0 } ( 3 ) We define the affine component of a layer function ` k as the intersection of the upper parts of its units , and its zero set as the intersection of the lower parts , as follows : A ( ` k ) = nk⋂ j=1 upper ( ukj ) , Z ( ` k ) = nk⋂ j=1 lower ( ukj ) ( 4 ) Remark 2.1 ( Dead unit ) . In a given ReLU -DNN F : Rn → Rk , we say that the j-th unit of layer ` k , ukj is dead with respect to a data set X ⊂ Rnk if and only if X ⊂ lower ( ukj ) . ( 5 ) Observe that if a unit is dead , the output of the unit will be zero for the entire dataset , rendering the unit useless . Moreover , since the gradient is zero as well , it remains in this state for the rest of the training ( See Lu et al . ( 2019 ) ; Shin & Karniadakis ( 2019 ) ) . This effectively reduce the network learning capacity . Remark 2.2 ( Dead point with respect to a layer ) . Given ReLU -DNN F : Rn → Rk , we say that a point x ∈ X is dead with regards to layer ` k if x ∈ Z ( ` k ) . ( 6 ) Remark 2.3 ( Dead point ) . In particular , if X ⊂ Rn , we say that point x ∈ X is dead with respect to a network F with depth D if it gets mapped to the zero set of a layer in its transit through the network . This is ( ∃k , 1 < k ≤ D| ` k−1 ◦ . . . ` 1 ( x ) ∈ Z ( ` k ) ) . ( 7 ) Any dead point fulfilling Equation 7 will show zero gradient in the layers previous to ` k. This hinders the learning process by effectively reducing the data set size . Again , this condition is not reversible using standard back-propagation . In a similar fashion to the case with dead units , the probability of finding a dead point increases with network ’ s depth and decreases with layer ’ s width . 3 INTRODUCING SEPARABILITY CONSTRAINTS . In this section we introduce the desiderata for units and points to remain alive . Then , we proceed to formulate the separability contraints that fulfill the desired conditions . Let us introduce the concept of a separating unit with respect to an arbitrary set X . Definition 3.1 ( Separating Unit ) . Given an arbitrary set X ⊂ Rnk , we say that the j-th unit on layer k , ukj , is able to separate through X if the following predicate is satisfied : RX ( u k j ) ≡ ∅ 6= { lower ( ukj ) ∩ X } ⊂ X ( 8 ) Thus , by construction , a separating unit can not be dead , and if RX ( ukj ) is valid , u k j can not degrade set X to zero . In other words , this condition ensures that each unit always separates at least one data point . In terms of points , we can define a separating point as follows : Definition 3.2 ( Separating point ) . Given an arbitrary set X ⊂ Rnk , we say that point x ∈ X is separating a layer function ` k if there exist indices j , l ∈ { 1 , . . . , mk } for which the following predicate is satisfied . Rxk ( j , l ) ≡ x ∈ { upper ( ukj ) ∩ lower ( ukl ) } ( 9 ) Again , by construction , a separating point ensures that each point in the data set has at least one unit in each layer with an activation different to zero and another with an activation equal to zero . 3.1 MODELLING UNIT-BASED SEPARATION CONSTRAINT ( SEP-U ) Unit based separation contraint ( Sep-U ) is designed to model predicate 8 with the goal of avoiding the presence of dead units . Given a unit ukj in layer ` k : Rnk → Rnk+1 from a ReLUDNN F of depth D , predicate 8 can be simply modelled imposing the following constraints , max i=1 , ... , |X| { wkj · xi + bkj } > 0 min i=1 , ... , |X| { wkj · xi + bkj } < 0 ( 10 ) These strict inequalities can not be directly optimized . It is easily to see that these can be rewritten as max i=1 , ... , |X| { wkj · xi + bkj } ≥ 1 min i=1 , ... , |X| { wkj · xi + bkj } ≤ −1 ( 11 ) The use of the former constraints makes most problems unfeasible . Thus , in a similar fashion to soft-margin SVM ( Cortes & Vapnik , 1995 ) , we introduce a set of positive slack variables { ξ+jk , ξ − jk } that account for constraint violations as follows , max i=1 , ... , |X| { wkj · xi + bkj } ≥ 1− ξ+jk , min i=1 , ... , |X| { wkj · xi + bkj } ≤ −1 + ξ−jk , ξ+jk , ξ − jk ≥ 0 , ( 12 ) for k = 1 , . . . , D and nk the number of units of layer k. The intuition behind the introduction of these constraints is as follows : by minimizing ξ+ at least one pre-activation value is forced to be greater ( or equal ) than 1 . Simmetrically , minimizing ξ− promotes at least one pre-activation to be below −1 . This effectively fulfills Predicate 8 and penalize the apparition of dead units . We show how to differentiate the Separation Constraints with regards to the parameters in the Appendix Section A.3 . At a global network scale , we can aggregate all the slack variable in a single optimization objective as follows , gU ( ξ + , ξ− ) = 1 2 D∑ k=1 nk∑ j=1 ( ξ+jk + ξ − jk ) . ( 13 ) 3.2 MODELLING POINT BASED SEPARATION CONSTRAINT ( SEP-P ) The derivation of the Point Based Separation Constraints ( Sep-P ) follows a parallel process to Sep-U . In order to avoid the presence of dead points it suffices to fulfill Predicate 9 . Similarly to the former derivation , we introduce a set of slack variables for each each point on the batch . That is , given xi ∈ X , and uk1 , . . . , ukn unit functions in a layer ` k , we define slack variables ξ−ik , ξ + ik in the context of the following constraints , max j=1 , ... , nk { wkj · xi + bkj } ≥ 1− ξ+ik , min j=1 , ... , nk { wkj · xi + bkj } ≤ −1 + ξ−ik , ξ+jk , ξ − jk ≥ 0 . ( 14 ) Observe that the minimization of the slacks makes that for any data point at least one activation is above 1 and another is below -1 . We can summarize all the point-based slack variables in a single optimization objective as follows , gP ( ξ + , ξ− ) = 1 2 D∑ k=1 |X|∑ i=1 ( ξ+ik + ξ − ik ) . ( 15 )

Summary: In this paper, a problem of training DNNs having dead points and neurons was addressed. The problem was posed as a constraint optimization problem. The constraints were augmented to the loss functions of DNNs as a regularizer. The slack variables of the constrained loss functions were optimized together with parameters of DNNs to train DNNs avoiding dead points and neurons.