Year: 2019

Abstract:
In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense connections, data normalization or model scaling. We accomplish that by alleviating two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is also known as dead neurons. The other is a less studied problem, dead points. Dead points refers to data points that are mapped to zero during the forward pass of the network. As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant values or even zero and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.

heading: 1 INTRODUCTION
text: The success of Deep Neural Networks (DNN for short) is linked to its ability to learn abstract representations from input data in a hierarchical fashion (LeCun et al. (2006); Ramachandran et al. (2017)). However, the concepts of depth and width in networks are often used as instrumental elements to address different DNN pathologies during the learning process. Examples of these are: vanishing gradient (Hochreiter (1991; 2001)), exploding gradient (Pascanu et al. (2013)), dead units (Maas (2013); Douglas & Yu (2018); Guerraoui et al. (2017)), or the degradation problem (He et al. (2015b)).
In order to address the former issues, we find many methods and techniques that we can roughly classify in two families: data manipulation and architectural modifications. The most commonly used data manipulation technique is data normalization on the output of the layers, for example using batch normalization (Ioffe & Szegedy (2015)). Examples of architectural modifications include the use of additional connections, as done in ResNets (He et al. (2015b)) or DenseNets (Huang et al. (2016)); unit augmentation, as in leaky-ReLU (Maas (2013)), PReLU (He et al. (2015a)), C-ReLU (Shang et al. (2016)), or linked neurons (Riera & Pujol (2017)); or the increment of layer’s width with depth (Zagoruyko & Komodakis (2016); Szegedy et al. (2014)). This last widely used approach effectively increases the the size of the network requiring larger computational power. This has led to several works (Hasanpour et al. (2018); Tan & Le (2019)) that address it by offering heuristics on how to scale the network. Recent research (Liu et al. (2018)) suggests that this inverted pyramidal architecture is not optimal.
Besides of the width scaling strategy, the concept of dead units1 is of particular interest to this work. A dead neuron is defined as the neuron with a constant or zero output for all training data points. This effectively renders this unit ineffective during the learning process. In (Lu et al., 2019) it is shown that as the depth increases the probability of finding dead neurons also increases, to the point
1In this work we use unit and neuron indistinctly.
that the entire network can be dead even at initialization. Additionally, as expected, as the width of a layer increases, the probability of having dead neurons decreases.
In this article we characterize another pathology, the dead point. The dead point is a dual concept to the dead neuron. A dead point corresponds to a data point that does not reach the output of the network. This is, the activation is zero for all the units in a given layer. As a result, this data point will have no influence in the training process. As in the case of dead units, both condition are not recoverable by back-propagation. This virtually reduces the size of the training set.
In order to solve the former issues, we propose and present a geometrical optimization constraint that is added to the loss function. As a result of adding this constraint we ensure that all neurons are active and no points are dead. The rationale behind the geometrical constraint is to control that all units/neurons and data points are alive by constraining pre-activation values in such a way that the hyperplanes associated to the non-linearity of each ReLU neuron separates, at least, one data point from the rest of the dataset. This has the additional effect of enabling the learning process to propagate all information through all the network avoiding the instrumental need of using additional connections or inverted pyramidal architectures.
We test our proposal in a series of controlled experiments to showcase the effect of applying the proposed Separation Constraints. We find that we can arbitrarily increase the network depth using the same constant width when compared to standard feedforward network and Batch Normalization Ioffe & Szegedy (2015) networks even when the width of the layer is extremely small (4 or 8 neurons). We additionally provide evidences that using this same approach we can initialize network parameters to zero and still achieve reasonable performance. These promising results gives insight on learning dynamics and suggest potential lines for future checks and research.
The article is organized as follows: Section 2 introduce the characterization of dead neurons and dead points, Section 3 introduces the two geometrical constraints that ensure that neurons and data points are alive, Section 4 describes the experiments and result, and, finally, Section 5 concludes the paper and suggests future lines of research.

heading: 2 CHARACTERIZING DEAD NEURONS AND DEAD POINTS
text: A standard feed-forward ReLUDNN (LeCun et al., 2015) F can be formally written as a multivalued real function, F (x), that is created by composing a collection of D vector layer functions ` : Rnk−1 → Rnk . Layer k is defined as the sum of a collection of scalar functions (units):
`k(x) = nk∑ j=1 ukj (x)êj (1)
that affinely depend on a weight vector wkj ∈ Rnk and a bias parameter bkj ∈ R. When using rectified linear units this value is truncated on negative values:
ukj (x) = max(0,w k j · x+ bkj ). (2)
Considering the hyperplane defined by wkj · x + bkj = 0, each unit defines a partition of the space Rnk in two sets: the upper part of unit ukj and the lower part of ukj :
upper(ukj ) = {x : wkj · x+ bkj > 0} lower(ukj ) = {x : wkj · x+ bkj ≤ 0} (3)
We define the affine component of a layer function `k as the intersection of the upper parts of its units, and its zero set as the intersection of the lower parts, as follows:
A(`k) = nk⋂ j=1 upper(ukj ), Z(`k) = nk⋂ j=1 lower(ukj ) (4)
Remark 2.1 (Dead unit). In a given ReLU -DNN F : Rn → Rk, we say that the j-th unit of layer `k, ukj is dead with respect to a data set X ⊂ Rnk if and only if
X ⊂ lower(ukj ). (5)
Observe that if a unit is dead, the output of the unit will be zero for the entire dataset, rendering the unit useless. Moreover, since the gradient is zero as well, it remains in this state for the rest of the training (See Lu et al. (2019); Shin & Karniadakis (2019)). This effectively reduce the network learning capacity.
Remark 2.2 (Dead point with respect to a layer). Given ReLU -DNN F : Rn → Rk, we say that a point x ∈ X is dead with regards to layer `k if
x ∈ Z(`k). (6)
Remark 2.3 (Dead point). In particular, if X ⊂ Rn, we say that point x ∈ X is dead with respect to a network F with depth D if it gets mapped to the zero set of a layer in its transit through the network. This is
(∃k, 1 < k ≤ D|`k−1 ◦ . . . `1(x) ∈ Z(`k)). (7)
Any dead point fulfilling Equation 7 will show zero gradient in the layers previous to `k. This hinders the learning process by effectively reducing the data set size. Again, this condition is not reversible using standard back-propagation. In a similar fashion to the case with dead units, the probability of finding a dead point increases with network’s depth and decreases with layer’s width.

heading: 3 INTRODUCING SEPARABILITY CONSTRAINTS
text: In this section we introduce the desiderata for units and points to remain alive. Then, we proceed to formulate the separability contraints that fulfill the desired conditions.
Let us introduce the concept of a separating unit with respect to an arbitrary set X. Definition 3.1 (Separating Unit). Given an arbitrary set X ⊂ Rnk , we say that the j-th unit on layer k, ukj , is able to separate through X if the following predicate is satisfied:
RX(u k j ) ≡ ∅ 6= {lower(ukj ) ∩ X} ⊂ X (8)
Thus, by construction, a separating unit can not be dead, and if RX(ukj ) is valid, u k j can not degrade set X to zero. In other words, this condition ensures that each unit always separates at least one data point.
In terms of points, we can define a separating point as follows:
Definition 3.2 (Separating point). Given an arbitrary set X ⊂ Rnk , we say that point x ∈ X is separating a layer function `k if there exist indices j, l ∈ {1, . . . ,mk} for which the following predicate is satisfied.
Rxk(j, l) ≡ x ∈ {upper(ukj ) ∩ lower(ukl )} (9)
Again, by construction, a separating point ensures that each point in the data set has at least one unit in each layer with an activation different to zero and another with an activation equal to zero.
3.1 MODELLING UNIT-BASED SEPARATION CONSTRAINT (SEP-U )
Unit based separation contraint (Sep-U ) is designed to model predicate 8 with the goal of avoiding the presence of dead units.
Given a unit ukj in layer `k : Rnk → Rnk+1 from a ReLUDNN F of depth D, predicate 8 can be simply modelled imposing the following constraints,
max i=1,...,|X|
{wkj · xi + bkj } > 0
min i=1,...,|X|
{wkj · xi + bkj } < 0 (10)
These strict inequalities can not be directly optimized. It is easily to see that these can be rewritten as
max i=1,...,|X|
{wkj · xi + bkj } ≥ 1
min i=1,...,|X|
{wkj · xi + bkj } ≤ −1 (11)
The use of the former constraints makes most problems unfeasible. Thus, in a similar fashion to soft-margin SVM (Cortes & Vapnik, 1995), we introduce a set of positive slack variables {ξ+jk, ξ − jk} that account for constraint violations as follows,
max i=1,...,|X|
{wkj · xi + bkj } ≥ 1− ξ+jk,
min i=1,...,|X|
{wkj · xi + bkj } ≤ −1 + ξ−jk,
ξ+jk, ξ − jk ≥ 0,
(12)
for k = 1, . . . , D and nk the number of units of layer k. The intuition behind the introduction of these constraints is as follows: by minimizing ξ+ at least one pre-activation value is forced to be greater (or equal) than 1. Simmetrically, minimizing ξ− promotes at least one pre-activation to be below −1. This effectively fulfills Predicate 8 and penalize the apparition of dead units. We show how to differentiate the Separation Constraints with regards to the parameters in the Appendix Section A.3.
At a global network scale, we can aggregate all the slack variable in a single optimization objective as follows,
gU (ξ +, ξ−) =
1
2 D∑ k=1 nk∑ j=1 (ξ+jk + ξ − jk). (13)
3.2 MODELLING POINT BASED SEPARATION CONSTRAINT (SEP-P )
The derivation of the Point Based Separation Constraints (Sep-P ) follows a parallel process to Sep-U . In order to avoid the presence of dead points it suffices to fulfill Predicate 9. Similarly to the former derivation, we introduce a set of slack variables for each each point on the batch. That is, given xi ∈ X, and uk1 , . . . , ukn unit functions in a layer `k, we define slack variables ξ−ik, ξ + ik in the context of the following constraints,
max j=1,...,nk
{wkj · xi + bkj } ≥ 1− ξ+ik,
min j=1,...,nk {wkj · xi + bkj } ≤ −1 + ξ−ik, ξ+jk, ξ − jk ≥ 0.
(14)
Observe that the minimization of the slacks makes that for any data point at least one activation is above 1 and another is below -1.
We can summarize all the point-based slack variables in a single optimization objective as follows,
gP (ξ +, ξ−) =
1
2 D∑ k=1 |X|∑ i=1 (ξ+ik + ξ − ik). (15)

heading: 3.3 TRAINING WITH SEPARATING CONSTRAINTS
text: The new optimization objectives can now be added to the original loss objective using a simple scalarization (Boyd & Vandenberghe (2004)) as follows,
argmin θ,ξ+,ξ−
L(T , θ) + λ ( gU (ξ +, ξ−) + gP (ξ +, ξ−) ) , (16)
where λ is a hyper-parameter that introduces a trade-off between the constraint fulfillment and the main loss function.
In terms of memory complexity, the former constraints introduce a very small memory overhead. In particular, Sep-U places a pair of constraints on each of the units of the network, so the complexity with respect to Sep-U scales with the size of the network as 2 ∑D k=1 nk. Alternatively, Sep-P places a pair of constraints in each of the points of the data set or selected subset X. In practice, one can use X as the training batch. Thus, the memory complexity scales with the size of the batch and the number of layers, i.e. 2D|X|. Furthermore, since the resulting gradient of both types of constraints depends only on the input of the layer that is already computed in the forward pass, we only add the cost of storing the slacks. Therefore, the total cost in terms of number of constraint is the addition of the former terms, i.e. 2 ∑D k=1 nk + 2D|X|.

heading: 4 EXPERIMENTS AND RESULTS
text: In this section we explore the application of the proposed constraints in different datasets. For that task we train all methods with different choices of depth and width parameters. The network architecture used is rectangular, i.e. networks with a fixed layer width for all the layers.
Datasets: Due to the large amount of computational resources required for the depth and width grid training, we are forced to chose three controlled datasets in our experimentation: the MOONS dataset (sampling 100 points, 85 for training and 15 for validation), the MNIST dataset as described in LeCun and Cortes. LeCun & Cortes (2010) and the CIFAR-10 dataset described in Krizhevsky (2009).
Experimental setting: We compare the combination of Sep-U and Sep-P (Sep-UP from now on) to feed-fordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in Ioffe & Szegedy (2015) using the same architecture.
For the MOONS dataset, we use depths from 1 to 120 in steps of 10, and width from 1 to 25 in steps of 1 between 1 and 5, and steps of 5 afterwards. In the case of the MNIST dataset, we use depth from 2 to 64 and width from 2 to 8 in steps of 4. Finally, for CIFAR-10we use depths in {2, 10, 25, 30, 40} with a fixed width of 10 due memory constraints.
Training Parameters: All the networks used were optimized using Adam (Kingma & Ba, 2014). More specifically, for the MOONS dataset we used a learning rate of 0.01 for 5000 epochs and a batch size of 85. Meanwhile, for both MNIST and CIFAR-10 we used a learning rate of 0.0001 for 50 epochs and a batch size of 1024. We used convolutional layers with a (3, 3) kernel size for both MNIST and CIFAR-10. We used λMOONS = 10−4 λMNIST = λCIFAR = 10−8 for Sep-UP . Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), fixing the random seed to an arbitrary value of 10.
As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010) for all the methods and datasets.

heading: 4.1 RESULTS
text: Figure 1 shows the results obtained in the MOONS dataset. Our proposal Sep-UP is able to train networks successfully without increasing the width up to 60 layers deep (see Figures 1c and 1f), while ReLU breaks down at only 30 layers (see Figures 1a and 1d) and ReLU + BN suffers from severe accuracy degradation (see Figures 1b and 1b).
Observe that no configuration with lower width than 2-3 is successful in achieving maximum accuracy. We understand that there exists a minimum width required and this is related to the complexity of the problem. When using wider layers, the rest of the width is instrumentally used to enable the training of deeper networks. As previously commented, the larger the layer’s width, the higher the chances of finding active units that do not cause dead points and dead units during initialization. In opposition, Sep-UP succesfully overcomes that constraint.
Notice that though Sep-UP is superior to all its competitors, it starts showing performance degradation after reaching depth 60 with the minimum width of 3. Considering that the number of parameters increases with the depth and width of the network and that we have a finite number of trained epochs, we conjecture that the displayed degradation is strictly due to the lack of convergence of the constrained network.

heading: 1 30 60 90 120

heading: 1 30 60 90 120
text: The separation constraint also proves successful on convolutional networks, as tested in MNIST and CIFAR-10 datasets. Figure 2 shows a similar behaviour to the MOONS dataset (Figure1). ReLU breaks down after a few layers, ReLU + BN delays the degradation of accuracy, while Sep-UP remains functional regardless of the depth. In the case of our experiments with the CIFAR-10 dataset (as presented on Figure 3) all the methods degrade with depth, but Sep-UP is the most robust. In regards to accuracy ReLU performs best closely followed by Sep-UP , while ReLU + BN clearly overfits. The poorer accuracy values shown are due to a limited choice of width, clearly inferior to the minimum required by the dataset, and a potential lack of convergence of the separating constraints.
Additional results in the Appendix further elaborate the contributions of each term Sep-U and Sep-P , independently.

heading: 4.2 RESULTS USING ZERO INITIALIZATION
text: In order to test the invariace to initialization scheme of Sep-UP , we use Zero Initialization. As its name states, in this initialization scheme all weights and biases are set to zero. However, a small variation of the scheme must be introduced in order to break symmetry for the constraints to apply. Since all the units are initialized to the same value (zero), we use Annealed Dropout (Rennie et al., 2014). Additionally, instead of adding ξ+ and ξ− pairs as in Equations 13 and 15, we use a convex combination with a small perturbation ρ. In our experiments, we use a value of ρ = 0.01.
( 1
2 + ρ)ξ+ + (
1 2 − ρ)ξ− (17)
Figure 4 summarizes the results found using zero initalization with our constraint formulation. In comparison to Glorot, we observe that zero initalization requires wider networks. Indeed, at a depth of 60, Zero initialization requires a width of 25 units while the Glorot scheme work with only 2 units. However, if we contrast ReLU + BN to our constraint formulation we find that it is unable to train networks of past depth 50, while Zero initialization achieves 70 layers.
Figure 4 shows the results using zero initialization. Although reported values and behavior are slightly worse compared to Glorot initialization, our results are promising. Notice how at depth of 60 layers, the zero initialization requires a width of 25 units in contrast to 2 required for Glorot. In addition, zero initialization rquires only 70 layers to reach perfect accuracy in contrast to the 100 required in Glorot (see Figures 1c and 1f). Moreover, in comparison to ReLU + BN , zero initialized networks show superior performance and behavior (check Figures 1b and 1e).

heading: 5 CONCLUSIONS
text: Through the Separation Constraints, we have shown that deeper networks can be trained without increasing the width of the layers. Moreover, this increment can be done using very small width values. In this sense we consider that effective training of deeper networks can be achieved by better accommodating the network to the input data. This departs from many proposal that achieve similar effects by modifying the architecture of the network or manipulating the data. We believe that this work shows an alternative research path in the pursuit of effective and efficient learning techniques for deep neural networks. This also opens the possibility of avoiding the use of pyramidal
architectures commonly employed in DNNs, thus removing the computational burden caused by the additional units and reducing the dimensionality of the internal representations.
We additionally show that our proposal enables the use of Zero Initialization. The results are promising, and though further experimentation is still needed, they still manage to surpass Batch Normalization in terms of depth, width, and accuracy. Nevertheless, the modifications used to break the symmetry of constraints and units need further consideration in order to achieve the same performance than random initialization.
Despite this article provides no information about the dynamics of the training process using Separation Constraints, preliminary revision shows interesting properties. In particular, training displays slight instabilities which appear as the units/points are reactivated. Although this does not hinder the training process, further study is needed in order to guarantee a smoother convergence.
The extension of this work to other activation functions is still to be explored. However, we conjecture that in cases where the activation function display a flat region, e.g. ELU, tanh, or even sigmoid, the current proposal can be applied with minor changes.
Finally, while the separation constraints prevent the vanishing gradient effect, the exploding gradient problem still remains. Extending the Separation Constraint with an upper bound on the magnitudes of the pre-activation, similar to -insensitive loss, might address it. It could be also helpful to explore other activation functions whose gradient vanishes with high pre-activations, such as the logistic family.

heading: A APPENDIX

heading: A.1 UNIT BASED AND POINT BASED SEPARATION CONSTRAINT EXPERIMENT
text: In order to justify the decision of combining Sep-U and Sep-P into Sep-UP , we provide Figure 5 showing the same experiment than Figure 1 for Sep-U and Sep-U . We find how although Sep-U is able to train deeper and thinner networks it suffers from inferior accuracy (see Figures 5a and 5c), requiring the use of the layer width increase technique. Contrarily, Sep-P is able to train without increasing the width from 1 to 40 layers deep, but it breaks down afterwards. Provided how complementary are their strengths and weaknesses, it is sensible to combine them, resulting in the superior performance (larger constant width area together with greater accuracy in higher depths) of Sep-UP , see Figures 1c and 1f.

heading: A.2 PROBABILITY OF A POINT TO BE DEAD IN A NEURAL NETWORK
text: We conjecture that difficulty arising when training neural network as the depth increases or the width decreases, is due the incorrect positioning of the hyperplanes of the units after initialization which causes the apparition of dead points (recall Remark 2.3), that ultimately break back-propagation. In this section we offer a formal treatment of this idea.
Theorem A.1 (The probability of a point to be dead increases with depth and decreases with width). Let F : Rn1 → RnD be a feed-forward neural network composed by a collection of D ∈ N layer functions ` : Rnk−1 → Rnk with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of nk ∈ N ReLU unit functions u : Rnk−1 → R+, we denote nk as the width of the layer. Let the network be initialized following a random distribution. Then, given a point x ∈ Rn1 we say that the probability of the point dying (Recall Definition 7) increases with D and decreases with nk. Lemma A.2 (Probability of a point to be dead for a ReLU unit follows Bernoulli). Let be a point x ∈ Rn1 and a ReLU unit u parametrized by a weight vector w ∈ Rn and a bias b ∈ R initialized following a random distribution, then the probability of point x being dead for u follows a Bernoulli distribution of unknown parameter p.
P (x ∈ lower(u)) ∼ Bernoulli(p) (18)
Proof. Since the unit is initialized randomly according to a given distribution (i.e. normal, uniform, Glorot, ...), the probability of its hyper-plane parametrized by w, b facing x is an unknown p according to the distribution used. Therefore, the probability of x belonging to lower(u) (Recall Equation 3) must be p.
As the lower set (Recall Definition 3) pre-activation is truncated, its gradient with respect the parameters will be zero (See Rey-Torres et al. (2019, Section 2)). Therefore, any point belonging to lower(u) will be dead in regards to u.
Lemma A.3 (Probability of a point to be dead for a ReLU layer). Let `k be a layer composed of nk ∈ N ReLU units u initialized following the same random distribution. Let be a point x ∈ Rnk−1 . Then the probability of x being dead for `k is:
P (x ∈ Z(`k)) = P (x ∈ lower(u))nk (19)
Proof. Similarly to the unit case, points belonging to the zero set of a layer (Recall Definition 4) have zero gradient with respect the parameters of the layer due the truncation (See Rey-Torres et al. (2019, Section 2)).
The probability of a point belonging to the zero set of a layer P (x ∈ Z(`k)) is equal to the probability of belonging to all the lowers of the units of the layer P (∀u ∈ `|x ∈ lower(u)). Since unit parameters are sampled independently during initialization and the probability of a point being dead for a unit follows Bernoulli, then the probability of a point belonging to Z(`k) is the product of the probabilities of belonging to the lower of each of the units.
Lemma A.4 (The probability of a point to be dead for a layer diminishes with its width). Let `k be a layer composed of nk ∈ N ReLU units u initialized following a random distribution. Let be a point x ∈ Rnk−1 . Then the probability of x being dead diminishes with nk:
lim nk→∞
P (x ∈ Z(`)) = 0 (20)
Proof. The probability of a point being dead for a layer is the multiplication of the probabilities of being dead of each of its units, recall Lemma A.3. Therefore, as the probabilities are lower than 1 since the upper set of a unit is never empty, in the limit where nk tends to infinity the probability tends to zero.
Note how Lemma A.4 can explain why increasing width allows us to train deeper networks (See Hasanpour et al. (2018); Huang et al. (2016)), it simply improves the chances of finding a hyperplane that allows the points to live. Definition A.1 (Dead point set in a neural network). Let F : Rn1 → RnD be a feed-forward neural network composed by a collection of D ∈ N layer functions ` : Rnk−1 → Rnk with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of nk ∈ N ReLU unit functions u : Rnk−1 → R+, we denote nk as the width of the layer. Then, we define the set of dead points of a neural network following Equation 7 as.
Z(F ) = {x|(∃k|1 < k ≤ D : `k−1 ◦ . . . `1(x) ∈ Z(`k)) : x} (21)
Lemma A.5 (Probability of a point to be dead for a ReLU feed-forward network). Let F : Rn1 → RnD be a feed-forward neural network composed by a collection of D ∈ N layer functions ` : Rnk−1 → Rnk with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of nk ∈ N ReLU unit functions u : Rnk−1 → R+, we denote nk as the width of the layer. Let the network be initialized following a random distribution. Then, given a point x ∈ Rn1 we say that the probability of the point dying (Recall Definition 7) is
P (x ∈ Z(F )) = 1− D∏ i=1 (1− P (x ∈ Z(`i))) (22)
Proof. The probability of not dying in a given layer is equal to 1 minus the probability of dying.
P (x ∈ Z(`)) = 1− P (x /∈ Z(`)) (23)
As the sampling of the parameters during intialization is independent among layers, the probabiliity of not dying in any of the layers of the network is the multiplication of the probabilities of not dying in each of the layers.
P (x /∈ Z(F )) = D∏ i=1 (1− P (x /∈ Z(`i))) (24)
The probability of a point to be dead for a network is equal to the probability of being dead at least in one layer, see 7. This is equivalent to 1 minus the probability of not dying in any of the layers.
P (x ∈ Z(F )) = 1− P (x /∈ Z(F )) (25)
Plugging Equation 24 into Equation 25 results in Equation 22.
Lemma A.6 (The probability of a point to be dead for a layer increases with its depth). Let F : Rn1 → RnD be a feed-forward neural network composed by a collection of D ∈ N layer functions ` : Rnk−1 → Rnk with k ∈ 1, . . . , D, we denote D as the depth of the network. Let the network be initialized following a random distribution. Then, given a point x ∈ Rn1 we say that the probability of the point dying as it traverses the network (Recall Definition 7) increases with D.
lim D→∞
P (x ∈ Z(F )) = 1 (26)
Proof. Since the hyper-planes of the units split the space in two half-spaces, the probability of point not being dead is always lower than 1.
P (x /∈ Z(`)) < 1 (27)
Therefore, the limit of the probability of x not dying in any of the layers when D tends to infinity is zero, recall Equation 24.
lim D→∞ P ( D∏ i=1 (1− P (x ∈ Z(`i))))) = 0 (28)
Which when plugged into Equation 22 leads to Equation 26.

heading: A.3 ON THE OPTIMIZATION OF THE CONSTRAINTS
text: In this section we show how to differentiate the Separation Constraints with regards to the weights. Notice that under our formulation, we are examining the set diameter of the preactivation of a dataset X . That is, given a unit u with vector parameters w ∈ Rn and b ∈ R, its preactivation is given by the function
z = p(x) = w · x+ b (29)
given a finite discrete set X ⊂ Rn ( a compact set of Rn), we can define the extreme values of X with regards to u as
zumin = min x∈X p(x)
zumax = max x∈X
p(x) (30)
on a similar argument, given a layer ` with units u1, u2, . . . , unk with pre-activations denoted by p1, p2, . . . , pnk , we can define the extreme values of layer ` provided with a fixed x ∈ X as
zpmin = min j=1...,nk pj(x)
zpmax = max j=1...,nk
pj(x) (31)
notice that since X is a finite discrete set of Rnk , z?min, z?max exist (where ? stands for u or p). This fact is instrumental in our Equations (10) to (12). Since X is bounded, and we are working in the real number system, we can find numbers ξ?+, ξ ? − such that
z?min ≤ −1 + ξ?− z?max ≥ 1− ξ?+
(32)
notice that the parameters w ∈ Rnk and b are free in the definition of zu, while x is free in the definition of zp. In this sense, for ξp± we can find w p ± ∈ Rnk and b p ± ∈ R such that
ξp+ = max(1− (w+ · x+ b+), 0) ξp− = max(1 + (w− · x+ b−), 0)
(33)
that constraints all possible configurations of parameters in units of `. Naturally, ξ± depend on w and b. This is at the heart of the constraint formulation: choosing w and b so that zmin < 0 and zmax > 0.
Meanwhile, given a fixed selection of w and b, since X ⊂ Rnk is discrete and bounded, we must have that there exist x+ and x− points of X such that
ξu− ≤ max(1− (w · x− + b), 0) ξu+ ≤ max(1 + (w · x+ + b, 0)
(34)
We wish to stress a geometric intuition, beyond calculations. We seek a combination of w and b so that they ’cut’ through X in the case of Sep-U, noticing that the preactivation p defines a signed distance from the separating hyperplane p(x) = 0.
The same argument is used in the field of Support Vector Machines, as cited in our references, particularly Vapnik & Cortes’ Vector Support Networks from 1995 and Boyd & Vandenberghes’ Convex Analysis from 2004.

heading: A.4 ON THE RELATION BETWEEN CONSTRAINTS AND PARAMETER UPDATE
text: Notice that up to Equation 12, we have only stated constraints upon the slack variables ξ± (that involve the dataset and parameters w and b). In Equation 13 we introduce the objective function to optimize the values of ξ±. Namely, the sum of all ξ.
Since we have different indications of the ξ, we must define different objective functions for unit-wise constraints or gU of Equation 13 and point-wise constraints or gP from Equation 15. We presented the construct for unit-based constraints (transit between Equations 12 and 13) and extended the notation for point-wise constraints on Equation 15.
If we wish to differentiate with regards to w and b equations describing the relation between x and ξ unit-wise or point-wise, we must notice that∇wξu± as stated, must be defined by parts:
∇wξu− = { x−, p(x−) > −1 0, otherwise
(35)
∂ξu+ ∂b = { 1, p(x−) > −1 0, otherwise
(36)
while for ξu+:
∇wξu+ = { −x+, p(x+) < 1 0, otherwise
(37)
∂ξu+ ∂b = { −1, p(x+) < 1 0, otherwise
(38)
while under the unit-based formulation x± is fixed by the geometry of X, for the point-wise formulation, we must have
∇w−ξ p − = { x, p(x) > −1 0, otherwise
(39)
∂ξp− ∂b = { 1, p(x) > −1 0, otherwise
(40)
and for ξp+:
∇w+ξ p + = { −x, p(x) < 1 0, otherwise
(41)
∂ξp+ ∂b = { −1, p(x) < 1 0, otherwise
(42)
that is parallel to the loss gradient∇wL.

heading: A.5 EFFECT OF THE SEPARATION ON THE INTERNAL REPRESENTATIONS
text: In this section we explore the effect of the proposed constraints in the activation of the network. For that task we train all methods fixing the architecture at 50 layers of 4 units each. We use the MOONS dataset for easy visualization, in the same configuration than Section 4(sampling 100 points, 85 for training and 15 for validation) .
Experimental setting: We compare the combination of Sep-U , Sep-P and Sep-UP to feedfordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in Ioffe & Szegedy (2015) using the same architecture.
Training Parameters: All the networks used were optimized using Adam (Kingma & Ba, 2014). We used a learning rate of 0.01 for 1000 epochs and a batch size of 85. We used λ = 10−4 when required. As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010). We also test the use of the Separation Constraints alone, without categorical crossentropy, to assess its effect by itself. Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), fixing the random seed to an arbitrary value of 10.
Table 1 summarizes the results. Figures 6, 7, 11, 10 and 8 show the activation plots for each of the methods tested.
In terms of accuracy ReLU reaches a trivial accuracy of 0.51 while ReLU + BN reaches 0.60, both architectures fail to solve the problem, see Table 1. When comparing the results presented in Figures 6 and 7, we find that at the fourth layer ReLU has collapsed the points of the dataset over a line (parts 6b and 6c), while ReLU + BN still manages to warp the dataset. However, both methods at layer 25th have collapsed the dataset into a small number of points (see Figures 7d, 7e for ReLU + BN and Figures 6d and 6e for ReLU ) that is then pushed to the point zero for ReLU (as shown in the feature layer: 6f and 6g), while ReLU + BN collapses to few points (Figures 7f and 7g).
This collapse is congruent with our expectations, displaying the born-dead network condition from Lu et al. (2019). Note how layer after layer its effect is worse, as we predicted in A.2. In addition, since the gradient is only back-propagated through the points lying on the upper sets of units, geometric collapse also stops learning. Notice that the standarization and the affine transformation (dependent on γ and β) from ReLU + BN are not able to prevent from killing the dataset. Thus, training thin and deep neural networks is difficult with ReLU or ReLU + BN .
On the extent of our experimentation ReLU networks featuring separation constraints are able to solve the problem with higher degrees of accuracy than ReLU and ReLU + BN (see the figures in Table 1). Moreover, Figures 8h, 11h and 10h), show that the separation function behaves intuitively in the sense of Hauser & Ray (2017).
The internal representations not only are non-trivial (as in ReLU + BN ) but also preserve geometrical structures like shape and connectivity, as shown in Figures 11d, 10d or 8e. Indeed, Figures 11c, 8c and 10c showcase how at the 4th layers a much a solution is already found. This proves that the gradient of the main loss is back-propagated to the input, unlike ReLU and ReLU + BN (Figures 6b, 6b, 7b, 7c).
The constraints enforce richer representations when used without main loss (cross-entropy),
0.2 0.0 0.2 0.4 0.6 0.2
0.1
0.0
0.1
0.2
0.3
(a) Input layer
0.0 0.2 0.4 0.6 0.8 1.0 1.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
(b) 4th layer
0.00 0.25 0.50 0.75 1.00 1.25 1.50
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
(c) 4th layer
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(d) 25th layer
0.00 0.02 0.04 0.06 0.08 0.10 0.12
0.0
0.2
0.4
0.6
0.8
1.0
(e) 25th layer
0.006 0.004 0.002 0.000 0.002 0.004 0.006
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(f) Feature layer
0 1 2 3 4 5
0
1
2
3
4
5
6
(g) Feature layer
0.2 0.0 0.2 0.4 0.6
0.2
0.0
0.2
0.4
0.6
4.20E-01
4.80E-01
5.40E-01
6.00E-01
6.60E-01
7.20E-01
7.80E-01
8.40E-01
9.00E-01
9.60E-01
(h) Output
Figure 7: Data transformed across a 50x4 ReLU + BN network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The dataset is collapsed in few points at the feature layer. As the gradient cannot be backpropagated across the truncation after the affine transform of γ and β despite the standarization, it fails in the same manner than ReLU only that with non-zero activations. This results in topological mixing of the datasets. Therefore, the representational capability of the network is hindered to such extent that the resulting output, although non-trivial, is totally arbitrary.
0.2 0.0 0.2 0.4 0.6 0.2
0.1
0.0
0.1
0.2
0.3
(a) Input layer
0.006 0.004 0.002 0.000 0.002 0.004 0.006
0
1
2
3
4
5
(b) 4th layer
0.0 0.5 1.0 1.5 2.0
0
10
20
30
40
50
(c) 4th layer
0 5 10 15 20 25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
(d) 25th layer
0 50 100 150 200
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
(e) 25th layer
0.0 0.2 0.4 0.6 0.8
0
2
4
6
8
10
12
14
16
(f) Feature layer
0.006 0.004 0.002 0.000 0.002 0.004 0.006
0
5
10
15
20
25
(g) Feature layer
0.2 0.0 0.2 0.4 0.6
0.2
0.0
0.2
0.4
0.6
0.00E+00
1.50E-01
3.00E-01
4.50E-01
6.00E-01
7.50E-01
9.00E-01
1.05E+00
(h) Output
Figure 8: Data transformed across a 50x4 Sep-UP network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The network displays internal representations without collapsing the dataset like Sep-P and retaining discriminative power like Sep-U .
see Figure 9. whereas the representation reached by Sep-UP is more complex, yet preserves connectivity. Furthermore, both avoid mapping the entire dataset to few values as observed with ReLU (Figures 9e and 9f) and ReLU + BN (Figures 9b and 9c).
The only exception is Sep-U that apparently solves the problem at layer 25th on Figures 11d and 11e, yet collapses the dataset in two points at the feature layer (Figures 11f and 11f). Our intuition is is that, although Sep-U prevents dead units, it cannot prevent points from falling into the intersection lower set of the units of a layer, resulting in a dead point). The Sep-P separation constraint was designed precisely to prevent the presence of dead points. However, Sep-P allows affine and dead units (as shown in Figures 10d and 10e), so the the decision function reached becomes too linear (see Figure 10h).
Therefore, if we combine both Sep-U and Sep-Pwe should have best of both worlds, as we say in Section A.1. Figure 8, shows how Sep-UP is able to separate both classes perfectly (recall the 0.93 in accuracy from Table 1). Furthermore Sep-UP , produces a feature layer that does not concentrates the dataset in two points like Sep-U , nor distributes the dataset linearly like Sep-P .

heading: A.6 RELATION BETWEEN SEPARATION CONSTRAINTS AND MAIN LOSS
text: In this section we explore the effect of the proposed constraints in the convergence of the network during optimization. We train a network of 50 layers of 4 units each using Sep-UP . We use the MOONS dataset, in the same configuration than Section 4 (sampling 100 points, 85 for training and 15 for validation).
Experimental setting: We analyze the interaction between of the main loss (cross-entropy) and the loss arising from the Separation Constraints. We plot them, together with training accuracy in Figure 12. Below we attach the decision boundaries at the most relevant epochs.
Training Parameters: The networks wes optimized using Adam (Kingma & Ba, 2014). We used a learning rate of 0.01 for 17000 epochs and a batch size of 85. We used λ = 10−4. As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010). Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), fixing the random seed to an arbitrary value of 10.
We see at Figure 12 how the use of the separation constraints enables back-propagation, and thus cross-entropy minimization. According to it, the training begins minimizing the constraint loss until a certain level is reached (approximately 10−2 circa epoch 250). During this phase, the cross-entropy loss stays constant. Only when this critical point in the constraint loss is reached, back-propagation is restored and the accuracy starts climbing up to 100%. However, using an this additional loss (from the separation constraints) induces aggressive transient states (e.g. instabilities) during training (approximately around epochs 6200, 11883 and 16432). We see also how after each of the transient states, the network finds a new solution different from earlier ones. Moreover, those solutions show an increasing level of complexity, in the form of smoother and more rouded boundaries. For instance, the first solution (Figure 12f) is linear and the second is composed of few lines (Figure 12h, whereas the last and second to last (Figure 12ai and 12ao) are much more curved.
We conjecture that this behaviour must be due dead units and points being revived during training. As a dead unit or point comes back from being dead, the weight configuration set by the Separation Constraints is likely to be unrelated to cross-entropy. That causes a break down in the convergence that we can see around epochs 6200, 11883 and 16432. As back-propagation is now working for those points or units, their error is eventually corrected by normal cross-entropy minimization and the network converges to a solution again. However, as now there is an additional working unit or point involved, the network has greater representational expressiveness, which results in a more complex decision boundary.

Review summaries:
In this paper, a problem of training DNNs having dead points and neurons was addressed. The problem was posed as a constraint optimization problem. The constraints were augmented to the loss functions of DNNs as a regularizer. The slack variables of the constrained loss functions were optimized together with parameters of DNNs to train DNNs avoiding dead points and neurons.