Title: SUPERSEDING MODEL SCALING BY PENALIZING DEAD UNITS AND POINTS WITH SEPARATION CON-STRAINTS Anonymous

Abstract: In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense connections, data normalization or model scaling. We accomplish that by alleviating two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is also known as dead neurons. The other is a less studied problem, dead points. Dead points refers to data points that are mapped to zero during the forward pass of the network. As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant values or even zero and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.

1. INTRODUCTION:

The success of Deep Neural Networks (DNN for short) is linked to its ability to learn abstract representations from input data in a hierarchical fashion (LeCun et al. (2006) ; Ramachandran et al. (2017) ). However, the concepts of depth and width in networks are often used as instrumental elements to address different DNN pathologies during the learning process. Examples of these are: vanishing gradient (Hochreiter (1991; 2001) ), exploding gradient (Pascanu et al. (2013) ), dead units (Maas (2013); Douglas & Yu (2018) ; Guerraoui et al. (2017) ), or the degradation problem (He et al. (2015b) ).

In order to address the former issues, we find many methods and techniques that we can roughly classify in two families: data manipulation and architectural modifications. The most commonly used data manipulation technique is data normalization on the output of the layers, for example using batch normalization (Ioffe & Szegedy (2015) ). Examples of architectural modifications include the use of additional connections, as done in ResNets (He et al. (2015b)) or DenseNets (Huang et al. (2016) ); unit augmentation, as in leaky-ReLU (Maas (2013)), PReLU (He et al. (2015a) ), C-ReLU (Shang et al. (2016) ), or linked neurons (Riera & Pujol (2017) ); or the increment of layer's width with depth (Zagoruyko & Komodakis (2016) ; Szegedy et al. (2014) ). This last widely used approach effectively increases the the size of the network requiring larger computational power. This has led to several works (Hasanpour et al. (2018) ; Tan & Le (2019) ) that address it by offering heuristics on how to scale the network. Recent research (Liu et al. (2018) ) suggests that this inverted pyramidal architecture is not optimal.

Besides of the width scaling strategy, the concept of dead units1 is of particular interest to this work. A dead neuron is defined as the neuron with a constant or zero output for all training data points. This effectively renders this unit ineffective during the learning process. In (Lu et al., 2019) it is shown that as the depth increases the probability of finding dead neurons also increases, to the point that the entire network can be dead even at initialization. Additionally, as expected, as the width of a layer increases, the probability of having dead neurons decreases.

In this article we characterize another pathology, the dead point. The dead point is a dual concept to the dead neuron. A dead point corresponds to a data point that does not reach the output of the network. This is, the activation is zero for all the units in a given layer. As a result, this data point will have no influence in the training process. As in the case of dead units, both condition are not recoverable by back-propagation. This virtually reduces the size of the training set.

In order to solve the former issues, we propose and present a geometrical optimization constraint that is added to the loss function. As a result of adding this constraint we ensure that all neurons are active and no points are dead. The rationale behind the geometrical constraint is to control that all units/neurons and data points are alive by constraining pre-activation values in such a way that the hyperplanes associated to the non-linearity of each ReLU neuron separates, at least, one data point from the rest of the dataset. This has the additional effect of enabling the learning process to propagate all information through all the network avoiding the instrumental need of using additional connections or inverted pyramidal architectures.

We test our proposal in a series of controlled experiments to showcase the effect of applying the proposed Separation Constraints. We find that we can arbitrarily increase the network depth using the same constant width when compared to standard feedforward network and Batch Normalization Ioffe & Szegedy (2015) networks even when the width of the layer is extremely small (4 or 8 neurons). We additionally provide evidences that using this same approach we can initialize network parameters to zero and still achieve reasonable performance. These promising results gives insight on learning dynamics and suggest potential lines for future checks and research.

The article is organized as follows: Section 2 introduce the characterization of dead neurons and dead points, Section 3 introduces the two geometrical constraints that ensure that neurons and data points are alive, Section 4 describes the experiments and result, and, finally, Section 5 concludes the paper and suggests future lines of research.

2. CHARACTERIZING DEAD NEURONS AND DEAD POINTS:

A standard feed-forward ReLU DNN (LeCun et al., 2015) F can be formally written as a multivalued real function, F (x), that is created by composing a collection of D vector layer functions : R n k-1 → R n k . Layer k is defined as the sum of a collection of scalar functions (units):

EQUATION

that affinely depend on a weight vector w k j ∈ R n k and a bias parameter b k j ∈ R. When using rectified linear units this value is truncated on negative values:

EQUATION

Considering the hyperplane defined by w k j • x + b k j = 0, each unit defines a partition of the space R n k in two sets: the upper part of unit u k j and the lower part of u k j :

EQUATION

We define the affine component of a layer function k as the intersection of the upper parts of its units, and its zero set as the intersection of the lower parts, as follows:

EQUATION

Remark 2.1 (Dead unit). In a given ReLU -DNN F : R n → R k , we say that the j-th unit of layer k , u k j is dead with respect to a data set X ⊂ R n k if and only if

EQUATION

Observe that if a unit is dead, the output of the unit will be zero for the entire dataset, rendering the unit useless. Moreover, since the gradient is zero as well, it remains in this state for the rest of the training (See Lu et al. (2019) ; Shin & Karniadakis (2019) ). This effectively reduce the network learning capacity.

Remark 2.2 (Dead point with respect to a layer). Given ReLU -DNN F : R n → R k , we say that a point x ∈ X is dead with regards to layer k if

EQUATION

Remark 2.3 (Dead point). In particular, if X ⊂ R n , we say that point x ∈ X is dead with respect to a network F with depth D if it gets mapped to the zero set of a layer in its transit through the network. This is

EQUATION

Any dead point fulfilling Equation 7will show zero gradient in the layers previous to k . This hinders the learning process by effectively reducing the data set size. Again, this condition is not reversible using standard back-propagation. In a similar fashion to the case with dead units, the probability of finding a dead point increases with network's depth and decreases with layer's width.

3. INTRODUCING SEPARABILITY CONSTRAINTS:

In this section we introduce the desiderata for units and points to remain alive. Then, we proceed to formulate the separability contraints that fulfill the desired conditions.

Let us introduce the concept of a separating unit with respect to an arbitrary set X.

Definition 3.1 (Separating Unit). Given an arbitrary set X ⊂ R n k , we say that the j-th unit on layer k, u k j , is able to separate through X if the following predicate is satisfied:

R X (u k j ) ≡ ∅ = {lower(u k j ) ∩ X} ⊂ X (8)

Thus, by construction, a separating unit can not be dead, and if R X (u k j ) is valid, u k j can not degrade set X to zero. In other words, this condition ensures that each unit always separates at least one data point.

In terms of points, we can define a separating point as follows: Definition 3.2 (Separating point). Given an arbitrary set X ⊂ R n k , we say that point x ∈ X is separating a layer function k if there exist indices j, l ∈ {1, . . . , m k } for which the following predicate is satisfied.

EQUATION

Again, by construction, a separating point ensures that each point in the data set has at least one unit in each layer with an activation different to zero and another with an activation equal to zero.

3.1. MODELLING UNIT-BASED SEPARATION CONSTRAINT (SE P-U ):

Unit based separation contraint (Sep-U ) is designed to model predicate 8 with the goal of avoiding the presence of dead units.

Given a unit u k j in layer k : R n k → R n k+1 from a ReLU DNN F of depth D, predicate 8 can be simply modelled imposing the following constraints,

EQUATION

These strict inequalities can not be directly optimized. It is easily to see that these can be rewritten as

max i=1,...,|X| {w k j • x i + b k j } ≥ 1 min i=1,...,|X| {w k j • x i + b k j } ≤ -1 (11)

The use of the former constraints makes most problems unfeasible. Thus, in a similar fashion to soft-margin SVM (Cortes & Vapnik, 1995) , we introduce a set of positive slack variables {ξ + jk , ξ - jk } that account for constraint violations as follows,

EQUATION

for k = 1, . . . , D and n k the number of units of layer k. The intuition behind the introduction of these constraints is as follows: by minimizing ξ + at least one pre-activation value is forced to be greater (or equal) than 1. Simmetrically, minimizing ξ -promotes at least one pre-activation to be below -1. This effectively fulfills Predicate 8 and penalize the apparition of dead units. We show how to differentiate the Separation Constraints with regards to the parameters in the Appendix Section A.3.

At a global network scale, we can aggregate all the slack variable in a single optimization objective as follows,

EQUATION

3.2 MODELLING POINT BASED SEPARATION CONSTRAINT (SE P-P )

The derivation of the Point Based Separation Constraints (Sep-P ) follows a parallel process to Sep-U . In order to avoid the presence of dead points it suffices to fulfill Predicate 9. Similarly to the former derivation, we introduce a set of slack variables for each each point on the batch. That is, given x i ∈ X, and u k 1 , . . . , u k n unit functions in a layer k , we define slack variables ξ - ik , ξ + ik in the context of the following constraints,

EQUATION

Observe that the minimization of the slacks makes that for any data point at least one activation is above 1 and another is below -1.

We can summarize all the point-based slack variables in a single optimization objective as follows,

EQUATION

)

3.3. TRAINING WITH SEPARATING CONSTRAINTS:

The new optimization objectives can now be added to the original loss objective using a simple scalarization (Boyd & Vandenberghe (2004) ) as follows,

EQUATION

)

where λ is a hyper-parameter that introduces a trade-off between the constraint fulfillment and the main loss function.

In terms of memory complexity, the former constraints introduce a very small memory overhead.

In particular, Sep-U places a pair of constraints on each of the units of the network, so the complexity with respect to Sep-U scales with the size of the network as 2 D k=1 n k . Alternatively, Sep-P places a pair of constraints in each of the points of the data set or selected subset X. In practice, one can use X as the training batch. Thus, the memory complexity scales with the size of the batch and the number of layers, i.e. 2D|X|. Furthermore, since the resulting gradient of both types of constraints depends only on the input of the layer that is already computed in the forward pass, we only add the cost of storing the slacks. Therefore, the total cost in terms of number of constraint is the addition of the former terms, i.e. 2 D k=1 n k + 2D|X|.

4. EXPERIMENTS AND RESULTS:

In this section we explore the application of the proposed constraints in different datasets. For that task we train all methods with different choices of depth and width parameters. The network architecture used is rectangular, i.e. networks with a fixed layer width for all the layers.

Datasets: Due to the large amount of computational resources required for the depth and width grid training, we are forced to chose three controlled datasets in our experimentation: the MOONS dataset (sampling 100 points, 85 for training and 15 for validation), the MNIST dataset as described in LeCun and Cortes. LeCun & Cortes (2010) and the CIFAR-10 dataset described in Krizhevsky (2009) .

Experimental setting: We compare the combination of Sep-U and Sep-P (Sep-UP from now on) to feed-fordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in Ioffe & Szegedy (2015) using the same architecture.

For the MOONS dataset, we use depths from 1 to 120 in steps of 10, and width from 1 to 25 in steps of 1 between 1 and 5, and steps of 5 afterwards. In the case of the MNIST dataset, we use depth from 2 to 64 and width from 2 to 8 in steps of 4. Finally, for CIFAR-10 we use depths in {2, 10, 25, 30, 40} with a fixed width of 10 due memory constraints.

Training Parameters: All the networks used were optimized using Adam (Kingma & Ba, 2014) . More specifically, for the MOONS dataset we used a learning rate of 0.01 for 5000 epochs and a batch size of 85. Meanwhile, for both MNIST and CIFAR-10 we used a learning rate of 0.0001 for 50 epochs and a batch size of 1024. We used convolutional layers with a (3, 3) kernel size for both MNIST and CIFAR-10. We used λ M OON S = 10 -4 λ M N IST = λ CIF AR = 10 -8 for Sep-UP . Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015) , fixing the random seed to an arbitrary value of 10.

As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010) for all the methods and datasets.

4.1. RESULTS:

Figure 1 shows the results obtained in the MOONS dataset. Our proposal Sep-UP is able to train networks successfully without increasing the width up to 60 layers deep (see Figures 1c and 1f ), while ReLU breaks down at only 30 layers (see Figures 1a and 1d ) and ReLU + BN suffers from severe accuracy degradation (see Figures 1b and 1b ).

Observe that no configuration with lower width than 2-3 is successful in achieving maximum accuracy. We understand that there exists a minimum width required and this is related to the complexity of the problem. When using wider layers, the rest of the width is instrumentally used to enable the training of deeper networks. As previously commented, the larger the layer's width, the higher the chances of finding active units that do not cause dead points and dead units during initialization. In opposition, Sep-UP succesfully overcomes that constraint.

Notice that though Sep-UP is superior to all its competitors, it starts showing performance degradation after reaching depth 60 with the minimum width of 3. Considering that the number of parameters increases with the depth and width of the network and that we have a finite number of trained epochs, we conjecture that the displayed degradation is strictly due to the lack of convergence of the constrained network. The separation constraint also proves successful on convolutional networks, as tested in MNIST and CIFAR-10 datasets. Figure 2 shows a similar behaviour to the MOONS dataset (Figure1). ReLU breaks down after a few layers, ReLU + BN delays the degradation of accuracy, while Sep-UP remains functional regardless of the depth. In the case of our experiments with the CIFAR-10 dataset (as presented on Figure 3 ) all the methods degrade with depth, but Sep-UP is the most robust. In regards to accuracy ReLU performs best closely followed by Sep-UP , while ReLU + BN clearly overfits. The poorer accuracy values shown are due to a limited choice of width, clearly inferior to the minimum required by the dataset, and a potential lack of convergence of the separating constraints. Additional results in the Appendix further elaborate the contributions of each term Sep-U and Sep-P , independently.

4.2. RESULTS USING ZERO INITIALIZATION:

In order to test the invariace to initialization scheme of Sep-UP , we use Zero Initialization. As its name states, in this initialization scheme all weights and biases are set to zero. However, a small variation of the scheme must be introduced in order to break symmetry for the constraints to apply. Since all the units are initialized to the same value (zero), we use Annealed Dropout (Rennie et al., 2014) . Additionally, instead of adding ξ + and ξ -pairs as in Equations 13 and 15, we use a convex combination with a small perturbation ρ. In our experiments, we use a value of ρ = 0.01.

(

EQUATION

Figure 4 summarizes the results found using zero initalization with our constraint formulation. In comparison to Glorot, we observe that zero initalization requires wider networks. Indeed, at a depth of 60, Zero initialization requires a width of 25 units while the Glorot scheme work with only 2 units. However, if we contrast ReLU + BN to our constraint formulation we find that it is unable to train networks of past depth 50, while Zero initialization achieves 70 layers.

Figure 4 shows the results using zero initialization. Although reported values and behavior are slightly worse compared to Glorot initialization, our results are promising. Notice how at depth of 60 layers, the zero initialization requires a width of 25 units in contrast to 2 required for Glorot.

In addition, zero initialization rquires only 70 layers to reach perfect accuracy in contrast to the 100 required in Glorot (see Figures 1c and 1f ). Moreover, in comparison to ReLU + BN , zero initialized networks show superior performance and behavior (check Figures 1b and 1e ).

5. CONCLUSIONS:

Through the Separation Constraints, we have shown that deeper networks can be trained without increasing the width of the layers. Moreover, this increment can be done using very small width values. In this sense we consider that effective training of deeper networks can be achieved by better accommodating the network to the input data. This departs from many proposal that achieve similar effects by modifying the architecture of the network or manipulating the data. We believe that this work shows an alternative research path in the pursuit of effective and efficient learning techniques for deep neural networks. This also opens the possibility of avoiding the use of pyramidal Figure 4 : Depth vs width accuracy heatmap a for a grid of rectangular networks with (width from 1 to 25 and depth from 1 to 120), trained using Adam using a learning rate of 0.01, over the MOONS dataset (5000 epochs and Zero Initialization). The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black at 0.5. Notice how although being inferior to Glorot (Figures 1c and 1f ) is superior to ReLU + BN (Figures 1b and 1e ).

architectures commonly employed in DNNs, thus removing the computational burden caused by the additional units and reducing the dimensionality of the internal representations.

We additionally show that our proposal enables the use of Zero Initialization. The results are promising, and though further experimentation is still needed, they still manage to surpass Batch Normalization in terms of depth, width, and accuracy. Nevertheless, the modifications used to break the symmetry of constraints and units need further consideration in order to achieve the same performance than random initialization.

Despite this article provides no information about the dynamics of the training process using Separation Constraints, preliminary revision shows interesting properties. In particular, training displays slight instabilities which appear as the units/points are reactivated. Although this does not hinder the training process, further study is needed in order to guarantee a smoother convergence.

The extension of this work to other activation functions is still to be explored. However, we conjecture that in cases where the activation function display a flat region, e.g. ELU, tanh, or even sigmoid, the current proposal can be applied with minor changes.

Finally, while the separation constraints prevent the vanishing gradient effect, the exploding gradient problem still remains. Extending the Separation Constraint with an upper bound on the magnitudes of the pre-activation, similar to -insensitive loss, might address it. It could be also helpful to explore other activation functions whose gradient vanishes with high pre-activations, such as the logistic family.

A APPENDIX A.1 UNIT BASED AND POINT BASED SEPARATION CONSTRAINT EXPERIMENT:

In order to justify the decision of combining Sep-U and Sep-P into Sep-UP , we provide Figure 5 showing the same experiment than Figure 1 for Sep-U and Sep-U . We find how although Sep-U is able to train deeper and thinner networks it suffers from inferior accuracy (see Figures 5a and 5c ), requiring the use of the layer width increase technique. Contrarily, Sep-P is able to train without increasing the width from 1 to 40 layers deep, but it breaks down afterwards. Provided how complementary are their strengths and weaknesses, it is sensible to combine them, resulting in the superior performance (larger constant width area together with greater accuracy in higher depths) of Sep-UP , see Figures 1c and 1f . 

A.2 PROBABILITY OF A POINT TO BE DEAD IN A NEURAL NETWORK:

We conjecture that difficulty arising when training neural network as the depth increases or the width decreases, is due the incorrect positioning of the hyperplanes of the units after initialization which causes the apparition of dead points (recall Remark 2.3), that ultimately break back-propagation. In this section we offer a formal treatment of this idea.

Theorem A.1 (The probability of a point to be dead increases with depth and decreases with width). Let F : R n1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions : R n k-1 → R n k with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of n k ∈ N ReLU unit functions u : R n k-1 → R + , we denote n k as the width of the layer. Let the network be initialized following a random distribution. Then, given a point x ∈ R n1 we say that the probability of the point dying (Recall Definition 7) increases with D and decreases with n k . Lemma A.2 (Probability of a point to be dead for a ReLU unit follows Bernoulli). Let be a point x ∈ R n1 and a ReLU unit u parametrized by a weight vector w ∈ R n and a bias b ∈ R initialized following a random distribution, then the probability of point x being dead for u follows a Bernoulli distribution of unknown parameter p.

EQUATION

Proof. Since the unit is initialized randomly according to a given distribution (i.e. normal, uniform, Glorot, ...), the probability of its hyper-plane parametrized by w, b facing x is an unknown p according to the distribution used. Therefore, the probability of x belonging to lower(u) (Recall Equation 3) must be p.

As the lower set (Recall Definition 3) pre-activation is truncated, its gradient with respect the parameters will be zero (See Rey-Torres et al. (2019, Section 2)). Therefore, any point belonging to lower(u) will be dead in regards to u.

Lemma A.3 (Probability of a point to be dead for a ReLU layer). Let k be a layer composed of n k ∈ N ReLU units u initialized following the same random distribution. Let be a point x ∈ R n k-1 . Then the probability of x being dead for k is:

EQUATION

Proof. Similarly to the unit case, points belonging to the zero set of a layer (Recall Definition 4) have zero gradient with respect the parameters of the layer due the truncation (See Rey-Torres et al.

(2019, Section 2)).

The probability of a point belonging to the zero set of a layer P (x ∈ Z( k )) is equal to the probability of belonging to all the lowers of the units of the layer P (∀u ∈ |x ∈ lower(u)). Since unit parameters are sampled independently during initialization and the probability of a point being dead for a unit follows Bernoulli, then the probability of a point belonging to Z( k ) is the product of the probabilities of belonging to the lower of each of the units.

Lemma A.4 (The probability of a point to be dead for a layer diminishes with its width). Let k be a layer composed of n k ∈ N ReLU units u initialized following a random distribution. Let be a point x ∈ R n k-1 . Then the probability of x being dead diminishes with n k :

lim n k →∞ P (x ∈ Z( )) = 0 (20)

Proof. The probability of a point being dead for a layer is the multiplication of the probabilities of being dead of each of its units, recall Lemma A.3. Therefore, as the probabilities are lower than 1 since the upper set of a unit is never empty, in the limit where n k tends to infinity the probability tends to zero. 

EQUATION

Lemma A.5 (Probability of a point to be dead for a ReLU feed-forward network). Let F : R n1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions : R n k-1 → R n k with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of n k ∈ N ReLU unit functions u : R n k-1 → R + , we denote n k as the width of the layer. Let the network be initialized following a random distribution. Then, given a point x ∈ R n1 we say that the probability of the point dying (Recall Definition 7) is

EQUATION

Proof. The probability of not dying in a given layer is equal to 1 minus the probability of dying.

EQUATION

As the sampling of the parameters during intialization is independent among layers, the probabiliity of not dying in any of the layers of the network is the multiplication of the probabilities of not dying in each of the layers.

EQUATION

The probability of a point to be dead for a network is equal to the probability of being dead at least in one layer, see 7. This is equivalent to 1 minus the probability of not dying in any of the layers.

EQUATION

Plugging Equation 24into Equation 25 results in Equation 22.

Lemma A.6 (The probability of a point to be dead for a layer increases with its depth). Let F : R n1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions : R n k-1 → R n k with k ∈ 1, . . . , D, we denote D as the depth of the network. Let the network be initialized following a random distribution. Then, given a point x ∈ R n1 we say that the probability of the point dying as it traverses the network (Recall Definition 7) increases with D.

lim D→∞ P (x ∈ Z(F )) = 1 (26)

Proof. Since the hyper-planes of the units split the space in two half-spaces, the probability of point not being dead is always lower than 1.

P (x / ∈ Z( )) < 1 (27)

Therefore, the limit of the probability of x not dying in any of the layers when D tends to infinity is zero, recall Equation 24.

EQUATION

Which when plugged into Equation 22 leads to Equation 26.

A.3 ON THE OPTIMIZATION OF THE CONSTRAINTS:

In this section we show how to differentiate the Separation Constraints with regards to the weights. Notice that under our formulation, we are examining the set diameter of the preactivation of a dataset X. That is, given a unit u with vector parameters w ∈ R n and b ∈ R, its preactivation is given by the function 

z = p(x) = w • x + b (29) given a finite discrete set X ⊂ R n ( a compact set of R n ),

EQUATION

notice that since X is a finite discrete set of R n k , z min , z max exist (where stands for u or p). This fact is instrumental in our Equations ( 10) to ( 12). Since X is bounded, and we are working in the real number system, we can find numbers ξ + , ξ -such that

EQUATION

notice that the parameters w ∈ R n k and b are free in the definition of z u , while x is free in the definition of z p . In this sense, for ξ p ± we can find

w p ± ∈ R n k and b p ± ∈ R such that ξ p + = max(1 -(w + • x + b + ), 0) ξ p -= max(1 + (w -• x + b -), 0) (33)

that constraints all possible configurations of parameters in units of . Naturally, ξ ± depend on w and b. This is at the heart of the constraint formulation: choosing w and b so that z min < 0 and z max > 0.

Meanwhile, given a fixed selection of w and b, since X ⊂ R n k is discrete and bounded, we must have that there exist x + and x -points of X such that

EQUATION

We wish to stress a geometric intuition, beyond calculations. We seek a combination of w and b so that they 'cut' through X in the case of Sep-U, noticing that the preactivation p defines a signed distance from the separating hyperplane p(x) = 0.

The same argument is used in the field of Support Vector Machines, as cited in our references, particularly Vapnik & Cortes' Vector Support Networks from 1995 and Boyd & Vandenberghes' Convex Analysis from 2004.

A.4 ON THE RELATION BETWEEN CONSTRAINTS AND PARAMETER UPDATE:

Notice that up to Equation 12, we have only stated constraints upon the slack variables ξ ± (that involve the dataset and parameters w and b). In Equation 13we introduce the objective function to optimize the values of ξ ± . Namely, the sum of all ξ.

Since we have different indications of the ξ, we must define different objective functions for unit-wise constraints or g U of Equation 13and point-wise constraints or g P from Equation 15. We presented the construct for unit-based constraints (transit between Equations 12 and 13) and extended the notation for point-wise constraints on Equation 15.

If we wish to differentiate with regards to w and b equations describing the relation between x and ξ unit-wise or point-wise, we must notice that ∇ w ξ u ± as stated, must be defined by parts:

EQUATION

EQUATION

while for ξ u + :

EQUATION

EQUATION

while under the unit-based formulation x ± is fixed by the geometry of X, for the point-wise formulation, we must have

EQUATION

EQUATION

and for ξ p + :

EQUATION

EQUATION

that is parallel to the loss gradient ∇ w L.

A.5 EFFECT OF THE SEPARATION ON THE INTERNAL REPRESENTATIONS:

In this section we explore the effect of the proposed constraints in the activation of the network.

For that task we train all methods fixing the architecture at 50 layers of 4 units each. We use the MOONS dataset for easy visualization, in the same configuration than Section 4(sampling 100 points, 85 for training and 15 for validation) .

Experimental setting: We compare the combination of Sep-U , Sep-P and Sep-UP to feedfordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in Ioffe & Szegedy (2015) using the same architecture.

Training Parameters: All the networks used were optimized using Adam (Kingma & Ba, 2014) . We used a learning rate of 0.01 for 1000 epochs and a batch size of 85. We used λ = 10 -4 when required. As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010) . We also test the use of the Separation Constraints alone, without categorical crossentropy, to assess its effect by itself. Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015) , fixing the random seed to an arbitrary value of 10. In terms of accuracy ReLU reaches a trivial accuracy of 0.51 while ReLU + BN reaches 0.60, both architectures fail to solve the problem, see Table 1 . When comparing the results presented in Figures 6 and 7 , we find that at the fourth layer ReLU has collapsed the points of the dataset over a line (parts 6b and 6c), while ReLU + BN still manages to warp the dataset. However, both methods at layer 25th have collapsed the dataset into a small number of points (see Figures 7d, 7e for ReLU + BN and Figures 6d and 6e for ReLU ) that is then pushed to the point zero for ReLU (as shown in the feature layer: 6f and 6g), while ReLU + BN collapses to few points (Figures 7f and 7g ).

This collapse is congruent with our expectations, displaying the born-dead network condition from Lu et al. (2019) . Note how layer after layer its effect is worse, as we predicted in A.2. In addition, since the gradient is only back-propagated through the points lying on the upper sets of units, geometric collapse also stops learning. Notice that the standarization and the affine transformation (dependent on γ and β) from ReLU + BN are not able to prevent from killing the dataset. Thus, training thin and deep neural networks is difficult with ReLU or ReLU + BN .

On the extent of our experimentation ReLU networks featuring separation constraints are able to solve the problem with higher degrees of accuracy than ReLU and ReLU + BN (see the figures in Table 1 ). Moreover, Figures 8h, 11h and 10h ), show that the separation function behaves intuitively in the sense of Hauser & Ray (2017) .

The internal representations not only are non-trivial (as in ReLU + BN ) but also preserve geometrical structures like shape and connectivity, as shown in Figures 11d, 10d or 8e. Indeed, Figures 11c, 8c and 10c showcase how at the 4th layers a much a solution is already found. This proves that the gradient of the main loss is back-propagated to the input, unlike ReLU and ReLU + BN (Figures 6b, 6b, 7b, 7c) .

The constraints enforce richer representations when used without main loss (cross-entropy), Under review as a conference paper at ICLR 2020 Figure 7 : Data transformed across a 50x4 ReLU + BN network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The dataset is collapsed in few points at the feature layer. As the gradient cannot be backpropagated across the truncation after the affine transform of γ and β despite the standarization, it fails in the same manner than ReLU only that with non-zero activations. This results in topological mixing of the datasets. Therefore, the representational capability of the network is hindered to such extent that the resulting output, although non-trivial, is totally arbitrary. Under review as a conference paper at ICLR 2020 see Figure 9 . whereas the representation reached by Sep-UP is more complex, yet preserves connectivity. Furthermore, both avoid mapping the entire dataset to few values as observed with ReLU (Figures 9e and 9f ) and ReLU + BN (Figures 9b and 9c ).

The only exception is Sep-U that apparently solves the problem at layer 25th on Figures 11d and 11e , yet collapses the dataset in two points at the feature layer (Figures 11f and 11f ). Our intuition is is that, although Sep-U prevents dead units, it cannot prevent points from falling into the intersection lower set of the units of a layer, resulting in a dead point). The Sep-P separation constraint was designed precisely to prevent the presence of dead points. However, Sep-P allows affine and dead units (as shown in Figures 10d and 10e ), so the the decision function reached becomes too linear (see Figure 10h ).

Therefore, if we combine both Sep-U and Sep-P we should have best of both worlds, as we say in Section A.1. Figure 8 , shows how Sep-UP is able to separate both classes perfectly (recall the 0.93 in accuracy from Table 1 ). Furthermore Sep-UP , produces a feature layer that does not concentrates the dataset in two points like Sep-U , nor distributes the dataset linearly like Sep-P .

Under review as a conference paper at ICLR 2020 We conjecture that this is due the dead point addressed with Sep-P .

A.6 RELATION BETWEEN SEPARATION CONSTRAINTS AND MAIN LOSS:

In this section we explore the effect of the proposed constraints in the convergence of the network during optimization. We train a network of 50 layers of 4 units each using Sep-UP . We use the MOONS dataset, in the same configuration than Section 4 (sampling 100 points, 85 for training and 15 for validation).

Experimental setting::

We analyze the interaction between of the main loss (cross-entropy) and the loss arising from the Separation Constraints. We plot them, together with training accuracy in Figure 12 . Below we attach the decision boundaries at the most relevant epochs.

Training Parameters: The networks wes optimized using Adam (Kingma & Ba, 2014) . We used a learning rate of 0.01 for 17000 epochs and a batch size of 85. We used λ = 10 -4 . As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010) . Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015) , fixing the random seed to an arbitrary value of 10.

We see at Figure 12 Moreover, those solutions show an increasing level of complexity, in the form of smoother and more rouded boundaries. For instance, the first solution (Figure 12f ) is linear and the second is composed of few lines (Figure 12h , whereas the last and second to last (Figure 12ai and 12ao) are much more curved.

We conjecture that this behaviour must be due dead units and points being revived during training. As a dead unit or point comes back from being dead, the weight configuration set by the Separation Constraints is likely to be unrelated to cross-entropy. That causes a break down in the convergence that we can see around epochs 6200, 11883 and 16432. As back-propagation is now working for those points or units, their error is eventually corrected by normal cross-entropy minimization and the network converges to a solution again. However, as now there is an additional working unit or point involved, the network has greater representational expressiveness, which results in a more complex decision boundary. ). This process occurs several times (6200, 11883 and 16432), resulting in increasingly complex solutions (notice how solution at 16630 is much more rounded and smoother). We conjecture that this due to dead units and points recovered during training by the Separation Constraint augment the capacity of the network.

:

In this work we use unit and neuron indistinctly.

Figures and Tables:

Figure 1: Depth vs width accuracy heatmap a for a grid of rectangular networks with width from 1 to 25 and depth from 1 to 120, trained using Adam with a learning rate of 0.01 in the MOONS dataset for 5000 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black at 0.5. Notice how ReLU breaks down at 20 layers and ReLU + BN requires more units per layer as increasing depth, while Sep-UP works with the minimum width (3) up to 60 layers deep.

Figure 2: Depth vs width accuracy heatmap a for a grid of rectangular networks with width from 2 to 8 and depth from 2 to 68, trained using Adam with a learning rate of 0.0001 in the MNIST dataset for 50 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black at 0.1. Notice how ReLU breaks down at 20 layers and ReLU + BN accuracy degrades with depth, while Sep-UP shows constant accuracy disregarding the number of layers.

Figure 3: Depth vs Width accuracy heatmap a for a grid of rectangular networks with width 10 and depth from 2 to 40, trained using Adam with a learning rate of 0.0001 in the CIFAR-10 dataset for 50 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black at 0.1. Observe how Sep-UP shows inferior degradation in accuracy as depth increases compared to ReLU + BN .

Sep-UP Zero-init val.

Figure5: Depth vs width accuracy heatmap a for a grid of rectangular networks with width from 1 to 25 and depth from 1 to 120, trained using Adam with a learning rate of 0.01 in the MOONS dataset for 5000 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black at 0.5. Notice how Sep-U enables train thinner and deeper networks (120 layers of single unit), but with reduced performance compared to Sep-UP . It also requires increasing the width of the layers in order to achieve perfect accuracy as ReLU + BN . Alternatively, Sep-P shows a constant width area between 1 and 40 layers, but its performance is much worse above. This justifies the decision of combining them both into Sep-UP to combine their strengths.

Note how Lemma A.4 can explain why increasing width allows us to train deeper networks(See  Hasanpour et al. (2018);Huang et al. (2016)), it simply improves the chances of finding a hyperplane that allows the points to live. Definition A.1 (Dead point set in a neural network). Let F : R n1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions : R n k-1 → R n k with k ∈ 1, . . . , D, we denote D as the depth of the network. The layers are composed by a collection of n k ∈ N ReLU unit functions u : R n k-1 → R + , we denote n k as the width of the layer. Then, we define the set of dead points of a neural network following Equation 7 as.

Figure 6: Data transformed across a 50x4 ReLU classification network. Notice how the the dataset is progressively mapped to zero as it traverses the network. This renders the output layer unable to solve the problem.

Figure 8: Data transformed across a 50x4 Sep-UP network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The network displays internal representations without collapsing the dataset like Sep-P and retaining discriminative power like Sep-U .

Figure 9: Data transformed across a 50x4 network with no main loss (cross-entropy) with constraints Sep-L and Sep-UP , versus ReLU and ReLU + BN . Notice how effectively ReLU and ReLU + BN collapse the dataset into few points whereas Sep-UP force the network to learn representations that preserve geometrical structure useful for back-propagation.

Figure11: Data transformed across a 50x4 Sep-U network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. Notice how dead units have been reduced. The interal representations are much richer than ReLU or ReLU + BN . Despite collapsing the dataset in two points at the feature layer, the classification performed in the output layer is approximately correct. We conjecture that this is due the dead point addressed with Sep-P .

Figure12: Evolution of training throughout epochs (cross-entropy, constraint loss, accuracy, and decision surface) in 50x4 network trained on the MOONS dataset. We used Adam with a lr of 0.01 for 17000 epochs and a batch size of 85 and Glorot uniform. We used λ = 10 -4 . Figure12ashows both cross-entropy and constraint loss for each epoch of the training phase in the horizontal axis, whereas Figure12bdoes the same for training accuracy. The rest of subfigures show the decision surface at the indicated epoch. We see how learning only starts after the constraint loss has been minimized below approximately 10 -2 , circa epoch 250. After that, back-propagation is restored and cross-entropy falls quickly, reaching a solution at epoch 1135. However, around epoch 6205 the training breaks down and the next epochs are spent in a transient state, until the geometry of the boundary changes and the network converges to a different solution (note how the solution found at 1135 is composed of 5 lines, whereas 9162 consists of 7). This process occurs several times (6200, 11883 and 16432), resulting in increasingly complex solutions (notice how solution at 16630 is much more rounded and smoother). We conjecture that this due to dead units and points recovered during training by the Separation Constraint augment the capacity of the network.

, given a layer with units u 1 , u 2 , . . . , u n k with pre-activations denoted by p 1 , p 2 , . . . , p n k , we can define the extreme values of layer provided with a fixed x ∈ X as

Table content: <table><tr><td/><td/><td/><td/><td>we can define the extreme values of X</td></tr><tr><td>with regards to u as</td><td/><td/><td/></tr><tr><td>z u min</td><td/><td>= min x∈X</td><td colspan="2">p(x)</td></tr><tr><td/><td/><td/><td/><td>(30)</td></tr><tr><td>z u max</td><td/><td colspan="2">= max x∈X</td><td>p(x)</td></tr><tr><td>on a similar argumentz p min</td><td>=</td><td colspan="3">min j=1...,n k</td><td>p j (x)</td></tr></table>



Table content: <table><tr><td/><td colspan="2">Accuracy</td><td>Loss</td></tr><tr><td/><td>Train</td><td>Val.</td><td>Train</td><td>Val.</td></tr><tr><td>ReLU</td><td>0.5176</td><td colspan="3">0.4 0.6925 0.6938</td></tr><tr><td colspan="2">ReLU + BN 0.8117</td><td colspan="3">0.6 0.6331 0.6636</td></tr><tr><td>Sep-P</td><td colspan="4">0.9294 0.8000 0.1765 0.6476</td></tr><tr><td>Sep-U</td><td colspan="4">0.9058 0.8000 0.4161 1.5228</td></tr><tr><td>Sep-UP</td><td colspan="4">0.9882 0.9333 0.6988 1.0810</td></tr></table>

Maximal performance experiment using the MOONS dataset. From left to right, accuracy and loss (for train and validation sets) for ReLU , ReLU + BN , and Sep-Cons in all its variants.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td/><td>0.008</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td/><td>0.006</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td/><td>0.004</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td/><td>0.002</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.000</td><td/><td/><td/><td/><td/><td>0.000</td><td/><td/><td/><td/><td/><td/><td>0.000</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td/><td>0.002</td><td/><td>5.18E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td/><td>0.004</td><td/><td>0.6</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td>5.18E-01</td></tr><tr><td>0.3</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td/><td>0.008</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.4</td><td>5.18E-01</td></tr><tr><td>0.0 0.1</td><td/><td/><td/><td/><td/><td/><td colspan="5">(b) 4th layer</td><td/><td/><td colspan="5">(d) 25th layer</td><td/><td colspan="2">(f) Feature layer</td><td>0.2</td><td>5.18E-01</td></tr><tr><td>0.1</td><td/><td/><td/><td/><td/><td>0.40</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td/><td>0.008</td><td/><td>5.18E-01</td></tr><tr><td>0.2</td><td>0.2</td><td>0.0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.35</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td>0.0</td><td>5.18E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.30</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td/><td>0.004</td><td/></tr><tr><td/><td colspan="5">(a) Input layer</td><td>0.15 0.20 0.25</td><td/><td/><td/><td/><td/><td>0.002 0.002 0.000</td><td/><td/><td/><td/><td/><td/><td>0.002 0.002 0.000</td><td/><td>0.2</td><td>5.18E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.05 0.10</td><td/><td/><td/><td/><td/><td>0.006 0.004</td><td/><td/><td/><td/><td/><td/><td>0.006 0.004</td><td/><td>0.2</td><td>0.0 (h) Output 0.2 0.4</td><td>0.6</td><td>5.18E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.00</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td/><td>0.008</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="5">(c) 4th layer</td><td/><td/><td colspan="5">(e) 25th layer</td><td/><td colspan="2">(g) Feature layer</td></tr></table>

Data transformed across a 50x4 Sep-P network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The network displays a richer internal representation without collapsing the dataset like Sep-U or ReLU + BN . However, plenty of dead units appear since they are not penalized, causing underfitting.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.000</td><td/><td/><td/><td/><td/><td>0.000</td><td/><td/><td/><td/><td/><td>0.000</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/><td/><td/><td/><td/><td>0.002</td><td/><td>1.05E+00</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td>0.6</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>0.006</td><td/><td>9.00E-01</td></tr><tr><td>0.3</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>0.008</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>50</td><td>100</td><td>150</td><td>200</td><td>250</td><td/><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.4</td><td>7.50E-01</td></tr><tr><td>0.0 0.1</td><td/><td/><td/><td/><td/><td/><td colspan="5">(b) 4th layer</td><td/><td/><td colspan="4">(d) 25th layer</td><td/><td colspan="2">(f) Feature layer</td><td>0.2</td><td>6.00E-01</td></tr><tr><td>0.1</td><td/><td/><td/><td/><td/><td>10</td><td/><td/><td/><td/><td/><td>0.008</td><td/><td/><td/><td/><td/><td>90</td><td/><td>4.50E-01</td></tr><tr><td>0.2</td><td>0.2</td><td>0.0</td><td>0.2</td><td>0.4</td><td>0.6</td><td/><td/><td/><td/><td/><td/><td>0.006</td><td/><td/><td/><td/><td/><td>80</td><td/><td>0.0</td><td>3.00E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>8</td><td/><td/><td/><td/><td/><td>0.004</td><td/><td/><td/><td/><td/><td>70</td><td/></tr><tr><td/><td colspan="5">(a) Input layer</td><td>6</td><td/><td/><td/><td/><td/><td>0.002 0.000 0.002</td><td/><td/><td/><td/><td/><td>50 60</td><td/><td>0.2</td><td>1.50E-01</td></tr><tr><td/><td/><td/><td/><td/><td/><td>2 4</td><td/><td/><td/><td/><td/><td>0.008 0.006 0.004</td><td/><td/><td/><td/><td/><td>20 30 40</td><td/><td>0.2</td><td>0.0 (h) Output 0.2 0.4</td><td>0.6</td><td>0.00E+00</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td><td>200</td><td>400</td><td>600</td><td>800</td><td>1000</td><td>1200</td><td>0.006</td><td>0.004</td><td>0.002</td><td>0.000</td><td>0.002</td><td>0.004</td><td>0.006</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="5">(c) 4th layer</td><td/><td/><td colspan="4">(e) 25th layer</td><td/><td colspan="2">(g) Feature layer</td></tr><tr><td colspan="4">Figure 10:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

how the use of the separation constraints enables back-propagation, and thus cross-entropy minimization. According to it, the training begins minimizing the constraint loss until a certain level is reached (approximately 10 -2 circa epoch 250). During this phase, the cross-entropy loss stays constant. Only when this critical point in the constraint loss is reached, back-propagation is restored and the accuracy starts climbing up to 100%. However, using an this additional loss (from the separation constraints) induces aggressive transient states (e.g. instabilities) during training (approximately around epochs 6200, 11883 and 16432). We see also how after each of the transient states, the network finds a new solution different from earlier ones.

Table content: <table/>

Peer review (iclr2020):
This papers suggests new penalties to avoid the apparition of dead units and minimize the existence of dead points. Decision I vote to reject this paper because the formulation of the penalties lacks clarity and because the experiments are incomplete and likely provide misleading results. Justification Formulation of the penalty It is unclear to me how the slack variables xi depends on the weights, and therefore how they can be minimized by adjusting the weights using a penalty. I believe this should be clarified between equations (12) and (13). The optimizer used is Adam instead of a plain SGD. How is the penalty integrated in Adam? Is it left out as it is best to do for weight decay or is it integrated in the adaptive estimations? I believe a plain SGD would be better justified when the goal is to understand an improvement to the flow of information in the network. Experiments The experimental setup have many problems. 1) The protocol is creating a bias 2) The comparisons of zero initializations are indirect 3) The tasks chosen cannot corroborate that the penalty is helping information flow deeper without harming generalisation. 1) Protocol Results may be misleading both because learning rates are not adjusted, which can cause deeper models to diverge while they could still be trained and because number of epochs is limited instead of letting the models train until convergence. I note that plots comparing train and validation accuracy in Figure 1 shows better results on the validation set than on the training set for ReLU and ReLU + BN. This suggests indeed that the optimization diverged, breaking the training accuracy while leaving the validation accuracy close to random. It is difficult however to evaluate the color-maps without an axis for the colors, so my assumption that dark purple is close to random accuracy may be wrong. A related note, description of Figure 4 in section 4.2 discuss about the achieved maximum performance at depth 60 and width 25. I could not read the same, to me it seems to be achieved right at depth 1, width 3. 2) Comparisons of zero initializations The experiments on the zero initialization scheme is an interesting investigation, but it is incomplete as there is no direct comparison with training without Sep-UP. The Annealed Dropout is important to allow training with zero initialization, and may well be more important than Sep-UP. We cannot measure their respective importance without a direct comparison as in Figure 1. 3) Too simple tasks The Figure 3 points to an important issue of the paper. Generally, the depth of networks is shown to provide higher capacity which leads to improved accuracy when not overfitting. In the experimental setup of this paper, the tasks and architectures do not improve with depth, or more precisely the improvement in capacity (training accuracy is maximal at depth 10) is associated with overfitting. If empirical results show that the contributed penalty helps improving the flow of information, it should also be shown that it is not at the cost of better generalisation otherwise there is no point in having greater depth. The chosen experimental setup does not allow this however. Even though it does fair better at generalization than the others, they all suffer from worst generalisation with depth. More difficult tasks may be better suited for these experiments. I am not recommending to use very large datasets such as ImageNet, difficult synthetic problems could be sufficient. It is necessary however that depth can be shown to be beneficial for baselines, so that we can confirm that training with the penalty is not degrading the beneficial effect of depth. Other comments The problem of dead points should be shown empirically since the contributions of this paper are supported empirically. Fourth paragraph of section 4.1 mentions results using convolutional layers, but it is not clear which of the provided results are on fully connected or convolutional networks. Minor comments In equation (1), hat{e}_j is not specified. My guess is that represents a basis, but I don*t see why this basis necessary in the equation. In equation (5), the notation is limited to a single layer, with x being the input of the layer. I believe this should be generalized with a composition operator as in equation (7). There seems to be a mistake in equation (8). The inclusion in X on the right is useless since the set on the left is an intersection with X. Section 3.1, third paragraph: It is *easily* to see [...] Section 3.2, first paragraph: [...] slack variables for *each each* point on the batch. Section 4.1, first paragraph: [...] (see Figures 1b and *1b*). (should be e) Edit: I still believe this paper lacks empirical evidence that the proposed loss would help train deep models without harming performance on tasks where depth matters. In light of all the corrections and clarifications, I upgrade my vote to weak reject.