Superseding Model Scaling by Penalizing Dead Units and Points with Separation Constraints
Authors: Carles Riera
Camilo Rey-Torres
Eloi Puertas
Oriol Pujol
Venue:  (2020)
Keywords: Dead Point
Dead Unit
Model Scaling
Separation Constraints
Dying ReLU
Constant Width
Deep Neural Networks
Backpropagation

Abstract: In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense connections, data normalization or model scaling. We accomplish that by alleviating two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is known to the academic community as \emph{dead neurons}. The other is a less studied problem, dead points. Dead points refers to data points that are mapped to zero during the forward pass of the network. As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant or even zero values and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.

Under review as a conference paper at ICLR 2020
SUPERSEDING MODEL SCALING BY PENALIZING DEAD UNITS AND POINTS WITH SEPARATION CON- STRAINTS
Anonymous authors Paper under double-blind review
ABSTRACT
In this article, we study a proposal that enables to train extremely thin (4 or neu- rons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense con- nections, data normalization two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is also known dead neurons. The other is a less studied problem, dead points. Dead points refers to data points that are mapped zero during the forward pass of the network. As such, the gradi- ent generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant values even zero and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.
model scaling. We accomplish that by alleviating
INTRODUCTION
The success of Deep Neural Networks (DNN for short) is linked to its ability to learn abstract representations from input data in a hierarchical fashion (LeCun et al. (2006); Ramachandran et al. (217 However, the concepts of depth and width in networks are often used as instrumental elements to address different DNN pathologies during the learning process. Examples of these are: vanishing gradient (Hochreiter/ (091], exploding gradient (Pascanu et al. (2013)) dead units MMas (2013); Douglas & Yu (2018); Guerraoui et al. (017)), or the degradation problem (He al. 2015b)).
In order address the former issues, we find many methods and techniques that we can roughly classify in two families: data manipulation and architectural modifications. The most commonly used data manipulation technique is data normalization on the output of the layers, for example using batch normalization (Toffe & Szegedy (2015)). Examples of architectural modifications include the use of additional connections, as done in ResNets (He et al.) (2015b)) or DenseNets (Huang et al. (016)); unit augmentation, in leaky-Re LU (Maas (2013)), PReLU (He et al.) (2015a)), C- ReLU Shang et al. (2016)), or linked neurons (Riera Pujol| (2017); or the increment of layer's 's width with depth (Zagoruyko & Komodakis (2tiaaakii(((()))))) (001)) et al. (2014)). This last widely used approach effectively increases the the size of the network requiring larger computational power. This has led several works (Hasanpour et al. (2018); Tan & Te (2019) that address it by offering heuristics how to scale the network. Recent research (Liu et al. (2018) suggests that this inverted pyramidal architecture is not optimal.
Besides of the width scaling strategy, the concept of dead unite is of particular interest to this work. A dead neuron is defined the neuron with constant or zero output for all training data points. This effectively renders this unit ineffective during the learning process. In (Lu et al.) 2019) it is shown that as the depth increases the probability of finding dead neurons also increases, to the point
In this work we wiit and neuron indistinctly.

---

Under review as a conference paper at ICLR 2020
that the entire network can be dead even initialization. Additionally, as expected, the width of a layer increases, the probability of having dead neurons decreases.
In this article characterize another pathology, the dead point. The dead point is dual concept to the dead neuron. A dead point corresponds to a data point that does not reach the output of the network. This is, the activation is zero for all the units in given layer. As result, this data point will have no influence the training process. As in the case of dead units, both condition are not recoverable by pack-propagation. This virtually reduces the size of the training set.
In order to solve the former issues, we propose and present geometrical optimization constraint that is added the loss function. As result of adding this constraint we ensure that all neurons are active and no points are dead. The rationale behind the geometrical constraint is to control that all units/ncurons and data points are alive by constraining pre-activation values in such way that the hyperplanes associated to the non-linearity of each ReLU neuron separates, at least, one data point from the rest of the dataset. This has the additional effect of enabling the learning process to propagate all information through all the network avoiding the instrumental need of using additional connections or inverted pyramidal architectures.
We test our proposal in series of controlled experiments to showcase the effect of applying the proposed Separation Constraints. We find that we can arbitrarily increase the network depth using the same constant width when compared to standard feedforward network and Batch Normalization lofte & Szegedy (2015) networks even when the width of the layer is extremely small (4 or neurons). We additionally provide evidences that using this same approach can initialize network parameters to zero and still achieve reasonable performance. These promising results gives insight on learning dynamics and suggest potential lines for future checks and research.
The article is organized as follows: Section 2 introduce the characterization of dead neurons and dead points, Section introduces the two geometrical constraints that ensure that neurons and data points are alive, describee the experiments and result, and, finally, Section concludes the paper and suggests future lines of research.
CHARACTERIZING DEAD NEURONS AND DEAD POINTS
A standard feed-forward ReLU DNN (LeCun et al. 2015 F can be formally written multi- valued real function, F(x), that is created by composing a collection of D vector layer functions f: 1 + RR Layer l is defined the sum of collection of scalar functions (units):
(x
u (x)è,
that affinely depend on a weight vector wh E R and a bias parameter by R. When using rectified linear units this value is truncated on negative values:
xx(x) = ma w x x 65).
Considering the hyperplane defined by w x + by = each unit defines partition of the space I in two sets: the upper part of unit uz and the lower part of uf:
upper (u²) = {x: (/:
x + 1/ > 0}
(3)
lower(uş) {x: w/ x + b2 < 0}
We define the affine component of layer function la as the intersection of the upper parts of its units, and its zero set the intersection of the lower parts, as follows:
12
A(lx) n upper(ut))
Z(ex)
n lower(u f=1
(4)
Remark 2.1 (Dead unit). In a given ReLU -DNN F R" + IR we say that the j-th unit of layer l u/ is dead with respect to data set X c R if and only if
X c Cowerruy))
(5)
2

---

Under review as a conference paper at ICLR 2020
Observe that if unit is dead, the output of the unit will be zero for the entire dataset, rendering the unit useless. Moreover, since the gradient zero as well, it remains in this state for the rest of the training (See Lu et al. (2019); Shin & Karniadakis (2019)). This effectively reduce the network learning capacity.
Remark 2.2 (Dead point with respect to a layer). Given ReLU-DNN R + IRk we say that point X is dead with regards to layer & if
E Z(Ex).
(6)
Remark 2.3 (Dead point). In particular, if C I", we say that point x is dead with respect to network F with depth D if it gets mapped to the zero set of a layer in its transit through the network. This is
(3k. < Di4x-1 o... 0...11...,1110 ( Z(ex)).
Any dead point fulfilling Equation 7 will show zero gradient in the layers previous to l.- This hinders the learning process by effectively reducing the data set size. Again, this condition is not reversible using standard back- prakk-rrpagaionn In a similar fashion to the case with dead units, the probability of finding dead point increases with network's 's depth and decreases with layer's ' width.
INTRODUCING SEPARABILITY CONSTRAINTS
In this section we introduce the desiderata for units and points to remain alive. Then, we proceed to formulate the separability contraints that fulfill the desired conditions.
Let us introduce the concept of a separating unit with respect to an arbitrary set X.
Definition 3.1 (Separating Unit). Given an arbitrary set X C TR we say that the j-th unit on layer k. u is able separate through X if the following predicate is satisfied:
P& (u 0 Allworrrrr nx) CX
(8)
Thus, by construction, separating unit can not be dead, and if Rx Rx(u() is valid, uz can not degrade set X to zero. In other words, this condition ensures that each unit always separates at least one data point.
In terms of points, we can define a separating point as follows:
Definition 3.2 (Separating point). Given an arbitrary set C. R", we say that point x X is separating layer function Ea if there exist indices j, E (l....,mu. m.. for which the following predicate is satisfied.
RE = (upper(ul) lower (ut)r
Again, by construction, a separating point ensures that each point in the data set has at least one unit in each layer with an activation different to zero and another with an activation equal to zero.
3.1 MODELLING UNIT-BASED SEPARATION CONSTRAINT (SE?-U)
Unit based separation contraint (Sep-U) U) is designed to model predicate g with the goal of avoiding the presence of dead units.
Given a unit w in layer l R + R from a ReLU DNN F of depth D, predicate g can be simply modelled imposing the following constraints,
max (w x; + by} > 0 t=.....|
(10)
(w) x; + by 0 [=.....|
min
These strict inequalities can not be directly optimized. It is easily to see that these can rewritten
3

---

Under review as a conference paper at ICLR 2020
max [w] + 6 21 i=....,
(11)
min {w] x; + 5
The of the former constraints makes most problems unfeasible. Thus, in similar fashion to soft-margin SVM (Cortes & Vapnik 1995), we introduce set of positive slack variables (Ejke Spp) that account for constraint violations as tollows,
max {w x; + b1} 2 1 - Ejh. i=
min
{w] x; + byt -1 Eyk-
(12)
Sjk Syk 2 0,
for 1,..., D and ny the number units of layer k. The intuition behind the introduction of these constraints is follows: by minimizing ç+ at least one pre-activation value is forced to be greater (or equal) than 1. Simmetrically, minimizing { promotes at least one pre-activation to be below -1. This effectively fulfills Predicate g and penalize the apparition of dead units. We show how to differentiate the Separation Constraints with regards to the parameters in the Appendix Section A.3
At a global network scale, we can aggregate all the slack variable in a single optimization objective follows,
n G5tt (55))
gEt (
(13)
3.2 MODELLING POINT BASED SEPARATION CONSTRAINT (SER-P)
The derivation of the Point Based Separation Constraints (Sep-P) follows parallel process Sep-U. In order to avoid the presence dead points suffices to fulfill Predicate 9 Similarly to the former derivation, we introduce given X; E X, and ...... ue unit functions in layer la. we define slack variables E Sik in the context of the following constraints.
of slack variables for each each point on the batch. That is, et
max {w) x; + b/} 2 E j=1...,
min (wf x; + 1// S Etk. j= 5 Ejk 0.
(14)
Observe that the minimization of the slacks makes that for any data point at least one activation is above and another is below -1.
We can summarize all the point-based slack variables in a single optimization objective as follows,
gp(E+, (ET)
Σ
ST.
(15)
3.3 TRAINING WITH SEPARATING CONSTRAINTS
The new optimization objectives can now be added to the original loss objective using a simple scalarization (Boyd& Vandenberghe (2004)) as follows,
arg min L(T, 0) (g ,ET) + (o((ttt57)) 0
(16)
where À is a main loss function.
that introduces a trade-off between the constraint fulfillment and the
4

---

Under review as a conference paper at ICLR 2020
In terms of memory complexity, the former constraints introduce very small memory overhead. In particular, places a pair of constraints on each of the units of the network, so the com- plexity with respect to Sep-U scales with the size of the network as 2 E Tuk. Alternatively, Sep-P places a pair of constraints in each of the points of the data set or selected subset X. In prac- tice, one can use X as the training batch. Thus, the memory complexity scales with the size of the batch and the number of layers, i.e. 2D Furthermore, since the resulting gradient of both types of constraints depends only on the input of the layer that is already computed in the forward pass, we only add the cost of storing the slacks. Therefore, the total cost in terms of number of constraint is the addition of the former terms, i.e. 2 >k==1 n4g + +
4 EXPERIMENTS AND RESULTS
In this section we explore the application of the proposed constraints in different datasets. For that task we train all methods with different choices of depth and width parameters. The network architecture used is rectangular, i.e. networks with fixed layer width for all the layers.
Datasets: Due the large amount of computational resources required for the depth and width grid training, we are forced to chose three controlled datasets in our experimentation: the MOONS dataset (sampling 100 points, 85 for training and 15 for validation), the MNIST dataset as described in LeCun and Cortes. LeCun & Cortes (2010) and the CIFAR-10 dataset described in Krizhevsky (2009).
Experimental setting: We compare the combination of Sep-U and Sep-P (Sep-UP from now on) to feed-fordward ReLU networks (Glorot et al. (01I]) and Batch Normalization as described in loffe & Szegedy (2015) using the same architecture.
For the MOONS dataset, we use depths from 1 to 120 in steps of 10, and width from to 25 in steps of 1 between 1 and 5, and steps of 5 afterwards. In the case of the MNIST dataset, use depth from 2 to 64 and width from to 8 in steps of 4. Finally, for CIFAR- 10 we use depths in {2. 10, 25, 30, 40) with fixed width of 10 due memory constraints.
Training Parameters: All the networks used were optimized using Adam (Kingma & Ba 2014 More specifically, for the MOONS dataset we used learning rate of 0.01 for 5000 epochs and a batch size of 85. Meanwhile, for both MNIST and CIFAR-10 we used a learning rate of 0.0001 for 50 epochs and a batch size of 1024. We used convolutional layers with (3,3) kernel size for both MNIST and CIFAR 10. We used AMOONS = 10 AMNIST ACTFAR = 10 for Sep-UP. UP, Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al. 2015), fixing the random seed to an arbitrary value of 10.
As initialization scheme, we used Glorot uniform from Glorot Bengio 2010) for all the methods and datasets.
4.1 RESULTS
Figure I shows the results obtained in the MOONS dataset. Our proposal Sep-UP is able to train networks successfully without increasing the width up 60 layers deep (see Figures Ic and IT). while ReLU breaks down at only 30 layers (see Figures la and ld) and ReLU + BN suffers from
severe accuracy degradation (see Figures I (b).
Observe that no configuration with lower width than 2-3 is successful in achieving maximum accu- racy, We understand that there exists a minimum width required and this is related the complexity of the problem. When using wider layers, the rest of the width is instrumentally used to enable the training of deeper networks. As previously commented, the larger the layer's width, the higher the chances of finding active units that do not cause dead points and dead units during initialization. In opposition, Sep-U2 succesfully overcomes that constraint.
Notice that though Sep- UP is superior to all its competitors, it starts showing performance degra- dation after reaching depth 60 with the minimum width of 3. Considering that the number of param- eters increases with the depth and width of the network and that we have finite number of trained epochs, we conjecture that the displayed degradation is strictly due to the lack of convergence the constrained network.
5

---

Under review as a conference paper at ICLR 2020
1.0
1.0
1.0
HNwa 00
25
20
15
Width
0.8
10
0.8
0.8
0.7
0.7
-0.6
0.6
0.6
120
30
60 Depth
90
120
30
120
Depth
Depth
(a) ReLU train
(b) ReLU BN trainin
(c) Sep-UP UP train
1.0
1.0
1.0
25
25
20
20
0.9
20
15
Width
10
0.88
10
0.8
10
0.8
0.66
0.6
-066
120
30
60
90
120
120
Depth
Depth
Depth
(d) ReLU validation
(e) ReLU +
validation
(f) Sep-Up UP validation
Figure 1: Depth width accuracy heatmap a for grid of rectangular networks with width from to 25 and depth from 1 to 120, trained using Adam with learning rate of 0.01 in the MOONS dataset for 5000 epochs. The color shows the accuracy attained of each the combinations of width and depth, with clear beige at and black at 0.5. Notice how ReLU breaks down at 20 layers and ReLU BN requires more units per layer as increasing depth, while Sep-UP works with the minimum width (3) up to 60 layers deep.
1.0
1.0
1.0
0.8
0.8
0.8
Width
Widtt 2
0.4
12
36 Depth
52
60
68
0.2
28 Depth
44
52
60
-.2
20
Depth
28
44
68
0.
ReLU training
(b) ReLU
training
(c) Sep-UP training
wiath
0.6
Width
0.6
Whdth
0.6
-0.4
-04
004
12
28
Depth
-0.2
12.20
Depth
60
-02
28
Depth
-02
(d) RelJ validation
(e) ReLU BN validation
(f) Sep-UP validation
Figure 2: Depth vs width accuracy heatmap a for a grid rectangular networks with width from 2 to 8 and depth from 2 to 68, trained using Adam with learning rate of 0.0001 in the MNIST dataset for 50 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at 1 and black 0.1. Notice how ReLU breaks down at 20 layers and ReLU + BN accuracy degrades with depth, while Sep- UP shows constant accuracy disregarding the number of layers.
The separation constraint also proves successful on convolutional networks, as tested in MNIST and CIFAR-10 datasets. Figure 2 shows similar behaviour to the MOONS dataset (FigureT). ReLU breaks down after few layers, ReLU + BN delays the degradation of accuracy, while Sep-UP remains functional regardless the depth. In the case our experiments with CIFAR-10 dataset (as presented on Figure 3 all the methods degrade with depth, but Sep- UP is the most robust. regards to accuracy ReLU performs best closely followed by Sep-UP, while ReLU + BN clearly overfits. The poorer accuracy values shown due to limited choice of width, clearly inferior to the minimum required by the dataset, and a potential lack of convergence of the separating constraints.

---

Under review as a conference paper at ICLR 2020
ReLU
ReLV
ReLU-BN
0.91
0.82
0.70
0.67
0.60
ReLU-BN
0.57
0.54
0.54
0.53
0.45
Sep-Up
0.69
0.79
0.74
0.70
0.68
Sep-UP
0.59
0.57
0.54
0.53
0.53
10
25
25 Depth
Depth
(a) Training accuracy
(b) Validation accuracy
Figure 3: Depth Width accuracy heatmap for grid of rectangular networks with width 10 and depth from 2 to 40, trained using Adam with a learning rate of 0.0001 in the CIFAR-10 dataset for 50 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at and black at 0.1. Observe how Sep-UP shows inferior degradation in accuracy depth increases compared to ReLU + BN.
Additional results in the Appendix further elaborate the contributions of each term Sep-U and Sep-P, independently.
4.2 RESULTS USING ZERO INITIALIZATION
In order to test the invariace to initialization scheme of Sep UP, use Zero Initialization. As its name states, in this initialization scheme all weights and biases are set to zero. However, a small variation of the scheme must be introduced in order break symmetry for the constraints apply. Since all the units are initialized to the same value (zero), we use Annealed Dropout (Rennie et al.) 2014). Additionally, instead of adding ç+ and & pairs in Equations 13 and 15 combination with a small perturbation In our experiments, we use a value of 0.01.
+ + + -PE
(17)
Figure summarizes the results found using zero initalization with our constraint formulation. In comparison Glorot, observe that zero initalization requires wider networks. Indeed, at depth of 60, Zero initialization requires a width of 25 units while the Glorot scheme work with only units. However, we contrast ReLU BN to our constraint formulation we find that it is unable to train networks of past depth 50, while Zero initialization achieves 70 layers.
Figure A shows the results using zero initialization. Although reported values and behavior slightly worse compared to Glorot initialization. our results are promising. Notice how at depth of 60 layers, the zero initialization requires width of 25 units contrast required for Glorot. In addition, zero initialization rquires only 70 layers reach perfect accuracy in contrast to the 100 required Glorot (see Figures Tc and Tf). Moreover, in comparison to ReLU BN, zero initialized networks show superior performance and behavior (check Figures lb and lc
5 CONCLUSIONS
Through the Separation Constraints, we have shown that deeper networks can be trained without increasing the width the layers. Moreover, this increment can be done using very small width values. In this sense consider that effective training of deeper networks can be achieved by better accommodating the network to the input data. This departs from many proposal that achieve similar effects by modifying the architecture of the network or manipulating the data. We believe that this work shows an alternative research path in the pursuit of effective and efficient learning techniques for deep neural networks. This also opens the possibility of avoiding the use of pyramidal

---

Under review as a conference paper at ICLR 2020
1.0
1.0
25
Width
0.8
0.8
0.7
0.7
-0.6
120
60
90
120
Depth
Depth
(a) Sep-UP UP Zero-inie train
(b) Sep-UP UP Zero-init val.
Figure 4: Depth vs width accuracy heatmap for grid of rectangular networks with (width from to 25 and depth from to 120), trained using Adam using learning rate of 0.01, over MOONS dataset (5000 epochs and Zero Initialization). The color shows the accuracy attained of each of the combinations of width and depth, with clear beige at and black at 0.5, Notice how although being inferior to Glorot (Figures Tc and Tf is superior ReLU
BN (Figures Tb and
Te
architectures commonly employed in DNNs, thus removing the computational burden caused by the additional units and reducing the dimensionality of the internal representations.
We additionally show that our proposal enables the use of Zero Initialization. The results are promis- ing, and though further experimentation is still needed, they still manage surpass Batch Normal- ization in terms of depth, width, and accuracy. Nevertheless, the modifications used to break the symmetry of constraints and units need further consideration in order to achieve the same perfor- mance than random initialization.
Despite this article provides no information about the dynamics of the training process using Sepa- ration Constraints, preliminary revision shows interesting properties. In particular, training displays slight instabilities which appear as the units/points are reactivated. Although this does not hinder the training process, further study is needed in order to guarantee smoother convergence.
The extension of this work to other activation functions still be explored. However, we conjec- ture that in cases where the activation function display flat region, e.g. ELU, tanh, even sigmoid, current proposal can be applied with minor changes.
Finally, while the separation constraints prevent the vanishing gradient effect, the exploding gradient problem still remains. Extending the Separation Constraint with an upper bound on the magnitudes of the pre-activation, similar e-insensitive loss, might address it. It could be also helpful to explore other activation functions whose gradient vanishes with high pre-activations, such as the logistic family.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens. Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https://wwwtt www.rrfonsor.oooo org/ Software available from tensorflow.org.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004. ISBN 0521833787,
François Chollet et al. Keras. htttps:/keerss. 2015.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-29) 1995.
8

---

Under review as a conference paper at ICLR 2020
Scott C. Douglas and Jiutian Yu. Why relu units sometimes die: Analysis of single-unit emror backpropagation in neural networks. CoRR, abs/1812.05981, 2018. URL httt:://axxii
htg/abs/1812. 05981
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATSIO). Society for Artificial Intelligence and Statistics, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Ge- offrey Gordon, David Dunson, and Miroslav Dudk (eds.), Proceedings of the Fourleenth Interna- tional Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learming Research, pp. httt::/prroceedi: ngs m www.pres//vlll. orot 1att1mla..mlll
Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL
Rachid Guerraoui et al. When neurons fail. In 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 1028-1037. IEEE, 2017.
Seyyed Hossein Hasanpour, Mohammad Rouhani, Mohsen Fayyaz, Mohammad Sabokrou, and Ehsan Adeli. Towards principled design of deep convolutional networks: Introducing simpnet. CoRR, aws//182..0200000 2018. URL
org watss//80....0 06205
Michael Hauser and Asok Ray. Principles of riemannian geometry in neural networks. In NIPS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. CoRR, abs/ 50000150..0185, 2015a. URL
.01852
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recog- nition. CoRR, abs/ 512.0333,,, 2015b. URL URL.stte///arxx...rr//bb//////
Sepp Hochreiter.
Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Uni- versität München, 91(1), 1991.
Sepp Hochreiter. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. 2001.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR,
2016. /oLLiitppp///rxiioooo///ass//
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http:://axxi...rrrr//.
abs/1502. 03167
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. CRLhttp:// /ttp///avvvv/..////v//eee/eaaav/ borg/abs/1410 6980
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL/ttpp://yan:
lecun com/ /e////ddbbmnniiee
Yann LeCun, Sumit Chopra, and Raia Hadsell. A tutorial energy-based learning. 2006.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep leariin. Nature, 521(7553))436444 May 2015. ISSN 0028-0836. doi: 10. 1038///turee45399 URL /ttp://xxx.....//.....
1038/nature14533
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. CoRR, abs/18 1oo.1800,,,270, 2018. URL
05270
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Kamiadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv: 1rXiv:/903.0677 2019.

---

Under review as a conference paper at ICLR 2020
Andrew L. Maas. Rectifier nonlincarities improve neural network acoustic models. 2013.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning- Volume 28, ICML'13, pp. I11-130--11-111..10 JMLL.. 2013. URLhttp: 7/dl. acm. org/ /iitatiitnnnnoo fm? wtt.:ooo...mmmmmm121201e 3042811 3043083
P. Ramachandran, B. Zoph, and Q. V. Le. Swish: Self-Gated Activation Function. ArXiv e-prints, October 2017.
Steven Rennie, Vaibhava Goel, and Samuel Thomas. Annealed dropout training of deep networks. In 2014 IEEE Spoken Language Technology Workshop (SLT), pp. 159-164. IEEE, 2014.
Camilon Rey-Torres, Carles Riera-Molina, and Eloi Puertas-Prats. A naive explanation model for relu based classification phenomena, 2019. to be released on ArXiv.
Carles Riera and Oriol Pujol. Solving internal covariate shift in deep learning with linked neurons. arXiv preprint arXiv: 111x:v:7122.00099 2017.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. CoRR, (tss/6603.0201,, 2016.
Yeonjong Shin and George E. Karniadakis. Trainability and data-dependent initialization of over- parameterized relu neural networks. CoRR, abs/1907.09696, 2019. URL http://arxiv.
org/abs/1900 09696
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. hRL/tttp://arxvv.orgg wttpe/iiivvrrgaassoooa/oo/// wttp://////////////////////// 4842
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.1 1946, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/ lass//6050011 2016. URL http::pp/aaavv..rraaaa/////eeeee
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
UNIT BASED AND POINT BASED SEPARATION CONSTRAINT EXPERIMENT
In order justify the decision of combining Sep-U U and Sep-P into Sep-UP, provide Fig- ure 2 showing the same experiment than Figure ! for Sep-Uand Sep-U. We find how although Sep is able to train deeper and thinner networks it suffers from inferior accuracy (see Figures 5a and 5c), requiring the use of the layer width increase technique. Contrarily, Sep-P is able to train without increasing the width from 40 layers deep. but breaks down afterwards. Provided how complementary are their strengths and weaknesses, it is sensible to combine them, resulting in the superior performance (larger constant width area together with greater accuracy higher depths) of Sep-UP, see Figures
1c Gnnd Tf
1.0
1.0
25
25
20
0.9
20
0.9
15
15
wid
10
0.8
Wid
10
0.8
4
0.7
4
0.7
3
2
0.6
0.66
1
1
30
60 Depth
90
120
1
30
60
90
120
Depth
(a) Sep-Utrain
(b) Sep-P train
1.0
1.0
25
25
20
0.9
20
0.9
15
15
wioth
10
0.8
10
0.8
5
5
4
0.7
0.7
2
0.6
1
0.6
30
60
90
120
30
60
90
120
Depth
Depth
(c) Sep-U validation
(d) Sep-P validation
Figure 5: Depth vs width accuracy heatmap a for grid of rectangular networks with width from 25 and depth from 120, trained using Adam with learning rate of 0.01 the MOONS dataset for 5000 epochs. The color shows the accuracy attained of each of the combinations of width and depth, with clear beige and black at 0.5. Notice how Sep-U enpblen train thinner and deeper networks (120 layers of single umit), but with reduced performance compared to Sep-UP It also requires increasing the width of the layers in order to achieve perfect accuracy as ReLU BN. Alternatively, Sep-P shows constant width area between 1 and 40 layers, but its performance is much worse above. This justifies the decision of combining them both into Sep-UP UP to combine their strengths.
A.2 PROBABILITY OF A POINT TO BE DEAD IN NEURAL NETWORK
We conjecture that difficulty arising when training neural network as the depth increases or the width decreases, is due the incorrect positioning of the hyperplanes of the units after initialization which causes the apparition dead points (recall Remark 2.3), that ultimately break back- pack-propggatio.. In this section offer a formal treatment of this idea.

---

Under review as a conference paper at ICLR 2020
Theorem A.1 (The probability of a point to be dead increases with depth and decreases with width). Let F R + RDD be a feed-forr ferdaforward neural network composed by a collection of D E N layer functions & I 1 + R with ic 1..... D, denote D as the depth of the network. The layers are composed by a collection of ny E N ReLūunit functions u : R + R4, we denote ng as the width of the layer. Let the network be initialized following random distribution. Then, given point x E I. we say that the probability of the point dying (Recall Definition 7 increases with D and decreases with Ti.,
Lemma A.2 (Probability of point to be dead for a ReLU unit follows Bernoulli). Let be a point X E IR' and a ReLU unit u parametrized by a weight vector w E R" and a bias E IR initialized following random distribution, then the probability of point x being dead follows Bernoulli distribution of unknown parameter p.
P(x E lower(u)) ~ Berroulliiy)
(18)
Proof. Since the unit is initialized randomly according to a given distribution (i.e. normal, uniform, Glorot, ...). the probability of its hyper-plane parametrized by w, b facing is unknown p accord- ing to the distribution used. Therefore, the probability of x belonging to lower(u) (Recall Equation 3 must be p.
As the lower set (Recall Definition 3) pre-activation is truncated, its gradient with respect the pa- rameters will be zero (See Reyr-Toress et al. (2019 Section 2)). Therefore, any point belonging to lower(u) will be dead in regards to u.
Lemma A.3 (Probability of a point to be dead for a ReLU layer). Let Eg be a layer composed of n& E N ReLU units initialized following the same random distribution. Let be point E R=* 1, Then the probability of x being dead for lx is:
P(x E Z(Ex)) = P(x E lower)u)))**
(19)
Proof. Similarly to the unit case, points belonging to the zero set of a layer (Recall Definition 4 have zero gradient with respect the parameters of the layer due the truncation (See Rey- Torres et al. 2019 Section 2)).
The probability of point belonging to the zero set of a layer P(x E Z((x)) is equal to the proba- bility of belonging to all the lowers of the units of the layer P(Vu E f|x E lower(u)). Since unit
parameters are sampled independently during initialization and the probability of point being dead
for a unit follows Bernoulli, then the probability of a point belonging to Z(4) is the product of the
probabilities of belonging to the lower of each of the units.
O
Lemma A44 (The probability of point to be dead for a layer diminishes with its width). Let la be a layer composed of n& E ReLÚ u initialized following a random distribution. Let be
point x E IR Then the probability of x being dead diminishes with ng::
lim P(x E Z(0)== 0 78g +00
(20)
Proof. The probability of a point being dead for a layer is the multiplication of the probabilities of being dead of each of its units, recall Lemma A.3] Therefore, as the probabilities are lower than since the upper set of a unit is never empty. in the limit where ny tends to infinity the probability tends zero
Note how Lemma A.4 4 can explain why increasing width allows us to train deeper networks (See Hasanpour et al. (2018) ; Huang et al. (2016)), it simply improves the chances of finding hyper- plane that allows the points to live.
Definition A.1 (Dead point set in a neural network), Let F: RT + RnD be a foedfforrard neural network composed by collection of D E N layer functions e R=--- - R with k E 1...., D, we denote D as the depth of the network. The layers are composed by a collection of ny E N ReLUunit unit finctions u BR - R,, we denote n& as the width of the layer. Then, we define the
set of dead points of a neural network following Equation 2 as.
7(F)= (3k|1 k D 6--1- o (x(x) E 7(ex)) xx
(21)
12

---

Under review as a conference paper at ICLR 2020
Lemma A.5 (Probability of a point to be dead for ReLU feed-forward network). Let F I RTD be a feed-forward neural network composed by a collection of D E N layer functions R + T with & ................................ D. we denote D the depth of the network. The layers are composed by a collection of ng E N RetUunit functions u RRax-1 + R2, we denote n& as the width of the layer. Let the network be initialized following random distribution. Then, given point X E IR we say that the probability of the point dying (Recall Definition 7 is
P(x & Z(F)) = II P(x E ((4)))
(22)
Proof. The probability of not dying a given layer is equal to minus the probability of dying.
P(x E Z(0) = =1 1- P & Z()))
(23)
As the sampling of the parameters during intialization independent among layers, the probabiliity of not dying in any of the layers of the network is the multiplication of the probabilities of not dying each of the layers.
P(x Z(F) II P(x & Z(4)))
(24)
The probability of point to be dead for a network is equal to the probability of being dead at least in one layer, see This is equivalent to 1 minus the probability of not dying in any of the layers.
P(x E Z(F)) = 1 Z(F))
(25)
Plugging Equation 24 into Equation results in Equation 22
Lemma A.6 (The probability of a point be dead for layer increases with its depth). Let F I + nnD be feed-forward neural network composed by collection of D E N layer functions R= + g with k E l,....,. D, we denote as the depth of the network. Let the network be initialized following random distribution. Then, given point R. say that the probability of the point dying as traverses the network (Recall Definition 7 increases with D.
lim D
P(x E Z(F)) =
(26)
Proof. Since the hyper-planes of the units split the space in two half-spaces, the probability of point not being dead always lower than
P(x & Z())) < 1
(27)
Therefore, the limit of the probability of x not dying in any of the layers when D tends to infinity is zero, recall Equation 24
lim P
P
(1 P(x z(()))) =
(28)
D-500 =1
Which when plugged into Equation 22 leads to Equation 26
13

---

Under review as a conference paper at ICLR 2020
A.3 ON THE OPTIMIZATION OF THE CONSTRAINTS
In this section we show how to differentiate the Separation Constraints with regards to the weights. Notice that under formulation,
examining the diameter of the preactivation of a dataset X. That is, given a unit u with vector parameters w E IR" and b E R, its preactivation is given by the function
Z = p x
(29)
given a finite discrete set X C I" ( a compact set of IR"), we can define the extreme values of X with regards to
min p(x) XEX
Pmin
(30)
maa
maxp(x) XEX
on similar argument, given a layer & with units u1, u2..... Ung with pre-activations denoted by P1, P2.... ................................ Prgz define the extreme values of layer e provided with fixed x X as
min (x(x)
min
j=I...m
(31)
max Px(x)
maz
j=
notice that since a finite discrete set of TM, min mar exist (where stands for u or p). This fact is instrumental in our Equations (10) to (12). Since X is bounded, and we are working in the real number system, we can find numbers ç4 E such that
zmii
+ 1 E
(32)
maa
notice that the parameters w E Th and are free in the definition of 2", while x is free in the definition of 2P, In this sense, for L we can find w E R. and by E R such that
c = max(1 (w+ x b+), )) _ max(1 + (w_ x b_),0)
(33)
that constraints all possible configurations of parameters in units of L. Naturally, & depend on w and b. This is at the heart of the constraint formulation: choosing w and b that 2min 0 and Zmar > 0.
Meanwhile, given fixed selection of and b, since X C R is discrete and bounded, we must have that there exist x4 and x_ points such that
<" max(1 (w x + b). max(1 + (w x + 0,
(34)
Ex
We wish to stress a geometric intuition, beyond calculations. We seek a combination of w and that they "cut' through X in the of Sep-U, noticing that the preactivation p defines signed distance from the separating hyperplane p(x) = 0.
The same argument is used in the field of Support Vector Machines, as cited in our references. particularly Vapnik & Cortes' Vector Support Networks from 1995 and Boyd Vandenberghes' Convex Analysis from 2004.
A.4 ON THE RELATION BETWEEN CONSTRAINTS AND PARAMETER UPDATE
Notice that up to Equation 12 we have only stated constraints upon the slack variables & (that involve the dataset and parameters and b). In Equation 13 we introduce the objective function to optimize the values f.. Namely, the sum of all ç.
14

---

Under review as a conference paper at ICLR 2020
Since we have different indications of the Ç. must define different objective functions for unit-wise constraints or gu of Equation [3] and point-wise constraints or gP from Equation 15 We presented the construct for unit-basch constraints (transit between Equations 12 and 13 and extended the notation for point-wise constraints on Equation 15
If we wish to differentiate with regards to w and equations describing the relation between and unit-wise or point-wise, we must notice that C as stated, must be defined by parts:
x== "=
p(x)> -1
(35)
0,
otherwise
054
11,
p(x_)>-1
(36)
06
0,
otherwise
while for ex:
X4+
p(x+) < 1
V 54
(37)
0,
otherwise
-1,
p(x+) <1
(38)
06
0,
otherwise
while under the unit-based formulation x fixed by the geometry of X, for the point-wise formu- lation, we must have
p(x) -1 0, otherwise
x,
V
(39)
[1,
p(x) > -1 otherwise
(40)
Ob
10
and for c.:
-xx
p(x) <
V wi
cP
(41)
0,
otherwise
0c5 Øb
11,
p(x) <1
(42)
0.
otherwise
that is parallel to the loss gradient L.
A.5 EFFECT OF THE SEPARATION ON THE INTERNAL REPRESENTATIONS
In this section we explore the effect of the proposed constraints in the activation of the network. For that task we train all methods fixing the architecture at 50 layers of units each. We use the MOCNS dataset for easy visualization, in the same configuration than Section ssmpllig 100 points, 85 for training and 15 for validation).
Experimental setting: We compare the combination of Sep-U, Sep-P and Sep-UP to feed- fordward networks (Glorot et al., [011) and Batch Normalization as described in loffe & Szegedy (0155) using the same architecture.
2014). We used learning rate of 0.01 for 1000 epochs and batch size of 85. We used À 10 when required. As initialization scheme, we used Glorot uniform from Glorot & Bengio (0010). We also test the use of the Separation Constraints alone, without categorical crossentropy, assess its effect by itself. Our experiments were conducted using Keras (Chollet et al.) 2015 and TensorFlow (Abadi et al. 2015), fixing the random seed to arbitrary value of 10.
Training Parameters: All the networks used were optimized using Adam (Kingma
Table I summarizes the results. Figures
and g show the activation plots for each of the
methods tested.
15

---

Under review as a conference paper at ICLR 2020
Aceuracy
Loss
Train
Val.
Train
Val.
ReLU ReLU
0.5176
0.4
0.6925
0.6938
BN
0.8117
0.6
0.6331
0.6636
Sep-t 0.9294
0.8000
0.1765
0.6476
Sep-U U
0.9058
0,8000
0,4161
1,5228
Sep-UP 0.9882
0.9333
0.6988
1.0810
Table 1: Maximal performance experiment using the MOONS dataset. From left to right, accuracy and loss (for train and validation sets) for ReLU, ReLU BN, and Sep-Cons in all its variants.
(b) 4th layer
(d) 25th layer
(f) Feature layer
(a) Input layer
(h) Output
(c) 4th layer
(e) 25th layer
(g) Feature layer
Figure 6: Data transformed across 50x4 ReLU classification network. Notice how the the dataset is progressively mapped to zero it traverses the network. This renders the output layer unable to solve the problem.
In terms of accuracy ReLU reaches trivial accuracy of 0.51 while ReLU BN reaches 0.60, both architectures fail to solve the problem, see Table When comparing the results presented in Figures and 7 we find that the fourth layer ReLU has collapsed the points of the dataset BN BN still manages to warp the dataset. However, both methods layer 25th have collapsed the dataset into small number of points (see Figures] 7d
line (parts 6b and 6c). while ReLU
7e for ReLU
BN and Figures 6d and be for ReLU) that is then pushed to the point zero for ReLU (as shown in collapses to few points (Figures
the feature layer: and while ReLU
This collapse is congruent with our expectations, displaying the born-dead network condi- tion from Lu et al. (2019). Note how layer after layer its effect is worse, as we predicted in A.2 In addition, since the gradient is only back propagated through the points lying on the upper sets of units, geometric collapse also stops learning. Notice that the standarization and the affine transformation (dependent on " and (3) from ReLJ + BN are not able to prevent from killing the dataset. Thus, training thin and deep neural networks is difficult with ReLU ReLU
BN.
On the extent of our experimentation ReLU networks featuring separation constraints able to solve the problem with higher degrees of accuracy than ReLU and ReLU + BN (see the figures in Table [ Moreover. Figures 8h ITh and 10h), show that the separation function behaves intuitively in the sense of Hauser & Ray (2017)
The internal representations not only are non-trivial (as in ReLU + BN but also preserve geometrical structures like shape and connectivity, as shown in Figures [Id [od or 8e Indeed, Figures
and 10c showcase how at the 4th layers much solution is already found. This proves that the gradient of the main loss is back packapropagated to the input, unlike ReLU and ReLU
Ic 8c
BN (Figures 6b
6b
7b
7c
The constraints enforce richer representations when used without main loss (cross-entropy),
16

---

Under review as a conference paper at ICLR 2020
(b) 4th layer
(d) 25th layer
(f) Feature layer
(a) Input layer
(h) Output
(c) 4th layer
(e) 25th layer
Feature layer
Figure 7: Data transformed across 50x4 ReLU + BN network trained using Adam with learning rate 0.01 in the MOONS dataset for 1000 epochs. The dataset is collapsed in few points the fea- ture layer. As the gradient cannot be backpropagated across the truncation after the affine transform of y and 3 despite the standarization, it fails in the same manner than ReLU only that with non-zero activations. This results in lopological mixing of the datasets. Therefore, the representational ca- pability of the network hindered to such extent that resulting output, although non-trivial, totally arbitrary.
(b) 4th layer
(d) 25th layer
(f) Feature layer
(a) Input layer
(h) Output
(c) 4th layer
(e) 25th layer
(g) Feature layer
Figure 8: Data transformed across a 50x4 Sep-UP network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The network displays internal representations without collapsing the dataset like Sep-P and retaining discriminative power like Sep-U.
17

---

Under review as a conference paper at ICLR 2020
(a) ReLU input layer
(b) RoLU 50th layer
(c) ReLU 50th layer
(d) ReLU EN input layer
(e) ReLU
50th layer
(I) ReLU
BN 50th layer
(g) SPp-UP Input
(h) SPp-UP 50th layer
Sep-UP 50th layer
Figure 9: Data transformed across 50x4 network with main loss (cross-entropy) with constraints
Sep L and Sep-UP, versus ReLU and ReLU BN. Notice how effectively ReLU and ReLU BN collapse the dataset into few points whereas Sep-UP force the network learn representations that preserve geometrical structure useful for back soeck-roppggtton.
Figure whereas the representation reached by Sep-UP is more complex, yet preserves connectivity. Furthermore, both avoid mapping the entire dataset to few values as observed with ReLU (Figures 9e and 9f) and ReLU BN (Figursss) 9b and 9c).
The only exception is Sep-U U that apparently solves the problem at layer 25th on Figures 1ld and [Te]] yet collapses the dataset in two points the feature layer (Figures If and TIn.. Our intuition is is that, although Sep-U U prevents dead units, it cannot prevent points from falling into the intersection lower set the units of a layer, resulting in a dead point). The Sep-P separation constraint was designed precisely to prevent the presence of dead points. However, Sep-P allows affine and dead units (as shown in Figures 10d and 10e), so the the decision function reached becomes too linear (see Figure 10h).
Therefore, if we combine both Sep-Uand Sep-P we should have best of both worlds, as we say in Section A.I Figure g shows how Sep-UP able to separate both classes perfectly (recall the 0.93 in accuracy from Table I]. Furthermore Sep-UP, produces feature layer that does not concentrates the dataset in two points like Sep-U, nor distributes the dataset linearly like Sep-P.
18

---

Under review as a conference paper at ICLR 2020
(b) 4th layer
(d) 25th layer
(f) Feature layer
(a) Input layer
(h) Output
(c) 4th layer
(c) 25th layer
(g) Feature layer
Figure 10: Data transformed across 50x4 Sep-P network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. The network displays richer internal representation without collapsing the dataset like Sep-U or ReLU BN. However, plenty of dead units appear since they are not penalized, causing underfitting.
(b) 4th layer
(d) 25th layer
(f) Feature layer
(a) Input layer
(h) Output
(c) 4th layer
(e) 25th layer
(g) Feature layer
Figure 11: Data transformed across 50x4 Sep-U network trained using Adam with a learning rate of 0.01 in the MOONS dataset for 1000 epochs. Notice how dead units have been reduced. The interal representations are much richer than ReLU ReLU + BN. Despite collapsing the dataset in two points at the feature layer, the classification performed in the output layer is approximately correct. We conjecture that this is due the dead point addressed with Sep-P.
19

---

Under review as a conference paper at ICLR 2020
A.6 RELATION BETWEEN SEPARATION CONSTRAINTS AND MAIN LOSS
In this section we explore the effect of the proposed constraints in the convergence of the network during optimization. We train network of 50 layers of 4 units each using Sep-Ur We the MOONS dataset, in the same configuration than Section (sampling 100 points, 85 for training and 15 for validation).
Experimental setting: We analyze the interaction between of the main loss (cross-entropy) and the loss arising from the Separation Constraints. We plot them, together with training accuracy in Figure 12 Below we attach the decision boundaries at the most relevant epochs.
Training Parameters: The networks wes optimized using Adam (Kingma & Ba 2014). We used learning rate of 0.01 for 17000 epochs and a batch size of 85. We used À = 10 As mitialization scheme, we used Glorot uniform from Glorot & Bengio (2010). Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al. 2015), fixing the random seed to an arbitrary value 10.
We see at Figure 12 how the use of the separation constraints enables back-propagation, and thus cross-entropy minimization. According to it, the training begins minimizing the constraint loss until certain level is reached (approximately 10 circa epoch 250). During this phase, the cross-entropy loss stays constant. Only when this critical point in the constraint loss is reached, back-propagation is restored and the accuracy starts climbing up to 100%. However, using this additional loss (from the separation constraints) induces aggressive transient states (e.g. instabilities) during training (approximately around epochs 6200, 11883 and 16432). We see also how after each of the transient states, the network finds a new solution different from earlier ones. Moreover, those solutions show an increasing level of complexity, in the form of smoother and more rouded boundaries. For instance, the first solution (Figure 12f is linear and the second is composed of few lines (Figure [2h whereas the last and second to last (Figure [2ai and I2ao are much more curved.
We conjecture that this behaviour must be due dead units and points being revived during training. As a dead unit or point comes back from being dead, the weight configuration set by the Separation Constraints is likely be unrelated to cross-entropy. That causes a break down in the convergence that we can see around epochs 6200, 11883 and 16432. As back-propagation is now working for those points units, their error is eventually corrected by normal cross-entropy minimization and the network converges to solution again. However, as now there is an additional working unit or point involved, the network has greater representational expressiveness, which results in more complex decision boundary.
20

---

Under review as a conference paper at ICLR 2020
Coopanergy (88)
1187
(a) Evolution of cross-entropy and constraint loss during training.
1.0
3500
5000
7504
3009
135W
1569
1750
(b) Evolution of accuracy during training.
(c) 0
(d) 246
(e) 269
(I) 400
(g) 874
(h) 1135
(i) 6200
(j) 6205
(k) 7025
()) 7104
(m) 8131
(n) 8132
(0) 8133
(p) 8137
(q) 9119
(r) 9162
(8) 11881
(1) 11883
(u) 11887
(v) 11998
(w) 12000
(x) 12111
(y) 12112
(2) 12124
(aa) 12402
(ab) 12700
(ac) 14021
(ad) 14022
(ae) 14023
(af) 14075
(ag) 14340
(ah) 14464
(ai) 16424 (aj) 16432 (ak) 16433 (al) 16434 (am) 16436 (an) 16442 (ao) 16630 (ap) 16632
Figure 12: Evolution of training throughout epochs (cross-entropy, constraint loss, accuracy, and decision surface) 50x4 network trained on the MOONS dataset. We used Adam with Ir 0.01 for 17000 epochs and batch size of 85 and Glorot uniform. We used X = 10-1 Figure 12a shows both cross-entropy and constraint loss for each epoch of the training phase in the horizontal axis, whereas Figure 12b does the same for training accuracy. The rest of subtigures show the decision surface at the indicated epoch. We see how learning only starts after the constraint loss has been minimized below approximately 10-2 circa epoch 250. After that, prok-propag is restored and cross-entropy falls quickly, reaching solution at epoch 1135. However, around epoch 6205 the training breaks down and the next epochs spent in transient state, until the geometry of the boundary changes and the network converges to a different solution (note how the solution found 1135 composed of lines. whereas 9162 consists of 7). This process occurs several times (6200, 11883 and 16432), resulting in increasingly complex solutions (notice how solution at 16630 much more rounded and smoother). We conjecture that this due dead units and points recovered during training by the Separation Constraint augment the capacity of the network.
21


Reviews:
[{"review_id": "SJldceB15H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers suggests new penalties to avoid the apparition of dead units and minimize the existence of dead points.\n\nDecision\n\nI vote to reject this paper because the formulation of the penalties lacks clarity and because the experiments are incomplete and likely provide misleading results.\n\nJustification\n\n    Formulation of the penalty \n\nIt is unclear to me how the slack variables xi depends on the weights, and therefore how they can be minimized by adjusting the weights using a penalty. I believe this should be clarified between equations (12) and (13).\n\nThe optimizer used is Adam instead of a plain SGD. How is the penalty integrated in Adam? Is it left out as it is best to do for weight decay or is it integrated in the adaptive estimations? I believe a plain SGD would be better justified when the goal is to understand an improvement to the flow of information in the network.\n\n    Experiments\n\nThe experimental setup have many problems.\n1) The protocol is creating a bias\n2) The comparisons of zero initializations are indirect\n3) The tasks chosen cannot corroborate that the penalty is helping information flow deeper without harming generalisation.\n\n1) Protocol\nResults may be misleading both because learning rates are not adjusted, which can cause deeper models to diverge while they could still be trained and because number of epochs is limited instead of letting the models train until convergence. I note that plots comparing train and validation accuracy in Figure 1 shows better results on the validation set than on the training set for ReLU and ReLU + BN. This suggests indeed that the optimization diverged, breaking the training accuracy while leaving the validation accuracy close to random. It is difficult however to evaluate the color-maps without an axis for the colors, so my assumption that dark purple is close to random accuracy may be wrong. A related note, description of Figure 4 in section 4.2 discuss about the achieved maximum performance at depth 60 and width 25. I could not read the same, to me it seems to be achieved right at depth 1, width 3.\n\n2) Comparisons of zero initializations\nThe experiments on the zero initialization scheme is an interesting investigation, but it is incomplete as there is no direct comparison with training without Sep-UP. The Annealed Dropout is important to allow training with zero initialization, and may well be more important than Sep-UP. We cannot measure their respective importance without a direct comparison as in Figure 1.\n\n3) Too simple tasks\nThe Figure 3 points to an important issue of the paper. Generally, the depth of networks is shown to provide higher capacity which leads to improved accuracy when not overfitting. In the experimental setup of this paper, the tasks and architectures do not improve with depth, or more precisely the improvement in capacity (training accuracy is maximal at depth 10) is associated with overfitting. If empirical results show that the contributed penalty helps improving the flow of information, it should also be shown that it is not at the cost of better generalisation otherwise there is no point in having greater depth. The chosen experimental setup does not allow this however. Even though it does fair better at generalization than the others, they all suffer from worst generalisation with depth. More difficult tasks may be better suited for these experiments. I am not recommending to use very large datasets such as ImageNet, difficult synthetic problems could be sufficient. It is necessary however that depth can be shown to be beneficial for baselines, so that we can confirm that training with the penalty is not degrading the beneficial effect of depth.\n\nOther comments\n\nThe problem of dead points should be shown empirically since the contributions of this paper are supported empirically. \n\nFourth paragraph of section 4.1 mentions results using convolutional layers, but it is not clear which of the provided results are on fully connected or convolutional networks.\n\nMinor comments\n\nIn equation (1), \\hat{e}_j is not specified. My guess is that represents a basis, but I don't see why this basis necessary in the equation.\nIn equation (5), the notation is limited to a single layer, with x being the input of the layer. I believe this should be generalized with a composition operator as in equation (7).\nThere seems to be a mistake in equation (8). The inclusion in X on the right is useless since the set on the left is an intersection with X.\nSection 3.1, third paragraph: It is *easily* to see [...]\nSection 3.2, first paragraph: [...] slack variables for *each each* point on the batch.\nSection 4.1, first paragraph: [...] (see Figures 1b and *1b*). (should be e)\n\nEdit:\n\nI still believe this paper lacks empirical evidence that the proposed loss would help train deep models without harming performance on tasks where depth matters. In light of all the corrections and clarifications, I upgrade my vote to weak reject.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgL7CxOqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed two constrains to tackle the problems of dead points and dead neurons. In the toy datasets,  these two problems are successfully solved. In addition, the network weights can be initialized naively with the proposed method.\n\nPros:\n- The idea sounds interesting. Dead points and dead neurons are solved in the toy examples.\n- Networks can be initialized with zeros with the proposed idea.\n\nCons:\n- The experiments are not sufficient to validate the arguments.  MNIST and CIFAR-10 datasets are toy datasets. As the memory complexity is reasonable, Experiments should be done on ImageNet.\n- Why aren't the constrains 12 and 13 in the final objective 16? Am I missing something?\n- Too many typos.\n\nminors:\ne hat is not defined in equation 1.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJeUAR0wKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a problem of training DNNs having dead points and neurons was addressed. The problem was posed as a constraint optimization problem. The constraints were augmented to the loss functions of DNNs as a regularizer. The slack variables of the constrained loss functions were optimized together with parameters of DNNs to train DNNs avoiding dead points and neurons.\n\nThe paper was written well in general. However, the work is incomplete in terms of explanation of the details of the proposed method, and their experimental analyses. For instance:\n\n- In the code, you implemented the constraints in a ReLU.  In this case, could you please compare the proposed SeparatingReLU with the other variation of ReLU such as parametric ReLUs etc.\n\n- Could you please analyze convergence of parameters of DNNs together with slack variables during training? \n\n- Do you apply additional constraints on slack variables to control their scale?\n\n- How do the proposed methods scale with larger datasets and networks? Please provide more detailed analyses for training larger/wider DNNs with different structures (such as VGG and ResNets) on larger datasets, such as at least Cifar 100 and ImageNet.\n\nAfter the discussion:\n\nI checked the response of the authors and comments of the other reviewers.\n\nThe authors consider analyses of various crucial properties of the methods in comparison with the other works as the future work. There are also some statements need to be verified and explained more precisely, such as regarding generalization properties and the relationship between network complexity, constraints and regularization. Therefore, I suggest authors to improve the related statements, and provide more detailed analyses for verification of the proposed results in the following submissions. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes constraints to tackle the problems of dead neurons and dead points. The reviewers point out that the experiments are only done on small datasets and it is not clear if the experiments will scale further. I encourage the authors to carry out further experiments and submit to another venue.
Decision: Reject