Farkas layers: don't shift the data, fix the geometry
Authors: Aram-Alexandre Pooladian
Chris Finlay
Adam M Oberman
Venue:  (2020)
Keywords: initialization
deep networks
residual networks
batch normalization
training
optimization

Abstract: Successfully training deep neural networks often requires either {batch normalization}, appropriate {weight initialization}, both of which come with their own challenges. We propose an alternative, geometrically motivated method for training. Using elementary results from linear programming, we introduce Farkas layers: a method that ensures at least one neuron is active at a given layer. Focusing on residual networks with ReLU activation, we empirically demonstrate a significant improvement in training capacity in the absence of batch normalization or methods of initialization across a broad range of network sizes on benchmark datasets.

Under review as a conference paper at ICLR 2020
FARKAS LAYERS: DON'T SHIFT THE DATA, FIX THE GE- OMETRY
Anonymous authors Paper under double-blind review
ABSTRACT
Successfully training deep neural networks often requires either batch normaliza- tion, appropriate weight initialization, both of which come with their own chal- lenges. We propose an alternative, geometrically motivated method for training. Using elementary results from linear programming, we introduce Farkas layers: method that ensures at least one neuron is active at given layer. Focusing on residual networks with ReLU activation, we empirically demonstrate a significant improvement in training capacity in the absence of batch normalization or meth- ods of initialization across a broad range of network sizes on benchmark datasets.
INTRODUCTION
The training process of deep neural networks has gone through significant shifts in recent years, primarily due revolutionary network architectures, such as Residual neural networks (He al., 2016) and normalization techniques, initially proposed as batch normalization (loffe & Szegedy, 2015). The former the backbone for current "state-of-thhearr" results for image-based tasks such as classification. Normalization can be found in variety of flavors and applications; (Ba et al., 2016; Ulyanov et al., 2016; Wu He, 2018; Vaswani et al., 2017; Zhu et al., 2017). For reasons that not fully understood, normalization gives rise many desirable traits in network training, e.g. fast convergence rate, but also comes with a cost by increasing vulnerability to adversarial attacks (Galloway et al., 2019). Apart from normalization, network weight initialization has been driving force of improved performance. Throughout the years, various initialization schemes have been proposed, in particular Xavier initialization (Glorot & Bengio, 2010), FixUp (Zhang et al., 2019), and a recent asymmetric initialization (Lu et al., 2019). These approaches are often connected to probabilistic notions, or balancing learning rates while training.
Modern network architectures catered for imnag-classificatton tasks incorporate various one- sided activation functions, the canonical example being the Rectified Linear Unit (ReLU) (Nair & Hinton, 2010). Other one-sided activations include the Exponential Linear Unit (ELU) (Clevert et al., 2015), Scaled ELU (SELU) (Klambauer et al., 2017), and LeakyReI.U (Maas et al.). ELU and SELU were constructed with the purpose of climinating batch normalization (BN); these activation functions minimize the internal covariate shift (ICS), which is what BN was intended to accomplish. Recent developments have shown that BN has little impact on ICS, but affects the smoothness of the loss landscape, allowing for easier (and faster) training regimes (Santurkar al., 2018 2018).
A geometric interpretation of batch normalization can be readily seen in the context of the dying ReLU problem. This phenomenon occurs when the gradient of the neuron is zero (when the neuron outputs only negative values), in which case the neuron is "dead". This effectively freezes the neuron during training. If too many neurons are dead, the network learns slowly. In fact, is entirely possible for a network to be "born dead",, where it does not allow learning at all (Lu et al., 2019). To motivate this, consider a simple binary classification problem: suppose two sets of points in R are sufficiently separated and we wish to classify them using simple 1-layer ReLU network (with standard Cross Entropy loss). If the network is initialized such that one cloud of points is not "observed", as in Figure la, the network will not learn to classify those points. Batch normalization will shift the points to behave like lb; the network is no longer dead. There a far simpler geometric solution in R2: given one hyperplane, construct another face the missing data,

---

Under review as a conference paper at ICLR 2020
shown in Figure Ic by the green hyperplane, which is non-dead component of the network.
(a) Example of dead layer
(b) Effect of normalization
(c) Effect of Farkas layers
Figure I: motivating geometric interpretation
OUR CONTRIBUTIONS
We present geometrically motivated method for network training in the absence of initialization and normalization. Our novel layer structure, called Farkas layers, ensures that at least one neuron is active in every layer. Using off-the-shelf networks, Farkas layers can recover significant part of the explanatory power of DNNs when batch normalization removed, sometimes up to 10%, as demonstrated by empirical results benchmark image classification tasks, such CIFARIO and CIFARI00. When used in conjunction with batch normalization on ImageNet- Ik networks, empirically show approximate 20% improvement on the first epoch over only using batch normalization. Finally, we claim that input data normalization is a critical component for using large learning rates in FixUp, which is not the case for Farkas layer-based networks. All in all, this work provides new research direction for architecture design beyond initialization methods, normalization layers, or new activation functions.
2 BACKGROUND
2.1 NOTATION
We define neural networks as iterative compositions activation functions with linear functions. More specifically, given an input (*()) E T to the k-th layer, the output (k++1) is typically written
x (k+1) (R) + (())
where (K)) m is weight matrix, the rows are written as f 1,..., m,) 6(k) R is the layer's 's bias, and is the activation function of the layer. For the remainder of our analysis, we consider the ReLU activation function, written
p(x) max fzr 0},
which acts component-wise in the case of vector arguments.
2.2 RELATED WORK
Previous work on the dying ReLU problem, or vanishing gradient problem, in the case of using sigmoid-like activation functions, has heavily revolved around novel weight initializations. To ad- dress the vanishing gradient problem, (Glorot & Bengio, 2010) proposed an initialization that is still often used with ReLU activations (Shang et al., 2017). (He al., 2015) proposed scaled version of a normal distribution, which improved training for pure convolutional neural networks. More recently, (Lu al., 2019) studied the dying ReLU problem from a probabilistic perspective and addressed networks that are "born dead"., i.c. not capable of learning. Their analysis is based on the reasonable assumption that all weights and biases are initialized with non-zero probability of being dead:
PP (w x tb, < < 2>>0 (i=1) 1,...,,., m),
2

---

Under review as a conference paper at ICLR 2020
where is the number of neurons in a given layer, and is a fixed positive constant. They show that, as the number of layers approaches infinity, the probability of having network be born dead approaches one. They also find that initializing weights with symmetric distribution is more likely to cause a network to be born dead and thus propose initializing weights using asymmetric distributions to mitigate this problem. Their analysis is relevant in the that the network is not very wide, which is distinct from the case we study in this paper.
Normalization has been one of the driving forces of recent state-of-the-ard results; where the general idea to manipulate the neuron activation with various statisties, such as subtracting the mean and dividing by the variance. Popular normalization techniques include batch normalization (loffe & Szegedy, 2015), Layer normalization Ba et al. (2016), Instance normalization (Ulyanov et al., 2016) and Group normalization (Wu & He, 2018). However, there is desire to eliminate normalization from training. recent work that does this is FixUp (Zhang et al., 2019), a novel initialization technique that works by scaling the weights as a function of the network structure, allowing the network to take large learning rate steps during training. Reportedly, FixUp provides the same level of accuracy as that of BN-enhanced network, and scales to networks that train ImageNet- Ik. As stated in FixUp, the inherent structure of residual networks already prevents some level of gradient vanishing, since the variance of the layer output grows with depth. In the case of positively- howoeeeeoooos blocks (e.g. no bias in a Linear or Convolutional layer), this type of variance increase can lead gradient explosion (Hanin & Rolnick, 2018), which makes training more difficuut.
3 FARKAS LAYERS
3.1 AUGMENTING RELU ACTIVATED LAYERS
We present a novel approach to understanding how weight matrices and bias vectors contribute to neural network learning, called Farkas layers. As the name suggests, our method is influenced by Farkas' lemma. In particular, we use the following representation of Farkas' lemma; the proof is an exercise in linear programming and is left in the appendix.
Lemma 3.1 (Representation of Farkas' lemma). Let W E gm b E R.m with x E R" Then the set for Wx + < 0} is empty if and only if there exists a E R" such that d 2 (for i i= 1, :...:: m.) with W X = 0 and X T 0.
We present the construction of W and b, and prove using Lagrangian duality (Boyd & Vandenberghe, 2004) that such weights and bias vectors will satisfy Farkas' lemma, resulting in least one positive component, In essence, this solves the dying ReLU problem while training.
Theorem 3.2. Lei L be a layer of a neural network with ReLU activation function, with weights W E max (with rows w:) ) and bias vector b E R, and let X E R with 2 (with at least one X; > 0, without loss of generality Am > 0). Further, suppose W has the property that
-1
Wm
Ajw3
Am
and that 6m
ΣØ byby.
Then this layer has a non-zero gradient.
Am
Proof. Let W E IR. mx (with rows wo) and bias b E R", with the aforementioned assumptions,, and arbitrary input x. The output of this layer is <pWr x + b), and has non-zero gradient if and only there exists least one component such that
= max ! x + by > 0.
(1)
The left-hand side of (1) (i.e. omitting the strict inequality constraint) can be written as the following minimization problem:
p min s
subject to w + b, Sa (i 1, m.....m.
2,5
3

---

Under review as a conference paper at ICLR 2020
equivalently,
p" min 8 subject to Wz + + S s1,
where 1 is the vector of all ones. The Lagrangian for this problem is
( 8, X) = s + ( =1)
where À 2 We compute the dual problem.
d(A) inf C(x, 8,
2,6
inf s(1 - 11 + b (W x)=x
2,6
AT ifA1=11, 0, (A20)
(-x otherwise.
The conditions on W are met by assumption, and there are many ways to ensure that À is on the simplex, and thus has at least one strictly positive component. Both the primal and dual problems are linear optimization problems: the objective and constraints are linear. By weak duality, p 2 sup, d(A). Thus to ensure that p* >> 0 (as in (I)), it suffices to show that A 0. This is guaranteed by the assumptions on the bias vectors, and so we are done.
highly unlikely that given layer in deep neural network has all dead neurons. However, we believe that guaranteeing the explicit activity of one neuron is beneficial in that it improves training. Figure motivates the use of Farkas layers as a potential substitute for (BN) in the context of learning: the arrows of the corresponding hyperplane indicate the "non-zero" part of the ReLU. In the case of la, we see that all data points end up on the "zero" side, which would typically lead to learning. Heuristically, the effect of BN mimics Figure 1b, where the data points centered and scaled by a diagonal matrix (which geometrically acts like an ellipse) allowing for learning to occur. Farkas layers will instead append another hyperplane, that will be guaranteed to all the points (in a "third" dimension).
3.2 EXTENSION TO GENERAL ONE-SIDED ACTIVATIONS
The use of Farkas layers can extended to other one-sided activation functions, such as a smooth- ReLU (i.e. replacing the kink with polynomial), ELU, and several others. In all these cases, one can impose a scalar cutoff value e such that "beyond" this cutoff, we have very small (or zero) gradients. This is shown in the following corollary, which has the same proof as Theorem 3.2.
Corollary 3.3. Let L he layer of neural network with an arbitrary one-sided activation function (or exactly zero). Denote the weights by W E mm (with rows wi) and bias vector b E 1. and let X Rm, with X O, with at least one positive component. To ensure that (W x + b) has at least one component greater than c, we require that
d with cutoff e E R. i.e. for all x < c, we have vxx)
Wm
AjWl
Am
and that bm
Amo En Xzbj-
3.3 ALGORITHM AND LIMITATIONS
We denote a Farkas layer by satisfy Theorem 3.2. The construction of W and b imposes certain restrictions on the existence of Farkas layers, namely we require that dim(Dom) 2 2. Finally, some network architectures claim to perform better without bias vectors present; we require bias vectors by construction. We focus on replacing Convolutional and Linear layers in (deep) neural networks, with efficient implementations in PyTorch. The general method of incorporating Farkas layers is provided in Algorithm 1. To minimize computational cost, we consider À to be the evenly spaced simplex element i.e. X; dim(D,m)) (for i 1,..., di...,d) but for smaller problems (such as the
W,b) Din + Donts where the weights and biases
4

---

Under review as a conference paper at ICLR 2020
motivating l-layer example) À can be learnable. Farkas layers be made into Farkas blocks in conjunction with residual networks as well, outlined in Algorithm 2, which we leave in the Ap- pendix. In the case of multiple convolutional layers in a given residual block, the extension is natural.
In both Algorithm and 2, consider choices for the aggregating function (AggFune): either the sum or the mean of the previous outputs and biases. Indeed, let dim(Dour) = m, and thus Ag m- Then we have the following two aggregation methods:
12 Σ
m-11
Wm
w/
(sum),
Wm
Σ
w (mean),
m
and similarly for the bias vectors. Theorem 3.2 is constructed using "sum", " but the result holds in the case of using "mean" since ReLUS are 1-positive homogenous. Furthermore, by using "mean", we induce stability with respect the -inducee matrix norm for matrix A RR* defined as
|| s0
max ||a] 11a,
15/5m
where of are the rows of A. Let Ā e (( x) and a T be the mean of the rows of A. Define
A = aA: E Rmxn then
|| max (l aae::laal 1la111] z max ||Al|bo (m- 1)-1 n
Σ
lla
S ||
Thus, in the context of network weights, the burden of stability lies on the trainable weights and not the aggregated component. similar analysis can be done for the bias vectors. Thus, for networks that train ImageNet-1K, k, potentially larger learning rates, we believe that using "mean" as the aggregation function is necessary.
Algorithm FarkasLayer with ReLU activation; F(; (,W, with AggFunc "sum" or "mean"
Initialize W (m-m-)) and b R"
Forward pass:
= Wz
c.g. Linear and/or Conv with no bias
Compute the Aggregated Components:
=
-AggFunc(y)
max - Agggnucc -1]), bm)
concatenate y)
concatenate(b(0-
+ y -
ReLU(y)
Optional: apply batch normalization to y
Return y
EXPERIMENTS - IMAGE CLASSIFICATION
Image classification is standard benchmark for measuring new advancements in deep learning. Our goal to show how training with Farkas layers, henceforth referred to as FLs, impacts learning in these settings relative other training techniques such as normalization and initialization (e.g. FixUp).
We consider CIFAR10, CIFAR100, and ImageNet-1k 1k datasets for the purposes of testing FL- based residual networks on image classification tasks. For CIFARIO, we consider 18, 34, 50, and 101 layer residual networks. The latter two of these use a BottleNeck block structure. On CIFAR100, we only consider 34, 50, and 101 layers. All models are trained using the same SGD schedule for 200 epochs. We make the distinction between "large" learning rate, meaning 0.1, and a "small" learning rate, which is 0.01. We use standard data augmentation (RandomCrop and RandomiHorizonallFinn but do normalize the inputs with respect to the mean and standard
5

---

Under review as a conference paper at ICLR 2020
deviation of the dataset in the case of CIFARIOICIFARIOD The weights are left at the default PyTorch initialization. Additionally, we use cutout to improve generalization Vries & Taylor, 2017), and use weight-decay. For ImageNet-1k, we modified code from the DAWNBench competition (Coleman et al., 2018; Shaw et al.). In this set of experiments, we augment the data via RandomCrop, RandomHorizontalFiig the case of networks using BN and see how FLS can improve performance, and train with 30 epochs.
and normalize the input data. We only consider
Henceforth, we call residual network FarkasNet if is it comprised of only FLs, and we always use the "sum" aggregation function unless otherwise specified. We make the distinction of networks that use or do not use BN in the tables. When omitting BN, use the small learning rate. For CIFARIO 10 and CIFARIOO 100, the results presented are averaged across three runs; if one of the runs failed! to train, we omit it from the average but place an asterisk. Full training curves are in the Appendix.
INTERPRETING DEPENDENCE OF BATCHI NORMALIZATION
Table 1: Percent misclassification comparison for CIFARIO (lower is better)
18 Layers
34 Layers
50 Layers
101 Layers
ResNet (with BN, large LR) FarkasNet (with BN, large LR)
7.01
6.79
5.04
4.95
6.87
6.50
5.03
4.98
ResNet (no BN, small LR)
12.51 10.83
9.90
24.53
25.25
FarkasNet (no BN, small LR)
9.45
19.84
15.38
Table 2: Percent misclassification comparison for CIFARI00 (lower is better)
34 Layers
50 Layers
101 Layers
ResNet (with BN)
30.95
23.43
22.90
FarkasNet (with BN)
30.34
23.88
23.22
ResNet (no BN)
40.10
55.05*
54.02
FarkasNet (no BN)
34.70
43.93
42.94
Tables and 2 both show strong disparity in the learning capacity of DNN when batch nor- malization is removed, which is unsurprising. On CIFARIO, we primarily observe similar, if not better, test errors when batch normalization used. Without normalization, our test error always better; the same can be said for the CIFAR100 dataset. Thus, while FarkasResNets also diminish quality, we note that adding a guaranteed undead neuron to all layers heavily impacts learning in the case of non-normalized network. This is not at all to say that training deep neural network is only possible using FLs, or that we achieve state-of-the-an performance; however, our implemen- tation demonstrates the strong dependency of getting low test accuracy with normalization, and is geometrically motivated.
IMPROVEMENT FIRST BPOCH FOR IMAGENET-IK
The final table shows similar behaviour on ImageNet- Ik, where FL.s² neither dramatically improve nor inhibit the learning capacity towards the end of training except in the case of 50-layers, where performs significantly better. Figure 2 shows that FarkasNets have dramatic advantage over standard ResNets at the start of training. Evidently, this advantage gradually diminishes: nonethe- less, its presence alludes to a bigger problem in network initialization. We notice roughly 20% improvement on the first epoch, simply due to the use of FLs.
Failed implics that the network did not make any progress in the first five epachs.
Forr ImageNet-16 1k, use "mean" aggregation function..

---

Under review as a conference paper at ICLR 2020
Table 3: Topl/Top5 percent misclassification comparison for ImageNet-Ik Ik (lower is better)
50 Layers
101 Layers
152 Layers
ResNet (with BN)
25.64/7.68
22.26/5.95 23.74/6.67
23.23/6.49
FarkasNet (with BN)
23.70/6.65
22.17/5.86
100
ResNet50
FarkasNet50
ResNet101
FarkasNet101
90
ResNet152
FarkasNet152
85
80
65
60
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Figure 2: Illustration of improved performance at starting epochs with Farkas layers
IMPACT OF DATA NORMALIZATION AND BEST PRACTICE COMPARISON
We briefly compare against other best practice learning regimes on CIFAR1O. Table 4 highlights the various differences between methods. the core, we compare FarkasNets to FixUp, recently proposed initialization scheme that allows the use of larger learning rates in the absence of batch normalization, and thus faster convergence. We consider a 34-layer ResNet with BasicBlock structure using the official implementation of FixUp' for the network architecture. We compare using a 34-layer FarkasNet, where the last layer in a block initialized to zero (as FixUp, which motivated by several other works (Srivastava et al., 2015; Hardt & Ma, 2016; Goyal et al., 2017)) but use standard initialization otherwise, as well as "mean" AggregationFunction We maintain the setup of the previous experiments.
Data normalization is standard "trick" for training deep neural networks, and acts like initial batch normalization step, where the incoming data is scaled to have zero mean and unit variance. In the case of CIFARIO and CIFARI00, any attempts at normalization, which is why we have omitted this from our training methodology. In the absence of data normalization, we observed that FixUp requires a small learning rate for training. Thus, while (Zhang al., 2019) claims present theory for addressing training in the absence of batch normalization, we find that still heavily hinges on normalization principles, namely the first layer pass. This observation is not to diminish the benefits of FixUp, as faster convergence observed (see Figure 3), but the learning rate does need adjustment. On the other hand, our 34-layer FarkasNet was able to use medium learning rate of 0.05, allowing for improved performance relative to Table
strove to study the problem of training without
From of the authors' Github pages

---

Under review as a conference paper at ICLR 2020
Table 4: Best practice comparison on CIFARIO (lower error better)
Batch Normalization
LR capacity
Test Error
ResNet34
Large
6.77
ResNet34
Small
9.18
FixUp34
X
Small
7.49
FarkasNet34
Medium
7.12
With BN
Without
Fxupp
50 75
100
325
150
375
200
Figure 3: Training curves for best practice comparison on CIFAR10
Finally, we also address the notion that training a very deep neural network is challenging using maximal learning rate without normalization, which has been experimentally observed in FixUp and Layer- Sequentelulttal
Varianar (LSUV) orthogonal initialization (Mishkin & Matas, 2015). Despite the shortcomings of previous work, by incorporating the "mean" aggegration function and default initialization, we are able to successfully train 101-layer FarkasNet with large learning rate on CIFARIO. We present the averaged training curve in Figure left in the Appendix. where we repeated the experiment three times. We remark that using default initialization alone results in complete failure train.
5 CONCLUSION
To date, successful training of deep neural networks requires either some form of normalization, weight initialization, and/or choice of activation function, of which come with their own chal- lenges. In this work, we provide a fourth option, Farkas layers, which can be used alone or in conjunction with the aforementioned techniques. The Farkas layer based on the interaction of the geometry of the weights with the data: by simply adding one linearly dependent row to the weight matrix, we ensure that no neurons are dead. Using our method, we have shown that training with larger learning rates is possible even in the absence of batch normalization, and with default initial- ization. In this work, we only touched on image classification, but Farkas layers could used in many deep learning applications, which is left for future work.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.000 2016.
8

---

Under review as a conference paper at ICLR 2020
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 1.XV:1511..02899 2015.
Cody Coleman, Daniel Kang. Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Christopher Ré, and Matei Zaharia. Analysis of dawnbench, time- to-accuracy machine learning performance benchmark. CoRR, abs/1800.01427, 2018. URL http://arxiv/ org/ abs/ 888806.0 01427.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
170.:010822.2 2017.
Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W. Taylor. Batch normalization is a cause of adversarial vulnerability. CoRR, abs/1905.02161, 2019. URL httt:::
/ //arxiv. org abs 1905. 02161.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: ToXiv:/700.000 770.02667, 2017.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems, pp. 571-581, 2018
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.0423) 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. 2015. URL BtLLtttp//axxiv..oo abs/ 15C2. 01852.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-645, Cham, 2016. Springer International Publishing.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 502.03167, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:1903.06733 2019.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models.
Dmytro Mishkin and Jiri Matas. All you need good init. arXiv preprint arXiv:1511.0642 2015.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018.
Wenling Shang. Justin Chiu, and Kihyuk Sohn. Exploring normalization in deep residual networks with concatenated rectified linear units. In Thirty-Fira AAAI Conference on Artificial Intelligence, 2017.

---

Under review as a conference paper at ICLR 2020
Andrew Shaw, Yaroslav Bulatov, and Jeremy Howard. Imagenet in 18 minutes.
URL https:
//github. com http://uuxdddeeeee http///duuuudd/////neneeeee
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv: 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini
2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint UrXiv:1901.0933 2019.
Jan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2222322 2017.
A APPENDIX
A.1 ELEMENTARY LINEAR PROGRAMMING
Proposition A.I (Farkas' Lemma). Let W E mxxn and b E Rm
{z Wæ + b= = 0, 2 0} + 0 & s.t. ATW 20, A > 0.
Corollary A.2. Let W E m and mm. Then
{or W r + b < 03 = 0 = W. =0, A 0 (A2 0).
Proof. Suppose by contradiction there exists an x such that Wx + s 0. Let x+ = max 0} and x: max { -x, 0}. and so x+ x Then:
3r s.t Wx + 550
Art x s.t W(W (x+ x - +b50
3rt x As W(x"' x + x 0 (A20) 0)
- 8== (a+, x x=,,) , 2 0, W = W W []] s.t Wi+b=0
which follows from Farkas' Lemma.
A.2 ALGORITHM FOR RESIDUAL FARKAS LAYER
Algorithm 2 Residual Farkas layer with ReLU activation
Initialize W(1,, W(2) W(2),
Forward pass:
F(x; ((),(1))
+ (2) y
e.g. Linear and/or Conv with no bias
Compute aggregated components:
= AggFunc(z + y)
residual component
= max{ ( (bl - -1]), (22
f- concatenate y, y)
6 concatenate (6 10: 11] b)
f y y
+ ReLU(y)
Optional: apply batch normalization to
Return y
10

---

Under review as a conference paper at ICLR 2020
A.3 TRAINING CURVES FOR CIFARIO
(a) 18 layers
(b) 34 layers
Vanil,
Nu
(c) 50 layers
(d) 101 layers
Figure 4: Average training curves for CIFARIO; "Vanilla" refers to standard ResNet. Dotted lines omit batch normalization, solid lines incorporate it.
60
60
59
40
25
50
75
100
125
3.50
175
200
Figure 5: Average training curve for 101-layer FarkasNet. with "mean" aggregation function, maxi- mal learning rate, and without batch normalization.
11

---

Under review as a conference paper at ICLR 2020
A.4 TRAINING CURVES FOR CIFARI00
For 50-layers, one of the networks (standard ResNet without BN) failed to train, and we omit from the average.
1MP1
1.9
WIT
Parkin withier
W
(a) 34 layers
(b) 50 layers
22-
(c) 101 layers
Figure 6: Average training curves for CIFAR100; "Vanilla" refers to standard ResNet
A.S TRAINING CURVES FOR IMAGENET-IK 1K Here, dotted line means Farkas layer-based network.
Restect
102
BakkeEt
05
(a) 50 layers
(b) 101 layers
(c) layers
Figure 7: Training curves for ImageNet
12


Reviews:
[{"review_id": "rklkPJJnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new `normalization' approach called Farkas layer for improving the training of neural networks. The main idea is to augment each layer with an extra hidden unit so that at least one hidden unit in each layer will be active. This is achieved by making the extra hidden unit dependent on the rest of the units in the layer, so that it will become active if the rest are inactive, and they name it after Farkas' lemma in linear programming. This avoids the gradient becoming zero when all the units in a layer are dead. \n\nThe empirical results show that this normalization method is effective, and improves the training of deep ResNets when no batch normalization is used. The accuracies on CIFAR10 and CIFAR100 are improved with the use of Farkas layers. Unfortunately it still cannot beat or replace batch normalization. When batch normalization is used, the benefit of using this Farkas layer becomes marginal (Tables 1 and 2). \n\nI am also not completely satisfied with the authors' explanation on why Farkas' layers work. The authors motivate the design of the layer with dead hidden units, but in the experiments they do not show if any layer actually becomes completely `dead' (or gradient becomes very small) when Farkas' layer is not used. There could be other reasons why the layer helps, other than keeping some units in a layer active. \n\nOverall I think the idea is novel and interesting, but the improvement is not big enough to replace existing normalization methods that makes this paper slightly below the acceptance threshold in my opinion.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xqcPXpKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a new method to deal with the zero gradient problem of ReLU (all input values are smaller than zero) when BN is removed, called Farkas layer. This Farkas layer concatenates one positive value to guarantee that, at least one neuron is active in every layer to reduce the challenge for optimization. Compared with the method without BN, Farkas layer shows better results on CIFAR-10 and CIFAR-100.\n\nThough, I still have several concerns:\n1.\tThe proposed Farkas layer is too simple and seems not work well. With BN, the FarkasNet does not show significant improvements than the traditional ResNet with BN on CIFAR-10, CIFAR-100 and ImageNet.\nWithout BN, though FarkasNet shows significant improvements than ResNet. But FarkasNet without BN cannot achieve comparable performance with FarkasNet with BN. With deeper networks, the performance further goes down, which really downgrade the rating of this paper.\nIn Fixup, ResNet w/o BN with mixup can achieve comparable performance with ResNet with BN on ImageNet. Could FarkasNet further improve the performance in the setting of ResNet w/o BN with mixup and fixup init? This would much more improve the application value of the proposed FarkasNet.\n\n2.\tFor the results of CIFAR-10 and CIFAR-100, the error bar should be added to make the results more convincing. \n\n3.\tFigure 7, the training curves seem weird. Why the training error goes up in some stages? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lkFTxotH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new type of layer, Farkas layers, that are designed to ameliorate the \"dying ReLU problem\". The idea of repurposing Farkas' lemma from linear programming to a component in NNs is very nice. However, the paper doesn't have convincing baselines and doesn't dig deep enough into what the Farkas layer is actually doing.\n\nComments:\n\n1. Start of section 2.2: “Previous work on the dying ReLU problem, or vanishing gradient problem, in the case of using sigmoid-like activation functions, has heavily revolved around novel weight initializations”. Dying ReLUs and vanishing gradients are different problems. In particular, it doesn’t make sense to talk about the dying ReLU problem for sigmoid-like activation functions.\n\n\n2. Batchnorm (BN) consists in two operations: shifting and scaling. Both are relevant to vanishing gradients. However, only shifting (that is, subtracting the mean and resetting to a learned value) is relevant to dying ReLUs because rescaling the input to a ReLU by a positive number doesn’t affect whether the ReLU is subsequently ON or OFF.\n\n\n3. Given point #2, a natural baseline to compare the Farkas layer against is a “pared-down BN”, which shifts but does not rescale.\n\n\n4. Similarly, when combining the Farkas operation with BN, it might be worth considering keeping BN’s rescaling but dropping the shift -- since Farkas is analogous to the shift operation in BN.\n\n\n5. I claimed above that Farkas is analogous to the shift in BN, but haven’t thought about it deeply. Do you agree? Any comments on how they differ and why?\n\n\n6. Our contributions, p2: “We empirically show an approximate 20% improvement on the first epoch over only using batch normalization.” I’m not sure what to make of this; improvements on the first epoch are only useful if they lead to overall improvements. \n\n\n7. Figure 4 of “Shattered gradients”, https://arxiv.org/abs/1702.08591 looks at ReLU activations by layer, both with and without BN. It’s worth doing a similar analysis for Farkas layers. Concretely: how do Farkas layers change the pattern and frequency of activation, both at activation and during training?\n\n\n8. Guaranteeing the activity of a single neuron per layer seems very weak. What is the empirical effect of Farkas on the number of live neurons? Is it really just making sure one ReLU is on, or does it do better? Is it possible to ensure more neurons are ON in each layer? The “shattered” paper above suggests BN sets close to half ReLUs as ON at initialization, and approximately controls how often ReLUs are ON or OFF during training via the shift. \n\n\n9. As a broader point, the paper proposes an algorithm based on a hypothesis: that having (at least one) ReLU on helps training. It’s worth digging into the hypothesis a bit rather than just plotting training curves. \n\n\n10. I would expect ResNets have much *less* of a problem with dying ReLUs than standard NNs because of the skip-connections. One would therefore expect Farkas layers to help more with standard NNs than ResNets. However, the reported results are for ResNets. What happens when there are no skip-connections? \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new normalization scheme that attempts to prevent all units in a ReLU layer from being dead. The experimental results show that this normalization can effectively be used to train deep networks, though not as well as batch normalization. A significant issue is that the paper does not sufficiently establish that their explanation for the success of Farkas layer is valid. For example, do networks usually have layers with only inactive units in practice?
Decision: Reject