Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper provides a new method to deal with the zero gradient problem of ReLU (all input values are smaller than zero) when BN is removed, called Farkas layer. This Farkas layer concatenates one positive value to guarantee that, at least one neuron is active in every layer to reduce the challenge for optimization. Compared with the method without BN, Farkas layer shows better results on CIFAR-10 and CIFAR-100.

Though, I still have several concerns:
1.	The proposed Farkas layer is too simple and seems not work well. With BN, the FarkasNet does not show significant improvements than the traditional ResNet with BN on CIFAR-10, CIFAR-100 and ImageNet.
Without BN, though FarkasNet shows significant improvements than ResNet. But FarkasNet without BN cannot achieve comparable performance with FarkasNet with BN. With deeper networks, the performance further goes down, which really downgrade the rating of this paper.
In Fixup, ResNet w/o BN with mixup can achieve comparable performance with ResNet with BN on ImageNet. Could FarkasNet further improve the performance in the setting of ResNet w/o BN with mixup and fixup init? This would much more improve the application value of the proposed FarkasNet.

2.	For the results of CIFAR-10 and CIFAR-100, the error bar should be added to make the results more convincing. 

3.	Figure 7, the training curves seem weird. Why the training error goes up in some stages? 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a new `normalization' approach called Farkas layer for improving the training of neural networks. The main idea is to augment each layer with an extra hidden unit so that at least one hidden unit in each layer will be active. This is achieved by making the extra hidden unit dependent on the rest of the units in the layer, so that it will become active if the rest are inactive, and they name it after Farkas' lemma in linear programming. This avoids the gradient becoming zero when all the units in a layer are dead. 

The empirical results show that this normalization method is effective, and improves the training of deep ResNets when no batch normalization is used. The accuracies on CIFAR10 and CIFAR100 are improved with the use of Farkas layers. Unfortunately it still cannot beat or replace batch normalization. When batch normalization is used, the benefit of using this Farkas layer becomes marginal (Tables 1 and 2). 

I am also not completely satisfied with the authors' explanation on why Farkas' layers work. The authors motivate the design of the layer with dead hidden units, but in the experiments they do not show if any layer actually becomes completely `dead' (or gradient becomes very small) when Farkas' layer is not used. There could be other reasons why the layer helps, other than keeping some units in a layer active. 

Overall I think the idea is novel and interesting, but the improvement is not big enough to replace existing normalization methods that makes this paper slightly below the acceptance threshold in my opinion.  

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper introduces a new type of layer, Farkas layers, that are designed to ameliorate the "dying ReLU problem". The idea of repurposing Farkas' lemma from linear programming to a component in NNs is very nice. However, the paper doesn't have convincing baselines and doesn't dig deep enough into what the Farkas layer is actually doing.

Comments:

1. Start of section 2.2: “Previous work on the dying ReLU problem, or vanishing gradient problem, in the case of using sigmoid-like activation functions, has heavily revolved around novel weight initializations”. Dying ReLUs and vanishing gradients are different problems. In particular, it doesn’t make sense to talk about the dying ReLU problem for sigmoid-like activation functions.


2. Batchnorm (BN) consists in two operations: shifting and scaling. Both are relevant to vanishing gradients. However, only shifting (that is, subtracting the mean and resetting to a learned value) is relevant to dying ReLUs because rescaling the input to a ReLU by a positive number doesn’t affect whether the ReLU is subsequently ON or OFF.


3. Given point #2, a natural baseline to compare the Farkas layer against is a “pared-down BN”, which shifts but does not rescale.


4. Similarly, when combining the Farkas operation with BN, it might be worth considering keeping BN’s rescaling but dropping the shift -- since Farkas is analogous to the shift operation in BN.


5. I claimed above that Farkas is analogous to the shift in BN, but haven’t thought about it deeply. Do you agree? Any comments on how they differ and why?


6. Our contributions, p2: “We empirically show an approximate 20% improvement on the first epoch over only using batch normalization.” I’m not sure what to make of this; improvements on the first epoch are only useful if they lead to overall improvements. 


7. Figure 4 of “Shattered gradients”, https://arxiv.org/abs/1702.08591 looks at ReLU activations by layer, both with and without BN. It’s worth doing a similar analysis for Farkas layers. Concretely: how do Farkas layers change the pattern and frequency of activation, both at activation and during training?


8. Guaranteeing the activity of a single neuron per layer seems very weak. What is the empirical effect of Farkas on the number of live neurons? Is it really just making sure one ReLU is on, or does it do better? Is it possible to ensure more neurons are ON in each layer? The “shattered” paper above suggests BN sets close to half ReLUs as ON at initialization, and approximately controls how often ReLUs are ON or OFF during training via the shift. 


9. As a broader point, the paper proposes an algorithm based on a hypothesis: that having (at least one) ReLU on helps training. It’s worth digging into the hypothesis a bit rather than just plotting training curves. 


10. I would expect ResNets have much *less* of a problem with dying ReLUs than standard NNs because of the skip-connections. One would therefore expect Farkas layers to help more with standard NNs than ResNets. However, the reported results are for ResNets. What happens when there are no skip-connections? 




review_title: Official Blind Review #2

Meta-review: This paper proposes a new normalization scheme that attempts to prevent all units in a ReLU layer from being dead. The experimental results show that this normalization can effectively be used to train deep networks, though not as well as batch normalization. A significant issue is that the paper does not sufficiently establish that their explanation for the success of Farkas layer is valid. For example, do networks usually have layers with only inactive units in practice?
Decision: Reject