Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed two constrains to tackle the problems of dead points and dead neurons. In the toy datasets,  these two problems are successfully solved. In addition, the network weights can be initialized naively with the proposed method.

Pros:
- The idea sounds interesting. Dead points and dead neurons are solved in the toy examples.
- Networks can be initialized with zeros with the proposed idea.

Cons:
- The experiments are not sufficient to validate the arguments.  MNIST and CIFAR-10 datasets are toy datasets. As the memory complexity is reasonable, Experiments should be done on ImageNet.
- Why aren't the constrains 12 and 13 in the final objective 16? Am I missing something?
- Too many typos.

minors:
e hat is not defined in equation 1.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This papers suggests new penalties to avoid the apparition of dead units and minimize the existence of dead points.

Decision

I vote to reject this paper because the formulation of the penalties lacks clarity and because the experiments are incomplete and likely provide misleading results.

Justification

    Formulation of the penalty 

It is unclear to me how the slack variables xi depends on the weights, and therefore how they can be minimized by adjusting the weights using a penalty. I believe this should be clarified between equations (12) and (13).

The optimizer used is Adam instead of a plain SGD. How is the penalty integrated in Adam? Is it left out as it is best to do for weight decay or is it integrated in the adaptive estimations? I believe a plain SGD would be better justified when the goal is to understand an improvement to the flow of information in the network.

    Experiments

The experimental setup have many problems.
1) The protocol is creating a bias
2) The comparisons of zero initializations are indirect
3) The tasks chosen cannot corroborate that the penalty is helping information flow deeper without harming generalisation.

1) Protocol
Results may be misleading both because learning rates are not adjusted, which can cause deeper models to diverge while they could still be trained and because number of epochs is limited instead of letting the models train until convergence. I note that plots comparing train and validation accuracy in Figure 1 shows better results on the validation set than on the training set for ReLU and ReLU + BN. This suggests indeed that the optimization diverged, breaking the training accuracy while leaving the validation accuracy close to random. It is difficult however to evaluate the color-maps without an axis for the colors, so my assumption that dark purple is close to random accuracy may be wrong. A related note, description of Figure 4 in section 4.2 discuss about the achieved maximum performance at depth 60 and width 25. I could not read the same, to me it seems to be achieved right at depth 1, width 3.

2) Comparisons of zero initializations
The experiments on the zero initialization scheme is an interesting investigation, but it is incomplete as there is no direct comparison with training without Sep-UP. The Annealed Dropout is important to allow training with zero initialization, and may well be more important than Sep-UP. We cannot measure their respective importance without a direct comparison as in Figure 1.

3) Too simple tasks
The Figure 3 points to an important issue of the paper. Generally, the depth of networks is shown to provide higher capacity which leads to improved accuracy when not overfitting. In the experimental setup of this paper, the tasks and architectures do not improve with depth, or more precisely the improvement in capacity (training accuracy is maximal at depth 10) is associated with overfitting. If empirical results show that the contributed penalty helps improving the flow of information, it should also be shown that it is not at the cost of better generalisation otherwise there is no point in having greater depth. The chosen experimental setup does not allow this however. Even though it does fair better at generalization than the others, they all suffer from worst generalisation with depth. More difficult tasks may be better suited for these experiments. I am not recommending to use very large datasets such as ImageNet, difficult synthetic problems could be sufficient. It is necessary however that depth can be shown to be beneficial for baselines, so that we can confirm that training with the penalty is not degrading the beneficial effect of depth.

Other comments

The problem of dead points should be shown empirically since the contributions of this paper are supported empirically. 

Fourth paragraph of section 4.1 mentions results using convolutional layers, but it is not clear which of the provided results are on fully connected or convolutional networks.

Minor comments

In equation (1), \hat{e}_j is not specified. My guess is that represents a basis, but I don't see why this basis necessary in the equation.
In equation (5), the notation is limited to a single layer, with x being the input of the layer. I believe this should be generalized with a composition operator as in equation (7).
There seems to be a mistake in equation (8). The inclusion in X on the right is useless since the set on the left is an intersection with X.
Section 3.1, third paragraph: It is *easily* to see [...]
Section 3.2, first paragraph: [...] slack variables for *each each* point on the batch.
Section 4.1, first paragraph: [...] (see Figures 1b and *1b*). (should be e)

Edit:

I still believe this paper lacks empirical evidence that the proposed loss would help train deep models without harming performance on tasks where depth matters. In light of all the corrections and clarifications, I upgrade my vote to weak reject.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, a problem of training DNNs having dead points and neurons was addressed. The problem was posed as a constraint optimization problem. The constraints were augmented to the loss functions of DNNs as a regularizer. The slack variables of the constrained loss functions were optimized together with parameters of DNNs to train DNNs avoiding dead points and neurons.

The paper was written well in general. However, the work is incomplete in terms of explanation of the details of the proposed method, and their experimental analyses. For instance:

- In the code, you implemented the constraints in a ReLU.  In this case, could you please compare the proposed SeparatingReLU with the other variation of ReLU such as parametric ReLUs etc.

- Could you please analyze convergence of parameters of DNNs together with slack variables during training? 

- Do you apply additional constraints on slack variables to control their scale?

- How do the proposed methods scale with larger datasets and networks? Please provide more detailed analyses for training larger/wider DNNs with different structures (such as VGG and ResNets) on larger datasets, such as at least Cifar 100 and ImageNet.

After the discussion:

I checked the response of the authors and comments of the other reviewers.

The authors consider analyses of various crucial properties of the methods in comparison with the other works as the future work. There are also some statements need to be verified and explained more precisely, such as regarding generalization properties and the relationship between network complexity, constraints and regularization. Therefore, I suggest authors to improve the related statements, and provide more detailed analyses for verification of the proposed results in the following submissions. 
review_title: Official Blind Review #2

Meta-review: This paper proposes constraints to tackle the problems of dead neurons and dead points. The reviewers point out that the experiments are only done on small datasets and it is not clear if the experiments will scale further. I encourage the authors to carry out further experiments and submit to another venue.
Decision: Reject