Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper uses ADMM to justify hard thresholding heuristic, which thresholds out DNN's weights if the value is less than some amount.
I think the empirical case study is well carried, and recommend this for the workshop to have a discussion. Note that the convergence of non-convex ADMM is unknown, and this paper does not use ADMM in a strict sense. ADMM splits the optimization problem to multiple instances, and we need to solve those subproblems EXACTLY. This paper worked around this by obtaining very crude approximation of solution for the subproblem. But for practical purposes, I think this maybe enough.

Also I would encourage the authors to report training AND TESTING loss and accuracy. Non-convex ADMM has stability issue, so it's best to see 

1. Run the algorithm 10 times.
2. Report error bar in training and testing loss and accuracy.

In the optimization point of view, reducing the number of non-zero weights while preserving training loss makes sense, but in the machine learning point of view, if the procedure hurts the generalization property significantly, it is not useful.
So I recommend to the authors to include a performance of weight-pruned classifier on the test set.

review_title: Intersting empirical study

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to use an l0 norm regularizer (although the authors do not identify it as such) to achieve weight sparsity in DNN training. The authors propose to upper bound the l0 norm (equivalently, the number of non-zero components) of the weights of each layer, by adding an indicator function to the underlying loss function. The resulting (non-convex) optimization problem is then tackled via ADMM. The experimental results (though not very extensive) show that the approach is able to yield networks with many zero weights.  

The paper suffers from a few minor issues and unexplained aspects. How is the update with respect to W_i carried out? Is it solved exactly, or simply a few steps of SGD? The notation in Equation (5) is wrong since it is the notation used to represent products; the authors should use \Pi rather than \prod. Furthermore, notice that this projection may not be unique, due to the non-convexity of S_i. How are the \rho_i parameters selected? Does this choice affect the results?

review_title: A decent work, but with a few minor issues and unexplained aspects

Meta-review: Congratulations, your paper was accepted to the ICLR workshop.
Decision: Accept