Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper attempts to explain the recent practical success of binary weight neural networks by using the relationships between angles of binarized or non-binarized vectors in high dimensional spaces. Even though the paper is short, I think the main idea presented in the paper is important and insightful, and it can lead to interesting follow-up work and would be a good contribution to the workshop track.

I agree with all of the other reviewer's points (including the questions that the paper addresses however does not answer, which I consider to be okay for a workshop paper considering the ideas that it contributes), yet I think the paper is sufficient for acceptance. The exposition is a bit short and you can include your derivations even though you think it is straightforward, for the sake of completeness and ease of reading.


review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: Deep networks typically rely on relatively high precision arithmetic. There are a variety of advantages to reducing this precision requirement, by coming up with high performance networks with, eg, binary weights. Recent methods have been able to achieve good performance with binary weights (though they make use of continuous weights during training). This paper attempts to explain this performance with the simple but potentially important observation that the angle between a continuous weight vector and its binarized version is typically much larger than the angle between two random vectors in high dimensions.

-The theory could be more explicitly spelled out. Suppose the weight vectors are drawn randomly from a Gaussian or uniform(-1,1) distribution. What is the resulting distribution of the angle with the binarized vector?

-The argument for the prediction that the inner product with the binarized weights should be equal to the inner product with the continuous weights would appear to be neglecting the potential change in the norm of the weights. Eg, w^c \dot x will only equal w^b \dot x if the norm of w^c and w^b are similar. One possibility is that this highlights that a binarized network can only be sensitive to the angle to an input, but will struggle to be sensitive to the magnitude of an input. The magnitude can be extremely important: it may decide whether the activity lies in the saturating regime of a sigmoid, for instance. 

-It seems like a prediction of the theory is that, if the binarized copy is giving the same value as the original, you should be able to just train a standard continuous network and afterward binarize all the weights and obtain good performance. That this does not work may point to the significance of the magnitude of the weights.

-The paper could benefit from a more extensive overview of related work. Seung, Sompolinksy, & Tishby Physical Review A 1992, for instance, work out the training and generalization error of a perceptron with binary synapses. It would be useful to contrast this solution (where they directly search in the discrete weight space), with the gen/train error achieved by first learning a continuous perceptron and then binarizing the weights, using the arguments developed here.

-How does this interact with sparsity in the weights? 
review_title: null

Decision: Accept