Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors introduce a simple trick to improve the training of binary-weighted RNNs.
The motivation of this work is to consider the difference in the distribution of weights between the input-to-hidden connection and hidden-to-hidden connection. The main idea is to use different scaling terms for each weight (i.e., input-to-hidden and hidden-to-hidden). This is shown with GRU-RNN implementation.

As shown in Fig. 1 (b), the performance is dramatically improved by using this simple scaling trick. I will complain a bit about the term 'contextual discretization'. It seems a bit overselling the idea after I learned that it is about hand-picking different scaling variables to different multiplication terms in the update function of GRU-RNN. The scaling terms are chosen after observing the distribution of weights in a GRU-RNN using full-precision, therefore the values reported in this paper may not apply in general. However, the method is surprisingly effective based on the reported results. The paper is clearly written, and the method is tested on two different domains, text and audio. 

Minor comments: citation to GRU-RNN is missing.

Pros:
A simple trick that works well.
Cons:
The method is not entirely new, I believe a similar trick was already proposed for non-recurrent nets.

review_title: A simple trick for binary recurrent nets

final_score: rating: 3: Clear rejection
review_content: In this paper, the authors propose a weight discretization scheme for training recurrent neural networks. The work shows that RNNs can be successfully trained on language modelling and music generation tasks when the binary weights are chosen to be much smaller and different for the hidden-to-hidden and input-to-hidden connections.

The idea of the paper is very simple but apparently effective. Some of the earlier research about binary and ternary weights did involve different choices for different parts of networks (e.g, the different GRU gates in Ott et al., 2016). Since recurrent neural networks are complicated non-linear dynamical systems, it makes sense that the scaling of the weights should be chosen carefully to allow for useful dynamics which are useful for solving practical problems. It's good that the paper draws attention to this. I think that the branding of this method as 'contextual discretization' is making it seem more elaborate than it really is. 

While the authors cite the work by Rastegari et al. (2016), they don't point out that the Binary-Weight-Networks described there are almost identical to the method proposed for implementing binary GRU's. In Binary-Weight-Networks, a multiplication with a binary weight vector is followed by a multiplication with a scalar which has been chosen to make the end result as similar as possible to a multiplication with the original real-valued weights. While that method has apparently not been applied to RNNs yet, it is more elegant than picking these values by hand and it can be seen as a more general method than the one proposed in this paper. I think that this similarity severely undermines the novelty of the ideas presented.

The paper is clearly written and easy to understand. Given how short the paper is, there could have been a more elaborate discussion of related work.

Pros:
The method is very simple yet effective for the tasks it is evaluated on and this is an interesting result.

Cons:
The method is almost identical to Binary-Weight-Nets.
The hand-picked scaling values could be very task specific.
The paper exaggerates the novelty of the method by giving it a new name.

review_title: null

Decision: Reject