Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper propose a new "gate" function for LSTM to enable the values of the gates towards 0 or 1. The motivation behind is a  flat region of the loss surface is likely to generalize well. It shows the experimental results are comparable or better than vanilla LSTM and much more robust to low-precision approximation and low-rank approximation.

In section 3.2, the paper claimed using a smaller temperature cannot guarantee the outputs to be close to the boundary. Is there any experimental evidence to show it's not working? It also claimed pushing output gate to 0/1 will drop the performance. It actually quite interesting because there are bunch of paper claimed output gate is not important for language modeling, e.g. https://openreview.net/pdf?id=HJOQ7MgAW . 

In the sensitive analysis, what if apply rounding / low-rank for all the parameters? 

How was this approach compare to binarynet https://arxiv.org/abs/1602.02830 ? Applying the same idea, but only for forget gate/ input gate. Also, can we apply this idea to the binarynet? 

Overall, I think it's an interesting paper but I feel it should compare with some simple baseline to binarized the gate function.  

Updates: Thanks a lot for all the clarification. It do improve the paper quality but I'm still thinking it's higher than "6" but lower than "7". To me, improve ppl from "52.8" to "52.1" isn't very significant. For WMT, it improve on DE->EN but not for EN->DE (although it improve both for the author's own baseline). So I'm not fully convinced this approach could improve the generalization. But I feel this work can have many other applications such as "binarynet". 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper argues for pushing the input and forget gate’s output toward 0 or 1, i.e., the LSTM tends to reside in flat region of surface loss, which is likely to generalize well. To achieve that, the sigmoid function in the original LSTM is replaced by a function G that is continuous and differentiable with respect to the parameters (by applying the Gumbel-Softmax trick). As a result, the model is still differentiable while the output gate is approximately binarized.  

Pros:
-	The paper is clearly written
-	The method is new and somehow theoretically guaranteed by the proof of the Proposition 1
-	The experiments are clearly explained with detailed configurations
-	The performance of the method in the model compression task is promising 

Cons:
-	The “simple deduction” which states that pushing the gate values toward 0 or 1 correspond to the region of the overall loss surface may need more theoretical analysis
-	It is confusing whether the output of the gate is sampled based on or computed directly by the function G  
-	The experiments lack many recent baselines on the same dataset (Penn Treebank: Melis et al. (2017) – On the State of the Art of Evaluation in Neural Language Models; WMT: Ashish et.al. (2017) – Attention Is All You Need) 
-	The experiment’s result is only slightly better than the baseline’s
-	To be more persuasive, the author should include in the baselines other method that can “binerize” the gate values such as the one sharpening the sigmoid function. 


In short, this work is worth a read. Although the experimental results are not quite persuasive, the method is nice and promising. 

review_title: Interesting, but not impressive

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to push the LSTM gates to be binary. To achieve this, the paper proposes to employ the recent Gumbel-Softmax trick to obtain end-to-end trainable categorical distribution (taking 0 or 1 value). The resulted G2-LSTM is applied for language model and machine translation in the experiments. 

The novelty of this paper is limited. Just directly apply the Gumbel-Softmax trick. 

The motivation is not explained clearly and convincingly. Why need to pursue binary gates? According to the paper, it may give better generalization performance. But there is no theoretical or experimental evidence provided by this paper to support this argument. 

The results of the new G2-LSTM are not significantly better than baselines in the experiments.
review_title: The technical novelty is limited and experiments do not show much benefits of the proposed model.

Meta-review: This paper proposes training binary-values LSTMs for NLP using the Gumbel-softmax reparameterization.  The motivation is that this will generalize better, and this is demonstrated in a couple of instances.

However, it's not clear how cherry-picked the examples are, since the training loss wasn't reported for most experiments.  And, if the motivation is better generalization, it's not clear why we would use this particular setup.
Decision: Reject