Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I find some "key" terms not proper. It would be great if you can explain more. 

1. You said that you proposed a series of architectural modifications for LSTM networks. But in fact many of them are not "architectural modifications". In the Monte Carlo model averaging, you didn't change the model at all. What you did is to use a different method for inference. In Deep vector averaging, what you did is to simply combine an LSTM network with a MLP at the (pre)-output layer, you thus didn't modify the LSTM architecture. 

2. "Deep vector averaging": I was wondering if you are abusing the term "deep". Basically any MLPs can be used in spite of their depth. Can you also report the size of the MLPs you used? 
review_title: terms used in the paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: I agree with the other reviewer that the application areas are limited in the paper. I agree with the overall sentiment of the paper to evaluate effectiveness of some of the more recent techniques in this area, in conjunction with the recurrent networks. 

The paper advertises itself as a method (or a list of methods) of improving the recurrent baselines when performing experiments, however fails (or not shown) to generalize to other tasks. Effectiveness of these methods need to be shown across a wide variety of tasks if we intend to replace traditional baselines in general, rather than a specific subset of applications.

I like the desire to evaluate many of the recent techniques and having many replications of experiments towards this end (which is a strong point of the paper). However, whether there are synergies of some of the enhancements with sentiment analysis or not, we cannot see from these results. It would be interesting to see whether some of these results generalize across a wide variety of tasks.  
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents three improvements to the standard LSTM architecture used in many neural NLP models: Monte Carlo averaging, embed average pooling, and residual connections. Each of the modifications is trivial to implement, so the paper is definitely of interest to any NLP researchers experimenting with deep learning. 

With that said, I am concerned about the experiments and their results. The residual connections do not seem to consistently help performance; on SST the vertical residuals help but the lateral residuals hurt, and on IMDB it is the opposite. More fundamentally, there need to be more tasks than just sentiment analysis here. I'm not quite sure why the paper's focus is on text classification, as any NLP task using an LSTM encoder could conceivably benefit from these modifications. It would be great to see a huge variety of tasks like QA, MT, etc., which would really make the paper much stronger. 

At this point, while the experiments that are included in the paper are very thorough and the analysis is interesting, there need to be more tasks to convince me that the modifications generalize, so I don't think the paper is ready for publication.
review_title: review

final_score: rating: null
review_content: I find the experiments not convincing because the two datasets are quite similar (sentiment analysis). I was wondering if you have tried your proposed models/methods on much more different tasks (e.g. machine translation, question answering, etc.)
review_title: Datasets

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes and analyses three methods applied to traditional LSTMs: Monte Carlo test-time model averaging, average pooling, and residual connections. It shows that those methods help to enhance traditional LSTMs on sentiment analysis. 

Although the paper is well written, the experiment section is definitely its dead point. Firstly, although it shows some improvements over traditional LSTMs, those results are not on par with the state of the art. Secondly, if the purpose is to take those extensions as strong baselines for further research, the experiments are not adequate: the both two datasets which were used are quite similar (though they have different statistics). I thus suggest to carry out more experiments on more diverse tasks, like those in "LSTM: A Search Space Odyssey"). 

Besides, those extensions are not really novel.
review_title: official review

Meta-review: The paper attempts to perform an interesting exploration (how to combine different tricks for LSTM training) but does not take it far enough. 
 
 Pros:
 - interesting attempt at studying different techniques to improve LSTM training results
 Cons:
 - not very strong baselines
 - limited set of domains were explored
 - low in novelty (which wouldn't be a problem if the comparison was more thorough -- see above 2 points).
Decision: Reject