Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a data noising technique for language modeling. The main idea is to noise a word history by using a probabilistic distribution based on N-gram smoothing techniques. The paper is clearly written and shows that such simple techniques improve the performance in various tasks including language modeling and machine translation. May main concern is that the method is too simple and sounds ad hoc, e.g., there is no theoretical justification of why n-gram smoothing based data noising would be effective for recurrent neural network based language modeling.

Comments: 
- p. 3 “can be seen has a way” -> “can be seen as a way” (?)
- p. 3. In general, the explanation about blank noising should be improved. Why does it avoid overfitting on specific contexts?
- p. 4. It would be better to provide more detailed derivations for a general form of unigram and blank noising equations.
- p. 5, Section 3.6: Is there any discussions about noising either/both input and output sequences with some numbers? This would be helpful information.

review_title: null

final_score: rating: null
review_content: Can you provide the detailed information of the data size (number of words and sentences) for both Penn Treebank and IWSLT2015 corpus so that readers can figure out the data size range of experiments?
Since the method is motivated by a smoothing method, and the discussion of the smoothing method depends on the scale of data (and the order of n-gram), it would be interesting to have such information.
Providing the computational costs of the proposed method would be more helpful.
Also, is it possible to apply it to larger scale data?


review_title: Amount of data.

final_score: rating: null
review_content: How do you compare the noising method proposed in this paper for LM with the counterpart in speech, which is often called multi-style learning (Large-vocabulary speech recognition under adverse acoustic environments, Interspeech, 2000), in terms of practical efficacy? Does the more effective method (noise-adaptive training) than noising or multi-style training described there for speech apply to LM here?  
review_title: comparison with noising methods in speech

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths
-	A simple “noising” method for improving LM
-	“noise” added to word history by using a probabilistic distribution using N-gram smoothing
-	Experimental evidence that such simple techniques improve LM and MT

Weaknesses
-	Purely empirical, with no theoretical justification
-	Rather primitive step so far; would be nice to see future work in modeling different types of LM “noise” as hidden variables and then “denoise” them.


review_title: A simplistic, empirical but practically useful approach to improve LM and MT

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper discusses data noising as a regularization technique for language modelling as an alternative to dropout regularization. The key idea is to adapt smoothing methods from ngram language modelling in such a fashion that they can be applied to continuous language models through noise. Through this motivation, the authors present noising analogies to standard discounting, as well as Kneser-Ney smoothing.

The experiments are convincing in that the smoothed (noised) models outperform their unregularized baselines.

My main issue with the evaluation in this paper is that there is no comparison between the noising/smoothing idea and more conventional regularizers (such as L2 and dropout) which were discussed in the paper. Likewise, it would have been interesting if the techniques proposed here had been applied to stronger base models (such as the models compared with in Table 2). Seeing that the noising technique is effectively just data augmentation it should have been reasonably trivial to blackbox the model and plug in Zaremba's or Gal's!

Those weaknesses aside (and I recommend the authors investigate improving their paper by adding these experiments), this paper presents a novel method for improving neural network learning for a number of sequence based problems, and does so convincingly. I strongly recommend the paper for acceptance.
review_title: null

Meta-review: The reviewers are reasonably supportive of this paper. The ideas presented in the paper are nice and the results are encouraging. The authors should consider, for the final version of this work, providing comparisons to other approaches on the text8 corpus (or on the 1 Billion Words corpus, Chelba et al.).
Decision: Accept (Poster)