Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to add a regularizing term to the objective function which penalizes the estimation of distributions with small entropy. It is one of these small tricks that were tried out by various groups even though only few people mention it in publications because the changes in performance are small and the additional hyperparameter makes it unattractive. This is also reflected in the paper here, as the improvements are very small compared to the baselines and usually vanish if more care is taking w.r.t. traditional regularization approaches. 

Further remarks:

- The evaluation is done on a broad spectrum of tasks, but the selections of the baseline systems is questionable. Especially on WSJ, there is no good reason to take an attention based seq2seq model but not also a network trained in a hybrid fashion or with CTC. Especially the CTC experiment would have been of great interest since the criterion tends to favor sharp probabilities.

- A theoretical perspective on the convergence is not well established and a proper justification why this is should be able to improve neural network training is missing (except for the norms of the gradients on MNIST).  If the argument is that gradients saturate too quickly if probabilities go too high then I would like to see an experiment with the squared error criterion as comparison, where this effect is not that large. 

In total I appreciate the work and broad evaluation but would suggest to include this method in a larger comparison paper that describes several of these tricks. The paper is well written and certainly correct, and the required scope is clearly limited within a workshop. Yet I would like to see some of the points here addressed before recommending acceptance. 
review_title: Another softmax smoothing technique

final_score: rating: 7: Good paper, accept
review_content: The paper proposes using confidence as a term for regularization in neural networks, helping to prevent overfitting by penalizing overly confident predictions.  The experiments range across a number of fields and architectures, helping to show both the generality of the technique and where it appears to be most helpful.

The work and experiments are rather detailed and exhaustive, especially when delving in to the Appendix for specific details of the various experiments. The confidence penalty regularization is compared to and combined with dropout and label smoothing. I do agree with another reviewer that some of the baseline systems are weaker than others. Seeing an LSTM used without recurrent dropout (variational dropout, zoneout, ...) as a baseline for language modeling is unfortunate for example. Even with that acknowledged, the results and analysis over a variety of datasets is enough to convince me of the capability of confidence penalization as a regularization technique.

Overall, I think the paper makes a good contribution to the knowledge and application of various smoothing techniques and introduces the benefits of confidence penalization as a competing and/or complementary regularization technique. The paper is clearly written and detailed in the number and variety of experiments performed.
review_title: Thorough evaluation of smoothing techniques, interesting introduction of confidence penalization

Decision: Accept