Task: Language Identification
Input (Spanish, English): This manuscript tries to tackle neural network regularization by blending the target distribution with predictions of the model itself. In this sense it is similar in spirit to scheduled sampling (Bengio et al) and SEARN (Daume et al) DAgger (Ross et al) which consider a "roll-in" mixture of the target and model distributions during training. It was clarified in the pre-review questions that these targets are generated on-line rather than from a lagged distribution, which I think makes the algorithm pseudocode somewhat misleading if I understand it correctly.  This is an incremental improvement on the idea of label softening/smoothing that has recently been revived, and so the novelty is not that high. The author points out that co-label similarity is better preserved by this method but it doesn't follow that this is causal re: regularization; a natural baseline would be a fixed, soft label distribution, as well as one where the softening/temperature of the label distribution is gradually reduced (as one would expect for this method to do as the model gets closer and closer to reproducing the target distribution).  It's an interesting and somewhat appealing idea but the case is not clearly made that this is all that useful. The dropout baselines for MNIST seem quite far from results already in the literature (Srivastava et al 2014 achieves 1.06% with a 3x1024 MLP with dropout and a simple max norm constraint; the dropout baselines here fail to break 1.3% which is rather high by contemporary standards on the permutation-invariant task), and results for CIFAR10 are quite far from the current state of the art, making it difficult to judge the contribution in light of other innovations. The largest benchmark considered is SVHN where the reported accuracies are quite bad indeed; SOTA for single net performance has been less than half the reported error rates for 3-4 years now. It's unclear what conclusions can be drawn about how this would help (or even hurt) in a better-tuned setting.  I have remaining reservations about data hygiene, namely reporting minimum test loss/maximum test accuracy rather than an unbiased method for model selection (minimum validation set error, for example). Relatedly, the regularization potential of early stopping on a validation set is not considered. See, e.g. the protocol in Goodfellow et al (2013).
Output (English): en