Task: Language Identification
Input (Spanish, English): The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be?   The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning.   The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point. In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy. Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective. But the paper doesn't indicate precisely what.  I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms?   The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing.   At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications.   Typo: In related work: "Penalizing entropy" - you mean penalizing low entropy
Output (English): en