Title: Regularizing Neural Networks by Penalizing Confident Output Distributions
Authors: Gabriel Pereyra
George Tucker
Jan Chorowski
Lukasz Kaiser
Geoffrey Hinton
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning
Speech
Structured prediction

Abstract: We propose regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. We connect our confidence penalty to label smoothing through the direction of the KL divergence between networks output distribution and the uniform distribution. We exhaustively evaluate our proposed confidence penalty and label smoothing (uniform and unigram) on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and our confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyper-parameters.


TL;DR: We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning.

Review discussion:
In this draft, the models are only evaluated (as is standard) by accuracy.
But often model probabilities are used as input to decision theory.
It seems one consequence of the entropy regularization is that it could un-calibrate probabilities.
Consider a problem with two classes. The first class has base rate 20% and the second has base rate 80%.
In the extreme case, if you crank up the entropy regularization you might get a model which always gives both classes 50% probability.
But this overestimates the frequency of the first label and underestimates the frequency of the second.
In short, this problem arises in part because this form of the confidence penalty pushes label distributions towards uniform.
Another formulation might push label distributions towards the base rate distribution instead. Have you considered such an alternative formulation?
*UPDATE* Thanks for the answer. This makes sense.
Question 2:
It might seem that achieving arbitrarily high entropy while preserving prediction accuracy should be trivial to accomplish. Any softmax softening maintains the order of the predictions exactly while increasing the entropy.
So while the experimental results appear strong and are clearly the interesting component of this paper, I suggest there's a big question mark over why precisely this method works. This regularizer doesn't limit the model complexity.
One explanation given in the paper suggests that the confident predictions are a symptom of overfitting, therefore by addressing overconfident predictions we might mitigate overfitting.
This argument appears doubly dubious.
First, there's a logical error: curing a symptom doesn't cure a disease (although it might be a valid source of hypothesis generation).
Second It's not clear that having more confident predictions is necessarily bad, or precisely at what point it becomes bad.
An alternative hypothesis is that you might get sharp errors (from cross entropy) when probabilities are very confident. Entropy regularization might effectively be providing some gradient clipping.
This could lead to faster convergence and a growing body of work (see Moritz Hardt's "Train faster, generalize better" suggests that faster convergence leads to better generalization).
I'm interested to hear your thoughts and a deeper explanation / discussion of what precisely is working - especially in light of the triviality of satisfying the entropy maximization goal while generating predictions with identical accuracy.
Thank you for highlighting this point.
Indeed, entropy regularization can mess up calibration.  We do report using the unigram frequency to push the predictive distribution towards the base rate and it works well on a language task where the class frequencies are very unequal.
Moreover, we also noticed that the confidence penalty is KL with the uniform flipped (compared to label smoothing), which suggests a way to do "unigram" confidence penalty (i.e., KL with the unigram flipped from unigram label smoothing) which we leave to future work.
Although accuracy does not change when smoothing the distribution, the log likelihood does change, which is what we optimize during training. This is similar to weight decay, where again scaling all of the weights trivially reduces the L2 norm without changing accuracy, but it does change the log likelihood.
In particular, what changes is the ratio between the different classes and as described in the distillation paper, we think this plays a role in the regularization effect we see.
Another observation is that once an example is correctly predicted, the only way to increase the log likelihood is to make the prediction sharper.
Hence the outputs become uncalibrated and unless capacity is controlled in some way, the network will put very sharp distributions on its predictions. These regularizers prevent this behavior.
Moreover, the Inception paper notes that the gradient vanishes on confident correctly predicted samples. This does not happen with label smoothing and the confidence penalty, which means that we do get some training signal for the lower layers even on correct predictions. This may improve data efficiency when doing multiple epochs (normally during the later passes only the few samples that the net doesn’t get confidently have any influence). This is similar to the trick from Yann LeCun’s “Efficient Backprop” in which the hyperbolic tangent was used on the output of the net, but it was scaled to have a range of [-1.1, 1.1], while the targets were +-1.
Thus the net was never driven into saturation.
We can interpret our approach as a regularizer encouraging the predictive distribution to be close to uniform.
So, when the model has little evidence it is regularized to the uniform, however, when sufficient evidence is accumulated, the predictive distribution matches the data.
Regarding AnonReviewer5's calibration concern and your reply:
In my pre-review questions, I mentioned the line of work on deterministic annealing.  This typically pushes the distribution toward uniform (at least in early stages of optimization).
But the idea of pushing it instead toward some other known reasonable distribution does appear in the literature, under the name "skewed deterministic annealing."
However, the motivation in that case was not to calibrate the final results, but rather to improve search by starting at the known distribution (rather than at uniform) and annealing away from it.
Your current related work section claims: "Penalizing the [low] entropy of a network's output distribution has not been explored in supervised learning."  Could you please clarify the sense in which this is true?
Surely you've heard of the maximum entropy principle (Jaynes 1957), which breaks ties among distributions that satisfy the data constraints by choosing the one with maximum entropy?  Or the whole line of work in the 1990's on deterministic annealing and information bottleneck methods?  These explicitly add an entropy regularizer similar to yours.  And it's not as if these papers were limited to unsupervised methods (Rao et al. 1990; Tishby et al., 1999) or to single-layer probabilistic classifiers (Berger et al. 1996).
For example, consider Miller et al. (1995) https://scl.ece.ucsb.edu/sites/scl.ece.ucsb.edu/files/publications/b95_4_0.pdf : "... probability distributions chosen to maximize entropy subject to a constraint on the expected classification error.  ... Our method is applicable to a variety of classifier structures, including ... multilayer perceptron-based classifiers."
Conversely, in some settings it makes sense to take beta < 0 and penalize HIGH entropy, on the grounds that the true classification is unambiguous and thus the true classifier will be confident.  This yields the entropy regularization technique of Grandvalet and Bengio (2005), which, in their semi-supervised setting, can be seen as a probabilistic analogue of transductive SVMs.
These ideas are well known and have been applied in many settings and with many variants.
Thus, I hope you can spend a day identifying a wider swath of relevant previous work (including work before 2014, and any work that may have conducted experiments similar to yours) and situating your own contribution more precisely with respect to it.
I'm not sure I understand your result tables.  Usually they have two sections.  Is the idea that the numbers above the line are copied from the literature, while the numbers below the line are from your own implementations?  You seem to boldface the lowest test number in each section, except in Table 4 - why is Table 4 different?  And are the differences among systems statistically significant?
In some sections you compare label smoothing or confidence penalty to dropout, while in other sections you combine them with dropout and compare to dropout alone.  Why the change?
How sensitive were your results to the choice of hyperparameter?  (You say something about this in section 4.4.2, although you don't seem to report your actual hyperparameters in section 4.4.2.)  Also, are you sure you did enough of a grid search in sections 4.1.1 and 4.1.2, given that you often picked one of the extreme values?
The results are missing in the last line of Table 6.  Do you have these results now?
I'm interested in AnonReviewer5's question 2, but AnonReviewer5 seems to be assuming that the main objective only cares about the ranking of the classes y, e.g., that it scores argmax_y p(y | x).  Is your answer that it's ok because your main objective term is log p(y|x)?  (\sum_y p(y|x) loss(y,y*) would also work, as in the many papers on deterministic annealing in supervised settings.)
NOTE: I may edit this post to add additional questions here.  Thought I'd be able to post them separately, but apparently not.
Thank you for pointing out these references. We will expand the relevant works situation to address these omissions and better situate our contributions, which are primarily experimental.
For our tables, above the line are results from the literature and below the line is our own implementation. We changed the tables to only bold the best result.
We compare with and without dropout depending on how severely performance degrades without dropout.
For example, if dropout is removed from the recurrent neural network used for Penn Treebank, it degrades to 120 perplexity. In these cases, we only compare with dropout. The goal was to show that our entropy penalty was something that could be added to models without retuning their hyper-parameters and achieve improved performance.
We will add supplementary plots showing performance as the hyperparameters are varied. We will also expand the range of hyperparameters considered for the experiments in sections 4.1.1 and 4.1.2.
We have completed the results in Table 6.
Yes, we agree, see our response to AnonReviewer5’s question 2 for more detail.
A bit more related work to discuss. My review says:
"I generally like this idea.  Regularizing the behavior of the model often makes more sense than regularizing its parameters.  The behavior is interpretable, whereas the parameters are uninterpretable and work together in mysterious ways.
So one might be able to choose a more sensible prior over the behavior.
In other words, prefer parameters if they lead to a plausible distribution."
The following group of papers all independently had the idea, at the same time, of regularizing toward a plausible distribution:
* posterior regularization (http://jmlr.csail.mit.edu/papers/volume11/ganchev10a/ganchev10a.pdf)
* generalized expectation criteria (http://www.jmlr.org/papers/volume11/mann10a/mann10a.pdf)
* learning from measurements (https://cs.stanford.edu/~pliang/papers/measurements-icml2009.pdf)
I was going to put this explanation in the review, but it's probably better here:
> Surely you've heard of the maximum entropy principle (Jaynes 1957), which breaks ties among distributions that satisfy the data constraints by choosing the one with maximum entropy?
A maximum-likelihood log-linear distribution (e.g., exponential family distribution) falls out of the goal to maximize the entropy while exactly predicting some expectations in the data.  The log-linear coefficients emerge as Lagrange multipliers on the predictive constraints.  One good tutorial is https://aclweb.org/anthology/J/J96/J96-1002.pdf , which I cited above.
If some slack is allowed in the predictions, then one gets a Lagrangian objective rather like the one you use, which balances high entropy with low slack penalty (high predictive accuracy).  That leads to the log-linear distribution that maximizes a regularized likelihood.  A reasonable review with refs is http://www.aclweb.org/anthology/W03-1018.
Differences from your work: These maximum entropy papers did not make any assumptions about the form of the distribution.  The parametric form simply emerged from the constraint that it should approximately predict a vector of feature expectations.
In contrast, you are restricting the distribution a priori to a parametric form, e.g., a neural network.  And instead of constraining the distribution to approximately predict a vector of feature expectations, you are constraining it to achieve good performance on some task.
But the various later papers I cited are closer to your setup, e.g., deterministic annealing, entropy regularization, ...
Have the authors compared their results with other smoothing techniques? E.g.:
- the virtual adversarial training
- Other smoothing technique is to force the network to estimate uniform output, which is then interpolated with the true CE objective:
L = -log(p_c) - lambda * Sum_i(t_i * log(p_i))  where t_i = 1/N, N: num of classes, p_c is the estimated posterior of the correct output.
MNIST result: Comparison with similar model size or compared to label smoothing: is the 0.08% or 0.06% improvement really significant?
confidence penalty: 1.0 is maybe not optimal. Have the authors tried larger value?
CIFAR: confidence penalty, 0.1 might be not optimal, have you tried \beta<0.1?
We did not experiment with virtual adversarial training, however will add it to the related work section. The primary reasons for not using it are:
* Two hyper-parameters
* The approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations,
which is significantly more computation in grid-searching and training than simply penalizing low entropy.
The second smoothing technique is denoted label smoothing in our manuscript.
We will add error bars. The difference between label smoothing and confidence penalty on MNIST is likely not statistically significant, but the later experiments back up the performance of the method.
We will extend the range of hyperparameters for the MNIST and Cifar10 experiments.
The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be?
The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning.
The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point.
In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy.
Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective.
But the paper doesn't indicate precisely what.
I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms?
The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing.
At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications.
Typo:
In related work: "Penalizing entropy" - you mean penalizing low entropy
Thank you for the encouraging review and for highlighting the empirical rigor of the manuscript.
Thank you for pointing out this confusing use of terminology. The “unigram smoothing” applies to the datasets where the input is a sequence of tokens (e.g., language modeling), where n-gram statistics would be applicable.
We are running experiments to quantify the gradient norm during training and will add the results to the paper. We have now added these to the paper.
Specifically, this paper suggests regularizing the estimator of a probability distribution to prefer high-entropy distributions.  This avoids overfitting.
I generally like this idea.  Regularizing the behavior of the model often makes more sense than regularizing its parameters.
After all, the behavior is interpretable, whereas the parameters are uninterpretable and work together in mysterious ways to produce the behavior.
So one might be able to choose a more sensible prior over the behavior.
In other words, prefer parameters not because they are individually close to 0 but because they jointly lead to a distribution that is plausible or low-risk a priori.
Pro: I believe that the idea is natural and sound (that is, I do not share the doubts of AnonReviewer5).
Pro: It's possible that this hasn't been well-explored yet in neural networks (not sure).
Pro: The experimental results look good.
So maybe everyone should use this kind of regularizer.
Con: It is a kind of pollution of the scientific literature to introduce this idea to the community as if it were unconnected to (almost) anything else in machine learning.  There are many, many papers that include a scaled entropy term in the optimization objective!  It's not just for reinforcement learning.  Please see the long list of connections in my pre-review questions / comments.
Con: Experimental results should always be accompanied by significance tests and error analysis.  Is your trained model actually doing better on the distribution of test data, or was your test set too small to tell?  Are the improvements robust across many different training sets?  What errors does your model fix, and what errors does it introduce?
Summary recommendation: Revise and resubmit.  ICLR has lots of submissions.  I would prefer to reward authors who not only tried something, but who properly contextualized it and carefully evaluated it.
Otherwise, there's a race to the bottom where everyone wants to be the first to try something, so that readers are confronted with a confusing sea of slapdash papers with unclear relationships.
Thank you for the encouraging review.
Thank you for the many citations to related literature; they have been very interesting to read. We will revise the related work section to appropriately situate our work.
I agree with your statement that experimental results should always be accompanied by significance tests and error analysis, however, this is rarely done (to the detriment of the field) and thus standards are not well established. Do you have an example with a good presentation of this analysis? As you mention, I can see three sources of randomness that we might want to quantify: randomness due to the particular train set, randomness due to the particular test set, and randomness inherent in the model training procedure. One approach would be to use the bootstrap estimate, is that what you would suggest? We will make a best effort to address this concern, however, for the larger models on the large datasets running repeated trials is particularly onerous.
We have extended the related work section and better contextualized our contributions.
We have added error bars for some tasks.
Although we have not strictly quantified the uncertainty due to the finite test set size, we feel that the results taken as a whole over multiple independent tasks support our conclusions.
The paper experimentally investigates a slightly modified version of label smoothing technique for neural network training, and reports results on various tasks. Such smoothing idea is not new, but was not investigated previously in wide range of machine learning tasks.
Comments:
The paper should report the state-of-the-art results for speech recognition tasks (TIMIT, WSJ), even if models are not directly comparable.
The error back-propagation of label smoothing through softmax is straightforward and efficient. Is there an efficient solution for BP of the entropy smoothing through softmax?
Although the classification accuracy could remain the same, the model will not estimate the true posterior distribution with this kind of smoothing.
This might be an issue in complex machine learning problems where the decision is made on higher level and based on the posterior estimations, e.g. language models in speech recognition.
More motivation is necessary for the proposed smoothing.
We thank the reviewer for their helpful comments.
We now list state-of-the-art results for the speech recognition tasks.
Yes, the gradient has an intuitive form. We now describe this in the text.
Yes, we agree that it could be a problem. We partially address this question with the speech experiments which use a beam search during decoding (but do not use a separately trained LM) and these models do benefit from the increased uncertainty. Without the penalty, the model may overly penalize deviations from the most likely path, making the beam search less effective.

Meta-review:
role: PC
data: comment: The reviewers agreed that the idea proposed in this paper is sensible and possibly very useful, and that the experiments are thorough with good results. However, they share strong doubts regarding the novelty of the proposed approach. Hopefully the discussion will help the authors refine this work. With a more thorough discussion of related ideas, such as entropy maximization, within the machine learning literature and a careful placement of this idea within that context, this could be a strong submission to a future conference.

Decision: Reject