Title: An Analysis of Feature Regularization for Low-shot Learning
Authors: Zhuoyuan Chen
Han Zhao
Xiao Liu
Wei Xu
Venue: ICLR 2017
Keywords: Deep learning
Computer vision

Abstract: Low-shot visual learning, the ability to recognize novel object categories from very few, or even one example, is a hallmark of human visual intelligence. Though successful on many tasks, deep learning approaches tends to be notoriously data-hungry. Recently, feature penalty regularization has been proved effective on capturing new concepts. In this work, we provide both empirical evidence and theoretical analysis on how and why these methods work. We also propose a better design of cost function with improved performance. Close scrutiny reveals the centering effect of feature representation, as well as the intrinsic connection with batch normalization. Extensive experiments on synthetic datasets, the one-shot learning benchmark “Omniglot”, and large-scale ImageNet validate our analysis.

TL;DR: An analysis of adding regularization for low-shot learning

Review discussion:
- Do you have an intuition why the alphas are not important in eq 3?
- Did you actually compare feature penalty to batch norm empirically?
Great thanks for the really good questions and suggestions!
About dropping alpha:
My insight is that empirically we found the centralizing phenomena of feature \phi(x) after we add the regularization, regardless of alpha. Since modifying alpha during iteration means modifying the cost function every iteration, we suggest that by simply dropping the weight alpha we can achieve better robustness.
Comparison with batch-normalization:
For the first few layers, we already added batch-normalization. Since the regularization is only added on the final feature representation, we recently compare all 4 cases: with/without adding our regularization and with/without batch-normalization on MNIST, Omniglot and Cifar. For the baseline case with neither regularization nor batch-norm, the algorithm converges much slower. With only batch-normalization, the algorithm slightly outperforms our regularization term. With both batch-normalization and regularization, we are able to achieve best performance and fastest convergence. We will try on larger dataset such as ImageNet for a more systematic study.
I liked the analysis, and the unexpected relation to covariate shift, but I am also curious wrt batch-norm:
- did you try adding feature penalty on more than the last hidden layer? Considering BN is usually added to every layer, why not do this as well?
- BN really forces a "normal distribution" on the features. Did you measure the density/distribution of learned features with and without the feature penalty? vs with BN?
- you argue that \phi "can take a quite general form [...], many common nonlinear operations such as the ReLU." I think I see how this might be done for piecewise linear units such as the ReLU, but I don't see how it's going to hold to "real" non-linear functions. Do you think it will?
Thanks!
Great thanks for your questions and deeply sorry for the late reply. I've been in NIPS these days.
1. Penalizing each layer is a very interesting idea. Thanks for reminding. We just tried it on MNIST and CIFAR, and found by tuning lambda the result is almost as good as penalizing only the last layer. It would be more interesting to try penalizing each layer on ImageNet, and we are trying it recently.
2. It is a very good suggestion to study the influence of penalization and batch-normalization on the distribution. We just tried it. If we directly apply feature penalization or batch-normalization after a linear layer such as a fully-connected or convolutional layer, the distribution looks very similar to a Gaussian with a little heavier tail;
in contrast, if conducted after Relu, feature distribution after BN becomes more skewed, while feature penalty makes the distribution observe a one-side Gaussian. We will further study the influence on different dataset with different network structure.
3. I think Relu is a special case of non-linear.
In case of other forms such as a sigmoid or tanh function, the distribution does change.
I'm not quite sure what point to take away from section 2.1. I understand how the regularizer centers feature representations, but I'm not quite sure why that should lead to better validation accuracy (as in figure 3) than without the feature regularizer. In the XOR example this seems heavily dependent on the choice of problem, the starting point b = [0, 0], and the x^2 non-linearity, which is non-standard. Why should centered features lead to better validation accuracy for other problems and networks? Should it perform better than weight decay (as in eq. 9)?
Great thanks for your question! Sorry for the delay. I'm in Barcelona attending NIPS...
About centering:
Yes. I think the XOR is a special case to demonstrate how centering makes the learning problem easier, especially in case of x1 * x2 non-linear activation.
In real learning problems, suppose we have an input X, prediction Y and we want to learn W s.t. WX=Y. Generally, a second-order method leads to W = (XX')^-1 (X'Y).
In case of centered X, the correlation (X*X') has better condition number \lambda_max / \lambda_min and its inversion makes better sense. We guess that is why our model works. We will make this part more clear in our paper.
About weight decay:
As we know, adding weight decay improves generalizing ability and is better than adding nothing. According to our experiment, adding both feature penalty and weight penalty achieves the best result. It is a little tricky to compare adding weight decay only and feature penalty only,  since the latter sometimes degenerates. We believe a good prior may include both in the cost function.
The paper proposes to use a last-layer feature penalty as regularization on the last layer of a neural net.
Although the equations suggest a weighting per example, dropping this weight (alpha_i) works equally well.
The proposed approach relates to Batch Norm and weight decay.
Experiments are given on "low-shot" settting.
There seem to be two stories in the paper: feature penalty as a soft batch norm version, and low-shot learning; why is feature penalty specifically adapted to low-shot learning and not a more classical supervised task?
Regarding your result on Omniglot, 91.5, I believe it is still about 2% worse than the Matching Networks, which you refer to but don't put in Table 1. Why?
Overall, the idea is simple but feels like preliminary: while it is supposed to be a "soft BN", BN itself gets better performance than feature penalty, and both together give even better results. Is something still missing in the explanation?
-- edits after revised version:
Thank you for adding more information to the paper. I feel it is still too long but hopefully you can reduce it to 9 pages as promised.
However, I'm still not convinced the paper is ready to be accepted, mainly for the following reasons:
- on Omniglot, the paper is still significantly far from the current state of the art.
- the new experiments do not really confirm/infirm the relationship with BN.
- you added an explanation of why FP works for low-shot setting, by showing it controls the VC dimension and hence is good to control overfitting with a small number of training examples, but this discussion is basic and does not really shed more light than the obvious.
I'm pushing up your score from 4 to 5 for the improved version, but I still think it is below acceptance level.
To AnonReview2:
We would like to thank your reviews and suggestions. We have modified our paper accordingly and submitted a revised version on Jan 11th. Sorry for the long delay. It takes us a while to carry out experiments on the large-scale ImageNet benchmark.
* About why feature regularizer works and how it improves in case of low-shot learning and normal supervised learning:
This is the central question we try to answer in our paper, and we are carefully rethinking about this problem. In the origin paper of SGM, the intuitive explanation is that a large gradient might be outlier.
We observe that in supervised learning the CNN model achieves almost 100% accuracy on training and lower accuracy on testing experimentally.
Regarding the performance discrepancy from over-fitting, the training/testing performance discrepancy could be reduced if a good regularizer (with both feature penalty and weight decay) is introduced. The regularizer acts like a "max-margin" to limit the selection of parameter space and thus further reduce the "VC-dimension".
This is our preliminary guess and we have included some analysis in our revised version. We are working on a more strict answer.
* Comparison with Batch-Normalization:
Great thanks for the very good suggestion. We add classification performance comparison between our feature penalty method (FP) and batch normalization (BN). It is a little tricky to set up a fair comparison, since our model only includes regularization on the last hidden layer and BN modules are generally added on every layer. For now we still keep BN layers in previous layers. Our current comparison on supervised learning tasks indicates that the two methods achieves similar performance on MNIST, CIFAR-10 and Omniglot; on ImageNet, BN is slightly better than FP (75% v.s. 74%). Both BN and FP outperforms baseline CNN, and the best classification performance can be achieved with both modules added.
We include this part in our revised version. We notice that FP can substitute BN in every layer rather than only the last layer. We are working on a more complete comparison.
* Inclusion of performance of Matching Network
We have already added it in our revised version.
* Though some of the analysis in our paper is preliminary on shallow networks, some insight could still be valuable and extended to more general network structures. We wish the revised version could address some of the issues and we are now trying to improve our analysis to make it more general.
This paper proposes analysis of regularization, weight Froebius-norm and feature L2 norm, showing that it is equivalent to another proposed regularization, gradient magnitude loss. They then argue that: 1) it is helpful to low-shot learning, 2) it is numerically stable, 3) it is a soft version of Batch Normalization.
Finally, they demonstrate experimentally that such a regularization improves performance on low-shot tasks.
First, this is a nice analysis of some simple models, and proposes interesting insights in some optimization issues. Unfortunately, the authors do not demonstrate, nor argue in a convincing manner, that such an analysis extends to deep non-linear computation structures. I feel like the authors could write a full paper about "results can be derived for φ(x) with convex differentiable non-linear activation functions such as ReLU", both via analysis and experimentation to measure numerical stability.
Second, the authors again show an interesting correspondance to batch normalization, but IMO fail to experimentally show its relevance.
Finally, I understand the appeal of the proposed method from a numerical stability point of view, but am not convinced that it has any effect on low-shot learning in the high dimensional spaces that deep networks are used for.
I commend the authors for contributing to the mathematical understanding of our field, but I think they have yet to demonstrate the large scale effectiveness of what they propose.
At the same time, I feel like this paper does not have a clear and strong message. It makes various (interesting) claims about a number of things, but they seem more or less disparate, and only loosely related to low-shot learning.
notes:
- "an expectation taken with respect to the empirical distribution generated by the training set", generally the training set is viewed as a "montecarlo" sample of the underlying, unknown data distribution \mathcal{D}.
- "we can see that our model learns meaningful representations", it gets a 6.5% improvement on the baseline, but there is no analysis of the meaningfulness of the representations.
- "Table 13.2" should be "Table 2".
- please be mindful of formatting, some citations should be parenthesized and there are numerous extraneous and missing spacings between words and sentences.
To AnonReview1:
We appreciate your valuable comments and suggestions. We have modified our paper accordingly and submitted a revised version on Jan 11th. Sorry for the long delay. It takes us a while to carry out experiments on the large-scale ImageNet benchmark.
* About Non-linear cases to derive similar results:
Great thanks. More general idea could be derived for (1) non-linear ReLU and max-pooling as well as (2) deeper models with 3+ layers. Many other popular forms such as tanh could also be used.
We are working on this question and focus on ReLU case here. The ReLU operator on a hidden "h" and changes the 1st order gradient of dE/dh. A tricky problem is that ReLU is not 2nd-order differentiable with infinite Hessian. If we substitute ReLU(x)=max{0,x} with a 2nd-order differentiable version CReLU(x)=ln(1+e^x), the revised Hessian of Eqn(11) is still convex and could be numerically more stable with regularizer added.
Also, for max-pooling case, the selected max-value among the max operation channels will dominate the computation and set the 1st and 2nd order derivatives of non-maximum elements to zero.
These are our preliminary extension to the more common non-linear scenario and added in the revised version.
Though linear case is also non-trivial due to the non-convexity of the optimization problem as a whole, a more general analysis will make our conclusion more complete. We are working on the extension now.
* About comparison with batch-normalization
Great thanks for the very good suggestion. This issue is raised by several reviewers and is of importance to evaluate the proposed model completely. We add classification performance comparison between our feature penalty method (FP) and batch normalization (BN). It is a little tricky to set up a fair comparison, since our model only includes regularization on the last hidden layer and BN modules are generally added on every layer. For now we still keep BN layers in previous layers. Our current comparison on supervised learning tasks indicates that the two methods achieves similar performance on MNIST, CIFAR-10 and Omniglot; on ImageNet, BN is slightly better than FP (75% v.s. 74%). Both BN and FP outperforms baseline CNN, and the best classification performance can be achieved with both modules added.
We include this part in our revised version. We notice that FP can substitute BN in every layer rather than only the last layer. We are working on a more complete comparison.
* About why feature regularizer works in case of low-shot learning:
This is the central question we try to answer in our paper, and we are carefully rethinking about this problem from the angle of generalization ability.
In the origin paper of SGM, the intuitive explanation is that a large gradient might be outlier.
We observe that in the supervised learning the CNN model achieves almost 100% accuracy on training and lower accuracy on testing, especially the several low-shot scenarios experimentally. We regard the performance discrepancy is actually from over-fitting, due to the complexity and parameter amount of neural network models. The training/testing performance discrepancy could be reduced if a good regularizer (with both feature penalty and weight decay) is introduced. The regularizer acts like a "max-margin" (an analogy to SVM, the distance from support vectors to the plane) to limit the selection of parameter space and thus further reduce the "VC-dimension".
This is our preliminary guess and we have included some analysis in our revised version. We are working on improving it.
* Strict in presentation and reorganization of the paper:
Great thanks for the suggestions of presentation improvement. We have already modified some of them accordingly in our revised version. We will further proof-read to make it better.
Also, current version is a little bit too dense. We manage to include new experimental results and analysis in our revised paper. We will trim it down within 9 pages, with some detailed derivations left in supplemental materials.
I have read the updated version of the paper, I think it is improved and reads better, but for similar reasons to Reviewer2, I think this paper still would benefit from either a clearer experimental demonstration of the effects of the proposed method (from other measures than error metrics) or a task for which the improvements are more than marginal, either unsolved or new low-shot tasks that would emphasize the potential of your approach.
Summary
This paper extends and analyzes the gradient regularizer of Hariharan and
Girshick 2016. In that paper a regularizer was proposed which penalizes
gradient magnitudes and it was shown to aid low-shot learning performance.
This work shows that the previous regularizer is equivalent to a direct penalty
on the magnitude of feature values weighted differently per example.
The analysis goes to to provide two examples where a feature penalty
favors a better representation. The first example addresses the XOR
problem, constructing a network where a feature penalty encourages
a representation where XOR is linearly separable.
The second example analyzes a 2 layer linear network, showing improved stability
of a 2nd order optimizer when the feature penalty is added.
One last bit of analysis shows how this regularizer can be interpreted as
a Gaussian prior on both features and weights. Since the prior can be
interpreted as having a soft whitening effect, the feature regularizer
is like a soft version of Batch Normalization.
Experiments show small improvements on a synthetic XOR test set.
On the Omniglot dataset feature regularization is better than most baselines,
but is worse than Moment Matching Networks. An experiment on ImageNet similar
to Hariharan and Girshick 2016 also shows effective low-shot learning.
Strengths
* The core proposal is a simple modification of Hariharan and Girshick 2016.
* The idea of feature regularization is analyzed from multiple angles
both theoretically and empirically.
* The connection with Batch Normalization could have broader impact.
Weaknesses
* In section 2 the gradient regularizer of Hariharan and Girshick is introduced.
While introducing the concept, some concern is expressed about the motivation:
"And it is not very clear why small gradients on every sample produces
good generalization experimentally." This seems to be the central issue to me.
The paper details some related analysis, it does not offer a clear answer to
this problem.
* The purpose and generality of section 2.1 is not clear.
The analysis provides a specific case (XOR with a non-standard architecture)
where feature regularization intuitively helps learn a better representation.
However, the intended take-away is not clear.
The take-away may be that since a feature penalty helps in this case it
should help in other cases. I am hesitant to buy that argument because of the
specific architecture used in this section. The result seems to rely on the
choice of an x^2 non-linearity, which is not often encountered in recent neural
net literature.
The point might also be to highlight the difference between a weight
penalty and a feature penalty because the two seem to encourage
different values of b in this case.
However, there is no comparison to
a weight penalty on b in section 2.1.
* As far as I can tell, eq. 3 depends on either assuming an L2 or cross-entropy
loss. A more general class of losses for which eq. 3 holds is not provided. This
should be made clear before eq. 3 is presented.
* The Omniglot and ImageNet experiments are performed with Batch Normalization,
yet the paper points out that feature regularization may be similar in effect
to Batch Norm. Since the ResNet CNN baseline includes Batch Norm and there are
clear improvements over that baseline, the proposed regularizer has a clear
additional positive effect.
However, results should be provided without
Batch Norm so a 1-1 comparison between the two methods can be performed.
* The ImageNet experiment should be more like Hariharan and Girshick.
In particular, the same split of classes should be used (provided in
the appendix) and performance should be measured using n > 1 novel examples
per class (using k nearest neighbors).
Minor:
* A brief comparison to Matching Networks is provided in section 3.2, but the
performance of Matching Networks should also be reported in Table 1.
* From the approach section: "Intuitively when close to convergence, about half
of the data-cases recommend to update a parameter to go left, while
the other half recommend to go right."
Could the intuition be clarified? There are many directions in high
dimensional space and many ways to divide them into two groups.
* Is the SGM penalty of Hariharan and Girshick implemented for this paper
or using their code? Either is acceptable, but clarification would be appreciated.
* Should the first equal sign in eq. 13 be proportional to, not equal to?
* The work is dense in nature, but I think the presentation could be improved.
In particular, more detailed derivations could be provided in an appendix
and some details could be removed from the main version in order to increase
focus on the results (e.g., the derviation in section 2.2.1).
Overall Evaluation
This paper provides an interesting set of analyses, but their value is not clear.
There is no clear reason why a gradient or feature regularizer should improve
low-shot learning performance.
Despite that, experiments support that conclusion,
the analysis is interesting by itself, and the analysis may help lead to a
clearer explanation.
The work is a somewhat novel extension and analysis of Hariharan and Girshick 2016.
Some points are not completely clear, as mentioned above.
To AnonReview3:
We appreciate your valuable comments and suggestions. We have modified our paper accordingly and submitted a revised version on Jan 11th. Sorry for the long delay. It takes us a while to carry out experiments on the large-scale ImageNet benchmark.
* About why feature regularizer works and how it improves in case of low-shot learning:
This is the central question we try to answer in our paper, and we are carefully rethinking about this problem. In the origin paper of SGM, the intuitive explanation is that a large gradient might be outlier and should be penalized.
In several supervised learning tasks in the paper, the CNN model achieves almost 100% accuracy on training and lower accuracy on testing. We regard the performance discrepancy is actually from over-fitting, due to the complexity and parameter amount of neural network models. The training/testing performance discrepancy could be reduced if a good regularizer (with both feature penalty and weight decay) is introduced. The regularizer acts like a "max-margin" (an analogy with SVM) to limit the selection of parameter space and thus further reduce the "VC-dimension".
This is our preliminary guess and we have included some analysis in our revised version. We are working on improving it.
* Comparison with Batch-Normalization:
Great thanks for the very good suggestion. We add classification performance comparison between our feature penalty method (FP) and batch normalization (BN). It is a little tricky to set up a fair comparison, since our model only includes regularization on the last hidden layer and BN modules are generally added on every layer. For now we still keep BN layers in previous layers in our FP. Our current comparison on supervised learning tasks indicates that FP and BN achieves similar performance on MNIST, CIFAR-10 and Omniglot; on ImageNet, BN is slightly better than FP (75% v.s. 74%). Both BN and FP outperforms baseline CNN, and the best classification performance can be achieved with both modules added.
We include this part in our revised version. We notice that FP can substitute BN in every layer rather than only the last layer. We are working on a more complete comparison.
* More general forms of cost functions; Experiment setup of ImageNet; Inclusion of Matching Network in Table; Implementation of SGM and more:
Great thanks. We carried out some derivation on other cost function forms and found that general convex costs (e.g., the L2 and cross entropy loss) will favor our model when SGD is applied in optimization.
We use our own implementation by TensorFlow to compare with original SGM paper. We are currently asking the original authors for their setup on ImageNet and some details of their implementation. We will make it clear in our final version.
We already include the performance of Matching Network in our revised paper.
We have modified some presentations and will carefully proof-read it.
* About the case study of XOR classification:
Thanks for the suggestion. We did some re-derivation on the model and find that the choice of the uncommon non-linear layer h2=h11*h12 favors centralizing the features by moving "offsets".
Previously, we chose this special form on purpose to emphasize the whitening effect of feature penalty.
In case of common non-linear activation like ReLU, we find that the XOR classification becomes moving points on the simplex and regularization still helps. We will try to work out an example to better demonstrate the influence of the regularizer intuitively and understand the problem better.
* Reorganization of the paper:
Admittedly, current version is a little bit too dense. And our recent revised version is beyond page-limit (11 pages now).
We manage to include new experimental results and analysis in our revised paper.
It is a great idea to trim it down within 9 pages, with some detailed derivations left in supplemental materials.
We would like to thank all reviewers for their valuable comments and suggestions. We modify our paper accordingly.
Especially, we add classification performance comparison between our methods and batch normalization.
Moreover, we also add our preliminary analysis why the proposed regularizer could improve generalization performance, which is closely related to the improved "low-shot" improvement.
Admittedly, current version is a little bit beyond page-limit (11 pages now). We managed to include some new experimental results and analysis in our revised paper, and will trim it within 9 pages with some detailed deductions left in supplemental materials.

Meta-review:
role: PC
data: comment: The paper extends a regularizer on the gradients recently proposed by Hariharan and Girshick. I agree with the reviewers that while the analysis is interesting, it is unclear why this particular regularizer is especially relevant for low-shot learning. And the experimental validation is not strong enough to warrant acceptance.

Decision: Reject