Title: AN ANALYSIS OF FEATURE REGULARIZATION FOR LOW-SHOT LEARNING

Abstract: Low-shot visual learning, the ability to recognize novel object categories from very few, or even one example, is a hallmark of human visual intelligence. Though successful on many tasks, deep learning approaches tends to be notoriously datahungry. Recently, feature penalty regularization has been proved effective on capturing new concepts. In this work, we provide both empirical evidence and theoretical analysis on how and why these methods work. We also propose a better design of cost function with improved performance. Close scrutiny reveals the centering effect of feature representation, as well as the intrinsic connection with batch normalization. Extensive experiments on synthetic datasets, the one-shot learning benchmark "Omniglot", and large-scale ImageNet validate our analysis.

1. INTRODUCTION:

The current success of deep learning hinges on the ability to apply gradient-based optimization to high-capacity models. It has achieved impressive results on many large-scale supervised tasks such as image classification Krizhevsky et al. (2012) ; He et al. (2016) and speech recognition Yu & Deng (2012) . Notably, these models are extensively hungry for data.

In contrast, human beings have strong ability to learn novel concepts efficiently from very few or even one example. As pointed out in Lake et al. (2016) , human learning is distinguished by its richness and efficiency. To test whether machines can approach this goal, Lake et al. propose the invaluable "Omniglot" hand-written character classification benchmark Lake et al. (2011) , where each training class has very few examples and the ability to fast-learn is evaluated on never-seen classes with only one example.

There has been previous work on attaining rapid learning from sparse data, denoted as meta-learning or learning-to-learn Thrun. (1998); Baxter. (1998) . Although used in numerous senses, the term generally refers to exploiting meta-knowledge within a single learning system across tasks or algorithms. In theory, a meta-learning is able to identify the right "inductive bias shifts" from previous experiences given enough data and many tasks Baxter. (1998) . However, even if a well-designed convolutional neural network is a good "inductive bias shift" for a visual recognition task, it is still elusive to find the optimal parameter from a small training set without any prior knowledge.

To alleviate this issue, low-shot learning methods have been proposed to transfer knowledge from various priors to avoid over-fitting, such as Bart & Ullman (2005) . Recently, Hariharan & Girshick. (2016) propose a novel prior of gradient penalty, which works pretty well experimentally. Although an intuitive explanation is provided in Hariharan & Girshick. (2016) that a good solution of the network parameters should be stable with small gradients, it is mysterious why adding such a regularization magically improves the low-shot task by a large margin. Mathematical derivation shows that gradient penalty is closely related to regularizing the feature representation.

In this paper, we give more analysis, both empirically and theoretically, on why adding a gradient regularization, or feature penalty performs so well. Moreover, we carefully carry out two case studies: (1) the simplest non-linear-separable XOR classification, and (2) a two-layer linear network for regression. The study does give insight on how the penalty centers feature representations and make the learning task easier. Furthermore, we also theoretically show that adding another finallayer weight penalty is necessary to achieve better performance. To be added, close scrutiny reveals its inherent connection with batch normalization Ioffe & Szegedy (2015) . From a Bayesian point of view, feature penalty essentially introduces a Gaussian prior which softly normalizes the feature representation and eases the following learning task.

1.1. RELATED WORK:

There is a huge body of literature on one-shot learning and it is beyond the scope of this paper to review the entire literature. We only discuss papers that introduce prior knowledge to adjust the neural network learning process, as that is the main focus of this work.

Prior knowledge, or "inductive bias " Thrun. (1998) , plays an important role in one-shot or low-shot learning. To reduce over-fitting problems, we need to regularize the learning process. Common techniques include weight regularization Bishop (1995) . Recently in Hariharan & Girshick. (2016) , a gradient penalty is introduced, which works well experimentally in low-shot scenarios.

Various forms of feature regularization have been proposed to improve generalization: Dropout Srivastava et al. ( 2014) is effective to reduce over-fitting, but has been eschewed by recent architectures such as batch-normalization Ioffe & Szegedy (2015) and ResNets He et al. (2016) . Other forms have also been proposed to improve transfer learning performance, such as minimizing the correlation of features Cogswell et al. (2016) and the multiverse loss Littwin & Wolf (2015) .

Our work is also closely related to metric learning and nearest neighbor methods, in which representations from previous experience are applied in cross-domain settings Fink (2005) ; Koch et al. (2015) ; Goldberger et al. (2005) ; Chopra et al. (2005) . The insight lies in that a well-trained representational model have strong ability to generalize well on new tasks. In a recent work Santoro et al. (2016) , DeepMind proposed a Memory Augmented Neural Network (MANN) to leverage the Neural-Turing-Machine for one-shot tasks. However, in Vinyals et al. (2016) , it is found that a good initialization such as VGG-Net largely improves the one-shot performance. In our opinion, a good feature representations still play a central role in low-shot tasks.

1.2. CONTRIBUTIONS:

The main contributions of our comprehensive analysis are three-fold:

1. We carefully carry out two case studies on the influence of feature regularization on shallow neural networks. We observe how the regularization centers features and eases the learning problem. Moreover, we propose a better design to avoid degenerate solutions. 2. From Bayesian point of view, close scrutiny reveals internal connections between feature regularization and batch normalization. 3. Extensive experiments on synthetic, the "Omniglot" one-shot and the large-scale ImageNet datasets validate our analysis.

2. AN ANALYSIS OF FEATURE REGULARIZATION:

We briefly introduce the notations in our work: we denote uppercase A, bold a and lowercase a for matrices, vectors and scalars respectively. For a vector a i , we denotes a i,j as its j-th element. ||.|| F stands for the Frobenius norm of a matrix. Given N examples {(x i , y i )|i = 1, ..., N }, we define E{.} as an expectation taken with respect to the empirical distribution generated by the training set.

Following Hariharan & Girshick. (2016) , we aim to learn a neural network model to extract the feature representations φ(x i ) and make predictions ŷi = W φ(x i ) with W = [w 1 , ..., w |C| ]. This setting includes both classification and regression problems, with |C| as the number of classes or target dimension, respectively. The problem can be generally formulated as:

EQUATION

where l(.) can be any reasonable cost function. In this paper, we focus on cross-entropy and L 2 loss due to their convexity and universality.

In Hariharan & Girshick. (2016) , it is suggested that adding a squared gradient magnitude loss (SGM) on every sample can regularize the learning process.

W * , φ * = arg min W,φ E{l(W, φ(x i ), y i ) + λ||∇ W l(W, φ(x i , y i ))|| 2 } (2)

The insight is that for a good solution, the parameter gradient should be small at convergence. However, we know that the convergence of a neural network optimization is a dynamic equilibrium. In other words, at a stationary point, we should have E{∇ W l(W, φ(x))} → 0. Intuitively when close to convergence, about half of the data-cases recommend to update a parameter to move positive, while the other half recommend to move negative. It is not very clear why small gradients on every sample E{||∇ W l(W, φ(x))|| 2 } produces good generalization experimentally.

Mathematical derivation shows that the optimization problem with gradient penalty is equivalent with adding a weighted L 2 regularizer φ(x i ):

arg min W,φ E{l(W, φ(x i ), y i ) + λα i ||φ(x i )|| 2 } (3)

where the example-dependent α i measures the deviation between the prediction ŷi and the target y i . In a regression problem, we have

α i = r 2 = ||ŷ i -y i || 2 , with the residual r = ŷi -y i ; in a classification problem, we have α i = k (p i k -I(y i = k)) 2 .

Intuitively, the misclassified high-norm examples might be outliers, and in a low-shot learning scenario, such outliers can pull the learned weight vectors far away from the right solution. In Hariharan & Girshick. (2016) , the authors compare dropping α i and directly penalizing ||φ(x i )|| 2 , which performs almost equally well.

In our work, we argue that a better and more reasonable design should be:

arg min W,φ E{l(W, φ(x i ), y i ) + λ 1 α i ||φ(x i )|| 2 } + λ 2 ||W || 2 F (4)

where we add another weight regularizer ||W || 2 F , which is necessary to avoid degenerate solutions. We will give further explanation in our second case study. In following analysis, we denote the cost in Eqn(4) with example-dependent α i as weighted L 2 feature penalty, and the example-independent (setting α i ≡ 1) as uniform L 2 feature penalty.

We carry out two case studies: (1) an XOR classification and (2) a regression problem, both empirically and theoretically to analyze how the uniform and weighted L 2 feature penalty regularize the neural network. In our paper, we will focus on the uniform feature regularization and will also cover the weighted scenario as well.

2.1. CASE STUDY 1: AN EMPIRICAL ANALYSIS OF XOR CLASSIFICATION:

First, we study the simplest linear-non-separable problem-exclusive-or (XOR). Suppose that we have four two-dimensional input points

x = [x 1 , x 2 ] T ∈ R 2 : {x 1 = [1, 1] T , x 2 = [0, 0] T } ∈ C + belongs to the positive class, while {x 3 = [1, 0] T , x 4 = [0, 1] T } ∈ C -the negative.

As shown in Figure 1 , we use a three-layer neural network to address the problem (left figure):

h 1 = x + b is a translation with b = [b 1 , b 2 ] T ∈ R 2 as the offset; h 2 = h 1,1 * h 1,2

is a non-linear layer, multiplying the first and second dimension of h 1 and producing a scalar; y is a linear classifier on h 2 parametrized by w. The original classification problem can be formulated as:

arg min w1,b,w 4 i=1 log(1 + exp(-y i * w * h i 2 ))

Suppose that we start from an initialization b = [0, 0] T , all three samples {x 2 , x 3 , x 4 } from different classes will produce the same representation h 2 = 0, which is not separable at all. It takes efforts to tune the learning rate to back-propagate w, b updates from the target y.

However, if we introduce the uniform L 2 feature regularization as: Then, we have:

arg min w1,b,w 4 i=1 log(1 + exp(-y i wh i 2 )) + λ 1 2 ||h i 2 || 2

λ 1 2 ∂||h 2 || 2 ∂b = λ 1 E{h 2 (x 2 + b 2 )} E{h 1 (x 1 + b 1 )} (5)

the gradient descent pulls Eqn(5) towards zero, i.e., pulling b towards b 1 = -E{x 1 } = -0.5 and b 2 = -E{x 2 } = -0.5.

As shown on the right of Figure 1 , the gradient of feature regularization pulls h 1 along the direction of red arrows. Then, we have h 2 > 0 for positive examples and h 2 < 0 for negative ones, which means h 2 is linearly-separable. In summary, we can observe that: Empirically, the feature regularization centers the representation h 2 = φ(x) and makes the following classification more learnable.

For the weighted case, the offset b have similar effects. It can be derived that when converged the feature representation will satisfy E{h 1 } = 0 and E{h 2 }=0. 

2.2. CASE STUDY 2: A COMPREHENSIVE ANALYSIS ON:

E{||y -W 2 W 1 x|| 2 }

where the latent feature is h = φ(x) = W 1 x. The optimization of {W 1 , W 2 } in this multi-layer linear neural network is not trivial, since it satisfies following properties:

1. The regression loss is non-convex and non-concave. It is convex on W 1 (or W 2 ) when the other parameter W 2 (or W 1 ) is fixed, but not convex on both simultaneously; 2. Every local minimum is a global minimum; 3. Every critical point that is not a global minimum is a saddle point; 4. If W 2 * W 1 is full-rank, the Hessian at any saddle point has at least one negative eigenvalue.

We refer interested readers to Baldi & Hornik (1989) ; Kawaguchi (2016) for detailed analysis.

In case of the uniform L 2 feature penalty, the problem becomes:

E(W 1 , W 2 ) = E{ 1 2 ||y -W 2 W 1 x|| 2 + λ 1 2 ||W 1 x|| 2 } + λ 2 2 ||W 2 || 2 F (6)

At the global minimum {W * 1 , W * 2 }, we should have:

∂E ∂W 1 | W * 1 = W T 2 Σ xy -W T 2 W 2 W 1 Σ xx + λ 1 W 1 Σ xx = 0 (7) ∂E ∂W 2 | W * 2 = Σ xy W T 1 -W 2 W 1 Σ xx W T 1 + λ 2 W 2 = 0 (8)

where we define the variance and covariance matrix as

Σ xx = E{xx T }, Σ xy = E{yx T }. Carrying out Eqn(7) * W T 1 -W T 2 * Eqn(8) = 0 reveals a very interesting conclusion: λ 1 E{||W 1 x|| 2 } = λ 2 ||W 2 || 2 F (9)

This reads as the expected L 2 feature penalty should be equal to final-layer weight regularizer when converged. Or equivalently, when close to convergence, the L 2 feature penalty reduces overfitting by implicitly penalizing the corresponding weight matrix W . A more generalized form is:

Lemma 1 For a cost function of form in Eqn (4) with uniform L 2 feature regularization:

arg min W,φ E{l(W, φ(x i ), y i ) + λ 1 ||φ(x i ))|| 2 } + λ 2 ||W || 2 F we have: λ 1 E{||φ(x)|| 2 } = λ 2 ||W || 2 F (10)

The φ(.) can take a quite general form of a convolutional neural network with many common nonlinear operations such as the ReLU, max-pooling and so on. One can follow the derivation of Eqn(9) to easily derive Lemma 1.

Lemma 1 also reveals the importance of adding the weight penalty ||W || 2 F in Eqn(4). If we only include the the feature penalty and drop the weight penalty (λ 2 = 0 in our case), then a scaling as φ(.) = γφ(.) and W = 1 γ W with γ < 1 will always decrease the energy and the solution will become very ill-conditioned with γ → 0.

2.2.1. L 2 FEATURE REGULARIZATION MAKES OPTIMIZATION EASIER:

Moreover, we analyze numerically how the L 2 feature penalty influences the optimization process in our regression problem.

We study a special case {x ∈ R d , y ∈ R} with W 1 ∈ R 1×m , W 2 ∈ R and include offsets b 1 ∈ R and b 2 ∈ R to make the problem more general. Then, the latent representation becomes h = W 1 x + b 1 and the prediction is ŷ = W 2 h + b 2 . The cost function of Eqn(4) becomes:

E(W 1 , b 1 , W 2 , b 2 ) = 1 2 E{(W 2 h + b 2 -y) 2 + λ 1 2 r 2 † h 2 } + λ 2 2 W 2 2 (11)

We define the prediction residual r and r † as ŷ -y for better readability, and substitute α i = (r i † ) 2 . Numerically, we apply a two-step process: in the first step, we calculate the sample-dependent r i † in the feed-forward process to obtain our L 2 feature regularization weights α i for each (x i , y i ); in the second step, we treat r † as a constant in the optimize. The gradient and Hessian matrix of Eqn(11) can be derived as:

∂E ∂W 2 = E{rh T } + λ 2 W 2 ∂E ∂b 2 = E{r} ∂E ∂h i = W 2 r i + λ 1 (r i † ) 2 h i ∂E ∂W 1 = W 2 E{rx T } + λ 1 E{r 2 † hx T } ∂E ∂b 1 = W 2 E{r} + λ 1 E{r 2 † h} and ∂ 2 E ∂W 2 2 = E{hh T } + λ 2 ∂ 2 E ∂b 2 2 = 1 ∂ 2 E ∂(h i ) 2 = W 2 2 + λ 1 (r i † ) 2 ∂ 2 E ∂W 2 1 = (W 2 2 + λ 1 E{r 2 † })E{xx T } ∂ 2 E ∂b 2 1 = W 2 2 + λ 1 E{r 2 † }

Suppose that we apply a second-order optimization algorithm for the network parameter θ ∈ {w 1 , w 2 , b 1 , b 2 }, the updates should be θ = -η * (∂ 2 E/∂θ 2 ) -1 (∂E/∂θ) with η > 0 as the step-size. If we unluckily start from a bad initialization point W 2 → 0, the updates of h i , W 1 and b 1 are of the form:

h i = -η(W 2 2 + λ 1 (r i † ) 2 }) -1 (W 2 r i + λ 1 (r i † ) 2 h i ) w 1 = -η[(W 2 2 + λ 1 E{r 2 † })E{xx T }] -1 (W 2 E{r} + λ 1 E{r 2 † h})E{x} b 1 = -η(W 2 2 + λ 1 E{r 2 † }) -1 (W 2 E{r} + λ 1 E{r 2 † h})

The updates will become very ill-conditioned without the regularization term (λ 1 = 0), since spectrum of the Hessian matrix is two-orders of infinitesimal O(W 2

2 ) and the gradient is of one-order O(W 2 ). In comparison, with a reasonable choice of λ 1 > 0, the computation can be largely stabilized when E{r 2 † } = 0. When the algorithm finally converges to a local minimum

{W * 1 , b * 1 , W * 2 , b * 2 }

, the expectation of parameter and latent feature should have gradient close to 0:

∂E ∂W 2 = E{rh T } + λ 2 W 2 → 0 ∂E ∂b 2 = E{r} → 0

Substituting this in the analysis of b 1 , we have:

EQUATION

In other words, the feature penalty centralizes the final hidden layer representation h(x) = φ(x).

Especially, in the uniform L 2 -feature penalty case, we simply drop α in Eqn ( 12) and have E{h} = E{φ(x)} → 0.

In summary, the feature penalty improves the numerical stability of the optimization process in the regression problem. The conclusion also holds for the classification.

Similar results for φ(x) can be extended to deeper multilayer perceptrons with convex differentiable non-linear activation functions such as ReLU and max-pooling. In an m-layer model parametrized by {W 1 , W 2 , ..., W m }, the Hessian matrix of hidden layers becomes strongly convex by backpropagating from the regularizer ||σ m-1

(W m-1 * σ m-2 (W 2 * (... * σ 1 (W 1 x))))|| 2 . 2.3 UNIFORM L 2 FEATURE PENALTY IS A SOFT BATCH NORMALIZATION

In our two case studies, we can observe that L 2 feature penalty centers the representation φ(x), which reminds us of the batch normalization Ioffe & Szegedy (2015) with similar whitening effects.

We reveal here that the two methods are indeed closely related in spirit.

From the Bayesian point view, we analyze a binary classification problem: the probability of prediction ŷ given x observes a Bernoulli distribution p(ŷ|x, φ, W ) = Ber(ŷ|sigm(W φ(x)), where φ(x) ∈ R d is a neural network parametrized by φ and W ∈ R 1×d . Assuming a factorized Gaussian prior on φ(x) ∼ N (0, Diag(σ 2 1 )) and W ∼ N (0, Diag(σ 2 2 )), we have the posterior as:

EQUATION

Applying the maximum-a-posteriori (MAP) principle to Eqn(13) leads to the following objective:

E{l(W, φ(x i ), y i ) + 1 2σ 2 1 ||φ(x i )|| 2 } + 1 2σ 2 2 ||W || 2 F + C (14) with C = -d ln( √ 2πσ 1 ) -d ln( √ 2πσ 2 ) is a constant. This is exactly the uniform L 2 version of Equation (4) with λ 1 = 1 2σ 2 1 , λ 2 = 1 2σ 2 2

and α i = 1. The i.i.d. Gaussian prior on W and φ(x) has the effect of whitening the final representation.

The difference between uniform L 2 feature penalty and batch normalization is that the former whitens φ(x) implicitly with an i.i.d. Gaussian prior during training, while the latter explicitly normalizes the output by keeping moving average and variance. We could say that the uniform L 2 penalty is a soft batch normalization.

As analyzed in Wiesler & Ney (2011) this kind of whitening improved the numerical behavior and makes the optimization converge faster. In summary, the L 2 feature regularization on φ(x) indeed tends to reduce internal covariate shift.

2.4. FEATURE REGULARIZATION MAY IMPROVE GENERALIZATION:

As pointed out in Hariharan & Girshick. (2016) , the gradient penalty or feature regularization improves the performance of low-shot learning experimentally. Here, we give some preliminary analysis on how and why our modified model in Eqn 4 may improve generalization performance in neural network learning. Denoting the risk functional (testing error) R(W ) as:

R(W ) = 1 2 E{||y -W φ(x)|| 2 } = 1 2 (x,y)∼P (x,y) ||y -W φ(x)|| 2 dP (x, y)

and empirical risk function (training error) R emp (W ) on {(x i , y i )|i = 1, ..., N } as:

R emp (W ) = 1 2N N i=1 ||y i -W φ(x i )|| 2

As we know, the training and testing discrepancy depends on the model complexity Vapnik (1998) :

EQUATION

where O * (.) denotes order of magnitude up to logarithmic factor. The upper bound 15 holds true with probability 1 -η for a chosen 0 < η < 1. In the equation, h is a non-negative integer named the VC-dimension. The right side of Eqn 15 contains two term: the first one ν(W ) is the error on training set while the second one models complexity of the learning system. In a multilayer neural network with ρ parameters and non-linear activations, the system has a VC dimension Sontag (1998) of O(ρ log ρ).

In our model in Eqn 4, the final cost function includes both feature and weight penalty as:

arg min W,φ E{l(W, φ(x i ), y i ) + λ 1 α i ||φ(x i )|| 2 } + λ 2 ||W || 2 F

Empirically, the term λ 2 ||W || 2 F enforces large margin which limits the selection space of W . The term λ 1 ||φ(x i )|| 2 not only improves numerical stability by feature whitening as discussed in the above sections, but also limits the selection of hidden layer parameter. These regularization terms thus reduces the VC-dimension of our learning. According to Eqn 15, the reduction of VC dimension of our model further reduces the theoretical discrepancy of training and testing performance.

3. EXPERIMENTAL RESULTS:

We evaluate our algorithm on three datasets: synthetic XOR datasets, the Omniglot low-shot benchmark and the large-scale ImageNet dataset. We use our own implementation of SGM approach Hariharan & Girshick. (2016) for a fair comparison.

3.1. SYNTHETIC DATASETS:

We first evaluate our model on the XOR dataset. Without loss of generality, we assume the data points x = [x 1 , x 2 ] T are uniformly sampled from a rectangle x 1 ∈ [-1, 1], x 2 ∈ [-1, 1], and y(x) = I(x 1 * x 2 > 0). The structure we use is a two-layer non-linear neural network with one 

Model:

One-shot Evaluation Random Guess 5% Pixel-KNN 26.7% MANN Santoro et al. (2016) 36.4% CNN Metric Learning 72.9% CNN (Our implementation) 85.0% Low-shot Hariharan & Girshick. (2016) 89.5% Matching Network Vinyals et al. (2016) 93.8% Ours 91.5%

Table 1 : Experimental results of our algorithm on the Omniglot benchmark Lake et al. (2011) .

latent layer h = σ(W 1 x + b) where σ(.) is the rectified linear unit. ADAM Kingma & Ba (2014) is leveraged to numerically optimize the cross entropy loss.

During training, we make use of only 4 points

{[1, 1], [-1, 1], [1, -1],

[-1, -1]}, while randomly sampled points from the whole rectangle as the test set. It is a very low-shot task. As shown in Figure 3 , our model with both feature and weight regularization outperforms the gradient penalty Hariharan & Girshick. (2016) and no regularization. We use uniform feature penalty as in Eqn(4) and set λ 1 = λ 2 = 0.1 in our experiments.

3.2. LOW-SHOT LEARNING ON OMNIGLOT:

Our second experiment is carried out on the Omniglot one-shot benchmark Lake et al. (2011) . Omniglot training set contains 964 characters from different alphabets with only 20 examples per each character. The one-shot evaluation is a pairwise matching task on completely unseen alphabets.

Following Vinyals et al. (2016) , we use a simple yet powerful CNN as feature representation model, consisting of a stack of modules, each of which is a 3 × 3 convolution with 128 filters followed by batch normalizationIoffe & Szegedy (2015), a ReLU and 2 × 2 max-pooling. We resized all images to 28×28 so that the resulting feature shape satisfies φ(x) ∈ R 128 . A fully connected layer followed by a softmax non-linearity is used to define the Baseline Classifier.

We set λ 1 =1e-4 in SGM Hariharan & Girshick. (2016) (2015) with the 20-way one-shot setting. 

3.3. LARGE-SCALE LOW-SHOT LEARNING ON IMAGENET:

Our last experiment is on the ImageNet benchmark Russakovsky et al. (2015) . It contains a wide array of classes with significant intra-class variation. divide the 1000 categories randomly into 400 base for training and evaluate our feature representation on the 600 novel categories.

We use a 50-layer residual network He et al. (2016) as our baseline. Evaluation is measured by top-1 accuracy on the 600 test-set in a 20-way setup, i.e., we randomly choose 1 sample from 20 test classes, and applies a nearest neighbor matching. As shown in Table 2 , we can see that our model learns meaningful representations for unseen novel categories even with large intra-class variance.

3.4. COMPARISON WITH BATCH-NORMALIZATION:

As discussed in Section 2.3, feature penalty has similar effects with batch normalization Ioffe & Szegedy (2015) . It is of interest to compare the influence of two modules influence training performance of neural networks. We study the performance of the classification with and without each modules on four classic image classification benchmarks. For CIFAR-10 and ImageNet, we applied the Residual Net architecture He et al. (2016) , while stacked convolution layers with ReLU and max-pooling is applied for MNIST and Omniglot. For ImageNet benchmark evaluation, we test the top-1 accuracy on the validation set with 50,000 images.

Since in our model the feature penalty regularizer is applied only on the last hidden layer, we still keep the batch-normalization modules in previous layers in our "FP" model. As shown in Figure 4 , we observe that baseline models with neither "BN" nor "FP" takes much longer to converge and achieve inferior performance; our "FP" regularizer achieves almost the same performance on MNIST (both 99%), CIFAR-10 (both 93%) and Omniglot 1-shot (88% BN v.s. 89% FP); on Ima-geNet, "BN" performs better than our "FP" (75% BN v.s. 74% FP). With both batch-normalization and feature penalty modules added, we achieve the best classification performance on all four benchmarks.

4. CONCLUSION:

In this work, we conduct an analysis, both empirically and theoretically, on how feature regularization influences and improves low-shot learning performance. By exploiting an XOR classification and two-layer linear regression, we find that the regularization of feature centers the representation, which in turn makes the learning problem easier with better numerical behavior. From the Bayesian point of view, the feature regularization is closely related to batch normalization. Evaluation on synthetic, "Omniglot" one-shot and large-scale ImageNet benchmark validates our analysis.

:



Figures and Tables:

Figure 1: Case 1: an empirical study of the XOR classification task. The left figure: the network structure we use: h 1 is a linear transformation, h 2 is a non-linear transform of h 1 and y is the prediction; The right column: The linear transformation maps x to h 1 . As shown in the red arrow, an L 2 norm penalty on h 2 centers the feature of h 1 and make the points from different sets separable. 'X's refer to positive examples, and 'O's are negative ones.

Figure 2: Case 2: a comprehensive study of a two-layer linear neural network for regression task. We minimize the L 2 distance between the prediction ŷ = W 2 (W 1 x + b 1 ) + b 2 and y. The latent representation h = W 1 x + b 1 is a linear mapping.

A REGRESSION PROBLEM Next, we analyze a two-layer linear neural network as shown in Figure 2. Denoting the input as X = [x 1 , x 2 , ...] and the target as Y = [y 1 , y 2 , ...]. The regression loss can be formulated as:

Figure 3: Evaluation on the XOR classification task. The red, blue and green lines stand for the accuracy of the Neural Network without any regularization, only the L 2 feature penalty, and our model with both weight and feature regularizer, respectively.

Figure 4: Classification accuracy comparison of our feature penalty (termed "PN") with batchnormalization (termed "BN") Ioffe & Szegedy (2015) on MNIST, CIFAR-10, Omniglot and Im-ageNet benchmarks. We compare baseline methods with neither batch-normalization nor feature penalty, with each module added, and with modules included.

Experimental results of our algorithm on the ImageNet benchmark Russakovsky et al.

Table content: <table/>

Peer review (iclr2017):
Summary === This paper extends and analyzes the gradient regularizer of Hariharan and Girshick 2016. In that paper a regularizer was proposed which penalizes gradient magnitudes and it was shown to aid low-shot learning performance. This work shows that the previous regularizer is equivalent to a direct penalty on the magnitude of feature values weighted differently per example. The analysis goes to to provide two examples where a feature penalty favors a better representation. The first example addresses the XOR problem, constructing a network where a feature penalty encourages a representation where XOR is linearly separable. The second example analyzes a 2 layer linear network, showing improved stability of a 2nd order optimizer when the feature penalty is added. One last bit of analysis shows how this regularizer can be interpreted as a Gaussian prior on both features and weights. Since the prior can be interpreted as having a soft whitening effect, the feature regularizer is like a soft version of Batch Normalization. Experiments show small improvements on a synthetic XOR test set. On the Omniglot dataset feature regularization is better than most baselines, but is worse than Moment Matching Networks. An experiment on ImageNet similar to Hariharan and Girshick 2016 also shows effective low-shot learning. Strengths === * The core proposal is a simple modification of Hariharan and Girshick 2016. * The idea of feature regularization is analyzed from multiple angles both theoretically and empirically. * The connection with Batch Normalization could have broader impact. Weaknesses === * In section 2 the gradient regularizer of Hariharan and Girshick is introduced. While introducing the concept, some concern is expressed about the motivation: *And it is not very clear why small gradients on every sample produces good generalization experimentally.* This seems to be the central issue to me. The paper details some related analysis, it does not offer a clear answer to this problem. * The purpose and generality of section 2.1 is not clear. The analysis provides a specific case (XOR with a non-standard architecture) where feature regularization intuitively helps learn a better representation. However, the intended take-away is not clear. The take-away may be that since a feature penalty helps in this case it should help in other cases. I am hesitant to buy that argument because of the specific architecture used in this section. The result seems to rely on the choice of an x^2 non-linearity, which is not often encountered in recent neural net literature. The point might also be to highlight the difference between a weight penalty and a feature penalty because the two seem to encourage different values of b in this case. However, there is no comparison to a weight penalty on b in section 2.1. * As far as I can tell, eq. 3 depends on either assuming an L2 or cross-entropy loss. A more general class of losses for which eq. 3 holds is not provided. This should be made clear before eq. 3 is presented. * The Omniglot and ImageNet experiments are performed with Batch Normalization, yet the paper points out that feature regularization may be similar in effect to Batch Norm. Since the ResNet CNN baseline includes Batch Norm and there are clear improvements over that baseline, the proposed regularizer has a clear additional positive effect. However, results should be provided without Batch Norm so a 1-1 comparison between the two methods can be performed. * The ImageNet experiment should be more like Hariharan and Girshick. In particular, the same split of classes should be used (provided in the appendix) and performance should be measured using n > 1 novel examples per class (using k nearest neighbors). Minor: * A brief comparison to Matching Networks is provided in section 3.2, but the performance of Matching Networks should also be reported in Table 1. * From the approach section: *Intuitively when close to convergence, about half of the data-cases recommend to update a parameter to go left, while the other half recommend to go right.* Could the intuition be clarified? There are many directions in high dimensional space and many ways to divide them into two groups. * Is the SGM penalty of Hariharan and Girshick implemented for this paper or using their code? Either is acceptable, but clarification would be appreciated. * Should the first equal sign in eq. 13 be proportional to, not equal to? * The work is dense in nature, but I think the presentation could be improved. In particular, more detailed derivations could be provided in an appendix and some details could be removed from the main version in order to increase focus on the results (e.g., the derviation in section 2.2.1). Overall Evaluation === This paper provides an interesting set of analyses, but their value is not clear. There is no clear reason why a gradient or feature regularizer should improve low-shot learning performance. Despite that, experiments support that conclusion, the analysis is interesting by itself, and the analysis may help lead to a clearer explanation. The work is a somewhat novel extension and analysis of Hariharan and Girshick 2016. Some points are not completely clear, as mentioned above.