Title: DISCRIMINATIVE K-SHOT LEARNING USING PROBABILISTIC MODELS

Abstract: This paper introduces a probabilistic framework for k-shot image classification. The goal is to generalise from an initial large-scale classification task to a separate task comprising new classes and small numbers of examples. The new approach not only leverages the feature-based representation learned by a neural network from the initial task (representational transfer), but also information about the classes (concept transfer). The concept information is encapsulated in a probabilistic model for the final layer weights of the neural network which acts as a prior for probabilistic k-shot learning. We show that even a simple probabilistic model achieves state-of-the-art on a standard k-shot learning dataset by a large margin. Moreover, it is able to accurately model uncertainty, leading to well calibrated classifiers, and is easily extensible and flexible, unlike many recent approaches to k-shot learning.

1. INTRODUCTION:

A child encountering images of helicopters for the first time is able to generalize to instances with radically different appearance from only a handful of labelled examples. This remarkable feat is supported in part by a high-level feature-representation of images acquired from past experience. However, it is likely that information about previously learned concepts, such as aeroplanes and vehicles, is also leveraged (e.g. that sets of features like tails and rotors or objects like pilots/drivers are likely to appear in new images). The goal of this paper is to build machine systems for performing k-shot learning, which leverage both existing feature representations of the inputs and existing class information that have both been honed by learning from large amounts of labelled data. K-shot learning has enjoyed a recent resurgence in the academic community (Lake et al., 2015; Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Srivastava & Salakhutdinov, 2013) . Current stateof-the-art methods use complex deep learning architectures and claim that learning good features for k-shot learning entails training for k-shot specifically via episodic training that simulates many k-shot tasks. In contrast, this paper proposes a general framework based upon the combination of a deep feature extractor, trained on batch classification, and traditional probabilistic modelling. It subsumes two existing approaches in this vein (Srivastava & Salakhutdinov, 2013; Burgess et al., 2016) , and is motivated by similar ideas from multi-task learning (Bakker & Heskes, 2003) . The intuition is that deep learning will learn powerful feature representations, whereas probabilistic inference will transfer top-down conceptual information from old classes. Representational learning is driven by the large number of training examples from the original classes making it amenable to standard deep learning. In contrast, the transfer of conceptual information to the new classes relies on a relatively small number of existing classes and k-shot data points, which means probabilistic inference is appropriate.

While generalisation accuracy is often the key objective when training a classifier, calibration is also a fundamental concern in many applications such as decision making for autonomous driving and medicine. Here, calibration refers to the agreement between a classifier's uncertainty and the frequency of its mistakes, which has recently received increased attention. For example, Guo et al., 2017 show that the calibration of deep architectures deteriorates as depth and complexity increase. Calibration is closely related to catastrophic forgetting in continual learning. However, to our knowledge, uncertainty has so far been over-looked by the k-shot community even though it is high in this setting.

Our basic setup mimics that of the motivating example above: a standard deep convolutional neural network (CNN) is trained on a large labelled training set. This learns a rich representation of images at the top hidden layer of the CNN. Accumulated knowledge about classes is embodied in the top layer softmax weights of the network. This information is extracted by training a probabilistic model on these weights. K-shot learning can then 1) use the representation of images provided by the CNN as input to a new softmax function, 2) learn the new softmax weights by combining prior information about their likely form derived from the original dataset with the k-shot likelihood.

The main contributions of our paper are: 1) We propose a probabilistic framework for k-shot learning. It combines deep convolutional features with a probabilistic model that treats the top-level weights of a neural network as data, which can be used to regularize the weights at k-shot time in a principled Bayesian fashion. We show that the framework recovers L 2 -regularised logistic regression, with an automatically determined setting of the regularisation parameter, as a special case.

2) We show that our approach achieves state-of-the-art results on the miniImageNet dataset by a wide margin of roughly 6% for 1-and 5-shot learning. We further show that architectures with better batch classification accuracy also provide features which generalize better at k-shot time. This finding is contrary to the current belief that episodic training is necessary for good performance and puts the success of recent complex deep learning approaches to k-shot learning into context.

3) We show on miniImageNet and CIFAR-100 that our framework achieves a good trade-off between classification accuracy and calibration, and it strikes a good balance between learning new classes and forgetting the old ones.

2. PROBABILISTIC K-SHOT LEARNING:

K-shot learning task. We consider the following discriminative k-shot learning task: First, we receive a large dataset D = { u i , y i } N i=1 of images u i and labels y i ∈ {1, . . . , C} that indicate which of the C classes each image belongs to. Second, we receive a small dataset D = {u i , y i } N i=1 of C new classes, y i ∈ { C + 1, C + C}, with k images from each new class. Our goal is to construct a model that can leverage the information in D and D to predict well on unseen images u * from the new classes; the performance is evaluated against ground truth labels y * .

Summary. In contrast to several recent k-shot learning approaches that mimic the k-shot learning task by episodic training on simulated k-shot tasks, we propose to use the large dataset D to train a powerful feature extractor on batch classification, which can then be used in conjunction with a simple probabilistic model to perform k-shot learning. In 2003, Bakker & Heskes introduced a general probabilistic framework for multi-task learning with multi-head models, in which all parameters of a generic feature extractor are shared between a set of tasks, and only the weights of the top linear layer (the "heads") are task dependent. In the following, we frame k-shot learning in a similar setting and propose a probabilistic framework for k-shot learning in this vein. Our framework comprises four phases that we refer to as 1) representational learning, 2) concept learning, 3) k-shot learning, and 4) k-shot testing, cf. Fig. 1 (right). We then show that, for certain modelling assumptions, the obtained method is equivalent/related to regularised logistic regression with a specific choice for the regularisation parameter. 

2.1. A FRAMEWORK FOR PROBABILISTIC K-SHOT LEARNING:

We provide a high-level description of the probabilistic framework and present a more detailed derivation in Appendix A. While it might appear overly formal, the resulting scheme will be simple and practical, and the probabilistic phrasing will make it extensible and automatic (no free parameters).

Feature extractor and representational learning. We first introduce a convolutional neural network (CNN) Φ ϕ as feature extractor whose last hidden layer activations are mapped to two sets of softmax output units corresponding to the C classes in the large dataset D and the C classes in the small dataset D, respectively. These separate mappings are parametrized by weight matrices W for the old classes and W for the new classes. Denoting the output of the final hidden layer as x = Φ ϕ (u), the first softmax units compute p(

y n | x n , W) = softmax( W x n ) and the second p(y n | x n , W) = softmax(Wx n ), cf. Fig. 1 (left).

For representational learning (phase 1) the large dataset D is used to train the CNN Φ ϕ using standard deep learning optimisation approaches. This involves learning the parameters ϕ of the feature extractor up to the last hidden layer, as well as the softmax weights W. The network parameters ϕ are fixed from this point on and shared across later phases.

Probabilistic modelling. The next goal is to build a probabilistic method for k-shot prediction that transfers structure from the trained softmax weights W to the new k-shot softmax weights W and combines it with the k-shot training examples. Thus, given a test image u * during k-shot testing (phase 4), we compute its feature representation x * = Φ(u * ), and the prediction for the new label y * is found by averaging the softmax outputs over the posterior distribution of the softmax weights given the two datasets,

EQUATION

To this end, we consider a general class of probabilistic models in which the two sets of softmax weights are generated from shared hyperparameters θ, so that p( W, W, θ) = p(θ)p( W|θ)p(W|θ) as indicated in the graphical model in Fig. 1 (right). In this way, the large dataset D contains information about θ that in turn constrains the new softmax weights W. We further assume that there is very little uncertainty in W once the large initial training set is observed and so a maximum a posteriori (MAP) estimate, as returned by standard deep learning, suffices. As a consequence of this approximation and the structure of the model, the original data D are not required for the k-shot learning phase. Instead, the weights learned from these data, W MAP , can themselves be treated as observed data, which induce a predictive distribution over the k-shot weights p(W| W MAP ) via Bayes' rule. This argument is fully explained in Appendix A. We refer to this step as concept learning (phase 2) and note that all probabilistic modelling happens in the definition of p( W, W, θ), (see Secs. 2.2 and 2.3).

During k-shot learning (phase 3) we treat this predictive distribution as our new prior on the weights and again use Bayes' rule to combine it with the softmax likelihood of the k-shot training examples D to obtain a new posterior over the weights that now also incorporates D,

EQUATION

Finally, we approximate Eq. ( 2) by its MAP estimate W MAP , so that the integral in Eq. (1) becomes

p(y * | x * , D, D) ≈ p(y * | x * , D, W MAP ) ≈ p(y * | x * , W MAP ).

(3)

2.2. CHOOSING A MODEL FOR THE WEIGHTS:

The probabilistic model over the weights is key: a good model will transfer useful knowledge that improves performance. However, the usual trade-off between model complexity and learnability is particularly egregious in our setting as the weights W are few and high-dimensional and the number of k-shot samples is small. With an eye on simplicity, we make two simplifying assumptions. First, treating the weights from the hidden layer to the softmax outputs as a vector, we assume independence.

Second, we assume the distribution between the weights of old and new classes to be identical,

EQUATION

After extensive testing, we found that a Gaussian model for the weights strikes the best compromise in the trade-off between complexity and learnability, cf. Sec. 4.2 for a detailed model comparison.

2.3. GAUSSIAN MODEL AND ITS RELATION TO LOGISTIC REGRESSION:

Our method. We use a simple Gaussian model p(w|θ) = N (w|µ, Σ) with its conjugate Normalinverse-Wishart prior p(θ) = p(µ, Σ) = N IW(µ, Σ | µ 0 , κ 0 , Λ 0 , ν 0 ), and estimate MAP solutions for the parameters θ MAP = {µ MAP , Σ MAP }. The approximations discussed in Sec. 2.1 lead to

p(W | D) ≈ p(W | W MAP ) = N (W | µ MAP , Σ MAP ),

and the posterior at k-shot time becomes

EQUATION

For details see Appendix C.1. For k-shot testing we use the MAP estimates for the weights of the new classes. We found that restricting the covariance matrix to be isotropic, Σ = σ 2 I, performed best at k-shot learning, probably due to the small number of data points to learn from as mentioned above.

Relation to logistic regression. Standard logistic regression corresponds to the maximum likelihood (MLE) solution of the softmax likelihood p(y n | x n , W) = softmax(Wx n ). Often, L 2regularisation on the weights W with inverse regularisation strength 1/C reg is used; the solution to this regularised optimisation problem corresponds to the MAP solution of a model with isotropic Gaussian prior on the weights with zero mean:

p(W | D) ∝ N (W|0, 1 2 C reg I) N n=1 p(y n | x n , W

). This method is analogous to Eq. ( 5). However, the probabilistic framework has several advantages: i) modelling assumptions and approximations are made explicit, ii) it is strictly more general and can incorporate non-zero means µ MAP , whereas standard regularised logistic regression assumes zero mean, and iii) the probabilistic interpretation provides a principled way of choosing the regularisation constant using the trained weights W: C reg = 2σ 2 W , where σ 2 W is the empirical variance of the weights W MAP . In k-shot learning, alternative (frequentist) methods such as cross-validation suffer in the face of the small number of k-shot examples, and are not applicable in 1-shot learning at all.

3. RELATED WORK:

Embedding methods map the k-shot training and test points into a non-linear space and perform classification by assessing which training points are closest, according to a metric, to the test points. Siamese Networks (Koch et al., 2015) train the embedding using a same/different prediction task derived from the original dataset and use a weighted L 1 metric for classification. Matching Networks (Vinyals et al., 2016) construct a set of k-shot learning tasks from the original dataset to train an embedding defined through an attention mechanism that linearly combines training labels weighted by their proximity to test points. More recently, Prototypical Networks (Snell et al., 2017) are a streamlined version of Matching Networks in which embedded classes are summarised by their mean in the embedding space. These embedding methods learn representations for k-shot learning, but do not directly leverage concept transfer.

Amortised optimisation methods (Ravi & Larochelle, 2017 ) also simulate related k-shot learning tasks from the initial dataset, but instead train a second network to initialise and optimise a CNN to perform accurate classification on these small datasets. This method can then be applied for new k-shot tasks.

Importantly, both embedding and amortised inference methods improve when the system is trained for a specific k-shot task: to perform well in 5-shot learning, training is carried out with episodes containing 5 examples in each class. The general statement appears to be that training specifically for k-shot learning is essential for building features which generalise well at k-shot testing time. The approach proposed in this paper is more flexible; it is not tailored for a specific k and, thus, does not require retraining when switching, e.g., from 5-shot to 10-shot learning. Moreover, Snell et al. (2017) find that using a larger number of k-shot classes for the training episodes (e.g., train with 20 k-shot classes per episode when testing on only 5 new k-shot classes) can be beneficial, and they choose this number by cross-validation on a validation-set. This is in alignment with our finding that training with more data and more classes improves performance at k-shot time.

Deep probabilistic methods include the approach developed in this paper. The methods in this family are not unique to deep learning, and the idea of treating weights as data from which to transfer has been widely applied in multi-task learning (Bakker & Heskes, 2003) . The work most closely related to our own is not an approach to k-shot learning per se, but rather a method for training CNNs with highly imbalanced classes (Srivastava & Salakhutdinov, 2013) . It is similar in that it trains a form of Gaussian mixture model over the final layer weights using MAP inference that regularises learning. Burgess et al. (2016) propose an elegant approach to k-shot learning that is an instance of the framework described here: a Gaussian model is fit to the weights with MAP inference. The evaluation is promising, but preliminary. One of the goals of this paper is to provide a comprehensive evaluation. While not using a probabilistic approach, Qiao et al., 2017 develop a method for k-shot learning that trains a recognition model to amortise MAP inference for the softmax weights which can then be used at k-shot learning time. While this method trains the mapping from activation to weights jointly with the classifier, and thus does not learn from the weights per se, it does exploit the structure in the weights for k-shot learning.

4. EXPERIMENTS:

The code used to produce the following experiments will be made available after review.

Dataset. miniImageNet has become a standard testbed for k-shot learning and is derived from the ImageNet ILSVRC12 dataset (Russakovsky et al., 2015) by extracting 100 out of the 1000 classes. Each class contains 600 images downscaled to 84 × 84 pixels. We use the 100 classes (64 train, 16 validation, 20 test) proposed by Ravi & Larochelle (2017) . As our approach does not require a validation set, we use both the training and validation data for the representational learning.

Representational learning. We employ standard CNNs that are inspired by ResNet-34 (He et al., 2016) and VGG (Simonyan & Zisserman, 2014) for the representational learning on the C base classes, cf. Phase 1 in Sec. 2.1. These trained networks provide both W MAP and the fixed feature representation Φ ϕ for the k-shot learning and testing. We employed standard data augmentation from ImageNet for the representational learning but highlight that no data augmentation was used during the k-shot training and testing. For details on the architecture, training, and data augmentation see Appendix D.4. t-SNE embeddings (Van der Maaten & Hinton, 2008) of the learned last layer weights show sensible clusters, which highlights the structure exploited by the probabilistic model, see Appendix E.1.

Baselines and competing methods. We compare against several baselines as well as recent stateof-the-art methods mentioned in Sec. 3. The baselines are computed on the features x = Φ φ (u) from the last hidden layer of the trained CNN: (i) Nearest Neighbours with cosine distance and (ii) regularized logistic regression with regularisation constant set either by cross-validation or (iii) using the variance of the weights, C = 2σ 2 W , as motivated by our probabilistic framework, cf. Sec. 2.3. We also compare against three recent k-shot methods: (i) Matching Networks1 (Vinyals et al., 2016) , (ii) Prototypical Networks, with numbers reported from Snell et al., 2017 and (iii) Meta-learner LSTM, with numbers reported from Ravi & Larochelle, 2017.

Testing protocol. We evaluate the methods on 600 random k-shot tasks by randomly sampling 5 classes from the 20 test classes and perform 5-way k-shot learning. Following Snell et al. (2017) , we use 15 randomly selected images per class for k-shot testing to compute accuracies and calibration.

4.1. RESULTS ON miniIMAGENET:

Overall k-shot performance. We report performance on the miniImageNet dataset in Tab. 1 and Figs. 2 and 3 . The best method uses as feature extractor a modified ResNet-34 with 256 features, (Ravi & Larochelle, 2017) 43.4 ± 0.8% 60.6 ± 0.7% -Prototypical Nets (1-shot) (Snell et al., 2017) 49.4 ± 0.8% 65.4 ± 0.7% -Prototypical Nets (5-shot) (Snell et al., 2017) 45.1 ± 0.8% 68.2 ± 0.7% -Table 1 : Accuracy on 5-way classification on miniImageNet. Our best method, an isotropic Gaussian model using ResNet-34 features consistently outperforms all competing methods by a wide margin. trained with all 600 examples per training class, and a simple isotropic Gaussian model on the weights for concept learning. Despite its simplicity, our method achieves state-of-the-art and beats prototypical networks by a wide margin of about 6%. The baseline methods using the same feature extractor are also state-of-the-art compared to prototypical networks and both logistic regressions show comparable accuracy to our methods except for on 1-shot learning. In terms of log-likelihoods, Log Reg (C = 2σ 2 W ) fares slightly better, whereas Log Reg (cv) is much worse. Deeper features lead to better k-shot learning. We investigate the influence of different feature extractors of increasing complexity on performance in Fig. 3 : i) a VGG style network (500 train images per class), ii) a ResNet-34 (500 examples per class), and iii) a ResNet-34 (all 600 examples per class). We find that the complexity of the feature extractor as well as training set size consistently correlate with the accuracy at k-shot time. For instance, on 5-shot, Gauss (iso) achieves 65% accuracy with a VGG network and 74% with a ResNet trained with all available data, a significant increase of almost 10%. Moreover, Gauss (iso) outperforms Log Reg (C = 2σ 2 W ) on 1-shot learning across models, and performs similarly on 5-and 10-shot. We attribute the difference to the former's ability to also model the mean of the Gaussian, whereas logistic regression assumes a zero mean.

Importantly, this result implies that training specifically for k-shot learning is not necessary for achieving high generalisation performance on this k-shot problem. On the contrary, training a powerful deep feature extractor on batch classification using all of the available training data, then building a simple probabilistic model using the learned features and weights achieves state-of-the-art. Recent models that use episodic training cannot leverage such deep feature extractors as for them the depth of the model is limited by the nature of training itself. The reference baseline in the k-shot learning literature is nearest neighbours, which performs on par with Gauss (iso) on 1-shot learning but is outperformed by all methods on 5-and 10-shot. This is evidence that building a simple classifier on top of the learned features works significantly better for k-shot learning than nearest neighbours.

Calibration. A classifier is said to be calibrated when the probability it predicts for belonging to a given class is on par with the probability of it being the correct prediction. In other words, when 2017), we consider the log likelihood on the k-shot test examples as well as Expected Calibration Error (ECE) as summary measures of calibration. ECE can be interpreted as the weighted average of the distance of the calibration curve to the diagonal. We find that Log Reg (C = 2σ 2 W ) and Gauss (iso) provide better accuracy and calibration than Log Reg (cross-validation), cf. Fig. 2 . The difference in calibration quality for different regularisations of logistic regression highlights the importance of choosing the right constant, as we discuss now.

Choice of the regularisation constant for logistic regression. The results so far suggest that training a simple linear model such as regularised logistic regression might be sufficient to perform well in k-shot learning. However, while the accuracy at k-shot time does not vary dramatically as the regularisation constant changes, the calibration does, and jointly maximizing both quantities is not possible, cf. the first two plots of Fig. 4 . The standard (frequentist) method to tune this constant is cross validation, which is not applicable in the 1-shot setting, and suffers from lack of data in 5-and 10-shot. Contrary, our probabilistic framework provides a principled way of selecting this regularisation parameter by transfer from the training weights: Log Reg (C = 2σ 2 W ) strikes a good balance between accuracy and log-likelihood. The third plot in Fig. 4 reports log-likelihood as a function of accuracy and provides further visualisation of the achieved trade-off between accuracy and calibration for Log Reg (C = 2σ 2 W ), as well as the failure of Log Reg (cross-validation) to achieve a good compromise in 5-and 10-shot.

Evaluation in an online setting. We also briefly consider the online setting, in which we jointly test on 80 old and 5 new classes, for which catastrophic forgetting (French, 1999) is a well known problem. During k-shot learning and testing we employ a softmax which includes both the new and the old weights resulting in a total of 85 weight vectors. We utilise ResNet-34 trained on 500 images per class to retain 100 test images on the old classes. While the k-shot weights were modelled probabilistically, we use the MAP estimate W MAP for the old weights. Accuracies are reported in W , only new), which has not been trained in the presence of the old weights, either completely forget the old classes or do not learn anything, respectively. Fig. 5 for i) all the 85 classes, ii) the old 80 classes only, and iii) the new 5 classes only. For 5-and 10-shot, Gauss (iso) and Log Reg (2σ 2 W ) only lose a couple of percent on the accuracy of the old classes, and perform well on the new classes, striking a good trade-off between forgetting and learning at k-shot time. For unregularised (MLE) logistic regression, the new weights completely dominate the old ones, highlighting that the right regularisation is important. Yet, cross-validation in this setting is often very challenging. When training Logistic Regression without including the old weights ("only new"), the new weights are dominated by the old ones and fail to learn the new classes, making training in the presence of the old weights an essential component for online learning.

4.2. MODEL COMPARISON ON CIFAR-100:

We performed an extensive comparison between different probabilistic models of the weights using different inference procedures, which we present in Appendix E.4. We report results on the CIFAR-100 dataset on (i) Gaussian, (ii) mixture of Gaussians, and (iii) Laplace, all with either MAP estimation or Hybrid Monte Carlo sampling. We found that the simple Gaussian model is on par with or outperforms other methods at k-shot time, which we attribute to it striking a good balance between choosing a complex model, which may better fit the weights, and statistical efficiency, as the number of weights C (80 in our case) is often smaller than the dimensionality of the feature representation (256 in our case), cf. Sec. 2. This finding is supported by computing the log-likelihood of held out training weights under such model, with the Gaussian model performing best. Experiments using Hybrid Monte Carlo sampling for k-shot learning returned very similar performance to MAP estimation and at a much higher computational cost, due to the difficulty of performing sampling in such a high dimensional parameter space. Our recommendation is that practitioners should use simple models and employ simple inference schemes to estimate all free parameters thereby avoiding expending valuable data on validation sets.

5. CONCLUSION:

We present a probabilistic framework for k-shot learning that exploits the powerful features and class information learned by a neural network on a large training dataset. Probabilistic models are then used to transfer information in the network weights to new classes. Experiments on miniImageNet using a simple Gaussian model within our framework achieve state-of-the-art for 1-shot and 5-shot learning by a wide margin, and at the same time return well calibrated predictions. This finding is contrary to the current belief that episodic training is necessary to learn good k-shot features and puts the success of recent complex deep learning approaches to k-shot learning into context. The new approach is flexible and extensible, being applicable to general discriminative models and kshot learning paradigms. For example, preliminary results on online k-shot learning indicate that the probabilistic framework mitigates catastrophic forgetting by automatically balancing performance on the new and old classes.

The Gaussian model is closely related to regularised logistic regression, but provides a principled and fully automatic way to regularise. This is particularly important in k-shot learning, as it is a low-data regime, in which cross-validation performs poorly and where it is important to train on all available data, rather than using validation sets.

Appendix to "Discriminative k-shot learning using probabilistic models":

A DETAILS ON THE DERIVATION AND APPROXIMATIONS FROM SEC. 2.1

As stated in the main text, the probabilistic k-shot learning approach comprises four phases mirroring the dataflow:

Phase 1: Representational learning. The large dataset D is used to train the CNN Φ ϕ using standard deep learning optimisation approaches. This involves learning both the parameters ϕ of the feature extractor up to the last hidden layer, as well as the softmax weights W. The network parameters ϕ are fixed from this point on and shared across phases. This is a standard setup for multitask learning and in the present case it ensures that the features derived from the representational learning can be leveraged for k-shot learning.

Phase 2: Concept learning. The softmax weights W are effectively used as data for concept learning by training a probabilistic model that detects structure in these weights which can be transferred for k-shot learning. This approach will be justified in the next section. For the moment, we consider a general class of probabilistic models in which the two sets of weights are generated from shared hyperparameters θ, so that p( W, W, θ) = p(θ)p( W|θ)p(W|θ) (see Fig. 1 ).

Phases 3 and 4: k-shot learning and testing. Probabilistic k-shot learning leverages the learned representation Φ ϕ from phase 1 and the probabilistic model p( W, W, θ) from phase 2 to build a (posterior) predictive model for unseen new examples using examples from the small dataset D.

PROBABILISTIC MODEL OF THE WEIGHTS:

Given the dataflow and the assumed probabilistic model in Fig. 1 (right), a completely probabilistic approach would involve the following steps.

In the concept learning phase, the initial dataset would be used to form the posterior distribution over 

The graphical model in Fig. 1 entails that D is conditionally independent from D given W, such that

EQUATION

We recover Eq. ( 6) by adding p( D) to the constant of proportionality.

Inference in this model is generally intractable and requires approximations. The main challenge is computing the posterior distribution over the hyper-parameters given the initial dataset. However, progress can be made if we assume that the posterior distribution over the weights can be well approximated by the MAP value p( W | D) ≈ δ( W -W MAP ). This is an arguably justifiable assumption as the initial dataset is large and so the posterior will concentrate on narrow modes (with similar predictive performance). In this case p(θ | D) ≈ p(θ | W MAP ) and, due to the structure of the probabilistic model, all instances of D in Eq. ( 6) and Eq. ( 1) can be replaced by the analogous expressions involving W MAP . This greatly simplifies the learning pipeline as the probabilistic modelling only needs to have access to the weights returned by representational learning. Remaining intractabilities involve only a small number of data points D and can be handled using standard approximate inference tools. The following summarizes the approximations and computational steps for each phase of training. 

B APPROXIMATE INFERENCE METHODS:

In this section we briefly discuss different inference methods for the probabilistic models. In the main text we only considered MAP inference as we found that other more complicated inference schemes do not yield a practical benefit. However, in Appendix E.4 we provide a detailed model comparison, in which we also consider other approximate inference methods.

In all cases the gradients of the densities w.r.t. W can be computed, enabling MAP inference in the k-shot learning phase to be efficiently performed via gradient-based optimisation using L-BFGS (Liu & Nocedal, 1989) . Alternatively, Markov Chain Monte Carlo (MCMC) sampling can be performed to approximate the associated integral, see Eq. ( 1). Due to the high dimensionality of the space and as gradients are available, we employ Hybrid Monte Carlo (HMC) (Neal et al., 2011) sampling in the form of the recently proposed NUTS sampler that automatically tunes the HMC parameters (step size and number of leapfrog steps) (Hoffman & Gelman, 2014) . For the GMMs we employed pymc3 (Salvatier et al., 2016) to perform MAP inference.

C MODELS FOR THE PRIOR ON THE WEIGHTS:

As discussed in Sec. 2.1, we specify our model through p(W, W, θ) thus defining p(W | W M AP ) in Eq. ( 2). This section analyses different priors on the weights: (i) Gaussian models, (ii) Gaussian mixture models, and (iii) Laplace distribution. In the main paper, we only use a Gaussian model with MAP inference, as we saw no significant advantage in using other, more complex models. However, we provide an extensive comparison of the different models in Appendix E.4.

C.1 GAUSSIAN MODEL:

Possibly the simplest approach consists of modelling p(W | W) as a Gaussian distribution:

EQUATION

)

Details for this section can be found in Murphy, 2012 . The normal-inverse-Wishart distribution for µ and Σ is a conjugate prior for the Gaussian, which allows for the posterior to be written in closed form. More precisely,

EQUATION

)

where Z is the normalising constant. The posterior p(µ, Σ | W) also follows a normal-inverse-Wishart distribution:

EQUATION

where

µ N = κ 0 κ 0 + N µ 0 + N κ 0 + N W κ N = κ 0 + N Λ N = Λ 0 + S + κ 0 N κ 0 + N ( W -µ 0 )( W -µ 0 ) t ν N = ν 0 + N ,

and S is the sample covariance of W.

For this model, we can integrate (9) in closed form, which results in the following multivariate Student t-distribution:

p(W | W) = t ν N -p+1 µ N , Λ N (κ N + 1) κ N (ν N -p + 1)

.

As with other approaches, one can also compute the MAP solutions for the mean µ MAP and covariance

Σ MAP , such that p(W | W) = N (W | µ MAP , Σ MAP ).

For both the analytic posterior and the MAP approximation, p(W | W) depends on the hyperparameters of the normal-inverse-Wishart distribution: µ 0 , ν 0 , κ 0 and Λ 0 . There are different ways to choose these hyperparameters. One way would be by optimising the log probability of held out training weights, see Appendix E.4 for a brief discussion. In practise, it is common to choose uninformative or data dependent priors as discussed by Murphy (2012, Chapter 4) .

C.2 MIXTURE OF GAUSSIANS (GMM):

A Gaussian mixture model can potentially leverage cluster structure in the weights (animal classes might have similar weights, for example). This is related to the tree-based prior proposed in Srivastava & Salakhutdinov (2013) . MAP inference is performed because exact inference is intractable. Similarly to the Gaussian case, different structures for the covariance of each cluster were tested. In our experiments, we fit the parameters of the GMM via maximum likelihood using the EM algorithm. GMM consists on modelling p(W | W) as a mixture of Gaussians with S components:

EQUATION

where S s=1 π s = 1. In this work, we only compute the MAP mean and covariance for each of the clusters, as opposed to averaging over the parameters of the mixture. The resulting posterior is

EQUATION

)

The components of the mixture are fit in two ways. For CIFAR-100, the classes are grouped into 20 superclasses, each containing 5 of the 100 classes. One option is therefore to initialize 20 components, each fit with the data points in the corresponding superclass. For each such individual Gaussian, the MAP inference method presented in the previous section can be used. In order to increase the number of weight examples in each superclass, we merge the original superclasses into 9 larger superclasses. The merging of the superclasses is the following:

• Aquatic mammals + fish

• flowers + fruit and vegetables + trees

• insects + non-insect invertebrates + reptiles

• medium-sized mammals + small mammals

• large carnivores + large omnivores and herbivores

• people

• large man-made outdoor things + large natural outdoor things

• food containers + household electrical devices + household furniture

• Vehicles 1 + Vehicles 2.

The parameters of the mixture can also be fit using maximum likelihood with EM. We use the implementation of EM in scikit-learn. Both 3 and 10 clusters are considered in CIFAR-100. Weight log-likelihoods under this model and k-shot performance can be found in Appendix E.4.

Note that, similarly to the Gaussian model, we consider isotropic, diagonal or full covariance models for the covariance matrices.

C.3 LAPLACE DISTRIBUTION:

Sparsity is an attractive feature which could be helpful for modelling the weights. Indeed, it is reasonable to assume that each class uses a set of characteristic features which drive classification accuracy, while others are irrelevant. Sparse models would then provide sensible regularization. As such, we consider a product of independent Laplace distribution. Sec. 2.3 highlights the relation between a Gaussian prior on the weights and L 2 regularised logistic regression. One can similarly show that the Laplace prior is related to L 1 regularised logistic regression, which is well known for encouraging sparse weight vectors.

We consider a prior which factors along the feature dimensions:

p( W | {µ j }, {λ j }) = p j 1 2λ j exp   - C i | W ij -µ j | λ j   .

where the product over j is along the feature dimensions and the sum over i is across the classes. We fit the parameters µ and λ via maximum likelihood:

µ MLE,j = median i ( W ij ) λ MLE,j = 1 N i | W ij -µ j |, such that p(W | W) = p j 1 2λ MLE,j exp - C i |W ij -µ MLE,j | λ MLE,j .

An isotropic Laplace model with mean µ and scale λ is also considered:

p( W | µ, λ) = 1 2λ exp - ij | W ij -µ| λ ,

where

µ MLE = median( W) λ MLE = 1 N p ij | W ij -µ|,

D TRAINING AND EVALUATION PROCEDURE DETAILS D.1 miniIMAGENET:

To construct miniImageNet we use the same classes as initially proposed by Ravi & Larochelle (2017) and used in (Snell et al., 2017) , which is split into 64 training classes (cf. Tab. 2), 16 validation classes (cf. Tab. 3), and 20 test classes (cf. Tab. 4). We will make a full list of image files available.

As we do not require a validation set, we combine the training and validation set to form an extended training set. We extract 600 images per class from the ImageNet 2012 Challange dataset (Krizhevsky et al., 2012) , scale the shorter side to 84 pixels and then centrally crop to 84 × 84 pixels, that is, we preserve the original aspect ratio of the image content. We use these coloured 84 × 84 × 3 images as input for representational and k-shot learning and testing.

In order to train very deep models, such as a ResNet, we need to perform data augmentation as is the case when training full ImageNet. We use the following standard data augmentation from ImageNet that we adapt to the size of the input images:

• random horizontal flipping • randomly paste image into 100 × 100 frame and cut out central 84 × 84 pixels • randomly change brightness, contrast, saturation and lighting

We highlight that we do not perform any data augmentation for the k-shot learning and k-shot testing but use the original 84 × 84 colour images as input to the feature extractor. 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 32, 34, 35, 36, 38, 40, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 73, 74, 75, 76, 77, 78, 79, 80, 82, 83, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 ] classes_heldout = [ 8, 11, 12, 20, 23, 26, 29, 30, 31, 33, 37, 39, 41, 47, 57, 68, 71, 72, 81, 84 ] We provide an exhaustive comparison of different probabilistic models for this k-shot learning task in Appendix E.4.

D.3 NETWORK ARCHITECTURE AND TRAINING: RESNET INSPIRED:

The network architecture is inspired by the ResNet-34 architecture for ImageNet (He et al., 2016) that uses convolution blocks, with two convolutions each, that are bridged by skip connections. As a base, we utilise the example code2 provided by tensorpack (https://github.com/ppwwyyxx/tensorpack), a neural network training library built on top of tensorflow (Martín Abadi et al., 2015) . We adapt the number of features as well as the size of the last fully connected layer to account for the smaller number of training samples and training classes. The final architecture is detailed in Tab. 5. (Srivastava, Hinton, et al., 2014) and regularisation of the weights in the fully connected layers. The networks are trained with the ADAM optimiser Kingma & Ba, 2014 with decaying learning rate.

ResNet:

The network is implemented in tensorpack using tensorflow. The points are coloured according to their respective superclass. The colouring by superclass makes the structure in the weights evident, as t-SNE overall recovers the structure in the dataset. For instance, oak tree, palm tree, willow tree and pine tree form a cluster on the bottom right. This structure motivates our approach, as the training weights contain information which may be useful at k-shot time, for instance given a few example from chestnut trees. Structure is still present and we observe meaningful patterns, even though the classes in miniImageNet are more unique than in CIFAR-100. For instance, goose, house finch, toucan, Arctic fox, green mamba and other animals are clustered on the top, with birds close to each other. Examples of other small clusters include poncho and miniskirt, or organ and oboe. For readability, not all class names are plotted.

E EXTENDED EXPERIMENTS E.1 T-SNE EMBEDDING OF THE WEIGHTS:

We provide t-SNE embeddings for the weights of a VGG network trained in CIFAR-100 and a ResNet-34 trained on miniImageNet. A structure in the weights is apparent and provides motivation for our framework. The results can be seen in Fig. 6 and Fig. 7 . ) to bright (C = 10). In addition to Fig. 4 we also provide results for calibration in terms of ECE (lower is better), which are consistent with log likelihoods (higher is better): The Bayesian inspired choice of the regularisation parameter strikes a good balance between accuracy and calibration and consistently outperforms cross-validated choice of the parameter.

E.2 EXTENDED RESULTS ON miniIMAGENET:



E.4 MODEL ASSESSMENT IN CIFAR-100:

This section reports an extensive model comparison on CIFAR-100, both for the model of the weights p(W | W) and for the inference procedure at k-shot time (MAP or Hybrid Monte Carlo (HMC) sampling using NUTS (Hoffman & Gelman, 2014), see the description of approximate inference algorithms in Appendix B). We report log-likelihood of the weights under different models, as well as accuracy, log-likelihood and calibration in a k-shot learning task. Tab. 7 and Tab. 8 show descriptions of the methods analysed for respectively phase 2 (concept learning) and phase 3 (k-shot learning) of our k-shot pipeline described in Sec. 2.1. 8 : Methods and inference procedure during phase 3 (k-shot learning) for the models used in Fig. 10 . This specifies the inference procedure used when computing p(W | D, W) for the specified prior distribution.

Method name:

In the main text, we only consider an isotropic Gaussian model with MAP inference since we do not observe benefits from using alternative methods in terms of k-shot performance and calibration. Moreover, while we report results on a VGG-like architecture, we could also use a ResNet architecture, and preliminary results point to the same conclusion as experiments on miniImageNet when switching from VGG to ResNet: the deeper features consistently lead to higher k-shot performance on all methods whereas the ordering of the methods stays roughly the same.

Analysis of the models on held-out training weights. First, we analyse how well the different prior models for the new softmax weights are able to fit the C training weights W. We randomly excluded 10 of those weights and evaluated their held-out negative log likelihood given the remaining C -10 weights. We emphasise that this approach also constitutes a principled way to set the hyperparameters of the prior and, critically, relies on an explicit probabilistic model.

The negative log probabilities are averaged over 50 random splits and results of best optimised values w.r.t. hyperparameters are shown in Tab. 9 for CIFAR-100 (lower is better). We find that all models of weight-space which means learning even simple models is difficult. These observations suggest that the use of mixture models in this type of k-shot learning framework is not beneficial and is in contrast to the approach of Srivastava & Salakhutdinov (2013) , who employ a tree-structured mixture model. The authors show compare a model in which the assignments to the superclasses in the tree are optimized over against a model with a naive initialisation of the superclass assignments, and show that the first outperforms the second. However, they do not compare against a simpler baseline, e.g., a single Gaussian model.

Overall, we observe that there is no significant benefit of more complex methods over the simple isotropic Gaussian, either in terms of accuracy, log-likelihood or calibration. Thus, our recommendation is that practitioners should use simple models and employ simple inference schemes to estimate all free parameters thereby avoiding expending valuable data on validation sets

:

as reimplemented and optimised by https://github.com/AntreasAntoniou/MatchingNetworks to produce results that are superior to those originally published.

https://github.com/ppwwyyxx/tensorpack/tree/master/examples/ResNet

Figures and Tables:

Figure 1: left: Shared feature extractor Φ ϕ and separate top linear layers W and W with corresponding softmax units on old and new classes. right: Graphical model for probabilistic k-shot learning.

Figure 2: Results for miniImageNet with ResNet-34 style architecture and 600 training images per class.From left to right: accuracy and log likelihood (higher is better) for different k, Expected Calibration Error (ECE, lower is better) vs accuracy for 5-shot learning, and Calibration curve for 5-shot learning. Results on other architectures can be found in Appendix E.2 .

Figure 3: Comparison of different network architectures and training set sizes on the k-shot learning task: VGG style network (trained on 500 images per class) and ResNet-34 style network (trained on 500 and 600 images per class, respectively). Both, deeper networks and larger number of training images, give rise to features that transfer better to k-shot learning.

Figure5: Online learning with ResNet-34 features. Gauss (iso) and Log Reg (2σ 2 W ) strike a good trade-off between learning on new classes and forgetting of old classes. Unregularised Log Reg (MLE) and Log Reg (2σ 2 W , only new), which has not been trained in the presence of the old weights, either completely forget the old classes or do not learn anything, respectively.

the concept hyperparameters p(θ | D). The k-shot learning phase combines the information about the new weights provided by D with the information in the k-shot dataset D to form the posterior distribution p(W | D, D) ∝ p(W | D) n p(y n | x n , W) where p(W | D) = p(W | θ)p(θ | D)dθ. (6) To see this, notice that p(W | D, D) ∝ p(W, D, D) = p( D)p(W | D)p(D | D, W).

Representational learning. Deep learning is used to train a CNN. The representation of input images at the last hidden layer, x = Φ ϕ (u), is used in subsequent phases. The final layer softmax weights are assumed to be MAP estimates W MAP . Phase 2: Concept learning. A probabilistic model is fit directly to the MAP weights p(θ | W MAP ) ∝ p(θ)p( W MAP |θ). For conjugate models a full posterior can be retained, otherwise a MAP estimate p(θ | W MAP ) ≈ δ(θθ MAP ) is used. Phase 3: k-shot learning. The posterior distribution over the new softmax weights p(W | D, W MAP ) ∝ p(W | W MAP ) N n=1 p(y n |x n , W) is generally intractable. The posterior can, however, be approximated using the MAP estimate p(W | D, W MAP ) ≈ δ(W -W MAP ) or through sampling W m ∼ p(W | D, W MAP ). Note that p(W | W MAP ) = p(W|θ)p(θ | W MAP )dθ is analytic for conjugate models and, if instead a MAP estimate for θ is provided by the concept modelling stage, then p(W | W MAP ) ≈ p(W|θ MAP ). Phase 4: k-shot testing. Approximate inference is used to compute p(y * | x * , D, W MAP ) = p(y * | x * , W)p(W | D, W MAP )dW. If the k-shot learning phase provides a MAP estimate of W then p(y * | x * , D, W MAP ) ≈ p(y * | x * , W MAP ). If samples are returned then p(y * | x * , D, W MAP ) ≈ 1 M M m=1 p(y * | x * , W m ).

consists of 100 classes each with 500 training and 100 test images of size 32 × 32. The classes are grouped into 20 superclasses with 5 classes each. For example, the superclass "fish" contains the classes aquarium fish, flatfish, ray, shark, and trout. Unless otherwise stated, we used a random split into 80 base classes and 20 k-shot learning classes. For k-shot learning and testing, we split the 100 classes into 80 base classes used for network training and 20 k-shot learning classes. classes_base = [

Figure 6: t-SNE embedding of the CIFAR-100 weights W trained using a VGG style architecture.The points are coloured according to their respective superclass. The colouring by superclass makes the structure in the weights evident, as t-SNE overall recovers the structure in the dataset. For instance, oak tree, palm tree, willow tree and pine tree form a cluster on the bottom right. This structure motivates our approach, as the training weights contain information which may be useful at k-shot time, for instance given a few example from chestnut trees.

Figure 7: t-SNE embedding of the miniImageNet weights trained using a ResNet-34 architecture.Structure is still present and we observe meaningful patterns, even though the classes in miniImageNet are more unique than in CIFAR-100. For instance, goose, house finch, toucan, Arctic fox, green mamba and other animals are clustered on the top, with birds close to each other. Examples of other small clusters include poncho and miniskirt, or organ and oboe. For readability, not all class names are plotted.

Fig. 8 provides extended results on k-shot learning for the miniImageNet dataset for different network architectures. We investigate the influence of different feature extractors of increasing complexity and training data size on performance on: i) a VGG style network trained on 500 images per class, ii) a ResNet-34 trained on 500 examples per class, and iii) a ResNet-34 trained on all 600 examples per class.E.3 CHOICE OF REGULARISATION CONSTANTFig.9reports accuracy and calibration in terms of Expected Calibration Error (ECE) (lower is better) and log likelihoods (higher is better) for different regularisations of logistic regression for all three model architectures considered.

Network architecture. All unnamed layers are 2D convolutions with stated kernel size and padding SAME; the output of the shaded layer corresponds to Φ ϕ (u), the feature space representation of the image u, which is used as input for probabilistic k-shot learning.The network is trained using a decaying learning rate schedule and momentum SGD and is implemented in tensorpack using tensorflow.

Table content: <table><tr><td colspan="2">-34 inspired for miniImageNet</td></tr><tr><td>Output size</td><td>Layers</td></tr><tr><td>84 × 84 × 3 42 × 42 × 32 42 × 42 × 32 21 × 21 × 64 11 × 11 × 128 6 × 6 × 256 256</td><td>Input patch 5 × 5, 32, stride 2 3 × 3, 32 × 3 3 × 3, 32 3 × 3, 64 × 4 3 × 3, 64 3 × 3, 128 × 6 3 × 3, 128 3 × 3, 256 3 × 3, 256 × 3 global average pooling</td></tr><tr><td>C</td><td>fully connected, softmax</td></tr></table>

Network architectures. All 2D convolutions have kernel size 3 × 3 and padding SAME; max-pooling is performed with stride 2. The output of the shaded layer corresponds to Φ ϕ (u), the feature space representation of the image u, which is used as input for probabilistic k-shot learningThe network architecture was inspired by the VGG networks Simonyan & Zisserman, 2014, but does not employ batch normalisationIoffe & Szegedy, 2015. To speed up training, we employ exponential linear units (ELUs), which have been reported to lead to faster convergence as compared to ordinary ReLUsClevert et al., 2015. To regularise the networks, we employ dropout

Table content: <table/>

Description of the inference for the parameters of the prior in phase 2 (concept learning) for the models in from Fig.10. This specifies the inference procedure for θ in p(w | θ) after observing the training weights W.

Table content: <table><tr><td/><td/><td colspan="2">Phase 2: Concept learning Prior distribution Inference</td></tr><tr><td>Gauss (iso) Gauss (MAP prior) Gauss (integr. prior) GMM (supercl.) GMM (3, iso) GMM (3, diag) GMM (10, iso) Laplace (diag)</td><td colspan="2">Gaussian isotropic covariance Gaussian isotropic covariance Gaussian full covariance GMM on superclasses iso. cov. GMM on 3 clusters iso. cov. GMM on 3 clusters diagonal cov. GMM on 10 clusters iso. cov. Laplace diagonal covariance</td><td>MAP MAP Integrated MAP MLE MLE MLE MLE</td></tr><tr><td>Method name</td><td/><td colspan="2">Phase 3: k-shot learning Prior distribution Inference</td></tr><tr><td colspan="2">Gauss (iso) MAP Gauss (MAP prior) MAP Gauss (MAP prior) HMC Gauss (integr. prior) MAP Gauss (integr. prior) HMC GMM (supercl.) MAP GMM (3, iso) MAP Laplace (diag) HMC Laplace (diag) MAP</td><td>Gaussian Gaussian Gaussian Gaussian Gaussian GMM on superclasses GMM on 3 isotropic comp. Laplace (diagonal) Laplace (diagonal)</td><td>MAP MAP HMC MAP HMC MAP MAP HMC MAP</td></tr></table>



Table content: <table/>

Peer review (iclr2018):
 Authors present a k-shot learning method that is based on generating representations with a pre-trained network and learning a regularized logistic regression using the available data. The regularised regression is formulated as a MAP estimation problem with the prior estimated from the weights of the original network connected final hidden layer to the logits — before the soft-max layer. The motivation of the article regarding “concepts” is interesting. It seems especially justified when the training set that is used to train the original network has similar objects as the smaller set that is used for k-shot learning. Maps shown in Figures 6 and 7 provide good motivation for this approach. Despite the strong motivation, the article raises some concerns regarding the method. 1. The assumption about independence of w vectors across classes is a very strong one and as far as I can see, it does not have a sound justification. The original networks are trained to distinguish between classes. The weight vectors are estimated with this goal. Therefore, it is very likely that vectors of different classes are highly correlated. Going beyond this assumption also seems difficult. The proposed model estimates using only one W matrix, the one that is estimated by training the original network in the most usual way. In this case, the prior over would have a large influence on the MAP estimate and setting it properly becomes important. As far as I can see, there is no good recipe presented in the article for setting this prior. 2. How is the prior model defined? It is the most important component of the method while precise details are not provided. How are the hyperparameters set? Furthermore, this detail needs to be in the main text. 3. With the isotropic assumption on the covariance matrix, the main difference between logistic regression, which is regularized by L2 norm and coefficient set proportional to the empirical variance, and the proposed method seems to be the mean vector . From the details provided in the appendix — which should be in the main text in my opinion — I believe this vector is a combination of the prior and mean of w_c across classes. If the prior is set to 0, how different is this vector from 0? Authors should focus on this in my opinion to explain why methods work differently in 1-shot learning. In the other problems, the results suggest they are pretty much the same. 4. Authors’ motivation about concepts is interesting however, if the model bases its prediction on mean of w_c vectors over classes, then I am not sure if authors really achieve what they motivate for. 5. Results are not very convincing. If the method was substantially different than baseline, I believe this would have been no problem. Given the proximity of the proposed method to the baseline with regularised logistic regression, lack of empirical advantage is an issue. If the proposed model works better in the 1-shot scenario, then authors should delve into it to explain the advantage. Minor comments: Evaluation in an online setting section is unclear. It needs to be rewritten in my opinion.