Title: ATTENTIVE WEIGHTS GENERATION FOR FEW SHOT LEARNING VIA INFORMATION MAXIMIZATION

Abstract: Few shot image classification aims at learning a classifier from limited labeled data. Generating the classification weights has been applied in many metalearning approaches for few shot image classification due to its simplicity and effectiveness. However, we argue that it is difficult to generate the exact and universal classification weights for all the diverse query samples from very few training samples. In this work, we introduce Attentive Weights Generation for few shot learning via Information Maximization (AWGIM), which addresses current issues by two novel contributions. i) AWGIM generates different classification weights for different query samples by letting each of query samples attends to the whole support set. ii) To guarantee the generated weights adaptive to different query sample, we re-formulate the problem to maximize the lower bound of mutual information between generated weights and query as well as support data. As far as we can see, this is the first attempt to unify information maximization into few shot learning. Both two contributions are proved to be effective in the extensive experiments and we show that AWGIM is able to achieve state-of-the-art performance on benchmark datasets.

1. INTRODUCTION:

While deep learning methods achieve great success in domains such as computer vision (He et al., 2016) , natural language processing (Devlin et al., 2018) , reinforcement learning (Silver et al., 2018) , their hunger for large amount of labeled data limits the application scenarios where only a few data are available for training. Humans, in contrast, are able to learn from limited data, which is desirable for deep learning methods. Few shot learning is thus proposed to enable deep models to learn from very few samples (Fei-Fei et al., 2006) .

Meta learning is by far the most popular and promising approach for few shot problems (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017; Ravi & Larochelle, 2016; Rusu et al., 2019) . In meta learning approaches, the model extracts high level knowledge across different tasks so that it can adapt itself quickly to a new-coming task (Schmidhuber, 1987; Andrychowicz et al., 2016) . There are several kinds of meta learning methods for few shot learning, such as gradient-based (Finn et al., 2017; Ravi & Larochelle, 2016) and metric-based (Snell et al., 2017; Sung et al., 2018) . Weights generation, among these different methods, has shown effectiveness with simple formulation (Qi et al., 2018; Qiao et al., 2018; Gidaris & Komodakis, 2018; 2019) . In general, weights generation methods learn to generate the classification weights for different tasks conditioned on the limited labeled data. However, fixed classification weights for different query samples within one task might be sub-optimal, due to the few shot challenge.

We introduce Attentive Weights Generation for few shot learning via Information Maximization (AWGIM) in this work to address these limitations. In AWGIM, the classification weights are generated for each query sample specifically. This is done by two encoding paths where the query sample attends to the task context. However, we show in experiments that simple cross attention between query samples and support set fails to guarantee classification weights fitted to diverse query data since the query-specific information is lost during weights generation. Therefore, we propose to maximize the lower bound of mutual information between generated weights and query, support data. As far as we know, AWGIM is the first work introducing Variational Information Maximization in few shot learning. The induced computational overhead is minimal due to the nature of few shot problems. Furthermore, by maximizing the lower bound of mutual information, AWGIM gets rid of inner update without compromising performance.

AWGIM is evaluated on two benchmark datasets and shows state-of-the-art performance. We also conducted detailed analysis to validate the contribution of each component in AWGIM.

2. RELATED WORKS:



2.1. FEW SHOT LEARNING:

Learning from few labeled training data has received growing attentions recently. Most successful existing methods apply meta learning to solve this problem and can be divided into several categories. In the gradient-based approaches, an optimal initialization for all tasks is learned (Finn et al., 2017) . Ravi & Larochelle (2016) learned a meta-learner LSTM directly to optimize the given fewshot classification task. Sun et al. (2019) learned the transformation for activations of each layer by gradients to better suit the current task.

In the metric-based methods, a similarity metric between query and support samples is learned. (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Li et al., 2019a) . Spatial information or local image descriptors are also considered in some works to compute richer similarities (Lifchitz et al., 2019; Li et al., 2019b; Wertheimer & Hariharan, 2019) .

Generating the classification weights directly has been explored by some works. Gidaris & Komodakis (2018) generated classification weights as linear combinations of weights for base and novel classes. Similarly, Qiao et al. (2018) and Qi et al. (2018) both generated the classification weights from activations of a trained feature extractor. Graph neural network denoising autoencoders are used in (Gidaris & Komodakis, 2019) . Munkhdalai & Yu (2017) proposed to generate "fast weights" from the loss gradient for each task. All these methods do not consider generating different weights for different query examples, nor maximizing the mutual information.

There are some other methods for few-shot classification. Generative models are used to generate or hallucinate more data in (Zhang et al., 2018; Wang et al., 2018; Chen et al., 2019) . Bertinetto et al. (2019) and Lee et al. (2019) used the closed-form solutions directly for few shot classification. Liu et al. (2019) integrated label propagation on a transductive graph to predict the query class label.

2.2. ATTENTION:

Attention mechanism shows great success in computer vision (Xu et al., 2015; Parmar et al., 2018) and natural language processing (Bahdanau et al., 2015; Vaswani et al., 2017) . It is effective in modeling the interaction between queries and key-value pairs from certain context. Based on the fact that keys and queries point to the same entities or not, people refer to attention as self attention or cross attention. In this work, we use both types of attention to encode the task and query-task information. The work most similar to ours is Attentive Neural Processes (Kim et al., 2019) , which also employs self and cross attention. However, we are using attention for few-shot image classification via maximizing the mutual information. In stark contrast, Kim et al. (2019) worked on regression from the perspective of a stochastic process and the variational objective is optimized.

2.3. MUTUAL INFORMATION:

Given two random variables x and y, mutual information I(x; y) measures the decrease of uncertainty in one random variable when another is known. It is defined as the Kullback-Leibler divergence between joint distribution p(x, y) and product of marginal distributions p(x) ⊗ p(y), I(x; y) = D KL (p(x, y) p(x) ⊗ p(y)).

(1)

When x and y are independent, p(x, y) = p(x) ⊗ p(y) so that I(x, y) = 0, indicating that knowing x does not reveal any information about y. When y is a deterministic function of x, I(x, y) achieves its maximum value. Mutual information has been widely applied in applications such as Generative Adversarial Networks (Chen et al., 2016) , self-supervised learning (Hjelm et al., 2019) The weight generator g takes these concatenated representations as input to generate classification weights W specific for x, denoted by the colorful matrix with slash. It can be used to predict the class label for x and X. W is also used to reconstruct the inputs of the generator g by two networks r 1 and r 2 . In this way, the lower bound of mutual information is maximized and g is forced to generate classification weights sensitive to different query samples.

3. PROPOSED METHOD:

In this section, we provide the problem formulation first. Then the proposed model is described in Sec. 3.3. The objective function, which maximizes the mutual information between certain variables, and theoretical analysis are provided in Sec. 3.4.

3.1. PROBLEM FORMULATION:

Following many popular meta-learning methods for few shot classification, we formulate the problem under episodic training paradigm (Vinyals et al., 2016; Finn et al., 2017) . One N -way K-shot task sampled from an unknown task distribution P (T ) includes support set and query set:

EQUATION

where S = {(x cn;k , y cn;k )|k = 1, ..., K; n = 1, ..., N }, Q = {(x 1 , ..., x|Q| )}. Support set S contains N K labeled samples. Query set Q includes x and we need to predict label ŷ for x based on S. During meta-training, the meta-loss is estimated on Q to optimize the model. During metatesting, the performance of meta-learning method is evaluated on Q, provided the labeled S. The classes used in meta-training and meta-testing are disjoint so that the meta-learned model needs to learn the knowledge transferable across tasks and adapt itself quickly to novel tasks.

Our proposed approach follows the general framework to generate the classification weights (Qi et al., 2018; Qiao et al., 2018; Rusu et al., 2019; Gidaris & Komodakis, 2018; 2019) . In this framework, there is a feature extractor to output image feature embeddings. The meta-learner needs to generate the classification weights for different tasks.

3.2. LATENT EMBEDDING OPTIMIZATION:

Latent Embedding Optimization (LEO) (Rusu et al., 2019) is one of the weights generation methods that is most related to our work. In LEO, the latent code z is generated by h conditioned on support set S, described as z = h(S). h is instantiated as relation networks (Santoro et al., 2017) . Classification weights w can be decoded from z with l, w = l(z). In the inner loop, we use w to compute the loss (usually cross entropy) on the support set and then update z:

EQUATION

where L S indicates that the loss is evaluated on S only. The updated latent code z is used to decode new classification weights w with generating function l. w is adopted in the outer loop for query set Q and the objective function of LEO then can be written as

EQUATION

Here θ stands for the parameters of h and l and we omit the regularization terms for clarity. LEO avoids updating high-dimensional w in the inner loop by learning a lower-dimensional latent space, from which sampled z can be used to generate w. The most significant difference between LEO and AWGIM is that we do not need inner updates to adapt the model. Instead, AWGIM is a feedforward network trained to maximize the mutual information so that it fits to different tasks well. On the other hand, AWGIM learns to generate optimal classification weights for each query sample while LEO generates fixed weights conditioned on the support set within one task. In Section 3.4 we will show LEO can be casted as a special case of AWGIM under certain conditions.

3.3. ATTENTIVE WEIGHTS GENERATION:

The framework of our proposed method is shown in Figure 1 . Assume that we have a feature extractor, which can be a simple 4-layer Convnet or a deeper Resnet. All the images included in the sampled task T are processed by this feature extractor and represented as d-dimensional vectors afterwards, i.e., x cn;k , x ∈ R d . There are two paths to encode the task context and the individual query sample respectively, which are called contextual path and attentive path. The outputs of both paths are concatenated together as input to the generator for classification weights. Generated classification weights are used to not only predict the label of x, but also maximize the lower bound of mutual information between itself and other variables, which will be discussed in the following section 3.4.

3.3.1. CONTEXTUAL AND ATTENTIVE PATHS:

The encoding process includes two paths, namely the contextual path and attentive path. The contextual path aims at learning representations for only the support set with a multi-head self-attention network f cp sa (Vaswani et al., 2017) . The outputs of contextual path X cp ∈ R N K×d h1 thus contain richer information about the task and can be used later for weights generation.

Existing weights generation methods generate the classification weights conditioned on the support set only, which is equivalent to using contextual path. However, the classification weights generated in this way might be sub-optimal. This is because estimating the exact and universal classification weights from very few labeled data in the support set is difficult and sometimes impossible. The generated weights are usually in lack of adaptation to different query samples. We address this issue by introducing attentive path, where the individual query example attends to the task context and then is used to generate the classification weights. Therefore, the classification weights are adaptive to different query samples and aware of the task context as well.

In the attentive path, a new multi-head self-attention network f ap sa on the support set is employed to encode the global task information. f ap sa is different from f cp sa in contextual path because the selfattention network in contextual path emphasizes on generating the classification weights. On the contrary, outputs of self-attention here plays the role of providing the V alue context for different query samples to attend in the following cross attention. Sharing the same self-attention networks might limit the expressiveness of learned representations in both paths. The cross attention network f ap ca applied on each query sample and task-aware support set is followed to produce Xap ∈ R |Q|×d h . We use multi-head attention with h heads in both paths. In one attention block, we produce h different sets of queries, keys and values. Multi-head attention is claimed to be able to learn more comprehensive and expressive representations from h different subspaces (Vaswani et al., 2017; Voita et al., 2019) . More details of these two paths can be found in A.2.

3.3.2. WEIGHTS GENERATOR:

We replicate X cp ∈ R N K×d h and Xap ∈ R |Q|×d h for |Q| and N K times respectively and reshape them afterwards. Then we have X cp ∈ R |Q|×N K×d h and Xap ∈ R |Q|×N K×d h . These two tensors are concatenated to become X cp⊕ap ∈ R |Q|×N K×2d h . X cp⊕ap can be interpreted that each query sample has its own latent representations for support set to generate specific classification weights, which are both aware of the task-context and adaptive to individual query sample.

X cp⊕ap is decoded by the weights generator g : R 2d h → R 2d . We assume that the classification weights follow Gaussian distribution with diagonal covariance. g outputs the distribution parameters and we sample the weights from learned distribution during meta-training. The sampled classification weights are represented as W ∈ R |Q|×N K×d . To reduce complexity, we compute the mean value on K classification weights for each class to have W f inal ∈ R |Q|×N ×d . Therefore, ith query sample has its specific classification weight matrix W f inal i,:,:

∈ R N ×d . The prediction for query data can be computed by XW f inalT . The support data X is replicated for |Q| times and reshaped as X s ∈ R |Q|×N K×d . So the prediction for support data can also be computed as X s W f inalT .

Besides the weights generator g, we have another two decoders r 1 : R d → R d h and r 2 : R d → R d h . They both take the generated weights W as inputs and learn to reconstruct X cp and Xap respectively. The outputs of r 1 and r 2 are denoted as X cp re , Xap re ∈ R |Q|×N K×d h . The reason we are using reconstruction as auxiliary tasks will be discussed in following Sec. 3.4.

3.4. INFORMATION MAXIMIZATION:

In this section, we perform the analysis for one query sample without loss of generality. The subscripts for classification weights are omitted for clarity. In general, we use (x, y) and (x, ŷ) to represent support and query samples respectively.

Since the classification weights w generated from g are encoded with attentive path and contextual path, it is expected that we can directly have the query-specific weights. However, we show in the experiments that simply doing this does not outperform a weight generator conditioned only on the S significantly, which implies that the generated classification weights from two paths are not sensitive to different query samples. In other words, the information from attentive path is not kept well during the weights generation.

To address this limitation, we propose to maximize the mutual information between generated weights w and support as well as query data. The objective function can be described as maxI((x, ŷ); w) + 

Directly computing the mutual information in Equation 7 is intractable since the true posteriori distributions like p(ŷ|x, w), p(x|w) are still unknown. Therefore, we use Variational Information Maximization (Barber & Agakov, 2003; Chen et al., 2016) to compute the lower bound of Equation 5. We use p θ (x|w) to approximate the true posteriori distribution, where θ represents the model parameters. As a result, we have

EQUATION

H(•) is the entropy of a random variable. H(x) is a constant value for given data. We can maximize this lower bound as the proxy for the true mutual information.

Similar to I(x; w), (13) p θ (x|w), p θ (x, y|w) are used to approximate the true posteriori distribution p(x|w) and p(x, y|w).

EQUATION

Put the lower bounds back into Equation 7. Omit the constant entropy terms and the expectation subscripts for clarity, we have the new objective function as

EQUATION

The first two terms are maximizing the log likelihood of label for both support and query data with respective to the network parameters, given the generated classification weights. This is equivalent to minimizing the cross entropy between prediction and ground-truth. We assume that p θ (x|w) and p θ (x|w) are Gaussian distributions. r 1 and r 2 are used to approximate the mean of these two Gaussian distributions. Therefore maximizing the log likelihood is equivalent to reconstruct x cp and xap with L2 loss. Thus the loss function to train the network can be written as

EQUATION

)

CE here stands for cross entropy. x cp and xap are the inputs to weights generator g. x cp re ∼ p θ (x|w) and xap re ∼ p θ (x|w) are the reconstruction of x cp and xap . Since we convert the log likelihood in Equation 14 to mean square error or cross entropy in Equation 15 to optimize, the value of each term in Equation 15is not equal to real log likelihood and we have to decide the weightage for each one. λ 1 , λ 2 , λ 3 are thus hyper-parameters for trade-off of different terms. With the help of last three terms, the generated classification weights are forced to carry information about the support data and the specific query sample.

In LEO (Rusu et al., 2019) , the inner update loss is computed as cross entropy on support data. If we merge the inner update into outer loop, then the loss becomes the summation of first two terms in Equation 15. However, the weight generation in LEO does not involve specific query samples, thus making reconstructing xap impossible. In this sense, LEO can be regarded as a special case of our proposed method, where (1) only contextual path exits and (2) λ 2 = λ 3 = 0.

3.5. COMPLEXITY ANALYSIS:

The encoding process in contextual path results in computational complexity O((N K) 2 ) due to self-attention. Similarly, the computational complexity of attentive path is O((N K) 2 + |Q|(N K)). In total, the complexity is O((N K) 2 + |Q|(N K)). However, because of the nature of few-shot learning problem, the value of (N K) 2 is usually negligible. The value of |Q| depends on the setting and the cross attention can be implemented parallelly via matrix multiplication. Therefore, the induced computational overhead will be negligible. AWGIM avoids the inner update without compromising the performance, which furthers reduces both training and inference time significantly. The empirical evaluation is presented in A.3.4.

4. EXPERIMENTS:



4.1. DATASETS AND PROTOCOLS:

We conduct experiments on miniImageNet (Vinyals et al., 2016) and tieredImageNet (Ren et al., 2018) , two commonly used benchmark datasets, to compare with other methods and analyze our model. Both datasets are subsets of ILSVRC-12 dataset (Russakovsky et al., 2015) . miniImageNet contains 100 randomly sampled classes with 600 images per class. We follow the train/test split in (Ravi & Larochelle, 2016) , where 64 classes are used for meta-training, 16 for meta-validation and 20 for meta-testing. tieredImageNet is a larger dataset compared to miniImageNet. There are 608 classes and 779,165 images in total. They are selected from 34 higher level nodes in ImageNet (Deng et al., 2009) hierarchy. 351 classes from 20 high level nodes are used for meta-training, 97 from 6 nodes for meta-validation and 160 from 8 nodes for meta-testing. tieredImageNet are evaluated by Liu et al. (2019) . The results of Dynamic on miniImageNet with WRN-28-10 as the feature extractor is reported in (Gidaris & Komodakis, 2019) . The other results are reported in the corresponding original papers. We also include the backbone network structure of the used feature extractor for reference. The results on miniImageNet and tieredImageNet are shown in Table 1 and 2 respectively. The top half parts of Table 1 and 2 display the methods belonging with different meta learning categories, such as metric-based(Matching Networks, Prototypical Nets), gradient-based (MAML, MTL), graph-based (TPN). The bottom part shows the classification weights generation approaches including Dynamic, Prediction, DAE-GNN, LEO and our proposed AWGIM.

AWGIM can outperform all the methods in top parts of two table. Comparing with other classification weights generation methods in the bottom part, AWGIM still shows very competitive performance, namely the best on tieredImageNet and close to the state-of-the-art on miniImageNet. We note that all the classification weights generation methods are using WRN-28-10 as backbone network, which makes the comparison fair. In particular, AWGIM can outperform LEO in all settings.

4.4. ANALYSIS:

Table 3 : Analysis of our proposed AWGIM. In the top half, the attentive path is removed to compare with LEO. In the bottom part, ablation analysis with respective to different components is provided. We also shuffle the generated classification weights randomly to show that they are indeed optimal for different query samples.

Model:

miniImageNet tieredImageNet 5-way 1-shot 5-way 5-shot 5-way 1-shot 5-way We perform detailed analysis on AWGIM, shown in Table 3 . We include the results of LEO Rusu et al. (2019) for reference. "Generator in LEO" means that there is no inner update in LEO. In the upper part of the table, we first studied the effect of attentive path. We implemented two generators including only the contextual path during encoding. "Generator conditioned on S with IM" indicates that we add the cross entropy loss and reconstruction loss for support set. It can be observed that "Generator conditioned on S only" is trained with cross entropy on query set, which is similar to "Generator in LEO" without inner update. It is able to achieve similar or slightly better results than "Generator in LEO", which implies that self-attention is no worse than relation networks used in LEO to model task-context. With information maximization, our generator is able to obtain slightly better performance than LEO.

The effect of attention is investigated by replacing the attention modules with 2-layer MLPs, which is shown as "MLP encoding". More specifically, one MLP in contextual path is used for support set and another MLP in attentive path for query samples. We can see that even without attention to encode the task-contextual information, "MLP encoding" can achieve accuracy close to LEO, for the sake of information maximization. However, if we let λ 1 = λ 2 = λ 3 = 0 for MLP encoding, the performance drops significantly, which demonstrates the importance of maximizing the information.

We conducted ablation analysis with respective to λ 1 , λ 2 and λ 3 to investigate the effect of information maximization. First, λ 1 , λ 2 and λ 3 are all set to be 0. In this case, the accuracy is similar to "generator conditioned on S only", showing that the generated classification weights are not fitted for different query samples, even with the attentive path. It can also be observed that maximizing the mutual information between weights and support is more crucial since λ 1 = λ 2 = 0 degrades accuracy significantly, comparing with λ 3 = 0. We further investigate the relative importance of the classification on support as well as reconstruction. λ 1 = 0 affects the performance noticeably. We conjecture that the support label prediction is more critical for information maximization.

The classification weights are generated specifically for each query sample in AWGIM. To this point, we shuffle the classification weights between query samples within the same classes and between different classes as well to study whether the classification weights are adapted for different query samples. Assume there are T query samples per class in one task. W f inal ∈ R |Q|×N ×d can be reshaped into W f inal ∈ R N ×T ×N ×d . Then we shuffle this weight tensor along the first and second axis randomly. The results are shown as "random shuffle between classes" and "random shuffle in class" in Table 3 . For 5-way 1-shot experiments, the random shuffle between classes degrades the accuracy noticeably while the random shuffle in class dose not affect too much. This indicates that when the support data are very limited, the generated weights for query samples from the same class are very similar to each other while distinct for different classes. When there are more labeled data in support set, two kinds of random shuffle show very close or even the same results in 5-way 5-shot experiments, which are both worse than the original ones. This implies that the generated classification weights are more diverse and specific for each query sample in 5-way 5-shot setting. The possible reason is that larger support set provides more knowledge to estimate the optimal classification weights for each query example.

More analysis is provided in Appendix A.3.

5. CONCLUSION:

In this work, we introduce Attentive Weights Generation via Information Maximization (AWGIM) for few shot image classification. AWGIM learns to generate optimal classification weights for each query sample within the task by two encoding paths. To guarantee this, the lower bound of mutual information between generated weights and query, support data is maximized. As far as we know, AWGIM is the first work utilizing mutual information techniques for few shot learning.

The effectiveness of AWGIM is demonstrated by state-of-the-art performance on two benchmark datasets and extensive analysis.

A.3 EXPERIMENTAL ANALYSIS:



A.3.1 FEW SHOT REGRESSION:

AWGIM can be applied to few shot regression task by slight modification. During meta-training, we set the number of classes N equal to 1 and adapt the cross entropy loss to mean square error. We use the data points (x, y) as inputs to AWGIM and generate weight as well as bias parameters for a three layer MLP with hidden dimension 40. This is consistent with few shot regression experimental setting in LEO.

The few shot regression tasks are constructed as either sinusoidal or linear regression tasks. For sinusoidal regression tasks, the amplitude range is 

A.3.2 EFFECT OF MULTI-HEAD ATTENTION:

We replace the multi-head attention in the two paths with single-head attention and conduct the 5way 1-shot and 5-way 5-shot experiments on miniImageNet dataset. The results are shown in Table 4 . We can see clearly that multi-head attention improve the performance. In particular, for 5-way 1-shot experiment, single head attention gives results close to MLP encoding, which indicates that single head attention struggles when data are extremely scarce. 

A.3.4 INFERENCE TIME:

We measure the inference time of AWGIM to show that it induces minimal computational overhead. In comparison, we use "MLP encoding" in two paths, which has time complexity O(N K + |Q|).

We use two set-ups on miniImageNet and the batch size is set to be 64. 100 batches are processed and we report the average consumed time for one batch. All these experiments on done with the same GPU and workstation. The results are shown in Table 5 . It can be observed that the usage of self-attention and cross attention in AWGIM occurs negligible overhead, compared with MLP encoding. This is because the values of N, K, |Q| are all relatively small and matrix multiplication further can be processed very fast by GPU. denotes the classification weights for two query samples from two classes within one task. It can be observed that g can generate adapted weights for different query samples. This is consistent with Table 3 , where the results of "random shuffle between classes" suggest that query samples from different class have distinct classification weights. 

:

d h < d is the hidden dimension. We use matrix form here to be consistent with the description in 3.3.2.

https://github.com/deepmind/leo

Figures and Tables:

chain rule of mutual information, we have I((x, ŷ); w) = I(x; w) + I(ŷ; w|x). (6) Equation 6 stands for both terms in 5. So the objective function can be written as maxI(x; w) + I(ŷ; w|x) + (x,y)∈S [I(x; w) + I(y; w|x)].

x, y))+E w∼p(w|x,S) [E (x,y)∼p((x,y)|w) [log p θ (x|w)+log p θ (y|x, w)]].

[0.1, 5], phase range [0, 2π], frequency range [0.5, 2.0]. For linear regression tasks, the slope range is [-1, 1], intercept range [-5, 5]. Input x is randomly sample from [-5, 5]. Gaussian noise with standard deviation 0.3 is added to y during meta-training. We show some qualitative results in Figure 2. (a) and (b) are examples that can be tackled easily. For some non-trivial cases such as (c) and (d), AWGIM produces predictions slightly mixing with another regression family, despite that overall results are still faithful.

Figure 2: 5-shot regression results for a multi-modal task distribution. Regression targets are plotted in red and prediction in black. 5 training samples per task are plotted with blue solid circles.

Figure 3: The meta-validation accuracy during meta-training.

We visualize the generated classification weights by t-SNE(Maaten & Hinton, 2008). First we sample 400 tasks from meta-validation set of 5-way 1-shot miniImageNet experiment. Each task contains 5 query samples from 5 different classes. Thus in total there are 400 × 5 × 5 = 10, 000 weight vectors to visualize. As comparison, inputs to the generator g are also plotted. The visualization results are shown in Figure4. The inputs to g are displayed in (a, b) and the generated classification weights in(c, d). From the comparison between (a) and (c), we can see the decoded weights for each class in (c) are clustered closer than (a) in general. Red and blue dots in(b, d)

Figure 4: t-SNE visualization of the inputs to g in (a, b) and the generated classification weights in (c, d). Blue and red dots in (b) and (d) are the classification weights for two query samples in the same task.

The overview of our proposed AWGIM. The input task is 5-way 1-shot with X as support set and x as one query example. Different colors of the data in support set indicate different categories. The encoding process in contextual path produces context-aware support representations X cp . Similarly, the attentive path enables the query sample x to be equipped with task knowledge. Both paths are achieved by attention mechanism. xap is repeated to concatenate with X cp .

Table content: <table><tr><td/><td/><td/><td>SA</td><td>CA</td></tr><tr><td>attentive path</td><td colspan="2">contextual path</td><td>self-attention</td><td>cross-attention information maximization</td></tr><tr><td>� 𝐱𝐱</td><td/><td/><td>� 𝐱𝐱 𝑎𝑎𝑐𝑐</td><td>� 𝐱𝐱 𝑟𝑟𝑟𝑟 𝑎𝑎𝑐𝑐</td></tr><tr><td colspan="2">SA 1</td><td>CA</td><td/><td>𝑟𝑟 1</td></tr><tr><td>𝐗𝐗</td><td/><td/><td>𝑔𝑔</td><td>� 𝐲𝐲 𝑐𝑐𝑟𝑟𝑟𝑟𝑝𝑝</td><td>𝒀𝒀 𝑐𝑐𝑟𝑟𝑟𝑟𝑝𝑝</td></tr><tr><td/><td/><td/><td/><td>𝐖𝐖</td></tr><tr><td colspan="2">SA 2</td><td/><td/><td>𝑟𝑟 2</td></tr><tr><td/><td/><td/><td>𝐗𝐗 𝑐𝑐𝑐𝑐</td><td>𝐗𝐗 𝑟𝑟𝑟𝑟 𝑐𝑐𝑐𝑐</td></tr><tr><td>Figure 1:</td><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>, visual</td></tr><tr><td colspan="4">question generation Krishna et al. (2019) and so on.</td></tr></table>

Accuracy results on miniImageNet with 4 heads or single head in attention networks.We compare AWGIM with LEO in terms of convergence speed. The batch size is set to be 16 for both methods. We use the hyper-parameters tuned by authors to train LEO. The accuracy of metavalidation set during meta-training on 5-way 1-shot miniImageNet is plotted, shown in Figure3. we can see clearly that AWGIM converges faster than LEO and outperforms LEO except for the first few iterations.

Table content: <table><tr><td>Method</td><td colspan="2">5-way 1 -shot 5-way 5-shot</td></tr><tr><td>4 heads</td><td>63.12%</td><td>78.40%</td></tr><tr><td>single head</td><td>62.35%</td><td>77.75%</td></tr><tr><td>A.3.3 CONVERGENCE</td><td/><td/></tr></table>

The comparison of inference time between AWGIM and MLP encoding.

Table content: <table/>

Peer review (iclr2020):
This paper proposes a new model for few-shot classification. Their meta-learning approach falls into the category of weight generation models that, given a new classification task, generate the weights of a linear layer on top of learned embeddings that can be used for classifying query examples of that task. The weights of their model are meta-learned via a sequence of training tasks. More concretely, in each training task, the support and query examples are first embedded using into a d-dimensional space. Then, the support set together with a given query example are processed to form a hidden vector, via two “paths”. This hidden vector, is fed into a weight generator which outputs the linear weights that will be used for classifying the query example. Both processing paths contain a projection layer, followed by self-attention on the support set, and the second path additionally performs cross-attention for the query example on the processed support set. They observed empirically that when training this system simply using the cross-entropy query loss as usual (with the loss computed using the generated weights for each query) did not lead to significant gains. They therefore added terms in their loss function to maximize the mutual information between the generated weights and the support and query sets. The resulting objective is comprised by terms that maximize the cross-entropy on the support and the query sets and reconstruction terms for the support and query images. Pros [+] Interesting new objective for training a few-shot classification model [+] Thorough ablation analysis [+] Seems to perform well on standard benchmarks Cons [-] Lacking sufficient details on LEO. Specifically, the paper largely emphasizes the comparison of their method with LEO. For example, certain of their ablations are described based on their relationship to LEO. Given this, it would be useful to give more background on that method. While it is described in related work, I found that description too high-level to understand LEO’s training objective, its mechanism for conditioning on the support set, etc. [-] Some inconsistencies. For example, the last paragraph of section 3.3 suggests that LEO is similar to using only the first two terms of Equation 13 (ie. lambda_2 = lambda_3 = 0). However, later in the paper, they say that the ablation called “Generator conditioned on S with IM” (which includes the cross-entropy on the support set and reconstruction of support set terms only) is “equivalent to LEO to some extent”. This, however, entails using the second and third terms of Equation 13, seemingly contradicting the previous claim of which version of their method corresponds to LEO. It would be good to clear up this confusion. [-] I found the names of the ablations hard to understand. For example, “Generator conditioned on S only” is the same as “Generator conditioned on S only with IM” except that the reconstruction term for the support set is omitted? Or is the full objective used, but with the weights w having been generated using only the contextual path? It seems that which path(s) are used to generate the weights is orthogonal to which terms of the loss function are used for optimization. So it would be really useful to clarify exactly what each of the ablations is doing. Additional comments / questions: - Is the embedding network pre-trained as in LEO (and held fixed during the meta-training?), or are all the weights meta-learned end-to-end? - Do the rest of the models that are being compared against also re-train on the union of the meta-training and meta-validation sets? This procedure does not seem standard and I’m wondering if the comparison is fair. - A recent related work: “Cross attention networks for few-shot classification. Hou et al. NeurIPS 2019.” Their method also modifies the feature extraction of the support set differently for classifying each given query example, motivated by the need to possibly highlight different regions of the support examples depending on the given query. They also use attention to accomplish this. Given these similarities, it would be useful to compare to them. - This paper largely emphasizes comparisons with LEO. Along those lines, can a version of the proposed approach also tackle few-shot regression as LEO does?