Title: META-LEARNING WITH INDIVIDUALIZED FEATURE SPACE FOR FEW-SHOT CLASSIFICATION

Abstract: Meta-learning provides a promising learning framework to address few-shot classification tasks. In existing meta-learning methods, the meta-learner is designed to learn about model optimization, parameter initialization, or similarity metric. Differently, in this paper, we propose to learn how to create an individualized feature embedding specific to a given query image for better classifying, i.e., given a query image, a specific feature embedding tailored for its characteristics is created accordingly, leading to an individualized feature space in which the query image can be more accurately classified. Specifically, we introduce a kernel generator as meta-learner to learn to construct feature embedding for query images. The kernel generator acquires meta-knowledge of generating adequate convolutional kernels for different query images during training, which can generalize to unseen categories without fine-tuning. In two standard few-shot classification data sets, i.e. Omniglot, and miniImageNet, our method shows highly competitive performance.

1. INTRODUCTION:

A well-performing model for image classification is often equipped with a discriminative feature space and a powerful classifier, which heavily rely on training with large quantities of labeled data, especially for those high-capacity deep models. The scalability of most deep models to new tasks with few samples is severely limited because of the dependence on big data. In contrast, humans excel at recognizing objects and can rapidly learn to recognize a new class even with a single example by utilizing prior knowledge Lake et al. (2011) . The significant gap between human and machine learning motivates the development of one/few-shot learning that aims at achieving better generalization in the tasks with scarce labeled data.

The task of few-shot classification is to recognize previously unseen classes with very few labeled examples. A straightforward approach is to fine-tune a pre-trained model on the new classes, as is usually done in transfer learning Finn et al. (2017) . But it becomes awkward in few-shot classification, as it would cause severe over fitting when only a few training examples are available. Data augmentation and regularization techniques can alleviate the over fitting problem Sung et al. (2018) , but they are still far from obtaining a satisfactory model in few-shot scenarios. One promising approach to address few-shot classification is meta-learning Thrun (1998) ; Hochreiter et al. (2001) , in which transferable knowledge is learned on a set of tasks. The idea is that by learning on task τ following a distribution p(τ ), a meta-learner may figure out an effective learning strategy tailored for the task distribution p(τ ). Therefore, a meta-learner is able to quickly adapt to new tasks following the same distribution p(τ ) with only a few examples per class.

Meta-learner plays a crucial role in meta-learning. Generally, a meta-learner is a model designed to acquire learning algorithms (meta-knowledge), which are used to further teach a learner so that the learner can quickly adapt to new tasks. In existing meta-learning methods, the meta-learner is designed as an optimizer Andrychowicz et al. (2016) ; Ravi & Larochelle (2017) ; Li & Malik (2017) , an initializer Finn et al. (2017) ; Li et al. (2017) ; Nichol et al. (2018) , or as a kind of distance metric Vinyals et al. (2016) ; Koch et al. (2015) ; Snell et al. (2017) ; Sung et al. (2018) . A meta-learner for optimization usually learns to predict gradients of parameters or learns SGD-like parameter updating policy to make learners converge quickly. A meta-learner that learns to initialize model parameters aims to initialize a learner so that it can adapt to new tasks with a small amount of data in a few steps. In those metric-learning based methods, the meta-learner is often designed as a kind

Features of Sample Images:

Features of Query Images (a) Common Feature Space (b) Individualized Feature Space (IFS) Different from existing works, in this paper, we propose to learn how to create an individualized feature embedding specific to a given query image for more accurately classifying during metalearning. To classify a query image, we need to compare it with labeled images of all classes. In general, as different query images own different characteristics because of inter-class variations or intra-class variations (e.g. category, pose, illumination, and some other factors), one tends to focus on different aspects of those labeled images when comparing them to different query images. Thus, the characteristics of query images influence the perception of labeled images. Inspired from this, our meta-learner is designed to learn how to create feature embedding tailored for characteristics of different query images, so that the query image can be more accurately classified.

IFS for Q 1 IFS for Q 2 IFS for Q 3 IFS for Q 4 IFS for Q 5

The construction of individualized feature spaces naturally leads to formulation of binary classification tasks. Most existing methods map all images into a common feature space. In such situation, to distinguish different classes from each other, images are expected to form multiple clusters according to their class labels, leading to a multi-class classification task. By contrast, when an individualized feature space is constructed specifically for a given query image in our method, one only needs to distinguish the class which the query image belongs to from the rest of the classes, without concerning whether the rest of the classes can be distinguished from each other. Therefore, the goal of our meta-learner is to learn how to create individualized feature embedding tailored for the query image that will minimize (or maximize) the distances between the query image and those belonging to the same class (or different classes). Figure 1 illustrates the differences between the multi-class classification task in a common feature space and the binary classification tasks in our individualized feature spaces.

Specifically, the meta-learner in our proposed method is designed as a kernel generator to construct the feature embedding for a specific query image. It generates distinct sets of convolutional kernels as the characteristics of query images vary. By convolving features of images with those generated kernels, they are mapped into the individualized feature space constructed with the corresponding query image. During meta-training, the learning of the kernel generator is at higher level than that of ordinary network modules. It acquires meta-knowledge of generating adequate convolutional kernels for different query images, which can generalize to new classes without fine-tuning.

Overall, our contributions lie in three folds. First, we propose to learn how to create an individualized feature embedding tailored for a given query image. By considering the characteristics of the query image in feature embedding, an individualized feature space is constructed to accurately classify the query image. Second, we offer an effective implementation of the meta-learner as a kernel generator, which learns to generate convolutional kernels based on the given query image.

Third, on two standard few-shot classification data sets, including Omniglot Lake et al. (2011) , and miniImageNet Vinyals et al. (2016) , our method achieves competitive results.

2. RELATED WORK:

Meta learning Meta-learning approaches Thrun (1998) ; Schmidhuber (1987) ; Naik & Mammone (1992) involve training a meta-learner to acquire meta-knowledge, which generally can be trans-ferred into new tasks with scarce data and within a few iterations Andrychowicz et al. (2016) ; Schmidhuber et al. (1997) .

Some works design the meta-learner as an optimizer that learns to update model parameters Bengio et al. (1992) ; Schmidhuber (1992) . Similar method has been applied to deep networks to compose gradients of parameters for quick convergence of training Hochreiter et al. (2001) ; Andrychowicz et al. (2016) ; Li & Malik (2017) . Recently, Ravi & Larochelle (2017) proposes a meta-learner based on LSTM Hochreiter & Schmidhuber (1997) to learn a SGD-like parameter updating policy which is used to train another learner in the few-shot regime. It also learns a general initialization of the learner network to accelerate its convergence.

Meta-learner learning how to initialize a learner is more general and generic in meta-learning. Finn et al. (2017) proposes an algorithm, which is referred to as Model-Agnostic Meta-Learning (MAML), learning how to initialize a learner so that the learner can adapt to new task with a small amount of training data in just a few steps. MAML is compatible with any model trained with gradient descent. Li et al. (2017) improves MAML with not only learning the learner initialization, but also the learner update direction and learning rate. Nichol et al. (2018) analyzes the first-order MAML, and points out it is simpler to implement than was widely recognized prior, forming a new algorithm, called Reptile.

Metric-learning methods design the meta-learner as a kind of distance metric. Generally, those approaches attempt to learn a common feature space where categories can distinguish from each other based on the defined or learned distance metric, forming a multi-class classification task. Our work is related to metric learning. The difference is that our meta-learner goals for learning how to create an individualized feature space tailored for a given query image, where we only need to distinguish class the query image belongs to from all other classes, forming a binary classification task. More clear comparison can be seen in Figure 1 . Koch et al. (2015) formulates the oneshot classification task as the matching problem and train Siamese neural networks to calculate the similarity between images in the support set and a query image. 2016) trains a memory-augmented LSTM for few-shot learning, where the learner is trained to adapt to new tasks as the LSTM rolls out. Mishra et al. (2017) proposes a temporal convolutional network that outputs the prediction for the test example based on the previous labeled examples it has seen given as input a sequence of example-label pairs followed by an unlabeled example.

Parameter Prediction Our work is also related to parameter prediction. To accelerate the convergence rate of model, Ha et al. (2017) proposes to generate the weights of main network via a HyperNetwork with fewer learnable parameters, which can be viewed as a relaxed form of weightsharing across layers. The difference between HyperNetwork and our work is that, for a CNN, parameters of the main network are fixed after training in HyperNetwork, while kernels are generated dynamically conditioned on input query images even when testing in our method. De Brabandere et al. ( 2016) and Klein et al. (2015) apply input conditioned kernel to image prediction task, and the dynamically-generated filters are mainly used to predict the movement of pixels between frames. Han et al. (2018) proposes the contrastive convolutional kernels that is created based on the input face pair to focus on the different features between the input face pair for better certifying whether they belong to the same identity. Differently, our dynamic generated kernels corresponding to a given query image are used to build a individualized feature space, where the query image can be more accurately classified. Qiao et al. (2018) and Gidaris & Komodakis (2018) 

3.2. FRAMEWORK:

Suppose there is a sample set {(x ij , y ij )} i∈{1,..,C},j∈{1,...,K} and a query image q with label l, where x ij means the j-th image of the i-th class in the sample set, and y ij is the label of image x ij . As is shown in Figure 2 , the framework of our method mainly consists of two modules: a base feature extractor and a kernel generator. The base feature extractor c, which is shared across all images, consisting of several cascaded convolutional layers, extracts shared base features for all input images.

Taking x as one image in the sample set, as well as F x and F q as feature maps of image x and q respectively, c can be formulated as:

F x = c(x), F q = c(q) ∈ R h F ×w F ×c F (1)

where h F , w F and c F are the height, width, and number of channels of output feature maps respectively. At this step, images are mapped from the raw image space to a common feature space.

To better distinguish the query image from other objects, the individualized feature space is created based on the base feature space, where only characteristics that can best distinguish the query image from other objects are focused. Specifically, a kernel generator G is introduced to generate individualized kernels for different query images. The kernel generator G takes the feature maps F q as input, and outputs a set of individualized kernels K q for the query image q, which can be generally formulated as follow:

K q = G(F q ) (2)

The set of individualized kernels K q of image q can be used to create a query specific feature embedding:

f q : S Com -→ S Ind q

, where the S Com and S Ind q means common feature space corresponding to the base feature extractor and individualized feature space of q respectively. f q is constructed as the form of convolution with K q as convolutional kernels, that is:

f q (F x ) = K q * F x = F q x , f q (F q ) = K q * F q = F q q (3)

here, * means the conventional convolution. Thus, f q is a dynamic feature embedding, and varied with the query image.

An individualized feature space of image q is the range of function f q . In the individualized feature space S Ind q , the distance between images in the sample set and image q can be defined as any distance metric, such as cosine distance, Manhattan distance, and Euclidean distance. Based on the distance metric we used, images in the sample set are expected to gather into two clusters: images owning the same class label as q, and images with different labels from q.

3.3. KERNEL GENERATOR:

Convolutional kernels generated via the kernel generator G should be able to obtain adequate information of the given query image to create expressive individualized feature space. Inspired by Han et al. (2018) , our kernel generator is designed as a hierarchical structure to obtain kernels with multiple scales as feature maps in different layers usually own different receptive field. Specifically, there are T layers in kernel generator network, one sub-generator for each layer, forming T sub-generators in total:

The learning of kernel generator is higher level than vanilla kernel, acquiring meta-knowledge of generating adequate convolutional kernels for different query images during training, which can generalize to unseen categories without fine-tuning.

3.4. LOSS FUNCTION:

Given a query image q with label l in a query set of an episode, we hope features of images in the sample set to be close to or far from the features of q in the individualized feature space according to whether they have the same label with q. The possibility p q l that label l is assigned to image q can be calculated by a softmax function based on the defined distance metric. Following the cross entropy loss, the loss function is formulated as follow:

EQUATION

)

where N is the number of query images. Minimizing the above loss function can guarantee that the true label l be assigned to the query images q. Moreover, individualized kernels are expected to capture the intrinsic characteristics of a object, that is, individualized kernels of different images with the same object should be the same even with different poses, illuminations, forming another cross entropy loss:

EQUATION

where e l is a one-hot vector with 1 in the l-th position, and H(K q ) aims to regress the possibility distribution of kernels K q to a one-hot code for classification. The network can be trained in an end-to-end manner by jointly optimizing L 1 and L 2 with the gradient decent based optimization algorithm.

4. EXPERIMENTS:



4.1. SETTINGS:

We implement our method with PyTorch. In all experiments, the parameters of our models are randomly initialized, and Adam Kinga & Adam (2015) is used for optimization. Models are iterated 100k with learning rate 0.0005 during training. The batch size is set as 8. As for the kernel generator, it consists of two sub-generators. The first sub-generator takes 6×6 feature maps as input, generating 36 kernels, and the second one takes 4×4 feature maps as input, generating 16 kernels. For a K-shot task, we average the features of K images with the same label in the individualized feature space tailored for a given query image, and then calculate the distance between the query image and the mean features as is done in Snell et al. (2017) .

As most few-shot learning models utilize four convolutional blocks for feature embedding module 2018), the base feature extractor in our model is also designed as a convolution architecture with 4 modules, where each module consists of a 3 × 3 convolution layer with 64 filters, followed by batch normalization Ioffe & Szegedy (2015) , a ReLU nonlinearity, and 3 × 3 max-pooling. The stride for all convolution layers is 1. On Omniglot, step size in the first two max-pooling layers is 2, and in the last two max-pooling layers is 1 to make sure the network outputs bigger feature map, while on miniImageNet, all step size of max-pooling layers is 2 due to the increased image size. We refer the network described above to as C 4 . Besides, as Mishra et al. (2017) uses the small version of the ResNet He et al. (2016) to improve the accuracy, we also use a similar ResNet structure but with less filters on miniImageNet, which is referred to as ResNet.

For all experiments, on a C-way K-shot experiment, the episode is formed with C classes, and each class includes K sample images, and 6 and 15 query images for training and testing respectively. Following Vinyals et al. (2016) ; Snell et al. (2017); Finn et al. (2017) ; Sung et al. (2018) , on Omniglot, we compute few-shot classification accuracies by averaging over 1000 randomly generated episodes from the testing set, while on miniImageNet and tieredImageNet, results are computed by averaging over 600 randomly generated episodes from the testing set. Results on Omniglot is shown in Table 1 . As can be seen, accuracies of most methods approach 100%, which means the performance differences between all methods are small. However, our method still achieves about 0.4 % point promotion on 20-way 1-shot experiment, despiting that some methods fine-tune on the support set Vinyals et al. (2016); Finn et al. (2017) ; Li et al. (2017) ; Mishra et al. (2017) .

4.3. miniIMAGENET:

The miniImageNet dataset is a subset of the large ILSVRC-12 dataset 

Influence of Number of Sub-generator:

The kernel generator is designed with a hierarchical structure to integrate information from multiple scales. In our experiments, there are 2 layers in the kernel generator, each layer for one sub-generator. Here, to investigate the influence of the number of sub-generator, we compare the results of model with 1 sub-generator and model with 2 sub-generators. Experiments are performed on the setting of 5-way 1-shot and 5-way 5-shot with the 4-layer network on miniImageNet. The accuracies of model with 2 sub-generators increase from 53.15% to 54.47% and from 67.95% to 68.27% on 5-way 1-shot and 5-way 5-shot respectively relative to model with only 1 sub-generator, which indicates the effectiveness of hierarchical structure of the kernel generator.

Influence of Distance Metric we evaluate three different distance metrics in the individualized feature space on miniImageNet, including cosine similarity, Manhattan distance and Euclidean distance. The results are shown in Table . 2. As can be seen, the distance metric has limited influence on accuracies, and Euclidean distance performs slightly better on both setting of 5way-1shot and 5way-5shot.

Visualizating of Individualized features As shown in Figure 3 , we visualize the feature maps of the support set images in the individualized feature space corresponding to different query images. The first row shows three images S 1 to S 3 from support set. In each row of next 2-4, the leftmost image is the query image Q i used to generate kernels that map S 1 to S 3 into the feature space tailored for Q i , and the rest images are the feature maps of S 1 to S 3 in Q i 's feature space. Clearly, different aspects of a support set image are focused when mapped to distinct individualized feature spaces of different query images, demonstrating the rationality of our individualized feature space. 

Q 1 Q 2 Q 3 S 1 S 2 S 3

5. CONCLUSION:

In this paper, we introduced a meta-learning method based on learning how to create feature embedding dynamically for different query images in the few-shot classification task. When recognizing a given query image, its characteristics are considered into contrusting the feature embedding, leading to an individualized feature space tailored for the query image, where the query image can be better distinguished from other objects. The meta-learner is designed as a kernel generator to create the dynamic convolution kernels. Good generalization of our models benefits from the kernel generator which can obtain the meta-knowledge of creating adequate convolutional kernels for different query images. The good performance on two data sets, i.e., Omniglot, and miniImageNet, demonstrates the superiority of the proposed individualized feature space.

:



Figures and Tables:

Figure 1. Comparison of common feature space and individualized feature space. (a) The common feature space. Feature distribution has to be discriminative enough so that all classes can be distinguished from each other. (b) The individualized feature space. There is an individualized feature space tailored for each query image (i.e. the image to be classified), where images are classified into two categories according to whether they belong to the same class as the query image.

Figure 2. Network architecture for a 5-way 1-shot classification task.

Vinyals et al. (2016); Snell et al. (2017); Finn et al. (2017); Li et al. (2017); Sung et al. (

Figure 3. Illustration of feature maps in different individualized feature spaces.

Few-shot classification accuracies on Omniglot. '-': not reported. Omniglot Lake et al. (2011) consists of 1623 characters collected from 50 different alphabets. Each character contains 20 samples drawn by different people. Following Santoro et al. (2016); Finn et al. (2017); Snell et al. (2017), we resize all images into 28 × 28, and augment new classes through randomly rotating images by 90 degrees multiples times. We randomly choose 1200 classes for meta-training and the remaining 423 classes for meta-testing, both with rotation augmentation.

Table content: <table><tr><td>Model</td><td>Fine Tune</td><td colspan="4">5-way accuracy(%) 20-way accuracy(%) 1-shot 5-shot 1-shot 5-shot</td></tr><tr><td>MANN Santoro et al. (2016)</td><td>N</td><td>82.8</td><td>94.9</td><td>-</td><td>-</td></tr><tr><td>Convolutional Siamese Nets Koch et al. (2015)</td><td>N</td><td>96.7</td><td>98.4</td><td>88.0</td><td>96.5</td></tr><tr><td>Convolutional Siamese Nets Koch et al. (2015)</td><td>Y</td><td>97.3</td><td>98.4</td><td>88.1</td><td>97.0</td></tr><tr><td>Matching Nets Vinyals et al. (2016)</td><td>N</td><td>98.1</td><td>98.9</td><td>93.8</td><td>98.5</td></tr><tr><td>Matching Nets Vinyals et al. (2016)</td><td>Y</td><td>97.9</td><td>98.7</td><td>93.5</td><td>98.7</td></tr><tr><td>Siamese Nets with Memory Hertwig et al. (2004)</td><td>N</td><td>98.4</td><td>99.6</td><td>95.0</td><td>98.6</td></tr><tr><td>Neural Statistician Edwards &amp; Storkey (2017)</td><td>N</td><td>98.1</td><td>99.5</td><td>93.2</td><td>98.1</td></tr><tr><td>Meta Nets Munkhdalai &amp; Yu (2017)</td><td>N</td><td>99.0</td><td>-</td><td>97.0</td><td>-</td></tr><tr><td>Prototypical Nets Snell et al. (2017)</td><td>N</td><td>98.8</td><td>99.7</td><td>96.0</td><td>98.9</td></tr><tr><td>MAML Finn et al. (2017)</td><td>Y</td><td>98.7</td><td>99.9</td><td>95.8</td><td>98.9</td></tr><tr><td>Meta-SGD Li et al. (2017)</td><td>Y</td><td>99.53</td><td>99.93</td><td>95.93</td><td>98.97</td></tr><tr><td>Relation Net Sung et al. (2018)</td><td>N</td><td>99.6</td><td>99.8</td><td>97.6</td><td>99.1</td></tr><tr><td>SNAIL Mishra et al. (2017)</td><td>Y</td><td>99.07</td><td>99.78</td><td>97.64</td><td>99.36</td></tr><tr><td>ours</td><td>N</td><td>99.49</td><td>99.83</td><td>98.03</td><td>99.22</td></tr><tr><td>4.2 OMNIGLOT</td><td/><td/><td/><td/><td/></tr></table>

Russakovsky et al. (2015), consisting of 60000 image, with 600 images per class. In our experiments, the splits, introduced byVinyals et al. (2016), are used where data is divided into three disjoint subsets: 64 classes for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing. Following Santoro et al. (2016); Vinyals et al. (2016); Finn et al. (2017); Snell et al. (2017); Qi et al. (2018), we resize all images into 84 × 84 for training and testing.

Table content: <table/>

Comparion of different distance metrics on miniImageNet for 5-way 1-shot and 5-way 5shot.

Table content: <table><tr><td>Distance Metric</td><td colspan="2">5-way accuracy(%) 1-shot 5-shot</td></tr><tr><td>cosine</td><td>53.54</td><td>67.89</td></tr><tr><td>Manhattan</td><td>54.39</td><td>67.26</td></tr><tr><td>Euclidean</td><td>54.47</td><td>68.27</td></tr></table>

Few-shot classification accuracies on miniImageNet. C4 and ResNet are described in section 4.1.Comparison with State-of-the-art Here we compare our proposed method with other state-ofthe-art approaches on the miniImageNet data set. As is shown in 3, under both 5-way 1-shot and 5-way 5-shot setting, our accuracies are much higher than those metric learning based method, such as Matching NetsVinyals et al. (2016), Prototypical Nets Snell et al. (2017), and Relation Net Sung  et al. (2018), indicating the superiority of individualized feature space.

Table content: <table><tr><td>Model</td><td colspan="2">Fine Tune Networks</td><td colspan="2">5-way accuracy(%) 1-shot 5-shot</td></tr><tr><td>Matching Nets Vinyals et al. (2016)</td><td>Y</td><td>C 4</td><td>43.56</td><td>55.31</td></tr><tr><td>Meta-Learn LSTM Ravi &amp; Larochelle (2017)</td><td>Y</td><td>C 4</td><td>43.44</td><td>60.60</td></tr><tr><td>Prototypical Nets Snell et al. (2017)</td><td>N</td><td>C 4</td><td>49.42</td><td>68.20</td></tr><tr><td>MAML Finn et al. (2017)</td><td>Y</td><td>C 4</td><td>48.7</td><td>63.11</td></tr><tr><td>Meta-SGD Li et al. (2017)</td><td>Y</td><td>C 4</td><td>50.47</td><td>64.03</td></tr><tr><td>Relation Net Sung et al. (2018)</td><td>N</td><td>C 4</td><td>50.44</td><td>65.32</td></tr><tr><td>SNAIL Mishra et al. (2017)</td><td>Y</td><td>C 4</td><td>45.1</td><td>55.2</td></tr><tr><td>SNAIL Mishra et al. (2017)</td><td>Y</td><td>ResNet</td><td>55.71</td><td>68.88</td></tr><tr><td>ours</td><td>N</td><td>C 4</td><td>54.47</td><td>68.27</td></tr><tr><td>ours</td><td>N</td><td>ResNet</td><td>56.89</td><td>70.51</td></tr></table>

Peer review (iclr2019):
 Summary: One of popular approach to few-shot classification is to learn an embedding function to a common feature space where the similarity between two examples is expected to be well determined. The current work claims that query-dependent feature space (referred to as individualized feature space) gains over the common feature space, in the task of few-shot classification. To this end, the paper employed a technique *kernel generator* which has been recently proposed in [Han et al., 2018]. Few-shot classification is done using distance (e.g. Euclidean) in the query-dependent feature space. The paper evaluates this method using Omniglot and miniImagenet. Strengths: - Constructing individualized feature space tailored to each query is a novel idea. - The paper shows strong quantitative results. Weaknesses: - The clarity is a big obstacle in this paper. Section 3 contains the main idea on *kernel generator* which is the critical technique to map input images to individualized feature spaces. Unfortunately Section 3 is hard to follow. - Moreover, the idea of kernel generator is the one used in [Han et al., 2018], so the contribution of this paper is very limited. - In a nutshell, the current work can be considered as a mix of matching network and kernel generator. Specific comments: - Regarding terminology, authors state that "there are three sets of examples in a few-shot classification task: a training set, a support set, and a testing set. The training set and the support set have disjoint label spaces with each other while the testing set shares the same label space with the support set." I am very confused with what authors mean by support set. In general, each episode has a support set as well as queries in both meta-training and meta-test phases. Meta-training and meta-test has disjoint label spaces. - It is not clear to me what the problem setting is here. Queries in training and test phases have different label spaces. So, I am wondering feature space tailored to queries in the training phase can be well generalized to the test phase. Or you assumes that both cases have the same label space? -Fig 2: The kernels and the conv features interact in a node which says “X”, making it seem like we are either pointwise multiplying or taking an outer product. The figure would be clearer if it somehow expressed that the two interact via convolution. (To add to this confusion, the kernels are thin which makes them look like vectors) -eq(6): the index i is used to denote two things at once (g_i, c_ij). This notation should be different. -eq(6): it says g_i is a fully-connected layer, but P_q^ij is a 3d tensor. Is g_i a 1x1 convolution, or do you flatten P? -eq(9): what is H? Does it mean entropy? How is the set K_q a distribution of kernels? How does this loss relate to capturing the intrinsic characteristics of an object? This whole part should be clearer.