Title: IMAGE DEFORMATION META-NETWORKS FOR ONE-SHOT LEARNING

Abstract: Humans can robustly learn novel visual concepts even when images undergo various deformations and loose certain information. Incorporating this ability to synthesize deformed instances of new concepts might help visual recognition systems perform better one-shot learning, i.e., learning concepts from one or few examples.
Our key insight is that, while the deformed images might not be visually realistic, they still maintain critical semantic information and contribute significantly in formulating classifier decision boundaries. Inspired by the recent progress on meta-learning, we combine a meta-learner with an image deformation network that produces additional training examples, and optimize both models in an endto-end manner. The deformation network learns to synthesize images by fusing a pair of images -a probe image that keeps the visual content and a gallery image that diversifies the deformations. We demonstrate results on the widely used oneshot learning benchmarks (miniImageNet and ImageNet 1K challenge datasets), which significantly outperform the previous state-of-the-art approaches.
RELATED WORK

1. INTRODUCTION:

Deep architectures have made significant progress in various visual recognition tasks, such as image classification and object detection. Such a success typically relies on supervised learning from huge amounts of labeled examples. In real-world scenarios, however, one might not have enough resources to collect large training sets or need to deal with rare visual concepts. It is also unlike the human visual system, which can grasp a novel visual concept under very little supervision. One-shot or low/few-shot learning has thus attracted more and more attention, which aims to build a classifier for a new concept from one or very few labeled examples.

Recent efforts to address this problem have leveraged a learning-to-learn or meta-learning paradigm, which frames one-shot learning as an optimization problem (Finn et 2017)). Meta-learning algorithms train a learning procedure (i.e., learner), which is a parameterized function that maps labeled training sets to classifiers. Meta-learners are trained by sampling a collection of one-shot learning tasks and the corresponding datasets from a large universe of labeled examples of known (base) categories, feeding the sampled small training set to the learner to obtain a classifier, and then computing the loss of the classifier on the sampled test set. The hope is that the learner is able to tackle the recognition of unseen (novel) categories from few training examples.

Despite their noticeable performance improvement, these generic meta-learning algorithms typically treat images as black boxes and ignore the structure of the visual world. By contrast, our biological vision system is very robust and trustable in understanding images that undergo a variety of deformations (Vermaak et al. (2005) ; Boccolini et al. ( 2018)). For instance, we can easily recognize the concepts/objects in Fig. 1 as the ghost (Fig. 1(a, b )), stitched (Fig. 1(c )), montage (Fig. 1(d) ), and partially erased images (Fig. 1(e) ). While these deformed images might not be visually realistic, our key insight is that they still maintain critical semantic information and presumably serve as "hard examples" that contribute significantly in formulating classifier decision boundaries. Hence, by leveraging such modes of deformation shared across categories, the synthesized deformed images could be used as additional training data to build better classifiers.

A natural question then is how we could produce informative deformations. As shown in Fig. (2) , we propose a simple parametrization that linearly combines a pair of images to generate the deformed image 1 . We use a probe image to keep the visual content and overlay a gallery image at a patch level to introduce appearance variations, which could be attributed to semantic diversity, art effects, or even random noise. Importantly, inspired by Wang et al. (2018) , we learn to deform images that are useful for classification by the end-to-end meta-optimization of a classification objective that includes image deformation in the model.

Our Image Deformation Meta-network (IDeMe-Net) thus consists of two components: a deformation subnetwork and an embedding subnetwork. The deformation network learns to generate the deformed image by linearly fusing the patches of probe and gallery images. Specifically, we treat the given small training set as the probe images and sample addition images from the base categories to form the gallery images. We evenly divide the probe and gallery images into nine patches, and the deformation network estimates the combination weight of each patch. The synthesized deformed images are used to augment the probe images and train the embedding network, which maps images to feature representations and performs one-shot classification. The entire network is trained in an end-to-end manner on base categories.

Our contributions are three-fold. (1) We propose a novel meta-learning based image deformation framework to address one-shot learning, which uses the rich structure of shared modes of deformation in the visual world. (2) Our deformation network learns to synthesize diverse deformed images, which effectively exploits the complementarity and interaction between the probe and gallery image patches. (3) By using the deformation network, we effectively augment and diversify the one-shot training images, leading to the significant performance boost in one-shot learning tasks. Remarkably, our approach achieves the state-of-the-art performances on both the challenging ImageNet1K and miniImagenet datasets by large margins.

at training a parametrized mapping from a few training instances to some hidden parameters that impact the optimization procedure. Wang et al. (2018) employed meta-learner -"hallucinator" to produce additional training examples. Intrinsically, the hallucinator is a generative adversarial network to synthesize the realistic but imaginary images by using noise and training images. In contrast, also as a meta-learner, our IDeMe-Net dynamically learns to fuse the probe and gallery images as new images for one-shot learning. Our network aims at producing the deformed but real (i.e. not synthesized) images (as illustrated in Fig. 2 ). Furthermore, there are also other more general meta-learning strategies in one-shot learning, such as graph CNN (Garcia & Bruna (2018) Comparing with these models, our IDeMe-Net learns to dynamically fuse patches of two real images in an end-to-end manner. Our newly fused image has the best of two worlds: the IDeMe-Net can save the important patches of original images, while, the image is visually different from both images and thus help to train the one-shot classifier. Additionally, Wang et al. (2018) and Hariharan & Girshick (2017) both synthesize images in the feature domain while we can directly produce images in the image domain. To mimic the one-shot learning scenario, meta-learning algorithms learn from a collection of Nway-m-shot classification tasks/datasets sampled from D base and are evaluated in a similar way on D novel . Each of these sampled datasets is termed as an episode, and we thus have different metasets for meta-training and meta-testing. Specifically, we randomly sample N classes L ∼ C k for meta-training (i.e., k = base) and meta-testing episode (i.e., k = novel). We then randomly sample m and q labeled images per class in L to construct the support set S and query set Q, respectively, i.e., | S |= N × m; and | Q |= N × q. During meta-training, we sample S and Q to train our model. During meta-testing, we evaluate by averaging the classification accuracy on query sets Q of many meta-test episodes.

3. ONE-SHOT LEARNING SETUP:

We view the support set as supervised prob images and different from the previous work, we introduce an additional gallery image set G that serves as an unsupervised image pool to help generate deformed images. To construct G, we randomly sample some images per base class from the base dataset, i.e., G ∼ D base . The same G is used both in the meta-training and meta-testing episodes. Note that since it is purely sampled from D base , the newly introduced G does not break the standard 

4. IMAGE DEFORMATION META-NETWORKS:

We now explain our image deformation meta-network (IDeMe-Net) for one-shot learning. Figure 3 shows the architecture of IDeMe-Net f θ (•) parametrized by θ. IDeMe-Net is composed of two modules -a deformation subnetwork and an embedding subnetwork. The deformation network adaptively fuses the prob and the gallery images to synthesize the deformed images. The embedding network maps the images to feature representations and construct the one-shot classifier. The entire meta-network is trained in an end-to-end manner.

4.1. DEFORMATION SUBNETWORK:

This subnetwork f θ def (•) learns to explore the interaction and complementarity between the prob images I probe ({I probe , y probe } ∈ S) and the gallery images I gallery ∈ G, and fuses them to generate the synthesized images I syn , i.e., I syn = f θ def (I prob , I gallery ). Our key insight is to synthesize meaningful deformed images such that y syn = y probe . This is achieved by using two strategies:

(1) y syn = y probe is explicitly enforced as a constraint during the end-to-end optimization of the network; (2) we propose an approach to sampling I gallery that are visually or semantically similar to the images of y probe . Specifically, for a prob image {I probe , y probe }, we directly use the feature extractor and one-shot classifier learned in embedding network to select the top % images from G that has the highest class probability as y probe as the I gallery . We randomly sampled I gallery from the selected set, which are visually or semantically similar to the probe image Iprobe.

Two branches, ANET and BNET, are used to parse I probe and I gallery , respectively. Each of them is a residual network (He et al. ( 2015)) without fully-connected layers. The outputs of ANET and BNET are then concatenated to be fed into a fully-connected layer, which produces a 9-D weight vector w. We use w to construct a weight matrix W as shown in Fig. 3 . The deformed image is thus produced as a simple linear weighted combination of I probe and I gallery as follows:

EQUATION

where is the element-wise multiplication. We assign the class label y probe to the synthesized image I syn . For any probe image I i probe , we sample n aug gallery images from the selected set and produce n aug synthesized images I i syn . We thus obtain an augmented support set Sk = S k , (I i syn , y i probe )

N * m i=1 .

4.2. EMBEDDING SUBNETWORK:

As shown in Fig. 3 2017), we calculate the prototype vector p j θ for each class j in S as

EQUATION

where Z = Σ (Ii,yi)∈ S y i = j is the normalization factor and f θ emb (I i ) is the feature extracted by the embedding network. • is the Iverson's bracket notation: x = 1 if x is true, and 0 otherwise. Given any query image I i ∈ Q, its probability of belonging to class k is compute as

EQUATION

where • indicates the Euclidean distance. The one-shot classifier thus predicts the class label of I i as the highest probability over N classes. update θ def with the prototypical Loss 20: end procedure

5. TRAINING STRATEGY OF IDEME-NET:



5.1. TRAINING LOSS:

Training the entire IDeMe-Net includes two subtasks: (1) training the deformation network which maximally improves the one-shot classification accuracy; and (2) building the robust embedding network which effectively deals with various synthesized images. Note that our one-shot classifier has no parameters, which does not need to be trained. We use the prototypical loss and the crossentropy loss to train these two subnetworks, respectively. Update the deformation network. We optimize the following prototypical loss function to endow the deformation network with the desired one-shot classification ability:

EQUATION

where P θ (y i | I i ) is the one-shot classifier in Eq. ( 3). Using the prototypical loss to update the deformation network encourages to generate diverse instances to augment the support set.

Update the embedding network. We use the cross-entropy loss to train the embedding network to directly classify the augmented support set S. Note that with the augmented support set S, we have enough training instances to train this subnetwork and the cross-entropy loss is the standard loss function in training a supervised classification network. Empirically, we found that using the additional cross-entropy loss speeds up the convergence and improves the recognition performance than using the prototypical loss solely.

5.2. TRAINING STRATEGY:

We summarize the entire training procedure of our IDeMe-Net on the base dataset D base in Alg. 1.

During meta-training, we have the gallery G and sample the N -way-m-shot training episode to produce S and Q. The embedding subnetwork learns an initial one-shot classifier g (•) on the original support set S using Eq. ( 3). Given a probe image I probe , we then sample the gallery images I gallery ∼ G and train the deformation subnetwork to produce the augmented support set S using Eq. ( 1). S is further used to update the embedding subnetwork and learn a better one-shot classifier.

We then conduct the final one-shot classification on the query set Q and backpropagate the prediction error to update the entire network. During meta-testing, we sample the N -way-m-shot testing episode to produce S and Q from the novel dataset D novel .

6. EXPERIMENTS:

Our IDeMe-Net is evaluated on two standard benchmarks: miniImageNet and ImageNet 1K challenge datasets. The codes and models will be released upon acceptance. 

6.1. RESULTS ON IMAGENET1K CHALLENGE DATASET:

Setup. We use ResNet-10 architectures for ANET and BNET. For a fair comparison with Hariharan & Girshick (2017), we evaluate the performance of using ResNet-10 (Table 1 ) and ResNet-50 (Table 2 ) for the embedding network. Stochastic gradient descent (SGD) is used to train IDeMe-Net in an end-to-end manner. It gets converged over 300 epochs. The initial learning rates of ANET, BNET, and the embedding network are set as 3 × 10 -3 , 3 × 10 -3 and 10 -1 , respectively and decreased by 1/10 every 30 epochs. The batch size is set as 32. We randomly sample 10 images per base category to construct the gallery G and we set as 2. Note that G is fixed during the entire experiments. ANET, BNET, and the embedding network are trained from scratch on D base . Our model is evaluated on D novel . n aug is cross-validated as 8, which balances between the computational cost and the augmented training data scale.

Baselines and competitors. We compare against several baselines and competitors as follows. (1) We directly train a ResNet-18 feature extractor on D base and use it to compute image features on D novel . We then train standard supervised classifiers on D novel , including neural network(Softmax), support vector machine (SVM), logistic regression (LR), and prototype classifier. The neural network classifier consists of 1 fully-connected layer and 1 softmax classification layer. (2) We also compare with state-of-the-art approaches to one-shot learning, such as matching network ( (3) The standard data augmentation methods are also compared here: "flipping": the input image is flipped from left to right; "Gaussian noise": cross-validated Gaussian noise N (0, 10) is added to each pixel of the input image; "Gaussian noise (feature level)": cross-validated Gaussian noise N (0, 0.3) is added to each dimension of the ResNet feature of each image. For fair comparisons, all theses augmentation methods use the prototype classifier as the one-shot classifier.

Results. Tables 1 and 2 summarize the results of using ResNet-10 and ResNet-50 as the embedding sub-network, respectively. Fig. 4 (a) further highlights that our IDeMe-Net consistently outperforms all the baselines and competitors by large margins. For example, using ResNet-10, the top-1 accuracy of IDeMe-Net in Table 1 is superior to prototypical network by 6% when n = 1, 2, 5, showing the sample efficiency of IDeMe-Net for one-shot learning. The top-5 accuracy in Table 1 demonstrates the similar trend, and our IDeMe-Net beats prototypical network, the second best competitors, Table 3 : Top-1 / Top-5 accuracy of the ablation study on ImageNet1k challenge dataset (ResNet-10). We use ResNet-10 as the embedding subnetwork. by more than 9% when n = 1. Using ResNet-50 as the embedding sub-network, the performance of all the approaches improves and our IDeMe-Net consistently achieves the best performance, as shown in Table 2 .

6.2. ABLATION STUDY ON IMAGENET1K:

We conduct extensive ablation study to evaluate the contribution of each component in our model.

Variants of IDeMe-Net. We consider seven different variants of our IDeMe-Net, as shown in Fig. 4 (b) and Table 3 . (1) IDeMe-Net-CE Loss: the embedding sub-network is trained using only the prototypical loss without the cross-entropy loss (CE loss). ( 2) IDeMe-Net -Proto Loss: the embedding sub-network is trained using only the cross-entropy loss without the prototypical loss. (3) IDeMe-Net -g (I gallery ): the gallery images are randomly chosen in IDeMe-Net without predicting their class probability. (4) IDeMe-Net -Aug. Testing: the deformed images are not produced in the testing phase. (5) IDeMe-Net -Def. Network: the weight matrix W in Eq. ( 1) is randomly generated instead of using the learned deformation sub-network. (6) IDeMe-Net -Gallery: the gallery images are directly sampled from the support set instead of constructing an additional Gallery. (7) Gallery Baseline: we simply use the gallery images to serve as the deformed images. As shown in Fig. 4 (b), our full IDeMe-Net model outperforms all these seven variants, showing that each component is essential and complementary to each other. We note that (1) Using CELoss and prototypical loss to update the embedding sub-network and the deformation sub-network, respectively, achieves the best result. As shown in Fig. 4 (b), the accuracy of IDeMe-Net -CELoss is marginally lower than IDeMe-Net but still higher than the prototypical classifier baseline, while IDeMe-Net -Proto Loss underperforms the baseline. (2) Our strategy for selecting the gallery images is the key to diversify the deformed images. As we can see, randomly choosing the gallery image (IDeMe-Net g (I gallery )) or sampling the gallery images from the support set (IDeMe-Net -Gallery) obtains no performance improvement. One reasonable explanation is that they only introduce noise or redundancy and do not bring in efficient information. (3) Our improved performance comes from the diversified deformed images, rather than the embedding sub-network. Without producing the deformed images in the testing phase (IDeMe-Net -Aug. Testing), the performance is close to the baseline, suggesting that training on the deformed images does not obviously benefit from the embedding sub-network. That is, the performance gain of our IDeMe-Net mainly results from the deformed images generated in the testing phase. (4) Our meta-learned deformation sub-network effectively exploits the complementarity and interaction between the probe and gallery image patches, producing the key information in the deformed images. To show this point, we investigate two deformation strategies: randomly generating the weight vector w (IDeMe-Net -Def. Network) and setting all the weights to be 0 (Gallery Baseline); in the latter case, it is equivalent to purely using the gallery images to serve as the deformed images. Both strategies perform worse than the prototypical classifier baseline, indicating the importance of meta-learning a deformation strategy. Different division schemes. In the deformation sub-network and Eq. ( 1), we equally split the image into 3 × 3 patches. Some alternative division schemes are compared in Table 1 and Fig. 4(c ). Specifically, we consider the 1 × 1, 5 × 5, 7 × 7, and pixel level division schemes in Eq. ( 1) and report the results as IDeMe-Net (1 × 1), IDeMe-Net (5 × 5), IDeMe-Net (7 × 7), and IDeMe-Net (pixel level), respectively. The experimental results suggest the patch-level fusion, rather than image-level or pixel-level fusion in our IDeMe-Net. The image-level division (1 × 1) ignores the local combination and deforms through a global combination, thus decreasing the diversity. The pixel-level division is particularly suspect to the disarray of the local information while region-level division (3×3, 5×5, 7×7) considers the region as the basic unit to maintain some local information.

In addition, the results show that using a fine-grained patch size (i.e., 5×5 division and 7×7 division) may achieve slightly better results than our 3 × 3 division. This also makes senses since fine-grained patch deformed information is learned. In brief, our region-level division not only maintains the significant region information but also deforms it to increase diversity.

Number of synthesized images. We also show how the top-1 accuracy changes with respect to the number of synthesized images in Fig. 4(d) . Specifically, we change the number of synthesized images n aug in the deformation sub-network, and visualize the 5-shot top-5 accuracy on the Imagenet-1K dataset. It shows that when n aug is changed from 0 to 8, the performance of our IDeMe-Net is gradually improved. The performance saturates when enough synthesized images are generated (n aug > 8).

Visualization. Fig. 5 shows t-SNE van der Maaten & Hinton (2008) visualizations for novel classes from our IDeMe-Net, the Gaussian baseline, and the Gallery baseline. For the Gaussian baseline, the synthesized images are heavily clustered and close to the prob images. By contrast, the synthesized Our IDeMe-Net has different react according to the contents of the gallery image: ignore most patches in the misleading gallery image and utilize the significant patch in the highly relevant gallery image. We can observe that the weight(first row) in the second sample is much larger than in the third sample. This intuitively shows why our model work.

6.3. RESULTS ON miniIMAGENET:

Setup and competitors. We use a ResNet-18 architecture as the embedding sub-network. We use the same experimental setting as the ImageNet1k challenge dataset. As summarized in Table 4 , we mainly focus on three groups of the competitors: (1) meta-learning algorithms, such as MAML (Finn et Results. We report the results in Table 4 (a). Impressively, our IDeMe-Net consistently outperforms all these state-of-the-art competitors by large margins in 5-shot classification scenarios. This further validates the general effectiveness of our proposed model in addressing one-shot learning tasks.

7. CONCLUSIONS:

In this paper, we proposed a conceptually simple but powerful approach to one-shot learning that uses a trained image deformation network to generate additional examples. Our deformation network leverages unsupervised gallery images to synthesize deformed images and is trained end-toend with meta-learning. Extensive experiments demonstrate that our approach achieves the state-ofthe-art performance on multiple one-shot learning benchmarks by large margins.

:

The two images could be the same image.

Figures and Tables:

al. (2017); Li et al. (2017); Vinyals et al. (2016); Snell et al. (2017); Ravi & Larochelle (

Figure 1: Deformation illustration. Left to right: ghost, stitched, montage, and partly erased images.

Figure 2: Examples of the deformed images generated by our image deformation network.

), memory network (Santoro et al. (2016); Cai et al. (2018)) and relation network (Sung et al. (2018)). The attention mechanism is studied in Wang et al. (2017); Mishra et al. (2016): Wang et al. (2017) analyzed the relation between visual and semantic representations; Mishra et al. (2016) learned the combination of temporal convolutions and soft attention in the labeled training set. Quite different from Wang et al. (2017); Mishra et al. (2016), our IDeMe-Net concentrates on learning to use the complementarity and interaction among visual patches. Metric Learning. This is another important research line in one-shot learning. The goal of metriclearning is to learn a metric space which can be optimized for one-shot learning. The recent works include Deep Siamese Networks (Koch et al. (2015)), Matching Nets (Vinyals et al. (2016)), PROTO-NET (Snell et al. (2017)) and Gidaris & Komodakis (2018). Data Augmentation. The key limitation of one-shot learning is the lack of sufficient training images. Augmenting additional instances can help train supervised classifiers (Krizhevsky et al. (2012); Chatfield et al. (2014); Zeiler & Fergus (2014)). The standard techniques include flipping, rotating, adding noise and randomly cropping images. The classical data augmentation includes adding Gaussian perturbation, transforms, or rescales of training images. However, adding noise or jittering on the origin images are particular suspect to visual similarity with the origin images. Furthermore, previous works either seek additional training images by the semi-supervised manners (Ren et al. (2018); Rasmus et al. (2015)), augment new instances by transforms, rescales, or directly synthesize new instance in the feature domain (Wang et al. (2018); Hariharan & Girshick (2017)).

the recent work (Vinyals et al. (2016); Ravi & Larochelle (2017); Finn et al. (2017); Snell et al. (2017); Wang et al. (2018)), we frame one-shot learning in a meta-learning formulation: we have a base category set C base and a novel category set C novel , in which C base ∩ C novel = ∅; correspondingly, we have a base dataset D base = {(I i , y i )}, y i ∈ C base , and a novel dataset D novel = {(I i , y i )}, y i ∈ C novel . We aim to learn a classification algorithm on D base that is able to generalize to unseen categories C novel with one or few training examples per class.

Figure 3: Overall architecture of our image deformation meta-network (IDeMe-Net).

, the embedding subnetwork f θ emb (•) consists of a deep convolutional network for feature extraction and a non-parametric one-shot classifier which will be explained later. Given an input image I, we use a residual network(He et al. (2015)) to produce the corresponding feature rep-resentation f θ emb (I). To facilitate the training process, we introduce an additional fully-connected layer on top of the embedding network with cross-entropy loss (CELoss) that outputs |C base | scores.4.3 ONE-SHOT CLASSIFIERDue to its superior performance, we use the non-parametric prototypical classifier(Snell et al. (2017)) as the one-shot classifier. During each episode, given the sampled S, Q, and G, the deformation network produces the augmented support set S. FollowingSnell et al. (

The training procedure of the IDeMe-Net. G is the fixed gallery. f θ is our IDeMe-Net. 1: procedure TRAIN EPISODE The procedure of one training episode ← randomly sample instances belong L sample the support set 4: Q ← randomly sample instances belong L sample the query set 5: learn the prototypical classifer P from f θ emb (S) 6: S ← S initialize the augment support set 7: for c in L do enumerate the choosing class 8: pool ←use P to select % images in G having highest class probability of c 9: for (I prob , c) in S c do enumerate the instance of class c in S 10: I gallery ← randomly sampled from pool 11: I syn ← f θ def (I prob , I gallery ) 12: S ← S ∪ (I syn , c) classifer P from f θ emb ( S) 16: use P to classify f θ emb (Q) and get the prototypical Loss 17: use f θ emb to classify S and get the CELoss 18: update θ emb with the CELoss 19:

Vinyals et al. (2016)), generation SGM (Hariharan & Girshick (2017)), prototypical network (Snell et al. (2017)), Cosine Classifier & Att. Weight Gen(Cos & Att.) (Gidaris & Komodakis (2018)),PMN and PMN w/H (Wang et al. (2018)) .

Figure 4: Ablation study on the ImageNet1k challenge dataset: (a) highlights the comparison with several competitors; (b) shows the impact of different components in our IDeMe-Net; (c) analyzes the impact of different division schemes; (d) shows how the performance changes with respect to the number of synthesized images. Best viewed in color with zoom.

images of our IDeMe-Net scatter widely in the class manifold and tend to locate more around the class boundaries. For the Gallery baseline, the synthesized images are the same as the gallery images and occasionally fall into manifolds of other classes. Interesting, comparing Fig. 5(b) and Fig. 5(c), our IDeMe-Net effectively deforms those misleading gallery images back to the correct class manifold. Deformed Images. Here we show some examples of our deformed images in the Fig.2. In the first example, IDeMe-Net synthesizes a ghost image combining two lions with different head poses. In the second and the third samples, we have the same prob image but with different gallery images.

al. (2017)) and Meta-SGD (Li et al. (2017)); (2) metric-learning algorithms, including Matching Nets (Vinyals et al. (2016)), Prototypical Network (Snell et al. (2017)), RELATION NET (Sung et al. (2018)), SNAIL(Mishra et al. (2016)), Delta-encoder(Schwartz et al. (2018)) and Cosine Classifier & Att. Weight Gen(Cos & Att.) (Gidaris & Komodakis (2018)).

Top-1 / Top-5 accuracy on ImageNet1K Challenge Dataset (ResNet-10). We use ResNet-10 as the embedding subnetwork. n indicates the number of training examples per class.

Table content: <table><tr><td/><td>Methods</td><td>n = 1</td><td>2</td><td>5</td><td>10</td><td>20</td></tr><tr><td/><td>Softmax</td><td>-/16.3</td><td>-/35.9</td><td>-/57.4</td><td>-/ 67.3</td><td>-/72.1</td></tr><tr><td>Baselines</td><td>LR SVM</td><td>18.3/42.8 15.9/36.6</td><td>26.0/54.7 22.7/48.4</td><td>35.8/66.1 31.5/61.2</td><td>41.1/71.3 37.9/69.2</td><td>44.9/74.8 43.9/74.6</td></tr><tr><td/><td>Prototype Classifier</td><td>17.1/39.2</td><td>24.3/51.1</td><td>33.8/63.9</td><td>38.4/69.9</td><td>44.1/74.7</td></tr><tr><td/><td>Matching Network</td><td>-/ 43.0</td><td>-/ 54.1</td><td>-/ 64.4</td><td>-/ 68.5</td><td>-/72.8</td></tr><tr><td/><td>Prototypical Network</td><td>16.9/41.7</td><td>24.0/53.6</td><td>33.5/63.7</td><td>37.7/68.2</td><td>42.7/72.3</td></tr><tr><td>Competitors</td><td>Generation-SGM PMN</td><td>-/ 34.3 -/ 43.3</td><td>-/ 48.9 -/ 55.7</td><td>-/ 64.1 -/ 68.4</td><td>-/ 70.5 -/ 74.0</td><td>-/74.6 -/ 77.0</td></tr><tr><td/><td>PMN w/ H</td><td>-/ 45.8</td><td>-/ 57.8</td><td>-/ 69.0</td><td>-/ 74.3</td><td>-/ 77.4</td></tr><tr><td/><td>Cos &amp; Att.</td><td>-/ 46.0</td><td>-/ 57.5</td><td>-/ 69.1</td><td>-/ 74.8</td><td>-/ 78.1</td></tr><tr><td/><td>Flipping</td><td>17.4/39.6</td><td>24.7/51.2</td><td>33.7/64.1</td><td>38.7/70.2</td><td>44.2/74.5</td></tr><tr><td>Augmentation</td><td>Gaussian Noise</td><td>16.8/39.0</td><td>24.0/51.2</td><td>33.9/63.7</td><td>38.0/69.7</td><td>43.8/74.5</td></tr><tr><td/><td colspan="2">Gaussian Noise(feature level) 16.7/39.1</td><td>24.2/51.4</td><td>33.4/63.3</td><td>38.2/69.5</td><td>44.0/74.2</td></tr><tr><td>Ours</td><td>IDeMe-Net</td><td>23.1/51.0</td><td colspan="4">30.1/60.9 39.0/69.8 42.7/73.4 45.0/75.1</td></tr></table>

Top-1 / Top-5 results on ImageNet1K Challenge Dataset (ResNet-50). We use ResNet-50 as the embedding subnetwork. n indicates the number of training examples per class. Our model is also evaluated on the large-scale ImageNet 1K dataset. Following the data split in Hariharan & Girshick (2017), we divide the original 1K categories into 389 base (D base ) and 611 novel (D novel ) categories. The base categories are further divided into two disjoint subsets: base validation set D cv base (193 classes) and base evaluation set D f in base (196 classes) and the novel categories are divided into two subsets as well: novel validation set D cv novel (300 classes) and novel evaluation set D f in novel (311 classes). We use the base/novel validation set D cv for cross-validating hyper-parameters and use the base/novel evaluation set D f in to conduct the final experiments. The same experimental setup is used as in Hariharan & Girshick (2017) and the reported results are averaged over 5 trails. Here we focus on synthesizing novel instances and we thus evaluate the performance primarily on novel categories, i.e., 331-way-n-shot settings, which is also consistent with most of the contemporary work (Vinyals et al. (2016); Snell et al. (2017); Ravi & Larochelle (2017)).

Table content: <table><tr><td>Methods</td><td>n=1</td><td>2</td><td>5</td><td>10</td><td>20</td></tr><tr><td>Softmax</td><td>-/28.2</td><td>-/51.0</td><td>-/ 71.0</td><td>-/78.4</td><td>-/82.3</td></tr><tr><td>SVM</td><td>20.1/41.6</td><td>29.4/57.7</td><td>42.6/72.8</td><td>49.9/79.1</td><td>55.8/83.2</td></tr><tr><td>LR</td><td>22.9/47.9</td><td>32.3/61.3</td><td>44.3/73.6</td><td>50.9/78.8</td><td>56.2/82.4</td></tr><tr><td>Prototype Classifier</td><td>20.8/43.1</td><td>29.9/58.1</td><td>42.4/72.3</td><td>49.5/79.0</td><td>56.0/83.0</td></tr><tr><td colspan="2">Generation SGM Hariharan &amp; Girshick (2017) -/47.3</td><td>-/60.9</td><td>-/73.7</td><td>-/79.5</td><td>-/83.3</td></tr><tr><td>IDeMe-Net</td><td>30.3/60.1</td><td>39.7/69.6</td><td>47.5/77.4</td><td>51.3/80.2</td><td>56.5/83.6</td></tr></table>

Top-1 accuracy on miniImageNet. The "±" indicates 95% confidence intervals over tasks.

Table content: <table><tr><td>Methods</td><td colspan="2">miniImageNet (%) 1-shot 5-shot</td></tr><tr><td>MAML</td><td>48.70±1.84</td><td>63.11±0.92</td></tr><tr><td>Meta-SGD</td><td>50.47±1.87</td><td>64.03±0.94</td></tr><tr><td>Matching Nets</td><td>43.56±0.84</td><td>55.31±0.73</td></tr><tr><td colspan="2">Prototypical Network 49.42±0.78</td><td>68.20±0.66</td></tr><tr><td>RELATION NET</td><td>57.02±0.92</td><td>71.07±0.69</td></tr><tr><td>SNAIL</td><td>55.71±0.99</td><td>68.88±0.92</td></tr><tr><td>Delta-encoder</td><td>58.7</td><td>73.6</td></tr><tr><td>Cos &amp; Att.</td><td>55.45±0.89</td><td>70.13 ±0.68</td></tr><tr><td>Prototype Classifier</td><td>52.54±0.81</td><td>72.71±0.73</td></tr><tr><td>IDeMe-Net</td><td>57.71±0.89</td><td>74.34±0.78</td></tr></table>

Peer review (iclr2019):
 This paper addresses the problem of few-shot learning, specifically by incorporating a mechanism for generating additional training examples to supplement the given small number of true training examples in an episode. The example generation method takes as input a support set image and a gallery image (which can be any random image) and produces a new synthesized support example that is a linear weighted combination of the two input images (in terms of pixel patches) and is assumed to belong to the same class as the support set example input. The synthesis process is parameterized as an “image deformation network” whose weights must be learned. Additionally, there is an embedding network that takes as input the full support set (with true and synthesized examples) and is used in a prototypical-net fashion (Snell 2017) to classify the given query set. A set number of gallery images are created from the training base classes and this is the fixed set of gallery images that is used for both training and testing. The image deformation network and embedding network weights are trained end-to-end using the standard meta-learning loss on the query set given the support set. Experiments are conducted on the ImageNet-1K Challenge and Mini-Imagenet benchmarks. Pros: - Sizable improvement on ImageNet-1K benchmark relative to previous work. - Detailed ablation study is done to display the benefits of choices made in model. Cons: - Idea is not that novel relative to all the recent work on learning to supplement training data for few-shot learning (Hariharan 2017, Wang 2018, Gao 2018, and Schwartz 2018). Additionally, it is not made clear why we expect the idea proposed in this paper to be better relative to previous ideas. For example, Delta-Encoder (Schwartz 2018) also supplements training data by working in image-space rather than in feature-space (as done in the other previous work mentioned). One big difference seems to be that for the Delta-Encoder work, the image generation process is not trained jointly with the classifier but this does not seem to have a big impact when we compare the performance between Delta-Encoder and proposed model on Mini-Imagenet benchmark. - Benefit in Mini-Imagenet benchmark is not very large. Compared to Delta-Encoder (Schwartz 2018), the proposed model does worse for 1-shot case and intersects the confidence interval for 5-shot case. Remarks: - Could add citation for NIPS 2018 paper *Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks* (https://arxiv.org/pdf/1810.11730.pdf) - Delta-Encoder (Schwartz 2018) paper should be mentioned in related work and contrasted against. - How many supplemented examples are used per class? I believe it is n_aug=8 but this could be stated more clearly. Additionally, how many supplemented examples are used in the previous work that are being compared to? It would be useful to have this information and to make sure the comparison is consistent in that the number of supplemented examples are comparable across different models that synthesize examples. - In algorithm 1, it seems that only CELoss is used to update parameters of embedding network but on page 6, it says "we found that using additional cross-entropy loss speeds up convergence and improves recognition performance than using prototypical loss solely"? - At many points in the paper, *prob image* is used instead of *probe image*. - Page 10: *Our IDME-Net has different react...* => *Our IDME-Net has different output...* - Page 10: *...why our model work* => *...why our model works* Gao et al. Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks. 2018. Hariharan et al. Low-shot Visual Recognition by Shrinking and Hallucinating Features. 2017. Schwartz et al. Delta-Encoder: an effective sample synthesis method for few-shot object recognition. 2018. Snell et al. Prototypical Networks for Few-Shot Learning. 2017. Wang et al. Low-Shot Learning from Imaginary Data. 2018.