Title: VOCABULARY-INFORMED VISUAL FEATURE AUGMEN-TATION FOR ONE-SHOT LEARNING

Abstract: A natural solution for one-shot learning is to augment training data to handle the data deficiency problem. However, directly augmenting in the image domain may not necessarily generate training data that sufficiently explore the intra-class space for one-shot classification. Inspired by the recent vocabulary-informed learning, we propose to generate synthetic training data with the guide of the semantic word space. Essentially, we train an auto-encoder as a bridge to enable the transformation between the image feature space and the semantic space. Besides directly augmenting image features, we transform the image features to semantic space using the encoder and perform the data augmentation. The decoder then synthesizes the image features for the augmented instances from the semantic space. Experiments on three datasets show that our data augmentation method effectively improves the performance of one-shot classification. Extensive study shows that data augmented from semantic space are complementary with those from the image space, and thus boost the classification accuracy dramatically. Source code and dataset will be available.
The success of recent machine learning (especially the deep learning) greatly relies on the training process on hundreds or thousands of labelled training instances of each class. However in practice, it might be extremely expensive or infeasible to obtain many labelled data, e.g. for objects in dangerous environment with limited access. On the other hand, human can recognize an object category easily with only a few shots of training examples Thrun (1996). Inspired by such an ability of humans, one-shot learning aims at building classifiers from a few or even a single example.
The major obstacle of learning good classifiers in one-shot learning setting is the lack of enough training data. Thus a natural recipe for one-shot learning is to augment the data, which has been conducted in various ways. The dominant approach adopted by previous work is to bring in more images Krizhevsky et al. (2012) for each category as training data. These additional augmented training images could be borrowed from unlabelled data Fu et al. (2015) or other relevant categories Wang & Hebert (2016a;b); Li & Hoiem (2016); Lim et al. (2011) in an unsupervised or semisupervised fashion; however the semantic signals of augmented data are often noisy and unreliable and may suffer from negative transfer when the augmented data are from different classes. On the other hand, synthetic images rendered from virtual examples Movshovitz-Attias (2015); Park & Ramanan (2015); Movshovitz-Attias et al. (2015); Dosovitskiy et al. (2015); Zhu et al. (2016b); Opelt et al. (2006) are semantically correct but require careful domain adaptation to transfer the knowledge to the real image domain. In contrast, we propose to directly augment training data in the image feature domain rather than the original image. Augmenting data in image feature domain allows us to interact with useful discriminative signals more directly. The most similar work to us is Zhu et al. (2016b); Opelt et al. (2006) , where the feature patches (e.g. HOG) of the object parts are combined to synthesize new feature representation. However, their approach requires strong heuristics and spatial information to learn the combination. On the contrary, we augment data in compact deep learning based feature space, which is stronger for classification but contains limited the spatial information.
A straightforward approach to augment image feature is to add random vectors to the feature of each single training image. However, the cutting plane for the classification is usually not in a regular shape, e.g. a hyper ball, and such a simple disturbance, e.g. sampled from Gaussian distribution, may not sufficiently explore the intra-class space for each individual category. Our idea of data

:

We extract image feature and project it to the semantic space using the encoder. After augmenting data in semantic space, we use the decoder to obtain the corresponding image feature for augmented semantic feature. Both real and augmented data are used to train the classification model.

augmentation is to bring in the additional knowledge by leveraging the semantic space pre-trained by linguistic models such as google word2vec Mikolov et al. (2013) from a huge repository of linguistic corpora. In semantic space, similar concepts are supposed to have similar feature, such that the overall space demonstrates superior manifold continuity over semantic meaning, which is ideal for data augmentation. To leverage such good property, we learn a mapping (f (x)) to transform the image feature into the semantic space, where we add Gaussian random vectors on the semantic feature from the only training image. We then learn an inverse mapping (g(x)), from the semantic to image, to convert the synthesized semantic features into image features, which are then fed into the classification model during the training. In experiment, we found the data augmented in semantic space complements the data augmented in the image feature domain. Each set of augmented data can improve the performance individually, and the best performance is achieved by adding them together.

The semantic space also allows us to leverage the knowledge resides in the vocabulary space. Like in the image feature domain where not all the spots corresponds to a meaningful input image Sabour et al. (2016) , it might also be the case for the semantic space where not all the points represents a real concept. As such, we adopt a non-parametric approach to synthesize training data in the semantic domain leveraging the huge repository of linguistic corpora. Specifically, we calculate the semantic feature for a large number of vocabulary and phrases Mikolov et al. (2013) as a pool of candidate semantic features, from where we search for nearest neighbors for the given training instance as the augmented data. The semantic feature augmented in this way is then guaranteed to represent certain concepts that can be described by some words. Such nearest neighbors set is semantically similar to the training data and can thus help increase the intra-class coverage and differentiate the one-shot categories. For example, as visualized in Fig. 5 (b), the neighbors of training instances in "killer whale" are corresponding to the semantic features of related topics, such as "sea lions" and "orcas", which are different from the other one-shot training categories such as "grizzly bear".

The mapping between feature spaces, f (x) and g(x), can be learned with a variety of models. In practice, we found that the auto-encoder is an elegant and effective framework to learn two mapping at the same time. Taken an image feature as input, the encoder (f (x)) learns to produce semantic feature, and the decoder (g(x)) learns to recover the image feature. The auto-encoder also allows an end-to-end training, which optimizes the image feature extractor, feature space mappings, and the classification model jointly.

The contribution of this paper are mainly in three aspects. First, we propose to perform data augmentation directly in compact and discriminative image feature space rather than the original image domain. Second, we leverage the semantic space for data augmentation, which brings additional chance to explore the intra-class space thoroughly. Last, we propose a simply yet elegant deep learning architecture that allows end-to-end training for optimal performance.

1.1. RELATED WORK:

One-shot learning is inspired by the celebrated aspects of human learning Jankowski et al. (2011) ; Lake & Salakhutdinov (2013) -Fei et al. (2003; 2006) ; Tommasi & Caputo (2009) , or deep generative models Rezende et al. (2016) ; Santoro et al. (2016) , or Bayesian auto-encoders Kingma & Welling (2014) . These approaches directly learn the one-shot learning model, rather than assume an auxiliary set of helping transfer knowledge as in Sec. 1.1.2. Thus these works only employ a rich class of generative models to explain the observed data, rather than directly augmenting the training data in the feature space like our work.

1.1.2. KNOWLEDGE TRANSFER BASED ONE-SHOT LEARNING APPROACHES:

This category of one-shot learning can also be explored via learning to learn Thrun (1996) , which is also known as transfer learning Pan & Yang (2010) (2016) . However, this type of approaches still requires and relies on the well-labelled, well-organized and large enough auxiliary set of similar visual information, while such auxiliary set requires expertise knowledge and might be expensive to obtain in many real world scenario. In contrast, the idea of our framework solves one-shot learning from the perspective of directly augmenting visual features and thus can also work when we do not have enough auxiliary dataset.

1.1.3. ONE-SHOT LEARNING BY AUGMENTING TRAINING INSTANCES:

In principle, augmenting training instances have been exploited in supervised learning settings by many previous works Krizhevsky et al. (2012) ; thus training data augmentation can also be employed to alleviate the problem of lacking instances in one-shot learning settings. In particular, this type of approaches have been explored via various ways: (1) Learning one-shot models by utilizing the manifold information of large amount of unlabelled data in a semi-supervised or transductive learning way Previous methods may suffer from different problems: the manifold information may be not useful for one-shot learners; there may be negative transfer when the off-shell models or relevant categories are very different from one-shot classes; rendering, composing new virtual examples or distorting training examples may either need experience of domain expertise or only works for different domains. GANs based approaches mostly focused on learning good generators to synthesize "realistic" images to "cheat" the discriminators, while the synthesized images are not learned to preserve the discriminative information, which, is in contrast to our network structure, where the discriminative instances are synthesized in visual feature domain.

2. METHODOLOGY:

In this section, we introduce the detail of our algorithm for one-shot classification. In a high level, our approach relies on continuity of the manifold in semantic space to synthesize training data, which is complementary with the data augmented directly from the image feature domain, and thus increase the generalization capability of the model. To achieve this, we learn a two-sided mapping between the image feature space and the semantic space in a single deep auto-encoder. With the help of these mappings, the data augmentation can be done in the semantic space, which is combined with the data augmented in the image space together to train the one-shot classification model.

Problem setup.:

The training image dataset D s = {I i , z i } Ns i=1 of N s samples. I i indicates the raw pixel matrix of image i. z i ∈ W s is a class label and the corresponding vocabulary w i taken from the vocabulary set W. The vocabulary W is learned by word2vec Mikolov et al. (2013) on large-scale corpus; each vocabulary entity w ∈ W is projected as a semantic vector u ∈ R d .

2.1. DEEP AUTO-ENCODER EMBEDDING NETWORK:

The deep auto-encoder embedding network learns the mapping between the Image Feature Space (IFS) to the Semantic Space (S-S). Linear mapping has been used in previous work Fu & Sigal (2016) . However, practically, such a mapping should be highly non-linear; and thus a deep auto-encoder framework with supervision on the bottleneck layer is introduced here. The whole deep auto-encoder includes two main components, i.e. image feature extractor, and auto-encoder.

Image feature extractor converts the raw images into image feature vectors by the pre-trained deep convolutional network. We use pre-trained VGG-16 network Chatfield et al. (2014) which consists of 13 convolutional and 3 fully connected layers. For an input image I i , the image feature extractor output the 4096-dimensional feature vector x i from the fc7 layer.

Auto-encoder network can be divided into the encoder and decoder. The encoder is composed of five fully connected layers with the 4096, 2048, 1024, 512, and 256 neurons accordingly followed by non-linear kernel RELU. The structure is detailed in Fig. 2(a) . The encoder part learns the mapping from IFS to S-S, i.e., ûi = f (x i ) by projecting the image feature vectors to corresponding semantic feature vectors. The decoder has exactly the symmetry architecture with the encoder learning the map from S-S to IFS, i.e., xi = g (u i ). It is attached after the semantic feature to reconstruct the image feature. Note that different from the standard unsupervised auto-encoder, our deep auto-encoder network tries to embed the IFS and S-S. In other word, the latent representation space learned by auto-encoder network is the semantic space, which should be posed as additional supervised information when learning such an embedding. Thus the loss function is

EQUATION

where Θ indicates the parameter set of auto-encoder network and P (•) is the regularized penalty function. We use L 2 -penalty in experiments. The Image feature extractor and auto-encoder networks are trained on D s . 

2.2. DATA AUGMENTATION:

We perform data augmentation in two domains and combine them together to train the classification model. Not necessary to have the augmented images, our goal is to generate more image features as the training data for each category.

Augmentation via Image Feature Space. A natural way of augmenting the training data is via the image feature space. Given the features x i of training instance i, the augmented data x F can be sampled from the corresponding Feature Gaussian, i.e., x F i ∼ N x i , σ F E , σ F ∈ R is the variance of each feature dimension; E is the identity matrix. σ F controls the deviation of noise added. To make the augmented feature vector x F i still be representative enough of the class of x i , we empirically set σ F as 10% of the distance between x i and its nearest training instance x j (z i = z j ) as this gives the best performance.

Augmentation via Semantic Space. We introduce two types of augmentation in semantic space.

(1) Semantic Gaussian: the same as Feature Gaussian, we can explore the continuity of the manifold in the semantic space. Given the features x i of training instance i, the augmented data x G can be sampled from the corresponding Semantic Gaussian, i.e., x G i ∼ g N ûi , σ G E , σ G ∈ R is the variance of each semantic dimension. The σ G is also empirically set as the 10% of the distance between ûi and its nearest semantic vector ûj (z i = z j ).

(2) Semantic Neighborhood (Fig 2(b) ): Note that the semantic space is not uniformly dense; the distributions of words of vocabulary may depend on the frequency of words in the linguistic corpora. This indicates that the intra-class deviation of different category might be different, and a gaussian distribution with a fixed standard deviation may not suit all the categories. Thus, given the feature x i of training instance i, the augmented data x H can be sampled from the corresponding Semantic Neighborhood , i.e., x H i ∼ g (u j ), u j ∈ N eigh (û i ), w j ∈ W and N eigh (û i ) ⊆ W. The N eigh (û i ) indicates the nearest neighborhood vocabulary set of ûi . These neighbors correspond to the most similar examples in the semantic space to our real training instance, and thus can be used to augment training data.

2.3. ONE-SHOT CLASSIFICATION AND TRAINING STRATEGY:

As mentioned above, the image feature extractor and auto-encoder networks are trained from D s ; and training set together with augmented dataset are employed to train the one-shot classifiers. Specifically, we can now employ the augmented dataset

x i ; x A i , z i , x A i ∈ x F i , x G i , x H i

to better train the one-shot classification model. Specifically, the original and augmented image features are grouped together and fed into the classification network. As shown in Fig. 1 , the one-shot classification model consists of two fully connected layer (with ReLu) and a soft-max layer for classification. 

Arm chair Folding chair:



Double couch L-shaped:



3. EXPERIMENTS:

In this section, we evaluate our method on three datasets which cover a variety of cases in real application. We also conduct analysis and ablation study to demonstrate the effectiveness of our data augmentation method.

3.1. EXPERIMENTAL SETUP:

Dataset. We choose three datasets that are representative for different types of classification problem.

To highlight the effectiveness of our data augmentation, we focus on more challenging scenario where the difference between classes are comparatively smaller than usual.

Animals with Attributes (AwA) dataset: AwA consists of 50 classes of animals (30,475 images in total). As the classes are all about animal, the difference between classes are smaller than arbitrary cases, such as table and dog in which data augmentation would be comparatively easy.

Cub-bird 200-2011 dataset Wah et al. (2011) contains 11,788 images of 200 categories of birds. This dataset is especially designed for fine-grained recognition, where the data augmentation is extremely hard as the difference between categories are fairly small.

ShapeNet Multiview dataset: Indoor scene understanding is yet another scenario where one-shot learning is especially useful. The capability to recognize furniture in a few shots is a desirable feature for indoor robots. To evaluate in this scenario, we create a synthetic dataset containing 4660 photo-realistic rendering of indoor furnitures. We select 233 object 3D models from 24 fine-grained furniture related classes from ShapeNet Core V2. For each model, we set up environment illumination and render photo-realistic color images from 20 cameras looking at the object from different direction and distance. Fig. 3 shows some example images from our synthetic indoor furniture dataset.

Settings and Parameters. The dropout rate and learning rate of the auto-encoder network is set to 0.8 and 1e -4 respectively to prevent overfitting. The batch size is set to 50. The network is trained by Stochastic Gradient Descendent (SGD), and usually converges in 300 epochs. The learning rate of image feature extractor is set to 1e -8 in the fine-tuning step. To prevent randomness due to the small training set, all experiments are repeated for 10 times, and the average performances over all runs are reported. We employ the mean accuracy (i.e. mean of the diagonal of the confusion matrix) to evaluate the classification. The source codes and the synthetic dataset will be released. The same 100-dimensional vocabulary dictionary used in Fu & Sigal (2016) is used here.

3.2. DOES THE DATA AUGMENTATION MATTER?:

To answer this question, we conduct the experiments in one-shot learning scenario on all three datasets.

( Most importantly, on all of the three datasets, training with our augmented data significantly improves the classification accuracy over the cases without any data augmentation. Particularly on AwA dataset, our framework dramatically increase the accuracy by 13.07 absolute point, which almost double the baseline. This validates that our deep auto-encoder embedding framework effectively augments the training features which bring in additional information that can be leveraged by the classification model. Note that all the results in Tab. 1 are significantly better than the accuracy of chance-level.

Fig. 5 (a) shows the confusion matrix on AwA before (top) and after (bottom) augmenting 10 instances per class. Without data augmentation, the confusion matrix has vertical bands indicating consistent misclassification; in contrast, the one with data augmentation has the clearer diagonal structure, which demonstrates the data augmentation helps to resolve the ambiguity across categories.

3.3. DOES THE SEMANTIC SPACE AUGMENTATION MATTER?:

To answer this question, we further conduct the one-shot learning experiment to evaluate the effectiveness of each type of data augmentation. Specifically, on each dataset, we use 1 instance per class for training, and each training instance is augmented by 5 instances. We compare the results of ( 1) without data augmentation (No Aug); and data augmentation by ( 2) Feature Gaussian (FeatG), ( 3) Semantic Gaussian (SemG), (4) both Feature Gaussian and Semantic Gaussian (FeatG+SemG), (5) Semantic Neighborhood (SemN); ( 6) Feature Gaussian and Semantic Neighborhood (FeatG+SemN); and (7) all together (FeatG+SemN+SemG). The results are shown in Tab. 2.

We notice that on all three datasets, the results of combining all three types of augmentation achieve the best performance, outperforming all the other methods by a clear margin. Specifically, the result of each single type of data augmentation can already beat the situation with no data augmentation, and combining data augmented from both image feature space and semantic space outperforms the case of using only one feature space. This indicates that these three types of data augmentation are complementary to each other. The combinations of Feature Gaussian and Semantic Neighborhood (FeatG+SemN) have the second best results on all three dataset. This shows that these two types of data augmentation are most complementary to each other. To visually demonstrate the effect of exploring semantic space, we visualize some classification results on AWA in Fig. 4 . Each training image is followed by three testing images that are misclassified as the categories below before the augmentation, which are later correctly classified after the augmentation. These testing images are visually similar to other categories, but different with the only training image on the left, and thus are extremely hard to recognize correctly. Our augmentation explore the semantic space and build connection between the testing images and the training image. We evaluate on AWA to demonstrate the behavior of our data augmentation approach with different number of training instances per category. We compare the one-shot learning results in Tab. 3 and few-shot learning results in Tab. 4. For each training instance, we augment 1, 5, 10 instances respectively. Surprisingly, our data augmentation managed to improve the performance dramatically for all the cases, though the performance get saturated earlier with more training instances given for each category, i.e. augmenting 1 instance per category gives the best performance for 5 instance per class. This validates that our data augmentation can effectively utilize the information from each individual training instance to generate good augmented data. Interestingly, the performance does not keep increasing all the time as more instances are augmented, as shown in Tab. 4. It is largely due to the increasing possibility of augmenting non-discriminative noisy instances with a large number of instances augmented.

3.5. DOES THE DEEP LEARNING ARCHITECTURE MATTER?:

To demonstrate if our deep learning architecture is optimal and suitable for data augmentation, we run the same experiment in the previous section but replacing deep auto-encoder embedding with other supervised model, such as logistic regression (LR)1 , and support vector regressor (SVR), and the results are shown in Tab. 3 as well.

As can be seen, our data augmentation method consistently improve the classification accuracy with either LR or SVR. This demonstrates that our method is not constrained to work only with deep learning architecture, but can be generalized to work ubiquitously well with many different kinds of models. Nevertheless, the deep auto-encoder based model still shows superior base performance with no data augmentation and brings in much larger performance gain compared to the other two traditional models. This implies that the auto-encoder achieves a better mapping between two feature spaces, and the end-to-end learning allows joint optimization of the whole system in order to achieve optimal performance. Note that our vocabulary-informed feature augmentation framework is actually orthogonal to the one-shot learning approaches. Thus it is also interesting to see whether our augmented features can also be used to help those previous work. To demonstrate this point, we run the codes of SS-Voc Fu & Sigal (2016) on the 10-way target classes from AWA dataset without and with the data augmentation. The classification accuracy is improved from 0.432 to 0.453 after using the data agumentation. This validates that the our feature augmentation complements typical one-shot learning algorthms.

4. CONCLUSION:

We propose a vocabulary-informed data augmentation method for one-shot classification. We demonstrate that the data augmented from the semantic space complements the image feature domain and thus can further increase the overall classification performance. Future research can be held by investigating more generalized feature representation, extending the current system to zero-shot and open-set learning problem, or deploying in real applications.

Note SVM is not used here since it is unstable when trained by 1 training instance per class.

Figures and Tables:

Figure1: Overview of our framework. We extract image feature and project it to the semantic space using the encoder. After augmenting data in semantic space, we use the decoder to obtain the corresponding image feature for augmented semantic feature. Both real and augmented data are used to train the classification model.

Fu et al. (2015);(2)  Adaptively learning the one-shot classifiers from off-shelf trained modelsWang & Hebert (2016a;b);Li & Hoiem (2016); (3) Borrowing examples from relevant categories Lim et al. (2011) or semantic vocabularies Fu & Sigal (2016); Ba et al. (2015) to augment the training set; (4) synthesizing new labelled training data by rendering virtual examples Movshovitz-Attias (2015); Park & Ramanan (2015); Movshovitz-Attias et al. (2015); Dosovitskiy et al. (2015) or composing synthesized representations Zhu et al. (2016b); Opelt et al. (2006) or distorting existing training examples Krizhevsky et al. (2012); (5) Generating new examples by Generative Adversarial Networks (GANs) Zhu et al. (2017; 2016a); Goodfellow et al. (2014); Reed et al. (2016); Radford et al. (2016); Mao et al. (2017); Durugkar et al. (2017);Huang et al. (2017)

Figure 2: (a) Auto-encoder Network. We use fully connected layers and the number indicates the amount of neurons. Each fully connected layer is followed by non-linear kernel RELU. (b) Augmenting data in semantic space using auto-encoder. The training image (in blue box) feature is mapped to the semantic space (blue text), where nearest neighbor (red text) are searched. The neighboring semantic features are then projected back to the image feature domain as the augmented features (red box). Noted that for purpose of visualization, we demonstrate with images and words which are actually feature vectors.

Figure 3: Example images from ShapeNet Multiview dataset. As a reflection of the real indoor scene, the categories are chosen to cover various types of furniture, whereas including fine-grained categories as well.

Fig. 5(b)  shows the visualization of semantic space. Intuitively, the neighborhood vocabulary of each training prototype can help differentiate the corresponding prototypes with the other training prototypes. Thus these vocabulary in the neighborhood set of one prototype, i.e. Semantic Neighborhood, can be used to reliably augment the data.

Figure 4: Visualization of the classification results. Each training image is followed by three testing images that are misclassified as the categories below before the augmentation, which are later correctly classified after the augmeantation. These testing images are visually different with the only training data, ambiguous with other categories, and thus hard to recognize. Our augmentation can help to correctly recognize them.

Figure 5: (a) Confusion matrix on AWA before (top) and after (bottom) data augmentation. Learning with our augmented data shows clear diagnal pattern in the confusion matrix. Visualization of the semantic space. We show the t-SNE visualization of the semantic feature space. The words in boxes are the mapping of training image in the semantic space, and close neighbors are shown. The neighborhoods extend the single training data to a space semantically meaningful.

DOES THE DATA AUGMENTATION CAN ALSO HELP THE OTHER STATE-OF-THE-ART ONE-SHOT LEARNING METHODS?

leads to catastrophic fails of generalizability of models. We can roughly divide the existing one-shot learning into two groups as discussed below in term of whether transferring extra knowledge to help learning one-shot tasks.

Table content: <table><tr><td>1.1.1 DIRECT ONE-SHOT LEARNING APPROACHES</td></tr><tr><td>One-shot learning in this category can be implemented via instance-based learning (such as nearest</td></tr><tr><td>neighbour or K-nearest neighbour), non-parameteric methods Fei</td></tr></table>

The classification accuracy of one-shot learning on three dataset. "Number of instances augmented" indicates how many virtual instances are generated from one training instance in each type of data augmentation method. The standard deviation is 0.5 ∼ 1.3%.Comparing with/without data augmentation. For each dataset, we randomly sample 1 instance from each category for training, and take all the other images for testing. All three types of data augmentation are used, and the results are shown in Tab. 1. We vary the number of instances augmented for each training instance. As a standard baseline for comparison, we also show the performance with the same classification network introduced in Sec. 2.3 without any data augmentation.

Table content: <table><tr><td>%)</td><td>No data augmentation</td><td colspan="4">Number of instances augmented Chance-level 1 5 10</td></tr><tr><td>AwA</td><td>17.50</td><td colspan="2">27.98 30.43</td><td>30.57</td><td>2.50</td></tr><tr><td>Cub-bird</td><td>5.29</td><td>7.16</td><td>7.89</td><td>8.21</td><td>0.50</td></tr><tr><td>Shapenet</td><td>14.21</td><td colspan="2">17.04 18.51</td><td>18.18</td><td>4.17</td></tr></table>

The results of combining different types of data augmentation in our framework. "FeatG", "SemG", and "SemN" denote the data augmentation methods by Feature Gaussian, Semantic Gaussian and Semantic Neighborhood respectively. Combining augmented data from both space achieves the best performance.

Table content: <table><tr><td colspan="3">Number of training Models No data augmentation instances/class</td><td>Number of instances augmented 1 5 10</td></tr><tr><td/><td>NN</td><td>17.50 ± 2.8</td><td>27.98 ± 1.3 30.43 ± 0.9 30.57 ± 0.8</td></tr><tr><td>1</td><td>LR Near-N</td><td>5.07 ± 1.1 21.96 ± 1.2</td><td>6.19 ± 1.5 26.45 ± 0.9 27.26 ± 1.6 26.85 ± 0.4 7.42 ± 0.2 8.31 ± 0.4</td></tr><tr><td/><td>SVR</td><td>18.28 ± 1.6</td><td>24.78 ± 1.4 24.96 ± 0.8 24.83 ± 0.1</td></tr></table>

One-shot Learning Performance on AwA dataset w.r.t number of training instances per category. NN, LR, Near-N and SVR indicate neural network, logistic regression, nearest neighbour and support vector regressor for classification and feature mapping. The chance rate is 2.5.

Table content: <table><tr><td colspan="3">Number of training Models No data augmentation instances/class</td><td colspan="2">Number of instances augmented 1 5 10</td></tr><tr><td/><td>NN</td><td>35.87</td><td>45.69 45.61</td><td>47.15</td></tr><tr><td>3</td><td>LR</td><td>16.77</td><td>19.06 18.23</td><td>18.65</td></tr><tr><td/><td>SVR</td><td>25.13</td><td>32.92 32.63</td><td>30.41</td></tr><tr><td/><td>NN</td><td>47.11</td><td>54.75 53.78</td><td>51.36</td></tr><tr><td>5</td><td>LR</td><td>25.92</td><td>26.72 26.24</td><td>26.70</td></tr><tr><td/><td>SVR</td><td>37.73</td><td>37.87 36.92</td><td>40.10</td></tr></table>

Few-shot learning performance on AwA dataset w.r.t number of training instances per category. NN, LR, and SVR indicate neural network, logistic regression, and support vector regressor for classification and feature mapping. The standard deviation is 0.5 ∼ 1.3%.

Table content: <table><tr><td>Training data</td><td/><td>Testing data</td><td/><td>Training data</td><td/><td>Testing data</td><td/></tr><tr><td>Grizzly bear</td><td>Deer</td><td>Gorilla</td><td>Cow</td><td>Cow</td><td>Deer</td><td>Antelope</td><td>Collie</td></tr><tr><td>Deer</td><td>Sheep</td><td>Tiger</td><td>Cow</td><td>Lion</td><td>Siamese cat</td><td>Gorilla</td><td>Bobcat</td></tr></table>

Peer review (iclr2018):
 Summary: This paper proposes a data augmentation method for one-shot learning of image classes. This is the problem where given just one labeled image of a class, the aim is to correctly identify other images as belonging to that class as well. The idea presented in this paper is that instead of performing data augmentation in the image space, it may be useful to perform data augmentation in a latent space whose features are more discriminative for classification. One candidate for this is the image feature space learned by a deep network. However they advocate that a better candidate is what they refer to as "semantic space" formed by embedding the (word) labels of the images according to pre-trained language models like word2vec. The reasoning here is that the image feature space may not be semantically organized so that we are not guaranteed that a small perturbation of an image vector will yield image vectors that correspond to semantically similar images (belonging to the same class). On the other hand, in this semantic space, by construction, we are guaranteed that similar concepts lie near by each other. Thus this space may constitute a better candidate for performing data augmentation by small perturbations or by nearest neighbour search around the given vector since 1) the augmented data is more likely to correspond to features of similar images as the original provided image and 2) it is more likely to thoroughly capture the intra-class variability in the augmented data. The authors propose to first embed each image into a feature space, and then feed this learned representation into a auto-encoder that handles the projection to and from the semantic space with its encoder and decoder, respectively. Specifically, they propose to perform the augmentation on the semantic space representation, obtained from the encoder of this autoencoder. This involves producing some additional data points, either by adding noise to the projected semantic vector, or by choosing a number of that vector*s nearest neighbours. The decoder then maps these new data points into feature space, obtaining in this way the image feature representations that, along with the feature representation of the original (real) image will form the batch that will be used to train the one-shot classifier. They conduct experiments in 3 datasets where they experiment with augmentation in the image feature space by random noise, as well as the two aforementioned types of augmentation in the semantic space. They claim that these augmentation types provide orthogonal benefits and can be combined to yield superior results. Overall I think this paper addresses an important problem in an interesting way, but there is a number of ways in which it can be improved, detailed in the comments below. Comments: -- Since the authors are using a pre-trained VGG for to embed each image, I*m wondering to what extent they are actually doing one-shot learning here. In other words, the test set of a dataset that is used for evaluation might contain some classes that were also present in the training set that VGG was originally trained on. It would be useful to clarify whether this is happening. Can the VGG be instead trained from scratch in an end-to-end way in this model? -- A number of things were unclear to me with respect to the details of the training process: the feature extractor (VGG) is pre-trained. Is this finetuned during training? If so, is this done jointly with the training of the auto-encoder? Further, is the auto-encoder trained separately or jointly with the training of the one-shot learning classifier? -- While the authors have convinced me that data augmentation indeed significantly improves the performance in the domains considered (based on the results in Table 1 and Figure 5a), I am not convinced that augmentation in the proposed manner leads to a greater improvement than just augmenting in the image feature domain. In particular, in Table 2, where the different types of augmentation are compared against each other, we observe similar results between augmenting only in the image feature space versus augmenting only in the semantic feature space (ie we observe that "FeatG" performs similarly as "SemG" and as "SemN"). When combining multiple types of augmentation the results are better, but I*m wondering if this is because more augmented data is used overall. Specifically, the authors say that for each image they produce 5 additional "virtual" data points, but when multiple methods are combined, does this mean 5 from each method? Or 5 overall? If it*s the former, the increased performance may merely be attributed to using more data. It is important to clarify this point. -- Comparison with existing work: There has been a lot of work recently on one-shot and few-shot learning that would be interesting to compare against. In particular, mini-ImageNet is a commonly-used benchmark for this task that this approach can be applied to for comparison with recent methods that do not use data augmentation. Some examples are: - Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. (Finn et al.) - Prototypical Networks for Few-shot Learning (Snell et al.) - Matching Networks for One-shot Learning (Vinyals et al.) - Few-Shot Learning Through an Information Retrieval Lens (Triantafillou et al.) -- A suggestion: As future work I would be very interested to see if this method can be incorporated into common few-shot learning models to on-the-fly generate additional training examples from the "support set" of each episode that these approaches use for training.