Title: NETWORK REPARAMETERIZATION FOR UNSEEN CLASS CATEGORIZATION

Abstract: Many problems with large-scale labeled training data have been impressively solved by deep learning. However, Unseen Class Categorization (UCC) with minimal information provided about target classes is the most commonly encountered setting in industry, which remains a challenging research problem in machine learning. Previous approaches to UCC either fail to generate a powerful discriminative feature extractor or fail to learn a flexible classifier that can be easily adapted to unseen classes. In this paper, we propose to address these issues through network reparameterization, i.e., reparametrizing the learnable weights of a network as a function of other variables, by which we decouple the feature extraction part and the classification part of a deep classification model to suit the special setting of UCC, securing both strong discriminability and excellent adaptability. Extensive experiments for UCC on several widely-used benchmark datasets in the settings of zero-shot and few-shot learning demonstrate that, our method with network reparameterization achieves state-of-the-art performance.

1. INTRODUCTION:

The rich and accessible labeled data has fueled the revolutionary successes of deep learning in various tasks, e.g., visual recognition (He et al. (2016) ), object detection (Ren et al. (2015) ), machine translation (Bahdanau et al. (2014) ), etc. However, requiring numerous annotated data severely limits the applicability of deep learning algorithms to Unseen Class Categorization (UCC) for which we only have access to a limited amount of information, which is frequently encountered in industrial applications. Recently, an increasing number of approaches have been proposed to solve UCC with the help of either attribute descriptions (zero-shot learning (ZSL)) (Kodirov et al. (2017) ; Zhang et al. (2017) ) or one/a few labeled samples for each class (few-shot learning (FSL)) (Snell et al. (2017) ; Xian et al. (2018) ).

Previous approaches to UCC mainly have the following characteristics and limitations: (i) To obtain powerful discriminative feature representation, they often train a deep classification model employing state-of-the-art multi-class classification techniques. However, such models are hard to be adapted to new classes with limited supervision information due to the high volume of model parameters and the gradual updating scheme. (ii) To ensure the consistency of training and test settings and adaptability to new classes, previous methods often train a deep model in an episode fashion (Vinyals et al. (2016) ), sometimes along with some specially designed meta-learning updating rules (Finn et al. (2017) ). With episode-based training, the model acquires adaptability to new tasks after many training episodes using the knowledge it grasps during the training. However, the episode-based training strategy severely limits the model's capability of extracting discriminative features, because it does not fully exploit the diversity and variance of all classes within the training dataset. The trained model treats the classes in each episode as new classes and attempts to separate them. Therefore, it does not have memory of the competing information of these classes against all the other ones in the whole dataset beyond the current episode. Due to the neglect of this global (dataset-wise rather than episode-wise) discriminative information, the feature extraction capability of the model is suppressed, thus limiting the UCC performance.

To address these issues, we propose to secure both powerful discriminability of feature extraction and strong adaptability of model classification through network reparameterization, i.e., reparametrizing the learnable weights of a network as a function of other variables. We decouple the feature extraction module and the classification module of a deep classification model, learn the former as a standard multi-class classification task to obtain a discriminative feature extractor, and learn the latter employing a light deep neural network that generates generic classification weights for unseen classes given limited exemplar information. We train the classification weight generator by following the episode-based training scheme to secure the adaptability. Our method can be flexibly applied to both ZSL and FSL, where the exemplar information about unseen classes are provided in the form of either the semantic attributes or one/a few labeled samples. Extensive experiments show that our proposed method achieves state-of-the-art performance on widely-used benchmark datasets for both tasks.

2. RELATED WORK:

With regard to the form of the exemplar information provided about unseen classes, UCC can be classified as zero-shot learning and few-shot learning.

2.1. ZERO-SHOT LEARNING:

ZSL requires recognizing unseen classes based on their semantic descriptions. It is approached by finding an embedding space where visual samples and semantic descriptions of a class are interacted so that the semantic description of an unseen class can be queried by its visual samples. Since the embedding space is often of high dimension, finding the best match of a given vector among many candidates shall inevitably encounter the hubness problem (Radovanović et al. (2010) ), i.e., some candidates will be biased to be the best matches for many of the queries. Depending on the chosen embedding space, the severeness of this problem varies. Some approaches select the semantic space as the embedding space and project visual features to the semantic space (Lampert et al. (2014) ; Frome et al. (2013) . Projecting the visual features into a often much lower-dimensional semantic space shrinks the variance of the projected data points and thus aggravates the hubness problem. Alternatively, some methods project both visual and semantic features into a common intermediate space (Akata et al. (2015) ; Sung et al. (2018) ; Zhang & Saligrama (2015) ). However, due to lacking training samples from unseen classes, these methods are prone to classify test samples into seen classes (Romera-Paredes & Torr (2015) ) (for the generalized ZSL setting, seen classes are included when testing). Recently, Zhang et al. (2017) proposed to choose the visual space as the embedding space and learned a mapping from the semantic space to visual space. Benefiting from the abundant data diversity in visual space, this method can mitigate the hubness problem at some extent. However, the limitation of this method is that it strives only to learn a mapping from semantic space to visual space such that the visual samples of a class coincide with the associated semantic description; it however neglects the separation information among visual features of different classes. Our method avoids this problem. We formulate bridging the semantic space and the visual space as a visual feature classification problem conditioned on the semantic features. We learn a deep neural network that generates classification weights for the visual features when fed with the corresponding semantic features. By nature of a classification problem, both intra-class compactness (visual features of the same classes are assigned with the same label) and inter-class separability (visual features of different classes are assigned with different labels) are exploited, hence resulting in a better mapping.

2.2. FEW-SHOT LEARNING:

FSL aims to recognize unseen classes when provided with one/a few labeled samples of these classes. A number of methods address it from the perspective of deep metric learning by learning deep embedding models that output discriminative feature for any given images (Ren et al. (2018) ; Vinyals et al. (2016) ; Snell et al. (2017); Triantafillou et al. (2017) ; Sung et al. (2018) ). The difference lies in the loss functions used. More common approaches are based on meta-learning, also called learning to learn, which is to learn an algorithm (meta-learner) that outputs a model (the learner) that can be applied on a new task when given some information (meta-data) about the new task. Following this line, approaches such as META-LSTM (Ravi & Larochelle (2017) ), MAML (Finn et al. (2017) ), Meta-SGD (Li et al. (2017) ), DEML+Meta-SGD (Li et al. (2017) ), Meta-Learn LSTM (Ravi & Larochelle (2017) ), Meta-Networks (Munkhdalai & Yu (2017) ), and REPTILE (Nichol et al. (2018) ) aim to optimize the meta-learned classifiers to be easily fine-tuned on new Besides, it cannot bridge information across different domains (due to dimension inconsistency) so that is not applicable for ZSL. We instead learn a network to generate classification weights directly from feature embeddings of support images; it is more powerful and flexible to solve both ZSL and FSL within the same framework.

3. METHOD:

We focus on Unseen Class Categorization (UCC), which is to recognize objects of unseen classes given only minimal information (a few labeled samples or the attributes) about the classes. Formally, suppose we have three sets of data We adopt the cosine similarity based cross entropy loss to train the weight generator g φ . Traditional multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. Luo et al. (2017) recently showed that replacing the dot product with cosine similarity can bound and reduce the variance of the neurons and thus result in models of better generalization. Gidaris & Komodakis (2018) further showed that using the cosine similarity instead of dot product for calculating classification score in the last fullyconnected layer of deep neural network brings benefit for classification, with some minor revisions. We adopt this technique to train our weight generator g φ . The classification score of a sample (e x , y) is calculated as

D = {D t , D s , D u }, where D t = {X t , Y t }

p(y = n|e x ) = exp(s cos(w y , e x )) N j=1 exp(s cos(w j , e x ))

,

(1)

e x = f θ (x),

(2) w j = g φ (a j ), for ZSL,

(3)

EQUATION

)

where s is a learnable scalar controlling the peakiness of the probability distribution generated by the softmax operator (Gidaris & Komodakis (2018) ), w j is the classification weight for class j generated by neural network g φ taking supporting information of the class as input, x is the input image, a j is the attribute vector for class j for ZSL, x i,j is the i-th input image of class j for FSL, j = 1, ..., N f , and N f is the number of shots for FSL.

In a typical UCC task T , the loss function is calculated as

EQUATION

)

where λ is a hyper-parameter weighting the l 2 -norm regularization of the learnable parameters of neural network g φ .

For ZSL, we are provided with semantic class attributes S = A t ∪ A u as the assistance for UCC.

The basic assumption for existing ZSL algorithms is that the visual-attribute relationship learned from seen classes in a certain embedding space is class-invariant and can be applied to unseen classes. With this assumption, existing methods either project visual features to semantic space or reversely project semantic features to visual space, or alternatively project both visual and semantic features to an intermediate space. In any case, the coincidence of visual and semantic features of a class is utilized to learn the visual-attribute relationship. Zhang et al. (2017) recently showed that it is advantageous to select the visual space as the embedding space because the abundance of data diversity in the visual space can significantly mitigate the so-called "hubness" problem. Their objective function is as follows:

EQUATION

where f θ is a feature extraction model which outputs a representation vector f θ (x i ) using image x i as input. h ψ is a mapping function which projects attribute vector a yi of class y i to the embedding space where f θ (x i ) lies. Through minimizing the least square embedding loss, the visual-attribute relationship can be established. With this relationship, in the testing stage, the attributes A u of unseen classes are mapped to the visual feature embedding space in which the visual feature of an images of any unseen class can find the best class attribute through nearest neighbor searching.

One can observe that this method learns the visual-attribute relationship by only utilizing the coincidence of the visual samples of a class with the associated semantic description. It however neglects to explore the inter-class separation of different classes, which shall be crucial to further avoid the hubness problem. To remedy this, we reformulate the learning of visual-attribute relationship from a regression problem to a visual feature classification problem. We directly learn a network g φ that outputs the classification weights for classifying visual features and use the cross-entropy loss on top of Softmax outputs to guide learning g φ . Through this reformulation, both intra-class compactness and inter-class separability are elegantly exploited for learning the visual-attribute relationship: and

Bq = {{x k,j } Q f k=1 , yj} M f j=1

3. Calculate loss according to Eq. 5 4. Update g φ through back-propagation. end while Visual features of the same classes should be assigned with the same label (compactness), while visual features of different classes are assigned with different labels (separability).

We follow the network reparameterization scheme by decoupling the feature extraction module f θ and the classification weight module which is generated by g φ . The feature extraction module f θ is trained as a standard multi-class classification task to enable us to obtain a discriminative feature representation for any given image. To learn g φ , we adopt the episode based training scheme by continuously exposing g φ with new (randomly sampled) ZSL tasks so as to secure good performance when new real tasks arrive in the testing stage. More specifically, we keep randomly sampling from D t = {X t , Y t } and A t ZSL tasks and feeding them to the network. Each task consists of M z classes and the associated M z attribute vectors. For each class, we randomly sample N z images. With a batch of M z N z images B v and M z attribute vectors B a , we train g φ by minimizing the loss function defined in Eq. 5. In the testing stage, given attributes of unseen classes A u , or S = A t ∪ A u for all (seen and unseen) classes as in generalized ZSL setting, we generate the corresponding classification weights using g φ . The generated classification weights, integrated with the feature extraction network f θ serve to classify images of unseen classes. Algorithm 1 outlines the main steps of our method for ZSL.

For FSL, one/a few labeled samples D s = {X s , Y s } for each unseen class are provided to help recognize objects of these classes. Our novel categorization framework can be easily extended from ZSL to FSL, simply by replacing the semantic attribute vectors with feature embedding vectors as the input to the classification weight generation network g φ . To train g φ , we keep randomly sampling FSL tasks from D t = {X t , Y t }, each of which consists of a support set and a query set. Images in the both sets are from the same classes. The support set consists of M f classes and N f images for each class. With the feature embeddings B e of the M f N f images as input, g φ generates the classification weights for the M f classes, which are then used to classify the feature embeddings of images from the query set. Note that if N f > 1, i.e., each class has multiple support samples, we average the embeddings of all images belonging to the same class and feed the averaged embedding to g φ . Similar to ZSL, we learn the resulting model by optimizing the loss function defined in Eq. 5. Algorithm 2 outlines the main steps for FSL.

One of the most distinct aspects of our method from the existing ones is that we decouple the feature extraction module and the classifier module of the deep classification model, and train each module on the most beneficial tasks. We train the feature extraction module as a standard multi-class classification task. This is motivated by the observation that a simple classifier (e.g., nearest neighbor), when taking as input features obtained by a powerful extractor, can outperform some sophisticated FSL models that use weaker feature extraction models. For example, as shown in Fiugre 2, using nearest neighbor (NN) as the classifier, we can achieve better one-shot classification accuracy than a recent FSL algorithm PROTO NET (Snell et al. (2017) ), when using features extracted by ResNet18 (He et al. (2016) ) trained on the Mini-Imagenet or CUB datasets. Moreover, if we train ResNet18 on ImageNet where massive data are available for obtaining a even more powerful feature extractor ResNet18 , we can achieve The reason for this surprising result is that the episode-based training scheme of existing FSL methods inherently suppresses obtaining a powerful feature extractor: In each episode, the model is fed with a new FSL task that is assumed to have no relationship with the previous ones. The model is trained to separate well the several classes within the task. However, since all training tasks are sampled from the training dataset, one class shall appear in many tasks. The inter-class separation across the whole dataset is neglected by existing FSL methods. Therefore, there is a dilemma for existing FSL algorithms: They need to be trained in an episodebased fashion to ensure flexibility, but which in return compromises feature discriminability. To avoid this awkward situation, our proposed method decoupling the network and training different components in different ways ensures powerful discriminability and strong adaptability.

4. EXPERIMENTS:

We evaluate our framework for both zero-shot learning and few-shot learning tasks.

Datasets and evaluation settings. We employ the most widely-used zero-shot classification datasets for performance evaluation, namely, AwA1 (Lampert et al. ( 2014)), AwA2 (Xian et al. (2018) ), CUB (Wah et al. (2011) ), SUN (Patterson & Hays (2012) ) and aPY (Farhadi et al. (2009) ).

The statistics of the datasets are shown in Table 1 . We follow the GBU setting proposed in (Xian et al. (2018) ) and evaluate both the conventional ZSL setting and the generalized ZSL (GZSL) setting. In the conventional ZSL, test samples are restricted to the unseen classes, while in the GZSL, they may come from either seen classes or unseen classes.

Implementation details. Following (Xian et al. (2018) ), we adopt ResNet101 as our feature extraction model f θ which results in a 2048-dimension vector for each input image. For the weight generation model g φ , we utilize two FC+ReLU layers to map semantic vectors to visual classification weights. The dimension of the intermediate hidden layer are 1600 for all the five datasets. We train g φ with Adam optimizer and a learning rate 10 -5 for all datasets by 1,000,000 randomly sample ZSL tasks. Each task consists of 32 randomly sampled classes, 4 samples for each class, i.e., M z = 32 and N z = 4. The hyper-parameters λ is chosen as 10 -4 , 10 -3 , 10 -3 , 10 -5 and 10 -4 for AwA1, AwA2, CUB, SUN and aPY, respectively. Our model is implemented with PyTorch.

Experimental results. Table 2 shows the experimental results. For the conventional ZSL setting, our method reaches the best for three out of the five datasets, while being very close to the best for one of the left two. Remarkably, our method consistently outperforms DEM (Zhang et al. (2017) ) for all the five datasets, which substantiates the benefit of our method of taking consideration of interclass separability when learning the mapping from semantic space to visual space. For GZSL setting where seen classes are also included to be the candidates, our method significantly outperforms all competing methods, reaching performance gains over the second best even about 30% in the AWA1 dataset. We analyze the reason for our dramatic advantage is that our method considers inter-class separation during the training stage so that the resultant classification weights for the seen classes possess good separation property after training. When they are concatenated with the classification weights generated from semantic descriptions of unseen classes in the testing stage, they shall be quite discriminative to discern that the incoming images do not belong to their classes. From the perspective of hubness problem, since the classification weights for seen class have good separation property, the weight vectors are less likely to be clustered in the embedding space, so that the risk is reduced that some candidates are selected as the nearest neighbors for many query images.

Datasets and evaluation settings. We evaluate few-shot classification on two widely-used datasets, Mini-ImageNet (Vinyals et al. (2016) ) and CUB (Wah et al. (2011) ). The Mini-ImageNet dataset has 60,000 images from 100 classes, 600 images for each class. We follow previous methods and use the splits in Ravi & Larochelle (2017) for evaluation, i.e., 64, 16, 20 classes as training, validation, and testing sets, respectively. The CUB dataset is a fine-grained dataset of totally 11,788 images from 200 categories of birds. As the split in Ravi & Larochelle (2017) , we use 100, 50, 50 classes for training, validation, and testing, respectively. For both datasets, we resize images to 224×224 to meet the requirement of our adopted feature extraction network. Following the previous methods, we evaluate both 5-way 1-shot and 5-way 5-shot classification tasks where each task instance involves classifying test images from 5 sampled classes with 1 (1-shot) or 5 (5-shot) randomly sampled images for each class as the support set. In order to reduce variance we repeat the evaluation task 600 times and report the mean of the accuracy with a 95% confidence interval.

Implementation details. We use ResNet18 as our feature extraction model f θ which results in a 512-dimension vector for each input image after average pooling. We train f θ on the two experimental datasets by following the standard classification learning pipeline: We use Adam optimizer with an initial learning rate 10 -3 which decays to the half every 10 epochs. The model is trained with 100 epochs. As for g φ , we use two FC+ReLU layers, same as in ZSL. The dimension of the intermediate hidden layer is 512 for both datasets. We train g φ using Adam optimizer with a learning rate 10 -5 and set the hyper-parameters λ = 10 -5 for both datasets. The model is trained with 60000 randomly sampled FSL tasks, each of which consist of 5 classes, with 1 or 5 samples as the support samples and another 15 as the query samples. 2017)) (Left) and our method (Right). Seen and unseen classes are in orange and blue, respectively.

Experimental results. Table 3 shows the results of the proposed method and the most recent ones.

From the table, we can get some interesting observations. First, the baseline method "ResNet18 + NN" beats most competing FSL algorithms where various sophisticated strategies are used. Meanwhile, the accuracy of feeding the classifier of PROTO NET with features obtained by ResNet18 ("ResNet18 feat. + PROTO NET classifier") is much higher than that obtained by training PROTO NET end to end with ResNet18 as the base model ("ResNet18 + PROTO NET"). These results support our analysis that the episode-based training scheme adopted by existing FSL approaches suppresses the discriminability of the feature extraction model. Second, compared with the baseline methods "ResNet18 feat. + NN" and "ResNet18 feat. + PROTO NET classifier", which use the same feature representations as our method, we get obvious improvements. This substantiates the benefit of the proposed weight generation strategy for FSL. Third, compared with the existing methods, our method reaches the best in the both datasets for both 1-shot and 5-shot evaluation settings, often by large margins. This shows the great advantage of our method for handling the FSL problem.

4.3. FURTHER ANALYSIS:

As we can see above, our method dramatically outperforms existing methods for the GZSL setting.

The advantage is much more significant than that for the ZSL setting. We have analyzed the reason is that the classification weights generated from the attributes of seen classes show good separation property so that the hubness problem is not as severe as that for other methods. The hubness problem refers that in ZSL, some candidate points are prone to be the nearest neighbors of many query points when the dimension is high. So, if the candidate points are more evenly distributed in the space, the less severe of the hubness problem should be. To validate this, we use t-SNE (Van Der Maaten (2014)) to visualize the classification weight vectors generated from all 200 class semantic vectors in the CUB dataset. As a comparison, we do the same thing for DEM (Zhang et al. (2017) ) which also learns mapping from semantic space to visual space. The result is shown in Figure 3 . We can observe that the points are more evenly distributed for our method than that for DEM. This further validates the benefit of our method in avoiding the hubness problem.

5. CONCLUSIONS:

In this paper, we propose a flexible framework for unseen class categorization with limited information provided about these classes. We secure two key factors, a powerful feature extractor and a flexible classifier, through network reparameterization. We decouple the feature extraction module and the classification module of a deep model for UCC. The feature extraction module is learned in a standard multi-class classification framework and the classification weight vector is generated by a network from exemplar information of the unseen classes. We train the classification weight generator in an episode-by-episode fashion to enable it flexibility for new tasks. Applying our framework for zero-shot learning (ZSL), we achieve much better results especially for the generalized ZSL setting than the state-of-the-art owing to our incorporation of inter-class separation information for learning the mapping from semantic space to visual space. For few-shot learning (FSL), we also achieve remarkable performance gains relative to existing methods due to the flexible scheme that make it possible a powerful feature extraction model and a flexible weight generation model.

:



Figures and Tables:

Figure 1: Frameworks for zero-shot (Left) and few-shot (Right) classification.few-shot tasks using the small-scale support set provided. The common limitation of the above methods is that they adopt the episode-based training scheme to secure adaptability to new classes, which however compromises the capability of discriminative feature extraction due to the forgetting of global (dataset-wise) competing information of among classes beyond individual episodes. Perhaps closest to our approach,Gidaris & Komodakis (2018) proposed the DFSVL algorithm which approaches FSL also in virtue of classification weight generation. The major limitation of DFSVL is that it obtains classification weights for unseen classes simply as a mixture of feature embeddings of support images of novel classes and attended pretrained weights of base (seen) classes, which is too weak to bridge feature embeddings and classification weights. Besides, it cannot bridge information across different domains (due to dimension inconsistency) so that is not applicable for ZSL. We instead learn a network to generate classification weights directly from feature embeddings of support images; it is more powerful and flexible to solve both ZSL and FSL within the same framework.

Figure 2: One-shot classification accuracy on two datasets.

Figure 3: The Barnes-Hut t-SNE (Van Der Maaten (2014)) of the projections of semantic vectors in the visual space obtained by DEM (Zhang et al. (2017)) (Left) and our method (Right). Seen and unseen classes are in orange and blue, respectively.

In other words, we decouple the feature extraction network f θ and the classification weight W of a standard classification network. We train f θ as a standard multiclass classification task and learn another network g φ to generate the classification weight W. Since f θ is trained as a standard multi-class classification task to distinguish all classes within the training set, it is supposed to be able to generate more discriminative feature representations for images of unseen classes than that generated by a model trained in episode-based fashion where the model is train to distinguish several classes within mini-batches. Meanwhile, we train g φ in episode-based fashion by constantly sampling new classes and minimizing the classification loss (cross entropy loss on top of Softmax outputs) using the weights generated by g φ . After training, whenever some new classes come, along with supporting information in the form of either attribute vectors (ZLS)

Table content: <table/>

Algorithm 1. Network reparam. for ZSL Input: Training set Dt = {Xt, Yt}, attribute set of training classes At.

Table content: <table><tr><td/><td>Algorithm 2. Network reparam. for FSL</td></tr><tr><td/><td>Input: Training set Dt = {Xt, Yt}</td></tr><tr><td/><td>Output: Feature extraction network f θ and</td></tr><tr><td>Output: Feature extraction network f θ and classification weight generation</td><td>classification weight generation network g φ .</td></tr><tr><td>network g φ .</td><td>1. Train f θ as a standard multi-class</td></tr><tr><td>1. Train f θ as a standard multi-class classification task while not done do 2. Randomly sample from Dt and At a ZSL task T = {Bv, Ba}, where Bv = {{xi,j} Nz i=1 , yj} Mz j=1 and Ba = {aj} Mz j=1</td><td>classification task while not done do 2. Randomly sample from Dt a FSL task T = {Bs, Bq}, where Bs = {{xi,j} N f M f i=1 , yj} j=1</td></tr><tr><td>3. Calculate loss according to Eq. 5</td><td/></tr><tr><td>4. Update g φ through back-propagation.</td><td/></tr><tr><td>end while</td><td/></tr></table>

Information of zero-shot classification datasets. much higher accuracy. (Note that for experiments on Mini-Imagenet, we train ResNet18 not on the whole ImageNet but on the subset excluding the classes appearing on Mini-Imagenet.)

Table content: <table><tr><td/><td/><td colspan="5">AwA1 AwA2 CUB aPY SUN</td></tr><tr><td>#Class</td><td>#Seen #Unseen</td><td>40 10</td><td>40 10</td><td>150 50</td><td>20 12</td><td>645 72</td></tr><tr><td colspan="2"># VisDim</td><td>2048</td><td>2048</td><td colspan="3">2048 2048 2048</td></tr><tr><td colspan="2"># AttDim</td><td>85</td><td>85</td><td>312</td><td>64</td><td>102</td></tr><tr><td/><td/><td/><td/><td/><td>1</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>PROTO NET</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>ResNet18 + NN</td></tr><tr><td/><td/><td/><td/><td/><td>0.8</td><td/><td>ResNet18* + NN</td></tr><tr><td/><td/><td/><td/><td/><td>0.6</td><td/></tr><tr><td/><td/><td/><td/><td/><td>0.4</td><td/></tr><tr><td/><td/><td/><td/><td/><td>0.2</td><td/></tr><tr><td/><td/><td/><td/><td/><td>0</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>Mini-Imagenet</td><td>CUB</td></tr></table>

Few-shot classification accuracy. The best results are in bold.

Table content: <table/>

Peer review (iclr2019):
 The paper addresses both zero-shot and few-shot learning in a unified framework. The key idea is to separate the deep network model into two networks. The first, aimed at learning a feature representation, is trained using standard multi-class but using cosine instead of dot product . The second, is trained using episode-based meta learning. The paper suggests that by training the features in a standard way, the representation is more rich and provides better space for learning the ZS and FS classes. The idea is very reasonable, and the experimental evaluations show large improvement over existing baselines. My main concern is that evaluating ZSL and FSL, is often tricky, and one should be careful about the details such that no information is leaked to the unseen classes. The paper is written in a way that makes it hard to evaluate if all the experiments were done in a proper way. Some of gains claimed, like the accuracy improvement for GZSL are very large, reporting by +70% or even +150%(aPY). Sometimes this is a sign that the evaluations are not done in the same way as the baselines, or other issues. More careful analysis is expected to convince that the gains are due to the proposed method. I would have liked to see (1) more detailed analysis of why the approach works (2) Ablation experiments and more convincing evidence that the improvement is indeed due to the way the representation is trained. (3) Evidence that the feature representation is better. (4) more details about the evaluation procedure. For example these can include (1) repeat the experiments with the same setup, but with a feature representation learned in an episodic way. (2) Evaluate the quality of the representation on another task (3) For the GZSL experiments, report separately the accuracy on seen and unseen classes. (4) qualitative analysis linking wins and losses to changes in the representation. (5) tuning the size of episodic training to support the claim that small episodes lead to worse representation. Other comments: Some more recent baselines are not included, e.g. better performance on CUB (57.8%) was reported by Atzmon et al, Probabilitis AND-OR Attribute Grouping for Zero-Shot Learning, UAI 2018. Other improvement by Zhang and Koniusz. Zero-shot kernel learning. CVPR, 2018. Minor comments : -- clarify which classes were used in train, validation, test -- Eq 6: L is defined as a function of phi. not used.