Title: CORRECTION NETWORKS: META-LEARNING FOR ZERO-SHOT LEARNING

Abstract: We propose a model that learns to perform zero-shot classification using a metalearner that is trained to produce a correction to the output of a previously trained learner. The model consists of two modules: a task module that supplies an initial prediction, and a correction module that updates the initial prediction. The task module is the learner and the correction module is the meta-learner. The correction module is trained in an episodic approach whereby many different task modules are trained on various subsets of the total training data, with the rest being used as unseen data for the correction module. The correction module takes as input a representation of the task module's training data so that the predicted correction is a function of the task module's training data. The correction module is trained to update the task module's prediction to be closer to the target value. This approach leads to state-of-the-art performance for zero-shot classification on natural language class descriptions on the CUB and NAB datasets.

1. INTRODUCTION:

The ability to solve a task without receiving training examples -zero-shot learning -is desirable. This ability is particularly valuable when training data is limited, difficult to obtain, and/or expensive. We as humans can learn new tasks from descriptions of the tasks, as we learn from reading encyclopedia entries, manuals, handbooks, textbooks, etc. Our artificial agents should be able to do the same. However, generalizing to novel tasks unseen during training is challenging. The agent must integrate its previous training experience to solve the new task, without individual training samples from the new task, while avoiding over-fitting on its previous training experience.

We propose a model that learns a correction on predictions in the zero-shot setting, based on the training data set used to generate the initial prediction. Hence, our model is called Correction Networks. The meta-learner corrects for the zero-shot predictions of the learner.

The intuition for our model is that a zero-shot query sample that is different from samples in the training data will require a different correction than a zero-shot query sample that is similar to samples in the training data. Consider a simple classification model trained to identify different color classes. The training dataset for this simple classification problem consists of the classes of {yellow, green, indigo}. Let there be two zero-shot classes: pink and blue. The deviation of predicted values from true values of pink are different, possibly larger, than deviations of predicted values from true values of blue. These deviations would again be different if the training dataset consisted of the classes of {white, red, purple}.

Correction Networks update the predictions based on the training data. This updated prediction is trained to be closer to the target value than the original prediction. Correction Networks consist of two modules: a task module that supplies an initial prediction, and a correction module that provides a correction to the initial prediction. The task module is the learner and the correction module is the meta-learner. The final prediction is the task module's initial prediction combined with the correction module's correction. This method is illustrated in Figure 1 .

The prediction of the meta-learner is used to modify the output of the learner, unlike other metalearning algorithms that learn to update, initialize, or optimize the learner's weights (Finn et al., 2017) (Ravi & Larochelle, 2017) . Our approach is for zero-shot learning while previous metalearning approaches focus on few shot learning and require a few training samples for the new task.

Figure 1 : Correction Networks consist of two modules: a task module and a correction module. The task module produces an initial prediction. The correction module provides a correction such that when the correction is combined with the initial prediction, the resulting prediction is better than the initial prediction.

Another novelty is that the correction module (the meta-learner) takes as input the dataset used to train the task module.

Correction Networks can be used for problems for which an updated prediction can be obtained by applying a correction. Correction Networks are independent of the representation of the task module. Existing models that provide predictions can be treated as task modules. The correction module only need the inputs to the task module, the predictions of the task module, and the training data of the task module. While our focus is on neural network architectures, both the task module and correction modules can be, but needs not be, neural networks.

Training proceeds by randomly partitioning the training data D to create disjoint sets D S and D U . In the case of zero-shot classification, the partition is done by class so that the classes in D S and D U are disjoint. The task module is trained on D S . After the task module is trained, D U is used as the 'zero-shot' queries for the task module. The task module supplies initial predictions on D U . The correction module is trained to predict the corrections that need to be applied to the task module's initial predictions to obtain the true values associated with D U . One novelty is that the correction module, the meta-learner, also takes as input the dataset used to train the task module. The prediction of the meta-learner is used to modify the output of the learner, unlike other meta-learning algorithms that learn to update or initialize the learner's weights (Finn et al., 2017) (Ravi & Larochelle, 2017) .

We demonstrate Correction Networks on the problem of zero-shot image classification based on natural language class descriptions. Zero-shot image classification involves classifying images into classes that were unseen in the training data, given a natural language description of the new class.

The main contribution of this paper is a zero-shot learning model that corrects zero-shot predictions based on training data. We demonstrate our model on zero-shot image classification with class text descriptions. Our evaluation shows that Correction Networks compares favorably to state-of-the-art zero-shot learning methods.

The rest of this paper is as follows: Section 2 covers related work, Section 3 describes our Correction Network model, Section 4 summarizes the experiments and results, and Section 5 finishes with conclusions.

2. RELATED WORKS:

For zero-shot learning for images, the majority of state-of-the-art methods are embedding-based methods. This often involves learning a mapping from the visual space to the semantic space of class labels or vice versa. Alternatively, the embedding function between the visual and semantic spaces is jointly learned through a latent space.

To represent the new, novel classes, hand crafted attributes are often used. The idea is that given a defined attribute ontology, each class name can be converted to an attribute vector against which image features are compared (Lampert et al., 2009) (Palatucci et al., 2009) (Farhadi et al., 2009) (Akata et al., 2013) . Attribute based embedding spaces more recently gave way to text based embeddings. These text-based embeddings learn from a large external text corpus e.g. Wikipedia or WordNet on a natural language task to learn word embeddings into which class names are then projected (Oquab et al., 2014) (Socher et al., 2013) (Frome et al., 2013) (Fu et al., 2014) .

Semantic representations for zero-shot classes have also been created from text documents of the classes eg. Wikipedia articles corresponding to the class. This includes learning attribute representations from a web source by ranking the visualness of attribute candidates (Berg et al., 2010) . Internet sources have also been mined for semantic relatedness of attribute-class association (Rohrbach et al., 2013) . Directly measuring the compatibility between documents describing the class and visual features have also been modeled using deep learning (Zhu et al., 2018 ) (Ba et al., 2015) and non-deep learning models (Elhoseiny et al., 2013) (Romera-Paredes & Torr, 2015) (Fu et al., 2015 ) (Qiao et al., 2016) (Elhoseiny et al., 2017a) .

A more recent strategy frames zero-shot recognition as a conventional supervised classification problem by hallucinating samples for unseen classes. Classification performance thus depends on the quality of the hallucinated samples. Guo et al. (2017a) assumes the visual features of each class is distributed as a Gaussian and estimates the distribution of unseen classes by linearly combining the distributions of seen classes. Long et al. (2017) synthesizes visual data from attributes of classes in a one-to-one mapping. Guo et al. (2017b) assigns pseudo labels to samples from seen classes. Zhu et al. (2018) uses generative adversarial training with the generator learning to generate image features from text and then query samples are classified based on k-nearest neighbors to labeled or generated samples. In contrast, our approach avoids the data generation phase and instead, directly predicts the center of classes in image feature space. Classification regions are thus defined based on the class of the nearest class center.

Our work is similar to gradient boosting which produces a prediction model using an ensemble of weak models, where each additional model adds an estimator to provide a better model. Unlike gradient boosting, our approach takes in the training data used to create the initial estimate so that the model learns how the initial estimate deviates due to the training data. While the initial learner in gradient boosting is a weak learner, we use a strong learner. The task module M T is so-called because it is task specific and related to the application. The task module is trained on D s S . The output of M T is an estimate μ of a target µ. The predictions of the task module on its training data D s S is μs S . Training the task module proceeds by minimizing the distance between μs S and the ground truth µ s S . A sparsity regularization term is added to the input layer of M T . The loss is: where T is the class text description, µ is the empirical mean of samples that belong to the class, and d is a distance function. We use the L2 norm as the distance function.

3. CORRECTION NETWORKS:

L M T = E[d(M T (T s S ), µ s S )] + α||w M T ,input || 2 (1)

The task module M T is not trained on D u S nor D U . The task module's predictions on D u S are μu S . Likewise, the task module's predictions on D U are μU . The correction module M C computes a correction ˆ u S that is applied to the prediction μu S of the task module M T , where ˆ u S is calculated based on the data used to train M T , such that the corrected prediction (μ u S + ˆ u S ) is closer than μu S to the ground truth µ u S . Training the correction module proceeds by minimizing the distance between µ u S and (μ u S + ˆ u S ). We use the L2 norm. The training data for the task module D s S is input into the correction module by representing the training data D s S as an un-ordered collection of data by using a pooling function. We use linear layers followed by sum pooling. The objective function of the correction module is to minimize:

L M C = E[d(M C (T u S ), µ u S -M T (T u S ))]

(2) We adopt the meta-learning sampling strategy for training as in Snell et al. (2017) . Training data for Correction Networks is formed by randomly selecting a subset D s S from the training data D S . Then, the task module M T , is trained on D s S . The remaining tasks that the task module M T does not train on are treated as D u S for M T . The algorithm is detailed in Algorithm 1. To use Correction Networks for evaluation, the task module M T outputs μU and the correction network supplies ˆ U . The output of the Correction Networks is μU = μU + ˆ U . The evaluation algorithm is outlined in Algorithm 2.

Correction Networks can be used for zero-shot problems for which an updated prediction can be obtained by applying a correction. Appropriate problems are those with continuous valued outputs. For classification, samples can be represented as continuous values in a feature space and then classified based on the nearest class center. In our experiments, both the base network and the Correction Network are deep neural networks.

3.3. CORRECTION NETWORKS FOR ZERO-SHOT CLASSIFICATION:

In zero-shot classification, there are seen classes S available during training and unseen classes U during evaluation, where S and U are disjoint.

We are given data D S = {(x n , y n ), t m }) containing samples {x n , n = 1, ..., N } ∈ X and class labels {y n , i = 1, ..., N, y n ∈ Y } such that y n ∈ S, along with class text descriptions {t m , m = 1, ..., M } ∈ T S where there exists exactly one t m for each class in S. For image classification, each sample x n is an image's features extracted from a pre-trained model. There is no textual data per image. The only text data is t m with one text description per class.

At evaluation, we are given new class descriptions for classes in U , denoted by T U . We seek to learn a function f to minimize the 0-1 loss between the predicted f (x i ) and true class labels y i for each sample x i where the samples and class labels can be from the new classes in U .

Correction Networks will map the class text descriptions T U to their corresponding class centers in image feature space μU . Classification of a single image is done by assigning it the class label of the closest class center, measured by L2 distance. We subdivide the training classes S into S s and S u . Then, D s S corresponds to the data from D S that belong to the classes in S s . Similarly, D u S corresponds to the data from D S that belong to the classes in S u . Training and evaluation then proceed as described in the previous section. Correction Networks predict class cluster centers for zero-shot classes. Individual samples are classified to the nearest class cluster center using L2 distance. For classes in the training data, the class cluster center is the empirical mean of samples from the respective class.

In terms of implementation, both the task module and the correction module are feed-forward neural networks consisting of linear layers, activation functions, and dropout. We found the task module performance improves slightly when the output of the task module is fed into a classifier with a single hidden layer that is also trained to classify samples from the task model's training dataset. Additional model and training details are in Appendix A.

Algorithm 1 Correction Networks training algorithm. K is the number of classes in the training set S. S is partitioned into S s and S u by class. T S denotes the class descriptions of classes in S. D S is the set of individual samples of (x i y i ) for the classes in the training set S. D s S denotes the subset of D S containing all samples from classes in S s . D k is the subset of D S containing the individual elements (x i y i ) such that y i = k. MEAN is a function that returns the mean of its inputs. OPTIMIZER is an optimizer. d is a distance function, for example, L2 distance. M T is the task module. M C is the correction module. 

μs:

S ← M T (T s S )

9:

J M T ← d(μ s S , µ s S ) + α||w M T || 2 10: w M T ← OPTIMIZER(∇ w M T J M T , w M T ) 11:

end while 12:

μu S ← M T (T u S )

13:

ˆ u S ← M C (T u S , μu S , T s S ) 14: u S ← µ u S -μu S 15: J M C ← d(ˆ u S , u S )

16: 

w M C ← OPTIMIZER(∇ w M C J M C , w M C ) 17:

4. EXPERIMENTS:



4.1. TASK SETUP:

We demonstrate Correction Networks on fine-grained zero-shot classification based on natural language text descriptions of the class. Fine-grained image classification is image classification when classes are very similar. This requires the ability to distinguish between minute details and subtle differences between classes. An example of fine-grained classification is distinguishing between different birds that may be very similar.

Features of a class described in a text description may not be visible in all images belonging to that class. This loss of visibility can be due to image cropping, posture, camera angle, age, and gender of the bird, among other reasons. For example, an image shown from the front results in obfuscation of the back. An image may only be a head shot. Other objects such as branches or leaves may also block the object of interest. This is illustrated in Figure 5 .

4.2. DATASETS:

Image Data We evaluate our method on Caltech UCSD Birds 2011 (CUB) (Wah et al., 2011) and North America Birds (NAB) (Van Horn et al., 2015) . The CUB dataset contains 200 classes of birds with a total of 11,788 images. We only use the images and their associated labels. We use a newer version of the NAB dataset that contains 404 classes with a total of 48,562 images. Two splits were proposed for each dataset (Elhoseiny et al., 2017b) . The splits are named Super-Category-Shared (SCS) and Super-Category-Exclusive (SCE). In the SCS split, unseen classes and seen classes share the same parent category. In the SCE split, the parent categories are disjoint between seen and unseen classes. The SCE splits are more difficult than the SCS splits because the unseen classes are more different than the seen classes. Where published validation sets are not available, we create our own validation sets by holding out part of the training set, where the classes in the validation set are randomly sampled from among the training classes.

Visual Features:

The visual features are activations extracted from the FC layer of a pre-trained detector network (Elhoseiny et al., 2017b) . We use the published features from (Zhu et al., 2018) .

Text Features Each class is associated with a Wikipedia article. The articles are tokenized into words, stop words are removed, and words are reduced to their word stems. Then, the processed text is represented by TF-IDF features. We use the published text features from (Zhu et al., 2018) . Examples of the text and images are shown in Figure 3 .

Table 1 : Zero-shot learning classification results accuracy @ 1 on the CUB-200-2011 dataset and the NAB dataset using class descriptions from Wikipedia on the Super-Category-Shared (SCS) and Super-Category-Exclusive (SCE) zero-shot splits METHOD CUB NAB SCS SCE SCS SCE MCZSL (Akata et al., 2016) 34.7 ---WAC-Linear (Elhoseiny et al., 2013) 27.0 5.0 --WAC-Kernel (Elhoseiny et al., 2017a) 33.5 7.7 11.4 6.0 ESZSL (Romera-Paredes & Torr, 2015) 28.5 7.4 24.3 6.3 SJE (Akata et al., 2015) 29.9 ---ZSLNS (Qiao et al., 2016) 29.1 7.3 24.5 6.8 SynC fast (Changpinyo et al., 2016) 28.0 8.6 18.4 3.8 SynC OVO (Changpinyo et al., 2016) 12.5 5.9 --ZSLPP (Elhoseiny et al., 2017b) 37.2 9.7 30.3 8.1 GAZSL (Zhu et al., 2018) 43 (Elhoseiny et al., 2013) 23.9 4.9 23.5 -WAC-Kernel (Elhoseiny et al., 2017a) 22.5 5.4 0.7 2.3 SynC Fast (Changpinyo et al., 2016) 13.1 4.0 2.7 0.8 ESZSL (Romera-Paredes & Torr, 2015) 18.5 4.5 9.2 2.9 ZSLNS (Qiao et al., 2016) 14.7 4.4 9.3 2.3 SynC OvO (Changpinyo et al., 2016) 1.7 1.0 0.1 -ZSLPP (Elhoseiny et al., 2017b) 30.4 6.1 12.6 3.5 GAZSL (Zhu et al., 2018) 35.4 8.7 20.4 5.8 CorrectionNet 41.9 9.0 25.4 7.6

4.3. CONVENTIONAL ZERO-SHOT RECOGNITION:

The top-1 accuracy of our method and eight state-of-the-art algorithms for the CUB and NAB datasets for both the SCS split and the SCE split are tabulated in Table 1 . The eight comparison models are MCZSL (Akata et al., 2016) , ZSLNS (Qiao et al., 2016) , SJE (Akata et al., 2015) , WAC (Elhoseiny et al., 2017a) , SynC (Changpinyo et al., 2016) , ZSLPP (Elhoseiny et al., 2017b) , and GAZSL (Zhu et al., 2018) . The performance numbers are copied from (Zhu et al., 2018) . MCZSL directly uses manual part annotations to extract visual representations. On the other hand, our approach, GAZSL, and ZSLPP merely use detected parts for both training and testing. Thus, methods that use detected parts are expected to perform poorer than MCZSL, which uses manual parts annotations. Our model performs favorably against the other models, showing a relative improvement of up to 10% over the previous state-of-the-art. Qualitative results are shown in Figure 4 in Appendix B.

4.4. GENERALIZED ZERO-SHOT LEARNING:

The conventional zero-shot learning setting considers queries that come from unseen classes D U and classification of queries is restricted to be among the unseen classes U . In contrast, the generalized zero-shot learning setting classifies queries from both seen D S and unseen classes D U into S ∪ U .

A metric for generalized zero-shot learning performance is the area under the seen-unseen curve (Chao et al., 2016) . This is tabulated in Table 2 with values from competitors copied from Zhu et al. (2018) . For Correction Networks, Correction Networks is used to predict μ for the unseen classes while the empirical µ is used for the seen classes. Correction Networks consistently outperforms previous approaches in the generalized zero-shot learning setting. This improvement is between 3-31% relative to the runner-up.

4.5. ABLATION STUDIES:

To examine the contributions of separate components of our model, we conduct ablation studies by removing selected components. Then, the model is retrained and evaluated on the test set. The resulting performance of the ablated models are reported in Table 3 .

Removing the correction module degrades the performance, but the task module itself already achieves the state of the art. The correction module improves the performance by 2% in absolute accuracy. This suggests that the correction modules makes a slight change to the output of the task module. When we remove the task module's training data as input into the correction module, the zero-shot accuracy decreases by 2.4%. This demonstrates that the task module's training data is important to the correction module's prediction.

4.6. EXTENSIONS:

Correction Networks are independent of the architectures of the modules and can be used for problems for which an updated prediction can be obtained by applying a correction. Additional investigations include applying Correction Networks to other tasks with outputs that can be updated, including for example, estimates for regression problems, classification probabilities, or probability distributions for reinforcement learning policies. The type of update can also be multiplicative, weighted sum, or another function. In addition to the zero-shot classification problem presented here, Correction Networks can also be applied to correct outputs in the few shot setting.

The magnitude of the correction vector predicted by the Correction Network can be interpreted as the size of the update applied. It would be interesting to use an interpretable output space where different dimensions correspond to human-understandable variables. Then, the size and direction of corrections in individual dimensions can be interpreted. The size of the correction can be compared against the number of training samples given to the base network, defined as the number of seen classes. One would expect that the size of the correction decreases as the number of seen classes increases. Also, one would expect that the accuracy stays constant or else improves with the number of seen classes. It would be interesting to see if correction networks can be used to understand bias in the training data.

5. CONCLUSION:

We propose a zero-shot learning model that consists of a task module and a correction module that is trained to extrapolate from training data to unseen data. The training data is partitioned into a set of data used to train the task module and a disjoint set of data used to train the correction module. This later data is zero-shot with respect to the task module and the correction module is trained to update the task module's zero-shot predictions to create a better prediction.

Our model is demonstrated on zero-shot fine-grained classification using only a single natural language description per zero-shot class. Our model performs favorably against the state-of-the-art on zero-shot classification. This framework is flexible to different representations of the task and correction modules and can be extended to other types of model predictions that can be updated to improve predictions.

:



Figures and Tables:

3.1 NOTATION Let D S denote our training data and D U our testing data. D S is subdivided into disjoint sets D s S and D u S . For classification, the classes in D s S are disjoint from the classes in D u S . 3.2 MODEL Correction Networks M consists of two modules: a task module M T and a correction module M C . The model components and their inputs and outputs are illustrated in Figure 2.

Figure 2: Inputs and outputs of Correction Networks demonstrated on zero-shot image classification based on a natural language description of the zero-shot class. The task module takes as input the natural language description of the zero-shot class and makes an initial prediction of the cluster center of the class in image feature space. The correction module improves this initial prediction by applying a correction, taking as input the task module's initial prediction, training data, and the zero-shot class description.

for k in 1 to K do 2: µ k ← MEAN(D k ) 3: end for 4: while not done do 5: S s ← RANDOMSAMPLE({1, ..., K}, s) 6: S u ← {1, ..., K}\S s 7: while M T has not finished training do 8:

end while Algorithm 2 Correction Networks evaluation algorithm. S is the set of classes in the training set. U is the set of zero-shot classes. T U denotes the class descriptions of zero-shot classes in U . Require Trained M T on T s S , trained M C Output: Prediction μU 1: μU ← M T (T U ) 2: ˆ ← M C (T U , μU , T s S ) 3: μU = μU + ˆ U

Figure 3: Examples from the dataset where there is one natural language description of a class and the goal is to classify images for new classes without training sample images for the new classes.

Generalized Zero-shot learning classification area under Seen-Unseen Curve on CUB

Table content: <table><tr><td>.7 10.3 35.6</td><td>8.6</td></tr></table>

Effects of different components on zero-shot classification accuracy (%) on CUB SCS

Table content: <table><tr><td>METHOD</td><td>CUB</td></tr><tr><td>Task module only</td><td>43.8</td></tr><tr><td colspan="2">Correction module without task module's training dataset 43.4</td></tr><tr><td>Correction Networks</td><td>45.8</td></tr></table>

Peer review (iclr2019):
 === Post-rebuttal update === The authors* rebuttal provided many of the details I was seeking. I asked a few additional questions which were also recently addressed, and I encourage the authors to include these clarifications into the final draft of the paper. Hence, I*ve increased my score for this paper. === Pre-rebuttal review === This paper presents a meta-learning approach to zero-shot learning. The idea is to train a correction module which is trained to produce a correction to the output of a previously trained task module. The hypothesis is that the correction should depend on the nature of the training data of the task module, and so the correction module receives as input a representation of the training data of the task module. An episodic approach is then used for training the correction module, whereby many different task modules are trained on various subsets of the total training data, the rest being used as unseen data for the correction module. The proposed idea is original and the results are strong. Generally, I*d be inclined to see this paper published. However right now, the paper lacks A LOT of details on how the experiments were run. I would like to see these answered in the rebuttal, before I consider raising my rating for this paper: - What are the architectures used for M_T and M_C? - What distance functions was used for training? - What optimizer was used for training? - How was convergence established in the inner and outer while loops of algorithm 1? - Text mentions that before evaluation, M_T is trained on all data in D_S. How is this done exactly (e.g. how is convergence assessed)? - How is the T_S computed exactly? - How expensive is it to run Algorithm 1 (i.e. to train the correction module)? Since a new task module M_T needs to be trained for each subset S^s, it seems like it might be expensive to run... if not, why? I would also strongly suggest the authors release their code if this paper ends up being published. In summary: Pros - Claims SOTA results on two good benchmarks for zero-shot learning - Approach is original Cons - Paper lacks a lot of methodological and experimental details Some minor details: - "We found the task module performance improves slightly when the output of the task module is feed into a classifier with a single hidden layer that is also trained to classify samples from the task model’s training dataset." => I don*t understand what this means. Isn*t the output of the task module already trained to classify samples from its training dataset? So why is this additional single hidden layer needed? - Typos: - on few shot learn => on few shot learning - but needs not => but need not - image image classification => image classification - the the compatibility => the compatibility - psuedo => pseudo - "The task module is trained to minimize" => that reads like an unfinished sentence - .hat{.mu}_U .hat{.mu}_U => .hat{.mu}_U - inputted => input - FOr => For - it*s inputs => its inputs - otherhand => other hand