Year: 2018

Abstract:
We propose a model that learns to perform zero-shot classification using a metalearner that is trained to produce a correction to the output of a previously trained learner. The model consists of two modules: a task module that supplies an initial prediction, and a correction module that updates the initial prediction. The task module is the learner and the correction module is the meta-learner. The correction module is trained in an episodic approach whereby many different task modules are trained on various subsets of the total training data, with the rest being used as unseen data for the correction module. The correction module takes as input a representation of the task module’s training data so that the predicted correction is a function of the task module’s training data. The correction module is trained to update the task module’s prediction to be closer to the target value. This approach leads to state-of-the-art performance for zero-shot classification on natural language class descriptions on the CUB and NAB datasets.

heading: 1 INTRODUCTION
text: The ability to solve a task without receiving training examples – zero-shot learning – is desirable. This ability is particularly valuable when training data is limited, difficult to obtain, and/or expensive. We as humans can learn new tasks from descriptions of the tasks, as we learn from reading encyclopedia entries, manuals, handbooks, textbooks, etc. Our artificial agents should be able to do the same. However, generalizing to novel tasks unseen during training is challenging. The agent must integrate its previous training experience to solve the new task, without individual training samples from the new task, while avoiding over-fitting on its previous training experience.
We propose a model that learns a correction on predictions in the zero-shot setting, based on the training data set used to generate the initial prediction. Hence, our model is called Correction Networks. The meta-learner corrects for the zero-shot predictions of the learner.
The intuition for our model is that a zero-shot query sample that is different from samples in the training data will require a different correction than a zero-shot query sample that is similar to samples in the training data. Consider a simple classification model trained to identify different color classes. The training dataset for this simple classification problem consists of the classes of {yellow, green, indigo}. Let there be two zero-shot classes: pink and blue. The deviation of predicted values from true values of pink are different, possibly larger, than deviations of predicted values from true values of blue. These deviations would again be different if the training dataset consisted of the classes of {white, red, purple}. Correction Networks update the predictions based on the training data. This updated prediction is trained to be closer to the target value than the original prediction. Correction Networks consist of two modules: a task module that supplies an initial prediction, and a correction module that provides a correction to the initial prediction. The task module is the learner and the correction module is the meta-learner. The final prediction is the task module’s initial prediction combined with the correction module’s correction. This method is illustrated in Figure 1.
The prediction of the meta-learner is used to modify the output of the learner, unlike other metalearning algorithms that learn to update, initialize, or optimize the learner’s weights (Finn et al., 2017) (Ravi & Larochelle, 2017). Our approach is for zero-shot learning while previous metalearning approaches focus on few shot learning and require a few training samples for the new task.
Another novelty is that the correction module (the meta-learner) takes as input the dataset used to train the task module.
Correction Networks can be used for problems for which an updated prediction can be obtained by applying a correction. Correction Networks are independent of the representation of the task module. Existing models that provide predictions can be treated as task modules. The correction module only need the inputs to the task module, the predictions of the task module, and the training data of the task module. While our focus is on neural network architectures, both the task module and correction modules can be, but needs not be, neural networks.
Training proceeds by randomly partitioning the training data D to create disjoint sets DS and DU . In the case of zero-shot classification, the partition is done by class so that the classes inDS andDU are disjoint. The task module is trained on DS . After the task module is trained, DU is used as the ‘zero-shot’ queries for the task module. The task module supplies initial predictions on DU . The correction module is trained to predict the corrections that need to be applied to the task module’s initial predictions to obtain the true values associated with DU . One novelty is that the correction module, the meta-learner, also takes as input the dataset used to train the task module. The prediction of the meta-learner is used to modify the output of the learner, unlike other meta-learning algorithms that learn to update or initialize the learner’s weights (Finn et al., 2017) (Ravi & Larochelle, 2017).
We demonstrate Correction Networks on the problem of zero-shot image classification based on natural language class descriptions. Zero-shot image classification involves classifying images into classes that were unseen in the training data, given a natural language description of the new class.
The main contribution of this paper is a zero-shot learning model that corrects zero-shot predictions based on training data. We demonstrate our model on zero-shot image classification with class text descriptions. Our evaluation shows that Correction Networks compares favorably to state-of-the-art zero-shot learning methods.
The rest of this paper is as follows: Section 2 covers related work, Section 3 describes our Correction Network model, Section 4 summarizes the experiments and results, and Section 5 finishes with conclusions.

heading: 2 RELATED WORKS
text: For zero-shot learning for images, the majority of state-of-the-art methods are embedding-based methods. This often involves learning a mapping from the visual space to the semantic space of class labels or vice versa. Alternatively, the embedding function between the visual and semantic spaces is jointly learned through a latent space.
To represent the new, novel classes, hand crafted attributes are often used. The idea is that given a defined attribute ontology, each class name can be converted to an attribute vector against which image features are compared (Lampert et al., 2009) (Palatucci et al., 2009) (Farhadi et al., 2009) (Akata et al., 2013). Attribute based embedding spaces more recently gave way to text based embeddings. These text-based embeddings learn from a large external text corpus e.g. Wikipedia or WordNet on a natural language task to learn word embeddings into which class names are then projected (Oquab et al., 2014) (Socher et al., 2013) (Frome et al., 2013) (Fu et al., 2014).
Semantic representations for zero-shot classes have also been created from text documents of the classes eg. Wikipedia articles corresponding to the class. This includes learning attribute representations from a web source by ranking the visualness of attribute candidates (Berg et al., 2010). Internet sources have also been mined for semantic relatedness of attribute-class association (Rohrbach et al., 2013). Directly measuring the compatibility between documents describing the class and visual features have also been modeled using deep learning (Zhu et al., 2018) (Ba et al., 2015) and non-deep learning models (Elhoseiny et al., 2013) (Romera-Paredes & Torr, 2015) (Fu et al., 2015) (Qiao et al., 2016) (Elhoseiny et al., 2017a).
A more recent strategy frames zero-shot recognition as a conventional supervised classification problem by hallucinating samples for unseen classes. Classification performance thus depends on the quality of the hallucinated samples. Guo et al. (2017a) assumes the visual features of each class is distributed as a Gaussian and estimates the distribution of unseen classes by linearly combining the distributions of seen classes. Long et al. (2017) synthesizes visual data from attributes of classes in a one-to-one mapping. Guo et al. (2017b) assigns pseudo labels to samples from seen classes. Zhu et al. (2018) uses generative adversarial training with the generator learning to generate image features from text and then query samples are classified based on k-nearest neighbors to labeled or generated samples. In contrast, our approach avoids the data generation phase and instead, directly predicts the center of classes in image feature space. Classification regions are thus defined based on the class of the nearest class center.
Our work is similar to gradient boosting which produces a prediction model using an ensemble of weak models, where each additional model adds an estimator to provide a better model. Unlike gradient boosting, our approach takes in the training data used to create the initial estimate so that the model learns how the initial estimate deviates due to the training data. While the initial learner in gradient boosting is a weak learner, we use a strong learner.

heading: 3 CORRECTION NETWORKS

heading: 3.1 NOTATION
text: Let DS denote our training data and DU our testing data. DS is subdivided into disjoint sets DsS and DuS . For classification, the classes in DsS are disjoint from the classes in DuS .

heading: 3.2 MODEL
text: Correction Networks M consists of two modules: a task module MT and a correction module MC . The model components and their inputs and outputs are illustrated in Figure 2.
The task module MT is so-called because it is task specific and related to the application. The task module is trained on DsS . The output of MT is an estimate µ̂ of a target µ. The predictions of the task module on its training data DsS is µ̂sS . Training the task module proceeds by minimizing the distance between µ̂sS and the ground truth µ s S . A sparsity regularization term is added to the input layer of MT . The loss is:
LMT = E[d(MT (T sS), µsS)] + α||wMT,input ||2 (1)
where T is the class text description, µ is the empirical mean of samples that belong to the class, and d is a distance function. We use the L2 norm as the distance function.
The task module MT is not trained on DuS nor DU . The task module’s predictions on DuS are µ̂uS . Likewise, the task module’s predictions on DU are µ̂U . The correction module MC computes a correction ̂uS that is applied to the prediction µ̂ u S of the task module MT , where ̂ u S is calculated based on the data used to train MT , such that the corrected prediction (µ̂uS + ̂ u S) is closer than µ̂ u S to the ground truth µuS . Training the correction module proceeds by minimizing the distance between µuS and (µ̂ u S + ̂ u S). We use the L2 norm. The training data for the task module D s S is input into the correction module by representing the training data DsS as an un-ordered collection of data by using a pooling function. We use linear layers followed by sum pooling. The objective function of the correction module is to minimize:
LMC = E[d(MC(TuS ), µuS −MT (TuS ))] (2) We adopt the meta-learning sampling strategy for training as in Snell et al. (2017). Training data for Correction Networks is formed by randomly selecting a subsetDsS from the training dataDS . Then, the task module MT , is trained on DsS . The remaining tasks that the task module MT does not train on are treated as DuS for MT . The algorithm is detailed in Algorithm 1. To use Correction Networks for evaluation, the task module MT outputs µ̂U and the correction network supplies ̂U . The output of the Correction Networks is µ̄U = µ̂U + ̂U . The evaluation algorithm is outlined in Algorithm 2.
Correction Networks can be used for zero-shot problems for which an updated prediction can be obtained by applying a correction. Appropriate problems are those with continuous valued outputs. For classification, samples can be represented as continuous values in a feature space and then classified based on the nearest class center. In our experiments, both the base network and the Correction Network are deep neural networks.

heading: 3.3 CORRECTION NETWORKS FOR ZERO-SHOT CLASSIFICATION
text: In zero-shot classification, there are seen classes S available during training and unseen classes U during evaluation, where S and U are disjoint.
We are given data DS = {(xn, yn), tm}) containing samples {xn, n = 1, ..., N} ∈ X and class labels {yn, i = 1, ..., N, yn ∈ Y } such that yn ∈ S, along with class text descriptions {tm,m =
1, ...,M} ∈ TS where there exists exactly one tm for each class in S. For image classification, each sample xn is an image’s features extracted from a pre-trained model. There is no textual data per image. The only text data is tm with one text description per class.
At evaluation, we are given new class descriptions for classes in U , denoted by TU . We seek to learn a function f to minimize the 0-1 loss between the predicted f(xi) and true class labels yi for each sample xi where the samples and class labels can be from the new classes in U .
Correction Networks will map the class text descriptions TU to their corresponding class centers in image feature space µ̂U . Classification of a single image is done by assigning it the class label of the closest class center, measured by L2 distance. We subdivide the training classes S into Ss and Su. Then, DsS corresponds to the data from DS that belong to the classes in Ss. Similarly, DuS corresponds to the data from DS that belong to the classes in Su. Training and evaluation then proceed as described in the previous section. Correction Networks predict class cluster centers for zero-shot classes. Individual samples are classified to the nearest class cluster center using L2 distance. For classes in the training data, the class cluster center is the empirical mean of samples from the respective class.
In terms of implementation, both the task module and the correction module are feed-forward neural networks consisting of linear layers, activation functions, and dropout. We found the task module performance improves slightly when the output of the task module is fed into a classifier with a single hidden layer that is also trained to classify samples from the task model’s training dataset. Additional model and training details are in Appendix A.
Algorithm 1 Correction Networks training algorithm. K is the number of classes in the training set S. S is partitioned into Ss and Su by class. TS denotes the class descriptions of classes in S. DS is the set of individual samples of (xiyi) for the classes in the training set S. DsS denotes the subset of DS containing all samples from classes in Ss. Dk is the subset of DS containing the individual elements (xiyi) such that yi = k. MEAN is a function that returns the mean of its inputs. OPTIMIZER is an optimizer. d is a distance function, for example, L2 distance. MT is the task module. MC is the correction module.
1: for k in 1 to K do 2: µk ←MEAN(Dk) 3: end for 4: while not done do 5: Ss ← RANDOMSAMPLE({1, ...,K}, s) 6: Su ← {1, ...,K}\Ss 7: while MT has not finished training do 8: µ̂sS ←MT (T sS) 9: JMT ← d(µ̂sS , µsS) + α||wMT ||2
10: wMT ← OPTIMIZER(∇wMT JMT , wMT ) 11: end while 12: µ̂uS ←MT (TuS ) 13: ̂uS ←MC(TuS , µ̂uS , T sS) 14: uS ← µuS − µ̂uS 15: JMC ← d(̂uS , uS) 16: wMC ← OPTIMIZER(∇wMC JMC , wMC ) 17: end while
Algorithm 2 Correction Networks evaluation algorithm. S is the set of classes in the training set. U is the set of zero-shot classes. TU denotes the class descriptions of zero-shot classes in U . Require Trained MT on T sS , trained MC Output: Prediction µ̄U
1: µ̂U ←MT (TU ) 2: ̂←MC(TU , µ̂U , T sS) 3: µ̄U = µ̂U + ̂U

heading: 4 EXPERIMENTS

heading: 4.1 TASK SETUP
text: We demonstrate Correction Networks on fine-grained zero-shot classification based on natural language text descriptions of the class. Fine-grained image classification is image classification when classes are very similar. This requires the ability to distinguish between minute details and subtle differences between classes. An example of fine-grained classification is distinguishing between different birds that may be very similar.
Features of a class described in a text description may not be visible in all images belonging to that class. This loss of visibility can be due to image cropping, posture, camera angle, age, and gender of the bird, among other reasons. For example, an image shown from the front results in obfuscation of the back. An image may only be a head shot. Other objects such as branches or leaves may also block the object of interest. This is illustrated in Figure 5.

heading: 4.2 DATASETS
text: Image Data We evaluate our method on Caltech UCSD Birds 2011 (CUB) (Wah et al., 2011) and North America Birds (NAB) (Van Horn et al., 2015). The CUB dataset contains 200 classes of birds with a total of 11,788 images. We only use the images and their associated labels. We use a newer version of the NAB dataset that contains 404 classes with a total of 48,562 images. Two splits were proposed for each dataset (Elhoseiny et al., 2017b). The splits are named Super-Category-Shared (SCS) and Super-Category-Exclusive (SCE). In the SCS split, unseen classes and seen classes share the same parent category. In the SCE split, the parent categories are disjoint between seen and unseen classes. The SCE splits are more difficult than the SCS splits because the unseen classes are more different than the seen classes. Where published validation sets are not available, we create our own validation sets by holding out part of the training set, where the classes in the validation set are randomly sampled from among the training classes.
Visual Features The visual features are activations extracted from the FC layer of a pre-trained detector network (Elhoseiny et al., 2017b). We use the published features from (Zhu et al., 2018).
Text Features Each class is associated with a Wikipedia article. The articles are tokenized into words, stop words are removed, and words are reduced to their word stems. Then, the processed text is represented by TF-IDF features. We use the published text features from (Zhu et al., 2018). Examples of the text and images are shown in Figure 3.

heading: 4.3 CONVENTIONAL ZERO-SHOT RECOGNITION
text: The top-1 accuracy of our method and eight state-of-the-art algorithms for the CUB and NAB datasets for both the SCS split and the SCE split are tabulated in Table 1. The eight comparison models are MCZSL (Akata et al., 2016), ZSLNS (Qiao et al., 2016), SJE (Akata et al., 2015), WAC (Elhoseiny et al., 2017a), SynC (Changpinyo et al., 2016), ZSLPP (Elhoseiny et al., 2017b), and GAZSL (Zhu et al., 2018). The performance numbers are copied from (Zhu et al., 2018). MCZSL directly uses manual part annotations to extract visual representations. On the other hand, our approach, GAZSL, and ZSLPP merely use detected parts for both training and testing. Thus, methods that use detected parts are expected to perform poorer than MCZSL, which uses manual parts annotations. Our model performs favorably against the other models, showing a relative improvement of up to 10% over the previous state-of-the-art. Qualitative results are shown in Figure 4 in Appendix B.

heading: 4.4 GENERALIZED ZERO-SHOT LEARNING
text: The conventional zero-shot learning setting considers queries that come from unseen classesDU and classification of queries is restricted to be among the unseen classes U . In contrast, the generalized zero-shot learning setting classifies queries from both seen DS and unseen classes DU into S ∪ U . A metric for generalized zero-shot learning performance is the area under the seen-unseen curve (Chao et al., 2016). This is tabulated in Table 2 with values from competitors copied from Zhu et al. (2018). For Correction Networks, Correction Networks is used to predict µ̂ for the unseen classes
while the empirical µ is used for the seen classes. Correction Networks consistently outperforms previous approaches in the generalized zero-shot learning setting. This improvement is between 3-31% relative to the runner-up.

heading: 4.5 ABLATION STUDIES
text: To examine the contributions of separate components of our model, we conduct ablation studies by removing selected components. Then, the model is retrained and evaluated on the test set. The resulting performance of the ablated models are reported in Table 3.
Removing the correction module degrades the performance, but the task module itself already achieves the state of the art. The correction module improves the performance by 2% in absolute accuracy. This suggests that the correction modules makes a slight change to the output of the task module. When we remove the task module’s training data as input into the correction module, the zero-shot accuracy decreases by 2.4%. This demonstrates that the task module’s training data is important to the correction module’s prediction.

heading: 4.6 EXTENSIONS
text: Correction Networks are independent of the architectures of the modules and can be used for problems for which an updated prediction can be obtained by applying a correction. Additional investigations include applying Correction Networks to other tasks with outputs that can be updated, including for example, estimates for regression problems, classification probabilities, or probability distributions for reinforcement learning policies. The type of update can also be multiplicative, weighted sum, or another function. In addition to the zero-shot classification problem presented here, Correction Networks can also be applied to correct outputs in the few shot setting.
The magnitude of the correction vector predicted by the Correction Network can be interpreted as the size of the update applied. It would be interesting to use an interpretable output space where different dimensions correspond to human-understandable variables. Then, the size and direction of corrections in individual dimensions can be interpreted. The size of the correction can be compared against the number of training samples given to the base network, defined as the number of seen classes. One would expect that the size of the correction decreases as the number of seen classes increases. Also, one would expect that the accuracy stays constant or else improves with the number of seen classes. It would be interesting to see if correction networks can be used to understand bias in the training data.

heading: 5 CONCLUSION
text: We propose a zero-shot learning model that consists of a task module and a correction module that is trained to extrapolate from training data to unseen data. The training data is partitioned into a set of data used to train the task module and a disjoint set of data used to train the correction module. This later data is zero-shot with respect to the task module and the correction module is trained to update the task module’s zero-shot predictions to create a better prediction.
Our model is demonstrated on zero-shot fine-grained classification using only a single natural language description per zero-shot class. Our model performs favorably against the state-of-the-art on zero-shot classification. This framework is flexible to different representations of the task and correction modules and can be extended to other types of model predictions that can be updated to improve predictions.

heading: APPENDIX A: EXPERIMENTAL DETAILS
text: Architecture
The task module is a feed-forward network consisting of linear layers. The input is the TF-IDF feature for a class. There are two outputs: the predicted mean in image embedding space and softmaxes over the classes in TS . The loss function is the sum of the L2 loss between µ̂uS and µ u S and the classification loss, with a sparsity regualarizer on the input layer.
The correction module is also a feed-forward network consisting of linear layers. Each sample in T sS is fed through linear layers, then sum pooled across classes, and fed through an additional layer. The final transformation of T sS is concatenated with T u S and µ̂ u S and then passed through linear layers to output ̂uS . The loss function is L2 loss between ̂ u S and u S . Architectures are illustrated in Figure 4.
Training
The publicly available SCS and SCE splits (Elhoseiny et al., 2017b) define a test set. The remaining classes are randomly divided into a validation set and a training set. The size of the validation set is the same size as the training set. For example, CUB has 200 classes with the conventional SCS split consisting of 50 test classes. We divided the remaining 150 classes into a 50 class validation set and a 100 class training set.
The Correction Network is first trained by considering the training set as TS . Training proceeds as in Algorithm 1. That is, TS is randomly divided into disjoint sets T sS and T u S . A task module is trained on T sS , stopping based on performance on the validation set. The trained task module predicts µ̂ u S based on TuS . The correction module is updated using T u S , µ̂ u S , and T s S . The correction module is trained across multiple episodes of task module training, where the task module in each episode is trained on a randomly sampled T sS from TS . Training of different episodes of task modules is parallelizable as they are independent of one another and independent of the correction module.
Training of the correction module stops based on performance on the validation set. The stopping iterations of the task modules and correction module on the validation set is used to determine the stopping iterations of the task modules and the correction module on the test set.
To evaluate on the test set, the validation and training sets are combined and treated as TS while the test set becomes TU . Training again proceeds as in Algorithm 1. The Adam optimizer is used to train the models.

heading: APPENDIX B: RESULTS VISUALIZATIONS
text: Correctly and incorrectly classified images are shown in Figure 4 below. Each row is a different class. The left-most three images are correctly classified. The middle three images are false negatives, and the right-most images are false positives.

Review summaries:
This paper proposes a “meta-learning” approach for zero-shot learning. There is a Task Module that works in a conventional zero-shot way: Training to predict a class prototype using the auxiliary/text data description of that task. The new part is the added Correction Module that inputs both the target/zero-shot task description, the training task description, and the current prediction of the task module, and then outputs a correction vector that is added to the output of the task-module to produce the final output. The resulting system achieves state of the art results on zero-shot fine-grained classification (CUB and NAB).