Correction Networks: Meta-Learning for Zero-Shot Learning

1 INTRODUCTION . The ability to solve a task without receiving training examples – zero-shot learning – is desirable . This ability is particularly valuable when training data is limited , difficult to obtain , and/or expensive . We as humans can learn new tasks from descriptions of the tasks , as we learn from reading encyclopedia entries , manuals , handbooks , textbooks , etc . Our artificial agents should be able to do the same . However , generalizing to novel tasks unseen during training is challenging . The agent must integrate its previous training experience to solve the new task , without individual training samples from the new task , while avoiding over-fitting on its previous training experience . We propose a model that learns a correction on predictions in the zero-shot setting , based on the training data set used to generate the initial prediction . Hence , our model is called Correction Networks . The meta-learner corrects for the zero-shot predictions of the learner . The intuition for our model is that a zero-shot query sample that is different from samples in the training data will require a different correction than a zero-shot query sample that is similar to samples in the training data . Consider a simple classification model trained to identify different color classes . The training dataset for this simple classification problem consists of the classes of { yellow , green , indigo } . Let there be two zero-shot classes : pink and blue . The deviation of predicted values from true values of pink are different , possibly larger , than deviations of predicted values from true values of blue . These deviations would again be different if the training dataset consisted of the classes of { white , red , purple } . Correction Networks update the predictions based on the training data . This updated prediction is trained to be closer to the target value than the original prediction . Correction Networks consist of two modules : a task module that supplies an initial prediction , and a correction module that provides a correction to the initial prediction . The task module is the learner and the correction module is the meta-learner . The final prediction is the task module ’ s initial prediction combined with the correction module ’ s correction . This method is illustrated in Figure 1 . The prediction of the meta-learner is used to modify the output of the learner , unlike other metalearning algorithms that learn to update , initialize , or optimize the learner ’ s weights ( Finn et al. , 2017 ) ( Ravi & Larochelle , 2017 ) . Our approach is for zero-shot learning while previous metalearning approaches focus on few shot learning and require a few training samples for the new task . Another novelty is that the correction module ( the meta-learner ) takes as input the dataset used to train the task module . Correction Networks can be used for problems for which an updated prediction can be obtained by applying a correction . Correction Networks are independent of the representation of the task module . Existing models that provide predictions can be treated as task modules . The correction module only need the inputs to the task module , the predictions of the task module , and the training data of the task module . While our focus is on neural network architectures , both the task module and correction modules can be , but needs not be , neural networks . Training proceeds by randomly partitioning the training data D to create disjoint sets DS and DU . In the case of zero-shot classification , the partition is done by class so that the classes inDS andDU are disjoint . The task module is trained on DS . After the task module is trained , DU is used as the ‘ zero-shot ’ queries for the task module . The task module supplies initial predictions on DU . The correction module is trained to predict the corrections that need to be applied to the task module ’ s initial predictions to obtain the true values associated with DU . One novelty is that the correction module , the meta-learner , also takes as input the dataset used to train the task module . The prediction of the meta-learner is used to modify the output of the learner , unlike other meta-learning algorithms that learn to update or initialize the learner ’ s weights ( Finn et al. , 2017 ) ( Ravi & Larochelle , 2017 ) . We demonstrate Correction Networks on the problem of zero-shot image classification based on natural language class descriptions . Zero-shot image classification involves classifying images into classes that were unseen in the training data , given a natural language description of the new class . The main contribution of this paper is a zero-shot learning model that corrects zero-shot predictions based on training data . We demonstrate our model on zero-shot image classification with class text descriptions . Our evaluation shows that Correction Networks compares favorably to state-of-the-art zero-shot learning methods . The rest of this paper is as follows : Section 2 covers related work , Section 3 describes our Correction Network model , Section 4 summarizes the experiments and results , and Section 5 finishes with conclusions . 2 RELATED WORKS . For zero-shot learning for images , the majority of state-of-the-art methods are embedding-based methods . This often involves learning a mapping from the visual space to the semantic space of class labels or vice versa . Alternatively , the embedding function between the visual and semantic spaces is jointly learned through a latent space . To represent the new , novel classes , hand crafted attributes are often used . The idea is that given a defined attribute ontology , each class name can be converted to an attribute vector against which image features are compared ( Lampert et al. , 2009 ) ( Palatucci et al. , 2009 ) ( Farhadi et al. , 2009 ) ( Akata et al. , 2013 ) . Attribute based embedding spaces more recently gave way to text based embeddings . These text-based embeddings learn from a large external text corpus e.g . Wikipedia or WordNet on a natural language task to learn word embeddings into which class names are then projected ( Oquab et al. , 2014 ) ( Socher et al. , 2013 ) ( Frome et al. , 2013 ) ( Fu et al. , 2014 ) . Semantic representations for zero-shot classes have also been created from text documents of the classes eg . Wikipedia articles corresponding to the class . This includes learning attribute representations from a web source by ranking the visualness of attribute candidates ( Berg et al. , 2010 ) . Internet sources have also been mined for semantic relatedness of attribute-class association ( Rohrbach et al. , 2013 ) . Directly measuring the compatibility between documents describing the class and visual features have also been modeled using deep learning ( Zhu et al. , 2018 ) ( Ba et al. , 2015 ) and non-deep learning models ( Elhoseiny et al. , 2013 ) ( Romera-Paredes & Torr , 2015 ) ( Fu et al. , 2015 ) ( Qiao et al. , 2016 ) ( Elhoseiny et al. , 2017a ) . A more recent strategy frames zero-shot recognition as a conventional supervised classification problem by hallucinating samples for unseen classes . Classification performance thus depends on the quality of the hallucinated samples . Guo et al . ( 2017a ) assumes the visual features of each class is distributed as a Gaussian and estimates the distribution of unseen classes by linearly combining the distributions of seen classes . Long et al . ( 2017 ) synthesizes visual data from attributes of classes in a one-to-one mapping . Guo et al . ( 2017b ) assigns pseudo labels to samples from seen classes . Zhu et al . ( 2018 ) uses generative adversarial training with the generator learning to generate image features from text and then query samples are classified based on k-nearest neighbors to labeled or generated samples . In contrast , our approach avoids the data generation phase and instead , directly predicts the center of classes in image feature space . Classification regions are thus defined based on the class of the nearest class center . Our work is similar to gradient boosting which produces a prediction model using an ensemble of weak models , where each additional model adds an estimator to provide a better model . Unlike gradient boosting , our approach takes in the training data used to create the initial estimate so that the model learns how the initial estimate deviates due to the training data . While the initial learner in gradient boosting is a weak learner , we use a strong learner . 3 CORRECTION NETWORKS . 3.1 NOTATION . Let DS denote our training data and DU our testing data . DS is subdivided into disjoint sets DsS and DuS . For classification , the classes in DsS are disjoint from the classes in DuS . 3.2 MODEL . Correction Networks M consists of two modules : a task module MT and a correction module MC . The model components and their inputs and outputs are illustrated in Figure 2 . The task module MT is so-called because it is task specific and related to the application . The task module is trained on DsS . The output of MT is an estimate µ̂ of a target µ . The predictions of the task module on its training data DsS is µ̂sS . Training the task module proceeds by minimizing the distance between µ̂sS and the ground truth µ s S . A sparsity regularization term is added to the input layer of MT . The loss is : LMT = E [ d ( MT ( T sS ) , µsS ) ] + α||wMT , input ||2 ( 1 ) where T is the class text description , µ is the empirical mean of samples that belong to the class , and d is a distance function . We use the L2 norm as the distance function . The task module MT is not trained on DuS nor DU . The task module ’ s predictions on DuS are µ̂uS . Likewise , the task module ’ s predictions on DU are µ̂U . The correction module MC computes a correction ̂uS that is applied to the prediction µ̂ u S of the task module MT , where ̂ u S is calculated based on the data used to train MT , such that the corrected prediction ( µ̂uS + ̂ u S ) is closer than µ̂ u S to the ground truth µuS . Training the correction module proceeds by minimizing the distance between µuS and ( µ̂ u S + ̂ u S ) . We use the L2 norm . The training data for the task module D s S is input into the correction module by representing the training data DsS as an un-ordered collection of data by using a pooling function . We use linear layers followed by sum pooling . The objective function of the correction module is to minimize : LMC = E [ d ( MC ( TuS ) , µuS −MT ( TuS ) ) ] ( 2 ) We adopt the meta-learning sampling strategy for training as in Snell et al . ( 2017 ) . Training data for Correction Networks is formed by randomly selecting a subsetDsS from the training dataDS . Then , the task module MT , is trained on DsS . The remaining tasks that the task module MT does not train on are treated as DuS for MT . The algorithm is detailed in Algorithm 1 . To use Correction Networks for evaluation , the task module MT outputs µ̂U and the correction network supplies ̂U . The output of the Correction Networks is µ̄U = µ̂U + ̂U . The evaluation algorithm is outlined in Algorithm 2 . Correction Networks can be used for zero-shot problems for which an updated prediction can be obtained by applying a correction . Appropriate problems are those with continuous valued outputs . For classification , samples can be represented as continuous values in a feature space and then classified based on the nearest class center . In our experiments , both the base network and the Correction Network are deep neural networks .

Summary: This paper proposes a “meta-learning” approach for zero-shot learning. There is a Task Module that works in a conventional zero-shot way: Training to predict a class prototype using the auxiliary/text data description of that task. The new part is the added Correction Module that inputs both the target/zero-shot task description, the training task description, and the current prediction of the task module, and then outputs a correction vector that is added to the output of the task-module to produce the final output. The resulting system achieves state of the art results on zero-shot fine-grained classification (CUB and NAB).