MahiNet: A Neural Network for Many-Class Few-Shot Learning with Class Hierarchy

1 INTRODUCTION . The representation power of deep neural networks ( DNN ) has dramatically improved in recent years , as deeper , wider and more complicated DNN architectures ( He et al. , 2016 ; Huang et al. , 2017 ) have emerged to match the increasing computation power of new hardwares . Although this brings hope for complex tasks that could be hardly solved by previous shallow models , more training data is usually required . Hence , the scarcity of annotated data has become a new bottleneck for training more powerful DNNs . For example , in image classification , the number of candidate classes can easily range from hundreds to tens of thousands ( i.e. , many-class ) , but the training samples available for each class can be less than 100 ( i.e. , few-shot ) . Additionally , in life-long learning , models are always updated once new training data becomes available , and those models are expected to quickly adapt to new classes with a few training samples . This “ many-class few-shot ” problem is very common in various applications , such as image search , robot navigation and video surveillance . Although enormous previous works have shown the remarkable power of DNN when “ many-class many-shot ” training data is available , their performance degrades dramatically when each class only has a few samples available for training . In practical applications , acquiring samples of rare species is usually difficult and often expensive . In these few-shot scenarios , the model ’ s capacity can not be fully utilized , and it becomes much harder to generalize the model to unseen data . Recently , several approaches have been proposed to address the few-shot learning problem . Most of them are based on the idea of “ meta-learning ” , which trains a meta-learner that can generalize to different tasks . For classification , each task targets a different set of classes . Meta-learning can be categorized into two types : methods based on “ learning to optimize ” , and methods based on metric learning . The former type adaptively modifies the optimizer ( or some parts of it ) applied to the training process . It includes methods that incorporate an RNN meta-learner ( Andrychowicz et al. , 2016 ; Li & Malik , 2017 ; Ravi & Larochelle , 2017 ) , and model-agnostic meta-learning ( MAML ) methods aiming to learn a generally compelling initialization ( Finn et al. , 2017 ) . The latter type learns a similarity/distance metric ( Vinyals et al. , 2016 ) or a support set of samples ( Snell et al. , 2017 ) that can be generally used to build KNN classifiers for different tasks . Instead of using meta-learning , some other approaches , such as Douze et al . ( 2018 ) , address the few-shot learning problem through data augmentation by generating artificial samples for each class . However , most existing few-shot learning approaches only focus on “ few-class ” case ( e.g. , 5 or 10 ) per task , and performance usually collapses when the number of classes grows to hundreds or thousands . This is because the samples per class no longer provide enough information to distinguish them from other possible samples within a large number of other classes . And , in real-world problems , tasks are usually complicated involving many classes . Fortunately , in practice , class hierarchy is usually available or cheaper to obtain . As shown in Figure 1 , coarse class labels might reveal the relationships among the targeted fine classes . Moreover , the samples per coarse class are sufficient to train a reliable coarse classifier , whose predictions are able to narrow down the candidates for fine classes . For example , a sheepdog with long hair could be easily mis-classified as mop when training samples of sheepdog are insufficient . However , if we could train a reliable dog classifier , it would be much simpler to predict an image as a sheepdog than a mop given a correct prediction of the coarse class as “ dog ” . Hence , class hierarchy might provide weakly supervised information to help solve the “ many-class few-shot ( MCFS ) ” problem . 1.1 OUR APPROACH . In this paper , we study how to explore the class hierarchy to solve MCFS problem in both traditional supervised learning and in meta-learning . We develop a DNN architecture “ memory-augmented hierarchical-classification networks ( MahiNet ) ” that can be applied to both learning scenarios . MahiNet uses a CNN , i.e. , ResNet by He et al . ( 2016 ) , as a backbone network to extract features from raw images . The CNN feeds features into coarse-class and fine-class classifiers , and the results are combined to produce the final prediction according to fine classes as probabilities . In this way , both the coarse-class and the fine-class classifiers mutually help each other within MahiNet : the former helps to narrow down the candidates for the latter , while the latter provides multiple attributes per coarse class that can regularize the former . This design leverages the relationship between fine classes , and mitigates the difficulty caused by “ many class ” problem . To the best of our knowledge , we are the first to successfully employ the class hierarchy information to improve few-shot learning . Previous works ( Ren et al. , 2018 ) can not achieve improvement after using the same information . To address the “ few-shot ” problem , we apply two types of classifiers in MahiNet , i.e. , MLP and Knearest neighbor ( KNN ) , which have advantages in many-shot and few-shot situations , respectively . We always use MLP for coarse classification , and KNN for fine classification . With a sufficient amount of data in supervised learning , MLP is combined with KNN for fine classification ; and in meta-learning when less data is available , we also use KNN for coarse classification to assist MLP . In Table 1 , we provide a brief comparison of MahiNet with other popular models on the learning scenarios they excel . To make the KNN learnable and more adaptive to classes with few-shot data , we use an attention module to learn the similarity/distance metric used in KNN , and a re-writable memory of limited size to store and update KNN support set during training . In supervised learning , it is necessary to maintain and update a relatively small memory ( 7.2 % of the dataset ) by selecting a few samples , because conducting a KNN search over all available training samples is too computationally expensive in computation . In meta-learning , the attention module can be treated as a meta-learner that learns a universal similarity metric for different tasks . We extract a large subset of ImageNet ( Deng et al. , 2009 ) “ mcfsImageNet ” as a benchmark dataset specifically designed for MCFS problem . It contains 139,346 images from 77 non-overlapping coarse classes composed of 754 randomly sampled fine classes , each has only ∼ 180 images . Imbalance between the different classes are preserved to reflect the imbalance in practical problems . We further extract “ mcfsOmniglot ” from Omniglot ( Lake et al. , 2011 ) for the same purpose . We will make them publicly available later . In experiments on these two datasets , MahiNet outperforms the widely used ResNet ( He et al. , 2016 ) in supervised learning . In meta-learning scenario where each task convers many classes , it shows more promising performance than popular few-shot methods including prototypical networks ( Snell et al. , 2017 ) and relation networks ( Yang et al. , 2018 ) . 2 MEMORY-AUGMENTED HIERARCHICAL-CLASSIFICATION NETWORK . 2.1 PROBLEM FORMULATION . We study supervised learning and meta-learning given a training set of n samples D = { ( xi , yi , zi ) } ni=1 , where each sample xi ∈ X is associated with a fine-class label yi ∈ Y and a coarse-class label zi ∈ Z , and is sampled from a data distribution D , i.e. , ( xi , yi , zi ) ∼ D. Here , Y denotes the set of all the fine classes , and Z denotes the set of all the coarse classes . To define a class hierarchy for Y and Z , we further assume that each coarse class z ∈ Z covers a subset of fine classes Yz , and that distinct coarse classes are associated with disjoint subsets of fine classes , i.e. , for any z1 , z2 ∈ Z , we have Yz1 ∩ Yz2 = ∅ . Our goal is fine-class classification by using the class hierarchy information . In particular , the supervised learning in this case can be formulated as : min Θ E ( x , y , z ) ∼D − log Pr ( y|x ; Θ ) , ( 1 ) where Θ is the model parameters . In practice , we solve the corresponding empirical risk minimization ( ERM ) during training , i.e. , min Θ n∑ i=1 − log Pr ( yi|xi ; Θ ) . ( 2 ) In contrast , meta-learning aims to maximize the expectation of the prediction likelihood of a task drawn from a distribution of tasks . Specifically , we assume that the subset of fine classes T for each task is sampled from a distribution T , and the problem is formulated as min Θ ET∼T [ E ( x , y , z ) ∼DT − log Pr ( y|x ; Θ ) ] , ( 3 ) where DT refers to the distribution of samples with label yi ∈ T . The corresponding ERM is min Θ ∑ T [ ∑ i∈DT − log Pr ( yi|xi ; Θ ) ] , ( 4 ) where T is a task ( defined by a subset of fine classes ) sampled from distribution T , and DT is a training set sampled from DT . To leverage the coarse class information of z , we write Pr ( y|x ; Θ ) in Eq . ( 1 ) and Eq . ( 3 ) as Pr ( y|x ; Θ ) = ∑ z∈Z Pr ( y|z , x ; Θf ) Pr ( z|x ; Θc ) , ( 5 ) where Θf and Θc are the model parameters for fine classifier and coarse classifier , respectively1 . Accordingly , given a specific sample ( xi , yi , zi ) with its ground truth labels for coarse and fine 1For simplicity , we neglect model parameters θCNN for feature extraction here . classes , we can write Pr ( yi|xi ; Θ ) in Eq . ( 2 ) and Eq . ( 4 ) as follows . Pr ( yi|xi ; Θ ) = Pr ( yi|zi , xi ; Θf ) Pr ( zi|xi ; Θc ) . ( 6 ) Suppose that a DNN model already produces a logit ay for each fine class y , and a logit bz for each coarse class z , the two probabilities in the right hand side of Eq . ( 6 ) are computed by applying softmax function to the logit values in the following way . Pr ( yi|zi , xi ; Θf ) = exp ( ayi ) ∑ y∈Yzi exp ( ay ) , Pr ( zi|xi ; Θc ) = exp ( bzi ) ∑ z∈Z exp ( az ) . ( 7 ) Therefore , we integrate both the fine-class label and coarse-class label in an ERM , whose goal is to maximize the likelihood of the ground truth fine-class label . Given a DNN that can produce two logit vectors a and b for fine class and coarse class , we can train it for supervised learning or meta-learning by solving the ERM problems in Eq . ( 2 ) or Eq . ( 4 ) ( with Eq . ( 6 ) and Eq . ( 7 ) plugged in ) .

Summary: This study explores the class hierarchy to solve many-class few-short learning problem in both traditional supervised learning and meta-learning. The model integrates both the coarse-class and fine-class label as the supervision information to train the DNN, which aims to leverage coarse-class label to assist fine-class prediction. The core part in the DNN is memory-augmented attention model that includes at KNN classifier and Memory Update mechanism. The re-writable memory slots in KNN classifier aim to maintain multiple prototypes used to describe the data sub-distribution within a class, which is insured by designing the memory utility rate, cache and clustering component in Memory Update mechanism. This study presents a relatively complex system that combines the idea of matching networks and prototypical networks.