Multi-Representation Ensemble in Few-Shot Learning

Deep neural networks ( DNNs ) compute representations in a layer by layer fashion , producing a final representation at the top layer of the pipeline , and classification or regression is made using the final representation . A number of DNNs ( e.g. , ResNet , DenseNet ) have shown that representations from the earlier layers can be beneficial . They improved performance by aggregating representations from different layers . In this work , we asked the question , besides forming an aggregation , whether these representations can be utilized directly with the classification layer ( s ) to obtain better performance . We started our quest to the answer by investigating the classifiers based on the representations from different layers and observed that these classifiers were diverse and many of their decisions were complementary to each other , hence having the potential to generate a better overall decision when combined . Following this observation , we propose an ensemble method that creates an ensemble of classifiers , each taking a representation from a different depth of a base DNN as the input . We tested this ensemble method in the setting of few-shot learning . Experiments were conducted on the mini-ImageNet and tiered-ImageNet datasets which are commonly used in the evaluation of fewshot learning methods . Our ensemble achieves the new state-of-the-art results for both datasets , comparing to previous regular and ensemble approaches . 1 INTRODUCTION . The depth of a deep neural network is a main factor that contributes to the high capacity of the network . In deep neural networks , information is often processed in a layer by layer fashion through many layers , before it is fed to the final classification ( regression ) layer ( s ) . From a representation learning point of view , a representation is computed sequentially through the layers and a final representation is used to perform the targeted task . There have been deep neural networks that try to exploit the lower layers in the sequence to achieve better learning results . GoogLeNets ( Szegedy et al. , 2015 ) added auxiliary losses to the lower layers to facilitate training . Skip links ( such as the ones used in ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) ) may be added to connect the lower layers to the higher ones in a deep architecture . Even though the main purposes of these approaches are to assist the training process or to help the gradient back-propagation , the success of these approaches suggests that the representations from the lower layers may be beneficial to many learning tasks . Therefore , it is worth to rethink the standard sequential structure where a final representation is used to make the prediction . In this work , we ask the question whether the representations from the lower layers can be used directly ( instead of being auxiliary or being aggregated into a final representation ) for decision making . If so , how can we take advantage of these lower-level representations and what are good practices in doing so ? We first investigated the problem by conducting classifications using the representations from different layers . We took the convolutional layers of a trained network as an encoder . The representations ( feature maps ) from different layers of the encoder were tested for their classification performance . We observed that although overall , the feature maps from the higher layers led to better performance , there was a significant number of cases that correct predictions could be made with the lower feature maps but the higher-level feature maps failed to do so . This suggested that the lower-level representations have the potential to help the classification directly ( detailed analysis in Section 3 ) . Based on the inspiration from the prior models ( i.e. , GoogLeNet , ResNet and DenseNet ) and our own observations , we propose an ensemble approach to directly take advantage of the lower-level representations . By integrating multiple models , an ensemble is likely to compensate the errors of a single classifier , and thus the overall performance of the ensemble would be better than that of a single classifier . This makes ensemble a suitable technique for our purpose . A variety of methods exist for ensemble construction . Some utilize sampling to obtain individual models from different subsets of the training data . Others construct models with different structures or initialization . However , these common ensemble methods can not achieve our goal to exploit the lower-level representations . Instead , we propose a special type of ensembles , different from the existing ones . In particular , each classifier in our ensemble takes a feature map from a different depth of a CNN encoder as input and the whole ensemble utilizes the feature maps from multiple convolutional layers . We call this approach the multi-representation ensemble . Figure 1 illustrates our ensemble approach and compares it to the common ensemble method . We evaluate our ensemble method on the few-shot learning ( FSL ) problem ( Snell et al. , 2017 ) . FSL aims to learn a network capable of recognizing instances ( query images ) from novel classes with only few labeled examples ( support images ) available in each class . Given the demanding nature ( learning from a few examples ) of the problem , many FSL approaches first train an encoder following a regular training paradigm and then further-train the encoder and the classifier using a FSL paradigm . Because the encoder plays an important role in few-shot learning , it is a good learning task to apply and test our ensemble method which takes advantage of multiple representations from the encoder . Note that in recent years , many FSL works have employed extra data from the test ( novel ) classes for better performance . The extra data can be unlabeled and given at the test time ( transductive learning ) ( Kye et al. , 2020 ; Yang et al. , 2020 ) or during the training phase ( semi-supervised learning ) ( Rodrı́guez et al. , 2020 ; Lichtenstein et al. , 2020 ) . Our problem scope focuses on the traditional FSL setting , where only a few ( one or five ) support images per novel class are available at the test time . Experiments with our ensemble model were conducted on two FSL benchmark datasets and we obtained new state-of-the-art results for both . Besides evaluating our ensemble and comparing it to the existing methods for FSL tasks , we also conducted experiments that demonstrated that the utilization of multiple representations in the ensemble is crucial for the success of our method . Our main contributions are as follows : 1 ) We propose a novel ensemble method that creates a collection of models by employing multiple representations from different depth of a deep neural network . 2 ) We demonstrated the advantage of our ensemble model on the FSL problems and achieved new stateof-the-art results on two benchmark datasets . Our experiments also showed that multi-representation is necessary for the improved performance of the ensemble . 2 RELATED WORK . Ensemble methods . Ensemble methods are commonly used to improve prediction quality . Some example ensemble strategies include : ( 1 ) manipulate the data , such as data augmentation or dividing the original dataset into smaller subsets and then training a different model on each subset . ( 2 ) apply different models or learning algorithms . For example , train a neural network with varied hyperparameter values such as different learning rates or different structures . ( 3 ) hybridize multiple ensemble strategies , e.g. , random forest . Ensembles have also been applied to FSL problems . Liu et al . ( 2019b ) proposed to learn an ensemble of temporal base-learners , which are generated along the training time , producing encouraging results on the mini-ImageNet and the Fewshot-CIFAR100 datasets . Dvornik et al . ( 2019 ) introduced mechanisms to encourage cooperation and diversity of individual classifiers in an ensemble model . The main difference between our method and the previous ones is the ensemble construction that utilizes multiple representations from different depth of a neural network . Few-shot learning . Meta-learning method has shown great success in FSL ( Finn et al. , 2017 ; Grant et al. , 2018 ; Lee & Choi , 2018 ) . MAML ( Finn et al. , 2017 ) used a meta-learner that learns from the training images to effectively initialize a base-learner for a new learning task in the test dataset . Further works aimed to enhance the generalization ability by improving the learning algorithm ( Nichol et al. , 2018 ) , fine-tuning the image embedding space ( Sun et al. , 2019 ; Rusu et al. , 2018 ) . Another popular direction for FSL is metric-learning which targets on learning metric space where classes can be easily separated . For example , Prototypical Networks use euclidean distance with each class prototype set to be the mean of the support embeddings ( Snell et al. , 2017 ) . Relation network ( Sung et al. , 2018 ) was proposed to compute the similarity score between a pair of support and query images . The query image is classified into the category with the highest similarity . Each individual model in our ensemble employs a relation network for classification while the relational network in different models receives different representation as input . Many recent FSL studies proposed approaches that utilized extra unlabled data or other additional information ( Li et al. , 2019b ; Kye et al. , 2020 ; Yang et al. , 2020 ; Hu et al. , 2020 ; Rodrı́guez et al. , 2020 ; Lichtenstein et al. , 2020 ) . They are not in the scope of the problem we were considering and thus not compared in the result section . 3 MOTIVATION . As a beginning investigation on the lower layer representations , we conducted a set of experiments to gain insight into their classification power . We took the FSL setting as the experiment environment and measured the performance of the representations from different convolutional layers in the encoder . ( ResNet-18 and its two variants were used as encoders . After encoder pretraining , FSL was conducted using the representation from an encoder layer combined with a classification network . Representations from different layers were tested . ) Figure 2 ( a ) shows the classification accuracy across different convolutional layers . The best performance did not come from the final convolutional layers ( layer 18 in ResNet-18-v1 and 16 in ResNet-18 and ResNet-18-v2 ) . Instead the fourth or the fifth layer from the last ( layer 14 in ResNet-18-v1 and 12 in ResNet-18 and ResNet-18v2 ) generated the highest accuracy . Going further towards the lower layers , we observe that overall for the three models , the lower the convolutional layer is , the worse the classification performance becomes . We looked further into the predictions made by different layers and investigated whether these predictions can be complementary , that is , is there enough diversity among the predictions such that by combining them together , we may obtain a better prediction . Taking layers 16 and 14 of ResNet-18 as examples , we examined the FSL predictions on a random sample of 75,000 images . The classification results using the representations from these two layers are shown in Figure 2 ( b ) . The model using the representation from layer 14 had a little higher overall accuracy ( ( 44.9 + 8.4 + 6.5 ) % = 59.8 % ) than that of the one using layer 16 ( ( 44.9 + 5.6 + 8.3 ) % =58.8 % ) . We can categorize the results into 4 scenarios : 1 ) Both classified correctly ( labeled and colored as “ Both ” in Figure 2 ( b ) ) ; 2 ) Only the model using representation from layer 16 made correct prediction ( labeled and colored as “ Layer16 ” ) . The model with layer 14 gave incorrect results to these images ; 3 ) Only the model with layer 14 predicted correctly ( labeled and colored as “ Layer14 ” ) ; 4 ) Neither made correct prediction ( labeled and colored as “ Neither ” ) . Although many images were classified correctly by both models ( 44.9 % of the total ) , some images could be only recognized by one of them ( scenario 2 and 3 ) . In an ideal situation , if we find a way to resolve perfectly the conflicts between the two models in both scenarios 2 and 3 and make correct prediction for these images , a classification accuracy as high as 73.7 % may be achieved . Ensemble is a potential approach towards this goal . We did a quick test by constructing an ensemble from the two models and using the average of their outputs as the output of the ensemble . Figure 2 ( b ) also shows the result of this quick ensemble . The ensemble reached a accuracy of 61.6 % , higher than any individual model . Clearly , for images in scenario 1 , the ensemble still made the correct predictions . For images in the scenarios 2 and 3 , the ensemble was able to resolve more than half of them and make correct classifications . ( There were 13.9 % of the images in scenario 2 , 8.3 % were resolved and had correct predictions from the ensemble . There were 14.9 % of the images in scenario 3 , 8.4 % were resolved . ) Although the ensemble did not reach the ideal limit mentioned above , it did provide a better classification , more accurate than any of the individual model in the ensemble . This quick experiment shed light on the possibility to exploit representations from different layers in a deep neural network as a way to obtain better learning performance . Following this direction , we designed our ensemble method for FSL tasks . We conducted more thorough experiments to validate the approach and to determine the design choices that can optimize the performance gain .

Summary: This paper presents a deeply supervised few-shot learning model via ensemble achieving state-of-the-art performance on mini-ImageNet and tiredImageNet. The authors first studied the classification accuracy on mini-Image across convolutional layers and found the network could perform well even in the middle layer. Therefore, they added classification headers on the selected layers, so that these layers can directly output predictions. The final result is the ensemble of all the select layer predictions, called the Multiple Representation Emsemble. To improve the result, they further average the results of two models with different network backbones, called Multi-Model Emsemble. The results show this method can achieve state-of-the-art performance on the two datasets.