Multi-Representation Ensemble in Few-Shot Learning
Authors: Qing Chen
Jian Zhang
Venue:  (2021)
Keywords: Ensemble learning
Few shot learning
Multi-representaion

Abstract: Deep neural networks (DNNs) compute representations in a layer by layer fashion, producing a final representation at the top layer of the pipeline, and classification or regression is made using the final representation. A number of DNNs (e.g., ResNet, DenseNet) have shown that representations from the earlier layers can be beneficial. They improved performance by aggregating representations from different layers. In this work, we asked the question, besides forming an aggregation, whether these representations can be utilized directly with the classification layer(s) to obtain better performance. We started our quest to the answer by investigating the classifiers based on the representations from different layers and observed that these classifiers were diverse and many of their decisions were complementary to each other, hence having the potential to generate a better overall decision when combined. Following this observation, we propose an ensemble method that creates an ensemble of classifiers, each taking a representation from a different depth of a base DNN as the input. We tested this ensemble method in the setting of few-shot learning. Experiments were conducted on the mini-ImageNet and tieredImageNet datasets which are commonly used in the evaluation of few-shot learning methods. Our ensemble achieves the new state-of-the-art results for both datasets, comparing to previous regular and ensemble approaches.

Under review as a conference paper at ICLR 2021
MULTI-REPRESENTATION ENSEMBLE IN FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs) compute representations in layer by layer fash- ion, producing a final representation at the top layer of the pipeline, and classifi- cation regression is made using the final representation. A number of DNNs (e.g., ResNet, DenseNet) have shown that representations from the earlier lay- ers can be beneficial. They improved performance by aggregating representations from different layers. In this work, we asked the question, besides forming an aggregation, whether these representations can be utilized directly with the classi- fication layer(s) to obtain better performance. We started our quest to the answer by investigating the classifiers based on the representations from different layers and observed that these classifiers were diverse and many of their decisions were complementary to each other, hence having the potential to generate a better over- all decision when combined. Following this observation, we propose ensemble method that creates an ensemble of classifiers, each taking a representation from a different depth of base DNN as the input. We tested this ensemble method in the setting of few-shot learning. Experiments were conducted on the mini-ImageNer and ImageNet datasets which are commonly used in the evaluation of few- shot learning methods. Our ensemble achieves the new state-of-the-art results for both datasets, comparing to previous regular and ensemble approaches.
INTRODUCTION
The depth of deep neural network main factor that contributes to the high capacity of the network. In deep neural networks, information is often processed in layer by layer fashion through many layers, before it is fed the final classification (regression) layer(s). From representation learning point of view, a representation is computed sequentially through the layers and final representation is used to perform the targeted task. There have been deep neural networks that try to exploit the lower layers in the sequence to achieve better learning results. GoogLeNets (Szegedy) cet al., 2015) added auxiliary losses to the lower layers to facilitate training. Skip links (such as the used in ResNet (He et al., 2016) and DenseNet (Huang et al. (017)) may be added to connect the lower layers to the higher ones a deep architecture. Even though the main purposes of these approaches are assist the training process or to help the gradient pack-propagatton, the success of these approaches suggests that the representations from the lower layers may be beneficial many learning tasks. Therefore, it is worth to rethink the standard sequential structure where final representation is used to make the prediction. In this work, we ask the question whether the representations from the lower layers can be used directly (instead of being auxiliary or being aggregated into final representation) for decision making. If so, how can we take advantage of these lower-level representations and what are good practices in doing so?
We first investigated the problem by conducting classifications using the representations from differ- ent layers. We took the convolutional layers of a trained network as an encoder. The representations (feature maps) from different layers of the encoder were tested for their classification performance. We observed that although overall, the feature maps from the higher layers led to better performance, there significant number of cases that correct predictions could be made with the lower fea- ture maps but the higher-level feature maps failed to do so. This suggested that the lower-level representations have the potential help the classification directly (detailed analysis in Section 3).

---

Under review as a conference paper at ICLR 2021
Traditional ensemble
Multi- Nulti-representatioon ensemble
Model
Model n
Model 1
Model 2
Model
Classifier
Classifier
CNN
Encoder
CNN Encoder
Figure 1: Multi-representation classifier ensemble.
Based on the inspiration from the prior models (i.e., GoogLeNet, ResNet and DenseNet) and our own observations, we propose ensemble approach to directly take advantage of the lower-level representations. By integrating multiple models, an ensemble is likely to compensate the errors of single classifier, and thus the overall performance of the ensemble would be better than that of a sin- gle classifier. This makes ensemble a suitable technique for our purpose. A variety methods exist for ensemble construction. Some utilize sampling to obtain individual models from different sub- sets of the training data. Others construct models with different structures or initialization. However, these common ensemble methods cannot achieve our goal exploit the lower-level representations. Instead, we propose special type of ensembles, different from the existing ones. In particular, each classifier in ensemble takes feature map from different depth of CNN encoder as input and the whole ensemble utilizes the feature maps from multiple convolutional layers. We call this ap- proach the multi-representation ensemble. Figure fillustrates our ensemble approach and compares it to the common ensemble method.
We evaluate our ensemble method on the few-shot learning (FSL) problem (Snell et al. 2017 FSL aims to learn a network capable of recognizing instances (query images) from novel classes with only few labeled examples (support images) available in each class. Given the demanding nature (learning from few examples) of the problem, many FSL approaches first train an encoder following regular training paradigm and then further-train the encoder and the classifier using a FSL paradigm. Because the encoder plays important role in few-shot learning, it is good learning task to apply and test our ensemble method which takes advantage multiple representations from the encoder. Note that in recent years, many FSL works have employed extra data from the test (novel) classes for better performance. The extra data can be unlabeled and given at the test time (transductive learning) (Kye et al. 2020 Yang et al., (2020) during the training phase (semissupervised learning) [Rodríguez et al. 2020 Lichtenstein et al. (020). Our problem scope focuses on the traditional FSL. setting, where only few (one five) support images per novel class are available the test time.
Experiments with our ensemble model were conducted on two FSL benchmark datasets and we obtained new state-of-the-art results for both. Besides evaluating our ensemble and comparing to the existing methods for FSL tasks, we also conducted experiments that demonstrated that the utilization of multiple representations in the ensemble is crucial for the success of our method. Our main contributions are as follows: 1) We propose novel ensemble method that creates a collection of models by employing multiple representations from different depth of a deep neural network. 2) We demonstrated the advantage of our ensemble model on the FSL problems and achieved new state- of-the-an art results on two benchmark datasets. Our experiments also showed that multi multiirepressntttionn is necessary for the improved performance of the ensemble.
2 RELATED WORK
Ensemble methods.
Ensemble methods are commonly used to improve prediction quality. Some
example ensemble strategies include: (1) manipulate the data, such as data augmentation or divid- ing the original dataset into smaller subsets and then training a different model on each subset. (2) apply different models or learning algorithms. For example, train a neural network with varied hy- perparameter values such different learning rates different structures. (3) hybridize multiple ensemble strategies, e.g., random forest. Ensembles have also been applied to FSL problems. Liu et al. (2019b) proposed to learn ensemble of temporal base-learners, which are generated along
2

---

Under review as a conference paper at ICLR 2021
Individual
59
5.6%
8.3%
Layer16
57
26.4%
Both
55
53
Layer14
er1
51
49
ResNet-18
Net 18-v1
6.5%
6.5
8. 6
44.9%
Ensemble
Neither
18
Wrong
5
6
7
8
Convolational byer
12
14
15
Correct
(b)
Figure 2: (a) 5-way, 1-shot classification accuracy mini-ImageNet across convolutional layers. (b) Classification results for convolutional layer and 16 of ResNet-18 18 and the ensemble incorpo- rating both.
the training time, producing encouraging results on the mini-ImageNet and the datasets. Dvornik et al. (2019) introduced mechanisms to encourage cooperation and diversity of individual classifiers in an ensemble model. The main difference between our method and the previ- ones is the ensemble construction that utilizes multiple representations from different depth of a neural network.
2017 Grant et al. 2018 Lee & Choi 2018). MAML (Finn et al. (017] used a meta-learner that learns from the training images effectively initialize base-learner for new learning task in the test dataset. Fur- ther works aimed to enhance the generalization ability by improving the learning algorithm (Nichol al., 2018). fine-tuning the image embedding space (Sun al.) 2019 Rusu al., 2018). Another popular direction for FSL is metric-learning which targets on learning metric space where classes can be easily separated. For example, Prototypical Networks use euclidean distance with each class prototype set to be the mean of the support embeddings (Snell et al. 2017). Relation network (Sung) cct al. 2018) was proposed to compute the similarity score between a pair of support and query im- ages. The query image is classified into the category with the highest similarity. Each individual model in our ensemble employs relation network for classification while the relational network in different models receives different representation
Few-shot learning. Meta-lcarning method has shown great success in FSL (Finn
input. Many recent FSL studies proposed approaches that utilized extra unlabled data or other additional information (Li et al.] 2019b Kye al. 2020); Yang al. 2020 Hu et al. 2020) Roddiiuuez et al., 20200 Lichtenstein et 2020). They are not in the scope of the problem we were considering and thus not compared in the result section.
MOTIVATION
As a beginning investigation on the lower layer representations, we conducted a set of experiments gain insight into their classification power. We took the FSL setting as the experiment environ- ment and measured the performance of the representations from different convolutional layers in the encoder. (ResNet-18 and its two variants were used encoders. After encoder pretraining, FSL was conducted using the representation from an encoder layer combined with a classification network. Representations from different layers were tested.) Figure 2 a) shows the classification accuracy across different convolutional layers. The best performance did not come from the final convolutional layers (layer 18 in ResNet- 18-v1 and 16 in ResNet- 18 and ResNet-18-v2. fourth the fifth layer from the last (layer 14 in ResNet- 18-v1 and 12 in ResNet- 18 and ResNet- 88- v2) generated the highest accuracy. Going further towards the lower layers, we observe that overall for the three models, the lower the convolutional layer is, the worse the classification performance becomes.
Instead the
We looked further into the predictions made by different layers and investigated whether these pre- dictions can be complementary, that is, is there enough diversity among the predictions such that by combining them together, may obtain better prediction. Taking layers 16 and 14 of ResNet- 18 as examples, we examined the FSL predictions on a random sample of 75,000 images. The classifi- cation results using the representations from these layers shown in Figure bb). The model
3

---

Under review as a conference paper at ICLR 2021
using the representation from layer 14 had a little higher overall accuracy ((44. + 8.4 + 6.5)%
59.8%) than that of the one using layer 16 ((44.9 + 5.6 + 8.3)% =5..%%) We can categorize the re- sults into 4 scenarios: 1) Both classified correctly (labeled and colored as "Both" in Figure 2 b)); 2) Only the model using representation from layer 16 made correct prediction (labeled and colored as "Layer16")) The model with layer 14 gave incorrect results these images; 3) Only the model with layer 14 predicted correctly (labeled and colored as "Layer14") 4) Neither made correct prediction (labeled and colored as "Neither"). Although many images were classified correctly by both models (44.9% of the total), some images could be only recognized by one of them (scenario 2 and 3).
In an ideal situation, if we find way to resolve perfectly the conflicts between the two models in both scenarios 2 and 3 and make correct prediction for these images, classification accuracy as high as 73.7% may be achieved. Ensemble is a potential approach towards this goal. We did a quick test by constructing an ensemble from the two models and using the average of their outputs as the output of the ensemble. Figure 2]b also shows the result of this quick ensemble. The ensemble reached a accuracy of 61.6%, higher than any individual model. Clearly, for images in scenario 1, the ensemble still made the correct predictions. For images in the scenarios and 3, the ensemble was able to resolve more than half of them and make correct classifications. (There were 13.9% of the images in scenario 2, 8.3% were resolved and had correct predictions from the ensemble. There were 14.9% of the images in scenario 3, 8.4% were resolved.) Although the ensemble did not reach the ideal limit mentioned above, it did provide a better classification,, more accurate than any of the individual model in the ensemble. This quick experiment shed light on the possibility to exploit representations from different layers in deep neural network as way to obtain better learning performance. Following this direction, we designed our ensemble method for FSL tasks. We conducted more thorough experiments to validate the approach and to determine the design choices that can optimize the performance gain.
4 METHODOLOGY
4.1 PROBLEM DEFINITION
Formally, we have three disjoint datasets: a training set Dimin a validation set Deat= and a testing set Diest In the traditional C-way N-shot classification in FSL. we are tasked obtain model that can perform classification among C' classes (in the testing set Drea) while we have access to only N samples from each class. (The model can be pre-trained using data in Dunin or even data in D However, there are no overlapping classes between Diest and Duain or Dvat.) The set of C x N samples is often referred to as the support set. In many FSL works, N is commonly set to be 1 or 5. We remark that some recent FSL researches have started to explore helps from additional information. In particular, a number of methods have been proposed to leverage unlabeled data beyond the N examples from the C' classes to enhance model accuracy. They either use the unlabeled data in the training seemi-supervised learning) or perform classifications with a set of query data together (transductive learning) (Li et al. 2019b Kye et al., 2020; Yang et al. 2020: Hu et al, 2020 Rodríguez et al. 2020) Lichtenstein et al., 2020). In this work, we focus on the traditional FSL setting where additional information or unlabeled data are available and the prediction for query data point is made independently from (without knowing) any other query data points.
4.2 MULTI-MODEL MULTI- SELLTT-EERSSENTTIION ENSEMBLE
Our ensemble contains multiple encoders (encoders of different network structures). Hence the ensemble is also molti-model. An encoder generates multiple representations. Each is then fed to a classifier network. The combination of a particular representation and classifier network forms an individual classification model in the ensemble. Figure g shows the overall architecture of the ensemble. (For simplicity, the figure illustrates only one encoder with multiple representations. An actual ensemble may include several encoders.) In the following, We describe the details of the ensemble.
Same as many other research work on FSL, we consider the task of image classification. Therefore Dusin= D and Dust are collections of images and their labels. Before being employed in an ensemble to provide multiple representations, an encoder is pre-trained following the pretraining protocol in FSL (Rusu al. 2018 Sun et al. 2019). If multiple encoders are employed, each of them
4

---

Under review as a conference paper at ICLR 2021
Encodere
SU))
Milti-ruprss mp
S
g
q
r
(n-D
Figure Classification by multi-representation ensemble for a 2-way 1-shot problem.
will undergo the pretraining independently. After the pretraining stage, we have a set of encoders and each encoder provide multiple representations (image features) from different layers. We
refer to the features for image x from n-th convolutional layer in the encoder e by (x). Our ensemble uses relation network (Sung et al.) 2018) to perform classification. The relation network takes a pair of instances from a particular representation as input and outputs a similarity score. We denote by g the score function computed by the relation network.
Let ((1) and ((2) be two images from two different classes (class and 2) in the support set. Let q be a query image to be classified. The images are first fed into the encoders and for each im- age, multiple representations (from different encoders and different layers) are generated. Take an encoder and the n-th layer of e as an example, We illustrate the classification process using the representations ot (s()) (orange in Figure (s( (green in Figure and ((") (q) (blue in Figure 3) (Note that Figure 3 contains two sets of representations ol (-) and Φe "))))))) with the same color scheme. Colors are used to indicate the source image from which the representation is
computed. They are not related to the location, i.e., the encoder and the layer, where the representa- tion is generated.) The representation of (*) (q) is concatenated (on the channel dimension) to the representation of o!n)) (s(")). The relation network is applied to the concatenation and score is calculated to measure the similarity between the query and the class (class 1) representative image (11), i.e. y()) sion. In an N-shot (N > 1) setting, for each class, we have N examples (c.g., S(k), s(() ...., s(h)
o(o5") ((( where || indicates concatenation on the channel dimen-
for class k). A prototype is obtained by averaging the representations (i (s) ()) The score computed using the concatenation of de
(n) (g) and the prototype, i.e., in general,
(b) g(((m) (g)l
Σ N
diml (s:
(k)))
This calculation is performed for each class to produce C scores (1), (2)) (()) for C-way classification. The query image is classified into the category that has the maximum score.
The above presents the classification process of an individual model in the ensemble. There are set of encoders 61, c.. and for each encoder es, multiple representations from different layers (es) ,... are used to produce models in the ensemble. Let T {n(e:) (e:) } be the set of repre-
n(()), .12
sentations, each leading to a classifier. Let rt (q), ( E 2,,,. C} be the score for image q with respect to class and computed by the classifier using representation The ensemble's fi- nal classification is made following the average scores over the models, i.c., the class label c* (g) is predicted be:
(4) arg max
(0)
(2)
EE
5

---

Under review as a conference paper at ICLR 2021
where [T] is the size of the set T. Each relation network in the ensemble will be trained indepen- dently following the training procedure in (Sung et al. 2018) while the encoders remain unchanged after pre-training.
5 EXPERIMENT RESULTS
5.1 EXPERIMENTAL, SETUP
Datasets. We used two standard benchmark datasets in FSL: mini- mini-ImageNet (Vinyals et al. 2016) and fiered-ImageNet (Ren et al. 2018). Mini-ImageNer consists of 100 categories, 64 for training, 16 for validation and 20 for testing, with 600 images in each set. Tiered- Imrred-I includes 608 classes (779, 165 images) split as 351 training. 97 validation and 160 testing classes, each with about 1300 images. The image size of both datasets is 84 x 84.
Implementation details. We adopt the same ResNet- 18 model in (Han et al. 2020) and two variants of ResNet- 18 as the encoders. The first variant is ResNet-18-v1 in (Sun et al., (019) which has residual sections with 3 basic blocks, each including 2 convolutional layers, and the second one is ResNet- 18-v2 consisted of 2 residual sections with basic blocks. In order to pretrain an encoder that produces image features/representations, the first step is similar to the common pre-training stage used in FSL (Rusu et al. 2018: Sun et al. 2019). We merged data of all classes in Derain for pre-training. After pre-training, we added shift and scaling parameters for the convolutional layers in the encoder and trained the parameters by the MTL approach used in (Sun et al. (019). To further improve generalization, we also fine-tuned the upper layers in each encoder using Da as unlabelled data, following the method proposed in (Han et al. 2020). Specifically, we fine-tuned the upper half layers of the encoder while freezing the rest. After pre-training, the encoder remains unchanged.
The evaluation of the ensembles was conducted using C-way N-shot classification tasks with C' C 5 and N = 1 or N = 5. In an episode of evaluation, a classification task was constructed by randomly selecting C classes and N samples per class from Drest to serve as the support set. 15 random query data points from the C' classes were also selected as the query images to test the classification. The evaluation process consisted of 1000 episodes. The mean accuracy (in %) over the 1000 episodes and the 95% confidence interval are reported in the experiment results. Input images were re-scaled to the size 80 80 and normalized before fed into the model. The relation networks produce the similarity scores between the support and the query images were composed of 2 convolutional layers followed by a fully-comeettdd layer with a sigmoid function. Our implementation was based on PyTorch (Paszke et al., 2017a). Pre-training phase used default parameters in the works we cited. For optimization of the relation networks we used stochastic gradient descent (SGD) with the Nesterov momentum 0.9. The initial learning rate was set to le-2.
5.2 MAIN FSL RESULTS
The ensemble used in this section employed representations from the last (from top) 9 convolutional layers in R8 and the last (from top) 6 layers in ResNet-18-v1. The results on the two bench- mark datasets are shown in Table | As stated in previous sections, we focused on traditional FSL. In Table ! we compare our results only to the best prior results on traditional FSL. Table [ shows that our model gives the new state-of-the-arr performance on the 1-shot and the 5-shot tasks for both the mini-ImageNet and the sered-Imagett datasets. This illustrates the effectiveness of the multi-model multi-rrpressentation ensemble for the FSL problems.
5.3 ABOLITION STUDY AND ENSEMBLE DESIGN
In this section, we further demonstrate the benefit of lower-level features using results from the abo- lition experiments and discuss the design choices for constructing better ensemble. For abolition study, we compared ensembles that utilized different sets (subsets) of represeetatioosseeccoeer, We first present and discuss the performance results from ensembles that involve multi-representation from single encoder and then continue to ensembles that employ both multiple representations and encoders. The ensemble performances (accuracy) were measured for few shot classification on the mini-ImageNNe dataset.

---

Under review as a conference paper at ICLR 2021
mini-Iman
ImageNet
diered-maageNet
Model
Encoder
I-shot
5-shot
I-shot
5-shot
TADAM (Oreshkin al.
2018
ResNet-12
58.50 0.30
76.70 0.30
MTL (Sun et
2019
ResNet-11
62.10 = 1.80
78.50
0.90
67.8 i 1.8
83.0 1 0.7
TapNet Yoon al.
2019
ResNet-12
61.65 = 0.15
76.36
0.10
63.08
1 0.15
80.26 0.12
MetaOpt-SWM
al.
2019
ResNet-12
62.64 + 0.61
78.63
+ 0.46
65.99
+ 0.72
81.56 +
CAN (Hou et al
2019
ResNet-12
63.85 = 0.48
79.44 0.34
69.89
+ 0.51
84.23 0.37
CTM
La ct al.
2019a
ResNet-18
61.12 0.82
80.51 + 0.13
68.41
+ 0.39
84.28 + 1.73
Ensemble
Robust-dist++ (Dvorikk al.
2019
ResNet-18
59.48 + 0.62
75.62 + 0.48
(Lau TB**
2019A
ResNet-25
64.3
81.0
70.0
85.0
Ours
ResNet-18
65.01
82.31 + 0.49
70.87:
85.31 0.31
Table I: The 5-way, 1-shot and 5-shot classification accuracy (%) on mini-ImageNet and tiered- ImageNet datasets. Average classification performance over 1000 randomly generated episodes, with 95% confidence intervals. (* Confidence intervals are not reported in the original paper.)
Single Encoder Multiple Representation. Figure 2 shows that the best accuracy obtained by classifier using a single representation from ResNet-18 is 59.53% (with representation from layer 12). In Table we observe that ensembles utilizing metti-repressntation have boosted performance. For example, the ensemble containing classifiers using representations from layer 16 to layer 14 of R8 achieves an accuracy of 62.59% (row 1 of Table D. Including more classifiers that use representations from layers 13, 12 and 11, the ensemble can reach an even higher performance (63.54%, row 2). The topmost accuracy (64.03%) from the ensembles based on ResNet- 18 comes from the one that utilizes representations from layers 16 to 8. The improvement by including multi- ple representations into the ensemble can also be observed for the ResNet-18-v and the ResNet- v2 encoders. In general, for single encoder ensembles, coming down from the top layer, the more representations we include in the ensemble, the better performance the ensemble can achieve, until turning-point layer is reached. Adding representations after the turn point may lead to reduced performance.
Multiple Encoder Multiple Representation. Incorporating different models (e.g., neural networks of different structures) is a common method to construct an ensemble. Clearly, this multi-model construction can be combined with our multi-representation construction to create ensembles that employ both multiple models and multiple representations from each model. The last row of Ta- ble 2 shows the configuration and the performance of one of our modti, neutettreprssnntaaioo ensembles used in the experiments. There is significant performance gain comparing this ensem- ble to those on row 13 and 14. The ensembles on row 13 and 14 are multi-model but not multi- representation since only one representation is used from each of the encoders in the ensembles. Although a multi-model ensemble already performs better than the individual models in the ensem- ble, adding multi-representatiin on top of multi-model leads to even better performance, giving rise to the best performer among the ensembles (row 15).
Selection of Encoders and Representations for the Ensemble Not all encoders or representations are helpful for an ensembles. Some representations (such as the ones from layers 7 and 6 of ResNet- 18, layers 12 and 11 of ResNet-18-vile when incorporated into the ensemble, led to performance decline (rows and in these individual representations also showed lower performance (Figure aa)) Similarly, the per- formance of a multi-encoder ensemble may degrade considerably if an encoder that falls behind the others is included. Our final ensemble did not include ResNet- 18-v2 as an encoder because its performance is significantly lower than that of ResNet- 18 and ResNet- Mer-Net-18-- 18-v1. To select an encoder or a representation be included in an ensemble, we test, under the same task using the validation data, the performance of the individual encoders and representations. (For an encoder, we use the best performance across its layers as the measure for the encoder.) A relative threshold < < established. An encoder (representation) is selected if its performance is above of the performance of the best encoder (the best representation from the same encoder). We set T 0.93 to construct our final ensemble.
to rows 3, 7). We observe the classifiers that employed
Ensemble vs Aggregated Representation
7

---

Under review as a conference paper at ICLR 2021
Row#
Encoder(s)
Cony layers
Accuracy(
ResNet-18
16-14
62.59
ResNet-18
16-11
63.54
ResNet-18
16-8
64.03
ResNet-18
16-6
63.71
ResNet-18-vl
18-17
58.57
ResNet-18-vl
18-15
59.67
ResNet-118-v
18-13
60.72
8
ResNet-18-v1
18-11
59.21
ResNet-18-v2
16-14
56.05
10
ResNet-18-v2
16-12
57.49
ResNet-18-v2
16-10
58.26
12
ResNet-18-v2
16-8
58.77
ResNet-18, ResNet-18-vl
16, 18
60.87
14
ResNet-18, ResNet-18-v1. ResNet-18-y
16, 18, 16
58.99
ResNet-18, ResNet-18-vl
16-8, 18-13
65.01
Table 2: Classification accuracy of multi-mode multi-representation ensembles. (The second col- umn lists encoders and the third lists layers (or ranges of layer) from which representations were used. If there multiple encoders, their layer(s) information is separated by comma.)
As discussed in the introduction, one the mo- tivations for us to explore representations from
65
64
different layers the skip links in neural net-
works such as DenseNet. We conducted exper-
iments to compare our multi-represenlation en- semble to alternative that uses DenseNet-like aggregated representation.
62
61
Ensemble A ggregated
the alternative approach, representations from different layers were aggregated, by concatenation, into a sin- gle representation and classifications, using the relation network, were performed on this sin- gle representation. The results of comparison are shown in Figure 4 We tested these two methods on the mini- ImageNet dataset in 5- way, 1-shot setting. The performance of our multi-representation ensemble is much better than that of the aggregated representation alter- native. Moreover, aggregating more representa- tions does not necessarily benefit the classification since the model accuracy slightly drops and then stands still as representations from more layers are added in. This shows that not all approaches which try utilize multiple representations Our multi-representation ensemble is one approach that highly effective in taking advantage of the multiple representations.
60
59
58
16-14
15-12
16-12
16-11
16-8
16-6
Layers
Figure 4: Comparison of classification accuracy between multi-representation ensemble and ag- gregated representation. Marks on the x-axis in- dicate the range of convolutional layers.
attain performance gain from these representations.
6 CONCLUSIONS
In this paper we propose new ensemble method that creates an ensemble of classifiers, each using the representationnfeatuue map from different depth in a CNN encoder. Through experiments, we validated the effectiveness of our model on FSL problems. Our ensemble achieved the new state-of- the-art results on two commonly-used FSL benchmark datasets. We further conducted experiments and analysis to investigate the selection of representations for creating better ensemble. It is quite possible that the multi-representation approach is not limited to the few-shot learning problems. As be applied. Further-
future work, we plan to explore more scenarios where multi nulti-representation more, the multi-representation approach does not have conflict with many other ensemble methods. It can be combined with other types of ensemble constructions to produce better ensemble.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. Proceedings of the IEEE International Conference on Computer Vision, pp. 3723-3731, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint
2017.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes. arXiv preprint arXiv: 08930000009,,, 10X990000 2018.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. arXiv preprint arXiv:2002.057/1 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Advances in Neural Information Processing Systems, pp. 4003- 4014, 2019.
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Exploiting unsupervised inputs for accurate few-shot classification. arXiv preprint arXiv:2001.09849, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Scong Min Kye, Hae Beom Lee, Hoirin Kim, and Sung Ju Hwang. Transductive few-shot learning with meta-learned confidence. arXiv preprint w:Xiv:2002.12011 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. arXiv preprint 1801 05558, 2018
Hongyang Li, David Eigen, Samuel Dodge. Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In Proceedings of the IEEE Confer- on Computer Vision and Pattern Recognition, pp. 1-10, 2019a.
Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning self-train for semi-supervised few-shot classification. In Advances in Neural Infor- mation Processing Systems, pp. 10276-10286, 2019b.
Moshe Lichtenstein, Prasanna Sattigeri, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Tafssl: Pask-adaptive feature sub-space learning for few-shot classification. arXiv preprint arXiv:2003.00 06670. 2020.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble of epoch-wise empirical bayes for few- shot learning. arXiv preprint urXiv: 190440847,, 2019a.
Yaoyao Liu, Qianru Sun, An-An Liu, Yuting Su, Bernt Schiele, and Tat-Seng Chua. Lee: Learning to customize and combine neural networks for few-shot learning. arXiv preprint arXiv:1904.08444 2019b.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta- leta-learning algorithms. arXiv preprint 1rxiv:1803 1803.00999 2018

---

Under review as a conference paper at ICLR 2021
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint TrXXv:/803.00677 2018.
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propagation: Smoother manifold for few-shot classification. arXiv preprint arXiv:2003.04151, 2020.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05566. 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Putlern Recognition, pp. 1199-1208, 2018
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-366 0638,,63.--000 2016.
Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distribution propagation graph network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13390-13399, 2020.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. arXiv preprint 195500549, 2019.
10


Reviews:
[{"review_id": "ozcq0lKqJYU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a simple approach, which obtains competitive results with the state of the art of few shot learning. However, I have the following concerns:\n \n- the proposed method is somewhat incremental. The authors propose to average the predictions of classifiers that take as input different features from the backbone. \n- While it’s a sensible thing to try, in my understanding, the proposed method is equivalent to a simpler approach, that would simply concatenate those features and learn a classifier on the concatenated features. I believe that this approach, and a number of other simple baselines employing a wider representation space extracted from the backbone, would be important to strengthen the analysis of the proposed method.\n- The presentation of state of the art results is incomplete. Dvornik et al. also report results for tiered-Imagenet (which surpass the reported results). Some other relevant works would need to be cited and compared to [1, 2] (some of their results also surpass the reported results.)\n- Organisation: method and results should be presented separately: the current flow of the paper alternates between empirical findings (motivation), a formal approach (methodology) and experimental results. This structure suggests that the submission would likely be better suited for a more technical venue. It would also better to isolate in a background section the presentation of the baseline approach (Sung et al. 2018), before presenting the proposed method itself, to make it more evident what the contributions are.\n- the authors do not motivate the chosen experimental setting (FSL): there is no analysis of why the proposed approach should be particularly well suited to address the specificities and challenges of this task.\n- it seems to me that employing so many linear classifiers (on increasingly larger dimensional features) would lead to a large increase in parameter count - but the authors do not perform any analysis regarding this aspect. \n- Overall, a lot of polishing of the paper is needed prior to publication. Please find a few comments in that respect below.\n\nComments:\n- Figure 2: what is ResNet-18 (in red) if it’s not v1 or v2 ? On this note, both papers should be cited when they are introduced in Section 3 (only ResNet v1 is cited.)\n- “Our ensemble contains multiple encoders (encoders of different network structures).” At this point, this is not very clear: is the method used on top of a traditional ensemble ?\n- MULTI-MODEL MULTI-REPRESENTATION ENSEMBLE sounds tautological.\n- abolition should probably be ablation\n\n\n[1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Function, Ye et al. CVPR 2020\n[2] Adaptive Subspaces for Few-Shot Learning Simon et al CVPR 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "njBfDFRtbfo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to tackle the problem of few-shot learning (FSL) using ensembling diverse classifiers. The diverse classifiers are obtained using the outputs from different intermediate layers of a pre-trained CNN feature extractor (or multiple CNNs). As a result, the authors demonstrate state-of-the-art accuracy on two mini-ImageNet and tiered-ImageNet datasets.\n\nPros:\n- The idea totally makes sense since, in few-shot learning, the test distribution may be quite different from the training one. Hence, employing lower-layer features that are more class-invariant must be helpful, even though the space of semantic concepts learned by earlier layers is probably not as reach as for the deeper layers.\n- The results in mini-ImageNet and tiered-ImageNet are impressive. The experimental section is informative and clear.\n- The paper is well written and is easy to follow.\n\nCons:\n- Limited contribution. None of the introduced ideas in this paper is novel. For example, the idea of using ensemble methods for FSL was introduced in [1]. Then, the idea of aggregating information from intermediate layers of a feature extractor to build a reacher classifier for FSL was introduced in [2]. The authors of [3] also used intermediate layers for better classification results. Basically, the contribution of the current work is to combine the ideas of [1] and [2] while using a different backbone network (a new ResNet18) and a different classifier (RelationNet). \n- I would call the need to manually select the layers from which to build classifiers a down-side of the approach since selecting all representation would lead to degraded performance.\n\nOverall, I like how the paper reads. However, the contribution of this work boils down to combining existing ideas and methods into a new pipeline, which I don't find sufficient for the ICLR acceptance standard.\n\n[1] - Dvornik et.al. \"Diversity with cooperation: Ensemble methods for few-shot classification\"\n[2] - Dvornik et.al. \"Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification\"\n[3] - Rusu et.al. \"Meta-Learning with Latent Embedding Optimization\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "n37cJxHydtl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a deeply supervised few-shot learning model via ensemble achieving state-of-the-art performance on mini-ImageNet and tiredImageNet. The authors first studied the classification accuracy on mini-Image across convolutional layers and found the network could perform well even in the middle layer. Therefore, they added classification headers on the selected layers, so that these layers can directly output predictions. The final result is the ensemble of all the select layer predictions, called the Multiple Representation Emsemble. To improve the result, they further average the results of two models with different network backbones, called Multi-Model Emsemble. The results show this method can achieve state-of-the-art performance on the two datasets.\n\nAdvantage:\n1. The motivation and idea in this paper are clear and simple, so the reader is easy to understand it.\n2. Figures 2 and 3 are nice, which are clearly demonstrate the motivation and algorithm.\n3. The find in Figure 2(a) is very interesting. The middle layer has a better representation than the end on the few-shot image classification task.\n4. The results are positive.\n\nDisadvantage:\n1. The idea in the paper is not very novel. The main contribution of this paper is doing a deep supervision ensemble. However, people have studied deep supervision learning for a while on image classification [1], segmentation [2], and depth estimation [3]. Specifically, [2] [3] also fuse the multi-layers' outputs.\n2. The authors only show the ensemble results via averaging scores over the models. It will be good to study more ensemble methods. For example, the deep layer has higher accuracy than the shallow layer. Is it possible to assign a different ensemble weight for each layer based on the accuracy?\n3. In figure 2(a), why the middle layer performs better than the last layer? It will be good to show some analysis?\n4. In table 1, since the proposed model has done a model ensemble, it cannot directly compare with CAN and CTM. Should add the result without ensemble in table 1. If I put the third-row result \"64.03\" in table 2 to table 1, the improvement would be marginal.\n5. Both mini-ImageNet and tired-ImageNet are the subsets of ImageNet. To verify the generalization, it will be good to add CIFAR, meta-iNat [4], or CUB [5] results.\n\nMinor mistakes,\n1. Equation 1, should add the superscript `n` to r.\n2. Figure 1, the characters are not evenly spaced.\n3. Figure 2 (a), the axis label is too small.\n4. In section 4.1, the sentence \"The model can be pre-trained ......Dtrain or Dval.)\" is redundant, which is common sense.\n5. In section 5.1, \"After pre-training, we added shift and scaling parameters for the convolutional layers in the encoder and trained the parameters by the MTL approach used in\". Might add more details about the shift and scale, so that the reader does not have to read another paper.\n6. Table 1, the standard deviations in \"our\" results are not aligned.\n\n----- post rebuttal ----\nThe authors haven't addressed my questions. I would keep my score unchanged. \nOne more comment: I suggest the authors compare to a related baseline SimpleShot [6] that is arguably less complicated.\n\nOverall, given that the novelty and improvement are minor, I think this paper might be not ready at this time.\n\n[1] Lee, Chen-Yu, et al. \"Deeply-supervised nets.\" Artificial intelligence and statistics. 2015.\n\n[2] Xie, Saining, and Zhuowen Tu. \"Holistically-nested edge detection.\" Proceedings of the IEEE international conference on computer vision. 2015.\n\n[3] Chang, Jia-Ren, and Yong-Sheng Chen. \"Pyramid stereo matching network.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[4] Wertheimer, Davis, and Bharath Hariharan. \"Few-shot learning with localization in realistic settings.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n[5] Wah, Catherine, et al. \"The caltech-ucsd birds-200-2011 dataset.\" (2011).\n\n[6] Wang, Yan, et al. \"Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.\" arXiv preprint arXiv:1911.04623 (2019).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WINm80EVQR5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks to the authors for providing such an ensemble approach. This paper aims to find a way to directly utilize representations with the classification layer(s) to obtain better performance.  The ensemble method is able to create an ensemble of classifiers. And the ensemble achieves the new state-of-the-art results in a few-shot setting, comparing to previous regular and ensemble approaches.\n\nThis topic is very straightforward and would be very easy for the audience to understand. While the results might not be that convincing enough. The biggest concern is the contribution of this paper, to be more specific, the proposed method might not be useful and might need to be tuned in other few-shot settings. The mini-ImageNet and tiered-ImageNet results are good, while the authors could provide more evidence to show its strength and how to balance the computation and model performance. For the experimental setup, it is good and reproducible. However, when digging deeper, the reason for the ensemble is that we want to find a way to calculate the features through different classifiers, maybe this is because a single classifier is not able to learn all the features from the images at once. But why it is necessary to use this approach (it also needs pre-training) instead of using a more powerful network to achieve a similar performance?\n\nIt is really good to see those analyses on Single Encoder multiple Representation, Multiple Encoder Multiple Representation, and Selection of Encoders and Representations for the Ensemble. It would be suggested if the author could give a detailed interpretation of the selected layer and how it could be used in other settings. \n\nThis paper is well-written, with not many typos.\n\nThe topic is inspiring and interesting while it is not clear how the ensemble could help FSL tasks. The improvement is not obvious and the results are not enough. Also, it would be better the authors could provide more analysis about why this ensemble works. It would be better the authors could give an analysis of the hyper-parameters of the proposed method. For example, in 5.3 Selection of Encoders and Representations for the Ensemble, τ = 0.93, but how the model performs when τ is different and how we could find an appropriate τ when doing ensemble. The authors should provide enough support to justify the validity of the methods and why this method is worth doing in comparison with other methods. Also, it is worth discussing other aspects such as flops, params, etc. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an ensemble method to few-shot learning. 
Although the introduced method yields competitive results, it is fair to say it is more complicated than much simpler algorithms and does not necessarily perform better. Given that ensembling for few-shot learning has been around for a while, it is not clear that this paper will have a significant audience at ICLR. 
Sorry about the bad news, 

AC. 

Decision: Reject