FSL-QuickBoost: Minimal-Cost Ensemble for Few-Shot Learning

Authors: yunwei  bai
bill yang cai
zangwei  zheng
shiming  chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 04:53:35

Abstract: Few-shot learning (FSL) usually trains models on data from one set of classes, but tests them on data from a different set of classes, providing a few labeled support samples of the unseen classes as a reference for the trained model. Due to the lack of training data relevant to the target, there is usually high generalization error with respect to the test classes. Some existing methods attempt to address this generalization issue through ensemble. However, current ensemble-based FSL methods can be computationally expensive. In this work, we conduct empirical explorations and propose an ensemble method (namely QuickBoost), which is efficient and effective for improving the generalization of FSL. Specifically, QuickBoost includes an alternative-architecture pretrained encoder with a one-vs-all binary classifier (namely FSL-Forest) based on random forest algorithm, and is ensembled with the off-the-shelf FSL models via logit-level averaging.  Extensive experiments on three benchmarks demonstrate that our method achieves state-of-the-art performance with good efficiency.

Main paper: 
 related works: 
 Few Shot Learning. Few-shot learning classification algorithms can be broadly categorized into data-based approach, algorithmbased approach and model-based approach [56] . Among data-based algorithms, Gao et al. [21] , Hariharan and Girshick [26] , Mishra et al. [37] , Schwartz et al. [46] , Verma et al. [52] , Wang et al. [56] expands the training dataset by, for example, combining object characteristics across base classes, but the generation may involve bias since certain combinations can be invalid [57] . Among algorithmbased approaches, meta-learning [4, 18, 32, 45, 49] learns a good initialization of model parameters, and model-based approaches mainly involve metric-based learning, where models learn an embedding space that contrasts input samples at a lower dimension [10, 11, 34, 48, 50, 53, 59] . Most of the methods are purely deeplearning-based, which are data-hungry and time-consuming to train [54] . In particular, meta-learning algorithms like MAML [18] can be unstable to train [2] . Moreover, due to the challenging nature of FSL, most methods have limited performance. Therefore, we are motivated to improve the performance of existing FSL algorithms while keeping the ensemble procedure simplistic.In this paper, we examine a few typical FSL models to be ensembled with our methods, or namely the Prototypical Network (ProtoNet), Matching Network (MatchNet), Relation Network (Re-lationNet), FEAT, DeppSet and Infopatch. The FSL model encoders include the Conv4, Res12 or Res18 structures. The Conv4 encoder [50, 59] is a relatively simple 4-layer convolutional neural network (CNN) [31] . Res12 and Re18 [59] are of a more complex Resnet [27] structure. The Prototypical Network [48] is similar to the Matching Network [53] . Both of these models produce embeddings for pairs of support set sample and query, before measuring their similarity scores using either the euclidean distance or cosine similarity. When there is just one shot for the support set, the Matching Network and Prototypical Network are equivalent; when there are multiple shots, the Prototypical Network considers the average embedding of all support set samples within one class, while the Matching Network considers the support samples individually. The Relation Network [50] is similar, except that the algorithm uses a neural network to calculate the degree of similarity between two image embeddings. FEAT [59] , DeepSet [59] and InfoPatch [34] have a similar set-up, but FEAT and DeepSet add in transformative layers (e.g., self-attention-based mechanism [51] ) to produce more flexible embedding. InfoPatch tries to produce more accurate embeddings by blocking part of the query images, making the task harder [34] . Ensemble. Ensemble can be achieved using different approaches, including independently constructed ensembles and coordinated constructed ensembles [15, 28] . For an independently constructed ensemble, boosting ensemble method combines weak learners to produce a strong learner; random forest is also a type of independently constructed ensemble that combines the predictions of decision trees through majority voting [5] . The collection of tree structured classifiers can be decision trees [6, 7] . For coordinated constructed ensemble, Adaboost [15, 20] algorithm constructs new hypotheses incrementally. In the intersection field of FSL and model ensemble, Dvornik et al. [16] proposes to train multiple few-shot models with a loss function that encourages diversity via KL-divergence and cooperation via cosine similarity. Another work, Liu et al. [35] 𝐸 3 𝐵𝑀 learns and combines an ensemble of epoch-wise Bayes models. There are also other works involving ensemble of features or classifiers for few-shot learning [1, 3, 9, 25, 33, 58] . The major disadvantage of these ensemble schemes lies in the extra computation overhead and complication in implementation introduced. Importantly, most of these methods (e.g., Dvornik et al. [16] , Liu et al. [35] ) involve training of ensemble itself. Orthogonal to these methods, our design focuses on efficiency, enabling easy and quick performance boost via test-time-only ensemble, without involving any extra training of the original model. 
method 3.1 problem formulation: 
 FSL classification tasks commonly follow an 𝑁 -way-𝐾-shot set-up; 𝑁 represents the number of classes being classified and 𝐾 denotes the number of labeled samples per class. These 𝑁 × 𝐾 samples, denoted as 𝑆 = {𝑥 (1)1 , 𝑥 (1) 2 , . . . , 𝑥 (𝑁 ) 𝐾 -1 , 𝑥 (𝑁 )𝐾 }, are referred to as the support set. It is commonly assumed that ℎ is trained on a set of base classes 𝐶 𝑏 with complete supervision information, and then tested on a disjoint set of novel classes 𝐶 𝑏 (i.e., 𝐶 𝑏 ∩ 𝐶 𝑛 = ∅). ∀𝑐 ∈ 𝐶 𝑛 , 𝑐 contains limited supervision information, or just a few labeled samples. These few labeled samples serve as the support set 𝑆 during testing [14, 48, 50, 55, 59] .During training, an FSL classifier ℎ is presented with a labelled support set comprising samples from 𝑁 classes, and a query 𝑞 subsampled from the 𝑁 classes, and ℎ has to learn or predict which category 𝑞 belongs to by referring to 𝑆. The 𝑁 classes are a subset of 𝐶 𝑏 . During testing, ℎ has to predict classes of 𝑞 based on labels of 𝑆, where both 𝑞 and 𝑆 are sampled from 𝐶 𝑛 . Note that according to inductive FSL set-up [3] , the support set sample labels from 𝐶 𝑛 are not used during training.Overview. There are two important characteristics of ensemble. Firstly, ensemble consumes extra computation resources, which needs to be considered in practical applications. Secondly, in general, to maximize the effectiveness of ensemble, ensembled models need to be as accurate as possible individually, while possessing differences in logit output against one another's, so as to achieve a complementary effect. As such, it is desirable if we can have ensemble which achieves both computation efficiency and the complementary difference. Therefore, guided by these principles, we aim to develop an effective ensemble scheme for FSL, which boosts single classifiers' performance at minimal cost, thus minimizing the pain points of both FSL and ensemble.We introduce the FSL-QuickBoost ensemble scheme, where we restrain our scope to just one FSL-Forest model ℎ 2 for ensemble with a trained FSL model ℎ 1 . ℎ 1 comprises an encoder Φ 1 and a classifier 𝑡 1 , and ℎ 2 also has an encoder Φ 2 and a classifier 𝑡 2 . Essentially, we try to achieve the aforementioned complementary difference at both feature level and classifier level. At the feature level, we adopt different encoders for ℎ 1 and ℎ 2 respectively. At the classifier level, we apply logit weighted-averaging of a random-forest-based [5] one-vs-all [39] (a) Pretraining of Encoder. Some works propose that the mainstream meta-learning algorithms in FSL are sub-optimal [14, 23, 40, 41] , and they demonstrate that a fine-tuned pretrained encoder can achieve equivalent or even more superior performance as complex meta-learning models [13, 14, 23, 41] . Essentially, while simple, pretrained features can be effective for FSL classification. Following the practise, as the first step, we pretrain an encoder Φ 2 on base classes 𝐶 𝑏 , via standard cross entropy loss minimization [14] . Specifically, given a dataset 𝐷 𝑠 consisting of data from base classes 𝐶 𝑏 , we obtain the model parameters 𝜃 * for an encoder Φ 𝜃 * 2 :EQUATIONWe use Φ 𝜃 * 2 (Φ 2 in short) for the subsequent preparation of dataset. (b) Preparation of Dataset. After obtaining Φ 2 , we sub-sample positive and negative image pairs {𝑥 1 , 𝑥 2 } from the base classes 𝐶 𝑏 , and produce pairwise supervision labels as:EQUATIONFor each pair {𝑥 1 , 𝑥 2 }, where 𝑥 𝑖 is ∈ R 𝐹 , we calculate the squared elementwise feature difference, denoted as diff(𝑥 1 , 𝑥 2 ). In the context of 𝑁 -way-𝐾-shot tasks, it is:EQUATIONfor 𝑛 ∈ [1, 𝑁 ] and 𝑓 ∈ [1, 𝐹 ]. Note that 𝐹 is the total number of channels in Φ 2 (𝑥), and 𝑞 is the query. We save tuples 𝐷 ≔ {(diff(𝑥 𝑖 , 𝑥 𝑗 ), 𝒚)} for 𝑖, 𝑗 ∈ [1, 𝑀], where 𝑀 denotes the total number of samples collected from base classes 𝐶 𝑏 . These tuples are stored as data files for subsequent training of the random forest classifier 𝑡 2 . (c) FSL-Forest. We perform training on a random forest classifier 𝑡 2 with the prepared input 𝐷. The random forest classifier is named "FSL-Forest". The classifier constitutes a set of decision trees, each taking in the squared feature difference and returning a similarity relation score for the input image pair. Each decision tree partitions the input features recursively, based on some threshold, such that samples with the same labels or target values are grouped together. This procedure is also known as "impurity minimization".Let data at node 𝑚 of the decision tree be denoted as 𝑄 𝑚 with 𝑁 𝑚 samples. Here, each element of 𝑄 𝑚 , 𝑑, is each element-wise squared difference of two input vectors, originated from the pair of images for comparison. Let 𝑑 𝑚 = 1 𝑁 𝑚 𝑑 ∈𝑄 𝑚 𝑑. The "impurity" 𝐻 (•) is defined as the following mean squared error function [6] [7] [8] :EQUATION)For each decision tree in the random forest classifier 𝑡 2 , let 𝜔 denote the model parameters. Let 𝑒 𝑚 be each candidate's split threshold value. Let threshold 𝑒 𝑚 partition a node into two subsets, which are respectively denoted as 𝑄 left 𝑚 (𝜔) = {𝑑 |𝑑 < 𝑒 𝑚 , 𝑑 ∈ 𝑄 𝑚 } and 𝑄 right 𝑚 (𝜔) = 𝑄 𝑚 \𝑄 left 𝑚 (𝜔). The training procedure of each decision tree is tantamount to the following equation [6] [7] [8] :EQUATIONEQUATIONOutput scores 𝑡 2 (•) from the random forest are based on the average of individual output of each decision tree parameterized by 𝜔 * , in a set of decision trees [6] [7] [8] .The rationale of the design is that each value of the squared feature difference represents the visual strength of a particular feature difference in the original data pair. If certain discrepancies corresponding to important features are strong enough, the data pair tends to be of different classes. (d) Ensemble. Given two pretrained encoders Φ 1 and Φ 2 , and two classifiers 𝑡 1 and 𝑡 2 for ℎ 1 and ℎ 2 respectively, assuming that the output from ℎ are normalized, ensemble returns𝛼 1 • ℎ 1 (𝑞, 𝑆) + 𝛼 2 • ℎ 2 (𝑞, 𝑆) = 𝛼 1 • 𝑡 1 • Φ 1 (𝑞, 𝑆) + 𝛼 2 • 𝑡 2 • Φ 2 (𝑞, 𝑆)at the logit level. Note that 𝑞 is the query, and 𝑆 is the average embedding of each support set class. Besides, 𝛼 1 is the testaccuracy of ℎ 1 , and 𝛼 2 is the test-accuracy of ℎ 2 on few labelled support samples. Essentially, for ensemble with a high-accuracy FSL model, QuickBoost gives more priority to the original predictions. For an 𝑁 -way-𝐾-shot support set 𝑆 and a query 𝑞, 𝑡EQUATION2 ), . . . , diff(𝑞, 𝑥 (𝑁 ) 𝐾 -1 ), diff(𝑞, 𝑥 (𝑁 ) 𝐾 )} for 𝑥 as a sample from one of the novel classes 𝐶 𝑛 . Here, diff(•, •) denotes the set of element-wise squared difference between two features. The final ensemble prediction for each query corresponding to one support set is rendered as:EQUATION)where the normalization function norm(𝑢) for a real value 𝑢 ∈ 𝑈 and 𝑈 ∈ R 𝑁 is defined as:EQUATION)Note that the ensemble scheme is plug-and-play, happening only during inference stage, where the trained FSL model does not need to be trained further. 
theoretical insights: 
 In essence, FSL suffers from a high degree of approximation and estimation error, and ensemble can tackle such a pain point precisely. Core Challenge(s) in FSL? The core challenge is high difference between the best hypothesis obtainable via existing training data, and the best hypothesis in theory. Such a difference is mainly caused by the lack of supervision for the target tasks.We define a hypothesis space H , the expected risk 𝑅(ℎ) as∫ ℓ (ℎ(𝑥), 𝑦)𝑑𝑝 (𝑥, 𝑦) = E[ℓ (ℎ(𝑥), 𝑦)]. Empirical risk as 𝑅 𝐼 (ℎ) as 1 𝐼 𝐼 𝑖=1 ℓ (ℎ (𝑥 𝑖 ) , 𝑦 𝑖 ),where 𝐼 is the total number of available training samples. Furthermore, we have ĥ = arg min ℎ 𝑅(ℎ) as the function that minimizes 𝑅(ℎ); ℎ * = arg min ℎ∈ H 𝑅(ℎ) as the function in H that minimizes 𝑅(ℎ); ℎ 𝐼 = arg min ℎ∈ H 𝑅 𝐼 (ℎ) as the function in H that minimizes 𝑅 𝐼 (ℎ). Essentially, ĥ is the most ideal optimal classifier, ℎ * is the best classifier within the hypothesis space considered, and ℎ 𝐼 is the classifier we can obtain via thorough empirical risk minimizing training over available training data. The expected risk difference between ℎ 𝐼 and ĥ can be written as: E 𝑅 (ℎ 𝐼 ) -𝑅( ĥ) = E 𝑅 ℎ * -𝑅( ĥ)E app ( H) + E 𝑅 (ℎ 𝐼 ) -𝑅 ℎ * E est ( H,𝐼 ) . (9)Here, E app (H, 𝐼 ) is the approximation error, and E est (H ) is the estimation error [55] .The approximation error, representing the expected risk difference between ℎ * and ĥ, arises when the optimal classifier does not exist within the considered hypothesis space. In deep learning, this error can be caused by both the lack of relevant supervision data and limited representational ability of deep neural networks. Generally, in FSL, as the target-task classes are novel, the chance of the optimal classifier existing further outside of the hypothesis space is high compared to standard classification problems [55] . This error is also known as the representational issue in Zhou [62] .The estimation error arises when there is a difference between the obtainable trained classifier and the best classifier within the hypothesis space considered [55] . This error can be caused by getting stuck at local minima during training, which relates to computational and statistical issues in Zhou [62] . Generally, the issues can be mitigated with training on a large amount of labelled data [29, 39, 55] . FSL remains challenging given the lack of such access. Why Ensemble? In short, ensemble is a simple-yet-effective solution which tackles the aforementioned issues (i.e., representational, computational and statistical) precisely [62] .Given FSL models {ℎ 1 𝐼 . . . ℎ 𝑇 𝐼 }, where ℎ 𝐼 (𝒙) returns the logits of 𝒙, simple averaging gives the combined output 𝑃 (𝒙) as:EQUATION)Suppose the underlying true function to learn is ĥ(𝒙), and 𝒙 is sampled according to a distribution 𝑝 (𝒙). The output of each learner can be written as the true value plus an error item 𝜖, which is a result of both E app (H, 𝐼 ) and E est (H ) i.e., ℎ 𝑖 𝐼 (𝒙) = ĥ(𝒙) + 𝜖 𝑖 (𝒙), 𝑖 = 1, . . . ,𝑇 .The mean squared error of ℎ 𝑖 𝐼 can be written as:EQUATION)The averaged error made by the individual learners is:EQUATIONThe expected error of the combined learner (i.e., ensemble) is:EQUATION)If we assume that the errors 𝜖 𝑖 's have zero mean and are uncorrelated, i.e.,𝜖 𝑖 (𝒙)𝑝 (𝒙)𝑑𝒙 = 0 andEQUATIONfor 𝑖 ≠ 𝑗, 𝑗 ∈ [1,𝑇 ], we can have:EQUATION)This suggests that ensemble error is smaller by a factor of 𝑇 than the averaged error of the individual learners [62] . When 𝜖 increases in magnitude, which is usually the case in FSL, err(ℎ) increases, leading to a larger improvement brought by ensemble. From the perspective of approximation and estimation error, on the one hand, approximation error can be reduced via better representational ability of ensemble, which can cover a larger hypothesis space; on the other hand, estimation error characterized by local minima can be reduced through combining multiple local search results. The errors can be reduced best when ensemble involves accurate classifiers sharing little correlation (See assumption of Formula 15), which is also the complementary difference we refer to earlier. 
experiments 5.1 experimental configurations: 
 Benchmark Datasets. We use three popular FSL datasets, miniImagenet, tieredImagenet and Cifar-FS [49] . Both the miniImagenet dataset and the tieredImagenet are sampled from the imagenet dataset [44] , and the Cifar-FS dataset are sampled from the Cifar-100 dataset [30] . In both the miniImagenet dataset and the Cifar-FS dataset, there are 100 classes (train/validation/test classes = 64/16/20) [32] . Each of the class contains 600 images. In the tieredImagenet dataset, there are more categories (train/validation/test classes = 351/97/160) [59] . Implementation Details. For pretraining of the feature encoders, we use a randomly initialized Res18 [27] model, whose last fullyconnected layer. For the input data, we only apply normalization without any augmentation. We use a stochastic gradient descent (SGD) optimizer. The batch size is 128. The momentum is 0.9. The weight decay rate is 0.0001. The initial learning rate is 0.1. We train the model for 100 epochs, and the learning rate is reduced by a factor of 0.1 during the 30th, 60th and 90th epoch. The pretraining only involves the respective train split of the FSL datasets to prevent data leakage. After obtaining the image feature encoder for each of the dataset, we combine pairs of image feature vectors obtained via the pretrained encoder, and merge two 512-vectors into one 512-vector by computing their squared vector difference. Each merged 512-vector corresponds to either a positive label representing same-class pairs, or a negative label representing different-class pairs. We sample 12,800 pairs, with half of them as positive pairs and half of them negative pairs. Finally, we train a random forest classifier implemented in the scikit-learn library [8] using the prepared dataset. We set the number of estimators of random forest to 200, the maximum features to 4 and the random seed to 0. Evaluation Protocols. For evaluation of our method, we test trained FSL models with or without QuickBoost on standard 5way-1-shot and 5-way-5-shot few-shot classification tasks. We use the Res12 and the Conv4 [50, 59] backbones for different FSL algorithms (i.e., ProtoNet [48] , RelationNet [50] , MatchingNet [53] , FEAT [59] , DeepSet [59] , InfoPatch [34] ). For evaluation of ensemble performance, we show the performance of original models before and after ensemble. All the reported accuracy scores of the original methods are based on our reproduced experimental results. The respective official implementations are used for the reproduction. For each reported number, we report the top-1 accuracy score as well as the 95% confidence interval. Note that all the results included are of the inductive set-up, while transductive set-up are of a slightly different problem formulation [3] .Table 1 summarizes the performance of FSL-Forest in QuickBoost, original FSL model and Quickboost respectively on 5-way-1-shot and 5-way-5-shot tasks. The proposed ensemble helps trained FSL models achieve considerable performance improvement, despite the limited performance of FSL-Forest as a standalone classifier (e.g., 42.01% for 5-way-1-shot tasks on the miniImagenet). For example, on Conv4-based Prototypical Network trained on the miniImagenet, QuickBoost can improve its performance by 5.99% on 5-way-1-shot tasks, and 7.77% on 5-way-5-shot tasks. In general, as expected, the improvement is more salient on 5-way-5-shot tasks compared to 5-way-1-shot tasks, which is likely because of the amplification of benefits introduced by ensemble; when the number of shot size increases, QuickBoost can return more predictions to average out bias with greater efficacy. While FSL classification is challenging, our simplistic approach is competitive on state-of-the-art (SOTA) methods. Table 2 summarizes the SOTA methods in few-shot classification 5-way-1-shot or 5-way-5-shot tasks. Non-ensemble methods for comparison include DeepEMD [60] , TADAM [40] , MetaOptNet [32] , LEO [45] , SNAIL [38] , COSOC [36] and Shot-Free [42] . Ensemble methods include Robust 20 Full [16] , 𝐸 3 𝐵𝑀 [35] , CHEF [1] , MetaOptNet+MIMO [25] and EASY [3] . As indicated, when ensembled with SOTA algorithm like InfoPatch [34] , we can achieve the enhanced SOTA results on the common FSL benchmarks (e.g., 71.29% for miniImagenet and 81.81% for Cifar-FS 5-way-1-shot classification). Importantly, when compared to other ensemble algorithms (e.g., Robust 20 Full [16] , 𝐸 3 𝐵𝑀 [35] ), our approach is capable of better performance despite its little required computation resources and simplicity of usage.Visualization of Ensemble Features. We visualize the attention maps for the two different encoders, which are illustrated by Figure 3 . In the figure, we have 6 pairs of attention maps. In each pair, we have, from left to right column, pairs of the original images, together with their Res12 and Res18 attention. In the first row, we have attention map comparison which is more precise when generated from a Res12 encoder, while the Res18 encoder becomes less precise in the second row. However, in the bottom-right-corner pair, the Res12 attention is inaccurate in identifying the relevant region of interest. In terms of prediction correctness, for the first row, the Res12 encoder is correct in prediction (in a 2-way-1-shot classification) while Res18 encoder is wrong, and the second row accounts for the reverse case (i.e., Res18 encoder is right and Res12 is wrong). This visualization illustrates the complementary effect among different encoders.Note that the attention maps here are different from traditional attention maps [24, 47] , and is named mutual attention map in this work. Instead of highlighting regions of interest in a traditional classification setting, the mutual attention map highlights regions of interest during pairwise images comparisons. Originally, attention map considers the backward-gradient-weighted features [24, 47] . Here, we visualize a set of features Φ(𝑥) = {𝑥 𝑖 } of an original image input 𝑋 , where 𝑖 is the channel index from the output of an encoder Φ. The output feature sum to be visualized for one particular image 𝑥 can be represented by𝑖 𝑥 𝑖 •𝑥 ′ 𝑖 ∥𝑥 𝑖 ∥ • ∥𝑥 ′ 𝑖 ∥ • 𝑥 𝑖 ,where 𝑖 is the channel index, and 𝑥 ′ 𝑖 is per-channel-feature from the image compared. Encoder Combinations. Different encoder combinations can affect ensemble results, which is summarized in Table 3 . As stated in past ensemble research works [62] , in general, the higher the accuracy of each model in ensemble and the difference among the ensemble components, the better the results of ensemble. This phenomenon can be reflected in the table, which are the 5-way-1-shot and 5-way-5-shot results on miniImagenet, by Prototypical Networks with different encoders. Among the three different encoders, Conv4 is the most shallow network consisting of 4 CNN layers.Backbone miniImagenet Cifar-FS tieredImagenet 1-shot (%) 5-shot (%) 1-shot (%) 5-shot (%) 1-shot (%) 5 ). As expected, the better result (60.81% for 5-way-1-shot and 76.86% for 5-way-5-shot) belongs to the the combination of Res12 and Res18. This is because both of these two encoders are accurate on their own while possessing complementary differences against each other. Conversely, combinations involving Conv4 (e.g., Res12 + Conv4: 50.53% for 5-way-1-shot and 66.83% for 5-way-5-shot) tend to have worse performance, since Conv4-based models have a lower independent accuracy (47.79% for 5-way-1-shot tasks).Hyperparameter Tuning. We study different hyperparameters for the random forest classifier of FSL-Forest on 5-way-1-shot miniImagenet classification tasks, as summarized in Table 4 . We observe that FSL-Forest performance can be further improved through tuning of hyperparameters. For example, when we increase the maximum features to consider from 4 to 8 and the number of estimators from 200 to 800, the standalone FSL-Forest accuracy on 5-way-1shot tasks increases from 42.01% to 45.17%. Besides, we observe that the classifier can be further enhanced when combined with new features like cosine similarity, which can help achieve an even higher standalone 5-way-1-shot accuracy score at 46.74%. In brief, QuickBoost has potential to achieve better performance through hyperparameter tuning and alternative designs of input features, but usually at a higher computation cost. Computation and Data Efficiency.FSL-Forest in QuickBoost is computationally efficient. In principle, for small-scale training, traditional machine learning models such as those consisting of simple decision tree stumps can be much more computationally efficient when compared to DNNs, in terms of both time and data [54] . Table 5 summarizes the comparison between the Prototypical Network, 𝐸 3 𝐵𝑀 ensemble algorithm [35] and FSL-Forest. Note that the Prototypical Network computation requirement is also the basic computation required for ensemble algorithms like Dvornik et al. Model Interpretability. We use Local Interpretable Modelagnostic Explanations (LIME) [43] to qualitatively understand the inner mechanisms of the model. LIME allows us to understand the effects of superpixels in test images on the prediction of class similarity. LIME scores are based on superpixels identified via unsupervised image segmentation using Felzenszwalb's graph-based image segmentation [17] . We choose LIME as it can be applied to the QuickBoost FSL-Forest classifier, which is a mixture of gradient and non-gradient based models. Furthermore, recent studies find theoretical guarantees and connections between LIME and integrated gradient methods [22] . We randomly sample two classes (class A and B) and select two images from each class (images A and B), and one image from another class (image C). We then create 2 image pairs, (i) image A and B, (ii) image A and C. We designate the randomly-selected image A as the input image for LIME's perturbations. Using a trained FSL-Forest on miniImagenet, we run LIME on the model while feeding the model with pairs (i) and (ii). We find qualitative results that validate our hypothesis that the random forest classifier is able to identify image features that contribute to similarity scoring. Figure 4 shows samples of our qualitative results.  
conclusions: 
 To conclude, motivated by the nature of ensemble and FSL, we explore empirically how ensemble can be effective in terms of both performance and resource. Our proposed ensemble instance strengthens the original FSL prediction scores with alternative predictions, achieving boost for FSL performance at minimal cost. 
: 
 on the target task [48, 55] . In many real-world scenarios, sufficient labeled datasets are usually not readily available [19, 55, 61] , especially for rare object types. Therefore, the few-shot learning classification algorithms serve to overcome the problem of generalization in the face of such data constraints. It is commonly assumed that FSL models undergo training based on data from a different set of classes compared to those of the test data set [3, 50, 55, 59] . With few-shot training, models can predict classes of data they have not been trained on.However, the data constraint also limits the FSL model performance. In deep neural networks (DNN), lower generalization error is generally achieved by having a large number of labelled training examples coupled with models with sufficient parameters [29, 39, 55] . With less supervision information, local search (e.g., gradient descent) tends to get stuck at local minima [62] . Meanwhile, due to the lack of relevant data, the best hypothesis in the hypothesis space * may not represent the target well [62] . Furthermore, on the one hand, a single model with less parameters can be limited in its expressiveness to represent the target accurately; on the other hand, with limited data, models with more parameters can overfit [39] , composing a dilemma for FSL.To tackle the high generalization error, or the core challenge, in FSL classification, different categories of methods are proposed. Notably, in the data aspect, data augmentation techniques are used to augment the training dataset for FSL models, providing more supervision based on datasets related to the target tasks [21, 26, 37, 46, 52, 56] . In the model aspect, representation learning techniques project training data to lower dimension, setting the similar samples closer together while the dissimilar further apart [12, 34, 48, 50, 53, 59] . Both the data augmentation and the embedding learning techniques tackle the FSL problems mainly via integrating more prior information to FSL models [55] . In the algorithm aspect, metalearning algorithms [4, 32, 45, 49] can learn good model parameter initialization which enables quick adaptation to the new unseen task. However, the augmentation can introduce bias as argued in works like Xu and Le [57] . Meanwhile, model-based embeddinglearning approaches are limited in performance, while algorithmbased meta-learning algorithms can be unstable to train with nested training loops [2] .To analyse the problem further, we define a hypothesis space H . We have ĥ as the most ideal optimal classifier in theory, ℎ * as the best classifier within the hypothesis space considered, and ℎ 𝐼 as the classifier we obtain via thorough empirical risk minimizing over available training data. The expected difference between the risk of ℎ 𝐼 and ĥ in FSL (i.e., the best room of improvement that can be achieved in theory) can be decomposed into approximation error and estimation error, as illustrated in Formula 9 and Figure 1 . When there is sufficient and relevant supervision data, both estimation error (distance between ℎ * and ℎ 𝐼 ) and approximation error (distance between ĥ and ℎ * ) are relatively small, as illustrated in Figure 1 (a). When the relevant labelled data reduce, both of the errors increase. In particular, the estimation error increases due to a higher chance of the estimator ℎ 𝐼 getting stuck at local minima during the optimum search. The approximation error increases due to a decreased amount of labelled data learnt relevant to the target; without enough relevant data, the best target of search ℎ * becomes relatively inaccurate, as illustrated in Figure 1(b) . Ensemble mitigates these two errors. Specifically, despite accessing less labelled training data, the estimation error can be reduced by considering alternative local search results which average out bias of local minima. Meanwhile, the approximation error can be reduced through greater model expressiveness; as ensemble involves more model parameters, it can consider a larger hypothesis space and thus more accurate targets of search [39, 55, 62] , as illustrated in Figure 1(c) .According to existing literature, plenty of empirical and theoretical results have demonstrated the effectiveness of ensemble [62] , in tackling the approximation and estimation errors, which are the precise pain points of FSL. Meanwhile, "how to ensemble effectively" is a question equally worthy of consideration. In fact, ensemble has * Hypothesis space size is defined as the number of hypotheses. A hypothesis is a way of classification that best classifies the training data seen so far, given the representational ability of the function. In general, hypothesis space grows when there are more training data or model parameters [39] . certain preconditions to fulfill in order to unleash and maximize its potential benefits. It also has disadvantages. Specifically, to achieve good ensemble performance, the predictors have to possess a fair accuracy on their own and enough difference against other predictors to achieve complementary difference in their predictions. The predictions can then complement one another well to achieve improved overall accuracy [16, 62] . Besides, as multiple models need to be trained, ensemble can be disadvantageous in terms of computation efficiency and implementation convenience, which are relevant considerations for real-world applications [15, 28, 62] .Therefore, our goal is to reduce generalization error in trained FSL models, boosting their performance via resource-efficient ensemble. In this paper, we design an ensemble method for FSL, called FSL-Quickboost (Quickboost for short). Essentially, QuickBoost involves pretrained features from an alternative encoder and a one-vsall [39] binary classifier to be ensembled with trained FSL models. The binary classifier, called FSL-Forest, is random-forest-based [5] . We choose the random-forest algorithm due to its widely demonstrated decency in performance. Importantly, it is ready for few-shot classification tasks at minimal cost. To implement the FSL-Forest classifier, we directly use the pretrained encoder to produce image features, whose pairwise feature difference are fed as the input features to FSL-Forest. The extensive experiments on standard benchmarks demonstrate the superior performance.Our main contributions can be summarized as follows:• We analyse why ensemble can be a simple-yet-effective solution to the challenging set-ups in FSL. • We propose an ensemble scheme for trained FSL models, which involves a pretrained encoder paired with a randomforest-based FSL classifier. The classifier predicts based on the average among a set of decision tree stumps, each of which makes predictions based on pairwise differences among pretrained data features. The decision trees consider the elementwise value differences of each feature pair, strenghthening initial predictions of the original FSL model. • We provide extensive experiments to show the advantages and potential of FSL-QuickBoost. For example, our FSL-QuickBoost obtains approximately 6%, 6% and 7% performance improvement in 5-way-5-shot tasks on tieredImagenet [44] , miniImagenet [44] and Cifar-FS [30] , respectively. 
 
 Appendix:  


Decision: In this paper, the authors propose a novel ensemble method, QuickBoost, which is efficient and effective for improving the generalization of few-shot learning. After the rebuttal period, the reviewers gave consistent positive ratings. Therefore, I recommend accepting this paper. However, as Reviewers xeS1 and wjY8 pointed out, this paper still has some remaining minor concerns. The authors are required to update their paper according to the reviewers' comments and fix these issues in the final version.

Reviews: review: paper_summary: paper_summary: This work proposed a ensemble learning framework for few-shot learning (FSL). To be specific, the authors proposed to add a random-forest-based model on top of a trained standard FSL model. The added random forest is trained on pairwise feature difference generated on another feature extractor pre-trained on some base classes. While the random forest itself is a weak FSL model, it consistently improves the standard FSL model by a very significant margin on all dataset and model combinations considered in this work.
strength_weakness: strength_weakness: + This work is clearly written. The methodology is described clearly with vivid illustration of the architecture and intuition (Fig. 1).
+ The proposed framework is simple and clean, thus has good potential to be widely used by the community.
+ Empirical results are very strong. The improvements over the baseline FSL models are significant.
+ The authors also provided theoretical analysis and careful ablation study, which are informative, such as the encoder combination results presented in Table 3, which successfully decoupled the effect of introducing the base encoder.
limitations: limitations: - The (d) stage in Figure 2 is kind of confiusing. It looks like the output logits by the QuickBoost module is fed as input to the standard FSL model.
paper_summary: paper_summary: This paper introduces QuickBoost, a method designed to enhance the generalization of Few-shot Learning (FSL). QuickBoost integrates a one-vs-all binary classifier (FSL-Forest) with standard FSL models via logit-level averaging. FSL-Forest utilizes decision tree stumps to evaluate input pairs based on their feature-level elementwise value differences. Tested across three benchmarks, the proposed method demonstrates remarkable efficiency and effectiveness.
strength_weakness: strength_weakness: - The introduction of the ensemble random forest method to the Few-Shot Learning (FSL) task is interesting.    
- The ability to train on a CPU with notable efficiency is particularly impressive.
limitations: limitations: Weakness   
- The comparison in Table 2 seems biased as it incorporates the baseline method InfoPatch[31], which is already effective. Adding QuickBoost to InfoPatch does not clearly demonstrate the distinct advantages of QuickBoost.
- For clearer comparison, it would be beneficial to separate the results for miniImageNet and Cifar-FS into two distinct sections in Table 2, similar to the format used in Table 1. The current combined presentation is confusing.
- It is unclear whether the reported training time of 7.7 seconds for FSL-Forest includes only stage C or the entire FSL-QuickBoost ensemble. Clarification is needed to accurately assess the method's efficiency.
- To better validate the effectiveness of FSL-Forest in identifying relevant features, LIME results should also be applied to other FSL methods for comparative analysis.
paper_summary: paper_summary: This paper introduces a new ensemble method called QuickBoost for FSL to improve the generalization performances of existing FSL models with limited training data available. In particular, QuickBoost includes a deep neural network based classifier and a random forest based classifier and their output logits are averaged as the ensemble output. QuickBoost is an efficient ensemble method as the computational cost of training a random forest based classifier is much lower than that of deep models. Extensive experiments are conducted on three public FSL datasets and QuickBoost shows clear improvements compared to different baseline methods.
strength_weakness: strength_weakness: 1.	The paper is well organized overall and is easy to follow.
2.	The proposed QuickBoost is easy to implement and could be applied to different baseline methods.
3.	The authors provide theoretical analysis which would help readers understand the benefit of ensemble methods. However, the analysis is for general ensemble methods and is not specific to the proposed method. 
4.	The authors conduct experiments on three FSL datasets and QuickBoost is compared with both ensemble and non-ensemble methods to show its effectiveness. Besides, QuickBoost also clearly boosts the performances of several existing baseline methods.
limitations: limitations: 1.	Several parts of this paper, i.e., (c) FSL-Forest and (d) Ensemble, are not well presented and many important details are missing. Some expressions are also confusing. For example, for each decision tree in the random forest, which feature is selected as its input? What is the actual element in $Q_m$ and is $d$ in Eq.4 a scaler or a vector? As the random forest based classifier is the main part of the proposed method, I think the meaning of these variables and the learning procedure of the random forest should be explained more clearly. Moreover, the authors do not formally explain how to obtain the logits of different classes with random forest based classifier as each decision tree only produces a single score. Besides, at line 452, how to obtain the test accuracy of the two classifiers and which part of data is used for test? Also, I feel confused about the sentence at line 455-457.
2.	Although the random forest based classifier is an essential component of the proposed method, not only the important settings about the random forest algorithm (e.g., the depth of the decision tree, the stop criterion) but also the related ablation studies are missing. It is important to illustrate the influence of the hyper-parameters of the random forest algorithm (e.g., the number of trees, the depth of trees, the number of selected features, the number of training samples, different pretrained backbones) on the ensemble performance.
3.	Although QuickBoost achieves preferable empirical results, it is still unclear what makes it a good ensemble method and why the introduction of a random forest based classifier is important. This also weakens the motivation of QuickBoost. The authors claim that different classifiers should be diverse so as to obtain a good ensemble result but the diversity is not shown appropriately in QuickBoost. For example, the authors could analyze the difference between the logits output by the neural network based classifier and the random forest based classifier with some statistical methods, which would provide a better guidance for us to select appropriate models for ensemble. I think the visualization results with only a few cases are not sufficient for explaining this point. 
4.	In table 4, as the random forest based classifier is based on the image features from a trained res18 network, I think it is unfair that the training time of this res18 network is not included for comparison.