Pruning Feature Extractor Stacking for Cross-domain Few-shot Learning
Authors: Hongyu Wang
Eibe Frank
Bernhard Pfahringer
Geoff Holmes
Venue: Accepted by TMLR ()

Abstract: Combining knowledge from source domains to learn efficiently from a few labelled instances in a target domain is a transfer learning problem known as cross-domain few-shot learning (CDFSL). Feature extractor stacking (FES) is a state-of-the-art CDFSL method that maintains a collection of source domain feature extractors instead of a single universal extractor. FES uses stacked generalisation to build an ensemble from extractor snapshots saved during target domain fine-tuning. It outperforms several contemporary universal model-based CDFSL methods in the Meta-Dataset benchmark. However, it incurs higher storage cost because it saves a snapshot for every fine-tuning iteration for every extractor. In this work, we propose a bidirectional snapshot selection strategy for FES, leveraging its cross-validation process and the ordered nature of its snapshots, and demonstrate that a 95% snapshot reduction can be achieved while retaining the same level of accuracy.

Published in Transactions on Machine Learning Research (04/2025)
Pruning Feature Extractor Stacking for Cross-domain Few- shot Learning
Hongyu Wang
wonggn.wangghuakaatoocom
Deportment of Computer Science University of Waikalo
Eibe Frank
cbbe.frankddusacattoo.com
Department Computer Science University of Waikato
Bernhard Pfahringer
lermhard.ppacriingerruuukkk.
Department of Computer Science University of Waikato
Geoffrey Holmes
geafrrey.holmesstuaaiaa ac..z
Department of Computer Science Universily of Waikato
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= p499mXacI
Abstract
Combining knowledge from source domains to learn efficiently from few labelled instances in a larget domain is a transfer learning problem known as cross-domain few-shot learning (CDFSL). Feature extractor stacking (FES) a state-of-thh-art CDFSL method that main- tains a collection of source domain fcature extractors instead of a single universal extractor. FES uses stacked generalisation to build an ensemble from extractor snapshots saved dur- ing target domain fine-tuning. It outperforms several contemporary universal model-based CDFSL methods in the Meta-Dataset benchmark. However, it incurs higher storage cost be- cause it saves snapshot for every fine-tuning itcration for every extractor. In this work, we propose a bidirectional snapshol selection strategy for FES, leveraging its cross-validation process and the ordered nature of its snapshots, and demonstrate that a 95% snapshot reduction can be achieved while retaining the same level of accuracy.
1
Introduction
Deep learning methods generally require a large amount of labelled training data to obtain high predictive performance when training model for a new target domain. Cross-domain few-shot learning (CDFSL) attempts to address this by transferring knowledge gained from one or more source domains, with plenty of labelled training instances, the target domain, for which only few labelled training instances are available. The target domain may only be weakly related to any particular source domain. CDFSL aims to achieve high accuracy when predicting labels for new unlabelled instances in the target domain the so-called "query bey"-by aggregating knowledge from the source domains and placing emphasis those most relevant to the small labelled target domain training set-the so-called "support set". In practice, to evaluate CDFSL methods and compare them to each other, a large collection of query-and-support set pairs is sampled from a large labelled dataset such as CIFAR10, and average predictive performance on the query sets is reported. Each auery-and-suppor set pair is called an "episode".
Most recent top-performing CDFSL algorithms, such URL (Li et al. 2021 ), its TSA-based variant et al. 2022), and FLUTE. (Triantafilouu et. al. 2021), derive universal model from source domains and

---

Published in Transactions on Machine Learning Research (04/2025)
then fit it to support set using small set of trainable parameters. In contrast, FES (Wang et al, 2024), CDFSL method based on stacked generalisation (Wolpert 1992], maintains a collection of source domain-specific models instead of a universal one and produces a linear combination of model snapshots FES outperforms
saved during support set fino-tuning that have been evaluated through cross-validatte several contemporary universal model-based methods on the Meta-Dalase benchmark (Triantafillou et al. 2020), achieving the current state of the art (Wang et al., 2024). FES can be applied with semi-supervised STC classifiers (Wang et al. 2023) to further improve its accuracy using additional unlabelled data. FES also exhibits good interpretability as each snapshot's weight reflects its contribution to the learned task. However, it has high storage cost as it saves a snapshot for every fine-tuning iteration for every model. In this paper, we address this shortcoming by taking inspiration from the results in Wang et al. (2024), which show that only a minority of snapshots are assigned significant weights: most are assigned trivial weights and have no meaningful impact on predictions individually.
The key observation for the work presented here is that snapshot pruning in FES can be framed as feature subset, selection problem where the logits provided by every snapshot are the input features for the linear combination trained in the stacking approach- the "stacking classifier". Wrapper-basse based approaches using greedy scarch methods such as forward selection or backward climination can be applicd to such problems by using cross-validation performance to determine the relevance of feature subsets (Kohavi & John 1997). However, naive greedy search yields an infeasibly large search space when applying FES with a non-trivial number of snapshots, as it evaluates combinations of a selected subset with each of the remaining features, yielding runtime that scales quadratically in the total number of snapshots. Fortunately, we can exploit the ordering of snapshots, provided by the sequence of fine-tuning steps for each extractor, to limit the set of available choices, only evaluating small pool of potentially relevant candidates in each step of the search. This strategy can be interpreted as a form of linear forward selection (Gütlein et al. 2009
We propose a bidirectional snapshot selection (BSS) strategy for pruning FES that performs linear forward selection with an initial candidate pool comprising snapshots at the beginning and the end of every model's fine-tuning. When a candidate is selected, the pool is replenished with the candidate's adjacent snapshot in fine-tuning. Given a fixed number of extractors, BSS runtime scales linearly in the total number of snapshots. We show BSS outperforms several baseline strategios and achieves over 95% average snapshot reduction while retaining the same level of accuracy as unpruned FES in both supervised and supi-supe settings.
2 Related Work
We first review work in CDFSL before covering FES. Lastly, we consider wrapper-based feature selection. To keep this review concise, we do not discuss other, potentially orthogonal, machine learning approaches for settings where it is expensive to label data, such as semi-supervised learning, exemplified by the classic self-training approach (Chapelle et al. 2006), or active learning, exemplified by uncertainty sampling (Lewis)
& Gale 1994) and query by committee (Seung et al. 1992).
2.1 Cross-domain few-shot learning
Early few-shot learning work is in-domain (Vinyals et al. 2016 Snell et al. 2017 Finn et al. 2017 y where source and target domain classes are different partitions of the same dataset. Guo et al. (2020) showed that simple transfer learning outperforms these methods in CDFSL. Meta-Dataset (Triantafillou et al. 2020 is benchmark for evaluating CDFSL methods, originally containing eight domains, ilsvre__012, omniglot, aircraft, cu birds, dtd, quickdraw, fungi, and vgg flower, and two target domains from which episodes can be sampled: traffic__sig sign and mscoco. Throc additional target domains-mnist. cifar10, and cifar100~ were added by Requeima el al.] (2019), and a further five, namely, CropDisease, EuroSAT, ISIC, ChestX, and Food101, were added by Wann et al. (2024). Note that only target domain tasks qualify as cross- domain. Using established terminology, an algorithm's accuracy in the target domains represents its "strong generalisation' (SG) performance, while that in souree domains, even when measured on episodes sampled from data of these domains that has been held out, represents performance in "weak generalisation" (WG).
2

---

Published in Transactions on Machine Learning Research (04/2025)
CDFSL can be divided into three stages: pretraining, fine-tuning, and inference. The pretraining stage fits model(s) to source domain data. This stage generally has abundant training data in one or multiple source domains. Some CDFSL methods pretrain a model specifically on each source domain, while others pretrain one model on all source domains. The fine-tuning stage fits the pretrained model(s) to few-shot target domain data. The inference stage uses the fine-tuned model(s) for predictions in the target domain.
Early work in CDFSL includes SUR (Dvornik al. 2020 URT (Liu et al., 2021], CNAPs (Requeima et al.
2019) simple CNAPs (Bateni et al. 2020), and transductive CNAPs (Bateni et all 2022). URL [Li
et al.
2021 and FLUTE (Triantafillou et al. 2021 are newer methods that outperform these algorithms Meta- -atasttt URI. distils a universal model by optimising it to match feature and logit output of source
domain extractors using source domain instances. Given an episode, feature projection attached at the end of the universal model is fine-tuned on the support set with a nearest-centroid classifier. In contrast, FLUTE trains a universal template with source domain-specific batch normalisation layers and a set of convolutional layers shared by all source domains. Given an episode, a set encoder (Zaheer et al., 2017) blends the batch normalisation weights based on the predicted likeliness of the support set to the source domains. The blended weights are fine-tuned on the support set.
Task-specific adaptors (TSA) (Li et al. 2022). originally proposed for URL, are architectural components specifically designed to improve fine-tuning for CDFSL Given a feature extractor, these adaptors are attached to its convolutional layers. Multiple adaptor configurations are viable, and Li et al. (022 found residual channel-wise projections to be most effective, fine-tuned along with the feature projection from URL. We use
TSA with URL as the few-shot reference method because it outperforms the original URL and FLUTE (Li
et al. 2022).
We also use TSA for obtaining the feature extractor snapshots in FES, as in Wang et al. (0024)
2.2 Feature extractor stacking
FES Wang et al. 2024 composes an ensemble of source domain-specifie feature extractor snapshots fine- tuned on the support set, using stacking to estimate each snapshot's performance via cross-validation and combine all snapshots linearly. To oblain snapshols in FES for a given episode, each feature extractor is combined with a classifier and fine-tuned on the support set, and a snapshot of the extractor and classifier is saved before any fine-tuming and after each fine-tuning iteration. To yield a linear combination of the snap- shots by training the stacking classifier, pross-validation is used: the support set is split into two partitions using stratified cross-validation, and these two partitions alternate as training and test splits to produce logits, where an extractor and its classifier are fine-tuned one split and produce logits from the other split after each fine-tuning ileration. These logits are used as training data for the weights of the stacking classifier, constituting a linear combination of snapshots. The trained stacking classifier is then applied to combine the logits of the snapshots obtained by fine-tuning on the full support set to obtain classifications for the query set. The weights of the stacking classifier can be visualised as a heatmap to indicate each snapshot's impact on final predictions. In essence, FFS is a stacked generalisation (Wolpert, 1992) method adopting stratified two-fold cross-validation and treating each snapshot as a separate base model.
Wang et al. 2023) showed that self- trained centroids (STC) can be applied to adapt FES for semi-supervised learning and improve its accuracy by utilising unlabelled instances. Here, a semi-supervised CDFSL episode comprises a labelled support set and an unlabelled set pertaining to the same classes. The supervised fine- tuning procedure is first performed using the labelled set to obtain fine-tuned feature extractor(s) with which the labelled and unlabelled sets are then converted into feature vectors. Self-training (Rosenberg et al., 2005) is applied to the centroid-based classification head based on the extracted fcature vectors: 1) the centroids of labelled feature vectors are used to soft-label the unlabelled feature vectors with pseudo labels, 2) the unlabelled feature vectors are aggregated to obtain class centroids by forming weighted average using their soft pscudo labels, and 3) the two sets of class centroids from the labelled and unlabelled feature vectors are combined using a simple arithmetic average, leading to semi-supervised self-trained centroids for classifying query instances. STC can be considered a generalisation ion of the "inference only" baseline in Ren et al. (0018).
For FES, STC is applied to each extractor during both cross-validation and full support set fino-tuning. During pross-validation STC is applied using the training split as the labelled set, and test split logits produced by the self-trained centroid classifier are used as training data for the FES stacking classifier. Tn
3

---

Published in Transactions on Machine Learning Research (04/2025)
1
snapshots
selection
2
snapshots
selection
3
snapshots
selection
4
snapshots
selection
Figure 1:
Bidirectional snapshot selection: The candidate pool is marked with light circles. The arrow marks the selected smapshot yielding the highest decrease in cross-validated loss. Its neighbour replaces it in the pool.
short, compared to supervised FES, semi-supervised FES with STC uses logits obtained from STC classifiers instead of supervised centroid classifiers whenever logits are needed.
2.3 Wrapper-based feature selection
Greedy feature subset selection using wrapper methods that apply cross- validatioo popularised by sem- inal work presented in Kohavi & John (1997). Commonly, it is implemented using forward selection as the search strategy, it starts with an empty subset of features. Then, given set of candidate features, it separately includes each candidate in the existing subset and evaluates the subset's features in concert using cross-validation. The candidate whose inclusion leads to the highest cross-validatte performance is selected into the feature subset. Intuitively, the feature with the highest cross- pross-validation performance on its own is first selected into the empty subset, then the one with the highest cross-validation performance in concert with the first selection is selected into the subset, and so on. Commonly, the search proceeds until no can- didate improves coss-validation performance when added to the existing feature subset, and this subset is returned as the resulting feature selection. This approach of evaluating a candidate jointly with the existing subset is more robust against correlated features than evaluating the candidate independently, as feature needs to yield the highest improvement of the existing subset's predictive performance to merit its selection. Note that consecutive FES snapshots in fine-tuning are inherently correlated.
Lincar forward selection is variant of forward selection that only evaluates limited pool of candidates each step Güttlein et al. 2009]. Given feature set without explicit order, this method ranks the features by their individual cross-vall validatiio performance and initialises pool of user-specified size with the top- ranked features. During the scarch, this pool can be optionally replenished by the remaining features in ranked order. Gütlein et al. (2009) found linear forward selection is faster, finds smaller subsets, and sometimes improves accuracy compared to naive forward selection. As FES snapshots the same extractor are implicitly ordered because they are sequentially generated during fine-tuning, this search technique is a natural fit for computationally efficient pruning of FES ensembles.
3 Pruning FES with BSS
We now present bidirectional snapshot selection (BSS) for pruning FES ensembles, which is shown as pseudo code in Algorithm [ and visualised Figure [ In essence, BSS initialiscs an empty selection and a pool of candidates containing the first and last fine-tuning snapshots of each extractor; it iteratively selects the candidate snapshot from the pool into the selection that leads to the highest reduction in vross-validation
4

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 1 Bidirectional snapshot selection
Imput: Support set iustances X and labels Y, K extractors Pt,..., 2x
Parameter: fine-tuning iterations
patience
Output: Snapshot subset &
1: Split the support set into X1, X2 and Y1, Y2 using stratified cross-validation
2: for E [1, K] da
3: Fine-tune 62 using X; and Yi and save snapshots
@ CVV, N.O CV2 1
1 CV2
4: Pine-tune ON using X2 and Y/2 and save snapshots @ k.0
K.J
5: end for
Let selected subset 8 2, temporary subset = 03 hest loss foest oc, pationce value PV P, candidate pool P [(R, E
[1, K]. E (H),
7: while # Ø A PV 20 do
8: Let temporary best valsstvali loss feemy
9:
for p E P do
10:
Apply ( (p) to X2 to extract logits L
11:
Apply e {p]] to Xi to extract logits LeV.
12:
Train FES stacking classifier weights wev using D and Y1 and evaluate it using L L and Y2 to obtain loss tov.
13:
Train PES stacking classifter weights W using C and Y/ and evaluate it using 1. 1. and Y1 to obtain loss tcv. if fev, + tev, < fremy then
14:
15:
Let best candidate p, keemp = tovi lov--
16:
end if end for
18:
Let T T=TU P=
( U Pres U Sucs) VT.
19:
if Eeomp < baesr then
20:
Let 8 3= T,, facst Eeemp: PV P.
21:
else
Let PV -V
23:
end If
24:
end while
25:
return &
loss when incorporated into the FES ensemble, replenishing the pool with adjacent unexplored snapshots; the iterative process stops when no snapshot in the pool reduces cross- volssvvaliottin loss or no further snapshots remain, and the selection is returned as the final ensemble.
Given a support set containing instances X and labels Y, K source domain feature extractors D OK. each to be fine-tuned for iterations, and a patience value P E N, BSS returns a subset of the K x +J++) snapshots.
BSS leverages the 2-fold stratified cross-validation performed by FES to guide the search, where the support set is split into two partitions of instances, which alternate to serve for extractor fine-tuning and logit extraction, leading to two partitions of logits. Fine-tuning extractors is generally the most computationally heavy process of FES, and BSS can utilise its outcome without extra cost. BSS alternates the two logit partitions to use one to train the FES stacking classifier weights and the other as input to the trained stacking classifier to obtain leta-logits. The meta-logits are used with their correspouding labels to compute cross-validated cross-entropy, and losses of the two partitions are summed to represent cross-va validation loss of the evaluated subset. Selection of subsets is based on minimising this cross-validation loss.
The search starts with an empty subset and initialises its candidate pool with the first and last snapshots of each extractor's fine-tuning process, i.e., 2 x K snapshots. Then, it evaluates each candidate by measuring cross-validation loss on an expanded version of the existing subset (initially empty) to which the candidate has been added. The candidate that leads to the highest decrease in cross-validation loss is selected into the subset and removed from the pool; an adjacent snapshot is used to replenish the pool. This snapshot is the selected candidate's successor and it is only added to the pool if not already in the subset. For brevily, in Algorithm D given a snapshot of the k-th extractor at the j-th iteration b = (k,j), its predecessor set is denoted as Preş {(k.j 11) if
at the beginning of fino-tuning or predecessor if it at the end,
1 2 0 else Ø, and its successor set is denoted as Sue, = [(ks + 1)} if j +1 S J else Ø.
If the patience parameter P has value zero, the search terminates, and the current subset is returned if no candidate in the pool decreases cross-validation loss when added to the subset. For P > 0, the process only terminates if no loss decrease achieved in P consocutive steps. In this case, the candidate leading to the lowest loss increase is tentatively selected even if a previous subset achieved a lower loss, and its selection is only confirmed when later step within the patience limit achieves lower loss than the previous best subset..
5

---

Published in Transactions on Machine Learning Research (04/2025)
Termination returns the subset with the lowest cross-validation loss explored by the search. Depletion of the candidate pool also leads to termination of search.
Once BSS returns a subset, the final FES model is trained on this subset only: weights for the stacking classifier are trained on cross-validation logits of the snapshots in the subset; when producing predictions, these weights are used aggregate the output of the corresponding subset of snapshots fine-tuned using all support set data.
BSS is proposed to address the complexity issue of naively selecting from all snapshots, which can be prohibitive for long snapshot sequences. At the beginning, selecting from all snapshots has to perform J times as many evaluations as BSS to select each snapshot. On average, the complexity for the mumber of evaluations is O(KJ) for selecting each snapshot from all snapshols, while it is O(K) for selecting each snapshot using BSS.
4 Experimental Setup
We perform evaluation on Meta-Dataset with the extended set of target domains and adhere to the official sampling method to generate 600 few-shot episodes for each domain: each episode contains 5 to 50 classes, up to 100 support set instances per class, up to 500 (potentially class-imballnced) support set instances in total, and 10 query set instances per class. The sampled episodes are cached ensure that all pruning strategies are evaluated using the same episodes.
We use FES with TSA fine-tuning to evaluate pruning strategies. FES hyperparameters are consistent with Wang et al. (2024), and TSA hyperparamcters are consistent with Li ct al. (2022). ResNet18 (He et al. 2016) extractors are used throughout. Each of the eight. source domain extractors is fine-tuned with TSA for 40 iterations, leading to 8 x 41 = 328 snapshots in total. We also evaluate semi-supervised FES with TSA fine-tuning and STC, using the hyperparameters from Wang et al. (2023).
We evaluate three baselines: 1) FES without pruning, 2) exhaustively searching through all extructor com- binations, where selecting an extractor leads to inclusion of all of its 41 snapshots, and 3) exhaustively searching through all combinations of the extractors' final snapshots. For reference, we include a URL model with TSA fine-tuning (Li et al. 2022). We evaluate BSS with patience 0 as our main method. In an ablation study, we evaluate two unidirectional snapshot selection strategies that are akin to BSS but only search in one direction: unidirectional forward snapshot selection (UFSS) initialises its candidate pool with only snapshots from the beginning of fine-tuning and conversely, unidirectional backward snapshot selection (UBSS) initialises its candidate pool with only snapshots from the end of fine-tuning. We also evaluate BSS, UFSS, and UBSS with patience values from i.c., patience, to 328, which ensures depletion of the candidate pool. We perform paired I-tests with p 0.05 using accuracy of individual episodes to determine statistical significance of differences in accuracy on individual datasets.
5 Results
We first compare BSS to the baseline strategies. Then, we visualise pruning results as heatmaps. Follow- ing this, we conduct an ablation study with UFSS and UBSS and consider varying patience values. We then evaluate BSS pruning in semi-supervised learning with STC. Lastly, we analyse runtime and memory consumption of FES pruned with BSS and discuss the trade-off between accuracy and cost.
5.1 Comparison to baselines
Table I compares BSS to the basclines. Accuracy and percentage of snapshots remaining after pruning, i.c., SI/(K x ()), are reported with 95% confidence intervals. Mean WG and SG performance and ranks are also reported. Each baseline is evaluated against BSS in paired I-tests. detected, i.., < 0.05, indicates better BSS accuracy, whereas o indicates better baseline accuracy.
statistical significance is
Compared naive FES without pruning, BSS achieves higher average SG accuracy while the two exhaustive baselines perform worse. Exhaustive extractor search only removes approximately half of all snapshots. On

---

Published in Transactions on Machine Learning Research (04/2025)
the other hand, while exhaustively searching through combinations of the last snapshots achieves impressive ion, only 8 snapshots out of the total 328 are considered, its SG performance is the weakest among the four methods. BSS achieves the highest accuracy in SG while achieving over 95% snapshot reduction. The URL model with TSA fine-tuning achioves the highest WG accuracy but the lowest SG accuracy comparod to the pruned and unpruned FES methods.
We have also evaluated several other baselines but they proved inferior to the strategies presented in Table I they are omitted. These baseline strategies include: 1) exhaustive search involving only each extractor's s snapshot with the lowest individual cross-validation loss, 2) greedy forward selection with at most one snapshot per extractor, performed by including all snapshots in the initial pool and when a snapshot is selected, removing all candidates of the same extractor, 3) greedy forward selection with at most one snapshot per extractor but allowing replacement of a snapshot in the selected subset with another snapshot of the same extractor, 4) ranking all snapshots by individual cross-validation loss, iterating through the ranking, and selecting snapshots whose inclusion in the subset decreases vross-validation loss, and 5) pruning by applying L1 regularisation to FES stacking classifier training to force the weights of some snapshots to zero, hereby removing them.
5.2 Visualisation
Figure 2 presents stacking classifier weight matrices as heatmaps. In each pair of graphs, the left one represents average weight assigned to each snapshot by unpruned FES, and the right one represents the version by FES pruned with BSS. (Unselected snapshots receive zero weight.) Results are averages from the 600 episodes of the corresponding target domain. Five domains are presented in Figure g and analysed in detail: traffic sign, mscoco, ISIC, ChestX, and Food101. The other five target domains are omitted as their umpruned weight matrices focus on one or two source domains where weights change gradually as fine- tue-tuning progresses, while the pruned version focuses on the same domains with sparse weights concentrating al the end of fine-tuning
In traffic sign, three source domains are focused on: ImageNet, textures (dtd), and quickdraw. The un- pruned version features gradual snapshot weight changes in these three source domains, while the pruned version heavily focuses the snapshots the end of fine-tuning, which is sufficient according to Table T the pruned version achieves significantly higher accuracy according to a paired 1-test.
ImageNet is the main focus in both insocco and Food101. Unpruned FES focuses heavily on early snapshots in mscoco, while pruned FES focuses on the final snapshot. In Food101, the unpruned version focuses on two snapshots in particular: the one before any fine-tuning and the one after two fine-tuning iterations, whereas the pruned version focuses on the former only. Table I shows that the pruned version performs significantly better in these two domains than the unpruned version. Pruning also makes FES heatmaps more concise and thus more interpretable in these two domains well as the aforementioned traffic_ sign.
FES appears unfocused in ISIC and ChestX, likely because there is no strong relationship between any source domain and these two target domains. Pruned FES has weights concentrated at the beginning and end of finettnnng, and Table | shows that BSS pruming significantly decreases average accuracy in these two domains. The unpruned heatmaps show significant weights assigned to snapshots in the middle of fine- tuning, which may be the reason for this. The results indicate that the existence of target domains that are related to the source domains can lead to a smoother and more monotonic cross-validatiin accuracy curve during fine-tuning, resulting in concentration of the best snapshots at the beginning or the end of fine-tuning, which BSS can leverage to achieve significant pruning without tangible accuracy loss. When such properties are potentially absent, like in ISIC or ChestX, BSS is more likely to suffer a loss in accuracy.
Overall, these figures show that BSS able to remove many snapshots while retaining and sometimes even enhancing interpretability. Moreover, snapshots at the beginning and the end of fine-tuning can both be relevant to a task. However, may occasionally fail to include relevant snapshots observed during the middle of fine-tuning.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: BSS compared to the baselines: pruning (full), exhaustively searching through extractors (EE), and exhaustively searching through the last snapshot of each extractor (EL). URL extractor with TSA fine-tuning is evaluated for reference.
B35
URL-TSA
Dataset
radio
Hevro
56.33 35.32.1111
3-761:0.22
06.22211:16
281:1 1.13
50 1441-33
56-321:1
1.4020.03
56. 60-7881:1111
amniglat
93-09120-70
13:3950.80
83.33200664
252.0
47. 11.5011:-00
83.333..000
1.2820.04
90.88.00441
Atreratt
87-75220.78
1.882:0
87-6120-79
87
58-15:11-31
67-722:00
1.3220.03
68-43.0..0
70-774:00
1.30=0.08
76.174:0
2.95=:0.15
91.85-+0.70
01.92-0.66
69.36++1.50
1.45-0.03
02.07+0..5
WC uYa
79.88
4.99
79-74
79. 70-78
48.10
79.
1.34
80.14
WC rank
2:97
2:95
2.02
3.02
3:11
tzuffic
66::0.9G
2.85
0..90=0.07
80::00
95.731:0.04
1.45=0.00
2:00
54.54111.03
3.38
64.1421.00
54 28::
50::1.50
53.633:
1.51 20.00
maitt
98 28.899-00.52
6.35
9T.06=0.48
85::0.03 52
40-73-4 1.20
97.014:00 0:1::00000
1.19=0.05
00.352
cifar10
78.32+0..59
2.65+ 21
78.14-0.87
T8:24 78.24+00.87 +0.87
48 33- 1.49
78.62+ 0.
1.24-0.03
70.04 70.04+0.76
cifar100
70.04+1.00
3.65+0.38 0.33
70.36+1.05
T0.32+1105
49 00-40++...0 1.54
70.62+1.00
1.49=0.63
70.26 1.00
CropDiseare
57.800 01.66200668
4.572 35
66.15-0.08
ST 91.9111.0000 88
08..331111 1:211:200
67 87.85200000
1.5820.03
8- 20077
EureBAT
68 58-1552--844
2.502 20
65.85 05.6520.633
68 00.97100.81
41 15:111335 1:33
80 89:372 0.
1:21 12.22
88.82.9.554
ISIC
48 92220.94
3.42
40. 0
49 162.0
56. 984
47.58 47:522:00
1:4720.03
2.0.58
ChestX
27 07-04.20.58
3:2012:
27.07-0.01 0.00
27 651.0 60
56 1 600
00.00210.050
1.3720.03
27 17.00.7
Pood101
1..1121.12
2.0411
05.2221:11
50. 3511511111
48
1-111
54.0 54.8821.18
1.6120.03
53.37 1:221
69.44
3.55
51.43
68
3G rank
2.76
2.80
2.84
3.00
3.40
09.
(a) Unpruned traffic sign
(b) Pruned traffic sign
tra
(c) Unpruned mscoco
(d) Pruned micocc
0005
5006
Unpruned ISCC
(f) Pruned ISIC
cooo
coess
lea
Unpruned ChesttX
(h) Pruned ChestX
sudddion
325
Unpruned Food101
(I) Pruned Food101
Figure 2: Pairs of umpruned and pruned stacking classifier weight matrices

---

Published in Transactions on Machine Learning Research (04/2025)
Table Ablation study results with BSS, UFSS, and UBSS, performed with either zero or maximum patience
n88 (BIP)
GPS8 (F)
UP88
UB88
UB8S
(BI)
(FP)
(Ba)
(BaP)
Dotasot
noc
resto
000
raslo
Baaee. 2012
56.32:12
3.820.33
5..421:2
22-841.3
56.021-14 1
6.7.0.33
56.24: 56.241.14 1.
44.022.3
56 0.:32:1.2
7.920.4
06.32:11
32-721.5
omniglot
63-120077
12:320.8
83 00.311..7
31 012.
90.820.00
14.320.8
03.349.45
30 30-3223.
90-220-7
11.5 11.520..8
83.329.7
32-722.2
atroratt
67-810.8
4.920-1
57.74
30-141-6
67.820.8
11-120.4
87-720.8
55-522
57-720.8
5.520-4
87-740.5
29-511.5
cu_ cu_bird
80.0+00.5
4.0=0.3
80.1+0.8
24.0+1.4
19.6+0.5
8.3-0.4
19.8+0.9
44.4+22
70.5+0.5
6.9+0.4
79.9+0.8
66.642.4
did
70.5+0.8
8.2-0.3
76.4+0.5
18.641.4
75.740.5
7.8=0.3
76.0+0.8
80.8221
76.620.5
8.420.33
76.4+0.5
24.341.6
quicledraw
83.620.6
2.920.2
63.0-20.6
16:211.1
83.250..
8.720.3
83.314.8
40.9211
83.420.6
5.620.4
83.420.8
26.011:8
Fungs 19.511.1
3:2-0.33
89.0-11.1
26.0411.6
69.T=111
1.4=0.3
09.511.1
4
89.7:1.1
8.6:20.4
89.611111
92.4119
VRO Rower
01.8+0.7
91.9++0.7
19.9+1144
15.1+0.6
91.740.74
02.010.7
02.010.7
20.8+1.7
WG
79.83
4.99
70.85
28.64
79.68
9.86
79.69
43.91
79.58
6.33
79.83
31.75
WC rank
3.33
8.85
3.87
8.62
3.43
3,39
Lrafiie sign
T1.0111+
84.5+10
96.740.0
2.340.8
aucoco
1..5
54
63.0+1.0
3.3-0.2
68.84100
76.0=11
63-7=1.0
2.7:0.22
cifar100
3.5+0.3
10.0+111
T0.6+1.1
6.340.4
70.4+1.1
63.2+2.5
Creppiieeae
88.01.0.7
4.620.4
86.110.7
31441177
85.410
5.7.0.2
87.710.7
00-013
87.8
4.310.4
87.910.77
10.811.5
KaroSA'
89.32:0.6
2.520.2
89.310.6
20-741.6
85.720-8
3.120.2
88-21...
53.3228
89.410.6
2.120.2
89.310.5
23-121.6
ISIC
45.3+00.9
3.4=0.22
40 40.0400.00
38.8+2.0
47.2=0.9
4.120.2
49.450.9c
66.1
47.7+1.0
2.6+0.1
48.0+00.9
37.1+1.
CheatX
27:0#:0.6
3.2=0.33
27.020.00
08.0422.
25.420.54
3.0=0.2
27.010.0:
60.842.
27.020.0
2.2:0.88
27.220.6
40.023.11
Food101
6..721.1
2.920.2
65.721.1
20.121.7
53.32111
3.720.1
04
51.722.8
54.621.1
6.00333
05.221.1
91.222.1
avg
Table 3: Ablation study comparing BSS to selecting the number of random snapshots in each episode
Dataset
BSS
random
ilsvre 2012
16.35::1.16
50.87:11.35
omniggot
93.0910.70
91.8010.78
aircraft:
87.75:0.79
80.85:: 1.33
eu birds
80.05:0.88
75.26+1.10
dtd
78.48:0.82
72.13+0.94
quickdraw
83.56:10.59
77.14:1..01
fungi
69.50:11114
60. 14:1: 1.55
VEg flower
01.85
90.42100.3
WG
79.83
71.84
traffic_sign
85.66:10.95
80.211:1.07
mscoco
51.541::103
18.541:1.15
mnist
96.54100.51
cifar10
78.32:10.89
68.9311..22
cifar100
70.6441.06
03.57+1..29
CropDisease
87.96=00.68
00.31
EuroSAT
89.33:10.64
85.86+0.722
ISIC
48.32-10.94
07.54+0.90
ChestX
27.01::0.58
27. 1.58
Food101
55.7111.12
48.59.1. 1...1.26
SG avg
69.44
65.32
5.3 Ablation Study
Table 2 shows BSS, UFSS, and UBSS results. Each strategy is performed either with no patience or with maximum patience, which guarantees candidate pool depletion.
For both, zero and maximum patience, BSS achieves higher accuracy and snapshot reduction than the two unidirectional variants. This again indicates that snapshots relevant to task can be at both the beginning and the end fine-tuning and adopting the model at the end of fine-tuning may not be best in every CDFSL scenario. Given the same strategy, maximum patience leads to better accuracy but lower reduction. The mean accuracy difference between zero and maximum patience is small for BSS, while it is more substantial for UFSS and UBSS. Given these observations, we conclude that BSS without patience achieves the best balance between accuracy and reduction.
Table 3 compares BSS to a random selection strategy which matches the number of snapshots selected by BSS in each episode but samples the snapshots randomly from a uniform distribution. BSS significantly outperforms the random strategy in all domains except ChestX, where no statistically significant difference is found. This indicates that. BSS outperforms the uniformly random null hypothesis in most cases. This
9

---

Published in Transactions on Machine Learning Research (04/2025)
Table Results FES with wwihhutt STC and/or BSS
BSS STC
full sup
full STC
BSS sup
acc
ilavre 2012
16.72.11.15
4.07:10.25
56.22:1.14
56. 13.633::1.13
56.35:11.16
3.7610.28
omniglnt
93.68.00.67
13.14:10.82
93.33.10.44
94.00.10.58
93.09:10.70
12.29
aircraft
88.01=0073
4.88:
87.64=0.79
87 87:8880077
87.76=0.79
4.86=20336
birda
1.81+0.30
79.93=0.85
$0.17+00.82
80.05+0.88
1.03+0.26
dtd
76.46:10.7
3.34
76.21
76.18:10.77
06.48:0082
3.23
quickdraw
3.39
83.38.10.61
83.85.10.55
83.56
2.93.10.18
fungi
70.86:
3.07+:0.25
09.43+1.11
70.88-11.00
1. 1.14
3.24::.26
vgg Rlwer
92.3410066
5.88 0..50
01.9210..60
92. 02.43110.61
91.85.00.77
5.62 0..49
WG avg rank
80.29
5.39
79.74
80.25
79. 89.83
4.99
traffic sign
96.48-0.99
2.774:.22
84.90=0.97
85,99 $5.999--.996
85.6640.995
2.85 0..31
1..01
3.49.00.28
54.1411.0F
55.4811.01
5..5411.03
3.38 0.222
mnist
97.31 10.47
6.77:10.64
97.08-10.48
97 97.371100 97.37110.48
96.99110.52
6.35
cifar10
78.96:0.84
2.69+0011
78.14=0.87
78.61+00.8
78.3240.88
2.65
cifar100
T1.18+1.03
3.58+0034
70.35=1.05
70.87
70.61#1.06
3.55
Diseaee
85.83++0.66
0.45+0.35
88. 13-0.8.
89.07:10.64
87.9640.66
4.67
EuroSAT
89.56+0.61
2.51+0.21
88. 38.85+0.63
89.02+=0.61
89.33#0.61
2.50=0.20
ISIC
49.66:10.95
3.191:0.20
49.52-+0,93
51.43:10.93
48.32:10.94
3.42-10.17
ChestX
27.16.10.61
3.00.10.17
27.67
28.4410.64
27.04
0..33
Food101
65.99-11.11
3.1240..0
55. 65.2221.11 1.111.11
15.47:
55.71 1.12
2.94
SG
70.08
3.56
69.39
70.18
69.44
3.55
rank
2.22
2.83
2.25
2.71
Table Computational cost the following methods, applied with iterations of TSA fine-tuning: a single URL universal feature extractor, unpruned FES using the eight Meta-Datace source domain extractors, and the same FES setup with BSS pruning. The values are obtained using the largest traffic sign support set (497 instances) in evaluation.
single extractor
FES full
FES BSS
fine-tune time prune time stacking time
9.30s
150.95a
150.95a
0.33s
0.03s
frozen parameters
1IMx8
11Mx6
trainable parameters
1.5M
1.5M
12M
1.5Xx6
9M
stored parametors
11M + 1.5M
12.5M
11Mx8 + 1.5Mx328
580M
11Mx6 1.5Xx51
142.5M
stacking classifier parameters training GPU memory
328
51
8.2G
8.3G
8.3G
also suggests that ChestX is different from the other target domains, likely due to its lack of similarity to all source domains.
5.4 Semi-supervised Pruning with STC and BSS
Table shows FES results with BSS pruning (P = 0) and STC semi-supervised learning with 1000 unlabelled instances. The results are compared those obtained from supervised FES without pruning, as well FES with only STC semi-supervised lcarning and FES with only BSS pruning. FES with BSS and STC is sot the reference method in Table 4 and the other methods are evaluated against it using the paired I-test. A indicates significantly better reference method performance and indicates significantly better performance from the other method.
Table 4 shows that semi-supervised learning with STC consistently improves FES performance in both pruned and unpruned settings. Semi-supervised FES pruned with BSS achieves slightly lower average strong gencralisation accuracy than unpruned counterpart, but with only 3.56% of the snapshots on average. Also, five of its wins are statistically significant while there are only three statistically significant losses, and its average rank is higher too.
10

---

Published in Transactions on Machine Learning Research (04/2025)
5.5 Runtime and Memory Consumption
Table 5 shows computational cost and accuracy of a single URL extractor, unprumed FES, and FES with BSS pruning. The ResNet18 architecture contains 11M parameters, while TSA adaptors for one such model contain 1.5M parameters. As only the adaptors are fine-tuned, only onc copy of the 11M basc paramoters needs to be saved for each extractor, and each snapshot only contains the 1.5M TSA adaptor parameters. Note that we use the largest traffic sign support set, which has 497 training instances, and Table represents an upper bound in our experimental setup. Fine-tuning and stacking are performed on an NVIDIA A6000 GPU, while pruning is performed on Intel Core i7-6700K CPU combinations more quickly than a GPU. For this particular episode, pruning leads to 51 snapshots belonging to six extractors, and the other two extractors can be excluded entirely. Note that FES can fine-tune its extractors sequentially, so its GPU memory requirement is close to using a single extractor of the same architecture.
CPU iterates through snapshot
The main time consumption of FES due to its fine-tuning process, as the extractors need to be finc-tuned both for cross-validation and on the full support set. Time consumption of pruning significant but not overwhelming, constituting approximate 20% runtime increase compared to the FES version without pruming. However, the pruned version requires approximately 75% less storage space for its parameters compared to the unpruned version. BSS scales well with the number of snapshots as discussed in Section 3 while exhaustive search scales exponentially with the number of snapshots considered. Likely due to the larger number of training instances in the episode considered, the ratio of remaining snapshots, i.e., 51 out of 328, is greater than the traffic_sigh average reported in Table [T] Users are advisod to refer Tables [ and to consider trade-off among accuracy, runtime, and storage cost when choosing among a universal model, unpruned FES, and FES pruned with BSS.
6 Conclusion
We propose BSS, a pruning strategy for FES based on linear forward selection that leverages its cross- validation process and the ordered nature of FES snapshots. BSS achieves over 95% average snapshot reduction while retaining state-of-the-at supervised and semi-supervised accuracy. It outperforms several baselines and unidirectional snapshot selection strategies, indicating that relevant snapshots can be found at both the beginning and the end of fine-tuning. We also show that visualisation of BSS models aids analysis of larget domains.
References
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In TEFE/CVP CCFF Conference on Compuler Vision and Pallern Recognition, Seattle, WA, USA, pp. 14481-1444 14490. Computer Vision Foundation / IEEE, 2020.
Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, and Frank Wood. Enhancing few-shot image classification with unlabelled examples. In JEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, pp. 1597-1606. IEEE, 2022.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. MIT PRess, 2006.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi-domain rep- resentation for few-shot classification. In 16th European Conference on Computer Vision, Glasgow, UK, pp. 769 786. Springer, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In 34th International Conference on Machine Learning, Sydney, NSW, Austrulia, pp. 1126 1135. PMLR, 2017.
Yunhui Guo, Noel Codella, Lconid Karlinsky, James V. Codella, John R. Smith, Kate Saenko, Tajana Rosing, and Rogério Feris. A broader study of pross-domain few-shot learning. In 16th European Conference on Computer Vision, Glasgow, UK, pp- 141. Springer, 2020.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Martin Gütlein, Eibe Frank, Mark A. Hall, and Andreas Karwath. Large-scale attribute selection using wrappers. In Symposium on Compulational Intelligence and Data Mining, Nashwille, TN, USA, pp. 332- 339. LEEE, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, pp. 770-778. IEEE Computer Society, 2016.
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artif. Intell., (7(2)))27 324, 1997.
David D Lewis and William Gale, A sequential algorithm for training text classifiers, sigir'94: Proceedings of the 17th annual international acm sigir conference rescarch and development in information retrieval, 1994.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few- shot classification. In IEEE/CVF International Conference on Compuler Vision, Montreal, QC, Canada, pp. 9506 9515. IEEE, 2021.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task-specific adapters. In IEEE/CVF Conference on Computer Vision and Pattern Recognation, New Orleans, LA, USA, pp. 7151 7160. IEEE, 2022.
Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal represen- (ation transformer layer for few-shot image classification. In 9th International Conference on Learning Representations, Virtual Event, Austria. OpenReview net, 2021.
Mengye Ren, Eleni Triantafilouu, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tencnbaum, Hugo Tarochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In 6th Inter- national Conference on Learning Representations, Vancowver, BC, Canada. OpenReview.nee net, 2018.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Advances in Neural Information Processing Systems 32, Vancoueer, BC, Canada, pp. 7957-7968, 2019.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self- telf-training of object detection models. In 7th IEEE Workshop on Applications of Computer Vision IEEE Workshop on Molion and Video Computing, Breckenridge, CO, USA, pp. 29 36, IEEE Computer Society, 2005.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp- 287-294, 1992.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems 30, Long Beach, CA, USA, pp. 4077 4087, 2017.
Eleni Triantafillon, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning to Iearn from few examples. In 8th International Conference on Learning Repre- sentalions, Addis Abaha, Ethiopia. OpenReview.ned. 2020.
Eleni Triantafillou, Hugo Larochelle, Richard S. Zemel, and Vincent Dumoulin. Learning a universal template for few-shot dataset generalization. In 38th International Conference on Machine Learning, Virtual Event, pp. 10424-10433. PMLR, 2021.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29, Barcelona, Spain, pp. 3630 3638, 2016.
Hongyu Wang, Eibe Frank, Bernhard Pfahringer, and Geoffrey Holmes. Self-trained centroid classifiers for semi-supervised cross-domain few-shot learning. In 2nd Conference on Lifelong Learning Agents, pp. 481-492 PMLRR 2023.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Hongyu Wang, Eibe Frank, Bernhard Pfahringer, Michael Mayo, and Geoff Holmes. Feature extractor stacking for cross-domain few-shot learning. Machine Learning, 113(1):121-158 2024.
David H. Wolpert. Stacked generalization. Neural Networks, 6(2):241-255) 1992.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabés Póczos, Ruslan Salakhutdinov, and Alexan- der J. Smola. Deep sets. In Advances in Newral Information Processing Systems 30, Long Beach, CA, USA, pp. 3391 3401, 2017.
13


Reviews:
[{"review_id": "HzAauPK345", "reviewer": "Reviewer_t8j9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. The introduction of Bidirectional Snapshot Selection (BSS) for pruning Feature Extractor Stacking (FES) is an effective solution to reduce storage requirements while preserving model accuracy.\n\n2. BSS reduces storage and computational costs of FES, which makes it suitable for real-world applications with limited resources.\n\n3. Comprehensive Evaluation: The method is rigorously evaluated on the Meta-Dataset benchmark across various target domains, highlighting its general applicability.\n\n4. The use of heatmap visualizations to display the weights assigned to snapshots offers clear insights into the model’s behaviors, which enhances interpretability.\n\n### Weaknesses:\n\n1. Based on my understanding, FES, by retaining and combining model snapshots based on their performance through cross-validation (query and support sets), is very similar to the settings of active learning in selecting the most informative examples. A more detailed discussion of active learning, including methods like uncertainty sampling or query-by-committee, could improve the paper’s literature review by situating FES within the broader landscape of model selection and optimization strategies. This would provide readers with a clearer understanding of how FES relates to and differs from active learning approaches.\n\n2. The manuscript lacks a detailed theoretical explanation of why the bidirectional pruning strategy works effectively, which could provide a deeper understanding of its underlying principles.\n\n3. Limited Discussion on Edge Cases: The performance of BSS in domains where source and target domains are weakly related (e.g., ISIC and ChestX) is not thoroughly explored, and the paper does not address potential limitations in such scenarios.\n\n4. My main concern is the contribution and novelty of the study. While FES demonstrates strong performance, its significant computational overhead can be a major drawback compared to other CDFSL methods, especially when the performance lifts are not substantially greater. Given the high storage costs and computational complexity, further studies focused on FES may face challenges in terms of efficiency and practicality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes an effective pruning strategy, Bidirectional Snapshot Selection (BSS), for Feature Extractor Stacking (FES) in Cross-domain Few-shot Learning (CDFSL). FES has demonstrated state-of-the-art performance by stacking multiple model snapshots from source domain extractors. However, FES suffers from the significant storage costs due to the retention of snapshots after each fine-tuning iteration. The authors address this issue by leveraging the ordered nature of the snapshots and a bidirectional pruning method, which can reduces the snapshot storage by over 95% while maintaining the same level of accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Based on the similarities between FES and active learning in selecting the most informative examples, I suggest expanding the literature review to include a more detailed discussion of active learning. Specifically, including methods such as uncertainty sampling or query-by-committee could help situate FES within the broader landscape of model selection and optimization strategies.\n\n2. The manuscript would benefit from a deeper theoretical explanation of why the bidirectional pruning strategy is effective. A more formal analysis of the underlying principles would help strengthen the contribution and allow readers to better understand the mechanism behind its success.\n\n3. The manuscript could explore and discuss the performance of BSS in scenarios where source and target domains are weakly related (such as ISIC and ChestX). A more thorough examination of potential limitations in these edge cases would provide a more comprehensive view of the method’s applicability across various domains.\n\n4. My main concern is with the overall contribution and novelty of the study. While FES demonstrates strong performance, its high computational overhead compared to other CDFSL methods is a significant drawback, particularly when the performance improvements are not significantly larger. The study appears to have limited applicability, as it is entirely dependent on FES. I suggest that the motivations for focusing on FES be clearly stated. Merely claiming it achieves state-of-the-art performance is not sufficient justification for the study. A more detailed explanation of why FES is chosen and its practical relevance would strengthen the contribution.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5r6v1vLasn", "reviewer": "Reviewer_nLF2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nPruning unnecessary information from multiple extractors for meta-classifiers is a reasonable approach. The proposed method is both simple and straightforward, effectively determining which extractors should be selected to maintain performance.\n\n**Weaknesses**\nAccording to the TMLR guidelines, a key criterion for evaluation is whether the problem addressed holds substantial importance for the broader research community. In this case, the paper focuses on reducing storage and inference costs in a CDFSL setting, but the significance of this contribution is not entirely clear. Furthermore, the small-scale experiments presented are not sufficiently convincing to establish that pruning snapshots genuinely offers meaningful computational benefits.\n\nRegarding the methodology, why does the search start with only the first and last snapshots instead of considering the entire set of snapshots? While a bidirectional search certainly reduces the computational effort during pruning, examining all available snapshots could potentially allow for more aggressive pruning without harming overall performance. Naturally, this broader approach would increase the complexity of the pruning process during training, as we would need to evaluate which candidate pruning step leads to the greatest reduction in cross-validation loss. However, in my view, the primary advantage of pruning lies in improving inference throughput, even if this comes at the expense of a more intensive training process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The problem addressed in this paper is cross-domain few-shot learning (CDFSL). The work explores the use of a weighted averaging classifier derived from multiple source domain-specific feature extractors. In this category of methods, many existing approaches rely on a straightforward weighted averaging across numerous feature extractors, which can be computationally expensive. To address this overhead, the paper proposes a pruning approach that selects a subset of feature extractors while maintaining classification performance.\n\nThe proposed method is relatively simple: it iteratively selects candidates from multiple feature extractors based on a classification metric obtained through cross-validation. Despite its simplicity, this approach is effective. Experiments on the Meta-Dataset demonstrate that the proposed pruning strategy achieves comparable generalization performance while significantly reducing the number of parameters and improving inference throughput.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I suggest that the manuscript should be revised. In Section 3.2, the first sentence states that Figure 1 and Algorithm 1 present the main idea, but it does not provide sufficient detail, making it difficult to understand the overall method. While the following sentences do describe the method in more detail, I personally recommend revising the first paragraph to clearly present the main idea and emphasize an overview of the method. This improvement would enhance the clarity and reliability of the manuscript.\n\nAlthough pruning 328 snapshots into a smaller subset is a promising approach for significantly enhancing inference throughput, its broader importance and applicability could be better demonstrated through experiments on a larger number of snapshots or by utilizing more complex feature extractors. Additionally, replacing ResNet-18 with a more robust and powerful feature extractor could more effectively highlight the advantages of the proposed pruning method in terms of memory and throughputs.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6iO5eT2CYD", "reviewer": "Reviewer_RGkX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nThe author proposes a method that applies novel bidirectional snapshot selection strategy to feature stacking based cross domain few shot learning method. Since the feature stacking method uses all the feature snapshot during training which takes much memory, the motivation to reduce the snapshots by feature selection is well understandable.\n\nThe author conducts experimental evaluation on the several standard image recognition benchmarks and compares with multiple benchmark methods. The proposed method demonstrates comparable accuracy to the baseline that uses full snapshots with much less computational memory.\n\nThe author conducts several ablation study, visualizations and measurement of the computation cost to investigate the effectiveness of the proposed method.\n\n\nWeakness\n\nThe proposed method depends much on the existing methods (e.g. Wang et al., 2024) and the novelty is small.\nFurther, since the paper only explains the proposed bidirectional snapshot selection strategy in detail, it is difficult to understand the whole few shot learning pipeline without reading such previous papers.\nIt would be better to make the paper self-contained by adding the explanation about the problem setting of the cross domain few shot learning in pretraining, fine-tuning, and inference phase and about the feature extractor stacking approach in more detail.\n\nThe motivation why adopting bidirectional strategy is unclear. It would be more discussed in the introduction and the proposed method section.\n\nAs for the experimental comparison, it seems last snapshot method (EL) demonstrates not so bad accuracy with lower cost than the proposed bidirectional snapshot selection strategy. Is there clear benefit of the proposed method compared to EL?\n\nSince the motivation is the reduce in the inference cost, I would like to see the comparison with the full snapshot method with some sparse method in the final combination weight training stage (e.g. l1 regularization). Further, from Figure 2, most of the weights are assigned to first and last snapshots. Therefore, I would also like to see the comparison to the exhaustive search through first and last snapshot as an extension of EL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel cross domain few shot learning method. Based on existing feature stacking approach, the author proposes to reduce the candidate feature extractors by applying bidirectional feature selection method. The proposed method demonstrates comparable accuracy to the full feature case with much lower number of feature extractor snapshot used in the inference.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More detailed discussion on the motivation, full overview of the problem setting and the novelty compared to the existing work would improve the clarity of the paper.\n\nDiscussion about the advantage of the proposed method on the last snapshot method (EL) and comparison to more baselines written in the weakness section would help to make the advantage of the proposed method more evident.", "broader_impact_concerns": "The paper has no broader impact discussion. But since this is an algorithmic paper, I think the potential negative impact of the paper is small.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The core contribution of this paper is a bidirectional search method to prune snapshots from the feature extractor stacking (FES) method of Wang et al for cross-domain few-shot learning (CDFL). It's a pretty simple approach that seems quite effective at reducing the memory and inference footprint of FES.

The reviewers were mainly concerned about the novelty and significance aspect of the work. It is a rather specific approach to a fairly esoteric method. For those interested in CDFL and FES specifically, this could help with practicality. There may not be much interest from a more general audience. More work on general insight and a larger scale could help with this. Otherwise, if we restrict ourselves to the narrow scope of improving storage costs for FES, then the claims are justified.

I would like to see a comparison against a random baseline, where a random subset of snapshots are selected equal to the size of the subset found by BSS.
Decision: Accept (poster)