Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings
Authors: Heng Wang
Tan Yue
Xiang Ye
Zihang He
Bohan Li
Yong Li
Venue: ICLR 2023 poster (2023)
Keywords: Few-Shot Learning
Finetuning
Equivariance

Abstract: Few-Shot Learning (FSL) aims to learn a simple and effective bias on limited novel samples. Recently, many methods have been focused on re-training a randomly initialized linear classifier to adapt it to the novel features extracted by the pre-trained feature extractor (called Linear-Probing-based methods). These methods typically assumed the pre-trained feature extractor was robust enough, i.e., finetuning was not needed, and hence the pre-trained feature extractor does not change on the novel samples. However, the unchanged pre-trained feature extractor will distort the features of novel samples because the robustness assumption may not hold, especially on the out-of-distribution samples. To extract the undistorted features, we designed Linear-Probing-Finetuning with Firth-Bias (LP-FT-FB) to yield an accurate bias on the limited samples for better finetuning the pre-trained feature extractor, providing stronger transferring ability. In LP-FT-FB, we further proposed inverse Firth Bias Reduction (i-FBR) to regularize the over-parameterized feature extractor on which FBR does not work well.	The proposed i-FBR effectively alleviates the over-fitting problem of the feature extractor in the process of finetuning and helps extract undistorted novel features. To show the effectiveness of the designed LP-FT-FB, we conducted a lot of experiments on the commonly used FSL datasets under different backbones, including in-domain and cross-domain FSL tasks. The experimental results show that the proposed FT-LP-FB outperforms the SOTA FSL methods. The code is available at https://github.com/whzyf951620/LinearProbingFinetuningFirthBias.

Under review conference paper at ICLR 2023
REVISIT FINETUNING STRATEGY FOR FEW-SHOT LEARN- ING TO TRANSFER THE EMDEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Few-Shot Learning (FSL) aims to learn a simple and effective bias on limited novel sam- ples. Recently, many methods have been focused re-training randomly initialized linear classifier to adapt it to the novel features extracted by a pre-trained feature ex- tractor (called Sneear-robiingbasee methods). These methods typically assumed the pre-trained feature extractor was robust enough, i.e., finetuning was not needed, and hence the pre-trained feature extractor does not be adapted the novel samples. How- ever, the unadapted pre-trained feature extractor distorts the features of novel samples because the robustness assumption may not hold, especially on the out-of-distribution samples. To extract the undistorted features, we designed Linear-Prodee Probing-Finetunne with Firth-Bias (LP-FT-FB) to yield an accurate bias on the limited samples for better finetuning the pre-trained feature extractor, providing stronger transferring ability. In LP-FT-FB, we further proposed inverse Firth Bias Reduction (i-FBR) to regularize the ower-parameterized feature extractor which FBR does not work well. The proposed i-FBR effectively alleviates the wver-fitting problem of the feature extractor in the pro- cess of finetuning and helps extract undistorted novel features. To show the effectiveness of the designed LP-FT-FB, we conducted comprehensive experiments on the common- ly used FSL datasets under different backbones for in-domain and cross-domain FSL tasks. The experimental results show that the proposed FT-LP-FB outperforms the SO- TA FSL methods. The code is available at httttts: /ttttp://githbb wwt.w//wwwwww..... www..ooo/t/1l22 GinearProbing inet uninnirthiiiii
1 INTRODUCTION
Few-shot Learning (FSL. has recently developed quickly in the limited data regime. FSL aims learn suitable inductive bias on the given limited samples of novel classes. At the very start, the whole model consisting of the feature extractor and the classifier is pre-trained on the samples of base classes, and then is finetuned on the limited novel samples to obtain an inductive bias. The performance of the finetuned model drops significantly due to the over-fitting problem of the pre-trained model. To address the over- fitting problem, meta-learning- based methods such as Prototypical Networks (nell et al. (211)) and MAML Finn al. (017)) were proposed learn the learning strategies for suitable inductive bias. Then, Chen el al. proposed Baseline++ Chen et al.) (2019) to show that simple Linear Probing (LP) strategy can also methods. I.P re-trained a linear classifier to adapt to the novel samples without updating the whole model. Following Baselinct+, many researchers studied the LP-based FSL methods, such as S2M2 Mangla et al. (2020), RFS Tian et al. (2020), and EMD Zhang jet al. (2020), proposed to obtain
get comparable performance to meta-learning-basee
powerful tullnet feature extractor. L.--baeed FSL methods assumed that pre-trained feature extractor is robust enough to novel samples Yang et al. (2021): Tian et al. (2020) and hence does not need to be finetuned.

---

Under review conference paper at ICLR 2023
Lingar Probing
Pre-trannd
Feature
Liner Classifler (R))
Extracto
Linear Probing
FBR
FirsTTuiigg
Pre-trained
Feature
Limar Classifior (L.P)
Extracto
Finetuning
FFB
Pre-trained
Feature
Linear Classifie (LP-FT-FB)
Extracto
(LP-FT-FD)
Figure 1: The proposed LP-FT-FB. The flow is divided into Linear Probing and FineTTuning in two gray lines of dashes. (Left) At the linear probing stage, the parameters of the pre-trained feature extractor are frozen; at the finetuning stage, the parameters of the feature extractor are changeable. In the Linear Probing box, the RI means the linear classifier randomly initialized. In the FineTTniin box, the LP means the linear classifier is fully re- re-trained on the novel samples. (Right) The FBR is used to get an unbiased estimation for linear probing by encouraging the distribution of logits to be far away from the uniform distribution (U(a, b) in the figure). The proposed i-FBR addressed the over-fitting problem of the finetuned feature extractor by encouraging the distribution of logits to close the uniform distribution.
However, the robustness assumption may not hold, especially on the put-of-distributioo (OOD) novel sam- ples, because the existing regularization methods only provided effective robustness in-distribution (ID) samples. This leads to "wrong" features of novel samples CumplesKuumar Kumar et al. (2022). This problem cannot be well addressed by only strengthening the robustness of the pre-trained feature extractor without transferring the extracted features. Finetuning is a typical and simple technology in the transferring learning on large-scale datasets to quickly transfer the features Kornblith et al. (2019);
9): He et al. (2020); Chen et al. (2020). Howev- er, the existing FSL finetuning strategies cannot appropriately finetune the over-fitted feature extractor (hhnn eet al. (202T). Consequently, dexigning an appropriate finetuning strategy for the fully-truined feuture ex- tractor the limited samples is critical point to transfer the features and hence make the finetuned feature extractor yield "good" novel features.
To design appropriate finetuning strategy, we started by analyzing the existing finetuning strategies Kumar et al. 2022); Kanavati & Tsuneki (202T); Levine et al. (::16); Radford et al. (202)) especially on the ODD samples| Kumar al. (2022). Linear- Linear-Probing- Finettnin-nnnniggggggg (LP-FT) Kumar al. (2022) theoretically proved that the existing finetuning strategies underperformed due to distorted novel features. This is because when trying fit the novel samples with a randomly initialized linear classifier, the extracted ID features and OOD features change inconsistently. The inconsistent problem is caused by the unchanged features of the samples in the space orthogonal to the space spanned by the base samples, i.c., out-of-distribus features unchanged (details in Section 2.2). Kumar et al. proposed using a linear classifier well- cell-trained on the fitting novel samples to address this distorted problem.
Although LP-FT is effective, it is used to address the finetuning problem on large-scale -iitti--iitiittoonn datasets, which contain enough samples to provide an unbiased estimation. FSL tasks, limited novel samples lead to a biased estimation of the updated parameters. Seeing this, we reduce the Firth Bias of the model Ghaffari et al. (2022); y FIRTH (1993) to obtain an unbiased estimation (called Firth Bias Reduction). However, Firth Bias Reduction (FBR) gives unbiased estimation only for the linear classifier but not the feature extractor. Experimental result also shows that FBR decreased the performance of the finetuned fea- ture extractor. A deep analysis (Section 2.3)) shows that when the scaling factor À (see Eq, 2 is positive, FBR encourages the distribution of the linear classifier output to be far away from the uniform distribution (See Fig. [ FBR strengthens the influence of novel samples for linear probing. This accelerates the convergence
2

---

Under review conference paper at ICLR 2023
of the low-parameterized linear classifier but hampers the generalization of the over-parameterized feature extractor. According to the analysis, we proposed inverse-FBR (i-FRB, negative À in Eq. the distribution of the linear classifier output to be close to uniform distribution (See Fig. to address the
2 to encourage
over-fitting problem Müller et al. (2019).
Combining LP-FT, FBR, and the proposed i-FBR, we proposed (LP-FT-FBB to appropriately finetune the fully-trained feature extractors to adapt them to the target domain on limited novel samples. Firstly, we use linear probing to address the distorted feature problem. The linear probing is regularized with FBR for unbiased estimation. Then the whole pre-trained model is finetuned regularized with the proposed i-FBR on the limited novel samples to strengthen the transferring ability of the feature extractor for more separable features. The proposed LP-FT-FB makes the extracted features of novel samples undistorted and separable. The whole LP-FT-FB finetuning strategy is visualized Fig. [ Furthermore, the unbiased finetuned feature extractor transfers the novel features close the novel domain. This is verified in the cross-domain FSL tasks (Section 3.5
waan--Poooonn--nttuiiiie with Firth Bias
Our main contributions include (1) studying and finding the gaps between finetuning-baset and L.P-baeed FSL methods: distorted features and biased estimation; (2) designing novel and simple finetuning s- trategy, LP-FT-FB, quickly transfer the extracted features to the novel domain to bridge the gaps; (3) proposing inverse Firth Bias Reduction (i-FBR) to address the biased estimation problem of finetuning the feature extractor due limited samples.
2 METHODS
LP-FT is first described in Section 2.2 Then FBR is formulated and analyzed in Section 2.3 to address the unique limited-sampii problem in FSL, i.c., the biased estimation problem. However, FBR is not suitable for the over-paramen wveepparameterized feature extractor. Based on the analysis of FBR, i-FBR is proposed to finetune the feature extractor. In Section 2.4 LP-FT-FB is stated.
2.1
SET CP
Following FBR Ghaffari et al. (002)), assume multinomial logitic regression model for the classi-
fier v(.,3). 3 (e Jisss and Be denotes the logistic regression weights for class c. B 0) de- notes the feature extractor. The dataset D {(x;, JtlssiMM totally contains C classes and M samples. The whole inference process is formulated as: ý; v(z,, v(B(x, 0), where z, = B (xi, (xxi,)) 0). In the multinomial logistic regression model, the assignment probability of z; to class c is formulated p
pf
Prly; clz,) I+E0
Then the logistic log-likelihood function is formulated as
Llogistic -M 1/y, c) log where 1|-1 denotes the y; in the one-hot manner. For the linear probing and finetuning, we used the Cross Entropy Loss LCe = M P. = [P P. = y, denotes the predicted probability vector.
1 D. [y+ logo))(P.)], where
2.2 LINEAR-PROBINE PROBING-FINETUNINS
LP-FT explored the reason why LP underperforms FT on in-distribution samples but outperforms FT out- of-distribution samples large-scale datasets. LP-FT theoretically proved that when a pre-trained feature extractor tried to fit the ID samples, the transfer of the model will not change the feature of out-of-distribution samples in the space orthogonal the space spanned by pre-training samples.
To show this, let S denote the subspace spanned the training samples X, and the training loss L(v(B(X, 0), B), Y) |X0T Y/ B is assumed to be linear model; Y denotes the one-hot la-
3

---

Under review conference paper at ICLR 2023
bels of X. The gradients of the training loss with respect to the parameter 0 of the feature extractor B computed as:
VvL(v 0), Y) 2 - x)"3))>
(1)
With Eq. 1 u sample in the subspace orthogonal to S. the features of u do not change with the finetuned B(, 8): Az = VoL(v(:, (), ( 0))u = 2 XeT 3) (X . u) = 0. This leads to the distorted features
extracted by the finetuned B(,, 0) because the in-distribution and aut-of-distribution features inconsistently change.
However, LP-FT explores the feature extractor finetuning problem on large-scale datasets. Observing this, we wish to address the feature extractor finetuning problem with LP-FT in FSL if we can address the limited- sample problem. Because the novel samples used to finetune the pre-trained feature extractor are out-of- distribution in FSL models. The limited-sample problem is actually biased estimation problem.
2.3 INVERSE FIRTH BIAS REDUCTION
For linear probing, we use Firth Bias Reduction (FRB) to obtain an unbiased estimation. PMLE FIRTH (1993) added a log-determinant penalty to remove the ((n=1) term of the asymptotic bias of the maxi- mum likelihood estimate parameters for the unbiased estimation. FBR Ghaffari et al. (2022) addressed the situation when (et 0 and proposed an approximate format for the linear classifier and cosine classifier. In FhRGhaffari et al. 2022). the FBR loss is formulated as
M
1
== L= + FFirth = LCE A
DKL (U..Ull))) M
(2)
where U C) denotes the uniform distribution between 0 and C, and À denotes the scaling factor of FBR. From Eq. 2 we concluded that when À > O, the Crinth encourages the distribution of the logits z; to be far away from U Uj, which totally differs from Label Smoothing Müller et al. (2019). When À < 0, Fiirhh encourages the distribution of the logits z; to be close to Ul and hence avoids too high confidence of z;-
This coincides with the effectivvness of Label Smoothing Müller et al. (2119; Hein et al. (2019).
However, the vanilla FBR is not suitable for finetuning feature extractor because when À >> O (used in FBR Ghaffari et al. (2022), ). the logits z encouraged be far away from U|u.c The distribution drag leads too high similarity between the distribution of the logits z and y, and hence makes the per-parameterize feature extractor over-fittedHHer
al. (2019). Seeing this, according to the analysis of Eq. 2 we proposed inverse FBR (i-FBR) to encourage the distribution of the z to be close to U|o.c with X O (we use the Xine to denote the scaling factor i-FBR) to address the over-fitting problem. The proposed i-FBR works similarly to Label Smoothing the model to be close to the uniform distribution. Hence they can reduce the influence of the training novel samples and alleviate the too high confidence problem.
certain extent because both of them encourage the output distribution of
addition to the above distribution drag, we computed the gradients of LCE and L with respect to r; G B (x; 0) as follows to show the gradient variation brought by X.
ôLoE Ør; Ør:
dC
X
(CP: E),
(C + 1)
where E is a one-full vector with the same size as Pa and P ý; the predict of the whole model. The detailed derivation is given in Appendix. According to Eq. we concluded that when À >>> O, the gradient Dr is increased for the oww-parameterized linear classifier quickly reach the optimal point, and hence the linear probing model performs better Prabhu et al. (202T). The overfitting problem does not need to be considered because of the low- now-parameterized property.
(l)(()e(F)) where F Hesss (Ciogietic) Ey V Plogiittic A
4

---

Under review conference paper at ICLR 2023
However, it is not suitable for finetuning the over-parameterized feature extractor because of too large sim- ilarity between 2; and Vi. From Eq. we found that when À 0 and the class of x; is c, the gradient is decreased and är (k: + c) is increased. This reduces the influence of x,, which used to train the dr;
over-parameterized feature extractor, and hence alleviates the over-fitting problem. Furthermore, different from Label Smoothing, the proposed i-FBR decreases the gradients, which is suitable for finetuning the over-parameterized model, and hence outperforms Label Smoothing (See Table 6).
2.4 LP-FT-FB
With FBR, LP-FT, and the proposed i-FBR, we proposed L.P-T-FBB to quickly transfer the extracted features instead of the robustness assumption to address the over-fitting problem. Firstly, randomly initialized linear classifier v(. (3) re-trained on the novel samples and regularized with FBR. For the feww-way-K-Sh FSL tasks, the re-trained classifier v' (", is updated as
K-Csew Σ
ac y.). âr,
B' B+ K
(4)
KCeew
Ōr 03
where r; = B Bo(x; 00): 8 = 18:) Crow. yy :
(y:)
Crow :n is the LP learning rate.
Then the pre-trained B. (() and (', are together finetuned on the same novel samples and regularized with i-FBR. FT is stated as follows. The forward process of FT is formulated as = =v'(Bo (o),3")
(5)
The loss of FT is computed as:
K- K-C Σ
C(S'
(6)
Feow
The feature extractor Bo(- 0o) is updated as
K K-Cjew OL(F).y)) or 00,*
02
= 0o +
(7)
K Cjew
Ör,
where r' (rr) (r] and r = B' Bo(x;, 0o).
The linear classifier v',, is updated as
K.C OL(Y', y:)
B 3' +
U22
Σ
or!
(8)
K K-Cfew
Ōr
0
where 02 is the learning rate of FT. The analytic formula of Eq.
Eq.
8
and Eq, 7 are given in Appendix.
With B(, θ) and v(,, ()), the performance of LP-FT-FB is evaluated on the novel query samples. The whole flow of LP-FT-FB is given in Appendix.
3 EXPERIMENTS
To comprehensively show the effectiveness of the proposed i-FBR and LP-FT-FB, conducted many in-domain experiments: 1) 5-way-11,5sshot tasks on mini- Inni--maaenne, hereeeImagenet, and CUB datasets under the typical FSL. backbone, WideResNNe-22-10: 2) few-shot tasks under multiple scale backbones including ResNet-18 and ResNet-34; 3) N-way-K-shan (N > 5, > 5) tasks on mini-Imagenes and tiered- Imagenet; 4) few-shot tasks on the extracted features augmented by Distribution Calibration (DC) Yang al. (2021). Also, the cross-domain few-shot tasks, mini-emagenet + CUB and tiered- + CUB, are also evaluated.
Here, we used Cayw-way---Shee to correspond with in Sec. 2.1
2.2 and 2.3 instead of K-way.K-Shot.
5

---

Under review conference paper at ICLR 2023
3.1 EXPERIMENTAL SETUP
1) Datasets. The experiments are evaluated on three typical FSL datasets, mini- Vinyals et al. (2016), fiered-Imagenet Ren et al. (2018), and CUB Wah et al. (201T). mini-Imagenet consists of 100 classes from the ImageNet, which class has 600 samples of size 84 x 84. tiered- Imagenet consists of 608 classes from the ImageNet, which are split randomly into 351 base, 97 validation, and 160 novel classes. CUB contains 200 classes with a total of 11 11,788 images of size 84 x 84. The base, validation, and novel split contain 100, 50, and 50 classes. 2) Backhones. To show the effectiveness of LP-FT-FBB,
split randomly into 64 base, 16 validation, and 20 novel classes. Each
used many backbones. WideResNet-228. 10 Zagoruyko& & Komodakis (2016) with Dropout Hinton et al. (2012) is typical backbone which used in the SOTA methods, such 22M2Mange et al. (2020). FBR Ghaffari et al.) (2022), and Yang al. (202T). Also, multiple scale backbones including ResNet-18 He2 (016)) and ResNet-34, are used. The ResNets are the same as Mangla et al. (2020). 3) Finetuning hyper-parameter
The pre-trained hyper-parameters are the same as |Mnglla et al. (2020). We only give the hyper-parameter in the LP-FT process. For LP, we used the linear classifier proposed in the Baseline++ Chen et al. (2019). For the optimízer. the SGD is used with the learning rate a1 0.01, momentum 0.9, dampening 0.9, and weight decay 1e-3. For the FBR of classifier, the factor X in Eq. 2 is set to 1. For FT, the feature extractor and the classifier are together manually finetuned with the learning rate a2 LE and the i-FBR factor Xino in Eq. 7 is set -le 3 4) Evaluation. The reported results are averaged in percent with 95% confidence interval on 1 10000 tasks randomly selected by the Episodic Sampler. As the settings in Yang (202T) and al. (2022) the query sample number is 15 in each task for evaluating the performance.
3.2 6-WAY-1\5-S1BP TASKS
In-domain 5-way-1 (5-shot tasks are the most typical FSL tasks. We directly used the pre-trained WideResNNet28-110 provided by S2M2R Mangla et al. 2020). The evaluated results are reported in Ta- [ On mini-lmagenet, LP-FT-FB outperforms FBR by 1.45% and 0.86% for 1'5-shot tasks respectively. With the same pre-trained model, LP-FT-FB outperforms S2M2R x by 2.56% and 1.23% for 1' 115-shot tasks. Similarly, on the tiered Imagenett LP-FT-FB outperforms FBR by 3.04% and 3.75% for 1|5-shot tasks. On CUB. L.P.FTFBB outperforms $2M2n by 1.48% and 1.03% for 1/5-shot tasks.
Table I: The evaluation experiments for 5-way FSL tasks are conducted under the WideResNer28-10 on
three typical FSL datasets.
mini-Imagennt 5-shot
Shered-Imagent
CUB
Methods
1-shot
1-shot 5-shot
1-shot
5-shot
MAML Finn et al. (2017)
48.70
63.10
71.29
80.33
ProtoNet Snell et al. (201)
54.16
73.68
65.65
83.40
71.88
87.42
Baseline++ Chen al. 2019
51.87
75.68
68.00
84.20
69.40
87.50
S2M2 Mangla et al. (2020)
64.48
82.67
73.50
88.00
80.68
90.85
DeepEMI Zhang al. (2020)
65.91
82.41
71.16
86.03
75.65
88.69
FBR Ghaffari et al. (022
65.59
83.04
73.64
88.31
80.92
91.03
LP-FT-FB
67.04
83.90
76.68
92.06
82.16
91.88
3.3 EVALUATION UNDER DIFFERENT BACKBONES
In addition to WideResNet-28-10, 10, we also evaluated LP-FT-FB under different backbones, ResNet-18 and ResNet-34. Because S2M2 R did not provided the pre-trained models of ResNet-18 and ResNet-34, we used the code of S2M2R to reproduce the pre-trained models.
/ com htt.uu..coo/nuuuueeaeee homp/nupurkmtt9/2220 fewsshot
6

---

Under review conference paper at ICLR 2023
Table 2: The evaluation results for 5-way FSL tasks under different backbones.
ResNet- 18
ResNet-34
Dataset
Methods
I-shot
5-shot
1-shot
5-shot
Baseline++
Chen et al 2019
53.56
74.02
54.41
74.14
mini-lmagenet
S2M2 Mangla al. 2020)
64.06
80.58
63.74
79.45
LP.FT-FB
66.20
83.06
66.16
82.96
Baseline++ Chen et al.
(2019)
67.68
82.26
68.09
83.16
CUB
S2M2 Mangla al.
2020)
71,43
85.55
72.92
86.55
LP-FI-FB
73.36
86.88
76.48
90.02
Baseline++ Chen
(2019)
68.27
82.46
58.83
82.60
fieredImagenn
S2M2 Mangla et al.
2020)
69.86
85.52
69.38
84.41
LP-FI-FB
73.98
89.48
73.44
89.02
As given in Table 2 for ResNet-18, LP-FT-FB outperforms S2M2n by 2.14% and 2.48% for 11,5-hot tasks mini-Imagenet. LP-FT-FB outperforms S2M2n by 1.93% and 1.33% for 1 tasks CUB. For ResNet-34, LP-FT-FB outperforms S2M2R by 2.42% and 3.51% for 11o5-hot tasks on mini-Imagenet. LP-FT-FB outperforms S2M2R by 3.56% and 3.47% for 1\5-shot tasks on CUB.
3.4 EVALUATION ON AUGMENTED SAMPLES
The feature extractor is finetuned, and the novel features are since equivariant while novel samples totally differ from the base samples. The equivariant features are since closer the distribution of the novel sample than the unchanged feature extractor used in DC. To show this, we used DC to augment the yielded features before evaluation. The evaluated results are given in T
The data augmented experiments are conducted on the tiered-Imagenet. For the 1-shot- 10-way task, LP- FT-FB outperforms DC and FBR by 0.74% and 0.79%, respectively. For the 5-shot-10-way task, LP-FT- FB outperforms DC and FBR by 1.13% and 1.54%, respectively. Also, for the 15-way tasks, LP-FT-FB outperforms DC and FBR.
Table 3: The results of FSL tasks with DC data augmentation.
Imree-Imagenet
Methods
1& 10
5&10
1&15
5&15
DC Yang et al. (2021)
61.85
79.66
54.57
73.88
FBR Ghaffari et al. (2022)
61.90
80.07
54.62
74.40
LP-FT-FB+DC
62.64
81.20
55.02
75.64
3.5 CROSS-DOMAIN TASKS
To show the transferring ability brought by LP-FT-FB, we conducted cross-domain FSL experiments. LP- FT-FB evaluated on the CUB dataset, but the feature extractor is pre-trained on mini-Imagenet and tiered- Imagenet, respectively. In this experiment, used DC to augment the novel samples, just the evaluated experiments in FBR Ghaffari et al. (2022). L.P-FT-FB + DC denotes the proposed method in Table
As is shown in Table 4 for mini- for 5-shot- 10-way and 5-shot-15-way tasks. For tiered-Imagenet + CUB tasks, LP-FT-FB outperforms DC by 1.65% and 2.09% for 5-shot-10-way 10-way and 5-shot-15 1ahhot-1--ay tasks.
CUB tasks, LP-FT-FB outperforms DC by 2.07% and 2.29%
7

---

Under review conference paper at ICLR 2023
Table 4: The results cross-domain FSL tasks.
N denotes N-way-K-shot tasks.
mini-Imagenet + CUB
tiered-Imagene, - CUB
Methods
1&10
5&10
1&15
5&15
1&10
5&10
1&15
5&15
DC Yang al. (2021)
37.14
59.77
30.22
52.73
64.36
86.23
57.73
82.16
FBR Ghaffan et al. 2022
37.40
60.77
30.37
53.84
64.52
86.66
57.74
83.05
LP-FTFB+ +DC
37.96
61.84
30.98
55.02
65.02
87.88
58.26
84.25
3.6 ABLATION STUDY
LP-FT v.s. FBR v.s. i-FBR. We give the ablation study to explore the effectiveness of each part. The study is evaluated mini-Imagenet and tiered-Imagenet for 5-way-1 1--ssot tasks. As given in Table 5 show the effectiveness of LP-FT-FB's each part.
Table 5: The ablation study on LP-FT v.s. FBR v.S. i-FBR.
Strategy
LP
FT
Datasets
LP-FT
FBR
FBR
i-FBR
mini-Imagennt
tiered- leree--magenet
65.80
73.82
65.59
73.64
66.40
74.59
67.04
76.68
65.89
73.82
66.38
74.70
i-FBR factor Ane tuning. We give the tuning of Airo follows to show the pattern of i-FBR.
Amv Tuning
67
Acc
66
65
-le:1 -7e-3
1e:3
5e.2
1e-1
Amn
Figure 2: i-FB factor Xine tuning.
As given in Fig. 2 we show the tuning trend of Aine- The ablation experiments is evaluated on mini- Imagenet for the 5-way-1-shee task with À = 1. We obtained the best performance with Xine = -7e While Xine 2 0, i..., i-FBR is turned into FBR, the performance is decreased.
i-FBR v.s. smoothing regulariers. The proposed i-FBR is a sort regularizer smoothing the output distribution of the model. We compared the performance of different smoothing regularizers. As given in Table 6, the proposed i-FBR outperforms the smoothing regulaizers. The detailed experiment settings are given Section Appendix.
8

---

Under review conference paper at ICLR 2023
Table 6: The ablation study different smoothing regularizers. LP-FT-None denotes smoothing regu- larizers used.
mini-Imagenet
tiered- Imagen-m
Methods
1-shot
5-shot
1-shot
5-shot
LP-FT-None
66.40
83.20
75.66
90.85
LP-FT-LS
66.60
83.24
76.02
91.67
LP-FT-Mixup
66.64
83.37
76.00
91.54
LP-FT-CutMix
66.48
83.26
75.84
90.98
LP-FT-FB
67.04
83,90
76.68
92.06
4
RELATED WORKS
The FSL methods are divided into veta-learning-baser methods and fiettniiggbbase methods. The finetuning methods are divided into Linear- Seetar-roobnn-beee based and finetuning ceeiiingttt--eeee tho-l methods.
Hinettuing--earning--eeee Methods. In 2019, Chen al. proposed Baseline++ Chen al. (2019)
which showed that a re-trained linear classifier could get a comparable performance to the meta-learning- based methods when the feature extractor is pre-trained. The pre-trained feature extractors easy be obtained without the need of training starting from scratch (used in meta-learning- based methods). Fol- lowing Baseline++, Tian al. proposed RFS Tian et al. (2020) to show that learning a supervised or self-supervised representation on the meta-training set, followed by training linear classifier on top of this representation, gets good performance better than most neeta-learning-baae
methods. Similarly, S2M2 Mangla et al. (2020) and SKD SKD Rajasegaran et al. (2020) proposed that the feature extractor pre-trained with Rotation Augmentation, Manifold Mixup Verma et al. (2019), and Knowledge Distillation Hinton et al. (2015) can make the feature extractor more robust. These methods are called Linear Probing methods be- cause they only re-traine re-trained the linear classifier but fixed the pre-trained feature extractor, and they are based strong robustness assumption.
However, the robustness assumption may not hold. To address the problem, Shen et al. proposed Shen al. [2021) searching algorithm finding the layers suitable for finetuning. The searching algorithm consuinn and performs poorly on a single dataset because does not address the distorted feature problem. Additionally, the related meta-leaat veta-learning FSL methods is left to Appendix.
5 CONCLUSION
In this paper, we proposed LP-FT-FB finetuning the pre-trained feature extractor bring strong transferring ability to the feature extractor instead of strong robustness assumption. The transferring ability addressed the distorted feature problem caused by the OOD novel samples. However, the strong transferring ability is at the cost of the over-parameterized feature extractor finetuning. The proposed LP-FT-FB is also time- consuming, just like the first-order gradient computation of Reptile Nichol al. (2018). In future works, we will try to design filter-wise layer-wise finetuning method instead of the unit-wise one.
6 DECLARATIONS
This work supported by the National Key R&D Program of China under Grant 2019YFF0302601, National Natural Science Foundation of China (No. 62071060) and the Beijing Key Laboratory of Work Safety and Intelligent Monitoring Foundation.

---

Under review conference paper at ICLR 2023
REFERENCES
Deep Residual Learning for Image Recognition, 06 2016. doi: 10.1109/CVPR.2016.990
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference Learning Representations, 2019.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta- Learning for Fast Adaptation of Deep Networks. 2017. URL U.LLEtpp.//axxvv.oo/////////.... 03400
DAVID FIRTH. Bias reduction of maximum 0006-3444, doi: 10. 10.1093/biomet/80..22 UT//a/tpp.///00...rr/00011/////0 1..27
Biomeriika, 80(1))7733 03 1993. ISSN 1.27
Saba Ghaffari, Ehsan Saleh, David Forsyth, and Yu-Xiong Wang. On the importance of firth bias reduction in few-shot classification. In International Conference on Learning Representations, 2022. URL
//openreview.o net
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 9726-9735, 2020.
doi: 10.1 100/CC/P442200.2200000000
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. pp. 41-50, 06 2019. doi:
10.1109/CVPR.20 WWWp:/VVP..211.000000 2019.0013..
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint, arXiv, 07 2012.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.
Fahdi Kanavati and Masayuki Tsuneki. Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning, 2021.
Simon Korbliihh, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In Proceed- ings of In Computer Vision and Pattern Recognition (CVPR), 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distributtion In International Conference on Learning Representations, 2022. URL htt.aats///peeeeeeeeeeeeeeee ott.::/eooooommmmmeeeeeee.00
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomo- tor policies. Journal of Machine Learning Research, 17(39):1-40, 2016. URL http:://jmm /tap:////mml..r////// org/ papers/
v17/15 papers/v17/13 522 html
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Bal- asubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In The IEEE Winter Conference on Applications of Computer Vision, pp. 2218-2227, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NewrIPS, 2019.
10

---

Under review conference paper at ICLR 2023
Alex Nichol, Joshua Achiam, and J. Schulman. On first-order meta-learning algorithms. ArXiv, ab- //100300299, 2018.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry:
Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Inter-
national Conference on Computer Vision, pp. 5558-8567, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. Self- supervised knowledge distillation for few-shot learning. http:://urxivoorg/aax200066 09785, 2020.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In Pro- ceedings of 6th International Conference on Learning Representations ICLR, 2018.
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang- Drran-Tiiggaa Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9594-9600 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Ad-
vances in Neural Information Processing Systems, volume 30, 2017. URL lttps: neurips. cc/ Sttpiips.ccc/pppe//0/0 /tapperr/20///11ee//bdd666666663
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? arXiv preprint arXiv:2003 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4438-6447 PMLR, 09 15 Jun 2019. httphttss//pproeeeeeeee ngs mll press / ve 9a html
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing System- volume 29, 2016. URL https:
48c Paper.pdr
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR T0--0R--01111 1-001, California Institute Technology, 2011.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. International Conference on Learning Representations, 2021. URL https://ppenreeeewweettt/eeeeeee www.neet/ooiittto.oeeeee
JW01YxMG92s
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, (bbs1605.07146, 2016. URL
07146
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In IEEE/CVF Conference on Computer Vision and Puttern Recognition ( (CVPR), June 2020.
11


Reviews:
[{"review_id": "slMKL5Z1cp", "reviewer": "Reviewer_C9jy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out a potential issue with the linear-probing framework many SoTA methods are based on. Freezing a pre-trained feature encoder could get away with the feature inconsistency problem when one fine-tunes the feature extractor on the limited novel samples. However, the effectiveness of the frozen feature is built upon the assumption that the pre-trained feature extractor is robust enough for the novel samples, which nonetheless is usually not the case for out-of-distribution samples. The authors thus propose a method that effectively fine-tunes the features given limited samples from the novel classes. Specifically, on top of Firth bias reduction, they proposed an inverse FBR that decreases the influence of the novel classes when the extractor is fine-tuned.", "summary_of_the_review": "Overall, this paper points out the issue of prevalent SoTA papers on few-shot learning and offers a solution, which the authors showed effective in the experiments compared with other SoTA methods. Despite the effectiveness of the proposed method, the method is slightly limited in terms of novelty as it is built upon the previous FBR paper. Summarizing all these perspectives, I would recommend a weak accept for now.", "strength_and_weaknesses": "Strengths\n1. Provided an intuitive explanation alongside the mathematical proofs as to why the proposed inverse FBR works.\n2. The authors compared the proposed method with many SoTA methods and showed consistent improvements over them.\n3. One experiment demonstrated in Table 4 shows the method is complementary to one of the SoTA papers, Distribution Calibration.\n\nWeaknesses \n1. While the overall explanation seemed well-motivated, I had a question regarding the existence of samples that lie in the subspace orthogonal to the one spanned by the base samples. My concern is the following. If the feature extractor is pre-trained on the base classes, wouldn't the feature extractor tend to \"put\" whatever samples it sees in the subspace spanned by the pre-trained samples? I would appreciate more explanations from the authors pertaining to this argument.\n2. What do the authors mean that in-distribution and out-of-distribution features inconsistently change? It's also not crystal clear to my mind although the authors do provide an explanation with a linear model.\n3. The authors keep using \"equivariance\" in the article. However, for their scenario, e.g., using a feature trained on ImageNet on CUB200, it's not accurate (not sure if \"invariance\" would be a better choice). I would suggest the authors define what they mean by \"equivariant\" more clearly as what I would think of equivariance is when someone performs an operation on the image, the feature would change accordingly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ZGvVTNDFF9", "reviewer": "Reviewer_Yd7H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discussed the distorted/biased features problem between base set and novel set in few-shot learning task, and proposed a Linear-Probing-Finetuning method with Firth-Bias to extract the undistorted features for shot-few learning. It also introduced an inverse Firth Bias Reduction (i-FBR) method for training the model in the few-shot learning setting. The learning procedures in the whole approach are as follows:\n\n1)\tpre-train backbone model B0(·, θ) on the base set with a regular loss function.\n\n2)\ta randomly initialized linear classifier v(·, β) is retrained on the novel samples and regularized with FBR.\n\n3)\tthe pre-trained B0(·, θ) and v0(·, β) are together finetuned on the same novel samples and regularized with i-FBR.\n\nExperiments in this paper are conducted on mini-Imagenet, tiered-Imagenet, and CUB datasets with ResNet-18, ResNet-34, and WideResNet28-10 backbones.\n", "summary_of_the_review": "Overall, I think this is an interesting paper to explore the finetuning strategy for few-shot learning. However, I feel the writing can be improved with a few minor mistakes in the current shape, also some ablations on each component are encouraged to be put in the main text. I currently give borderline with the positive side. I'm open to increasing the rating according to the authors’ responses to my concerns.", "strength_and_weaknesses": "Strengths:\n\n    - It’s interesting to explore a better finetuning strategy for the few-shot learning problem.\n\n    - The proposed method is clear to understand and easy to follow.\n\n    - The method seems reasonable to obtain performance gain and the analysis and explanation are sufficient.\n\n    - The improvement over other methods is satisfied.\n\nWeaknesses:\n\n    - Writing of this paper can be improved. Some sentences are colloquial and not scientific, for example: in the abstract, “we conducted a lot of experiments on the commonly …”. In related work “… and Knowledge Distillation Hinton et al. (2015) can make the feature extractor” is incomplete.\n\n    - In Table 4, the authors provided the results using DC together with the proposed method, I wonder how about the results without DC. It's better to also provide this to demonstrate the effectiveness of the proposed method.\n\n    - There is no ablation on each stage of the proposed learning procedure, for example:\n\n     (1) train the linear classifier v(·, β) on the novel samples without FBR; \n\n     (2) finetune B0(·, θ) and v0(·, β) together on the same novel samples without the proposed i-FBR. (better to put in the main paper instead of appendix)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CNfIoPkIe6I", "reviewer": "Reviewer_rDwz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets at few-shot classification. Motivated by Label Smooth, it designs LP-FT-FB to impose equivariance on the feature extractor to address the over-fitting problem. It first train a randomly initialized linear classifier on novel samples with FBR. Then the pretrained feature extractor and the classifier are finetuned on the same novel samples regularized with i-FBR. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "Although this paper has some minor weakness, considering its clear motivation, simplicity of the method, and good experimental results, the reviewer thinks the paper is above the acceptance threshold.", "strength_and_weaknesses": "Pros:\n1. The proposed method is well motivated and clearly stated.\n2. The experimental results look good. \n\n\nCons:\n1. The organization of the paper can be improved. (a) Table1 in the Supplementary Material is interesting. The reviewer suggests moving it to the main paper. Why not merge it with Table 1 in the Main paper? \n2. The reviewer thinks adding experiments to show the results of different i-FBR factor lambda_inv might be interesting, ranging from values smaller than zero to values larger than zero. \n3. Some experimental results are missing. Some experiments (Table.1) are conducted on mini-Imagenet, tiered-Imagenet, and CUB. But some only reported part of the results (e.g. in Table.2, the results of tiered-Imagenet are missing.). This makes the reader wondering whether the results of tiered-Imagenet do not \"look perfect\" and something is hidden behind. If possible, please add these kinds of missing resutls.\n4. Minor issues: (1) The paper uses the format of ICLR 2021, instead of following the recent ICLR'2023 format. (2) The citations in the paper look strange, please fix them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5qF3N6JTh9", "reviewer": "Reviewer_Qxg7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to adapt LP-FT to Few shot learning and designs LP-FT-FB learning framework, in which inverse-FBR is proposed to regularize the feature extractor.  The experiments are extensive, but no theoretical proof is provided for the proposed i-FBR.\n\n", "summary_of_the_review": "Without theoretical proof for i-FBR, I think the proposed method is not convincing and the novelty is limited.", "strength_and_weaknesses": "1. The experiments, including those in the supplementary file, are extensive and show the effectiveness of the designed method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "557k6algmY", "reviewer": "Reviewer_nALG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use the \"linear probing then fine-tuning\" strategy (LP-FT) introduced in [1], as well as the firth bias reduction (FBR) introduced in [2], to learn the adaptation to novel-classes step in few-shot learning. The proposed method is evaluated on a variety of FSL tasks and demonstrate good results in the fine-tuning for FSL methods.", "summary_of_the_review": "Combining LP-FT and FBR for few-shot learning is promising but the current results are far from competitive with recent approaches for few-shot learning. The paper does not introduce a new concept and therefore should at least demonstrate strong results. For these reasons I recommend a reject.", "strength_and_weaknesses": "\nStrength:\n\n1) This is the first time linear probing then fine-tuning is used for few-shot learning\n\n2) The proposed method achieves state-of-the-art results among the fine-tuning methods (as opposed to meta-learning methods) on several few-shot learning benchmarks.\n\n\nWeaknesses:\n\n1) The results are outdated in comparison to the recent state-of-the-art in few-shot learning (For exemple: [3, 4, 5]). Only comparing with fine-tuning methods and not with all meta-learning baselines is unfair. Maybe an idea would be to use more recent pre-trained backbone, for exemple, how would it perform using similar backbones as in [6] ? Is there a reason to only use a pre-trained backbone from [7] ?\n\n2) Other than inverting FBR, which is just using a negative coefficient for FBR, there is no conceptual novelty introduced in the paper. The linear probing then fine-tuning pipeline is introduced in [1] and FBR is introduced in [2].\n\n3) A more in-depth ablation than what is provided in Appendix D is really necessary to understand the contribution of each component of the method. In particular, how does LP-FT performs without FRB in few-shot learning ?\n\n4) The benchmarks used are standard, but very simple, both mini-imagenet and tired-imagenet use 5-way classification novel tasks, which is not challenging. It would be great to consider a much harder benchmark such as ImageNet-1k which has 311-way novel tasks. (See Table of [8], and Table 1.c) of [9]).\n\n\nQuestions and remarks:\n\n1) How hard is it to tune $\\lambda$ both in the linear probing step and in the fine-tuning step ? How do you do it in practice ? More generally, to add on the critic about the lack of ablation, it would be great to provide a full ablation on lambda on all configuration possible, i.e. $\\lambda$ positive, negative and equal to 0, which correspond to FBR, i-FBR on no FBR, for both linear probing and fine-tuning steps (9 configurations in total).\n\n2) Have you tried different probe than the linear probe ? For exemple non-parametric such as k-NN or 2-layer MLP ?\n\n3) Would it be possible to use some kind of regularization on the weights of the linear probe, instead of FBR ? l2/l1-regularization, sparsity, ect.. ?\n\nReferences\n\n[1] A. Kumar et al., Fine-tuning can distort pretrained features and underperform out-of-distribution, ICLR 2022\n\n[2] S. Ghaffari et al., On the importance of firth bias reduction in few-shot classification, ICLR 2022\n\n[3] P. Rodriguez et al., Embedding Propagation: Smoother Manifold for Few-Shot Classification, ECCV 2020\n\n[4] I. M. Ziko et al., Laplacian Regularized Few-Shot Learning, ICML 2020\n\n[5] X. Chen et al., Few-Shot Learning by Integrating Spatial and Frequency Representation, CRV 2021\n\n[6] S. Xu Hu et al., Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference, CVPR 2022\n\n[7] P. Mangla et al., Charting the right manifold: Manifold mixup for few-shot learning, CACV 2020\n\n[8] Y-X. Wang et al., Low-Shot Learning from Imaginary Data, CVPR 2018\n\n[9] L. Gui et al., Learning to Hallucinate Examples from Extrinsic and Intrinsic Supervision, ICCV 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster