On the Importance of Firth Bias Reduction in Few-Shot Classification
Authors: Saba Ghaffari
Ehsan Saleh
David Forsyth
Yu-Xiong Wang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Few-shot Classification
Firth Regularization
MLE Bias

Abstract: Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision. The performance in few-shot classification suffers from the bias in the estimation of classifier parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classifiers has been overlooked. In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classification. Theoretically, Firth bias reduction removes the $O(N^{-1})$ first order term from the small-sample bias of the Maximum Likelihood Estimator. Here we show that the general Firth bias reduction technique simplifies to encouraging uniform class assignment probabilities for multinomial logistic classification, and almost has the same effect in cosine classifiers. We derive an easy-to-implement optimization objective for Firth penalized multinomial logistic and cosine classifiers, which is equivalent to penalizing the cross-entropy loss with a KL-divergence between the predictions and the uniform label distribution. Then, we empirically evaluate that it is consistently effective across the board for few-shot image classification, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes. Furthermore, we demonstrate the effectiveness of Firth bias reduction on cross-domain and imbalanced data settings. Our implementation is available at https://github.com/ehsansaleh/firth_bias_reduction.

Published as a conference paper at ICLR 2022
ON THE IMPORTANCE OF FIRTH BIAS REDUCTION IN FEW-SHOT CLASSIFICATION
Saba Ghaffari' Ehsan Saleh' David A. Forsyth Yu-Xiong Wang Department of Computer Science, University of Illinois Urbana- Champaainn (sabag2, ehsans2, daf, yxw) @ www.s.eeeii.
ABSTRACT
Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision. The performance in few-shot classification suffers from the bias in the estimation of classifier parameters, however, an effective underlying bias reduction technique that could alleviate this issue training few-shot classifiers has been overlooked. In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classification. Theoretically, Firth bias reduction removes the O(N-1) first order term from the small-sample bias of the Maximum Likelihood Estimator. Here we show that the general Firth bias reduction technique simplifies encouraging uniform class assignment probabilities for multinomial logistic classification, and almost has the same effect in cosine classifiers. We derive an easy-to-implement optimization objective for Firth penalized multinomial logistic and cosine classifiers, which is equivalent to penalizing the cross-entropy loss with KL-divergence between the uniform label distribution and the predictions. Then, we empirically evaluate that it is consistently effective across the board for few-shot image classification, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes. Furthermore, we demonstrate the effectiveness of Firth bias reduction on cross-domain and imbalanced data settings. Our implementation is available at https://gitub..oo//cssssaaa
bias reduction.
INTRODUCTION
Few-shot image classification is the practice of learning accurate classifiers using a small number of labeled samples (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang and Hebert, 2016; Finn et al., 2017; Snell et al., 2017; Wang et al., 2020). has a wide range of applications from face and gesture recognition (Pfister et al., 2014) to visual navigation in robotics (Finn et al., 2017). Essentially, modern few-shot classification methods can be viewed as combination of learning (1) strong feature representation through a backbone network (e.g., Verma et al. (2019); Gidaris et al. (2019); Tian et al. (2020)), and (2) an accurate small-sample classifier (e.g., Wang and Hebert (2016); Chen et al. (2019)). Therefore, two key questions arise in few-shot image classification: (1) How can we obtain a strong feature representation? ? and (2) How can we train accurate classifiers using a small number of samples? There have been many existing methods addressing the former question using host of different techniques (Snell et al., 2017; Rusu et al., 2019; Verma et al., 2019; Mangla et al., 2020). Here focus the less-explored second question. For our purposes, will use standard feature representations and methods for training the backbone model for few-shot classification. That still leaves us with severe classifier problem than most people realize.
There is a substantial difficulty with training a classifier using a small number of samples. In particular, with very few samples, standard classification machinery is biased. In other words, although the Maximum Likelihood Estimators (MLEs) are statistically consistent and asymptotically normal (Fahrmeir and Kaufmann, 1985), it is well-established that MLEs are biased for small number of N samples, with bias of O(N-1) (Cox and Snell, 1968; Box, 1971; Whitehead, 1986; Firth, 1993). Since common logistic regression models are a type of MLEs, they are also biased (Schaefer,
Botth authors contributed equally and were ordered alphahetically.

---

Published as a conference paper at ICLR 2022
1983; Cordeiro and McCullagh, 1991; Firth, 1993; Steyerberg al., 1999). Such biases increase the error rate of the few-shot trained classifiers, and so are important in few-shot learning.
In fact, there is standard solution for bias prevention by modifying the ordinary MLEs - known as Firth's Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993). In the case of the exponential family of distributions, Firth has simplified form that penalizes the likelihood by Jeffrey's s invariant prior (Firth, 1993; Poirier, 1994), which is proportional to the determinant of the Fisher Information Matrix F. For logistic and cosine classifiers (Chen et al., 2019) which are widely used in few-shot classification, since they belong to the exponential family, Firth bias reduction can be further cast as adding a log-dets -egeemeemnnnnn penalty (og((et(E)) (F)) to the cross-entropy loss. While such standard strategies can control the bias in classifiers trained with very few samples, they have not been utilized in few-shot image classification tasks.
In this paper, we show that using Firth bias reduction produces reliable improvements in wide range of circumstances. We achieve this by deriving a simplified yet effective Firth formulation that penalizes the Kullback-Leible (KL) divergence between the uniform distribution of classes and the predictions, for both multinomial logistic regression models and cosine classifiers. Note that common regularization techniques (such as L2-regularization and label smoothing (Szegedy et al., 2016)) cannot reduce the estimation bias of classifier weights in small-sample regimes (Liu et al., 2020) as the Firth penalty does; these regularization techniques are mainly used to control model complexity of deep neural networks for training feature extractor backbones in large-sample regimes.
More concretely, our results indicate that the improvements produced by Firth bias reduction for few-shot image classification tasks are consistent across the board (1) on a wide range of feature representations, (2) with both balanced and imbalanced data, (3) for both single-layer and multi-layer classifiers, (4) for both logistic and cosine classifiers. and (5) over multiple datasets and problems. Importantly. we found Firth bias reduction to consistently yield statistically significant and positive improvements, and we did not observe any performance penalty for utilizing it. Such improvements are on the order of 0.5-2.5% and up 3% in challenging tasks with large number of classes.
Our main contributions include (I) deriving generalized expression for Firth bias reduction in few- shot multinomial logistic regression models, and providing geometrical insight into its effect on the classification probability space; (2) evaluating the efficacy of the Firth penalized multinomial logistic model few-shot scenarios, with both balanced and imbalanced data distributions; (3) showing that Firth bias reduction can be extended beyond typical logistic models, and can be successfully adopted in cosine classifiers; and (4) providing an empirical comparison of Firth bias reduction with common regularizers such as L2 and label smoothing.
2 BACKGROUND
Mathematical Notations: In this work, we assume multinomial logistic regression model for the classifier, with a total of J +-1 classes {0, 1,2, ,J.. The logistic regression weights for class j is denoted as 3j (1 S < J). The class j = 0 is the reference class with zero logistic regression weights. We assume to have total number of N samples D {(a) v1), (æw, [N)]]. The 7th
target y; is the one-hot encoding of the 7th label. The assignment probability of the 7th sample to class j is denoted as Pay:
P.. == Pr(y: = =]z,,)
(1)
1 +
The likelihood of the sample set D given the weights ,3 is denoted as Pr(v(=;8):
Pr(yle; 8) = II M 1my = PP.j.
(2)
where 1[:] denotes the binary indicator function. Therefore, the logistic log-likelihood function Clogistic is defined as
N ΣΣ
Liogistic :=
[ys= = 5l Iog((Pg)
(3)
i=1 i=1 j=0
2

---

Published as a conference paper at ICLR 2022
0.70
bias(ßmLE) = ElßmLE] B"
-0.5
Mean Estimate
l0g1o B3)
El (N)
- B
4N
0.65
11.0
0.60
E[ßMLL]]
-1.5
0.55
-2.0
FiiRinn
0.50
-2.5
5 6
7
8
10
0.0
0.5
1.0
1.5
2.0
Sample Size (N)
logio(N)
Figure 1: The MLE bias and Firth bias reduction visualized in geometric experiment. Here, the task to estimate the coin flip probability based upon the number of tosses until the first head shows up. (Left) The average MLE (shown blue) and the Firth bias-reduced estimator (shown in red) against increasing number of samples. The true generative parameter 3 = 0.5 is annotated with a star, and the MLE bias is also visualized. The Firth bias-reduced estimator removes the leading ONN term from the MLE bias even with small N. (Right) To show that the MLE bias is asymptotically of O), MLE bias plotted against the sample size N in the log-log scale.
The Fisher Information Matrix (FIM) defined the Hessian of the negative log- likelihhohoo function Llogistio:
F == - H (Clggistie) By V Clogistic V & logistic
(4)
The Maximum Likelihood Estimator (MLE) for logistic regression is defined
ML := arg max Pr(y|æ;;8)
(5)
8
The dimension of the feature space is denoted as d in the derivations. Ula.6) denotes the (discrete) uniform distribution in the (a, b) interval. The cross-entropy and the KL-divergence of distributions q are defined as
p(x)
CE(plla) = = - log(1(x)),
DKL (kl = Σ p(z) log
(6)
g(x))
Table A2 in the Appendix summarizes these notations.
Small-Sample Bias of MLE: Assume 3 is the true generative parameter. When the sample size N is small, it is shown that the MLE bias b((MLE) R[ (3r] is non-zero and of O(N-1) (Cox and Snell, 1968). Therefore, while MLE is unbiased as N -> 0o, it is inaccurate for few-shot learning.
Firth Bias Reduction for MLE: Firth's PMLE (Firth, 1993) is modification to the ordinary MLE, which removes the O term from the small-sample bias. In particular, Firth has a simplified form for the exponential family. When Pr(y|æ;; 8) belongs the exponential family of distributions, the effect is to penalize the likelihood by Jeffrey's 's invariant prior (Poirier, 1994), which is proportional the determinant of F' (det(F)), Logistic and cosine classifiers (Chen al., 2019) - the widely-used classification models in few-shot learning belong to the exponential family; so for them, Firth bias reduction simplifies to adding penalty (log(det( (F)) to the cross-entropy loss. what follows, we will derive further simplified yet effective Firth formulation for logistic and cosine classifiers, which is computationally more efficient, deals with the case when det(F)) 0, and generalizes to multinomial distributions.
MILE vs. Firth's PMLE for a Simple Case: To demonstrate the extent of the MLE bias and how Firth's PMLE removes the leading O(N-1) bias term, simulated data from the geometric distribution with probability of success 3 = 0.5 as its only parameter. The geometric experiment was chosen since closed-form solution for the MLE and Firth's PMLE can be derived. Given the
samples (V1, ... (NN from the geometric distribution, the sample mean is N Σ y: and we have
MMLE & and Brinh N-1 N--1-
Note that since the sample mean noisy, the MIE suffers from
3

---

Published as a conference paper at ICLR 2022
noisy denominator, making it biased. Figure/ shows that Srirth is close to the true parameter 3 for all sample sizes, whereas ÍMLE has a significant bias away from 3 for small N. To further validate that the MLE bias indeed of O(N-1), we plotted the MLE bias against the sample size in the log-log scale in Figure which shows that is closely following a line with negative unit slope.
FIRTH BIAS REDUCTION IN LOGISTIC AND COSINE CLASSIFIERS
In logistic models, the penalized likelihood function proposed by Firth is equivalent to imposing Jeffreys' prior (Poirier, 1994) the parameters and making maximum posteriori estimation. In particular, Firth bias reduction encourages models with "large" by multiplying the likelihood by (et(F). This penalty degenerates when det(F)) = O. We work in the highest dimensional subspace where F has full rank, and use (er(FFr) to denote the product of all non-zero eigenvalues of F. obtaining
1
Pr(B]æ, v) Pr(ula;; (Flr))))
where Z is normalization constant and det(F)r) is the Jeffery's prior. Taking the log of both sides yields the log-pone posteroor as a sum of the logistic log-likelihood function and the Firth bias reduction term:
C : log((r(3)1,,)))) v)) = Llagistic + LFinh
(8)
where we have
Crith log(det( (dc((F())) + cte.
The definition of Crinh was left ambiguous up to constant with respect to 3 to facilitate the Firth bias reduction term's interpretation and avoid the definition of similar terms. Furthermore, À controls for the impact of the Firth term on the outcome relative to Liogistie We then apply series of derivation steps to simplify Equation (9), which are left to Section A the Appendix. Finally, the Firth bias reduction term can be expressed as
Crinh
ST., log
Σ 3'
=
1 N
log(Pr.)) (P)))s)
(10)
j=0
For the cosine classifier, the proof that log(det(F)) simplifies to Equation (10) involves straightfor- ward manipulation of the proof for the logistic classifier, and is left to Section B in the Appendix. The normalization of the 3; weights in the cosine classifier turns into a pure scale term in the optimization, and for a cosine classifier, scaling of the B, weights does not affect predictions. Therefore, this term should be ignored, and the bias reduction term effectively becomes the same as Equation (10).
Interpreting the Firth Bias Reduction for Logistic Models: It is well known that Jeffery's prior shrinks the parameter estimates towards zero, which equivalent to encouraging uniform class assignment probabilities (Firth, 1993; Bull et al., 2002). We take an alternative approach to reach the same conclusion in the following. By re-arranging Equation (10), one see the Σ/ log (P..)) term as scaled average over uniform distribution of classes:
1 N
Fiith x
Σ
Σ j=0
log(P..) -1 Σ (E(U,.,)IP.))
(11)
J+1
Therefore, we can abuse the notation, and redefine the coefficient À and the constant to have
Lrint X -1
Σ (Ko, cte.
(12)
This means that by dropping the constants, the optimization objective L can be rewritten
-1 N
L
CE(y: [|+)) + A AU..
(13)
4

---

Published as a conference paper at ICLR 2022
Since the KL-divergence and the FIM define an information geometry and Riemanian metric on probabilisstic measure spaces (Nielsen, 2020), a geometrical insight into Equation (13) is provided in Figure A5 in the Appendix as well.
Firth Bias Reduction vs. Common Regularization: While this insight brings Firth bias reduction closer to the common regularization techniques (e.g.. I.2-regularizationn, it is worth noting that (1) common regularization techniques mainly focus on controling model complexity, instead of reducing small-sample estimation bias; (2) Firth bias reduction operates on much lower-dimensional target distribution space, unlike L2 which operates in the high-dimensional parameter space; (3) Firth uses the same kind of metric as the logistic loss; and (4) Firth bias reduction dimensionally consistent like the Natural Gradients (Amari, 1998; Pascanu and Bengio, 2013), whereas L2 is not.
Firth Bias Reduction vs. Label Smoothing: Notice that the original form of label smoothing used for training in large-sample regimes (Szegedy et al., 2016) has the same formulation as the simplified Firth penalty term Equation (12) for multinomial logistic classifiers. However, Firth bias reduction is inherently different in that it reduces the classifier estimation bias in the small-sample regimes, whereas label smoothing penalizes over-confident predictions when training deep neural networks with large amounts of samples. Generally, the Firth bias reduction term (i.e., log(det( (F)) is not the same as the label smoothing penalty (i.e., DKL (U][P:)) for deep neural networks. Additional analysis and empirical comparisons are provided in Section 4.4 and Section G in the Appendix.
4 EXPERIMENTAL RESULTS
Here we show that for a wide range of experiments, Firth bias reduction is a reliable source of small yet useful improvements in the performance. Since we report improvements for a wide range of methods and settings, the absolute accuracy improvements were reported. These absolute improvements sometimes constitute significantly to the bascline accuracy in terms of relative importance.
Datasets: We perform experiments on four widely-used and publicly available benchmarks: mini- ImageNet (Vinyals al., 2016), CIFAR-FS (Bertinetto al., 2019), tiered- ImageNet (Ren et al., 2018), and CUB (Wah et al., 2011). Each dataset consists of non-overlapping base, validation, and novel classes. The detailed class splits are described in Section D in the Appendix. Following the standard practice (Chen et al., 2019), we train feature backbones on base classes, cross-validate bias reduction coefficients on validation classes, and train classifiers and measure test accuracy over multiple trials novel classes.
Details regarding the setup, implementation, statistical significance, and reducing the effect of randomized factors are covered in Appendix Section D.
Implementation Details:
Baselines and Evaluation Metric: Non-penalized classifiers are used as the baseline all exper- iments compare Firth bias reduction and L2-reuularization. Absolute accuracy improvements over the bascline averaged across multiple trials are used as the evaluation metric in all experiments. Relative improvements are also shown in the Appendix, which demonstrate similar behaviors.
Summary of Results: Section 4.1 shows the efficacy of Firth bias reduction on standard feature backbones (ResNets with varying depth) and single-layer logistic classifiers. Section 4.2 shows and argues that Firth bias reduction outperforms L2-regularization on few-shot classification tasks. Next, we investigate the driving factor in Firth's improvement in Section 4.3, and show evidence for the efficacy of Firth's 's bias suppression property. We also show that Firth bias reduction can be effectively, and without any modifications, applied to imbalanced data distribution settings. Section 4.4 com- pares Firth bias reduction against label smoothing variants, and shows that Firth outperforms label smoothing. In Section 4.5 Firth bias reduction is applied to modern few-shot methods with advanced feature backbones (i.e.. WideResNet trained with strong regularization (Mangla et al., 2020)) and cosine classifiers. Experiments with additional feature backbones (DenseNet and MobileNet (Wang et al., 2019)) are included in Appendix Section F. Finally, Section 4.6 demonstrates that Firth bias reduction produces reliable improvements over the state-of-ther arr feature calibration method (Yang et al., 2021). Our collective results clearly indicate consistent pattern of improvements over large array of (1) feature representations, (2) datasets, (3) classification ways, (4) number of shots, (5) types of classifiers, and (6) with both balanced and imbalanced data distributions.
5

---

Published as a conference paper at ICLR 2022
Firth Bias Reduction
L2 Regularization
2.5
2.5
ResNet10
ResNet10
2.0
ResNet18
2.0
ResNet18
ResNet34
ResNet34
M(CC))
1.5
ResNet50
AACCC)
1.5
ResNet50
ResNet101
ResNet101
1.0
1.0
0.5
0.5
0.0
0.0
1
5
10
15
20
25
10 15
20
25
Number of Shots
Number Shots
Figure 2: Firth bias reduction produces statistically significant improvements over a wide range of backbone architectures and number of shots, whereas the common IZ-regularization cannot. Firth bias reduction (left) delivers novel class accuracy improvement over the baseline classifier with an order of around 0.5-2.5% By contrast, L2-regularization (right) is mostly statistically insignificant in the same setting. 16-way logistic classification on the mini-ImageNet dataset was conducted this experiment, with over 800 randomized and matching trials for each combination of method, backhone, and number of samples, and the confidence intervals are covered by the blobs. Absolute and delta accuracies are provided in Table A9. The same experiments were repeated with 3-layer classifier instead of a I-layer classifier in Figure A7 in the Appendix and show similar results.
4.1 FIRTH BIAS REDUCTION IMPROVES THE ACCURACY OF LOGISTIC CLASSIFIERS
Figure 2 summarizes the results. For all the k-shot tasks and backbones, the average absolute test accuracy improvement is significant and increases as increases. Also, Firth bias reduction does not hurt the 1- and 5-shot performance, with slight improvements for some backbones. Extra experiments are included in Appendix Section E.
4.2 L 22-REGULLARIZAIION IS NOT AS EFFECTIVE AS FIRTH BIAS REDUCTION
To explore whether the performance improvement pattern for Firth bias reduction can be reproduced by common regularization techniques such as L2, the same experiments were repeated with L2- regularized logistic classifiers. For almost all the k-shot tasks and feature backbones, the average absolute test accuracy improvement is close to zero, as shown in Figure 2. This suggests that the bias reduction property of Firth plays a significant part in improving the classifier's performance in the few-shot regime which cannot be achieved by L2-regularization
4.3 BIAS OR PRIOR?
Firth bias reduction clearly helps. It may be doing so because is an effective way of suppressing bias. Alternatively, reinforeing the prior information might be helping. This section offers evidence that bias suppression is what is important. We consider data where the class frequencies are imbalanced. We apply variant of Firth bias reduction that uses class prior frequencies to this data. This variant is not as successful as routine Firth bias reduction, suggesting that bias correction is what is important. One might consider replacing the uniform distribution in the Firth prior with a class probability distribution. That is, replacing Lrinh = - DKI. (U oo.] (P.) by Lrinhh = - DRL. (A][P.) with A being the imbalanced class distribution, when training Firth penalized few-shot logistic classifier.
To test this, we designed two schemes generate imbalanced datasets. These schemes differ in the imbalanced count vectors used to create the few-shot dataset validation and novel splits. The increments in the counts per class were designed to result in two different average counts of 7.5 (Scheme 1) and 15 (Scheme 2) over the classes (experimental details are left to Section D in the Appendix). For each scheme, the experiments on mini-Imagee ImageNet, except that in each trial three classifiers penalized (KI. with the uniform prior), and (3) modified Firth penalized (KI with the imbalanced
carried out similar to the "balanced few-shot" case (1) baseline (not penalized), (2) Firth

---

Published as a conference paper at ICLR 2022
7.5 Average Shots
15 Average Shots
1.25
1.25
Non-uniform Prior
Non-uniform Prior
1.00
Uniform Prior
1.00
Uniform Prior
A(CCC)
0.75
4/ACC)%
0.75
0.50
0.50
0.25
0.25
0.00
0.00
RN10
RN18
RN34
RN50
RN101
RN10
RN18
RN34
RN50
RN101
Backbone Architecture
Backbone Architecture
Figure 3: Penalizing KI. to the imbalanced class distribution worse than using the Firth bias reduction. The accuracy improvement is compared (1) when DxL. (U ([)) is penalized (i.e., the blue bars with the uniform prior label) vs. (2) when the KL divergence to the imbalanced class distribution A (i.c., ( (A))P.) is penalized (i.e., the red bars with the non-uniform prior label). The left and right plots use a class distribution with an average of 7.5 and 15 shots, respectively. The range of shots vary from 2- 16-shots in the left plot, and from 1- to 29-shots in the right plot. The difference of accuracy improvements between the uniform and the non-uniform priors is more striking for the 7.5-average-shot case, which suffers from greater deal of estimation bias than the 1 15-average-shot scenario. This shows that the improvements of Firth bias reduction are driven by its bias suppression property, and are not a mere consequence of imposing uniform class prior. The same experiments were repeated with a 3-layer classifier in Figure A10 in the Appendix.
prior) were trained. In both schemes. the improvement achieved by the Firth penalized classifier is higher than that of the modified Firth penalized classifier over the baseline for all backbones, as shown in Figure The difference between the improvements is more substantial in Scheme where the average of samples per class is less than Scheme 2. This implies that the Firth penalized classifier is indeed reducing the bias and should not be naively considered as if prior simply defined over the class assignment probabilities.
4.4 FIRTH BIAS REDUCTION IMPROVES OVER LABEL SMOOTIING
Even though the original version of label smoothing (Szegedy al., 2016) has the same formulation as the Firth bias reduction term in Equation (13), there is family of label smoothing techniques that aim to reduce overfitting and over-confident predictions (Pereyra et al., 2017). Namely, the confidence penalty (i.c., regularizing DKL (P. UU)) and unigram label smoothing (i.c., regluarizing the KL-divergence with class priors) are two label smoothing variants, which are shown to outperform the original label smoothing, when training complex networks with large amounts of samples (Pereyra et al., 2017). Figure 3 shows that Firth bias reduction outperforms unigram label smoothing, and Table A4 in the Appendix shows that Firth bias reduction yields better results than applying confidence penalty and entropy regularization. This further suggests that the improvements obtained by Firth are the result of its bias reduction property.
4.5 FIRTH BIAS REDUCTION IMPROVES THE PERFORMANCE OF COSINE CLASSIFIESS
As cosine classifiers have recently proven to be useful for few-shot classification (Gidaris and Komodakis, 2018; Chen al., 2019), we investigated the impact of Firth bias reduction on them. The difference between a logistic and cosine classifier parameter space is that the parameter space of the cosine classifier is constrained to a unit sphere, therefore is likely for the Firth bias reduction to improve cosine classifier performance as well. To test this, we based our experiments the implementation the S2M2R method (Mangla et al., 2020), and used their pre-trained WideResNet- 28- 10 (Zagoruyko and Komodakis, 2016) the base classes Firth penalized and non penalizdl cosine classifiers on the mini ImageNet, tiered- prreed-ImgeNet, and CIFAR-FS datasets. Few-shot classifiers were trained for varying number of classes, depending on
strong feature extractor. We trained
7

---

Published as a conference paper at ICLR 2022
mini-ImageNett
CIFAR-FS
1-shot
0.40
1-shot
1.00
5-shot
5-shot
AACC))
10-shot
MACC)
0.30
10-shot
0.75
0.20
0.50
0.10
0.25
0.00
10 Number of Classes
10
20
Number of Classes
tiered-ImageNet
3.00
1-shot
5-shot
A(ACC)%
2.00
10-shot
1.00
0.00
10 20
50
Number of Classes
Figure 4: Firth bias reduction consistently improves the accuracy of the cosine classifier (a better few-shot classifier) in the presence of strong backbone features, regardless of (1) the type of dataset, and (2) the number of classes in few-shot classification. For each dataset, an advanced backbone architecture (WideResNet-28- 10) with strong mixup regularization and self- supervision was trained. The error bars are almost non-ex non-existent (i.e., less than 0.02%), since over 10,000 trials with matching randomization factors were performed for each point. Firth bias reduction improvements are always positive, and it never hurts to use the Firth bias reduction technique. Absolute and delta accuracies are provided in Tables A6, A7, and A8 in the Appendix.
the dataset, and varying number of samples per class. We followed the same scheme as Mangla et al. (2020) to generate the k-shot support and query datasets for training and testing, respectively.
As shown in Figure 4, in all datasets, Firth bias reduction improves the performance of 5- and 10-shot tasks in a monotonically increasing fashion with respect to the number classes. This behavior is well represented for fered-ImageNNt, as increasing 5-way 150-way classification results in more than 2.5% improvement of the average test accuracy. In the 1-shot task, the improvement by Firth bias reduction is monotonically decreasing with the number of classes all the datasets except tiered- ImageNet, but it does not hurt the performance. The improvement almost falls within (0. %.0.5.). (0-0.1%% 1%), and (0-0.2%), for mini-ImageNNet, CIFAR-FS, and tiered- Imaee-e,,,eNe,, respectively.
This experiment suggests that not only is Firth bias reduction effective for cosine classifiers, but it also could help with the performance even when strong features are used. This is well justified as Firth bias reduction targets the bias introduced in the classifier parameters not the features.
4.6 COMPARISON WITH ADDITIONAL STATE- ARE-ART METHOD
We further applied the Firth bias reduction term to the recent state-of-the-art method in few-shot classification (Yang et al., 2021), 1), called Distribution Calibration (DC). DC computes the closest base classes to each sample in the feature space, and samples artificial examples from a Gaussian distribution centered around the mean of the closest base class features. To make the setup challenging, we consider data where the base classes categorically different from the novel classes: i.e., by performing cross-dare pross-dttaset fho-shot classification, where the features are trained on the base classes of hered-Imagenh
and the novel sets instead sampled from the CUB dataset. The results are shown in Table 1, and indicate that the Firth bias reduction improves over Yang al. (2021). We also show the Firth bias reduction improvements the fiered-ImageNe dataset in
Imageee
8

---

Published as a conference paper at ICLR 2022
mini -mtgeeet ++ CUB
tiered- Brredd-Imgeeettt -
CUB
Way
Shot
Before
After
Improvement
Before
After
Improvement
10
37.14
37.40 0.13
0.26 0.03
64.36
64.52 = 0.16
0.15 + 0.04
10
5
59.77
60.77 +: 0.12
1.00 + 0.04
86.23
86.66 1 0.10
0.43 1 0.02
15
1
30.22
30.37 + 0.09
0.15 + 0.03
57.73
57.74 + 0.13
0.01 + 0.01
15
52.73
53.84 0.10
1.12 = 0.03
82.16
83.05 + 0.09
0.90 + 0.02
Table I: The cross-domain experiments for the DC method with and without Firth bias reduction. The columns containing the novel set accuracy obtained by DC and Firth penalized DC methods are tagged Before and After, respectively. Each setting (a combination of way, shot, and method) was tested with and without data augmentation (addition of 750 samples), and the maximum accuracy was reported. Note that the confidence intervals are much smaller for the improvement column, thanks to the random-effect matching procedure we used in this study. The Before confidence intervals were similar to the After confidence intervals, and thus not repeated due to space constraints.
Appendix Section H and Table A5. This further supports the observation that Firth bias reduction yields small but reliable improvements under different methods.
5 RELATED WORK
Bias Reduction of the MLE: A myriad of statistical work has been proposed to mitigate the small- sample bias of the MLE under different settings (Anderson and Richardson, 1979; Kenward and Roger, 1997; Bull et al., 2002; Kosmidis and Firth, 2009). Originally, the asymptotic bias of MLE was shown to be of O(N-1), with N being the sample size (Firth, 1993). To counter such an estimation bias, many approaches have existed. To name few, (1) additive penalization terms the main logistic loss were proposed to reduce the MLE's bias (Firth, 1993; Bull et al., 2002; Greenland and Mansournia, 2015), and (2) some methods have been proposed to directly approximate and remove such a small sample bias (Cox and Hinkley, 1979). While the latter approach may sound appealing. estimating the MLE's bias can be impractical. For instance, in few-shot scenarios, a perfect separation of the classes may be achievable, causing the logistic MLE to be unbounded (Heinze and Schemper, 2002). On the other hand, the penalization methods do not modify the estimated parameters directly, and instead gently push for preference towards less biased estimates. Such penalization methods can be generally applicable to a vast array of models.
Theoretically, Firth's s PMLE reduces the bias by removing the leading O(N term from the MLE's bias (Firth, 1993) property that does not exist in common regularization techniques such as I.2rreulaariation. Furthermore, PMLE of the logistic model has been shown to have smaller variance than MLE as well (Copas, 1988; Kosmidis and Firth, 2009). Firth's PMLE has been well studied for binomial logistic regression (Firth, 1993), and applied and tested against other penalization techniques in other fields (Rainey and McCaskey, 2015; Muchlinski et al., 2016; Rahman and Sultana, 2017).
Additional related work on few-shot image classification was left to Appendix Section C.
6 CONCLUSION
We show that Firth bias reduction consistently improves the accuracy across the board in few-shot classification regardless (1) the employed feature backbone, (2) the number of classes and samples, and (3) the dataset and problem setting. Furthermore, experiments show that Firth bias reduction can improve the performance domain shot settings without any necessary modifications. Overall, our evaluations suggest that Firth bias reduction is a useful and general bias reduction tool that has been missing in few-shot classification, and should be incorporated in few-shot classification tasks for accuracy improvements.
the cosine classifiers, and is applicable to imbalanced and cross-

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGMENT
This work was supported part by (1) the National Science Foundation's 's Major Research Instru- mentation program (Kindratenko al., 2020), grant number 1725729, (2) the National Science Foundation's 's Creating Knowledge with All-Novel-Class Computer Vision program, grant number 2106825, and (3) the Jump ARCHES endowment through the Health Care Engineering Systems Center. The majority of NSF funding contributions were designated to Ehsan Saleh and Saba Ghaffari equally in the form of computational resource allocations on pereh-eeformacee computing platforms for conducting the experiments. Overall, this work consumed more than 32 CPU years and one Nvidia V-100 GPU year from the NSF-funded resource allocations in the course of its analysis. Also, this work made use of the Illinois Campus Cluster, a computing resource that is operated by the Illinois Campus Cluster Program (ICCP) in conjunction with the National Center for Supercomputing Applications (NCSA) and is supported by funds from the University of Illinois Urbana- Oibann-Champaighh
REFERENCES
K. R. Allen, E. Shelhamer, H. Shin, and J. B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML, 2019. 21
S.-I. Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-776 1998.
Anderson and Richardson. Logistic discrimination and bias correction maximum likelihood estimation. Technometrics, 21(1):71-78, 1979. 9
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and
N. De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016. 21
J. Baxter. A Bayesiaan/informationn theoretic model of learning to learn via multiple task sampling. Machine Learning, 28(1):7-39, 1997. 21
Bengio, Y. Bengio, J. Cloutier, and Gescei. On the optimization of synaptic learning rule. In Optimality in Biological and Artificial Networks?, pages 281-303. 2013. 21
I.. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi. Learning feed-forward one-shot learners. In NeurlPS, 2016. 21
L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi. Meta-learming with differentiable closed- form solvers. In ICLR, 2019. 5
M. Box. Bias in nonlinear estimation. Journal of the Royal Statistical Society: Series B (Methodolog- ical), 33 1971.
S. B. Bull, C. Mak, and C. M. Greenwood. A modified score function estimator for multinomial logistic regression in small samples. Computational Statistics and Data Analysis, 39(1):57774, 2002. 4, 9
W.- Chen, Y.-.. Liu, Z. Kira, Y.-. F. Wang, and J.-B. Huang. A closer look at few-shot classifica- tion. In ICLR, 2019. 1, 2, 3, 5, 7,21
J. B. Copas. Binary regression models for contaminated data. Journal of the Royal Statistical Society: Series B (Methodologicaal) 50(2):225 2533, 1988. 9
G. M. Cordeiro and P. McCullagh. Bias correction in generalized linear models. Journal of the Royal Statistical Society: Series B (Methodological), 53(3):629-66 1991. 2
D. R. Cox and D. V. Hinkley. Theoretical statistics. CRC Press, 1979. 9
D. R. Cox and E. J. Snell. A general definition of residuals. Journal of the Royal Statistical Society. Series B (Methodological), 30(2):248-22 1968. 1, 3
G. S. Dhillon, P. Chaudhari, A. Ravichandran, and Soatto. A baseline for few-shot image classification. In ICLR, 2020. 21
10

---

Published as a conference paper at ICLR 2022
M. Douze, A. Szlam, B. Hariharan, and H. Jégou. Low-shot learning with large-scale diffusion. In CVPR, 2018. 21
N. Dvomikk C. Schmid, and J. Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In ICCV, 2019. 21
N. Dvorikk C. Schmid, and J. Mairal. Selecting relevant features from a multi-domain representation for few-shot classification. In ECCV, 2020. 21
H. Edwards and A. Storkey. Towards a neural statistician. In ICLR, 2017. 21
L. Fahrmeir and H. Kaufmann. Consistency and asymptotic normality of the maximum likelihood estimator in generalized linear models. The Annals of Statistics, 13(1):342-368 1985.
L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning object categories. IEEE TPAMI, 28(4):
594-611 2006. 1, 20
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICMI, 2017. 1, 20, 21
C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. In NeurIPS, 2018. 21
D. Firth, Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27-38, 1993. 1, 2, 3, 4, 9. 15
D. George, W. Lehrach, K. Kansky, M. Lázaro-Gredilla, C. Laan, B. Marthi, X. Lou, Z. Meng, Liu, H. Wang, A. Lavin, and D. S. Phoenix. A generative vision model that trains with high data efficiency and breaks eest- CAPTCHAS. Science, )ea(662)) 2017.
S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018. 7, 21
Gidaris, A. Bursue, Komodakis, Perez, and M. Cord. Boosting few-shot visual learning with self-supervision. In ICCV, 2019.
Greenland and M. Mansournia. Penalization, bias reduction, and default priors in logistic and related categorical and survival regressions. Statistics in Medicine,
2015, 9
L. Gui, A. Bardes, R. Salakhutdinov, A. Hauptmann, M. Hebert, and Y.-X. Wang. Learning to hallucinate examples from extrinsic and intrinsic supervision. In ICCV, 2021. 21
B. Hariharan and R. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017. 20
D. A. Harville, Matrix algebra from a statistician's perspective. Technometrics, 40(2):164, 1998. 17
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 21
G. Heinze and M. Schemper. A solution to the problem of separation in logistic regression. Statistics in Medicine, 21(16):2409-2 -2119, 2002.
M. G. Kenward and H. Roger. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3):983 1997. 9
V. Kindratenko, D. Mu, Y. Zhan, Maloney, S. H. Hashemi, B. Rabe, K. Xu, R. Camphell, Peng, and W. Gropp. HAL: Computer system for scalable deep learning. In Practice and Experience in Advanced Research Computing, pages 41-48. 2020.
G. Koch, R. Zemel, and R. Salakhudtinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, 2015. 21
I. Kosmidis and D. Firth. Bias reduction in exponential family nonlinear models. Biometrika, 96(4): 793-804, 2009.
11

---

Published as a conference paper at ICLR 2022
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009. 21
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, Human-level concept learning through proba- bilistic program induction. Science, 350(6266):1332. 1338, 2015. 20
H. Li, Dong, X. Mei, C. Ma, F. Huang, and B.-G. Hu. LGM-Net: Learning to generate matching networks for few-shot learning. In ICML, 2019a. 21
H. Li, D. Eigen, S. Dodge, M. Zeiler, and X. Wang. Finding task-relevant features for few-shot learning by category traversal. In CVPR, 2019b. 21
Z. Li, F. Zhou, F. Chen, and H. Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint 1tXv:1707.00933, 2017. 21
Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc. Dense classification and implanting for few-shot learning. In CVPR, 2019. 21
B. Liu, Y. Cao, Y. Lin, Q. Li, Z. Zhang, M. Long, and Hu. Negative margin matters: Understanding margin in few-shot classification. In ECCV, 2020. 2
L. Liu, W. Hamilton, G. Long, J. Jiang, and H. Larochelle. A universal representation transformer layer for few-shot image classification. In ICLR, 2021. 21
Mangla, N. Kumari, A. Sinha, M. Singh, B. Krishnamurthy, and V. N. Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In WACV, 2020. 1, 7, 8, 22, 23
E. G. Miller, N. E. Matsakis, and P. A. Viola. Learning from one example through shared densities transforms. In CVPR, 2000. 20
N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. simple neural attentive meta-learning. In ICLR, 2018. 21
D. Muchlinski, D. Siroky, He, and M. Kocher. Comparing random forest with logistic regression for predicting class-imbalanced civil war onset data. Political Analysis, 24(1):87-103, 2016.
Munkhdalai and H. Yu. Meta networks. ICML, 2017. 21
A. Nichol and J. Schulman. Reptile: A scalable metalearning algorithm. arXiv:/803.00299, 2018.
arXiv preprint
F. Nielsen. An elementary introduction to information geometry. Entropy, 22(10):1100, 2020.
B. Oreshkin, P. R. López, and A. Lacoste. TADAM: Task dependent adaptive metric for improved
few-shot learning. In NeurlPS, 2018. 21
R. Pascanu and Y. Bengio. Revisiting natural gradient for deep networks. arXiv preprint 1rX1v:3301.358.0 2013.
G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR, 2017. 7, 26
Pfister, J. Charles, and A. Zisserman. Domain-adaptive discriminative one-shot learning of gestures. In ECCV, 2014.
C. P. Phoo and B. Hariharan. Self-training for few-shot transfer across extreme task differences. In ICLR, 2021. 21
D. Poirier. Jeffreys' prior for logit models. Journal of Econometrics, 63(2):327-339, 1994. 2,
H. Qi, M. Brown, and D. G. Lowe. Low-shot learning with imprinted weights. In CVPR, 2018. 21
S. Qiao, C. Liu, W. Shen, and A. L. Yuille. Few-shot image recognition by predicting parameters from activations. In CVPR, 2018. 21
12

---

Published as a conference paper at ICLR 2022
M. S. Rahman and M. Sultana. Performance of Firth-and logF-type penalized methods in risk prediction for small or sparse binary data. BMC Medical Research Methodology, 17(1):1-15,001,
C. Rainey and K. McCaskey, Estimating logit models with small samples. Political Science Research and Methods, 2015. 9
Ravi and Larochelle. Optimization as a model for few-shot learning. ICLR, 2017. 21
M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S.
Zemel. Meta-learning for cemi-supervised few-shot classification. In ICLR, 2018. 5,
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 11533)211-252, 2015. 21
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019. 1, 21
E. Saleh, S. Ghaffari, D. Forsyth, and Y.-X. Wang. Dataset for the importance of Firth bias http:/doiooogg///g//// org/oo.g///.........oooo 1ttp:/11..1111211111111 reduction few-shot classification, 2022. URL https: /tt////oii/// 1016367_v1. 23
A. Santoro, Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One-shot learning with memory-augmented neural networks. In ICML, 2016. 20
R. L. Schaefer. Bias correction maximum likelihood logistic regression. Statistics in Medicine, (1):71-78, 1983.
Schmidhuber. Evolutionary principles in self-referential learning. On learning how learn: The meta-meta-.. hook. Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1987. 21
Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm. adaptive Levin search, and incremental self-improvement, Machine Learning, 28(1):105-13) 1997. 21
J. Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017. 1, 20, 21
E. W. Steyerberg, M. Eijkemans, and D. F. Habbema. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of Clinical Epidemiology, 52(10): 935-942, 1999. 2
F. Sung. Y. Yang. L. Zhang, T. Xiang, H. S. Torr, and T. M. Hospedales. Learning compare: Relation network for few-shot learning. In CVPR, 2018. 21
C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 2, 7,
S. Thrun. Lifelong learning algorithms. Learning to learn, 8:181-209, 1998. 21
Y. Tian, Wang, D. Krishnan, J. B. Tenenbaum, and Isola. Rethinking few-shot image classifica- tion: A good embedding is all you need? In ECCV, 2020. 1, 21
E. Triantafillou, R. Zemel, and R. Urtasun. Few-shot learning through an information retrieval lens. In NewrlPS, 2017. 21
E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.A. Manzagol, and H. Larochelle. Meta-dataset: dataset of datasets for learning to learn from few examples. In ICLR, 2020. 21
V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICMI, 2019.
13

---

Published as a conference paper at ICLR 2022
O. Vinyals, C. Blundell, T. P. Lillierap. K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In NeurlPS, 2016. 1,5, 20, 21
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-201 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011. 5
Y. Wang, W.-L. Chao, K. Q. Weinberger, and L. van der Maaten. Simpleshot: Revisiting nearest- neighbor classification for few-shot learning, arXiv preprint arXiv: 2019. 5, 25
Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
Y.-X. Wang and M. Hebert. Learning to learn: Model regression networks for easy small sample learning. In ECCV, 2016. 1, 20, 21
Y.-X. Wang, D. Ramanan, and M. Hebert. Learning to model the tail. In NeurIPS, 2017. 21
Y.-X. Wang, R. Girshick, M. Hebert, and Hariharan. Low-shot learning from imaginary data. In CVPR, 2018, 21
J. Whitchead. On the bias of maximum likelihood estimation following sequential test. Biometrika, 73(3)5573-581, 1986.
S. Yang. L. Liu, and M. Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021. 5, 21, 22, 23, 24, 27
H.JJ Ye, H. Hu, D.-C. Zhan, and Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020. 21
S. W. Yoon, J. Seo, and J. Moon. TapNet: Neural network augmented with task-adaptive projection for few-shot learning. In /CML, 2019. 21
S. Zagoruyko and N. Komodakis. Wide residual networks. BMVC, 2016. 7,22
C. Zhang, Y. Cai, G. Lin, and C. Shen. DeepEMD: Few-shot image classification with differentiable earth mover's distance and structured classifiers. CVPR, 2020. 21
X. Zhang, Meng, H. Gouk, and T. M. Hospedales. Shallow Bayesian meta learning for real-world few-shot recognition. In ICCV, 2021. 21
14

---

Published as a conference paper at ICLR 2022
A FIRTH BIAS REDUCTION FOR FEW-SHOT MULTINOMIAL LOGISTIC REGRESSION
Table A2 summarizes the notations used throughout the main paper and here. Given dataset of N samples D = {(11 y1), (22, 92), ... (IN, yN)) the multinomial logistic model for a total of J + 1
classes be formulated
Pr(y = و [.))
log
Pr(y = 0/ = 2
j (1,2,
, ,
(A14)
where class j was chosen as the reference class in the log odds ratio. other words, w.l.o.g. we assume Bn 0 in this formulation. Given the decision rule in Equation (A14), we write
P..
P.j 1 Ey'=1
A1 VisssJ.
(A15)
+ Ey=
E Bya
Under this notation, the log- likelihood would be Liogistic data matrix X D is given
EN Ey=o 1ly = j| log(Pi.)) The
ENlaxN- Also, X == XD & Iy where the @ operator denotes the Kronecker matrix product, and the J- dimensional identity matrix is denoted as L,.
X [2]
22
(A16)
Firth (1993) has established that the bias of logistic regression can be removed by maximizing the sum of (1) the log-likelln li-likelihood Liogiaue and (2) the log-determinant of the Fisher Information Matrix (FIM). For our purposes, this presents some challenges: we have a few number samples and the FIM determinant is zero. Instead, we use the product of all non-zero eigenvalues of the FIM as its "amended determinant". To obtain this efficiently, we need to know the specific structure of the FIM.
It is important in what follows that the FIM be defined as the matrix product
F(anxx(s)= ((J)x(NJ) M(NJ)X(NJ) X(NJ)x(aJ)-
(A17)
where M is block-diagonal matrix whose 7th diagonal block is denoted Mj. We leave the definition of X and M, to the "FIM Formulation for Logistic Regression" subsection. Next, we focus on:
Determinant Amendment and Constant Dropping: Generically, we show that
log(det(F)NNJ)
Σ log(det(Ane (M,) + cte,
(A18)
where det(F)NJ) is an amended version of det(F).
Efficient Computation of lo,()et(MM))): Next, we show that
log(det(M:))
Σ log(P:.)-
(A19)
j=0
Combining these two points will lead us the simplified Firth bias reduction objective:
N
Fiith A
log(Pi.))
(A20)
=11=0
Determinant Amendment and Constant Dropping: Having F = XT. M X prompts us utilize the SVD of as
XNJxdJ UNJXNJ SNJXaJ VJJxdJ- Therefore, the FIM can written as V. VST K S) VT, where K UT. M U. Since V and U are rotation matrices, we can write det(FF) SSt K KSS), and
(A21)
N
det(K) = det(M) = det().).
(A22)
15

---

Published as a conference paper at ICLR 2022
Notation
Description
J+1
Total Number of Classes in Multinomial Logistic Regression
Bg
The Logistic Regression Weights for Class j (j E {1.2, ........................ ,J})
Po
Classification Probability of Sample Belonging to Class
Pa
The 5th Sample's s Soft Classification Probabilities
N
Number of Samples
D = ((11,))))
Logistic Regression Sample Dataset
y;
The One-Hot Encoding of the Label y/s
1[a
The Binary Indicator Function (i.e., when a b and otherwise)
F
The Fisher Information Matrix
d
Dimension of the Features
The J x J Identity Matrix
A
The Kronecker Product of Matrix by Matrix
1,xc
The All Ones Matrix with Rows and e Columns
The Maximum Likelihood Estimator (MLE)
Llogistic
The Logistic Log-Likelihood Function
Fiint
The Firth Bias Reduction Function
A
The Firth Bias Reduction Coefficient
CE(plla)
The Cross-Entropy of p and g
(pl())
The KL Divergence of p and g
Ulo.J]
The Uniform Class Assignment Probabilities
(el(A)r)
The Amended Determinant of the Degenerate Matrix A with at most r Non-zero Figenvalues (See Section A in the Appendix)
Table A2: The mathematical notations used throughout the paper.
As we have d > for most few-shot tasks, the matrix S can be viewed in the following form:
SNJ = [SNINNS o].
(A23)
where S is diagonal square matrix. Since ST = S, we have
ST K = S K
NJXNJ
(A24)
Equation (A24) and det(FF) = (St K -S) show why det(F) is zero. For mitigation, we replace del(F) with the product of the non-zero eigenvalues of F, namely (M(FFN) and call it the "amended determinant" F. Thanks to = V. (ST K S) VT, even the amended determinants J) and det(STT N) are the same:
dct(FFNN)) dSt K K- S[S det($ K - det(M) SS2).
(A25)
16

---

Published as a conference paper at ICLR 2022
yi
Llyittte DRL
P. = fz(x))
u L À (UI]PP
Figure A5: Firth bias reduction is effectively the same as minimizing the KI. divergence to the uniform class probabilities for logistic regression. Here, P. denotes the predicted distribution of classes for the sample T3, and U denotes the uniform distribution of classes. The logistic objective minimizes the KL-divergence between the true label y; and P.. while Firth bias reduction Fiith tries to tie P. with KL divergence rope to the uniform distribution over the classes.
Could J) be zero? The answer is "not" generically; (M)(MM is generically positive we will show det(M) oo; Pi.j later in Equation (A28). Also, det(S²) > holds with probability 1 for continuous data distributions. In fact, det( can only be zero when the data contains linearly dependant samples, which happens with zero probability for non-atomic data distributions. Therefore, have
log(det(F)NNJ)) Σ log(det(M:)) + log(det(SS))
(A26)
This is the same Equation (A18): since the log(det(SS) term is independent of the model's parameters, we can treat it as an optimization constant and drop it.
Efficient Computation of (og(det(M,)) We define the soft predictions of the 7th sample (excluding
the reference class) as P..1:J = Pa1 ... Pa.J Given Mi's definition in Equation (A31), can write
M (P.1)) - Pi,N. PT 1.1:
(A27)
Next, we use the
Lemma (Harville, 1998) to compute de,(M.):
det(M,) det((Dag(PP..... (1 (1-PT. PT,,.. Diag((P....)) P..1.))
J
I Pay) (1 -1- PT P 1xx]
[I Pa.) P.o j=1
II Pa.. 3=0
(A28)
Taking the log will give us Equation (A19).
The FIM Formulation for Logistic Regression: Elementary methods established that the FIM of the logistic classifier is composed of x J block matrices, each with a dimension of d x (where is the dimension of the features). These block matrices can be expressed
N
F35 =
Σ
Pas (1-P
x6. x!
A 1 15j5J,
1=1
Fj, =
Σ t=1
Pij Ps.k 24 x!
V
15j4ksJ
(A29)
17

---

Published as a conference paper at ICLR 2022
5-way
10-way
2.5
2.5
ResNet10
ResNet10
2.0
ResNet18
2.0
ResNet18
ResNet34
ResNet34
M(CC))
1.5
ResNet50
A(ACC)%
1.5
ResNet50
ResNet101
ResNet101
1.0
1.0
0.5
0.5
0.0
0.0
1
5
10
15
20
25
10
15
20
25
Number of Shots
Number Shots
Figure A6: Improvements of Firth-penalized logistic classifier over the baseline on mini- ImageNet with 5 and 10 number of classes. Except for the number of classes, the experiments performed in the same setting as in Figure 2.
This facilitates the expression of the FIM as shown in Equation (A17): We can define X as the Kronecker product of (1) the data matrix XD and (2) the J-dimensional identity matrix T,:
x22
XD ==
x XD @ Ls.
(A30)
N
Nxd
M would then be block-diagonal matrix whose th block is defined
(M.),. = Pi.(1 - Pi.s)
A 1 15j
(M:)aa- Pi.
I5jAkEJ.
(A31)
B FIRTH BIAS REDUCTION FOR COSINE CLASSIFIERS
Section A derived the Firth bias reduetion term for the logistic classifier DKL (U](P,). Here, generalize this observation to cosine classifiers, and prove that the Firth bias reduction for cosine classifiers reduces down to the same form the obtained for logistic classifiers.
By defining the normalization transformation
3
13T
T
T(Ø)
T(x)
N
(A32)
113
||
x
|INI
|l
we have the log-liieldhod relation between the cosine classifier and the logistic model:
Leosine (3: x, y) Liogistic (T)))
(A33)
According to Equation (4) in the main paper and the chain rule, we can write
F Feusine (8,r, y) = Fragistic (T(3). T(x),)) AT,
(A34)
where Ad(J+1)xd(J+1) is the Jacobian matrix of T(3) with respect 3. It can be shown that A is symmetric block-diagonal matrix, whose jth diagonal block is
3
A₁
Laxd
(A35)
||8
3,
18

---

Published as a conference paper at ICLR 2022
Firth Bias Reduction
L2 Regularization
1.0
1.0
ResNet10
ResNet18
0.8
0.8
ResNet34
AACC))
0.6
MACC)
0.6
ResNet50
ResNet10
ResNet101
0.4
0.4
ResNet18
0.2
ResNet34
0.2
ResNet50
0.0
ResNet101
0.0
1
5
10
15
20
25
10 15
20
25
Number of Shots
Number Shots
Figure A7: Same experiment as Figure 2 in the main paper but with a 3-layer logistic classi- fier instead of I-layer logistic classifier (in the main paper). Again, Firth bias reduction improves the accuracy for different backbones and number of samples (left), whereas L2-regularization is not effective at all (right). In the very hard cases, i.e., and 5-shots, there is mild improvement by Firth and no deterioration. The error bars, representing the 95% confidence intervals, are covered by the blobs.
Firth Bias Reduction
L2 Regularization
12.5
12.5
ResNet10
10.0
ResNet18
AACCY/CCC%
A(ACC)/CC%
ResNet34
7.5
7.5
ResNet50
ResNet10
ResNet101
5.0
5.0
ResNet1 8
2.5
ResNet34
2.5
ResNet50
ResNet101
0.0
1
5
10
15
20
25
10
15
20
25
Number of Shots
Number of Shots
Figure A8: Relative accuracy improvements corresponding to Figure A7 with the 3-layer logistic classifier. The behavior of the relative accuracy improvements is similar to the absolute accuracy improvements for all the backbones both techniques.
Therefore,
log(det(F (Fuusime) log(dct Flogistic) )) + log(det(A))
(og(de((4iggun logistic) )) +2 Σ
log(det(A,..))
(A36)
j=0
It is obvious that
0
|| 8g
Aj
(A37)
v.
TP.
Therefore, Aj.y can be thought as an identity- proportional matrix in the sub-space perpendicular to By. Therefore, its amended determinant
Log(det(A),;i d 1)) = (-dd-1) 1og(113,
(A38)
For the plain logistic model, the Firth penalized optimization proble is:
max
Liogiatic (3; + log(det (Flogisst (8, y)))
(A39)
19

---

Published as a conference paper at ICLR 2022
ResNet10 Backbone
ResNet18 Backbane
ResNet34 Backbone
1-Layer Classifier
L-yayer Classifier L-yayer Classifier
Layee Classifier
3-Layer Classifier
Layer Classifier
9.5
9.5
9.0
ACC%
9.0
ACC%
9.0
8.5
8.5
8.5
8.0
8.0
8.0
7.5
7.5
7.0
10
20
25
10
Number of Shots
Number Shots
Number of Shots
ResNet50 Backbone
ResNet101 Backbone
10.0
1-Layer Classifier
1-Layer Classifier
Classifier
3-Layer Classifier
8.5
8.5
8.0
8.0
7.5
7.5
7.0
7.0
1
10
15
20
25
10
15
20
25
Figure A9: Absolute test accuracy values of 16-way classification for the non-penalized 3-layer and single-layer logistic classifiers on the novel set of mini-ImageNet for different backbones and varying number of samples.
For the cosine classifiers, the Firth penalized optimization problem is:
max Llogistic (T()): + A log((det (F (T(3), 8))) - C log( [|3]|11,]]] 11] (A40)
==0
where have C == 2X(d 1). By re-parameterizing $ (3) with us ((3) and
$9 111B11],, we have
maxx 48.08
Liogistic (Ma;; T(x), Logídet (ug, T(x), log(sgy)
(A41)
Since u. and s3 are independent optimization parameters, this problem can be restated as
max
Liogistic (u3; T()) + A log(det((igggng (uz, T(x), w)))) + max
C
(og(s
(3yz)
j=0
(A42)
Since Sg has no effect on the predictions of the model, it can ignored. Therefore, we end up with the optimization problem
max Ng
Aiogistic (ux; + log(del (Faedddee(Fogtttti (ug, T(x),
(A43)
Essentially, this suggests applying the same bias reduction form to cosine classifiers as the one used for logistic classifiers log(dct( F Flogisne T(x). y))) reduces down to DKL (U)( as proven in Section and Liogistic (u.:: (x), y) the cross-entropy loss with the true labels.
C ADDITIONAL RELATED WORK ON FEW-SHOT IMAGE CLASSIFICATION
As one of the unsolved problems in machine learning, few-shot learning (Miller al., 2000; Fei-Fei et al., 2006) has attracted growing interest in the deep learning era (Lake et al., 2015; Santoro et al., 2016; Wang and Hebert, 2016; Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017: Hariharan and
20

---

Published as a conference paper at ICLR 2022
7.5 Average Shots
15 Average Shots
1.0
Non-uniform Prior
1.0
Non-uniform Prior
Uniform Prior
Uniform Prior
0.8
0.8
MACC)
AACC))
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
RN10
RN18
RN34
RN50
AN101
RN10
RN18
RN34
RN50
RN101
Backbone Architecture
Backbone Architecture
Figure A10: Same experiment as Figure: 3 in the main paper but with a 3-layer logistic classifier instead of single-layer (in the main paper). Again, in both schemes (7.5 and 15 average shots), the Firth penalized classifier results in more accuracy improvements than the classifier penalized with KL-divergence the non-un non-uniform prior. This further suggests that Firth bias reduction indeed reducing the bias and is not simply imposing a uniform prior. Also, it reaffirms that plain Firth bias reduction can safely be used on imbalanced data.
Girshick, 2017; George et al., 2017; Triantafillou al., 2017; Edwards and Storkey, 2017; Mishra
et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019; Dvornik et al., 2019; Allen et al., 2019; Li et al., 2019a; Yoon et al., 2019; Lifchitz ct al., 2019; Li et al., 2019b; Zhang et al., 2020; Ye et al., 2020; Dhillon et al., 2020; Triantafillou et al., 2020; Tian et al., 2020; Dvornik et al., 2020; Yang et al., 2021; Phoo and Hariharan, 2021; Gui et al., 2021; Liu et al., 2021; Zhang et al., 2021), Successful generalization from few training samples requires "inductive biases" shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning, and more recently, meta-learning (Thrun, 1998; Schmidhuber, 1987; Schmidhuber al., 1997; Bengio et al., 2013). By explicitly "tearning-to-learn" over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task -ggoostcc knowledge to few-shot learning problems of novel classes. Within this paradigm, various types of meta- knowledge has been explored, including (1) a generic feature embedding or metric space, which images are easy to classify using distance-based classifier such as cosine similarity or nearest neighbor (Koch al., 2015; Vinyals al., 2016; Snell al., 2017: Sung al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol and Schulman, 2018; Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi and Larochelle,
2017; Munkhdalai and Yu, 2017; Li et al., 2017; Rusu et al., 2019); (3) transferable strategy to estimate model parameters based on few class examples (Bertinetto et al., 2016; Qiao et al., 2018; Qi et al., 2018; 8; Gidaris and Komodakis, 2018), or from an initial small dataset model (Wang and Hebert, 2016; Wang et al., 2017). Some most recent work (Gidaris and Komodakis, 2018; Chen et al., 2019; Tian al., 2020) also showed that the performance of these complex models can be matched by simple representation learning on base classes and classifier fine-tuning on novel classes.
D DATASETS AND ADDITIONAL EXPERIMENT DETAILS
mini-ImageNNet consists of 64, 16, and 20 classes from ImageNet (Russakovsky et al., 2015) for base, validation, and novel sets, respectively. Each class contains 600 images of size 84 x 84. tiered- consists 351, 97, and 160 classes from ImageNet for base, validation, and novel sets, respectively. Unlike mini- Imageem, the classes could have varying number of samples in ticred-ImageNet, but the images are of the same size. CIFAR-FS is a random split of CIFAR-100 (Krizhevsky and Hinton, 2009) with images of size 32 32 into 64, 16, and 20 classes for base, validation, and novel sets, respectively. CUB consists of 11,788 images of size 84 x 84 which are split into 100, 50, and 50 classes for base, validation, and novel sets, respectively.
Datasets:
Setups and Implementation Details: Our experiments fall into two main categories: (1) the stan- dard ResNet (He et al., 2016) feature experiments for logistic classifiers, and (2) more powerful,
21

---

Published as a conference paper at ICLR 2022
Setting
Hyper-parameter
Value
Learning Rate
0.005
Mini-bated Size
10
Number of Classes
16
All Standard Backbone Experiments
Optimizer
SGD
Splits
90%-10%
Backbones
ResNet 10, 18, 34, 50, 101
Balanced Data Experiments in Sections 4.1 and 4.2 in the Main Paper
Number of Shots
1, 10, 15, 20, 25
Firth Coefficients Set*
0, 0.01, 0.03, 0.1, 0.3, 3,
L2 Coefficients Sct**
0, 10, 30, 100, 300, 1000
7.5-Shot Class Distribution
2, 2, 2, 4, 4, 4, 8, 8, 8, 16, 16, 16, 16
Imbalanced Data Experiments in Section 4.3 in the Main Paper
15-Shot Class Distribution
1, 1, 9, 13, 13, 17, 17, 21, 21, 25, 25, 29, 29
Experiments
Number of Epochs
400
Sections 4.1, 4.2, and 4.3 in the Main Paper with I-Layer Logistic Classifier
Classifier Architecture
Number of Trials
More than 800
Number of Epochs
100
Experiments in Section E.2 with 3-Layer Logistic Classifier
Classifier Architecture
Peatures=>100-ReLUUU
ReLL -Softma
Number of Trials
More than 400
Table A3: Experimental settings used for the standard backbone experiments. The table is partitioned into sections, where the first section shows the global hyper-parameters used in standard backbone experiments. The same set of Firth bias reduction and L2 regularization coefficients were used for all validation experiments. *The Firth regularization coefficients were chosen for Equation (13) in the main paper. #*We defined the L2-regularization the mean squared value of all classifier parameters, which is why the normalized set of coefficients seems large. The typical unnormalized regularization coefficients can be obtained by dividing these normalized coefficients by the number of classifier parameters.
Number of Shots
Confidence Penalty Improvements
Firth Improvements
0.13 + 0.13 %
0.23 + 0.06
10
0.52 + 0.14 %
0.73 + 0.07 %
15
0.57 + 0.18 %
1.00 + 0.07
Table A4: Comparing Firth bias reduction against the confidence penalty label smoothing technique. The confidence penalty is defined as DKL (P,|IU) regularization term, whereas Firth bias reduction for logistic and cosine classifiers reduces to DKL (U)(P) penalty. The experimental setting is the same as Figure A7 with the ResNet-10 backbone. This suggests that the improvements obtained by Firth the result of its bias suppression property, and Firth cannot be replaced by standard label smoothing techniques.
thtte-tf- WideResNet (Zagoruyko and Komodakis, 2016) features trained with strong reg- ularization techniques (manifold mixup) and additional supe-superviiion (Mangla et al., 2020), or further calibrated via feature transformations (Yang al., 2021), for logistic and cosine classifiers.
22

---

Published as a conference paper at ICLR 2022
DenseNet
3.00
MACC))
2.00
1-Shot
5-Shot
1.00
10 10-Shot
0.00
5 10 20
50
100
150
Number of Classes MobileNet
3.00
A(ACC)%
2.00
1-Shot
5-Shot
1.00
10-Shot
0.00
5 10 20
50
100
150
Number of Classes
Figure A11: Firth bias reduction improves the accuracy of the logistic classifier on tiered- ImageNet for both DenseNet (top) and MobileNet (bottom) as backbone architectures. The improvement is consistent over the number of classes in few-shot classification. The error bars are almost non-existent (i.e., less than 0.02%), since over 10,000 trials with matching randomization factors were used for obtaining each single point. Firth bias reduction improvements always positive, and never hurts to use it.
For the first category of experiments, we averaged our results over 400 random trials. An array of 5 different ResNet architectures were trained in this category: ResNet10, 18, 34, 50, and 101, following simple pipeline in the Pytorch library example'. It is worth noting that deliberately did not engineer strong features for this category experiments. The mini-ImageNet dataset was used for these experiments, and 16-way classification was performed both the validation and novel classes. For this purpose, 16 out of 20 novel classes were chosen at random once and fixed for all the evaluations. We split the validation and novel classes into 90% training and 10% held-out for accuracy evaluation. The imbalanced data settings were chosen to either have 7.5 or 15 average number shots per class. For the second category of experiments, pre-trained feature backhones from Mangla et al. (2020); Yang et al. (2021). These experiments were conducted on three benchmarks, and each hyper-pann apper-parameter configuration was averaged over 10,000 random trials. The backbone parameters and extracted features for all datasets are publicly available the Illinois Data Bank repository (Saleh al., 2022). Additional details and hyper- parem-parmeeters are covered in the subsequent sections.
used the WideResNet-28-- 10
Additional Implementation Details for Sections 4.1 and 4.2 in the Main Paper:
The L.2-
regularization coefficient chosen for the mean-squared value of all classifier parameters, and the Firth bias reduction coefficient was chosen for Equation (13) in the main paper.
Statistical Significance and Reducing the Effect of Randomized Factor:
Our study investigates
improvements over the baseline. Random effects (random number seed; batch ordering; parameter initializatinn; and on) complicate the study by creating variance in the measured improvement. We used a matching procedure (so that the baseline and the Firth penalized models share the same values of all random effects) control this variance. As long that yield good improvement (we did not), this yields an unbiased estimate of the improvement. Each experiment is repeated multiple times obtain confidence intervals. Note that (1) contidence intervals are small; and (2) improvements occur over a large range of feature backbones and datasets.
does not search for random effects
thub. http::/ghhhtbb..ommpppttcr//////
23

---

Published as a conference paper at ICLR 2022
No Artificial Samples
750 Artificial Samples
Way
Shot
Before
After
Improvement
Before
After
Improvement
1
59.44
60.07 = 0.16
0.63 + 0.04
61.85
61.90 0.17
0.05 0.02
10
5
80.52
80.85 = 0.12
0.33 + 0.03
79.66
80.07 + 0.13
0.42 + 0.04
15
1
52.68
53.35 + 0.13
0.67 + 0.03
54.57
54.62 + 0.14
0.05 0.02
15
5
75.18
75.64 + 0.11 0.46 + 0.03
73.88
74.40 0.11 0.53 0.04
Table A5: The Firth bias reduction accuracy improvements the (eered-IaageNe dataset when 0 or 750 artificial samples were generated from the calibrated normal distribution in Yang al. (2021).
Before
After
Improvement
Before
After
Improvement
5-way
10-way
1-shot
74.96
75.03 + 0.19
0.07 + 0.01
61.46
61.49 + 0.13
0.03 + 0.00
5-shot
87.43
87.48
0.13
0.06 + 0.00
77.73
77.83 + 0.10
0.10 + 0.00
10-shot
89.83
89.88 + 0.11
0.05 + 0.00
81.52
81.64 + 0.09
0.11 + 0.00
15-way
20-way
1-shot
53.45
53.47
0.10
0.02
0.00
47.78
47.79 + 0.07
0.01 + 0.00
5-shot
70.70
70.99 + 0.07
0.28 + 0.00
65.26
65.60 + 0.03
0.34 + 0.00
10-shot
75.37
75.71
0.06 0.34 0.00
70.57
70.99 + 0.02
0.42 + 0.00
Table A6: The Firth bias reduction improvements on the CIFAR-FS dataset shown in Figure 4 in the main paper. "Before" stands for the novel set accuracy without having any Firth bias reduction, and "After" stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column, thanks the random-effect matching procedure we used in this study. The "Before" confidence intervals were similar to the "After" confidence intervals, and thus not repeated due space constraints.
is safe conclude that Firth bias reduction reliably offers small but useful improvement in accuracy for few-shot classifiers.
Additional Implementation Details for Section 4.3 in the Main Paper: We used two non-uniform count vectors with different average counts, 7.5 (scheme 1) and 15 (scheme 2), to generate the datasets both validation and novel sets. The count vector with the average of 7.5 7.5-shots had classes for each count from the geometric sequence {2, 4, 8, 16). and the count vector with the average of 15-shots had 2 classes for each count from the arithmetic sequence (1, 5, 9, 13, 17, 25, 29). The same 1- 1-layer logistic classifier of Sections 4.1 and 4.2 was used in Section 4.3 in the main paper.
Table A3 summarizes the Syper-parametert used in all the standard backbone experiments. Also, Figure A8 shows the relative accuracy improvements corresponding to Figure A7. Figure A12 contains the validation accuracy versus Firth coefficient X for the experiments of Figure and Figure A7.
Additional Implementation Details for Section 4.5 in the Main Paper:
the experiments carried out in Section 4.5 in the main paper, a I-layer cosine classifier was used. Also, for the Firth bias-reduced cosine classifier, the regularization coefficient was tuned for each (N, J) pair, with N representing the number of samples per class and J being the number of classes. For J-way classification on the novel set when J happened to be larger than the number of classes in the validation set (Jaa), the coefficient tuned for - classification adopted.
24

---

Published as a conference paper at ICLR 2022
Before
After
Improvement
Before
After
Improvement
5-way
10-way
1-shot
65.17
65.59
0.18
0.41 0.02
50.38
50.64 + 0.11
0.26 0.01
5-shot
82.60
83.04 + 0.12
0.44 + 0.01
71.15
71.91 = 0.10
0.76 + 0.02
10-shot
86.82
87.04 + 0.09
0.22 + 0.01
77.34
77.87 + 0.08
0.52 + 0.01
15-way
20-way
1-shot
42.65
42.85 + 0.08
0.20 + 0.01
37.56
37.76 + 0.07
0.20 + 0.00
5-shot
63.73
64.76
0.07
1.03 0.01
58.35
59.52 + 0.04
1.17 + 0.01
10-shot
70.87
71.71 + 0.05
0.84 + 0.01
66.06
67.12 + 0.03
1.06 + 0.01
Table A7: The Firth bias reduction improvements on the mini-ImageNet dataset shown in Figure 4 the main paper. "Before" stands for the novel set accuracy without having any Firth bias reduction, and "After" stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column, thanks to the random-effect matching procedure used in this study. The "Before" confidence intervals were similar to the "After" confidence intervals, and thus not repeated due to space constraints.
For the standard backbone experiments on mini- Imagenet,,,,, we trained more than 384,000 3-layer and 768,000 1-layer logistic classifiers for the balanced data settings. For the imbalanced settings on mini-Imagennt, we trained more than 64,000 3-layer and 128,000 I-layer logistic classifiers. For the cosine classifier experiments, we trained over 1.92, 1.92, and 3.36 million classifiers for mini-ImageNet, CIFAR-FS, and tiered- Imaeedet datasets, respectively.
E ADDITIONAL EXPERIMENTS ON THE STANDARD BACKBONES
E ADDITIONAL. LOGISTIC CLASSIFIER EXPERIMENTS
The experiments of Figure were repeated to perform 16 16-way classification using logistic classifier on tiered- ImageNet and CIFAR-FS in Figure A13. Moreover, 5-way and 10-way classification was tested for mini- ImageNe Figure A6 in the same setting as Figure 2. The results show that Firth improvements always exist and is even more effective as the number of classes increases.
E.2 3-LAYER LOGISTIC CLASSIFIERS FOR THE STANDARD BACKBONES
We conducted the same experiments as in Sections 4.1 and 4.2 in the main paper but with a 3- layer logistic classifier. Again, see consistent accuracy improvements for the Firth bias-reduced classifier over the non-penalized (baseline) classifier in Figure A7. This further supports the idea that Firth bias reduction boosts the performance of any reasonable classifier. Needless to say. L2-regularization is not effective as shown in Figure A7.
Furthermore, the imbalanced few-shot classification in Section 4.3 in the main paper was repeated with the 3-layer logistic classifier in Figure A10. Again for both schemes. the Firth penalized classifier has larger accuracy improvement than the classifier penalized with the Kl.-divergence the non-uniform prior over the class probabilities. This further validates the effectiveness of Firth bias reduction reducing the parameter estimation bias present the few-shot setting.
F ADDITIONAL FEATURE BACKBONES
To test the Firth bias reduction technique for additional backbones, we used pre-trained DenseNet and MobileNet backbones tiered-Imas ImageNet from Wang al. (2019). The accuracy improvements of Firth penalized logistic classifier over the baseline averaged over 10,000 trials are plotted in Figure A11. Regardless of the number of classes, the improvements always positive.
25

---

Published as a conference paper at ICLR 2022
Before
After
Improvement
Before
After
Improvement
5-way
10-way
1-shot
73.50
73.64 0.25
0.14 0.03
61.20
61.44 + 0.16
0.24 + 0.01
5-shot
88.00
88.31 + 0.12
0.30 + 0.01
79.41
80.01 = 0.11
0.60 + 0.01
10-shot
90.94
91.14
0.10
0.21 + 0.01
83.88
84.47 + 0.09
0.58 + 0.01
15-way
20-way
1-shot
53.90
53.97 + 0.15
0.07 + 0.01
48.81
48.96 + 0.11
0.15 + 0.01
5-shot
73.33
74.21
0.09
0.88 + 0.01
68.58
69.71 + 0.08
1.13 + 0.01
10-shot
78.70
79.57 + 0.08
0.86 + 0.01
74.58
75.74 + 0.07
1.16 + 0.01
50-way
100-way
1-shot
33.91
34.13
0.06
0.22 + 0.01
24.80
25.00 + 0.03
0.20 + 0.00
5-shot
52.60
54.71 + 0.05
2.10 + 0.01
41.03
43.59 +:2 0.03
2.56 + 0.01
10-shot
59.67
61.94
0.04
2.27 0.01
47.77
50.81 + 0.02
3.04 + 0.01
150-way
1-shot
20.37
20.56 + 0.02
0.19 + 0.00
5-shot
34.89
37.54
0.01
2.65 + 0.01
10-shot
41.04
44.28 + 0.01
3.25 + 0.01
Table A8: The Firth bias reduction improvements on the tiered-ImageNee dataset shown in Figure 4 in the main paper. "Before" stands for the novel set accuracy without having any Firth bias reduction, and "After" stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column, thanks the random-effect matching procedure used in this study. The "Before" confidence intervals were similar to the "After" confidence intervals, and thus not repeated due to space constraints.
G COMPARING FIRTH BIAS REDUCTION AGAINST STANDARD LABEL SMOOTHING TECHNIQUES
To demonstrate that Firth bias reduction cannot simply be replaced with label smoothing, we tested two advanced variants of label smoothing that are superior the original version as proposed by Pereyra et al. (2017). The first variant, called confidence penalty, uses the entropy of the classifier's output (or equivalently, reverses the direction of the KL divergence in the original version of label smoothing (Szegedy et al., 2016)); and the second variant, called unigram label smoothing, uses prior distribution over the classes instead of uniform, which has been shown be advantageous when the output labels' distribution is imbalanced in Pereyra et al. (2017). Note that both variants were investigated for training full deep neural network with a feature extractor backbone in large-sample regimes in Pereyra et al. (2017). Our experiments in Figure 3 evaluate the effect of unigram label smoothing when training the classifier the small-sample regime.
We also performed more experiments in the same setting to compare Firth bias reduction against confidence penalty regularization. As summarized in Table A4, all the settings Firth bias reduction has larger significant improvements than the confidence penalty technique. This further supports the value using Firth bias reduction and the fact that its impact on few-shot classification cannot be reproduced with well-known and widely-used label smoothing techniques.
26

---

Published as a conference paper at ICLR 2022
Before
After
Improvement
Before
After
Improvement
ResNet10
ResNet18
1-shot
7.96
7.97 + 0.06
0.01 0.01
7.57
7.58 0.05
0.01 + 0.01
5-shot
8.22
8.23 + 0.06
0.01 + 0.01
7.94
7.95 + 0.06
0.01 + 0.01
10-shot
8.19
8.24 0.05
0.06 0.05
8.01
8.15 0.05
0.14 + 0.06
15-shot
8,21
8.36 + 0.05
0.15 + 0.06
8.09
8.47 + 0.05
0.38 + 0.06
20-shot
8.22
8.51 + 0.05
0.28 + 0.06
8.15
8.75 + 0.06
0.60 + 0.06
25-shot
8.25
8.51 0.05
0.25 + 0.06
8.27
8.92 + 0.05
0.65 + 0.06
ResNet34
ResNet50
1-shot
7.47
7.48 + 0.05
0.01 0.01
7.51
7.52 + 0.05
0.01 + 0.01
5-shot
7.69
7.70 + 0.05
0.01 + 0.01
7.73
7.78 +: 0.05
0.05 + 0.05
10-shot
7.73
7.96 + 0.05
0.23 + 0.05
7.83
8.59 + 0.05
0.76 + 0.06
15-shot
7.79
8.22 0.05
0.43 0.06
7.89
9.02 0.05
1.13 + 0.06
20-shot
7.87
8.41 + 0.05
0.55 + 0.06
8.15
9.67 + 0.06
1.52 + 0.06
25-shot
7.97
8.54 + 0.05
0.57 0.06
8.44
10.71 + 0.06
2.27 + 0.06
ResNet101
1-shot
7.63
7.65 + 0.05
0.02 + 0.02
5-shot
7.81
7.88
0.05
0.07 0.05
10-shot
7.96
8.81 + 0.05
0.86 + 0.06
15-shot
8.10
9.50 + 0.05
1.40 + 0.06
20-shot
8.37
10.29
0.06
1.91 0.06
25-shot
8.74
10.62 + 0.06
1.88 + 0.06
Table A9: The Firth bias reduction improvements on the mini-ImageNet dataset shown in Figure 2 in the main paper. "Before" stands for the novel set accuracy without having any Firth bias reduction, and "After" stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column, thanks the random-effect matching procedure we used in this study. The "Before" confidence intervals were similar to the "After" confidence intervals, and thus not repeated due to space constraints. worth noting that we deliberately did not engineer strong features for this experiment (stronger feature backbone results are shown Sections 4.5 and 4.6 in the main paper). This diversifies Firth's 's performance portfolio, demonstrating its robustness to the strength of the feature backbones, with weak features, Firth bias reduction substantially improves the accuracy with high relative improvements as shown here and in Figure A8.
H ADDITIONAL COMPARISON WITH STATE OF THE ART
Table A5 summarizes the accuracy improvements obtained by integrating Firth bias reduction with the distribution calibration method (Yang et al., 2021) under different shots and ways on the tiered- ImageNet dataset. This method calibrates the features to follow a normal distribution, and generates artificial samples from the estimated normal distribution classification. In its state-of-the setting, the features are transformed using Tukey transformations, 750 artificial samples generated per class, and a logistic classifier is used. We tested Firth bias reduction two scenarios: (1) state-of oftee---t--t-t-t---e setting without generating artificial samples from the calibrated distribution; and (2) hate-of-the-art setting with 750 artificial samples generated per class,
data augmentation to aid few-shot
27

---

Published as a conference paper at ICLR 2022
3-Layer Logistic Classifier
25-Shol
20-Shot
15-Shat
10.5%
11.0%
Valdiatinn Accurscy
10.0%
9.2%%
10.0%
0.5%
9.0%
8.8%
8.5%
8.5%
8.0%
10-
10-2
10
10-
10"2
10-2
10-Shot
1-Shot
0.0%
7.4%
0.6%
8.4%
7.8%
7.2%
Valcaaioon
0.2%
7.6%6
8.0%
7.4%
7.0%
7.6%
7.2%
6.6%
10
10
10-1
10"
10-
10
10
10-1
10
10-
10
10
10-2
10
10
Regularizstion Coefficient
Firth Confficient
Regularization Coofficient
+ ResNee10
R8sNot18
+
ResNet34
+
ReaNet50
ResNet201
L-Layer Logistic Classifier
25-Shot
20-Shot
15-Shot
11.0%
Validation Accuracy
10.0%
10.0%
9.0%
9.0%
9.0%
8.0%
8.0%
7.5%
7.0%
10'
9.0%
8.455
Valdditin Accurwcy
8.5%
8.25
8.0%
7.5%
8.0%
7.5%
7.6%
7.0%
7.2%
6.5%
10-3
10-3
10-1
10c
10:
10-3
10-2
10-1
100
10:
10-5
10-2
10-3
10
102
Firth Regularization Coefficient
Firth Regularization Coefficient
Firth Regularizatin Coefficient
+ ResNet10
18
+ R4
+ ResNet50
ResNet101
Figure A12: The effect of the validation coefficient À on the validation accuracy for different number of shots and backbone architectures. The top two rows belong to the 3-layer logistic classifier Figure A7 in the Appendix, and the bottom two rows belong to the 1-layer logistic classifier in Figure 2 in the main paper.
shown in Table A5, The 95% confidence intervals for the accuracy improvements reported in both cases.
As shown in Table A5, Firth bias reduction produces positive improvements in all cases, which are statistically significant in all the cases. As expected, Firth bias reduction produces larger improve- ments when artificial sample generation is disabled, and thus there is larger maximum likelihood estimation bias on the logistic classifier (despite the use of Tukey transformation). In the case that artificial samples added and they tuned follow the original normal distribution, having 750
28

---

Published as a conference paper at ICLR 2022
Firth Bias Reduction (tiered- ImageNet)
L2 Regularization (tiered-ImageNed)
ResNet10
ResNet10
ResNet18
ResNet18
ResNet34
ResNet34
((CCC)%
ResNet50
A/ACC)
ResNet50
ResNet101
4
ResNet101
2
2
0
1
5
10
15
20
25
10
15
20
25
Number of Shots
Number of Shots
Firth Bias Reduction (CIFAR-FS)
L2 Regularization (CIFAR-FS)
8
ResNet10
ResNet10
ResNet18
ResNet18
6
ResNet34
6
ResNet34
S/ACC)%
ResNet50
AAACC%
ResNet50
ResNet101
ResNet101
2
2
0
0
1
5
10 15
20
25
10
15
20
25
Number Shots
Number of Shots
Figure A13: Comparing the effect of Firth bias reduction against I.-reuularization for the novel set classification accuracy on the tiered- Ierred- and CIFAR-FS datasets. Firth bias reduction (left) delivers a novel class accuracy improvement over the baseline classifier up to 8.0%. By contrast, L2- L2-regularization (right) is mostly statistically insignificant in the same few-shot setting. 16-way logistic classification was conducted in this experiment, with over 1,000 randomized and matching trials for each combination of method, backbone, and number of samples.
of them can to some extent alleviate the bias in the estimation of the logistic classifier's weights in the few-shot regime. However, the results show that even in the presence of more data (artificial samples), Firth bias reduction is still effective. This suggests that producing artificial samples to augment data cannot resolve the estimation bias issue, as they are more likely to be similar to the limited real samples available. As shown in Section 4.6, this becomes the cross-domain few-shot setting; producing artificial samples significantly less effective than Firth bias reduction, due to the domain shift.
more severe problem
29


Reviews:
[{"review_id": "zaiMy6-4X78", "reviewer": "Reviewer_4kx4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Main review\n\nThis paper is clearly written with extensive experiments to support the proposed approach.\n\nThe strength of this paper is related to the effective computation of the Firth bias reduction term resulting to a simple pragmatical implementation. This pragmatical implementation turns out to be a theoretical justification of the Label smoothing proposed in Szegedy et al.\n\nIn the mathematical notations the logistic regression weights are defined so that $\\beta_0=0$. This allows to have a non-redundant notation for logistic regression weights and to easily carry out computation of the loss associated to Firth bias reduction. However it came with some drawbacks:\n- when using L2 regularization, the regularization could be non optimal. What about the impact here on the performance of L2 regularization? (from table A2, we can also observe that the set of weights for L2 regularization is quite high - usually when using L2 regularization we observe the use of weights of order 10-4 ... 10-2)\n- when extending the computation of the Firth Loss to cosine classifiers, logistic regression weights are normalized... this is not compatible with the setting $\\beta_0=0$. The associated demonstration needs to be revisited. (although intuitively if we omit this constraint on notations, we should be able to redo the same reasoning)\n\nExtensive experiments clearly show the benefits of using the proposed bias reduction. To complement those results, it would be also interesting to have a final table with resuming the newly obtained SOTA results. \n\nAlso it would be interesting to have an idea on the effective value of $\\lambda$ to be used. Table A.2 shows a large set of values there. Could the author share what are the typical values observed on some configuration tested? (e.g. for figure 3?). Also what is the sensitivity of $\\lambda$ on the performance?\n\n# Minor remarks\n- equation 1: $j$ is used in the summation in the denominator... it should be changed to e.g. $j'$\n- just after equation (10), it should be mentioned in the text what is $\\lambda$ used for.\n- annex B: in the definition of the C term, $\\lambda$ should appear\n", "summary_of_the_paper": "The paper proposes to reduce the bias inherent to the estimation of the classifier used in Few Shot Classification settings. To this purpose Firth bias reduction is proposed to be used. Theoretical computation of this bias reduction is carried out within the context of Few Shot Classification resulting to a practical implementation. It turns out that it could be associated to a regularization term encouraging uniform class assignment probabilities. This results is also generalized to the context of using a cosine classifier on the features.\n\nExtensive experiments are carried out to show the benefits of this bias reduction on SOTA approaches and with various backbone architectures.", "summary_of_the_review": "Overall I am in favour of accepting this paper. The strength of this paper is on the theoretical justification of the proposed regularization term introduced associated to the general Firth bias reduction. Also extensive results are carried out to evaluate fairly the benefits of the proposed approach on SOTA solution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_Sh1J8-gxGu", "reviewer": "Reviewer_37fi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Q1- According to the intro, which I quote here ``We achieve this by deriving a simplified yet effective Firth formulation that penalizes the KL-divergence between the predicted distribution and the uniform distribution of classes'', the proposed solution assumes that the distribution of query samples is uniform. One wonders how restrictive this assumption might be, can authors comment on this? I understand that in FSL, current protocols have that property but when it comes to learning, many algorithms do not make such an assumption.\nI have to add that I understand that authors performed experiments in section 4.3 about imbalanced datasets and discussed replacing the uniform prior with non-uniform one, but from a theoretical point of view, is it possible to provide more insights?\n\nQ2- Based on Fig.1, it seems that the proposed method works better if number of shots is large, and specifically for very deep models. Would be very useful if authors can provide some insights about this.\n\nQ3- I can see some sort of inconsistency in the experiments. It would be good to see why authors didn't use the same datasets in Fig.3 for experiments in Fig.1. Also, the choice of 16class in Fig.1 is not well-justified. How does the method work if less classes were considered?\n\nQ4- Out of curiosity, did authors evaluate their method in conjunction with other FSL techniques, say other embedding techniques?\n\nA suggestion,  it is very hard for me to fully read the improvements in Fig.3, why not simply providing a table so the reader can better understand the improvements?\n\n", "summary_of_the_paper": "This work introduces a method to reduce the bias of MLE for FSL problems. Authors developed the bias reduction formulation based on the Firth bias reduction method and extended it for cosine classifier as well. A through set of experiments demonstrate that the proposed method provides statistically significant improvement over the baselines. ", "summary_of_the_review": "I believe the proposed method is novel and interesting. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D8iv1-6Ez5S", "reviewer": "Reviewer_gyef", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the submission. It's clear, well-written, and it explores a simple idea with rigor. The proposed idea is theoretically grounded and the experimental results are convincing.\n\nStrengths:\n- Clarity, writing quality.\n- Novelty: as far as I know, the issue of MLE's bias in the finite-data regime has not been discussed in the few-shot classification literature.\n- Rigor: the proposed approach is compared against sensible baselines (L2 regularization, label smoothing), controls for randomized factors, computes and compares confidence intervals, and considers multiple benchmarks and backbone architectures.\n\nWeaknesses:\n- The submission goes a bit quickly over the small-sample bias of MLE and Firth bias reduction for MLE. Without going into complete proofs, sketching the main arguments in both cases would make the submission accessible to a wider audience. As it stands, I feel like the submission is fully clear only to readers with prior knowledge of these notions.\n\nAdditional questions/comments:\n- Can the authors elaborate on why the number of novel classes subsampled for Figure 1 is 16, specifically?\n- In Figure 1, the improvements obtained from using Firth bias reduction are magnified by the backbone capacity. Is there an explanation for this?", "summary_of_the_paper": "The submission examines the impact of Maximum-Likelihood Estimation (MLE)'s small-sample bias in the few-shot classification setting. The authors point out that while asymptotically consistent, MLE is biased in the finite-data regime, and propose to apply Firth bias reduction to few-shot classifiers to counter this bias. In this setting, the technique simplifies to a KL divergence regularizer between the uniform distribution over class labels and the learner's predictions.\n\nExperiments on mini-ImageNet, CIFAR-FS, and tiered-ImageNet using ResNet and WideResNet network architectures show modest but consistent improvements across all settings. Firth bias reduction is shown to outperform other regularization approaches such as L2 regularization and label smoothing alternatives (confidence penalty and unigram label smoothing).\n", "summary_of_the_review": "The submission is clear, well-written, and it explores a simple idea with rigor. The proposed idea is theoretically grounded and the experimental results are convincing. The one thing in the way of a higher score is that the submission could be made more accessible to readers with less experience with the theory of small-sample bias in MLE and Firth's bias reduction for MLE.\n\n---\n\n**Post-rebuttal update**: The authors' response addresses my concerns, and I now feel comfortable making a clear acceptance recommendation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0gafaMWEKT7", "reviewer": "Reviewer_s811", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has multiple advantages.\n-This is the first paper addressing the effectiveness of Firth bias reduction in few-shot learning.\n-Vast empirical study validates the proposed method consistently improves the performance of the baseline method, and no performance penalty is introduced.\n-Apart from typical logistic regression models, the firth bias reduction can also be applied to cosine classifiers.\n-Comprehensive discussion about other regularization techniques is provided.\n\nI also have several questions and concerns about this paper.\n-The relative improvements are validated through empirical investigations. I expect to see the theoretical guarantee for the improvement of firth bias reduction in few-shot learning.\n-The idea of this paper partially comes from [1], which weakens the novelty of this manuscript. I would suggest the authors discuss more the extensions of [1] in current machine learning tasks, and discuss the difference with them.\n-All the experimental metrics are reported by relative improvement, making it hard to be compared for other upcoming works. I suggest the authors report the absolute accuracy on benchmark datasets in the supplementary material.\n\n[1] D. Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27–38, 1993\n", "summary_of_the_paper": "This paper tackles the few-shot learning problem, which is a hot topic in the representation learning field. Specifically, the authors aim to train accurate classifiers using a small number of samples – Maximum Likelihood Estimators are biased for them. As a result, the authors propose to utilize Firth’s Penalized Maximum Likelihood Estimator to modify the ordinary MLEs, and prevent the bias when training classifiers for few-shot instances. The authors have validated the effectiveness of Firth bias reduction from multiple aspects, and the experimental results show the performance improvement is consistent across various circumstances, e.g., with different backbones, data distributions, and classifiers.", "summary_of_the_review": "To conclude, this paper is clearly elaborated with sufficient experimental results and promising improvement. I suggest a ‘weak accept.’", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work starts from the observation that maximum likelihood estimation, while consistent, has a bias on a finite sample which is likely to hurt for small sample sizes. From this, they apply Firth bias reduction to the few-shot learning setting and demonstrate its empirical benefits, notably relatively to L2 regularization or label smoothing alternatives.

After some discussion with the authors, all reviewers are supportive of this work being accepted. Two are also suggesting this work be featured as a spotlight.

The proposed method is simple, well motivated, and appears to be effective. Therefore, I'm happy to recommend this work be accepted and receive a spotlight presentation.
Decision: Accept (Spotlight)