Year: 2021

Abstract:
Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision. The performance in few-shot classification suffers from the bias in the estimation of classifier parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classifiers has been overlooked. In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classification. Theoretically, Firth bias reduction removes the first order term O(N−1) from the small-sample bias of the Maximum Likelihood Estimator. Here we show that the general Firth bias reduction technique simplifies to encouraging uniform class assignment probabilities for multinomial logistic classification, and almost has the same effect in cosine classifiers. We derive the optimization objective for Firth penalized multinomial logistic and cosine classifiers, and empirically evaluate that it is consistently effective across the board for few-shot image classification, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes. Finally, we show the robustness of Firth bias reduction, in the case of imbalanced data distribution. Our implementation is available at https://anonymous.4open.science/r/firth_bias_reduction-2B60.

heading: 1 INTRODUCTION
text: Few-shot image classification is the practice of learning accurate classifiers using a small number of labeled samples (Fei-Fei et al., 2006; Wang et al., 2020; Snell et al., 2017; Vinyals et al., 2016; Finn et al., 2017). It has a wide range of applications from face and gesture recognition (Pfister et al., 2014) to visual navigation in robotics (Finn et al., 2017). Essentially, modern few-shot classification methods can be viewed as a combination of learning (1) a strong feature representation through a backbone network (e.g., Verma et al. (2019); Tian et al. (2020); Gidaris et al. (2019)), and (2) an accurate small-sample classifier (e.g., Chen et al. (2019); Wang and Hebert (2016)). Therefore, two key questions arise in few-shot image classification: (1) How can we obtain a strong feature representation? and (2) How can we train accurate classifiers using a small number of samples? There have been many existing methods addressing the former question using a host of different techniques (Snell et al., 2017; Rusu et al., 2019; Verma et al., 2019; Mangla et al., 2020). Here we focus on the less-explored second question. For our purposes, we will use standard feature representations and methods for training the backbone model for few-shot classification. That still leaves us with a severe classifier problem than most people realize.
There is a substantial difficulty with training a classifier using a small number of samples. In particular, with very few samples, standard classification machinery is biased. In other words, although the Maximum Likelihood Estimators (MLEs) are statistically consistent and asymptotically normal (Fahrmeir and Kaufmann, 1985), it is well-established that MLEs are biased for a small number ofN samples, with bias ofO(N−1) (Firth, 1993; Whitehead, 1986; Box, 1971; Cox and Snell, 1968). Since common logistic regression models are a type of MLEs, they are also biased (Schaefer, 1983; Steyerberg et al., 1999; Cordeiro and McCullagh, 1991; Firth, 1993). Such biases increase the error rate of the few-shot trained classifiers, and so are important in few-shot learning.
In fact, there is a standard solution for bias prevention by modifying the ordinary MLEs – known as Firth’s Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993). In the case of the
exponential family of distributions, Firth has a simplified form that penalizes the likelihood by Jeffrey’s invariant prior (Firth, 1993; Poirier, 1994), which is proportional to the determinant of the Fisher Information Matrix F . For logistic and cosine classifiers (Chen et al., 2019) which are widely used in few-shot classification, since they belong to the exponential family, Firth bias reduction can be further cast as adding a log-determinant penalty (log(det(F ))) to the cross-entropy loss. While such standard strategies can control the bias in classifiers trained with very few samples, they have not been utilized in few-shot image classification tasks.
In this paper, we show that using Firth bias reduction produces reliable improvements in a wide range of circumstances. We achieve this by deriving a simplified yet effective Firth formulation that penalizes the KL-divergence between the predicted distribution and the uniform distribution of classes, for both multinomial logistic regression models and cosine classifiers. Note that common regularization techniques (such as L2-regularization and label smoothing (Szegedy et al., 2016)) cannot reduce the estimation bias of classifier weights in small-sample regimes (Liu et al., 2020) as Firth’s penalty does; these regularization techniques are mainly used to control model complexity of deep neural networks for training feature extractor backbones in large-sample regimes.
More concretely, our results indicate that the improvements produced by Firth bias reduction for few-shot image classification tasks are consistent across the board (1) on a wide range of feature representations, (2) with both balanced and imbalanced data distributions, (3) for both single-layer and multi-layer classifiers, (4) for both logistic regression and cosine classifiers, and (5) over multiple datasets and problems. Importantly, we found that Firth bias reduction consistently yields statistically significant and positive improvements, and we did not observe any performance penalty for utilizing Firth bias reduction. Such improvements are on the order of 0.5-2.5% and up to 3% in challenging few-shot classification tasks with a large number of classes.
Our main contributions include (1) deriving a generalized expression for Firth bias reduction in few-shot multinomial logistic regression models, and providing geometrical insight into its effect on the classification probabilities space; (2) evaluating the efficacy of the Firth penalized multinomial logistic model in few-shot scenarios, both with balanced and imbalanced data distributions; (3) showing that Firth bias reduction can be extended beyond typical logistic regression models, and can be successfully adopted in cosine classifiers, and (4) providing an empirical comparison of Firth bias reduction to common regularization techniques, such as L2-regularization and label smoothing.

heading: 2 BACKGROUND
text: Mathematical Notations: In this work, we assume a multinomial logistic regression model for the classifier, with a total of J + 1 classes {0, 1, 2, · · · , J}. The logistic regression weights for class j is denoted as βj (1 ≤ j ≤ J). The class j = 0 is the base class with zero logistic regression weights. We assume to have a total number of N samples D = {(x1, y1), · · · , (xN , yN )}. The ith target yi is the one-hot encoding of the ith label. The assignment probability of the ith sample to class j is denoted as Pi,j :
Pi,j := Pr(y = j|xi) = eβ
ᵀ j x 1 + ∑J j′=1 e βᵀ j′x . (1)
The likelihood of the sample set D given the weights β is denoted as Pr(y|x;β):
Pr(y|x;β) = N∏ i=1 J∑ j=1 1[yi = j] · Pi,j , (2)
where 1[·] denotes the binary indicator function. We may abuse the notation and use Pr(y|x) instead of Pr(y|x;β) in the derivation processes. Therefore, the logistic log-likelihood function Llogistic is defined as
Llogistic := N∑ i=1 J∑ j=0 1[yi = j] · log(Pi,j). (3)
The Fisher Information Matrix (FIM) is defined as the Hessian of the negative log-likelihood function Llogistic:
F := −Hessβ(Llogistic) = Ey[∇βLlogistic · ∇βLᵀlogistic]. (4)
The Maximum Likelihood Estimator (MLE) for logistic regression is defined as
β̂MLE := arg max β
Pr(y|x;β). (5)
The dimension of the feature space is denoted as d in the derivations. U[a,b] denotes the (discrete) uniform distribution in the [a, b] interval. The cross-entropy and the KL-divergence of distributions p, q are defined as
CE(p‖q) := − ∑ x p(x) · log(q(x)), DKL(p‖q) := ∑ x p(x) · log ( p(x) q(x) ) . (6)
Table A2 of the Appendix summarizes these notations.
Small-Sample Bias of MLE: Assume β∗ is the true generative parameter. When the sample size N is small, it is shown that the MLE bias b(β̂MLE) := E[β̂MLE − β∗] is non-zero and of O(N−1) (Cox and Snell, 1968). Therefore, while MLE is unbiased as N →∞, it is inaccurate for few-shot learning. Firth Bias Reduction for MLE: Firth’s Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993) is a modification to the ordinary MLE, which removes theO(N−1) term from the small-sample bias. In particular, Firth has a simplified form for the exponential family. When Pr(y|β, x) belongs to the exponential family of distributions, the effect is to penalize the likelihood by Jeffrey’s invariant prior (Poirier, 1994), which is proportional to det(F ). Logistic and cosine classifiers (Chen et al., 2019) – the widely-used classification models in few-shot learning – belong to the exponential family; so for them, Firth bias reduction simplifies to adding a penalty (log(det(F ))) to the crossentropy loss. In what follows, we will derive a further simplified yet effective Firth formulation for logistic and cosine classifiers, which is computationally more efficient, deals with the case when det(F ) = 0, and generalizes to multinomial distributions.
MLE vs Firth’s PMLE for a Simple Case: To demonstrate the extent of the MLE bias and how Firth’s PMLE removes the leading O(N−1) bias term, we simulated data from the geometric distribution with probability of success β = 0.5 as its only parameter. The geometric experiment was chosen since a closed-from solution for the MLE and Firth’s PMLE can be derived; Given the samples (y1, · · · , yN ) from the geometric distribution, the sample mean is ȳ = 1N ∑ yi and we have β̂MLE = 1 ȳ and β̂Firth = N−1 Nȳ−1 . Note that since the sample mean is noisy, the MLE suffers from a noisy denominator making it biased. Figure 1 shows that β̂Firth is close to the true parameter β∗ for all sample sizes, whereas β̂MLE has a significant bias away from β∗ for small N . To further validate that the MLE bias is indeed of O(N−1), we plotted the MLE bias against the sample size in the log-log scale in Figure1 which shows that it is closely following a line with a negative unit slope.

heading: 3 FIRTH BIAS REDUCTION IN LOGISTIC AND COSINE CLASSIFIERS
text: In logistic models, the penalized likelihood function proposed by Firth is equivalent to imposing Jeffreys’ prior (Poirier, 1994) on the parameters and making an MAP estimate. In particular, Firth bias reduction encourages models with “large” F by multiplying the likelihood by det(F ). This penalty degenerates when det(F ) = 0. We work in the highest dimensional subspace where F has full rank, and use det(A|r) to denote the product of all r non-zero eigenvalues of A, obtaining
Pr(β|x, y) = 1 Z · Pr(y|x;β) · det(F |r)1/2, (7)
where Z is a normalization constant and det(F |r)1/2 is the Jeffery’s prior. Taking the log of both sides yields the log-posterior as a sum of the logistic log-likelihood function and the Firth bias reduction term: L := log(Pr(β|x, y)) = Llogistic + LFirth, (8) where we have
LFirth := λ
2 log(det(F |r)) + c.t.e.. (9)
The definition of LFirth was left ambiguous up to a constant w.r.t. β to facilitate the Firth bias reduction term’s interpretation and avoid the definition of similar terms. Furthermore, λ controls for the impact of the Firth term on the outcome relative to Llogistic. We then apply a series of derivation steps to simplify Equation (9), which are left to Section A of the Appendix. Finally, the Firth bias reduction term can be expressed as
LFirth = λ · 1
N N∑ i=1 [ J∑ j=0 ( xᵀi βj − log J∑ j′=0 ex ᵀ i βj′ )] = λ · 1 N N∑ i=1 [ J∑ j=0 log(Pi,j) ] . (10)
For the cosine classifier, the proof that log(det(F )) simplifies to Equation (10) involves straightforward manipulation of the proof for the logistic classifier, and is left to Section B of the appendix. The normalization of the βj weights in the cosine classifier turns into a pure scale term in the optimization, and for a cosine classifier, scaling of the βj weights does not affect predictions. Therefore, this term should be ignored, and the bias reduction term effectively becomes the same as Equation (10).
Interpreting the Firth Bias Reduction for Logistic Models: It is well known that Jeffery’s prior shrinks the parameter estimates towards zero which is equivalent to encouraging uniform class assignment probabilities (Firth, 1993; Bull et al., 2002). We take an alternative approach to reach the same conclusion in the following. By re-arranging Equation (10), one can see the ∑J j=0 log(Pi,j) term as a scaled average over a uniform distribution of classes:
LFirth ∝ 1
N N∑ i=1 [ J∑ j=0 1 J + 1 · log(Pi,j) ] = −1 N N∑ i=1 [ CE ( U[0,J]‖Pi )] . (11)
Therefore, we can abuse the notation, and re-define the coefficient λ and the constant to have
LFirth = λ · −1 N N∑ i=1 DKL(U[0,J]‖Pi) + c.t.e.. (12)
This means that by dropping the constants, the optimization objective L can be rewritten as
L = −1 N N∑ i=1 [ DKL(yi‖Pi) + λ ·DKL(U[0,J]‖Pi) ] . (13)
Since the KL-divergence and the FIM define an information geometry and a Riemanian metric on probabilistic measure spaces (Nielsen, 2020), a geometrical insight into Equation (13) is provided in Figure A6 as well.
Firth Bias Reduction vs. Common Regularization: While this insight brings Firth bias reduction closer to the common regularization techniques (e.g., L2-regularization), it is worth noting that (1) common regularization techniques mainly focus on controling model complexity, instead of reducing
small-sample estimation bias; (2) Firth bias reduction operates on a much lower-dimensional target distribution space unlike L2 which operates in the high-dimensional parameter space; (3) Firth uses the same kind of metric as the logistic loss; and (4) Firth bias reduction is dimensionally consistent like the Natural Gradients (Amari, 1998; Pascanu and Bengio, 2013), whereas L2 is not.
Firth Bias Reduction vs. Label Smoothing: Notice that the original form of label smoothing used for training in large-sample regimes (Szegedy et al., 2016) has the same formulation as the simplified Firth penalty term in Equation (12) for multinomial logistic classifiers. However, Firth bias reduction is inherently different in that it reduces the classifier estimation bias in the small-sample regimes, whereas label smoothing penalizes over-confident predictions when training deep neural networks with large amounts of samples. Generally, the Firth bias reduction term (i.e., log(det(F ))) is not the same as the label smoothing penalty (i.e., DKL(U‖Pi)) for deep neural networks. Additional analysis and empirical comparisons are provided in Section 4.4 and Section F of the Appendix.

heading: 4 EXPERIMENTAL RESULTS
text: Here we show that for a wide range of experiments, Firth bias reduction is a reliable source of small yet useful improvements in the performance. Since we report improvements for a wide range of methods and settings, the absolute accuracy improvements were reported. These absolute improvements sometimes constitute significantly to the baseline accuracy in terms of relative importance.
Datasets: We perform experiments on three widely-used and publicly available benchmarks: miniImageNet (Vinyals et al., 2016), CIFAR-FS (Bertinetto et al., 2019), and tiered-ImageNet (Ren et al., 2018). Each dataset consists of non-overlapping base, validation, and novel classes. The detailed class splits are described in Section D of the Appendix. Following the standard practice (Chen et al., 2019), we train feature backbones on base classes, cross-validate bias reduction coefficients on validation classes, and train classifiers and measure test accuracy over multiple trials on novel classes.
Setups and Implementation Details: Details are covered in Appendix Section D.
Baselines and Evaluation Metric: Non-penalized classifiers are used as the baseline in all the experiments to compare against Firth bias reduction and L2-regularization. Absolute accuracy improvements over the baseline averaged across multiple trials are used as the evaluation metric
in all the experiments. Relative accuracy improvements are also shown in the Appendix, which demonstrate similar behaviors.
Statistical Significance and Reducing the Effect of Randomized Factors: Details are discussed in Appendix Section D.
Summary of Results: Section 4.1 shows the efficacy of Firth bias reduction on standard feature backbones (ResNets with varying depth) and single-layer logistic classifiers. Section 4.2 shows and argues that Firth bias reduction outperforms L2-regularization on few-shot classification tasks. Next, we investigate the driving factor in Firth’s improvement in Section 4.3, and show evidence for the efficacy of Firth’s bias suppression property. We also show that Firth bias reduction can be effectively, and without any modifications, applied to imbalanced data distribution settings. Section 4.4 compares Firth bias reduction against label smoothing variants, and shows that Firth outperforms label smoothing. In Section 4.5 we show that Firth bias reduction can also be applied to modern few-shot methods with advanced feature backbones (i.e., WideResNet trained with strong regularization (Mangla et al., 2020)) and cosine classifiers. Experiments with additional feature backbones (DenseNet and MobileNet (Wang et al., 2019)) are included in Appendix Section G. Finally, Section 4.6 demonstrates that Firth bias reduction produces reliable improvements over the state-of-the-art feature calibration method (Shuo et al., 2020). Our collective results clearly indicate a consistent pattern of improvements over a large array of (1) feature representations, (2) datasets, (3) classification ways, (4) number of shots, (5) types of classifiers, and (6) with both balanced and imbalanced data distributions.

heading: 4.1 FIRTH BIAS REDUCTION IMPROVES FEW-SHOT CLASSIFICATION ACCURACY OF LOGISTIC CLASSIFIERS
text: Figure 2 summarizes the results. For all the k-shot tasks and backbones, the average absolute test accuracy improvement is significant and it increases as k increases. Also, Firth bias reduction does not hurt the 1 and 5-shot performance, with slight improvements for some backbones. Extra experiments are included in Appendix Section E.1.

heading: 4.2 L2-REGULARIZATION IS NOT AS EFFECTIVE AS FIRTH BIAS REDUCTION IN FEW-SHOT CLASSIFICATION
text: To explore whether the performance improvement pattern for Firth bias reduction can be reproduced by common regularization techniques such as L2, the same experiments were repeated with L2regularized logistic classifiers. For almost all the k-shot tasks and feature backbones, the average absolute test accuracy improvement is close to zero, as shown in Figure 2. This suggests that the bias reduction property of Firth plays a significant part in improving the classifier’s performance in the few-shot regime which cannot be achieved by L2- regularization.

heading: 4.3 BIAS OR PRIOR?
text: Firth bias reduction clearly helps. It may be doing so because it is an effective way of suppressing bias. Alternatively, the prior information might be helping. This section offers evidence that bias suppression is what is important. We consider data where the class frequencies are unbalanced. We apply a variant of Firth bias reduction that uses class prior frequencies to this data. This variant is not as successful as routine Firth bias reduction, suggesting that bias correction, not prior information, is what is important. One might consider replacing the uniform distribution in the Firth prior with a class probability distribution. That is, replacing LFirth = −λ · CE(U[0,J]‖Pi) by LFirth = −λ · CE(A‖Pi), when training a Firth penalized few-shot logistic classifier. The main question is (1) whether such a replacement is necessary for Firth bias reduction to work in this example, and (2) whether this prior is responsible for any gained improvements.
To test this we designed two schemes to generate imbalanced datasets. These schemes differ in the imbalanced count vectors used to create the few-shot dataset in validation and novel splits. The increments in the counts per class were designed to result in two different average counts of 7.5 (scheme 1), and 15 (scheme 2) over the classes (experimental details are left to Section D of the Appendix). For each scheme, the experiments were carried out similar to the “balanced few-shot” case on mini-ImageNet, except that in each trial three classifiers, (1) baseline (not penalized), (2) Firth
penalized (KL with the uniform prior), and (3) modified Firth penalized (KL with the imbalanced prior) were trained. In both schemes, the improvement achieved by the Firth penalized classifier is higher than that of the modified Firth penalized classifier over the baseline for all backbones, as shown in Figure 3. The difference between the two improvements is more substantial in scheme 1, where the average of samples per class is less than scheme 2. This implies that the KL divergence to the uniform class probabilities in a Firth penalized classifier is indeed reducing the bias and should not be naively considered as if a prior is simply defined over the class assignment probabilities.

heading: 4.4 FIRTH BIAS REDUCTION IMPROVES OVER LABEL SMOOTHING
text: Even though the original version of label smoothing (Szegedy et al., 2016) has the same formulation as the Firth bias reduction term in Equation (13), there is a family of label smoothing techniques that aim to reduce overfitting and over-confident predictions (Pereyra et al., 2017). Namely, the confidence penalty (i.e., regularizing DKL(U‖Pi)) and unigram label smoothing (i.e., regluarizing the KL-divergence with class priors) are two label smoothing variants, which are shown to outperform the original label-smoothing when training complex networks with large amounts of samples Pereyra et al. (2017). Figure 3 shows that Firth bias reduction outperforms unigram label smoothing, and Table A4 of the Appendix shows that Firth bias reduction yields better results than applying confidence penalties and entropy regularization. This further suggests that the improvements obtained by Firth are the result of its bias reduction property.

heading: 4.5 FIRTH BIAS REDUCTION IMPROVES THE PERFORMANCE OF COSINE CLASSIFIERS
text: As cosine classifiers have recently proven to be useful for few-shot classification (Chen et al., 2019; Gidaris and Komodakis, 2018), we investigated the impact of Firth bias reduction on them. The difference between a logistic and cosine classifier parameter space is that the parameter space of the cosine classifier is constrained to a unit sphere, therefore it is likely for the Firth bias reduction to improve cosine classifier performance as well. To test this, we based our experiments on the implementation of the S2M2R method (Mangla et al., 2020), and used their pre-trained WideResNet-
28-10 (Zagoruyko and Komodakis, 2016) on the base classes as a strong feature extractor. We trained Firth penalized and non-penalized cosine classifiers on the mini-ImageNet, tiered-ImageNet, and CIFAR-FS datasets. Few-shot classifiers were trained for varying number of classes, depending on the dataset, and varying number of samples per class. We followed the same scheme as (Mangla et al., 2020) to generate the k-shot support and query datasets for training and testing, respectively, in validation and novel splits.
As shown in Figure 4, in all datasets, Firth bias reduction improves the performance of 5-, and 10-shot tasks in a monotonically increasing fashion w.r.t. the number of classes. This behavior is well represented for tiered-ImageNet as increasing 5-way to 150-way classification results in more than 2.5% improvement of the average test accuracy. In the 1-shot task, the improvement by Firth bias reduction is monotonically decreasing with the number of classes in all the datasets except tieredImageNet, but it does not hurt the performance. The improvement almost falls within (0.1%-0.5%), (0-0.1%), and (0-0.2%), for mini-ImageNet, CIFAR-FS, and tiered-ImageNet, respectively.
This experiment suggests that not only is Firth bias reduction effective for cosine classifiers, but it also could help with the performance even when strong features are used. This is well justified as Firth bias reduction targets the bias introduced in the classifier parameters not the features.

heading: 4.6 COMPARISON WITH ADDITIONAL STATE-OF-THE-ART METHOD
text: We further applied the Firth bias reduction term to the recent state-of-the-art method in few-shot classification (Yang et al., 2021), called Distribution Calibration (DCL). DCL computes the closest base classes to each sample in the feature space, and samples artificial examples from a Gaussian distribution centered around the mean of the closest base classes features. To make the setup challenging, we consider data where the base classes are categorically different from the novel classes; i.e., by performing cross-dataset few shot classification where the features are trained on the
base classes of mini-ImageNet and tiered-ImageNet, and the novel sets are instead sampled from the CUB dataset. The results are shown in Table 1 and indicate that the Firth bias reduction improves over Yang et al. (2021). We also show the Firth bias reduction improvements on the tiered-ImageNet dataset in Appendix Section H and Table A5. This further supports the observation that Firth bias reduction yields small but reliable improvements under different methods.

heading: 5 RELATED WORK
text: Bias Reduction of the MLE: A myriad of statistical work has been proposed to mitigate the smallsample bias of the MLE under different settings (Kosmidis and Firth, 2009; Bull et al., 2002; Anderson and Richardson, 1979; Kenward and Roger, 1997). Originally, the asymptotic bias of MLE was shown to be of O(N−1), with N being the sample size (Firth, 1993). To counter such an estimation bias, many approaches have existed. To name a few, (1) additive penalization terms to the main logistic loss were proposed to reduce the MLE’s bias (Firth, 1993; Bull et al., 2002; Greenland and Mansournia, 2015), and (2) some methods have been proposed to directly approximate and remove such a small sample bias (Cox and Hinkley, 1979). While the latter approach may sound appealing, estimating the MLE’s bias can be impractical. For instance, in few-shot scenarios, a perfect separation of the classes may be achievable, causing the logistic MLE to be unbounded (Heinze and Schemper, 2002). On the other hand, the penalization methods do not modify the estimated parameters directly, and instead gently push for a preference towards less biased estimates. Such penalization methods can be generally applicable to a vast array of models.
Theoretically, Firth’s PMLE reduces the bias by removing the leading O(N−1) term from the MLE’s bias (Firth, 1993); a property that does not exist in common regularization techniques such as L2regularization. Furthermore, PMLE of the logistic model has been shown to have smaller variance than MLE as well (Kosmidis and Firth, 2009; Copas, 1988). Firth’s PMLE has been well studied for binomial logistic regression (Firth, 1993), applied and tested against other penalization techniques in other fields (Rahman and Sultana, 2017; Rainey and McCaskey, 2015; Muchlinski et al., 2016).
Additional related work on Few-Shot Image Classification was left to Appendix Section C.

heading: 6 CONCLUSION
text: We showed that Firth bias reduction consistently improves the accuracy across the board in fewshot classification regardless of (1) the employed feature backbone, (2) the number of classes and samples, and (3) the dataset and problem setting. Furthermore, our experiments show that Firth bias reduction can improve the performance of the cosine classifiers, and is applicable to imbalanced few-shot settings without any necessary modifications. Overall, our evaluations suggest that Firth bias reduction is a useful and geneal bias reduction tool that has been missing in few-shot classification, and should be incorporated in few-shot classification tasks for accuracy improvements.

heading: A FIRTH BIAS REDUCTION FOR FEW-SHOT MULTINOMIAL LOGISTIC REGRESSION
text: Table A2 summarizes the notations used throughout the main paper and here. Given a dataset of N samples D = {(x1, y1), (x2, y2), · · · , (xN , yN )}, the multinomial logistic model for a total of J + 1 classes can be formulated as
log [ Pr(y = j|xi) Pr(y = 0|xi) ] = βᵀj xi, j ∈ {1, 2, · · · , J}, (A14)
where class j = 0 was chosen as the base class in the log odds ratio. In other words, w.l.o.g. we assume β0 = 0 in this formulation. Given the decision rule in Equation (A14), we can write
Pi,0 = 1 1 + ∑J j′=1 e βᵀ j′xi , Pi,j = eβ ᵀ j xi 1 + ∑J j′=1 e βᵀ j′xi
∀1 ≤ j ≤ J. (A15)
Under this notation, the log-likelihood would be Llogistic = ∑N i=1 ∑J j=0 1[yi = j] · log(Pi,j).The data matrix XD is given as
XD := [x1 x2 · · · xN ]d×N (A16) Also, X := XD ⊗ IJ where the ⊗ operator denotes the Kronecker matrix product, and the Jdimensional identity matrix is denoted as IJ .
Firth (1993) has established that the bias of logistic regression can be removed by maximizing the sum of (1) the log-likelihood Llogistic and (2) the log-determinant of the Fisher Information Matrix (FIM). For our purposes, this presents some challenges: we have a few number of samples and the FIM determinant is zero. Instead, we use the product of all non-zero eigenvalues of the FIM as its “amended determinant”. To obtain this efficiently, we need to know the specific structure of the FIM.
It is important in what follows that the FIM can be defined as the matrix product
F(dJ)×(dJ) = X ᵀ (dJ)×(NJ) ·M(NJ)×(NJ) ·X(NJ)×(dJ), (A17)
where M is a block-diagonal matrix whose ith diagonal block is denoted as Mi. We leave the definition of X and Mi to the “FIM Formulation for Logistic Regression” subsection. Next, we focus on:
• Determinant Amendment and Constant Dropping: Generically, we show that
log(det(F |NJ)) = N∑ i=1 log(det(Mi)) + c.t.e., (A18)
where det(F |NJ) is an amended version of det(F ). • Efficient Computation of log(det(Mi)): Next, we show that
log(det(Mi)) = J∑ j=0 log(Pi,j). (A19)
Combining these two points will lead us to the simplified Firth bias reduction objective:
LFirth = λ · 1
N N∑ i=1 J∑ j=0 log(Pi,j). (A20)
Determinant Amendment and Constant Dropping: Having F = Xᵀ ·M ·X prompts us to utilize the SVD of X as XNJ×dJ = UNJ×NJ · SNJ×dJ · V ᵀdJ×dJ . (A21) Therefore, the FIM can be written as F = V · ( Sᵀ ·K · S ) · V ᵀ, where K := Uᵀ ·M · U . Since V and U are rotation matrices, we can write det(F ) = det(Sᵀ ·K · S), and
det(K) = det(M) = N∏ i=1 det(Mi). (A22)
Notation Description
J + 1 Total Number of Classes in Multinomial Logistic Regression
βj The Logistic Regression Weights for Class j (j ∈ {1, 2, · · · , J})
Pi,j Classification Probability of Sample i Belonging to Class j
Pi The ith Sample’s Soft Classification Probabilities
N Number of Samples
D = {(xi, yi)}Ni=1 Logistic Regression Samples Dataset
yi The One-Hot Encoding of the Label yi
1[a = b] The Binary Indicator Function (i.e., 1 when a = b and 0 otherwise)
F The Fisher Information Matrix
d Dimension of the Features
IJ The J × J Identity Matrix
A⊗B The Kronecker Product of Matrix A by Matrix B
1r×c The All Ones Matrix with r Rows and c Columns by Matrix B
β̂MLE The Maximum Likelihood Estimator (MLE)
Llogistic The Logistic Log-Likelihood Function
LFirth The Firth Bias Reduction Function
λ The Firth Bias Reduction Coefficient
CE(p‖q) The Cross-Entropy of p and q (See Equation (6) of the main paper)
DKL(p‖q) The KL Divergence of p and q (See Equation (6) of the main paper)
U[0,J] The Uniform Class Assignment Probabilities
det(A|r) The Amended Determinant of the Degenerate Matrix A with at most r Non-zero Eigenvalues (See Section A of the Appendix).
Table A2: The mathematical notations used throughout the paper.
As we have d > N for most few-shot tasks, the matrix S can be viewed in the following form: SNJ×dJ = [ ŜNJ×NJ 0 ] , (A23)
where Ŝ is a diagonal square matrix. Since Ŝᵀ = Ŝ, we have
Sᵀ ·K · S =  ( Ŝ ·K · Ŝ ) NJ×NJ 0
0 0  . (A24) Equation (A24) and det(F ) = det(Sᵀ ·K · S) show why det(F ) is zero. For mitigation, we replace det(F ) with the product of the non-zero eigenvalues of F , namely det(F |NJ), and call it the “amended determinant” of F . Thanks to F = V · ( Sᵀ ·K · S ) · V ᵀ, even the amended determinants det(F |NJ) and det(Sᵀ ·K · S|NJ) are the same:
det(F |NJ) = det(Sᵀ ·K · S|NJ) = det(Ŝ ·K · Ŝ) = det(M) · det(Ŝ2). (A25)
1 5 10 15 20 25 Number of Shots
0.0
0.2
0.4
0.6
0.8 1.0 (A CC )%
Firth Bias Reduction
ResNet10 ResNet18 ResNet34 ResNet50 ResNet101
1 5 10 15 20 25 Number of Shots
0.0
0.2
0.4
0.6
0.8
1.0
(A CC
)%
L2 Regularization ResNet10 ResNet18 ResNet34 ResNet50 ResNet101
Figure A5: Same experiment as Figure 2 of the main paper but with a 3-layer logistic classifierlogistic classifier instead of a single-layer logistic classifier (in the main paper). Again, Firth bias reduction improves the accuracy for different backbones and number of samples (left panel), whereas L2-regularization is not effective at all (right panel). In the very hard cases, i.e. 1- and 5-shots, there is a mild improvement by Firth and no deterioration. The error bars, representing the 95% confidence interval, are covered by the blobs.
Could det(F |NJ) be zero? The answer is “not” generically; det(M) is generically positive as we will show det(Mi) = ∏J j=0 Pi,j later in Equation (A30). Also, det(Ŝ
2) > 0 holds with probability 1 for continuous data distributions. In fact, det(Ŝ2) can only be zero when the data contains linearly dependant samples, which happens with zero probability for non-atomic data distributions. Therefore, we have
log(det(F |NJ)) = N∑ i=1 log(det(Mi)) + log(det(Ŝ 2)). (A26)
This is the same as Equation (A18): since the log(det(Ŝ2)) term is independent of the model’s parameters, we can treat it as an optimization constant and drop it.
Efficient Computation of log(det(Mi)): We define the soft predictions of the ith sample (excluding the base class) as Pi,1:J := [ Pi,1 · · · Pi,J ]ᵀ . Given Mi’s definition in Equation (A31), we can
write Mi = Diag(Pi,1:J)− Pi,1:J · Pᵀi,1:J . (A27)
Next, we use the Matrix-Determinant Lemma (David, 1998) to compute det(Mi):
det(Mi) = det(Diag(Pi,1:J)) · (1− Pᵀi,1:J · Diag(Pi,1:J) · Pi,1:J)
= ( J∏ j=1 Pi,j) · (1− Pᵀi,1:J1J×1) = ( J∏ j=1 Pi,j) · Pi,0 = J∏ j=0 Pi,j . (A28)
Taking the log will give us Equation (A19).
The FIM Formulation for Logistic Regression: Elementary methods established that the FIM of the logistic classifier is composed of J × J block matrices, each with a dimension of d× d (where d is the dimension of the features). These block matrices can be expressed as
Fj,j = N∑ i=1 Pi,j · (1− Pi,j) · xi · xᵀi ∀ 1 ≤ j ≤ J,
Fj,k = − N∑ i=1 Pi,j · Pi,k · xi · xᵀi ∀ 1 ≤ j 6= k ≤ J. (A29)
This facilitates the expression of the FIM as shown in Equation (A17): We can define X as the Kronecker product of (1) the data matrix XD and (2) the J-dimensional identity matrix IJ :
XD := 
xᵀ1
xᵀ2
...
xᵀN
 N×d , X := XD ⊗ IJ . (A30)
M would then be a block-diagonal matrix whose ith block is defined as
(Mi)j,j = Pi,j(1− Pi,j) ∀ 1 ≤ j ≤ J, (Mi)j,k = −Pi,jPi,k ∀ 1 ≤ j 6= k ≤ J. (A31)

heading: B FIRTH BIAS REDUCTION FOR COSINE CLASSIFIERS
text: Section A derived the Firth bias reduction term for the logistic classifier as DKL(Pi‖U). Here, we generalize this observation to cosine classifiers, and prove that the Firth bias reduction for cosine classifiers reduces down to the same form as the one obtained for logistic classifiers.
By defining the normalization transformation
T (β) = [ βᵀ0 ‖β0‖ , · · · , βᵀJ ‖βJ‖ ]T , T (x) = [ xᵀ1 ‖x1‖ , · · · , xᵀN ‖xN‖ ]T (A32)
we have the log-likelihood relation between the cosine classifier and the logistic model:
Lcosine(β;x, y) = Llogistic(T (β);T (x), y) (A33)
According to Equation (4) of the main paper and the chain rule, we can write
Fcosine(β, x, y) = A · Flogistic(T (β), T (x), y) ·Aᵀ, (A34)
where Ad(J+1)×d(J+1) is the Jacobian matrix of T (β) w.r.t. β. It can be shown that A is a symmetric block-diagonal matrix, whose jth diagonal block is
Aj,j = 1
‖βj‖
( Id×d − βjβ ᵀ j
βᵀj βj
) (A35)
Therefore,
log(det(Fcosine)) = log(det(Flogistic)) + 2 log(det(A))
= log(det(Flogistic)) + 2 J∑ j=0 log(det(Aj,j)), (A36)
It is obvious that
Aj,j · v = { 0 v ‖ βj v ‖βj‖ v⊥βj
(A37)
Therefore, Aj,j can be thought as a identity-proportional matrix in the sub-space perpendicular to βj . Therefore, its amended determinant is
log(det(Aj,j ; d− 1)) = −(d− 1) · log(‖βj‖) (A38)
For the plain logistic model, the Firth penalized optimization problem is:
max β
[ Llogistic(β;x, y) + λ · log(det(Flogistic(β, x, y))) ] (A39)
KL
KL
Figure A6: Firth bias reduction is effectively the same as minimizing the KL divergence to the uniform class probabilities for logistic regression. Here, Pi denotes the predicted distribution of classes for the sample xi, and U denotes the uniform distribution of classes. The logistic objective minimizes the KL-divergence between Pi and the true label yi, while Firth bias reduction LFirth tries to tie Pi with a KL divergence rope to the uniform distribution over the classes.
For the cosine classifiers, the Firth penalized optimization problem is:
max β
[ Llogistic(T (β);T (x), y) + λ · log(det(Flogistic(T (β), T (x), y)))− C J∑ j=0 log(‖βj‖) ] (A40)
where C := 2λ(d − 1). By re-parameterizing β = (uβ , sβ) with uβ := T (β), sβ :=[ ‖β0‖, · · · , ‖βJ‖ ] , we have
max uβ ,sβ
[ Llogistic(uβ ;T (x), y) + λ · log(det(Flogistic(uβ , T (x), y)))− C J∑ j=0 log(sβj) ] (A41)
Since uβ and sβ are independent optimization parameters, this problem can be restated as
max uβ
[ Llogistic(uβ ;T (x), y)+λ·log(det(Flogistic(uβ , T (x), y))) ] +max
sβ
[ −C J∑ j=0 log(sβj) ] (A42)
Since sβ has no effect on the predictions of the model, it can be ignored. Therefore, we end up with the optimization problem
max uβ
[ Llogistic(uβ ;T (x), y) + λ · log(det(Flogistic(uβ , T (x), y))) ] (A43)
Essentially, this suggests applying the same bias reduction form to cosine classifiers as the one used for logistic classifiers; log(det(Flogistic(uβ , T (x), y))) reduces down to DKL(Pi‖U) as proven in Section A, and Llogistic(uβ ;T (x), y) is the cross-entropy loss with the true labels.

heading: C ADDITIONAL RELATED WORK ON FEW-SHOT IMAGE CLASSIFICATION
text: As one of the unsolved problems in machine learning, few-shot learning (Miller et al., 2000; Fei-Fei et al., 2006) has attracted growing interest in the deep learning era (Lake et al., 2015; Santoro et al., 2016; Wang and Hebert, 2016; Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Hariharan and Girshick, 2017; George et al., 2017; Triantafillou et al., 2017; Edwards and Storkey, 2017; Mishra et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019; Dvornik et al., 2019; Allen et al., 2019; Li et al., 2019a; Yoon et al., 2019; Lifchitz et al., 2019; Li et al., 2019b; Zhang et al., 2020; Ye et al., 2020; Dhillon et al., 2020; Triantafillou et al., 2020; Tian et al., 2020; Dvornik et al., 2020; Yang et al., 2021; Phoo and Hariharan, 2021; Liu et al., 2021; Zhang et al., 2021). Successful generalization from few training samples requires “inductive biases” or shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning, and more
recently, meta-learning (Thrun, 1998; Schmidhuber, 1987; Schmidhuber et al., 1997; Bengio et al., 1992). By explicitly “learning-to-learn” over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task-agnostic knowledge to few-shot learning problems of novel classes. Within this paradigm, various types of meta-knowledge has been explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using a distance-based classifier such as cosine similarity or nearest neighbor (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol and Schulman, 2018; Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi and Larochelle, 2017; Munkhdalai and Yu, 2017; Li et al., 2017; Rusu et al., 2019); (3) a transferable strategy to estimate model parameters based on few class examples (Bertinetto et al., 2016; Qiao et al., 2018; Qi et al., 2018; Gidaris and Komodakis, 2018), or from an initial small dataset model (Wang and Hebert, 2016; Wang et al., 2017). Some most recent work (Gidaris and Komodakis, 2018; Chen et al., 2019; Tian et al., 2020) also showed that the performance of these complex models can be matched by simple representation learning on base classes and classifier fine-tuning on novel classes.

heading: D DATASETS AND ADDITIONAL EXPERIMENT DETAILS
text: Datasets: mini-ImageNet consists of 64, 16, and 20 classes from ImageNet for base, validation, and novel sets, respectively. Each class contains 600 images of size 84× 84. tiered-ImageNet consists of 351, 97, and 160 classes from ImageNet for base, validation, and novel sets, respectively. Unlike mini-ImageNet, the classes could have varying number of samples in tiered-ImageNet but the images are of the same size. CIFAR-FS is a random split of CIFAR-100 (Krizhevsky et al., 2009) with images of size 32× 32 into 64, 16, and 20 classes for base, validation, and novel sets, respectively. Setups and Implementation Details: Our experiments fall into two main categories: (1) the standard ResNet (He et al., 2016) feature experiments for logistic classifiers, and (2) more powerful, state-of-the-art WideResNet (Zagoruyko and Komodakis, 2016) features trained with strong regularization techniques (manifold mixup) and additional self-supervision (Mangla et al., 2020), or further calibrated via feature transformations (Shuo et al., 2020), for logistic and cosine classifiers. For the first category of experiments, we averaged our results over 400 random trials for both methods. An array of 5 different ResNet architectures with varying depths were used in this category: ResNet10, 18, 34, 50, and 101. The mini-ImageNet dataset was used for these experiments, and 16-way classification was performed on both the validation and novel classes. For this purpose, 16 out of 20 novel classes were chosen at random once and fixed for all the evaluations. We split the validation and novel classes into 90% training and 10% held-out for accuracy evaluation. The imbalanced data settings were chosen to either have 7.5 or 15 average number of shots per class. For the second category of experiments, we used the WideResNet-28-10 pre-trained feature backbones from Mangla et al. (2020); Shuo et al. (2020). These experiments were conducted on all three benchmarks, and each hyper-parameter configuration was averaged over 10,000 random trials. Additional details and hyper-parameters are covered in the subsequent sections.
Additional Implementation Details for Sections 4.1 and 4.2 of the Main Paper: In all the experiments, a 3-layer logistic classifier with 100 and 50 hidden units and ReLU activation was used. The L2-regularization coefficient was chosen for the mean-squared value of all classifier parameters, and the Firth bias reduction coefficient was chosen for Equation (13) of the main paper.
Statistical Significance and Reducing the Effect of Randomized Factor: Our study investigates improvements over the baseline. Random effects (random number seed; batch ordering; parameter initialization; and so on) complicate the study by creating variance in the measured improvement. We used a matching procedure (so that the baseline and the Firth penalized models share the same values of all random effects) to control this variance. As long as one does not search for random effects that yield a good improvement (we did not), this yields an unbiased estimate of the improvement. Each experiment is repeated multiple times to obtain confidence intervals. Note that (a) confidence intervals are small; and (b) improvements occur over a large range of feature backbones and datasets. It is safe to conclude that Firth bias reduction reliably offers a small but useful improvement in accuracy for few-shot classifiers.
Additional Implementation Details for Section 4.3 of the Main Paper: We used two non-uniform count vectors with different average counts, 7.5 (scheme 1) and 15 (scheme 2), to generate the datasets
Setting Hyper-parameter Value
All Standard Backbone Experiments
Learning Rate 0.005
Mini-batch Size 10
Number of Classes 16
Optimizer SGD
Train-Heldout Shares 90%-10%
Backbones ResNet 10, 18, 34, 50, 101
Balanced Data Experiments of Sections 4.1 and 4.2 of the
Main Paper
Number of Shots 1, 5, 10, 15, 20, 25
Firth Coefficients Set* 0, 0.01, 0.03, 0.1, 0.3, 1, 3, 10
L2 Coefficients Set** 0, 1, 3, 10, 30, 100, 300, 1000
Imbalanced Data Experiments of Section 4.3 of the Main
Paper
7.5-Shots Class Distribution 2, 2, 2, 2, 4, 4, 4, 4, 8, 8, 8, 8, 16, 16, 16, 16
15-Shots Class Distribution 1, 1, 5, 5, 9, 9, 13, 13, 17, 17, 21, 21, 25, 25, 29, 29
Experiments of Sections 4.1, 4.2, and 4.3 of the Main Paper with 1-Layer
Logistic Classifier
Number of Epochs 100
Classifier Architecture Features→100→ReLU→50 →ReLU→Classes→Softmax
Number of Trials More than 400
Experiments of Section E.2 with 3-Layer
Logistic Classifier
Number of Epochs 400
Classifier Architecture Features→Classes→Softmax Number of Trials More than 800
Table A3: Experimental settings used for the standard backbone experiments. The table is partitioned into 5 sections, where the first section shows the global hyper-parameters used in all standard backbone experiments. The same set of Firth bias reduction and L2 regularization coefficients were used for all validation experiments. *The Firth bias reduction coefficients were chosen for Equation (13) of the main paper. **We defined the L2-regularization as the mean squared value of all classifier parameters, which is why the normalized set of coefficients seems large. The typical unnormalized regularization coefficients can be obtained by dividing these normalized coefficients by the number of classifier parameters.
in both validation and novel sets. The count vector with the average of 7.5-shots had 4 classes for each count from the geometric sequence {2, 4, 8, 16}, and the count vector with the average of 15-shots had 2 classes for each count from the arithmetic sequence {1, 5, 9, 13, 17, 21, 25, 29}. The same 3-layer logistic classifier of Sections 4.1 and 4.2 was used in Section 4.3 of the main paper.
Table A3 summarizes the hyper-parameters used in all the standard backbone experiments. Also, Figure A11 shows the relative accuracy improvements corresponding to Figure A5. Figure A12 contains the validation accuracy versus Firth coefficient λ for the experiments of Figure 2, and Figure A5.
Additional Implementation Details for Section 4.5 of the Main Paper: In the experiments carried out in Section 4.5 of the main paper, a single-layer cosine classifier was used. Also, for the Firth bias-reduced cosine classifier, the coefficient was tuned for each (N, J) pair with N representing the number of samples per class and J being the number of classes. For J-way classification on the novel set when J happened to be larger than Jval–the number of classes in the validation set– the coefficient tuned for Jval-way classification on the validation set was adopted.
Computational Resources: For the standard backbone experiments, we trained more than 384,000 3-layer and 768,000 single-layer logistic classifiers for the balanced data settings. For the imbalanced settings, we trained more than 64,000 3-layer and 128,000 single-layer logistic classifiers. All classifiers were trained on CPUs, and these experiments alone consumed more than 16 CPU years. The 5 backbones were trained once using an Nvidia-V100 GPU and took 3-days for training.
For the cosine classifier experiments, we trained over 1.92, 1.92, and 3.36 million classifiers for mini-ImageNet, CIFAR-FS, and tiered-ImageNet datasets, respectively. These experiments used more than 5 CPU years in total.
Number of Shots Confidence Penalty Improvements Firth Improvements
5 0.13 ± 0.13 % 0.23 ± 0.06 %
10 0.52 ± 0.14 % 0.73 ± 0.07 %
15 0.57 ± 0.18 % 1 ± 0.07 %
Table A4: Comparing Firth bias reduction against the confidence penalty label smoothing technique. The confidence penalty is defined as a DKL(Pi‖U) regularization term, whereas Firth bias reduction for logistic and cosine classifiers reduces to a DKL(U‖Pi) penalty. The experimental setting is the same as Figure 2 with the ResNet-10 backbone. This suggests that the improvements obtained by Firth are the result of its bias suppression property, and Firth cannot be replaced by standard label smoothing techniques.
No Artificial Samples 750-Artificial Samples
Way Shot Before After Firth Improvement Before After Firth Improvement
10 1 59.44 60.07 ± 0.16 0.63 ± 0.04 61.85 61.90 ± 0.17 0.05 ± 0.02 10 5 80.52 80.85 ± 0.12 0.33 ± 0.03 79.66 80.07 ± 0.13 0.42 ± 0.04 15 1 52.68 53.35 ± 0.13 0.67 ± 0.03 54.57 54.62 ± 0.14 0.05 ± 0.02 15 5 75.18 75.64 ± 0.11 0.46 ± 0.03 73.88 74.40 ± 0.11 0.53 ± 0.04
Table A5: The Firth bias reduction accuracy improvements on the tiered-Imagenet dataset when 0 or 750 artificial samples were generated from the calibrated normal distribution in Shuo et al. (2020).

heading: E ADDITIONAL EXPERIMENTS ON THE STANDARD BACKBONES
text: E.1 ADDITIONAL LOGISTIC CLASSIFIER EXPERIMENTS
The experiments of Figure 2 were repeated to perform 16-way classification using a logistic classifier on tiered-ImageNet and CIFAR-FS Figure A13. Moreover, 5-way and 10-way classification was tested for mini-ImageNet Figure A9 in the same setting as Figure 2. The results show that Firth improvements always exist and it’s even more effective as the number of classes increases.
E.2 3-LAYER LOGISTIC CLASSIFIER FOR THE STANDARD BACKBONES
We conducted the same experiments as in Sections 4.1, and 4.2 of the main paper but with a 3- layer logistic classifier. Again, we see consistent accuracy improvements for the Firth bias-reduced classifier over the non-penalized (baseline) classifier in Figure A5. This further supports the idea that Firth bias reduction boosts the performance of any reasonable classifier. Needless to say, L2-regularization is not effective as shown in Figure A5.
Furthermore, the imbalanced few-shot classification in Section 4.3 of the main paper was repeated with the 3-layer logistic classifier in Figure A7. Again for both schemes, the Firth penalized classifier
RN 10
RN 18
RN 34
RN 50
RN 10
1
Backbone Architecture
0.0
0.2
0.4
0.6
0.8 1.0 (A CC )%
7.5 Average Shots Non-uniform Prior Uniform Prior
RN 10
RN 18
RN 34
RN 50
RN 10
1
Backbone Architecture
0.0
0.2
0.4
0.6
0.8
1.0
(A CC
)%
15 Average Shots Non-uniform Prior Uniform Prior
Figure A7: Same experiment as Figure 3 of the main paper but with a 3-layer logistic classifier instead of a single-layer (in the main paper). Again, in both schemes (7.5 and 15 average shots), the Firth penalized classifier results in more accuracy improvements than the classifier penalized with KL-divergence to the non-uniform prior. This further suggests that Firth bias reduction is indeed reducing the bias and is not simply imposing a uniform prior. Also, it reaffirms that plain Firth bias reduction can safely be used on imbalanced data.
1 5 10 15 20 25 Number of Shots
7.0
7.5
8.0
8.5
9.0
9.5
10.0
AC C%
ResNet10 Backbone 1-Layer Classifier 3-Layer Classifier
1 5 10 15 20 25 Number of Shots
7.0
7.5
8.0
8.5
9.0
9.5
10.0
AC C%
ResNet18 Backbone 1-Layer Classifier 3-Layer Classifier
1 5 10 15 20 25 Number of Shots
7.0
7.5
8.0
8.5
9.0
9.5 10.0 AC C%
ResNet34 Backbone 1-Layer Classifier 3-Layer Classifier
1 5 10 15 20 25 Number of Shots
7.0
7.5
8.0
8.5
9.0
9.5
10.0
AC C%
ResNet50 Backbone 1-Layer Classifier 3-Layer Classifier
1 5 10 15 20 25 Number of Shots
7.0
7.5
8.0
8.5
9.0
9.5
10.0
AC C%
ResNet101 Backbone 1-Layer Classifier 3-Layer Classifier
Figure A8: Absolute test accuracy values of 16-way classification for the non-penalized 3-layer and single-layer logistic classifiers on the novel set of mini-ImageNet for different backbones and varying number of samples. 3-layer logistic classifier works better than the single-layer logistic classifier in most of the regimes. This justifies the choice of the 3-layer logistic classifier for the experiments carried out in the main paper.
has larger accuracy improvement than the classifier penalized with the KL-divergence to the nonuniform prior over the class probabilities. This further validates the effectiveness of Firth bias reduction in reducing the parameter estimation bias present in the few-shot setting.
1 5 10 15 20 25 Number of Shots
0.0
0.5
1.0
1.5
2.0
2.5
(A CC
)%
5-way ResNet10 ResNet18 ResNet34 ResNet50 ResNet101
1 5 10 15 20 25 Number of Shots
0.0
0.5
1.0
1.5
2.0
2.5
(A CC
)%
10-way ResNet10 ResNet18 ResNet34 ResNet50 ResNet101
Figure A9: Improvements of Firth-penalized logistic classifier over the baseline on miniImageNet with 5 and 10 number of classes. Except for the number of classes, the experiments were performed in the same setting as Figure 2 experiments.

heading: F COMPARING FIRTH BIAS REDUCTION AGAINST STANDARD LABEL SMOOTHING TECHNIQUES
text: To demonstrate that Firth bias-reduction cannot simply be replaced with label smoothing, we tested two advanced variants of label smoothing that are superior to the original version as proposed by Pereyra et al. (2017). The first variant, called confidence penalty, uses the entropy of the classifier’s output (or equivalently, reverses the direction of the KL divergence in the original version of label smoothing (Szegedy et al., 2016)); and the second variant, called unigram label smoothing, uses prior distribution over the classes instead of uniform, which has been shown to be advantageous when the output labels’ distribution is imbalanced in Pereyra et al. (2017). Note that both the variants were investigated for training a full deep neural network with a feature extractor backbone in large-sample regimes in Pereyra et al. (2017). Surprisingly, our evaluation of these two variants for training only classifiers on novel classes in few-shot regimes indicates that they are not as helpful as Firth bias-reduction.
Our experiments in Figure 3 show that, in imbalanced few-shot classification tasks, the improvements achieved by unigram label smoothing is significantly less than the Firth penalized classifier. This validates the bias suppression effect of Firth – a property that cannot be achieved by unigram label smoothing.
We also performed more experiments in the same setting to compare Firth bias reduction against confidence penalty regularization. As summarized in Table A4, in all the settings Firth bias reduction has larger significant improvements than the confidence penalty technique. This further supports the value of using Firth bias-reduction and the fact that its impact on few-shot classification cannot be reproduced with well-known and widely-used label smoothing techniques.

heading: G ADDITIONAL FEATURE BACKBONES
text: To test Firth bias reduction technique for additional backbones, we used pretrained DenseNet and MobileNet backbones on tiered-ImageNet from (Wang et al., 2019). The accuracy improvements of Firth penalized logistic classifier over the baseline averaged over 10000 trials are plotted in Figure A10. Regardless of the number of classes the improvements are always positive.

heading: H COMPARISON WITH ADDITIONAL SOTA METHOD
text: Table A5 summarizes the accuracy improvements obtained by integrating Firth bias reduction with the distribution calibration method (Yang et al., 2021) under different shots and ways on the tieredImageNet dataset. This method calibrates the features to follow a normal distribution, and generates
5 10 15 20 50 100 150 Number of Classes
0.00
1.00
2.00
3.00
(A CC
)%
DenseNet
1-Shot 5-Shot 10-Shot
5 10 15 20 50 100 150 Number of Classes
0.00
1.00
2.00
3.00
(A CC
)%
MobileNet
1-Shot 5-Shot 10-Shot
Figure A10: Firth bias reduction improves the accuracy of the logistic classifier on tieredImageNet for both DenseNet (top) and MobileNet (bottom) as backbone architectures. The improvement is consistent over the number of classes in few-shot classification.The error bars are almost non-existent (i.e., less than 0.02%) since over 10,000 trials with matching randomization factors were used for obtaining each single point. Firth bias reduction improvements are always positive, and it never hurts to use it.
artificial samples from the estimated normal distribution as data augmentation to aid the few-shot classification. In its SOTA setting, the features are transformed using Tukey transformations, 750 artificial samples are generated per class, and a logistic classifier is used. We tested Firth bias reduction in two scenarios: (i) SOTA setting without generating artificial samples from the calibrated distribution; and (ii) SOTA setting with 750 artificial samples generated per class, as shown in Table A5. The 95% confidence intervals for the accuracy improvements are reported in both cases.
As shown in Table A5, Firth bias reduction produces positive improvements in all cases, which are statistically significant in all the cases. As expected, Firth bias reduction produces larger improvements when artificial sample generation is disabled and thus there is a larger maximum likelihood estimation bias on the linear classifier (despite the use of Tukey transformation). In the case that artificial samples are added and they are tuned to follow the original normal distribution, having 750 of them can to some extent alleviate the bias in the estimation of the logistic classifier’s weights in the few-shot regime. However, the results show that even in the presence of more data (artificial samples), Firth bias reduction is still effective. This suggests that producing artificial samples to augment data cannot resolve the estimation bias issue, as they are more likely to be similar to the limited real samples available. We suspect this will become an even more severe problem in the cross-domain few-shot setting; producing artificial samples will be significantly less effective than Firth bias reduction, due to the domain shift. We leave this investigation as future work.
5-way 10-way
Before After Improvement Before After Improvement
1-shot 74.96 75.03 ± 0.19 0.07 ± 0.01 61.46 61.49 ± 0.13 0.03 ± 0.00 5-shot 87.43 87.48 ± 0.13 0.06 ± 0.00 77.73 77.83 ± 0.10 0.10 ± 0.00
10-shot 89.83 89.88 ± 0.11 0.05 ± 0.00 81.52 81.64 ± 0.09 0.11 ± 0.00
15-way 20-way
Before After Improvement Before After Improvement
1-shot 53.45 53.47 ± 0.10 0.02 ± 0.00 47.78 47.79 ± 0.07 0.01 ± 0.00 5-shot 70.70 70.99 ± 0.07 0.28 ± 0.00 65.26 65.60 ± 0.03 0.34 ± 0.00
10-shot 75.37 75.71 ± 0.06 0.34 ± 0.00 70.57 70.99 ± 0.02 0.42 ± 0.00
Table A6: The Firth bias reduction improvements on the CIFAR-FS dataset shown in Figure 4 of the main paper. “Before” stands for the novel set accuracy without having any Firth bias reduction, and “After” stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column thanks to the random-effect matching procedure we used in this study. The “Before” confidence intervals were similar to the “After” confidence intervals, and thus not repeated due to space constraints.
5-way 10-way
Before After Improvement Before After Improvement
1-shot 65.17 65.59 ± 0.18 0.41 ± 0.02 50.38 50.64 ± 0.11 0.26 ± 0.01 5-shot 82.60 83.04 ± 0.12 0.44 ± 0.01 71.15 71.91 ± 0.10 0.76 ± 0.02
10-shot 86.82 87.04 ± 0.09 0.22 ± 0.01 77.34 77.87 ± 0.08 0.52 ± 0.01
15-way 20-way
Before After Improvement Before After Improvement
1-shot 42.65 42.85 ± 0.08 0.20 ± 0.01 37.56 37.76 ± 0.07 0.20 ± 0.00 5-shot 63.73 64.76 ± 0.07 1.03 ± 0.01 58.35 59.52 ± 0.04 1.17 ± 0.01
10-shot 70.87 71.71 ± 0.05 0.84 ± 0.01 66.06 67.12 ± 0.03 1.06 ± 0.01
Table A7: The Firth bias reduction improvements on the mini-ImageNet dataset shown in Figure 4 of the main paper. “Before” stands for the novel set accuracy without having any Firth bias reduction, and “After” stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column thanks to the random-effect matching procedure we used in this study. The “Before” confidence intervals were similar to the “After” confidence intervals, and thus not repeated due to space constraints.
5-way 10-way
Before After Improvement Before After Improvement
1-shot 73.50 73.64 ± 0.25 0.14 ± 0.03 61.20 61.44 ± 0.16 0.24 ± 0.01 5-shot 88.00 88.31 ± 0.12 0.30 ± 0.01 79.41 80.01 ± 0.11 0.60 ± 0.01
10-shot 90.94 91.14 ± 0.10 0.21 ± 0.01 83.88 84.47 ± 0.09 0.58 ± 0.01
15-way 20-way
Before After Improvement Before After Improvement
1-shot 53.90 53.97 ± 0.15 0.07 ± 0.01 48.81 48.96 ± 0.11 0.15 ± 0.01 5-shot 73.33 74.21 ± 0.09 0.88 ± 0.01 68.58 69.71 ± 0.08 1.13 ± 0.01
10-shot 78.70 79.57 ± 0.08 0.86 ± 0.01 74.58 75.74 ± 0.07 1.16 ± 0.01
50-way 100-way
Before After Improvement Before After Improvement
1-shot 33.91 34.13 ± 0.06 0.22 ± 0.01 24.80 25.00 ± 0.03 0.20 ± 0.00 5-shot 52.60 54.71 ± 0.05 2.10 ± 0.01 41.03 43.59 ± 0.03 2.56 ± 0.01
10-shot 59.67 61.94 ± 0.04 2.27 ± 0.01 47.77 50.81 ± 0.02 3.04 ± 0.01
150-way
Before After Improvement
1-shot 20.37 20.56 ± 0.02 0.19 ± 0.00 5-shot 34.89 37.54 ± 0.01 2.65 ± 0.01 10-shot 41.04 44.28 ± 0.01 3.25 ± 0.01
Table A8: The Firth bias reduction improvements on the tiered-ImageNet dataset shown in Figure 4 of the main paper. “Before” stands for the novel set accuracy without having any Firth bias reduction, and “After” stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column thanks to the random-effect matching procedure we used in this study. The “Before” confidence intervals were similar to the “After” confidence intervals, and thus not repeated due to space constraints.
ResNet10 ResNet18
Before After Improvement Before After Improvement
1-shot 7.96 7.97 ± 0.06 0.01 ± 0.01 7.57 7.58 ± 0.05 0.01 ± 0.01 5-shot 8.22 8.23 ± 0.06 0.01 ± 0.01 7.94 7.95 ± 0.06 0.01 ± 0.01
10-shot 8.19 8.24 ± 0.05 0.06 ± 0.05 8.01 8.15 ± 0.05 0.14 ± 0.06 15-shot 8.21 8.36 ± 0.05 0.15 ± 0.06 8.09 8.47 ± 0.05 0.38 ± 0.06 20-shot 8.22 8.51 ± 0.05 0.28 ± 0.06 8.15 8.75 ± 0.06 0.60 ± 0.06 25-shot 8.25 8.51 ± 0.05 0.25 ± 0.06 8.27 8.92 ± 0.05 0.65 ± 0.06
ResNet34 ResNet50
Before After Improvement Before After Improvement
1-shot 7.47 7.48 ± 0.05 0.01 ± 0.01 7.51 7.52 ± 0.05 0.01 ± 0.01 5-shot 7.69 7.70 ± 0.05 0.01 ± 0.01 7.73 7.78 ± 0.05 0.05 ± 0.05 10-shot 7.73 7.96 ± 0.05 0.23 ± 0.05 7.83 8.59 ± 0.05 0.76 ± 0.06 15-shot 7.79 8.22 ± 0.05 0.43 ± 0.06 7.89 9.02 ± 0.05 1.13 ± 0.06 20-shot 7.87 8.41 ± 0.05 0.55 ± 0.06 8.15 9.67 ± 0.06 1.52 ± 0.06 25-shot 7.97 8.54 ± 0.05 0.57 ± 0.06 8.44 10.71 ± 0.06 2.27 ± 0.06
ResNet101
Before After Improvement
1-shot 7.63 7.65 ± 0.05 0.02 ± 0.02 5-shot 7.81 7.88 ± 0.05 0.07 ± 0.05
10-shot 7.96 8.81 ± 0.05 0.86 ± 0.06 15-shot 8.10 9.50 ± 0.05 1.40 ± 0.06 20-shot 8.37 10.29 ± 0.06 1.91 ± 0.06 25-shot 8.74 10.62 ± 0.06 1.88 ± 0.06
Table A9: The Firth bias reduction improvements on the mini-ImageNet dataset shown in Figure 2 of the main paper. “Before” stands for the novel set accuracy without having any Firth bias reduction, and “After” stands for the novel set accuracy after applying Firth bias reduction. Note that the confidence intervals are much smaller for the improvement column thanks to the random-effect matching procedure we used in this study. The “Before” confidence intervals were similar to the “After” confidence intervals, and thus not repeated due to space constraints.
3-Layer Logistic Classifier

Review summaries:
The submission examines the impact of Maximum-Likelihood Estimation (MLE)'s small-sample bias in the few-shot classification setting. The authors point out that while asymptotically consistent, MLE is biased in the finite-data regime, and propose to apply Firth bias reduction to few-shot classifiers to counter this bias. In this setting, the technique simplifies to a KL divergence regularizer between the uniform distribution over class labels and the learner's predictions.  Experiments on mini-ImageNet, CIFAR-FS, and tiered-ImageNet using ResNet and WideResNet network architectures show modest but consistent improvements across all settings. Firth bias reduction is shown to outperform other regularization approaches such as L2 regularization and label smoothing alternatives (confidence penalty and unigram label smoothing). 

This work introduces a method to reduce the bias of MLE for FSL problems. Authors developed the bias reduction formulation based on the Firth bias reduction method and extended it for cosine classifier as well. A through set of experiments demonstrate that the proposed method provides statistically significant improvement over the baselines. 

The paper proposes to reduce the bias inherent to the estimation of the classifier used in Few Shot Classification settings. To this purpose Firth bias reduction is proposed to be used. Theoretical computation of this bias reduction is carried out within the context of Few Shot Classification resulting to a practical implementation. It turns out that it could be associated to a regularization term encouraging uniform class assignment probabilities. This results is also generalized to the context of using a cosine classifier on the features.  Extensive experiments are carried out to show the benefits of this bias reduction on SOTA approaches and with various backbone architectures.

This paper tackles the few-shot learning problem, which is a hot topic in the representation learning field. Specifically, the authors aim to train accurate classifiers using a small number of samples – Maximum Likelihood Estimators are biased for them. As a result, the authors propose to utilize Firth’s Penalized Maximum Likelihood Estimator to modify the ordinary MLEs, and prevent the bias when training classifiers for few-shot instances. The authors have validated the effectiveness of Firth bias reduction from multiple aspects, and the experimental results show the performance improvement is consistent across various circumstances, e.g., with different backbones, data distributions, and classifiers.