On the Importance of Firth Bias Reduction in Few-Shot Classification

1 INTRODUCTION . Few-shot image classification is the practice of learning accurate classifiers using a small number of labeled samples ( Fei-Fei et al. , 2006 ; Wang et al. , 2020 ; Snell et al. , 2017 ; Vinyals et al. , 2016 ; Finn et al. , 2017 ) . It has a wide range of applications from face and gesture recognition ( Pfister et al. , 2014 ) to visual navigation in robotics ( Finn et al. , 2017 ) . Essentially , modern few-shot classification methods can be viewed as a combination of learning ( 1 ) a strong feature representation through a backbone network ( e.g. , Verma et al . ( 2019 ) ; Tian et al . ( 2020 ) ; Gidaris et al . ( 2019 ) ) , and ( 2 ) an accurate small-sample classifier ( e.g. , Chen et al . ( 2019 ) ; Wang and Hebert ( 2016 ) ) . Therefore , two key questions arise in few-shot image classification : ( 1 ) How can we obtain a strong feature representation ? and ( 2 ) How can we train accurate classifiers using a small number of samples ? There have been many existing methods addressing the former question using a host of different techniques ( Snell et al. , 2017 ; Rusu et al. , 2019 ; Verma et al. , 2019 ; Mangla et al. , 2020 ) . Here we focus on the less-explored second question . For our purposes , we will use standard feature representations and methods for training the backbone model for few-shot classification . That still leaves us with a severe classifier problem than most people realize . There is a substantial difficulty with training a classifier using a small number of samples . In particular , with very few samples , standard classification machinery is biased . In other words , although the Maximum Likelihood Estimators ( MLEs ) are statistically consistent and asymptotically normal ( Fahrmeir and Kaufmann , 1985 ) , it is well-established that MLEs are biased for a small number ofN samples , with bias ofO ( N−1 ) ( Firth , 1993 ; Whitehead , 1986 ; Box , 1971 ; Cox and Snell , 1968 ) . Since common logistic regression models are a type of MLEs , they are also biased ( Schaefer , 1983 ; Steyerberg et al. , 1999 ; Cordeiro and McCullagh , 1991 ; Firth , 1993 ) . Such biases increase the error rate of the few-shot trained classifiers , and so are important in few-shot learning . In fact , there is a standard solution for bias prevention by modifying the ordinary MLEs – known as Firth ’ s Penalized Maximum Likelihood Estimator ( PMLE ) ( Firth , 1993 ) . In the case of the exponential family of distributions , Firth has a simplified form that penalizes the likelihood by Jeffrey ’ s invariant prior ( Firth , 1993 ; Poirier , 1994 ) , which is proportional to the determinant of the Fisher Information Matrix F . For logistic and cosine classifiers ( Chen et al. , 2019 ) which are widely used in few-shot classification , since they belong to the exponential family , Firth bias reduction can be further cast as adding a log-determinant penalty ( log ( det ( F ) ) ) to the cross-entropy loss . While such standard strategies can control the bias in classifiers trained with very few samples , they have not been utilized in few-shot image classification tasks . In this paper , we show that using Firth bias reduction produces reliable improvements in a wide range of circumstances . We achieve this by deriving a simplified yet effective Firth formulation that penalizes the KL-divergence between the predicted distribution and the uniform distribution of classes , for both multinomial logistic regression models and cosine classifiers . Note that common regularization techniques ( such as L2-regularization and label smoothing ( Szegedy et al. , 2016 ) ) can not reduce the estimation bias of classifier weights in small-sample regimes ( Liu et al. , 2020 ) as Firth ’ s penalty does ; these regularization techniques are mainly used to control model complexity of deep neural networks for training feature extractor backbones in large-sample regimes . More concretely , our results indicate that the improvements produced by Firth bias reduction for few-shot image classification tasks are consistent across the board ( 1 ) on a wide range of feature representations , ( 2 ) with both balanced and imbalanced data distributions , ( 3 ) for both single-layer and multi-layer classifiers , ( 4 ) for both logistic regression and cosine classifiers , and ( 5 ) over multiple datasets and problems . Importantly , we found that Firth bias reduction consistently yields statistically significant and positive improvements , and we did not observe any performance penalty for utilizing Firth bias reduction . Such improvements are on the order of 0.5-2.5 % and up to 3 % in challenging few-shot classification tasks with a large number of classes . Our main contributions include ( 1 ) deriving a generalized expression for Firth bias reduction in few-shot multinomial logistic regression models , and providing geometrical insight into its effect on the classification probabilities space ; ( 2 ) evaluating the efficacy of the Firth penalized multinomial logistic model in few-shot scenarios , both with balanced and imbalanced data distributions ; ( 3 ) showing that Firth bias reduction can be extended beyond typical logistic regression models , and can be successfully adopted in cosine classifiers , and ( 4 ) providing an empirical comparison of Firth bias reduction to common regularization techniques , such as L2-regularization and label smoothing . 2 BACKGROUND . Mathematical Notations : In this work , we assume a multinomial logistic regression model for the classifier , with a total of J + 1 classes { 0 , 1 , 2 , · · · , J } . The logistic regression weights for class j is denoted as βj ( 1 ≤ j ≤ J ) . The class j = 0 is the base class with zero logistic regression weights . We assume to have a total number of N samples D = { ( x1 , y1 ) , · · · , ( xN , yN ) } . The ith target yi is the one-hot encoding of the ith label . The assignment probability of the ith sample to class j is denoted as Pi , j : Pi , j : = Pr ( y = j|xi ) = eβ ᵀ j x 1 + ∑J j′=1 e βᵀ j′x . ( 1 ) The likelihood of the sample set D given the weights β is denoted as Pr ( y|x ; β ) : Pr ( y|x ; β ) = N∏ i=1 J∑ j=1 1 [ yi = j ] · Pi , j , ( 2 ) where 1 [ · ] denotes the binary indicator function . We may abuse the notation and use Pr ( y|x ) instead of Pr ( y|x ; β ) in the derivation processes . Therefore , the logistic log-likelihood function Llogistic is defined as Llogistic : = N∑ i=1 J∑ j=0 1 [ yi = j ] · log ( Pi , j ) . ( 3 ) The Fisher Information Matrix ( FIM ) is defined as the Hessian of the negative log-likelihood function Llogistic : F : = −Hessβ ( Llogistic ) = Ey [ ∇βLlogistic · ∇βLᵀlogistic ] . ( 4 ) The Maximum Likelihood Estimator ( MLE ) for logistic regression is defined as β̂MLE : = arg max β Pr ( y|x ; β ) . ( 5 ) The dimension of the feature space is denoted as d in the derivations . U [ a , b ] denotes the ( discrete ) uniform distribution in the [ a , b ] interval . The cross-entropy and the KL-divergence of distributions p , q are defined as CE ( p‖q ) : = − ∑ x p ( x ) · log ( q ( x ) ) , DKL ( p‖q ) : = ∑ x p ( x ) · log ( p ( x ) q ( x ) ) . ( 6 ) Table A2 of the Appendix summarizes these notations . Small-Sample Bias of MLE : Assume β∗ is the true generative parameter . When the sample size N is small , it is shown that the MLE bias b ( β̂MLE ) : = E [ β̂MLE − β∗ ] is non-zero and of O ( N−1 ) ( Cox and Snell , 1968 ) . Therefore , while MLE is unbiased as N →∞ , it is inaccurate for few-shot learning . Firth Bias Reduction for MLE : Firth ’ s Penalized Maximum Likelihood Estimator ( PMLE ) ( Firth , 1993 ) is a modification to the ordinary MLE , which removes theO ( N−1 ) term from the small-sample bias . In particular , Firth has a simplified form for the exponential family . When Pr ( y|β , x ) belongs to the exponential family of distributions , the effect is to penalize the likelihood by Jeffrey ’ s invariant prior ( Poirier , 1994 ) , which is proportional to det ( F ) . Logistic and cosine classifiers ( Chen et al. , 2019 ) – the widely-used classification models in few-shot learning – belong to the exponential family ; so for them , Firth bias reduction simplifies to adding a penalty ( log ( det ( F ) ) ) to the crossentropy loss . In what follows , we will derive a further simplified yet effective Firth formulation for logistic and cosine classifiers , which is computationally more efficient , deals with the case when det ( F ) = 0 , and generalizes to multinomial distributions . MLE vs Firth ’ s PMLE for a Simple Case : To demonstrate the extent of the MLE bias and how Firth ’ s PMLE removes the leading O ( N−1 ) bias term , we simulated data from the geometric distribution with probability of success β = 0.5 as its only parameter . The geometric experiment was chosen since a closed-from solution for the MLE and Firth ’ s PMLE can be derived ; Given the samples ( y1 , · · · , yN ) from the geometric distribution , the sample mean is ȳ = 1N ∑ yi and we have β̂MLE = 1 ȳ and β̂Firth = N−1 Nȳ−1 . Note that since the sample mean is noisy , the MLE suffers from a noisy denominator making it biased . Figure 1 shows that β̂Firth is close to the true parameter β∗ for all sample sizes , whereas β̂MLE has a significant bias away from β∗ for small N . To further validate that the MLE bias is indeed of O ( N−1 ) , we plotted the MLE bias against the sample size in the log-log scale in Figure1 which shows that it is closely following a line with a negative unit slope . 3 FIRTH BIAS REDUCTION IN LOGISTIC AND COSINE CLASSIFIERS . In logistic models , the penalized likelihood function proposed by Firth is equivalent to imposing Jeffreys ’ prior ( Poirier , 1994 ) on the parameters and making an MAP estimate . In particular , Firth bias reduction encourages models with “ large ” F by multiplying the likelihood by det ( F ) . This penalty degenerates when det ( F ) = 0 . We work in the highest dimensional subspace where F has full rank , and use det ( A|r ) to denote the product of all r non-zero eigenvalues of A , obtaining Pr ( β|x , y ) = 1 Z · Pr ( y|x ; β ) · det ( F |r ) 1/2 , ( 7 ) where Z is a normalization constant and det ( F |r ) 1/2 is the Jeffery ’ s prior . Taking the log of both sides yields the log-posterior as a sum of the logistic log-likelihood function and the Firth bias reduction term : L : = log ( Pr ( β|x , y ) ) = Llogistic + LFirth , ( 8 ) where we have LFirth : = λ 2 log ( det ( F |r ) ) + c.t.e .. ( 9 ) The definition of LFirth was left ambiguous up to a constant w.r.t . β to facilitate the Firth bias reduction term ’ s interpretation and avoid the definition of similar terms . Furthermore , λ controls for the impact of the Firth term on the outcome relative to Llogistic . We then apply a series of derivation steps to simplify Equation ( 9 ) , which are left to Section A of the Appendix . Finally , the Firth bias reduction term can be expressed as LFirth = λ · 1 N N∑ i=1 [ J∑ j=0 ( xᵀi βj − log J∑ j′=0 ex ᵀ i βj′ ) ] = λ · 1 N N∑ i=1 [ J∑ j=0 log ( Pi , j ) ] . ( 10 ) For the cosine classifier , the proof that log ( det ( F ) ) simplifies to Equation ( 10 ) involves straightforward manipulation of the proof for the logistic classifier , and is left to Section B of the appendix . The normalization of the βj weights in the cosine classifier turns into a pure scale term in the optimization , and for a cosine classifier , scaling of the βj weights does not affect predictions . Therefore , this term should be ignored , and the bias reduction term effectively becomes the same as Equation ( 10 ) . Interpreting the Firth Bias Reduction for Logistic Models : It is well known that Jeffery ’ s prior shrinks the parameter estimates towards zero which is equivalent to encouraging uniform class assignment probabilities ( Firth , 1993 ; Bull et al. , 2002 ) . We take an alternative approach to reach the same conclusion in the following . By re-arranging Equation ( 10 ) , one can see the ∑J j=0 log ( Pi , j ) term as a scaled average over a uniform distribution of classes : LFirth ∝ 1 N N∑ i=1 [ J∑ j=0 1 J + 1 · log ( Pi , j ) ] = −1 N N∑ i=1 [ CE ( U [ 0 , J ] ‖Pi ) ] . ( 11 ) Therefore , we can abuse the notation , and re-define the coefficient λ and the constant to have LFirth = λ · −1 N N∑ i=1 DKL ( U [ 0 , J ] ‖Pi ) + c.t.e .. ( 12 ) This means that by dropping the constants , the optimization objective L can be rewritten as L = −1 N N∑ i=1 [ DKL ( yi‖Pi ) + λ ·DKL ( U [ 0 , J ] ‖Pi ) ] . ( 13 ) Since the KL-divergence and the FIM define an information geometry and a Riemanian metric on probabilistic measure spaces ( Nielsen , 2020 ) , a geometrical insight into Equation ( 13 ) is provided in Figure A6 as well . Firth Bias Reduction vs. Common Regularization : While this insight brings Firth bias reduction closer to the common regularization techniques ( e.g. , L2-regularization ) , it is worth noting that ( 1 ) common regularization techniques mainly focus on controling model complexity , instead of reducing small-sample estimation bias ; ( 2 ) Firth bias reduction operates on a much lower-dimensional target distribution space unlike L2 which operates in the high-dimensional parameter space ; ( 3 ) Firth uses the same kind of metric as the logistic loss ; and ( 4 ) Firth bias reduction is dimensionally consistent like the Natural Gradients ( Amari , 1998 ; Pascanu and Bengio , 2013 ) , whereas L2 is not . Firth Bias Reduction vs. Label Smoothing : Notice that the original form of label smoothing used for training in large-sample regimes ( Szegedy et al. , 2016 ) has the same formulation as the simplified Firth penalty term in Equation ( 12 ) for multinomial logistic classifiers . However , Firth bias reduction is inherently different in that it reduces the classifier estimation bias in the small-sample regimes , whereas label smoothing penalizes over-confident predictions when training deep neural networks with large amounts of samples . Generally , the Firth bias reduction term ( i.e. , log ( det ( F ) ) ) is not the same as the label smoothing penalty ( i.e. , DKL ( U‖Pi ) ) for deep neural networks . Additional analysis and empirical comparisons are provided in Section 4.4 and Section F of the Appendix .

Summary: This paper tackles the few-shot learning problem, which is a hot topic in the representation learning field. Specifically, the authors aim to train accurate classifiers using a small number of samples – Maximum Likelihood Estimators are biased for them. As a result, the authors propose to utilize Firth’s Penalized Maximum Likelihood Estimator to modify the ordinary MLEs, and prevent the bias when training classifiers for few-shot instances. The authors have validated the effectiveness of Firth bias reduction from multiple aspects, and the experimental results show the performance improvement is consistent across various circumstances, e.g., with different backbones, data distributions, and classifiers.