Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective

1 Introduction . Balanced and large-scaled datasets [ 49 , 39 ] have promoted deep neural networks to achieve remarkable success in many visual tasks [ 23 , 48 , 21 ] . However , real-world data typically exhibits a long-tailed ( LT ) distribution [ 34 , 42 , 25 , 17 ] , and collecting a minority category ( tail ) sample always leads to more occurrences of common classes ( head ) [ 53 , 26 ] , resulting in most labels associated with limited instances . The paucity of samples may cause insufficient feature learning on the tail classes [ 64 , 12 , 33 , 42 ] , and such data imbalance will bias the model towards dominant labels [ 52 , 53 , 37 ] . Hence , the generalization of minority categories is an enormous challenge . The intuitive approaches such as directly over-sampling the tail [ 9 , 4 , 45 , 50 , 5 ] or under-sampling the head [ 18 , 4 , 20 ] will cause serious robustness problems . mixup [ 60 ] and its extensions [ 55 , 59 , 10 ] are effective feature improvement methods and contribute to a well-calibrated model in balanced datasets [ 54 , 61 ] , i.e. , the predicted confidence indicates actual accuracy likelihood [ 16 , 54 ] . However , mixup is inadequately calibrated in an imbalanced LT scenario ( Fig.1 ) . In this paper , we raise a conception called ⇠-Aug to analyze mixup and figure out that it tends to generate more head-head pairs , resulting in unsatisfactory generalization of the tail . Therefore , we propose Uniform Mixup ( UniMix ) , which adopts a tail-favored mixing factor related to label prior and a inverse sampling strategy to encourage more head-tail pairs occurrence for better generalization and calibration . ⇤Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous works adjust the logits weight [ 31 , 12 , 52 , 58 ] or margin [ 6 , 44 ] on standard Softmax cross-entropy ( CE ) loss to tackle the bias towards dominant labels . We analyze the inconstancy of label prior , which varies in LT train set and balanced test set , and pinpoint an inherent bias named Bayes Bias ( Bayias ) . Based on the Bayesian theory , the posterior is proportional to prior times likelihood . Hence , it ’ s necessary to adjust the posterior on train set by compensating different prior for each class , which can serve as an additional margin on CE . We further demonstrate that the Bayias-compensated CE ensures classification calibration and propose a unified learning manner to combine Bayias with UniMix towards a better-calibrated model ( see in Fig.1 ) . Furthermore , we suggest that bad calibrated approaches are counterproductive with each other , which provides a heuristic way to analyze the combined results of different feature improvement and loss modification methods ( see in Tab.4 ) . In summary , our contributions are : 1 ) We raise the concept of ⇠-Aug to theoretically explain the reason of mixup ’ s miscalibration in LT scenarios and propose Unimix ( Sec.3.1 ) composed of novel mixing and sampling strategies to construct a more class-balanced virtual dataset . 2 ) We propose the Bayias ( Sec.3.2 ) to compensate the bias incurred by different label prior , which can be unified with UniMix by a training manner for better classification calibration . 3 ) We conduct sufficient experiments to demonstrate that our method trains a well-calibrated model and achieves state-of-the-art results on CIFAR-10-LT , CIFAR-100-LT , ImageNet-LT , and iNaturalist 2018 . 2 Analysis of mixup The core of supervised image classification is to find a ✓ parameterized mapping F✓ : X 2 Rc⇥h⇥w 7 ! Y 2 RC⇥1 to estimate the empirical Dirac delta distribution P ( x , y ) = 1 N PN i=1 ( xi , yi ) of N instances x 2 X and labels y 2 Y . The learning progress by minimizing Eq.1 is known as Empirical Risk Minimization ( ERM ) , where L ( Y = yi , F✓ ( X = xi ) ) is xi ’ s conditional risk . R ( F✓ ) = Z x2X L ( Y = y , F✓ ( X = x ) ) dP ( x , y ) = 1 N XN i=1 L ( Y = yi , F✓ ( X = xi ) ) ( 1 ) To overcome the over-fitting caused by insufficient training of N samples , mixup utilizes Eq.2 to extend the feature space to its vicinity based on Vicinal Risk Minimization ( VRM ) [ 8 ] . ex = ⇠ · xi + ( 1 ⇠ ) · xj ey = ⇠ · yi + ( 1 ⇠ ) · yj ( 2 ) where ⇠ ⇠ Beta ( ↵ , ↵ ) , ↵ 2 [ 0 , 1 ] , the sample pair ( xi , yi ) , ( xj , yj ) is drawn from training dataset Dtrain randomly . Hence , Eq.2 converts P ( X , Y ) into empirical vicinal distribution P⌫ ( ex , ey ) = 1 N PN i=1 ⌫ ( ex , ey|xi , yi ) , where ⌫ ( · ) describes the manner of finding virtual pairs ( ex , ey ) in the vicinity of arbitrary sample ( xi , yi ) . Then , we construct a new dataset D⌫ : = { ( exk , eyk ) } Mk=1 via Eq.2 and minimize the empirical vicinal risk by Vicinal Risk Minimization ( VRM ) : R⌫ ( F✓ ) = Z ex2 eX L ⇣ eY = ey , F✓ ( eX = ex ) ⌘ dP⌫ ( ex , ey ) = 1 M XM i=1 L ⇣ eY = eyi , F✓ ( eX = exi ) ⌘ ( 3 ) mixup is proven to be effective on balanced dataset due to its improvement of calibration [ 54 , 16 ] , but it is unsatisfactory in LT scenarios ( see in Tab.4 ) . In Fig.1 , mixup fails to train a calibrated model , which surpasses baseline ( ERM ) a little in accuracy and seldom contributes to calibration ( far from y = x ) . To analyze the insufficiency of mixup , the definition of ⇠-Aug is raised . Definition 1 ⇠-Aug . The virtual sample ( exi , j , eyi , j ) generated by Eq.2 with mixing factor ⇠ is defined as a ⇠-Aug sample , which is a robust sample of class yi ( class yj ) iff ⇠ 0.5 ( ⇠ < 0.5 ) that contributes to class yi ( class yj ) in model ’ s feature learning . In LT scenarios , we reasonably assume the instance number n of each class is exponential with parameter [ 12 ] if indices are descending sorted by nyi , where yi 2 [ 1 , C ] and C is the total class number . Generally , the imbalance factor is defined as ⇢ = ny1/nyC to measure how skewed the LT dataset is . It is easy to draw = ln ⇢/ ( C 1 ) . Hence , we can describe the LT dataset as Eq.4 : P ( Y = yi ) = RR xi2X , yj2Y 1 ( X = xi , Y = yi ) dxidyjRR xi2X , yj2Y 1 ( X = xi , Y = yj ) dxidyj = e e C e yi , yi 2 [ 1 , C ] ( 4 ) Then , we derive the following corollary to illustrate the limitation of naïve mixup strategy . Corollary 1 When ⇠ ⇠ Beta ( ↵ , ↵ ) , ↵ 2 [ 0 , 1 ] , the newly mixed dataset D⌫ composed of ⇠-Aug samples ( exi , j , eyi , j ) follows the same long-tailed distribution as the origin dataset Dtrain , where ( xi , yi ) and ( xj , yj ) are randomly sampled from Dtrain . ( See detail derivation in Appendix A.2 . ) Pmixup ( Y ⇤ = yi ) = P 2 ( Y = yi ) + P ( Y = yi ) ZZ yi 6=yj Beta ( ↵ , ↵ ) P ( Y = yj ) d⇠dyj = e e C e yi , yi 2 [ 1 , C ] ( 5 ) In mixup , the probability of any ( exi , j , eyi , j ) belongs to class yi or class yj is strictly determined by ⇠ and E ( ⇠ ) ⌘ 0.5 . Furthermore , both ( xi , yi ) and ( xj , yj ) are randomly sampled and concentrated on the head instead of tail , resulting in that the head classes get more ⇠-Aug samples than the tail ones . 3 Methodology . 3.1 UniMix : balance-oriented feature improvement . mixup and its extensions tend to generate head-majority pseudo data , which leads to the deficiency on the tail feature learning and results in a bad-calibrated model . To obtain a more balanced dataset D⌫ , we propose the UniMix Factor ⇠⇤i , j related to the prior probability of each category and a novel UniMix Sampler to obtain sample pairs . Our motivation is to generate comparable ⇠-Aug samples of each class for better generalization and calibration . UniMix Factor . Specifically , the prior in imbalanced train set and balanced test set of class yi is defined as Ptrain ( Y = yi ) , ⇡yi , and Ptest ( Y = yi ) ⌘ 1/C , respectively . We design the UniMix Factor ⇠⇤i , j for each virtual sample exi , j instead of a fixed ⇠ in mixup . Consider adjusting ⇠ with the class prior probability ⇡yi , ⇡yj . It is intuitive that a proper factor ⇠i , j = ⇡yj/ ( ⇡yi + ⇡yj ) ensures exi , j to be a ⇠-Aug sample of class yj if ⇡yi ⇡yj , i.e. , class yi occupies more instances than class yj . However , ⇠i , j is uniquely determined by ⇡yi , ⇡yj . To improve the robustness and generalization , original Beta ( ↵ , ↵ ) is adjusted to obtain UniMix Factor ⇠⇤i , j . Notice that ⇠ is close to 0 or 1 and symmetric at 0.5 , we transform it to maximize the probability of ⇠i , j = ⇡yj/ ( ⇡yi + ⇡yj ) and its vicinity . Specifically , if note ⇠ ⇠ Beta ( ↵ , ↵ ) as f ( ⇠ ; ↵ , ↵ ) , we define ⇠⇤i , j ⇠ U ( ⇡yi , ⇡yj , ↵ , ↵ ) as : ⇠⇤i , j ⇠ U ( ⇡yi , ⇡yj , ↵ , ↵ ) = 8 > > < > > : f ( ⇠⇤i , j ⇡yj ⇡yi + ⇡yj + 1 ; ↵ , ↵ ) , ⇠⇤i , j 2 [ 0 , ⇡yj ⇡yi + ⇡yj ) ; f ( ⇠⇤i , j ⇡yj ⇡yi + ⇡yj ; ↵ , ↵ ) , ⇠⇤i , j 2 [ ⇡yj ⇡yi + ⇡yj , 1 ] ( 6 ) Rethink Eq.2 with ⇠⇤i , j described as Eq.6 : exi , j = ⇠⇤i , j · xi + ( 1 ⇠⇤i , j ) · xj eyi , j = ⇠⇤i , j · yi + ( 1 ⇠⇤i , j ) · yj ( 7 ) We have the following corollary to show how ⇠⇤i , j ameliorates the imbalance of Dtrain : Corollary 2 When ⇠⇤i , j ⇠ U ( ⇡yi , ⇡yj , ↵ , ↵ ) , ↵ 2 [ 0 , 1 ] , the newly mixed dataset D⌫ composed of ⇠-Aug samples ( exi , j , eyi , j ) follows a middle-majority distribution ( see Fig.2 ) , where ( xi , yi ) and ( xj , yj ) are both randomly sampled from Dtrain . ( See detail derivation in Appendix A.3 . ) P⇤mixup ( Y ⇤ = yi ) = P ( Y = yi ) Z yj < yi 1 ✓Z ⇠⇤i , jU ( ⇡i , ⇡j , ↵ , ↵ ) d⇠ ⇤ i , j 0.5 ◆ P ( Y = yj ) dyj = ( e e C ) 2 ⇣ e ( yi+1 ) e 2 yi ⌘ , yi 2 [ 1 , C ] ( 8 ) UniMix Sampler . UniMix Factor facilitates ⇠-Aug samples more balance-distributed over all classes . However , most samples are still ⇠-Aug for the head or middle ( see Fig.2 ( green ) ) . Actually , the constraint that pair xi , xj drawn from the head and tail respectively is preferred , which dominantly generates ⇠-Aug samples for tail classes with ⇠⇤i , j . To this end , we consider sample xj from Dtrain with probability inverse to the label prior : Pinv ( Y = yi ) = P⌧ ( Y = yi ) R yj2Y P ⌧ ( Y = yj ) dyj ( 9 ) When ⌧ = 1 , UniMix Sampler is equivalent to a random sampler . ⌧ < 1 indicates that xj has higher probability drawn from tail class . Note that xi is still randomly sampled from Dtrain , i.e. , it ’ s most likely drawn from the majority class . The virtual sample exi , j obtained in this manner is mainly a ⇠-Aug sample of the tail composite with xi from the head . Hence Corollary3 is derived : Corollary 3 When ⇠⇤i , j ⇠ U ( ⇡yi , ⇡yj , ↵ , ↵ ) , ↵ 2 [ 0 , 1 ] , the newly mixed dataset D⌫ composed of ⇠-Aug samples ( exi , j , eyi , j ) follows a tail-majority distribution ( see Fig.2 ) , where ( xi , yi ) is randomly and ( xj , yj ) is inversely sampled from Dtrain , respectively . ( See detail derivation in Appendix A.4 . ) PUniMix ( Y ⇤ = yi ) = P ( Y = yi ) Z yj < yi 1 ✓Z ⇠⇤i , jU ( ⇡i , ⇡j , ↵ , ↵ ) d⇠ ⇤ i , j 0.5 ◆ Pinv ( Y = yj ) dyj = ( e e C ) ( e ⌧C e ⌧ ) ⇣ e yi ( ⌧+1 ) e ( ⌧+yi ) ⌘ , yi 2 [ 1 , C ] ( 10 ) With the proposed UniMix Factor and UniMix Sampler , we get the complete UniMix manner , which constructs a uniform ⇠-Aug samples distribution for VRM and greatly facilitates model ’ s calibration ( See Fig.2 ( red ) & 1 ) . We construct D⌫ : = { ( exk , eyk ) } Mk=1 where { exk , eyk } is ( exi , j , eyi , j ) generated by ( xi , yi ) and ( xj , yj ) . We conduct training via Eq.3 and the loss via VRM is available as : L ( eyk , F✓ ( exk ) ) = ⇠⇤i , jL ( yi , F✓ ( exi , j ) ) + ( 1 ⇠⇤i , j ) L ( yj , F✓ ( exi , j ) ) ( 11 )

Summary: The authors proposed a new method based on two approaches, UniMix and Bayias, to sample input-output pairs from unbalanced datasets and use them in Mixup. The authors report experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, showing superior performance in terms of accuracy and calibration scores over a variety of baselines.  While I could get the main idea behind the proposed method, which seems appealing, I had problems understanding all the details, mainly because of a substantial lack of clarity. Sentences are often disconnected which makes it difficult to understand what the authors mean.  My score is borderline but I am open to increase it if the authors can provide a solid rebuttal and substantially improve the paper in terms of clarity.