Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer

1 Introduction . Learning with imbalanced datasets is a common yet still very challenging scenario in many machine learning applications . Typical scenarios include : ( i ) rare events , where the event prevalence is extremely low while their implications are of high cost , e.g. , severe risks that people seek to avert [ 56 ] ; ( ii ) emerging objects in a dynamic environment , which call for quick adaptation of an agent to identify new cases with only a handful target examples and plentiful past experience [ 31 ] . A typical scenario in natural datasets is that the occurrence of different objects follows a power law distribution . And in many situations , the accurate identification of those rarer instances bears more significant social-economic values , e.g. , fraud detection [ 24 ] , driving safety [ 27 ] , nature conservation [ 50 ] , social fairness [ 17 ] , and public health [ 63 , 92 ] . Notably , severe class imbalance and lack of minority labels are the two major difficulties in this setting , which render standard learning strategies unsuitable [ 57 ] . Without explicit statistical adjustments , the imbalance induces bias towards the majority classes . On the other hand , the lack of minority representations prevents the identification of stable correlations that generalize in predictive settings . In addition , due to technological advancements , more and higher dimensional data are routinely collected for analysis . This also inadvertently exacerbates the issue of minority modeling as there is an excess of predictors relative to the limited occurrence of minority samples . ∗Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Various research efforts have been directed to address the above issues , with class re-balancing being the most popular heuristic . The two most prominent directions in the category include statistical resampling and reweighting . Resampling alters the exposure frequency during training , i.e. , more for the minorities and less so for the majorities [ 29 ] . Alternatively , reweighting directly amends the relative importance of each sample based on their class [ 22 ] , sampling frequency [ 7 ] and associated cost for misclassification [ 30 ] . Recent developments also have considered class-sensitive and dataadaptive losses to more flexibly offset the imbalance [ 10 , 60 , 65 , 55 , 80 , 51 ] . While being intuitive and working reasonably well in practice , an important is that these approaches offer no protection against the over-fitting of minority instances ( see Figure 1 ) [ 91 ] , a fundamental obstacle towards better generalization . To circumvent this key limitation , inductive biases are often solicited to impose strong constraints that suppress spurious correlations that hinder out-of-sample generalization . A classical strategy is few-shot learning [ 31 , 89 , 40 ] , where the majority examples are used to train meta-predictors and transferable features , leaving only a few parameters to tune for the minority data . In anomaly detection , methods such as one-class classification instead regard minority classes as outliers that do not always associate with stable , recognizable patterns [ 67 , 75 ] . Despite their relatively strong assumptions , these methods capitalize on their superior ability in generalizing in the low sample regime in empirical settings . More recently , establishing causal invariance has emerged as a new , powerful learning principle for better generalization ability even under apparent distribution shifts [ 73 ] . In contrast to standard empirical risk minimization ( ERM ) schemes , where the generalization to similar data distributions is considered , causally-inspired learning instead embraces robustness against potential perturbations [ 5 ] . This is achieved via only attending to causally relevant features and associations postulated to be invariant under different settings [ 70 ] . Specifically , contributions from spurious , unstable features are effectively blocked or attenuated . Interestingly , compromise of performance can be expected in those models [ 74 ] , as a direct consequence of discarding useful ( but non-causal ) correlations in exchange for better causal generalization . Recently , [ 48 ] proposed non-parametric causal disentanglement of data representation to identify invariant relations . This work explores the advancement of imbalanced data learning via adopting causal perspectives , with the insight that the key algorithm can be reformulated as an energy-based contrastive learning procedure to drastically improve efficiency and flexibility . In recognition of the limitations discussed above , we present energy-based causal representation transfer ( ECRT ) : a novel imbalanced learning scheme that brings together ideas from causality , contrastive learning , energy modeling , data-augmentation and weakly-supervised learning , to address the identified weakness of existing solutions . Our key contributions are : ( i ) a causal representation encoder informed by an invariant generative mechanism based on generalized contrastive learning ; ( ii ) integrated data-augmentation and source representation regularization techniques exploiting feature independence to enrich mi- nority representations that better balance the trade-off between utility and invariance ; ( iii ) a key novelty is the derivation of an energy-based contrastive learning algorithm that greatly enhance model parallelism for large-label settings and extends generalized contrastive learning ; and ( iv ) insightful discussions on the justifications for the use of the proposed approach . Our claims are supported by strong experimental evidence . 2 Preliminaries . Notation and problem definition . We use x ∈ Rp to denote the input data and z ∈ Rd for the predictive features extracted from x . Let y ∈ { 1 , . . . , M } be the class label . The number of training samples and those with label m are denoted as n and nm , respectively . We use E [ · ] to denote the expectation ( average ) of an empirical distribution , ai to denote a vector associated with the i-th sample , and [ a ] b to denote the b-th entry of vector a . For simplicity , we assume class M is the minority class , i.e. , nm nM , for m ∈ { 1 , . . . , M − 1 } . Throughout , we refer to X , Z and S as data , feature , and source spaces , respectively , as shown in Figure 2 , and defined below . Our goal is to accurately predict the label of minority instances with very limited training examples of it . Generalization to multiple minority categories is straightforward . Generalized contrastive learning and ICA The proposed approach is based on a generalized form of independent component analysis ( ICA ) , which addresses the inverse problem of signal disentanglement [ 46 ] . Specifically , ICA decorrelates features Z of the observed signal X into a source signal representation S = fψ ( Z ) , where fψ ( z ) is a smooth and invertible mapping known as the de-mixing function , and while assuming that the components of S are statistically independent , i.e. , with density q ( s ) = ∏ j qj ( [ s ] j ) . Notationally , we call [ s ] j the j-th independent component ( IC ) of Z . While nonlinear ICA ( NICA ) is generally infeasible [ 19 , 47 ] , [ 48 ] has recently proposed a setting in which the identification of NICA can be achieved , by requiring an additional auxiliary label y . Specifically , NICA assumes that source signals are conditionally independent given y , i.e. , q ( s|y ) = ∏ j qj ( [ s ] j |y ) , then fψ ( z ) can be identified using generalized contrastive learning ( GCL ) , whose implementation is detailed below . Real Samples Feature Space Estimated Source Space GCL solves a generalized regression problem in which a critic function predicts whether label y and representation z are correctly paired , i.e. , congruent . We call ( yi , zi ) a congruent pair and ( yj , zi ) an incongruent pair if i 6= j . Specifically , the critic is defined as rν ( y , z ) =∑d a=1 r a ν ( y , [ s ] a ) , where r a ν ( · , · ) is a neural network with parameters ν , whose inputs are the label y and the a-th coordinate of s = fψ ( z ) , denoted as [ s ] a . Then , GCL optimizes the following objective : argmin fψ , rν Ei [ h ( −rν ( yi , zi ) ) ] + Ej 6=i [ h ( rν ( yj , zi ) ) ] ︸ ︷︷ ︸ LGCL ( fψ , rν ) , ( 1 ) where h ( r ) = log ( 1 + exp ( r ) ) is the softplus function . ( 1 ) seeks to optimize fψ ( · ) and rν ( · , · ) by maximizing the discriminative power to tell apart congruent and incongruent pairs . In fact , an interesting result showed by [ 48 ] revealed that maximizing the ability of the critic function rν ( · , · ) for correctly identifying matching pairs ( z , y ) leads to the identification ( up to univariate transformation ) of fψ ( · ) , such that the components of s are conditionally independent given y . See the Supplementary Material ( SM ) for a formal exposition . 3 Energy-based Causal Representation Transfer . In this section , we describe the construction of energy-based causal representation transfer ( ECRT ) : a causally informed data transformation and augmentation procedure to improve learning with imbalanced datasets . Our model assumes a shared causal data-generation procedure , which can be accurately identified by learning with the majority classes under assumed class-conditional representation independence . We obtain decorrelated representations that facilitate data augmentation and efficient learning with minority classes . The proposed model consists of the following components : ( i ) a feature encoder module z = eθ ( x ) ; ( ii ) two classification modules , hφ′ ( z ) and hφ ( s ) , for predicting label y from features z and sources s , respectively ; ( iii ) a nonlinear ICA module for the de-mixing function s = fψ ( z ) ; ( iv ) a critic function rν ( y , z ) for GCL ; and ( v ) a data augmentation module . Further , ( θ , φ′ , φ , ψ , ν ) denote the parameters of all the neural-network-based modules . Algorithm 1 outlines a general workflow , and below , we elaborate on our assumptions and detail the implementation of ECRT . 3.1 Model assumptions Algorithm 1 Energy-based Causal Representation Transfer . 1 . Pre-train encoder and predictor : eθ , hφ′ ← argmin { E [ ` ( h ( e ( x ) ) , y ) ] } 2 . NICA estimation with fixed z = eθ ( x ) : fψ , rν ← GCL ( { ( z , y ) } ) % Equation ( 1 ) 3 . Source space augmentation with fixed s = fψ ( z ) : ( s̃ , y ) ← AUG ( { s|y =M } ) % Equation ( 2 ) 4 . Minority predictor modeling with augmented source : hφ ← argmin { E [ ` ( h ( s , y =M ) ] +λE [ ` ( h ( s̃ , y =M ) ] } To enable knowledge transfer across classes , we make the following assumptions : Assumption 3.1 . Let z be a sufficient statistic ( features ) of x for predicting label y , all class conditional feature distributions p ( z|y ) share a common ICA demixing function fψ ( z ) . This implies there exists a smooth invertible function fψ : Z → S , and a set of IC distributions { q ( s|y = m ) } Mm=1 , that are linked to the conditional feature distributions p ( z|y = m ) via S m = fψ ( Z m ) , where Sm ∼ q ( s|y = m ) and Zm ∼ p ( z|y = m ) . The subscript m in Sm and Zm indicates y = m , in a slight abuse of notation . Importantly , f−1ψ ( s ) : S → Z is the invariant causal mechanism underlying the features of observed data . This specification , which is consistent with Assumption 3.1 enables the identification of the shared generating process , namely , the de-mixing function fψ ( z ) that connects the likely dissimilar conditionals { p ( z|y = m ) } Mm=1 via the source conditionals { q ( s|y = m ) } Mm=1 , as demonstrated in the context of NICA by [ 48 ] A shortcoming of Assumption 3.1 is that the hypothesis supporting it is somewhat strong , untestable and may not hold in practice . However , we argue that the structural constraints imposed on the model by the assumption via the source conditionals q ( s|y = m ) and the invertibility of fψ ( z ) , restrict the search space of the otherwise over-flexible model space powered by neural networks . Further , the causal mechanism implied by f−1ψ ( s ) enables an effective knowledge transfer mechanism across classes via q ( s|y = m ) .

Summary: This paper consider the problem of learning in the presence of a severe class imbalance, specifically when there are more than two classes. It stipulates a causal generating mechanism to "enable efficient knowledge transfer from the dominant classes to the under-represented counterparts," which hinges on the important assumption that all classes share a common set of salient features. In detail, the proposed procedure involves: (1) pre-training an encoder and predictor, (2) NICA estimation using the learned features, (3) source-space augmentation, (4) minority predictor modeling with the augmented source. Experiments on CIFAR100, iNaturalist, TinyImagenet and ArXiv show that the model