Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer

1 Introduction . Learning with imbalanced datasets is a common yet still very challenging scenario in many machine learning applications . Typical scenarios include : ( i ) rare events , where the event prevalence is extremely low while their implications are of high cost , e.g. , severe risks that people seek to avert [ 56 ] ; ( ii ) emerging objects in a dynamic environment , which call for quick adaptation of an agent to identify new cases with only a handful target examples and plentiful past experience [ 31 ] . A typical scenario in natural datasets is that the occurrence of different objects follows a power law distribution . And in many situations , the accurate identification of those rarer instances bears more significant social-economic values , e.g. , fraud detection [ 24 ] , driving safety [ 27 ] , nature conservation [ 50 ] , social fairness [ 17 ] , and public health [ 63 , 92 ] . Notably , severe class imbalance and lack of minority labels are the two major difficulties in this setting , which render standard learning strategies unsuitable [ 57 ] . Without explicit statistical adjustments , the imbalance induces bias towards the majority classes . On the other hand , the lack of minority representations prevents the identification of stable correlations that generalize in predictive settings . In addition , due to technological advancements , more and higher dimensional data are routinely collected for analysis . This also inadvertently exacerbates the issue of minority modeling as there is an excess of predictors relative to the limited occurrence of minority samples . ∗Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Various research efforts have been directed to address the above issues , with class re-balancing being the most popular heuristic . The two most prominent directions in the category include statistical resampling and reweighting . Resampling alters the exposure frequency during training , i.e. , more for the minorities and less so for the majorities [ 29 ] . Alternatively , reweighting directly amends the relative importance of each sample based on their class [ 22 ] , sampling frequency [ 7 ] and associated cost for misclassification [ 30 ] . Recent developments also have considered class-sensitive and dataadaptive losses to more flexibly offset the imbalance [ 10 , 60 , 65 , 55 , 80 , 51 ] . While being intuitive and working reasonably well in practice , an important is that these approaches offer no protection against the over-fitting of minority instances ( see Figure 1 ) [ 91 ] , a fundamental obstacle towards better generalization . To circumvent this key limitation , inductive biases are often solicited to impose strong constraints that suppress spurious correlations that hinder out-of-sample generalization . A classical strategy is few-shot learning [ 31 , 89 , 40 ] , where the majority examples are used to train meta-predictors and transferable features , leaving only a few parameters to tune for the minority data . In anomaly detection , methods such as one-class classification instead regard minority classes as outliers that do not always associate with stable , recognizable patterns [ 67 , 75 ] . Despite their relatively strong assumptions , these methods capitalize on their superior ability in generalizing in the low sample regime in empirical settings . More recently , establishing causal invariance has emerged as a new , powerful learning principle for better generalization ability even under apparent distribution shifts [ 73 ] . In contrast to standard empirical risk minimization ( ERM ) schemes , where the generalization to similar data distributions is considered , causally-inspired learning instead embraces robustness against potential perturbations [ 5 ] . This is achieved via only attending to causally relevant features and associations postulated to be invariant under different settings [ 70 ] . Specifically , contributions from spurious , unstable features are effectively blocked or attenuated . Interestingly , compromise of performance can be expected in those models [ 74 ] , as a direct consequence of discarding useful ( but non-causal ) correlations in exchange for better causal generalization . Recently , [ 48 ] proposed non-parametric causal disentanglement of data representation to identify invariant relations . This work explores the advancement of imbalanced data learning via adopting causal perspectives , with the insight that the key algorithm can be reformulated as an energy-based contrastive learning procedure to drastically improve efficiency and flexibility . In recognition of the limitations discussed above , we present energy-based causal representation transfer ( ECRT ) : a novel imbalanced learning scheme that brings together ideas from causality , contrastive learning , energy modeling , data-augmentation and weakly-supervised learning , to address the identified weakness of existing solutions . Our key contributions are : ( i ) a causal representation encoder informed by an invariant generative mechanism based on generalized contrastive learning ; ( ii ) integrated data-augmentation and source representation regularization techniques exploiting feature independence to enrich mi- nority representations that better balance the trade-off between utility and invariance ; ( iii ) a key novelty is the derivation of an energy-based contrastive learning algorithm that greatly enhance model parallelism for large-label settings and extends generalized contrastive learning ; and ( iv ) insightful discussions on the justifications for the use of the proposed approach . Our claims are supported by strong experimental evidence . 2 Preliminaries . Notation and problem definition . We use x ∈ Rp to denote the input data and z ∈ Rd for the predictive features extracted from x . Let y ∈ { 1 , . . . , M } be the class label . The number of training samples and those with label m are denoted as n and nm , respectively . We use E [ · ] to denote the expectation ( average ) of an empirical distribution , ai to denote a vector associated with the i-th sample , and [ a ] b to denote the b-th entry of vector a . For simplicity , we assume class M is the minority class , i.e. , nm nM , for m ∈ { 1 , . . . , M − 1 } . Throughout , we refer to X , Z and S as data , feature , and source spaces , respectively , as shown in Figure 2 , and defined below . Our goal is to accurately predict the label of minority instances with very limited training examples of it . Generalization to multiple minority categories is straightforward . Generalized contrastive learning and ICA The proposed approach is based on a generalized form of independent component analysis ( ICA ) , which addresses the inverse problem of signal disentanglement [ 46 ] . Specifically , ICA decorrelates features Z of the observed signal X into a source signal representation S = fψ ( Z ) , where fψ ( z ) is a smooth and invertible mapping known as the de-mixing function , and while assuming that the components of S are statistically independent , i.e. , with density q ( s ) = ∏ j qj ( [ s ] j ) . Notationally , we call [ s ] j the j-th independent component ( IC ) of Z . While nonlinear ICA ( NICA ) is generally infeasible [ 19 , 47 ] , [ 48 ] has recently proposed a setting in which the identification of NICA can be achieved , by requiring an additional auxiliary label y . Specifically , NICA assumes that source signals are conditionally independent given y , i.e. , q ( s|y ) = ∏ j qj ( [ s ] j |y ) , then fψ ( z ) can be identified using generalized contrastive learning ( GCL ) , whose implementation is detailed below . Real Samples Feature Space Estimated Source Space GCL solves a generalized regression problem in which a critic function predicts whether label y and representation z are correctly paired , i.e. , congruent . We call ( yi , zi ) a congruent pair and ( yj , zi ) an incongruent pair if i 6= j . Specifically , the critic is defined as rν ( y , z ) =∑d a=1 r a ν ( y , [ s ] a ) , where r a ν ( · , · ) is a neural network with parameters ν , whose inputs are the label y and the a-th coordinate of s = fψ ( z ) , denoted as [ s ] a . Then , GCL optimizes the following objective : argmin fψ , rν Ei [ h ( −rν ( yi , zi ) ) ] + Ej 6=i [ h ( rν ( yj , zi ) ) ] ︸ ︷︷ ︸ LGCL ( fψ , rν ) , ( 1 ) where h ( r ) = log ( 1 + exp ( r ) ) is the softplus function . ( 1 ) seeks to optimize fψ ( · ) and rν ( · , · ) by maximizing the discriminative power to tell apart congruent and incongruent pairs . In fact , an interesting result showed by [ 48 ] revealed that maximizing the ability of the critic function rν ( · , · ) for correctly identifying matching pairs ( z , y ) leads to the identification ( up to univariate transformation ) of fψ ( · ) , such that the components of s are conditionally independent given y . See the Supplementary Material ( SM ) for a formal exposition . 3 Energy-based Causal Representation Transfer . In this section , we describe the construction of energy-based causal representation transfer ( ECRT ) : a causally informed data transformation and augmentation procedure to improve learning with imbalanced datasets . Our model assumes a shared causal data-generation procedure , which can be accurately identified by learning with the majority classes under assumed class-conditional representation independence . We obtain decorrelated representations that facilitate data augmentation and efficient learning with minority classes . The proposed model consists of the following components : ( i ) a feature encoder module z = eθ ( x ) ; ( ii ) two classification modules , hφ′ ( z ) and hφ ( s ) , for predicting label y from features z and sources s , respectively ; ( iii ) a nonlinear ICA module for the de-mixing function s = fψ ( z ) ; ( iv ) a critic function rν ( y , z ) for GCL ; and ( v ) a data augmentation module . Further , ( θ , φ′ , φ , ψ , ν ) denote the parameters of all the neural-network-based modules . Algorithm 1 outlines a general workflow , and below , we elaborate on our assumptions and detail the implementation of ECRT . 3.1 Model assumptions Algorithm 1 Energy-based Causal Representation Transfer . 1 . Pre-train encoder and predictor : eθ , hφ′ ← argmin { E [ ` ( h ( e ( x ) ) , y ) ] } 2 . NICA estimation with fixed z = eθ ( x ) : fψ , rν ← GCL ( { ( z , y ) } ) % Equation ( 1 ) 3 . Source space augmentation with fixed s = fψ ( z ) : ( s̃ , y ) ← AUG ( { s|y =M } ) % Equation ( 2 ) 4 . Minority predictor modeling with augmented source : hφ ← argmin { E [ ` ( h ( s , y =M ) ] +λE [ ` ( h ( s̃ , y =M ) ] } To enable knowledge transfer across classes , we make the following assumptions : Assumption 3.1 . Let z be a sufficient statistic ( features ) of x for predicting label y , all class conditional feature distributions p ( z|y ) share a common ICA demixing function fψ ( z ) . This implies there exists a smooth invertible function fψ : Z → S , and a set of IC distributions { q ( s|y = m ) } Mm=1 , that are linked to the conditional feature distributions p ( z|y = m ) via S m = fψ ( Z m ) , where Sm ∼ q ( s|y = m ) and Zm ∼ p ( z|y = m ) . The subscript m in Sm and Zm indicates y = m , in a slight abuse of notation . Importantly , f−1ψ ( s ) : S → Z is the invariant causal mechanism underlying the features of observed data . This specification , which is consistent with Assumption 3.1 enables the identification of the shared generating process , namely , the de-mixing function fψ ( z ) that connects the likely dissimilar conditionals { p ( z|y = m ) } Mm=1 via the source conditionals { q ( s|y = m ) } Mm=1 , as demonstrated in the context of NICA by [ 48 ] A shortcoming of Assumption 3.1 is that the hypothesis supporting it is somewhat strong , untestable and may not hold in practice . However , we argue that the structural constraints imposed on the model by the assumption via the source conditionals q ( s|y = m ) and the invertibility of fψ ( z ) , restrict the search space of the otherwise over-flexible model space powered by neural networks . Further , the causal mechanism implied by f−1ψ ( s ) enables an effective knowledge transfer mechanism across classes via q ( s|y = m ) .

Summary: The paper proposes to use an auxiliary feature space (S) via non-linear independent component anaylsis (NICA) on an existing feature space (Z), e.g., of a pre-trained deep neural network, to transfer causal information of majority classes to minority classes in the context of imbalanced (or long-tailed) classification. More specifically, the method trains an invertible neural networks Z → S using only majority samples to perform NICA, and uses this model to augment minority samples by simply permuting their S-features, thanks to the coordinate-wise independence of S. This augmentation has an effect of transferring the causal features of majority samples into the minority samples. Experimental results confirms that such techniques can indeed prevent classifiers from overly relying on non-generalizable features of minority samples.