Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer
Authors: Junya Chen
Zidi Xiu
Benjamin Goldstein
Ricardo Henao
Lawrence Carin
Chenyang Tao
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: imbalanced data learning
causal representation learning
rare-event modeling
contrastive learning

Abstract: Dealing with severe class imbalance poses a major challenge for many real-world applications, especially when the accurate classification and generalization of minority classes are of primary interest.
In computer vision and NLP, learning from datasets with long-tail behavior is a recurring theme, especially for naturally occurring labels. Existing solutions mostly appeal to sampling or weighting adjustments to alleviate the extreme imbalance, or impose inductive bias to prioritize generalizable associations. Here we take a novel perspective to promote sample efficiency and model generalization based on the invariance principles of causality. Our contribution posits a meta-distributional scenario, where the causal generating mechanism for label-conditional features is invariant across different labels. Such causal assumption enables efficient knowledge transfer from the dominant classes to their under-represented counterparts, even if their feature distributions show apparent disparities. This allows us to leverage a causal data augmentation procedure to enlarge the representation of minority classes. Our development is orthogonal to the existing imbalanced data learning techniques thus can be seamlessly integrated. The proposed approach is validated on an extensive set of synthetic and real-world tasks against state-of-the-art solutions. 

Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer
Junya Chen Zidi Xiu Benjamin Goldstein'. Rícardo Henao Lawrence Carin', and Chenyang Tao
Duke University KAUST (junya.hhe, chen, zidi xiu, chenyang. tao] @duke edu
Abstract
Dealing with severe class imbalance poses major challenge for many real-world applications, especially when the accurate classification and generalization minority classes are of primary interest. In computer vision and NLP, learning from datasets with long-tail behavior is a recurring theme, especially for naturally occurring labels. Existing solutions mostly appeal to sampling weighting ad- justments to alleviate the extreme imbalance, or impose inductive bias to prioritize generalizable associations. Here take novel perspective promote sample efficiency and model generalization based on the invariance principles of causality. Our contribution posits meta-distribel distributional scenario, where the causal generating mechanism for cobblicondiiional features invariant across different labels. Such causal assumption enables efficient knowledge transfer from the dominant classes to their under-represented counterparts, even if their feature distributions show ap- parent disparities. This allows to leverage causal data augmentation procedure to enlarge the representation of minority classes. Our development is orthogonal to the existing imbalanced data learning techniques thus can be seamlessly integrated. The proposed approach is validated on an extensive set of synthetic and real-world tasks against of-t--of-the-ar solutions.
Introduction
Learning with imbalanced datasets a common yet still very challenging scenario many machine learning applications. Typical scenarios include: (i) rare events, where the event prevalence is extremely low while their implications are of high cost, e.g., severe risks that people seek to avert [56]: (ii) emerging objects in dynamic environment, which call for quick adaptation of an agent to identify new cases with only handful target examples and plentiful past experience [31]. A typical scenario in natural datasets is that the occurrence of different objects follows power law distribution. And in many situations, the accurate identification of those rarer instances bears more significant social-economic values, e.g., fraud detection (24), driving safety [27). nature conservation [50), social
fairness [17], and public health 663 (22..
Notably, severe class imbalance and lack of minority labels are the two major difficulties in this setting, which render standard learning strategies unsuitable [57]. Without explicit statistical adjustments, the imbalance induces bias towards the majority classes. On the other hand, the lack of minority representations prevents the identification of stable correlations that generalize in predictive settings. In addition, due to technological advancements, more and higher dimensional data are routinely collected for analysis. This also inadvertently exacerbates the issue of minority modeling as there an excess of predictors relative to the limited occurrence of minority samples.
Equal Contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

F1 Score
Label Frequency
ECRT w/ Aug
LDAM
ECRT w/O Aug
ERM
66
64
Majority
Minority
Figure 1: Energy-based causal representation transfer alleviates tail imbalance for natural image classification. The shaded region denotes label frequency in the iMaturalist data, with some representative images shown. Solid curves are for label-conditional FI scores (higher is better) for the proposed energy-based causal representation transfer (ECRT), LDAM (state of the art) and the standard ERM. ECRT consistently outperforms the others, especially in the low-sample regime.
Various research efforts have been directed to address the above issues, with class re- re-balancing being the most popular heuristic. The two most prominent directions in the category include statistical resampling and reweighting. Resampling alters the exposure frequency during training, i.e., more for the minorities and less so for the majorities [29] Alternatively, reweighting directly amends the relative importance of each sample based their class [22] sampling frequency [7] and associated cost for misclassification 30) Recent developments also have considered lass-sensitive and data- 80 [5]] While being intuitive and working reasonably well in practice, an important is that these approaches offer no protection against the fittinitting of minority instances (see Figure 91 fundamental obstacle towards better generalization.
adaptive losses to more flexibly offset the imbalance [TO] 60 65 55
To circumvent this key limitation, inductive biases are often solicited to impose strong constraints that suppress spurious correlations that hinder out-of-sample generalization. A classical strategy is few-shot learning [3 89 40), where the majority examples used train meta-predictors and transferable features, leaving only a few parameters to tune for the minority data. In anomaly detection, methods such one-class classification instead regard minority classes outliers that do not always associate with stable, recognizable patterns [5].
Despite their relatively strong assumptions, these methods capitalize on their superior ability in generalizing the low sample regime empirical settings.
More recently, establishing causal invariance has emerged as new, powerful learning principle for better generalization ability even under apparent distribution shifts (73). In contrast standard empirical risk minimization (ERM) schemes, where the generalization to similar data distributions is considered, causally-inspire learning instead embraces robustness against potential perturbations [5]. This is achieved via only attending to causally relevant features and associations postulated to be invariant under different settings [70]. Specifically, contributions from spurious, unstable features are effectively blocked or attenuated. Interestingly, compromise of performance can be expected in those models 74, as a direct consequence of discarding useful (but non-causal) correlations in exchange for better causal generalization. Recently, 443 proposed non-parametrie causal disentanglement of data representation to identify invariant relations.
This work explores the advancement of imbalanced data learning via adopting causal perspectives, with the insight that the key algorithm can be reformulated as an energy-based contrastive learning procedure to drastically improve efficiency and flexibility. In recognition of the limitations dis- cussed above, we present energy- energy-based based causal representation transfer (ECRT): a novel imbalanced learning scheme that brings together ideas from causality, contrastive learning, energy modeling, data-augmentation and weakly-supervised learning, to address the identified weakness of existing solutions. Our key contributions are: (i) causal representation encoder informed by invariant generative mechanism based on generalized contrastive learning; (ii) integrated data-augmentation and source representation regularization techniques exploiting feature independence to enrich mi-

---

nority representations that better balance the trade-off between utility and invariance; (iit) a key novelty the derivation of an energy-based contrastive learning algorithm that greatly enhance model parallelism for large-label settings and extends generalized contrastive learning; and (iv) insightful discussions on the justifications for the use of the proposed approach. Our claims are supported by strong experimental evidence.
2 Preliminaries
Notation and problem definition. We use E IR to denote the input data and z E R for the predictive features extracted from æ. Let y E {l,..., ,,....,,, M) be the class label. The number training samples and those with label m are denoted expectation (average) of an empirical distribution, a, to denote a vector associated with the i-th sample, and [a]o to denote the b-th entry of vector a. For simplicity, we assume class M is the minority class, Le., P.m > n.M for m E (1...., M - }). Throughout, we refer to X' 2 and S as data, feature, and source spaces, respectively, is to accurately predict the label of minority instances with very limited training examples of it. Generalization multiple minority categories is straightforward.
and nmm respectively. We use [[|] to denote the
shown in Figure 2 and defined below. Our goal
Generalized contrastive learning and ICA The proposed approach is based on a generalized form of independent component analysis (ICA), which addresses the inverse problem of signal disentanglement [46]. Specifically, ICA decorrelates features Z of the observed signal X into a source signal representation S (Z), where (x) is smooth and invertible mapping known as the de-mixing function, and while assuming that the components of S are statistically independent, L.., with density ((s) II, 95 ([$})) Notationally, we call [s); the j-th independent component (IC) of Z. While nonlinear ICA (NICA) is generally infeasible [19] 47 48) has recently proposed setting in which the identification of NICA can be achieved, by requiring an additional auxiliary label y. Specifically, NICA assumes that source signals are conditionally independent given y, i.e., ((>ly) = II, マ ((s]]lv). then (z) can be identified using generalized contrastive learning (GCL), whose implementation is detailed below.
GCL solves a generalized regression problem in which a critic function predicts whether la- bel y and representation are correctly paired,
Samples
Space
Estimated Source Space
2
2
4 2
3
7
2. 5 /7
7
7
i.e., congruent. We call (ys, z;) a congruent pair and (Vj, 2.) an incongruent pair if i o Specifically, the critic is defined as ru 2)
0
4 1 6
0
[
3
2
9
5
9 07
Ee=1 (3, [s]]a, where r-(--) is neural net- work with parameters v, whose inputs are the
Figure 2: Illustration of data space X, feature space 2 (predictive but entangled) and source space S (independent, or disentangled) representations identified by ECRT for MNIST.
label y and the a-th coordinate of s = f (z),
denoted as [s]a. Then, GCL optimizes the fol-
lowing objective:
arg min foste
2())] + Ej 2;)]]
Cooi (fe.r))
where ((()- log(1 exp(r)) is the softplus function. seeks to optimize (e)()) and ru(s)) by maximizing the discriminative power to tell apart congruent and incongruent pairs.
In fact, an interesting result showed by [88 revealed that maximizing the ability of the critic function - for correctly identifying matching pairs (z, y) leads to the identification (up to univariate transformation) of (-), such that the components of are conditionally independent given See the Supplementary Material (SM) for a formal exposition.
3 EEnergy-based Causal Representation Transfer
In this section, we describe the construction of energy-based causal representation transfer (ECRT): a causally informed data transformation and augmentation procedure to improve learning with imbalanced datasets. Our model assumes a shared causal data-generation procedure, which can be accurately identified by learning with the majority classes under assumed class-conditional representation independence. We obtain decorrelated representations that facilitate data augmentation and efficient learning with minority classes.
3

---

Generalized Contrestive Leaming
Space Augmentation
Jackkian
Likellasod kos
Original Source Represemiation
Augment
bata
Fooeee Enceskrr MLAC Kollo xg(
Featurt
Souuce Encoder dewe 4 (7)
logpCa)
SiL
Source
Scunce Price P65)
A A2
Shuffle
GCT loss
Iroepren Land
Descakie
3
A4
Figure 3: Source space estimation module of ECRT. We Figure 4: Non-parametric source space use GCL to identify the demixing function s (z) via augmentation based on shuffling. telling apart congruent & incongruent pairs.
The proposed model consists of the following components: (i) feature encoder module = = ea (a:); (it) two classification modules, hag (z) and hs for predicting label y from features z and sources respectively; (tir) nonlinear ICA module for the de-mixing function 8 fo(2); (év) a critic function ru (y, z) for GCL; and (v) a data augmentation module. Further, (0, é', o 16, v) denote the parameters of all the neural-network. based modules. Algorithm [ottlines general workflow, and below, we elaborate on our assumptions and detail the implementation of ECRT.
3.1 Model assumptions
Algorithm 1 EEnergy-basee Causal Representation Transfer.
To enable knowledge transfer across classes, we make the following assump- tions:
1. Pre-train encoder and predictor:
eg. he < arg min ( [B(((((e(())) v)]
2. NICA estimation with fixed (x))
fe. +- GCL([(z, 3)})
Equation
Assumption 3.1. Let be sufficient statistic (features) of x for predicting la- bel y, all class conditional feature distri- butions p(zly) share a common ICA de- mixing function (z).
3. Source space augmentation with fixed
2
(3.4) AUG([s yy M)) % Equation 2
Minority predictor modeling with augmented source:
he + arg min E E(h(s, y = M))+ M)
This implies there exists smooth invertible function fa 2 + S, and set of IC distributions {g(sly m) M mm]m=1. that are linked to the conditional feature distributions p(zly = m)) via S (Z where 5t qss m) and z" p(z = m). The subscript in S and 2 indicates
in slight abuse of notation.
Importantly, (s) : S + is the invariant causal mechanism underlying the features of observed data. This specification, which is consistent with Assumption 3.1 enables the identification of the shared generating process, namely, the de-mixing function fo (z) that connects the likely dissimilar conditionals [p(zly m))
via the source conditionals {q( (sly = m demonstrated in m=1
the context of NICA by
A shortcoming of Assumption 3. is that the hypothesis supporting it is somewhat strong, untestable and may not hold in practice. However, we argue that the structural constraints imposed the model by the assumption via the source conditionals g(sly- = m) and the invertibility of (z), restrict the search space of the otherwise over-flexible model space powered by neural networks. Further, the causal mechanism implied by f3 (8) enables effective knowledge transfer mechanism across classes via q(sly m..
3.2 Easegy-based Causal Representation Transfer
Encoder pre-training. To implement ECRT, we first find good (reduced) feature representation of x highly predictive of label y. This can be achieved via supervised representation learning (see Figure 3. which optimizes encoder and predictor pair (eo(ar), (=)) to minimize the label prediction risk L(d') A E[e(h (eg(2)) v)], where E(...) suitable loss function, e,g., cross-ent cross-entropy, hinge loss, etc. To avoid capturing spurious (non-generale (on-geeerllizzble) features that overfit the minority class, we advocate training only with majority samples at this stage; assuming that M > 2. Alternatively, one could also consider unsupervised feature extraction schemes, such as auto-encoders. Further, we also recommend using statistical adjustments such as importance weighting, to reduce the impact of data imbalance.

---

De-mixing representation with GCL. After obtaining good feature representation Z = es (X), we proceed to learn the de-mixing function f (z), such that the coordinates of the source representa- tion S = fa( (Z) are (approximately) independent given the label y. This can be done by optimizing the GCL objective in ① with respect to the feature representation and label pairings (y, 2), adopting the masked auto-regressive flow (MAF) [69] to model the smooth, invertible transformation (2), which allows efficient parallelization of the autoregressive architecture via causal masking [14]. The
procedure is outlined in Figure 3
Augmenting the minority. Inspired by 83 we artificially augment the minority feature representa- tions z via random permutations in the source space S, as shown in Figure[4] Provided the assumed conditional independence of the sources, the features z" corresponding to label y = m are generated by 2 (S"), ) where each dimension in the source representation [St)], q(<))) /V m) independently and implicitly sampled described below. Specifically, using the (estimated) de-mixing function (z),), we can obtain an approximate empirical source distribution for each m. m. i.e., 5Th _ {s: = fo (x)())))) = m) = { where 5th is a collection of fm samples of g(sly= = m). Then, we can draw new artificial samples 8" q = m) by randomly permuting the coordinates within elements 5tt independently via
[8 0 = [
[g),,,
(2)
where o (01,... ; Od) is a random permutation of (1, (1...., mm). Note we have used 8 to emphasize that the source point artificially created via permutation o. We call this procedure nonparametric augmentation because it does not make distributional assumptions for g(s|y/ = m.) However, below we will discuss its limitations and consider an alternative where a parametric form is assumed. While aegeersuueeee it is tempting to refine the predictor hy (z) with artificial features augmented via nTm = f f="' in
Section 3.3 we will argue that it stands benefit more from training new predictor directly based on source representations, i.e., h (s). without the need for inverting (z).
Model refinement. Now we can leverage the augmented data to refine the prediction model. For minority class y = M, we optimize the following objective
( ((d)) + XEEz.' [e(har (#*), M)] - E-- (e(hrr( M)))
(3)
where [(o') is the loss used for pre- vreetraiiing. Conceptually, (3) replaces a portion of the minority samples with augmentations. The trade-off parameter X E [0. 1] encodes the relative confidence
for trusting the artificially generated representations 2 Ad obtained from S for the minority label y M. Further, at this stage we found it's 's beneficial to fix the encoder module to prevent the de- mixing function to accommodate the changes in the encoder which in practice may cause instability during training.
Challenges with naive implementation. We identify three major issues with naïvely implemented ECRT, to be addressed in the section below: (i) Representation conftict: since the GCL solution is not unique, do observe naïve GCL training drifts among viable source representations whose per- formance differ considerably, causing stability concerns; (ii) Costly augmentation: MAF inversions dominate the computation load during training, which becomes prohibitive in high dimensions; and (iii) Gridding artifact: a small minority sample size leads to pronounced augmentation bias when sampling nonparametrically via 2 manifested as gectangular-shapee grid (see Figure 5)
3.3 Improving causal representation transfer
Ensegy-based GCL. Our key insight improve GCL comes from the fact that Equation (1 is
essentially learning the density ratio between the joint and product of marginals, i.e., p(z.g) This immediately reminds us the recent literature on contrastive mutual information (MI) estimators, such as InfoNCE [7]]. In such works, variational lower bound of MI is derived, and the algorithm optimizes a critic function using the positive samples from the joint distribution, and the negative samples from the product of the marginals. At their optimal value, these critics recover the density ratio or a transformation of it. Our development is based on the recent work of (37], using an energy- perspective to improve contrastive learning. Specifically, we will be using a variant of the celebrated Donsker- Donsker-Varadhan (DV) estimator (28), and applied Fenchel duality trick to compute a solution Specifically, the Fenchel- -Donsker---oo Varudhan
p(z))(())
32
Donssee-aaarrahn (FDV) estimator takes the following form:
E, expl(go (g0 (I,, ,2) go (2,, 9²))
JOO 1bv (15,, v1,) Σ, exp (ōol F,, v,) Go (a.,, y. ))/T]
+ 1,
5

---

where go(x;, V:) is our critic of interest and (b Vzh) K log(Z>>> k exp(g(x1, (/)/(/)1)())))) k)) is the Donsker- Varadhan (DV) estimator (2 for the MI. Using the same parameterization used in GCL recovers the same causal identification property (see Appendix for details). Compared to the original GCL formulation, we are now using multiple negative samples instead of one, which greatly boosts learning efficiency [38. And this can be efficiently implemented with the bilinear critic trick [T5 13 37 that all in-batch data can be used
g yn)
as negatives. In our context, it greatly boosts training efficiency when dealing with a large number of different classes. See Algorithm S1 in Appendix.
Regularizing the data likelihood. Recall GCL solutions can only be identified up to an invertible transformation of each dimension [48], and the predictive performance of different valid GCL solutions can vary significantly. Empirically, we observe that a naive implementation of GCL often leads to source representations that are densely packed (see Figure S1 in the SM). This is undesirable. when decoding back to the feature space and making useful predictions, the neural network predictor will need be expansive, i.e., requiring large Lipschitz constant, thereby sacrificing optimization stability and model generalization according to existing learning theory 20 87].
To encourage source representations that are less condensed, consider simple, intuitive strategy consisting of regularizing the source representation with the log- big-likelihood in feature space. This likelihood can be easily obtained with MAF using fFLOw (fw) defined in the SM. Following common practice, we set source prior p(s) to the standard Gaussian, and optimize the following likelihood- regularized GCL objective:
Locr rv) = Locu ru) + PC (fo);
(5)
where p > 0 is the regularization strength. Naturally, this regularization will encourage the global source representation identified by GCL to be more consistent with Gaussian- -sapee distribution.
An alternative interpretation for the
Observations
Non con-parament Aug Parametric Aug
Oracle Aug
likelihhoddrrggulrizeedd objective in (5) is that it can be understood as a relax- ation to the conditional independence (Assumption B.I.. To see this, recall the likelihood objective obtained from the invertible neural networks (INN) alone attempts to map the source rep- resentations to be unconditionally in-
Figure 5: Comparison of different source augmentations over- laid on the ground-truth distribution (gray dots). A severe gridding artifact is observed in the low-sample regime for the nonparametric scheme, whereas the parametric augmentation closely matches the oracle in distribution.
dependent, as opposed to the con- ditional independence assumed by NICA. The regularized formulation (5) provides a safe "fall-back" " mode in case Assumption 3.1 is violated. This also motivates us to consider an important variant: making multiple class-dependent source priors, i.e., ps)(s) for each class label m E (1..... M) in (5). whose parameters (i.e., mean and variance) are jointly learned with other model parameters. Compared to the fully con-parametric objective (T). this strategy further encourages the source representations to be independent given the class labels, and it enables parametric data augmentation, i.e., sampling from the parametric label priors instead of permuting the indices. We refer to this variant as ECRT-MP, where MP stands for multiple priors. And similarly, ECRT- refers to the case when single prior is used. We have found that ECRT-MP performs better in most cases, and consequently, ECRT means ECRT-MP by default.
Modeling in the souree space. Rather than modeling the predictor har (z) we advocate instead for building the predictor directly the source space S, i.e., modeling with (s). This practice enjoys several benefits: (i) Easy & robust augmentation: many designs of high-dimensional flows are asymmetric computationally. and inverting a MAF is not only d times more costly than forward pass, it also numerically unstable at the boundary. Direct modeling in the source space circumvents the difficulties associated with MAF inversions during data augmentation; (ii) Feature whitening: the source representation identified by GCL is component- wise independent, and literature documents abundant empirical evidence that similar de-correlation based pre- prepproces,ing commonly known as whitening, benefits learning [5] 6 49).
(z) in the feature space 2,
Parametric augmentation. When the number of minority observations is scarce, the above non- parametric indices-shuffling augmentation suffers from the gridding artifact (Figure This artifact amplifies the augmentation bias in the low-sample regime. To overcome this limitation, we empirically observe that the estimated class-conditional source distributions are usually Gaussian- like after the

---

likelihood regularization (especially so when label conditional priors are used). In these situations, a parametric augmentation that draws synthetic source samples from a Gaussian distribution matched to the empirical mean and variance of minority source representations is more efficient.
3.4 Insights and remarks
To better appreciate the gains and limitations expected from ECRT, we compile a few complementary arguments below, through the lens of very different perspectives.
Why causal augmentation works. It is helpful to understand the gains from ECRT's causal augmen- tation beyond the heuristic that permuting the ICs provides more training samples for the minority class. [83] considered similar causal augmentation procedure for few-shot learning, and provided two major theoretical arguments: (i) the risk estimator based on the augmented examples is the uniformly minimum variance unbiased estimator given the accurate estimation of fu (see Theorem I, 833 ); and (it) with high probability, the generalization gap can be bounded by the approximation error of f (see Theorem 2, (83) In the SM, we give arguments that our causal augments give the 'best' label-conditional distribution estimate.
Speedup from shared embedding. While for typical supervised learning tasks the generalization bound scale as 3)) superior rate of O(n where 7) E 1 1] is possible, if there exists abundant data for an alternative, yet related task that shares the same feature embedding (see Theorem 3. 772). Note n refers to the size of labeled data directly related to the strong task of interest, in our case, prediction of minority labels. Our ECRT employs GCL to identify one such common embedding, i.e., the source space, using the majority examples, and consequently, improves predictions on the minority class.
78 via de- correlating the representations conditionally. Extensive empirical evidence has pointed to the fact that such representation de-correlation, more commonly known as data whitening [52], is expected to con- siderably improve learning efficiency [8]. This benefit has been attributed to the better conditioning of the Fisher information matrix for gradient-based optimization (25). thus rectifying second-order curvatures to accelerate convergence. Our source space modeling explicitly separates the task of representation disentanglement, and in turn, helps the prediction network to focus on its primary goal.
Representation whitening. Our ECRT causally disentangles representation
Potential limitations. The setting considered by ECRT restrictive in that precludes the learning of useful. yet non-transferable features predictive of the minority labels. For instance, there might be a feature unique to the minority class. However, since the de-mixer (z) is only trained on the majority domains absent of this feature, it can not be accounted for by the ECRT model. This is key limitation of causally inspired models, in that they are often too conservative for only retaining the invariant features, promoting cross-domain generalization at the cost of within-domain performance degradation 774.
4 Related Work
Causal invariance and representation learning, A major school of considerations for building ro- bust machine learning models to stipulate invariant causal relations across environments 76 62 12
such that one hopes to safely extrapolate beyond the training scenario 8 We broadly categorize such efforts into two streams, namely predictive causal models and generative causal models. Prominent examples from the first category include ICP 77 43 and IRM [5], which highlight the identification of invariant representation and causal relations via penalizing environmental heterogeneity. Our solution pertains to the second category, where the data distribution shifts across environments can be tethered by an invariant generation procedure [48] [3]. This work exploits the recovered causally invariant source representation to improve learning efficiency and mitigate the sample scarcity of minority labels in imbalanced sets.
Learning with imbalanced data. Resolving data imbalance is a heavily investigated topic [61] Standard sampling and weight adjustments suffer from caveats such as introducing bias and infor- mation loss. Due to these concerns, recent literature has actively explored adaptive strategies, such as redundancy-adjusted balancing weights [22]] and the theoretically grounded class-size adapted margins 10). Similar to boosting (34), adaptive weights are designed to prioritize the learning of less cels-classified examples (60 while excluding apparent outliers [59]. Much related to our setting are the meta- neaa-lea scenarios
[88 33, 90), where the model tries generalize & repurpose the
7

---

Extreme Classifiation Top Accuracy
MMMDPpay Pawe)
Unaugmented
Augmented
10 R30
Naturalla HSM embeising
20
Puram, Aug. Non Aug.
butterf
whete
Gillaam
FORT
ECRT w/b
serperi
ERM
10
damosely
10 20
50
75
100
150
10
20 40 80 160 320 640
2x
Number Samplers Per Class
Minority Sample Size
Figure 6: Extreme classifi-
Figure 7: Comparison of dif- ferent causal augmentations, lower is better. Parametric augmentation is more efficient with small samples.
Figure 8: ECRT source represen- tation trained on Naturalist visualized using tSNE embed- ding in two dimensions for eight random categories.
cation of 1K labels. ERM performs slightly better than random guess, while ECRT works significantly better.
knowledge learned from majority classes to minority predictions; and also the nugmentation-baser schemes to restore class balance 66 [68]
Data augmentation. Due to its exceptional effectiveness, augmentation schemes are widely adopted in practical applications [77]. These augmentation strategies are built on known invariant transforma- tions, e.g., rotation, scaling, noise corruption, etc. [79], simple interpolation heuristics [1 and more recently towards fully auto-mated procedures [21] Notably, recent trend in data augmentation highlights robust learning against adversarially crafted inputs (35] and generative augmentation procedures that compose realistic artificial samples 3
Closest to our contribution is causal mechanism transfer (CMT) 83 which focused on addressing few-shot learning for continuous regression. We note few key differences to our work: (i) CMT focused on domain adaptation and does not address classification; and (ii) it bundles (x,, y) for NICA which necessitates the flow inversion for sample augmentation. Grounded in the setting of imbalanced data learning, our ECRT extends applications and advocates source space modeling to simplify and improve causal augmentation. also features likelihood regularization to enhance representation regularity. We also offer new insights to justify the use of NICA -bssed augmentation, complementing the analysis from CMT by [83].
Domain adaption and causal mechanism transfer.
Energy- bbasd modeling for representation learning. There is growing recognition that the energy the lens of energy based modeling, the distribution of data is characterized as an (unnormalized) energy function 4 Optimization of the energy function is often considered challenging [8] and contrastive techniques have been proven effective [4] 39
perspective integral to representation learning [58] 71 82 37
Our work is a generalization of [48] that disentangles representations from an energy perspective. Interesting comparison can be made to (54) whose training objective bears resemblance to our ECRT. But the specific designs used by ECRT in the network architecture and scoring function allows ECRT to provably disentangle the feature representations and consequently capture causality for exploitation.
Experiments
To validate the utility of our model, we consider a wide range of (semi)-synthetic and real-world tasks experimentally. All experiments are implemented with PyTorch, and our code is available homtttts:////gg
com/ www/tiiiwii/////cooooaccco /idiXii/EERRRT More details of our setup & additional analyses are deferred to the SM Sections C-E.
5.1 Experimental setup
Baselines.
The following competing baselines are considered to benchmark the proposed solution: (i)
Empirical risk minimization (ERM), naïve bascline with no adjustment; (ii) importance-weiighinn
(IW) 19]. a class-weight balanced training loss; (iiz) Generative adversarial augmentation (GAN) 3 synthetic augmentations from adversarially-traine
sampler; (iv) Virtual adversarial training
(VAT) (66), robustness regularization with virtual perturbations; (v) FOCAL loss [60], cost-sensitive
adaptive weighting; (wi) dabel-distribution-- aware margin loss (LDAM) [0] margin-optimal class
weights All baselines tuned for best performance for TOP- 1 accuracy validation.
Evaluation metrics& & setup. We consider the following metrics to quantitatively assess performance:
(i) negative log- log-likeldoood (NLL); (ii) Fl score; (Mii TOP-k accuracy (k: = 1,5). Following the classical evaluation setup for imbalanced data learning, we learn on an imbalanced training set and report performance a balanced validation set.
8

---

0.95
INaturalist Minority F1
0.55
CIFAR100 Validation Top-1 Accuracy
x10
F1 Sensitivity to Aug.
Majorily Extegory Samplc Sizc
94
Reruraal
0.94
100
300
200
400
93
0.93
ERM
92
CRT
0.92
Miner
Sample Siz Aug)
0.2
CRT à 16-2
1E-4
91
Ninorit Sample Size
10
30
5D
0.91
0.1
10
10 w/Agg.
w/Aug
Training Epochs
100
10-"10-" 10" 10"
.5
Figure 9: Performance compar- Figure 10: Comparison of learn- Figure 11: Sensitivity analysis of ison with and without augmen- ing dynamics. ECRT enables tation. Augmented solutions im- both faster learning and better prove the effective sample size. model predictions.
augmentation strength X. Smaller minority sample sizes are more sensitive the augmentation.
Table 1: Comparison of performance on real-world datasets (f higher is better, + lower is better).
CIFAR100
INATURALIST ToP-5t
TinyINaGENET TOP-51
ARXIV
ToP-11
ToP-5t
NLL4
TOP-11
NLL4
ToP-11
NLL
Acct
NLL
ERM
49.29
78.22
2.95
66.73
87.86
1.70
58.52
79.01
3.22
44.64
0.0407
IW
43.97
68.96
3.89
67.63
88.94
1.66
60.50
80.23
2.92
46.02
0,0477
GAN
47.64
78.53
2.69
67.40
87.00
1.82
60.69
80.91
2.33
45.42
0,0407
VAT
46.47
74.23
3.19
67.06
87.39
1.90
59.69
82.05
2.42
45.82
0,0474
FOCAL
43.32
74.39
2.89
66.63
88.45
1.59
58.27
79.39
2.59
46.01
0.0416
LDAM
50.46
74.39
2.18
67.39
87.13
4.00
58.18
82.51
2.15
45.04
0.0450
ECRT-IP 52.31
81.14
1.98
68.38
88.13
1,48
62.46
83.50
1.79
48.33
0.0434
ECRT-MP
53.00
81.99
2.31
69.01
90.01
1,23
64.40
84.54
1.94
Toy model. We sample 2D standard Gaussians with different means and variances as source repre- sentation for each label class, which is then distorted by random affine and Hénon transformations 21 = 1- - 1.4 - + 52, 22 = 0.3 81. to induce sellwwordd-like complex association structures. See the SM Sec. D for detailed description and visualizations.
Real-world datasets. We consider the following semi-synthetic and real datasets: (i) Imbalanced MNIST and CIFAR100: standard image classification tasks with artificially created step-imbalanced following 10: (ii) Imbalanced ImageNet (2] a scaled-down version of the classic natural image dataset ImageNet, comprised of 200 classes, 500 samples per class and 10k validation images, with different simulated imbalances applied; (iid) iNaturalist 2019 [86]: challenging task for image classification in the wild comprised of 26k training and 3k validation images labeled with 14: classes, with a skewed distribution in label frequency; (ir) arXiv abstracts, imbalanced multi-label prediction of paper categories with 160k samples and 152 classes.
Preprocessing, model architecture, and tuning, We use pre-trained models to extract vectorized representations for big complex datasets: ResNet [42] for image models of TinyImageNer Incep- tion V3 for iNaturalist] and BERT 226 for the arXiv [ language model. These pre-trained representations are used as raw feature imputs, subsequently fed into a fully-connecter perceptrons (MLP) for source space encoding. For small image models (i.e., MNIST, CIFAR100) we directly train CNN from scratch for feature extraction. We used a random 8/2 split for training and validation, and applied Adam optimizers for training. We rely on the best out-of-sample cross-entropy and GCL. loss for hyperparameter tuning.
multi-layer
ECRT efficiency. We first examine the dynamics of efficiency gains from ECRT using toy model, with the results summarized in Figure Consistent with the shared embedding perspective, increasing the majority sample size improves the minority accuracy. Causal augmentation consistently improves performance, and it is most effective in the low-sample regime, offering more than 4x boosts in effective minority sample size. Figure I0 we compare modeling with feature and source representations, respectively. Source space modeling speeds up learning, in addition to the modest accuracy gains. Augmented training is slightly slower, but eventually converges to a better solution.
pth
httpa///bballllllyyyrrrroo/////s goggoooooooo_oooogeeeeeeeo 1a9a5a14.
http:://gggtaabb.cc///lll//////s

---

Interestingly, the major performance gain originates from the GCL-based source space modeling we proposed, and lesser extent, from the augmentation perspective presented by [3
Ablations on augmentation. We examined the contributions from augmentation as we vary the augmentation strength À (Figure 11. A relatively small À already yields improvement, but when the minority size is small, stronger augmentation degrades performance. This implies the augmentation gains are more likely to be originated from the exposure the diversity of synthetic augmenting, and the accuracy of augmented distribution is limited by the imperfectness of empirical estimation (e.g., limited sample size). In Figure we compare the MMD distance [36] between the parametric and nonparametric augmentation outputs and the ground-truth confirming the improved efficiency from the ECRT parametric augmentation in the low- low-sample regime.
Ablation on different model variants. We further compare the model performance with feature encoder trained with both majority and minority. (Table [used feature encoder trained only majority classes.) We have Cifar100 10P-1=51.79, TOP-5=81 .02, NLI1=2.03. This is slightly worse than the majority-only result but still outperformed other competing solutions. We caution, whether including minority examples in the pre-training of feature encoder differs case-by-case. They may not affect performance at all (majority dominant), improve performance (predictive features consistent with those used by majority) and completely devastating (containing spurious features, overfit).
Extreme classification. A related challenge of interest is extreme classification [16], where the label size is huge but there is only a handful of samples pertaining to each label class, This means there is no clear majority class. In Figure we show the proposed ECRT also fares very well in this scenario, while standard ERM struggles (slightly better than random guess). This result evidences that ECRT also efficiently extracts generalizable information from abundance of different class labels.
Imbalanced data learning. In Table D we compare the performance of ECRT to other competing solutions, with each baseline carefully tuned to ensure fairness. Minor discrepancies compared to results reported by prior literature are explained in the SM Sec. E, and in general our implementation performs slightly better. Overall, ECRT-based solutions consistently lead the performance chart, with the label-prior variant solidly outperforming vanilla ECRT and other competitors in most categories. While showing varying degrees of success, FOCAL and LDAM failed to establish dominance compared to ERM. GAN and VAT also verified the effectiveness of augmentation and adversarial perturbations in the imbalanced setting. We also compare the overall F1 Fi-score between different methods on CIFAR100 dataset: ERM: 0.439, SMOTE 11: 0.444, FOCAL:0.391 LDAM: 0,408, and ECRT: 0.482. FOCAL and LDAM gave very poor F1 scores in this case, even worse than the ERM baseline. SMOTE improved ERM, but the largest gain is obtained by ECRT. Figure [ compares performance of different learning schemes condition on different label sizes using the Fl score on the iNaturalist data. In Figure g augmentation, using tSNE embeddings.
we show the ECRT learned representations, with and without
6 Conclusion
This paper developed a novel learning scheme for imbalanced data learning. Leveraging causal per- spective, our solution cuts through the apparent data heterogeneity and identifies a shared, invariant, disentangled source representation, using only majority samples. We demonstrate this source repre- sentation modeling enables more efficient learning and allows principled augmentation. Importantly, we bridge the research between contrastive representation learning and energy modeling to causal machine learning, which constructs promising directions for future research.
Acknowledgements
This research was supported in part by NIH/NIDDK R01-DKI23062, NIH/NIBIB R01 001-EB025020, NIH/NINDS I IR61NS120246 DARPA, DOE, ONR and NSF. J. Chen was partially supported by Shanghai Municipal Science and Technology Major Project (No.201 8SHZDZX01) and National Key R&D Program of China 0.2008AAAA0000000
This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number Environment (XSEDE) PSC Bridges-2 and SDSC Expanse at the service- provider through allocation TG-ELE200002 and TG-C1S210044.
[85) This work used the Extreme Science and Engineering Discovery
10

---

References
wtt../ele..coooooaalllll------.- [1] dataset. http://////////////////////////a
[2] Tiny IMAGENET visual recognition challenge. httpps//wwww.agggle..co/////////
[3] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial
networks. arXiv preprini urXiv:1711.04340 2017.
[4] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In ICLR, 2020.
[5] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wwxxvv1907.028999 2019.
[6] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? In NeurlPS, 2018.
[7] Zdravko Botev and Dirk P Kroese, An efficient algorithm for rare-event probability estimation, combina- torial optimization, and counting. Methodology and Computing in Applied Probability, (0(4):47-5505 2008.
[8] Peter Bühlmann. Invariance, causality and robustness. arXiv preprint
2018.
[9] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning" In ICML, 2019.
[10] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datascts with label-distribution aware margin loss. In NeurIPS, 2019.
[11] Nitesh Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[12] Junya Chen, Jianfeng Feng. and Wenlian Lu. A wiener causality defined by divergence. Neural Processing Letters, 53(3):1773 1773-194777 2021.
[13] Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Yi Xu, Belinda Zeng, Wenlian Lu, et al. Simpler. faster, stronger: Breaking the log-k curse contrastive learners with flatnee. arXiv preprint orXiv:2107.0155: 2021.
[14] Junya Chen, Danni Lu. Zidi Xiu, Ke Bai, Lawrence Carin, and Chenyang Tao. Variational inference with holder bounds. arXiv preprint arXiv:2111.0294/ 2021.
[15] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020.
[16] Anna Choromanska. Alekh Agarwal, and John Langford. Extreme multi class classification. In NIPS Workshop: eXtreme Classification, submitted, 2013.
[17] Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. arXiv preprint 1rXivv1810.08811 2018.
[18] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. arXiv preprint arXiv:1511.06068. 2015.
[19] Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
[20] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-29) 2977,777799 1995.
[21] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
[22] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongic. Class- baass-blaanced loss based on effective number of samples. In CVPR, 2019.
[23] Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponential family estimation via adversarial dynamics embedding. In NeurIPS, pages 10979-10990, 2019.
[24] Andrea Dal Pozzolo, Giacomo Boracchi, Olivier Caelen, Cesare Alippi, and Gianluca Bontempi. Credit card fraud detection: realistic modeling and a novel learning strategy. IEEE transactions on neural networks and learning systems, 29(8):3784-377 3797, 2017.

---

[25] Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, et al. Natural neural networks. In NIPS, 2015,
[26] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACI,, 2019.
[27] Thomas A Dingus, Feng Guo, Suzie Lee, Jonathan F Antin, Miguel Perez, Mindy Buchanan- King, and Jonathan Hankey. Driver crash risk factors and prevalence evaluation using naturalistic driving data. Proceedings of National Academy of Sciences, 113(10):2636 2641, 2016.
[28] Monroe Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-22) 1983.
[29] Chris Drummond, Robert C Holte, al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. Workshop on learning from imbalanced datasets II, volume 11, pages 1-8, Citeseer, 2003.
[30] Charles Elkan. The foumdations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd, 2001.
[31] Li Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594 611, 2006.
[32] Wemer Fenchel. On conjugate convex functions. Canadian Journal of Muthematics, 1(1):73-77, 1949.
[33] Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdel-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: 703.003000 2017.
[34] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 15(1):119-11 139, 1997.
[35] Tan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[36] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Berhaard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.
[37] Qing Guo, Junya Chen, Dong Wang. Yuewei Yang. Xinwei Deng, Fan Li, Lawrence Carin, and Chenyang Tao. Tight mutual information estimation with contrastive Fenchel-Legendre optimization, 2021. [Available online; accessed 20-211-
[38] Michael Gutmann and Jun-ichiro Hirayama. Bregman divergence general framework estimate unnormalized statistical models. arXiv preprint arXiv:1202.3727 2012.
[39] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle unnormalized statistical models. In AISTATS, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[40] Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, pages 3018-3027, 2017.
[41] Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In WCCI, 2008
[42] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[43] Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018.
[44] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1778):177 2002.
[45] Lei Huang, Dawei Yang. Bo Lang, and Jia Deng. Decorrelated batch normalization. In CVPR, 2018
[46] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 I 1-300 2000.
[47] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
12

---

[48] Aapo Hyvarinen. Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In AISTATS, pages 859-868, 2019.
[49] Kui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 2019.
[50] Kaggle. Planet: Understanding the amazon from spacel kaggle. Consulté sur https:/www. kaggle. comichhlanee-mdderrtonddig the waaprdinnn-----mamo---oooooo wwwttooorroo-ppaceeeee (Consulté le: 2018-01-25), 2017.
[51] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yannis Kalantidis, Decoupling representation and classifier for long-tailed recognition. ICLR, 2019.
[52] Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309-314, 2018.
[53] Tlys Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In AISTATS, pages 2207-2217, 2020.
[54] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, volume 33, 2020.
[55] Jachyung Kim. Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to-minor translation. In CVPR, pages 13896-13905, 2020.
[56] Gary King and Langche Zeng. Logistic regression in rare events data. Political analysis, 9(2):137 163, 2001.
[57] Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al. Handling imbalanced datasets: A review. GESTS International Transactions on Computer Science and Engineering, 300 (0:15)25-300 36, 2006.
[58] Yann LeCun, Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, and Fu-Jic Huang. Predicting Structured Data, chapter A Tutorial on Bnergy-Basedl Learning. MIT Press, 2006.
[59] Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized dingle-stage detector. In AAAI, 2019.
[60] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[61] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in open world. In CVPR, 2019.
[62] Danni Lu, Chenyang Tao, Junya Chen, Fan Li, Feng Guo, and Lawrence Carin. Reconsidering generative objectives for counterfactual reasoning. In NewrIPS, 2020.
[63] JA Tenreiro Machado and António M Lopes. Rare and extreme events: the case of covid- 19idd-19pmm pandemic. Nonlinear Dynamics, page 1, 2020.
[64] Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi. BAGAN: Data augmentation with balancing GAN. arXiv preprint 1oXXiv:800309900 2018.
[65] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICIR, 2020.
[66] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularizationn method for supervised and semi- supervisd learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979 199999999999999 2018
[67] Mary Moya and Don R Hush. Network constraints and multi-objective optimization for one-class classification. Newral networks, 93))4633-47, 1996.
[68] Sankha Subhra Mullick, Shounak Datta, and Swagatam Das. Generative adversarial minority oversampling. In ICCV, 2019.
[69]
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In NIPS, 2017.
[70] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B (Statistical Methodology), 2016.
13

---

[71] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML. PMLR. 2019.
[72] Joshua Robinson, Stefanie Jegelka, and Suvrit Sra. Strength from weakness: Fast learning using weak supervision. In ICML, 2020.
[73] Mateo Rojas-Carulla, Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):139--1422, 2018.
[74] Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: heterogeneous data meets causality. arXiv preprint arXiv:1801.0060 06099, 2018
[75] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, 2018.
[76] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv:1206.64711 2012. 2012.
[77] Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of Big Data, 6(1 6(1):60, 2019.
[78] Narayanaswamy Siddharth, Brooks Paige, Jan- Willee Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In NIPS, 2017.
[79] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, 2016.
[80] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurlPS, volume 33, 2020.
[81] Chenyang Tao, Liqun Chen, Shuyang Dai, Junya Chen, Ke Bai, Dong Wang, Jianfeng Feng, Wenlian Lu, Georgiy V Bobashev, and Lawrence Carin. On fenchel mini-max learning. In NeurlPS, 2019.
[82] Chenyang Tao, Shuyang Dai, Liqun Chen, Ke Bai, Junya Chen, Chang Liu, Ruiyi Zhang, Georgiy Bobashev, and Lawrence Carin Duke. Variational annealing of gans: A langevin perspective. In ICMI., pages 6176-6185. PMLR, 2019.
[83] Takeshi Teshima, Issei Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mechanism transfer. In ICML, 2020.
composttionalrppreeetttttiiiooii [84] Pavel Tokmakov, Yu- Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recogmition. In ICCV, 2019.
[85] John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D Peterson, et al. Xsede: accelerating scientific discovery. Computing in science & engineering, 16(5):62-74, 2014.
[86] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018.
[87] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
[88] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, 2016.
[89] Yaqing Wang, Quanming Yao, James Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning ACM Computing Surveys, 53(3):1-35 34, 2020.
[90] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NIPS, 2017.
[91] Zidi Xiu, Chenyang Tao, Michael Gao, Connor Davis, Benjamin Goldstein, and Ricardo Henao. Variational disentanglement for rare event modeling. In AAAI, 2021.
[92] Zidi Xiu. Chenyang Tao, and Ricardo Henao. Variational learning individual survival distributions. In ACM CHIL, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please check the Supplemental Material.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to 5.1 and the SM.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A] All datasets are public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The datasets don't have any personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "o_OqpCHOyeJ", "reviewer": "Reviewer_sKzi", "summary": "The paper proposes to use an auxiliary feature space (S) via non-linear independent component anaylsis (NICA) on an existing feature space (Z), e.g., of a pre-trained deep neural network, to transfer causal information of majority classes to minority classes in the context of imbalanced (or long-tailed) classification. More specifically, the method trains an invertible neural networks Z → S using only majority samples to perform NICA, and uses this model to augment minority samples by simply permuting their S-features, thanks to the coordinate-wise independence of S. This augmentation has an effect of transferring the causal features of majority samples into the minority samples. Experimental results confirms that such techniques can indeed prevent classifiers from overly relying on non-generalizable features of minority samples.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the paper is clearly written with novel idea to tackle an important topic of imbalanced classification. I appreciate that the introduction is carefully put to articulate the messages the authors want to convey, and that the paper properly discusses the challenges and limitation of the current method, and several ideas to alleviate them. Although someone might concern that the proposed method is too complex in practice, or with increased training complexity, but I think it is OK to me in a sense that each of the components is generally well-motivated and contains many interesting idea, giving us more insights on leveraging NICA in the context of deep learning. The experimental results are also promising, compared to the state-of-the-art results of LDAM, although I slightly feel that the paper could have also covered some other recent approaches other than LDAM in their comparison to give readers a more clearer view.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not clearly states their limitations or societal impacts in their text, so I suggest the authors to add the respective paragraphs in the revision. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "iZ3KnH7s8Y", "reviewer": "Reviewer_8aQQ", "summary": "This paper consider the problem of learning in the presence of a severe class imbalance, specifically when there are more than two classes. It stipulates a causal generating mechanism to \"enable efficient knowledge transfer from the dominant classes to the under-represented counterparts,\" which hinges on the important assumption that all classes share a common set of salient features. In detail, the proposed procedure involves: (1) pre-training an encoder and predictor, (2) NICA estimation using the learned features, (3) source-space augmentation, (4) minority predictor modeling with the augmented source. Experiments on CIFAR100, iNaturalist, TinyImagenet and ArXiv show that the model ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is a lot going on in this paper, and it took a little while to wrap my head around it. One of the reasons is the organization of the paper, where the details of the proposed approach are spread over many sections. It may have been easier to understand the proposed approach if the final version was presented first, followed by further details about what didn't work as well (perhaps relegated to appendix). As is, several different versions of different aspects of the pipeline are presented in different parts of the paper, and I would like to see a consolidation the of the presentation around the best ideas.\n\nAnother issue, which is also a strength of the paper, is that this work draws on many recent ideas - normalizing flows, NICA, GCL, FDV, MI, etc. - which means that much *necessary* background introduction is relegated to passing citations. This paper is far from self-contained, and I found this particularly egregious in 3.3, which is very dense. The discussion of MAF is also confusing, where it is at times described as \"efficient\" (3.2), and others as \"costly.\" That being said, I found 3.4 to be quite helpful in clarifying the contributions.\n\nPerhaps the most critical step of the proposed approach is the pre-training of the encoder and predictor. Because all subsequent steps rely on this one (cf Assumption 3.1), if anything goes wrong at this stage (e.g. overfitting) one would expect everything else to fail. As such, I would have liked to more experimental ablations and discussion of this step. For example, it would be interesting to know how regularization at this stage affects downstream performance.\n\nOther Concerns:\n\n1) There is insufficient detail about how the baselines are tuned. For example, Focal loss has two important hyper-parameters: were they both tuned? In what range?\n\n2) The notion of discriminatively pre-training a feature extractor bears some resemblance to the recently proposed Supervised Contrastive Learning paper (https://arxiv.org/abs/2004.11362), which would have been a good further baseline, particularly if combined with a Focal loss for the classifier-learning stage.\n\n3) The results across four datasets in Table 1 are fairly convincing (with the caveat about the baseline tuning). \n\nQuestions:\n\n1) Can you comment on calibration beyond NLL? For example, have you computed ECE? Does the proposed procedure result in a consistent probability estimator?\n\n2) You suggest that auto-encoders might be used to learn features. Have you tried it? Generative feature representations strike me as being somewhat different from the discriminative ones used throughout the paper, and it seems dubious that that they would satisfy A3.1 or work with ICA.\n\n3) An important class of imbalanced problems are binary, e.g. credit card fraud, cancer detection, etc.  (In fact, this may be what comes to mind for many when talking about \"imbalanced data learning\"!) Can you comment on how the proposed method might work there?\n\n** Update after author response period ** Thanks to the authors for their replies. I remain positive about the work but keep my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DNrV8D2FVvq", "reviewer": "Reviewer_yZHA", "summary": "- This paper tackles the class imbalance problem in the perspective of knowledge transfer of the dominant classes to minority classes via data augmentation on the feature space. The proposed method first learns a feature extractor using the dominant classes. Based on these features, generalized contrastive learning is conducted with the masked auto-regressive flow to map the features into source space that the independency of each dimension is guaranteed with an invertible transformation. Then, a causal data augmentation procedure on the source space is conducted to enlarge the representation of minority classes. The empirical effectiveness of the proposed Energy-based Causal Representation Transfer (ECRT) is demonstrated on widely used imbalanced benchmarks with the comparison of strong baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros.**\n\n- **Well motivated problem and solution.** As the many real-world datasets have imbalanced class distributions, improving DNNs under class imbalance is an important and well-motivated problem. Also, data augmentation of minority classes is definitely one of the most natural solutions, and leveraging the knowledge of majority classes to conduct it has been known to be effective in the related literature.\n\n**Cons.**\n\n- **Vague source of empirical improvement and insufficient ablation study.** Figure 1 shows that there is no significant difference between empirical performances of ECRT with and without the proposed augmentation. Namely, the improvement does not come from the augmentation for minority classes itself but might come from the components which are not directly related to class imbalance, such as energy-based GCL and modeling the additional classifier in the source space themselves.\nAlso, the presented ablation study is not enough now. The independent gain from the following components should be additionally presented in the ablation study:(1) Energy-based GCL compared to the original GCL (currently, Figure S1 implicitly reveal the improvement, but a more direct comparison with test accuracy is required), (2) Additional classifier module \\phi for predicting labels from the source, (3) Restriction of feature encoder learning to the majority classes compared to the learning with all samples including minority classes, and (4) Parametric augmentation itself by comparing with non-parametric augmentation under the use of $\\tilde{\\mathcal{L}}_{\\text{GCL}}$\n- **Omitted details for reproducibility.** Many details of the proposed ECRT are currently omitted as enumerated follow; hence it is now hard to understand and reproduce the proposed method fully:\n    - According to lines 155-156, the feature encoder is only trained with the majority classes ($M >2$, i.e., $m ≤ 2$). Is this selection commonly applied to all datasets? As the many real-world imbalanced datasets have a long-tailed distribution (i.e., continuously decreased number of samples), such a hard threshold seems to be sub-optimal. Also, after the pre-training phase (step 1 in Algorithm 1), is the feature encoder never trained again? Algorithm 1 implies no fine-tuning of the encoder, but it is unnatural because the feature encoder is never trained with minority classes.\n    - In the proposed ECRT, there are two classifiers 1) $h_{\\phi^{'}}(\\boldsymbol{z})$ on features $\\boldsymbol{z}$ and 2) $h_{\\phi}(\\boldsymbol{s})$ sources $\\boldsymbol{s}$. Then, how is the prediction conducted? Just taking the average of the outputs of two classifiers? There is no description about it now.\n    - No mention about sensitivity and selected value for $\\rho$ in Equation (5).\n    - Omitted source of pre-training for image classification in lines 362.\n    - No descriptions about 1) toy model in lines 369-377 and 2) extreme classification in lines 386-390.\n- **Limited methodological and empirical contributions baselines for class imbalanced learning.** The methodological contributions of this paper can be summarized into two components: (1) modification of GCL to energy-based GCL and (2) adding regularization of data likelihood with FLOW. However, the improvement from (1) is not clearly validated, and the sensitivity of hyper-parameter $\\rho$ in Equation (5) is also not provided as previously mentioned. Additionally, the idea of casual representation transfer and augmenting minority samples are not the novel ideas of this paper but the ideas from the original papers. Also, the stronger baselines for handling class imbalance should be compared [1, 2], and more related works should also be added to related works [3, 4, 5].\n\n**Other comments.**\n\n- **Evaluation metrics.** Negative log-likelihood (NLL) is now used as one of the metrics, but its effectiveness is quite questionable. What is the behind rationale of such a choice, and how about considering other metrics for class imbalance such as Geometric Means (GM)?\n- **Visualization of class distribution.** Although the authors present the descriptions about the used datasets in lines 353-360, it would be better to visualize their class distributions to improve the readability.\n- **Inconsistency**\n    - Both CRT and ECRT terms are now used to indicate the proposed method\n    - In Table 1, there are no results about F1 score. Conversely, Figure 9 and 11 only report F1 score without any motivation of such choice. It would be better to unify the used metric, e.g., additional results of F1 score to Table 1.\n\n[1] Kang et al., Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 20 \\\\\\\n[2] Menon et al., Long-tail learning via logit adjustment., ICLR 21 \\\\\\\n[3] Tang et al., Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect., NeurIPS 20 \\\\\\\n[4] Hariharan and Girshick., Low-shot Visual Recognition by Shrinking and Hallucinating Features., ICCV 17 \\\\\\\n[5] Kim et al., M2m: Imbalanced Classification via Major-to-minor Translation., CVPR 20", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "6thLIsLZKYq", "reviewer": "Reviewer_weGp", "summary": "The paper presents a solution for imbalanced classification problem – which a well-known, but otherwise difficult machine learning problem, especially in high-dimensional space. The difficulty typically lies in the unknown nature of minority classes. Naïve methods such as resampling would easily overfit. A plausible approach would impose certain assumptions, such as the ability to ‘disentangle’ the feature representation; and utilising recent identifiability result in nonlinear ICA this paper assumed that the feature representation of a data point can be decomposed to several independent source features conditioned by the data point’s class label. Also, these source features can be effectively extracted from data by a general contrastive learning process. The author carefully designed a framework and a learning algorithm to achieve this goal.\n\nIn short, the key contribution is a combination of exploiting recent results in CMT (casual transfer mechanism, but limited to regression), generalize contrastive learning (GCL) and strategy to improve training. This is to learn from an input x to its source representation s = f(z) where z is the usual feature encoder. Now, elements of s are conditionally independent given the label. From here, enhancing minority class is fairly standard.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n-\tI like the general technical thoroughness of this paper, especially its attempts to discuss its shortcomings, relation to practical considerations. The supplementary materials have also made a reasonable effort to make the work as complete as possible.\n-\tThe exploitation of the results from GCL in [1] and CMT [66] to address the imbalanced problem is welcoming. \n-\tI can’t say for sure due to my lack of hands-on experience, but the new energy-based CGL, exploiting ratio density trick via Donsker-Varadhan seems very useful, especially when dealing with large number of classes as claimed in the paper.\n\nWeaknesses:\n\n-\tDespite empirical results reported, what missing is insight into when the proposed approach will fail? What assumption on the number of minority labels for it to work? What is the effect of correlations between label of minority classes and majority classes? (e.g., if have lots of major labels for horse, it might be easier to correct for minority class for zebra than for car) \n-\tIt is also well-know that the complexity and dimensionality of the data space is important. Having said that, a popular baseline SMOTE (Synthetic Minority Oversampling Technique), which can be applied on the latent space z, is missing. And while it is understood that CMT originally designed for regression, is it possible to twist it as baseline (since it is closest to this work) ?\n-\tFor the augmentation step, the new samples are created by randomly mixing of existing sample (in the naïve setting) or sample from a distribution. Although it is showed in figure 5 that these samples’ distribution as the ground truth distribution, there seems to be nothing to guarantee that the new created data point has the same label as the data points used to created it. Please clarify\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the imbalanced data learning issue, through the combination of two approaches:
* data augmentation based on non-linear independent representations
* causal modeling based on generalized contrastive learning.

The presented scheme is a quite complex mixture of recent ideas:
* generalized contrastive learning; 
* non-linear independent sources (NICA) 
* masked auto-regressive flow (MAF).

The approach proceeds mainly in 4 steps:
1. All non-minority classes (at least 2) are used to learn a feature encoder module $Z = e_\theta(X)$, and $Z$ supports a discriminant model for the non-minority classes;
2. NICA is used to build the sources $S = f_\psi(Z)$, using a masked auto-regressive flow, and ensuring that $f_\psi$ is invertible; its inverse $f_\psi^{-1}$ is viewed as the causal mechanism $S \rightarrow Z$; 
3. An augmentation module is defined to generate new samples in the minority class by crossing-over the independent components of examples in the minority class (or using a parametric augmentation, e.g. using a Gaussian distribution fitted from the samples), subject to the assumption below;
4. A discriminant model based on $S$ is then trained for all classes. 

The assumption done is that the causal mechanism ($Z = f_\psi^{-1}(S)$) is same for all classes. 

The contribution on the top of generalized contrastive learning is to use an energy modeling based formulation and enforcing its smoothness using a MAF.

Another insightful ingredient consists in learning in the source space as opposed to, in the representation space. 

The positioning and rationale of the approach, in supplementary material, is particularly interesting. 
Besides, the authors did an excellent and thorough job in answering the reviewers concerns (regarding the limitations of the approach, the fact that the augmented examples are not guaranteed to fall in the minority class, the reproducibility of the results, the relationship with related work, e.g. Supervised Contrastive Learning (SCL) [Khosla, et al. (2020)], the fact that Z is frozen and not adapted after the augmentation). 

While the paper contains more than one idea -- which is always dangerous -- it is the Area Chair's opinion that the material in it is sound, well thought from both theoretical and practical perspectives, and that it has the potential to inspire many NeurIPS attendees. 
Decision: Accept (Spotlight)