Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a solution for imbalanced classification problem – which a well-known, but otherwise difficult machine learning problem, especially in high-dimensional space. The difficulty typically lies in the unknown nature of minority classes. Naïve methods such as resampling would easily overfit. A plausible approach would impose certain assumptions, such as the ability to ‘disentangle’ the feature representation; and utilising recent identifiability result in nonlinear ICA this paper assumed that the feature representation of a data point can be decomposed to several independent source features conditioned by the data point’s class label. Also, these source features can be effectively extracted from data by a general contrastive learning process. The author carefully designed a framework and a learning algorithm to achieve this goal.

In short, the key contribution is a combination of exploiting recent results in CMT (casual transfer mechanism, but limited to regression), generalize contrastive learning (GCL) and strategy to improve training. This is to learn from an input x to its source representation s = f(z) where z is the usual feature encoder. Now, elements of s are conditionally independent given the label. From here, enhancing minority class is fairly standard.

main_review: Strength:
-	I like the general technical thoroughness of this paper, especially its attempts to discuss its shortcomings, relation to practical considerations. The supplementary materials have also made a reasonable effort to make the work as complete as possible.
-	The exploitation of the results from GCL in [1] and CMT [66] to address the imbalanced problem is welcoming. 
-	I can’t say for sure due to my lack of hands-on experience, but the new energy-based CGL, exploiting ratio density trick via Donsker-Varadhan seems very useful, especially when dealing with large number of classes as claimed in the paper.

Weaknesses:

-	Despite empirical results reported, what missing is insight into when the proposed approach will fail? What assumption on the number of minority labels for it to work? What is the effect of correlations between label of minority classes and majority classes? (e.g., if have lots of major labels for horse, it might be easier to correct for minority class for zebra than for car) 
-	It is also well-know that the complexity and dimensionality of the data space is important. Having said that, a popular baseline SMOTE (Synthetic Minority Oversampling Technique), which can be applied on the latent space z, is missing. And while it is understood that CMT originally designed for regression, is it possible to twist it as baseline (since it is closest to this work) ?
-	For the augmentation step, the new samples are created by randomly mixing of existing sample (in the naïve setting) or sample from a distribution. Although it is showed in figure 5 that these samples’ distribution as the ground truth distribution, there seems to be nothing to guarantee that the new created data point has the same label as the data points used to created it. Please clarify

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper consider the problem of learning in the presence of a severe class imbalance, specifically when there are more than two classes. It stipulates a causal generating mechanism to "enable efficient knowledge transfer from the dominant classes to the under-represented counterparts," which hinges on the important assumption that all classes share a common set of salient features. In detail, the proposed procedure involves: (1) pre-training an encoder and predictor, (2) NICA estimation using the learned features, (3) source-space augmentation, (4) minority predictor modeling with the augmented source. Experiments on CIFAR100, iNaturalist, TinyImagenet and ArXiv show that the model 
main_review: There is a lot going on in this paper, and it took a little while to wrap my head around it. One of the reasons is the organization of the paper, where the details of the proposed approach are spread over many sections. It may have been easier to understand the proposed approach if the final version was presented first, followed by further details about what didn't work as well (perhaps relegated to appendix). As is, several different versions of different aspects of the pipeline are presented in different parts of the paper, and I would like to see a consolidation the of the presentation around the best ideas.

Another issue, which is also a strength of the paper, is that this work draws on many recent ideas - normalizing flows, NICA, GCL, FDV, MI, etc. - which means that much *necessary* background introduction is relegated to passing citations. This paper is far from self-contained, and I found this particularly egregious in 3.3, which is very dense. The discussion of MAF is also confusing, where it is at times described as "efficient" (3.2), and others as "costly." That being said, I found 3.4 to be quite helpful in clarifying the contributions.

Perhaps the most critical step of the proposed approach is the pre-training of the encoder and predictor. Because all subsequent steps rely on this one (cf Assumption 3.1), if anything goes wrong at this stage (e.g. overfitting) one would expect everything else to fail. As such, I would have liked to more experimental ablations and discussion of this step. For example, it would be interesting to know how regularization at this stage affects downstream performance.

Other Concerns:

1) There is insufficient detail about how the baselines are tuned. For example, Focal loss has two important hyper-parameters: were they both tuned? In what range?

2) The notion of discriminatively pre-training a feature extractor bears some resemblance to the recently proposed Supervised Contrastive Learning paper (https://arxiv.org/abs/2004.11362), which would have been a good further baseline, particularly if combined with a Focal loss for the classifier-learning stage.

3) The results across four datasets in Table 1 are fairly convincing (with the caveat about the baseline tuning). 

Questions:

1) Can you comment on calibration beyond NLL? For example, have you computed ECE? Does the proposed procedure result in a consistent probability estimator?

2) You suggest that auto-encoders might be used to learn features. Have you tried it? Generative feature representations strike me as being somewhat different from the discriminative ones used throughout the paper, and it seems dubious that that they would satisfy A3.1 or work with ICA.

3) An important class of imbalanced problems are binary, e.g. credit card fraud, cancer detection, etc.  (In fact, this may be what comes to mind for many when talking about "imbalanced data learning"!) Can you comment on how the proposed method might work there?

** Update after author response period ** Thanks to the authors for their replies. I remain positive about the work but keep my original rating.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to use an auxiliary feature space (S) via non-linear independent component anaylsis (NICA) on an existing feature space (Z), e.g., of a pre-trained deep neural network, to transfer causal information of majority classes to minority classes in the context of imbalanced (or long-tailed) classification. More specifically, the method trains an invertible neural networks Z → S using only majority samples to perform NICA, and uses this model to augment minority samples by simply permuting their S-features, thanks to the coordinate-wise independence of S. This augmentation has an effect of transferring the causal features of majority samples into the minority samples. Experimental results confirms that such techniques can indeed prevent classifiers from overly relying on non-generalizable features of minority samples.
main_review: Overall, I found the paper is clearly written with novel idea to tackle an important topic of imbalanced classification. I appreciate that the introduction is carefully put to articulate the messages the authors want to convey, and that the paper properly discusses the challenges and limitation of the current method, and several ideas to alleviate them. Although someone might concern that the proposed method is too complex in practice, or with increased training complexity, but I think it is OK to me in a sense that each of the components is generally well-motivated and contains many interesting idea, giving us more insights on leveraging NICA in the context of deep learning. The experimental results are also promising, compared to the state-of-the-art results of LDAM, although I slightly feel that the paper could have also covered some other recent approaches other than LDAM in their comparison to give readers a more clearer view.
limitations_and_societal_impact: The paper does not clearly states their limitations or societal impacts in their text, so I suggest the authors to add the respective paragraphs in the revision.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: - This paper tackles the class imbalance problem in the perspective of knowledge transfer of the dominant classes to minority classes via data augmentation on the feature space. The proposed method first learns a feature extractor using the dominant classes. Based on these features, generalized contrastive learning is conducted with the masked auto-regressive flow to map the features into source space that the independency of each dimension is guaranteed with an invertible transformation. Then, a causal data augmentation procedure on the source space is conducted to enlarge the representation of minority classes. The empirical effectiveness of the proposed Energy-based Causal Representation Transfer (ECRT) is demonstrated on widely used imbalanced benchmarks with the comparison of strong baselines.
main_review: **Pros.**

- **Well motivated problem and solution.** As the many real-world datasets have imbalanced class distributions, improving DNNs under class imbalance is an important and well-motivated problem. Also, data augmentation of minority classes is definitely one of the most natural solutions, and leveraging the knowledge of majority classes to conduct it has been known to be effective in the related literature.

**Cons.**

- **Vague source of empirical improvement and insufficient ablation study.** Figure 1 shows that there is no significant difference between empirical performances of ECRT with and without the proposed augmentation. Namely, the improvement does not come from the augmentation for minority classes itself but might come from the components which are not directly related to class imbalance, such as energy-based GCL and modeling the additional classifier in the source space themselves.
Also, the presented ablation study is not enough now. The independent gain from the following components should be additionally presented in the ablation study:(1) Energy-based GCL compared to the original GCL (currently, Figure S1 implicitly reveal the improvement, but a more direct comparison with test accuracy is required), (2) Additional classifier module \phi for predicting labels from the source, (3) Restriction of feature encoder learning to the majority classes compared to the learning with all samples including minority classes, and (4) Parametric augmentation itself by comparing with non-parametric augmentation under the use of $\tilde{\mathcal{L}}_{\text{GCL}}$
- **Omitted details for reproducibility.** Many details of the proposed ECRT are currently omitted as enumerated follow; hence it is now hard to understand and reproduce the proposed method fully:
    - According to lines 155-156, the feature encoder is only trained with the majority classes ($M >2$, i.e., $m ≤ 2$). Is this selection commonly applied to all datasets? As the many real-world imbalanced datasets have a long-tailed distribution (i.e., continuously decreased number of samples), such a hard threshold seems to be sub-optimal. Also, after the pre-training phase (step 1 in Algorithm 1), is the feature encoder never trained again? Algorithm 1 implies no fine-tuning of the encoder, but it is unnatural because the feature encoder is never trained with minority classes.
    - In the proposed ECRT, there are two classifiers 1) $h_{\phi^{'}}(\boldsymbol{z})$ on features $\boldsymbol{z}$ and 2) $h_{\phi}(\boldsymbol{s})$ sources $\boldsymbol{s}$. Then, how is the prediction conducted? Just taking the average of the outputs of two classifiers? There is no description about it now.
    - No mention about sensitivity and selected value for $\rho$ in Equation (5).
    - Omitted source of pre-training for image classification in lines 362.
    - No descriptions about 1) toy model in lines 369-377 and 2) extreme classification in lines 386-390.
- **Limited methodological and empirical contributions baselines for class imbalanced learning.** The methodological contributions of this paper can be summarized into two components: (1) modification of GCL to energy-based GCL and (2) adding regularization of data likelihood with FLOW. However, the improvement from (1) is not clearly validated, and the sensitivity of hyper-parameter $\rho$ in Equation (5) is also not provided as previously mentioned. Additionally, the idea of casual representation transfer and augmenting minority samples are not the novel ideas of this paper but the ideas from the original papers. Also, the stronger baselines for handling class imbalance should be compared [1, 2], and more related works should also be added to related works [3, 4, 5].

**Other comments.**

- **Evaluation metrics.** Negative log-likelihood (NLL) is now used as one of the metrics, but its effectiveness is quite questionable. What is the behind rationale of such a choice, and how about considering other metrics for class imbalance such as Geometric Means (GM)?
- **Visualization of class distribution.** Although the authors present the descriptions about the used datasets in lines 353-360, it would be better to visualize their class distributions to improve the readability.
- **Inconsistency**
    - Both CRT and ECRT terms are now used to indicate the proposed method
    - In Table 1, there are no results about F1 score. Conversely, Figure 9 and 11 only report F1 score without any motivation of such choice. It would be better to unify the used metric, e.g., additional results of F1 score to Table 1.

[1] Kang et al., Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 20 \\\
[2] Menon et al., Long-tail learning via logit adjustment., ICLR 21 \\\
[3] Tang et al., Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect., NeurIPS 20 \\\
[4] Hariharan and Girshick., Low-shot Visual Recognition by Shrinking and Hallucinating Features., ICCV 17 \\\
[5] Kim et al., M2m: Imbalanced Classification via Major-to-minor Translation., CVPR 20
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The paper tackles the imbalanced data learning issue, through the combination of two approaches:
* data augmentation based on non-linear independent representations
* causal modeling based on generalized contrastive learning.

The presented scheme is a quite complex mixture of recent ideas:
* generalized contrastive learning; 
* non-linear independent sources (NICA) 
* masked auto-regressive flow (MAF).

The approach proceeds mainly in 4 steps:
1. All non-minority classes (at least 2) are used to learn a feature encoder module $Z = e_\theta(X)$, and $Z$ supports a discriminant model for the non-minority classes;
2. NICA is used to build the sources $S = f_\psi(Z)$, using a masked auto-regressive flow, and ensuring that $f_\psi$ is invertible; its inverse $f_\psi^{-1}$ is viewed as the causal mechanism $S \rightarrow Z$; 
3. An augmentation module is defined to generate new samples in the minority class by crossing-over the independent components of examples in the minority class (or using a parametric augmentation, e.g. using a Gaussian distribution fitted from the samples), subject to the assumption below;
4. A discriminant model based on $S$ is then trained for all classes. 

The assumption done is that the causal mechanism ($Z = f_\psi^{-1}(S)$) is same for all classes. 

The contribution on the top of generalized contrastive learning is to use an energy modeling based formulation and enforcing its smoothness using a MAF.

Another insightful ingredient consists in learning in the source space as opposed to, in the representation space. 

The positioning and rationale of the approach, in supplementary material, is particularly interesting. 
Besides, the authors did an excellent and thorough job in answering the reviewers concerns (regarding the limitations of the approach, the fact that the augmented examples are not guaranteed to fall in the minority class, the reproducibility of the results, the relationship with related work, e.g. Supervised Contrastive Learning (SCL) [Khosla, et al. (2020)], the fact that Z is frozen and not adapted after the augmentation). 

While the paper contains more than one idea -- which is always dangerous -- it is the Area Chair's opinion that the material in it is sound, well thought from both theoretical and practical perspectives, and that it has the potential to inspire many NeurIPS attendees. 
Decision: Accept (Spotlight)