Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper looks at uncertainty estimation for regression on imbalanced datasets. It proposes a new method called VIR which combines variational inference, smoothed statistics, and conjugate distribution parametrization. In the experiments, VIR is evaluated on 2 imbalanced datasets on accuracy and calibration metrics.
strength_and_weaknesses: Pros:

1. The task of uncertainty estimation for regression on imbalanced dataset is important.
2. The idea of combining smoothed statistics to fix imbalanced dataset and produce uncertainty estimates via parametrizing conjugate distributions is interesting.
3. The method achieves great improvement on 2 datasets.

##########################################################################

Cons:

- Related works: The paper misses many important related works for uncertainty estimation for regression like [1, 2, 3, 4, 5, 6, 7] but also many others. This includes very common baselines like Dropout [6] and Ensemble [1] which work for regression. The paper also does not discuss GP-based methods [2, 3] which can be used for regression. Further, the paper does not discuss Posterior Networks methods [4, 5, 7]. More specifically, [4, 5] shares many similarities with VIR. They use conjugate distributions, pseudo-count interpretations, posterior updates, and variational losses which are key parts of VIR. NatPN is also designed to work for regression. Action suggestion: discuss all of these methods in the related work section.
- Desiderata: The two desiderata are sometimes not clear. It feels that the two desiderata are high quality *uncertainty estimation* and high performance on *imbalanced dataset*. However the paper mentions that the desiderata are “performance improvement and uncertainty estimation”. The paper also phrases the identification of the desiderata of imbalanced dataset and uncertainty estimation as a contribution.  However people already looked at imbalanced datasets and uncertainty estimation separately in previous works. Finally, the paper does not discuss existing desiderata for uncertainty estimation [8, 7, 9]. Action suggestion: I would recommend to change the phrasing of this contribution and have a discussion on existing desiderata in uncertainty estimation.
- Loss: I felt that the description of the loss was sometimes confusing. I would be interested in the derivation fo the loss. This could also be provided in the appendix. In eq.(1), what is p_\theta(z_i) ? Does this ELBO loss assume any prior ? What is the importance of the regularization loss in the total loss ? Action suggestion: Beyond the answer to these questions, it would be intresting to show the important of the regularization term in an experiment.
- Clarity: The paper is sometimes hard to read. E.g. the paper mentions multiple times “see below” which leaves unclear where to exactly find the missing information. Action suggestion: make explicit reference when pointing to further details.
- Bins: It is unclear what is the sensitivity of the method to the number of bins. It was also unclear to me when readin sec. 3.1. what data are used to build the mean representation from the bins. Action suggestion: I would recommend to clarify the bins construction since it is an essential part of the method. I would also be interested in an experiment on the number of bins.
- Experiences: The experiments do not look very extensive to me. They consider only two datasets and a single backbone network. They do not compare to DropOut, Ensemble, GP, or NatPN which would be appropriate baselines. They do not look at common uncertainty estimation metrics like OOD detection scores. They also do not provide error bars which are key to assess the significance of the results. Action suggestion: I would recommend to add at least one dataset (e.g. depth estimation) with another backbone architecture. I would also recommend to add at least Ensemble which is a common and powerful baseline and NatPN which shares many similarities with VIR. I would also recommend to add error bars.

I am happy to improve my score if a majority of the above points are addresses (e.g. with the action suggestions).

[1] Simple and scalable predictive uncertainty estimation using deep ensembles, NeurIPS 2017

[2] On feature collapse and deep kernel learning for single forward pass uncertainty.

[3] Simple and principled uncertainty estimation with deterministic deep learning via distanceawareness, NeurIPS 2020.

[4] Posterior Network: uncertainty estimation without ood samples via density-based pseudo-counts. NeurIPS 2020

[5] Natural Posterior Network: deep bayesian uncertainty for exponential family distributions, ICLR 2022

[6] Dropout as a bayesian approximation: representing model uncertainty in deep learning, ICML 2016

[7] Graph Posterior Network: bayesian predictive uncertainty for node classification. NeurIPS 2021.

[8] Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift, NeurIPS 2019.

[9] NOMU: Neural Optimization-based Model Uncertainty. ICML 2022
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear even if it is sometimes hard to read with quite dense mathematical notations (e.g. eq. 2, 3, 4) and some unclear cross references (i.e. "see below" statements). The paper combines existing techniques (smoothed statistics and conjugate distribution parametrization)  to solve the important problem of prediction on imbalanced datasets. However, it misses many important related works which would allow to more correctly estimate the novelty of the paper for the reader.
summary_of_the_review: Overall, I vote for strong reject. The task is important and well motivated and the . My major concerns are about the related work, and the experiences (see cons beow). Hopefully the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel approach named variational imbalanced regression for the probabilistic deep imbalanced regression problem. It leverages data with similar regression labels to compute the latent representation’s variational distribution, and predicts the entire normal-inverse-gamma distributions and modulates the associated conjugate distributions to impose probabilistic reweighting on the imbalanced data.
strength_and_weaknesses: Strength

- The problem studied in this paper is interesting and valuable. Both balanced accuracy and uncertainty estimation are important topics in regression.
- The paper is well-organized and clearly written, which is easy to follow.
- This paper provides some novel perspectives for probabilistic deep imbalanced regression. Leveraging Neighboring and Identically Distributed data is a valuable attempt for handling imbalance and uncertainty issues. 

Weaknesses

- In Section 3, the authors declare some limitations of LDS and DER. However, the descriptions are somewhat confusing. In my opinion, the authors should provide theoretical perspectives or specific cases for the limitations, thus illustrating the advantage of VIR.
- Since the I.I.D. assumption has been replaced with the N.I.D. assumption, some theoretical guarantees could be provided to further demonstrate the effectiveness of the algorithm.
- The impact of $\lambda$ is unclear. Further ablation studies should be conducted to illustrate the impact of $\lambda$.
- One can observe in Table 4 that the uncertainty estimation performance of the proposed VIR is worse than LDS+FDS+DER on Medium labels. The authors should further analyze the reasons for this phenomenon.

clarity,_quality,_novelty_and_reproducibility: The paper is somewhat clear, but some details are missing. This paper is technically reasonable.
summary_of_the_review: This paper provides some new perspectives for imbalanced regression. However, I have concerns about whether this work can be accepted in its current form. I will update my reviews based on the authors' responses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a probabilistic deep learning model, namely variational imbalanced regression (VIR), when the distribution of outcome is imbalanced. This method uses data with similar outcomes to estimate the distribution of the latent representation, provides a point estimate of the outcome, and provides uncertainty quantification of the point estimate. The method can work well for the minority group in the data. 
strength_and_weaknesses: Strength: 

This method uses a probabilistic representation approach that can quantify the uncertainty in the estimated prediction model. In addition, the method can potentially work well for the minority group in the data. 


Weakness:

I think the paper can benefit from providing more practical guidance on implementing the proposed method. For example, how to select the number of bins B in Section 3.1? Why is it a good idea to partition the label space $\mathcal{Y}$ into equal-interval bins, but not equal-size bins (with the same number of data points)? How to select the bandwidth in the kernel function $k(\cdot)$ to compute the neighboring data and smoothed statistics in Section 3.3? How does the bandwidth vary with the sample size N?

In addition, can the authors provide some theoretical guarantee (e.g., unbiasedness, consistency) on the proposed method? 
clarity,_quality,_novelty_and_reproducibility: This paper is well written. The method is well-motivated. The experimental results well support the proposed method.
summary_of_the_review: This paper provides a solution to address the problem of probabilistic deep imbalanced regression. The proposed method can potentially accurately estimate the outcome and quantify the uncertainty in the estimation. I think the paper can benefit from providing more practical guidance on using the proposed method, and providing some theoretical guarantee on the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper simultaneously addresses the label imbalance problem and uncertainty qualification capability in regression.
The authors propose to enhance the reweighting technique dealing with the imbalance problem in (Yang et al., 2021) to be applicable to VAE and combine the output distribution and the corresponding loss in (Amini et al., 2020), which provides uncertainty qualification capability.
Experimental results on several real-world datasets demonstrate that the proposed method performs better than state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation.
strength_and_weaknesses: *Strength
- The authors simultaneously address the label imbalance problem and uncertainty qualification capability in regression, which is a novel problem setting.
- The non-trivial combination of two different approaches is proposed and works well.

*Weaknesses
- The intuition that each of the components of the proposed method works for their purpose is not described well in the main text. For example, there is no overall procedure for their reweighting technique, which is provided in the paper (Yang et al., 2021) though.
- In Section 3.3, the authors proposed to use the covariance of z for probabilistic overall covariance. However, it is unclear why they do not use the covariance of z^{mu} and z^{sigma} instead.
- There is no intuition why the proposed approach outperformed DIR, which could be more practical in terms of prediction performance in some cases, in my understanding.
clarity,_quality,_novelty_and_reproducibility: *Clarity
- Are there any drawbacks of computational cost since the proposed method requires to use of multiple x even when testing?
- Why IID assumption harms performance for data with minority labels?
- There is no definition for Omega in Eq. 6.
- Section 3.4 is hard to follow. For example, there is no definition for the parameters of NIG at the point of Eq. 6.
- What is the difference between LDS + FDS + DER  and the proposed method?
- Tables 5 and 6 are confusing. There is no result of "eoncoder only" in Table 6. Why?

*Quality
- Please see the above comments.

*Novelty
- The proposed method seems to be novel.

*Reproducibility
- Code is not available, and the description of the proposed method is also incomplete.

summary_of_the_review: Although the proposed method is a good combination of the two SOTA methods and performs well, the description of the proposed method does not seem self-complete.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a probabilistic model called Variational Imbalanced Regression (VIR) to deal with the problem of imbalanced regression. VIR borrows data with similar regression labels to compute the latent representation’s variational distribution. VIR first encodes a data point into a probabilistic representation and then mixes it with neighboring representations. The proposed method not only performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct. 
strength_and_weaknesses: Strength: 

(1) The authors introduce the setting of uncertainty estimation to the deep imbalanced regression. I appreciate this contribution since DIR was proposed very recently by Yang et al. (ICML 2021), and this can be a good contribution to the future investigation of DIR.

(2) The method is interesting, especially for the reweighting scheme of the balanced predictive distribution. 

Weakness:

(1) Considering the novelty, I think VIR is more like a combination of [1] and [2] from the current manuscript. Though VIR is empirically superior and introduces uncertainty estimation as a byproduct, it is still vague how VIR is motivated compared to the FDS in [1], since both methods have kernel estimation as the core contribution. And the idea of extending deterministic space to a probabilistic one is interesting but lacks more motivation. 

(2) Missing related work [3, 4]. Both were published before the submission deadline and the authors should discuss the difference and compare VIR with them.

(3) The experimental setup can be improved. First, the authors only choose two age estimation datasets whose distributions are similar. Second, the reported numbers are not very convincing (please see the next section for quality and reproducibility). 

(4) I’m not fully convinced by the choice of Gaussian distribution as the prior distribution. For some specific datasets with regression labels like ages, this may possibly be a suitable assumption. But the authors claim the contributions to the task of general deep imbalanced regression, so I encourage the authors to conduct experiments on other types of regression datasets. It will strongly improve the contribution of this work if the authors can consider more datasets proposed in [1] or [3]. 

[1] Yang et al.,  Delving into Deep Imbalanced Regression. ICML 2021 \
[2] Amini et al., Deep Evidential Regression. NeurIPS 2020 \
[3] Ren et al., Balanced MSE for Imbalanced Visual Regression, CVPR 2022 \
[4] Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022 \



clarity,_quality,_novelty_and_reproducibility: 
Clarity: 

[+] the paper is well-structured and easy to follow. 

Novelty: 

[+] it is essential for regression tasks to consider uncertainty estimation. 

[-] the overall method is more like a combination of FDS [1] and DER [2] to me in the current presentation. It could be better motivated why probabilistic latent code is better than FDS to improve deep imbalanced regression.

Quality: 

I am highly concerned about the experimental setup -- 1) fair comparison between VIR and other SOTAs. Reproducing worse SOTA results without error bars is not a good practice; 2) more datasets should be considered in order to claim the method as the contribution in general DIR; 3) no discussion or empirical comparison to the missing related work [3, 4]. 

[-] The SOTA results are much worse than the ones reported in the original paper [1]. For example, in Table 1, LDS+FDS: *MAE all* from original 7.55 to reported 7.82, *GM all* from original 4.72 to reported 5.01. Also in Table 2, LDS+FDS: *MAE all* from 7.78 to 8.08, *GM all* from 4.37 to 4.66. Did the authors reproduce all results? If the SOTA is not robust and the variance between different runs is high, please consider reporting the error bar for both methods (SOTA and VIR). It is not a good practice to replace the prior work results with single runs.
 
[-] I’m concerned about the conditional log-likelihood $\mathcal{L}^D$ in Eq. (1). Did the authors have any ablation study on the weighting parameter $\lambda$? The overall task is discriminative and evaluated on the accuracy of predicting the regression label. The weighting of this conditional log-likelihood term should be discussed. This term can dominate the objective since the authors use image-based age estimation datasets with high-dimensional $x$. 


[1] Yang et al.,  Delving into Deep Imbalanced Regression. ICML 2021 \
[2] Amini et al., Deep Evidential Regression. NeurIPS 2020 \
[3] Ren et al., Balanced MSE for Imbalanced Visual Regression, CVPR 2022 \
[4] Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022 \

summary_of_the_review: I like the idea of extending conventional deterministic encoding to be probabilistic for deep imbalanced regression based on the formulation of variational autoencoders. The presentation of the idea is clear and technically sound. However, the experimental evaluation needs to be improved to make the method more convincing. Therefore, I recommend rejection at this moment but encourage the authors to improve and submit to future venues.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has received mixed reviews, with three reviewers tending to accept (6-6-6) and two reviewers proposing to reject (3-5). The reviewers agree that the paper's contribution is valuable and the problem it addresses is important. Most reviewers have found the paper to be well-written, with clarifications added during the discussion. The most important concern expressed by the reviewers is the experimental validation of the proposed VIR method. Specifically, VIR was assessed on only two age estimation datasets with similar distributions. Preliminary results were obtained on STS-B-DIR during the rebuttal, but the validation would be more convincing and fair against baselines if it were performed on a larger number of datasets, including e.g NYUD2-DIR and SHHS-DIR (as in Yang et al, 2021). Additionally, the reviewers have requested that the authors provide empirical results against missing related work and include error bars in their results. Despite the strong engagement of the authors and the many improvements made during the discussion, I believe that these concerns remain and make the paper not entirely ready for publication. 
justification_for_why_not_higher_score: Weaknesses in the experimental validation.
Decision: Reject