Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel method to construct prediction intervals with pre-specified conditional coverage probability. Starting point of the paper is the observation that the length $|\hat{C}(X)|$ and conditional coverage indicator $1\{Y \in \hat{C}(X)\}$ are independent whenever the prediction interval $\hat{C}(X)$ has exact coverage probability.  The authors therefore suggest to compute lower and upper bounds of the prediction interval by solving a quantile regression problem augmented by a penalty term that penalizes correlation between $|\hat{C}(X)|$ and $1\{Y \in \hat{C}(X)\}$. The authors compare the performance of their procedure and competing methods on simulated data and nine benchmark data sets.
main_review: I enjoyed reading this paper. It is very clearly written and makes an interesting contribution. Using quantile regression to construct prediction intervals with (asymptotically) correct conditional coverage probability is well-established. Augmenting this traditional approach with a penalty to encourage independence of length and conditional coverage indicator to improve finite sample performance is creative.

My main concern is that the key reason for the poor finite sample performance of the vanilla QR approach might not be due to the violation of the independence property, but simple because of the variability of the quantile regression estimates themselves. Instead of using classical QR, it would be interesting to see whether weighted/ efficient QR (Koenker 2005, Ch. 5.3) already leads to improved conditional coverage. By how much would enforcing the orthogonality condition further improve coverage based on weighted QR?
limitations_and_societal_impact: Yes. They discuss limitations with regard to implementation of their method.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper discusses a regularization scheme that aims to improve the conditional coverage of prediction intervals that are learned while performing quantile regression for a pair of quantiles. The empirical experiments show the effect of applying the regularization scheme by demonstrating improvement across a suite of metrics.
main_review: This paper is clearly written and well motivated. Learning accurate conditional quantiles and prediction intervals (PIs) is a relevant problem, especially to the general UQ community.

The proposed method is based on the findings presented in Proposition 1, which is applied via a regularization term as stated in equation (7). I believe these are relevant and interesting findings, and provide insights in the problem of learning accurate quantiles and PIs.

I have some questions about the method, which is stated below in the Limitations section.

limitations_and_societal_impact: I have a couple of concerns and questions regarding the method, and empirical evaluations. 

Method:

1. How does the regularization term impact the optimization procedure? I can see multiple ways in which the R(L,V)=0 can be satisfied, e.g. a very wide or narrow prediction interval which always has V=1 or V=0, or any homoscedastic prediction which always satisfies L=constant. Will the regularization term push the converged solution towards these degenerate cases? 

2. In relation to the first question, how should $\gamma$ be tuned in (7)?

3. Because of 1), I am dubious of the proposed metrics. E.g. if we only consider a class of homoscedastic models, all models in this class will have R_{corr} and R_{HSIC} = 0? 

Empirical evaluation:

4. The evaluation mainly displays how appending the regularization R(L,V) on top of other standard QR losses can improve performance. It's been observed in previous literature that vanilla QR (just optimizing the pinball loss) in the deep learning setting often results in miscalibration, and post-hoc recalibration methods can adjust the quantiles to achieve better calibration. What improvements does orthogonal QR provide over vanilla QR + recalibration? I believe conformalized QR (Section 5.3 and Table 3) is a form of post-hoc recalibration, and I'm not sure if there are significant improvements when comparing the OQR columns in Table 2 with CQR columns in Table 3.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper shows a new loss function that can be used for quantile regression. The key idea is to combine loss functions often used in quantile regression with a new penalty that enforces conditional coverage. Results show improvements according to several metrics.
main_review: The language of the paper is very clear. The ideas and math are also technically correct. Quantile regression is relevant for uncertainty quantification, and thus the contributions are important.

The key to the development of the paper (in particular of the loss function) is Proposition 1, which is neat. Although similar results have been used in other contexts, I have never seen it being used for quantile regression. The empirical results are encouraging.

Major comments:

- How is gamma (the penalty factor) chosen in practice? How sensitive are the empirical results to such a choice? I think this is a very important point that is not addressed in the paper.

- It is natural that the method will outperform others in terms of Pearson and HSIC metrics (because they are explicitly used in the loss function); the improvements with respect to the other metrics are more appealing. However, if gamma is large, wouldn't it be the case that all metrics that were computed in the experiments would be better for OQR (because they basically measure conditional coverage), even though a large gamma does not guarantee that the quantiles would be well estimated? That is, a large gamma enforces good conditional coverage, but by itself, it would not enforce the quantiles to be correctly estimated. Other estimates also satisfy conditional coverage. If this is the case, I feel that other metrics would be necessary to complement the results (such as the pinball loss).

- I also miss comparisons to other quantile estimators that don't directly target the pinball loss, such as quantile regression forests.

Minor comments:

- is the reference to interval score loss "[17]" correct? I think it should be "[15]".

- it would be useful to explain what the worst-slab coverage means
limitations_and_societal_impact: Yes, limitations have been addressed.
review_title: null

Meta-review: This paper proposes a new type of regularization for quantile regression such that the accuracy of finite sample conditional coverage is improved. The proposed regularization is built on an interesting notion about independence between the size of the intervals and the indicator of a miscoverage event. All reviewers agreed that learning accurate conditional quantiles and prediction intervals (PIs) is an important problem in the uncertainty quantification community. The authors are encouraged to consider updating the paper based on the reviewers' comments and suggestions.
Decision: Accept (Poster)