Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a neural network approach for quantile regression on censored data, based on Portnoys's estimator. The authors also proposed a new algorithm to mitigate the computational challenge when combining neural networks and the existing sequential grid algorithm for Portnoys's estimator. In addition, the authors made the connection between the proposed CQRNN algorithm and the EM algorithm for further understanding and compared CQRNN with existing methods on both synthetic and real-world datasets.
strengths_and_weaknesses: Strengths:
* The proposed method is not just replacing the linear model with a neural network in the existing algorithm for Portnoy's estimator. They did identify new computational challenges when optimizing a neural network through the existing sequential grid algorithm, which was less problematic for linear models, and proposed a new CQRNN algorithm that achieves faster speeds and saves memory (while the improvement mainly depends on the size of the quantile grid).
* The connection to the EM algorithm under certain distribution assumptions and the self-correcting property are very interesting. I have a related question below.
* The authors provided solid numerical evaluations and comparisons with existing approaches both qualitatively and quantitatively, and in terms of both prediction accuracy and computation efficiency. 


Weaknesses:
* It would be worthwhile to add more discussion in "related work" so as to explicitly highlight the improvement or limitation in comparison to other existing methods.
* For partial maximization, the authors mentioned the alternative version that is closer to the standard EM algorithm. While in each iteration the proposed algorithm saves computational costs by taking a single gradient step, the number of iterations required for convergence might also change or even increase. Do the authors have any comments on this or have the authors investigated the impact of taking a single gradient instead of multiple gradient descents on the overall computing time/convergence?
* From Table 2, the censoring distribution in LogNorm (and Norm uniform, etc.) in synthetic datasets seems to be independent of covariates. In this case, the assumption in DeepQuantReg also holds while CQRNN still performs better than DeepQuantReg. Can the authors explain this? 

questions: Please see the weaknesses above.
limitations: Yes, the authors discussed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a neural network based approach for doing quantile regression on possibly censored data, i.e. when the target variable is sometimes not directly observed and instead a lower or upper bound is known, as in the case of survival data.
The authors first build on the linear approach on Portnoy 2003 and extend it straightforwardly to neural networks. However this approach requires the sequential optimization of a new NN for each quantile level to be predicted, which is computationally heavy.
Then a new method is proposed for simultaneous quantile regression , which is interpreted as a form of expectation maximization, allowing to estimate all the quantiles simultaneously.

strengths_and_weaknesses: Strengths:
- The paper is clearly written and the background on censored data and survival analysis is properly explained.
- The analysis gives interesting insights and useful guarantees on the algorithm
- The experiments are interesting and appropriate.

Weakness:
- Lacks a discussion on the recent literature on simultaneous quantile regression with NN


 

questions: I think it would be interesting to discuss and relate this work to the recent literature on simultaneous quantile regression with NN that alleviates or avoids the crossing quantile problem. In particular, it would be interesting to discuss how these methods could be adapted to censored data using your construction.

A non-exhaustive list:
- *Tagasovska, N., & Lopez-Paz, D. (2019). Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems, 32.*
- *Zhou, F., Wang, J., & Feng, X. (2020). Non-crossing quantile regression for distributional reinforcement learning. Advances in Neural Information Processing Systems, 33, 15909-15919.*
- *Brando, A., Center, B. S., Rodriguez-Serrano, J., & Vitria, J. (2022, May). Deep Non-Crossing Quantiles through the Partial Derivative. In International Conference on Artificial Intelligence and Statistics (pp. 7902-7914). PMLR.*

Minor comments:
Algorithm 1: since $\tau \leftarrow grid_\tau[i-1]$,  $\hat{\boldsymbol{q}}[K]$ and $\hat{\boldsymbol{q}}[\neg K_{cross}]$ are assigned the same value, is this correct?

Algorithm X, Equation X, Appendix X,… should start with a capital letter

To avoid confusion, I think it is better to say quantile level for $q_j$

Lines 187-191: $\bar{w}_j$ is the true weight, $\hat{w}_j$ is the estimate, what is $w_j$ then?

Line 324: which were the methods use to combat the crossing-quantile problem?


limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the task of quantile regression in a survival analysis context, i.e. one where some examples are right-censored. As in almost all recent works, the encoder backbone is a neural network. The authors start by adapting the sequential grid (SG) algorithm to work with neural networks, as opposed to strictly linear models. Noting the inneficiency of the adaptation, they propose CQRNN, a novel algorithm which is more efficient. CQRNN departs from SG in that the assignment weights are updated during training.
In addition to introducing their proposed algorithm and presenting experimental results jusitying it's adoption, the authors provide a qualitative discussion of their algorithm. In particular, they frame it as an Expectation-Maximization algorithm, and provide a proof sketch as to why their proposed solution should result in "sensible" weight allocations.
strengths_and_weaknesses: I. Strenghts
- The paper is well-written, and reasonably clear. The discussion is easy to follow, and the relevant prior works necessary to understand the novel approach are made explicit.
- The algorithm is clearly presented (good explanations + an explicit algorithm).
- Experiments are well detailed: hyperparameter optimization of the competing methods was done (to a certain extent).

II. Weaknesses.
- As the authors note, the performance gains over vanilla sequential grid are minor (I am referring to table 1). In particular, it is hard to tell whether they are statistically significant without error bars or similar information. As the difference between the two is a major point (otherwise the contribution is incremental), the question I ask in the next section has the goal to clarify this point.
questions: - Could the authors re-do the comparison in table 1? I would like to confirm whether the purported advantage for their method is statistically significant.
- Would it make sense to adapt classical survival models (the authors cite a few in their work) to this context by simply changing the target to be the quantile center? I am curious what would be the relative performance of these approaches: quite a few are very strong in their original field.

Edit: The authors have responded to my questions. Taking this into account, I am raising my score.
limitations: I don't see any potential negative societal impact for this work. If anything, survival analysis is a field that has the potential to *positively* impact medicine.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors use neural networks to replace the linear models in censored quantile regression. And the authors further improve the sequential grid algorithm with bootstrap weights. The authors explain the soundness of their method from an analogy of EM. The authors compare their method with three baselines on synthetic and real dataset.
strengths_and_weaknesses: Strengths:
1. The method is novel. Introducing nn into quantile regression and the new bootstrap algorithm are both contributions to the society.
2. The authors make some effort to explain why and how the bootstrap algorithm CQRNN works.
3. The writing is clear. It's easy for me to follow the paper.

Weakness:
1. I appreciate that the authors explain how CQRNN work from an EM analogy and self-correcting property. But they still sound vague to me. Can we have a simpler example to understand it in a more rigorous way? For example, if the model is linear, how the convergence property is like using the CQRNN bootstrap algorithm? 
2. The empirical comparison does not seem to be thorough if the authors want to prove that their model is a good "survival model".
- On synthetic datasets, the authors only compare the performance only on several $\tau$'s (0.1, 0.5, 0.9). But for a survival model, why do we only care about certain $\tau$'s? How do we get values at other $\tau$'s that the quantile regression has not modeled? For a survival model, quantile at certain $\tau$'s is not the only thing we care about. We may need to look at the whole survival distribution. And then we compare other scores, for example, Brier score.
- The authors only consider lognormal model as their baseline and the reason is that they believe "this distribution often suits properties of real-world time-to-event survival data." Why do you believe so? As far as I know, lognormal models have strict distributional assumptions and may not have good performance. It is also hard to train due to the exp transformation from normal. Other models like DeepHit, SODEN may have better performance on concordance or calibration.

questions: 1. Can we get a more concrete guarantee on the convergence of the propose algorithm? Some guarantee when the model is linear should also be good.
2. Quantile regression seems to give us quantiles for certain $\tau$'s . But for a valid survival model, we need the whole distribution. How do we get the whole distribution? Or why do we not need to whole distribution in survival modeling? In Portnoy (2003), $\tau$ seems to be part of the function so we do not infinite amount of $\tau$'s. But in the authors' setting, for a different $\tau$, we need a different model. This requires the infinite amount of $\tau$'s to get the whole distribution.
3. Why do authors choose lognormal model as the baseline for survival modeling?
limitations: One extra limitation seems to be that CQRNN could not get the full survival distribution.
review_title: null

Meta-review: metareview: This paper studies the quantile regression of censored data. Neural network models are used as the statistical model. Numerical results show the proposed algorithm is computationally efficient and attains high prediction accuracy compared to existing methods. Since reviewers agree that this paper is well written and interesting, I recommend accepting the paper.
confidence: Certain
Decision: Accept