Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work is after finding proper scoring rules in survival analysis. They have a parameter vector w whose true specification underpins the proofs their provide for the discussed scoring rules being proper. They approximate the parameter vector w using an EM algorithm which can in turn be plugged into their scoring rule.
strength_and_weaknesses: It is not obvious from the paper how explanatory covariates can be integrated. The parameter vector which is central in the correctness of the deduced proofs is distribution dependent. For each covariate combination, the values for w changes. Therefore, I cannot think of applying what this paper proposes in a survival analysis application in which the sample is not thought to be drawn from a homogeneous population.

The experiments are not shedding any light on what I mentioned above. Table 1 has several metrics for which several loss functions have been tried but the author is left guessing what to take out of it. Since this paper is concerned with survival analysis, it would have been better to show that the metrics that this paper proposes to be proper, when minimized, prove to be better than those normally minimized for which such guarantees do not exist.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear. However, given the parameter set w is so central in this paper I would have liked it explained better. The first encounter with the parameter w is after equation 2 and I find the description quite terse. 
summary_of_the_review: I have doubts about the applicability of this work. It is not obvious to me how this work can be applied to a real-world scenario where explanatory covariates exist and the population from which the data is sampled is not homogeneous. The experiments did not seem to counter my current understanding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies how to use proper scoring rules to solve the survival analysis problem, where we want to estimate the distribution F(t) of a random variable T using possibly censored data, which are samples of Z = min{T, C} and whether Z = T or C. While any proper scoring rule can be used as a loss function to train a model (e.g., a neural network) to estimate a distribution using uncensored data, to train on censored data the original proper scoring rule needs to be adapted. The authors adapt four proper scoring rules (Pinball, Logarithmic, Brier, and Ranked Probability Score), and theoretically prove that these adapted socring rules are proper, under the assumption that a parameter w (that depends on the distribution F(t)) is known.

In practice, however, the parameter w is unknown, so the authors use EM and CQRNN algorithms to choose w and train the model together. Experiments show that the adapted logarithmic scoring rule is the best.
strength_and_weaknesses: [Strength]

Survival analysis is definitely an important problem to study, and the use of proper scoring rules to solve this problem is also well motivated. The authors prove that the proper scoring rules adapted to the survival problem are proper under some condition. This is an interesting result.

[Weakness]

However, I can't really tell the novelty of this work. Just as the authors mention, many works like (Portnoy, 2003), (Pearce et al, 2022) already proposed to use proper scoring rules to solve the survival analysis problem. Although the authors claim to be the first to prove that Portnoy's estimator (which is adapted from Pinball loss) is proper, this conclusion is under the strong assumption that the parameter w (which depends on the unknown distribution F(t)) is known. Under this assumption, the conclusion that Portnoy's estimator (and other three adapted scores) is proper is not surprising. It seems to me that the most challenging part of using scoring rules to solve survival analysis is how to estimate the parameter w, instead of proving properness. The methods to estimate w in this paper, however, basically follow from previous works.

The relationship with previous works is unclear. Introduction says that Rindt et al (2022) show that variants of Brier score and ranked probability score are not proper, but the variants in this paper are claimed to be proper (if w is known). Are they different variants? Also, did Portnoy (2003) or Pearce et al (2022) already prove that Portnoy's estimator is proper if we know w?


[Feedback, not weakness]

It might help to include some results in Appendix B (additional experiments) in the body, in particular, the comparison between the proper scoring rules in this paper and the custom loss functions in other state-of-the-art neural networks. One motivation for the authors to study properness of scoring rules in the survival analysis problem is because the scoring rules are used as loss functions in training neural networks, where properness is desired. So, the empirical comparison between proper scoring rules and other custom loss functions is worth including.
clarity,_quality,_novelty_and_reproducibility: [Clarity]

The language part is generally clear. But some math part is not clear:

1. Introduction, "Brier score" item: "show that neither of them are not proper in terms of the theory of scoring rules" --> "neither of them is proper"
  
2. Related Work: "Since we do not require the theory of scoring rules under this assumption" --> "Since the theory of scoring rules does not need this assumption".
  
3. Assumption 3.1: What is feature X? Does X affect the distribution of T? Namely, is the CDF of T still equal to F(t) after conditioning on X? (I think it is, according to your proof of Theorem 4.5). Also, does X affect the distribution of C? If X does not affect the distributions of T and C then why do you define X?
  
4. Page 4: "Therefore, the inequality $S(\hat F_1, y) < S(\hat F_2, y)$ means that ...". Should be the expectation of S.
  
5. Equation (4): should "$w$" be "$w_i$" because it depends on i. Also, what if $\zeta_{i+1}$ is very large, so that $F(\zeta_{i+1})$ is close to $F(c)$ and $w_i$ is not close to 0?
  
6. Equation (6): $f_i$ -> $\hat f_i$
  
7. Equation (9): should "1" be "$\hat F(\zeta)$"?
  

[Novelty]

As written in [Weakness], I cannot access the novelty of this paper due to lack of comparison with previous works.
summary_of_the_review: I currently recommend weak reject due to the lack of novelty and clarity. But if the authors can provide more contexts, like comparisons with previous works, to demonstrate the novelty of their works, I might change my opinion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose four extensions of training objectives in survival analysis. Under some conditions, these metrics are strictly proper scoring rules. The main contributions are four new strictly proper objectives.
strength_and_weaknesses: Strength:
1. Strictly proper scoring rules are important for both training and evaluation in survival analysis. Due to the existence of censoring, strictly proper scoring rules are hard to find. The common training objectives are usually not strictly proper. The authors tackle this difficulty and propose four new extensions of existing objectives. 

2. The authors also propose another evaluation metric for calibration based on the KL-divergence between Kaplan-Meier estimator and the proposed estimation. 

Weakness:
1. The authors propose four strictly proper scoring rules under the condition parameters w are correct. However, unless we know the true distribution. But the true distribution is never known. So the authors need to discuss how hard we can get w in practice. And if we cannot get the w correctly, is using wrongly estimated w based "strictly proper scoring rule" still better than using those non-strictly proper ones?

2. In experiments, the authors only compare four new objectives. The authors have not compare with training with the non-proper version of the proposed algorithm. To make the things easier to compare with, the authors can use synthetic distributions, in which they can get the true distribution/true parameters w in the objectives, and the authors test the performance using the strictly proper scoring without estimation loss from w. The authors should also include some common training objectives, for example, likelihood during the comparison.

3. The authors should consider some other common evaluation metrics, for example, concordance. Though concordance is not a proper scoring rule, but it has a practical meaning in survival analysis, i.e., the patient with high risk should be estimated with high risk. This is helpful when allocating medical resources. 

4. Closely related to the first point. Are the proposed objectives really helpful in training deep models? Deep models may not find the global optima in optimization. Finding good local optima is more important. So are "strictly proper scoring rules" really an important point? Will non-proper rule be better for optimization. The authors do not need to answer this in details. This is a hard topic.


Minor:
1. In introduction, bullet point Brier score, "show that neither of them are not proper in terms of the theory of scoring rules." I think the authors may want only one negation here. maybe remove "not".
2. I think when the authors define four new objectives, they probably need to define with the expectation. For example, the expectation of the Portnoy’s estimator is strictly proper but not the estimator itself.
clarity,_quality,_novelty_and_reproducibility: I can understand the paper easily.  The contribution is novel as far as I know. 
summary_of_the_review: I think the authors deal with an interesting and important topic: proper scoring rules in survival analysis. They also made some contribution to propose new ones. But I think more evaluations need to be done as mentioned in weakness. I am happy to raise my score if the authors can address my point both theoretically and empirically in the weakness part.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is a theoretical study of scoring rules for survival analysis, including some empirical work.  The consider standard scoring rules such as logarithmic and Brier, and adaptations for survival analysis.  The reviewers pointed out a number of issues and the authors addressed quite a few of these.

The contribution and strength of the paper is to present extensions to the standard scoring rules and does some evaluations.  The paper gives a very nice contribution to statistical theory.

The review raised a number of issues with the paper and the authors clarified some of these but its not clear to what degree this prompted appropriate rewrites of the paper.   This is specialised work and would be excellent for a conference like AI & Stats.
justification_for_why_not_higher_score: The paper is borderline.  The work is very specialised and its not clear at all it belongs in ICLR.  Its a very nice contribution to statistical theory.
Decision: Reject