Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ali Parsaee
authors: Bei Jiang
authors: Russell Greiner
reviews: 
Comments: This paper focuses on a problem of great practical interest, when the dataset is partially labeled and the data is censored. It proposes a budgeted learning (a subfield of active learning) method in the context of survival analysis.
It's a niche subject, but the method is very new and original, I am convinced it can be used for practical purposes. 

I am not familiar with either active learning or survival analysis (so I'll put 2 on the confidence score).
Strengths: - This paper is clear, domains are well introduced (whereas the paper is at the intersection of survival analysis and active learning)
- The results are convincing, and the authors have made the effort to compare themselves with “naive” methods (sanity checks part), even though there is no work dealing with this case.
Weaknesses: - I have some minor comments, it's more a question of presentation than content.
Recommendation: 3
Reviewer confidence: 2
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The authors consider the learning of survival model with budget constrained active learning. The formulation of the problem (Section 3) requires that the training dataset has the complete information of event times and censoring times, which is not a typical survival problem in which the survival times cannot be all observed for training data. The active learning is only applied to the case where the budget is used to extend the study (expend the observation period) in order to observed possible more events. This scenario is not the typical consideration in survival analysis in medical or health studies, though in reliability study in industry, in which type II or Type I right censoring are often used to collect data, it is possible to observe more events by increasing the study time.  In clinical trial random right censoring is mostly the observational scheme and  extending study time may not be able to "acquire" the information assumed in the article.
Strengths: The use of active learning in survival analysis is an interesting idea.
Weaknesses: The use of de-censoring as the label in survival analysis does not have much practical usage. It could be used in "reliability" in industry, though. The written English is not clear. The use of words and punctuation like  "and" and "," make the sentences hard to understand. For example, see  lines 269 and 282.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: - Presented work focuses on the problem of Bayesian active learning with the learner trained on survival analysis data. 
- The number of query steps is constrained by the given budget. In particular: 
   - the underlying data are right censored with the underlying latent classifier which indicates whether the patient died during the observation period;
   - the censoring corresponds to a patient dropping out of study and so after a particular time period we lose the access to the information whether the patient survived, the censoring bound is known. 
   - the learner can query an unlabeled dataset and obtain partially labelled censored instances, where some time interval can be queried, e.g. data are censored up to 3 years and we query information about additional two more years about the patient.  This corresponds to following up with the patient after some period;     
   - the budget constraint indicates that different labels have associated different costs and information content. 

Authors propose a solution based on the Bayesian Active learning by Disagreement (BALD) applied to querying a batch of given size.  Authors provide a budgeted greedy algorithm and incorporate mutual information scores from the BALD-like estimate. To budget constrain problem, authors try to optimize the cost coverage over the batch. 

Authors use 3 real life survival datasets to demonstrate their proposed method.
Strengths: - The selected problem is very relevant and interesting for real life data applications. The paper well states the problem and aspire to provide a solution.
- Authors propose three alternative methods to demonstrate the strength of their approach.
Weaknesses: 1) Paper contains multiple typos, see e.g line 210. 
2) Terms are not clearly defined, e.g. Algorithm 1, line 275, what is a_BB_surv ? 
4) The paper is not very clearly written, there is missing description of  the learner, there is missing debate on the distributional properties of the BALD model.  
5) The results in the main section seems cherry-picked, e.g. it would be good to include more points in the budget constraints rather than 0, 1, 5, 20.
6) It would be good to provide study on the synthetic dataset to demonstrate how the proposed methods perform in the controlled setting.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2