Comparing the Performance of Radiation Oncologists versus a Deep Learning Dose Predictor to Estimate Dosimetric Impact of Segmentation Variations for Radiotherapy

Authors: amith  kamath
zahira  mercado
robert  poel
jonas  willmann
elena  riggenbach
nicolaus  andratschke
mauricio  reyes
z  kamath
r p mercado
Venue: MIDL_io_2024_Conference
Published: 2024-01-25 11:17:32

Abstract: Current evaluation methods for quality control of manual/automated tumor and organs-at- risk segmentation for radiotherapy are driven mostly by geometric correctness. It is however known that geometry-driven segmentation quality metrics cannot characterize potentially detrimental dosimetric effects of sub-optimal tumor segmentation. In this work, we build on prior studies proposing deep learning-based dose prediction models to extend its use for the task of contour quality evaluation of brain tumor treatment planning. Using a test set of 54 contour variants and their corresponding dose plans, we show that our model can be used to dosimetrically assess the quality of contours and can outperform clinical expert radiation oncologists while estimating sub-optimal situations. We compare results against three such experts and demonstrate improved accuracy in addition to time savings. Our code is available at https://github.com/ubern-mia/radonc-vs-dldp.

Main paper: 
 introduction: 
 Glioblastoma, accounting for about 45% of brain tumors (McFaline-Figueroa and Lee, 2018) , is an aggressive malignant tumor treated with surgery, radiotherapy (RT), and chemotherapy (Stupp et al., 2005) . RT aims to target the tumor while minimizing dose to healthy organs-at-risk (OAR). The planning involves a trade-off between tumor control and tissue toxicity (Scaringi et al., 2018) . A critical step is the segmentation of structures, which is time-consuming when done manually, can take up to seven hours per patient in the head and neck anatomy (Das et al., 2009) .With advancements in deep learning-based auto-segmentation, the role of radiation oncologists is shifting from manually drawing to monitoring and correcting these automated segmentations (Claessens et al., 2022) . Quality checks are hence crucial since it has been reported that incorrect tumor volumes cause 25% of non-compliant treatment plans, leading to untreated tumors or harmful radiotherapy doses (Peters et al., 2010) . While geometric metrics like Dice score coefficient (DSC) and Hausdorff distance are currently the de-facto metrics to evaluate segmentation quality, it has been reported that they do not correlate with dosimetric effects of contouring errors (Poel et al., 2021; Kofler et al.) . In RT, it has been postulated that auto-segmentation methods must be evaluated using a diverse range of performance metrics, including impact on delivered dose (Harrison et al., 2022) , which ultimately impacts clinical outcome. Related work: The clinical RT community has developed standards for target contouring (Niyazi et al., 2023) , which mainly includes geometrical and anatomical considerations. Dosimetry-based considerations require dose plan calculations, which are time-consuming and necessitate iterations between the radiation oncologist and dosimetrists or medical physicists. Hence, due to its time-consuming nature, dosimetric assessment of segmentation quality has not been conventionally employed in the clinics. Nonetheless, as recently pointed out by (Claessens et al., 2022) , dosimetry considerations are urgently needed in the quality control process of tumor and organs-at-risk segmentations.Previously proposed approaches to evaluate the quality of automated segmentations include methods that predict segmentation metrics, such as DSC, (Valindria et al., 2017) or use Graph Neural Networks to identify segmentation errors in radiotherapy (Henderson et al., 2022) . These approaches assume that these geometric metrics reflect the quality of dosimetry, which is not the case (Poel et al., 2021; Kofler et al.) . Furthermore, models predicting DSC perform poorly with low-quality segmentations (the main target of such QC system) due to a lack of representative training data for this performance range.Other approaches have explored the use of uncertainty estimation in OAR segmentation in head and neck cancer (Cubero et al., 2023) , under the premise that high uncertainty is linked with potentially low-quality segmentations. However, geometric variability and uncertainty estimates may not imply dosimetric effects (e.g., high uncertainty of a contour located in a non-dosimetrically relevant area), and uncertainties based on imaging information alone may not sufficiently guide quality assurance.In line with dosimetry-focused quality assurance, a deep learning-based dose prediction model is utilized in (Roberfroid et al., 2024) to guide radiation oncologists on which volume slices require manual adjustments. This segmentation editing tool demonstrates potential for time efficiency while maintaining dosimetric equivalence with distribution maps produced without its assistance. In (Kamath et al., 2023b) we introduced a method that uses a deep learning-based dose predictor to assess the impact of local segmentation changes on dosimetric outcomes. However, this work focused on organ-at-risk segmentation, and not on tumor lesions, which is clinically more important due to the higher complexity of this segmentation task. Hypothesis and Contributions: Beyond the state of the art, we postulate in this study that a deep learning based dose predictor can be employed within a quality control framework to detect dosimetrically worse segmentations, with levels of performance superior to human experts. We substantiate this by comparing the performance of our deep learningbased quality assurance method with that of three expert radiation oncologists, using a test dataset comprising 54 segmentation variations from brain tumor patients, and reference dose plans produced by a clinically validated treatment planning system (Varian Medical Systems Inc., Palo Alto, USA).To the best of our knowledge, this is the first study comparing the levels of dosimetric awareness on contour modifications between human experts and a deep-learning dose predictor model. Our study design, depicted in Figure 1 , involves a set of reference tumor segmentations (ground-truth) and corresponding expert-derived contour variations (four per reference segmentation). For each reference and contour variation (n=54 pairs), dose plans are computed. Each contour is classified as "Worse", "No change", or "Better" based on dosimetric variations relative to the reference segmentation. This classification scheme is used to evaluate the ability of three experienced radiation oncologists and the proposed deep-learning dose predictor model to accurately classify each contour variation. We report classification metrics and time taken to perform the task. 
data: 
 Our data set includes imaging and segmentation data from 100 patients diagnosed with glioblastoma. This includes CT and Magnetic Resonance Image (MRI) T1 contrast-enhanced images, and binary segmentation masks of 13 OARs as well as the tumor target volume. The OARs include Brainstem, optic chiasm, cochlea (left and right), eye (left and right), hippocampus (left and right), lacrimal gland (left and right), optic nerve (left and right), and the pituitary gland. Each of these subjects also has a reference dose plan, calculated using a standardized clinical protocol with Eclipse (Varian Medical Systems Inc., Palo Alto, USA). This reference is a double arc co-planar volumetric modulated arc therapy (VMAT) plan with 6 mega volt flattening filter free beams, optimized (Varian photon optimizer version 15.6.05) to deliver 30 times 2 Gray while maximally sparing the OARs. The dose is calculated with the AAA algorithm (Van Esch et al., 2006) , normalized so that 100% of the prescribed dose covers 50% of the target volume. All the volumes are resampled to an isometric 2x2x2 millimeter grid of size 128 3 voxels using PyRaDiSe (Rüfenacht et al., 2023) and converted to NIfTI files to use for training and evaluation.We train a cascaded 3D U-Net dose prediction model (Liu et al., 2021) , which has been previously evaluated to show a mean prediction error of 1.38 Gray (Poel et al., 2023; Kamath et al., 2023a ) on a subset of n=60 cases (from the original 100 cases). The inputs are the CT volume, the OAR and tumor binary segmentation masks (14 volumes), and the output is the dose prediction. Ten cases are used as validation, and 14 are used as the test set for this study (Implementation details below). We save the rest of the 16 cases for future evaluations. Contour modifications and replanning: For 14 test subjects, between three to four variations to the tumor target volume are made independently by an expert using the same rationale as in (Poel et al., 2021) . The reference segmentation follows current delineation standards. The dose is then re-planned with the same settings as earlier to construct a ground truth dose plan for each of the variations. This leads to 54 test scenarios, which are categorized into "Worse", "No Change" and "Better" from a dosimetry perspective.Figure . 2 shows how a variation (in red) looks compared to the reference (in green).Ground-truth dosimetric categorization of contour variations: We define three categories of dosimetric impact based on a 10% change on the maximum dose recorded within each of the 13 OARs with respect to each reference contour as computed using the commercial treatment planning system. A contour variation is considered as dosimetrically impactful if at least one OAR crosses this threshold. We define three categories: "Worse", "No Change" or "Better" -indicating if such a change to the tumor target volume leads to higher OAR dose, no change, or lower OAR dose as compared to the reference segmentation, respectively. On the set of 54 cases, this definition leads to 33 Worse, 17 No Change, and 4 Better scenarios based on the ground truth reference dose plans.To automate the categorization of contours, we use the trained dose prediction model on the reference contour and each analyzed contour variation, yielding dose maps D ref and D cv , respectively. Each contour variation is then classified following algorithm 1, which follows the class definitions presented above for ground-truth generation.Algorithm 1: Contour Variation Quality Classification -Q(cv)1: for each OAR do 2: if max(D cv ) > max(D ref )(1 + αT ) then 3:increment counter for Worse Q(cv) = No Change 16: end ifAlgorithm 1 examines each Organ-at-Risk (OAR) to identify those exceeding a specified percentage dose change, determined by threshold T . If the count of such OARs surpasses the hyperparameter nOAR, the case is deemed "Worse". In the absence of dose violations, the algorithm investigates potential dose improvements (line 5). If none are found, the contour variation is labeled as "No-Impact". The hyperparameter α serves as a calibration parameter, akin to the ROC-AUC threshold in classification models. The hyperparameter nOAR sets the model's overall sensitivity to dose violations. Implementation details: Each of the two U-Nets in the cascade (Liu et al., 2021) has a depth of five sets of convolution layers in the encoder and decoder with 16, 32, 64, 128, 256 channels in the first level and twice this number in the second. The model input is a normalized CT volume and binary segmentation masks for each of the 13 OARs and target volume, and predicts a continuous-valued dose volume (up-scaled from [0, 1] to 0 to 70 Gray) of the same dimension as the input. The loss is computed as a weighted sum of L1 losses between outputs of the first and second U-Nets versus the reference dose: Loss = 0.5 * L1(ref erence, A) + L1(ref erence, B), where A and B are the outputs of the first and second U-Nets respectively, ref erence is the reference dose and L1 refers to the L1 loss. The weights are randomly initialized using the 'He' method. Training runs for 80000 iterations and the model with the best validation dose score is saved. The training batch size is set to 2 and the learning rate to 1e -3 with a weight decay of 1e -4. Data augmentation is done with random flipping and random rotation along the z-axis (in the axial plane). T in Algorithm. 1 is set to 0.1. All experiments are run with PyTorch 1.12 on an NVIDIA RTX A5000 GPU, and each training run takes approximately 24 hours. Comparing against human expert baselines: For each test case, three expert radiation oncologists evaluated contour variations and were asked to classify them using the defined dosimetric categorization. We use 3D Slicer (version 5.6.0) for this evaluation and show all three slice planes (see Figure . 6 in the Appendix). We also include a 3D rendering of the geometric relationship between the OARs and the tumor target volume, highlighting where the contour change is made. We time their responses, and show all the variations for each subject simultaneously so that they can make visual comparisons against the reference for each contour variation.As classification metrics, we report precision and recall, and the confusion matrices. Average time to evaluate each variant is also presented to compare performance. 
results: 
 Table . 1 shows the weighted average (across the three categories) precision and recall as well as the average time taken to evaluate each of the 54 variants by the three radiation oncologists (in rows marked R1, R2 and R3) as compared to the deep learning dose prediction model in the last row. On both precision and recall, the model outperforms all three experts. Notably, we underline the high inter-rater variability in performance among the three experts. Expert R3, being the most meticulous and expert rater, used significantly more time than other experts. While the proposed model tends to classify more "No-Impact" contours as "Worse", we view this as a beneficial trade-off. In practice, it would lead to additional checks, which is preferable to potentially overlooking increased toxicity to the patient. The time taken by the model is dominated by two inference runs through the reference as well as the variant contours. Additionally, the range of time taken varies broadly between experts, from 19 to 138 seconds per variation, while the deep learning predictor always takes the same quantum of time irrespective of the difficulty in geometry.Figure . 3 shows the confusion matrices (normalized by true category) for the three expert radiation oncologists (R1, R2 and R3) and the dose predictor model (right-most panel marked "Prediction"). Darker green on the diagonal is better, while darker red on the off-diagonal is not. The model outperforms all three radiation oncologists in the "Worse" category. None of the experts mark any variant as "Better". Figure . 4 shows the sensitivity of (weighted average) precision and recall to the hyperparameters α and nOAR used to classify dosimetric impact. α (horizontal axis) ranges from 0.005 to 0.15 in each of the two heat maps. Specifically, α = 0.1 means that the percentage change threshold for the model is 0.1 times that used for the reference (in this case, 1%). The vertical axis is nOAR, where smaller values make the model more sensitive and strict, and larger values increase model robustness while trading off sensitivity. Both the precision and recall metrics show a reasonably smooth variation, except that the precision values drop significantly for small α and nOAR. As good trade-off we chose α = 0.1 and nOAR = 3 for the results presented in Figure . 3 and Table . 1. 
discussion and conclusion: 
 Radiation oncologists indicate that their mental model emphasizes proximity to OARs (closer and in the line of sight between the tumor target and OARs are more impactful) and the size of the variation (larger causes higher residual dose to OARs). Figure . 5 demonstrates two exemplar situations that we use to showcase the strengths as well as weaknesses of our proposed idea. The left half shows one such condition where the model correctly classifies a condition that conventionally would be considered to be not impactful. Conversely, the right half shows a situation where the model overestimates the severity. This can be attributed to the predicted dose being a smooth proxy to the actual beam structure. Recent advances (Teng et al., 2024) aim to account for this effect.We present a novel dosimetry-driven quality control framework, where our dose predictor model outperforms human experts, indicates a promising baseline on which to build on improvements. This work demonstrates human clinician baseline, upon which we plan to work on the next set of evaluations where radiation oncologists are shown assistive maps like (Kamath et al., 2023b) to measure if their performance improves both in time and accuracy. 
 
 Appendix:  


Decision: The authors describe a deep learning model that estimates the dosimetric impact of segmentation variations in radiotherapy. The problem and the rationale of the work is well explained and the paper is well written. During the rebuttal phase, the authors adequately addressed many of the concerns raised, which helped in improving the quality of the paper.

Reviews: review: paper_summary: paper_summary: In this interesting manuscript, the authors describe a deep learning model that estimates the dosimetric impact of segmentation variations in radiotherapy. The problem and the rationale of the work is well explained and the paper is well written. The results compare the model to three expert radiation oncologists and show promising results. Great work!
main_review: main_review: I would like to thank the authors for their responses and the rebuttal. My questions have been adequately answered! The paper would benefit from a larger scale validation, but I understand that that takes time and effort.
strength_weakness: strength_weakness: - The establishment of the ground truth segmentations is unclear to me.
- The test set actually only consists of 14 cases, which is relatively small.
- An external validation on data from another center would strenghten the paper.
questions: questions: Please address my points in the detailed comments.
paper_summary: paper_summary: The authors propose to automatically predict the impact of tumor segmentation variations on the planned RT dose. This allows them to compare deep learning segmentation and expert segmentation in terms of dosimetric impact rather than the usual segmentation metrics.
The analysis is performed on a small set of CT images (14) from patients with glioblastoma, with multiple modifications of tumor volumes. 
Variations on the max dose recorded in the OARs are used to classify the doses of modified volumes as “Sub-optimal”, “No Impact” or “Improved”.
main_review: main_review: The authors replied to my questions and clarified some of my concerns. The overall quality of the manuscript is improved. Some limitations remain, but I changed my recommendation to borderline accept.
strength_weakness: strength_weakness: 1) The method and setup are not very complex, but I found it very hard to understand from Figure 1 and Section 2.2.

2) There is already a validated automatic treatment planning system that you use to create the three GT categories "Sub-optimal", "No impact", "Improved". I don't understand the motivation for the analysis: creating a new DL dose predictor and comparing the OAR dose impact with this GT (and with the human dose impact prediction). 

3) How can you say that just lower OAR dose is an improved dosimetry (and vice-versa), without considering the dose in the tumor? 
With this definition, a smaller volume results in an "improved" dose, and no volume at all is a perfect dose ? 

4) 54 contour variations are derived from 14 test volumes. 
How are the modifications made? I think it is a very important point for the study and it is not mentioned. 

5) The test set is very small (derived from 14 volumes).

6) There are incoherences (e.g. number of derived volumes from the test cases, see below).
questions: questions: All of the main concerns in the "Weaknesses" part. 
I may have missed an important point that would answer my second point, in which case a clarification is welcome.
paper_summary: paper_summary: Since over- and under-segmentation of tumor volumes may lead to radiotherapy plans that do not treat the entire tumor and/or exposes certain risk structures to high radiation doses, properly assessing the quality of (deep learning based) segmentation tools is extremely important. The authors of this work investigate the use of an existing deep learning based dose predictor as a new quality metric for such automatic segmentation tools within the field of radiotherapy of brain tumors. Specifically, for the dose prediction model, the authors use 60 patients to train a cascaded UNet mapping the CT volume and the associated segmentations of both organs at risk (OAR) and the tumor to the dose map computed within the software Eclipse (Varian Medical Systems). 

Using 54 pairs of tumor segmentation and a segmentation variation, the DL-based dose predictor is compared against expert radiation oncologists to classify whether the shape variation is dosimetrically impactful, i.e., whether the shape variation leads to a positive, negative, or no impact. The experiment demonstrates that the DL-based dose predictor outperforms the three expert radiation oncologists both in terms of accuracy and run-time.
main_review: main_review: The authors have adequately addressed all of my questions and concerns. I also appreciate the changes to the manuscript, which has noticeably improved the quality of the paper. Even though there is still room for improvement (e.g., the discussion of the results), I am happy to change my initial recommendation to (weak) acceptance.
strength_weakness: strength_weakness: Even though the paper is interesting to read and the topic of quality assessment of automatic segmentation tools is of particular importance, there are several weaknesses to be addressed in the rebuttal:

First of all, the technological novelty may be limited since it seems like the authors presented the same cascaded UNet trained on the same data in a preceding publication. If there are any additional technological contributions, please clarify it.

In addition, the methodology and experiments section lacks details and clarity to replicate and fully assess the paper:
- How were the segmentation variations generated? Are the variations primarily additions to the reference segmentations or are there also subtractions?
- On what basis was the dose threshold for the classification of the impact chosen? Is a 10% change of the maximum dose clincally relevant? How would the results of the experiments change with a different threshold for the ground truth labels?
- How are the ground truth labels generated? If they follow Algorithm 1, what values for alpha and nOAR were chosen? If another algorithm was chosen, please add a short description and justify the difference in algorithms used.
- Figure 4: Over which dataset were the precision and recall values computed?
- Figure 5: What does the yellow error in the right case denote? In general, it is really difficult to understand the images. One easy improvement may be a separate image showing the OARs in the same volume slice and, ideally, a table with the maximum dose values in each OAR. 
- Considering the low number of patients, running a cross-validation over the training/validation split could help strengthen the results

Moreover, it is questionable to me whether the comparison between the proposed method and the experts is fair. First of all, the ground truth labels and the DL predictions are generated from dose values. The experts, however, are only seeing the segmentation masks and thus must judge based on geometric proximity, which the authors even state at the beginning of the discussion. The results could be strengthened if, on a subset of the data, the experts are presented with the dose maps. 
Second, it is not clear to me whether the 10% change of the maximum dose as a threshold to form the ground truth is clinically meaningful. Considering that the experts are more often stating "no impact" than "sub-optimal" it may also hint that they could accept a higher tolerance. In addition to showing the experts the dose maps, it may be worth to analyse how close to the threshold the ground truth dose values were on the misclassified cases. 
Ultimately, the proposed classification method has tuneable parameters nOAR and alpha, which are adjusting the sensitivity of the classifier and could significantly improve the precision and recall. In particular, I am having concerns about alpha being chosen as 0.1, which sets the classification of the impact for a specific OAR to 1% as opposed to 10% for the ground truth generation. Could the authors please comment on this? In addition, I strongly believe that it would strengthen the comparison if error metrics in estimating the dose map for the test set were presented. 

Finally, the paper could greatly benefit from a deeper and more critical discussion of the presented results. Please use the above weaknesses as inspiration.
questions: questions: Please see the weaknesses section. While I understand that addressing all topics may not be realizable in the short amount of time, I would appreciate if priority is given to adding the missing details of the methodology and providing more insights into the performance of dose predictor and the comparison to the experts.

Comments: title: Author responses to feedback (point-by-point, part 2)
Author responses to feedback (point-by-point, part 1)
Author responses to feedback (point-by-point, part 2)
Author responses to feedback (point-by-point, part 1)
Author responses to feedback (point-by-point)
comment: > Figure 4: Over which dataset were the precision and recall values computed?  

These were calculated on the test set.  
  
 
> Figure 5: What does the yellow error in the right case denote? In general, it is really difficult to understand the images. One easy improvement may be a separate image showing the OARs in the same volume slice and, ideally, a table with the maximum dose values in each OAR.   

Thank you for the comment. This is clarified in the caption now to say “The yellow arrow shows the beam artifacts in the reference dose plan which are not replicated by our model” 
 
> Considering the low number of patients, running a cross-validation over the training/validation split could help strengthen the results. Moreover, it is questionable to me whether the comparison between the proposed method and the experts is fair. First of all, the ground truth labels and the DL predictions are generated from dose values. The experts, however, are only seeing the segmentation masks and thus must judge based on geometric proximity, which the authors even state at the beginning of the discussion. The results could be strengthened if, on a subset of the data, the experts are presented with the dose maps.   
  
We acknowledge the concerns and clarify that in this first phase we are mostly interested in assessing the current dosimetric awareness of clinicians (i.e., where no dosimetric information is available to them) vs. that of a DL-based approach. We clarify that, according to the clinicians, proximity is not the only “feature” they use in their internal thought process, but rather a combination of experience-based and such proxy features. At this point our main objective is to establish a human baseline that reflects the current clinical scenario. On a second stage, we are indeed conducting what the reviewer refers to.   
  
We have revised the discussion section by clarifying the strategy for this and our next steps.
We thank you for the positive feedback regarding the relevance and potential impact of our work.  Beyond making changes to the text in the revision, here are responses to your comments:

> First of all, the technological novelty may be limited since it seems like the authors presented the same cascaded UNet trained on the same data in a preceding publication. If there are any additional technological contributions, please clarify it. 

We would like to clarify that the innovation of the study does not lie within the development of a new dose predictor model, but rather on a novel dosimetry-driven quality control framework. The ability of the model to outperform human experts, using current model architecture, indicates a promising baseline on which to build on improvements. As the framework is the main core message of the study, we refrain from employing more complex architectures/models at this time.  

We have clarified and enriched the discussion accordingly. 

> In addition, the methodology and experiments section lacks details and clarity to replicate and fully assess the paper: 

> How were the segmentation variations generated? Are the variations primarily additions to the reference segmentations or are there also subtractions? 

Thank you for the question. We clarify with the text now saying: “For 14 test subjects, between three to four variations to the tumor target volume are made independently by an expert using the same rationale as in (Poel et al., 2021). The reference segmentation follows current delineation guidelines and .....” (last paragraph on page 4). We clarify here further that the variations are primarily additive.  

> On what basis was the dose threshold for the classification of the impact chosen? Is a 10% change of the maximum dose clinically relevant? How would the results of the experiments change with a different threshold for the ground truth labels? 
> it is not clear to me whether the 10% change of the maximum dose as a threshold to form the ground truth is clinically meaningful. Considering that the experts are more often stating "no impact" than "sub-optimal" it may also hint that they could accept a higher tolerance.  

We thank you for your question. We show below the precision, recall and F1 scores for all three radio-oncologists and our dose predictor model across six different percentage thresholds (20%, 15%, 12%, 10%, 8%, 5%) to create the ground truth: 
 
|                 |                      R1                 |                      R2                 |                      R3                 |                  Prediction             | 
|-----------------|:---------------------------------------:|:---------------------------------------:|:---------------------------------------:|:---------------------------------------:| 
|      Precision  |     0.6, 0.51, 0.45, 0.41, 0.38, 0.78   |     0.61, 0.49, 0.53, 0.48, 0.52, 0.79  |     0.73, 0.56, 0.55, 0.55, 0.54, 0.76  |     0.64, 0.48, 0.57, 0.57, 0.60, 0.78  | 
|      Recall     |     0.65, 0.56, 0.44, 0.35, 0.26, 0.19  |     0.56, 0.50, 0.54, 0.46, 0.43, 0.39  |     0.59, 0.54, 0.57, 0.57, 0.52, 0.50  |     0.56, 0.44, 0.57, 0.57, 0.59, 0.72  | 
|      F1         |     0.62, 0.51, 0.38, 0.29, 0.21, 0.28  |     0.57, 0.50, 0.52, 0.44, 0.42, 0.51  |     0.61, 0.54, 0.56, 0.56, 0.51, 0.59  |     0.58, 0.44, 0.57, 0.57, 0.59, 0.75  | 

The Precision of R1 varies between 0.38 and 0.78, R2 between 0.48 and 0.79, R3 between 0.54 and 0.76, and the model between 0.48 and 0.78, with the model being the best at 3/6 thresholds. 

For recall and F1-score, the model is best at 4/6 thresholds, indicating that this choice does not change our inference significantly, especially given that there is no current clinical understanding of what constitutes unacceptable levels of dose change in relation to OAR tolerance. We believe for this reason that a 10% change is a reasonable choice, and that in the future,  a dose constraint related metric could help. There are many distinct guidelines (maximum dose, mean dose, volume coverage and so on) which we will incorporate into this work in the next several months. 

 

> How are the ground truth labels generated? If they follow Algorithm 1, what values for alpha and nOAR were chosen? If another algorithm was chosen, please add a short description and justify the difference in algorithms used. 

We apologize for the lack of clarity on this point. The GT labels were produced using Algorithm 1 (alpha = 1, nOAR=1), with the exception that the true dose distributions (from Eclipse) were used instead of those provided by the dose predictor. The predicted labels are generated with alpha=0.1, nOAR=3, with the dose predictor. The parameters are changed for higher robustness: Alpha is set to 0.1 to make the labels more sensitive to overall dose changes, where nOAR is set to 3 to avoid outliers due to a single OAR exceeding the limit.
> There are incoherences (e.g. number of derived volumes from the test cases, see below).  
> Between three to four variations to the tumor target. Why does this vary? Somewhere else in the paper, you say 4 (Section 2)  

Thank you for the question and apologies for the lack of clarity.  We have two test cases with only three (instead of four) alternatives because the tumor target volume was small, and making more alternatives would not be feasible, as they would not be distinct enough.  

> "inter-rater variability in performance" Actual inter-rater variability would be more relevant.  

Inter-rater variability amongst the three experts show the correlation is between weak and moderately positive (based on spearman rank coefficient): R1 and R2: statistic=0.4779, pvalue=0.0002; R1 and R3: statistic=0.3304, pvalue=0.0146; R2 and R3: statistic=0.6657, pvalue=3.9449e-08; This shows variability while also general agreement on the categories. (0.0 to 0.3 is considered weak, and 0.4 to 0.7 is considered moderate correlation).    

> "16 cases for future evaluation" Explain why.   

We are planning to use these cases for a future evaluation where we show dose maps (like suggested by reviewer 3) and additional heatmaps (specifically related to ASTRA, Kamath et al. 2023) along with the contour variations and evaluate if the human experts perform better than earlier, and, if there is higher inter-rater agreements between them than earlier.
We thank you for the detailed feedback and suggestions, allowing us to improve the manuscript. Here are our responses to points you raise:

> The method and setup are not very complex, but I found it very hard to understand from Figure 1 and Section 2.2. 

> There is already a validated automatic treatment planning system that you use to create the three GT categories "Sub-optimal", "No impact", "Improved". I don't understand the motivation for the analysis ...

We appreciate your feedback and apologize for the lack of clarity. Our work is motivated by the evolving role of radiation oncologists in the era of AI-assisted radiotherapy. As automated segmentation becomes more prevalent, the role of medical experts is shifting from manual segmentation to monitoring and correcting AI-generated contours.  

In this new role, it is crucial for experts to assess if such contours’ deviation from a hypothetic ground truth may affect the dose delivered to tumor or organs at risk in a clinically-relevant manner, i.e. altering the chances of tumor control or the risk of normal tissue toxicity.. In order to make such a comparison, one would have to prepare a treatment plan based on the AI-generated contours and assess the dose delivered to the ground truth organs and tumor in this plan, and compare it to the achieved doses when planning directly to the ground truth structures. However, due to the time-consuming nature of treatment planning and common lack of an immediately available ground truth structure to compare AI-generated contours with, such comparisons are not feasible in routine clinical practice.  Our proposed deep learning-based dose predictor addresses this challenge. It enables a novel paradigm where alternative segmentations can be ranked from a dosimetric perspective, making the comparative assessment more efficient and feasible. 

We have adapted the introduction to make the motivation of our work clearer. 
 
> How can you say that just lower OAR dose is an improved dosimetry (and vice-versa), without considering the dose in the tumor? With this definition, a smaller volume results in an "improved" dose, and no volume at all is a perfect dose ?  
 
The ground truth plans are constructed with the target volume receiving the full prescribed dose, normalized for the same coverage. This means that no target coverage is sacrificed when OAR dose is estimated. The metric for the deep learning-based dose predictor hence is subject to this criterion, which will fail if it predicts no target volume dose, thereby marking everything “improved”.  
 
We have revised the text accordingly to clarify this. 

> 54 contour variations are derived from 14 test volumes. How are the modifications made? I think it is a very important point for the study and it is not mentioned.  

We fully agree and acknowledge the concern. We have revised the text to clarify that the modifications were made manually by an expert, following a similar rationale as presented in Poel et al. 2022 and Poel et al. 2021, where variations follow clinically plausible variations.  

Poel R., et al., Impact of random outliers in auto-segmented targets on radiotherapy treatment plans for glioblastoma. Radiation Oncology, 17, 1-18, 2022.  

Poel R., et al., The predictive value of segmentation metrics on dosimetry in organs at risk of the brain. Medical Image Analysis, 73, 102161, 2021. 
 
> The test set is very small (derived from 14 volumes). 
We agree with this assessment. We, however, note that scheduling clinician's time for running these human-versus automated-model evaluations is not easily scalable. The segmentation and planning preparation process consumes more than 70% of the total time from diagnosis to treatment, typically spanning over two weeks. Consequently, producing meaningful alternate segmentations and re-planning doses using the clinical TPS is a significant time investment. We are aware of this limitation and are working towards enriching the dataset of cases, and believe this first evidence is of high interest to the scientific community.  
 
We note that the work of Poel et al. MedIA 2021, employed a total of 12 cases from which a similar multiplicative effect exists leading to a total of 60 alternatives RT struct sets (our total set comprises 54 RT struct sets). We also highlight the multi-expert evaluation presented in this paper, which in our opinion favorably compares to other multi-rater studies presented in our community and similar venues.  
Poel R. et al., The predictive value of segmentation metrics on dosimetry in organs at risk of the brain. Medical Image Analysis, 73, 102161, 2021.
We thank the reviewer for the positive feedback on our work and hope our revision based on these suggestions can lead to further improvement in the review score. Here is a summary of our responses:

> The establishment of the ground truth segmentations is unclear to me. 
 
Thank you for this remark. We clarify with the text now saying: “The reference segmentation follows the current delineation standards.” (last paragraph of page 4). 
 
> The test set actually only consists of 14 cases, which is relatively small. 
 
We agree with this assessment. We, however, note that scheduling clinician's time for running these human-versus automated-model evaluations is not easily scalable. The segmentation and planning preparation process consumes more than 70% of the total time from diagnosis to treatment, typically spanning over two weeks. Consequently, producing meaningful alternate segmentations and re-planning doses using the clinical TPS is a significant time investment. We are aware of this limitation and are working towards enriching the dataset of cases, and believe this first evidence is hypothesis generating and of high interest to the scientific community.  
 
We note that the work of Poel et al. MedIA 2021, employed a total of 12 cases from which a similar multiplicative effect exists leading to a total of 60 alternatives RT struct sets (our total set comprises 54 RT struct sets). We also highlight the multi-expert evaluation presented in this paper, which in our opinion favorably compares to other multi-rater studies presented in our community and similar venues.  
Poel R., Rufenacht E., Herrmann E., Scheib S., Manser P., Aebersold D., and Reyes M. The predictive value of segmentation metrics on dosimetry in organs at risk of the brain. Medical Image Analysis, 73, 102161, 2021. 

> An external validation on data from another center would strengthen the paper. 
 
Thank you for the suggestion. We do have plans to run this analysis on external data, and since scheduling clinician time to arrange these subjective evaluations is not easy, we will run more of these in the weeks and months to come.  
  
> Finally, there are 54 variations on 14 test cases, because the authors write that 3 to 4 variations are performed. Why not make it consistent and apply four variations to all test cases, so that 56 variations are obtained? 
 
Thank you for the question and apologies for the lack of clarity.  We have two test cases with only three (instead of four) alternatives because the tumor target volume was small, and making more alternatives would not be feasible, as they would not be distinct enough. 

> Please clarify how the reference segmentations are set. Since the authors also write that there is substantial interreader variability (Results section reads "Notably, we underline the high inter-rater variability in performance among the three experts."), it is important to make it very clear how the reference segmentations are obtained. 
 
Thank you for the recommendation. We clarify with the text now saying: “The reference segmentation follows the current delineation standards.” (last paragraph of page 4). For inter-reader variability amongst the three experts, the correlation is between weak and moderately positive (based on spearman rank coefficient): R1 and R2: statistic=0.4779, pvalue=0.0002; R1 and R3: statistic=0.3304, pvalue=0.0146; R2 and R3: statistic=0.6657, pvalue=3.9449e-08; indicating that there is variability, but they all generally agree on the categories. (0.0 to 0.3 is considered weak, and 0.4 to 0.7 is considered moderate correlation), supporting our statements. 
 
> I find the naming of the three categories (“Sub-optimal”, “No Impact” or “Improved”) suboptimal. I would suggest to rename to something more clear and simple, perhaps just "Worse, no change, Better", or "Detoriated, no change, Improved". 

Many thanks for this suggestion. We have renamed the categories: “Worse, No Change, Better”.