Weakly supervised deep learning model with size constraint for prostate cancer detection in multiparametric MRI and generalization to unseen domains

Authors: robin  trombetta
olivier  rouvière
carole  lartizien
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 18:28:32

Abstract: Fully supervised deep models have shown promising performance for many medical segmentation tasks. Still, the deployment of these tools in clinics is limited by the very time-consuming collection of manually expert-annotated data. Moreover, most of the state-of-the-art models have been trained and validated on moderately homogeneous datasets. It is known that deep learning methods are often greatly degraded by domain or label shifts and are yet to be built in such a way as to be robust to unseen data or label distributions. In the clinical setting, this problematic is particularly relevant as the deployment institutions may have different scanners or acquisition protocols than those from which the data has been collected to train the model. In this work, we propose to address these two challenges on the detection of clinically significant prostate cancer (csPCa) from bi-parametric MRI. We evaluate the method proposed by (Kervadec et al., 2018) , which introduces a size constaint loss to produce fine semantic cancer lesions segmentations from weak circle scribbles annotations. Performance of the model is based on two public (PI-CAI and Prostate158) and one private databases. First, we show that the model achieves on-par performance with strong fully supervised baseline models, both on in-distribution validation data and unseen test images. Second, we observe a performance decrease for both fully supervised and weakly supervised models when tested on unseen data domains. This confirms the crucial need for efficient domain adaptation methods if deep learning models are aimed to be deployed in a clinical environment. Finally, we show that ensemble predictions from multiple trainings increase generalization performance.

Main paper: 
 introduction: 
 Over the last years, deep learning models have become state-of-the-art methods in almost all medical imaging applications, including segmentation and detection. Among data-oriented methods, fully supervised models remain the most common and best performing ones. However, gathering numerous expert-annotated data to train such models is a very time and ressources consuming process, restraining the current use of such models in the medical field. For this reason, other promising paradigms have also been explored such as semi-, weakly-or unsupervised learning (Bosma et al., 2023; Baur et al., 2021) . They aimed to mitigate the need of annotated data to train deep learning models.Another known drawback of deep learning methods is the limited generalization capacity to unknown data distribution. It has been shown that when tested on out-of-distribution data, deep learning models can significantly underperform compared to in-distribution evaluation (Boone et al., 2023 ). Yet, the robustness of models to unseen domains is an absolute necessary condition for their use in clinical settings given the inherent heterogeneity among scanners and acquisition protocols between and within clinical institutions.In this work, we propose to tackle these two problems in the challenging task of detecting and localizing clinically significant (ISUP grade group ≥ 2) prostate cancer (csPCa) lesions in multi-parametric MRI. This is a task of primary clinical interest, as shown by the recent success in the urology community of the PI-CAI (The Prostate Imaging: Cancer IA)1 challenge. Many recent works have tried to improve the automatisation of cancerous prostate lesions detection (Bhattacharya, 2022) . Most proposed deep learning strategies focus on supervised models, with architectures such as nnUNet ranging among the top performing on the PI-CAI challenge dataset. A few recent works have proposed self-or weakly supervised approaches (Tardy and Mateus, 2021; Bateson et al., 2021) , leveraging bounding boxes (Baumgartner et al., 2021) , scribbles or patient-level annotations (El Jurdi et al., 2021; Yang et al., 2021) , partially lowering the gap with supervised approaches.Our contributions in this work are threefold:• We evaluate the method proposed by (Kervadec et al., 2018) for the challenging task of segmenting csPCa lesions in multiparametric MRI and achieve performances close to strong fully supervised baselines using only circle scribbles and image-level priors.• We evaluate how the scribble annotation process impacts performance of weakly supervised model and show that the model is robust to various weak annotation strategies.• We evaluate the models both on in-distribution validation data and unseen test images to evaluate the drop in performance in the generalization configuration, and show that our weakly supervised model is less prone to such effect.• We quantify to what extent ensemble predictions from multiple trainings improve generalization of deep learning models. 
a weak segmentation model based on object size constraint loss function: 
 In (Kervadec et al., 2018) , the authors proposed a loss function for partially annotated data that aims to impose a size constraint on the predicted segmentations of a model. The partial cross-entropy H, computed only on the annotated pixels Ω a , is combined with a constraint loss C that adds a quadratic penalty to the model on the total sum of its predictions for class c if it is outside a defined range [a, b] . More specifically, let V c = p∈Ω S p,c be the sum of the probabilities S p,c for class c of every pixel p in the image domain Ω. The constraint loss is given by :C(V c ) =      (V c -a) 2 if V s < a (V c -b) 2 if V S > b 0 otherwise (1)The total cost function is then defined asH(S) + λC(V S ) (2)where λ is a positive constant weighting the two terms, V S = p∈Ω S p with S p the softmax probability at pixel p in the image domain Ω. The size constraint loss term was initially used in a binary segmentation problem to improve prostate gland segmentation in multiparametric partially labeled MRI in (Kervadec et al., 2018) . (Duran et al., 2022) extended the binary formulation to a multi-class output with C classes and evaluated it on a prostate cancer detection task. They used this constraint at the image level with image tag priors, following the definition in Kervadec et al. (Kervadec et al., 2018) , that is enforcing the presence of the target class by setting a = 1 and b = |Ω| (the image domain) or the absence of the target with parameters a = b = 0. This implementation achieved promising performance for segmentation and grading of PCa lesions in a weakly supervised setting on the Prostatex-2 challenge and a private datasets.We extend the work of (Duran et al., 2022) on prostate cancer detection by leveraging the constraint term referred to as common bounds introduced by (Kervadec et al., 2018) , whose principle is to introduce more precise lower and upper bounds a and b depending on the size of the lesions in the ground truth. These bounds are a way to introduce prior knowledge on the objects to detect to compensate for the partially labeled data. We implement this method by imposing a common bounds constraint both on the prostate class and the CS lesion class.The experiments are conducted on three datasets, described hereunder :• PI-CAI challenge public training dataset. It contains 1500 multi-parametric MRI (T2w, DWI and ADC) exams from 3 Dutch centers acquired on 7 different scanners, 5 from Siemens Healthineers and 2 from Philips Medical Systems. It includes 328 cases from the Prostate-X challenge (Armato III et al., 2018) . Of all the exams available, we only use the 1295 that are manually annotated by expert clinicians, and do not leverage the 205 exams with AI-derived lesion segmentations. • The Prostate158 (Adams et al., 2022) Both T2-weigthed (T2w) and apparent diffusion coefficient (ADC) MR maps were used as input channels. The latter modality was registered to the former, all images were resampled to a 1 × 1 × 3 mm 3 pixel size and cropped to 96 × 96 × 20 volumes. Images intensities were linearly normalized into the range [0, 1] for each patient and each modality. More details about these datasets can be found on Appendices A and B, including lesion volume distributions and histograms of intensities for T2-weighted imaging and ADC maps.The aim of weak annotations is to mimic what could be an easier and faster way for clinicians to provide annotations on real images. For this purpose, we replace full segmentations by circles of maximum radius of 3 mm inside each individual lesion. The centers of the circles are drawn randomly and independently on each axial slices. If the lesion is too small to fit a circle of this size, the radius is reduced until a circle can fit inside the lesion. The prostate gland is also annotated is such way, with only one circle per slice. In total, weak annotations only represent 14% of the full masks of CS lesions, considerably reducing the amount and complexity of annotations and thus the time needed for experts to make these annotations. Illustrative circle annotations are depicted on Figure 3 . Appendix F evaluates how the best weakly supervised model performs when other annotation strategies are adopted.We compare several weakly supervised methods and fully supervised baseline models. For our proposed scribble based weak model, we consider two main configurations : one with partial cross-entropy (CE) and the image tag (IT) and one with partial CE, image tag and common bounds (CB) constraint loss terms. We compare them to a simpler weak model with partial cross-entropy and negative cross-entropy (denoted Partial CE) as well as to fully supervised baselines trained with cross-entropy and generalized DICE loss on the full available annotations. We use 2D and 3D MONAI's DynUNet (Cardoso et al., 2022) as backbone architectures for the proposed weak and fully supervised models. As for comparison to other weakly supervised models, we train nnDetection (Baumgartner et al., 2021) with ground truth segmentations masks being 3D rectangular cuboids framing full lesion annotations (nnDetection full ) or weak scribble annotations (nnDetection weak ). Note the comparison between these models and the ones with size constraints is not straightforward as they do not use the same kind of weak annotations.All models are trained in 5-fold cross-validation on the PI-CAI dataset. They are first evaluated in the in-distribution setup, meaning we report the mean performance on the 5 validation folds of the PI-CAI dataset. Then, we appraise the models in the generalization setup by testing them on data from two unseen domains, namely Prostate158 and our private database. Moreover, for each method, we combine the best models of each training fold into a single ensemble model, for which the lesion probability maps are computed as the average of the probability maps of the 5 aggregated models. These ensemble models are only tested on the two unseen data domains.The models are evaluated both at lesion and patient levels. Following PI-CAI guidelines, a detection map is made of non-overlapping and non connected clusters, representing predicted csPCa lesions. Each lesion is assigned a unique probability score, chosen as the average of the probabilities of the cluster's voxels. At a lesion level, we report metrics derived from the free-response receiver operating characteristics (FROC) curve which shows sensitivity as a function of the number of false positive detections per patient. In continuity of previous works done on this csPCa detection task (Bosma et al., 2023; Saha et al., 2021) , we consider a predicted lesion as a true positive if it intersects a ground truth lesion with an intersection-over-union (IoU) ratio of at least 0.1. Since there is no consensus metric to summarize a FROC curve, we choose to report the sensitivity at 1 false positive per patient. Another complementary indicator of the performances of detection models is the average precision (AP), defined as the area under the precision-recall curve. Finally, for the patient-level diagnosis performance, the area under the ROC curve (AUROC) is reported. The patient's overall likelihood of harboring csPCa is defined as the maximum score of the predicted lesion clusters.All hyperparameters were determined with grid search on the first fold of the PI-CAI training/validation splits (see Appendix C for more details). The lower and upper bounds associated with the CS lesions class were set to 5 and 500 voxels for 2D models and to 30 and 4 000 voxels for 3D models. For the prostate class, they were set to 100 and 2 500 in the 2D case and 10 000 and 40 000 in the 3D case. The fully supervised baselines were trained with cross-entropy and generalized DICE loss. All models were trained during 200 epochs with Adam optimizer, a learning rate of 10 -3 and a weight decay of 10 -4 . To compensate for the low amount of lesions in the PI-CAI dataset (see Table 1 ), sampling was weighted such that 2D transverse slices for 2D models -or 3D volumes for 3D models -with and without lesions have the same probability of being drawn in a batch. For post-processing, predicted lesions of size inferior than 15 voxels are discarded. 
classification and detection performances: 
 Figure 1 shows performance of all considered models for the three metrics of interest, namely sensitivity at 1 FP, AP and AUROC. Figure 3 provides examples of visual results of lesion detection maps for some 3D models. Extended visual results, including of 2D and ensemble models, are showcased in Appendix E.First of all, it is important to note that the best performing model, namely the 3D supervised DynUNet, achieves a mean AUROC of 0.82 and mean AP of 0.42 thus producing a mean aggregated score of 0.62. This performance compares well against the best achievable reported metric on the PI-CAI challenge leader-board. We thus consider it a reliable baseline for our comparison. Surprisingly, the 3D supervised DynUNet is still outperformed by 2D models in term of sensitivity at 1 FP, including by models trained with weak labels.The two weak models with size constraints (CE+IT, CE+IT+CB) clearly outperform the model trained only with partial and negative cross entropies (Partial CE), showing the interest of the additional size constraint cost functions. Between the image tag (IT) and the common bounds (CB) losses, the latter achieves a higher score in 22 of the 30 configurations. We use the term configuration to refer to a pairwise comparison between models with the same spatial dimension input (2D or 3D) and type of model (ensemble or not) evaluated on a given dataset and for a given metric. For instance, AUROC comparison between 2D CE+IT and 2D CE+IT+CB on Prostate158 accounts for one configuration. Remarkably, the models that have been trained with weak labels can outperform fully supervised models. They also perform favorably compared to nnDetection in most cases. The weakly supervised CE+IT+CB model achieves better scores than its supervised counterpart in almost all 2D configurations, but only 2 times out of 30 in 3D. For almost all models, metrics and datasets, model ensembling improves the generalization performances, with a mean improvement of 20% among all models and metrics. We can note that for sensitivity at 1FP and AUROC, some models, especially with ensemble predictions, equal or surpass the reference metrics, which is the mean performance over 5 folds of the 3D fully supervised DynUNet trained and tested on the Prostate158 or private datasets, respectively. However, these reference models remain better in terms of AP. Finally, the comparison study provided in Appendix F shows that the weakly supervised models are robust to several scribble annotation strategies and that the one we chose does not bias the model towards an overestimation of its performance.Figure 2 shows the relative performance of the models on the two test datasets, that is Prostate158 and our private database, compared to the performance of the same model evaluated on the in-distribution validation dataset. We did not report the results for the cases where the models are trained with partial and negative cross entropies as their absolute performances are much lower than the others. For most of the metrics and models, there is, as one could have expected, a notable drop in performances when the models are evaluated on a test set that has been acquired in a different setup from that of the training dataset. The average performance decrease ratio is of 28% among all models and metrics. This can reach values as low as -61% for supervised models. Quite surprisingly, the Average-Precision score is even or better -sometimes by a large amount -on the test datasets than on the validation datasets for many models.Compared to fully supervised models, the weak models trained with CB constraint loss has a more favorable relative change in 19 configurations out of 24 (6 configurations correspond to the PI-CAI dataset and are thus not considered here). This advantage is also found when compared to the models trained with the IT loss. Ensemble predictions, with a rule as simple as averaging the output probability maps of models obtained from several trainings, almost always help reducing the performance gap in generalization.  
discussion and conclusion: 
 Our proposed weakly supervised method achieves competitive results compared to fully supervised baselines, while requiring only 14% of annotation voxels of clinically significant lesions. It consistently outperforms 2D supervised DynUNet trained with cross-entropy and generalized DICE loss. In the 3D configuration, the model trained with full segmentation annotations remains better overall. The addition of the more precise common bounds (CB) size constraint gives better results compared to the image tag (IT) model that was proposed in (Duran et al., 2022) . Among all compared methods, the weak model with CB loss is the most robust to unseen data domains. As seen on Figure 2 , it indeed suffers the least from a performance drop when tested on data that do not belong to the training distribution.Our study confirms, for the task of csPCa lesion detection and segmentation, that heterogeneity between training and test databases noticeably impacts performance of deep learning models and is thus an issue of first interest if such models are aimed to be used in a clinical environment. We show that one simple way to mitigate this issue is to make ensemble predictions from multiple trainings, as this allows decreasing the performance drop in almost all the configurations we have tested. Finally, it is important to note that the best models trained on PI-CAI reach performances on unseen domains that can be on par with fully supervised models trained on these datasets. This is an encouraging result that supports the current trend of building models on a given large training dataset, in a weakly or fully supervised setup, and deploying it on other institutions that have less or no annotated data. Further work includes refining the hyperparameters of the scribble based weak models as well as designing task-specific and few shot domain adaptation methods to better handle dataset heterogeneities. • a and b : {5, 10} for a and {100, 200, 300, 400, 500, 600} for b in the two-dimensional case and {10, 30, 50, 70, 100} for a and {1500, 2000, 2500, 3000, 3500, 4000, 5000, 6000} for b in the three-dimensional case. The grid search in the 2D case is smaller because an optimization of the hyperparameters had already been done in (Duran et al., 2022) . • the class weights w c were set to 0.14 for the prostate and 0.22 for the lesion based on empirical values reported in (Duran et al., 2022) .  
 
 Appendix: © 2024 CC-BY 4.0, R. Trombetta, O. Rouvière & C. Lartizien.The PI-CAI grand challenge : https://pi-cai.grand-challenge.org/PI-CAI/ 


Decision: The Meta-Reviewer has read all reviews and rebuttal of authors.

The work has received mixed reviews. There is a consensus that the main value of the paper is rather on the evaluation of an existing approach for prostate cancer detection, rather than contributing novel technical methodology. It is acknowledged by reviewers that the paper is well written, the evaluation is performed on 3 databases (which is commendable) and that the experimental results provide some useful insights, eventhough there are limitations (eg further baselines/comparisons could have been performed).

Overall, the consensus between reviewers and this meta-reviewer is that the paper provides sufficient experimental insights to be of interest to the community. Therefore I will recommend accepting the paper

Reviews: review: paper_summary: paper_summary: The paper proposes to validate a popular size constraint loss proposed by (Kervadec et al., 2018) on large-scale databases of prostate cancer in a weakly supervised setting. Their extensive experiments show that for both in and out-of-distribution data model trained with weak supervision performs on par with the full supervision setting. They also show that the ensemble of these networks always gives better performance compared to a single model.
main_review: main_review: The authors provided a detailed rebuttal.  My main initial concern was regarding the focus of the work. In the revised manuscript, author clearly mentioned that the focus of the work is applying kervadec et al. 2018 method on challenging prostate lesion detection task. Considering that the main focus of the conference is not on novelty but rather on quality of the paper, I am leaning towards weak accept.
strength_weakness: strength_weakness: * The paper seems to be a direct application of Kervadec et al., 2018 on a prostate cancer dataset. The authors may want to highlight what is different in their paper compared to Kervadec et al., 2018 or state clearly that their work is focused on validating Kervadec et al., 2018 on prostate cancer datasets.
* The paper is missing some relevant references [1] [2] [3] which are related to weakly supervised settings and Kervadec et al., 2018.
* Authors mention that their full supervision model compares well against the PI-CAI challenge leaderboard. Maybe the authors want to clearly state the best performing method on the leaderboard and report those results for the sack of completeness. 
* Image Tag (IT) has been used in a weakly supervised setting. However, details or a citation about the same are missing from the paper. Could the authors please give more information about how IT has been used in a weakly supervised setting? 
* Section 2.3 mentions Figure 3 for examples of weak annotations, but Figure 3 doesn't provide that as it shows ground truth and predicted segmentation for different methods. The authors may want to add weak annotations in Figure 3.
* Authors mention that parameters for size constraint-based loss (a and b) were decided empirically. Looking at the histogram of Figure 4 in the appendix, clearly shows that the finalized values of a and b divide the histogram into 3 separate Gaussian distributions. It would be interesting to report model performance for these three different bins of lesion size. I am assuming that the performance would be drastically different for these. 
* It is not clear why the authors employed CE + Dice for the full supervision setting and not only CE. It might be a good idea to only use CE in the full supervision setting as it would be easier to compare against other methods.
* Why in the 2D case weakly supervised setting outperform the fully supervised setting and not in the case of 3D? Can authors comment on that?

[1] Tardy, M. and Mateus, D., 2021. Looking for abnormalities in mammograms with self-and weakly supervised reconstruction. IEEE Transactions on Medical Imaging, 40(10), pp.2711-2722.

[2] El Jurdi, R., Petitjean, C., Honeine, P., Cheplygina, V. and Abdallah, F., 2021. High-level prior-based loss functions for medical image segmentation: A survey. Computer Vision and Image Understanding, 210, p.103248.

[3] Bateson, M., Dolz, J., Kervadec, H., Lombaert, H. and Ayed, I.B., 2021. Constrained domain adaptation for image segmentation. IEEE Transactions on Medical Imaging, 40(7), pp.1875-1887.
questions: questions: Please address as many questions raised in the weakness and the detailed comment section as you can.
paper_summary: paper_summary: The study addresses challenges in deploying fully supervised deep models for medical segmentation due to the time-consuming collection of expert-annotated data and issues with domain shifts. It proposes a weakly supervised method using scribble annotations to detect clinically significant prostate cancer from MRI scans. The model achieves comparable performance to fully supervised models on validation and unseen test data but experiences decreased performance on unseen data domains, highlighting the need for domain adaptation methods in clinical deployment. Ensemble predictions from multiple trainings enhance generalization performance.
main_review: main_review: The evaluation did not incorporate state-of-the-art segmentation techniques.

Statistical significance, including confidence intervals, was not included in the analysis of model performance.

The introduction of an ensemble method aimed to narrow the performance gap, indicating possible enhancements for other models presented.

Regarding the better performance of 2D over 3D models, there is no clear justification for why the 3D supervised DynUNet is consistently outperformed by 2D models in terms of sensitivity at 1 false positive.
strength_weakness: strength_weakness: - no state of the art methods segmentation techniques employed.
-  statistically signify CSE of models not reported e.g CI
- ensemble method was introduced to reduce performance gap which underscores other models presented; more detail on how models were ensembled.
questions: questions: Why is 2d performing better than 3d is there a justification?
“Surprisingly, the 3D supervised DynUNet is still outperformed by 2D models in term of sensitivity at 1 FP, including by models trained with weak labels.”
What is the significance of introducing ensemble method, how were the models ensembled?
paper_summary: paper_summary: The paper presents a weakly supervised method with size constraints for the detection of prostate cancer lesions is MRI. Similar to (Kervadec et al., 2018), the proposed method combines a partial cross-entropy loss on scribble-annotated pixels with a size constraint loss on the predicted foreground. The size constraint is defined with fixed lower and upper bounds on the size, and is implemented with an L2 loss as in (Kervadec et al., 2018). While the previous work in (Kervadec et al., 2018) tackled the problem of segmentation, this paper instead focuses on detecting lesions (a lesion is detected if it has a sufficient overlap with a true lesion). The model is trained on the PI-CAI dataset and tested on hold-out samples of this dataset as well as on two different datasets, Clara-P and Prostate158, verifying its cross-dataset generalization ability. Results highlight the benefit of the size loss and the difficulty of generalizing to unseen datasets.
main_review: main_review: Authors have provided detailed answers to my questions, however the main issues with the paper remain: lack of methodological novelty and poor evaluation of the proposed method. Nevertheless, I raise my score to Borderline reject.
strength_weakness: strength_weakness: * The main weakness of the paper is its lack of methodological novelty. The proposed model is same as (Kervadec et al., 2018) and the overall method mainly differs by how the predicted segmentation is used (separate connected components are treated as different lesions)

* Experiments only compare the proposed method against variants (without the image tags or size losses, 2D vs 3D segmentation)
questions: questions: The authors need to clarify the novel contributions of their method and compare it against recent approaches for weakly-supervised segmentation/detection.

Comments: title: Final response
Final response
Thanks for the detailed answers
Response to Rebuttal
Comments and response to all reviewers
Response to Reviewer aYxy
Response to Reviewer 49yd
Response to Reviewer heuH
Response to reviewer heuH
comment: 1) Providing analysis for P0-P10, P10-90 and P90-100 was an easier way to discuss the bias with the lesion sizes, and we think that finer 2D analysis for 0-a, a-b and b-max(lesion size) would yield similar conclusions. We agree with you that the higher performance for large lesion if indeed due to the rather loose criteria for counting true positives. Extending the analysis to the influence of the hyperparameters on performance by lesion size is indeed a necesseray perspective of this paper, to assess the robustness of Kervadec's method on this task.  As a final addition, we have added to Appendix D the maximum sensitivity and average false positive per patient. It may help future readers to better understand the behaviour of the models, such as the difference between 2D and 3D models, or the impact of ensembling on the number of false positives.

2) Given the short amount of time before the closure of the discussion phase, we do not have time to train the weakly supervised model with CE + DICE (and redo the grid search) or to train and evalute the baselines with CE loss only (although we agree that it would have allowed for better comparison) and we will leave the paper in its current state. 


It would be our pleasure to continue the discussion about the choice of cost function and size constraint bounds at the conference if the paper is accepted. We would like to thank you once again for your interest in our work and for taking the time to exchange with us.
Thank you for the detailed feedback. 

1. Thank you for the provided analysis and correcting the histogram figure. From the revised figure-5 I can see that choice of b is still really small and majority of 2D lesions fall outside the range of  a-b.  Considering that, I am not sure why the provided analysis was down for P0-P10, P10-P90, and P90-P100. These percentiles were not used to select the hyperparameters. As per my original comment, the analysis would be more useful for 0-a, a-b, and b-max(lesion_size). Nonetheless, provided analysis indeed shows that there is indeed a bias. Especially, for the small lesion size, but this is expected. However, I was surprised that peformance was higher for the large lesion. But maybe this is due to the fact that the paper measures on detection performance and not segmentation. In the future, authors may want to show this as one of the features of their work that lesion detection performance, especially for the large lesions are not affected by the choice of hyperparameter b. Also, in the future authors may want to lesion wise dice scores in the table, and also number of false positive lesions detected per patient in the performance evaluation. 

2. Thank you the provided analysis. Indeed, as I was expecting, the performance for weakly supervised methods indeed seems to improve with addition of Dice. I would encourage the authors to either stick to only CE or CE+Dice for both supervised and weakly supervised methods (the choice is theirs). As it allows better comparison. 

Considering all the efforts by the authors, I am converting my rating to weak acceptance
1) We have updated the contribution section and the abstract to make our main contribution clearer. We hope that this new version will better meet your expectations.

2) We would like to apologize as there was a mistake on the distributions of the lesion sizes that we had displayed in the appendix. The main peak/bin represented in fact the size of the prostate, which explains its higher frequency compared to lesion and its largely superior mean size. We have corrected this mistake with the true distribution, which does not contain anymore the three main bins of lesion sizes that you mentionned in your review. However, to address the point raised by the reviewer concerning the differences in performance depending on the size of the lesions to be detected, we provide hereunder a table with the sensitivity at 1 FP per patient for the best weakly supervised and fully supervised models for three bins of sizes. P0-P10 is for lesions between under 10th percentile in size distribution (*small lesions*), P10-90 is for lesions between the 10th and 90th percentile (*normal size lesions*), and P90-P100 is for the lesions above the 90th percentile (*large lesions*). As expected, we observe some discrepancies in performance depending on the size of the lesion to be detected, smaller lesions being generally less well detected by both models. Moreover, the weakly supervised model, even though it sets lower and upper bounds on the object sizes to be segmented, which one intuits could have biased its performance towards underperforming on small and/or big lesions, does not seem to be more biased than the baseline 3D supervised baseline. Please let us think if you believe that this analysis on the performance depending on the lesion size is of interest to improve the quality of the paper.

|   PI-CAI                        | P0-P10 | P10-P90   | P90-P100 |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.391       |  0.745    | 0.913     |
| 3D Supervised                 |  0.348          | 0.610     | 0.826     |

|   Private dataset                       | P0-P10 | P10-P90   | P90-P100 |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.264      | 0.404    | 0.261     |
| 3D Supervised                 | 0.118     | 0.323    | 0.339     |

|   Prostate158                       | P0-P10 | P10-P90   | P90-P100 |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.733 | 0.526   | 0.667    |
| 3D Supervised        | 0.444    | 0.410    | 0.689     |


3) DICE loss is particularly well suited for cases where there is a disparity between the sizes of the object that are aimed to be segmented. Indeed, it penalizes a model based on the proportion of ground truth that is well segmented, therefore handling better class imbalance than cross-entropy. In our weakly supervised setup, and given how we obtain the weak annotations (point scribles), most of the objects have the exact same size, so we intuited that the model would not benefit from being trained with cross-entropy and DICE loss. We trained the best performing weakly supervised model, namely 2D CE+IT+CB, with additional DICE loss and provide in the table below its performance. It gives slightly better results overall on most metrics and databases. However, as the discussion period is coming to its end, we do not have time to train all the weakly supervised models, select the best hyperparameters and evaluate them with the same protocol as that of all the other experiments. Thus, we can not include the results below in the article but hope that it will convince the reviewer of the value of applying Kervadec's loss to this more challenging prostate cancer detection task.

|   PI-CAI                        | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.746 +/- 0.028       |  0.256 +/- 0.042    | 0.776 +/- 019     |
| 2D + CE + IT + CB + DICE                 |  0.783 +/- 0.059          | 0.268 +/- 0.080     | 0.790 +/- 0.017     |

|   Private dataset                       | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.376 +/- 0.056 (0.401)      | 0.280 +/- 0.047 (0.399)    | 0.641 +/- 0.031 (0.661)     |
| 32D + CE + IT + CB + DICE                 | 0.404 +/- 0.037 (0.457)     | 0.278 +/- 0.032 (0.414)    | 0.776 +/- 0.019 (0.686)     |

|   Prostate158                       | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.542 +/- 0.059 (0.635)  | 0.388 +/- 0.041 (0.421)   | 0.726 +/- 0.028 (0.781)    |
| 2D + CE + IT + CB + DICE        | 0.577 +/- 0.051 (0.635)     | 0.377 +/- 0.030 (0.500)    | 0.733 +/- 0.017 (0.786)     |
Dears Reviewers, 

As requested, we have extended our comparison to state-of-the-art models by adding nnDetection to it. nnDetection is a 3D object detection network that uses segmentation masks as supervision. To compare it with our models, *i.e.* in a weakly supervised setting, we only trained using bounding boxes by replacing full segmentations by bounding boxes segmentations. We considered two setups : a first using the real segmentations to create the bounding boxes (*whole lesions BB*) and a second where the boxes were created from the dotted annotations that we used for other weakly supervised models (*weak lesions BB*). We have trained and evaluated it on the same setup as all the other models.

We provide hereunder the results of these two additional models (and show in the table the best weakly supervised model on dot annotations and the best supervised model for comparison). We will add these results to the table, figures and discussion very soon in a new version of the article.

|   PI-CAI                        | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.746 +/- 0.028       |  0.256 +/- 0.042    | 0.776 +/- 019     |
| 3D Supervised                  |  0.705 +/- 0.058          | 0.412 +/- 0.047     | 0.825 +/- 0.013     |
| nnDetection (whole lesion BB)  |  0.476 +/- 0.042      | 0.276 +/- 0.043    | 0.758 +/- 0.034     |
| nnDetection (weak lesions BB)  | 0.327 +/- 0.057        | 0.189 +/- 0.019    | 0.746 +/- 0.044    |

|   Private dataset                       | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.376 +/- 0.056 (0.401)      | 0.280 +/- 0.047 (0.399)    | 0.641 +/- 0.031 (0.661)     |
| 3D Supervised                  | 0.304 +/- 0.045 (0.272)     | 0.337 +/- 0.030 (0.432)    | 0.627 +/- 0.037 (0.626)     |
| nnDetection (whole lesion BB)  | 0.332 +/- 0.045 (0.365)     | 0.240 +/- 0.059 (0.287)    | 0.614 +/- 0.045 (0.626)     |
| nnDetection (weak lesions BB)  | 0.215 +/- 0.025 (0.267)     | 0.122 +/- 0.014 (0.165)  | 0.602 +/- 0.025 (0.633)     |

|   Prostate158                       | Sensi at 1 FP | AP   | AUROC |
|--------------------------------|---------------|------|-------|
| 2D + CE + IT + CB              | 0.542 +/- 0.059 (0.635)  | 0.388 +/- 0.041 (0.421)   | 0.726 +/- 0.028 (0.781)    |
| 3D Supervised                  | 0.438 +/- 0.071 (0.438)     | 0.366 +/- 0.031 (0.484)    | 0.733 +/- 0.019 (0.780)     |
| nnDetection (whole lesion BB)  | 0.295 +/- 0.039 (0.322)    | 0.199 +/- 0.054 (0.237)   | 0.632 +/- 0.042 (0.673)     |
| nnDetection (weak lesions BB)  | 0.238 +/- 0.052 (0.311)   | 0.140 +/- 0.047 (0.197)    | 0.619 +/- 0.073 (0.668)     |
I thank the authors for answering my questions. I believe the paper has merits, however I still feel that applying an existing method to a different problem offers a limited contribution to the field. Also, the lack of comparison to weakly-supervised baselines remains an issue.
Thank you for providing detailed feedback: 

1) Can you please clearly state at the end of the introduction in the contribution section that the major contribution of your work is to evaluate the Kervadec et al. 2018 method to difficult lesion segmentation task? 

3) I agree that limited time for the rebuttal period might not allow you to perform this analysis, but I would argue that this would have been the easiest. This analysis doesn't require you to perform any new experiments and only extends the performed analysis by dividing the test set into three different sets based on lesion size. The analysis is important as the majority of the lesions' size is greater than the chosen value of b (Fig 4 and Fig 5). In this case, it is important to analyze if the proposed method indeed works well for all lesion sizes, or if it introduces some type of bias based on lesion load. 

4) It is indeed encouraging that choosing CE+Dice over CE was based on empirical performance and the authors chose the strongest option. However, this raises the question of why CB losses were only employed with CE and not CE+Dice. As it would make a better case of showing the usefulness of CB losses.
Dear Reviewers, 

We first would like to thank you all for the time spent on reviewing this paper and for your fruitful feedbacks. We have tried to address most of them in the following responses and append the manuscript as far as we could due to time and space constraints. We hope that the quality of the manuscript will meet your expectations. The main comments, some of them shared by the reviewers, are the followings:

- Clarify the link between this work and those of (Kervadec et al., 2018) and (Duran et al., 2022).

- Add an appendix in which we compare several ways of obtaining weak labels.

- We had forgotten to show the weak annotations in figures 3 and 8; this has been corrected. There was also a mistake in Figure 1, where the performance of the model 'Reference' was underevaluated (instead, we were reporting the performance of the ensemble 2D supervised model). We would like to apologize for this error, which does not change most of the analysis made in our work.

- Provide explanation of the differences in performances between certain models. In particular, please see answer to point 4) of reviewer 49yd and point 3) of reviewer aYxy.

Changes in the paper have been highlighted in red to facilitate the reviewers' revision.

To expand our comparison with concurrent models, we are currently running a 5-fold training of nnDetection (Baumgartner et al., 2022), a state-of-the-art model in medical imaging object detection. The results will be soon added to the paper. For now, we can report on this model's performance on : AP = 0.265 and AUROC = 0.735. Please note that the model uses a different kind of weak supervision than we do, namely bounding boxes, and we intuit than this is a stronger supervision when evaluation is performed with detection metrics. Please also read the reply to reviewer heuH for more details on how PI-CAI-s leaderboard results compare to our own.

We'll be pleased to have feedbacks from the reviewers and exchange with them during the discussion phase to further improve the quality of the paper.
Thank you for your interest in our work and your comments, Reviewer aYxy. We are glad you appreciate the work we put to evaluate the sensitivity of the models to distribution shifts. we have corrected the typos you pointed out and we hope to address your concerns below.

***1)The authors need to clarify the novel contributions of their method and compare it against recent approaches for weakly-supervised segmentation /detection.***

This important point was also raised by the other reviewers. We extensively discuss it in our *reply 1) to Reviewer heuH*, and would ask Reviewer aYxy to please refer to this response.

***2)”For this purpose, we replace full segmentations by circles of maximum radius of 3 mm inside each individual lesion. The centers of the circles are drawn randomly and independently on each axial slices.” This scenario is not very realistic as human annotators would probably put the circle near the center of the lesion on each slice. If the segmentation is performed in 3D, having the circles placed randomly may actually offer more information as it covers a greater extent of the lesion. An alternative strategy to define the annotations is to erode the lesion mask until the target size is reached.***

We would like to thank the reviewer for raising this point about how we characterize the placement of weak annoatations, which is crucial to justifying the relevance of our results. As suggested, we have added an appendix that compares several annotations methods. Among the methods compared, no one clearly stands out as superior to the others. Importantly, it shows that the way we initially choose to model the weak annotations does not tend to overestimate the performance of the weakly supervised models compared to other weak annotations types.

***3)”p6: ”Quite surprisingly, the Average-Precision score is even or better – sometimes by a large amount – on the test datasets than on the validation  atasets for many models.” Can you explain this result?***

We do not think there is an obvious explanation for this phenomenon, but a possible explanation, at least for our private database could be found in the characteristics of our dataset itself, especially with regards to the patient inclusion criterion. Indeed, our database was built up from patients who had all undergone radical prostatectomy after the presence of cancer had been proven by biopsies. This inclusion criterion, which is more restrictive than that of the PI-CAI training dataset, results in a much higher rate of positive patients and a much higher number of lesions per patient (see Table 1 in Appendix A for details). We hypothesize that the lesions in the private database are more advanced and more easily distinguishable than those in the PI-CAI training database, which would explain the higher AP. One result that would tend to validate our interpretation is that the reference model also has a much higher AP than its counterpart trained and tested on PI-CAI (see Figure 1 and Table 2).

***4)Weak evaluation. Experiments only compare the proposed method against variants (without the image tags or size losses, 2D vs 3D segmentation)*** 

This point was also discussed in detail in our response both to reviewers heuH and 49yd. For greater clarity and to avoid unnecessary repetition, we kindly ask reviewer aYxy to refer to the *answer 2) to reviewer heuH and answer 1) to reviewer 49yd*.

***5)”The constraint loss weight lambda was set to 10e-5 and 10e-8 for 2D and 3D models respectively” Why such small values?***

The constraint cost function is not normalized and is therefore typically of the order of magnitude of the square of the objects to be detected. Please note that this explains why the lambda factor is higher in 3D than in 2D models. We could have worked out a way to normalize this cost function to avoid having to use a lambda learning factor, but the grid search to determine a relevant value for lambda was easy to conduct so we chose to preceed this way.
Thank you for your interest in our work and your comments, Reviewer 49yd, we hope to address your concerns below.

***1)no state of the art methods segmentation techniques employed.***

We believe that the supervised models we are comparing with consitute solid baselines *for an evaluation with single models or simple ensembling*. The best-performing models in the PI-CAI challenge are typically designed to win such challenges, i.e. a clever ensembling of well-established models, such as nnUnet and nnDetection in this case, to improve the robustness of the predictions. For further details on the limits of comparing our models with those of the PI-CAI ranking and with the winning methods of the challenge, *please refer to the answer 2) given to reviewer heuH*.

As stated in the global response, we are currently training an nnDetection model on PI-CAI data and aim to include its results in the next few days, to provide for readers a comparison with a state-of-the-art model in weakly supervised segmentation. However, please note that that nnDetection is supervised by bounding boxes annotations, which we believe to be stronger ground truths than dots when models are evaluated with detection metrics.

***2)statistically signify CSE of models not reported e.g CI***

As all the metrics are patient-level or lesion-level, they have to be computed on a whole dataset. Hence, to including CI and statistical tests would mean doing additional training to make them significative, for instance via bootstraping or multiple iterations of the 5-fold cross-validation. We did not run such experiments for computational resources issues and only provided standard deviation on the results of the 5 folds. If the reviewer has precise ideas of statistical tests that could provide insight on the robustness of the models to multiple trainings or evaluations, we would be pleased to add in our work.

***3)ensemble method was introduced to reduce performance gap which underscores other models presented; more detail on how models were ensembled.***

The ensembling method used here is very simple : for each method, the probability scores output by each of the 5 models (one per training fold) are averaged. Then, as for a single model, argmax function was used to obtain the final segmentation masks on which the performances are computed.

***4)Why is 2d performing better than 3d is there a justification? “Surprisingly, the 3D supervised DynUNet is still outperformed by 2D models in term of sensitivity at 1 FP, including by models trained with weak labels.” What is the significance of introducing ensemble method, how were the models ensembled?***

Although it is difficult to rationalize why this happens, we have identified a point that might be part of the explanation. The 3D models are more cautious and make less predictions than 2D models. Consequently, the sensitivity at 1FP that we report for them are often their maximum sensitivity, achieved at a FP per patient rate lower than 1 (we have added to Table 2 an indicator to flag when this is the case), whereas this almost never happens for 2D models.
This phenomenon can be mitigated with a loss function, that biased the model toward making more predictions, such as Tversky loss, as employed by the winning model of the PI-CAI challenge. Note that this may results in a lower precision of detections. The strategy is particularly effective when different models are subsequently ensembled.
***3)Authors mention that parameters for size constraint-based loss (a and b) were decided empirically. Looking at the histogram of Figure 4 in the appendix, clearly shows that the finalized values of a and b divide the histogram into 3 separate Gaussian distributions. It would be interesting to report model performance for these three different bins of lesion size. I am assuming that the performance would be drastically different for these.***

With the other experiments we had to run, we didn't have time to get the final results on this point. Preliminary results show that sensitivity is indeed much lower for small lesions (volume less than 80 voxels). We are currently investigating this point in more detail to see if this effect is less present on certain models.

***4)It is not clear why the authors employed CE + Dice for the full supervision setting and not only CE. It might be a good idea to only use CE in the full supervision setting as it would be easier to compare against other methods.***

We are looking at the problem of prostate cancer detection from the point of view of how the annotations are obtained. Our aim is to show that it is possible to compete with fully supervised models using annotations that are quicker to obtain by adding an a priori to the model. With this in mind, we think that it has more sense to compare the weak models with strong supervised baselines (both 2D and 3D). From our experiments, both 2D and 3D supervised models achieve better performance using a combination of CE and generalized DICE loss, that is why we decided to include these results. 

As an extension of this work, we plan on testing the CB loss with a nnUNet. However, please note that even though nnUNet is often considered as a state-of-the-art model in medical imaging segmentation, a well-tuned UNet often achieves similar performances. For instance, on the PI-CAI leaderboard, they are 3 comparison databases and 2 training setup (supervised or semi-supervised), and on these 3*3 = 6 possible comparisons, the mean performance of UNet (0.64) is close to that of nnUNet (0.69).

***5)Why in the 2D case weakly supervised setting outperform the fully supervised setting and not in the case of 3D? Can authors comment on that?***

We believe that the explanation of this phenomenon is two-fold : the overperforming of 3D supervised compared to 2D counterparts *and* the underperforming of 3D weak models compared to 2D counterparts. For the first point, it is not really surprising and in most cases, 3D models benefit from larger context, thus achieving better results than 2D models. For the second point, we hypothesize that it is mainly due by the fact that size bounds are less informative as the number of dimensions increases. Indeed, the lesion size distribution is more spread-out/has more variance in 3D than in 2D, in particular due to the high anisotropy of the images, so the bounds a and b are looser.

***6)Usually, I prefer graphs. But I think, in this paper, the graphs are giving too much information (Fig 1 and Fig 2) as there is too much to unpack. Maybe the authors may want to convert it into a table for ease of reading. It would be beneficial if they could provide the table for the same in the appendix.***

Even if it is a little dense, we think that the figure allows a better visualization of the results and comparison between the models, so have decided to keep it like this for the main text. However, we do agree that a table providing the numerical results is a valuable addition to the paper, so, as suggested by the reviewer, we added it in the appendix.
Thank you for your interest in our work and your comments, Reviewer heuH. 

We are glad that you appreciate that we provided generalization results. we have corrected the typos you pointed out and added the references you indicated that were missing from the related works. We hope that we will address your main concerns hereinbelow.

***1) The paper seems to be a direct application of Kervadec et al., 2018 on a prostate cancer dataset. The authors may want to highlight what is different in their paper compared to Kervadec et al., 2018 or state clearly that their work is focused on validating Kervadec et al., 2018 on prostate cancer datasets.***

We would like to thank the reviewer for highlighting this point, which lacked clarity in the paper. The common bounds loss that we use in the paper was already introduced by (Kervadec et al. 2018), for the prostate segmentation task. In (Duran et al., 2022). only the image-tag loss has been adapted to prostate cancer detection. In this work, we validate the use of more precise size constraint (CB loss) on prostate cancer detection task.

Even though this works does not introduce any methodological novelty with regards to the size constraint loss proposed by Kervadec et al, we believe that it still provides a valuable and novel contribution first because it expands the application of this proposed general loss to a more difficult task than binary prostate segmentation illustrated in Kervadec et al. The detection of prostate cancer is indeed more challenging, because the objects to detect are intrinsically more subtle. We demonstrate the positive impact of the CB loss in this situation where the prostate lesion sizes are much less steadily distributed than the prostate ones. Second, we show that the weak learning paradigm of Kervadec allows very promising performance with regards to supervised learning settings.

Finally, we would like to emphasize that our study demonstrates the value of this model for an application of primary interest as recently demonstrated by the great success of the PI-CAI challenge within the urology community.

***2) Authors mention that their full supervision model compares well against the PI-CAI challenge leaderboard. Maybe the authors want to clearly state the best performing method on the leaderboard and report those results for the sack of completeness.***

The models referred to in this statement can be found here : https://pi-cai.grand-challenge.org/evaluation/open-development-phase/leaderboard/. Of the three sections, the one that is more comparable to the experiments we have carried out is 'Open Development Phase - Tuning Leaderboard', as it is an evaluation on 100 patients from the same centers as the training dataset, hence providing a smaller feature and label shifts. However, we have chosen not to discuss further the comparison of the performance of our reference supervised models with those of the PI-CAI leaderboard for several reasons: 
- The best teams, including the one that won the challenge, mainly leverage ensemble of different models (typically nnUNet and nnDetection) to make more robust predictions and thus improve their AUROC and AP. Moreover, they all combine their lesion segmentation predictions with the knownledge of PSA density, for instance with a logistic regression. Although such ensemble models are commonplace at the top end of any segmentation challenge rankings, we find it of limited interest to compare with them. The models that make the most sense to compare with are those whose performances have been shared by the challenge organizers, i.e. by the users *joeran.bosman* and *anindo* on the leaderboard. Their scores (i.e. mean between AUROC and AP) are as follows : 0.576 for UNet, 0.597 for nnUNet and 0.502 for nnDetection, which are in par with performance achieved by our model (0.62, see Figure 1 and Table 2).

- The data on which the models are evaluated are not the same. We evaluate in a 5-fold cross-validation setup on 1295, whereas the performances reported were obtained after training on all these 1295 patients and testing on 100 new patients, for which we do not have the exams. On these 100 patients, the final predictions are obtained as an ensemble of 25 models (the 5 best models of the 5 training folds), while on PI-CAI, we only report result of single models.

- The models are trained with T2w, ADC and DWI as input modalities while we only used the first two.

With all these elements, we hope that the reviewer has a better idea of why we are claiming that our 3D supervsied baseline model compares well against the PI-CAI challenge leaderboard. We haven't had time to share our model for evaluating them on the hidden cohorts, but we plan to do so in an extension of our work, to provide an even more relevant evaluation.