Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 4: Weak accept
review_content: summary: The paper presents a Computer-aided Laparoscopic Bleeding Management (CALBM) for early detection and management of intraoperative bleeding that exploits temporal information through a space-time memory network with domain adapttion. Moreover, a novel dataset for laparoscopic bleeding detection has been introduced to test the proposed method.
strengths: - Despite the limited novelty of the work, it address a relevant problem for the medical society
- A novel dataset of about 1000 labelled images has been proposed
- The paper is well written and easy to read
- An Ablation study helps validate the efficacy of the proposed method
weaknesses: - Despite a dataset is presentend, the amount of frames is limited and the cliam of the authors is misleading
- The improvements to the space-time network are not clearly presented
- The literature review on the spatio-temporal methods is limited
- The purpose of the domain adaptation is not clearly introduced 
- User evaluation is very limited and does not provide enough evidence to validate the results
recommendation: Poster
review_title: null

final_score: rating: 3: Borderline
review_content: summary: The paper proposed a space-time memory network that learns from real and semi-synthetic data, using adversarial domain adaptation for early detection and management of intraoperative bleeding during laparoscopic procedures. The paper compares and evaluated with SOTA methods and improves performance by a reasonable margin. 
strengths: - the paper is reasonable well written and clearly presented
- the combination of known approaches is sound and the results show good improvements 
- a new dataset is introduced comprising of 60 hours of videos including about 1000 labelled images
- a user evaluation is provided
weaknesses: - dataset probably not public
- methodological insights are limited, not going far beyond the addition of more input dimensions and more data with adversarial domain adaption
- no ablation study provided 
detailed_comments: 
The contribution section states "we introduce a dataset of over 60 hours of videos including about 1000 labelled images.", which sounds like that this dataset will be provided with the paper but I cannot see a download link of project page. 

since several user responses are available from the surveys, it could be considered to also evaluate inter-rater agreement. 

Kamnitsas K, Baumgartner C, Ledig C, Newcombe V, Simpson J, Kane A, Menon D, Nori A, Criminisi A, Rueckert D, Glocker B. Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. InInternational conference on information processing in medical imaging 2017 Jun 25 (pp. 597-609). Springer, Cham.
should be added to the discussion of domain adaption. 

The results are still not great, thus I am wondering how relevant such a method would be in practice. Is IoU perhaps not relevant for the downstream task, or is perhaps missing a bleeding much worse than a false positive (the something like a F2 score could also be evaluated if the metric should be either biased towards false positives or false negatives) 
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The authors propose to modify Space-time Memory Network (STM) by adding the Spatio-temporal positional encoding and unsupervised Adversarial Domain Adaptation for bleeding segmentation in laparoscopic videos. The authors evaluate their method with their gynecologic laparoscopic video dataset and compare their proposed method with DeepLabV3+. The authors claim that their method can be applied to online bleeding segmentation for surgeons.
strengths: 1. The idea of adding the Spatio-temporal positional encoding and unsupervised Adversarial Domain Adaptation to STM seems to be novel.

2. Besides metric-based evaluation, the authors also conduct user evaluations for their system.

3. The authors conduct a good literature review for prior work from the methodological development aspect.
weaknesses: 1. Not enough details are provided for the dataset. This has a negative effect on the evaluation part of the paper. Please check on my detailed comments on this.

2. One big challenge for a Video-based Computer-aided Laparoscopic Bleeding Management system is to distinguish bleeding from blood. However, the paper did not address enough on this issue. Please check on my detailed comments on this. 

3. The current method seems to be evaluated in an offline setting instead of an online setting. Please check on my detailed comments on this.
detailed_comments: For weakness point 1:

(1) It is very difficult for the reader to understand what is the detailed data split here. It seems that the author select some videos and annotate only part of them as the training set and test set. May I know how many videos are used for training/domain adaptation/testing? May I know how many images/video clips are used for training/domain adaptation/testing? How many semi-synthetic training videos are generated? A Table of the detailed dataset might be helpful for this part. 

(2) If the dataset size is small and the test data is limited,  cross-validation should be done in the evaluation part of the paper. 

For weakness point 2:

(1) Detecting bleeding is sometimes difficult due to the model failing to distinguish bleeding with blood presence. The authors claim their system can be used for online purposes. However, the authors did not share any analysis on how accurate the model can detect bleeding events.

(2) How many images in the training/test data have blood presence on them (Not bleeding)? Please share more details on this topic. For online applications, surgeons certainly do not want the model to misrecognize bleeding with blood presence. 

For weakness point 3:

(1) The evaluation seems in offline mode. To capture different edge cases in online settings, why not annotate full video for bleeding segmentation. Will selecting short video clips and annotating 20 representative images per video clip for ground truth be enough to generate a representative dataset for this challenging problem? 

(2) It will be great if the authors can provide us with error analysis. What are the edge cases where the model fails to detect bleeding? What are the edge cases where the model cannot segment the bleeding from images well? How these will affect the online application?
recommendation: Poster
review_title: null

Meta-review: metareview: According to the reviewers, the quality of the paper improved alot during the rebuttal phase (2x strong accept, 1x weak accept). I also recommend acceptance of the paper. The authors explained the data set split in more detail and provided other very important information, that helps in reproducing the work. The method was evaluated with SOTA methods and performance was improved by a reasonable margin. 


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)