Venue: MICCAI 2022 Challenge FLARE

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Pros:
- high dice
- proposed techniques are relevant to the task including post-processing for some organs
- overall, authors did a great job

Cons:
- techniques are hardly novel, yet effective
review_title: Uncertainty-Guided Self-Learning Framework for Semi-Supervised Multi-Organ Segmentation

final_score: rating: 7: Good paper, accept
review_content: 1, The training process is simple, but work for improving the segmentation results.
2, The word "AI" in the title is a large concept, and it is not sutable for the semi-supervised segmentation task in FLARE2022, in my opinon.
3. In Section 2.2, the authors describe the process of training multiple models, including multiple iterations, cross-validation, uncertainy calculation, and seleting the pseudo labels. I suggust the authors provide a diagram here, because it is the key parts of the proposed method. Moreover, The authors should emphasize how to use unlabeled data.
4. The method  has excellent DSC, however, since FLARE2022 challenge is a joint low resource and accuracy oriented segmentation challenge, the authors should better provide the prediction time both in abstract and results as well as DSC.
5. The authors should provide the visualization examples of successful and failed cases.
review_title: The authors propose a progressive training strategy to leverage the unlabeled images of FLARE2022. They trained multiple models using labeled data, and predict pseudo labels for the unlabeled images. Based on the pair-wise DSC, the pseudo labels are selected to train another model.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose an uncertainty guided framework for multi-organ segmentation on CT scans that uses a small labelled dataset to leverage a large unlabelled dataset in a semisupervised setting and modifies the nnU-Net framework for efficient inference, e.g., increasing the step size of strided inference from 0.5 to 0.9. The quality, clarity, and description of the paper are good except for some minor issues.

* It is recommended to add a map of uncertainty guided framework for multi-organ segmentation in the method introduction.
* In general, the code link should be at the end of the abstract rather than the introduction.
* It may be more pleasing if table 4 is horizontal and the table header is the abbreviation of the organ.
review_title: Uncertainty-Guided Self-Learning Framework

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Strengths: The proposed method first trains five models based on labeled data, then generate pseudo labels by the trained models, and finally train one model based on labeled data and selected pseudo-labeled data. The method increases 5% DSC by using selected pseudo-labeled data over using only the labelled data.
Weaknesses: 
- In five-fold cross-validation, are training epochs of five models same? How do you determine the training epochs?
- The calculation method of the mean pairwise DSC is not clear. It is better to add a equation.
- Which model is chosen after five-fold cross-validation? Does the method perform better if ensemble the network weights of five models?
- It is better to show some unlabeled samples that are not used for network training.

review_title: AI that knows when it does not know - Uncertainty-Guided Self-Learning Framework for Semi-Supervised Multi-Organ Segmentation

final_score: rating: 7: Good paper, accept
review_content: Strengths: The paper proposes a very interesting technique to measure the uncertainty in the pseudo-label prediction. The authors also make the code publicly available. 
Weaknesses: The preprocessing steps should be better described. The model description can be improved. 
Details: 
- Although the authors mention that they apply the nnU-Net cropping and normalization strategies, these should be described in more detail in the manuscript. 
- The text in Fig 1 is very small, increase the font. 
- The description of the method can be improved. Include figure that shows the psuedo-labelling process. 
- Justify why you use three iterations.  
review_title: The authors propose an uncertainty-guided self-learning framework for multi-organ segmentation that implements uses psuedo labels to train with unlabelled cases. 

Meta-review: metareview: Nice paper. Please address the reviewers' comments in the revised manuscript.
confidence: 5: The area chair is absolutely certain
Decision: Minor Revision