Venue: MICCAI 2022 Challenge FLARE

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this manuscript, the authors designed a co-training framework consisting of two strategies to generate high-quality pseudo labels and make efficient inferences, respectively. The resource-intensive uu U-Net model achieved a mean DSC of 0.9064 on 13 abdominal organ segmentation tasks and ranked first on the validation leaderboard. 

Strengths: The model achieved the top score on the leaderboard in terms of mean DSC.

Weaknesses and queries for authors: 

1: There is no ablation study that proves the effectiveness of the model. Suggested to add.

2: No NSD was reported in the manuscript. Suggested to add.

3: The paper structure could be modified and suggested to restructure; things are added in unarranged manners or have missing components/connections. 

3: While training the lightweight nn U-Net, do you use the sliding window? If so, then how could it be beneficial for faster inference?

4: Figure 1 should be modified entirely, which is more apparent concerning a clear explanation for each component.

5: Table 5 is split; it is recommended to make it unified for clarity.

6: Figure 4 has captions at the bottom rather than on the top.

7: Throughout the manuscript, there are many grammatical errors and typos. Therefore, it is suggested that the whole manuscript could be checked for such mistakes and should be fixed, such as,

- nnUUNet should be nnU-Net

- 3d should be 3D

review_title: The resource-intensive uu U-Net model achieved a mean DSC of 0.9064 on 13 abdominal organ segmentation tasks and ranked first on the validation leaderboard. However, still have some issues.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary:
This paper combines a noisy student approach with nnU-Net. Here, 5 large teacher models are trained on the labeled data, which are then ensembled to generate pseudo labels on the unlabeled dataset. The pseudo labels are re-used to iteratively improve the large teacher models on both the labeled and pseudo-labeled images. 
For the final efficient student network, a small 3D U-Net is trained on the union of labeled and filtered pseudo-labeled images. The filtering is based on an image-level consistency of segmentation maps between the different large-model re-training iterations.

Pros:
- Very well written, clear description of the strategy and methodology used.
- Figures support the understanding of the approach in a clear fashion.
- The filtering approach is very intuitive and a nice way to filter for stability.
- The efficient sliding window approach is a good approach to save inference time.

Missing information:
- How exactly is the filtering done (uncertainty threshold u)? The 76 filtered unreliable pseudo labels seem a bit ad-hoc.
- A comparison to the big nnU-Net in terms of inference time, GPU memory etc. would have been interesting.

Problems:
- The resolution of Figure 4 is a bit low, which makes it hard to inspect segmentation details (minor).
- Typo in section 4.3: dcoker instead of docker (minor).
review_title: Strong iterative noisy student approach based on nnU-Net, just some minor details missing.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary:**

This work is a nnU-Net based framework and achieves mDSC of 0.8773 in only about 12 seconds for each case. It's meaningful that this work is several times faster than the nnU-Net while maintaining the same performance as nnU-Net.

**Strengths:**

- Similar or even higher performance compared with nnU-Net but much faster than nnU-Net.
- Simple but efficacious pseudo Labeling scheme for semi-supervised learning.
- Efficient sliding window strategy. It will significantly shorten inference time by skipping windows without foreground area.

**Questions:**

- In this work, the supervised nnU-Net achieves mDSC of 0.8797 and even gets 0.9064 after re-training, as shown in Fig. 3. In my experience, the pure nnU-Net can not achieve this (most works based on nnU-Net achieve mDSC around 0.85~0.87 in the FLARE2022). Thus, my first question is, how does the big nnU-Net achieve such high performance in supervised training? Is it trained on nnU-Net default deployments or with some other tricks not mentioned?
- How much time does the efficient sliding window strategy reduce in the inference stage? How much performance will be dropped by using this sliding window strategy? These are not mentioned in this work.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose an iterative pseudo-labelling process based on nnU-Net to leverage the unlabeled data. Two nnU-Net models are developed:
a high-performance, high-resource nnU-Net for iterative pseudo label generation
a lower-performing, lower-resource nnU-Net for efficient inference
The authors also change the sliding-window inference approach of nnU-Net to make it more efficient.

Pros:
. A well-written paper, simple method with a thorough description
. Good performance on the validation set with both the big and small networks.
. Reproducible and open-source code.

Cons:
. No reasoning or evidence is shown for the design choices regarding the input patch size and spacing for the big/small nnU-Nets
. There is no ablation study to show the individual influence of every alteration made to nnU-Net for optimizing resource consumption. Namely, the authors changed the architecture, increased the downsampling, disabled test-time augmentations, reduced the patch overlap to 50%, and introduced a different sliding-window approach. Based on the presented results, it is impossible to say if the proposed sliding window approach makes a significant difference in resource consumption and performance.
. For the sliding window approach, the authors define a region of interest for abdominal organs but do not specify how this region is obtained.
review_title: Iterative pseudo labeling with nnUNet

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: Strength: The trade-off between model efficiency and accuracy is quite reasonable. The use of nnUnet ensures the generalizability of the model. And the inference speed is guaranteed with a low number of model parameters. Great illustration.

Weakness: No obvious logical error found in the paper except for some minor grammar mistakes. Good job.
review_title: Good redesign of classical nnunet

Meta-review: metareview: Nice paper. Please address the reviewers' comments in the revised manuscript.
confidence: 5: The area chair is absolutely certain
Decision: Minor Revision