Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the AdaWAC as an adaptive weighting algorithm
that assigns label-dense samples to supervised cross-entropy loss and label-sparse
samples to unsupervised consistency regularization.
This paper also proposes a WAC regularization that uses the consistency of encoder layer outputs as a natural reference to incentivize separation between samples with sparse and dense label. 
strength_and_weaknesses: Ads:
This paper proposes the AdaWAC as an adaptive weighting algorithm
that assigns label-dense samples to supervised cross-entropy loss and label-sparse
samples to unsupervised consistency regularization.
This paper also proposes a WAC regularization that uses the consistency of encoder layer outputs as a natural reference to incentivize separation between samples with sparse and dense labels. 
Experiments are conducted to validate the results showing the effectiveness of the proposed methods.

Dis:
The paper organization and the paper writing are poor, which makes the paper difficult to understand.
In general, there is a lack of explanation of the methods used in the study.
Furthermore, an explanation of why the authors did these experiments should be provided.
In experimental table 1, the AdaWAC proposed in this paper is still worse than the baseline results in some situations. The authors are advised to provide some explanations and justifications. 
The results of comparing current SOTA methods used in medical image segmentation should be presented. Now the article only shows the results of the baseline and the proposed AdaWAC. 

clarity,_quality,_novelty_and_reproducibility: The paper organization and the paper writing are poor, which makes the paper difficult to understand.
summary_of_the_review: This paper proposed AdaWAC as an adaptive weighting algorithm
that assigns label-dense samples to supervised cross-entropy loss and label-sparse
samples to unsupervised consistency regularization.
However, this paper is difficult to understand.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The manuscript proposed an adaptive data weighting algorithm for 3D medical image segmentation. The proposed algorithm dynamically adjusts weights between supervised loss and self-supervised consistency loss during model training. The weights are adjusted by the values of loss terms on-the-fly. And the self-supervised loss is based on different augmentations on the same images, and introduced as weighted augmentation consistency (WAC) regularization. Moreover, the experimental results validate the proposed algorithm on two public 3D medical image segmentation datasets. Both fully- and partially-labeled scenarios are studied in the experiments.
strength_and_weaknesses: <Strength>

•	The paper is well-organized and well-written.

•	The experimental results supports the claims made in the manuscripts.

•	The submitted content is related to the application of deep neural networks in 3D medical image segmentation, which is highly relevant to the ICLR audience.

<Weakness>

•	The novelty of the proposed algorithm is limited. The WAC regularization is similar to the existing works like MixMatch or FixMatch.

•	The effectiveness and practical values of the proposed algorithm is unconvincing. Only a single 2D backbone neural network is 
studied in the experiments. And the results of the proposed algorithm are not compared to other state-of-the-art segmentation algorithms on the same datasets or public leaderboards.

•	It is lack of comparison of other semi-supervised learning algorithms in medical image segmentation.

•	Discussion or ablation studies are not sufficient (see below).

<Comments>

1.	How do the (nearly) failure cases look like, especially the cases that the proposed algorithm cannot fix?

2.	Does the batch size matter for the performance of Algorithm 1?

3.	Would using a different random seed in model training give similar results?

4.	It would be necessary to conduct ablation studies on the effects of different backbone networks.

5.	How are the losses computed in the Table 1 for the baselines with partial annotations? Are the losses calculated only on the annotated 
areas?

6.	In Table 2, ACR is able to help improve Dice score for “trim-ratio”. Why does it hurt the Hausdorff distance a lot? Where is the improvement of Dice score from?

7.	It would be better to evaluate the algorithm via the public testing leaderboard of the Synapse dataset.

8.	The experiments are based on a 2D segmentation networks for 3D image segmentation tasks. It is necessary to conduct the experiments with 3D networks as well.

clarity,_quality,_novelty_and_reproducibility: The manuscript is of moderate quality, with good clarity and limited novelties.
summary_of_the_review: My concerns about the manuscript are about its novelties and practical impacts. The novelties are limited because the WAC regularization is similar to the existing works like MixMatch or FixMatch and no comparison of such works were conducted. Moreover, only a single 2D backbone neural network is studied in the experiments. The results of the proposed algorithm are not compared to other state-of-the-art segmentation algorithms on the same datasets or public leaderboards.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method, namely adaptively weighted augmentation consistency (AdaWAC), by combining adaptive weighting and augmentation consistency regularization. It models the separation between sparse and dense labels as a subpopulation shift in the label distribution. It is demonstrated in the experiments that the segmentation performance has been improved in terms of DSC and HD95.  
strength_and_weaknesses: Strength
+ An interesting solution to consistency regularization
+ Comparison with hard thresholding algorithms
+ Good study on robustness

Weakness
+ no comparison with the soft thresholding algorithm

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The idea is interesting and new. Experiments are presented with details. 
summary_of_the_review: The paper presents an interesting idea and good experimental results. However, the evaluation needs improvement.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new regularization term (ADAPTIVELY WEIGHTED AUGMENTATION CONSISTENCY REGULARIZATION) for volumetric segmentation in medical images. Based on my reading, the paper can be understood, but the presentation can be improved for easier understanding and highlight the main contributions. A major weakness is the validation as the authors only use Unet and Trans-Unet as baselines, lacking of comparison with state-of-the-art approaches. The reviewers got some point in the concern that the authors shall compare with some state-of-the-art algorithms. Although comparison with Unet and TransUnet show some improvement, it is important to see if such improvement is consistent with latest methods.

Another concern is the experimental justification. Although the title suggests that the approach is for volumetric segmentation. The actual experimental results are for segmenting volumetric medical images as 2D slices. This is quite misleading. Some experiments based on 3D network shall be conducted.
justification_for_why_not_higher_score: See the summary
justification_for_why_not_lower_score: N/A.
Decision: Reject