Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Annotation in the clinical dataset is hard to prepare, so it is important if self-learning algorithms can be applied to medical data which can do few-shot learning. In this paper, authors put forward a method to do segmentation which makes use of pretraining with contrastive learning with metadata, and then better fine tuning with self-paced learning. The second step is necessary to mitigate meta data noise. This paper shows experiments on three public datasets. With a few annotated data, the algorithms still show comparable performance with the fully supervised segmentation algorithm.
main_review: The major strength of this paper is the soundness of empirical methodology used to validate the empirical approach. Authors showed intensive experiments and persuasive results to demonstrate the advance of the proposed methods compared with prior works (table 2 and table 3).
The idea of this paper is also with great clinical impact: right now, fully supervised segmentation methods although show good performance, depends heavily on the expensive annotations. It is great to explore the self-/semi- supervised learning algorithms to reduce such strict requirement on data acquisition.

In this paper, authors discussed about the previous related work about self-supervision, contrastive learning and self-paced learning, and later clearly discussed how the proposed method was built upon the prior work.

This work is reproducible. They used public datasets to train and evaluate the algorithm, and the authors promised to make the codes public available if the paper got accepted.

Paper Strength:
+ A new self-paced strategy is proposed for contrastive learning by assigning importance weights to positive sample pairs. With self-paced strategy, the model learns from easy samples in the beginning and gradually add noisy/hard samples.
+ Meta-labels are incorporated in the contrastive learning process, which is reasonable and effective. Using all three meta-labels achieves the best performance.
+ Experiments show that the proposed method outperforms several existing methods on 3D medical image segmentation in the semi-supervised setting. Ablation experiments also demonstrate the effectiveness of the proposed self-paced learning strategy and the use of meta-label in contrastive learning.

limitations_and_societal_impact: One weakness is in the experiment design. In this paper, table 2, authors showed the ablation study of all combination of loss functions on all three datasets, and selected “SP-Con (both) + Mean Teacher” as the best model to report. It would be better if this ablation study and model selection was done only with one of the three datasets, and only report the best model’s performance on the rest, so that it can follow the best practice of model selection. In table 1 (section 5.1) the selection of hard vs. linear self-paced regularization is a good example of parameter selection.
Another limitation for the current submission is that authors did not discuss whether all organs can be segmented with the proposed algorithm design. Right now, from the dataset point of view, it works with prostate (PROMISE12) and heart (ACDC and MMWHS). Further experiments and discussion with different organs could better support the idea that proposed method could have a good clinical impact.

Overall, this paper is well written. However, it could be better if an algorithmic workflow figure is provided. Right now, although this paper put much effort to discuss about the methodology, it is still not straight forward enough for readers to understand how this algorithm works from input to contrastive learning, and to the self-paced learning. 
In table 1, performance on ACDC is shown with percentage. It would be better, if in the table title, the authors could explain what are these percentage numbers: are they dice or auc or any other metrics.

More graphics in the main body, instead of putting them in the supplementary material, could better help readers understand the idea. For example, choosing one row from Figure 1, Figure 2, and Figure 3 in the supplementary material respectively as a new figure, and moving it to the main body as examples of “meta-labels”. A selection of Figure 6 in the supplementary material could also be shown in the main body, to visually show the segmentation performance.

Additionally, 
- There are many hyper-parameters used in the paper, which are determined using grid search on the validation set. But they are not specified, including the loss weights in Equation (14) and (15). How is the temperature factor set in Equation (2)?
- The output of the encoder should be a 2D feature map. Why use a non-linear head to project it to 256 dimensional vector? Is the vector reshaped to a 2D feature map before fed to the decoder?
- It appears that both the training and the inference are done for 2D slices individually. The 3D continuity information between slices is lost. Such information is important for 3D medical image segmentation, although it might be outside the scope of this paper. It would be better to include a 3D segmentation model for comparison.
- The experiments show that the proposed method is useful when there are few training data (1 to 7 scans). How does the proposed method work when more labeled data are available? It should be studied since they are available in the three datasets. Besides, the training scans are randomly selected from the available labeled data. How does the random selection affect the performance?
- Table 1 does not have the baseline results for reference.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes the use of meta labels and self paced learning to improve performance of medical image segmentation when using contrastive pre-training. Through results on three different medical image segmentation tasks, the authors demonstrate the efficacy of the proposed method.

main_review: Originality:
While the paper is mostly built on top of https://arxiv.org/pdf/2006.10511.pdf, the addition of self paced learning and meta labels are interesting improvements on top of the prior work.

Quality and clarity:
The paper is mostly well-written and easy to follow. The authors demonstrate good experimental rigor with experiments on multiple datasets and good ablations

Significance:
Its challenging and expensive to obtain labels for medical segmentation. Efforts like this which aim to improve the sample efficiency for medical segmentation are great research directions in general

Detailed Comments:

> However, thanks to large datasets like ImageNet [11], the amount of labeled data for training has increased and, in such setting, pre-training often hinders performance [36].

This comment is a bit confusing. You may want to clarify that while supervised pre-training on Imagenet is still the norm for developing many vision models, this comment is for unsupervised pre-training

> Contrastive training as a regularization loss in semi-supervised learning. 

Please cite related work - https://arxiv.org/abs/2102.06605
There is a lot of recent work on self supervised learning for medical imaging and segmentation. Related work section is missing those papers.
Eq 2 is pretty much an instantiation of the supervised contrastive objective. Please cite this in the text

>  These update rules in Equ. (7) can be explained intuitively. For a given γ, the hard threshold rule only considers confident pairs with Lij ≥ γ and ignores the others

There seems to be an error here and i think this should be Lij < y

>  contrastive learning has mostly been used for pre-training the model. Here, we show that it can further boost results in a semi-supervised setting.

Please cite https://arxiv.org/abs/2006.10029 here

> How is gamma in eq 16 related to the gamma derived from equation 2? It is not clear they relate.

Please substantiate the relationship here.

It is difficult to read the results in table 3. Please bold the best ones

> Please add more details as to how the self paced objective was in the semi-supervised setup without the contrastive objective. 

Assume the loss in this case would be wrt individual training examples?

limitations_and_societal_impact: Major concern regarding the work is 
- why the experimentation setup doesn't exactly match https://arxiv.org/pdf/2006.10511.pdf? The papers seem to use different number of training examples
- it would be good to see the demonstration of self paced learning beyond segmentation or medical tasks. Without the results, it is difficult to ascertain the generality of the proposed method(s)

If the authors can address these concerns satisfactorily, will be happy to reconsider my scores.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose using “metal labels” in pertaining image encoders on unlabeled datasets. They use the metal-labels in pertaining the encoder, and also as an auxiliary task in training. They show their method works on multiple medical datasets.
main_review: The value of clustering 2D slices in a 3D volume seems not as helpful as just learning the model on 3D data in the first place (a 3D implementation of the U-net exists). 

More information could be provided on what meta-labels were included. It’s possible that manually viewing them with some domain expertise could already select those that are unlikely to contribute any signal.

Best performance is only achieved by combining everything, compared with the baselines that quite possibly focus on a single innovation. Compared with the other individual methods tried, the best performance is achieved by the baselines in some cases. So the paper becomes more about how much one can push performance than introducing a single innovation.
limitations_and_societal_impact: Limitations are short - I’ve mentioned a number above, but these are not discussed in the limitations section.

The social impact of the problems tackled is clear, as the authors have focused on medical use cases. I don’t think it is as hard to collect unlabeled medical data as they are claiming (there are many large open source datasets), but the quality of labels is poor so the problem remains an important one.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors show how to use weak labels to boost segmentation of medical datasets.
The idea hinges on a combination of contrastive learning with a new self-paced learning approach that gives preference to more confident labels (and therefore not being mislead by label noise or irrelevant weak labels).
main_review: I enjoyed reading the paper. It is written very clearly, addresses very relevant issues (learning with limited data, utilisation of weak labels), and convinced me by presenting a solid set of experiments as well as ablations/baselines.

The theoretical contributions are, as they are presented, useful and novel. Contributions might not be seen as earth shattering, are likely not the start of a new paradigm, but can still be labeled as solid. The main text points at all hyper-parameters and how good values have been found (grid search, mostly). The range for sensible $\gamma$ values, one of the more critical parameters to choose, are discussed in detail, and presented understandably.

While I'm fairly confident that I understand the manuscript very well, I want to point out that I do not know the related work very well and would not know if some work exists that was not mentioned/cited by the authors.

In summary, the manuscript presents a solid method that shows excellent performance wrt. to competing methods, even if only very limited training data is available. Contributions are good, presentation excellent.

Minor comments:
line 144 -- I would have preferred to find a more complete list of used meta-labels.
line 159 -- What is g(.)? Could the authors be more concrete about it?
line 170 -- Superfluous "A".
line 182 -- missing "y" in "by".
page 5 -- l_{i,j} is used with inconsistent font
line 198 -- shouldn't it be l_{i,j} <= \gamma ???
line 218 -- I was wondering how critical is it at the end to find the right \lambda_k? (How fine do you need to grid search, how far can you be off before results drop significantly in quality?)

limitations_and_societal_impact: A social impact and limitations sections exists and I agree with the authors comments in it.
review_title: null

Meta-review: The reviewers broadly agreed that the overall motivation for the work (reducing the dependency on labels in medical image segmentation) and the novelty and thorough evaluation of the proposed method are of general interest to the community. There were a few minor clarity points brought up that should be addressed in the revised version, and the authors should also include the additional experimental results from the discussion.  
Decision: Accept (Poster)