Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper studies few-shot segmentation for medical images where labeled data is hard to obtain.  It proposed a stage-wise pipeline where pseudo labels are first proposed by noisy learning functions (LFs), then aggregated through a PGM, and finally used to train a segmentation model. The authors argue the two biggest challenges are "injecting knowledge for segmentation model" and "measuring accuracies of pseudo labels", and propose two modules to address them. The experiments have shown that the proposed method outperforming several few-shot learning baselines.

### Pros
The studied application is indeed very important and requires few-shot learning techniques. The 2nd challenge ("measuring accuracies of pseudo labels") is very insightful and indeed a huge problem for many methods. The technical solution of using PGMs is interesting and should be encouraged.  The experimental results are promising and encouraging.

### Cons
1. The abbreviations are abused, especially about FSL, LLL, and WS. As a reviewer from vision community, there is a common agreement of the definition of FSL and WS. LLL is rarely used and usually represents a very broad concept rather then anything specific. This paper has different interpretations of these concepts, which is fine, but should make it really clear what they stands for and what are the differences between them. In my opinion, this paper is simply a FSL setting and there are no need to introduce other two terms. 
2. The notations are also not super clear to me. In Sec.3.1, it's said $X_i \in \mathbb{R}^{I,J,K,T}$ is a tensor of $P$ pixels. However, $P$ doesn't go into any places of the notation. Moreover, what are $I,J,K,T$? They are never properly introduced. Also,  the superscript and subscript in $X_{i=1}^{i=n}$, $Y_{i=1}^{i=n}$ are just self-contradictory. 
3. In Sec.3.2 four LFs are discussed (Physical models, pre-training networks, ...). Are these methods ever used in the proposed framework, or compared to as baselines in experiments? Also, the reference of these methods are missing.
4. It seems like I miss something. There seem to be only one label function:  a few-shot image classifier. What are the multiple LFs? 
5. The design choices are not properly ablated. It's unclear which part contributes the most.
6. The FSL baselines are very limited: only PANet. In fact there are many orthogonal solutions of FSL. It's preferable multiple different baselines are compared  for this relatively new task.
7. Tab.2 results are obtained by directly comparing accuracy numbers for two different dataset, which doesn't make sense to me as they are not comparable. A better way to evaluate generalization performance is to compare "test a model pretrained using A on B" and "test a model trained on B". 

I have some doubts about the evaluation:
1. What's the reason for using Dice as evaluation metric rather than standard IoU, Pixel-Acc metrics? DICE is not popular for any segmentation work, and from the formulation it seems to be a worse metric than IoU (union area summed twice in the denominator).

I have some doubts about the motivation:
1. Why "Injecting knowledge via LFs" is a big challenge? Isn't it the standard pseudo-label or knowledge distillation algorithm? Multiple semi-supervised/weakly-supervised segmentation papers have studied this idea. To name a few:
*. Papandreou and Chen etal., Weakly- and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation. ICCV 2015
*. Wei etal., STC: A simple to complex framework for weakly-supervised semantic segmentation. PAMI

### Misc
1. Since all the experiments have been runned 3 times and the labeled samples are few (5), it's worth reporting the variance/std together with the mean.  Some results in Tab.1 are very close and hard to tell which method is better. 
2. Sec.3.2 `` "LLL labeling functions" writs "one-shot learning". However, in experiments it seems like 5 labeled images are used. Better to be rigorous.
3. What's the dimension of $\theta_i$ in Eq.(1)?
4. Fig.4 can be improved to be more consistent via using the same color for proposed method.
5. I would change Fig.1 example to use medical images.

### Questions
1. Have the authors tried to sample different labeled images? Will the results be different? Compared to using random seeds, I would report mean results over different samples.

********************************
The rebuttal is very detailed and contains many useful new baselines and results. The authors answer to reviewers' questions carefully and with great supportive details. I appreciate the efforts and many of my questions are answered, therefore I have increased my score. 

I'm not  an expert from medical image field. So it's a little bit hard for me to evaluate the significance of the reported results and proposed methods. However, I think this paper is well-motivated and current version has greatly improved over the first draft.  For the machine learning community, it's an interesting and important application and thus should be encouraged.
review_title: Interesting application; Limited technical contribution, unclear writing, and unconvincing experiment results. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents weakly supervised framework for image segmentation tasks with limited annotated data. It first builds several labeling functions with limited annotation and then uses probability graph to fuse the labels.  Last, the final output will be generated via CNN network.  There are two key challenges in such setting. First, how to build the labeling functions. Second, how to measure the accuracies from the labeling functions. 

However, these two challenges are not well described in this paper. First, the labeling functions are built on prior knowledges such as physical models, clustering and pretrained network.  Especially , for the pretrained network, it is not fair in few shot learning setting.  Second, the author claims she/he uses one image to train the LFs, since applying one image to train the network is difficult, it is very confusing on how to train the network. Third,  the author uses 60% data for training, it is not clear how to differentiate the proposed weakly supervised training with traditional supervised training (it is not uses true label?but the paper not describe it).  Morever, since FS-5 has reasonable good performance, the author should also compares with FS-60%.  

Since current paper has lots parts need to be cleared, the current draft can not be accepted. 


-----------------------------------------------------------------------------------------------------------------------------------
The author answers to reviewers' questions clearly with supportive details.  In addition,  the newer version has added useful new baselines and results. I have increased my score as all my concerns get cleared.

Overall,  I think  this is an interesting paper and should be encouraged. But I still has some concerns as the application is very limited on medical images. The author has compared all the baselines and shown that the generalization capability for the proposed method is similar to fully supervised method, which seems not doing well.  I would be more convinced if the proposed can be tested on few shot benchmark which are natural images. 



review_title: This paper provides a weakly supervised method for medical image segmentation. However, it has several weaknesses need to be addressed in the future. 

final_score: rating: 7: Good paper, accept
review_content: The paper addresses the problem of performing medical image segmentation in the limited label scenario, i.e. the case where there is only a small number of manual expert segmentations available.

The proposed approach to address this challenge uses a combination of two approaches: First, multiple labelling functions are learned by training CNNs with a single labelled example. The resulting labelling functions are then combined using a label fusion approach. This label fusion approach is based on a conditional model that focuses on areas of contention. To the best of my knowledge, the proposed approach is novel.  I found the methodology interesting and well motivated. The formulation and writing is clear. The main weakness to me is the evaluation. First, the selected baselines are not always very strong. For example, the best performing method for segmentation of ACDC (cardiac MRI) reported in Bernard et al., 2018 achieves significantly higher results than the baseline segmentation reported (”FS-all”). 

The authors mention that, depending on task difficulty, more sophisticated limited label learning models could be used as labelling functions. It would be good to see some discussion of this.
review_title: Interesting approach to medical image segmentation with limited labelled data, evaluation could be stronger

final_score: rating: 6: Marginally above acceptance threshold
review_content: CUT OUT THE ANNOTATOR, KEEP THE CUTOUT: BETTER SEGMENTATION WITH WEAK SUPERVISION
The authors make a clear explanation of the problem and rationale behind their approach, but a few experimental questions should be addressed concerning the implementation of the models and needs some additional results.

*Review
The paper is well written with significant time spent setting up a clear explanation of the background information and challenges.
The authors also adequately discuss alternate approaches and other methods currently used to address this current problem.
The raw numeric results of the proposed method are very compelling.

The authors make a good case for the significance of this work if it is able to surpass current methods in this space.
There are several questions that are still left un-addressed:

-Because the size of a neural network impacts the quality of the segmentation, it is important to know the size of the U-Net that they used for each model. The proposed method uses several CNNs in combination. So depending on the size of networks used in the proposed method (compared to the network size of the other chosen methods), the results might be biased based solely on the size of the networks used.

-The authors explain that the stacked networks will not add computational time due to the fact that they are each trained on only 1 image. This might be small since they were only given 5 training images, but this time will scale with more training data. Also, when stacked networks are applied to large datasets at test time, the additional computational times will add up. It would be nice to see a numeric comparison of computational times.

-One problem of learning from small datasets is guaranteeing balance and the quality of the training images provided (Changing out even a single image can impact model training). How variable are the results on what training images are provided? Authors state that they trained the models three times with different seeds, but it is assumed they used the same training images in every iteration.

-Authors state that random image augmentations were used in all models except FS-5 (unstated what was used for FS-all). Since image augmentation is important to apply to test set and generalization in general, could authors provide the rationale as to why baseline models weren’t trained with image augmentation?

- In Table 1 three trained models were averaged. It would be nice to also show deviations alongside the means so we can see the consistency of each approach.

- In Figure 4. Scaling of the axis could be misleading. Keep colors consistent for the proposed method.
review_title: The authors make a clear explanation of the problem and rationale behind their approach, but a few experimental questions should be addressed concerning the implementation of the models and needs some additional results.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors employ ConvNets trained with a very small set of reference labeled images, and introduce a new weakly supervised model that focuses on certain areas of the image to fuse these weak sources (i.e. their approach introduces a conditional model that focuses on areas where the labeling functions disagree, more finely tracking the performance in such areas). I think the paper is interesting and well-presented, but there are several important aspects that, in my humble opinion, harm my overall rating of the paper:

- Not clear to me what is the main original contribution and novelty of this work. They use generic and simple labeling functions to obtain weak supervision and, then, they use a probabilistic model to refine the accuracy for each labeling function. As far as I knoe, both approaches (knowledge injection via weak segmentation algorithms, and probabilistic graphical models to model segmentation masks) are not novel, so what is the main methodological contribution and novelty of this work?

- Why the authors do not compare with other state-of-the-art few-shot medical image segmentation approaches (like [1])? In fact, [1] is already cited in the paper, but the experimental comparison does not include it. Why is that? Without a more extensive experimental comparison with prior methods is difficult to elucidate the actual empirical contribution of the proposed method. 
[1] Ouyang, Cheng, et al. "Self-supervision with Superpixels: Training Few-Shot Medical Image Segmentation Without Annotation." ECCV 2020.

There are other details that should be clarified or improved in the paper:

- Are the authors employing one ("we use one-shot learning: we train a CNN from a single labeled data point for each LF.") or five ("five images are annotated by hand and used to seed LFs that weakly label the rest of the training set.") labeled images to weakly label their training data?

- The conclusions of the paper are too succint (just two lines and a half). I would encourage the authors to extend this section. 

- What fully supervised network ("FS-all") do the authors use in their experiments?
review_title: Maybe insufficient methodological contribution; limited experimental evaluation and comparison with state-of-the-art

Meta-review: This article proposes a weakly supervised few-shot learning method for medical imaging segmentation. While initially, the article presented several problems indicated by the reviewers, e.g., the explanation of the novelty and contributions, the explanation of the method, and the experimental evaluation, the authors made a great effort addressing most of the reviewers' comments and uploaded an updated version of the article. However, still, the evaluation part of the article is a bit weak. But the article contains interesting contributions. Accordingly, I recommend accepting the paper at ICLR2021.
Decision: Accept (Poster)