Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper addresses the problem of learning segmentation models in the presence of weak and strong labels of different types. This is an important problem that arises while learning in data-scarce settings. The presented approach optimizes jointly over the various types of labels by treating each one as a task. They extend the U-net architecture to incorporate the different tasks.

Prior work:
There has been other work on incorporating multi-resolution or different types of labels. Here is one that can be cited:
Label super-resolution networks (https://openreview.net/forum?id=rkxwShA9Ym)

Major comments:
- The motivation for the specific structure of the multi-task blocks is not clear
- The object boundaries labels can be noisy (i.e s(2) can have noise). How does model deal with this?
- Is it the case that every image in I_3 is completely labeled - i.e all segments/classes marked?
- The assumption that s(3) is independent of s(1) and s(2) is not true. Instead of constraining the model to learn masks that respect the various types of labels, it seems they learn from each source independently. It is not clear how the sharing of parameters in the multitask block helps.
- Can they comment on the applicability of the prior work suggested above?

Minor comments:
- How do the rough labeling tools work on biomedical data where the objects are more heterogenous patterns where different labels can have very different distribution of pixels. How well will their method generalize in such settings?
- Can this work be used for segmentation and prediction on crop data?

Results:
- It seems as if the improvement over the PL baseline (pseudo labels) is incremental? Can the authors provide error bars so the reader knows what the significance of the results is?
- Can they give a more thorough comparison in terms of human effort? It is interesting to note that only 2 images give 0.82. Would 3 images give 0.94? They need to show the trade-off between additional effort vs gains in performance.
- What is the performance of MT U-net without the SL images (i.e without task-3)? Table-2 does give some intuition, but authors should add another row with multitask WL
- Table-3: How well does MDUnet do with 9.4% SL data?



review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a method for semantic segmentation using "lazy" segmentation labels. Lazy labels are defined as coarse labels of the segmented objects. The proposed method is a UNET trained in a multitask fashion whit 3 tasks: object detection, object separation, and object segmentation. The method is trained on 2 datasets: air bubbles, and ice crystals. The proposed method performs better than the same method using only the weakly supervised labels and the one that only uses the sparse labels.

The novelty of the method is very limited. It is a multitask UNET. The method is compared with one method using pseudo labels. However, this method is not SOTA. Authors should compare with current methods such as:
 - Where are the Masks: Instance Segmentation with Image-level Supervision
 - Instance Segmentation with Point Supervision
 - Object counting and instance segmentation with image-level supervision
 - Weakly supervised instance segmentation using class peak response
 - Soft proposal networks for weakly supervised object localization
 - Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation
These methods can use much less supervision (point-level, count-level or image-level) and may work even better.

The method should be compared on standard and challenging datasets like Cityscapes, PASCAL VOC 2012, COCO, KITTI...


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The submission presents a neural network for multi-task learning on sets of labeled data that are largely weakly supervised (in this case, partially segmented instances), augmented by comparatively fewer fully supervised annotations.
The multiple tasks are designed to make use of both the weak as well as as full (‘strong’) labels, such that performance on fully annotated machine-generated output is improved.

As noted in the related work section (Section 2), multi-task methods aim to use benefits from underlying common information that may be ignored in a single-task setting. The network presented here is quite similar to most of these multi-task approaches: a common feature encoder, and partially distinct feature decoding and classification parts.
The (minor) novelty mainly comes from the distinct types of weak/strong annotation data fed here: instance scribbles, boundary scribbles, and (some or few) full segmentations. 

The submission is overall well written and provides sufficient clarity and a good overview of the approach.
Section 3 presents a probabilistic decomposition of the proposed architecture. With some fairly standard assumptions and simplifications, the loss in Eq. 3 becomes rather straightforward (weighted cross entropy)
The actual network architecture described in Section 3.2 takes a standard U-Net as a starting point and modifies it in a fairly targeted way for the different expected types of annotations. These annotations (Section 3.3) are cheaper than full labels on a same-size dataset; it is not completely clear, however, if the mentioned scribbles need to capture each instance in the training set, or if some can also be left out. Without this being explicitly mentioned, I will assume the former.

The experimental evaluation is done reasonably well, although I am not familiar with any of the presented data sets. The SES set seems to be specific to the submission, while the H&E data set has been used at least one other relevant publication (Zhang et al.). My main issue here is that at least on the SES set, which does not seem to be that large, the score difference is not that big, so dataset bias could play some part (which is unproven, but so is the opposite).
Experimental evaluation does not leave the low-number-of-classes regime, and I’m left wondering how the method might compare on a semantically much richer data set, e.g. Cityscapes. Finally, unmodified U-Net is by now a rather venerable baseline, so I’m also wondering how the proposed multi-task learning could be used in other (more recent) architectures, i.e. whether the idea can be generalized sufficiently.

While I think the ideas per se have relatively minor novelty, the combination seems novel to me, and that might warrant publication.
review_title: Official Blind Review #3

Meta-review: The paper proposes an architecture for semantic instance segmentation learnable from coarse annotations and evaluates it on two microscopy image datasets, demonstrating its advantage over baseline. While the reviewers appreciate the details of the architecture, they note the lack of evaluation on any of popular datasets and the lack of comparisons with baselines that would be more close to state-of-the-art. The authors do not address this criticism convincingly. It is not clear, why e.g. the Cityscapes or VOC Pascal datasets, which both have reasonably accurate annotations, cannot be used for the validation of the idea. If the focus is on the precision of the result near the boundaries, then one can always report the error near boundaries (this is a standard thing to do). Note that the performance of the baseline models is far from saturated near boundaries (i.e. the errors are larger than mistakes of annotation).

At this stage, the paper lacks convincing evaluation and comparison with prior art. Given that this is first and foremost application paper, lacking some very novel ideas (as pointed out by e.g. Rev1), better evaluation is needed for acceptance.
Decision: Reject