Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes a new type of weakly supervised clustering / multiple instance learning (MIL) problem in which bags of instances (data points) are labeled with a "unique class count (UCC)*, rather than any bag-level or instance-level labels.  For example, a histopathology slide (the bag), consisting of many individual pixels to be labeled (the instances) could be labeled at the bag level only with UCC = 1 (for only healthy or only metastatic) or UCC = 2 (for mixed / border case).  The paper then proposes an approach for clustering instances based on the following two-step approach: (1) a UCC model is trained to predict the UCC given an input bag, and (2) the features of this learned UCC model are used in an unsupervised clustering algorithm to the get the instance-level clusters / labels.  The paper also provides a theoretical argument for why this approach is feasible.

Overall this paper proposes a (1) novel and creative approach that is well validated both (2) theoretically and (3) empirically, and relevant to a real-world problem, and thus I believe should be accepted.  In slightly more detail:

- (1) MIL where we are given bag-level labels only is a well-studied problem that occurs in many real world settings, such as the histopathology one used in this paper.  However to this reader's knowledge, this is a new variant that is both creative and motivated by an actual real-world study, which is exciting and alone warrants presentation at the conference in my opinion.

- (2) The theoretical treatment is high-level, but still serves a clear purpose of establishing feasibility of the proposed method- this modest and appropriate purpose serves the paper well.

- (3) The empirical results are thorough---e.g. the use of the two loss components for the UCC model are appropriately ablated, there are a range of baseline approaches compared to, multiple evaluation points are provided (i.e. both UCC prediction and final clustering metrics), and a real world use case is presented--and the results are impressive.

Some comments to improve the paper:
- The connection of the proposed UCC approach to the motivating histopath example should be explicitly stated upfront to help the reader understand how this method could be used and how it makes sense!
- It seems that the KDE element of the UCC model was chosen in part to enable the theoretical analysis?  If so, this should be clearly stated to help the reader understand the design rationale.
- Either way, it seems that a different approach than the KDE layer could have been taken- this should be added to the ablation experiments
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a MIL clustering method. The proposed MIL setup is called "unique class count (ucc)", this is, for a bag os samples ucc is the number of clusters in the bag. The method learns the features of the samples using two losses: an autoender loss and the ucc loss. Once trained on a dataset the method can perform clustering on classes (classify) better than a fully unsupervised clustering algorithm and worse than a fully supervised model. The method is evaluated on MNIST, CIFAR10, CIFAR100  and on binary breast cancer segmentation.

The main table of the paper (Table 1) compares the results of the proposed methods only with unsupervised methods and a fully supervised one. The results are reasonable. However, not a fair comparison. More fair comparisons can be found in Table 6 of appendix C.6 page 22 where the method is compared to semi-supervised methods. In this case, the proposed method performs worse than the semisupervised methods.

UCC model uses bags of sizes from 1 to 4. Assuming uniform distribution, 25% of the samples are fully labeled. The semi-supervised methods from Table 6 how many labeled samples do they use? Having that small bag samples the problem is quite easy. Moreover, during training, I guess that the same sample can go to different bags. Is this right? If so, the annotation time would be very big. And also, one could trivially get the full label of each sample.

The article is excessively long. 10 pages for the main article and 12 extra pages for supplementary material which are needed for understanding the paper. Many of the definitions are redundant and there is excessive detail in explanations that can be expressed more simply. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors present a novel weakly-supervised multiple instance learning model based on a bag level label called unique class count(UCC). The core content of the method is to learn mapping between bags and their associated bag level ucc labels and then to predict the ucc labels of unseen bags. 
Positive:
(1) The authors use the unique class count(UCC) in the bag as a weak, bag level label to design a deep learning based UCC model for extracting features and clustering the individual instances in the unseen bags. They also construct a new optimization objective function by combining ucc loss and autoencoder loss. This is a novel weakly-supervised clustering method rarely seen.
(2) In this paper, a large number of experiments show the effectiveness of the algorithm in different data sets and semantic segmentation of breast cancer metastases.
Negative:
(1) The authors use KDE method to estimate the distribution of extracted features as the input of distribution regression module, but it is not clear how the description of KDE method updates on the parameters of the auto-encoder.
(2) The proofs of some propositions in Appendix B are very obvious and redundant. The authors’ proofs are not enough to prove whether the designed UCC classifier is perfect.
review_title: Official Blind Review #3

Meta-review: The paper proposes a weakly supervised learning algorithm, motivated by its application to histopathology. Similar to the multiple instance learning scenario, labels are provided for bags of instances. However instead of a single (binary) label per bag, the paper introduces the setting where the training algorithm is provided with the number of classes in the bag (but not which ones). Careful empirical experiments on semantic segmentation of histopathology data, as well as simulated labelling from MNIST and CIFAR demonstrate the usefulness of the method. The proposed approach is similar in spirit to works such as learning from label proportions and UU learning (both which solve classification tasks).
http://www.jmlr.org/papers/volume10/quadrianto09a/quadrianto09a.pdf
https://arxiv.org/abs/1808.10585

The reviews are widely spread, with a low confidence reviewer rating (1). However it seems that the high confidence reviewers are also providing higher scores and better comments. The authors addressed many of the reviewer comments, and seeked clarification for certain points, but the reviewers did not engage further during the discussion period.

This paper provides a novel weakly supervised learning setting, motivated by a real world semantic segmentation task, and provides an algorithm to learn from only the number of classes per bag, which is demonstrated to work on empirical experiments. It is a good addition to the ICLR program.
Decision: Accept (Poster)