Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a semi-supervised method to classify and segment WSI histological images that are only labeled at the whole image level. Images are tiled and tiles are sampled and encoded into a feature vector via a ResNET-50 pretrained on ImageNET. A 1D convolutional layer followed by a min-max layer and 2 fully connected layer compose the network. The conv layer produces a single value per tile. The min-max layer selects the R min and max values, which then enter the FC layers. A multi-instance (MIL) approach is used to train the model by backpropagating only instances that generate min and max values at the min-max layer. Experiments are run on 2 public datasets achieving potentially top performance. Potentially, because all other methods supposedly make use of segmentation labels of tumor, while this method only uses the whole image label.

Previous publications have used MIL training on tiles with only top-level labels [1,2] and this is essentially an incremental improvement on the MIL approach by using several instances (both min-negative and max-positive) instead of a single instance for backprop, as described in [3]. So, the main contribution here, is to adapt min-max MIL to the histology domain. Although the result are good and the method interesting, I think that the technical contribution is a bit thin for a ML conference and this paper may be a better fit for a medical imaging conference.

The paper is well written and easy to understand. 



[1] Hou, L., Samaras, D., Kurc, T. M., Gao, Y., Davis, J. E., & Saltz, J. H. (2016). Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2424-2433).
[2]  Cosatto, E., Laquerre, P. F., Malon, C., Graf, H. P., Saito, A., Kiyuna, T., ... (2013). Automated gastric cancer diagnosis on H&E-stained sections; training a classifier on a large scale with multiple instance machine learning. Medical Imaging, 2. 2013.
[3] Durand, T., Thome, N., & Cord, M. (2016). Weldon: Weakly supervised learning of deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4743-4752).
review_title: Interesting MIL approach, lacks technical depth for this conference

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a deep learning (DL) approach (pre-trained CNNs) to the analysis of histopathological images for disease localization.
It correctly identifies the problem that DL usually requires large image databases to provide competitive results, while annotated histopathological data repositories are costly to produce and not on that size scale.
It also correctly identifies that this is a daunting task for human medical experts and therefore one that could surely benefit from the use of automated methods like the ones proposed.

The study seems sound from a technical viewpoint to me and its contribution is incremental, as it builds on existing research, which is correctly identified.
Results are not always too impressive, but authors seem intent on making them useful for pathogists in practice (an intention that is always worth the effort).
I think the paper would benefit from a more explicit statement of its original contributions (against contextual published research)

Minor issues:
Revise typos (e.g. title of section 2)
Please revise list of references (right now a mess in terms of format, typos, incompleteness
review_title: Down-to-earth practical application of DL in a medico-clinical context

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors approach the task of labeling histology images with just a single global label, with promising results on two different data sets. This is of high relevance given the difficulty in obtaining expert annotated data. At the same time the key elements of the presented approach remain identical to those in a previous study, the main novelty is to replace the final step of the previous architecture (that averages across a vector) with a multiplayer perceptron.  As such I feel that this would be interesting to present if there is interest in the overall application (and results of the 2016 CVPR paper), but not necessarily as a novel contribution to MIL and histology image classification.

Comments to the authors:

* The intro starts from a very high clinical level. A introduction that points out specifics of the technical aspects of this application, the remaining technical challenges, and the contribution of this work might be appreciated by some of your readers.
* There is preprocessing that includes feature extraction, and part of the algorithm that includes the same feature extraction. This is somewhat confusing to me and maybe you want to review the structure of the sections.  You are telling us you are using the first layer (P=1) of the ResNet50 in the method description, and you mention that you are using the pre-final layer in the preprocessing section. I assume you are using the latter, or is P=1 identical to the prefinal layer in your notation?  Tell us. Moreover, not having read Durand 2016, I would appreciate a few more technical details or formal description here and there.  Can you detail about the ranking method in Durand 2016, for example?
* Would it make sense to discuss Durand 2016 in the base line methods section? 
* To some degree this paper evaluates WELDON (Durand 2016) on new data, and compares it against and an extended WELDON algorithm called CHOWDER that features the final MLP step. Results in table 1 suggest that this leads to some 2-5% performance increase which is a nice result.  I would assume that experimental conditions (training data, preprocessing, optimization, size of ensemble) are kept constant in between those two comparisons? Or is there anything of relevance that also changed (like size of the ensemble, size of training data) because the WELDON results are essentially previously generated results? Please comment in case there are differences. 
review_title: Interesting application and results with incremental innovation on exististing work

Meta-review: Authors present a method for disease classification and localization in histopathology images. Standard image processing techniques are used to extract and normalize tiles of tissue, after which features are extracted from pertained networks. A 1-D convolutional filter is applied to the bag of features from the tiles (along the tile dimension, kernel filter size equal to dimensionality of feature vector). The max R and min R values are kept as input into a neural network for classification, and thresholding of these values provides localization for disease / non-disease.

Pro:
 - Potential to reduce annotation complexity of datasets while producing predictions and localization

Con:
- Results are not great. If anything, results re-affirm why strong annotations are necessary.
- Several reviewer concerns regarding novelty of proposed method. While authors have made clear the distinctions from prior art, the significance of those changes are debated.

Given the current pros/cons, the committee feels the paper is not ready for acceptance in its current form.
Decision: Reject