Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: This paper describes a solid piece of work for detecting and classifying cells of 5 different cell types in placenta H&E slides. The system contains two parts: detecting nuclei as a cell candidate detector, and classifying the detected cells in one of 5 types.

The data set was collected by the authors. 10 slides were scanned and tiles of 1600x1200 were cropped. 13k nuclei were annotated, 9.k cells were classified. The data set is not shared as far as I can see, although the github repo with the code, that is shared, contains some images. The authors could clarify if they intend to share the data. I think it is likely results can be substantially improved, and sharing the data could allow the community to assist in this effort. 

For the detection step two published approaches have been attempted, Retinanet, ref 22, and FCNN, ref 25. The authors choose Retinanet because FCNN has the issue that often two nuclei are detected in the nucleus is elongated. This is shown in Figure S2. I think it should be fairly easy to merge such double detections into one.

The results of the nuclei detection phase, with Retinanet, are not clearly described. Table 1 is not informative, or at least not easy to interpret. The text mentions a generalization performance of 0.66, but this is not an entry in the Table. The standard for evaluating detection tasks is to show FROC curves. My impression is that the performance is not very good. Figure 2 contains 9 cells that are missed. It seems the system consistently fails on detecting smaller nuclei. There is also one example where a detection bounding box covers two smaller reference boxes. How is the hit criterion defined, i.e., how do you handle this? Papers on detection tasks in medical image analysis tend to describe this in detail. There are no examples of false positive detections, surely the system produces these occasionally too. So Figure 2 alone is not a very good way of illustrating results.

Considering all the missed detections of Retinanet, I wonder if FCNN with a postprocessing step might not be better, or al least have additional value if the two approaches were ensembled.

The description of the classification step results suffers from similar issues. Why show a confusion matrix in the main text of only Inception V3, followed by a table of the ensembling experiments, and have another confusion matrix in the supplementary figures? You could easily have the two matrices side by side. You write "Misclassification error types are comprehensively visualized in Figure S4." but this figure is a confusion matrix and is not visualizing anything. It would be very interesting to see images of misclassified cells. 88% or 89% accuracy indicates there may be substantial space for improvements.

It would be great to compare the results with independent human expert classifications.

The subsequent results are preliminary, the discussion lists sensible next steps. To summarize the work as "a computational pipeline to comprehensively analyze placental histology imaging" is an overstatement, you have made a promising start in detecting and classifying cells, a comprehensive analysis of placental histology would be the next step. 

Pro:
* Solid work
* Very well written; a joy to read
* Probably a first attempt to analyze placental H&E slides in this manner; novel and relevant application

Con:
* The title is misleading. Deep Cellular Phenotyping sounds fantastic, but that is not covering the content of the paper. In general, the paper is overselling.
* Results of the detection seem rather poor and are not compared with human performance
* Result of the classification are not critically analyzed and not compared with human performance 
review_title: Solid work on cell detection and classification

final_score: rating: 2: Marginally below acceptance threshold
review_content: The authors propose a deep learning--based approach to the analysis of plactental cells, which consists of a nuclei localization and a nucleus classification step.

The topic is well introduced and motivated. The contributions of this work are not clearly stated. From my point of view however, the contributions are the application and comparison of existing deep learning approaches to a new data set. The data set seems to have a rather small but still reasonable size to train and test the proposed approach. The authors are aware of the limited size and also its imbalance and tackle these issues explicitly using transfer learning and bootstrapping.

For the nuclei localization task, RetinaNet and FCNN-Unet were applied and compared. The comparison is made in textual form, it would have been good to also state quantitative results for the FCNN-Unet. The evaluation of the RetinaNet is comprehensive, but the quality measure mAP should have been explained or at least referenced. This measure is assumed to be precision, thus a measure of recall would have been helpful as well.
The authors also compared two data augmentation methods (with and without stain transformation). They state that performance was improved by applying stain transformation, however, for the relevant cut-offs of the posterior probability, the improvement is not or only very marginally present.
The major criticism here is that Table 1 shows validation quality instead of test quality. Although test quality is given in the text, this should have been the other way round, as validation quality is much less relevant to get insights into the quality of the approach. Additionally, the validation quality is at 0.80 while test quality is at 0.66. This indicates severe over fitting to the validation set, as the optimal model is chosen based on the validation set. However, this issue is not mentioned in the paper and the discussion just states that the approach detects nuclei robustly.

For the nuclei classification task, the InceptionV3 network is applied and produces good results. The authors visualized that in a confusion matrix. Additionally, an ensemble of the InceptionV3, an InceptionResNetV2 and an Xception network is applied to further improve performance. The performance however improved only marginally at the cost of increasing the complexity of the approach.
Additionally, the authors incorporated t-SNE to generate a visualization of the classes in the deep feature space. In this visualization, the classes are well separable. The visualization is very interesting and helpful to corroborate the quality measures of the nuclei classification. With the visualization, the authors were able to discovered that the FIB class in fact consists of two subclasses and provide a medical explanation.

Summary of pros(+) and cons(-):
+ limits of data set noticed and tackled
+ Use of t-SNE visualization interesting and validates learned features
- Quality measures are not described
- Validation quality is highlighted instead of test quality
- Clear over fitting not explained or addressed in any way
review_title: Application of DL to an interesting task, but evaluation does not meet the standards

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: The authors suggest models for localizing (RetinaNet vs. FCNN) and classifying (Inception V3 + Inception ResNetV2 + Inception) placental cells. Care was taken on evaluating preprocessing steps (stain transformation), boosting performance using ensembles, and on balancing the data using bootstrap, class reweighting, and downsampling. The embedding of the learned classification representations is analyzed with tSNE and subclusters are visualized to gain insights into the subpopulations of the data.

I enjoyed reading this paper and am satisfied with the clarity and structure of the manuscript. The reported results look encouraging and the choice of model and parameters seem reasonable. I have a couple of remarks:

- I appreciate that the code will be publicly available. 

- It seems that the test images were extracted from around 8 individuals (approx. 1119/(13179/91)). Is this correct? This would be quite a small number of test individuals. If true, I would suggest to perform cross validation for improving the evaluation.

- How was the confidence of the annotators assessed (Section 3.1.2)?

- How were the hyperparameters (e.g., perplexity) for the tSNE visualization selected (Figure 5)?

- I would recommend to entirely leave out the word "comprehensively" and leave this assessment up to the reader.
review_title: Review

Decision: Poster