Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 3: Marginally above acceptance threshold
review_content: Summary: Author presented a neural network framework integrated with dense CRF model. Different with related work[25] which used CRF as a post-processing, NCRF model put the CRF at the top of CNN architectures which makes it possible to jointly train the CNN and CRF models, and decreases the computational cost as well. The experimental results also showed the higher classification and smoother visual results comparing to baseline methods.

Comments: This is a interest contribution that connect CRF directly to the CNN model. Unlike related approaches such as CRF-RNN, NCRF methods integrate CRF with CNN in a more simply and explicit way. It makes this framework more flexible to apply to other neural network architectures.


Questions: 

1. Hope to see a complete and successful comparison with previous work[24]. 

2. Please give a more detailed explanation of Eq. 3, the weight of smoothness kernel[15] become zero? 
How about the weight of position-related term in appearance term？ In Eq. 3, the position-related term is fixed to 1?
The original pairwise equation in [15] is more reasonable.

3. It is better to show the FROC in figure with other reference approaches, not only the mean score.

review_title: Simple and practical CRF-based CNN framework

final_score: rating: 3: Marginally above acceptance threshold
review_content: This work applies an end-to-end CRF-CNN for detecting metastasis in lymph nodes on high-dimensional whole-slide histology data for breast cancer. The CNN part is applied to individual image patches while the CRF work on the output of the CNN for a small grid of patches. The comprehensively validated results on an open data set supports the efficacy of the proposed method.

End-to-end training a CNN followed with a CRF to take into account the spatial correlation explicitly is not methodologically novel. However, applying this on the high-dimensional WSI seems well-motivated and, more importantly, showed improved performance due to the added CRF. The methods and experiments are well-organised and clearly-presented. The paper does not quite stress the small size of the grid of patches used for the CRF part. This is probably due to memory restrictions but it questions the need for large trainable regularisation. It would be interesting to better evaluate the impact of training the CRF as opposed to using CRF as a post-processing. This option would indeed limit the memory requirements.

The results are not reaching reported best numerical performance, this may invalidate the clinical relevance from the observed improvement due to the added CRF. Although likely given large validation/test data sizes, it helps to report the variance for the results from Table 1 and 2, in addition to appropriate statistical tests with p-values, to support the claimed significance. The numerical results, including the improvement has not been contexturised under the clinical application of interest.  

A few minor comments I do not think need to be fully addressed as a condition to accept this work:
“Significant discordance on detection results among different pathologists has also been reported.” This is an interesting area pointed out by the authors, but I am disappointed not to see any further discussion or quantitative analysis on this point with the obtained results, such as detailed discrepancy (e.g. size, CI, clinical consequence) between computational methods and those between machine and human, in this very task. 

Is there a quantitative measure to support that “smoother probability maps allow pathologists to focus more on potential tumor regions and less on isolated outliers” is beneficial?

VGG and AlexNet are fine, but more refined networks have been demonstrated in medical image segmentation tasks with better results, including using other loss functions other than CE.
It would be interesting to see the effect of varying grid sizes, especially the target is the detection and localisation instead of pure classification.

It would be interesting to see if even fewer iterations in mean-field approximation could achieve equivalent improvement, as random patch sampling may implicitly consider some level of spatial correlation.

Is excluding the background patches explicitly a better idea than weighting them down in loss?

It would be interesting to see if more explicit learning of localisation (predicting the coordinates) would encourage the network to learn the spatial correlation which was presumably omitted when not using CRF. 

review_title: A CRF-based approach to go beyond single patch processing in WSI analysis with a fair evaluation

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: Overall:
The paper presents an idea of using neural conditional random fields, a combination of neural networks with a conditional random fields on top, to learn spatial dependencies among parts of a slide (patches). The idea is not new but its application to medical imaging is very smart. The obtained results are solid and convincing. The analysis of the results is clear and sound.

Strengths:
+ The paper is very well-written and it is very easy to follow.
+ All ideas are clearly explained.
+ All experiments are solid and the results are analyzed in-depth.
+ The proposed idea is not new (see a remark below), however, it makes a lot of sense in the proposed approach.
+ The learning algorithm is simple (mean-field), nevertheless, it is fast and works well in practice. It is especially important from the practical point of view.

Remarks:
* Major
- How many times were the experiments repeated? The differences between the reported results in Table 1 are small and, therefore, it would be beneficial to provide also a standard error.
- The idea of combining neural networks and conditional random fields is not new and could be traced back to, for instance, the following paper:
Do, T. and Artieres, T. (2010). Neural conditional random fields, AISTATS 2010.

I suggest to refer to this paper and discuss it briefly in the introduction.

* Minor
- Since the proposed idea is closely related to the multiple-instance learning, it would be interesting to discuss it briefly in the introduction or the conclusion. 
review_title: A very interesting paper

Decision: Oral