Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 1: Strong rejection
review_content: In this paper the authors investigate segmentation of liver tumor from CT images, propose a weighted loss computation to handle the data imbalance issue, and a random forest post-processing to reduce false positives. In the evaluation, segmentations from LiTS challenge ground truth, a medical technical radiology assistant, and the proposed automated method are compared.

The contribution of this paper is rather unclear:
- The training of the proposed method requires additional liver masks in the training set (not clear how they are generated in this paper), however the tumor segmentation performance is inferior to the best performing entries which automatically segment both liver and tumor. Therefore the motivation of having the masks and using the proposed loss computation is not well-demonstrated.

- For the proposed random forest post-processing step, it is unclear what the features are used, if the features are extracted in 3D. Also, shape-based evaluations are necessary in order to show that the shape-based features are actually helpful (LiTS challenge adopts a few shape-based evaluation criteria, those could be considered in this paper).

-The algorithm for output/reference tumor correspondence establishment is interesting, it would be great to have further comments to link the evaluations to the clinical decision making/risk assessment in practice.

review_title: Decent evaluations comparing challenge ground truth, manual segmentation, and automated segmentation; but overall the paper is incremental with unclear contribution

final_score: rating: 2: Marginally below acceptance threshold
review_content: Overall:
The paper tackles a problem of automatic liver tumor segmentation in abdominal CT scans using a combination of a neural network (U-Net) and a random forest classifier. In general, the paper is very well-written and easy to follow. The obtained results are solid. The main drawback of the paper is its clear orientation on the MICCAI challenge. The paper combines well-known methods in an interesting manner, however, it does not introduce any novelty from the methodological point of view.

Strengths:
+ The method is well-thought and it is clear that the authors considered pros and cons of different architectures.
+ The considered problem is of great practical significance.
+ The proposed approach is well-suited to the problem.
+ The obtained results are impressive (the 3rd place in the MICCAI 2017 challenge).

Remarks:
* Major
- I have some doubts how the final structure of the model was chosen. After reading the following sentence: "We observed that some of the neural network outputs corresponded to false positives (...)", I wonder whether these observations were done basing on training or test data.
- Why did the authors pick a random forest classifier instead of other classifiers like SVM? What was the motivation for that?
- In general, the paper aims at a possibly best performance instead of developing a new method. Because of its pure engineering character, I decreased the overall assessment by 1 point.

* Minor
- Personally, I dislike that the authors add a random forest classifier on top of a trained neural network. It results in a two-stage learning algorithm where first a neural network is trained, and then a random forest is trained on extracted features from the neural network. However, this remark is really minor since the final results are in favor of the proposed approach. In general, it would be an interesting direction for future research to understand why a neural network alone cannot achieve similar or better results as the random forest classifier.
review_title: A solid paper but it lacks novelty

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: The authors present a method for detecting and segmenting liver tumors from CT. They evaluate their method on the publicly available LiTS dataset. The method is well described. Contributions of the paper are made clear. Clear description of the evaluation metric for the lesion matching. Overall and interesting and high quality contribution.

Positive aspects of the manuscript:
- Public data used for training and evaluation
- Authors contribute additional reference standard and discuss differences between observers and automated method
- Method is well described

Negative aspects of the manuscript:
- None

Detailed comments:

2 Methods
Training: what was the size of the image patches? Later in the paragraph the use of "whole axial slices" is mentioned. This is confusing? Are the terms "patch" and "slice" used interchangeably?
Where did the liver masks used in training come from, where they set manually?

3 Experimental setup
Detection: definition for recall is confusingly worded, would rephrase to something like: "Ratio of TP detections to the number of positives in the reference standard"
FPs/case: Shouldn't this be the average nr of FPs per case?

Algorithm 1: nice job on making the complex matching algorithm clear!


review_title: Nice manuscript, well described method, public data used.

Decision: Reject