Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 1: Strong rejection
review_content: The authors evaluate Unet for an automated localization and segmentation of mitral valve leavelet segmentation in echocardiographic images.

Pro: This is an important application, and results are promising. Evaluation seems to be well designed and thorough. Unet performs better than previous non-ANN applications.

Con: While the study features an interesting application and data set, there is little algorithmic novelty. It features little more than an application use case for Unet. 
review_title: Unet application

final_score: rating: 3: Marginally above acceptance threshold
review_content: Straight-forward application of a U-net to the task of segmenting mitral valve leaflets (anterior + posterior) in echocardiography. The paper has a solid overview of classical (pre-deep learning) approaches to this problem. No prior use of deep learning on this topic has been reported. The dataset is interesting, but apparently proprietary (~50 US videos from two manufacturers with more than 2500 frames for PLAX and A4C views each, 27 / 28 of which have two manually annotated mitral leaflets). The evaluation is based on voxel-wise classification measures plus subjective ratings on 23 unlabeled videos. The performance on the labeled test set looks quite good for the anterior leaflet (DSC around 0.75 can be considered a success for such a thin structure), while the posterior leaflet, which is also hard to make out for human experts, cannot be reliably segmented in all cases.
"The samples were randomly shuffled and divided into a training and validation set": I wonder if the random shuffling was done with care, or if different US frames from the same patient may have been in the training & validation sets.  Furthermore, normally an independent test set is also used for deep learning.  Update: Later, Table 1 reveals that there is a test set, but it should be addressed also in section 2. Furthermore, it looks like validation & training sets are not independent.
The authors should really try to understand the overlapping tile strategy (see e.g. the original Ronneberger U-net paper [13]) which does not only mean to use "valid" convolutions instead of zero padding, but also allows to process arbitrarily large images ("Due to memory management and taking into account the down-sampling process, samples’ dimensions were set to a 416 pixel height and 512 pixel width."). The training speed-up that "was not observed during preliminary tests" is due to the fully-convolutional nature and the ability to freely choose the training patch sizes, and cannot be observed of course when comparing training with small patches and zero padding against training with much larger, padded inputs.
The number of feature channels was quite low: 4 to 32 in the first layer (but Table 2 only contains results for 4, 8, 16, which surprises the reader before reading the next section), whereas the original paper suggested 64, and may not be doubled in every level, although that looks like an error in Fig. 2. (Note that Cicek et al. proposed a refined scheme in 2016 for 3D to save memory and prevent bottlenecks, which could also have been applied here, but was not.) That is probably because of the limited GPU (GTX 970 with 4 GiB memory), which I do not want to criticize, but could have been made better use of.
The fact that less filters worked better for PLAX (8 instead of 16) could be due to missing regularization / dropout. Furthermore, no data augmentation is mentioned, although it is commonly done with U-Nets and would likely improve the models (when chosing appropriate transformations, of course).
Overall, the paper structure could be improved a little bit; often things appeared to be missing, but would be mentioned later. There are some technical shortcomings (see above, plus some obvious options for improvements already mentioned by the authors), but they are not major, and the topic is interesting and has not been addressed before.
Small comments:
"… the DICE, precision and recall were used.", but that should be further defined. Are they defined on the voxel level? Aggregated per frame? or per batch? per object?
- "greed search optimization": probably "grid search"?
- "Preliminary tests have shown that lower depths result in less accurate results." does not belong into 2.1.1, but the results section.
- strange formatting of abbreviations (PLAX in LaTeX math mode? looks like P*L*A*X this way) and some more words (CNN, Sscore, …)
- "a batch size of 4 was set" → use unbreakable space before 4
review_title: Straight-forward application of U-net for interesting task

final_score: rating: 2: Marginally below acceptance threshold
review_content: The paper describes an automatic segmentation approach of the mitral leaflets. The solution is based on a well known UNet architecture and it is evaluated on two datasets: PLAX and A4C.

Structure of the introduction: 
	The section reviews nicely motivation and state-of-the-art; however, the introduction of the paper lacks mention about what is the contribution of the paper. 

Methodology:
	From Section 2, it seems that there is no methodological contribution. The paper uses UNet and adds zero padding in the convolutions as well as batch norm before concat operation. This modifications are rather minor and the effect of these modifications is not analyzed in the paper. Given this, it is hard to justify the the statement form the conclusion section: "A new method based on UNet architecture was proposed...".

	"This adds regularization effect by ensuring that the concatenated feature maps have the same order of magnitude". It is not clear why the regularization would be triggered by ensuring same order of magnitude. Could the authors comment on this?

Experimental section:
	Is there any data augmentation used? What is the setup of data pre-processing? 
	
	Although the UNet model is a good choice of baseline, there is no comparison to alternative architectures nor approaches making it hard to evaluate the results presented in the paper.
review_title: UNet-based segmentation for echocardiography

Decision: Reject