Venue: MIDL 2018 Conference

Reviews:

final_score: rating: 2: Marginally below acceptance threshold
review_content: The authors adopt a multiscale convolutional LSTM to classify mild cognitive impaired (MCI) from normal healthy control (NC) subjects. The model takes a sequence of correlation matrices as inputs that are intended to approximate brain dynamics. Before the classification stage, the model is pretrained using an LSTM unit. At each time step, the model predicts the correlation matrix of the next time step. The authors also incorporate multiscale information in their model by using tailored up- and downscaling layers. The proposed model is evaluated against three approaches from the literature.

The achieved results of the model look encouraging but I have some concerns with the clarity, evaluation, and maturity of the manuscript. 

* The title of the paper should reflect that the proposed model is a convolutional LSTM with pretraining, not unsupervised learning. Why do the authors call this model a D-net? Skip-connections do not justify a new model name in my point of view.
	
* The description of the architecture is interspersed with parameter settings, e.g., 3x3 kernel while explaining the layer operations. I would recommend to separate the architectural description from the parameterization of the model.
	
* Figure 1 seems overloaded and so I am wondering if it can be broken down in more parts. What are the dashed green, blue, black, and red arrows? What does the copy&add transformation mean and why is it important?
	
* The evaluation dataset is quite small (277 scans from 93 patients) which makes it hard to assess the performance of the model. What are the confidence intervals for the reported results? How many hyperparameters have been tried out?
	
* The manuscript contains many grammar mistakes and typos. I recommend to carefully proofread the manuscript.
	
* The notation is in some parts confusing or incomplete. For instance, a mix of superscripts (l=0, middle, last) is used.
review_title: Unclear model description and small evaluation data

final_score: rating: 1: Strong rejection
review_content: The paper introduces a D-network designed to learn connectivity pattern in rs-fMRI.

Pros:
+interesting medical application
Cons:
-model design is arbitrary
-week evaluation

I like the idea of using Convolutional LSTM to analyze spatio-temporal medical data. However, the paper's presentation could be improved. It is not clear why the authors call their method unsupervised. From abstract, it is seems that the method is evaluated on a classification task. Moreover, in Figure 1, there are two outputs: 1) prediction of the next time step and 2) classifier. The setup of D-net resembles more a semi supervised learning than unsupervised learning.  Thus, it seems more natural to present D-Net as a semi supervised approach (or maybe even supervised if we have label for each spatiotemporal datapoint).

It looks like D-net is trained in two stages (section 3.3). In the first stage, the model is trained only with prediction loss (pre-training) and, in the second stage, the model is fine-tuned with classification loss. Wouldn't it be possible to train the model in one stage?

The architecture design looks quite arbitrary. It is clear why one would want to combine LSTMs with Convolutional layers; however, the justification on why the D-net would work well is missing. There is a significant body of papers on architectures combining RNNs and convolutional layers. Could the authors comment on why they do not apply previously developed architectures and present a new one? What is the advantage of D-Net when compared to more standard models combining LSTMs and Convolutions?

In Figure 1, what is the meaning of rectangles? Why do we need to copy and add z^{middle}? There is no need to introduce Res_U_Block, same architecture could be explained with a single resblock together with upsampling and downsampling operations. 

The text in Figure 3 is too small.

"We found that involving unsupervised pre-training is crucial, in order to avoid over-fitting during the training of the networks ,..." The model pre-training play a role of model regularization. The overfitting depicted in Figure 5, might be due to the lack of proper model regularization.  Did the authors tried to improve regularization of the supervised model by using different regularization techniques: dropout, data augmentation and weight decay? Moreover, from Figure 5(b) we can see that the model with pre-training still gets to the overfitting regime. Further hyper-parameters tuning might be beneficial.

Figure 5 suggests that the authors monitor test set during the experiments. In standard ML approach, one would use the test set only to report the result for the model with the best score on the validation set. If the test set is used to drive design decisions, the reported results might reflect overfitting to test set.

It is not clear where the name D-net is coming from.

There are authors missing in ref 3 and 15.
review_title: Convolutional LSTM to analyze spatio-temporal medical data

final_score: rating: 3: Marginally above acceptance threshold
review_content: This paper presents a pipeline to diagnose AD/MCI using fMRI images using deep convolutional LSTMs. The network inputs are 28x28 dynamic functional connectivity matrices obtained from 4D fMRI scans. The authors used residual blocks in both the convolutional module and the LSTMs. The proposed methods were evaluated on the fMRI images released by the ADNI cohort. The authors showed the proposed pipeline could outperform several previous solutions that were reproduced by the authors. 

pros:
The idea of using convolutional LSTMs to learn from fMRI FC networks is interesting and inspiring. The preprocessing pipeline in this paper makes sense. It is also interesting to see that pre-training by predicting the following series could prevent overfitting after a certain number of iterations. The paper was written in a clear way.

cons:
The title is inaccurate and confusing. The main motivation of this paper is to diagnose AD/MCI as a supervised task. The unsupervised learning was only used for pre-training the network. Besides Fig.5, it is not clear if the unsupervised learning is really necessary here since the best test performance without the pre-training stage is not shown. Is it replaceable by the other regularization methods? eg. early stopping, dropout, batch-normalization, etc. Experimental evidence is needed to make such claim. It also remains unclear which choice of the architecture design is critical without ablative analysis made available. 

The naming of D-Net is indeed an overkill and non-necessary. It seems it is called D-Net only because of the Residual blocks. Then why bother naming it?

The evaluation falls short of a comprehensive analysis of the claims made by this paper. The paper claims the advantages of using residual blocks in cLSTMs which also resulted in the network naming. However, the numbers from a cLSTM without the residual blocks were not provided. It is intuitive nowadays that deep learning methods could outperform the hand-crafted features. It would be more interesting to see the results from other simpler deep learning approaches, for example, the performance of a 3D CNN or a CNN with anisotropic kernels.

The dataset used in this study is way too small (36+31+26 patients), to draw any solid conclusions in a machine learning study regarding the classification accuracy. 
review_title: Interesting idea to use cLSTM to learn from dynamic FC matrices. Confusing title, overkill naming, insufficient experiments 

Decision: Reject