Deep learning for neuroimaging: a validation study
Authors: Sergey M. Plis, Devon R. Hjelm, Ruslan Salakhutdinov, Vince D. Calhoun
ICLR 2014

Abstract: Deep learning methods have recently enjoyed a number of successes in the tasks of classification and representation learning. These tasks are very important for brain imaging and neuroscience discovery, making the methods attractive candidates for porting to a neuroimager's toolbox. Successes are, in part, explained by a great flexibility of deep learning models. This flexibility makes the process of porting to new areas a difficult parameter optimization problem. In this work we demonstrate our results (and feasible parameter ranges) in application of deep learning methods to structural and functional brain imaging data. We also describe a novel constraint-based approach to visualizing high dimensional data. We use it to analyze the effect of parameter choices on data transformations. Our results show that deep learning methods are able to learn physiologically important representations and detect latent relations in neuroimaging data.


Peer review:
[Review]:
Summary:

The paper applies deep learning methods to various problems in MRI data analysis, showing that DBNs can recover similar features as other standard techniques, and can possibly improve classification performance in certain tasks.

Major comments:

The classification experiments in this paper are hard to follow, and as written it seems they may be flawed:

For the results in Section 3.2, it sounds like the models may possibly have been pretrained on the testing data, and worse, possibly fine-tuned on the testing data as well. In particular, the model is pretrained and fine-tuned on 335 of 389 examples, and then, subsequently, the top layer activations of these fine-tuned models are supplied to a new shallow classifier that is trained on a different train/test split of half the 389 subjects in each. Each of these splits will thus necessarily contain many examples from the training set used for pretraining/fine-tuning. This means there will be many test examples on which the underlying DBN model has already been fine-tuned with the correct label. Hence the performance numbers are inaccurate. 

For the results in Section 3.3, all data examples are used for pretraining/finetuning, and hence it is not clear to what extent the learned classifiers will generalize to new examples. This is particularly important to check given that the three layer network attains perfect accuracy, which may indicate overfitting. These concerns are moderated by the fact that dimensionality reduction applied to the learned representation shows an interesting structuring by disease severity, and this data was not used in the training process. I also recognize that, in this data analysis application, there are other important goals beyond generalization accuracy. If the trained model identifies particular features that are interpretable in the light of other known data, that can be very helpful. Drawing out these other uses of DBNs is an interesting direction to pursue.

More details of the RBM model and training procedure should be included in the paper. RBMs typically use 0-1 valued inputs, and the sigmoidal form of the activation function arises by manipulating the energy function. Some details which would strengthen the paper: How was the switch to Tanh activations performed? Were weights updated using contrastive divergence to approximately maximize log likelihood? If so, how many sampling steps were used (e.g., CD-1, CD-10)? Was sampling used, or were the updates based on the mean field approximation? How were tanh units sampled from? How was L1 regularization added? E.g., was it ||tanh(Wx)||_1, or ||sigmoid(Wx)||_1 or ||Wx||_1? 

Depending on how the switch to tanh units was performed, the RBM algorithm can be nearly identical to ICA. See, for example, Q.V. Le, A. Karpenko, J. Ngiam, A.Y. Ng. ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning. NIPS, 2011. This connection should be cited and discussed. In particular, the statement 'In general, RBM performed competitively with ICA, while providing--perhaps, not surprisingly due to the used L1 regularization--sharper and more localized features' may need some adjustment. L1 regularization is the objective to be minimized in ICA too--if anything, the difference is due to less stringent orthogonalization.

Some of the given parameter values are confusing. The learning rate is given as epsilon = 0.08, but this is not in the 'workable range' of [1e-4, 1e-3]. Were these parameters hand optimized? How were network sizes chosen?

Some claims are stated without providing the relevant data. E.g., 'Moreover, the block structure of the correlation matrix (not shown) of feature time courses provide a grouping that is more physiologically supported than that provided by ICA.' More discussion of this, maybe in a supplementary materials section after the main paper, would be welcome. It is also stated that RBM bases are more 'spatially concentrated' but this is not quantified or clearly established in the presented data.

The text is often unclear and hard to follow (particularly section 3.2).

Pros/cons:
+ Interesting new application area for DL
+ Some results that hold promise

- Flawed experiments mix train/test data
- Insufficient details of model and training method


Minor comments:

The first page footnote should be removed