\documentclass[runningheads]{llncs}
\usepackage{graphicx}

\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} %
\usepackage{color}

\DeclareMathOperator*{\argmin}{argmin} %
\newcommand*\samethanks[1][\value{footnote}]{\footnotemark[#1]}

\begin{document}
\pagestyle{headings}
\mainmatter

\title{Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification} %
            
\titlerunning{Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification}
\author{Xin Xing\thanks{authors show equal contribution}\orcidID{0000-0001-7207-5149}\and 
Gongbo Liang\samethanks \orcidID{0000-0002-6700-6664}  \and
Hunter Blanton \orcidID{0000-0001-8058-4218} \and
Muhammad Usman Rafique\orcidID{0000-0001-5504-5482}\and
Chris Wang \orcidID{0000-0003-3898-3690}\and
Ai-Ling Lin \orcidID{0000-0002-5197-2219} \and
Nathan Jacobs \orcidID{0000-0002-4242-8967}
}
\authorrunning{X. Xing et al.}
\institute{University of Kentucky, Lexington KY 40506, USA \\
\email{\{xxi242, gli238\}@g.uky.edu}}

\maketitle

\begin{abstract}
We propose to apply a 2D CNN architecture to 3D MRI image Alzheimer's disease classification. Training a 3D convolutional neural network (CNN) is time-consuming and computationally expensive. We make use of approximate rank pooling to transform the 3D MRI image volume into a 2D image to use as input to a 2D CNN. We show our proposed CNN model achieves $9.5\%$ better Alzheimer's disease classification accuracy than the baseline 3D models. We also show that our method allows for efficient training, requiring only $20\%$ of the training time compared to 3D CNN models. The code is available online: https://github.com/UkyVision/alzheimer-project.
\keywords{Dynamic image, 2D CNN, MRI image, Alzheimer's Disease}
\end{abstract}
\section{Introduction}
Alzheimer's disease (AD) is the sixth leading cause of death in the U.S.~\cite{nih}. It heavily affects the patients' families and U.S. health care system due to medical payments, social welfare cost, and salary loss. Since AD is irreversible, early stage diagnosis is crucial for helping slow down disease progression. Currently, researchers are using advanced neuroimaging techniques, such as magnetic resonance imaging (MRI), to identify AD. MRI technology produces a 3D image, which has millions of voxels. Figure~\ref{fig1} shows example slices of Cognitive Unimpaired (CU) and Alzheimer's disease (AD) MRI images.

\begin{figure}
\centering
\includegraphics[scale=0.5]{fig1.pdf}
\caption{The MRI sample slices of the CU and AD participants and the corresponding dynamic images.}
\label{fig1}
\end{figure}

With the promising performance of deep learning in natural image classification, convolutional neural networks (CNNs) show tremendous potential in medical image diagnosis. Due to the volumetric nature of MRI images, the natural deep learning model is a 3D convolutional neural network (3D CNN)~\cite{3dcnn}. Compared to 2D CNN models, 3D CNN models are more computationally expensive and time consuming to train due to the high dimensionality of the input. Another issue is that most current medical datasets are relatively small. The limited data makes it difficult to train a deep network that generalizes to high accuracy on unseen data. To overcome the problem of limited medical image training data, transfer learning is an attractive approach for feature extraction. However, pre-trained CNN models are mainly trained on 2D image datasets. There are few suitable pre-trained 3D CNN models. In our paper, We propose to apply approximate rank pooling~\cite{dyi} to convert a 3D MRI volume into a 2D image over the height dimension. Thus, we can use a 2D CNN architecture for 3D MRI image classification. The main contributions of our work are following:
\begin{itemize}
\item We propose to apply a CNN model that transforms the 3D MRI volume image into 2D dynamic image as the input of 2D CNN. Incorporating with an attention mechanism, the proposed model significantly boosts the accuracy of the Alzheimer's Disease MRI diagnosis.
\item We analyze the effect of skull MRI images on the approximate rank pooling method, showing that the applied approximate rank pooling method is sensitive to the noise introduced by the skull. Skull striping is necessary before using the dynamic image technology.    
\end{itemize}

\section{Related Work}
Learning-based Alzheimer's disease (AD) research can be mainly divided into two branches based on the type of input: (1) manually selected region of interest (ROI) input and (2) whole image input. With ROI models~\cite{ref1}~\cite{ref2}, manual region selection is needed to extract the interest region of the original brain image as the input to the CNN model, which is a time consuming task. It is more straightforward and desirable to use the whole image as input. Korolev et al.~\cite{Korolev2017} propose two 3D CNN architectures based on VGGNet and ResNet, which is the first study to prove the manual feature extraction step for Brain MRI image classification is unnecessary. Their 3D models are called 3D-VGG and 3D-ResNet, and are widely used for 3D medical image classification study. Cheng et al.~\cite{Cheng2017} proposes to use multiple 3D CNN models trained on MRI images for AD classification in an ensemble learning strategy. They separate the original MRI 3D images into many patches (n=27), then forward each patch to an independent 3D CNN for feature extraction. Afterward, the extracted features are concatenated for classification. The performance is satisfactory, but the computation cost and training time overhead are very expensive. Yang et al.~\cite{Yang2018} uses the 3D-CNN models of Korolev et al.~\cite{Korolev2017} as a backbone for studying the explainability of AD classification in MRI images by extending class activation mapping (CAM)\cite{cam} and gradient-based CAM\cite{grad-cam} on 3D images. In our work, we use the whole brain MRI image as input and use 3D-VGG and 3D-ResNet as our baseline models. 
Dynamic images where first applied to medical imagery by Liang et al.~\cite{Liang2019} for breast cancer diagnosis. The authors use the dynamic image method to convert 3D digital breast tomosynthesis images into dynamic images and combined them with 2D mammography images for breast cancer classification. In our work, we propose to combine dynamic images with an attention mechanism for 3D MRI image classification.

\section{Approach}
We provide a detailed discussion of our method. First, we summarize the high-level network architecture. Second, we provide detailed information about the approximate rank pooling method. Next, we show our classifier structure and attention mechanism. Finally, we discuss the loss function used for training.
\subsection{Model Architecture}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{workflow1.pdf}
\caption{The architecture of our 2D CNN model.}
\label{fig2}
\end{figure}
Figure~\ref{fig2} illustrates the architecture of our model. The 3D MRI image is passed to the approximate rank pooling module to transform the 3D MRI image volume into a 2D dynamic image. We apply transfer learning for feature extraction with the dynamic image as the input. We leveraged a pre-trained CNN as the backbone feature extractor. The feature extraction model is pre-trained with the ImageNet dataset~\cite{imagenet}. Because we use a lower input resolution than the resolution used for ImageNet training, we use only a portion of the pre-trained CNN. The extracted features are finally sent to a small classifier for diagnosis prediction. The attention mechanism, which is widely used in computer vision community, can boost CNN model performance, so we embed the attention module in our classifier.

\subsection{Dynamic Image}
The temporal rank pooling~\cite{Fernando}~\cite{dyi} was originally proposed for video action recognition. For a video with T frames $I_{1}, ... , I_{T}$, the method compresses the whole video into one frame by temporal rank pooling. The compressed frame is called a dynamic image. The construction of the dynamic image is based on Fernando et al~\cite{Fernando}. The authors use a ranking function to represent the video. $\psi(I_{t})\in\Re^d$ is a feature representation of the individual frame $I_t$ of the video. $V_t=\frac{1}{t}\sum_{\tau=1}^{t}\psi(I_{\tau})$ is the temporal average of the feature up to time $t$. $V_t$ is measured by a ranking score $S(t|d)=<d, V_t>$, where $d\in\Re^m$ is a learned parameter. By accumulating more frames for the average, the later times are associated with larger scores, e.g $q>t\rightarrow S(q|d)>S(t|d)$ , which are constraints for the ranking problem. So the whole problem can be formulated as a convex problem using RankSVM:

\begin{equation}
d^*=\rho(I_1, ..., I_t; \tau)=\argmin_dE(d)
\label{eq:1}
\end{equation}
\begin{equation}
E(d)=\frac{\lambda}{2}||d||^2 + \frac{2}{T(T-1)}\times\sum_{q>t}\max\{0, 1-S(q|d)+S(t|d)\}
\label{eq:2}
\end{equation}
In Equation \eqref{eq:2}, the first term is a quadratic regularization used in SVMs, the second term is a hinge-loss counting incorrect rankings for the pairs $q>t$. 

The RankSVM formulation can be used for dynamic image generation, but the operations are computationally expensive. Bilen et al.~\cite{dyi} proposed a fast approximate rank pooling for dynamic images: 
\begin{equation}
\hat{\rho}(I_1, ..., I_t; \psi)=\sum_{t=1}^{T}\alpha_t \cdot\psi(I_t) %
\label{eq:3}
\end{equation}
where, $\psi(I_t)=\frac{1}{t}\sum_{\tau=1}^{t}I_{\tau}$ is the temporal average of frames up to time t, and $\alpha_t=2t-T-1$ is the coefficient associated to frame $\psi(I_t)$.  We take this approximate rank pooling strategy in our work for 3D MRI volume to 2D image transformation. In our implementation, the z-dimension of 3D MRI image is equal to temporal dimension of the video.

\subsection{Classifier with Attention Mechanism}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{att.pdf}
\caption{The attention mechanism structure in our CNN model.}
\label{fig3}
\end{figure}
The classifier is a combination of an attention mechanism module and a basic classifier. Figure~\ref{fig3} depicts the structure of attention mechanism, which includes four $1 \times 1$ convolutional layers. The first three activation functions of convolutional layers are ReLU, the last convolutional layer is attached with softmax activation function. The input feature maps $A \in R^{H\times W\times C}$ are passed through the four convolutional layers to calculate attention mask $S\in R^{H\times W\times 1}$. We apply element-wise multiplication between the attention mask and input feature maps to get the final output feature map $O \in R^{H\times W\times C}$. Our basic classifier contains three fully connected (FC) layers. The output dimensions of the three FC layers are 512, 64, and 2. Dropout layers are used after the first two layers with dropout probability 0.5. 

\subsection{Loss Function}
In previous AD classification studies, researchers mainly concentrated on binary classification. In our work, we do the same for ease of comparison. The overall loss function is binary cross-entropy. For a 3D image $V$ with label $l$ and probability prediction $p(l|V)$, the loss function is:
\begin{equation}
loss(l,V)=-[l \cdot log(p(l|V))+(1-l) \cdot log(1-p(l|V))]
\label{eq:4}
\end{equation}
where the label $l=0$ indicates a negative sample and $l=1$ indicates a positive sample.

\section{Evaluation}
We use the publicly available dataset from the Alzheimer’s Disease Neuroimaging Initiative (ADNI)~\cite{ADNI} for our work. Specifically, we trained CNNs with the data from the ``spatially normalized, masked, and N3-corrected T1 images” category. The brain MRI image size is $110 \times 110 \times 110$. Since a subject may have multiple MRI scans in the database, we use the first scan of each subject to avoid data leakage. The total number of data samples is 100, containing 51 CU samples and 49 AD samples. 

The CNNs are implemented in PyTorch. We use five-fold cross validation to better evaluate model performance. The batch size used for our model is 16. The batch size of the baseline models is 8, which is the maximum batch size of the 3D CNN model trained on the single GTX-1080ti GPU. We use the Adam optimizer with $beta_1=0.9$ and $beta_2=0.999$. The learning rate is 0.0001. We train for 150 epochs. To evaluate the performance of our model, we use accuracy (Acc), the area under the curve of Receiver Operating Characteristics (ROC), F1 score (F1), Precision, Recall and Average Precision (AP) as our evaluation metrics. 

\subsection{Quantitative  Results}
High quality feature extraction is crucial for the final prediction. Different pre-trained CNN models can output different features in terms of size and effective receptive field. We test different pre-trained CNNs to find out which CNN models perform best as our feature extractor. Table~\ref{table1} shows various CNN models and the corresponding output feature size. 
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{The different pre-trained CNN model as feature extractors and the output feature sizes}
\label{table1}
\begin{tabular}{lll}
\hline\noalign{\smallskip}
CNN model & & Output feature size\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
AlexNet~\cite{Alex}  & & $256\times5\times5$ \\
VggNet11~\cite{Vgg} & & $512\times6\times6$ \\
ResNet18~\cite{He2015} & & $512\times7\times7$ \\
MobileNet\_v2~\cite{Sandler_2018_CVPR} & &$1280\times4\times4$ \\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\setlength{\tabcolsep}{4pt}

Since our dynamic image resolution is $110\times110\times3$, which is much smaller than the ImageNet dataset resolution: $256\times256\times3$, we use only part of the pre-trained CNN as the feature extractor. Directly using the whole pre-trained CNN model as feature extractor will cause the output feature size to be too small, which decreases the classification performance. In the implementation, we get rid of the maxpooling layer of each pre-trained model except for the MobileNet\_v2~\cite{Sandler_2018_CVPR}, which contains no maxpooling layer. Also, because there is a domain gap between the natural image and medical image we set the pre-trained CNN models' parameters trainable, so that we can fine tune the models for better performance.

\begin{table}
\begin{center}
\caption{The performance results of different backbone models with dynamic image as input}
\label{table2}
\begin{tabular}{llccccc}
\hline\noalign{\smallskip}
Model  & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
AlexNet  & 0.87 & 0.90 & 0.86 & 0.89 & 0.83 & 0.82 \\
ResNet18 & 0.85 & 0.84 & 0.84 & 0.86 & 0.81 & 0.79 \\
MobileNet\_v2 & 0.88 & 0.89 & 0.87 & 0.89 & 0.85 & 0.83 \\
VggNet11 & 0.91 & 0.92 & 0.91 & 0.88 & 0.93 & 0.86 \\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\begin{table}
\begin{center}
\caption{The performance results of different 2D and 3D CNN models}
\label{table3}
\begin{tabular}{llcccccc}
\hline\noalign{\smallskip}
Model  &$\quad$ & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}

3D-VGG~\cite{Korolev2017} &$\quad$  & 0.80 & 0.78 & 0.78 & 0.82 & 0.75 & 0.74 \\
3D-ResNet~\cite{Korolev2017}&$\quad$  & 0.84 & 0.82 & 0.82 & 0.86 & 0.79 & 0.78 \\
\hline
Max. + VGG11&$\quad$ & 0.80 & 0.77 & 0.80 & 0.78 & 0.81 & 0.73 \\ 
Avg. + VGG11&$\quad$ & 0.86 & 0.84 & 0.86 & 0.83 & 0.89 & 0.79 \\

Max. + VGG11 + Att&$\quad$ & 0.82 & 0.76 & 0.82 & 0.80 & 0.83 & 0.75 \\ 
Avg. + VGG11 + Att&$\quad$ & 0.88 & 0.89 & 0.88 & 0.85 & \textbf{0.91} & 0.82 \\
\hline
Ours  &$\quad$  & \textbf{0.92} &\textbf{0.95} & \textbf{0.91} & \textbf{0.97} & 0.85 & \textbf{0.90} \\

\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
When analyzing MRI images using computer-aided detectors (CADs), it is common to strip out the skulls from the brain images. Thus, we first test the proposed method using the MRI with the skull stripped. Our proposed model takes dynamic images (Dyn) as input, VGG11 as feature extractor, and a classifier with the attention mechanism: $Dyn + VGG11 + Att $. The whole experiment can be divided into three sections: the backbone and attention section, the baseline model section, and the pooling section. In the backbone and attention section, we use 4 different pre-trained models and test the selected backbone with and without the attention mechanism. Based on the performance shown in Table~\ref{table2}, we choose VGG11 as the backbone model. In the baseline model section, we compare our method with two baselines, namely 3D-VGG and 3D-ResNet. Table~\ref{table3} shows the performance under different CNN models. The proposed model achieves $9.52\%$ improvement in accuracy and $15.20\%$ better ROC over the 3D-ResNet. In the pooling section: we construct two baselines by replacing the approximate rank pooling module with the average pooling (Avg.) layer or max pooling (Max.) layer. The pooling layer processes the input 3D image over the z-dimension and outputs the same size as the dynamic image. Comparing with the different 3D-to-2D conversion methods under the same configuration, the dynamic image outperforms the two pooling methods.

\subsection{Pre-processing Importance Evaluation}

\begin{table}
\begin{center}
\caption{The performance results of different 2D and 3D CNN models on the MRI image with skull.}
\label{table4}
\begin{tabular}{lcccccccc}
\hline\noalign{\smallskip}
Model &$\quad$ & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
3D-VGG~\cite{Korolev2017} &$\quad$  & 0.78 & 0.62 & 0.77 & 0.80 & 0.75 & 0.72 \\
Ours &$\quad$ & 0.63  & 0.52 & 0.63 & 0.62 & 0.64 & 0.57\\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\begin{figure}
\centering  
\includegraphics[scale=0.5]{MRIwskull.pdf}
\caption{The MRI sample slices with skull of the CU and AD participants and the corresponding dynamic images.}
\label{fig4}
\end{figure}

In this section, we show results using the raw MRI image (including skull) as input. We perform experiments on the same patients' raw brain MRI image with the skull included to test the performance of our model. The raw MRI image category is ``MT1,GradWarp,N3m". The image size of the raw MRI image is "$176 \times 256 \times 256$". Figure~\ref{fig4} illustrates the dynamic images of different participants' MRI brain images with the skull. The dynamic images are blurrier than the images under skull striping processing. This is because the skull variance can be treated as noise in the dynamic image. %
Table~\ref{table4} shows the significant performance decrease when using 3D Brain MRI images with skull. Figure~\ref{fig4} shows a visual representation of how the dynamic images are affected by including the skull in the image. In this scenario, the model can not sufficiently diagnose the different groups. A potential cause of this decrease in performance is that the approximate rank pooling module is a pre-processing step, and the module is not trainable. We believe an end-to-end, learnable rank pooling module would improve performance.%

\subsection{Models Training time}
\begin{table}
\begin{center}
\caption{The total 150 epochs training time of different CNN models.}
\label{table5}
\begin{tabular}{lcc}
\hline\noalign{\smallskip}
 &$\quad$ &Training time(s)  \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
3D-VGG~\cite{Korolev2017} & &2359 \\
3D-ResNet~\cite{Korolev2017} & &3916\\
Ours & &414\\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
Another advantage of the proposed model is faster training. We train all of our CNN models for 150 epochs on the same input dataset. Table~\ref{table5} shows the total training time of the different 2D and 3D CNN models. Compared with the 3D-CNN networks, the proposed model trains in about $20\%$ of the time. Also, due to the higher dimension of the 3D convolutional layer, the number of parameters of the 3D convolutional layer is naturally higher than the 2D convolutional layer. 
By applying the MobileNet~\cite{mobilenet} or ShuffleNet~\cite{shuffle} in medical image diagnosis, there is potential for mobile applications. We used MobileNet for our experiments. We used the MobileNet v1 achitecture as the feature extractor and obtained 84.84\% accuracy, which is similar in accuracy to the 3D ResNet.
\section{Conclusions}
We proposed to apply the approximate rank pooling method to convert 3D Brain MRI images into 2D dynamic images as the inputs for a pre-trained 2D CNN. The proposed model outperforms a 3D CNN with much less training time and improves 9.5\% better performance than the baselines. We trained and evaluated on MRI brain imagery and found out that brain skull striping pre-processing is useful before applying the approximate rank pooling conversion. We used an offline approximate rank pooling module in our experiments, but we believe it would be interesting to explore a learnable temporal rank pooling module in the future. 

\section*{Acknowledgement}
This work is supported by NIH/NIA R01AG054459. 
\bibliographystyle{splncs04}
\bibliography{egbib}
\end{document}


Peer review:
Simple and effective method to improve AD classification
Summary
-------

This paper presents a method for Alzheimer's Disease (AD) classification from
3D MRI scans. In contrast to earlier approaches, the 3D scans are first
converted into a 2D dynamic image, which is then processed by a 2D
convolutional neural network. The network consists of a pre-trained (and
fine-tuned) feature extractor, an attention module, and subsequent fully
connected layers for the final classification. Results on MRI scans with
stripped skulls demonstrate a significant increase compared to fully 3D
baselines, albeit using only 20% of the time needed for training.

Quality and Clarity
-------------------

The introduction, motivation, and technical description are very well written
and easy to follow. The provided figures are helpful to understand the method
and show qualitative results of the dynamic image generation.

Originality
-----------

Using 2D dynamic images instead of the full 3D scan is a compelling idea
(although already explored in other medical domains). The main contribution of
this paper lies therefore in the subsequent use of pre-trained 2D feature
extractors, which are enabled by the 2D input images. Furthermore, the authors
investigate the usefulness of an attention module between the feature
extraction and classification network.

Significance
------------

The results demonstrate consistent improvements over several baselines and
variations of the proposed model. On top of those improvements, the proposed
model trains faster and requires less resources for prediction.

The authors mention that "there is potential for mobile applications" (line
360). I am not sure what this could look like in a clinical setting and suggest
the authors elaborate if they want to make this point convincingly.

My main concern for a clinical application is that the improved results seem to
be achieved only on "skull stripped" MRI scans. From the paper, it is unclear
whether this is a manual, semi-automatic, or entirely automatic process. I
would appreciate if the authors could discuss this point in more detail to
provide context about the amount of manual labor needed in the proposed
pipeline.

Pros
----

* well written
* elegant architecture
* thorough comparison to baselines and model variations
* significant accuracy improvement

Cons
----

* unclear how much manual intervention is needed for "skull stripping"

Minor Comments
--------------

* title: capitalize "image" and "classification"
* line 56: "We" -> "we"
* line 144: "association" -> "associated"
* line 283: "choice" -> "choose"
* line 335: "decrease if" -> "decrease in"
* Equation 4: What is $i$ running over? According to the text, $l$ and $I$ are label and image of a single sample.
* Figure 4: It would be helpful to see same (or similar) images without skull in the same figure.
* notation: $\mathcal{R}$ vs $R$
Rating: 8: Top 50% of accepted papers, clear accept
Confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct