\documentclass[runningheads]{llncs}
\usepackage{graphicx}

\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} %
\usepackage{color}

\DeclareMathOperator*{\argmin}{argmin} %
\newcommand*\samethanks[1][\value{footnote}]{\footnotemark[#1]}

\begin{document}
\pagestyle{headings}
\mainmatter

\title{Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification} %
            
\titlerunning{Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification}
\author{Xin Xing\thanks{authors show equal contribution}\orcidID{0000-0001-7207-5149}\and 
Gongbo Liang\samethanks \orcidID{0000-0002-6700-6664}  \and
Hunter Blanton \orcidID{0000-0001-8058-4218} \and
Muhammad Usman Rafique\orcidID{0000-0001-5504-5482}\and
Chris Wang \orcidID{0000-0003-3898-3690}\and
Ai-Ling Lin \orcidID{0000-0002-5197-2219} \and
Nathan Jacobs \orcidID{0000-0002-4242-8967}
}
\authorrunning{X. Xing et al.}
\institute{University of Kentucky, Lexington KY 40506, USA \\
\email{\{xxi242, gli238\}@g.uky.edu}}

\maketitle

\begin{abstract}
We propose to apply a 2D CNN architecture to 3D MRI image Alzheimer's disease classification. Training a 3D convolutional neural network (CNN) is time-consuming and computationally expensive. We make use of approximate rank pooling to transform the 3D MRI image volume into a 2D image to use as input to a 2D CNN. We show our proposed CNN model achieves $9.5\%$ better Alzheimer's disease classification accuracy than the baseline 3D models. We also show that our method allows for efficient training, requiring only $20\%$ of the training time compared to 3D CNN models. The code is available online: https://github.com/UkyVision/alzheimer-project.
\keywords{Dynamic image, 2D CNN, MRI image, Alzheimer's Disease}
\end{abstract}
\section{Introduction}
Alzheimer's disease (AD) is the sixth leading cause of death in the U.S.~\cite{nih}. It heavily affects the patients' families and U.S. health care system due to medical payments, social welfare cost, and salary loss. Since AD is irreversible, early stage diagnosis is crucial for helping slow down disease progression. Currently, researchers are using advanced neuroimaging techniques, such as magnetic resonance imaging (MRI), to identify AD. MRI technology produces a 3D image, which has millions of voxels. Figure~\ref{fig1} shows example slices of Cognitive Unimpaired (CU) and Alzheimer's disease (AD) MRI images.

\begin{figure}
\centering
\includegraphics[scale=0.5]{fig1.pdf}
\caption{The MRI sample slices of the CU and AD participants and the corresponding dynamic images.}
\label{fig1}
\end{figure}

With the promising performance of deep learning in natural image classification, convolutional neural networks (CNNs) show tremendous potential in medical image diagnosis. Due to the volumetric nature of MRI images, the natural deep learning model is a 3D convolutional neural network (3D CNN)~\cite{3dcnn}. Compared to 2D CNN models, 3D CNN models are more computationally expensive and time consuming to train due to the high dimensionality of the input. Another issue is that most current medical datasets are relatively small. The limited data makes it difficult to train a deep network that generalizes to high accuracy on unseen data. To overcome the problem of limited medical image training data, transfer learning is an attractive approach for feature extraction. However, pre-trained CNN models are mainly trained on 2D image datasets. There are few suitable pre-trained 3D CNN models. In our paper, We propose to apply approximate rank pooling~\cite{dyi} to convert a 3D MRI volume into a 2D image over the height dimension. Thus, we can use a 2D CNN architecture for 3D MRI image classification. The main contributions of our work are following:
\begin{itemize}
\item We propose to apply a CNN model that transforms the 3D MRI volume image into 2D dynamic image as the input of 2D CNN. Incorporating with an attention mechanism, the proposed model significantly boosts the accuracy of the Alzheimer's Disease MRI diagnosis.
\item We analyze the effect of skull MRI images on the approximate rank pooling method, showing that the applied approximate rank pooling method is sensitive to the noise introduced by the skull. Skull striping is necessary before using the dynamic image technology.    
\end{itemize}

\section{Related Work}
Learning-based Alzheimer's disease (AD) research can be mainly divided into two branches based on the type of input: (1) manually selected region of interest (ROI) input and (2) whole image input. With ROI models~\cite{ref1}~\cite{ref2}, manual region selection is needed to extract the interest region of the original brain image as the input to the CNN model, which is a time consuming task. It is more straightforward and desirable to use the whole image as input. Korolev et al.~\cite{Korolev2017} propose two 3D CNN architectures based on VGGNet and ResNet, which is the first study to prove the manual feature extraction step for Brain MRI image classification is unnecessary. Their 3D models are called 3D-VGG and 3D-ResNet, and are widely used for 3D medical image classification study. Cheng et al.~\cite{Cheng2017} proposes to use multiple 3D CNN models trained on MRI images for AD classification in an ensemble learning strategy. They separate the original MRI 3D images into many patches (n=27), then forward each patch to an independent 3D CNN for feature extraction. Afterward, the extracted features are concatenated for classification. The performance is satisfactory, but the computation cost and training time overhead are very expensive. Yang et al.~\cite{Yang2018} uses the 3D-CNN models of Korolev et al.~\cite{Korolev2017} as a backbone for studying the explainability of AD classification in MRI images by extending class activation mapping (CAM)\cite{cam} and gradient-based CAM\cite{grad-cam} on 3D images. In our work, we use the whole brain MRI image as input and use 3D-VGG and 3D-ResNet as our baseline models. 
Dynamic images where first applied to medical imagery by Liang et al.~\cite{Liang2019} for breast cancer diagnosis. The authors use the dynamic image method to convert 3D digital breast tomosynthesis images into dynamic images and combined them with 2D mammography images for breast cancer classification. In our work, we propose to combine dynamic images with an attention mechanism for 3D MRI image classification.

\section{Approach}
We provide a detailed discussion of our method. First, we summarize the high-level network architecture. Second, we provide detailed information about the approximate rank pooling method. Next, we show our classifier structure and attention mechanism. Finally, we discuss the loss function used for training.
\subsection{Model Architecture}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{workflow1.pdf}
\caption{The architecture of our 2D CNN model.}
\label{fig2}
\end{figure}
Figure~\ref{fig2} illustrates the architecture of our model. The 3D MRI image is passed to the approximate rank pooling module to transform the 3D MRI image volume into a 2D dynamic image. We apply transfer learning for feature extraction with the dynamic image as the input. We leveraged a pre-trained CNN as the backbone feature extractor. The feature extraction model is pre-trained with the ImageNet dataset~\cite{imagenet}. Because we use a lower input resolution than the resolution used for ImageNet training, we use only a portion of the pre-trained CNN. The extracted features are finally sent to a small classifier for diagnosis prediction. The attention mechanism, which is widely used in computer vision community, can boost CNN model performance, so we embed the attention module in our classifier.

\subsection{Dynamic Image}
The temporal rank pooling~\cite{Fernando}~\cite{dyi} was originally proposed for video action recognition. For a video with T frames $I_{1}, ... , I_{T}$, the method compresses the whole video into one frame by temporal rank pooling. The compressed frame is called a dynamic image. The construction of the dynamic image is based on Fernando et al~\cite{Fernando}. The authors use a ranking function to represent the video. $\psi(I_{t})\in\Re^d$ is a feature representation of the individual frame $I_t$ of the video. $V_t=\frac{1}{t}\sum_{\tau=1}^{t}\psi(I_{\tau})$ is the temporal average of the feature up to time $t$. $V_t$ is measured by a ranking score $S(t|d)=<d, V_t>$, where $d\in\Re^m$ is a learned parameter. By accumulating more frames for the average, the later times are associated with larger scores, e.g $q>t\rightarrow S(q|d)>S(t|d)$ , which are constraints for the ranking problem. So the whole problem can be formulated as a convex problem using RankSVM:

\begin{equation}
d^*=\rho(I_1, ..., I_t; \tau)=\argmin_dE(d)
\label{eq:1}
\end{equation}
\begin{equation}
E(d)=\frac{\lambda}{2}||d||^2 + \frac{2}{T(T-1)}\times\sum_{q>t}\max\{0, 1-S(q|d)+S(t|d)\}
\label{eq:2}
\end{equation}
In Equation \eqref{eq:2}, the first term is a quadratic regularization used in SVMs, the second term is a hinge-loss counting incorrect rankings for the pairs $q>t$. 

The RankSVM formulation can be used for dynamic image generation, but the operations are computationally expensive. Bilen et al.~\cite{dyi} proposed a fast approximate rank pooling for dynamic images: 
\begin{equation}
\hat{\rho}(I_1, ..., I_t; \psi)=\sum_{t=1}^{T}\alpha_t \cdot\psi(I_t) %
\label{eq:3}
\end{equation}
where, $\psi(I_t)=\frac{1}{t}\sum_{\tau=1}^{t}I_{\tau}$ is the temporal average of frames up to time t, and $\alpha_t=2t-T-1$ is the coefficient associated to frame $\psi(I_t)$.  We take this approximate rank pooling strategy in our work for 3D MRI volume to 2D image transformation. In our implementation, the z-dimension of 3D MRI image is equal to temporal dimension of the video.

\subsection{Classifier with Attention Mechanism}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{att.pdf}
\caption{The attention mechanism structure in our CNN model.}
\label{fig3}
\end{figure}
The classifier is a combination of an attention mechanism module and a basic classifier. Figure~\ref{fig3} depicts the structure of attention mechanism, which includes four $1 \times 1$ convolutional layers. The first three activation functions of convolutional layers are ReLU, the last convolutional layer is attached with softmax activation function. The input feature maps $A \in R^{H\times W\times C}$ are passed through the four convolutional layers to calculate attention mask $S\in R^{H\times W\times 1}$. We apply element-wise multiplication between the attention mask and input feature maps to get the final output feature map $O \in R^{H\times W\times C}$. Our basic classifier contains three fully connected (FC) layers. The output dimensions of the three FC layers are 512, 64, and 2. Dropout layers are used after the first two layers with dropout probability 0.5. 

\subsection{Loss Function}
In previous AD classification studies, researchers mainly concentrated on binary classification. In our work, we do the same for ease of comparison. The overall loss function is binary cross-entropy. For a 3D image $V$ with label $l$ and probability prediction $p(l|V)$, the loss function is:
\begin{equation}
loss(l,V)=-[l \cdot log(p(l|V))+(1-l) \cdot log(1-p(l|V))]
\label{eq:4}
\end{equation}
where the label $l=0$ indicates a negative sample and $l=1$ indicates a positive sample.

\section{Evaluation}
We use the publicly available dataset from the Alzheimer’s Disease Neuroimaging Initiative (ADNI)~\cite{ADNI} for our work. Specifically, we trained CNNs with the data from the ``spatially normalized, masked, and N3-corrected T1 images” category. The brain MRI image size is $110 \times 110 \times 110$. Since a subject may have multiple MRI scans in the database, we use the first scan of each subject to avoid data leakage. The total number of data samples is 100, containing 51 CU samples and 49 AD samples. 

The CNNs are implemented in PyTorch. We use five-fold cross validation to better evaluate model performance. The batch size used for our model is 16. The batch size of the baseline models is 8, which is the maximum batch size of the 3D CNN model trained on the single GTX-1080ti GPU. We use the Adam optimizer with $beta_1=0.9$ and $beta_2=0.999$. The learning rate is 0.0001. We train for 150 epochs. To evaluate the performance of our model, we use accuracy (Acc), the area under the curve of Receiver Operating Characteristics (ROC), F1 score (F1), Precision, Recall and Average Precision (AP) as our evaluation metrics. 

\subsection{Quantitative  Results}
High quality feature extraction is crucial for the final prediction. Different pre-trained CNN models can output different features in terms of size and effective receptive field. We test different pre-trained CNNs to find out which CNN models perform best as our feature extractor. Table~\ref{table1} shows various CNN models and the corresponding output feature size. 
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{The different pre-trained CNN model as feature extractors and the output feature sizes}
\label{table1}
\begin{tabular}{lll}
\hline\noalign{\smallskip}
CNN model & & Output feature size\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
AlexNet~\cite{Alex}  & & $256\times5\times5$ \\
VggNet11~\cite{Vgg} & & $512\times6\times6$ \\
ResNet18~\cite{He2015} & & $512\times7\times7$ \\
MobileNet\_v2~\cite{Sandler_2018_CVPR} & &$1280\times4\times4$ \\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\setlength{\tabcolsep}{4pt}

Since our dynamic image resolution is $110\times110\times3$, which is much smaller than the ImageNet dataset resolution: $256\times256\times3$, we use only part of the pre-trained CNN as the feature extractor. Directly using the whole pre-trained CNN model as feature extractor will cause the output feature size to be too small, which decreases the classification performance. In the implementation, we get rid of the maxpooling layer of each pre-trained model except for the MobileNet\_v2~\cite{Sandler_2018_CVPR}, which contains no maxpooling layer. Also, because there is a domain gap between the natural image and medical image we set the pre-trained CNN models' parameters trainable, so that we can fine tune the models for better performance.

\begin{table}
\begin{center}
\caption{The performance results of different backbone models with dynamic image as input}
\label{table2}
\begin{tabular}{llccccc}
\hline\noalign{\smallskip}
Model  & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
AlexNet  & 0.87 & 0.90 & 0.86 & 0.89 & 0.83 & 0.82 \\
ResNet18 & 0.85 & 0.84 & 0.84 & 0.86 & 0.81 & 0.79 \\
MobileNet\_v2 & 0.88 & 0.89 & 0.87 & 0.89 & 0.85 & 0.83 \\
VggNet11 & 0.91 & 0.92 & 0.91 & 0.88 & 0.93 & 0.86 \\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\begin{table}
\begin{center}
\caption{The performance results of different 2D and 3D CNN models}
\label{table3}
\begin{tabular}{llcccccc}
\hline\noalign{\smallskip}
Model  &$\quad$ & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}

3D-VGG~\cite{Korolev2017} &$\quad$  & 0.80 & 0.78 & 0.78 & 0.82 & 0.75 & 0.74 \\
3D-ResNet~\cite{Korolev2017}&$\quad$  & 0.84 & 0.82 & 0.82 & 0.86 & 0.79 & 0.78 \\
\hline
Max. + VGG11&$\quad$ & 0.80 & 0.77 & 0.80 & 0.78 & 0.81 & 0.73 \\ 
Avg. + VGG11&$\quad$ & 0.86 & 0.84 & 0.86 & 0.83 & 0.89 & 0.79 \\

Max. + VGG11 + Att&$\quad$ & 0.82 & 0.76 & 0.82 & 0.80 & 0.83 & 0.75 \\ 
Avg. + VGG11 + Att&$\quad$ & 0.88 & 0.89 & 0.88 & 0.85 & \textbf{0.91} & 0.82 \\
\hline
Ours  &$\quad$  & \textbf{0.92} &\textbf{0.95} & \textbf{0.91} & \textbf{0.97} & 0.85 & \textbf{0.90} \\

\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
When analyzing MRI images using computer-aided detectors (CADs), it is common to strip out the skulls from the brain images. Thus, we first test the proposed method using the MRI with the skull stripped. Our proposed model takes dynamic images (Dyn) as input, VGG11 as feature extractor, and a classifier with the attention mechanism: $Dyn + VGG11 + Att $. The whole experiment can be divided into three sections: the backbone and attention section, the baseline model section, and the pooling section. In the backbone and attention section, we use 4 different pre-trained models and test the selected backbone with and without the attention mechanism. Based on the performance shown in Table~\ref{table2}, we choose VGG11 as the backbone model. In the baseline model section, we compare our method with two baselines, namely 3D-VGG and 3D-ResNet. Table~\ref{table3} shows the performance under different CNN models. The proposed model achieves $9.52\%$ improvement in accuracy and $15.20\%$ better ROC over the 3D-ResNet. In the pooling section: we construct two baselines by replacing the approximate rank pooling module with the average pooling (Avg.) layer or max pooling (Max.) layer. The pooling layer processes the input 3D image over the z-dimension and outputs the same size as the dynamic image. Comparing with the different 3D-to-2D conversion methods under the same configuration, the dynamic image outperforms the two pooling methods.

\subsection{Pre-processing Importance Evaluation}

\begin{table}
\begin{center}
\caption{The performance results of different 2D and 3D CNN models on the MRI image with skull.}
\label{table4}
\begin{tabular}{lcccccccc}
\hline\noalign{\smallskip}
Model &$\quad$ & Acc & ROC &F1 & Precision & Recall & AP\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
3D-VGG~\cite{Korolev2017} &$\quad$  & 0.78 & 0.62 & 0.77 & 0.80 & 0.75 & 0.72 \\
Ours &$\quad$ & 0.63  & 0.52 & 0.63 & 0.62 & 0.64 & 0.57\\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\begin{figure}
\centering  
\includegraphics[scale=0.5]{MRIwskull.pdf}
\caption{The MRI sample slices with skull of the CU and AD participants and the corresponding dynamic images.}
\label{fig4}
\end{figure}

In this section, we show results using the raw MRI image (including skull) as input. We perform experiments on the same patients' raw brain MRI image with the skull included to test the performance of our model. The raw MRI image category is ``MT1,GradWarp,N3m". The image size of the raw MRI image is "$176 \times 256 \times 256$". Figure~\ref{fig4} illustrates the dynamic images of different participants' MRI brain images with the skull. The dynamic images are blurrier than the images under skull striping processing. This is because the skull variance can be treated as noise in the dynamic image. %
Table~\ref{table4} shows the significant performance decrease when using 3D Brain MRI images with skull. Figure~\ref{fig4} shows a visual representation of how the dynamic images are affected by including the skull in the image. In this scenario, the model can not sufficiently diagnose the different groups. A potential cause of this decrease in performance is that the approximate rank pooling module is a pre-processing step, and the module is not trainable. We believe an end-to-end, learnable rank pooling module would improve performance.%

\subsection{Models Training time}
\begin{table}
\begin{center}
\caption{The total 150 epochs training time of different CNN models.}
\label{table5}
\begin{tabular}{lcc}
\hline\noalign{\smallskip}
 &$\quad$ &Training time(s)  \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
3D-VGG~\cite{Korolev2017} & &2359 \\
3D-ResNet~\cite{Korolev2017} & &3916\\
Ours & &414\\
\hline
\end{tabular}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
Another advantage of the proposed model is faster training. We train all of our CNN models for 150 epochs on the same input dataset. Table~\ref{table5} shows the total training time of the different 2D and 3D CNN models. Compared with the 3D-CNN networks, the proposed model trains in about $20\%$ of the time. Also, due to the higher dimension of the 3D convolutional layer, the number of parameters of the 3D convolutional layer is naturally higher than the 2D convolutional layer. 
By applying the MobileNet~\cite{mobilenet} or ShuffleNet~\cite{shuffle} in medical image diagnosis, there is potential for mobile applications. We used MobileNet for our experiments. We used the MobileNet v1 achitecture as the feature extractor and obtained 84.84\% accuracy, which is similar in accuracy to the 3D ResNet.
\section{Conclusions}
We proposed to apply the approximate rank pooling method to convert 3D Brain MRI images into 2D dynamic images as the inputs for a pre-trained 2D CNN. The proposed model outperforms a 3D CNN with much less training time and improves 9.5\% better performance than the baselines. We trained and evaluated on MRI brain imagery and found out that brain skull striping pre-processing is useful before applying the approximate rank pooling conversion. We used an offline approximate rank pooling module in our experiments, but we believe it would be interesting to explore a learnable temporal rank pooling module in the future. 

\section*{Acknowledgement}
This work is supported by NIH/NIA R01AG054459. 
\bibliographystyle{splncs04}
\bibliography{egbib}
\end{document}


Peer review:
Dynamic Image for 3D MRI image Alzheimer's Disease classification
Summary
This paper addresses the problem of 3D MRI volume segmentation of images collected from patients with Alzheimer's disease. The motivation for the work comes from automating the classification task for assigning labels such as cognitive unimpaired (CU) and Alzheimer’s disease (AD) to each patient based on the 3D MRI volume. The authors approach the problem by  leveraging a 3D to 2D conversion using dynamic images according to https://openaccess.thecvf.com/content_cvpr_2015/papers/Fernando_Modeling_Video_Evolution_2015_CVPR_paper.pdf and https://www.egavves.com/data/cvpr2016bilen.pdf and introducing an attention module in a transfer model with pre-training on ImageNet dataset. The evaluations include (a) matching feature dimensionality of the features extracted from four well-established architectures and the features coming from the dynamic image based conversion, (b) optimizing the steps in feature extraction and AI-based classification, (c) analyzing inclusion/exclusion of skull, and (d) comparing execution times when 3D vs 2D raw data are used as inputs into classifiers.
Strengths:
The classification framework is very interesting.
The introduction of dynamic image and attention module is novel

Weaknesses:
The experimental dataset is very limited.
The theoretical description is not very clear.

Comments:
Is there any reason (e.g., based on visual inspection) to believe that the features of 2D dynamic images are of the same nature as the features extracted from ImageNet?
Line 129: what do you refer to when mentioning the ImageNet resolution? 
Line 171: why did you choose three activation functions and 1x1 convolutional kernels? 
The section 3.3 also did not explain the details in Fig 3. For example, why do you have in Fig 3 the same blocks but the tensor sizes are HxWx512 -> HxWx256 -> HxWx64? Shouldn’t the last tensor size be HxWx128?
What implementation did you use for the CAM attention module?
What is the method in dynamic image based conversion that you are using to create 110 x 110 x 3 (i.e., 3 features)? The original paper in https://openaccess.thecvf.com/content_cvpr_2015/papers/Fernando_Modeling_Video_Evolution_2015_CVPR_paper.pdf refers to learning rank machines but your paper does not mention this important detail.


Minor comments:
Fig 1 caption: you have one row of pictures but the caption refers to two rows.
Why applying dynamic image-based 3D to 2D conversion is preferred over z-axis? Is there any motivation to prefer one of the three possible planes, i.e., sagittal vs transversal vs coronal plane?
Rating: 8: Top 50% of accepted papers, clear accept
Confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct