###### Abstract  
Assessment of magnetic resonance imaging (MRI) after neoadjuvant chemoradiation therapy (nCRT) is essential in rectal cancer staging and treatment planning. However, when predicting the pathologic complete response (pCR) after nCRT for rectal cancer, existing works either rely on simple quantitative evaluation based on radiomics features or partially analyze multi-parametric MRI. We propose an effective pCR prediction method based on novel multi-parametric MRI embedding. We first seek to extract volumetric features of tumors that can be found only by analyzing multiple MRI sequences jointly. Specifically, we encapsulate multiple MRI sequences into multi-sequence fusion images (MSFI) and generate MSFI embedding. We merge radiomics features, which capture important characteristics of tumors, with MSFI embedding to generate multi-parametric MRI embedding and then use it to predict pCR using a random forest classifier. Our extensive experiments demonstrate that using all given MRI sequences is the most effective regardless of the dimension reduction method. The proposed method outperformed any variants with different combinations of feature vectors and dimension reduction methods or different classification models. Comparative experiments demonstrate that it outperformed four competing baselines in terms of the AUC and F1-score. We use MRI sequences from 912 patients with rectal cancer, a much larger sample than in any existing work. convolutional neural network (CNN); magnetic resonance imaging (MRI); neoadjuvant chemoradiation therapy (nCRT); pathologic complete response (pCR); radiomics; rectal cancer +
Footnote †: journal:  
To predict the pCR of rectal cancer, radiologists have used various MRI sequences, such as T2-weighted images (T2), diffusion-weighted imaging (DWI) [4; 5], and contrast-enhanced imaging (CE) [6]. While T2 is considered as an essential MRI sequence, radiologists can achieve higher accuracy by using DWI along with T2 than using T2 alone [7]. This can be improved further by replacing T2 and DTW with T2/Gabor (T2 after applying the Gabor filter) and DWI/ADC (apparent diffusion coefficient of DWI) [8; 9]. To quantitatively evaluate MRI for the pCR prediction of rectal cancer, many prior studies [8; 10; 11; 12] have focused on radiomics features that can quantify the texture and non-texture characteristics of tumors. For example, a random forest classifier on T2/Gabor radiomics features outperformed qualitative analysis of T2 and DWI by radiologists [8]. By merging the radiomics features of multi-parametric MRI [10] and additional information, such as tumor length [11], simple classifiers based on multi-layer perceptron (MLP), and logistic regression, have shown high pCR prediction accuracy. Recently, convolutional neural network (CNN) architectures have been widely used to extract new features of tumors in medical images, such as MRI and CT/PET [13; 14; 15; 16; 17; 18]. Using 2D-CNN pre-trained on non-medical images, 2D features of the tumor are extracted from CE MRI and used for an effective logistic regression classifier [13; 14]. To improve the pCR prediction accuracy, 2D-features from multi-parametric MRI [15] and radiomics features [16] can be combined. Some approaches have used pre-trained 3D-CNN to extract 3D features of tumor volume [17; 18]. However, they neither analyze multi-parametric MRI nor consider radiomics features; they analyze 3D CT/PET images [18] or the DWI/ADC MRI sequence [17] only. In this study, given pre-operative MRI sequences, {T2, DWI/ADC, and CE}, we predict the pCR of rectal cancer after nCRT by using multi-parametric MRI embedding. Specifically, we focus on extracting 3D features of tumor volume and radiomics features and fusing them to generate novel and diverse features of multi-parametric MRI. To this end, we encapsulate multiple MRI sequences into a multi-sequence fusion image (MSFI) and extract features directly from it, instead of simply merging the features extracted from each MRI sequence.