## 3 Method
### Suggested Method  
#### 3.2.1 Representing Multiple MRI Sequences as MSFI Embedding  
Figure 2 depicts our pCR prediction process used to transform given multi-parametric MRI sequences into embedding. To extract features of tumor volumes, we highlight tumor areas in each MRI image and select the MRI images related to the major tumor volume as follows. First, we highlight the tumor area in each MRI image by filling the region outside its tumor mask with zeros. Then, to select contiguous slices capturing tumor volume, we find the slice with the largest tumor area and pick five and six slices above and below the slice, respectively, in each MRI sequence. Figure 1: Three types of MRI images {T2, DWI/ADC, and CE} of a pCR patient (**upper**) and a non-pCR one (**lower**). Yellow masks are rectal tumor areas segmented and validated by radiologists. If the tumor size exceeded 12 slices, a total of 12 slices were used above and below the central section with the largest tumor area. The reason is that after nCRT, the viable portion of the tumor is mainly found in the central region of the tumor, and the border region of the tumor has a very small volume or is observed as a streak-like fibrosis, making it difficult to represent the characteristics of the entire tumor volume. In our study, the number of patients whose tumor size exceeded 12 slices is relatively small (22/592 in the training set and 17/320 in the test set). Lastly, since the input resolution of the 3D-CNN used for transfer learning is 112 \(\times\) 112, we center-cropped each slice around the tumor area accordingly. To alleviate data scarcity and avoid overfitting, we apply data augmentation techniques and transfer learning. We use data augmentation techniques, such as 3D-rotation and 3D-shift [34] during the training stage to increase the size and variety of the training set. We exclude some image augmentation techniques, such as adding Gaussian noise and  
Figure 2: The pCR prediction process using the 3D-CNN classifier from which we extract MSFI embedding, given {T2, DWI/ADC, and CE} MRI sequences. applying a median filter, because they often distort the texture of tumor areas, which is an essential characteristic of tumors for pCR prediction [35]. In addition to extracting the diverse features from each MRI sequence, we aim to examine novel features that can be found only by considering multiple MRI sequences jointly. For this, we transform the given three MRI sequences, {T2, DWI/ADC, and CE}, into an MSFI. The MSFI is a sequence of slices containing 3D values \((v_{1},v_{2},v_{3})\), where \(v_{1}\), \(v_{2}\), and \(v_{3}\) are from T2, DWI/ADC, and CE, respectively. After encapsulating three MRI sequences into the MSFI, we use it as an input for deep learning to represent it as MSFI embedding. To extract the volumetric features of tumors, we use a 3D-CNN model that is known for its high classification performance on video data. Unlike 2D-CNN, 3D convolutional filters can identify patterns that appear across multiple image slices. As there are many 3D convolutional filters to tune in a deep 3D-CNN model, we perform transfer learning to improve generalization ability [36; 37], instead of training randomly initialized 3D filters. For this, we used 3D-ResNet [38] pre-trained on Kinetic [39], a large-scale, high-quality video dataset that contains 400 classes with at least 400 videos per class and is considered as a de facto standard for the research on 3D image processing. 3D-ResNet is known to distinguish 3D instances very effectively by reducing the gradient-vanishing effect through gradient flow. This means that its pre-trained 3D filters can already extract useful volumetric features from 3D instances. Therefore, by fine-tuning the pre-trained filters in 3D-ResNet, we can construct more effective 3D filters for MSFI embedding extraction. Figure 3 shows the architecture of 3D-ResNet, the 3D-CNN that we use for MSFI embedding extraction. We obtain MSFI embedding from the fully connected layer of the trained 3D-ResNet.