# BTDNet: A Multi-Modal Approach for Brain Tumor Radiogenomic Classification
## 3 Materials and Method
### Proposed Method  
First, all input 3D volumes for each modality (FLAIR, T1w, T1 wCE, and T2) are fed to the data augmentation component. During training, this component applies a series of geometric transformations to the inputs and then performs a novel augmentation technique (MixAugment [22]) on them. During testing, this component applies test-time data augmentation [13] to the inputs.  
Then, all input 3D transformed volumes for each modality (FLAIR, T1w, T1 wCE, and T2) are padded with black slices to have lengths \(t_{1},t_{2},t_{3},t_{4}\) (e.g., each input FLAIR volume consists of \(t_{1}\) slices, whereas each input T1w volume consists of \(t_{2}\) slices). Each input 3D volume is first fed to the 3D analysis component. This component consists of a CNN [23], which performs local (per 2D image/slice) analysis, extracting features from each slice. Then, these features are fed to an RNN [24], which is placed on top of the CNN, to capture their temporal dependencies. The CNN and RNN networks perform global (per 3D volume) analysis. The RNN output features are then fed to the routing component, where they are concatenated and fed to a mask layer. This step is essential since we have annotations at the volume level (not at the slice level) and thus we know that all slices (not just independent slices) may convey important information for the final prediction of the network. This step is also important as it is the mask layer that dynamically selects RNN outputs, taking into account the input length, i.e., the 'true' number of slices of the currently analyzed volume. The output of the mask layer is then fed to a dense layer, which is equipped with batch normalization [14] and a GELU [15].  
It should be noted that the above-described procedure is followed for each modality. For example, the input 3D FLAIR volume is fed to the 3D analysis component. Similarly, the input 3D T1 volume is fed to the 3D analysis component that is identical and shares the same weights with the corresponding component that the FLAIR modality was fed to. Then, the output of the 3D analysis component of the FLAIR modality is fed to the routing component. Likewise, the output of the 3D analysis component of the T1 modality is fed to the routing component, which is identical and shares the same weights with the corresponding component of the FLAIR modality.  
The output features of the routing component for each modality are fed to the modality fusion component. The output features are concatenated and fed to a dense layer that maps them to the same feature space. Finally, the output layer follows, providing the final classification. In the following, we explain each component of our proposed method in more detail. Figure 1 provides an overview of our proposed framework, BTDNet.  
Figure 1: The proposed multimodal BTDNet takes as input mpMRI 3D volumes (FLAIR, T1w, T1 wCE, and T2 modalities). BTDNet consists of the following components: (i) data augmentation (geometric transformations, adapted MixAugment, test-time data augmentation); (ii) 3D analysis (CNN plus RNN); (iii) routing (mask layer, routing, dense layer); and (iv) modality fusion (dense layer, output layer). The symbol ‘+’ in a circle denotes the concatenation operation.