# BTDNet: A Multi-Modal Approach for Brain Tumor Radiogenomic Classification
## 1 Introduction
However, the fact that the whole volume has one label does not mean that each slice in the volume exhibits the MGMT promoter methylation status; it could be the case that only some slices display the MGMT promoter methylation status. In cases where the volumes have variable lengths, traditional approaches use ad hoc strategies. They select a fixed volume length and either remove slices when a larger length is encountered (thus losing information that could be important for the final decision) or duplicate slices when the volume contains a smaller number of slices (this duplication negatively affects the final decision, as the model becomes biased toward the repeating data) [10; 11; 12]. Moreover, this ad hoc way of selecting the fixed volume length needs empirical tuning for each different database. In this paper, we propose BrainTumorDetectionNetwork (BTDNet), a multi-modal approach for predicting the MGMT promoter methylation status. We develop a multi-modal approach, as leveraging data from multiple modalities helps the method (i) have a richer data representation since different modalities provide complementary information; (ii) reduce redundancy and ambiguity in data interpretation; and (iii) mitigate data scarcity. BTDNet takes as input an mpMRI scan, i.e., a FLAIR 3D volume, a T1w 3D volume, a T1 wCE 3D volume, and a T2 3D volume. BTDNet consists of four components: (i) data augmentation; (ii) 3D analysis; (iii) routing; and (iv) modality fusion. At first, while training, each input volume is passed through the data augmentation component, which initially applies geometric transformations to the input volume and then performs a novel augmentation technique on it. It should be noted that, during inference, we apply test time data augmentation [13] to the input volume. The transformed input volume is then processed by the 3D analysis and routing components. The 3D analysis component consists of a CNN network acting as a feature extractor and is applied to each slice of the volume. The features are then fed to an RNN that captures temporal information within the slices of the same volume. The RNN's features are then passed to the routing component, consisting of a mask layer, which dynamically selects specific RNN outputs, followed by a dense layer (i.e., fully connected layer equipped with batch normalization [14] and a GELU [15]), whose output features are then fed to the modality fusion component. The mask layer is utilized for handling variable input feature lengths (due to variable slices per volume) when training the network. The 3D analysis and routing components are identical and share the same weights for each of the four input 3D volumes (modalities). This means that there exist four identical 3D analysis and routing components. The modality fusion component is fed with the outputs of the routing component of all four modalities; these outputs are first concatenated and then fed to a dense layer that maps them to the same feature space. Then, the output layer follows and performs the final classification. Our method is trained using an extension of the focal loss [16] for multi-class classification. The rest of this paper is organized as follows. Section 2 provides a comprehensive review of related works that address the problem of predicting the MGMT promoter methylation status. Section 3 presents the proposed method, along with its novelties. In addition, the dataset and pre-processing techniques utilized are described, along with the performance metrics used to evaluate our approach and the implementation details regarding our approach. Section 4 presents the rich experimental study in which we compare our method's performance to that of the state-of-the-art, as well as various ablation studies. Finally, Section 5 provides the conclusions and discusses future work and extensions of the proposed method.