## 4 Implementation Details
### Training and Implementation Details  
To alleviate the irregularities of variable contrast in some MRI images, we normalize all input images (both CT and MRI) to have zero mean and unit std. The volumes were cropped and zero-padded to a fixed size of \(176\times 224\times 144\) to provide a fine ROI for the network input while making sure all heart structures are inside the selected ROI. We apply three different data augmentation methods on input image channels to increase the sample size of training data and enhance the robustness and generalization ability, namely random axis mirror flip, random scaling, and intensity shift. Random axis mirror flip creates a mirror reflection of an original image along one (or more) selected axis and is commonly flipped at a rate of 50%. Random scaling operation \(S\) scales input image and performs independently in different directions. Intensity shift performs an element-wise addition of a scalar to the image and affects the brightness of the original image. Details about parameters of used data augmentation methods are presented in Table 1 while examples of input images after applying different data augmentation methods are shown in Figure 4. Moreover, we empirically found that advanced augmentation techniques, such as random histogram matching, or random image filtering, do not show any additional improvements to the final segmentation result. Figure 4: Examples of different augmentation methods on input CT and MRI images. Figure 3: An example of one slice with corresponding ground truth from 3D volume across axial, coronal, and sagittal planes. The ground truths include seven heart structures: LV (red), RV (magenta), LA (blue), RA (green), Myo (yellow), Ao (orange), and PA (cyan). While optimization methods such as stochastic gradient descent (SGD) or gradient descent with momentum provide a powerful optimization for training CNNs, choosing the most optimal learning rate \(\alpha\) is not a trivial task. If \(\alpha\) is chosen to be too large, the training may not converge, might oscillate, or skip over relevant local minima. On the other hand, if it is chosen to be too small, it significantly delays the convergence process. Therefore, in this work, we use adaptive learning rate optimizer Adam with initial learning rate of \(\alpha_{0}=10^{-4}\) and gradually decrease it according to the following expression:  
\[\alpha=\alpha_{0}*\left(1-\frac{c}{T_{c}}\right)^{0.9} \tag{20}\]  
where \(T_{c}\) is a total number of epochs (200 in our case) and \(c\) is an epoch counter. Furthermore, to ensure our models generalizes well on unseen data, i.e., to reduce the effect of overfitting or underfitting, we employ \(L2\) norm regularization with a weight of \(10^{-5}\) and the spatial dropout with a rate of 0.2 after the initial encoder convolution. Since early stoping aims to regularize the finding the network parameters at the point of the lowest validation loss, we implement early stopping with patience set to 50. In our experiments, we train four encoder-decoder based architectures: (1) 3D Pre-ResNet without VAE regularization, (2) 3D Pre-ResNet with VAE regularization, (3) FM-Pre-ResNet without VAE regularization, and (4) FM-Pre-ResNet with VAE regularization. All four networks are trained from scratch and separately for CT and MRI images. The whole experimental procedure is implemented in Pytorch and trained on two NVIDIA Titan V GPU simultaneously. The source code of our method is available at [33]. Our training and validation dataset consists of 20 CT volumes and 20 MRI volumes, with 80-20% training and validation split, respectively. The trained networks are evaluated using a testing dataset that includes 40 subjects for both CT and MRI images. The proposed 3D FM-PreResNet + VAE architecture produced the highest validation accuracy of 94.40% at the end of the 200 th epoch.