# A Multi-Channel Uncertainty-Aware Multi-Resolution Network for MR to CT Synthesis
## 1 Introduction  
Simultaneous positron emission tomography and magnetic resonance imaging (PET/MRI) is an important tool in both clinical and research applications, allowing a multiparametric evaluation of the subject. MRI provides information with high soft-tissue contrast and PET gives information about the radiotracer uptake distribution. For the PET reconstruction process, it is essential to perform photon attenuation correction (AC) throughout the subject. A multi-center study on brain images has shown that obtaining tissue attenuation coefficients from synthesised computed tomography (CT) images leads to state-of-the-art results for PET/MRI AC [1]. Within the last years, many research groups have utilised convolutional neural networks (CNNs) in the field of MR to CT synthesis that have proved to be a powerful tool in the image-to-image translation task, outperforming existing multi-atlas-based methods [2, 3]. However, little progress has been made in the area of whole-body MR to CT synthesis. The biggest hurdle with high-resolution whole-body data is its size and the fact that a large field of view is crucial for making accurate pseudo-CT (pCT) predictions. In 2019, [PERSON] et al. [4] attempted to translate whole-body MR images to CT images by introducing a multi-view adversarial learning scheme that predicts 2D pCT images along three axes (axial, coronal, sagittal). They obtain 3D volumes for each axis by stacking 2D slices together, followed by an average fusion that results inthe final 3D volume. The synthesis performance is evaluated on multiple sub-regions of the body (lungs, femur bones, spine etc.), however, the authors do not provide results on the full volume.  
Learning image features from multiple levels of resolution has been used for many computer vision tasks including dynamic scene deblurring [5], optical flow prediction [6] and depth map estimation [7]. In the field of medical imaging, multi-resolution learning has been utilised to solve image classification [8], super-resolution [9] and segmentation [10] tasks. These methods learn strong features at multiple levels of scale and abstraction, therefore finding the input/output voxel correspondence based on these features. The large image size of whole-body images and GPU memory constraints do not allow for high-resolution 3D image synthesis networks to be trained on full images. Thus, training is performed in a patch-wise manner, which only captures a limited amount of spatial context.  
Modelling uncertainty has gained popularity in the field of medical imaging, specifically in areas like image translation, segmentation, and super-resolution [11; 12; 13] and still remains an active field of research. It is important to distinguish between two types of uncertainty: _aleatoric_ and _epistemic_ uncertainty [14]. Aleatoric uncertainty captures the irreducible variance that exists in the data, whereas epistemic uncertainty accounts for the uncertainty in the model [15]. Aleatoric uncertainty can be further divided into two subcategories: _homoscedastic_ and _heteroscedastic_ uncertainty. Homoscedastic uncertainty is constant across all input data, while heteroscedastic uncertainty varies across the input data. In this work, we propose the use of a deep learning framework for multi-resolution image translation specifically designed for whole-body MR to CT synthesis (MultiRes). We demonstrate that incorporating feature maps learned at multiple resolutions results in a significantly better synthesis result compared to using high-resolution images only. We further present the benefits afforded by incorporating model uncertainty [16] at multiple resolutions for whole-body MR to CT synthesis.  
This paper is an extension of preliminary work [17]. We extend the framework to allow for multi-channel inputs and add additional validation on a dataset showing state-of-the-art synthesis results.