# Robust deep MRI contrast synthesis using a prior-based and task-oriented 3D network

Sergio Morell-Ortega, Marina Ruiz-Perez, Marien Gadea, Roberto Vivo-Hernando, Gregorio Rubio, Fernando Aparici, Mariam de la Iglesia-Vaya, Thomas Tourdias, Boris Mansencal, Pierrick Coupé, et al.

<figure><img src="image_1.png" /></figure>

To cite this version:

Sergio Morell-Ortega, Marina Ruiz-Perez, Marien Gadea, Roberto Vivo-Hernando, Gregorio Rubio, et al.. Robust deep MRI contrast synthesis using a prior-based and task-oriented 3D network. Imaging Neuroscience, 2025, 3, 10.1162/IMAG.a.116 . hal-05232296

HAL Id: hal-05232296

https://hal.science/hal-05232296v1

Submitted on 1 Sep 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Robust deep MRI contrast synthesis using a prior-based and task-oriented 3D network

Sergio Morell-Ortegaª, Marina Ruiz-Perezª, Marien Gadeaᵇ, Roberto Vivo-Hernandoᶜ, Gregorio Rubioᵈ, Fernando Apariciᵉ, Mariam de la Iglesia-Vayaᵉ, Thomas Tourdiasᵉ, Boris Mansencalᵇ, Pierrick Coupéᵇ, José V. Manjónª

Corresponding Author: Sergio Morell-Ortega (sermoor1@teleco.upv.es)

## ABSTRACT

Magnetic resonance imaging (MRI) is one of the most widely used tools for clinical diagnosis. Depending on the acquisition parameters, different image contrasts can be obtained, providing complementary information about the patient's anatomy and potential pathological findings. However, multiplying such acquisitions requires more time, additional resources, and increases patient discomfort. Consequently, not all image modalities are typically acquired. One solution to obtain the missing modalities is to use contrast synthesis methods. Most existing synthesis methods work with 2D slices due to memory limitations, which produces inconsistencies and artifacts when reconstructing the 3D volume. In this work, we present a 3D deep learning-based approach for synthesizing T2-weighted MR volumes from T1-weighted ones. To preserve anatomical details and enhance image quality, we propose a segmentation-oriented loss function combined with a frequency space information loss. To make the proposed method more robust and applicable to a wider range of image scenarios, we also incorporate a priori information in the form of a multi-atlas. Additionally, we employ a semi-supervised learning framework that improves the model's generalizability across diverse datasets, potentially improving its performance in clinical settings with varying patient demographics and imaging protocols. By integrating prior anatomical knowledge with frequency domain and segmentation loss functions, our approach outperforms state-of-the-art methods, particularly in segmentation tasks. The method demonstrates significant improvements, especially in challenging cases, compared with state-of-the-art approaches.

**Keywords:** MRI, contrast synthesis, deep learning, semi-supervised learning

## 1. INTRODUCTION

Magnetic resonance imaging (MRI) can noninvasively acquire images with different contrasts of the same anatomical region, with each contrast providing complementary

information. For instance, T1-weighted (T1) images clearly differentiate brain gray matter from white matter, while T2-weighted (T2) images distinguish cortical gray matter from cerebrospinal fluid, and FLAIR imaging assists in lesion detection.

<figure><img src="image_3.png" /></figure>

<footnote>ªInstituto de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Valencia, Spain</footnote>

<footnote>ᵇDepartment of Psychobiology, Faculty of Psychology, Universitat de València, Valencia, Spain</footnote>

<footnote>ᶜInstituto de Automática e Informática Industrial, Universitat Politècnica de València, Valencia, Spain</footnote>

<footnote>ᵈDepartamento de matemática aplicada, Universitat Politècnica de València, Valencia, Spain</footnote>

<footnote>ᵉÁrea de Imagen Médica, Hospital Universitario y Politécnico La Fe, Valencia, Spain</footnote>

<footnote n="1">¹Unidad Mixta de Imagen Biomédica FISABIO-CIPF, Fundación para el Fomento de la Investigación Sanitario y Biomédica de la Comunidad Valenciana—Valencia, Spain</footnote>

<footnote>ᵉUniv. Bordeaux, INSERM, Neurocentre Magendie, U1215, Bordeaux, France</footnote>

<footnote>ᵇCNRS, Univ. Bordeaux, Bordeaux INP, LABRI, UMR5800, in2brain, Talence, France</footnote>
The literature confirms that combining T1 and T2 MRI improves downstream tasks. For the segmentation of brain anatomy or lesions, multimodal pipelines (T1 + T2/ FLAIR) greatly reduce misclassification errors (Lindig et al., 2018; Morell-Ortega et al., 2025; Ruiz-Perez et al., 2024; Viviani et al., 2017). In disease classification and grading, adding T2 data significantly improves accuracy and AUC (Zeng et al., 2024). Besides, in image reconstruction, the use of multimodal information helps to recover high-quality images from limited data (Xiang et al., 2019).

However, acquiring different image contrasts requires modifying the MR acquisition parameters and obtaining additional images. This extended acquisition process is costly, time consuming, resource intensive, and increases patient discomfort. Consequently, in recent years there has been growing interest in automatic image synthesis as an efficient way to enhance image analysis (e.g., brain tissue segmentation; Manjón et al., 2022) at a lower acquisition cost.

For example, Iglesias et al. (2013) were among the first to propose image synthesis to enable the use of Free-surfer software by synthesizing T1 images from original T2 images, since the software was originally designed to work exclusively with T1 images.

Earlier methods for image synthesis included non-linear registration-based (Burgos et al., 2014; Miller et al., 1993) and intensity transformation-based methods (Iglesias et al., 2013; Jog et al., 2017; Roy et al., 2013). More recently, Deep learning techniques have emerged as a method capable of learning non-linear mappings for image synthesis. A recent review by Dayarathna et al. (2024), provides a comprehensive analysis of widely used methods, offering valuable insights into the current state of the field.

The most commonly used networks for image-to-image translation are based on GANs (Generative Adversarial Networks), as for example, the work done in Dar et al. (2019), where they also leverage correlated 2D information across neighboring cross sections within a volume. These methods used mainly 2D approaches due to the memory overhead of their generator-discriminator approach (Dar et al., 2019; Liu et al., 2021), but there are also 3D patch-based GAN methods that deal with the high memory consumption using small volume patches (Uzunova et al., 2020; Zhao et al., 2021).

Various models have been employed for the generator architecture. For example, the well-known U-Net and variants have been used in several methods (Manjón et al., 2022; Osman & Tamam, 2022). More advanced architectures, including recent methods based on vision transformers, have also been proposed (Dalmaz et al., 2022; Y. Li et al., 2022; J. Liu et al., 2022; K. Pan et al., 2022; Yan et al., 2022).

The most recent methods for image synthesis using deep learning are based on diffusion models due to their high ability to sample the target distribution. Meng et al. (2022) proposed the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM), using conditional diffusion and reverse generation in the complete modality space. Therefore, they can learn a comprehensive set of cross-modal conditional distributions. Due to promising results in diffusion, Yoon et al. (2023) proposed a sequence-aware diffusion model (SADM) to generate longitudinal medical images. This design facilitates learning longitudinal dependencies even in the presence of missing data during training and supports the autoregressive generation of image sequences during inference. Pan et al. (2023) developed a novel Cycle-guided Denoising Diffusion Probability Model (CG-DDPM) for cross-modality MRI synthesis. It utilizes two interconnected DDPMs that condition one another to generate synthetic images from two distinct MRI pulse sequences. During the reverse processes, the two DDPMs exchange random latent noise, which serves to regularize both models and produce consistent images across the two modalities. This approach enhances the accuracy of image-to-image translation. Arslan et al. (2024) proposed a novel self-consistent recursive diffusion bridge (Self-RDB) that employs an innovative noise scheduling strategy in its forward process, where the variance progressively increases until reaching the end point, representing a noise-enhanced source image.

Other studies have explored untrained generative priors (e.g., deep image priors; Ulyanov et al., 2018) and learned diffusion priors for medical images, which can provide structured regularization in reconstruction and synthesis tasks. Besides, hybrid architectures that embed generative modules or physical models into neural networks have also been proposed. For example, Atli et al. (2024) introduced I2I-Mamba, a hybrid state-space/CNN model for multi-modal MRI synthesis.

As noted, most image synthesis approaches are 2D-based or patch-based methods. However, it is well known that 3D methods are generally more consistent, accurate, and efficient (Crespi et al., 2022; Shivdeo et al., 2021). To reduce memory consumption, approaches such as 2D to 3D mapping (Zhu et al., 2023) or 3D spatial-to-depth encoding (Manjón et al., 2022) have been used. Manjón et al. (2022) proposed a modified version of U-Net designed to handle entire 3D volumes. This method works by splitting the input volume into eight smaller sub-volumes using a decimation technique in a mosaic pattern. This approach enables the use of a greater number of filters, enhancing the model's capability to process the data effectively. Zhang et al. (2022) introduced a novel 3D patch-based MRI synthesis framework and a pyramid
transformer network (PTNet3D) that relies on attention
mechanisms implemented through transformer layers.

In summary, many of the proposed methods rely on
GANs or diffusion models, which can generate highly
realistic results. However, GANs have the risk of halluci-
nating structural details, potentially introducing artifacts
in the generated images (Teneggi et al., 2023). However,
diffusion models, despite delivering excellent results, are
often time consuming due to their iterative nature (Teneggi
et al., 2023). Moreover, both approaches typically use 2D
techniques because their complexity and memory
demands make infeasible to work in 3D, which can lead
to inconsistencies across 3D slices, compromising the
continuity and accuracy of the generated images.

In this work, we propose a task-oriented approach for
synthesizing 3D T2 MR images from T1 ones. Our method
uses a lightweight 3D architecture specifically designed
for synthesis tasks, enhanced by prior information in the
form of a multi-atlas derived from real T2 images. Addi-
tionally, we explored both global and local loss terms—
such as spatial, frequency-based, and segmentation-task
losses—to ensure consistency, precision, and effective-
ness in the regression task. Furthermore, we imple-
mented a semi-supervised approach to increase the
variability of the training set, thereby enhancing the
robustness of the final model.

## 2. METHODS

### 2.1. Datasets

This study utilizes six distinct datasets. Because images
captured with different scanners or acquisition parame-
ters naturally exhibit varying characteristics, incorporat-
ing multiple databases helps enhance the model's
robustness to these variations (four of the datasets
include paired T1 and T2 MR brain images). These data-
sets are:

* **HCP dataset:** This dataset consists of 1,104 T1/T2 MR images from the Human Connectome Project (HCP) (Van Essen et al., 2012). This dataset consists of MR images taken on a 3T machine from healthy subjects aged between 22 and 35 years. For T1-weighted imaging, the acquisition was 3D MPRAGE (TR = 2,400 ms, TE = 2.14 ms, TI = 1,000 ms, Flip Angle: 8°, FOV = 224 mm × 224 mm, Voxel Size: 0.7 mm isotropic, BW = 210 Hz/Px, iPAT = Factor of 2, Acquisition Time: Approximately 7 minutes and 40 seconds). For T2-weighted imaging, the sequence is 3D T2-SPACE (TR = 3,200 ms, TE = 565 ms, Flip Angle: Variable, FOV = 224 mm × 224 mm, Voxel Size: 0.7 mm isotropic, BW = 744 Hz/Px, iPAT: Factor of 2, Acquisition Time: Approximately 8 minutes and 24 seconds).

* **IXI dataset:** This dataset consists of 580 T1/T2 MR images from the IXI Project (https://brain-development.org/ixi-dataset/) acquired in 1.5T and 3T machines. The 3T T1 images were acquired on a Philips Intera 3T scanner (TR = 9.6 ms, TE = 4.6 ms, flip angle = 8°, volume size = 256 × 256 × 150 voxels, voxel dimensions = 0.94 × 0.94 × 1.2 mm³). The 1.5T T1 images were acquired on a Philips Gyroscan 1.5T scanner (TR = 9.8 ms, TE = 4.6 ms, flip angle = 8°, slice thickness = 1.2 mm, volume size = 256 × 256 × 150, voxel dimensions = 0.94 × 0.94 × 1.2 mm³). The parameters of the 3T T2 images were (TR = 5,725 ms, TE = 100 ms, flip angle = 90°, volume size = 192 × 187 × 187 voxels, voxel dimensions = 0.94 × 0.94 × 1.2 mm³) and those for the 1.5T images were (TR = 8,178 ms, TE = 100 ms, flip angle = 90°, volume size = 192 × 187 × 187 voxels, voxel dimensions = 0.94 × 0.94 × 1.2 mm³). Images from subjects presenting excessive artifacts or inconsistencies with their corresponding paired image were excluded using a correlation analysis resulting in the final number of subjects used of 420. The age range of the subjects of this dataset is between 24 and 75 years.

* **OpenNeuro dataset:** This dataset comes from OpenNeuro platform (https://OpenNeuro.org). It contains 269 T1 and T2 MR images from various projects. Details can be found on their specific papers (Hanke et al., 2018; Rogers et al., 2022; Pinho et al., 2018; Racey et al., 2023; Wolna & Wodniecka, 2023). The age range of the subjects of this dataset is between 19 and 81 years.

* **Bordeaux dataset:** This dataset consists of 44 T1/T2 MR images from a private dataset acquired in a 3T scanner (Vantage Galan 3T/ZGO; Canon Medical Systems) in Bordeaux Hospital as a part of the DeepMultiBrain research project. In this dataset, T1 and T2 images had a matrix size of 256 x 376 x 368 voxels and a voxel size of 0.6 x 0.6 x 0.6 mm³. The age range of the subjects of this dataset is between 20 and 64 years.

* **OASIS1 dataset:** This dataset contains 416 T1 MR images from Open Access Series of Imaging Studies (OASIS) (https://sites.wustl.edu/oasisbrains/). We randomly select a set of 100 cases. The MPRAGE images (TR = 9.7 ms, TE = 4 ms, TI = 20 ms, flip angle = 10°, slice thickness = 1.25 mm, matrix size = 256 x 256, voxel dimensions = 1 x 1 x 1.25 mm³ resliced to 1 mm³, averages = 1) were acquired on a 1.5T Vision scanner (Siemens,
Erlangen, Germany). The age range of the subjects of this dataset is between 18 and 96 years.

* Lifespan dataset: This last dataset is made of a set of quality curated 3,067 standard resolution T1 images from various public databases (age from 3 to 90 years) previously used in a project to construct a lifespan model of the human brain. Details can be found in Coupé et al. (2017). These databases are the following:

1. NDAR (N = 403): The Database for Autism Research (NDAR) is a national database funded by NIH (https://ndar.nih.gov). This database included 13 different MRI cohorts acquired on 1.5T and 3T scanners. In our study, we used 206 images of control subjects from the NIHPD (https://www.bic.mni.mcgill.ca/nihpd_info/info2/data_access.html) dataset and 197 images of control subjects from the Lab Study 19 of National Database for Autism Research. For the NIHPD, T1-weighted images were acquired at six different sites with 1.5T systems by General Electric and Siemens Medical Systems. The MR images are 3D T1-weighted spoiled gradient recalled (SPGR) echo sequence (the parameters are TR = 22–25 ms, TE = 10–11 ms, flip angle = 30°, FoV= 256 mm IS x 256 mm AP, matrix size = 256 x 256: 1 x 1 x 1 mm³ voxels, 160–180 slices of sagittal orientation). The participants chosen from the Lab Study 19 of National Database for Autism Research (NDAR) were scanned using a 3T Siemens Tim Trio scanner at each site. The MR images are 3D MPRAGE sequence (voxel dimensions: 1.0 x 1.0 x 1.0 mm³; image dimensions: 160 x 224 x 256, TE = 3.16 ms, TR = 2,400 ms).

2. AIBL (N = 343): The Australian Imaging, Biomarkers and Lifestyle (AIBL) dataset. The imaging protocol was defined to follow ADNI's guideline on the 3T scanner (https://adni.loni.usc.edu/data-samples/adni-data/neuroimaging/mri/mri-scanner-protocols/) and a custom MPRAGE sequence was used on the 1.5T scanner.

3. ADNI (N = 1,772): The images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset consisted of subjects from the 1.5T and 3T MR in collections 1 and 2. These images were acquired at different sites across the United States and Canada. ADNI1: The images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (http://adni.loni.usc.edu)

used in this study consist of 228 control subjects from the 1.5T baseline collection. These images were acquired on 1.5T MR scanners at 60 different sites across the United States and Canada. A standardized MRI protocol to ensure cross-site comparability was used. Typical MR images are 3D sagittal MPRAGE (TR = 2,400 ms, minimum full TE, TI = 1,000 ms, flip angle: 8°, 24 cm FoV, and a 192 x 192 x 166 acquisition matrix in the x-, y-, and z- dimensions, yielding a voxel size of 1.25 x 1.25 x 1.2 mm³, later reconstructed to get 1 mm³ isotropic voxel resolution). ADNI2: The images from the ADNI2 database (second phase of the ADNI project) consist of 215 control subjects. Images were acquired on 3T MR scanners with the standardized ADNI-2 protocol, available online (www.loni.usc.edu). Typical MR images are T1-weighted 3D MPRAGE sequence (TR = 2,300 ms, TE = 2.98 ms, flip angle 9°, FoV = 256 mm, resolution 1.1 x 1.1 x 1.2 mm³).

4. ICBM (N = 293): The images from the International Consortium for Brain Mapping (ICBM) dataset (http://www.loni.usc.edu/ICBM/) were obtained on 293 subjects through the LONI website. The MR images are T1-weighted MPRAGE (fast field echo, TR = 17 ms, TE = 10 ms, flip angle = 30°, 256 x 256 matrix, 1 mm² in plane resolution, 1 mm thick slices) acquired on a 1.5T Philips GyroScan imaging system (Philips Medical Systems, Best, The Netherlands).

5. ABIDE (N = 256): The images from the Autism Brain Imaging Data Exchange (ABIDE) dataset (http://fcon_1000.projectsnitrc.org/indi/abide/) used in this study consist of 256 subjects acquired at 20 different sites on 3T scanners.

The datasets HCP, IXI, and OpenNeuro were used for supervised training for in-domain test. The Bordeaux and OASIS datasets were used for out-of-domain test. Finally, the Lifespan dataset was used for semi-supervised training. Table 1 summarizes the used datasets.

## 2.2. Preprocessing

The preprocessing stage is essential to locate the images into a standardized geometric and intensity space which simplifies the learning process.

Firstly, all the images (T1 and T2) were denoised using a spatially adaptive non-local means filter (Manjón et al., 2010). Then, for the standardization of geometric spaces, a rigid registration parameter estimation between T1 and
<table><caption>**Table 1.** Summary of the six datasets used throughout the study.</caption><thead><tr><th>DATASET</th><th>HCP (T1&T2)*</th><th>IXI (T1&T2)*</th><th>OpenNeuro (T1&T2)*</th><th>Bordeaux (T1&T2)†</th><th>OASIS (T1)†</th><th>Lifespan (T1)†</th></tr></thead><tbody><tr><td>Train</td><td>1,089</td><td>414</td><td>258</td><td></td><td></td><td>3,067</td></tr><tr><td>Test</td><td>10</td><td>3</td><td>7</td><td>44</td><td>100</td><td></td></tr><tr><td>Val</td><td>5</td><td>3</td><td>4</td><td></td><td></td><td></td></tr></tbody></table>

HCP, IXI, and OpenNeuro, which makes up the train dataset, are indicated by (*). Bordeaux and OASIS are only used for testing and are indicated by (†), and Lifespan for the semi-supervised training approach are indicated by (‡).

T2 images (T2 was registered to the space of the same subject T1) was performed. Later, an affine registration of the T1 image to the MNI152 space at 1 mm³ using the ANTS software (Avants et al., 2009) was done. Finally, the T2 image was also registered to the MNI152 space by concatenating the rigid (T2 to T1) and affine T1-based transformations. Afterward, we performed an inhomogeneity correction of the registered T1 and T2 images using the N4 method (Tustison et al., 2010).

Finally, we standardized the intensity levels across our data. It is crucial to recognize that generating a T2 image from a T1 image requires learning and applying a specific intensity transformation. Therefore, normalizing intensities during preprocessing is essential. This step aligns the intensity distributions between the source and target images, thereby reducing complications arising from the diverse acquisition protocols of T1 (MPRAGE, SPGR, etc.) and T2 images (a many-to-many mapping problem). By implementing contrast stretching techniques during preprocessing, we ensure that both sets of images share similar intensity properties, ultimately facilitating the synthesis process. This fact has already been studied in Reinhold et al. (2019) where they examined the importance of image normalization as an essential step in preprocessing, analyzing seven normalization algorithms. Their findings concluded that normalization is crucial for accurate MRI synthesis.

In our case, two techniques for intensity normalization were used. For T1 images, we used an approach based on Trimmed Means Segmentation (TMS) (Manjón et al., 2008) that adjusts the histogram using a piece-wise linear function. The method uses the mean values of the different brain tissues for matching. For T2 images, we used a modified version of histogram matching (Gonzales & Fittes, 1977). Histogram matching has been widely used to normalize image intensities. However, when the origin and target images have very different tissue proportions (for example, small and large ventricles), it can produce incoherent results. To minimize this problem, before applying histogram matching, we non-linearly register the reference image to the target image to minimize these shape variations using a deep learning-based non-linear registration network (Coupé et al., 2020). Finally, both T1 and T2

images were renormalized by dividing each by its mean value, mapping them into a range better suited for neural network training.

## 2.3. Neural network architecture

In medical image synthesis tasks, the U-Net architecture (Ronneberger et al., 2015) is a popular choice. However, when dealing with entire 3D images, the memory requirements are high, which limits the number of filters the network can use. This, in turn, restricts its effectiveness in more complex scenarios. As a result, most synthesis methods are based on 2D slices to overcome these limitations.

In this work, we propose to use an alternative architecture, the Deep Pyramidal Network (DPN) network (Morell-Ortega et al., 2025). Due to its reduced complexity and fewer parameters, the DPN requires less computational power and shorter training times while potentially maintaining comparable or superior performance in image synthesis. The DPN was first proposed for segmentation purposes (Morell-Ortega et al., 2025). This architecture is based on an incremental coarse-to-fine approach which enables a lighter and less memory-demanding model. Compared with a U-Net where an encoder-decoder strategy is used, the DPN consists only of the decoder part. The network starts with an input tensor, the original high-resolution volume, and progressively generates lower-resolution inputs at 1/2, 1/4, and 1/8 scales using a 3D Average Pooling layer. At the deepest level, features are extracted through three processing blocks (Convolution + ReLU + Batch Normalization). These features are then up-sampled using trilinear interpolation resizing and merged with higher-resolution features from the previous level to refine details. This process continues until the original resolution is restored, where a final Convolution + ReLU layer generates the output intensities. Each convolution is performed with a kernel of size 3 x 3 x 3, using the same number of filters in all blocks. In this project, we used 32 filters at each convolution, which results in a DPN network with a total of learnable 395.169 parameters. The general idea of this architecture is to use an incremental approach where the first layers generate the most general patterns that
are progressively refined as the network increases its resolution (see Fig. 1 for details).

## 2.4. Atlas generation

One of the primary challenges faced by deep learning-based methods is their sensitivity to images that differ from those used during training (i.e., out-of-domain generalization). Consequently, performance can decline significantly when these models are applied to different datasets. The goal of our proposed method is to ensure robustness across various image patterns and anatomical variations, thereby minimizing the risk of catastrophic errors. To achieve this, we propose incorporating prior information in the form of a T2 atlas to support the synthesis task. Although atlas-based methods may not achieve the peak accuracy of deep learning approaches, they are considerably more robust when handling diverse image conditions. Leveraging this a priori information can reduce model sensitivity to noise, artifacts, or variability in the input data, thus preventing catastrophic errors and implausible topologies.

We employ a multi-atlas strategy to create a subject-specific atlas, which has proven more robust than a single-atlas approach (Coupé et al., 2011). Specifically, the method estimates a nonlinear registration transform from a subset of anatomical T1 volumes in the training library, aligning each to the input T1 volume. These nonlinear transforms are then applied to the corresponding T2 volumes in the library, resulting in a subject-specific set of T2 volumes. Finally, these warped T2 volumes are merged voxel-wise according to a rule that assigns an intensity value to each voxel.

To perform the intensity fusion, an intensity-based weighted majority voting is used where voxels of the library with intensities similar to the voxel to be processed

have a higher weight than those more dissimilar. Each weight is calculated as follows:

$$w_{\rho} = \frac{1}{1 + d \cdot \|I_{\rho} - L_{p,j}\|}, \quad (1)$$

where d is a scaling coefficient, $I_{\rho}$ is the intensity of a voxel at a position p, and $L_{p,j}$ is the intensity of a voxel at a position p of an image j from library. We set d as 0.5 experimentally. To deal with the temporal limitations of multi-atlas approaches, the fusion algorithm was implemented in C language and parallelized using the OpenMP multiprocessing library.

Nonetheless, the primary limitation of this methodology, as previously identified in the article by Park et al. (2014), is the time cost associated with non-linear registrations and intensity fusion. Fortunately, deep learning-based non-linear registration methods have dramatically reduced the time cost of non-linear registration from minutes to less than 1 second (Balakrishnan et al., 2019). In the proposed method, we used a deep non-linear registration network developed in Coupé et al. (2020).

Another aspect that impacts the temporal cost of the atlas generation process is the number of templates used to generate it (the bigger the library, the more time will be spent on the atlas creation). We used a subset of 70 HCP T1/T2 volumes but selected only the N cases that are more similar to the sample under analysis (based on the L1 norm of image intensities) to perform the non-linear registration and the later fusion (T1 and T2 images were already intensity normalized). We empirically found that a plateau in terms of image quality is reached when the size of the library is N = 20 samples (see Fig. 2 for an example of a generated atlas). Consequently, the atlas creation process takes around 20 seconds on our hardware configuration. This atlas is normalized by its mean

<figure><img src="image_4.png" /><figcaption>Fig. 1. DPN synthesis network architecture. First the 3D average pooling layers are used to lower the input resolution at different scales and then the convolution layers at each scale are applied.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Example of an atlas obtained for an input T1 case. From left to right: original T1 image, its corresponding T2 image, and atlas obtained for the input T1 image. Note that although the fine details are not visible in the atlas, the overall appearance is well captured by the atlas.</figcaption></figure>

and integrated into the architecture alongside the input T1 image as an additional input channel.

## 2.5. Training

Selecting an appropriate loss function is crucial when training a neural network. To determine the most effective option for T2 synthesis, we evaluated several loss functions. Among the options tested, the mean absolute error (MAE) is commonly used in most synthesis studies (Chartsias et al., 2018; S. Pan et al., 2023; Zhu et al., 2023) because it directly measures the average difference in pixel intensity between the generated and real images, which is particularly useful in tasks where maintaining accuracy in pixel intensity is crucial. However, this loss tends to produce blurred images, failing to recover fine details (Chen et al., 2023).

Another widely used loss function is the adversarial loss, typically employed in GAN-based methods. In these approaches, a discriminator assesses the plausibility of generated samples by distinguishing them from real images, which in turn guides the generator to produce more realistic outputs. However, training GANs can be unstable due to challenges in balancing the generator and discriminator (Upadhyay & Awate, 2021). This instability often results from the delicate equilibrium required between the generator and discriminator. Consequently, the quality of generated images can vary significantly, with some outputs appearing highly realistic while others exhibit artifacts or hallucinated anatomical structures.

In contrast, perceptual loss takes a different approach to evaluate the quality of the generated sample by focusing on higher-level image features rather than focusing solely on pixel-level differences or image plausibility. It operates by comparing the feature activations of a pretrained deep network between the generated image and the real image (a common network used for this task is the VGG (Dar et al., 2019; Johnson et al., 2016) for 2D).

These networks are trained on large datasets to recognize high-level image features critical for perceptual quality. By aligning the features in the latent space, perceptual loss helps ensure that the generated images resemble the real ones visually and share their underlying structural and semantic information.

While perceptual loss functions, such as those utilizing the VGG network, have effectively enhanced image quality by focusing on high-level features, their applicability to medical imaging, particularly MRI neuroimaging, is limited. On the one hand, VGG typically operates in 2D, while our data are in 3D. On the other hand, models such as VGG are trained on natural images and may not capture the unique characteristics of medical images. For instance, M. Han et al. (2022) and Mei et al. (2022) highlight that using perceptual loss with networks trained on natural images may not be suitable for medical imaging tasks due to the distinct nature of medical images. To address this, employing the encoder of a self-trained autoencoder using MR data allows for extracting features more pertinent to the specific problem, thereby improving the relevance and effectiveness of the perceptual loss in the synthesis task.

However, despite its advantages, perceptual loss has some practical challenges. Training a synthesis deep network with perceptual loss, especially in 3D, is computationally and memory intensive, as it requires processing high-dimensional data at both networks (generator and latent space of the autoencoder). This limitation motivates the exploration of alternative loss functions, such as those based on the frequency domain.

Analyzing signals in the frequency spectrum is particularly relevant for MRI images, as these images were originally acquired in this domain. In theory, frequency components provide truly global indirect anatomical information compared with perceptual loss where the visual field depends on the depth of the autoencoder. The Fourier Transform enables analyzing and manipulating an
image's frequency content. In this domain, each point represents information from the entire spatial domain, capturing global image characteristics. This duality between spatial and frequency domains allows for the development of loss functions that emphasize global features, potentially improving the quality of synthesized images while serving as an efficient alternative to perceptual loss. Additionally, operations in the frequency domain are computationally efficient in most programming frameworks.

This type of loss function is not new, and it has been previously used in super-resolution problems where the objective is to recover the high-frequency content of the image (Fuoli et al., 2021). In image synthesis, it has also been recently used. Jiang et al. (2021) introduced Focal Frequency Loss (FFL), a novel loss function designed to enhance image reconstruction and synthesis by focusing on hard-to-learn frequency components. In this work, we propose an FFT-based loss computed as the mean squared error between the complex spectra of the predicted and ground truth samples.

Finally, we would like to prioritize also the practical applications of the network under design, alongside its reconstruction quality. We believe this balanced approach will enhance its overall effectiveness. For example, GANs or diffusion models can create images with impressive details. However, this realism is not always linked to probed clinical utility. Dohmen et al. (2024) examine how visual improvement in medical image synthesis is a subjective aspect that is challenging to quantify accurately. Therefore, considering that one of the goals of synthesizing T2 images from T1 images is to provide more information to improve segmentation, we propose to include a segmentation-based loss in the learning process. This

approach aims to synthesize a T2 image that is good in terms of image metrics but also generates a good segmentation. Measuring the Dice coefficient as an auxiliary evaluation metric informs us not only about the image's visual quality but also about its ability to produce an improved segmentation.

To incorporate this approach, we first trained a multimodal (T1 + T2) brain tissue segmentation network capable of segmenting seven brain regions (cerebral-spinal fluid, white matter, cortical gray matter, subcortical gray matter, cerebellar gray matter, cerebellar white matter, and brainstem). We decided to train a multimodal network (T1 + T2) instead of a monomodal (T2) to promote the consistency between the synthesized T2 and the real T1. The network was trained using 70 paired T1 and T2 images from the holiAtlas dataset (Manjón et al., 2025). Due to memory constraints, we utilized a compact DPN architecture with only 8 filters per level, resulting in a total of 87.674 parameters. Once trained, the segmentation network was frozen and integrated into the synthesis pipeline to compute the loss function (see Fig. 3 for details).

## 2.6. Semi-supervised learning

One of the primary objectives of this work is to develop a robust T2 synthesis model from T1 images, ensuring generalizability across diverse datasets and clinical scenarios (for instance, highly atrophic brain images or containing lesions). A significant challenge in achieving this goal is the limited availability of large enough datasets containing paired T1 and T2 images, which is essential for traditional supervised learning approaches. To address this limitation, we employed a semi-supervised

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** DPN synthesis network with Dice loss included in the learning process by adding the frozen segmentation network. The synthesis network receives T1 and T2 atlas as input. The segmentation network receives the same T1 and the synthetic T2 (T2\*) generated by the synthesis network. The reconstruction loss consists of an image fidelity term.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Schematic of the semi-supervised approach: 50% of the time, the model reads from the train dataset with real T2 as ground truth, and 50% of the time it reads from the Lifespan dataset with T2 synthesized with a previously trained model used as ground truth.</figcaption></figure>

strategy that enables the integration of unpaired data, significantly increasing both the diversity and volume of training samples.

To this end, we make use of the Lifespan dataset, a large collection of 3,067 T1 images from male and female participants aged 3 to 90 years. This dataset provides a rich anatomical and demographic diversity (including pathological images) but lacks paired T2 images. To incorporate this dataset into training, we generated synthetic T2 images using the best previously trained supervised model. These synthetic T2 images act as pseudo-images, allowing us to fine-tune the synthesis network with a broader set of anatomical variations. This approach is analogous to pseudo-labeling techniques commonly used in semi-supervised segmentation (Han et al., 2024; Kamraoui et al., 2022).

By integrating real paired and pseudo-synthetized paired data, our method benefits from an expanded training distribution, enabling better generalization to unseen cases. In particular, this strategy ensures that the model is exposed to a wide range of brain ages (or brain shapes), varying acquisition conditions, and potential pathologies such as atrophy and microvascular lesions, which are often underrepresented in limited paired datasets. The segmentation labels required for the loss function on the Lifespan dataset were obtained using a monomodal segmentation network (using 32 filters) trained only with T1 images from the holiAtlas dataset (the improved quality of the T1 segmentations forced the network to learn better mappings despite the limitation of using pseudo-images). To maintain a balance between real and semi-supervised data, training samples were selected with a 50% probability, preventing overfitting to synthetic cases while still leveraging the increased data diversity. The process is schematically illustrated in Figure 4.

All the experiments were performed with Tensorflow 1.15, using a V100-SXM2-32GB GPU with a system with 64GB RAM memory and an Intel Xeon processor Gold 5220R CPU @ 2.20GHz with 32 cores, all running under Linux Ubuntu 18.04.5. All networks were trained with 20 steps per epoch and a batch size of 1 for 1,500 epochs (1 day of processing), ensuring that their convergence was reached. Adam optimizer (Kingma & Ba, 2014) was used with its default parameters.

## 3. RESULTS

In this section, we summarize the results of the experiments performed to evaluate the proposed approach. First, we present the results of the supervised model with the optimal combination of loss functions and the results of the semi-supervised learning. Finally, a comparison with some methods of the state of the art is included.

Statistical differences were tested using the two-sided Wilcoxon signed-rank test (p < 0.05). Because multiple comparisons were performed, a correction was applied to control the false discovery rate (FDR) using the Benjamini-Hochberg method (Benjamini & Hochberg, 1995). The original p-values for each comparison were calculated and adjusted using the FDR correction.

### 3.1. Supervised learning results

The proposed synthesis network was first trained with the supervised dataset composed of paired T1 and T2 images from the HCP, OpenNeuro, and IXI datasets. The metrics used to evaluate the results were the Peak Signal to Noise Ratio (PSNR), the Pearson's Correlation Coefficient (CC), the Structural Similarity Index Measure (SSIM) (Wang et al., 2004), and the Dice coefficient. All metrics
were estimated inside the intracranial cavity, extracted from the T1 images using Deep ICE method (Manjón et al., 2020). The results of different analysis following this supervised paradigm are shown in Tables 2, 3, and 4.

As discussed in Section 2.5, we have experimented with diverse loss functions. In Table 2 we studied the impact of each of them carefully through an ablation study. The results are analyzed over the in-domain dataset, letting apart the out-of-domain one for an independent analysis. In the image space, we used as baseline loss the Mean Absolute Error (MAE). As W. Li et al. (2020) demonstrated, using L1 loss encourages less blurring compared with L2 loss. However, in the frequency domain, different studies use different L-norms, the L1 (MAE) in Fuoli et al. (2021) and L2 (MSE) in Jiang et al. (2021). Therefore, we experimented with both of them to

assess their suitability. The final loss used for training is a mix of several balanced losses.

$$L_{\text{Total}} = \text{MAE}(\text{space domain}) + \lambda_1 \cdot \text{MSE}(\text{FFT}) + \lambda_2 \cdot \text{DiceLoss}. \qquad (2)$$

The values of the different λ were set so that all loss terms have the same magnitude, in order to help maintain a balanced optimization process. The values are λ₁ = 0.01 and λ₂ = 150. The experiments in Table 2 demonstrate that using the MAE of the FFT instead of the MSE of the FFT results in lower performance metrics, indicating that MSE is more effective for this task. Furthermore, when the FFT computation is removed entirely, leaving only the MAE between the real and predicted images, the metrics

<table><caption>**Table 2.** Results for in-domain test data (N = 20).</caption><thead><tr><th></th><th>PSNR</th><th>CC</th><th>SSIM</th><th>Dice</th></tr></thead><tbody><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss + atlas</td><td>32.1432 ± 1.4662*</td><td>0.9865 ± 0.0052*</td><td>0.9603 ± 0.0100*</td><td>0.9082 ± 0.0982*</td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss</td><td>31.8454 ± 1.3386</td><td>0.9855 ± 0.0050</td><td>0.9582 ± 0.0089</td><td>0.9012 ± 0.1023</td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT)</td><td>31.8702 ± 1.2455</td><td>0.9856 ± 0.0050</td><td>0.9570 ± 0.0089</td><td>0.8813 ± 0.1299</td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MAE(FFT) +<br>λ<sub>3</sub> · Perceptual</td><td>27.6085 ± 0.9549</td><td>0.9649 ± 0.0060</td><td>0.9098 ± 0.0125</td><td>0.7879 ± 0.2406</td></tr><tr><td>MAE(space domain) +<br>λ<sub>3</sub> · Perceptual</td><td>31.4378 ± 1.6187</td><td>0.9849 ± 0.0060</td><td>0.9577 ± 0.0106</td><td>0.8861 ± 0.1254</td></tr><tr><td>MAE(space domain)</td><td>24.9342 ± 0.7783</td><td>0.9306 ± 0.0164</td><td>0.8518 ± 0.0135</td><td>0.7310 ± 0.2857</td></tr></tbody></table>

Best results in bold.

*Represents statistically significant differences between the first and second row (the best model and the second best).

<table><caption>**Table 3.** Results for out-of-domain Bordeaux dataset test data (N = 44).</caption><thead><tr><th>CONFIGURATION</th><th>PSNR</th><th>CC</th><th>SSIM</th><th>Dice</th></tr></thead><tbody><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss + atlas</td><td>28.4298 ± 0.7679*</td><td>0.9712 ± 0.0043*</td><td>0.9143 ± 0.0118*</td><td>0.8543 ± 0.0306</td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss</td><td>27.4602 ± 0.9670</td><td>0.9627 ± 0.0077</td><td>0.9091 ± 0.0133</td><td><b>0.8599 ± 0.1494</b></td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MSE(FFT)</td><td>27.4417 ± 0.9625</td><td>0.9625 ± 0.0067</td><td>0.9106 ± 0.0123</td><td>0.8314 ± 0.1954</td></tr><tr><td>MAE(space domain) +<br>λ<sub>1</sub> · MAE(FFT) +<br>λ<sub>3</sub> · Perceptual</td><td>27.2069 ± 1.0471</td><td>0.9603 ± 0.0089</td><td>0.8864 ± 0.0115</td><td>0.7865 ± 0.2509</td></tr><tr><td>MAE(space domain) +<br>λ<sub>3</sub> · Perceptual</td><td>26.7039 ± 0.8444</td><td>0.9580 ± 0.0074</td><td>0.9043 ± 0.0130</td><td>0.8307 ± 0.1935</td></tr><tr><td>MAE(space domain)</td><td>25.2364 ± 1.0470</td><td>0.9390 ± 0.0160</td><td>0.8343 ± 0.0159</td><td>0.7408 ± 0.2890</td></tr></tbody></table>

Best results in bold.

*Represents statistically significant differences between the best and second best (the best model and the second best).
drop significantly, suggesting that MAE alone is insufficient for the synthesis task. Additionally, analyzing the impact of perceptual loss, we observed that it achieves comparable results to the MSE of the FFT for in-domain testing in terms of reconstruction metrics, corroborating our assumption that FFT can work as an efficient alternative to perceptual loss.

The impact of segmentation loss was also analyzed in Table 2. Incorporating it into the generator's loss function quantitatively improves the Dice metric, as it encourages the preservation of segmentation-relevant structures in the synthesized images. Qualitatively, as shown in Figure 5,

<table><caption>**Table 4.** Dice results in OASIS dataset (N = 100).</caption><thead><tr><th>CONFIGURATION</th><th>Dice</th></tr></thead><tbody><tr><td>MAE(space domain)+</td><td>0.8158* ± 0.0776*</td></tr><tr><td>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss + atlas</td><td></td></tr><tr><td>MAE(space domain)+</td><td>0.8017 ± 0.0936</td></tr><tr><td>λ<sub>1</sub> · MSE(FFT) + λ<sub>2</sub>DiceLoss</td><td></td></tr><tr><td>MAE(space domain)+</td><td>0.7942 ± 0.0884</td></tr><tr><td>λ<sub>1</sub> · MSE(FFT)</td><td></td></tr><tr><td>MAE(space domain)+</td><td>0.7106 ± 0.1114</td></tr><tr><td>λ<sub>1</sub> · MAE(FFT)+</td><td></td></tr><tr><td>MAE(space domain)+</td><td>0.7859 ± 0.0950</td></tr><tr><td>λ<sub>3</sub> · Perceptual</td><td></td></tr><tr><td>MAE(space domain)</td><td>0.6346 ± 0.1444</td></tr></tbody></table>

Measuring the difference between the segmentation obtained with the unimodal segmentation network with a T1, and the segmentation obtained with the multimodal network, with the T1 and the T2 synthesized for each method. Indirect measure of the quality of the synthesized T2. Best results in bold.

*Represents statistically significant differences between the first and second row (the best model and the second best).

the absence of Dice loss leads to blurrier edges, particularly in small structures such as arteries and cerebrospinal fluid boundaries. Though small, these structures enhance both the image's perceived quality and its usefulness for downstream tasks. Segmentation loss acts as a focal mechanism, directing attention toward regions with lower Dice scores and improving their synthesis. Without it, the model struggles to generate these fine details accurately.

Furthermore, we observed that incorporating the atlas significantly improves both reconstruction and segmentation metrics statistically. As we will analyze later, the atlas provides the network with a *prior* that helps it learn the synthesis task more effectively—rather than starting from scratch—and enhances its robustness when handling input data that differ significantly from the training distribution.

In Table 3, we consider the results on the Bordeaux dataset, which serves as an out-of-domain evaluation. We aim to assess the model's ability to generalize to data from different sources, characterized by greater heterogeneity and, in some cases, the presence of lesions. On this dataset, we again observed the benefits of complementing MAE in image space with additional loss functions. Notably, the trend between MAE and MSE in the frequency domain remains consistent, confirming that MSE is the superior choice, and its performance is comparable with perceptual loss. Additionally, segmentation loss improves the Dice score and contributes to better overall reconstruction quality, as reflected in the reconstruction metrics.

Finally, we analyze the results from the second out-of-domain dataset from OASIS. In this case, we do not have a T2 image to compare the reconstruction quality. However, we can indirectly assess the quality of the generated

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** On the left is real T2, in the middle is synthetic T2 with the method without segmentation loss, on the right is synthetic T2 with the method with segmentation loss. The model without segmentation loss does not clearly delimit the area between the cerebrospinal fluid and the cerebellum or the brainstem.</figcaption></figure>
volume using the segmentation metrics as a proxy. As shown in Table 4, the same trend observed in the in-domain test dataset evaluation is repeated here. On one hand, the equivalence between perceptual loss and MSE in the spectrum is confirmed. On the other hand, the introduction of segmentation loss makes the synthesized images easier to segment, impacting in the measured DICE. Finally, it is revalidated that incorporating prior information in the form of an atlas is beneficial. Figure 6 shows data distribution to better understand the results.

## 3.2. Atlas impact

Previous studies have employed atlases as prior information to enhance segmentation quality by providing contextual localization data (Morell-Ortega et al., 2025). It is well established that atlas-based approaches leverage spatial information through registration, resulting in greater consistency and interpretability. In our approach, we integrate the atlas as an input channel in a 3D CNN deep learning network to help ensure that the network's predictions remain spatially consistent. The use of the multi-atlas-based synthesized image serves as a starting point for the network.

In order to further measure whether substantial improvements in robustness and consistency justify the

extra cost of using atlases, we performed an additional quantitative and qualitative analysis. Our quantitative evaluation used the out-of-domain test dataset from Bordeaux with data transformations applied via the TorchIO library (Pérez-García et al., 2021), which serves us to measure the model's ability to generalize under varying input conditions.

We applied realistic transformations that simulate clinical acquisition conditions, where image quality may vary compared with research datasets. These transformations, applied with random degrees, include anisotropy, blurring, gamma correction, bias field, motion artifacts, noise, and ghosting. Given that the network's primary objective is contrast synthesis rather than correction of spatial acquisition parameters, T2 images are also subjected to anisotropy and blurring to reflect their naturally lower resolution under such conditions. For the rest of the transformations, modifications are applied solely to the T1 input volume (from which the atlas is generated again), while the T2 original target image remains unchanged. The atlas is recalculated for these transformations based on the augmented T1 input, thereby simulating the network's response under real-world conditions. This approach allows us to categorize the transformations into geometric and intensity-based groups, facilitating a more rigorous analysis of the

<figure><img src="image_9.png" /><figcaption>**Fig. 6.** Dice distribution by model in OASIS dataset. Measuring the difference between the segmentation obtained with the unimodal segmentation network with a T1, and the segmentation obtained with the multimodal network, with the T1 and the T2 synthesized for each method. Indirect measure of the quality of the synthesized T2.</figcaption></figure>
model's performance. TorchIO augmentations were pro-
duced using the default parameters with the exception
of flip (done in the three axes) and loggamma
(range = [-0.1, 0.1]).

**Figure 7** presents a comparative analysis of different image quality metrics (DICE, PSNR, CC, and SSIM) across multiple transformations, evaluating the effect of incorporating an atlas. The results indicate that models using an atlas achieve higher mean values across all metrics, except DICE, while reducing dispersion in some transformations, suggesting improved robustness to perturbations. Statistical significance is assessed through central tendency and dispersion tests. For dispersion analysis, if both distributions are normal (Shapiro-Wilk test), Bartlett's test is applied; otherwise, Levene's test accounts for non-normal distributions. To mitigate the risk of false positives due to multiple comparisons, a Bonferroni correction is applied, adjusting the significance threshold

accordingly. Cases where significance is found in central
tendency, dispersion, or both tests are highlighted in red,
blue, and purple, respectively. Notably, bias field, motion,
and gamma transformations exhibit significant differ-
ences, with models without an atlas experiencing greater
performance degradation. While some transformations
show marginal differences, the overall trend highlights the
benefit of integrating an atlas, particularly in maintaining
structural consistency (DICE) and correlation (CC).

Visually, the effects of the atlas can also be compared
in Figure 8, where an example of the synthetic T2 obtained
with the atlas method and without the atlas is shown. The
model without atlas makes serious mistakes in the cere-
brospinal fluid of the ventricles.

To the best of our knowledge, this is the first work that
directly integrates atlas priors as an input channel into a
3D CNN for the specific task of T2 synthesis from T1
images. While previous approaches have leveraged

<figure><img src="image_10.png" /><figcaption>**Fig. 7.** Impact of Atlas Incorporation on image synthesis quality across different transformations over Bordeaux dataset. Dotted lines are the mean for all the transformations.</figcaption></figure>
atlas-based registration or deep learning techniques separately, our unified framework combines these strategies providing both enhanced contextual awareness and improved robustness.

### 3.3. Semi-supervised learning results

The semi-supervised approach was designed to enhance the robustness and generalization capabilities of the proposed method, enabling it to perform effectively on a large variety of unseen data. Incorporating unpaired images alongside paired data into the training process, the model leverages a more extensive and diverse dataset, capturing broader variations in anatomical structures and imaging protocols. The results of the semi-supervised training are shown in Table 5.

On the in-domain dataset, both methods perform nearly identically across all metrics, suggesting that when test images closely match the training distribution, each approach effectively captures the underlying image characteristics. For the out-of-domain (Bordeaux) data, the semi-supervised method achieved statistically significant improvements in SSIM and Dice score. Although these differences are modest in magnitude, the statistical significance suggests that the semi-supervised approach might better preserve structural details and segmentation quality when encountering data that differ from the training set. Nonetheless, the practical implications of these improvements should be interpreted with caution.

However, when we repeated the analysis by applying data augmentation to the Bordeaux test dataset (see Fig. 9), we observed that this approach once again enhances the model's robustness, achieving higher average values for Dice, Correlation Coefficient, and SSIM. In summary, atlas provides prior anatomical information, while semi-supervised training leverages unlabeled or partially labeled data to learn robust features. Combined, they enhance both image quality (PSNR, SSIM, CC) and segmentation accuracy (Dice), especially under transformations meant to simulate out-of-domain shifts.

Finally, results on OASIS dataset show a clear improvement on Dice index for the semi-supervised trained network (0.833 vs. 0.816). In Figure 10, the improvement for all 100 cases is shown visually, where we would like to highlight the enhancement in worst cases.

Finally, to visually provide some evidence of the robustness of the proposed method, it was applied to OASIS images containing lesions. A visual example of the results of the proposed method is shown in Figure 11. As can be noticed, the lesions were correctly synthetized even though the supervised training data had no lesioned images.

### 3.4. State-of-the-art comparison

The proposed method was compared with related state-of-the-art related synthesis methods. To make a fair comparison, the code of the selected methods was

<figure><img src="image_11.png" /><figcaption>**Fig. 8.** From left to right: T1 input, T2 image synthesized with the synthesis model without the atlas as a channel and image obtained with the synthesis model with atlas. Note that lateral ventricles are correctly synthetized with the atlas option but not without it.</figcaption></figure>

<table><caption>**Table 5.** Semi-supervised and supervised training comparison (for in-domain (N = 20) and out-of-domain (N = 44)).</caption><thead><tr><th></th><th></th><th>PSNR</th><th>CC</th><th>SSIM</th><th>Dice</th></tr></thead><tbody><tr><td rowspan="2">In-domain</td><td>Semi-supervised</td><td><strong>32.2718 ± 1.4842*</strong></td><td><strong>0.9869 ± 0.0050*</strong></td><td><strong>0.9612 ± 0.0097*</strong></td><td><strong>0.9095 ± 0.0975</strong></td></tr><tr><td>Supervised</td><td>32.1432 ± 1.4662</td><td>0.9865 ± 0.0052</td><td>0.9603 ± 0.0100</td><td>0.9082 ± 0.0982</td></tr><tr><td rowspan="2">Out-of-domain<br>(Bordeaux)</td><td>Semi-supervised</td><td>28.3450 ± 0.7679</td><td>0.9712 ± 0.0043</td><td><strong>0.9152 ± 0.0122*</strong></td><td><strong>0.8583 ± 0.0275*</strong></td></tr><tr><td>Supervised</td><td><strong>28.4298 ± 0.7679*</strong></td><td><strong>0.9717 ± 0.0043*</strong></td><td>0.9143 ± 0.0118</td><td>0.8543 ± 0.0306</td></tr></tbody></table>

Best results in bold.

*Represents statistically significant differences between semi-supervised and supervised training.
<figure><img src="image_12.png" /><figcaption>**Fig. 9.** Impact of semi-supervised training on image synthesis quality across different transformations. Dotted lines are the mean for all the transformations.</figcaption></figure>

downloaded and trained with our training library and tested with our test dataset. These include a 3D volume-based deep learning approach, a 2D diffusion-based generative model, and a slice-based generative adversarial network, having a representation of each technology. Below, we outline the details of each method:

* UNET3D (Manjón et al., 2022), this method employs a 3D volume-based deep learning approach using a stride-based variant of the U-Net architecture. The network processes entire 3D volumes, enabling it to capture spatial dependencies across slices effectively.

* SelfRDB (Arslan et al., 2024) is a diffusion-based generative model that leverages residual dense blocks (RDBs) for capturing hierarchical features during the synthesis process. Diffusion models excel at producing high-quality and realistic out-

puts by iteratively refining noisy initializations through a learned denoising process.

* The pGAN method (Dar et al., 2019) adopts a slice-based generative adversarial network architecture. Unlike 3D volume-based approaches, pGAN synthesizes slices independently, reducing the computational burden while maintaining high-quality outputs. This approach involves training a generator-discriminator pair, where the generator learns to produce realistic slices of data, and the discriminator refines the output by distinguishing real slices from generated ones. However, the slice-based nature of the approach can sometimes lead to discontinuities between slices in the final reconstructed volume.

When we compare the methods in our in-domain test data (n = 20 images), we can observe that for all metrics
<figure><img src="image_13.png" /><figcaption>**Fig. 10.** Dice value for the 100 images of the OASIS dataset. In orange the values for the semi-supervised approach showing higher mean and higher minimum values. Note that extreme cases show clear higher dice values.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**Fig. 11.** Qualitative results on the OASIS dataset. Top row: T1-weighted input images with visible white matter confluent lesions. Bottom row: synthesized T2-weighted images produced by the proposed method, showing consistent lesion depiction (red arrows).</figcaption></figure>
our method produces better results, followed by SelfRDB having the second-best Dice index (see Table 6).

Table 7 presents the results for the Bordeaux dataset, which was excluded from training for all methods, ensuring a more realistic assessment of performance. Notably, our model surpasses all competitors in every metric except SSIM, highlighting its superior image reconstruction capabilities and significantly improving its utility for downstream segmentation tasks.

Finally, we also compared the different methods with the OASIS dataset (N = 100), which has the presence of lesions. Again, we indirectly assess the quality of the generated volume using the segmentation metrics as a proxy (see Table 8). We can also see the results in Figure 12, where we can see that the proposed method is more robust than the other methods (our method dispersion is lower than the other methods and the dice value is much higher).

To visually inspect the differences between the compared methods, in Figures 13 and 14 an example of the synthesized T2 volume for each method is shown. In the example of Bordeaux dataset (Fig. 13), the UNET3D method produces consistent but blurred images (probably due to the loss function used during its training). In addition, pGAN shows very realistic axial slices, but in the sagittal

slice, pGAN has many artifacts, due to its 2D nature. Finally, SelfRDB produces convincing axial slices (with some vessel-like hallucinations) and (as pGAN) interslice artifacts due to its 2D nature. The proposed method shows a 3D coherent reconstruction and well-defined edges.

In the example case of OASIS dataset (Fig. 14), our proposed method and SelfRDB are the ones that generate better results, but again in the sagittal slice, SelfRDB has more artifacts and a worse image quality since it works in 2D and not in 3D as in our work.

A numerical analysis of these data indicates that the proposed method, along with pGAN and SelfRDB, performs competitively regarding classical metrics such as PSNR, SSIM, and CC. The key difference emerges when evaluating the Dice index, as discussed in Section 2.5. Here, the Dice index aims to assess visual quality in a more complex manner, aligning with the observations made from the reconstructed images. For instance, as shown in Table 7 (Bordeaux dataset) and Figure 13, pGAN achieves the highest SSIM performance. However, artifacts observed in the sagittal slice, which degrade visual quality of the reconstructed 3D volume, could explain the lower Dice index measured.

We finally compared the time required to train each model and their inference times. Regarding training time,

<table><caption>**Table 6.** Results for in-domain test data (N = 20).</caption><thead><tr><th></th><th>PSNR</th><th>CC</th><th>SSIM</th><th>Dice</th></tr></thead><tbody><tr><td>Our method</td><td><strong>32.2718 ± 1.4842</strong> *</td><td><strong>0.9869 ± 0.0050</strong> *</td><td><strong>0.9612 ± 0.0097</strong> *</td><td><strong>0.9095 ± 0.0975</strong> *</td></tr><tr><td>UNET3D</td><td>24.9035 ± 0.6391</td><td>0.9297 ± 0.0092</td><td>0.8251 ± 0.0159</td><td>0.7378 ± 0.2744</td></tr><tr><td>SelfRDB</td><td>28.4151 ± 4.6835</td><td>0.9458 ± 0.0479</td><td>0.8962 ± 0.0838</td><td>0.8166 ± 0.2435</td></tr><tr><td>pGAN</td><td>28.3846 ± 4.5307</td><td>0.9482 ± 0.0438</td><td>0.8810 ± 0.0884</td><td>0.6868 ± 0.2885</td></tr></tbody></table>

Best results in bold.

<table><caption>**Table 7.** Results for out-domain Bordeaux test data (N = 44).</caption><thead><tr><th></th><th>PSNR</th><th>CC</th><th>SSIM</th><th>Dice</th></tr></thead><tbody><tr><td>Our method</td><td><strong>28.3450 ± 0.7365</strong> * β</td><td><strong>0.9715 ± 0.0043</strong> * α β</td><td>0.9152 ± 0.0116 *</td><td><strong>0.8583 ± 0.0027</strong> * α β</td></tr><tr><td>UNET3D</td><td>25.5561 ± 0.6018</td><td>0.9426 ± 0.0049</td><td>0.8344 ± 0.0133</td><td>0.7481 ± 0.2794</td></tr><tr><td>SelfRDB</td><td>27.7835 ± 0.4756</td><td>0.9688 ± 0.0052</td><td>0.9160 ± 0.0101</td><td>0.8352 ± 0.1868</td></tr><tr><td>pGAN</td><td>27.7483 ± 0.9454</td><td>0.9688 ± 0.0056</td><td><strong>0.9165 ± 0.0117</strong> β</td><td>0.6700 ± 0.3091</td></tr></tbody></table>

Best results in bold.

<table><caption>**Table 8.** Dice results in the OASIS test set (N = 100).</caption><thead><tr><th></th><th>Our method</th><th>UNET3D</th><th>SelfRDB</th><th>pGAN</th></tr></thead><tbody><tr><td>Dice</td><td><strong>0.8350 ± 0.0623</strong>*</td><td>0.6610 ± 0.0771</td><td>0.6641 ± 0.0929</td><td>0.6441 ± 0.1064</td></tr></tbody></table>

Best results in bold.

<footnote>*Represents statistically significant differences between our method and the state-of-the-art methods.</footnote>

<footnote>*Represents statistically significant differences between our method and UNET3D</footnote>

<footnote>αRepresents statistically significant differences between our method and SelfRDB.</footnote>

<footnote>βRepresents statistically significant differences between our method and pGAN.</footnote>

<footnote>*Represents statistically significant differences between our method and the state-of-the-art methods.</footnote>
<figure><img src="image_15.png" /><figcaption>**Fig. 12.** Dice distribution by model in OASIS dataset. Note that our proposed method has a lower dispersion than the other methods and the Dice index value is higher than the other state-of-the-art methods.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Fig. 13.** Visual comparison of methods using an example from the Bordeaux dataset. Red arrows in the top row indicate for UNET3D the inability to reconstruct the fine details from the choroid plexus within the lateral ventricles, in the bottom row, they indicate the artifacts in sagittal slice for pGAN and SelfRDB.</figcaption></figure>

in the case of the UNET3D (Manjón et al., 2022), the network was trained for 1,800 epochs, 50 steps per epoch and lasted 1 day. The training used the same dataset, preprocessing pipeline, and optimization settings as our method to ensure a fair comparison. SelfRDB (Arslan et al., 2024) was trained for 50 epochs (as suggested by the authors), requiring approximately 16 days to complete due to the computationally demanding nature of the

2D diffusion models. We trained pGAN for 100 epochs, each taking approximately 6,100 seconds, resulting in a total training time of around 7 days. The training time of our method lasts around 1 day, with 1,500 epochs. Regarding inference time, UNET3D time is around 1.5 seconds, and the Self RDB model takes around 3 minutes per case, making it significantly slower than most deep learning-based models. The inference time for
<figure><img src="image_17.png" /><figcaption>**Fig. 14.** Comparison of the methods on an example of the OASIS dataset image. The real T1 is shown as a reference, as we do not have the T2 image for this dataset. Red arrows in the top row indicate fine details that cannot be well synthesized by the networks UNET3D and pGAN, in the bottom row, they indicate the artifacts in UNET3D and artifacts in pGAN and SelfRDB generated by obtaining the 3D image from the 2D slices.</figcaption></figure>

pGAN was 16.5 seconds per case. Finally, the inference time of our proposed method is around 2 seconds. However, we must add the time to generate the atlas ($T_{atlas}$ = 20 seconds), and the time to preprocess the image in terms of contrast stretching ($T_{tms}$ = 3.8 seconds) resulting in an overall time of 25.8 seconds.

## 4. DISCUSSION

In this paper, we have presented a novel deep learning-based method for synthesizing T2 MR images from T1 images. The proposed method works in 3D which eliminates interslice inconsistencies of previous 2D approaches and is designed with a segmentation-oriented loss function that improves the quality of the synthesized images.

We also explored incorporating prior anatomical information to enhance the model's robustness and improve its performance on more complex images. This prior knowledge helps guide the synthesis process, allowing the model to handle anatomical variability better and ensure the generated images' structural coherence. Our experiments demonstrate that models incorporating prior information outperform those relying solely on image-to-image translation, particularly in challenging cases with large lesions or advanced atrophy, such as cases with unusually large ventricles. Additionally, the proposed method incorporates a semi-supervised learning strategy, which enhances its robustness when applied to datasets with diverse characteristics, extending beyond the scope of the training data.

A critical component of the proposed method is the choice of loss function. To address this, we evaluated several loss functions, including those operating in both the spatial and frequency domains. We tested the mean

squared error (MSE) of the Fast Fourier Transform (FFT) of the images, which captures global structural information that is often overlooked by spatial domain loss functions. Our results show that combining this frequency domain loss with a segmentation-oriented loss function best fits the problem. This hybrid approach allows the model to synthesize anatomical structures while ensuring predicted intensity consistency.

One of the key strengths of our method lies in its semi-supervised approach, which enables greater generalization across diverse datasets. The model demonstrates robustness because it has been trained on images spanning a wide range of age groups, from 3 years old and above, and a wide range of image types from 1.5T to 3T and with different gradient echo approaches such as MPRAGE of FSPGR. This adaptability is particularly relevant in clinical settings where datasets often vary significantly in terms of patient demographics and imaging protocols. One of the primary limitations in T2 synthesis tasks is the scarce availability of paired T1 and T2 images, which our semi-supervised approach is designed to address. While the use of prior information significantly mitigates challenges related to anatomical variability, we are aware that further efforts are needed to refine the method's adaptability to extreme domain shifts or rare anatomical variations.

We compared the performance of our method against several state-of-the-art approaches, including SelfRDB (Arslan et al., 2024), a UNET3D-based method (Manjón et al., 2022) and a GAN-based method (Dar et al., 2019). The comparison highlights that our method is competitive, particularly in segmentation tasks. This improvement is attributed to the combined use of prior information, the segmentation loss function, and the frequency
domain loss, which collectively enhances the synthesized image's quality and utility for downstream tasks.

The proposed method has some practical implications. The improved Dice index and visual quality of the synthesized T2w MRI suggest that the proposed method can provide more accurate and clinically useful information, aiding in better diagnosis and treatment planning. The proposed method's training and inference times are competitive, with training completed in approximately 1 day, inference taking around 2 seconds per case, and additional time for atlas generation and preprocessing.

Finally, we are aware that, as with every method, our method has its limitations. In order to simplify the synthesis process, we standardized the position and resolution of the images by mapping them to MNI space rather than using them at native space which can introduce some blurring due to the interpolation applied during registration. Besides, while the proposed method is efficient, the generation of the atlas and preprocessing steps add to the overall time, which could be optimized further. Although the model showed robustness across diverse datasets, further validation on more varied and larger datasets could strengthen the findings. Future work should include more extensive clinical validation and pathological modifications to assess the practical impact of the synthesized images on diagnostic accuracy and treatment outcomes.

## 5. CONCLUSION

We proposed a new synthesis method based on deep learning 3D CNN, specifically leveraging a combination of segmentation loss and prior information to guide the synthesis process. Unlike other studies that predominantly rely on generative adversarial networks (GANs) or diffusion models, we have developed a framework that avoids the instability typically associated with adversarial training and the large training/test times of diffusion approaches. Instead, our method focuses on integrating anatomical prior information, a powerful loss function and a semi-supervised learning paradigm, resulting in a more robust and generalizable solution.

Through experiments, we have validated the hypothesis that incorporating prior information and using loss functions tailored to the task, such as the Fourier transform loss combined with segmentation loss, significantly improves synthesis quality. The experiments demonstrated that our approach generates visually accurate T2 images and preserves anatomical and semantic consistency, even in datasets with diverse age ranges and anatomical variability.

Furthermore, the semi-supervised strategy proved critical in addressing the challenge of limited paired T1

and T2 datasets, enabling the model to generalize effectively to unseen data. This makes the proposed method especially useful in clinical scenarios where access to paired datasets is limited or infeasible.

Our results show that the proposed method outperforms state-of-the-art techniques, particularly in segmentation tasks, demonstrating its potential for applications that require accurate downstream analysis. In the future, we will explore contrast synthesis of other MR sequences such as FLAIR and Gd-enhanced T1, or modalities and their ability to detect small lesions.

## DATA AND CODE AVAILABILITY

Public data are available online, while private data cannot be shared. Code will be available upon reasonable request.

## AUTHOR CONTRIBUTIONS

Sergio Morell-Ortega: Writing—original draft, Visualization, Validation, Methodology, Investigation, Data curation. Marina Ruiz-Perez: Software, Methodology, Investigation. Marien Gadea: Writing—review & editing, Data curation. Roberto Vivo-Hernando: Writing—review & editing, Formal analysis. Gregorio Rubio: Writing—review & editing, Formal analysis. Fernando Aparici: Writing—review & editing, Data curation. Mariam de la Iglesia-Vaya: Writing—review & editing, Data curation. Thomas Tourdias: Writing—review & editing, Data curation. Boris Mansencal: Software, Methodology. Pierrick Coupé: Writing—review & editing, Supervision, Investigation, Formal analysis, Conceptualization. José V. Manjón: Writing—review & editing, Validation, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization.

## DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## ACKNOWLEDGMENTS

This work has been developed thanks to the project PID2023-152127OB-I00 of the Ministerio de Ciencia, Innovación e Universidades of Spain. This work benefited from the support of the project HoliBrain of the French National Research Agency (ANR-23-CE45-0020-01). Moreover, this project is supported by the Precision and global vascular brain health institute funded by the France
2030 investment plan as part of the IHU3 initiative (ANR-23-IAHU-0001). Finally, this study received financial support from the French government in the framework of the University of Bordeaux's France 2030 program/RRI IMPACT and the PEPR StratifyAging. We thank the support of ITACA (Institute of Information and Communication Technologies) at UPV (Universitat Politècnica de València).

This work is based on multiple samples. We wish to thank all investigators of these projects who collected these datasets and made them freely accessible. The NDAR data used in the preparation of this manuscript were obtained from the NIH-supported National Database for Autism Research (NDAR). NDAR is a collaborative informatics system created by the National Institutes of Health to provide a national resource to support and accelerate research in autism. The NDAR dataset includes data from the NIH Pediatric MRI Data Repository created by the NIH MRI Study of Normal Brain Development. This is a multisite, longitudinal study of typically developing children from ages newborn through young adulthood conducted by the Brain Development Cooperative Group. A listing of the participating sites, and a complete listing of the study investigators can be found at https://nda.nih.gov/edit_collection.html?id=1151. Data collection and sharing for this project were funded by the Alzheimer's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer's Association; Alzheimer's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; Eurolmmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the

University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.

The OASIS data used in the preparation of this manuscript were obtained from the OASIS project. See http://www.oasis-brains.org/ for more details. The AIBL data used in the preparation of this manuscript were obtained from the AIBL study of aging. See www.aibl.csiro.au for further details. The ICBM data were used in the preparation of this manuscript. The IXI data used in the preparation of this manuscript were supported by the—http://www.brain-development.org/. The ABIDE data used in the preparation of this manuscript were supported by ABIDE funding resources listed at http://fcon_1000/projects.nitrc.org/indi/abide/. Primary support for the work by Adriana Di Martino was by ABIDE. Primary support for the work by Michael P. Milham and the INDI team was provided by gifts from Joseph P. Healy and the Stavros Niarchos Foundation to the Child Mind Institute, http://fcon_1000 projects.nitrc.org/indi/abide/. This manuscript reflects the views of the authors and may not reflect the opinions or views of the database providers.

## REFERENCES

Arslan, F., Kabas, B., Dalmaz, O., Ozbey, M., & Çukur, T. (2024). Self-consistent recursive diffusion bridge for medical image translation. arXiv:2405.06789. http://arxiv.org/abs/2405.06789

Atli, O. F., Kabas, B., Arslan, F., Demirtas, A. C., Yurt, M., Dalmaz, O., & Çukur, T. (2024). I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling. arXiv:2405.14022v5. https://arxiv.org/pdf/2405.14022

Avants, N., Tustison, N. J., & Song, G. (2009). Advanced Normalization Tools V1.0. *The Insight Journal*, 16(03), 31. https://doi.org/10.54294/uvnhin

Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J., & Dalca, A. V. (2019). VoxelMorph: A learning framework for deformable medical image registration. *IEEE Transactions on Medical Imaging*, 38(8), 1788–1800. https://doi.org/10.1109/TMI.2019.2897538

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. *Journal of the Royal Statistical Society: Series B (Methodological)*, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Burgos, N., Cardoso, M. J., Thielemans, K., Modat, M., Pedemonte, S., Dickson, J., Barnes, A., Ahmed, R., Mahoney, C. J., Schott, J. M., Duncan, J. S., Atkinson, D., Arridge, S. R., Hutton, B. F., & Ourselin, S. (2014). Attenuation correction synthesis for hybrid PET-MR scanners: Application to brain studies. *IEEE Transactions on Medical Imaging*, 33(12), 2332–2341. https://doi.org/10.1109/TMI.2014.2340135

Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A. (2018). Multimodal MR synthesis via modality-invariant latent representation. *IEEE Transactions on Medical Imaging*, 37(3), 803–814. https://doi.org/10.1109/TMI.2017.2764326

Chen, C., Raymond, C., Speier, W., Jin, X., Cloughesy, T. F., Enzmann, D., Ellingson, B. M., & Arnold, C. W. (2023).
Synthesizing MR image contrast enhancement using 3D high-resolution ConvNets. *IEEE Transactions on Bio-Medical Engineering*, 70(2), 401. https://doi.org/10.1109/TBME.2022.3192309

Coupé, P., Catheline, G., Lanuza, E., & Manjón, J. V. (2017). Towards a unified analysis of brain maturation and aging across the entire lifespan: A MRI analysis. *Human Brain Mapping*, 38(11), 5501–5518. https://doi.org/10.1002/hbm.23743

Coupé, P., Manjón, J. V., Fonov, V., Pruessner, J., Robles, M., & Collins, D. L. (2011). Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation. *NeuroImage*, 54(2), 940–954. https://doi.org/10.1016/j.neuroimage.2010.09.018

Coupé, P., Mansencal, B., Clément, M., Giraud, R., Denis de Senneville, B., Ta, V. T., Lepetit, V., & Manjon, J. V. (2020). AssemblyNet: A large ensemble of CNNs for 3D whole brain MRI segmentation. *NeuroImage*, 219, 117026. https://doi.org/10.1016/j.neuroimage.2020.117026

Crespi, L., Loiacono, D., & Sartori, P. (2022). Are 3D better than 2D convolutional neural networks for medical imaging semantic segmentation? In *Proceedings of the International Joint Conference on Neural Networks, July 2022* (pp. 1–8). IEEE. https://doi.org/10.1109/IJCNN55064.2022.9892850

Dalmaz, O., Yurt, M., & Cukur, T. (2022). ResViT: Residual vision transformers for multimodal medical image synthesis. *IEEE Transactions on Medical Imaging*, 41(10), 2598–2614. https://doi.org/10.1109/TMI.2022.3167808

Dar, S. U. H., Yurt, M., Karacan, L., Erdem, A., Erdem, E., & Cukur, T. (2019). Image synthesis in multi-contrast MRI with conditional generative adversarial networks. *IEEE Transactions on Medical Imaging*, 38(10), 2375–2388. https://doi.org/10.1109/TMI.2019.2901750

Dayarathna, S., Islam, K. T., Uribe, S., Yang, G., Hayat, M., & Chen, Z. (2024). Deep learning based synthesis of MRI, CT and PET: Review and analysis. *Medical Image Analysis*, 92, 103046. https://doi.org/10.1016/j.media.2023.103046

Dohmen, M., Truong, T., Baltruschat, I. M., & Lenga, M. (2024). Five pitfalls when assessing synthetic medical images with reference metrics. In A. Mukhopadhyay, I. Oksuz, S. Engelhardt, D. Mehrof, & Y. Yuan (Eds.), *Deep generative models*. DGM4MICCAI 2024 (pp. 150–159). Springer. https://doi.org/10.1007/978-3-031-72744-3_15

Fuoli, D., Van Gool, L., & Timofte, R. (2021). Fourier space losses for efficient perceptual image super-resolution. *arXiv:2106.00783*. http://arxiv.org/abs/2106.00783

Gonzales, R. C., & Fittes, B. A. (1977). Gray-level transformations for interactive image enhancement. *Mechanism and Machine Theory*, 12(1), 111–122. https://doi.org/10.1016/0094-114X(77)90062-3

Han, K., Sheng, V. S., Song, Y., Liu, Y., Qiu, C., Ma, S., & Liu, Z. (2024). Deep semi-supervised learning for medical image segmentation: A review. *Expert Systems with Applications*, 245, 123052. https://doi.org/10.1016/J.ESWA.2023.123052

Han, M., Shim, H., & Baek, J. (2022). Perceptual CT loss: Implementing CT image specific perceptual loss for CNN-based low-dose CT Denoiser. *IEEE Access*, 10, 62412–62422. https://doi.org/10.1109/ACCESS.2022.3182821

Hanke, M., Baumgartner, F. J., Ibe, P., Kaule, F. R., Pollmann, S., Speck, O., Zinke, W., & Stadler, J. (2018). "Forrest Gump." OpenNeuro. https://doi.org/10.18112/openneuro ds000113.v1.3.0

Iglesias, J. E., Konukoglu, E., Zikic, D., Glocker, B., Van Leemput, K., & Fischl, B. (2013). Is synthesizing MRI contrast useful for inter-modality analysis? In K. Mori, I. Sakuma, W. Sato, C. Barillot, & N. Navab (Eds.), *Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013* (pp. 631–638). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-40811-3_79

Jiang, L., Dai, B., Wu, W., & Loy, C. C. (2021). Focal frequency loss for image reconstruction and synthesis. *CVF Open Access*. https://github.com/EndlessSora/focal-frequency-loss

Jog, A., Carass, A., Roy, S., Pham, D. L., & Prince, J. L. (2017). Random forest regression for magnetic resonance image synthesis. *Medical Image Analysis*, 35, 475–488. https://doi.org/10.1016/j.media.2016.08.009

Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. *arXiv:1603.08155*. http://arxiv.org/abs/1603.08155

Kamraoui, R. A., Ta, V.-T., Tourdias, T., Mansencal, B., Manjon, J. V., & Coupé, P. (2022). DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation. *Medical Image Analysis*, 76, 102312. https://doi.org/10.1016/j.media.2021.102312

Kingma, D. P., & Ba, J. L. (2014). Adam: A method for stochastic optimization. *3rd International Conference on Learning Representations, ICLR 2015—Conference Track Proceedings*. https://arxiv.org/abs/1412.6980v9

Li, W., Li, Y., Qin, W., Liang, X., Xu, J., Xiong, J., & Xie, Y. (2020). Magnetic resonance image (MRI) synthesis from brain computed tomography (CT) images based on deep learning methods for magnetic resonance (MR)-guided radiotherapy. *Quantitative Imaging in Medicine and Surgery*, 10(6), 1223. https://doi.org/10.21037/QIMS-19-885

Li, Y., Zhou, T., He, K., Zhou, Y., & Shen, D. (2022). Multi-scale transformer network with edge-aware pre-training for cross-modality MR image synthesis. *IEEE Transactions on Medical Imaging*, 42(11), 3395–3407. https://doi.org/10.1109/TMI.2023.3288001

Lindig, T., Kotikalapudi, R., Schweikardt, D., Martin, P., Bender, F., Klose, U., Ernemann, U., Focke, N. K., & Bender, B. (2018). Evaluation of multimodal segmentation based on 3D T1-, T2- and FLAIR-weighted images—The difficulty of choosing. *NeuroImage*, 170, 210–221. https://doi.org/10.1016/j.neuroimage.2017.02.016

Liu, J., Pasumarthi, S., Duffy, B., Gong, E., Datta, K., & Zaharchuk, G. (2022). One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation. *IEEE Transactions on Medical Imaging*, 42(9), 2577–2591. https://doi.org/10.1109/TMI.2023.3261707

Liu, X., Xing, F., Prince, J. L., Carass, A., Stone, M., Fakhri, G. El, & Woo, J. (2021). Dual-cycle constrained bijective Vae-Gan for tagged-to-cine magnetic resonance image synthesis. In *Proceedings—2021 IEEE 18th International Symposium on Biomedical Imaging, April 2021* (pp. 1448–1452). IEEE. https://doi.org/10.1109/ISBI48211.2021.9433852

Manjón, J. V., Coupé, P., Martí-Bonmatí, L., Collins, D. L., & Robles, M. (2010). Adaptive non-local means denoising of MR images with spatially varying noise levels. *Journal of Magnetic Resonance Imaging: JMRI*, 31(1), 192–203. https://doi.org/10.1002/JMRI.22003

Manjón, J. V., Morell-Ortega, S., Ruiz-Perez, M., Mansencal, B., Le Bot, E., Gadea, M., Lanuza, E., Catheline, G., Tourdias, T., Planche, V., Giraud, R., Rivì Ere, D., Mangin, J.-F., Labra-Avila, N., Vivo-Hernando, R.,
Rubio, G., Aparici, F., De La Iglesia-Vaya, M., & Coupé, P. (2025). Ultra-high resolution multimodal MRI dense labelled holistic brain atlas. *arXiv:2501.16879*. https://arxiv.org/abs/2501.16879v1

Manjón, J. V., Romero, J. E., & Coupe, P. (2022). Deep learning based MRI contrast synthesis using full volume prediction using full volume prediction. *Biomedical Physics and Engineering Express*, 8(1), 015013. https://doi.org/10.1088/2057-1976/ac3c64

Manjón, J. V., Romero, J. E., Vivo-Hernando, R., Rubio-Navarro, G., De la Iglesia-Vaya, M., Aparici-Robles, F., & Coupé, P. (2020). Deep ICE: A deep learning approach for MRI intracranial cavity extraction. *arXiv:2001.05720*. https://arxiv.org/abs/2001.05720v1

Manjón, J. V., Tohka, J., García-Martí, G., Carbonell-Caballero, J., Lull, J. J., Martí-Bonmatí, L., & Robles, M. (2008). Robust MRI brain tissue parameter estimation by multistage outlier rejection. *Magnetic Resonance in Medicine*, 59(4), 866–873. https://doi.org/10.1002/mrm.21521

Mei, X., Liu, Z., Robson, P. M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K. E., Yang, T., Wang, Y., Greenspan, H., Deyer, T., Fayad, Z. A., & Yang, Y. (2022). RadImageNet: An open radiologic deep learning research dataset for effective transfer learning. *Radiology: Artificial Intelligence*, 4(5), e210315. https://doi.org/10.1148/RYAI.210315

Meng, X., Gu, Y., Pan, Y., Wang, N., Xue, P., Lu, M., He, X., Zhan, Y., & Shen, D. (2022). A novel unified conditional score-based generative framework for multi-modal medical image completion. *arXiv:2207.03430*. http://arxiv.org/abs/2207.03430

Miller, M. I., Christensen, G. E., Amitt, Y., & Grenander, U. (1993). Mathematical textbook of deformable neuroanatomies. *Proceedings of the National Academy of Sciences of the United States of America*, 90(24), 11944–11948. https://doi.org/10.1073/pnas.90.24.11944

Morell-Ortega, S., Ruiz-Perez, M., Gadea, M., Vivo-Hernando, R., Rubio, G., Aparici, F., Iglesia-Vaya, M. de la, Catheline, G., Mansencal, B., Coupé, P., & Manjón, J. V. (2025). DeepCERES: A deep learning method for cerebellar lobule segmentation using ultra-high resolution multimodal MRI. *NeuroImage*, 308, 121063. https://doi.org/10.1016/J.NEUROIMAGE.2025.121063

Osman, A. F. I., & Tamam, N. M. (2022). Deep learning-based convolutional neural network for intramodality brain MRI synthesis. *Journal of Applied Clinical Medical Physics*, 23(4), e13530. https://doi.org/10.1002/acm2.13530

Pan, K., Cheng, P., Huang, Z., Lin, L., & Tang, X. (2022). Transformer-based T2-weighted MRI synthesis from T1-weighted images. In *Proceedings of the 2022 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, July 2022* (pp. 5062–5065). IEEE. https://doi.org/10.1109/EMBC48229.2022.9871183

Pan, S., Chang, C.-W., Peng, J., Zhang, J., Qiu, R. L. J., Wang, T., Roper, J., Liu, T., Mao, H., & Yang, X. (2023). Cycle-guided denoising diffusion probability model for 3D cross-modality MRI synthesis. *arXiv:2305.00042*. https://doi.org/10.48550/arXiv.2305.00042

Park, M. T. M., Pipitone, J., Baer, L. H., Winterburn, J. L., Shah, Y., Chavez, S., Schira, M. M., Lobaugh, N. J., Lerch, J. P., Voineskos, A. N., & Chakravarty, M. M. (2014). Derivation of high-resolution MRI atlases of the human cerebellum at 3T and segmentation using multiple automatically generated templates. *NeuroImage*, 95, 217–231. https://doi.org/10.1016/J.NEUROIMAGE.2014.03.037

Pérez-García, F., Sparks, R., & Ourselin, S. (2021). TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. *Computer Methods and Programs in Biomedicine*, 208, 106236. https://doi.org/10.1016/j.cmpb.2021.106236

Pinho, A. L. G., Hertz-Pannier, L., & Thirion, B. (2018). *Individual brain charting*. OpenNeuro. https://doi.org/10.18112/openneuro.ds000244.v1.0.0

Racey, C., Rae, C., Simpson, I., Bowen-Hill, O., Bauer, M., Simner, J., Kampourei, C., & Ward, J. (2023). *Mapping the connectome of synaesthesia: An open access MRI dataset*. OpenNeuro. https://doi.org/doi:10.18112/openneuro.ds004466.v1.0.0

Reinhold, J. C., Dewey, B. E., Carass, A., & Prince, J. L. (2019). Evaluating the impact of intensity normalization on MR image synthesis. In *Proc. SPIE 10949, Medical Imaging 2019: Image Processing, 109493H* (pp. 890–898. SPIE https://doi.org/10.1117/12.2513089

Rogers, C. S., Jones, M. S., McConkey, S., & Peelle, J. E. (2022). Listening task. *OpenNeuro*. [*Dataset*]. https://doi.org/10.18112/openneuro.ds004285.v1.0.0

Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In N. Navab, J. Hornegger, W. Wells, & A. Frangi (Eds.), *Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015* (pp. 234–241). Springer International Publishing. https://doi.org/10.1007/978-3-319-24574-4_28

Roy, S., Carass, A., & Prince, J. L. (2013). Magnetic resonance image example-based contrast synthesis. *IEEE Transactions on Medical Imaging*, 32(12), 2348–2363. https://doi.org/10.1109/TMI.2013.2282126

Ruiz-Perez, M., Morell-Ortega, S., Gadea, M., Vivo-Hernando, R., Rubio, G., Aparici, F., de la Iglesia-Vaya, M., Tourdias, T., Coupé, P., Manjón, J. V., & Ruiz, M. (2024). DeepThalamus: A novel deep learning method for automatic segmentation of brain thalamic nuclei from multimodal ultra-high resolution MRI. *arXiv:2401.07751*. https://doi.org/10.48550/arXiv.2401.07751

Shivdeo, A., Lokwani, R., Kulkarni, V., Kharat, A., & Pant, A. (2021). Evaluation of 3D and 2D deep learning techniques for semantic segmentation in CT scans. In *IcABCD 2021–4th International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems, Proceedings* (pp. 1–8). IEEE. https://doi.org/10.1109/ICABCD51485.2021.9519338

Teneggi, J., Tivnan, M., Stayman, J. W., & Sulam, J. (2023). How to trust your diffusion model: A convex optimization approach to conformal risk control. *Proceedings of Machine Learning Research*, 202, 33940–33960. https://doi.org/10.1117/12.3007923

Tustison, N. J., Avants, B. B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., & Gee, J. C. (2010). N4ITK: Improved N3 bias correction. *IEEE Transactions on Medical Imaging*, 29(6), 1310–1320. https://doi.org/10.1109/tmi.2010.2046908

Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018). *Deep image prior* (pp. 9446–9454). http://www.cs.tut.fi/

Upadhyay, U., & Awate, S. P. (2021). A mixed-supervision multilevel GAN framework for image quality enhancement. In *Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, 11768 LNCS (pp. 556–564). Springer. https://doi.org/10.1007/978-3-030-32254-0_62

Uzunova, H., Ehrhardt, J., & Handels, H. (2020). Memory-efficient GAN-based domain translation of high
resolution 3D medical images. *Computerized Medical Imaging and Graphics*, 86, 101801. https://doi.org/10.1016/j.compmedimag.2020.101801

Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prior, L., Marcus, D., Michalareas, G., Moeller, S., ... Yacoub, E. (2012). The Human Connectome Project: A data acquisition perspective. *NeuroImage*, 62(4), 2222-2231. https://doi.org/10.1016/j.neuroimage.2012.02.018

Viviani, R., Pracht, E. D., Brenner, D., Beschoner, P., Stingl, J. C., & Stöcker, T. (2017). Multimodal MEMPRAGE, FLAIR, and R2\* segmentation to resolve dura and vessels from cortical gray matter. *Frontiers in Neuroscience*, 11, 258. https://doi.org/10.3389/fnins.2017.00258

Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. *IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society*, 13(4), 600–612. https://doi.org/10.1109/TIP.2003.819861

Wolna, A., & Wodniecka, Z. (2023). *Bilingual speech production with functional localizers*. OpenNeuro. https://doi.org/doi:10.18112/openneuro ds004456 v1.0.0

Xiang, L., Chen, Y., Chang, W., Zhan, Y., Lin, W., Wang, Q., & Shen, D. (2019). Deep-learning-based multi-modal fusion for fast MR reconstruction. *IEEE Transactions on*

Biomedical Engineering, 66(7), 2105-2114. https://doi.org/10.1109/TBME.2018.2883958

Yan, S., Wang, C., Chen, W., & Lyu, J. (2022). Swin transformer-based GAN for multi-modal medical image translation. *Frontiers in Oncology*, 12, 942511. https://doi.org/10.3389/fonc.2022.942511

Yoon, J. S., Zhang, C., Suk, H.-I., Guo, J., & Li, X. (2023). SADM: Sequence-aware diffusion model for longitudinal medical image generation. In A. Frangi, M. de Bruijne, D. Wassermann, & N. Navab (Eds.), *Information Processing in Medical Imaging*. IPMI 2023 (Vol. 13939, pp. 388–400). Springer. https://doi.org/10.1007/978-3-031-34048-2_30

Zeng, Y., Liu, N., Yang, X., Huang, C., & Liu, M. (2024). Enhanced multimodal brain tumor classification in MR images using 2D ResNet as backbone with explicit tumor size information. *Journal of Cancer*, 15(13), 4275-4286. https://doi.org/10.7150/JCA.95987

Zhang, X., He, X., Guo, J., Ettehadi, N., Aw, N., Semanek, D., Posner, J., Laine, A., & Wang, Y. (2022). PTNet3D: A 3D high-resolution longitudinal infant brain MRI synthesizer based on transformers. *IEEE Transactions on Medical Imaging*, 41(10), 2925-2940. https://doi.org/10.1109/TMI.2022.3174827

Zhao, P., Pan, H., & Xia, S. (2021). MRI-Trans-GAN: 3D MRI cross-modality translation, 2021 40th Chinese Control Conference (CCC), Shanghai, China, pp. 7229-7234. https://doi.org/10.23919/CCC52363.2021.9550256

Zhu, L., Xue, Z., Jin, Z., Liu, X., He, J., Liu, Z., & Yu, L. (2023). Make-A-Volume: Leveraging latent diffusion models for cross-modality 3D brain MRI synthesis. *arXiv:2307.10094*. http://arxiv.org/abs/2307.10094