Translation of Partially Paired Images with Generative Adversarial Networks

Authors: sachin  chhabra
yaoxin  zhuo
riti  paul
javad  sohankar
ji  luo
shan  li
wendy  lee
yi  su
teresa  wu
baoxin  li
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-05 07:47:30

Abstract: The integration of paired medical MRI (Magnetic Resonance Imaging) and PET (Positron Emission Tomography) images holds considerable significance in clinical evaluations and offers a richer source of clinical insights. 
However, acquiring paired MRI-PET images poses challenges due to the various practical constraints.
To address this, MRI-PET translation emerges as a valuable approach, enabling professionals to obtain complementary information from one modality and enhance decision-making using only single-modality images.
Existing approaches predominantly rely on either using paired MRI-PET images for training or treating the entire dataset as unpaired.
In this study, we introduce PaPaGAN, an innovative end-to-end Partially Paired Generative Adversarial Network specifically tailored for partially paired images. In a practical setting, where a mix of paired and unpaired data is available, PaPaGAN leverages the unpaired data to learn a mapping function capable of generating a noisy intermediate image. To refine this intermediate image and address the inconsistencies during the unpaired translation process, PaPaGAN employs a secondary image translation module. This module is specifically trained using the paired data, which provides a consistent mapping from source to target domain images. By effectively harnessing both paired and unpaired MRI-PET images, our method significantly enhances translation capabilities, facilitating precise image translation and elevating image quality for the target modality. Our quantitative and qualitative medical image translation experiments on two public datasets, ADNI and OASIS, demonstrate the superiority of PaPaGAN over alternative image translation methods.

Main paper: 
 i. introduction: 
 Recent advancements in medical image-to-image translation have revolutionized clinical practices, streamlining workflows and opening new avenues for diagnostic and treatment planning [15] , [20] , [23] . These methods are very effective in generating realistic medical images across diverse modalities, democratizing access to medical imaging, and potentially revolutionizing patient care and research. An exemplary application lies in the translation between Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). While MRI provides intricate details of soft tissues, aiding in the assessment of brain structure and pathology, PET highlights metabolic activities crucial for diseases like Alzheimer's. The synergistic combination of both modalities offers invaluable insights for accurate diagnosis and treatment decisions. However, obtaining both MRI and PET scans for every patient poses challenges due to cost constraints, equipment availability, and concerns over radiation exposure. In such cases, image-to-image translation models can be employed to enhance decision-making by providing complementary information of the missing modality.Generative adversarial networks (GANs) [9] is a popular image-generation technique and serve as the cornerstone for several deep learning-based image-generation techniques [17] , [19] , including image translation [1] , [4] , [12] , [30] . These image translation methods can broadly be classified into paired and unpaired approaches. Paired medical image translation relies on access to matched MRI-PET image pairs for model training, whereas unpaired methods operate on MRI and PET images without requiring pairwise correspondences.Pix2pix [12] stands out as a leading technique for image-toimage translation, particularly suited for paired data scenarios. It employs an image generator and an image discriminator. The discriminator's objective is to differentiate between the original target image and the images synthesized by the generator. The generator tries to outsmart the discriminator and, in turn, produces realistic images that closely mimic the target domain. By harnessing the power of paired images, Pix2pix excels in enhancing pixel-to-pixel level similarity, thereby preserving fine-grained details during translation. Other techniques for paired data follow the principle with some extensions. However, the availability of paired data is scarce, particularly in the medical domain, where data collection can be arduous and resource-intensive.For image translation using unpaired data, which is often easier to collect, CycleGAN [30] is the most popular technique. It employs two generator-discriminator pairs, one focusing on the source domain while the other tackles the target domain. One generator is tasked with converting images from the source domain to the target domain, while the other performs the inverse transformation. Similarly, one discriminator focuses on the source domain, and the other tackles the target domain. The generators are trained to fool the discriminator along with a cycle consistency loss that helps to preserve the main content of the image. The cycle consistency loss is an unreliable method for preserving the content as an image can be translated to something entirely different and then reconstructed as the original when translated back. Most unpaired data training approaches are based on CycleGAN with limited enhancements.Our method, PaPaGAN, is specifically designed to handle partially paired data, where some data is paired while the rest remains unpaired. This is a more realistic setting as it is always easy to get unpaired data, which can be combined with paired data to create a partially paired dataset. Traditional paired image translation methods overlook unpaired data, missing out on potential enhancements. Conversely, unpaired approaches neglect the valuable information available in paired data, potentially yielding suboptimal translations. PaPaGAN bridges this gap by leveraging both paired and unpaired data effectively.Our approach employs two generator-discriminator pairs trained with a mix of paired and unpaired data (treated as unpaired data only) and cycle consistency loss to translate images into the target domain. The output from these generators is then fed into a secondary generator-discriminator pair trained exclusively on paired data. Therefore, the input image passes through two image generator modules to achieve the final target image. The first generator creates an image in the target domain but emphasizes encoding the source properties during its reconstruction due to the cycle consistency loss. However, this loss can be ineffective when significant differences exist between domains. To address this, we introduce a second image generator that takes the source image and the output of the first generator to produce a more accurate version.PaPaGAN is a bi-directional translation method that simultaneously trains networks for both translation directions. This bi-directionality allows us to use the same discriminator for translating paired data in both directions, effectively regularizing the networks. In this work, our contributions can be summarized as follows:• PaPaGAN is designed to work effectively with partially paired datasets, which include both paired and unpaired data. This setting is more realistic and practical in many real-world applications where unpaired data is more readily available than paired data. • To address the limitations of cycle consistency loss, especially when there are significant differences between domains, PaPaGAN introduces a second generator to produce an enhanced translation. • Comprehensive evaluations on benchmark datasets such as ADNI [26] and OASIS [13] demonstrate that PaPa-GAN outperforms unpaired and paired baselines. Experiments show that the partially paired data-based frameworks provide advantages that neither setting (paired/unpaired) can achieve alone. This highlights the innovative aspect of PaPaGAN in integrating multiple levels of supervision for superior performance.While PET is a relatively new modality compared to noninvasive MRI, its application in diagnosis is rising. However, obtaining both MRI and PET scans for each patient is often impractical due to the high costs, limited availability of PET scanners, and concerns over radiation exposure. To mitigate these issues, recent research has focused on generating PET data from existing MRI or CT scans. For example, Li et al. [14] utilized a 3D-CNN to learn the non-linear mapping between paired MRI-PET scans from the ADNI dataset. Gao et al. [8] introduced RIED-Net, which aims to produce higher-quality PET images from MRI data (also from the ADNI dataset). These techniques, however, often result in smoothed images that lose detail because they use L1/L2 optimization methods. Notably, the task of MRI-to-PET translation is related to, but distinct from, MRI super-resolution (e.g., [6] ), where the output is higher-resolution MRI images, and domain adaptation (e.g., [2] , [3] ), where classification is typically the objective.Generative Adversarial Networks (GANs) have recently shown significant success in image generation and restoration tasks. Due to their ability to produce high-quality synthetic images, GANs have been integrated into various multi-modal or cross-modal medical imaging tasks. Pan et al. [18] proposed a 3D conditional GAN (cGAN) framework to model bi-directional mappings between MRI and PET scans for Alzheimer's Disease (AD) diagnosis. Yaakub et al. [28] designed a 3D GAN with residual connections to learn the mapping from MRI to PET for evaluating patients with focal epilepsy. Sikka et al. [22] employed a method that enhances both global structural integrity and local detail fidelity in synthetic PET images using a multi-path architecture. Other significant architectures utilizing paired data for training include Pix2Pix [12] , CoCaGAN [11] , and EA-GAN [29] .Despite their success, these frameworks depend heavily on paired cross-modal data, which is costly and labor-intensive to collect and annotate. Several unpaired image translation frameworks [4] , [5] , [30] have been developed recently to tackle the challenge of paired data scarcity. For instance, Zhu et al. [30] introduced CycleGAN, which incorporates a cycle consistency loss specifically designed for unpaired medical image generation. StarGAN [4] , [5] uses a discriminator that distinguishes between fake and real images and performs auxiliary domain classification. Another notable method for unpaired image-to-image translation is the UNsupervised Image-to-image Translation (UNIT) [16] , which is based on a Variational Autoencoder (VAE)-GAN framework.In our work, we address a more common scenario where data often exist in a partially paired form; some data is paired while most remain unpaired. Instead of relying solely on cycle consistency loss [30] , we propose to improve or correct a noisy approximation of the translated image using paired data mapping. The proposed method helps to preserve structural details from the source and incorporate target domain style adaptation for enhanced image quality. Consider a dataset P ∈ {x p , y p } Np comprising N p pairs of source and target domain images, where x p represents a source image and y p denotes its corresponding image from the target domain. Additionally, let U ∈ {x u } Nx , {y u } Ny represent an unpaired dataset containing N x source images and N y target images. The objective of PaPaGAN is to learn an optimal image translation mapping between the source and target domains by utilizing both the paired data P and the unpaired data U .Partially Paired data compromises of both paired P and unpaired U data. However, paired data can also be treated as unpaired data. Hence, we first combine P paired and U unpaired datasets to form a larger unpaired dataset U ′ of size N p + N u . By doing this, we effectively increase the size of available unpaired data. More data helps networks approximate their domains better and also regularize the networks.The training with unpaired data compromises two pairs of generator and discrimination, namely G u1 , G u2 , D u1 , and D u2 , which are trained using generative adversarial network framework [9] . The generators G u1 take x as input and outputs ȳ = G u1 (x) and similarly we get x = G u2 (y). We train the discriminators D u1 and D u2 to differentiate real vs generated images using,L ud = -E x,y∼U ′ [(1 -D u1 (y)) 2 + D u1 (G u1 (x)) 2 + (1 -D u2 (x)) 2 + D u2 (G u2 (y)) 2 ] (1)The generators are tasked to fool the discriminators, which in turn helps them to generate realistic images as,EQUATIONWe also train the image generators with cycle-consistency loss [30] to ensure network outputs do not refrain too much from the original input,EQUATIONThe two training loss functions of image generators are balanced using a hyper-parameter λ u = 10. The final training loss for the image generator is,EQUATIONUnpaired data training uses cycle consistency loss to ensure the content does not change much during the translation. However, in cases with higher domain differences, such as crossmodal medical imaging, the effectiveness of cycle consistency loss tends to deteriorate. In our case, MRI and PET modalities contain significant domain differences. In such cases, cycle consistency forces the network to encode source properties for reconstruction rather than preserving visual content.To combat this problem, we refine the generated target image using another set of generators G p1 and G p2 . The generators G p1 and G p2 are specialized for task of correcting mistakes of G u1 and G u2 respectively. They take the output from the unpaired generators G u1 and G u2 along with their inputs x and y and learn to generate the correct outputs using paired data ȳ = G p1 (x; G u1 (x)) and x = G p2 (G u2 (y); y) respectively.Unlike traditional GANs, we train the generators G p1 and G p2 using a single discriminator D p for both domains instead of two discriminators. The discriminator D p is a conditional discriminator that takes both source and target images as input.To provide multiple inputs in the generators and discriminator, we fuse the source and the target domain images along the channel dimension, doubling the number of channels in their inputs. This strategy enables G p1 , G p2 and D p to learn the structural similarities inherent in paired source and target domain images. By doing so, the discriminator is better equipped to ensure that the generator preserves the essential content and structural details during the translation, resulting in more accurate and realistic outputs. It also enables us to use the same discriminator for both domains, effectively increasing its training data. We train the D p to differentiate real vs fake pairs of both the domains as,EQUATIONThe 1 2 weightage is given to each fake generated image pair to avoid any bias towards the fake class due to the imbalance. The generators are tasked to fool the discriminator,EQUATIONIntegrating the GAN losses with conventional metrics such as L 1 and L 2 distances has proven beneficial for convergence [12] , [30] . This accelerates convergence rates and further encourages generated outputs to closely resemble the ground truth. Thus, we incorporate the L 1 distance into the training objective of the generators as,EQUATIONSimilar to unpaired data training, we balance the generator losses using λ p = 10 hyperparameter.The primary goal is to ensure that the final output images are as accurate as possible. To achieve this, the paired generators refine and correct the output of the unpaired generators. However, if the output of unpaired generators is optimal, we want the paired generators to recognize and retain optimal outputs from the unpaired generators. Hence, we add an identity loss for the paired generators as,EQUATIONThe only difference between 7 and 8 is that we use the ground truth target image as part of the input to the paired generators. By incorporating the ground truth target image into the input during training, the paired generators learn to recognize correct outputs. This training method ensures that paired generators can effectively utilize the correct inputs without altering them unnecessarily. It also augments their training data and enhances the overall performance.We follow the standard procedure of training discriminators and generators iteratively. First, we train the unpaired discriminators D u1 and D u1 on unpaired data and the paired discriminator D p on paired data, respectively. Following discriminator training, we train the combined generators framework G u1 , G u2 , G p1 , and G p2 of paired and unpaired data with the combined loss functions,EQUATIONλ u , λ p , and λ i are loss-weighing hyperparameters and are set to 10, 10, and 1, respectively. We generate the final output target image as ȳ = G p1 (x; G u1 (x)) and ȳ = G p2 (G u2 (y); y) for domains using the trained generators and inputs x and y respectively. Our approach is modeled in Figure 1 .IV. EXPERIMENTS A. Settings 1) Dataset: To assess the effectiveness of PaPaGAN, we conducted experiments on two prominent public MRI-PET datasets: ADNI (Alzheimer's Disease Neuroimaging Initiative) and OASIS-3 (Open Access Series of Imaging Studies-3). For a comprehensive evaluation, we performed image translation experiments in both directions: from MRI→PET and vice versa. ADNI is a multisite study aimed at enhancing clinical trials for Alzheimer's Disease (AD) prevention and treatment. From ADNI, we collected 2,717 paired 3D MRI-PET samples after excluding any corrupted or missing data. Additionally, we gathered an additional 2,174 MRI samples lacking paired PET counterparts. OASIS-3, collected by Washington University, is a third-generation database obtained with a waiver of informed consent. After filtering out corrupted or missing data, we obtained 770 3D paired MRI-PET and 2,071 extra MRI samples, respectively. The PET data for both of these datasets have been acquired with different radiotracers, heterogeneous patient demographics, and various disease-specificities. In this work, we only consider the criteria of a single radiotracer when filtering PET images. For ADNI and OASIS-3, FBP and AV45 PET images were selected during filtering, respectively.To facilitate our experiments, we divided each dataset randomly into five equal splits, each containing MRI-PET 3D paired samples. One split was allocated for testing, two splits for paired training data, and the remaining two splits for unpaired data. To ensure fairness, PET images from the last two splits (unpaired data) were paired only with MRI images from the extra unpaired MRI dataset, preventing any overlap between paired and unpaired images. This setup mirrors realworld application scenarios. This resulted in 1,088 paired and 1,088 unpaired samples in the ADNI training set, while the test set contained 543 paired samples. The OASIS training set comprised 308 paired and 308 unpaired samples, with 154 paired samples in the test set. 2) Data Preprocessing: The T1-weighted MR data are rigidly aligned with an MNI template and then analyzed using FreeSurfer [7] (version 7) for intensity inhomogeneity correction and standardization followed by volumetric segmentation and cortical parcellation. PET imaging analysis was performed using the standard protocols that included scanner harmonization, motion correction, and PET-to-MR registration. We further processed the PET images through a unified pipeline [24] to extract regional values, culminating in the generation of intensity-normalized SUVR images. For these images, the cerebellum served as the reference region, providing a stable baseline for comparing and analyzing metabolic activity across the brain region. All images are processed to 256 × 256 × 256.3) Implementation Details: All methods utilized a 3D cubic input dimension of size 256 × 256 × 256. To ensure regularization, a zero padding of 12 size was applied to each dimension, accompanied by RandomCrop. For paired data, the same cropping scheme was employed to maintain structural similarities between pairs. The training comprised 25,000 iterations with the Adam optimizer using β values of (0.5, 0.999) and a batch size of 4. The learning rate was set to 2e-4 and underwent linear decay to zero after half the total iterations, with a warm-up period of 500 iterations initially. The same configuration was employed across all experiments (including baselines). The code accompanying this work will be released on the author's website.We opt for a 3D-UNet [21] architecture with 3D convolution and 3D Instance Normalization operations and skip connections [10] to serve as our generator due to its efficient memory utilization. However, it's worth noting that any image translation framework could be utilized. The encoder of the UNet used a leaky relu activation, whereas the decoder used relu activation. The number of features increases from 12 to 192 in the encoder, and the decoder decreases it in reverse for unpaired generators. The paired generators are smaller and contain features ranging from 8 to 64 only with a shallower depth. All discriminators adopt a 3D Patch discriminator design [12] , enabling them to classify whether an image patch is real or synthetic. This choice of patch discriminators offers advantages such as parameter efficiency and improved regularization. It uses feature maps ranging from 16 to 128 based on the depth of the network. Each block uses a convolution layer followed by Instance Normalization and leaky relu activation except the first and the last block. The first block skips the normalization layer similar to [12] , [30] , and the last block consists of a single convolution that maps the features maps to a single dimension. The paired discriminator concatenates paired data along the channel dimension, whereas the unpaired discriminator uses the original input. The unpaired generator, paired generator, and each discriminator contain about 5.9 million, 1.8 million, and 1.75 million training parameters, respectively.1) Quantitative Results: We employ Root Mean Squared Error (RMSE), Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio (PSNR) to assess the performance of various image translation models. We compared our approach against the following image translation methods: Pix2Pix, PAN, and CycleGAN, which belong to different settings of supervision. Pix2Pix and PAN are paired data-based image translation methods requiring full supervision. Cycle-GAN, on the other hand, is a popular unsupervised imagetranslation method that doesn't require any paired data. Xu et al. [27] is a semi-paired data-based framework similar to ours. However, we exclude this from the baseline comparison due to a lack of code availability and dataset differences. CycleGAN, on the other hand, exhibits inferior results due to its reliance on unpaired data and the necessity to preserve unnecessary information for original image reconstruction. This leads to a degradation in its image translation quality.Furthermore, our analysis reveals a notable discrepancy in translation performance between PET→MRI and MRI→PET directions. This discrepancy primarily stems from MRI images containing richer detail and high-frequency information, which aids the model in learning more precise MRI→PET translations. Conversely, the scarcity of detailed information in PET images poses challenges for generating high-quality MRI images in the reverse PET→MRI translation task. We believe that more training data should alleviate this issue.To evaluate the statistical significance of our results, we conducted a paired t-test of all baseline methods with respect to our framework, PaPaGAN. We set the following standards for evaluation:• Null Hypothesis (p > 0.05): No significant difference between PSNR/SSIM of two methods • Alternate Hypothesis (p ≤ 0.05): The difference between PSNR/SSIM is significant (not by chance).Table II demonstrates that for all the methods, PaPaGAN has significantly different quantitative metrics for both PSNR and SSIM in all cases. For all baselines compared in Table II , our framework has significantly low p-values, indicating that the quality of the translated images using our framework is higher.2) Output Visualization: output images of the unpaired generator in Figure 2 (PaPaGAN -Unpaired in the 5 th row). We can observe that the unpaired output is missing all the essential intensities, which are then corrected by the paired generator in the final output. The comparison in these figures highlights the significant advantages of PaPaGAN's approach. By integrating unpaired and paired data through a two-stage generation process, PaPaGAN can produce images that adhere closely to the desired target domain characteristics and retain essential details that might be lost when relying solely on unpaired data.3) Impact of the number of Paired Images: We showcase the significance of paired and unpaired data in PaPaGAN by conducting experiments on the quantity of paired and unpaired images. Instead of employing two splits containing paired images (1, 088 paired images), we substitute paired data with unpaired data, i.e., we replace paired data with unpaired data. This keeps the total (paired+unpaired) the same but reduces paired data and increases unpaired data. Since Pix2Pix is trained exclusively on paired data, its training data is reduced, whereas PaPaGAN utilizes all the data. The findings of this experiment are illustrated in Figure 4 . It shows that as the number of paired images decreases, the performance of pix2pix suffers significantly, whereas PaPaGAN exhibits a comparatively less decline in performance, highlighting the substantial impact of unpaired data.In this study, we introduce PaPaGAN, an innovative endto-end medical image translation model leveraging GAN to utilize partially paired images. Our setting closely mimics realworld application scenarios, where a combination of paired and unpaired data is commonly encountered. PaPaGAN effectively utilizes both types of data to bolster the performance of the image translator by preserving the image content and providing high-quality outputs. Through comprehensive experiments across two-direction cross-modal image translation tasks and datasets, PaPaGAN demonstrates it outperformed the baselines in both quantitative and qualitative evaluations. 
 
 Appendix:  



Reviews: review: title: title: The review of 'PaPaGAN: Translation of Partially Paired Images with Generative Adversarial Networks'
paper_summary: paper_summary: The authors introduce PaPaGAN, a novel GAN-based model designed for translating between PET and MRI medical images. This approach employs dual generator-discriminator pairs, effectively leveraging both paired and unpaired image data. The proposed model addresses a key limitation observed in CycleGAN, which struggles to accurately translate features between domains with substantial differences.

PaPaGAN's performance was evaluated on the ADNI and OASIS datasets, where it demonstrated superior results compared to Pix2pix, PAN, and CycleGAN in both PET-to-MRI and MRI-to-PET translation tasks. These findings underscore the model's efficacy in utilizing partially paired data, leading to significant improvements in image translation accuracy.
strength_weakness: strength_weakness: The visualization for the more challenging PET-to-MRI tasks is notably absent, with no explanation provided. Additionally, the visualization lacks clarity regarding the biological significance of the generated images. For example, it is unclear whether the images accurately reflect cellular-level abnormalities or if a pathologist could reliably diagnose the disease based on these generated images.
questions: questions: Could a more systematic approach to model selection be provided? The ablation analysis in this study appears to be closely tied to hyperparameters like loss weight. I observed that the optimal number of unpaired samples saturates at a relatively high level (~544). A more thorough model selection analysis could potentially further enhance the impact of this work.
title: title: Translation of Partially Paired Images with Generative Adversarial Networks
paper_summary: paper_summary: The paper presents PaPaGAN, an end-to-end Partially Paired Generative Adversarial Network designed specifically for partially paired images. Additionally, PaPaGAN utilizes unpaired data to learn a mapping function that generates a noisy intermediate image, enabling it to work effectively with both paired and unpaired data.
strength_weakness: strength_weakness: In the introduction, the paper claims that GANs are the state-of-the-art method for image generation, which contradicts the fact that Stable Diffusion has been the leading approach for this task. Additionally, the authors assert that their setting mimics real-world scenarios. However, it's unclear how this is feasible, as clinicians typically compare images to ground truth. How can they perform such comparisons if they don't have a paired image to reference?
questions: questions: How can a clinician compare generated images to the ground truth if there isn't a paired image for reference?
title: title: GAN framework designed to improve the translation of partially paired MRI-PET images
paper_summary: paper_summary: The paper introduces PaPaGAN, a novel Generative Adversarial Network (GAN) designed to facilitate the translation of partially paired medical images, specifically between MRI (Magnetic Resonance Imaging) and PET (Positron Emission Tomography). This translation is valuable in clinical settings where paired MRI-PET data is often limited due to practical constraints such as cost and radiation exposure.The method demonstrates superior performance compared to existing paired and unpaired translation techniques, as validated by experiments on public datasets like ADNI and OASIS.
strength_weakness: strength_weakness: While PaPaGAN is designed to work with both paired and unpaired data, it would be critical to understand how sensitive its performance is to changes in the proportion of paired data available.The experiments primarily focus on MRI-PET translations using specific public datasets like ADNI and OASIS. It would be important to evaluate whether PaPaGAN can maintain its high performance across other imaging modalities or datasets with different characteristics, to assess its robustness and applicability in broader clinical contexts.
questions: questions: No