Breaking the Dilemma of Medical Image-to-image Translation

1 Introduction . Generative adversarial networks ( GANs ) [ 1 ] is a framework that simultaneously trains a generator G and a discriminator D through an adversarial process . The generator is used to translate the distribu- ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tion of source domain images X to the distribution of target domain images Y . The discriminator is used to determine if the target domain images are likely from the generator or from the real data . min G max D LAdv ( G , D ) = Ey [ log ( D ( y ) ) ] + Ex [ log ( 1−D ( G ( x ) ) ) ] ( 1 ) Supervised Pix2Pix [ 2 ] and unsupervised Cycle-consistency [ 3 ] are the two commonly used modes in GANs . Pix2Pix updates the generator ( G : X → Y ) by minimizing pixel-level L1 loss between the source image x and the target image y . Therefore , it requires well aligned paired images , where each pixel has a corresponding label . min G LL1 ( G ) = Ex , y [ ∥y −G ( x ) ∥1 ] ( 2 ) Well aligned paired images , however , are not always available in real-world scenarios . To address the challenges caused by misaligned images , Cycle-consistency was developed which was based on the assumption that the generator G from the source domain X to the target domain Y ( G : X → Y ) was the reverse of the generator F from Y to X ( F : Y → X ) . Compared to the Pix2Pix mode , the Cycle-consistency mode works better on misaligned or unpaired images . min G min F LCyc ( G , F ) = Ex [ ∥F ( G ( x ) ) − x∥1 ] + Ey [ ∥G ( F ( y ) ) − y∥1 ] ( 3 ) The Cycle-consistency mode , however , has its limitations . In the field of medical image-to-image translation , it requires not only the style translation between image domains , but also the translation between specific pair of images . The optimal solution should be unique . For example , the translated images should maintain the anatomical features of the original images as much as possible . It is known that the Cycle-consistency mode may produce multiple solutions [ 4 , 5 ] , meaning that the training process may be relatively perturbing and the results may not be accurate . The pix2pix mode is not ideal either . Even though it has a unique solution , it is difficult to satisfy the requirement asking for well aligned paired images . With misaligned images , the errors are propagated through the Pix2Pix mode which may result in unreasonable displacements on the final translated images . As of today , there is no image-to-image translation mode that can outperform both the Pix2Pix mode on aligned data and the Cycle-consistency mode on misaligned or unpaired data . Inspired by [ 6–10 ] , we consider the misaligned target images as noisy labels , which means that the existing problem is regarded as supervised learning with noisy labels . So we introduce a new image-to-image translation mode called RegGAN . Figure 1 provides a comparison of the three modes : Pix2Pix , Cycle-consistency and RegGAN . To facilitate reading , we summarize our contributions as follows . • We demonstrate the feasibility of RegGAN from the theoretical perspective of `` losscorrection '' . Specifically , we train the generator using an additional registration network to fit the misaligned noise distribution adaptively , with the goal to search for the common optimal solution for both image-to-image translation and registration tasks . • RegGAN eliminates the requirement for well aligned paired images and searches unique solution in training process . Based on our results , RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistencyăon misaligned or unpaired data . • RegGAN can be integrated into other methods without changing the original network architecture . Compared to Cycle-consistency with two generators and discriminators , RegGAN can provide better performance using less network parameters . 2 Related Work . Image-to-image Translation : Generative adversarial networks ( GANs ) have shown great potential in the field of image-to-image translation [ 11–16 ] . It has been successfully implemented in medical image analysis like segmentation [ 17 ] , registration [ 18 , 19 ] and dose calculation [ 20 ] . The existing modes , however , have their limitations . Specifically , the Pix2Pix mode [ 2 ] requires well aligned paired images which may not always be available . The Cycle-consistency mode can achieve unsupervised image-to-image translation . With a Cycle-consistency loss , it can be used for misaligned images . Based on Cycle-consistency , many methods [ 3 , 21–30 ] have been developed including CycleGAN [ 3 ] and its variants such as MUNIT [ 31 ] and UNIT [ 32 ] in which both image content and style information are used to decouple and reconstruct the image-to-image translation task ; U-gat-it [ 33 ] with a self-attention mechanism added ; and NICEGAN [ 34 ] proposed to reuse the discriminator for encoding . The main limitation of Cycle-consistency is that it may produce multiple solutions and therefore is sensitive to perturbation , making it difficult to meet the high accuracy requirements of medical image-to-image translation tasks . Learning From Noisy Labels : Neural network anti-noise training has made great progress . Current research are mainly focused on : estimating the noise transition matrix [ 7 , 35–40 ] , designing a robust loss function [ 41–44 ] , correcting the noise label [ 45–50 ] , sampling importance weighting [ 51– 55 ] and meta-learning [ 56–59 ] . Our work is in the category of estimating the noise transition matrix . Compared to conventional noise transition estimation , we mitigate the issue and simplify the task by acquiring prior knowledge of noise distribution . Deformable Registration : Traditional image registration methods have gained widespread acceptance , such as Demons [ 60 ] , B-spline [ 61 ] and elastic deformation model [ 62 ] . One of the most popular deep learning methods is Voxelmorph [ 63 ] . In this study , a CNN model was trained to predict the deformable vector filed ( DVF ) . The time-consuming gradient derivation process was thus skipped to improve the calculation efficiency . Affine registration and a vector momentum-parameterized stationary velocity field ( vSVF ) [ 64 ] was implemented to get better transformation regulation . Fast Symmetric method [ 65 ] used symmetric maximum similarity . Deep flash [ 66 ] outperformed other models in terms of training and calculation time . Closest to our work , Arar.M et al [ 67 ] introduced a multi-modal registration method for natural images based on geometry preserving . But their work focused only on registration and did not demonstrate results of image-to-image translation or discuss the relationship between registration and image-to-image translation . The key insight of our work is that we demonstrated the feasibility of using registration to significantly improve the performance of image-to-image translation because the noise could be eliminated adaptively during the joint training process . What we propose in the paper is a completely new mode for medical image-to-image translation . 3 Methodology . 3.1 Theoretical Motivation . If we considerămisaligned target images as noisy labels , ăthe training for image-to-image translation becomes aăsupervised learning processăwith noisy labels . Given a training dataset { ( xn , ỹn ) } Nn=1 with N noisy labels in which xn , ỹn are images from two modalities and assume yn is the correct label for xn , but it is unknown in real-world scenarios . Our goal is to train a generator using the dataset { ( xn , ỹn ) } Nn=1 with noisy labels and achieve the performance equivalent to trained on clean dataset { ( xn , yn ) } Nn=1 as much as possible . Direct optimization based on Equations 4 usually does not work and can lead to bad results because the generator can not squeeze out the influence of noise . Ĝ = argmin G 1 N N∑ n=1 L ( G ( xn ) , ỹn ) ( 4 ) To address the noise issue , we propose a solution based on `` loss-correction '' [ 7 ] shown in Equations 5 . Our solution corrects the output of the generator G ( xn ) by modeling a noise transition ϕ to match the noise distribution . Previously , Patrini et al [ 7 ] proved mathematically that the model trained with the noisy labels could be equivalent to the model trained with the clean labels , if the noise transition ϕ matches the noise distribution . Ĝ = argmin G 1 N N∑ n=1 L ( ϕ ◦G ( xn ) , ỹn ) ( 5 ) To achieve this , Goldberger et al [ 36 ] proposed to view the correct label as a latent random variable and explicitly model the label noise as a part of the network architecture , denoted by R. Then , Equations 5 can be rewritten in the form of log-likelihood , which is used as the loss function for neural network training . L ( G , R ) = − N∑ n=1 log ( p ( ỹn|yn ; R ) p ( yn|xn ; G ) ) = − N∑ n=1 log ( p ( ỹn|xn ; G , R ) ) ( 6 ) 3.2 RegGAN . Compared to existing methods that use expectation-maximum [ 7 , 36 ] , fully connected layers [ 35 ] , anchor point estimate [ 37 ] and Drichlet-distribution [ 38 ] to solve Equations 6 . In our problem , the type of noise distribution is clearer , it can be expressed as displacement error : ỹ = y ◦ T . Here T is expressed as a random deformation field , which produces random displacement for each pixel . So we adopt a registration network R after the generator G as label noise model to correct the results . The Correction loss is shown Equations 7 : min G , R LCorr ( G , R ) = Ex , ỹ [ ∥ỹ −G ( x ) ◦R ( G ( x ) , ỹ ) ∥1 ] ( 7 ) where , R ( G ( x ) , ỹ ) is the deformation field and ◦ represents the resamples operation . The registration network is based on U-Net [ 68 ] . A smoothness loss [ 63 ] is defined in Equations 8 to evaluate the smoothness of the deformation field and minimize the gradient of the deformation field . min R LSmooth ( R ) = Ex , ỹ [ ∥∇R ( G ( x ) , ỹ ) ∥2 ] ( 8 ) Finally , we add the Aversarial loss between the generator and the discriminator ( Equations 1 ) , and the total loss is expressed in Equations 9. min G , R max D LTotal ( G , R , D ) = LCorr + LSmooth + LAdv ( 9 ) 4 Experiments . Performance evaluation of RegGAN was conducted through three investigations to 1 ) demonstrate the feasibility and superiority of the RegGAN mode in various methods , and 2 ) assess RegGANs sensitivity to noise , and 3 ) explore the availability of the RegGAN on unpaired data . 4.1 Dataset . The open-access dataset ( BraTS 2018 [ 69 ] ) was used to evaluate the proposed RegGAN mode . The training dataset and testing dataset contained 8457 and 979 pairs of T1 and T2 MR images , respectively . BraTS 2018 was selected because the original images were paired and well aligned . We created misaligned images by randomly adding different levels of rotation , translation and rescaling to the original images . And we randomly sample one image from T1 and the other one from T2 when training on unpaired images . The availability of well aligned paired images , misaligned paired images , and unpaired images allow us to evaluate the performances of all three modes ( Pix2Pix , Cycle-consistency and RegGAN ) .

Summary: The authors proposed a novel method for conducting image-to-image translation integrating registration network (RegGAN), and demonstrated the superior performance in T1 to T2 MRI image translation to the current methods based on Pix2Pix and cycle-consistency techniques. The RegGAN performed slightly better than Pix2Pix method even without added noise likely due to inherent misalignment from the image acquisition. While the Pix2Pix method deteriorates rapidly with the added noise, RegGAN was robust against noise similarly to CycleGAN (with better performance). The authors also experimented to train them with unpaired images, and RegGAN could generate T2 images more similar to the actual images than the other two methods.