Breaking the Dilemma of Medical Image-to-image Translation
Authors: Lingke Kong
Chenyu Lian
Detian Huang
ZhenJiang Li
Yanle Hu
Qichao Zhou
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Image-to-image Translation

Abstract: Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of "loss-correction". In RegGAN, the misaligned target images are considered as noisy labels and the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively. The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available. Code and dataset are available at https://github.com/Kid-Liet/Reg-GAN.

Breaking the Dilemma of Medical Image-to-inage Translation
Lingke Kong" Manteia Tech
Chenyu Lian" Xiamen University cylianestu. xmu, edu. cn
kongllingee@@@arteiitttee.c
Detian Huang
Zhenjiang Li Shandong University
Yanle Hu* Mayo Clinic Arizona Hu. Yanle@mayo edu
Huaqiao University
huangdetian@hqu. edu. cn
zhenj1i1987011..coom
Qichao Zhou Manteia Tech
zhouqc@mant www.tch.ccnttcccoooooooooo
Abstract
Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dom- inate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But requires paired and well pixel-wise aligned images, which may always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical toageeto-image translation. It is based on the theory of "loss-correction" images are considered noisy labelsåand the generator is trained with an addi- tional registration network to fit the misaligned noise distribution adaptively. The goal is search for the common optimal solution to both amage-to-iman transla- tion and registration tasks. We incorporated RegGAN into a few state-of-the-ard image-to-image translation methods and demonstrated that RegGAN could be eas- ily combined with these methods to improve their performances. Such as a simple CycleGAN in mode surpasses latest NICEGAN even though using less net- work parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency insensitive to noises which makes a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available. Code and data used in this study can found at https://github.cm Let//Kid-Liiee RRelett/eeeGN...
RegGAN, the misaligned target
misaligned unpaired data. RegGAN is
1 Introduction
Generative adversarial networks (G) is a framework that simultaneously trains generator G and discriminator through an adversarial process. The generator is used translate the distribu-
"Equal contribution.
"Corespooniig author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

tion of source domain images X to the distribution of target domain images Y. The discriminator is used determine if the target domain images are likely from the generator or from the real data.
min max Adv (G, B, [log (D (y)) + Ex [log (1 D (G (G (z)))
(1)
G
Supervised [ix and unsupervised Cycle-consistency/l in GANs. Pix2Pix updates the generator (G X + Y) by minimizing pixel-level L1 loss between the source image æ and the target image y. Therefore, requires well aligned paired images, where each pixel has a corresponding label.
are the two commonly used modes
G CLi -G)=E,., [lly G (x) [h]]
Well aligned paired images, however, are not always available in real-world scenarios. To address the challenges caused by misaligned images, Cycle-consistency was developed which was based on the assumption that the generator G from the source domain X to the target domain Y (G + Y) was the reverse of the generator F from Y to X (F: Y + X). Compared to the Pix2Pix mode, the Cycle-consistency mode works better on misaligned or unpaired images.
min min LCye F) E. [II (x))
==|1] + By [[I (F (v)) ylla]
(3)
The Cycle-consistency mode, however, has its limitations. In the field of medical image-to-image translation, requires not only the style translation between image domains, but also the translation between specific pair of images. The optimal solution should be unique. For example, the translated images should maintain the anatomical features of the original images as much as possible. It is known that the Cycle-consistency mode may produce multiple solutions[[L D]. meaning that the training process may be relatively perturbing and the results may not be accurate. The pix2pix mode is not ideal either. Even though it has unique solution, it is difficult to satisfy the requirement asking for well aligned paired images. With misaligned images, the errors are propagated through the Pix2Pix mode which may result in unreasonable displacements on the final translated images.
As of today, there is no image-to-image translation mode that can outperform both the Pix2Pix mode aligned data and the Cycle-consistency mode misaligned or unpaired data. Inspired by[ū-II, we consider the misaligned target images as noisy labels, which means that the existing problem is regarded as supervised learning with noisy labels. So introduce new image-to-ing to-image translation mode called RegGAN. Figure D provides a comparison of the three modes: Pix2Pix, Cycle-consistency and RegGAN. To facilitate reading, summarize our contributions as follows.
We demonstrate the feasibility of RegGAN from the theoretical perspective of "loss- correction". Specifically, we train the generator using additional registration network to fit the misaligned noise distribution adaptively, with the goal to search for the common optimal solution for both image-to-image translation and registration tasks.
RegGAN eliminates the requirement for well aligned paired images and searches unique solution in training process. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and sycle-consisteeeyEon misaligned or unpaired data.
RegGAN can be integrated into other methods without changing the original network archi- tecture. Compared to consisteent with two generators and discriminators, RegGAN can provide better performance using less network parameters.
2 Related Work
Image-to-image Translation: Generative adversarial networks (GANs) have shown great poten- tial in the field of image-to-image rraslation[III-IIS It has been successfully implemented in medical image analysis like segmentation[117] registration[IIL [1]] and dose calculation The existing modes, however, have their limitations. Specifically, the Pix2Pix mode[E] requires well aligned paired images which may not always be available. The Cycle-consistency mode can achieve unsupervised image-to-image translation. With a Cycle-consistency loss, can used for mis- aligned images. Based on Cycle-consistency, many [B,, 21--30] have been developed in- cluding CycleGAN[[3] and its variants such as MUNIT[3I]] and UNIT[62] in which both image content and style information are used to decouple and reconstruct the image-to-image translation

---

Conettion loss
Adv loss
H(Gx))
Ad loss
y
R(G(x),))
Ade
Dj
F
D
F
R
D
Oyde www.sisaoncgggggooo loss
Cycle-onssee Cydb-onsddrrr YOSS
((x)
y
G(x)
F(y)
G(x)
loss
G
G
D
G
G
X
Adv loss
G(F(F))
X
(a) Pix2Pix
(b) Cyde-conssstenny
(c) RegGAN
Figure 1: Comparison among the modes of Pix2Pix, CycleGAN and RegGAN
task; U-g with a self-attention mechanism added; and NICEGAN proposed to reuse the discriminator for encoding. The main limitation of Cycle-consistency is that may produce multi- ple solutions and therefore sensitive to perturbation, making it difficult to meet the high accuracy requirements of medical image-to-image translation tasks.
Learning From Noisy Labels: Neural network anti-noise training has made great progress. Cur- rent research are mainly focused on: estimating the noise transition matrix [7A 35--00) designing robust loss function [ -.I, correcting the noise label[45--501, sampling importance weighting(SII-
55] and meta-learning (wriiinnn(5655555 (56-59). Our work is in the category of estimating the noise transition matrix. Compared to conventional noise transition estimation, we mitigate the issue and simplify the task by acquiring prior knowledge of noise distribution.
Deformable Registration:
Traditional image registration methods have gained widespread accep- tance, such as [60]] B-spline and elastic deformation model[62]]. One of the most popu- lar deep learning methods Voxelmorph|66))
this study, a CNN model was trained to predict the gradient derivation process was thus skipped to improve the calculation efficiency. Affine registration and vector stationary velocity field (ySVF)[6]] was implemented to get better transformation regulation. Fast Symmetric method[b3) used symmetric maximum similarity. Deep flash[[6f] outperformed other models in terms of training and calculation time.
deformable vector filed (DVF). The
Closest our work, Arar.M al[6?]] introduced multi-modal registration method for natural images based on geometry preserving. But their work focused only on registration and did not demonstrate results of mrage-to-image translation or discuss the relationship between registration and image-to-ine to-image translation. The key insight of our work is that we demonstrated the feasibility of using registration to significantly improve the performance of image-to-inee to-mgee translation because the noise could be eliminated adaptively during the joint training process. What we propose in the paper a completely new mode for medical image-to-image translation.
Methodology
3.1 Theoretical Motivation
we considerämissligned target images as noisy labels,äthe training for image-to-image translation becomes aäsupervised learning processäwith noisy labels. Given a training dataset { Zn " with N noisy labels in which Fn. Vn are images from modalities and assume yn the correct label for In, but it is unknown in real-world scenarios. Our goal is to train generator using the dataset (Z Th. N with noisy labels and achieve the performance equivalent to trained on clean

---

dataset (z Mrs) |n= much as possible. Direct optimization based on Equations A usually does work and can lead bad results because the generator cannot squeeze out the influence of noise.
G argmin G N Σ (xn)
To address the noise issue, we propose a solution based on "loss-correction [[Z]] shown in Equations Our solution corrects the output the generator G(x,,) by modeling a noise transition the noise distribution. Previously, Patrini et al[[ proved mathematically that the model trained with the noisy labels could be equivalent to the model trained with the clean labels, if the noise transition o matches the noise distribution.
"[Z]
match
N Σ
1
G arg min
o (zo), (o.
(5)
To achieve this, Goldberger et al G| proposed to view the correct label as a latent random variable and explicitly model the label noise a part of the network architecture, denoted by R. Then, Equations B can be rewritten in the form of log-likelihood, which is used as the loss function for neural network training.
C (G, R) =
Σ log (p (Umlvn: R) p In G))
(6)
log (p (#mlomi G, R))
3.2 RegGAN
Compared to existing methods that use pnnectationnmxximmm/20 [6], fully connected layers[[S], anchor point estimate[[2]] and Drichlet-distribttion(tY to solve Equations A In our problem, the type of noise distribution clearer, can be expressed as displacement error: ğ = y Here T expressed random deformation field, which produces random displacement for each pixel. So we adopt a registration network R after the generator G as label noise model to correct the results. The Correction loss is shown Equations 7
min CCorr (G, R) F., -lū - G o (G (x), lha] G.Rt
where, R (G (at), 77 is the deformation field and represents the resamples operation. The registra- tion network is based on U-Net[68]. smoothness loss[63] is defined in Equations evaluate the smoothness of the deformation field and minimize the gradient of the deformation field.
min Csmooth (R) Ez.ÿ (I (x) [
(8)
It
Finally, we add the Aversarial loss between the generator and the discriminator (Equations M. and the total loss expressed in Equations
min max Drotal (,, Loorr + Lsmooth + Lade
G.R
4 Experiments
Performance evaluation of RegGAN was conducted through three investigations to 1) demonstrate the feasibility and superiority of the RegGAN mode in various methods, and 2) assess RegGANs sensitivity to noise, and 3) explore the availability of the RegGAN on unpaired data.
4.1 Dataset
The open-access dataset (BraTS 2018[69]) was used to evaluate the proposed RegGAN mode. The training dataset and testing dataset contained 8457 and 979 pairs of T1 and T2 MR images, respec- tively. BraTS 2018 was selected because the original images were paired and well aligned. We

---

created misaligned images by randomly adding different levels of rotation, translation and rescaling the original images. And we randomly sample one image from T1 and the other one from T2 when training unpaired images. The availability of well aligned paired images, misaligned paired images, and unpaired images allow us evaluate the performances of all three modes (Pix2Pix, Cycle-consistency and RegGAN).
4.2 Performances in Different Methods
The primary motivation of introducing RegGAN was address challenges caused by misaligned data. Therefore, in this section, misaligned data were used in model training to demonstrate the feasibility and superiority of RegGAN. We selected the most popular CycleGAN[3]] and its vari- ants MU]] ( and NICEGAN [32] as the methods for evaluation and compared the following four modes for each method.
C(Cycle-consistencyy The most primitive mode of all methods, with Cycle-consistency loss (Equations () as the main constraint. Two generators and two discriminators are required in this mode.
C+R (Cycle-consistency- + Registration): The RegGAN mode is combined with the mode C. Registration network (R) and Correction loss (Equations M are added to the constraints.
NC(Non cycle-consistency): Only Adversarial loss (Equations ( is used for updating. Compared the mode C, Cycle-consistency loss removed. Only one generator and one discriminator are required in this mode.
NC+R(Non Cycle-consistency + Registration): registration network (R) and Correc- tion loss (Equations (1) are added to the mode NC. It the proposed RegGAN mode.
Table 1: Comparison CycleGAN, MUNIT, UNIT and NICEGAN using four training modes(C, C+R, NC and NC+R).
Modes
Methods
CycleGAN
MUNIT
UNIT
NICEGAN
Index
C
0.089
0.11
0.087
0.082
C+R
(-0.012)0.077
(-0.022)0.088
(-0.013)0.074
(-0.011)0.071
NMAE +
NC
0.11
0.10
0.098
0.089
NC+R
(-0.038)0.072
10.021)0.079
(-0.027)0.071
(-0.019)0.070
C
23.5
20.6
24.6
25.2
C+R
(+0.3)23.8
(+2.1)22.7
(+0.7)25.3
(+0.9)26.1
PSNR
NC
20.2
21.5
23.7
23.5
NC+R
(+5.4)25.66
(+2.3)23.8
(+ (-2.5)
(+2.8)26.3
C
0.83
0.80
0.84
0.83
C+R
(+0.02)0.85
(+0.03) 0.83
(+0.02)0.86
(+0,03)0.86
SSIM T
NC
0.79
0.81
0.83
0.84
NC+R
(+0.07)0.86
(+0.04)0.85
(+0.03) 0.86
(+0.02)0.86
To evaluate the performance of each method misaligned data, randomly added [-5, +5] degrees of angle rotation, [-5, +5] percent of translation, and [-5, +5] percent rescaling to the original and T2 images on the training dataset.
To ensure fair comparison, we used the same training strategy and hyperparameters for all meth- ods and modes (see supplementary materials for details). The Normalized Mean Absolute Error (NMAE), Peak Signal Noise Ratio (PSNR) and Structural Similarity (SSIM) were used as met- rics to evaluate the performances of trained models based the testing dataset. To avoid false high results of index, we excluded the image background from the calculation. Table summarized the results for all methods and modes under the current investigation.
5

---

CydleGAN
Exror map
MUNIT
Error map
UNIT
Exror map
NICEGAN
Error map
T2
NC
Cydisaan
Error map
MUNTT
Ervor map
UNIT
Error map
NICEGAN
Exror map
T2
NC
NC
0
Figure 2: The errors of different modes in different methods.
Based on the results from the Table M. we can reach several conclusions. First, adding the reg- istration network (+R) significantly improves the performances of the methods. This is true for all methods in both C and NC modes. It clearly demonstrates that RegGAN can be incorporated various methods combined with different network architectures to improve the performances. Second, the C mode is in general better than the NC mode for most of methods. Adding the reg- istration network (+R) improves the performance of the NC mode more than that of the C mode. In fact, our results show that the NC+R mode is even better than the C+R mode, implying that "Cycle-consistency loss" may play negative role when with the commonly used C mode with two generators and two discriminators, RegGAN has fewer parameters but provides better performance. The simple CycleGAN method in the NC+R mode outperforms the current state-of-the-art method NICEGAN in the C mode by 0.01, 0.4, 0.03 for NMAE, PSNR and SSIM, respectively. The NC+R mode can also used to improve the perfor- mance of NICEGAN, In fact, the performance of NICEGAN in the NC+R mode is the best among all combinations of the methods and modes.
combined with RegGAN. Compared
Figure D shows representative results from various combinations of the 4 methods (CycleGAN, MUNIT, UNIT and NICEGAN) and 4 modes (C, C+R, NC and NC+R). For all aspects of image (from the tumor areas and the details), the combinations that use the registration network (+R) always provide more realistic and accurate results than those that do not use the registration network (+R).

---

Noloc
Natel1
Ni0o2
Nis3
Nited
No0e
NateNA
Figure 3: Example images at seven different levels of introduced noise.
Table 2: Comparison of the NMAE, PSNR and SSIM for CycleGAN(C), Pix2Pix and RegGAN under levels noise.
Noise.0
Noise.
Noise.2
Noise.3
Noise.4
Noise.5
Noise.NA
Rotate
0
=
2
3
#
X
Setting
Translation
0%
+2%
+4%
16%
18%
100
x
Rescaling
0%
+2%
+.4%
+6%
+8%
+10%
X
NMAE +
0.084
0.095
0.087
0.094
0.087
0.110
0.091
CycleGAN(C)
PSNR t
23.9
22.5
23.7
23.3
23.9
23.7
23.5
SSIM f
0.83
0.83
0.82
0.81
0.82
0.79
0.83
NMAE +
0.075
0.103
0.139
0.161
0.175
0.181
0.086
Pix2Pix
PSNR 1
25.6
22.3
18.9
16.2
15.3
15.0
21.1
SSIM
0.85
0.82
0.78
0.76
0.74
0.74
0.82
NMAE 4
0.071
0.073
0.071
0.072
0.072
0.072
0.071
RegGAN
PSNR 1
26
25.6
25.9
25.7
25.4
25.2
25.9
SSIM
0.86
0.86
0.86
0.86
0.86
0.85
0.86
4.3 Performances in Different Noise Levels
To evaluate the sensitivity of RegGAN noise, we selected simple network architec- (CycleGCy)) with the intention to minimize interference from other factors. The same net- work architecture was used for all three modes: CycleGAN(C),, Pix2Pix and RegGAN, Seven levels of noise were used in the evaluation. Table E lists the specific noise setting and range for each noise level. Noise.0 means the original dataset with no added noise. Noise.5 is the highest level of noise. At 5oise.5, the data likely from different patients. In addition, noise(Noise.NA) settings. The non-affine noise is generated by spatially transforming TI and T2 using clastic transformations control points followed by Gaussian smoothing. Figure shows example images at different levels introduced noise.
also made non-affine
Table A lists the quantitative evaluation metrics from 3 modes at 7 levels of noise. It is clear that RegGAN outperforms CycleGAN(C) under all noise levels. Figure Ma) shows the test results from each epoch during the training process for both RegGAN and CycleGAN(C). Curves of different colors corresponds to different levels of noise. We notice that CycleGAN(C) is not very stable during the training process. The test results fluctuate significantly and cannot converge well. This may be caused by the fact that the solution of CycleGAN (C) is not unique. As a comparison, RegGAN is quite stable. Although the results from different levels of noise may vary at the beginning of training, all curves converge to a similar result after multiple epoches of training, indicating that RegGAN is more robust to noise compared to CycleGAN (C).
Based on Table we notice that the performance of Pix2Pix deteriorates rapidly as the noise in- creases. This is as expected because Pix2Pix requires well aligned paired images. Surprisingly. the performances of RegGAN at all noise levels exceed those of Pix2Pix with no noise. Figure (lb) shows the test results at each epoch of RegGAN and Pix2Pix under Noise.0 (i.e., no noise). Theo- retically, the performances RegGAN and Pix2Pix should be similar on perfectly aligned paired datasets because the registration network of RegGAN does not help and RegGAN is equivalent to
7

---

(b)
(4)
0.5
Falles Rogolo
Netch
0.19
0.922
0.30
3.00
0.15
MAAE
SM00TH
0.13
8390
8.10
437-
EPOCH
EPOCH
EPOCH
Figure 4: Quantitative evaluation metrics at different epochs in the training process. (a) Comparison of CycleGAN and RegGAN different levels of noise. (b) Comparison of Pix2Pix and RegGAN N (i.e., no noise). (c) RegGAN's Smoothness loss under different levels noise.
Case. 1
Case. .2
Case
Case
Case
Case .6
D.F
T2
Figure 5: The misalignment of orginal image pairs and corresponding deformation fields.
Pix2Pix. A possible explanation to our results is that in the medical field, the perfectly pixel-wise aligned dataset may not practically exist. Even for BraTS 2018[63]] which recognized as well aligned, it is still possible that there exists slight misalignment. As result, adding the registra- tion network is always likely to improve the performances in real-world scenarios. To verify our explanation, we plotted the Smoothness loss of RegGAN under different noise levels as shown in Figure BKc). Large Smoothness loss corresponds to large deformation field displacement. First, we notice that the Smoothness loss under Noise.0 never completely goes to O, indicating the existence of misalignment and potential usefulness the registration network. Second, the noise level and Smoothness loss show step-like positive correlation, which means that RegGAN can adaptively handle the noise distribution, i.c., the registration network can determine the range of deformation according to the noise level. In addition, we can see that even under the setting of non-affine noise, the above conclusion still holds. Because what the registration network corrects is deformation noise.
we also show some original image pairs and visualize the corresponding deformation fields output by registration network in Figure & Obviously, there is some misalignment between the original T1 and T2 images, and such misalignment is represented by the deformation fields (highlighted by red circle).

---

T1
12
Ph.0%s
LYdGAND
Fimm
Index
NMAEJ PSNR SSIM
Mode
Pix2Pix
0.180
15.5
0.71
CycleGAN(C)
0.094
23.6
0.83
RegGAN
0.086
24.0
0.83
Figure 6: Performance comparison of the three modes (CycleGAN(C), Pix2Pix and RegGAN) unpaired dataset.
12
Tramshted
Registered
DXP
T1
12
Tramsiaedd
Registered
Figure 7: Display of RegGAN's output unpaired data. T1 and T2 are unpaired images. The Translated represents the translation result of TI to T2. Registered represents the registration result of the translated images. D.F represents deformation fields.
4.4 Performances on Unpaired Dataset
So far, our investigations are based on paired datasets. We also want to explore how RegGAN per- forms using unpaired datasets. In practice, this is not recommended because different patients may have similarities in their body tissues of adjacent layers. For unpaired datasets, we can conduct rigid registration first in 3D space and then use RegGAN for training. Unpaired data can be treated as having larger scale noise. If the correction capability is strong enough, RegGAN can still work effectively. The comparison the performances of three modes on the unpaired dataset is shown in Figure K
With unpaired datasets, Pix2Pix longer considers the characteristics of the input TI images and thus has the worst performance. Due to the challenges in fitting the noise, the performance improve- from replacing CycleGAN(C) with RegGAN using unpaired datasets may be dramatic as that demonstrated using paired datasets, but RegGAN still has the best performance under unpaired conditions. In Figure we show some examples of how RegGAN corrects noise on unpaired dataset. It can be seen that RegGAN will try its best to eliminate the influnce of noise through registration.
Based on our results, it is reasonable to reach the conclusions below. In all circumstances, RegGAN demonstrates better performance compared to Pix2Pix and CycleGAN(C).
For paired and aligned conditions, RegGAN > Pix2Pix CycleGAN(C).
For paired but misaligned conditions, RegGAN > SycleGAN())>Piii>0
For unpaired conditions, RegGAN CycleGAN(C) Pix2Pix.

---

Conclusion
In this study, we introduced new image-to-image translation mode RegGAN the medical com- munity that can break the dilemma of mage-to-imags translation task. Using a public BraTS 2018 dataset, we demonstrated the feasibility of RegGAN and its superior performance compared to Pix2Pix and Cycle-consistency. We validated that RegGAN can be incorporated into various ex- isting methods to improve their performances. We also evaluated the sensitivity of RegGAN to noise. Our results confirmed that RegGAN could adapt well to various scenarios from no noise to large-scale noise. The superior performance of RegGAN makes better choice over Pix2Pix and Cycle-consistency whether datasets aligned not. However, this mode may not work well on natural images. The noise may cannot be considered simply as deformation errors due to the differences in natural images much greater than those in medical images.
Broader Impact
Image-to-imant translation has been one of the main focuses in medical image analysis, as it aids in diagnosis and treatment. Previously, physicians had to different medical imaging equipments if they wanted to get different image sequences of patient, which was time-consuming and expensive. Pix2Pix mode expected solve this problem by its outstanding performance in image-to-image translation. In most of clinical scenarios, however, it is not practical to create such a large well aligned dataset for Pix2Pix mode. Cycle-consistence mode does need well aligned dataset but can not meet the high-precision requirements of medical image analysis. Our work aims to provide a general image-to-iman
translation mode, which not only has no strict requirements on the dataset, but also can meet the clinical requirements in terms of image quality. In the future, we will attempt obtain multi-modal dataset(eg MR-CT) for clinical verification. We foresee positive impacts the mode is applied to diagnosis in radiology, treatment planning and research.
10

---

References
[1] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C[//Gaarrmmaii 2, Welling M, Cortes C, al. Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014.
[2] Isola P. Zhu J Y. Zhou T, et al. Image-to-image translation with conditional adversarial net- Works|CC/Proceedingg of the IEEE conference on computer vision and pattern recognition.
2017: 1125-1134.
[3] Zhu J Y, Park T, Isola P. al. Unpaired image-to-image translation using cycle-consistent adversarial (e//orrss of the IEEE international conference on computer vision. 2017: 2223-2232.
[4] Sim B, Oh G, Lim S. et al. Optimal transport, cyclegan, and penalized for unsupervised learning in inverse problems(J]. 2019.
[5] Moriakov Adler Teuwen J. Kernel of cyclegan as a principal homogeneous space[C]// International Conference on Learning Representations. 2020.
[6] Huang L. Zhang C, Zhang H. Self-adaptive training: beyond empirical risk minimization(]]. Advances in Neural Information Processing Systems, 2020, 33.
[7] Patrini G, Rozza A, Krishna Menon A, et al. Making deep neural networks robust to label noise: loss correction ApproachiC///Proeedding of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1944-1952.
[8] Dgani Y, Greenspan H. Goldberger Training a neural network based on unreliable human annotation of medical images[C]//2D IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, 2018: 39-42.
[9] Yao Y, Liu T, Han B, et Dual t: Reducing estimation error for transition matrix in label- noise Banningg//OOL//LLaoohooo H. Ranzato M, Hadsell R, et al. Advances in Neural Infor- mation Processing Systems: volume 33. Curran Associates, Inc., 2020: 7260-7271. https: Paper.pdf.
prroceddmgs.neumgg
[10] Zhang L, Tanno R, Xu M C, et al. Disentangling human error from ground truth in segmentation of medical Imaaess(/OL///Lrrchhee H, Ranzato M, Hadsell R.
Advances in Neural Information Processing Systems: volume 33. Cur- ran Associates, Inc., 2020: 1.750-15762.
b5d 7ed25502da
Saatlldddaaaaaaaaaaaaa 3aa727at0da 150865 Paper--a...pp...f Paper.pdf.
[1I] wang Yu L, van de Weijer J. Deepi2i: Enabling deep hierarchical image-to- image translation by transferring from cans///OL.//Larrcchee H, Ranzato M, Hadsell R,
et al. Advances in Neural Information Processing Systems: volume 33. Cur- ran Associates, Inc., 2020: I 11803-11815.
[12] Zhang P. Zhang B. Chen D, al. Cross-domain correspondence learning for exemplar-based image crunlattoonC///Pooceedngg of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition. 2020: 5143-5153.
[13] Kim Cha M, Kim al. Learning to discover cross-domain relations with generative adversarial (iwworks(C//Inteemmttinnll Conference on Machine Learning, PMLR, 2017: 1857- 1865.
[14] Cao Y, Wan X. Divgan: Towards diverse paraphrase generation via diversified generative adversarial networkkC///rroceding of the 2020 Conference Empirical Methods in Natural Language Processing: Findings. 2020: 241 2411-242..
[15] Choi Y, Choi M, Kim M, et al. Stargan: Unified generative adversarial networks for multi- domain image-to-image sion and pattern recognition. 2018: 8789-8797.
of the TEEE conference on computer vi-

---

[16] Choi Y, Uh Y, Yoo J, et al. Stargan v2: Diverse image synthesis for multiple domains [Cn]/ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8 8188-8197.
[17] Chen C, Dou Q. Chen H, et al. Synergistic image and feature adaptation: Towards cross- modality domain adaptation for medical image (ammennttionnCC//Poeeeeiii of the AAAI Conference on Artificial Intelligence: volume 33. 2019 865-872.
[18] Fan J, Cao X, Wang Q. et Adversarial learning for mono-or multi-modal registration(J]. Medical image analysis, 2019, 58:101545.
[19] Qin C, Shi B, Liao R, et al. Unsupervised deformable registration for multi-modal images via disentangled pureseenationscC//mmmmmatttttt Conference on Information Processing in Medical Imaging. Springer, 2019: 249-261.
[20] Sahiner B, Pezeshk A, Hadjiiski M. et al. Deep learning in medical imaging and radiation therapy[J]. Medical physics, 2019, 46(1):e1-e36.
[21] Hoffman J, Tzeng E, Park T, et al. Cycada: Cycle-consistent adversarial domain adaptation [CI/Annernational conference machine learning. PMLR, 2018: 1989-1998.
[22] Li M, Huang H. Ma L, et al. Unsupervised image-to-ing -mmiage translation with stacked cycle- consistent adversarial hetwoorks(CC//Proceeddng of the European conference on computer vi- sion (ECCV). 2018: 184-199.
[23] Deng W, Zheng I., Ye Q, et al. Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification (C//roccnn of the IEEE conference on computer vision and pattern recognition. 2018: 994-1003.
[24] Li J, Chen E, Ding Z. et al. Cycle-consistent conditional adversarial transfer networks(C\/ Proceedings of the 27th ACM International Conference on Multimedia. 2019: 747-755.
[25] Schmidt V, Luccioni A, Mukkavilli S K, et al. Visualizing the consequences of climate change using cycle-consistent adversarial networks(J]. arXiv preprint 1rXiv: 2019.
[26] Chen Z, J, Luo Y, et al. Canzsl: Cycle-consistent adversarial networks for zero-shot learning from natural of Computer Vision. 2020: 874-883.
of the IEEE/CVF Winter Conference on Applications
[27] Ren C X, Ziemann A, Theiler J, et al. Cycle-consistent adversarial networks for realistic pervasive change generation in remote sensing imagery[C]/2000 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI). IERE, 2020: 42-45.
[28] Liu J, Ding Y, Xiong J, et al. Multi-cycle-consissent adversarial networks for et image denois- ing/C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020: 614-618.
[29] Zheng C, Pan L, Wu P. Camu: Cycle-consistent adversarial mapping model for user alignment across social networks(J]. IEEE Transactions on Cybernetics, 2021.
[30] Shah M, Chen X, Rohrbach M, et al. Cycle-consistency for robust visual question answering [C]//Poocediigg of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 6649-6658.
[31] Huang Liu M Y. Belongie S, et al. Multimodal unsupervised image-to-image translation [C)//Poocediing of the European conference on computer vision (ECCV). 2018: 172-189.
[32] Liu Y, Breuel T, Kautz Unsupervised image-to-image translation networks(C/OL.j// Guyon I, Luxburg U V, Bengio S, et al. Advances in Neural Information Processing Sys- tems: volume 30. Curran Associates. Inc., 2017. http://ppoecddng....rrsssscopppp
12

---

of the IEEE/CVF International Conference on [33] Kim J, Kim M. Kang H, et al.
U-gat-it: Unsupervised generative attentional net- works with adaptive layer-instance normalization for image-to-image translation(//OL)/ International Conference on Learning Representations. 2020. https://openeeeeew.ee///ooun id=BIIZZySKPH,
[34] Chen R, Huang W, Huang B, et al. Reusing discriminators for encoding: Towards unsupervised image-to-image (atpllaaionnc////oeeeeiiii of the IEER/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8168-8177.
[35] Sukhbaatar P L. F, J. Bruna. Training convolutional networks with noisy labels[C)/ International Conference on Learning Representations. 2015.
[36] Goldberger J, Ben-Reuven E. Training deep 2016.
using a noise adaptation layer[J].
[37] Xia X. Liu T. Wang N, et al. noise |esrming/CCOOLL//NeetsS.. 8908-are 1088-re-aaae aure-ancor--oooo pocitr-ppooss--see rew.rrelllllllenaaaaeeeet laa-aablll-----a------------ loise-noose---------ooo neaarcinnns---aannnng http://ppprs..ps.c/ppppp
Are anchor points really indispensable in label- 2019: 6835-6846.
[38] Yao J, Wu H. Zhang Y, ct al. Safeguarded dynamic label regression for noisy supervision[C]/ Proceedings of the AAAI Conference on Artificial Intelligence: volume 33. 2019: 9103-9110.
[39] Xiao T, Xia I, Yang Y, al. Learning from massive noisy labeled data for image classification [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 2691-2699.
[40] Misra I, Lawrence Zitnick C, Mitchell M, et al. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels(C//Prooeeedngg of the IEEE Conference Computer Vision and Pattern Recognition. 2016: 2930-2939.
[41] Tanno R, Saeedi A, Sankaranarayanan S, al. Learning from noisy labels by regularized estimation of annotator caanfussoonC///Pooeeeeeee Vision and Pattern Recognition. 2019: 11244-11253.
the IEEE/CVF Conference on Computer
[42] Rodrigues F, Pereira F. Deep learning from crowdss[[//Proceeding of the AAAI Conference on Artificial Intelligence: volume 32. 2018.
[43] Branson S, Van Horn G, Perona P. Lean crowdsourcing: Combining humans and machines in an online System[C//Proceeinng of the IEEE Conference Computer Vision and Pattern Recognition. 2017: 7474-7483.
[44] Izadinia H, Russell C, Farhadi A, et al. Deep classifiers from image tags in the wild[C]/ Proceedings of the 2015 Workshop on Community-Organin Multimodal Mining: Opportu- nities for Novel Solutions. 2015: 13-18.
[45] Jaehwan L, Donggeun Y, Hyo-Eun K. Photometric transformer networks and label adjustment for breast density Computer Vision Workshops. 2019: 0-0.
[46] Veit A, Alldrin N, Chechik G, et al. Learning from noisy large-scale datasets with minimal su- pervision[C//Proceeee C1/Proceedingg of the IEEE conference on computer vision and pattern recognition. 2017: 839-847.
[47] Tanaka D. Ikami D, Yamasaki T, et al. Joint optimization framework for learning with noisy Sabels/C//Prooceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 5552-5560.
[48] Zheng S, Wu Goswami A, et al. Error-bounded correction of noisy labelss(C//nntemaational Conference on Machine Learning. PMLR, 2020: 11447-11457,
[49] Han J. Luo P. Wang X. Deep self-learning from noisy labels C/Peroccc/ss of the IERR/VV International Conference on Computer Vision. 2019: 5138-5147.
13

---

[50] Durand Т, Mehrasa N. Mori G. Learning deep convnet for multi-label classification with partial labels/C//Prooceedins of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 647-657.
[51] Wu X, He R, Sun Z, et al. A light enn for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13(1 (3(11))288-228%.
[52] Huang J, Qu I., Jia R, et al. O2u-net: A simple noisy label detection approach for deep neu- ral (rtwoorks[///roccediii of the IEEE/CVF International Conference on Computer Vision. 2019 3326-3334.
[53] Li J. Socher R. Hoi S C. Dividemix: Learning with noisy labels semi-supervised learning CC/OL\/Interaationall Conference on Learning Representations. 2020. https://openreveew.ndd
forum
[54] Yan Y, Xu 2, Tsang et al. Robust semi-supervised learning through label aggregation Proceedings of the AAAI Conference Artificial Intelligence: volume 30. 2016.
[55] Jiang J, Ja J. Wang Z, et al. Hyperspectral image classification in the presence of noisy labels [J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 57(2):851-865.
[56] Li J, Wong Y, Zhao Q et al. Learning to learn from noisy labeled data bata[[///Proceedingg of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 5051-5059.
[57] Li Y. Yang J, Song Y. et al. Learning from noisy labels with distillation((|//Poocceegg of the IEEE International Conference on Computer Vision. 2017: 1910-1918.
(58) Algan G, Ulusoy I. Meta soft label generation for noisy labels[C]//2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021: 7142-7148.
[59] Garcia L P. de Carvalho A C, Lorena A C. Noise detection in the meta-learning level[J]. Neurocomputing. 2016, 176:14-25.
[60] Thirion J P. Image matching diffusion process: analogy with maxwell's demons[J].
Medical image analysis, 1998, 2:243260.
[61] Rueckert D. Sonoda L I, Hayes C, al. Nonrigid registration using free-form deformations: application to breast mr images[J]. IEEE transactions on medical imaging, 1999, 18(8):712- 721.
[62] Shen D, Davatzikos C. Hammer: hierarchical attribute matching mechanism for elastic regis- tration[J]. IEEE transactions medical imaging, 2002, 21(11):1421-447
[63] Balakrishnan G, Zhao A, Sabuncu M R, et al. Voxelmorph: a learning framework for de- formable medical image registration(J]. IEEE transactions medical imaging, 2019, 38(8): 1788-1800.
[64] Shen Z, Han X, Xu Z, al. Networks for joint affine and non-parametric image registration [CI//Pooceeiings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019;
4224-4233.
[65] Mok C Chung A. Fast symmetric diffeomorphic image registration with convolutional neu- ral Crtworrs(C//Proceeedin of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 4644-4653.
[66] Wang Zhang M. Deepflash: An efficient network for learning- baaeing-based medical image registra- ciom[[//Proceedinns of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. 2020: 4444-4452.
[67] Arar M, Ginger Y, Danon D, et al. Unsupervised multi-modal image registration via geom- etry preserving image-to-image translaion(Cy/Prreedding the IEEE/CV conference on computer vision and pattern recognition. 2020: 13410-13419.
14

---

[68] Ronneberger O, Fischer P. Brox T. U-net: Convolutional networks for biomedical image seg- mentation(C//Intemaatioaalll Conference on Medical image computing and computer-assisted intervention. Springer, 2015: 234-241.
[69] Menze B H. Jakab A. Bauer S, et al. The multimodal brain tumor image segmentation bench- mark (brats)[J]. IEEE transactions on medical imaging, 2014, 34(10):1993-200
15


Reviews:
[{"review_id": "qz_CJQYCURO", "reviewer": "Reviewer_REoq", "summary": "The paper introduces a new method for image-to-image translation, which accommodates misalignment between training image pairs - a common problem particularly with medical images where motion or distortion often leads to misalignment that cannot be perfectly corrected through a-priori image registration.  The authors rightly state that this is a key missing component of existing image-to-image translation approaches such as pix2pix-type algorithms that assume matched pairs with perfect alignment, whereas cycle-consistency approaches do not exploit matched pairs when available.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am enthusiastic from the start, as this is a problem I have encountered and thought about, and have been waiting for a nice solution!  I'm amazed no-one has done this before actually, but I am not aware of prior work on this issue.\n\nThe implementation the authors propose is simple but sensible and the right place to start with solving this important problem.  They simply add estimation of a deformation field that is learned concurrently with the image-to-image translation during training, as well as a smoothing loss to keep the deformation under control.\n\nExperiments use the BRATS brain-MRI data set and add artificial misalignments to pairs of images with different MR contrast before learning image-to-image translations between those contrasts from the paired data set.  As expected, the proposed technique outperforms existing techniques that do not consider misalignment. The authors go on to show that the proposed technique also has benefit in situations where matched pairs are not available but simply unmatched examples of both contrasts even though the method is not designed for such situations.\n\nMy key criticism is that the experiments are rather artificial as the imposed misalignments are much larger than one would normally find between different MR contrasts.  Rotations scalings and displacements of the type the authors use to corrupt the alignment would be easily fixed by a-priori image registration. In practice the real problem arises from small residual misalignments after a-priori image registration - that tends to lead to blurring in estimated images as a consequence of learning from misaligned data sets. A fairer comparison with existing methods and more convincing demonstration of the benefits of the proposed technique would have come from registering the paired images prior to training, as that is what users of say pix2pix would naturally normally do.  We do get a hint of this in section 4.3 where the evaluation as a function of \"noise level\" (which here refers to the degree of alignment corruption) includes zero noise, ie no artificial misalignment but only residual misalignment after the registration step used in the original BRATS data.  The proposed technique does still show benefit in this \"zero noise\" test, which is encouraging, but I would have liked to see a more complete evaluation and comparison against the state of the art in this more realistic scenario.\n\nNevertheless, I remain enthusiastic about the proposed method and look forward to seeing more complete demonstrations of its capability as well as further refinement of the idea.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "Marked as N/A - the data is publicly available, so I guess that's fine.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "cR-T3HRIt6", "reviewer": "Reviewer_TPfs", "summary": "The authors proposed a novel method for conducting image-to-image translation integrating registration network (RegGAN), and demonstrated the superior performance in T1 to T2 MRI image translation to the current methods based on Pix2Pix and cycle-consistency techniques.\nThe RegGAN performed slightly better than Pix2Pix method even without added noise likely due to inherent misalignment from the image acquisition. While the Pix2Pix method deteriorates rapidly with the added noise, RegGAN was robust against noise similarly to CycleGAN (with better performance).\nThe authors also experimented to train them with unpaired images, and RegGAN could generate T2 images more similar to the actual images than the other two methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nWhile using deep learning network for medical image registration is not new (e.g. BIRNet), combining it with generator network for image-to-image translation and training it end-to-end is the original approach.\n\nQuality and clarity:\nThe paper is clearly written with enough details.\n\nSignificance:\nT1 and T2 MRI images are typically well aligned, and the performance gain in paired setting is not large. Still, the results were presented well to indicate the potential to be applied to other medical imaging tasks where distortion from motion artifacts are more evident.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper still have a few limitations.\nFirst, more traditional image (co)registration technique combined with Pix2Pix technique hasn't been explored. Especially in paired setting with small or zero noise, Pix2Pix followed by rigid alignment may perform similarly to RegGAN. While registering T2 images to T1 images can be challenging in some cases, another approach would be register both T1 and T2 images to reference T1 and T2 images, respectively at first and train Pix2Pix translation between the registered images. I still think the end-to-end approach of the authors have benefits over these in terms of simplicity, it would be nice to see such comparisons.\nSecond, the performance metric used is mostly based on image-level, and doesn't reflect actual utility in medical diagnosis. Slight improvement in aligning the normal tissues can improve these metrics, while not improving the clinical utility. Potentially, even with the higher score, the translated images may completely miss important abnormalities which is normally highlighted in T2 images (e.g. high: ischemic stroke, low: hemorrhage). This can be in particular problematic in unpaired settings. At least some examples should be included.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "T2H039vVom", "reviewer": "Reviewer_sQmz", "summary": "The authors present a new medical image-to-image generative model, called RegGAN, that combines an adversarially trained image generator with a registration module that enforces alignment between the generated image and ‘noisy’, (un)paired ground truth image. The aim of the network is to search for the single optimal solution to both image-to-image translation and registration tasks. The authors compared their proposed method against two other general model types, Pix2Pix (paired aligned data) and Cycle GAN (unpaired or misaligned). For training and testing, they use the BRATS dataset, which consists of brain tumor MRIs, and translate T1 to T2 images. The authors were able to show better performance on all major metrics for both paired and unpaired data. They showed that RegGAN was more resistant to label noise and increased stability during training.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: an original combination of a registration loss to a GAN objective in a medical image setting. \nQuality and Clarity: good quality, the paper is well written and the figures are excellent. \nSignificance: Good contribution to a medical imaging problem that needs to be addressed. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have restricted the noise to affine deformations only. RegGAN is capable of doing non-affine noise correction and these deformations may be encountered in clinical medicine (e.g. 1.5 vs 3 Tesla MRIs). The authors are encouraged to perform these experiments and add them to Table 2. \n\nThe current study is limited to a single image-to-image translation task: T1 to T2. While I understand that the authors are limited by the availability of open-source datasets, this task is highly artificial and generally would not occur in clinical medicine. It seems a better test, especially for demonstrating superior performance on unpaired or misaligned datasets, would be cross-modality, such as CT to MRI or vice versa. The authors state ‘Previously, physicians had to use different medical imaging equipment if they wanted to get multi-modal images of a patient, which was time-consuming and expensive... Our work aims to..meet the clinical requirements in terms of image quality’. No multi-modal testing is shown in the paper. I would either attempt to obtain a CT-MRI dataset (this would strengthen the impact of the results), or remove the previous statement.\n\nLine 235: BraST > BraTS\n\nOverall, good paper and a thoughtful contribution. \n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "3VmRd9t2KQN", "reviewer": "Reviewer_yp9h", "summary": "This paper considers the setting where paired images are available but some of them are misaligned. To address this issue, it proposes to add a registration network after the translation network as the \"correction loss\".", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n1. The setting is interesting. Pix2pix needs well-paired data while CycleGAN fails to utilize some paired data. Instead, this paper considers the noisy case where some paired data are misaligned.\n\n2. Extensive experiments are conducted and show the performance of the propose method.\n\n\nCons:\n\n1. The novelty is limited.  This paper introduces a registration network after the translation network to address the misaligned data. But a published noisy label paper [7] already introduces a noise transition $\\phi$. This paper adapts it to medical image translation and uses a registration network.  The theoretical motivation is also borrowed from [7].\n\nAfter reading authors' rebuttal, I think the technical novelty of this method is enough though more analysis (theoretical and more complex datasets) are needed. So I raised my score to 6.\n\nSuggestions:\n\n1. Authors may consider adding a brief introduction to registration network and deformation field before section 3.2, which may cause some confusion for people who are not in medical fields.\n\n2. It would be better if authors can provide some theoretical analysis about this method. It is still not very clear why introducing a registration network can help align the data without any other specific regularization. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel medical image-to-image generative model, called RegGAN, to combine an adversarially trained image generator with a registration module, which enforces the alignment between the generated image and noisy ground truth image. The paper illustrates the better performance of the proposed method on all major metrics for both paired and unpaired data and that RegGAN is more resistant to label noise and more stable in the training process. All reviewers agree that the results are convincing and that the work may bring insights to the image translation community.
Decision: Accept (Spotlight)