Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a novel method for conducting image-to-image translation integrating registration network (RegGAN), and demonstrated the superior performance in T1 to T2 MRI image translation to the current methods based on Pix2Pix and cycle-consistency techniques.
The RegGAN performed slightly better than Pix2Pix method even without added noise likely due to inherent misalignment from the image acquisition. While the Pix2Pix method deteriorates rapidly with the added noise, RegGAN was robust against noise similarly to CycleGAN (with better performance).
The authors also experimented to train them with unpaired images, and RegGAN could generate T2 images more similar to the actual images than the other two methods. 
main_review: Originality:
While using deep learning network for medical image registration is not new (e.g. BIRNet), combining it with generator network for image-to-image translation and training it end-to-end is the original approach.

Quality and clarity:
The paper is clearly written with enough details.

Significance:
T1 and T2 MRI images are typically well aligned, and the performance gain in paired setting is not large. Still, the results were presented well to indicate the potential to be applied to other medical imaging tasks where distortion from motion artifacts are more evident.
limitations_and_societal_impact: The paper still have a few limitations.
First, more traditional image (co)registration technique combined with Pix2Pix technique hasn't been explored. Especially in paired setting with small or zero noise, Pix2Pix followed by rigid alignment may perform similarly to RegGAN. While registering T2 images to T1 images can be challenging in some cases, another approach would be register both T1 and T2 images to reference T1 and T2 images, respectively at first and train Pix2Pix translation between the registered images. I still think the end-to-end approach of the authors have benefits over these in terms of simplicity, it would be nice to see such comparisons.
Second, the performance metric used is mostly based on image-level, and doesn't reflect actual utility in medical diagnosis. Slight improvement in aligning the normal tissues can improve these metrics, while not improving the clinical utility. Potentially, even with the higher score, the translated images may completely miss important abnormalities which is normally highlighted in T2 images (e.g. high: ischemic stroke, low: hemorrhage). This can be in particular problematic in unpaired settings. At least some examples should be included.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new method for image-to-image translation, which accommodates misalignment between training image pairs - a common problem particularly with medical images where motion or distortion often leads to misalignment that cannot be perfectly corrected through a-priori image registration.  The authors rightly state that this is a key missing component of existing image-to-image translation approaches such as pix2pix-type algorithms that assume matched pairs with perfect alignment, whereas cycle-consistency approaches do not exploit matched pairs when available.


main_review: I am enthusiastic from the start, as this is a problem I have encountered and thought about, and have been waiting for a nice solution!  I'm amazed no-one has done this before actually, but I am not aware of prior work on this issue.

The implementation the authors propose is simple but sensible and the right place to start with solving this important problem.  They simply add estimation of a deformation field that is learned concurrently with the image-to-image translation during training, as well as a smoothing loss to keep the deformation under control.

Experiments use the BRATS brain-MRI data set and add artificial misalignments to pairs of images with different MR contrast before learning image-to-image translations between those contrasts from the paired data set.  As expected, the proposed technique outperforms existing techniques that do not consider misalignment. The authors go on to show that the proposed technique also has benefit in situations where matched pairs are not available but simply unmatched examples of both contrasts even though the method is not designed for such situations.

My key criticism is that the experiments are rather artificial as the imposed misalignments are much larger than one would normally find between different MR contrasts.  Rotations scalings and displacements of the type the authors use to corrupt the alignment would be easily fixed by a-priori image registration. In practice the real problem arises from small residual misalignments after a-priori image registration - that tends to lead to blurring in estimated images as a consequence of learning from misaligned data sets. A fairer comparison with existing methods and more convincing demonstration of the benefits of the proposed technique would have come from registering the paired images prior to training, as that is what users of say pix2pix would naturally normally do.  We do get a hint of this in section 4.3 where the evaluation as a function of "noise level" (which here refers to the degree of alignment corruption) includes zero noise, ie no artificial misalignment but only residual misalignment after the registration step used in the original BRATS data.  The proposed technique does still show benefit in this "zero noise" test, which is encouraging, but I would have liked to see a more complete evaluation and comparison against the state of the art in this more realistic scenario.

Nevertheless, I remain enthusiastic about the proposed method and look forward to seeing more complete demonstrations of its capability as well as further refinement of the idea.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a new medical image-to-image generative model, called RegGAN, that combines an adversarially trained image generator with a registration module that enforces alignment between the generated image and ‘noisy’, (un)paired ground truth image. The aim of the network is to search for the single optimal solution to both image-to-image translation and registration tasks. The authors compared their proposed method against two other general model types, Pix2Pix (paired aligned data) and Cycle GAN (unpaired or misaligned). For training and testing, they use the BRATS dataset, which consists of brain tumor MRIs, and translate T1 to T2 images. The authors were able to show better performance on all major metrics for both paired and unpaired data. They showed that RegGAN was more resistant to label noise and increased stability during training.  
main_review: Originality: an original combination of a registration loss to a GAN objective in a medical image setting. 
Quality and Clarity: good quality, the paper is well written and the figures are excellent. 
Significance: Good contribution to a medical imaging problem that needs to be addressed. 
limitations_and_societal_impact: The authors have restricted the noise to affine deformations only. RegGAN is capable of doing non-affine noise correction and these deformations may be encountered in clinical medicine (e.g. 1.5 vs 3 Tesla MRIs). The authors are encouraged to perform these experiments and add them to Table 2. 

The current study is limited to a single image-to-image translation task: T1 to T2. While I understand that the authors are limited by the availability of open-source datasets, this task is highly artificial and generally would not occur in clinical medicine. It seems a better test, especially for demonstrating superior performance on unpaired or misaligned datasets, would be cross-modality, such as CT to MRI or vice versa. The authors state ‘Previously, physicians had to use different medical imaging equipment if they wanted to get multi-modal images of a patient, which was time-consuming and expensive... Our work aims to..meet the clinical requirements in terms of image quality’. No multi-modal testing is shown in the paper. I would either attempt to obtain a CT-MRI dataset (this would strengthen the impact of the results), or remove the previous statement.

Line 235: BraST > BraTS

Overall, good paper and a thoughtful contribution.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers the setting where paired images are available but some of them are misaligned. To address this issue, it proposes to add a registration network after the translation network as the "correction loss".
main_review: Pros

1. The setting is interesting. Pix2pix needs well-paired data while CycleGAN fails to utilize some paired data. Instead, this paper considers the noisy case where some paired data are misaligned.

2. Extensive experiments are conducted and show the performance of the propose method.


Cons:

1. The novelty is limited.  This paper introduces a registration network after the translation network to address the misaligned data. But a published noisy label paper [7] already introduces a noise transition $\phi$. This paper adapts it to medical image translation and uses a registration network.  The theoretical motivation is also borrowed from [7].

After reading authors' rebuttal, I think the technical novelty of this method is enough though more analysis (theoretical and more complex datasets) are needed. So I raised my score to 6.

Suggestions:

1. Authors may consider adding a brief introduction to registration network and deformation field before section 3.2, which may cause some confusion for people who are not in medical fields.

2. It would be better if authors can provide some theoretical analysis about this method. It is still not very clear why introducing a registration network can help align the data without any other specific regularization. 

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposes a novel medical image-to-image generative model, called RegGAN, to combine an adversarially trained image generator with a registration module, which enforces the alignment between the generated image and noisy ground truth image. The paper illustrates the better performance of the proposed method on all major metrics for both paired and unpaired data and that RegGAN is more resistant to label noise and more stable in the training process. All reviewers agree that the results are convincing and that the work may bring insights to the image translation community.
Decision: Accept (Spotlight)