Discriminative Cross-Modal Data Augmentation for Medical Imaging Applications
Authors: Yue Yang
Pengtao Xie
Venue:  (2021)
Keywords: Deep learning
Medical imaging
Cross-Modal Learning

Abstract: While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, it is oftentimes very difficult to obtain a lot of labeled medical images for model training. In this paper, we study cross-modality data augmentation to mitigate the data deficiency issue in medical imaging domain. We propose a discriminative unpaired image-to-image translation model which translate images in source modality into images in target modality where the translation task is conducted jointly with the downstream prediction task and the translation is guided by the prediction. Experiments on two applications demonstrate the effectiveness of our method. 

Under review as a conference paper at ICLR 2021
DISCRIMINATIVE CROSS-MODAL DATA AUGMENTA- TION FOR MEDICAL IMAGING APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, is oftentimes very difficult obtain a lot of labeled medical images for model training. In this paper, we study cross- modality data augmentation to mitigate the data deficiency issue in the medical imaging domain. We propose a discriminative unpaired image-to-image trans- lation model which translates images in source modality into images in target modality where the translation task is conducted jointly with the downstream pre- diction task and the translation is guided by the prediction. Experiments on two applications demonstrate the effectiveness of our method.
INTRODUCTION
Developing deep learning methods to analyze medical images for decision-making has aroused much rescarch interest in the past few years. Promising results have been achieved in using medical images for skin cancer diagnosis (Esteva et al. 2017 Tschandl et al.] 2019). chest diseases identifi- cation (Jaiswal et al. 2019), diabetic eye disease detection (Cheung et al., 2019) name few. It is well-known that deep learning methods are data-hungry. Deep learning models typically contain tens of millions of weight parameters. To effectively train such large-sized models, large number of labeled training images are needed. However, in the medical domain, it is very difficult to collect labeled training images due to many reasons including privacy barriers, unavailability of doctors for annotating disease labels, etc.
To address the deficiency of medical images, many approaches (Krizhevsky et al., 2012 Cubuk et al., 2018: Takahashi et al.) 2019; Zhong et al.) 2017 Perez & Wang, (017) have been proposed for data augmentation. These approaches create synthetic images based on the original images and use the synthetic images as additional training data. The most commonly used data augmentation approaches include crop, flip, rotation, translation, scaling, etc. Augmented images created by these methods are oftentimes very similar to the original images. For example, cropped image is part of the original image. In clinical practice, due to the large disparity among patients, the medical image of new patient (during test time) is oftentimes very different from the images of patients used for model training. If the augmented images are very close to the original images, they are not very useful in improving the ability of the model to generalize to unseen patients. It is important to create diverse augmented images that are non-redundant with the original images.
To create non-redundant augmented images for one modality such CT, one possible solution is to leverage images from other modalities such as X-ray, MRI, PET, etc. In clinical practice, for the same disease, many different types of imaging techniques are applied to diagnose and treat this disease. For example, to diagnose lung cancer, doctors can use chest X-rays, CT scans, MRI scans, to name few. As result, different modalities of medical images are accumulated for the same disease. When training deep learning model on an interested modality (denoted by X) of images, the number of original images in this modality is small, we may convert the images in other modalities into the target modality X and use these converted images as additional training data. For example, when a hospital would like to train a deep learning model for CT-based lung cancer diagnosis, the hospital can collect MRI, X-ray, PET images about lung cancer and use them to augment the CT training dataset. Images of different modalities are typically from different patients. Therefore, their clinical diversity large.

---

Under review as a conference paper at ICLR 2021
This motivates to study cross-modality data augmentation to address the data deficiency issue in training deep learning models for medical image based clinical reciisio-makingg The problem setup is as follows. The goal is to train a deep learning model to diagnose disease D based one modality (denoted by X) of medical images. However, the number of images in this modality is limited, which are not sufficient train effective models. Meanwhile, there another modality (denoted by Y) of medical images used for diagnosing disease D. Cross-modality data augmentation refers to leveraging the images in Y augment the training set in X. Specifically, we translate images in Y into images that have similar style as those in X and add the translated images together with their disease labels into the training dataset in X. Compared with simple augmentation such as cropping, scaling, rotation, cross-modality data augmentation can bring in more diversity since the images in different modalities are from different patients and hence are clinically more heterogeneous. More diverse augmented images are more valuable in improving the generalization ability of the model to unseen patients.
To perform cross-modality data augmentation, we propose discriminative unpaired image-to- image translation (DUIIT) method. Given images in source modality, we translate them into images in a target modality. The translated images, together with their associated disease labels, are added to the training set in the target modality to train the predictive model. Different from unsupervised translation methods such as CycleGAN (Zhu et al. (017a) which perform the trans- lation between images without considering their disease labels, our method conducts the translation in discriminative way, where the translation is guided by the predictive task. Our model performs cross-modality image translation and predictive modeling simultaneously, to enable these two tasks to mutually benefit each other. The translated images are not only aimed to have similar style as those in the target modality, but also are targeted to be useful in training the predictive model.
Our model consists of two modules: a translator and predictor. The translator transforms the images in the source modality into synthesized images in the target modality. Then the translated images (together with their class labels) are combined with the images in the target modality to train the predictor. The predictor takes an image as input and predicts the class label. The two modules are trained jointly so that the translator is learned to generate target images that are effective in training the predictor.
We apply our method to two medical imaging applications. In the first application, the source modality is MRI and the target modality is CT. Our method translates MRI images into CTs in a dis- criminative way and uses the combination of original CTs and translated CTs to train the predictive model, which achieves substantially lower prediction error compared with using original CTs only. In the second application, the source modality is PET and the target modality is CT. By translating PET images to CT's, our method significantly improves prediction performance.
The major contributions of this paper include:
We propose a discriminative unpaired image-to-image translation method to translate medical images from the source modality to the target modality to augment the training data in the target modality. The translation is guided by the predictive task.
On two applications, we demonstrate the effectiveness of our method.
The rest of the paper is organized as follows. Section 2 reviews related works. Section 3 and present the methods and experiments. Section 5 concludes the paper.
2
RELATED WORKS
2.1 MEDICAL IMAGE SYNTHESIS
Several attempts have been made to synthesize medical images. Frid-Adar et al. (2018) combine three DCGANs (Radford et al. 2013) to generate synthetic medical images, which are used for data augmentation and lesion classification. Jin et al.) (2018) develop a 3D GAN to learn lung nodule properties in the 3D space. The 3D GAN is conditioned on a volume of interest whose central part containing nodules has been erased. GANs are also used for generating segmentation maps Guibas et al. 2017) where a two stage pipeline is applied. Mok & Chung (2018) apply conditional GANs (Mirza& & Osindero 2014 Odena et al. 2017) synthesize brain MRI images in a coarse-to to-too---- fine manner, for brain tumour segmentation. To reserve fine- ame-ierinedd details of the tumor core, they encourage the generator to delineate tumor boundaries. mods-ty translation has
2

---

Under review as a conference paper at ICLR 2021
been studied synthesize medical images. In Wolterink al. 2017). it is found that using unpaired medical images for augmentation is better than using aligned medical images. Chartsias et al. (2017) apply CycleGAN for generating synthetic multi-modal cardiac data. Zhang al. (2018) combine the synthetic data translated from other modalities with real data for segmenting multimodal medical volumes. A shape-consistency loss is used to reduce geometric distortion.
2.2 IMAGE GENERATION
Generative Adversarial Networks (GANs) (Goodfellow et al. 2014; Radford et al. 2015 Ar-
jovsky et al. 2017) have been widely used for image generation from random vectors. Conditional GANs (Mizaa & Osindero 2014, Odena et al. 2017) generates images from class labels. Image-to-
image translation (Isola al. 2016 Wang et al. [0118] Zhu et al., 2017a) studies how to generate one image (set) from another (set) based on GANs. A number of works have been devoted to generating images from texts. Mansimov et al. (2015) propose an encoder-decoder architecture for text-to- image generation. The encoder of text and the decoder of image are both based on recurrent net- works. Attention is used between image patches and words. AttnGAN (Xu et al., (218)) synthesizes fine-grained details at different subregions of the image by paying attention to the relevant words in the natural language description. DM-GAN (Zhu et al., 2019) uses a dynamic memory module to refine fuzzy image contents, when the initial images are not well generated, and designs memory writing gate to select the important text information. Obj-GAN (Li et al. (019) proposes an object- driven attentive image generator synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. MirrorGAN (Qiao et al.) 2019) uses an autoencoder architecture, which generates an image from text, then reconstructs the text from the image. Many techniques have been proposed to improve the fidelity of generated images by GANs. Brock et al. (2018) demonstrate that GANs benefit remarkably from scaling: increasing model size and minibatch size improves the fidelity of generated images. To generate images with higher resolution, the progressive technique used in PGGAN (Karras et al.) 2017)) has been widely adopted. StackGAN (Zhang al., (017) first uses GAN to generate low-resolution images, which are then fed into another GAN to generate high-resolution images.
2.3
IMAGE AUGMENTATION
Image augmentation is widely used technique enlarge the training dataset and alleviate over- fitting. Basic augmentation methods (Krizhevsky et al. 2012) include geometric and color trans- formations, such as crop, flip, rotation, translation, scale, color jitter, contrast, etc. [Cbuk et al. (2018) propose a reinforcement learning based algorithm to automatically search for augmentation policies. Takahashi al. (2019) propose to randomly crop four images and patch them to create new training image. Zhong et al. (2017) propose Random Erasing which assigns random values to pixels in vandomly-sampled rectangle regions, to generate augmented images with different occlu- sion levels. |eerez& & Wang (2017) design two methods for data augmentation. One is using GAN to synthesize new images with different styles. The other training an augmentation network for generating augmented data. Zhao et al. (2020) propose DiffAugment that can improve the sample efficiency of training GANs. This method applies the same differentiable augmentation to both real and fake images when training the generator and discriminator.
METHOD
In this section, we introduce a discriminative unpaired image-to-image translation method for cross- modal data augmentation. We are given a set of images in modality X and another set of images in modality Y. For example, X could be CTs and Y could be MRIs. For each image in X and Y, it is associated with a class label r E R. where the label space R is shared by images in these two modalities. Our goal to learn a predictive model which maps images in X to R. However, the number of training images in X is limited. Therefore, we would like to use images in Y to augment the training set X. The basic idea is: we develop model which translates each image y in Y into an image where the modality of ÿ is aimed to be X. Then we add the translated images together with their associated class labels into the training set in and use the combined training set to train the predictive model. One way to perform such a translation is to use unpaired image-to- image translation methods such as CycleGAN (Zhu et al.) 2017b). However, CycleGAN performs translation in unsupervised way. The translated images may not be optimal for the predictive task. To address this issue, we propose a discriminative unpaired image-to-imade translation approach. In model, two tasks performed simultaneously: unpaired image-to-image translation and
3

---

Under review as a conference paper at ICLR 2021
Translator
Predictor
Real CT
Adversarial Loss
Discriminator:
Prediction Loss
Cycle Consistency Loss
Reconstructed
Generator:
Real MRI
The architecture of our model. The source modality and target modality CT and MRI respectively. For clarity, omit the adversarial loss in translating CTs MRIs. predictive modeling. The translated images are not only encouraged to be similar to the images in X, but also useful to train the predictive model. Figure illustrates this idea.
Figure 1:
Our model consists of two modules: translator and predictor. The translator takes images in the source modality as inputs and translates them into images that are similar to those in the target modality. The predictor takes the original target images and translated images as inputs and predicts their class labels. The translator and the predictor are trained jointly end-to-end. The translated images are required 1) be visually similar those the target modality; 2) informative for training the predictor. In the sequel, we introduce these two modules in detail.
3.1 TRANSLATOR
The translator is based CycleGAN. Generative adversarial network (GAN) Goodfellow et al. (2144) is a deep generative model. It consists of a generator and discriminator, The generator takes random vector as input and generates an image. The discriminator takes an image (either real generated) input and predicts whether this image is real or generated. The generator and discriminator are learned jointly by solving min-max problem where the loss is a classification image as real or generated. The discriminator aims to minimize
loss in the task of classifying this loss and the generator aims to maximize this loss. Intuitively, the discriminator tries to tell apart generated images from real ones while the generator tries to make the generated images as realistic as possible a way that they are not distinguishable from real images.
GAN generates images from random vectors. These random vectors do not have class labels. As result, the generated images do not have class labels. Therefore, they are not very useful for training supervised models. In task, images in the source modality a source image and its class label r, we can translate y into the target modality ÿ. Then we obtain a labeled pair (ú, r) in the target domain and use this pair to train the target predictor.
associated with class labels. Given
However, the images in source modality and those in the target modality do not have one-to-one correspondence. Namely, a source image and target image are from different patients. Therefore, we cannot perform the translation by training a model which maps a source image to a target image. To address this problem, we use CycleGAN (Zhu et al. (0017)), to perform unpaired image-to-image translation which translates one set of images to another set. Let G denote a conditional generative model which takes a source image as input and generates an image in the target modality. Let FF denote conditional generative model which takes a target image as input and generates an image the source modality. Let D. denote a discriminator which judges whether an image in the source modality is real or fake. Let D, denote discriminator which judges whether image in the target modality is real or fake. CycleGAN employs a cycle-consistency loss: given a source image y. it is first translated into the target modality using G: ((v), then G(v) is translated back to the source modality using F: F(G(y)) and F()()) is encouraged to be close to the original image y. Similarly, the cycle consistency loss be defined target image well, where ((F(x)) is
4

---

Under review as a conference paper at ICLR 2021
encouraged to be close to The overall loss is defined as:
Lg,ans(g F,, D., +deye ( Pansa (2) (x) y1l1 Pa (x) [[ =
Ladi (Dr, + Ladu (D., x,,7
(1)
where Lodv denotes the adversarial training loss in GAN and Aaye is tradeoff parameter.
3.2 PREDICTOR
(())) )) where yi is an image and r² is its label, we trans- Given N₂ labeled source images {(yi, ri
late ys into the target modality: (V:)), and obtain collection of translated images with labels:
((G)), r } Then we combine these generated pairs with the training data {(z,, ( ) in
the target modality and the combined data to train predictor. Let denote the predictor and (()()), denote the loss function defined on a training pair (zr, r). Then the discriminative loss is:
N. Σ (P(G(W)) + Σ (P(2)))))))
Cpred P)
3.3 OBJECT FUNCTIONS
In our method, the translation task and the prediction task are performed jointly by optimizing the following objective function:
D,, ( D,, + AG, P)
where À is tradeoff parameter.
EXPERIMENT
In this section, we present experimental re- sults. The target modality CT and the clin- ical task is to predict the physiological age of
Table 1: Statistic of the three datasets.
Image number
Image size
Patient number
MRI
3454
256 x 256
110
a patient based on his or her CT image. Phys- iological age (Wang et al. 2017) is measure of how well or poorly one's s body is function- ing. It may be different from the chronolog- ical age (which is calculated based on birth
CT
2438
650
650
82
PET
12780
128 x 128
298
date). For example, 30-year-old young person may have physiological age of 40 if his her biological system is not functioning well. Predicting physiological age has important clinical appli- cations in disease prognosis and treatment. We use images from two source modalities including MRI and PET to augment the CT training set. We translate MRI and PET images into CTs to help with the physiological age prediction task on CT. Each image in the three modalities labeled with physiological age. We use mean squared error to measure predictive performance.
4.1 DATASETS
Three datasets are used in our experiments: Brain MRI (Buda 2019), Brain CT Hssayeni 2019 Hssayeni al., 2020), and Head PET (Vallières et al. 2017 Vallieres al., 2017 Clark et al. 2013). Brain MRT contains 3454 brain MRI images of size 256 x 256 from 110 patients. MRI stands for magnetic resonance imaging, which is a medical imaging technique used in radiology to form pictures of the anatomy and the physiological processes of the body. Brain CT contains 2438 brain CT slices of size 650 650 from 82 patients.
CT stands for computed tomography, which is another type of medical imaging technique. On average, each patient has about 30 slices. Head PET (Vallières et al. 2017 Vallieres et al. [017 Clark al. 2013) contains 12780 head PET images of size 128 x 128 from 298 patients. PET stands for positron emission tomography, which a type of nuclear medicine procedure that measures metabolic activity of the cells of body tissues. Each image in the three dataset is labeled with the physiological age of the patient. Table T shows the statistics of the three datasets. From the original CT dataset, we randomly select 1887 images for training, 271 images for validation, and 280 images for testing. Synthetic CT's translated from MRIs and PETs are used for training. We perform the random splits 10 times and report the mean and standard deviation of performance numbers obtained from the 10 runs.
5

---

Under review as a conference paper at ICLR 2021
4.2 IMPLEMENTATION DETAILS
In the translator, the architectures of the generator and discriminator are the same as those in Cycle- GAN (Zhu et al. 2017b). The generator consists of several convolution layers with stride 2, residual blocks (He et al. 2016) and fractionally strided convolutions with stride 2-2 Different blocks are used for images with different resolution. As for the discriminator, we use the architecture of the 70 x 70 PatchGANs (Isola al. 2017 Ledig al., 2017: Li & Wand, 2016). For the predictor, we use ResNet50 (He et al., (016) as the backhone. For the translator, we follow the same train- ing setting CycleGAN (Zhu et al. 2017b). We use the least-square loss (Mao et al. 2017) for Ladv. Discriminators are updated using images from image buffer (Shrivastava et al. 2017). Aeye in Equation D is set to 10. We use Adam (Kingma translator are trained from scratch with a learning rate of 0.0002, which linearly decays from epoch 100. The learning rate set to 0.001 for the predictor and we adopt the same decay strategy as for the translator. À in Equation 3 is set to 0.001.
Ba (014) as the optimizer. Networks in the
4.3 BASELINES
We compare with the following baselines.
Transfer learning (TL). We first train the prediction network using images and their labels in the source modality. Then we transfer the learned network the target modality. We use the network trained on source images to initialize the network for the target task, then finetune this network using images and labels the target modality.
Multi-task learning (MTL). We develop single network to perform the prediction task in the source modality and target modality simultaneously. The tasks share the same visual feature learning network. A source-specific predictive head is used to make predictions on the source images and larget-specific predictive head is used make predictions on the target images. After training, the source head is discarded. The representation learning network and the target head retained form the final network.
Domain adaptation (DA). We treat the source and target modality as source and target domain respectively and perform adversarial domain adaptation using the -ovean-Adversariial Neural Network (DANN) (Ganin & Lempitsky/ 2015) method. DANN aims to learn feature extractor in way that the representations of images in the souree domain indistinguishable from the target domain, for the purpose of minimizing the discrepancy of these two domains.
CycleGAN. We use CycleGAN (Zhu et al., 201 to perform unsupervised translation of source images into the target modality. In the translation process, the age labels are not leveraged. After translation, the translated images are combined with real images for training the predictive model.
Data augmentation methods. We compare with three data augmentation methods including Simple Augment, AutoAugment (Cubuk et al.) 2018). and Random Erasing (Zhong et al. 2020)) These methods are applied the training CTs create augmented CTs which used addi- tional training data. In Simple Augment, we apply commonly used traditional data augmentation methods, such as crop, flip, translation, rotating, and color jitter. The augmentation operations in AutoAugment are automatically searched using reinforcement learning. Random Erasing ran- domly samples a rectangle from an image and replaces pixels in this rectangle with random values.
4.4 RESULTS
Table 2 shows the prediction errors under four settings:
PURE-CT. The predictive model is trained purely on original CT images.
MIX-CTMMRI. The predictive model is trained on CT images and MRT images. The MRI images are translated into synthesized CT images, which are combined with the original CT images to train the predictive model. The translation from MRI to CT is performed jointly with the training of the predictive model.
MIX-CT-PET. The setting is similar to MIX-CT-MRI, except the source modality is PET.
MIX-CT-MRI-PET The setting is similar to MIX-CT-MRI, except that CT and PET are both used as source modalities and are translated into CTs.
As can be seen from this table, the mean prediction errors under MIX-CT-MRI, MIX-CT-PET, and MIX-CT-MRI-PET much lower than that under PURE-CT. This demonstrates that

---

Under review as a conference paper at ICLR 2021
effective to translate images from other modalities (Lee., MRI, PET) into the target modal- ity (i.e.. CT) as augmented data for model training. Our approach is effective in perform- ing such translation. MIX-CT-MRI PET because MIX-CT-MRI-PET translates more source images for training the predictive model. Another observation is: the standard deviation of pre- diction errors under MIX- CT-MRI, MIX-CT-PET, and MIX-CT-MRI-PET are much lower than that under PURE-CT. This because with cross-modal translation, MIX-CT-MRI, PURE-CT. Increasing training data can reduce the variance of the model.
performs better than MIX-CT-MRI and MIX-CT
Table 2: Mean and standard deviation (std) of prediction errors under different data settings.
PURE-CT
MIX-CT-MRI
MIX-CT-PET
MIX-CT-MRI-PET
Mean 103.88
76.91
75.75
71.37
64.27
8.31
20.52
18.92
and MIX-CT-MRI-PE have more training data than
Table Comparison between our method and baselines
MRI
Method
MTL
CycleGAN
method
TL
CycleGAN
Our method
Mean
91.52
99.26
88.13
147.06
76.91
102.08
92.94
148.72
96.70
75.75
13.88
22.12
21.86
79.89
15.37
28.26
20.52
Table 4: Comparison between our method and data augmentation methods
Simple Augment
AutoAugment
Random Erasing
Our method
Our method (PET-CT) -CCT)
Mean
96.80
84.65
84.87
76.91
75.75
Std
26.00
15.28
8.55
8.31
20.52
3 compares our method with (TL). multi-task learning (MTL), domain adaptation As can be seen, our method outperforms the four baselines. The reason that our method outperforms TL, MTL, and DA is because these three baseline methods do not explicitly translate source images into the target modal- ity whereas our method performs this explicit translation. In TL, the source images used to pretrain the prediction network. In MTL, the source images used train the prediction network jointly with target images. In these two methods, the source images are directly used to train networks without translation. Since the source images have large modality discrep- ancy with target images, the networks trained by source images may not be suitable for mak- ing predictions on target images. Domain adaptation (DA) partially addresses this prob- lem by making the visual representations of source images and target images to be close. The adaptation is performed in the latent space which may not capture the fine- nhe-grained details the pixel level which are crucial for making accurate clinical predictions. In contrast, our method produces the clinical details are preserved. The rea- son that our method outperforms CycleGAN is because the source-to-targer translation in our method is discriminative and supervised by age labels, whereas CycleGAN performs the translation in an unsupervised way without leveraging the age labels. While the translated images by Cycle- GAN visually look like CTs, they may not optimal for predicting the chronological ages.
Table
baselines including (DA),
transfer and
learning CycleGAN,
Real MRI
MRI
Real PET
PET -
synthetic images where
Figure 2: Examples of translating MRI and PET images into CTs.
7

---

Under review as a conference paper at ICLR 2021
method with several data augmenta- tion methods including Simple Augment, Augmmnt and Random Erasing. As can be seen, der two settings translating MRIs to CT's and translat- ing PETs to CTs both performs better than the three augmentation methods. The three augmentation methods create augmented images from the training CTs. As re- sult, the augmented CTs are similar to training CTs with high redundancy. These augmented CTs do not bring in significantly new signals into the training set and hence are not substantially helpful in improving the generalization performance. In con- trast, our method translates real MRIs and PETs into CTs. These MRIs and PETs are from other patients and contain clinical traits that are significantly different from those in the training CT'.. The CTs translated from MRIs and PETs bring in substantial diversity to the training set and the model trained using them has better generalization ability.
Table 5: Evaluation of image quality us- ing IS and FID
method un-
IS
FID
CycleGAN
3.49 + 0.15
40.94
Ours 3.92 0.17
63.17
We evaluate the quality of translated images. Fig- ure 2 shows some examples of translating MRI and PET images into CTs by our method. As can be seen, these translated images look like real CTs. Many fine-grained details in the original MRI/PET images are well-preserved in the translated images. These details are important for correctly predicting physiological ages. Table [ compares the inception score (IS) (Salimans et al. (016)) and Frechet incep- tion distance (FID) (Heusel et al., 2017) achieved by CycleGAN and our method. Figure 3 shows the comparison between images generated by Cycle- GAN and our method. As can be seen, our method achieves higher (better) inception score. Incep- tion score measures the realisticness and diversity of generated images. This result demonstrates that the images generated by our method and diverse. The reason is that our method translates images in discriminative way, by encouraging the translated images to be suitable for the prediction task. In contrast, CycleGAN performs the transla- tion purely based on style matching and ignores the supervised information. On the other hand, the FID achieved by our method is larger (worse) compared to CycleGAN, FID measures the similarity between generated images and target images. This result demonstrates that images generated by our method are less similar to the target images compared with CycleGAN. The reason is that our method retains details of some soft tissues that are contained in source images but not in target images. These details render the translated CTs more different from the real CT's However, this is not necessarily a disadvantage. In our method, the goals of trans- lation are two-fold: 1) the translated images are visually similar to those in the target modality; 2) the translated images preserve important clinical details that are informative for predicting chrono- logical ages. Compared with CycleGAN, our method achieves the second goal better with a small sacrifice of the first goal. CycleGAN performs the translation to solely achieve the first goal. While the translated CTs by CycleGAN are more visually similar the real CTs, they are not necessarily good for predicting the physiological age. And the prediction task is ultimately what we care about.
CycleGAN
Our method
more realistic
Figure 3:
Some examples of translating MRIs to CTs by CycleGAN and our method. As shown in the regions marked with ovals, our method can better preserve fne-grained details in the translated images than Cycle- GAN.
5 CONCLUSION
In this paper, we propose discriminative unpaired image-to-ine -o-aage translation, which translates im- ages from source modalities into a target modality and use these translated images as augmented data for training the predictive model in the target modality. In our method, the training of the image-to- image translation module and the learning of the predictive module are conducted jointly so that the supervised information in the predictive task can guide the translation model. We apply our method for physiological age prediction. Experiments on three datasets demonstrate the effectiveness of our method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan.
arXiv preprint
arXiv: (eXiv:/701.0700 7777777775 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018.
Mateusz Buda. Brain mri segmentation, 2019. data retrieved from kaggle open dataset, https:
//www. kaggle com wetr:/aatusszzuuuu /lgg- Lig--riii- wogggggr-eemeeeteeeeeeeeee-eee-e secmentationn
Agisilaos Chartsias, Thomas Joyce, Rohan Dharmakumar, and Sotirios Tsaftaris. Adversarial image synthesis for unpaired multi-modal cardiac data. In International workshop simulation and synthesis in medical imaging, pp. 3-13. Springer. 2017.
Carol Y Cheung, Fangyao Tang, Daniel Shu Wei Ting, Gavin Siew Wei Tan, and Tien Yin Wong. Ar- tificial intelligence diabetic eye disease screening. The Asia-Pacipp Journal of Ophthalmology. 166,1,58-16, 2019.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating a public information repository. Journal of digital imaging, 26(6): 1045-1057, 2013.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1rXiv:8005.00 00551, 2018.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Scbastian Thrun. Dermatologist-lowwe classification of skin cancer with deep neural networks. Nature, 542 2017.
Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan-based synthetic medical image augmentation for increased enn performance in liver lesion classification. Neurocomputing, 321:321-331, 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning. pp. 1180-188, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
John T Guibas, Tejpal S Virdi, and Peter S Li. Synthetic medical images from dual generative 2017.
adversarial networks. arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem:770-7718 2016. ISSN 10636919, doi: 10.1109/CVP..2016.90
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637,2000 2017.
M.D. Hssayeni. Computed tomography images for intracranial hemorrhage detection and seg- mentation, 2019. Available online:
3. acccessed on 26 December 2019).
Murtadha D Hssayeni, Muayad Croock, Aymen D Salman, Hassan Falah Al-khafaji, Zakaria A Yahya, and Behnaz Ghoraani. Intracranial hemorrhage segmentation using deep convolutional model. Data, 5(1):14, 2020.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imet tomiia translation with conditional adversarial networks. arxiv, 2016.

---

Under review as a conference paper at ICLR 2021
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-ing -ongage translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Amit Kumar Jaiswal, Prayag Tiwari, Sachin Kumar, Deepak Gupta, Ashish Khanna, and Joel JPC Rodrigues. Identifying pneumonia in chest x-rays: A deep learning approach. Measurement, 145: 51 511-518, 2019.
Dakai Jin, Ziyue Xu, Youbao Tang, Adam P Harrison, and Daniel J Mollura. Ct-realistic lung nodule simulation from 3d conditional generative adversarial networks for robust lung segmentation. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp- 732-740. Springer, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. European conference on computer vision, pp. 702-716. Springer, 2016.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 12174-12182, 2019.
Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. CoRR, (bs//551.00299,, 2015.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 2794-2802, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. ArXiv, abs/1411.1784. 2014.
Tony CW Mok and Albert CS Chung. Learning data augmentation for brain tumor segmentation with coarse-to-fine generative adversarial networks. In International MICCAI Brainlesion Work- shop, pp. 70-80. Springer, 2018.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2642-2651. JMLR. org, 2017.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.0462) 2017.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 1505-1514, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 151 1.06444, 2015.
10

---

Under review as a conference paper at ICLR 2021
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2244, 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2107-2116, 2017.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions on Circuits and Systems for Video Technology, 2019.
Philipp Tschandl, Cliff Rosendahl, Bengu Nisa Akay, Giuseppe Argenziano, Andreas Blum, Ralph P Braun, Horacio Cabo, Jean- Gourhant, Jürgen Kreusch, Aimilios Lallas, et al. Expert-level diagnosis of nonpigmented skin cancer by combined convolutional neural networks. JAMA der- matology, 155(1):58 65, 2019.
Martin Vallières, Emily Kay-Rivest, I. Perrin, Xavier Liem, Christophe Furstoss, Nader Khaouam, Phuc Félix Nguyentan, Chang-Shu Wang. and Khalil Sultanem. Data from peed--eec--ee--tt The Cancer Imaging Archive, 10:K9, 2017.
Martin Vallieres, Emily Kay-Rivest, Léo Jean Perrin, Xavier Liem, Christophe Furstoss, Hugo JWL Aerts, Nader Khaouam, Phue Felix Nguyen-Tan, Chang-Shi Shu Wang, Khalil Sultanem. et al. Ra- diomics strategies for risk assessment of tumour failure in head-and cancer. Scientific re- ports, 7(1):1-14, 2017.
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018
Zichen Wang, Li Li, Benjamin S Glicksberg, Ariel Israel, Joel Dudley, and Avi Ma ayan. Predict- ing age by mining electronic medical records with deep learning characterizes differences between chronological and physiological age. Journal of biomedicul informatics, 76:59-68, 2017.
Jelmer Wolterink, Anna M Dinkla, Mark HF Savenije, Peter Seevinck, Cornelis AT van den Berg, and Ivana Iguum Deep mr to ct synthesis using unpaired data. In International workshop on simulation and synthesis in medical imaging, pp. 14-23. Springer, 2017.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316-1324, 2018.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-reale realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference computer vision, pp. 5907-5915, 2017.
Zizhao Zhang. Lin Yang. and Yefeng Zheng. Translating and segmenting multimodal medical vol- umes with eycle-and shape-consistency generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 9242-9251, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006. 10738, 2020.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. arXiv preprint arXiv: I 7708.048866 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmen- tation. In AAAI, pp. 13001-13008. 2020.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- fional Conference on, 2017a.
11

---

Under review as a conference paper at ICLR 2021
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Translation Using Cycle-Consisted Adversarial Networks. Proceedings of the IEEE Interna- tional Conference Computer Vision, 2017-Octob:2242-D550 2017b. ISSN 15505499. doi: 10.1 10910CC222211.2244
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adver- sarial networks for text-to-image synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5802-5810, 2019.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A DETAILS OF EXPERIMENTS
A.1 EXPERIMENT
A.1.1 DESCRIPTION
In this experiment, we train the model to generate fake CT images.
A.1.2 HYPERPARAMETER SETTINGS
Name
Value
Description
Optimizer
Adam
Algorithm to update model
Epoch
200
Number of total epochs to run
Learning rate (translator)
2e-4
Initial learning rate of translator
Learning rate (predictor)
le-3
Initial learning rate of predictor
Aeye
10
Tradeoff parameter for cycle consistency loss
Tradeoff parameter between translator and predictor le-3
Batch size
16
Batch size of all GPUs
Weight decay
Factor to regularize the model
A.1.3 RUNTIME & COMPUTING INFRASTRUCTURE
took about 60 hours train the model to translate MRI images to CT images on GeForce GTTX 1080 Ti x4. It took about 110 hours to train the model to translate PET images to CT images on GeForce GTX 1080 Ti x4. It took about 160 hours to train the model translate MRI and PET images to CT images on TITAN Xp x4.
A.2 EXPERIMENT 2
A.2.1 DESCRIPTION
In this experiment, train the model pure CT images.
A.2.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor regularize the model
A.2.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeForce GTX 1080 Ti x1.
A.3 EXPERIMENT 3
A.3.1 DESCRIPTION
In this experiment, train the model on MIXTURE of MRT and CT images.
13

---

Under review as a conference paper at ICLR 2021
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor to regularize the model
A.3.2
HYPERPARAMETER SETTINGS
A.3.3
RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours train the model GeForce GTX 1080 Ti x1.
A.4
EXPERIMENT 4
A.4.1
DESCRIPTION
In this experiment, we train the model on MIXTURE of PET and CT images.
A.4.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor to regularize the model
A.4.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeFFrce GTX 1080 Ti x1.
A.5 EXPERIMENT 5
A.5.1 DESCRIPTION
In this experiment, train the model on MIXTURE of MRI, PET and CT images. To stabilize the training, the model is pretrained on pure CT images.
A.5.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet18
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor to regularize the model
A.5.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours train the model GeForce GTX 1080 Ti x1.
14

---

Under review as a conference paper at ICLR 2021
A.6
EXPERIMENT 6
A.6.1
DESCRIPTION
In this experiment, the baseline method transfer learning(TL) is applied to MRI and CT images. The hyperparameter settings for pretraining and finetuning are the same.
A.6.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor regularize the model
A.6.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeForce GTX 1080 Ti x1.
A.7
EXPERIMENT 7
A.7.1 DESCRIPTION
In this experiment, the baseline method Domain-Adversariil Neural Network (DANN) (Ganin & Lempitsky 2015) is applied MRI and CT images.
A.7.2 HYPERPARAMETER SETTINGS
Name
Value
Description
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs run
Learning rate
le-5
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
le-3
Factor to regularize the model
A.7.3 RUNTIME & COMPUTING INFRASTRUCTURE
took about hours to train the model on GeForce GTX 1080
A.8
EXPERIMENT 8
A.8.1 DESCRIPTION
this experiment, the baseline method multitask learning (MTL) is applied MRI and CT images. To stabilize the training, the model is pretrained pure CT images.
A.8.2
HYPERPARAMETER SETTINGS
A.8.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours train the model GeForce GTX 1080 Ti x1
15

---

Under review as a conference paper at ICLR 2021
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to
Learning rate
le-4
Initial learning rate
Batch size
32
Batch size of all GPUs
Weight decay
Factor to regularize the model
Amri
0.05
Factor for task (training MRI images)
Act
0.95
Factor for task2(training CT images)
A.9
EXPERIMENT 9
A.9.1
DESCRIPTION
In this experiment, fake CT images translated from MRI by CycleGAN (Zhu al. 201 7a) are combined with real CT images to train the model.
A.9.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
Batch size of all GPUs
Weight decay
Factor regularize the model
A.9.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeForce GTX 1080 Ti x1.
A.10
EXPERIMENT 10
1010.1
DESCRIPTION
In this experiment, the baseline method transfer learning(TL) (TL) is applied to PET and CT images. The hyperparameter settings for pretraining and finetuning are the same.
A.10.2
HYPERPARAMETES SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
32
Batch size of all GPUs
Weight decay
Factor to regularize model
A. RUNTIME & COMPUTING INFRASTRUCTURE
took about hours train the model GeForce GTX 1080 x1.
A.11 EXPERIMENT
A. A.11 DESCRIPTION
In this experiment, the baseline method domain adaptation(DA) is applied to PET and CT images.
16

---

Under review as a conference paper at ICLR 2021
A.11.2 HYPERPARAMETER SETTINGS
Name
Value
Description
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
32
Batch size of all GPUs
Weight decay
Factor regularize the model
A. 1..3 RUNTIME & COMPUTING INFRASTRUCTURE
took about hours to train the model on GeForce GTX 1080 x1
A.12
EXPERIMENT 12
A.12.1
DESCRIPTION
this experiment, the baseline method multitask learning (MTL) is applied to PET CT images. To stabilize the training, the model is pretrained pure CT images.
A.12.2
HYPERPARAMETES SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
32
Batch size of all GPUs
Weight decay
Factor to regularize the model
Amri
0.1
Factor for task (trainngg MRI images)
Aat
0.9
Factor for task2(training CT images)
A. 1..3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeForce GTX 1080 Ti x1.
A.13 EXPERIMENT 13
1.13.1 DESCRIPTION
In this experiment, fake CT images translated from PET by CycleGAN Zhu et al. combined with real CT images to train the model.
2017a)
A.13.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet18
Model for training Algorithm to update model
Optimizer
Adam
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
Batch size of all GPUs
Weight decay
0
Factor to regularize the model
A. 13.3 RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours train the model GeForce GTX 1080 Ti x1.
17

---

Under review as a conference paper at ICLR 2021
A.14
EXPERIMENT 14
A.14.1 DESCRIPTION
In this experiment, basic data augmentation methods are applied train the model.
A.14.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet50
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number total epochs to run
Learning rate
le-3
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor regularize the model
A. 14.3
RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours to train the model on GeForce GTX 1080 Ti x1.
A.15
EXPERIMENT 15
A.15.1
DESCRIPTION
In this experiment, AutoAugment method is applied train the model.
A.15.2
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet18
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
32
Batch size of all GPUs
Weight decay
Factor to regularize the model
A. 15.3
RUNTIME & COMPUTING INFRASTRUCTURE
It took about hours train the model GeForce GTX 1080 Ti x1.
A.16
EXPERIMENT 16
A.16.1
DESCRIPTION
In this experiment, random erasing method is applied to train the model.
1.1
HYPERPARAMETER SETTINGS
Name
Value
Description
Model
Resnet18
Model for training
Optimizer
Adam
Algorithm to update model
Epoch
100
Number of total epochs to run
Learning rate
le-4
Initial learning rate
Batch size
64
Batch size of all GPUs
Weight decay
Factor to regularize the model
18

---

Under review as a conference paper at ICLR 2021
A.16.3 RUNTIME & COMPUTING INFRASTRUCTURE
took about hours to train the model on GeForce GTX 1080 x1
B ADDITIONAL EXPERIMENTAL RESULTS
Figure shows more medical images from different modalities (i.e., CT, MRI and PET). Some fake images are generated from other modalities using our method (DUIIT) and CycleGAN (Zhu al. 7017a).
Real CT
Real MRI
MRI (DUIIT)
MRI CT (CycleGAN)
Real PET
PET (DUIIT)
PET (OycleGAN)
Figure 4: More real and fake images from different modalities(CT, MRI and PET). The fake images are generated using our method (DUITT) and CycleGAN.
C SIGNIFICANCE TEST BETWEEN METHODS
The significance test based on student test is done between results gain from our methods and other methods. The significance threshold is set O.1. Table 6 shows the test results between
MIX-MRI-CT and other methods. Table 7 shows the test results between MIX-PET-CO and other methods.
19

---

Under review as a conference paper at ICLR 2021
Table 6: Significance test between MIX-MRI-CT and other methods
PURE-CT
TL
CycleGAN
SimpleAugment
AuroAugment
RandomBrasing
P-value
0.020
0.017
0.000
0.010
0.013
0.013
0.000
0.089
Different distribution or not
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Table 7: Significance test between MIX-PET-CT and other methods
PURE-CT
TE.
MTL
CycleGAN
SimpleAngment
AutoAugment
RandomErasing
P-value
0.026
0.001
0.015
0.000
0.003
0.025
0.000
0.043
Different distribution not
Yes
Yes
Yes
20


Reviews:
[{"review_id": "z0okLeUou9m", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an algorithm to enlarge the training set for image classification problems in certain medical applications where training data of the target modality is scarce. They do so by training an unpaired image-to-image translator network and an image classifier end-to-end in order to utilize labeled images acquired through various imaging modalities. Moreover, they demonstrate the effectiveness of the proposed algorithm via extensive numerical experiments on a prediction problem and compare their method to several different approaches ranging from transfer learning to data augmentation.\n\nPositives:\n+ The problem is well-motivated. Obtaining large medical image datasets is difficult, therefore it is very important to find ways to use the available data as efficiently as possible.\n+ In most parts, the paper is well-written, clear and easy to follow.\n+ Authors compared their method to a wide range of other possible techniques.\n+ The experimental results are very promising for the given prediction problem.\n\nNegatives:\n- The novelty of the paper seems to be somewhat limited. Using unpaired image-to-image translation networks for cross-modal medical image synthesis has been studied in [1, 2] (as the authors pointed out in the paper). The only novelty compared to [2] seems to be that authors apply the idea to an image classification problem instead of segmentation. Authors claim that part of the novelty is that in their method translation is guided by the predictive task, however in [2] the translator is also jointly trained with the segmentation network.\n- The consistency of anatomical  structures is not enforced in any way in the translated images (see shape consistency in [2]). For instance, CT images generated from PET are very dissimilar to original PET images (Fig. 2). This might lead to loss of features crucial for diagnosing certain diseases, or even to false diagnosis via 'hallucinated' features. Medical diagnosis is often made based on very specific, fine features on the image. The experiments performed in the paper pertain to physiological age prediciton where specific fine-grained features might not be necessary for correct prediction. Therefore further experiments might be needed to judge the wider applicability of the method.\n- There is some confusion around the experimental results. First, in Table 3 some results (CycleGAN for MRI-CT and DA for PET-CT) of competing methods are worse than simply training on the CT images (PURE-CT in Table 2). Second, based on the Appendix, in some experiments the backbone model is ResNet18 and in other experiments it is ResNet50. The data augmentation experiments from Table 4 were performed using ResNet-18, whereas for the proposed method ResNet50 has been used. It is not clear how the results are comparable given the difference in network capacity.\n\nEven though the paper shows promise, given the limited novelty and somewhat questionable applicability of the presented results as detailed above, in this form I would recommend rejection.\n\nCould the authors clarify what the main contribution of the paper is compared to other techniques exploiting cross-modal image synthesis for medical applications. Furthermore, could the authors comment on my concerns about geometric distortions originating in image translation and how it may effect medical diagnosis? Lastly, please address the issues raised about the experimental results above.\n\nAdditionally, I have some minor comments and suggestions:\n- The algorithm could be better differentiated from other results in the related work section.\n- It is not clear how the discussion on text-to-image generation methods in Section 2.2 is relevant to the paper.\n- Using the same colormap for all figures for different modalities would be helpful for better comparison.\n- It is difficult to match the experimental details in the appendix to the experiments in the paper.\n- There is a typo in the title of Section 3.3\n\n[1] Chartsias et al, Adversarial image synthesis for unpaired multi-modal cardiac data, International workshop on simulation and synthesis in medical imaging, 2017\n\n[2] Zhang et al, Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network, CVPR, 2018\n\nPost-rebuttal: The authors did not provide feedback and therefore I keep my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mkW1xWf2UJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper discusses an approach to augment a medical imaging dataset using images from another modality. The images in the other (i.e. source) modality should have been originally collected & labeled to perform the same discrimination or regression task as the target modality. A network consisting of a prediction network for the mixture of augmented and target images & another network based on CycleGAN for the image translation network are jointly optimized using end to end training.  \n \nReasons for score:\nThe objective of the paper (i.e. addressing small training set size in medical imaging) is quite important, and the approach is interesting. However, the evaluation, discussion, and generalizability of the approach to other tasks require additional clarification, simulations, and extra information.\n\nDetailed review:\n1) Is physiological age in the clinic estimated from a single slice in a CT volume, or an entire volume? If latter, the entire evaluation strategy should change to include the set of images from the volume rather than single images. Otherwise, the task selected will not have any bearing on the actual clinical task of estimation of physiological age. This would also impact the approach which uses a 2D image to image translation, as it will likely mean that a 3d to 3d image translation should be used.\n2) The authors have not provided sufficient information about the distribution of attributes of the different datasets. This information is critical in assessing the generalizability of results, as well as whether the experiments were set up in a meaningful way. \na) What is the distribution of actual age in the different datasets from the different modalities?\nb) What are the image acquisition / reconstruction attributes of source/target datasets? Images from the same patient will look very different and contain varying degrees of anatomical detail depending on slice thickness, dose in CT & imaging sequence in MRI, reconstruction algorithm, etc. When doing image to image translation, does e.g. slice thickness across the modalities need to match each other?\n3) In medical imaging we often have severe class imbalance, where disease positive samples are much more rare compared with disease negative samples. Given that datasets from different modalities will likely have different ratios of positive to negative samples, how will this affect the overall training strategy? E.g. would the authors only augment the positives in this way , or both positives & negatives? Otherwise, if images from source modality have different class balance than the one in target?\n4) The proposed approach has been described for a medical imaging task that involves macro anatomical features only. It is therefore not clear whether this approach would generalize to a task that pertains to more micro features (e.g. tumor classification/detection, or tumor segmentation). Based on Fig2, it appears that the MRI images converted to CT contain obvious anatomical inaccuracies (e.g. Fig 4 shows consistently thick skulls in the generated CT images, which would affect a brain volume estimation task). Have the authors used this approach for other tasks that require more micro level anatomical precision? If not, this should be explicitly stated as a shortcoming of the current approach.\n5) The selection of hyper-parameters, experimental setups, and split of data into train/test using different approaches requires more explanation:\na) It is not clear how the lambda values were selected. Also, for eq 3 the authors state that a lambda of 0.001 was used, which seems to severely favor the GAN loss rather than the prediction loss. Given that the prediction task is the more important among the two, no justification has been provided on why the weight of the corresponding loss value would be so small compared to the GAN loss.\nb) What was the stopping criteria for the different scenarios in Table 2? In the Appendix, the authors have the number of training epochs for each approach (which is different for different experiments), but it is not clear what determined the end of training. This is important, since all approaches should be trained using the same rule to ensure a fair comparison.\nc) On p5, the authors state that they randomly select images from the CT cases to split the data into train/val/test buckets; Does this mean that images from same patient are mixed into both train/test? This would not be appropriate, since the different slices of CT data from the same patient share anatomical similarity, which means that the train/test data are likely cross-contaminated.\nd) In Table 2, what error is being shown? Is it l1 norm, l2 norm, or absolute error of predicted age relative to annotated age?\ne) What is the actual loss function in eq 2? The loss for L_adv has been described on p6 but not the loss in eq2. Also, predicting age is a regression task, and not a discrimination task. Why is eq2 described as showing a discrimination task?\n6) In Table 3, why is the cyclegan result  significantly worse than other methods for MRI (it actually deteriorates the performance compared to baseline)? On the other hand for PET, cyclegan is better than other methods; The opposite trend is happening for domain adaptation (i.e. worse for PET and better for MRI). Is this related to the amount of training data? Otherwise, this would seem to indicate that perhaps cyclegan did not train properly due to implementation issues, etc.\n7) In Figs2-4, visual comparison of the translated images to real CT images have been provided. However, since unpaired data was used, it is not clear how anatomically correct the translated images are. It would be best if for a small number of samples that have paired data (i.e. patient was imaged using both CT and MR, or CT and PET), the authors show a comparison of the translated images from MR or PET compared to the actual patient images in CT. Such a comparison would show beyond doubt that the translated images are anatomically correct or not.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gaFPxfHMdzJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the problem of learning a predictor from a specified modality using a dataset where each example has images from only one modality. The proposed approach is to set up a CycleGAN to translate between the modalities and a predictor from the required modality to the prediction target. The authors propose to learn the CycleGAN and the predictor parameters jointly. Experimental results show that this performs better than either not doing the translation at all (i.e. only learning the predictor from examples with the required modality) or learning the translation followed by learning the prediction.\n\n\nStrengths:\n+ Interesting problem setting. I find the medical imaging use case convincing.\n+ The proposed idea is simple and clean\n+ Empirical evaluation uses a comprehensive set of baselines\n\n\nConcerns / weaknesses:\n- The technical novelty in this paper is unclear. It has been generally known that an end-to-end approach typically works better than learning the two parts (CycleGAN-translator and predictor) separately.\n- Authors state that they randomly split images into training, validation and test sets. However, since there are ~30 images per patient, this could result in images from the same patient ending up in training as well as test sets. I imagine nearby slices of the 3D volume represented by a single CT-scan to be very similar to each other, and so this raises a serious concern about train/test overlap.\n- Related to above (and even after using per-patient splits), the std-dev of the prediction errors needs to be estimated using independent samples (i.e. from different patients), since samples from the same patients would not be independent. It is unclear how these correlations were handled.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8PxXXzrE58r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines the Cycle-GAN with the predictor for downstream task to augment the training data in the target modality. On two applications, the experimental results illustrate the effectiveness of their method.\n\nStrengths:\n(1) The paper is very well-written and easy to be understood.\n(2) Compared with the baselines, the proposed method significantly improves the performance of downstream regression task.\n  \nWeakness:\n(1) The comparison between the baselines and proposed method is not fair. For example, Data augmentations methods only use the target dataset, but the proposed method use both the source and target datasets.\n(2) The combination between the Cycle-GAN and downstream predictor is not used at first. [1] has introduced the segmentor to constrain the generated images. The difference is that the proposed method is applied to regression task and [1] solved the segmentation task.\n \n[1] Zhang, Zizhao, Lin Yang, and Yefeng Zheng. \"Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018..\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for data augmentation by cross-modal data generation. While the reviewers agree that the paper addresses a relevant and important problem in medical imaging, they also agree on that the paper has limited novelty over the state of the art. Also the setup of experimental validation to comparison methods is questioned.
Decision: Reject