Construction of a Multi-modal Model of Pancreatic Tumors
by Integration of MRI and Pathological Images using Conditional Cycle α-GANTomoshige Shimomura1 , Mauricio Kugler1 , Tatsuya Yokota1 , Chika Iwamoto2 , Kenoki Ohuchida2 ,Makoto Hashizume2 , and Hidekata Hontani1
1

Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi, 466-8555, Japant.shimomura.525@nitech.jp, {kugler.mauricio,t.yokota,hontani}@nitech.ac.jp
2
Kyushu University, 744 Motooka Nishi-ku, Fukuoka, 819-0395, JapanAbstract
In this study, we constructed the multi-modal model ofa pancreas tumor by learning the correspondence between
MRI voxels and pathology image patches with a conditional
cycle α-GAN. In this framework, we constructs encoderdecoder networks that translate MRI voxels and pathology
image patches each other, and two discriminators for bothmodalities. When a voxel in a pancreas tumor region in
an MRI image is selected, this model can generate various
corresponded pathology image patches non-invasively, andvice versa. We made a training dataset by registrating between an MRI image and a 3D pathology image and trained
our multi-modal model. Using trained model, anyone canobserve the change of pathology image with respect to theMRI values, and we found the behaviors are closely related
to the growing process of pancreas tumor.

1. Introduction
Modeling the correlations between pathology images
and MRI images has been investigated. The former images
can be used for definitive diagnosis and the latter images
can be obtained non-invasively. Models that represent thecorrelations of these images would improve the confidenceof diagnosis and can be used for predicting the histopathological status from the corresponding MRI images.
In this study, we construct a multi-modal model that predicts pathology image patches in the condition of the voxel
in an MRI image of a pancreas tumor of a KPC mouse. For
constructing such a model, we made a training dataset byregistrating between the tumor region in the MRI image andthe 3D pathology image reconstructed from a spatial of the2D pathology images [1] as shown in Figure 1. We created a model with a generator trained on a conditional cycle

(a)

(b)

(c)

Figure 1. Registration between the tumor region in the MRI image and the 3D pathology image. (a): An MRI image of the tumor region. (b): The corresponding pathology image. (c): Thecorrespondence between the MRI voxels and the pathology imagepatches.

α-GAN (cc-α-GAN) extended from an α-GAN [2].

2. Method
We construct a multi-modal model using a cc-α-GAN
combining α-GANs [2] and cycle GANs [3]. This GANs
consist of encoder, generator, discriminator and code discriminator with parameters η, θ, φ and ω, respectively. We
define the loss function for the encoder and the generator:
L(θ, η) = Ex∼p(x) Ey,z∼qη (y,z|x)
λ1 ||x − Gθ (y, z)||1 + λ2 ||yreal − y||22Cω (y, z)
Dφ (Gθ (y, z))
− log
− log
1 − Dφ (Gθ (y, z))
1 − Cω (y, z)
+Ey,z∼p(y,z) Eŷ,ẑ∼qη (y,z|Gθ (y,z))

λ2 ||y − ŷ||22 + λ3 ||z − ẑ||22Dφ (Gθ (ŷ, ẑ))Cω (ŷ, ẑ)
− log
− log
1 − Dφ (Gθ (ŷ, ẑ))1 − Cω (ŷ, ẑ)(1)0Encoder:Generator:: distance fromcontour

100%000: voxelvalue: voxel
value

100%

100%

: distance fromcontour

100%

Figure 3. Image matrices of generated pathology images from conditional variables. Each image matrix has own latent variable. The
patches from same latent variable have similar texture and same
conditional variable have similar style.
Figure 2. cc-α-GAN frameworkx ∈ R96×96y ∈ R2z ∈ R1024qη : x → (y, z)Gθ : (y, z) → x
DφCω

Pathology image patch
MRI voxel intensity and 2D Euclidean distance from the contour ofthe pancreas tumor region
Latent variable sampled from normal distribution
Encoder
Generator
Discriminator for x
Code discriminator for (y, z)

Table 1. The description of each symbol

every 25 % of the cumulative frequency of the each histogram and same latent variable which is randomly sampled. Figure 3 shows the result of using two different latent variables. We found the patches from same latent variable have similar texture and same conditional variable have
similar style. In addition, if y1 and y2 increase, generated
patches become pink like necrosis portions. The tendency issimilar to the characteristic of the actual images that a voxelin MRI images with high intensity corresponds to necrosis
portion because of moisture arising in necrosis. As a result,
we succeeded in pseudo reproducing the process of growing
to necrosis of tumors.

4. Conclusion
where the coefficients λ1 , λ2 and λ3 are scale parameters, and yreal is the conditional variable corresponding tox ∼ p(x). The discriminator and the code discriminator
are trained with cross entropy loss as standard GANs.
We use a training dataset made from a MRI image andcorresponding 3D pathology image of a pancreas tumor.
Each sample consists of pairs of a pathology image patch asx and the voxel information of corresponding MRI image
as y, which has the voxel intensity y1 and the 2D Euclidean
distance y2 from the contour of the pancreas tumor region in
the MRI image. The latent variable z follows the standard
normal distribution with 1024 dimensions. Figure 2 shows
the framework of a cc-α-GAN and Table 1 summarizes thedescription of each symbol.

3. Experiments and Results
We trained our networks with the dataset composed ofabout one hundred thousand samples with augmentationsfor 20 epochs using Adam optimizer and a batch size of
100. The learning rate was 5 × 10−4 and the coefficients
λ1 , λ2 and λ3 were set to 50, 10 and 10, respectively.
Using trained generator, we generate pathology image
patches from conditional variables y1 and y2 sampled from

We constructed a multi-modal model of pancreas tumorsthat can predict pathology image patches from a voxel inthe corresponding MRI image non-invasively using a cc-αGAN. Our model succeeded in pseudo reproducing the process of growing to necrosis of tumors. The future works areimproving accuracy and developing the evaluation method.

References[1] M. Kugler, Y. Goto, Y. Tamura, N. Kawamura, H. Kobayashi,
T. Yokota, C. Iwamoto, K. Ohuchida, M. Hashizume,
A. Shimizu, and H. Hontani. Robust 3d image reconstructionof pancreatic cancer tumors from histopathological images
with different stains and its quantitative performance evaluation. International Journal of Computer Assisted Radiology
and Surgery, Jul 2019.
[2] M. Rosca, B. Lakshminarayanan, D. Warde-Farley, andS. Mohamed. Variational Approaches for Auto-Encoding
Generative Adversarial Networks.
arXiv e-prints, page
arXiv:1706.04987, June 2017.
[3] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer
Vision (ICCV), pages 2242–2251, 2017.