03/01, 2023

Image quality enhancement of medical images by use of deep learningwith a small amount of training data

Sho Ozaki
Hirosaki University

Interdisciplinary Science Conference in Okinawa (ISCO 2023) - Physics and Mathematics meet Medical Science -My career trajectory
2011. PhD (Physics) in Osaka Univ.
2011-2012. Univ. of Tokyo & Riken
Nuclear &Hadron physics

2012-2013. Yonsei Univ. (South Korea)
Postdoc

2013-2015. KEK
2015-2017. Keio Univ.
Medical Physics(Deep learning
in medical imagings)

2017-2021. Univ. of Tokyo Hospital

Medical physicist

2022-. Hirosaki University

Assistant Prof.

Medical physicist
Conventional 3D-CRT

IMRT(Optimization calculation)

By using the optimization calculation, the intensity modulated radiation therapy (IMRT)
results in a highly conformal dose distribution which covers the target volume with avoiding
normal tissue exposure.

Radiotherapy equipment in Univ. of Tokyo Hospital
Elekta Synergy

Accuray Tomotherapy

Cone-beam CT (CBCT)

Mega-voltage CT (MVCT)Radiotherapy units:

CT for registration:

For precise registration based on image guidance, the image quality of CTs for registration must be
ensured. However, the image qualities of these CTs are considerably lower than a normal CT for diagnosis
and a planning CT for radiotherapy.
We have studied image quality enhancement MVCT by using deep learning.

CautionsDifficulties in medical fields
ü Large amount of data for training
The high cost of acquiring a large data set is limitation to their utilization
in medical fields.

ü Particularity of medical images• Safety• Stability• Structure preservation in images
Of the 2,212 AI-based COVID-19 image diagnostic studies that were published in 2020,
none of them are suitable for clinical use.
(Roberts et al. Nature Machine Intelligence 2021)

Problems in previous studiesl Deep learning models such as GAN is developed fornatural image processing.
create variety of images
change original structure and shape
This is undesirable in medical imaging

input

outputgroundtruthl Assessment in the absence of the ground truth.

Solutionsü We extend and improve CycleGAN with several losses for structure preservation.
ü We propose several metrics for evaluating image quality without the ground truth.

CycleGAN
L1 loss

Low quality
CT images

Enhanced
CT images

Low quality
CT images.

High quality
CT images

CycleGAN
L1 loss

Low quality
CT images

Enhanced
CT images

Low quality
CT images.

High quality
CT images

CycleGAN

Structure preservation and stability conditions,such as Perceptual loss and Autoencoder loss,
which are need for medical images.
L1 loss

Low quality
CT images

Enhanced
CT images

Low quality
CT images.

High quality
CT images

200

where C is a constant equivalent to598 HU. This loss ensures that the generators do not201

alter the regions having values less than

598 HU to preserve the air-body boundaries.

Perceptual
loss
The
perceptual loss is defined as
202

203

Lpercep =

204

1
whncl

X

x2MV

|| (GMV!kV (x))(x)||2 +X

y2kV|| (GkV!MV (y))(y)||2

𝜙 : Output of the second layer of pretrained VGG with ImageNet database

!

,

(15)

205

where

is the output of the second convolutional layer of the VGG networks
pre-trained
MV : MVCT
imagesby206

the natural images derived from the ImageNet database. w, h, ncl denotekVthe
width,
height,images: kVCT(PlanCT)207

and number of channel of this layer, respectively. The second (and, in general, a shallow)

208

layer of the VGG network is known to learn low-level image features such as edges. The

209

perceptual loss helps to preserve the boundary of di↵erent regions in the CT image.

VGG

II.B. MVCT to kVCT conversion with extended GANs
T;2 3

Examples of 𝜙 outputs

aX PxFB 2i HX

6B;m`2 j, 1tKTH2 Q7 i?2 QmiTmi Q7 i?2 b2+QM/ Hv2` Q7 o::Re 7Q` M Jo*h BK;2 Q7
TiB2Mi O RX
kk9

r?2`2

!

LEARNING CT CONVERSION WITH SMALL DATA

3775

LEARNING CT CONVERSION WITH SMALL DATA

24734209, 2022, 6, Downloaded from https://aapm.onlinelibrary.wiley.com/doi/10.1002/mp.15626 by Cochrane Japan, Wiley Online Library on [15/02/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed

The autoencoder
loss
strengthens
the cycle
consistency
conversion,
such
that
the information
thetheinput
imageofandhence, adding
loss
terms generally
contributes
toward inizes
channels
𝜙(x)its
fo
loss tostability.
enforceMoreover,
the reversibilityof thetheconversion,
suchthe wheretheto suppress
alteration
patient
They represent lo
equivalent.
The total
variationofloss
is defined
as1.hyperparametersthat theinformationin theowing
inputtoimage
and its latentfollows:
anatomical
structure
the adversarialloss, we as asedges, and the boundarX
representationequivalent.
require anare
additionalloss The
term.total variation loss ishence, the body contour is
Ltv =||grad(GMV!kV (x))||1 ,definedasThe
autoencoderloss
ismaintaining
a low𝜆autoenc
perceptu
𝜆cycle = 10.0,=1
Auto-encoder loss

vation of the structures be∑∑
Decoder 𝜆 = 1.0,
𝜆tvEncoder= 0.001,
images.
where
is
the
image
gradient.
Owing
to
this
loss, the
gene
D
||x
− Decgrad(Enc(x))||
autoenc||grad(G(x))||,(7)
tv ==
latent→MV
MV→latent
1
MV→kV
1
Consequently, the loss fu
x∈MV
x∈MV
as
∑ uniform images. The air-region loss isAdefinedand decoders
defined byn
schemaof theisproposed+||y − Declatent→kV (Enc
XkV→latent (y))||1 .
X
x2MV

L1 Loss

where grad is the image gradient. Owing to this loss, theis presented
in Figure 2.
Generatory∈kVLair =|| (GMV!kV (x))(x)||+|| (G+kV!
G 1=to𝜆kVCT
𝜆aucycle cycle
generator produces spatially uniform images. The air- (6)In CBCT
conversi
x2MV
y2kV
region loss is defined as
the following +𝜆two loss+terms.
𝜆air ai
Air region loss
tv tv
where
∑
The autoencoder
loss strengthens the cycle consistency tion was defined by
(
air
loss
to =
enforce||𝜓(Gthe reversibilityof 𝜓(x)||the conversion,
such
MV→kV (x)) −1
where the∑hyperparameterxone=week,(ifwhichxwe||G<interval
C),after −
anGinterval(x))||of more
model
after
an
of more than one we= ||grad(x − G,(16)
DIG(x) model= ||grad(x, than(16)x∈MV(x) − G
that the informationin theDIG(x)input
image(x))||anditsbelieve
latent
idem
MV→kV
as
follows:is(x)sufficiently =long to nullify the effectbelieveof theisdocsufficiently
long to,nullify the effec
tors’MVCT
memory.
For theevaluation,
we adopted
thememory.
PlanCT For the evaluation, we adopted
where x is the input MVCT image. The DIG
wheremeasuresx is the input
image.
The
DIG measuresx∈MV
0pro-day astors’
(if
x
C),
∑are equivalent.
representation
total
variationloss
isimageson thesame(kVCT)
the MVCT
images acquired on the same day athe change in the The
pixel valuesof the original
the change
and pro-in the(kVCT)
pixel values
ofacquired
the original
andimagesforthesamepatient.
3D
rigid
registration
images
for
from
the
same
patient. 3D rigid regis
images(y))
in the gradientdomain, thus
cessedfacilitating
images in (8)the gradient domain, thus facilitating||𝜓(Gcessed−
𝜓(y)||
,
∑
kV→MV
1
the
PlanCT
imagestothe
MVCT
imagesthewasPlanCT
appliedimagesto
MVCT images
defined as+
an evaluation of the preservation of edges.an evaluation of the preservation of edges.
𝜆 average=
10.0,
𝜆theautoenc=−
1
theWe considered
the avusing a 3D Slicer. We considered thecycle
using
aof3D
Slicer.
+
||G
(y)
𝐶
=
−600
HU
kV→MV
y∈kV where C is a constant equivalent
to
598
HU.
This
loss
ensuresthseven contours on the registered PlanCT sevenimagescontours
proon the registered PlanCTF I G U R E 4 Visual comparison of image quality among
F I G UMVCT,R E 4 non-local
Visual comparison
means denoisingof image(NLM),
quality
non-local
amongmeans
MVCT,denoising
non-local withmeans
histogramdenoising (NLM), non-local means denoising
matching (NLM+HM), processed MVCT by data-full, andmatching
reference(NLM+HM),kVCT images
processedof patients
MVCT#1by (first
data-full,row) andand reference#2 (secondkVCTrow).images
A tiny structureof patients #1 (first row) and #2 (second row).
in the muscle of patient #1 is highlighted (third row). The
in thekVCTmusclewasofacquired
patient on#1 a
is different
highlightedday.(third
The displayrow). The
window
kVCT range
was acquiredis set ason(−300,
a different300) day. The display window range is set
HU
HU

MV→kV

2

MV→kV

2.4
Clinicalevaluation
vided
by the doctors and regarded the average
videdcontourbythe doctors and regarded the ave∑ 2.4 Clinical evaluationy∈kV
as the ground truth. The average of the contours
as thegroundreferstruth. The average
the co𝜆=0.001,
𝜆Dthe=ofof1.0,
tvto||grad(G(x))||, of therelevance(7)
tv = alter The
clinicalregions
relevanceof the quality
improvement
Thethequality improvement
of HU
the
to theofbinarizationof the598
averageof binary
totheimagesbinarization
atof the average
binar
the
with
valueslessthanpreserveai
MV→kV
1clinical
19

where

2

19

processed images was assessed through processed
a contouring
imageswas assessed
througha contouring
a thresholdof 0.5.
We computed
the Dicea coefficientsthreshold of 0.5. We computed the Dicex∈MV task. Seven MDs of radiotherapy from the task.
University
SevenofMDs betweenof radiotherapy
from the
University
of MVCTthe contours
in theprocessed
betweenand thethe contours in the processed MV
These loss functions properlymodelparameters,
andthegradient
wasdeTokyoregularize
Hospital contoured thetwo parotid
glandsTokyo Hospitalof the contoured
the two parotid
glandsofground truth,and
those
betweenthethecontours
groundin thetruth,and
orig- loss
those between
the contou
A
schemaoftheproposedfour patient images in the validation datafour
set,patient
includ-images
the validation
data
set,
includinal in
MVCTand the groundtruth.
inal MVCTandthe groundtruth.
which leads to structureand stability
with
small
amountof training data.
The
perceptual
lossmodel,
isorig-definedas{preservation
ing the processed MVCT by the proposeding
the processed
MVCT by the proposed model, orig-

differenceof thegradients
co
MVCT,
and <
PlanCT(kVCT)
images. toIninaltotal,
MVCT,
eightand
PlanCT the(kVCT) images. is
In total,eight
where grad is the image
gradient.
Owing
this
loss,
presented
inFigure2.
x inal(ifxC),
regions of interest (ROIs) were prepared underregions
fourofdifinterest3 (ROIs)
were prepared under four difRESULTS
3
RESULTS
images.
The
idempotenc
ferent modalities uniform
and contoured by images.
sevenferent
doctors.
modalities
To Theandcontoured
byput
seven doctors.
To(9)𝜓(x) = 0X
Xwith
generator produces
spatially
airIn
CBCT
toComparisonkVCT
convers
1 artifact in the
avoid(ifthe regionof the
metal
avoid
oral
thecavregion 3.1of the metal
artifact in theoralconventionalcav3.1Comparison
withconventx≥C),
2
ity, approximately
50% of the volume of ity,
each
approximately
parotid|| denoising
50% of the volumeof(x))
each parotid
denoisingformalrequirement
for
the
m
L
=
(G
(x)||
+
||
(G
MV!kV
region loss is definedpercep
asthefollowing
two loss terms
was used. We designed the experiments aswas
follows.
used.TheWe designed theexperiments asfollows.
The
whnthecl(1) MVCTMDsof were
MDs were asked to contour
all fouraskedFigure
to contour
the (1)
all four means
4 shows
theMVCT
MVCT,of nonlocal
Figuredenois4 shows the MVCT, nonlocal me
ators
and
helpedimprove
th
y2kV
tion
was
by
patients. After finishing the contouring, thex2MV
patients.
images were
After finishing
the contouring,
imageswere
ing (NLM),
non-local themeans
denoisingwithingdefined
histogram(NLM), non-local
means denoising wit∑
taken away and they were asked to contour
taken(2)away
pro-and matching
they were(NLM+HM),
asked to contour(2) proprocessed
MVCT matchingby the pro(NLM+HM), processed MVCTT;2 RkaX PxFB 2i HX

Visual comparison with conventional denoising methods
MVCT (Input)

Conventional Denoising
Conventional Denoising w/ histogram matching Proposed model

S. O., et al., Med. Phys. 49 (6) (2022)
kVCT (Plan CT)

6B;m`2 9, oBbmH +QKT`BbQM Q7 BK;2 [mHBiv KQM; Jo*h- MQM@HQ+H K2Mb /2MQBbBM;

r?2`2(NSelfSSIM),
aaAJ Bb i?2definedbi`m+im`H
bBKBH`Biv BM/2t K2bm`2 M/ N (0, σ ) Bb i?2
s, we propose a new metric, noised k3y
SelfSSIM
as
2

k3R

bBM /Bbi`B#miBQM rBi? σ = 20 >lÞ X h?2 `iBQMH2 Bb i?i B7 i?2 BK;2k3k

Bb bB;MB}+MiHv /Bz2`2Mi BM i2`Kb Q7 i?2 aaAJ r?2M i?2 :mbbBM MQBb2 B

NSelfSSIM(img) = SSIM(img, img + Gaussian noise with

= 6),

(16)

S. O., et al., Med. Phys. 49 (6) (2022)

M is the structural similarity index k3j
measure.
rationale 2t?B#Bib
is that if theoriginal
BM;Hv-The
La2H7aaAJ
HQr2`
pHm2b B7 i?2 BK;2 ?b  HQr2` MQBb2 H2p2H

Proposed evaluation metrics

has low noise, it is significantly di↵erent
in terms of the
SSIM when
Gaussiank39[mMiBiiBp2Hv
BM/B+i2
i?2 MQBb2`2/m+iBQM BM i?2 T`Q+2bb2/ BK;2bX

ded. Accordingly, NSelfSSIM exhibits lower values if the image has a lower noise[mMiB7v
i?2processedbi`m+im`HT`2b2`piBQM- r2 BMi`Q/m+2/ i?2 /Bz2`2M+2 B
metric can quantitatively indicate k38
the noise hQ
reductionin theimages.
k3er?B+? Bb /2}M2/ b

Metric measuring noise reduction

Metric measuring structure preservation

NSelfSSIM(img) = SSIM(img,
k3dimg + Gaussian noise),

DIG(x)(17)= ||grad(x − GMV→kV (x))||2 ,T;2 ReaX PxFB 2i HX

antify the structural preservation, we introduce two other indexes, namely, thek33

r?2`2 x Bb i?2 BMTmi Jo*h BK;2X h?2 .A: K2bm`2b i?2 +?M;2 BM i?2

Proposedn gradient (DIG) and cycleNSelfSSIM
consistency error (CCE). The DIG is definedas modelk3NOriginal CycleGAN

Q`B;BMH M/ T`Q+2bb2/ BK;2b BM i?2 ;`/B2Mi /QKBM- i?mb 7+BHBiiBM; i?2

MVCT

Proposed

PlanCT

0.90240.70790.7263

Lgrad (x) = ||grad(xGMV!kV (x))||2Q7, i?2 2/;2bX
kNy
T`2b2`piBQM

MeanT;2 Re(18)

ithub.com/shizuo-kaji/UnpairedImageTranslation

SD

0.01060.0794 Þ

0.0825
q2 ?p2 +?Qb2M σ = 20 >l b i?2 KBMBKmK bi`2M;i? Q7 +HBMB+HHv `2H2pMi MQBb2X
rBi? σ = 10 >l II.C.
M/ σEvaluation= 30 >lX methods
h?2 #bQHmi2 pHm2b r2`2 /Bz2`2Mi #mi `2HiBp2 #2?
TERIALS AND METHODS
KQ/2Hb rb b22M iQ #2 bBKBH`- b?QrBM; i?2 `Q#mbiM2bb Q7 i?2 K2i`B+X
Noise reduction

AAX

Jh1_AGa L. J1h>P.a

6B;m`2 3, oBbmHBxiBQM Q7 .A: U1[X UReVV 7Q`  b2H2+i2/ bHB+2 7`QK
TiB2Mi OR T`Q+2bb2/ #v i?2 Q`B;BMH *v+H2:L rBi? λ!/p = 1.0 UH27i
!

AAX*X

ZmMiBiiBp2 2pHmiBQM

6B;m`2 3, oBbmHBxiBQM
TiB2Mi OR T`Q+2bb2/ #v

S. O., et al., Med. Phys. 49 (6) (2022)

Data size dependence of DIG
†
†

†

DIG

†

Statistical analysis
Statistical analysis is performed
by Fridmanʼs test (p < 0.05) witha post hoc Shirleyʼs test.
The dagger † indicates p < 0.05
in Shirleyʼs test.

Data sizeSummary
We develop a novel low-quality to high quality image translation model based on
deep learning, which can be trained using only a few hundred unsupervised images.

We evaluate the performance of our model with proposed metrics, and show that
our model outperforms CycleGAN in terms of structure preservation.