IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on
Internet of Things and its Related Technologies
in Image Electronics

Vol. 7, No. 1 2019

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief

Mei KODAMA (Hiroshima University)

Vice Editors in Chief

Osamu UCHIDA (Tokai University)
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)

Advisory Board

Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Masayoshi AOKI (Seikei University Emeritus)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)

Editors

Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (Fuji Xerox Co., Ltd.)Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (Fuji Xerox Co., Ltd.)
Shoji KURAKAKE (NTT DOCOMO)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Koji MAKITA (Canon Inc.)
Junichi MATSUNOSHITA (Fuji Xerox Co., Ltd.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer

Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)
Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)
Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)

Munetoshi IWAKIRI (National Defense Academy of Japan)
Yuki IGARASHI (Meiji University)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya)
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ Office

Hidekazu SEKIZAWA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information

The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.7 No.1 June 2019
CONTENTS

Special Issue on Internet of Things and its Related Technologies in Image Electronics
1

Upon the Special Issue on Internet of Things and its Related Technologies
in Image Electronics

2

Fast Image Quality Enhancement for HEVC by Postfiltering via Shallow
Neural Networks

Shunichi KIMURA

Contributed Papers
Antoine CHAUVET, Tomo MIYAZAKI, Yoshihiro SUGAYA,
Shinichiro OMACHI

Regular Section
Contributed Papers
13

Deep Learning Based Uterus Localization and Anatomical Structure
Segmentation on Fetal Ultrasound Image

Yan LI, Rong XU, Artus KROHN-GRIMBERGHE, Jun OHYA,
Hiroyasu IWATA

24

An Ensemble Approach to Precancerous Cervical Lesion Classification from
Cervigram Images

Suleiman MUSTAFA, Akio KIMURA

36

Region Control in Stylized Shading Using Radial Transformation within
Texture Projection

Muhammad ARIEF, Hideki TODO, Koji MIKAMI,
Kunio KONDO

Short Paper
46

Region Mining of Fetal Head in Ultrasound Image Based on Weakly
Supervised Annotations and Deep Learning

Yan LI, Rong XU, Artus KROHN-GRIMBERGHE, Jun OHYA,
Hiroyasu IWATA

Announcements
52

Call for Papers : Special Issue on CG & Image Processing Technologies for
Generation and Post-Processing of Image/Animation

Guide for Authors
53

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan（IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Upon the Special Issue on
Internet of Things and its Related Technologies in Image Electronics
EditorShunichi KIMURA(Fuji Xerox Co., Ltd.)In recent years, a wide variety of devices designed to operate over the Internet are
increasingly coming into the market. These devices and related technologies, commonly
called Internet of Things (IoT), are expected to create new social values. Moreover, from aviewpoint of Image Electronics Technology, widespread of small optical devices in mobile
phones, automobiles, or personal computers are also regarded as IoT devices because thesedevices can be used as connected sensors to get image information. Image Electronics
Technology treating such imaging information will be one of the main fields to enhancestudies of the IoT.
At the call for submission, we received several interesting papers in this special issue.
Based on the extensive peer review processes, six papers are accepted for publication;“Diminished Reality for Privacy Protection by Hiding Pedestrians in Motion Image
Sequences Using Structure from Motion” shows a novel object hiding method, “A
Development of Wearable System for Information Providing Applications Using Scene Text
Recognition” and “Designs and Evaluation of Raccoon Detector for Anti-Vermin
Surveillance Cameras” report new systems for practical applications, “Green-Noise
Diffusion Watermarking Method Enabling to Embed Multiple Information” proposes a
sophisticated digital watermarking method embedding multiple information, and “A
Flicker Noise Reduction Method of Holographic Projected Images with the Use of the Fog
Screen” and “Fast Image Quality Enhancement for HEVC by Postfiltering via Shallow
Neural Networks” devise effective methods to improve image qualities. First five papers
are written in Japanese and already published in IIEEJ Journal in April, and the last one
written in English is included in this Transaction.
Finally, I would like to thank all the reviewers for their enormous effort in producing
careful and effective reviews for submitted papers. I am also deeply grateful to themembers of the editorial committee of IIEEJ and the staff at IIEEJ office for various kindsof support.

1

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
IIEEJ

Paper

-- Special Issue on Internet of Things and its Related Technologies in Image Electronics --Fast Image Quality Enhancement for HEVC by Postfiltering via
Shallow Neural Networks
Antoine CHAUVET† , Tomo MIYAZAKI† , Yoshihiro SUGAYA† , Shinichiro OMACHI† (Member)† Tohoku University<Summary> The study proposes a lightweight adaptive postfilter based on neural networks for use in
H.265 High Eﬃciency Video Coding (HEVC). The proposed filter is adaptive because it uses a diﬀerent
set of parameters based on encoding settings and most significantly on the quantization parameter. With
the aforementioned information, the filter most eﬃciently improves each block. We trained the filter for 4
diﬀerent QP values and we demonstrate that the use of the filter leads to a decrease in bitrate of over 4%
in a few cases and a decrease in bitrate of 1.5% on average for both All Intra and Random Access modes.
In contrast to a few filters that use several passes and require specific ordering, the proposed filter changes
each pixel at most once and the input uses only initial values, thereby allowing perfect parallelization.
Furthermore, the use of only one convolutional layer and eight feature layers maintains the computing costand memory footprint to the minimum possible extent, and this makes real-time processing possible evenon embedded hardware.
Keywords: block-based coding, video coding, video filtering, convolutional neural networks

1.

need for eﬃcient processing of data because costs andelectrical consumption should remain low. Several small
devices, such as phones and tablets, stream video fromthe Internet, and it is not surprising that a refrigerator
or an oven that performs the same will be developed ina few years. Naturally, algorithms that can eﬃciently
process video are desired. The aim of the proposed filter
is that it should be operable on the aforementioned devices, and thus it should be suﬃciently simple to respect
the processing power limitations of these devices.
As shown in the AVC in-loop deblocking filter, an understanding of the quantization parameter (QP) aids indetermining whether edges are more likely to be artifacts
or true edges and increase filtering eﬃciency. Thus, theproposed filter uses the QP value as a parameter in conjunction with the frame input.
The proposed filter runs after the decoding, and thus
the performance concerns from using an in-loop deblocking filter can be alleviated. The limitation of in-loop filters is that they should be processed in encoding orderbecause the frame is then used as a reference. This creates two issues when the filter becomes too complicated.
First, there are inherent limits on parallelization becausea frame must be filtered before commencing to process thenext frame. Second, skipping the filter when the frame is

Introduction

High Eﬃciency Video Coding (HEVC)1)is a standard ofvideo coding that is used extensively for High Definition
content. However, defects appear on videos coded via
HEVC. For example, blocking artifacts are visible defects
caused by the division in coding units. Furthermore, finedetails are typically blurred because the high frequency
components are more aggressively quantized. This canmake text unreadable at lower bitrates.
In order to reduce the artifacts caused by encoding, recent standards, such as Advanced Video Coding (AVC),
included a deblocking filter2). Specifically, HEVC is a
successor of AVC that also includes a deblocking filter3)
and a new Sample Adaptive Oﬀset (SAO) filter4). The
former exhibits a significant improvement by removing
block artifacts with the only disadvantage that it can filter excessively and blur a few details. The latter is mostuseful to reduce ringing artifacts that occur around edges
because of the high frequency and quantization. Both filters are in-loop, and this implies that the filtered frame
is used for reference in contrast to filters that process the
decoded stream and are also termed as postfilters.
Recently, the Internet of Things is an extremely popular subject of research and its popularity leads to the2IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

cant. However, in other modes, the number of filtered
blocks is significantly lower leading to less overall improvement. Filtering the whole picture was more eﬀective than only specific blocks. Furthermore, it removes
the need for preserving the encoding information because
a single parameter (QP) is used for the whole frame.

delayed to avoid dropping frames that aﬀect the qualityof all subsequent frames in the current Group of Pictures
because they rely on the correctly filtered frame for reference. Based on the key frame interval, reduced quality
can last a few seconds with errors compounding over time
until the next intra-frame. This implies that in-loop filters should be processed in a considerably short time torealize the correct real-time decoding.
With a postfilter, in the former case, all the filtering canbe performed as a completely independent process anddoes not exhibit any temporal dependencies, and thus it
possible to use separate processing units for each framein parallel. In the latter case, if processing cannot becompleted in time, then the current picture is left partially unimproved although other pictures do not suﬀer
any degradation and can be accurately filtered provided
that suﬃcient processing capacity is available for them.
In practice, a few frames that are more expensive to decode can use excessive processing power to allow filtering
although other frames can run the filter as intended.
The proposed filter operates independently on eachchannel. Hence, only luma can be processed if the computing power is limited such as in the case of embedded
or mobile devices. This oﬀers most of the gains for alower cost. The neural network-based architecture allows
changes in the number of filters and allows even the lesscapable devices to improve to a certain extent. Scalability
is important because it allows a larger adoption rate andallows implementers to determine the eﬃciency/cost ratio that optimally works for them. For example, the proposed filter uses eight feature layers although four layers
still exhibit improvement for half of the processing cost.
Over eight feature layers exhibit little improvement andprevent real-time processing on most hardware, and thus
eight feature layers correspond to a good compromise. As
opposed to aiming to obtain a higher eﬃciency than thatof image reconstruction state of the art approaches, theaim of the proposed filter is to be practical and usable inseveral devices.
The improvement of the in-loop filters is significant because they use additional information for increased eﬃciency, and thus the aim of the proposed filter is to supplement them and not to replace them. Previous studies5)ona filter that used the same information as the in-loop filter indicated that even in the All Intra case (wherein the
improvement from a frame helping the following frames
does not exist), the use of both filters exhibited optimal
results. The improvement for filtered blocks was signifi-

2.

Convolutional Neural Networks in Image Filtering

The proposed filter uses Convolutional Neural Networks (CNN) to perform filtering. Specifically, CNN isthe leading method in image recognition competitions.
CNN were also used in image filtering before to remove
Gaussian noise from images6),7), pulse noise8)or for upscaling9)–12).
In Jain et al.6), the CNN-based method achieved extremely impressive restoration of the original picture.
However, the deep structure with up to six layers makes
it considerably slow. This type of a deep architectureis still too complex for the speed required by real-timevideo filtering that our filter aims to achieve. Zhang etal.7)improved the aforementioned method although it stillrequired 60 ms to process an image with a GPU. It is possible to use this for video processing although it requires
expensive hardware.
Du et al.8)oﬀered impressive reconstruction of images
degraded with pulse noise. The Median Filter alreadyoﬀers good reconstruction because the degradation is extreme although it does not aﬀect every pixel: most of theinformation is still easily extracted. The proposed filter
extracts the information more eﬃciently than the medianfilter and avoids blurring the image.
Jampani et al.9)use bilateral filters inside CNN. The filter used sparse and high dimensional convolutions. It exhibited significant results for both color upsampling anddepth upsampling.
Shi et al.10)used a learned filter to perform upscaling
and achieves real-time processing of 1080p videos on aGPU. It used deconvolution to increase the size of the
input.
Lai et al.11)obtained better results than Shi et al.10)
by using a more complex architecture and maintaining
upscaling with deconvolutions while also adding a feature
extraction branch wherein results were then merged into
the deconvolution result.
Dong et al.12)discussed an interesting network that canperform real-time processing (above 40 fps) on a CPU
although this was for images that were 240×240 pixels,

3

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）high-end CPU or half that on a GPU for a resolution of
only 176×144. With respect to FullHD (1080p) video,
which is now considerably common, it takes 80 times asmuch time, making it considerably unpractical. Even additional processing power makes real-time processing unlikely. Hence, although it oﬀers improvement over Parket al.14)in terms of its general nature that allows it towork on all three components instead of only luma, it isimpractical for actual use due to its speed.

and it was not suitable for high definition content because
the cost increased with the number of pixels.
However, none of the aforementioned approaches areused for image degradation specifically resulting from encoding. Artifacts from encoding are considerably specific,
and thus a filter trained specifically for removing encoding noise is necessary. Super resolution and other upscaling methods use low-resolution pictures that are simply
downsampled occasionally with Gaussian noise althoughnothing similar to the loss results from encoding. The
computing cost of these methods is also excessively highfor the real-time processing of HD content without using
expensive hardware.
However, a current few studies focused on alternative in-loop filters or post-processing methods for HEVC13)–15). The issue with these methods is there are limitations despite a few improvements.
Li et al.13)detailed an overall reduction in bit rate of
1.6% although it used an extremely complex network of
20 layers. It used diﬀerent models for diﬀerent values of
QP, which required exporting some metadata while decoding or integrating directly with the decoder. Although
the need decreased while using constant QP, it filteredonly the luma component and appeared to use the same
sequences in training and testing. While they were encoded diﬀerently, it is likely that the results improved
because of the large similarities between the training andtesting sets.
The study by Park et al.14)was based on super resolution CNN (SRCNN) trained on degraded images thatwere reconstructed via the decoder. It used diﬀerent
models for variable values of QP to provide adequate filtering for the images. It exhibited an impressive reduction of 4.8% for All Intra mode and between 1.6 and 2.8%
with Low Delay P and All Intra. The computing timeexceeded13)with 1.2 s per frame on 832×480 sequences,which corresponded to one frame per 5.2 s for FullHD.
Furthermore, it exhibited the same limitations as13)because it used the same training and testing sequences.
Dai et al.15)revealed a 4.6% reduction in bitrate overmany sequences. The results for All Intra are interesting, although of little use in real applications that use
inter-picture prediction and where in-loop filtering oﬀers
large gains. In-loop filters are disabled, and thus runningthe filter only during post-processing is likely to lead to
extremely low to negative improvements on the more realistic Random Access configuration. It is also considerably slow and takes almost one second per frame on a3.

Proposed Method

3.1 Input and output blocks
The proposed filter receives 12×12 pixels blocks as input. As significant part of the noise introduced during
encoding is situated around block boundaries (and thus
the need and improvement is oﬀered by the deblocking
filter), each block is centered over an 8×8 grid. For example, the first filtered block is centered around (8, 8),
the second is centered around (16, 8), and the series continues. Figure 1 shows how the blocks are aligned. The
position is selected to place the HEVC 8×8 block boundaries immediately in the middle on the input block andoﬀer suﬃcient context information for filtering.
The input blocks overlap, and this is extremely helpful to avoid potential blocking artifacts at the edges of
the 8×8 output block because smaller 4×4 blocks exhibit edges overlapping the 8×8 block edges. The figure
shows only two input blocks to avoid confusion because
the overlapping makes it diﬃcult to specify where eachblock ends. It is important to use 12 pixels for the blocks
to provide information on neighboring pixels from bothsides. We use the same block size as the output to significantly reduce the eﬃciency, and thus 12 pixels is a goodcompromise between paucity of information and excessive
information that results in a significantly higher process-Fig. 1 Representation of filtered blocks (in red) and input blocks (in green)

4

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）is defined as follows:

ing time.
With respect to the output, the filter aﬀects only the
8×8 closest to the center. The output blocks do not overlap, which is extremely important because it allows parallel processing. It also covers the complete picture andpotentially aﬀects every pixel while the existing in-loop
filters mainly aﬀect the two pixels closest to the blockboundary while other pixels are significantly less aﬀected.
The output targets the residual, namely the diﬀerence between the original image and reconstructed image because
it makes training easier. The input is also normalized byremoving the average value of the block for the same reasons.
The filter also uses the same architecture for chromaeven while using chroma subsampling. Smaller blocks,
such as the ones used in the HEVC deblocking filter, did
not show good results.
The filter requires a larger block for input than output,
and thus the four pixels closest to the edge are excluded
from the filter. Although this can limit the eﬃciencyof the filter and especially for chroma while using 4:2:0
subsampling, the edges are typically less important forvisual quality because attention is focused on the centerof the image. Furthermore, the four pixels that are left
unfiltered represent a small part of the image when dealing with high definition content, thereby making this aminor issue.
3.2

f (x) = max(0, x).

(1)

The final layer is a fully connected layer that takes theoutput of the previous ReLU layer and outputs the 8×8
corrected values. In contrast to the in-loop filter, theproposed filter aﬀects eight pixels on each line as opposed
to only up to six. The use of a symmetric convolution
allows only one pass of the filter as opposed to in-loop
vertical and horizontal filtering that creates a temporaldependency because it is necessary to perform one prior
to the other to obtain the expected results.
In order to determine the number of filtering layers,
experiments are conducted with varying amounts. Increases in the amount improve the filtering in some cases
although it also increases the number of operations andnumber of parameters, thereby making learning more difficult and longer. This typically leads to over learning,which can be solved with a larger training dataset. In
the study, eight feature layers are used. Using a lower
number of features significantly decreases the improvement. However, increasing to 16 leads to an extremely
small improvement and even worse results for a few clips.
Increasing the training dataset also increases the training
time, which for the retained design corresponds to a week
for all the diﬀerent types of configurations.
A partial representation of the filter (normalization and
ReLU layers not shown) is given in Fig. 2.

Filtering layers

3.3With the number of blocks in a regular image, it is
impossible to perform real-time or even acceptable fast
filtering with methods via twenty layers of convolution asreported by Park et al.14). Thus, the proposed filter onlyuses two filtering layers and a normalization layer. The
normalization layer computes the average value of the
input blocks and subtracts it from the original values.
The proposed filter uses a 7×7 convolution for the convolutional layer. The convolution also exploits the correlation along the axis of the block boundary to obtain
improved results when compared to that of the in-loop filter. The in-loop filter uses 4 pixels from each side in thecomputation to oﬀer eﬃciency exceeding it, and thus theuse of several pixels is important in the convolution. Different designs are tested with deeper layers such as 5×5
followed by 3×3. However, optimal results are obtainedwith a 7×7 convolution.
The next layer corresponds to a rectified linear unit
(ReLU) layer, which acts as a thresholding function that

Training configuration

Training and testing is performed using Matlab andthe MatConvNet library16), which provides implementation of CNN for Matlab and training scripts. A common
metric to evaluate filters objectively corresponds to Peak
Signal to Noise Ratio (PSNR). Hence, it is logical to train
the filter to maximize PSNR, and thus the loss function
is set to Mean Square Error. This is a good loss function
because it is easy to invert and requires an extremely lessadditional computations during back-propagation.
All the sequences for training and testing used the same
encoding settings. Given the memory requirements andtraining time resulting from the use of an excessive number of frames, only thirty frames per video clip are usedduring training for luma and sixty for chroma. The training is performed several times over each encoding configuration.
The training and testing is obtained by using the original (before encoding) frame as an objective and the en-

5

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 2 Architecture of the proposed filter

filter processes one block per eight pixels on both verticaland horizontal direction, and thus this leads to approximately 32,000 blocks.
The cost for each block is easy to compute by assuming that the ReLU layer is trivial to compute (because it
performs a simple thresholding operation) and that mostof the required processing lays in the two convolutional
layers. Each multiply-add required in the convolution iscounted as a single operation because this is a commonly
implemented instruction. The first layer is a 7×7 convolution, and this leads to the following cost:

coded frame as an input.

4.

Computational Complexity and Parallelism

While the filtering performance was tested in the experiments, the processing was conducted on a processor
and without optimizations. This part discusses the parallelization possibilities, theoretical computing cost of thefilter, hardware implementation requirements, and current implementation performance.
4.1

Parallelism in image filtering

Nop1 = 7 × 7 × (12 + 1 − 7) × (12 + 1 − 7) × F

Most image filters allow some parallelization and especially in recent years because processing on a single core is
impossible. The deblocking filter in AVC was diﬃcult toparallelize, and thus a parallelizable deblocking filter was
introduced instead in HEVC. Thus, most filters consider
parallelization to allow fast processing.
CNN are typically parallel in their processing. However, in contrast to the AVC deblocking filter, a few filters
require the blocks to be processed in order, and this limits
the parallelization on a per block basis. Large networks
can use all resources for a single block although small networks with fewer operations cannot be significantly split,
and thus the ability to compute several blocks is simultaneously valuable.
The use of a postfilter also exhibits an advantage forparallelization, as mentioned in the introduction. The
ability to process frames completely independently significantly improves performance while processing on multiple devices with slow intercommunication such as multiple physical CPU or GPU.
4.2

= 1764F.

(2)

Where F corresponds to the number of feature layers.
With respect to the fully connected layer, each value ofthe 6×6×F output of the previous layer is connected tothe 8×8 output. This leads to the following expression:
Nop2 = 6 × 6 × 8 × 8 × F = 2304F.

(3)

The summation of all the terms leads to the following
expression:
Nopt = 4068F.

(4)

Given the current design of eight feature layers, this leads
to 32,544 operations. This indicates that a frame requires
approximately one billion operations just for luma. Whileadding chroma, this corresponds to 1.5 billion operations.
With respect to real-time processing and assuming 30
frames per second (fps), this requires 45 GFLOPS (floating point operations per second). Although the aforementioned figure still looked daunting a few years ago, consumer GPUs currently surpass 8,000 GFLOPS. Real-timeprocessing is possible even with consumer-grade hardware. However, with respect to a CPU, this approximately corresponds to the limit of a recent multi coreComputational cost of the filter

In the current test case, all videos correspond to FullHD(1920×1080 pixels) with 4:2:0 chroma subsampling. The

6

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

4.4

CPU, and thus further increases in the cost increase thediﬃculty of real-time processing. A good GPU can process even higher definition video without significant issues. Further testing also investigates half-precision or
fixed point computing for even faster computing and reduced memory footprint, thereby allowing the proposed
filter to run on smaller specialized hardware such as FPGAs where the reduction in memory requirements for theparameters is encouraged. Although the network is already small, any reduction in the memory footprint results in significant savings while considering mass production of integrated circuits. For example, hardware
vendors also pushed for limitations on the number of reference frames in AVC and HEVC due to memory requirements.
4.3

Current implementation

In practice, the current implementation using Matlab
and the MatConvNet library16)runs only at 2.5 fps perthread on an Intel Xeon E5-2630 (2.6 GHz) CPU and does
not consider the I/O operations on the raw video files.
However, it is possible to significantly improve the performance while using Intel Math Kernel Library (MKL)17)
as opposed to Matlab’s implementation of the Basic Linear Algebra Subprograms (BLAS). The biggest part of
the run time is spent on the convolutional layer. The
fully connected layer requires more operations although
the layer can be computed extremely eﬃciently with asingle matrix–matrix multiply (gemm) simultaneously for
all blocks. Conversely, it is significantly more diﬃcult to
optimize the 7×7 convolution in this manner. Even while
using Intel MKL, a frame still takes approximately 0.1 s
to process. In this case, the algorithm processes block perblock. Thus, with some parallelization, the time per image can be reduced to an average of 16 ms on the 6 coresof the CPU. The time associated with the fully connected
layer corresponds to 20 ms for Matlab’s BLAS and 15 ms
for MKL. Thus, the time to process an image is 31 ms,
which corresponds to 32 fps. However, the full processing
power of the CPU cannot be realistically used for filteringin practical uses, and thus a few further improvements or
optimizations are useful to ensure reliable real-time processing.
MatConvNet GPU implementation of the convolution
showed underwhelming results for the 7×7 convolution,
making it slower than the CPU implementation. It is also
slowed by the overhead of sending the data from CPU
memory to GPU memory and back. However, a better
implementation causes the decoded video to be alreadyin the GPU memory in the first place and eliminates theneed to return to the CPU because it is only displayedon the screen. Although this can optimize the GPU processing, this is not considered in the present study.

Hardware implementation

When porting an algorithm to a GPU, a common problem that appears corresponds to the limits on parallelization. However, the proposed filter does not exhibit inherent limitation for parallelization. Each block used as an
input corresponds to a block before any filtering is applied and each pixel is only filtered once. There are no
possible race conditions provided a buﬀer is maintained
for storing the original picture. It is possible to divide theimage into the maximum number of parts given the number of cores available to process each part independently
without any issues. There is no issue if it is necessary toseparate between two GPUs that cannot share memory
easily, and each GPU can process half the image. Onlya few redundant pixels are necessary because the input
exceeds the output and overlaps slightly. If latency is
not an issue, it is possible for diﬀerent GPUs to process
diﬀerent images and completely avoid redundancies.
With respect to embedded FPGA or specialized circuits, the 7×7 convolution can be implemented more efficiently and with less waste than in CPU or GPU (thesize and data are not aligned with SIMD instructions).
There are smaller gains for the fully connected layer although it is not necessary to support anything other thanarithmetic operations and this frees several transistors,
thereby achieving the required performance at a significantly lower cost. Apart from adapting to the image size
and QP value (once per frame), there are no branches,
and thus an extremely deep and wide architecture is possible.

5.
5.1

Testing Methodology

Encoding configuration

In order to obtain results that reflect practical use of
the filter, each sequence was encoded with a x265 HEVC
encoder18), which is an encoder that is available free ofcharge under the GNU GPL license version 2.
Two diﬀerent encoding modes were tested to evaluate
performance, namely All Intra (AI) and Random Access(RA) modes. While they slightly diﬀer from the reference encoder, the settings were kept as close as possible.

7

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

The encoding settings for both the modes are shown in
Table 1. Every loop filter in HEVC was kept as-is, andthus the comparison baseline used the deblocking and
SAO filters, and the proposed post-processing filter was
further applied.

by the proposed method. In order to assess the overall
performance and not only the performance on the Y, U,
or V channels, a combined PSNR using luma and chroma
was computed as follows:
PSNRYUV = (6×PSNRY +PSNRU +PSNRV )/8.(5)5.2

DatasetThis is the same expression as that used in a popular
comparison of video coding standards21). With respect
to each encoding mode and video, the BD-BR for eachchannel and overall improvement using PSNRYUV werecomputed. The BD-BR using YUV was computed fromPSNRYUV , and thus the values can slightly diﬀer thanthat while directly applying the formula to Y, U and V
BD-BR values.
The BD-BR values are shown in Table 3 and Table 4,
and PSNR values are shown in Table 5 and Table 6.
The detailed consideration of each mode is shown below. With respect to the aggregate results using averageand median, the BD-BR values before rounding are used.
Specifically, 10 clips are considered, and thus the medianvalue is the average between the 5th and 6th best clips.

We used video sequences from the ITE/ARIB Hi-Vision
Test Sequence 2nd Edition19). Ten clips from ten diﬀerent video sequences were used. The full list with extracted frame numbers is shown in Table 2. Most videos
still use an 8-bit depth and 4:2:0 chroma subsampling.
Thus, the source videos were cropped to 8 bits and halfof the chroma values were discarded from the 10-bit 4:2:2
source.
Training requires enough diﬀerent sequences for usability on clips outside the training clips. It is not extremely
useful to present results for only a couple of sequences,and thus we used cross-validation. Most clips were used
for training. The remaining clips were then tested andthe training set was switched. We tested videos in subsets of three and two by repeating the training twice onseven video sequences and eight video sequences such thatevery video sequence was tested.

6.

6.1

Results

Table 3 Filter results for the All Intra mode
Sequence
Y
U
V
YUVs201 (Ginko Tree)-2.6% -2.7% -0.8% -2.4%
s202 (Truck train)-1.6% -2.7% -2.8% -1.7%s203 (Cosmos flowers)-2.9% -1.4% -0.5% -2.6%
s209 (Fountain)-1.4% -1.7% -0.7% -1.4%
s214 (Basketball)-1.9% -5.7% -3.4% -2.5%s215 (Twilight scene)
3.0% 16.0% 4.2% 4.2%
s218 (Horse racing)-1.1% -1.2% -1.5% -1.1%
s251 (Rotating disk)-4.4% -1.8% -1.0% -3.9%
s259 (Colorful World)-3.5% -2.9% -1.2% -3.2%
s265 (Fountain)-0.9% -3.2% -2.9% -1.0%
Median
-1.7% -2.2% -1.1% -2.1%
Average-1.7% -0.7% -1.1% -1.6%

This section details the results of the proposed filter
using PSNR measurements 2 Bit Rate (BD-BR)20)wherea negative value indicates the percentage of bitrate saved
Table 1 Encoding settings
Codec
Version
Encoding settings(AI)
Encoding settings
(RA)

x265
HEVC Encoder version 2.1+69c97c64ab8b8e–preset veryslow –tune=psnr –nowpp –rd=6 –keyint 1 –qp $qp+3–preset veryslow –tune=psnr –nowpp –rd=6 –keyint 32 –min-keyint
32 –bframes 16 –qp $qp

Table 4 Filter results for Random Access mode
Sequence
Y
U
V
YUVs201 (Ginko Tree)-2.1% -2.0% -1.4% -2.0%
s202 (Truck train)-0.8% -0.3% -1.1% -0.8%s203 (Cosmos flowers)-4.7% -0.9% -0.3% -4.1%
s209 (Fountain)-1.5% -0.8% -0.3% -1.4%
s214 (Basketball)-1.8% -2.7% -1.5% -1.9%
s215 (Twilight scene)
3.2% 17.2% 6.8% 5.1%
s218 (Horse racing)-1.2% 0.1% -0.1% -1.0%
s251 (Rotating disk)-4.9% -0.3% -0.1% -4.0%
s259 (Colorful World)-4.4% -0.2% 1.4% -3.1%
s265 (Fountain)-1.2% -1.4% -1.0% -1.2%
Median
-1.6% -0.5% -0.3% -1.6%
Average-1.9% 0.9% 0.2% -1.4%

Table 2 Video Sequences
Sequence
Number
201
202
203
209
214
215
218
251
259
265

Name
Ginko trees
Truck train
Cosmos flowers
Fountain (dolly)
Basketball
Twilight scene (zoom out)
Horse racing (dirt)
Rotating disk 59.94P
Colorful world A
Fountain (chromakey)

Extracted
Frames100 - 159
100 - 159
100 - 159
100 - 159
100 - 159
100 - 159
100 - 159
100 - 159
100 - 159
100 - 159

All intra mode

The following results were obtained while encoding themovie clips with the All Intra configuration. Table 3

Test
Subset
1
1
1
2
2
2
3
3
4
4

8

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Table 5 PSNR values for two diﬀerent QP values on each sequence for the proposed filter and reference
HEVC (All Intra)
Sequence
Ginko Tree
Truck train
Cosmos flowers
FountainBasketball
Twilight scene
Horse racing
Rotating disk
Colorful World
Fountain
Average

QP22
QP32
Reference
Proposed Method
Reference
Proposed Method
Y
U
V
Y
U
V
Y
U
V
Y
U
V
40.69 40.92 40.28 40.84 40.97 40.30 31.74 35.32 33.96 31.97 35.46 34.00
41.23 41.55 42.63 41.37 41.56 42.65 33.09 38.72 39.43 33.16 38.77 39.49
40.70 40.27 40.70 40.93 40.28 40.69 32.65 35.37 36.01 32.87 35.44 36.04
41.25 44.61 43.87 41.33 44.61 43.87 35.99 43.50 40.80 36.03 43.52 40.81
42.17 44.87 44.17 42.22 44.99 44.21 37.43 39.83 38.86 37.54 40.16 39.06
41.31 41.85 41.55 41.24 41.76 41.52 37.02 39.11 37.37 36.91 38.63 37.18
42.25 43.84 46.19 42.27 43.85 46.20 37.69 39.65 41.64 37.75 39.71 41.71
41.49 42.50 43.32 41.68 42.52 43.34 35.62 39.59 38.57 35.91 39.65 38.62
40.84 40.96 41.14 41.04 41.01 41.17 33.92 35.51 34.58 34.14 35.68 34.66
41.42 43.43 44.72 41.49 43.44 44.73 36.18 42.64 42.37 36.19 42.67 42.44
41.33 42.48 42.86 41.44 42.50 42.87 35.13 38.93 38.36 35.25 38.97 38.40

Table 6 PSNR values for two diﬀerent QP values on each sequence for the proposed filter and reference
HEVC (Random Access)
Sequence
Ginko Tree
Truck train
Cosmos flowers
FountainBasketball
Twilight scene
Horse racing
Rotating disk
Colorful World
Fountain
Average

QP22
QP32
Reference
Proposed Method
Reference
Proposed Method
Y
U
V
Y
U
V
Y
U
V
Y
U
V
38.91 41.09 41.00 38.95 41.09 41.06 32.71 36.28 34.88 32.85 36.36 34.92
39.77 41.98 43.18 39.83 41.98 43.20 34.70 39.20 39.99 34.72 39.20 40.00
39.22 39.80 40.57 39.46 39.80 40.58 32.04 35.65 36.22 32.26 35.67 36.23
40.49 44.50 43.65 40.56 44.50 43.65 35.74 43.68 40.68 35.77 43.68 40.68
41.41 44.87 43.92 41.46 44.89 43.94 37.60 40.65 39.47 37.65 40.75 39.53
40.27 41.51 41.23 40.19 41.40 41.14 36.89 38.73 37.20 36.82 38.43 37.06
41.51 43.91 46.69 41.53 43.91 46.70 37.89 40.48 42.52 37.92 40.47 42.52
40.54 42.74 43.91 40.68 42.74 43.92 36.36 40.25 39.53 36.52 40.25 39.51
39.57 41.02 41.78 39.68 41.00 41.76 35.21 36.55 35.81 35.38 36.58 35.74
40.60 43.42 44.65 40.66 43.42 44.66 35.60 42.63 42.74 35.66 42.63 42.75
40.23 42.48 43.06 40.30 42.47 43.06 35.47 39.41 38.90 35.56 39.40 38.90

and made it worse. This indicated that it is importantthat the training set was comprised of movie clips withcharacteristics similar to that of the target clips.
While considering PSNRYUV , the improvement corresponds to a reduction in bitrate by approximately 1.7%
on average.
Figure 3 shows the improvement in the filter on the
Basketball and Rotating disk sequence. The large edges
due to the low bitrate encoding were smoothened and theimage was less degraded. In Fig. 3, top row is from theBasketball clip, bottom from the Rotating Disk clip, theleft column shows the image before using the filter, andthe right column shows the image after using the filter.

shows the improvement in the proposed filter for each
video and the results were averaged over each video. The
actual PSNR values for two values of QP are shown in
Table 5.
With the exception of a video (Twilight scene (zoomout)), each video exhibited an improvement and especially for luma with up to 4.4% reduction in the bitrate
for a video. Chroma exhibited less gain on average. However, outside the outlier, it was always improved by atleast half a percent and up to 5.7% and exhibited good
performance while considering the median.
The PSNR values indicated that the filter is most eﬃcient for low bitrate clips and that an improvement overhigh quality clips is more diﬃcult.
The poor performance on Twilight scene can be explained by looking at the other videos in the current set
as follows: this was the only video shot at night withlow light and significant grain due to the lack of light.
The characteristics of the clip (significantly diﬀer fromthe others) explained why the filter failed to improve it

6.2

Random access mode

The following results were obtained while encoding themovie clips with the Random Access configuration. Table 4 shows the improvement in the proposed filter foreach video. With the exception of a video (Twilightscene(zoom out)), which exhibited same problems as the9IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）(a) without the application filter(b) with the application of the filter

Fig. 3 Comparison of images

albeit with a few misgivings. Only Dai et al.15)discussed
results for chroma components, and thus only the luma
component can be compared. In this case, the proposed
method revealed inferior results to Dai et al.15)and Parket al.14), who show a reduction of approximately 4.5% inbitrate (with the latter using the same dataset for training
and testing). However, the proposed filter is significantly
faster than both methods.
With respect to the random access case, results reported by Li et al.13)and Park et al.14)can be comparedalbeit with a few caveats. In the former, the intra-frame
distance is fixed to 150 frames (–keyint 150)22)while forthe latter, it corresponds to 16 frames. The current study
used 32 frames because it oﬀers a good compromise between jumping in the middle of the stream and encoding
eﬃciency. Furthermore, 150 frames is not typically considered random access and is most used for archiving dataas the eﬃciency is more important than the ease of access. Hence, Park et al. indicated the highest eﬃciencyof the three methods with a 2.6% reduction in bitrate.
It is also the only in-loop filter solution of the three andrequired the encoder and the decoder to be modified andsupplied with the network parameters. The proposed filter corresponded to the second best filter with a 1.9%
reduction in bitrate, and the filter proposed by Li et al.
was last with 1.6%. Both methods also exhibited an additional advantage because they use the same training
and testing datasets, which means their true eﬃciency
for diﬀerent testing data is likely to be lower.

All Intra mode, each video improved with the proposed
filter with an overall BD-BR improvement correspondingto -1.4% by using PSNRYUV . Except for a few exceptions,luma exhibited maximum improvement with an averageof -1.9% and up to -4.9%.
Chroma was unfortunately improved to a lesser extent by the filter with only up to -2.7% for U and -1.5%
for V and very often merely corresponded to minor improvements, which resulted in an overall negative meanchange due to the outlier and a -0.3% reduction in bit rate
while considering the median. The actual PSNR values
as shown in Table 6 indicate the lack of improvement
on chroma with higher QP values, and thus potentiallyrequiring additional training for usability.
While the Random Access mode exhibited similar andoccasionally better performance on luma when comparedto the All Intra mode, performance on chroma was stillpoor for most clips. Only a clip exhibited an improvement over 2% on chroma and several others also exhibitedsome degradation. Diﬀerent types of frames can make thelearning more diﬃcult. Furthermore, as chroma was typically subject to more severe degradations and especiallyin B frames, the recovery of information was typicallymore diﬃcult than that for luma.
In order to solve the issue, a solution was assumed to
involve the use of a diﬀerent filter based on frame type.
However, this required significantly more encoding andsource material to exhibit suﬃcient I and P frames toform a suﬃciently large training set.
6.3

Comparison with other methods

6.4

Other methods did not use the same datasets, and thus
comparison proves diﬃcult.
With respect to the All Intra mode (which was present
in the two methods14),15)), the results can be compared

Processing speed

Given that other methods did not use as many optimizations to extract the maximum performance fromtheir computers, the use of the processing speed fromthe naı̈ve implementation in MatConvNet appeared as

10

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）the fairest option, especially since it was also used inPark et al.14). The proposed filter achieved a processing
time of 0.4 s for a FullHD(1920×1080) 4:2:0 subsampled
video. Park et al.14)revealed a processing time of 1.2 s on
a 832×480 sequence (only luma). Dai et al.15)takes 0.98
s on CPU and 0.45 s on a GPU for a QCIF (176×144),
4:2:0 subsampling video.
In order to make this comparable, the times were extrapolated assuming the processing scales linearly with
the number of pixels. This leads to 6.2 s per frame forPark et al. and 36 s for Dai et al. (given GPU time). This
makes the proposed filter at least an order of magnitude
faster than other methods.

7.

Acknowledgment
The study was partially supported by JSPS KAKENHI
Grand Numbers 16H02841 and 18K19772.

References
1) B. Bross, W.-J. Han, G. J. Sullivan, J.-R. Ohm, T. Wiegand: “High Eﬃciency Video Coding (HEVC) Text Specification Draft 9”, ITU-T/ISO/IEC Joint Collaborative Team on
Video Coding (JCT-VC) (2012).
2) P. List, A. Joch, J. Lainema, G. Bjøntegaard, M. Karczewicz:“Adaptive Deblocking Filter,” IEEE Trans. on Circuits andSystems for Video Technology, Vol. 13, No. 7, pp.614–619(2003).
3) A. Norkin, G. Bjøntegaard, A. Fuldseth, M. Narroschke, M.
Ikeda, K. Andersson, M. Zhou, G. Van der Auwera: “HEVC
Deblocking Filter”, IEEE Trans. on Circuits and Systems forVideo Technology, Vol. 22, No. 12, pp.1746–1754 (2012).

Conclusion

4) C.-M. Fu, E. Alshina, A. Alshin, Y.-W. Huang, C.-Y. Chen,
C.-Y. Tsa, C.-W. Hsu, S.-M. Lei, J.-H. Park, W.-J. Han: “Sample Adaptive Oﬀset in the HEVC Standard”, IEEE Trans. on
Circuits and Systems for Video Technology, Vol. 22, No. 12,
pp.1755–1764 (2012).

In the study, a post-processing filter for HEVC wasintroduced. The filter, based on CNN, uses only two convolutional layers to preserve processing speed.
The results indicated that the proposed filter significantly increased the quality of the output (based on the
PSNR metric). Based on the video, gains of up to 5%
reduction in bit rate on luma were possible. One video
exhibited negative results although a better training set
can potentially fix this.
The filter exhibits good average performance for bothAll Intra and Random Access modes with an overall 1.5%
reduction in bitrate. While other modes were not tested,similar values obtained on both modes suggest that they
result in similar values. For example, Park et al.14)indicate that the results for Low Delay P are similar to thosefor the Random Access mode.
The filter failed to achieve the improvements in thestate of the art post-processing methods although it
achieved a moderate improvement for a processing timethat was ten to a hundred times lower. It was also considerably simple to run because it only required the frames
and QP value, which can be passed as a global parameterto select the appropriate model. It allows the proposed
post-processing filter to be embedded inside a decoder or
television without excessive additional cost. Other existing filters are simply not worth the additional processing
cost.
Future studies should investigate videos of diﬀerent resolutions and from other sources. Results on 10-bit videos
using 4:2:2 chroma subsampling should also be examined.
In order to aid with the processing cost, it is important tofocus on separable filters since they allow faster and easier
implementation of computation for the convolutions.

5) A. Chauvet, T. Miyazaki, Y. Sugaya, S. Omachi: “Adaptive
Post Filter for Reducing Block Artifacts in High Eﬃciency
Video Coding”, Proc. of the 2016 International Conference on
Multimedia Systems and Signal Processing (ICMSSP), pp.22–
25 (2016).
6) V. Jain, S. Seung: “Natural Image Denoising with Convolutional Networks”, Proc. of the Advances in Neural Information
Processing Systems 21, pp.769–776 (2009).
7) K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang: “Beyond a
Gaussian Denoiser: Residual Learning of Deep CNN for Image
Denoising”, IEEE Trans. on Image Processing, Vol. 26, No. 7,
pp.3142–3155 (2017).
8) M. J. Du, W. Y. Wang, A. Liu: “Study for Image Optimal
Filter Based on Neural Network”, Proc. of the 2016 International Conference on Audio, Language and Image Processing(ICALIP), pp.391–394 (2016).
9) V. Jampani, M. Kiefel, P. V. Gehler: “Learning Sparse High
Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks”, Proc. of the 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.4452–
4461 (2016).
10) W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R.
Bishop, D. Rueckert, Z. Wang: “Real-Time Single Image andVideo Super-Resolution Using an Eﬃcient Sub-Pixel Convolutional Neural Network”, Proc. of the 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.1874–
1883 (2016).
11) W.-S. Lai, J.-B. Huang, N. Ahuja, M.-H. Yang: “DeepLaplacian Pyramid Networks for Fast and Accurate SuperResolution”, Proc. of the 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp.5835–5843 (2017).
12) C. Dong, C. Change Loy, X. Tang: “Accelerating the SuperResolution Convolutional Neural Network”, Proc. of the European Conference on Computer Vision, pp.391–407 (2016).
13) C. Li, S. Li, R. Xie, W. Zhang: “CNN based post-processing to
improve HEVC”, Proc. of the IEEE International Conference
on Image Processing (ICIP), pp.4577–4580 (2017).
14) W.-S. Park, M. Kim: “CNN-based in-loop filtering for coding eﬃciency improvement”, Proc. of the 2016 IEEE 12th Im-

11

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Yoshihiro SUGAYA

age, Video, and Multidimensional Signal Processing Workshop(IVMSP), pp.1–5 (2016).

He received B.E., M.E., and Ph.D. degrees
from Tohoku University, Japan in 1995, 1997,
and 2002, respectively. He is currently an associate professor at the Graduate School ofEngineering, Tohoku University. His researchinterests include the areas of pattern recognition, image processing, parallel processing,and distributed computing. Dr. Sugaya is amember of the Institute of Electronics, Information and Communication Engineers (IEICE) and the Information Processing Societyof Japan.

15) Y. Dai, D. Liu, F. Wu: “A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding”, Proc. of
the International Conference on Multimedia Modeling, pp.28–
39 (2016)
16) A. Vedaldi and K. Lenc: “MatConvNet: CNNs for MATLAB”,
Proc. of the ACM International Conference on Multimedia,
pp.689–692 (2016).
17) Intel, Math Kernel
software/ (2017).

Library,

http://developer.intel.com/

18) MulticoreWare, Inc., x265 HEVC Encoder, http://x265.org/(2016).
19) Institute of Image Information and Television Engineers,
ITE/ARIB Hi-Vision Test Sequence 2nd Edition, http://www.
ite.or.jp/contents/chart/emanual.pdf (2009).

Shinichiro OMACHI (Member)He received B.E., M.E., and Ph.D. degrees
from Tohoku University, Japan, in 1988,
1990, and 1993, respectively. He worked as a
research associate at the Education Center for
Information Processing at Tohoku University
from 1993 to 1996. Since 1996, he has beenwith the Graduate School of Engineering at
Tohoku University, where he is currently aprofessor. From 2000 to 2001, he was a visiting associate professor at Brown University.
His research interests include pattern recognition, computer vision, image processing, andimage coding. He received the IAPR/ICDARBest Paper Award in 2007, the Best Paper
Method Award of the 33rd Annual Conference of the GfKl in 2010, the ICFHR Best
Paper Award in 2010, and the IEICE Best Paper Award in 2012. Dr. Omachi is a memberof the IEEE, Institute of Electronics, Information and Communication Engineers, Information Processing Society of Japan, among
others.

20) G. Bjøntegaard: “Calculation of Average PSNR Diﬀerences
between RD-curves”, ITU-T Video Coding Experts Group(VCEG) 13th Meeting (2001).
21) J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, T. Wiegand: “Comparison of the Coding Eﬃciency of Video CodingStandards–Including High Eﬃciency Video Coding (HEVC)”,IEEE Trans. on Circuits and Systems for Video Technology,
Vol. 22, No. 12, pp.1669–1684 (2012).
22) Grand Challenge ICIP 2017, Grand Challenge on the Use Of
Image Restoration For Video Coding Eﬃciency Improvement,
https://storage.googleapis.com/icip-2017/index.html (2017).

(Received May 31, 2018)(Revised March 5, 2019)

Antoine CHAUVETHe received M.E. degrees from Tohoku University, Japan, and INSA Lyon, France in2017. He is currently a doctoral student in
Tohoku University, Japan. His research interests include video filtering, video coding andparallel processing.

Tomo Miyazaki
He received B.E. degree from Department ofInformatics, Faculty of Engineering, Yamagata University in 2006. He received M.E.
and Ph.D. degrees from Graduate School ofEngineering, Tohoku University in 2008 and2011, respectively. He joined Hitachi Ltd in
2011, and has worked at the Graduate Schoolof Engineering, Tohoku University from 2013
to 2014 as researcher. Since 2015 he has beenan Assistant Professor. His research interests
include pattern recognition and image processing. Dr. Miyazaki is a member of the Institute of Electronics, Information and Communication Engineers.

12IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
IIEEJ

Paper
Deep Learning Based Uterus Localization and Anatomical Structure
Segmentation on Fetal Ultrasound Image
Yan LI† (Student Member), Rong XU†† , Artus KROHN-GRIMBERGHE††† , Jun OHYA† †† (Member),
Hiroyasu IWATA†† Department of Modern Mechanical Engineering, Waseda University ,†† Global Information and Telecommunication Institute, Waseda University ,††† Department of Business Information Systems, Paderborn University<Summary> This paper proposes deep learning based methods for automatically detecting the uterus
in the ultrasound (US) image and segmenting the detected uterus into anatomical structures. For accurate
detection of the uterus and for segmentation of multiple fine-grained anatomical structures from the US
image, we use a two-tier deep learning based algorithm: (I) localizing the bounding box of the uterus, and(II) segmenting the areas of amniotic fluid and fetal body from uterine image. To achieve (I) we design
and train a convolutional neural network (CNN) based bounding box regression model, which regresses
candidate positions of the uterus. Then we use the cropped uterus region as the input to a semantic
segmentation approach. For (II) we apply fully convolution based architecture that segments the fetal
body and amniotic fluid from fetal US images in an end-to-end, supervised learning pipeline. We use
additional inner layers and intermediate supervisions to improve the segmentation accuracy and smooth out
the boundaries. We experimentally evaluate our methods and demonstrate the accurate uterus detection
and anatomical structure segmentation results.
Keywords: uterus detection, anatomical structure segmentation, ultrasound image, convolutional neuralnetwork

1.

Introduction

Accurate and automatic measurements of the anatomical structures on the ultrasound (US) image can bring
precise medical treatment to the prenatal examinationand solve the shortage of manpower. To achieve this goal,
we seek a way to automatically localize the uterus area,and precisely extract the border lines of the fetal bodyand amniotic fluid areas on US images. In this work,
we propose a two-tier approach of deep learning based
techniques to (I) locate the bounding box of the uterus,and (II) segment the pixels in the uterine region into fetal body, amniotic fluid, and background. An example offetal US image and its annotation is visualized in Fig.1.
Firstly, we expect to use localized uterus area to improve the semantic segmentation results. Conventional
methods use manual designed feature descriptors andtrained classifiers to detect the object from the images
in a sliding window fashion. For instance, Albayrak etal.5) use Histograms of Oriented Gradients (HOG) to extract the responses of detection widows from each of thepositions, and scores each of the positions using a Support

Fig. 1 Example of the fetal US image and its annotation

Vector Machine (SVM) classifier.
The deep learning based convolutional object detectors(such as F-RCNN9) or SSD17) etc) have shown large advantages on object detection tasks by well-designed neural network structures. They map the input image intoa target dimension space which corresponds to the vectors between sets of pre-defined positions to the groundtruths.
On the other hand, to achieve pixel-wise classification,

13

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）tal body and amniotic fluid in an end-to-end, fully supervised learning pipeline. We use a symmetrical designed
fully convolution structure1) to achieve a pixel-wise andmultiple category classification on fetal US images. To
further improve the predictive performance in complex
scenes and smooth out the segmentation results we modify the structure with additional layers and multi-scale
supervisions. We compare the proposed method with several recently published approaches through experiments.

2.
2.1

MethodOverview

The overall working flow is illustrated in Fig.2. The
input of our system is a still US image. We first feed the
raw US still image into a bounding box regression network
to localize the position of the uterus. The bounding box
regression network predicts the oﬀsets between a set ofpre-defined coordinates and the ground truth positions.
Then Non Maximum Suppression14) (NMS) is adopted as
post-processing approach to cluster the multiple candidate bounding boxes. Regarding the tier 2, we extract
the uterus area by obtained bounding box and then resize the sub region of image to a fixed size. Then weclassify each of the pixels to pre-defined categories on the
input region by a second segmentation network. Finally,
the pixel-wise probability map is transferred back to theorigin position by the localized uterus area.

Fig. 2 The flow map of proposed methodin the earlier study, low level patterns are used as theclues for clustering each pixel to groups or extracting theedge of objects on the US images, some of the related
works8) achieve the segmentation by morphology options
or gradient filters etc. Mitchell SC et al.10) propose theenergy function and statistic model based approaches. B.
Georgescu et al.7) propose a data-driven approach to detect and segment the edge of the left ventricle (LV) from
US image. Comparing with recent deep learning based
works, the methods lack of accuracy and rely on regularshape information.
Regarding the deep learning based US image segmentation, Carneiro et al.4) adopt deep belief network (DBN)
to localize the most likely positions on the perpendicularsof the contour of anatomical structures. The feature is
extracted on each of the positions of the anchor points toobtain the responses, which makes redundant computations on overlapping areas. Chen et al.6) use oﬀ-the-shelf
fully convolution network (FCN) structure2) to segment
the LV from designated US slices. They iteratively segment the desired object in the sub region of the US image.
The results show that the segmentation in the detailed
areas of complex scenes is still need to be improved.
To verify the performance of deep learning based approach for uterus localization and improve the accuracyof the follow-up segmentation task, we first propose a
CNN based bounding box regression model to regress theoﬀsets of fixed number of multiple pre-defined positions(which is named as reference box), and crop the uterus
area by the localized bounding box with some residuals.
Then we apply CNN based architecture to segment the fe-

2.2

Localization of uterus area

In this section, we describe the proposed method of how
to localize the position of uterus in the pregnant patient’s
raw US images through bounding box regression network.
2.2.1

Framework

The proposed approach is illustrated in Fig.3. We define the position of the uterus as tight bounding box,
which can be represented by P = (Xtl , Ytl , Xbr , Ybr ),where Xtl , Ytl , Xbr , Ybr indicate the top left and bottom
right x (horizontal) and y (vertical) coordinates, respectively. The ground truth position PGT of bounding box
is defined as the tight rectangle that starts from the leftmost pixel of uterus, and ends at the rightmost pixelof the uterus. Inspired by convolutional object detectors such as F-RCNN9) or SSD17), we use multiple reference boxes with fixed initial positions and calculate thedistance to the target positions. We expect the deeply
learned model learn the oﬀsets between initial position
and ground truth, and object confidence of each of thereference boxes as well.

14

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

width and height of the input US image. The normalization limits the output values so that the training loss
converges.
ij
The ground truth category CGTof each reference box
is determined by the area V ij in which the reference box
and ground truth bounding box overlap, such as,{0, if V ij > Θv
ij
CGT
=
,
(3)
1, if V ij ≤ Θv
where Θv is a pre-defined value (in our experiments we
use 0.3). The confidence labels of the corresponding reference boxes are set to 1 (positive); otherwise, the label is
set to 0 (negative). The specific computation of the overlapping area V ij , in which the ground truth boundingbox and reference box overlap, are performed by Eq.(4):Fig. 3 Framework of bounding box regression network

2.2.2 Oﬀset regression
To predict the position of the bounding box of uterus
by convolution neural network model, we design a
regression model which predicts the distance between
sets of pre-defined locations and the ground truth
locations. We define multiple reference boxes throughinitial positions PInit . For each image sample image,
we equally place n initial reference boxes at every
column and row. Given sample image I(v), the oﬀset
ij
ij
ij
is defined as Poﬀset= (∆Xtlij , ∆Ytlij , ∆Xbr
, ∆Ybr) forij
ij
reference box (i, j), where (∆Xtl , ∆Ytl ) is the distance
between the top left corner of reference box and theij
ij
ground truth, and (∆Xbr, ∆Ybr) is the distance between the bottom right corner of reference box andthe ground truth.
Additionally, we also learn theconfidence score C ij , which indicates if the referencebox (i, j) will be used as positive or negative sample,
as detailed below in the last paragraph of this section. Formally, we learn the following mapping of I(v) ∼ij
1111[(∆Xtl11 , ∆Ytl11 , ∆Xbr
, ∆Ybr, C 11 ), ...(∆Xtlij , ∆Ytlij , ∆Xbr,
ij
ij
∆Ybr , C )] for each of the reference box (i, j), where iand j range from 1 to n. All of the (4 + 1) × n × n outputs
are learned simultaneously by our network (Fig.3). In
Fig.3, rectangles in green and orange show predictedbonding boxes. This gives us (normalized) predicted
positions PPred for each of the reference boxes by
PPred = PInit + Fd (I(v)),

V ij = (S ij ∩ SGT )/(S ij ∪ SGT ),(4)

where S ij indicates the area of reference box (i, j), SGT
indicates the area of uterus bounding box, ∩ and ∪ areintersection and union of the two areas, respectively. During testing, the predicted category values are used as theconfidence of each reference boxes. We treat the boxes
which have higher positive confidence than negative ones
as the positive positions. In our implementation, all of
the outputs (the sets of regression oﬀsets and confidences)
are assigned as one vector, which is together calculated
by linear combination on previous fully connecting layer.
It brings advantage in robust global and context information because all of the positions and confidence scores of
the reference boxes are considered jointly.
2.2.3 Post-processing
As introduced, the method first transfers all of the initial coordinates to the target positions PP red by predicted
oﬀsets, as Eq.(1). To demonstrate the relations between
initial reference boxes and prediction results, the positions of initial reference boxes, the positions of transferred reference boxes, and the final prediction results
after NMS are visualized in Fig.4. Here, in Fig.4, columna) shows initial reference boxes with category annotations (green: positive, red: negative) which are assignedby overlapping area with ground truth, column b) shows
the regressed positive (in green) and negative (in red)bounding boxes, and column c) shows the uterus localization results after NMS, and the yellow rectangles indicate the ground truth bounding boxes. This initial prediction contains many overlapped bounding boxes which
are assigned to the same object in the image (as shownin Fig.4 (b)). Therefore, the approach needs to further(1)

where Fd (U ) indicates the trained bounding box regression network.
In our implementations, the oﬀset value ∆X and ∆Y
are normalized by the size of original image, such as,∆X = (XInit − XGT )/w, ∆Y = (YInit − YGT )/h,(2)where (XInit , YInit ) is the coordinate of initial position
while (XGT , YGT ) is the ground truth, w and h are the15

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）“encoder-decoder”1) CNN. The input image is scaled
down by a series of pooling and a second series of upsampling operations. Fig.5 illustrates the overall framework. The network first maps the input image into multichannel down-scaled feature maps then up-sample thefeature maps back to the original size and target channels (which has same number of categories). In Fig.5, theinput and output dimensions of feature maps are represented in width× height×channels. The optimizations
for improving segmentation accuracy and smoothness are
introduced as follows.

eliminate the redundant predictions. In this paper wecluster the multiple overlapping boxes by NMS14). The
NMS seeks for the position with the maximum confidence
value in a given region which might contain the same object by eliminating all of the bounding boxes whose overlapping areas are larger than a threshold. Consequently,
the merged bounding box is obtained as the final result of
the uterus detection. Among the obtained multiple candidate bounding boxes, we keep the bounding box which
has the largest confidence as per NMS.
2.3

Semantic segmentation of anatomical
structure
In this section we describe the semantic segmentation
model for anatomical structure segmentation. To preventthat the obtained bounding box is smaller than the actual
uterus area, we extend the bounding box area with a fixedfactor of 1.2 to leave some residuals for the areas near theuterine border.
To segment the pixels from the obtained uterine region into the fetal body, amniotic fluid and background,
we adopt framework based on symmetrical designed

2.3.1

Encoding-decoding architecture

The input image is first mapped to a set of down-scaledfeature maps by convolution and pooling operations. Our
work uses max-pooling, which keeps only the maximum
value of each pooling window. In the first half stage ofthe network (encoding stage), multiple max-pooling operations are adopted. Overall, the feature maps of the lastM
layer of the encoding stage are down scaled to (1/2) ofthe raw input by M max-pooling layers with stride two.
Then, to rescale the output of the encoder stage to thesame size as the input image we use un-pooling3) operations. Each pixel is assigned to the recorded maximum
position in each of the corresponding windows, while theother positions are filled with zeroes. For backward propagation, the derivatives are passed to the former layer
by only keeping the largest one in the window. Each
un-pooling layer is concatenated with convolution layers
which have learnable convolution kernels.
2.3.2

Inner layers

From the bottom layer to the top layer in the encoding
stage, we can assume that the responses of higher layers
represent more global information. In the first several layers, the learned convolution kernels work as low level fea-

Fig. 4 The visualized reference boxes and the prediction
results

Fig. 5 Illustration map of segmentation network for anatomical structure

16

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

the positions on the response map. In training we treat
all of the errors equally by summing the derivatives ofeach of the branch with same weights. We do not weight
the error of each branch layer by the size of the output
map since the loss is normalized by the number of pixelsof the output.
During testing, the auxiliary branches are discarded,
and only the output of the last layer is used as the final
segmentation results.

ture descriptors such as edge, corner, etc. As the network
gets deeper, the learned convolutional weights tend to focus on higher level representations such as shape, spatial
relationships etc. To this end, we add layers called “inner layers” between the encoding and decoding stages toenhance higher level representations. The stacked inner
layers are connected to the last layer of down samplingstage. The inner layer works as the smallest size local
receptive fields, where we use 1x1 sized convolution operations at each position to map the data into additional
hieratically feature spaces. For each of the inner layers,
we widen the network by using relatively large number ofconvolution kernels to enhance the hierarchical representation.
2.3.3 Intermediate supervision
In the decoding stage of segmentation network, we notice that the un-pooling operation causes un-smoothed
segmentation in the border areas. By comparing withinterpolation up-scaling, the un-pooling operation’s output are relative sparse because most of the positions arefilled with zeroes. It causes the final segmentation results
to be very non-smooth in some areas, especially in theboundaries.
We propose to improve the smoothness by using multiple output branches with multi-scaled supervision signals. The images with diﬀerent resolutions yield diﬀerent detailed information in the borders. The higher resolved segmentation maps contain more information in
the border area and have smoother edges. Thus, we expect to enhance the continuity of each group of convolution layers by diﬀerent resolved ground truth segmentation maps. Specifically, inspired by GoogLeNet15), we
not only use the error from the last output layer, butalso from inserted additional output branches among thedecoding structure. The additional outputs are used as
auxiliary branches for calculating the errors between thepredictions and down-scaled ground truth label maps asshown in Fig.5. They are concatenated to the output
of convolution layers in the up-sampling stage. We use
nearest neighbor interpolation to down-scale the groundtruth label map from the original size to the same size as
the corresponding output branch. To map the intermediate output to the same dimension number as the groundtruth, for each of the intermediate branch we use convolution layers with a fixed number of convolution kernels(the number of kernels equals to the number of channelsof the output). For each of the intermediate supervision
layer we target to minimize the cross-entropy loss of all of

2.4 Backbone network
We use the VGG1611) structure as the base feature extractor (backbone network) for both bounding box regression and semantic segmentation networks. It has beensuccessful in many computer vision and recognition tasks
such as ImageNet13) object classification challenge andhas relative low hardware requirements. In this work wetreat the backbone network as feature extractor and do
not measure the eﬀects of diﬀerent backbone networks.
Regarding the structure of output vector of boundingbox regression network, our model predicts the oﬀsets andconfidence scores C ij for each of the n×n bounding boxes
from per US image. As shown in Fig.3, we organize theoutput (the oﬀsets and confidences) as a one-dimensional
vector. Thus we convert the n × n × 4 oﬀsets values andn × n × 1 confidence values to one long vector DPred anduse it as the output of the model. The Euclidean loss is
adopted as our loss function of detection network by,2

Ld = ||DPred − DGT || ,(5)

where DGT is the ground truth vector.
The segmentation network adopts three output channels as we want to classify the pixels into three categories(fetal head, amniotic fluid, and background). We calculate the confidence for each of the pixels. For each input
image with H pixels, the corresponding loss function Ls
of our segmentation network is:
1 ∑Ls = −
log ph ,
H
H

(6)

h=1

where ph is the predicted confidence of pixel h which corresponds to its ground truth category. For each of theintermediate and the last output layer we use the sameprobabilistic function.

17

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）3.

Table 1 Structures of inner layers used by diﬀerent
modelsLayer name Segnet
1IL
2IL
3IL
Layer1None
5124096 4096
Layer2
None None 512 4096
Layer3
None None None 512

Experiment

3.1 Dataset
3.1.1 Experimental environment
To conduct our experiments, we received approval fromthe Ethics Review Committee on Research with Human
Subjects, Waseda University (2014-165). We use GE Voluson E8 and C1-5 linear array transducer with frequencies in the range of 4.0 Hz. The axial and lateral resolution is 2mm and 3mm, respectively. The field of views is
66 degrees and the depth setting is 15cm. The average
fetal age of our subjects is approximately 22 weeks. We
acquire four video clips from anonymous patients. The
original resolution of each frame is 640 × 480 pixels. The
raw data are sampled from each video clips every twoframes. We use 2-fold cross validation by first randomly
separating the diﬀerent patients’ data into the training
and testing sets. The contour annotations are made by
doctors who have years of US examination experiences.
3.1.2 Data cleaning
The duplicate or nearly duplicate images do not contribute to our training, and bring bias to the training or
testing sets. We one by one feed each of the data samples
through a fully trained CNN model (in this research weuse alexnet12) network structure and trained on ImageNet
dataset) and compare the Euclidean distance of output
feature vector of the last fully connected layer to all of theother samples. The image pairs which have close distanceare treated as the similar image samples. We discard allof the redundancy from the original data set. The cleaned
dataset has about 400 samples for training and testing.

20,000 iterations. The inference time on GPU is about
30ms for the detection and 60ms for the segmentation
networks per image, respectively.
3.2.2 Domain transferred learning
Studies proved that the low level feature representations are highly similar across many domains. Compared with randomly initialized weights, the fine-tuningon the pre-trained model converges at a faster speed andachieves better performance. The weights of the convolution layers of the segmentation model are initialized froma pre-trained model18). During fine-tuning of our models,
the newly added layers have five times as large learning
rate as the fine-tuned layers. For comparison, the result without using pre-trained model (“ w/oPre”) can befound in the last row of Table 1. Note that regarding
the uterus detection network, we just directly train themodel from scratch.
3.3 Evaluation criteria
We use intersection of union (IOU) to evaluate uterus
detection results. Specifically, IOU is the proportion of
the intersection of true positive pixels and union area ofthe predictions and ground truths. Since we use the relative confidences between positive and negative object,
and to avoid the single threshold limitation, the experiments further adopt Average Precision (AP)21)to evaluate
the detection performance. To demonstrate the diﬀerentalignment accuracy between ground truth bounding box
and prediction, we use three overlapping thresholds 0.5(AP.5), 0.6 (AP.6) and 0.7 (AP.7). The bounding boxes
which have overlapping ratio larger than the thresholds
are considered as true positives. The larger threshold isstricter than smaller one.
Regarding semantic segmentation, besides IOU, wedemonstrate the class specified evaluation results by ROC
curve. The true positive rate (TPR) and false positive
rate (FPR) over classes are calculated by:

3.2 Training details
3.2.1 Data augmentation and hyper parameters
We augment the training data by adding random disturbance to the predicted uterus bounding boxes. In particular, multiple sub regions are cropped from the original image with random transfer factor on the four pointsof the bounding box. The corresponded label maps also
undergo the same operations. It makes our training samples increase to more than 4000 training samples in eachsubset. For comparison, the result without data augmentation (“ w/oAug”) can be found in Table 1.
Regarding the hyper parameters, our experiments run
on a single NVIDIA GTX 1080. The deep learning platform is modified from Caﬀe16). In each iteration the error
between the ground truth and output is summed and averaged over entire batch. We stop model training after

TPR = NTP /NPos , FPR = NFP /NNeg ,(7)

where NTP , NFP , NPos , NNeg indicate the number of true
positive, false positive, positive and negative pixels ofeach category, respectively. We further adopt pixel-wise

18

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 7 Normalized uterine images by detected boundingbox

setting and compare them with F-RCNN9)and SSD17).
For F-RCNN we use multi-scale training scheme to randomly resize input image in 3 diﬀerent scales. During
testing we use original size as input. Regarding SSD, weadopt 300 × 300 as both training and testing input. For
both of above methods, we adopt VGG16 as the backbone
network.
The IOU shows that URN 5 × 5 achieves the best score
but the gaps between diﬀerent results are not obvious.
The AP demonstrates that URN 5 × 5 model achieve
higher accuracy especially on more strict criteria (AP.7).
It proves that our method has better alignment accuracy.
The major diﬀerence is the last fully connecting operation. It introduces global feature to the oﬀsets regression, which brings richer context information. The global
regression scheme is more robust to the case of uterus
localization in US image.
The advantages of usage of cropped areas can be explained from following 2 aspects: 1. constraint the location of the uterus to the area inside the uterus boundingbox; 2. alleviate the unbalanced data distribution. The
cropped uterus examples are shown in Fig.7. We noticethat the predicted uterus area smaller than uterus, it iscaused by unclear feature at left and right areas (as shownin Fig.6). It does not heavily aﬀect to the segmentation
scheme since we extend every bounding box with fixedfactor. The quantitative results of semantic segmentation w/ and w/o cropped uterus area can be found in
the column 1 (“SegNet nodet” , here, the “ nodet” indicates the model without using uterus detection as the
pre-processing. ) and 2 (SegNet) of Table 4. It provesthat the localization scheme improves the accuracy of following segmentation work.
3.4.2 Semantic segmentation of anatomical
structure
We named our optimized scheme as “EDN”. The basescheme without any modification is same as Segnet. Regarding the models with diﬀerent settings, in this study,
the models with intermediate supervisions are namedwith the suﬃx “ IS”. The models with k (k=1,2,3) inner layers are named with the suﬃx “ kIL”. For exam-

Fig. 6 Comparison between a) 3×3, b) 5×5 and c) 7×7
reference boxes (green: positive, red: negative,
yellow: ground truth); (left column: boundingboxes before NMS, and right column: boundingboxes after NMS)
Table

2 Evaluation results of uterus localization(IOU)(%)

Method

Sub.1
Sub.2
Avg.
Table

URN
3×3
55.7
55.0
55.3

URN
5×5
63.1
61.7
62.4

URN
7×7
64.6
59.7
62.1

FRCNN

SSD

61.0
60.2
60.6

60.7
59.1
59.9

3 Evaluation results of uterus localization(AP)(%)
Method
AP.5
AP.6
AP.7

URN 5 × 5
99.6
88.6
68.3

FRCNN
99.5
85.4
53.7

SSD
99.4
83.3
50.2

classification accuracy (Accu) to evaluate the segmentation results without the eﬀect of diﬀerent categories.
3.4 Results and discussions
3.4.1 Localization of uterus area
The visualized examples of the bounding box regression
network with diﬀerent settings are shown in Fig.6. The
IOU and AP results are shown in Table 2 and Table 3,
respectively. We name our proposed uterus regression
network as “URN”. The networks with diﬀerent numberof reference boxes are named with suﬃx “ 3×3”, “ 5 × 5”
and “ 7 × 7” in this study.
We evaluate the uterus localization results on diﬀerent

19

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

ple, the “EDN 3IL IS” is the results of EDN model with
3 inner layers and intermediate supervisions. Firstly, todemonstrate the performance of intermediate supervision
layers, in Fig.8, we show some of the segmentation results by highlighting the edge of the segmented blobs.
The smoothed boundaries which are predicted by modelwith intermediate supervision layers show significant improvements in the given image samples. The intermediate
supervision layers also improve the overall segmentation
accuracy (EDN IS of Table 4).
Then, to verify the eﬀectiveness of inner layers, we use
several settings with diﬀerent number of inner layers. We
list the detail of the structures of inner layers in Table 1.
The quantitative accuracy (Table 4 and Fig.9) and visualized results (Fig.10) show the additional inner layers
bring higher accuracy on the segmentation results.
Concerning comparison experiments, we adopt FCN
and Segnet as baseline methods. Besides the baseline
methods, we also evaluate Deeplabv3+19) and PSPNet20)
with same training and testing sets. Note that, all of thesegmentation models are trained based on the cropped
uterus area for fair comparison. The FCN8s model isiterative trained from FCN32s and FCN16s. Regarding
the Deeplabv3+, we adopt the ASPP scheme with batchnormalizations and an additional encoder-decoder layer.
For fair comparison purpose, the results are firstly evaluated and compared with Deeplabv3+ and PSPNet with
VGG16 backbone (V3+VGG16 and PSPVGG16). Numerical results of above models are demonstrated in Table 5 and Fig.9. In Table 5, the suﬃx of “ fb” indicatesfetal body, “ af” indicates to amniotic fluid, and “ bkg”
indicates to background. Some visualized examples ofextracted contours and segmented blobs with diﬀerent
approaches and proposed methods can be found in Fig.8
and Fig.10.
In addition, the paper further provides the results ofTable 4 Pixel-wise accuracy over all of the pixels (Accu)(%)
Methodw/
croppeduterus area
w/ intermediate
supervision
w/ 1 innerlayer
w/ 2 inner
layersw/ 3 inner
layersSubset 1
Subset 2
Avg.

SegNet SegNet EDN
nodet
IS

√

√

EDN
1IL
IS
√

EDN
2IL
IS
√

EDN
3IL
IS
√

√

√

√

√

√

√

√

√

√
√

91.99
90.41
91.20

92.23
91.05
91.64

93.43 93.97 93.81 94.08
92.22 92.39 92.49 93.71
92.83 93.18 93.15 93.90

Fig. 8 The visualized segmentation results; a) SegNet (w/o intermediate supervision layers), b)
EDN IS (w/ intermediate supervision layers) andc) PSPVGG16

Fig. 9 Category specified ROC curves of diﬀerent models

20

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 5 Class specified results (IOU) and pixel-wise accuracy over all of the pixels (Accu) (avg. of 2
sets.) (%)
MethodFCN8S
SegNetPSPVGG16
V3+VGG16
EDN w/oAug
EDN w/oPre
EDN IS
EDN 3IL IS

IOU fb IOU af
49.32
55.00
65.13
69.17
44.29

36.06
44.83
55.13
52.64
30.91

IOU
bkg
92.37
93.35
94.84
95.19
92.03

mIOU Accu
59.25
64.39
71.70
72.33
55.74

89.86
91.64
93.03
93.51
88.01

61.24

50.81

93.10

68.38

89.90

64.10
69.36

51.71
54.79

94.90 70.24 92.83
95.44 73.19 93.90

Table 6 Evaluation results of DeeplabV3+ and PSPNet
with Resnet50 as the backbone network (%)
PSPNetV3+RES50

IOU fb
61.48
61.79

IOU af
50.41
51.80

IOU bkg
94.17
94.83

mIOU
68.69
69.47

Accu
92.95
93.50

mentation results by using diﬀerent scales of ground truth
masks in training. From the visualized results, the FCN8s
model shows under fitting to the ground truth area. Ourmodel performs more complete and accurate contour information on the segmentation results than others. Despite this, from the visualized samples we can see that themodel with intermediate supervision have more smoothsegmentations in the border areas.

Fig. 10 Visualized segmentation results; a) FCN8S, b)
PSPVGG16, c) V3+VGG16, d) EDN IS, e)
EDN 3IL IS, and f) ground truth

Deeplabv3+ and PSPNet with Resnet5022) as the backbone network in Table 6 (V3+RES50 and PSPNet).
In Table 6, the suﬃx of “ fb” indicates fetal body,“ af” indicates to amniotic fluid, and “ bkg” indicates tobackground. Interestingly, the numerical results do notachieve better scores than using VGG16’s model in our
experiments. It might because in this paper we use relative small batch size (batch size=6 for both models) ona single GPU device. Even if we have trained the models
with more iterations, it seems that the performances of
networks trained with batch normalizations (i.e Resnet50,
Resnet101) are heavily depended on large batch size.
The best performance in the quantitative results is our
proposed optimized encoding-decoding model with 3 inner layers and intermediate supervisions. Although theresult does not have large gaps with V3+VGG16 andPSPVGG16 but we notice that it can proved more complete contour of objects. The proposed inner layers bringsimilar eﬀects as the atrous convolution or multi-scalepooling by introducing larger field of views to the featuremaps. Compare with PSPVGG16, the additional intermediate supervision branches not only give us faster converging speed in the training, it also brings smoother seg-

4.

Conclusion

This paper has proposed deep learning based uterus
detection and anatomical structure segmentation frameworks. Through experiments we verify our methods andconclude the results as follows:a) We use bounding box regression network to localize the positions of the uterus and achieve the best
performance within diﬀerent comparison models.
b) Compared with segmentations applied directly to
raw US images, segmentations in the cropped uterinearea optimize the segmentation results by relieving
the imbalance issue and aligning the region of target
pixels.
c) The additional 1 × 1 sized inner layers improve theoverall segmentation accuracy by extending the network to larger dimensional space and global representations.
d) The intermediate supervision layers smooth out thesegmentation results through adding multi-scaled supervisions after each group of convolution layers.
In this work we do not extend our proposed optimiza-

21

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

tions to other backbone network such as Resnet. The
reason is because the scheme records geometric representation and makes use of more local position information
which is obtained from max-pooling operations. However,
the original Resnet does not provide such down scaling
method which is required by our currently used decoding scheme. Regarding the limitation of the work, thesymmetric designed segmentation scheme has drawbacks
on high GPU memory cost in the training, which willcause the training process on our device meets the out ofGPU memory problem. It limits us to adopt the backbone networks with less learn-able weights. Therefore, inthis work we do not concern much about the backbone
networks.
To deal with above limitations, as for the future work,
firstly we will try to modify the base structure such as
Resnet backbone to fit the decoding structure used inour segmentation scheme. Then, we plan to reduce thememory cost caused by the symmetric designed scheme,
and make the network more light weight to avoid thecomputation limitation without losing too much performance. We are also considering integrating the proposed
optimization modules with other optimization modules
such as ASPP in related US image segmentation tasks.
On the other hand, we notice that our current two-stage
approach has redundancies in computations. We might
speed up the system and optimize the learning strategy
through joint learning of the bounding box regression andsegmentation.

Images”, Proc. of International Conference on Medical Image
Computing and Computer-Assisted Intervention, Vol. 9901,
pp.487–495 (2016).
7) B. Georgescu, X. S. Zhou, D. Comaniciu, A. Gupta: “Database
Guided Segmentation of Anatomical Structures with Complex
Appearance”, Proc. of IEEE Conference on Computer Vision
and Pattern Recognition, Vol. 2, pp.429-436 (2005).
8) R. Supriyanti, D. A. Putri, E. Murdyantoro, H. B. Widodo:“Comparing Edge Detection Methods to Localize Uterus Area
on Ultrasound Image”, Proc. of International Conference on Instrumentation, Communications, Information Technology andBiomedical Engineering, pp.152–155 (2013)9) S. Q. Ren, K. M, He, R. Girshick, J. Sun: “Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks”, Proc. of International Conference on Neural Information Processing Systems, Vol.1, pp.91–99 (2015)10) S. C. Mitchell, B. P. Lelieveldt, V. D. G. Rj, H. G. Bosch, J.
H. Reiber, M. Sonka: “Multistage Hybrid Active Appearance
Model Matching: Segmentation of Left and Right Ventricles in
Cardiac MR Images”, IEEE Trans. on Medical Imaging, Vol.20,
No.5, pp.415–423 (2001)
11) K. Simonyan, A. Zisserman: “Very Deep Convolutional Networks for Large-Scale Image Recognition”, arXiv:1409.1556,(2014)
12) A. Krizhevsky I. Sutskever, G. Hinton: “ImageNet Classification with Deep ConvolutionalNeural Networks”, Advancesin Neural Information Processing Systems 25, pp.1097–1105(2012)
13) J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, F. F. Li: “ImageNet: A Large-scale Hierarchical Image Database”, Proc. ofIEEE Conference on Computer Vision and Pattern Recognition, pp.248–255 (2009)14) A. Neubeck, L. V. Gool: “Eﬃcient Non-Maximum Suppression”, Proc. of IEEE Conference on International Conference
on Pattern Recognition, Vol.3, pp.850–855 (2006).
15) C. Szegedy, W. Liu, Y. Q. Jia, P. Sermanet, S. Reed, D.
Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich: “Going
Deeper with Convolutions”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp.1–9 (2015).
16) Y. Q. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R.
Girshick, S. Guadarrama, T. Darrell: “Caﬀe: Convolutional
Architecture for Fast Feature Embedding”, Proc. of ACM International Conference on Multimedia, pp. 675–678 (2014).

References
1) V. Badrinarayanan, A. Kendall, R. Cipolla:“SegNet: A Deep
Convolutional Encoder-Decoder Architecture for Scene Segmentation”, IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol. 39, No. 12, pp.2481–2495 (2017).

17) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Ful,
A. C. Berg: “SSD: Single Shot MultiBox Detector”, Proc. ofEuropean Conference on Computer Vision, Vol.9905, pp.21–37(2016).

2) E. Shelhamer, J. Long, T. Darrell:“Fully Convolutional Networks for Semantic Segmentation”, IEEE Trans. on PatternAnalysis and Machine Intelligence, Vol. 39, No. 4, pp.640–651(2017).

18) L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A.
L. Yuille: “Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs”, arXiv:1412.7062(2014).

3) M.D. Zeiler, R. Fergus: “Visualizing and Understanding Convolutional Networks”, Proc. of European Conference on Computer Vision, Vol.8689, pp.818–833 (2014).

19) L. C. Chen, Y. Zhu, G. Papandreou, F. Schroﬀ, H. Adam:“Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation”, arXiv:1802.02611 (2018).

4) G. Carneiro, J. C. Nascimento, A. Freitas: “The Segmentationof The Left Ventricle of The Heart from Ultrasound Data Using Deep Learning Architectures and Derivative-Based Search
Methods”, IEEE Trans. on Image Processing, Vol.21, No.3,
pp.968–982 (2012).

20) H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia: “Pyramid Scene Parsing Network”, Proc. of IEEE Conference on Computer Vision
and Pattern Recognition, pp. 6230–6239 (2017).

5) N. B. Albayrak, A. B. Oktay, Y. S. Akgul: “Prostate Detection
from Abdominal Ultrasound Images: A Part Based Approach”,
Proc. of IEEE International Conference on Image Processing,
pp.1955–1959 (2015).

21) M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn, A.
Zisserman: “The Pascal Visual Object Classes (VOC) Challenge”, International Journal of Computer Vision, Vol.88, No.2,
pp.3030-338 (2010).

6) H. Chen, Y. Zheng, J.H. Park, P. -A. Heng, S. K. Zhou: “Iterative Multi-domain Regularized Deep Learning for Anatomical Structure Detection and Segmentation from Ultrasound22) K. M. He, X. Zhang, S. Q. Ren, J. Sun: “Deep Residual Learning for Image Recognition”, Proc. of IEEE Conference on Com-

22

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Jun
OHYA (Member)
He earned B.S., M.S and Ph.D degrees in precision machinery engineering from the University of Tokyo, Japan, in 1977, 1979 and1988, respectively. He joined NTT ResearchLaboratories in 1979. He was a visiting research associate at the University of Maryland, USA, from 1988 to 1989. He transferredto ATR, Kyoto, Japan, in 1992. Since 2000,
he has been a professor at Waseda University,
Japan. He was a guest professor at the University of Karlsruhe, Germany, in 2005. His
research areas include computer vision andmachine learning. He is a member of IEEE,IIEEJ, IEICE, IPSJ and VRSJ.

puter Vision and Pattern Recognition, pp.770–778 (2016).

(Received April 13, 2018)(Revised December 2, 2018)

Yan

LI (Student Member)
He earned B.S. degree in Communication
University of China. He obtained M.S. degree in Graduate School of Global Information and Tele-communication Studies (GITS),
Waseda University. He is currently a doctoral
program student in Graduate School of Creative Science and Engineering, Waseda University. His research areas include computer
vision and machine learning.

Hiroyasu IWATA
Rong XU

He earned the B.S., M.S. and Dr. of Engineering degrees in Mechanical Engineeringfrom Waseda University in 1997, 1999 and2003, respectively. He is currently a Professorof Department of Modern Mechanical Engineering, School of Creative Science and Engineering, Waseda University and appointed
as a Next Generation Core Researcher in Key
Researchers Development Program by Boardof directors, Waseda University (since 2016)as well. His research fields include humanassistive and augmentation robotics, skinadhesive bio-sensors, supernumerary robotic
limb “Third Arm”, echography-based prenatal checkup assistive robot, perceptionassistive rehabilitation RT and artificial intelligence for construction machinery. He isa member of RSJ, JSME, SICE, EAJ andIEEE.

He earned B.S. degree in School of Information Science and Engineering, and M.S. degree in School of Computer Science and Engineering from Southeast University in 2003
and 2009, China, respectively. He obtained
his Ph.D degree in Graduate School of Global
Information and Tele-communication Studies(GITS), Waseda University in 2014. Currently, he is a senior researcher in Datasection
Inc., a Japanese listed company. His research
focuses on image processing, video analysis,
pattern recognition, deep learning, surgicalimage-guidance, etc.

Artus KROHN-GRIMBERGHE
He earned his MSc equivalent in informationengineering and management from KIT, Karlsruhe Institute of Technology in 2006. Afterroughly a year in the industry he went backto university and obtained his Ph.D. in Machine Learning from University of Hildesheim
in 2012. Afterwards he joined University ofPaderborn as an assistant professor in applied
Machine Learning in 2012 for six years till
2018. Currently, he is founder and CEO ofLYTIQ GmbH, a Germany-based startup focused on consulting in AI (www.lytiq.de).

23

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
IIEEJ

Paper
An Ensemble Approach to Precancerous Cervical Lesion Classificationfrom Cervigram Images
Suleiman MUSTAFA† , Akio KIMURA†† (Member)† Graduate School of Engineering, Iwate University , †† Faculty of Science and Engineering, Iwate University<Summary> In this study, we propose a semi-automated system for detecting cervical cancer fromcervigram photographs of aﬀected cervix regions. Cervical cancer is among the most common cancers
aﬀecting women in the world particularly in developing countries where few population have access toproper screening due to high costs of laboratory testing. For this reason a simple inexpensive test byvisual inspection with acetic acid (VIA) is used where the cervix region is observed with the naked eye forchange in color, texture and appearance. We consider that applying adequate image processing techniques
to the captured images during VIA is eﬀective to assist gynecologist (doctor) for detecting, diagnosing andexamining the cervix region based on the visual inspection observations. That is, it is possible to constructa kind of computer aided systems for detecting and diagnosing cervical cancers. In our framework we first
segment an input image into lesions of interest by GrabCut algorithm, and next extract many color- andtexture-based features by using image processing.Then based on these extracted features the segmented
image is categorized as cancerous “malignant” or non-cancerous “benign” by using ensemble classification
methods combined with 3 or 5 machine learning algorithms. We conducted some experiments using real
cervigram images and found through the statistical analysis that only 10–13 extracted features can be
suﬃcient to detect cervix cancer and our method comparatively improved the detection accuracy comparedto visual eye inspection.
Keywords: computer aided diagnosis, cervigram classification, cervix segmentation, feature extraction,ensemble learning

1.

been obtained, however, the capturing of cellular level
images is diﬃcult to sustain in resource-poor areas (likedeveloping countries) because of the complicated process;in collection, sampling, preparation, staining, reading, reporting and the period of time it takes before providing test results3). Due to the high cost of the equipment used for Pap smear screening, it is being expected
to develop low-tech and inexpensive screening tools that
could significantly reduce the burden of cervical cancer
deaths.2)There is therefore a need for low cost cervicalcancer screening systems that can be used in the developing countries has become an interesting research fieldrecently.
An easier and cost eﬀective alternative screening
method is the visual acetic acid (VIA) test3)performed attissue level rather than cellular level. It involves applying
acetic acid (vinegar) to the region and observing change
in color to whitish appearance. This test works in the remotest of settings, bringing access to virtually location.1)
VIA has shown ability to correctly identified between 45%

Introduction

Cervical cancer is among the most common cancers affecting women worldwide with an estimated 530,000 new
cases reported in 2012 and of the 270,000 deaths from thecancer in 2015 with about 85%–90% occurring in less developed countries1). Its incidence in Sub-Saharan Africa
is increasing due to lack of or poor screening2), while incontrast the incidence of human papillomavirus (HPV)
is declining rapidly in Western countries by the reducing of the rate of cervical cancer through appropriate
screening3). Cervical cancer screening is usually achievedby taking a papanicolaou test (Pap smear test), where
cells are scraped and collected from the cervix for microscopic examination. This is followed by investigated for
irregularities with the aim of detecting potentially precancerous symptoms, i.e., cervical intraepithelial neoplasia (CIN)1).
Microscopic images of Pap smear cells have been used invarious research recently4)–7). Several good results have24

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）representation of the interest region and computational
methods can provide suitable results for identifying cervix
regions. So far as the authors know, precancerous lesion
detection using the above segmentation-based machinelearning methods (not deep learning) are between 65.10–
82.46%(0.651–0.825), 69.78–83.21% and 54.00–94.79% respectively and therefore can be improved upon.
In this work we aim to achieve relatively comparablyhigh accuracy in precancerous cervical lesion classification based on simple and proper image processing. In
particular, we target over 3,600 real cervical images andnewly investigate a small number of eﬀective and usefulfeatures for diagnosing cervical cancer. We previously
proposed a classification framework for melanoma detection20), and we believe a similar approach is also eﬀective
to this work. We use GrabCut segmentation on the region of interest (ROI) in cervigram images and calculate
44 features based on color and texture information fromthe ROI by using typical image processing techniques.
Then we analyze which feature important is and how
many the number of features is for correct classificationby using several ensemble machine learning approaches,and finally we show from the experiments we did that
only 10–13 features yield satisfactory results to determine
most of precancerous cases.

to 79% of women at high risk of developing cervical cancer by well-trained medical personnel8). As well as change
in color, texture appearance is also observed for irregularsurface, coarseness and mosaic vessels9). Shape featureson the other hand have been shown to be not so usefulat tissue level detection classification10). Therefore, somesuitable image processing techniques can be applied tothe images captured during VIA in order to easily observe their color tone and texture diﬀerences which are
two main characteristics used by experts to diﬀerentiate
lesions. Furthermore, machine learning classification canalso potentially improve performance in identification of
precancerous cervical lesions.
There have been several related (image processingbased) studies that have used VIA image data-sets. Xueet al.10)developed the web-based cervigram image retrieval system and the system used visual characteristics
based on color, texture and shape are used to derive features. Acosta-Mes et al.11)proposed an aceto-white temporal pattern feature extraction model to identify precancerous cervical lesion. Zimmerman et al.12)utilized
information of gradient, color saturation and intensity
to detect specular reflections of cervix images. Song etal.9)proposed a method using pyramid histogram oriented
gradients features (P-HOG) on segmented regions of interest. Xu et al.13)improved the method of Song et al.
by applying the P-HOG features with color features from
LAB color space and local binary pattern (LBP) texture
features.
Moreover, some cancer classification techniques basedon the machine learning algorithms, such as k-nearest
neighbors (k-NN), support vector machines (SVMs) anddeep learning have been studied recently9),13),14), and it
has been reported that they work well but under limited conditions with only a few hundred images used forthe experiments making it a small dataset. Particularly
in deep learning-based method, precancerous lesion detection with a small dataset achieved relatively high accuracy, sensitivity and specificity of 88.9%(0.889), 87.8%
and 90.0% respectively. However, since the method utilize the rectangular cropped ROI (region of interest) inthe cervigram images for feature extraction, it will either
select unnecessary additional regions outside of the cervix
or not select some important cervix regions. An essential
requirement to the computer aided task of investigating ofcervix cancer is segmentation. Previous studies15)–19)have
highlighted that this segmentation step allows for better2.

Proposed Scheme

The proposed our framework has four major steps:
image acquisition, segmentation, feature extraction andclassification as illustrated in Fig.1.
2.1 Image acquisition
The images used for this study were provided from the
National Cancer Institute (NCI) of the United States.
(We could obtain test images based on the agreement for
the transfer of de-identified cervical images between NCI
and our university.) The data-set used here consist of
3,637 images from 1,819 patients. Among of them 2,841
images are of normal (no risk) and CIN1 (low risk) treatedas our “ Negatives”, and 796 images of CIN2, CIN3 andCIN4 as our “ Positives” (medium risk, high risk andcancer risk, respectively).
The original images have the cervicography medical
equipment from which they are captured from in the
background of the images as shown in Fig.2. Therefore, eﬀective segmentation is required to eliminate themand outer cervix lesions but still preserve the aﬀected
part.

25

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 1 Flow diagram of our proposed algorithm

Fig. 3 Colored quantized image and our defined ROI

selected images from our database.
Morphing is achieved by taking two of our color images
fu and fv . We create an intermediate morphed image
fmor using the following equation

Fig. 2 Sample of images with segmented regions of interest

2.2

Mask generation for GrabCut

Eﬀective segmentation of the important regions for distinguishing cervix cancer from non-cervix cancer is very
important but challenging because of similarities in structure and appearance10). For the acquired images segmentation is performed to separate the cervix region (foreground) from the outer region (background). We use
GrabCut segmentation21)for this purpose to capture as
much of the foreground region as possible. It is derivativefrom graph cuts22)with the initially identified information
about the foreground and the background represented bya rectangular selection around ROI.
The aﬀected part is located and covers most of the central region in almost all the acquired images, but theregion of the aﬀected part vary for each image. In previous studies involving cervix segmentation, semi-manual
methods have been adopted23),24). Therefore, to eliminate the need for user intervention to determine the desired rectangle (ROI) around the foreground object, wecreate a segmentation image mask by morphing randomly

fmor (i, j) = (1 − α)fu (i, j) + αfv (i, j),(1)where (i, j) represents pixel position and α is a parameter
which controls the merger rate between fu and fv . In
RGB color space, the vector function f can be denoted
f (i, j) = (R(i, j), G(i, j), B(i, j)), where R, G and B are
the red, green and blue components, respectively.
We used α = 0.5 and applied the morphing process
Eq.(1) iteratively to 100 of our randomly selected images
fu and fv , then we took the average of 100 morphed
images. Since the resulting averaged image being too
noisy, we additionally performed color quantization using k-means clustering25)to obtain the quantized image.
The quantized image is shown in Fig.3. Based on thiswe manually determined four ROIs as shown in Fig.3; areas specified with high probability of known foreground(KF), known background (KB), possible foreground (PF)and possible background (PB). The reason why we chosethese areas is that in most of our image data the fore-

26

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

ground area is inside of KF and the background is outsideof KB, but in some of data the foreground or backgroundis a little larger (inside of PF or outside of PB). Therefore we make an adjustment and use any two of ROIs
as the masks of the foreground and background in thesubsequent GrabCut segmentation, as necessary.
2.3

Cervix region segmentation using GrabCut

Fig. 4 Example of segmentation results

As already described, GrabCut21)is an extension ofgraph cut algorithm used as color image segmentation
method. We use the GrabCut to segment cervix region
from every image data. For the details of GrabCut algorithm, please refer to the literature21).
For each of pixels of an input image either foreground(cervix) or background (non-cervix) is labeled in advance
while referring to the masks KF and KB. Also KF is regarded as an initial specified area including cervix region.
Note that GrabCut usually requires user inputs bydrawing rectangles that provides three rough regions, i.e.
foreground, background, and the unknown areas. But our
method already estimates the four ROIs (KF, KB, PF,
and PB) according to the procedure described above, andthus the method can designate possible regions easily. In
other words it is easy to do the task of labelling evenfor a non-expert, that is, the need to directly draw therectangles on input image for specifying rough regions iseliminated.
Then we input labeled pixels and the initial region into
GrabCut algorithm. The algorithm estimates an initial
Gaussian mixture distribution (GMM) derived from color
information of the foreground and background pixels, andthen assigns one of distributions with maximum likelihood among GMM individually to every foreground pixel.
The parameters of GMM are updated after finishing all
assignments, and background pixels are separated andextracted from the foreground by using minimum cut algorithm22). The above procedure is repeated by a several
times, the GrabCut finally outputs well segmented foreground.
As shown in Fig.4, the aﬀected part is well segmented
by applying the GrabCut with our ROI. It should be
noted that when an obtained segmentation result is not
so good, we can perform the GrabCut again by replacing
the masks KF, KB with PF, PB and the initial specified
area KF with PF.
In this study, almost all tested images are well segmented by selecting ROIs appropriately. However, inFig. 5 Examples of poor segmentation results

some few cases, the result becomes either under- or oversegmentation due to noise in the image from tissue, characters, marks or light reflection as shown in Fig.5. In
Fig.5 the automatic segmented region of our proposedmodel is indicated by the white boundary. Only if suchcases, we manually modify the segmented boundary to beappropriate region indicated by the yellow line as shownin Fig.5.
To verify the eﬀectiveness of our segmentation in moredetail, we preliminarily examined how accurate the GrabCut detects cervix region pixels in an image. For this purpose, we first randomly selected 1,000 images (500 positives and 500 negatives) and then manually segmented
each correct cervix region by hand-drawing the boundary
one by one. Then we regarded those segmented regions as
the correct foregrounds and compared them with pixels
identified as the part of foregrounds in GrabCut imagesby counting overlapping pixels. As a result GrabCut detected correct foreground pixels accurately about 87.1%,
which is the average of 1,000 images. We think this factshows a potential segmentation ability of GrabCut andalso it suggests our method can capture suﬃcient lesions
for color and texture feature extraction, as it is necessary
later to consider both measures together.
2.4

Feature extraction

To determine what features are to be extracted, we use
the visual inspection with acetic acid (VIA) screening ofcervical cancer3)as a guide. In VIA, the degree of whiteness and degree of pinkish are observed and therefore we
consider color features. Moreover, the lesion smoothness
and appearance are also noticed in VIA so we also con-27

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 1 Extracted features based on color
Featurex1 - x4

x5 - x8

Entropy∑255 ∑

ing techniques and every feature is described by a scalarvalue. These values are denoted by xi , i = 1, . . . , 44 inthis paper.
‘Color features’ are firstly calculated from s(i, j). We
consider not only RGB as color space but HSV, LAB andYCr Cb as suitable color spaces, that can provide better
information for diﬀerentiating between lesions, diﬀer asreported in many studies13).
To analyze the cervix lesions we focus on color statistical measures; the mean, variance, skewness, kurtosis and entropy. Furthermore, we apply the individual
separated channels of the four color spaces and analyze
the statistical measure of average pixel intensity. Detailsof color-based features how to calculate and extract aresummarized in Table 1.
‘Texture features’ are derived from gray-scale imagesg (i, j), which can be calculated by using NTSC coeﬃcients as follows:

Description

k=0ch∈{R,G,B} Hch (k) ln Hch (k),where Hch (k) is the histogram ofchannel ch of s(i, j) and k is a levelof pixel value integers in [0, 255]. In
RGB color space, the subscript chmeans one of R,G,B channels.
x2 , x3 , x4 are calculated in the samemanner for HSV, LAB and YCr Cb
color spaces, respectively.
∑∑
Variance N13 (i,j)∈SR ch∈{R,G,B} (sch (i, j) −m)2 ,where N is the number of pixels inside of SR, sch (i, j) is thevalue of channel ch of s(i, j)
and∑m is the∑ mean calculated by
1(i,j)∈SRch∈{R,G,B} sch (i, j).
N3x5 is calculated from RGB color
space, and x6 , x7 , x8 are calculated in
the same manner for HSV, LAB andYCr Cb color spaces, respectively.

x9 - x12 Skewness M3 /(M2 )3/2 ,where M2 is the second moment
which is already calculated as x5 (incase of RGB space). M3 is the third
momentand ∑calculated by M3 =∑13(i,j)∈SRch∈{R,G,B} (sch (i, j) −Nm)3 .
x9 is calculated from RGB color
space, and x10 , x11 , x12 are calculated
in the same manner for HSV, LAB
and YCr Cb color spaces, respectively.

sg (i, j) = 0.299·R(i, j)+0.587·G(i, j)+0.114·B(i, j). (2)Note that each value of sg (i, j) is rounded to integer in[0, 255]. We adopt the gray-level co-occurrence matrix(GLCM)26)to examine texture features. GLCM is a wellknown techniques for analyzing various texture measures
and widely used for image classification.
The matrix element Q(li , lj |∆x , ∆y ) can be derived
from sg (i, j) and it means the relative frequency withwhich two pixels, separated by a pixel distance (∆x , ∆y ),occur within a given neighbourhood, one with gray level
intensity li and the other with lj . Note that the rangeof li and lj is [0, 255]. In the case of using the notation
Q(li , lj |d, θ) each element contains the second order statistical probability for changes between gray levels li andlj at a particular displacement distance d and at a particular angle θ. In this study we set d = 1 and choose
four directions θ = 0◦ , 45◦ , 90◦ , 135◦ , i.e., the matrices
are calculated in (1, 0◦ ), (1, 45◦ ), (1, 90◦ ) and (1, 135◦ )respectively. Then we take average of the calculated results and denote it by Q(li , lj ) again.
According to the matrix Q(li , lj ) obtained, many textural features can be defined. The features on texture weused are summarized in Table 2.
After calculating our features, the ranges of values forthe derived features x1 to x44 vary widely. Since feature
values with wider range will cause misclassification in thelatter cancer prediction process, we normalize every xi as
follows:

M4 /(M2 )2 ,where M4 is the fourth moment andcalculated by
M4 ∑
=
∑
1
(i,j)∈SRch∈{R,G,B} (sch (i, j) −N3m)4 .
x13 is calculated from RGB color
space, and x14 , x15 , x16 are calculated
in the same manner for HSV, LAB
and YCr Cb color spaces, respectively.
∑
x17 - x28 Average N1(i,j)∈SR sch (i, j),pixel in- where ch is one of channels RGB,
tensity
HSV, LAB and YCr Cb color spaces.
x17 is calculated regarding R channel,
x18 is G channel, . . . and x28 is Cbchannel.
x13 - x16 Kurtosis

sider texture features.
Hereafter, the area segmented as foreground (cervix region) is denoted by SR and the part of original image
within SR’s circumscribed rectangle by s(i, j) as shownin Fig.4. Note that s(i, j) is a vector function since it isa color image. In this study we extract 44 lesions properties from s(i, j), 28 are color-based and 16 texture-based.
All features are calculated by using typical image process-xi ← (xi − min{xi })/(max{xi } − min{xi }).
28(3)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 2 Extracted features based on texture (GLCM)
Featurex29

Averagex30

Contrast

x31

Variance

x32

Energyx33

Angularsecondmomentx34

Correlation

x35

x36
x37
x38
x39

x40
x41

As cervical cancer detection is a two class problem
where there are two possible classes of the outcome “cancerous”(positive) or “non-cancerous”(negative), the classof unknown feature vector x belongs to either positive or
negative. We determine the class of x by simply voting
combination denoted as:Positive for ∑ y (x) > 0,k kclass(x) =(4)
Negative for ∑ y (x) < 0.

Description∑255

li =0 li Q̇x (li )(≡ µx ),where Qx (li ) is marginal∑ probability
matrix calculated by 255
lj =0 Q(li , lj ).
∑255 2
k=0 k Qx−y (k),where
Qx−y (k) }is calculated by{∑255.
li ,lj =0 Q(li , lj )|li −lj |=k∑255

li ,lj =0 (li − µx ) Q(li , lj )2ksquare root of the angular second moment x33 .
∑255
2
li ,lj =0 Q(li , lj )where yk (x) is the k th classifier and outputs either 1 whenx is estimated as positive or -1 as negative.
We aim to train and predict whether a class-unknown
image is positive or negative based on our extracted features. For such purposes we select the following supervised learning models as classifiers yk : linear discriminant
analysis28)(LDA), support vector machine29)(SVM), AdaBoost30)(AB), random forest31)(RF), and bagging withdecision tree32)(BDT). These methods have shown to beimportant and popular algorithms widely used in computer aided diagnosis for image classification and havebeen eﬀective for many complex problems27). Note thatthe last three algorithms are known as ensemble classifierin even single use, but in this study we consider somecombinations of the above five learning models and makedecision based on majority voting from classification results obtained by multiple algorithms used.
All of these algorithms are implemented in scikit-learn
libraries33), and we utilize them in the subsequent classification experiments.

∑255

li ,lj =0 {li lj Q(li , lj ) − µx µy },whereµ∑255y is the average calculatedbylj =0 lj · Qy (lj ), and Qy (lj ) is
∑255
calculated by
Furli =0 Q(li , lj ).
ther, σx and σy are the standard
deviations
calculated by∑ respectively2
σx2 = 255(l−µ)
Q(li ) and σy2 =ixx∑255 li =02
lj =0 (lj − µy ) Qy (lj ).
∑2·255
Sum average
k=0 k · Qx+y (k),where
Qx+y (k) }is calculated by{∑255.
li ,lj =0 Q(li , lj )1
σx σy∑2·255|li +lj |=k

Sum entropy − k=0 Qx+y (k) log{Qx+y (k)}(≡eS ).
∑2·2552Sum
k=0 (k − eS ) Qx+y (k)variance∑
Entropy− 255
li ,lj =0 Q(li , lj ) log{Q(li , lj )}∑255
1
Inverseli ,lj =0 1+(li −lj )2 Q(li , lj )diﬀerence
moment∑255
1
Homogeneity
li ,lj =0 1+|li −lj | Q(li , lj )Diﬀerencevariance

variance of Qx−y (k).

x42

Diﬀerence
entropy−x43

Dissimilarity

x44

Inertia3.
3.1

k=0 Qx−y (k) log{Qx−y (k)}∑255

li ,lj =0 Q(li , lj )|li − lj |li ,lj =0 Q(li , lj )(li − lj )2That is, we re-scale each feature xi to a new range [0,1].
2.5

Experiments

Evaluating model

As already described, cervical cancer detection is a twoclass problem. For the experiments in this section we divided the acquired 3,637 images (2,841 negatives and 796
positives) into two sets in advance; training set with 70%of the data 2,545 images (1,988 negatives and 557 positives) and testing set with 30% of the data 1,092 images(853 negatives and 239 positives). Then we evaluatedthe prediction accuracy for testing data. Furthermore,to support the number of true instances we set the parameter average to weighted, due to the in-balance in our
data-set having more negatives than positives samples.
We then measured the performance of our model basedon the accuracy defined as follows:

∑255

∑255

k

Ensemble classification

To examine which features xi are best for cervical cancer classification, we use and evaluate with machine learning algorithm. Particularly it has been reported that ensemble approaches based on majority voting using multiple learning algorithms usually achieve better performance27), and therefore we also consider ensemble methods by stacking diﬀerent learning algorithms.

Accuracy =29

TP + TN,TP + FN + TN + FP

(5)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 3 Parameters used for experiments

where TP, TN, FP, and FN are true positive, true negative, false positive, and false negative, respectively. In our
experiments, TP refers to the number of cervix cancer detected in cervix cancer images, TN refers to the numberof non-cervix cancer detected in non-cervix cancer images, FP refers to the number of cervix cancer detectedin non-cervix cancer images, and FN refers to the numberof non-cervix cancer detected in cervix cancer images.
As our training data size is not so large (2,545 images) and we want results to generalize for unseen data as
well, we evaluate by applying 10-folds cross-validation34)
by randomly splitting the training data-set into 10 folds,
9 folds are used for model training and 1 fold is used fortesting. This processing is repeated 10 times by replacing testing data with one of the other folds, and then we
take the average of 10 estimate results as the final accuracy. We apply similar cross-validation for all subsequentevaluation processes.
3.2

Classifier
LDA
SVM
AB
RF
BDT

Parameterssolver= ‘svd’, tol=0.1
C = 1, γ = 0.1
max depth=2,
weak classifiers=1,000
trees=1,000
trees=500

Fig. 6 Sequential backward selection (SBS) of featuresdue to the default of scikit-learn, we searched for the tolerance (tol) in {1, 0.1, 0.01, 0.001, 0.0001, 0.00001}, andfound that solver=‘svd’ and tol=0.1 give the best result.
All parameters chosen from our experiments are summarized in Table 3.

Determination of parameters for classification

First of all we need to determine parameters of some ofmachine learning algorithms we will use. For this purpose
we executed exhaustive grid searches.
We first examined parameters of SVM (support vector machine). We decided to adopt Gaussian radial basis function (RBF) kernel for SVM, because preliminarily
experiment results further confirmed better performancewith RBF kernel compared to linear and polynomial kernels. Further to optimize SVM’s performance we adjusted
hyper parameters, i.e., cost parameter C and kernel parameter γ. We searched for the values in {0.001, 0.01,
0.1, 1, 10, 100}, and found that C = 1 and γ = 0.1 givethe best result.
Next we investigated parameters of AB (AdaBoost).
Since we selected the decision tree classifier as the baseestimator (weak classifier) due to the default of scikitlearn, we searched for the depth of each decision tree in{1,2,3,4,5}. Also we searched for the values in {100, 200,
500, 1,000, 2,000, 5,000, 10,000, 20,000} to determine thenumber of weak classifiers, and we chose 2 as depth and1, 000 as the number of weak classifiers.
In the same way we also optimized in RF (random forest) and BDT (bagging with decision tree) by searchingfor the number of trees. We found that 1, 000 and 500
trees produce the best result in RF and BDT, respectively. Finally, LDA (linear discriminate analysis) is optimized by searching for the solver in {‘svd’, ‘sqr’, ‘eigen’},3.3

Feature selection

Our aim here is to select the most useful features fromthose described in Table 1 and 2. All the derived features
may not be of importance in cervix cancer classification
and also the higher number of features will lead to an
over fitting easily.
We used sequential backward selection (SBS)33)to determine how many number of features that could be important. SBS starts from the full set of 44 features
Y = {x1 , x2 , . . . , x44 } and returns a subset of features
XNf = {xj |j ∈ 1, 2, . . . , Nf ; xj ∈ Y }; where Nf (< 44)is the pre-defined number of selected features. The results are shown in Fig.6. From the Fig.6 we can see that
performance improves from 10–13 features and peaks at
16-21 features. Therefore we concluded that ranges of
10–21 features are suﬃcient for cervix cancer classification.
We further analyzed the color and texture features using K-best and F-test univariate feature selection (provided scikit-learn library) to rank the features in terms ofimportance. Univariate feature selection examines eachfeature individually to determine the strength of the relationship of the feature with the response variable, andthese methods are simple and are in general particularly

30

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 4 Best feature combinations
Classifier

LDA
SVM
AB
RF
BDT

trice
more
twice

(a)
Features combination
x1 , x3 , x6 ,x14 , x15 , x24 ,x25 ,x30 , x32 , x40
x1 , x6 , x10 , x16 , x24 , x25 , x28 , x30 , x32 , x38
x1 , x3 , x6 , x24 , x28 ,x32 , x38 , x40 ,x42 , x44
x3 , x6 , x14 ,x15 , x24 , x25 , x28 , x32 , x38 , x40
x1 , x3 , x6 , x14 , x16 , x24 , x28 , x38 , x40 , x43(b)Features with multiple occurrenceor x1 , x3 , x6 , x14 , x24 , x25 , x28 , x32 , x38 , x40
x15 , x16 , x30

Fig. 7 Ranking of color-based features using univariate
selection

Fig. 8 Ranking of texture-based features using univariate selection

good for gaining a better understanding of data. (Formore details, please refer to the literature33).)
Results of the color- and texture-based feature rankings
are shown in Fig.7 and Fig.8 respectively. In the bar
graphs p-values for the univariate feature selection andthe corresponding weights of SVM are both plotted, andthose having higher score can be regarded as informativeand significant features. We found that the 21 featuresof which 13 are from color features: x1 , x3 , x6 , x10 , x14 ,x15 , x16 , x20 , x21 , x23 , x24 , x25 , x28 , and 8 from texture
features: x30 , x31 , x32 , x38 , x40 , x42 , x43 , x44 . These areprobably important for classification.
Then we examined the best combinations of the above
21 features on each learning algorithm and the result isshown in Table 4. Table 4 (a) shows 10 best features selected with each algorithm, while Table 4 (b) shows features selected two times or more throughout (bold textmeans trice or more times occurrence). We found 13
important features with multiple occurrences across our
five diﬀerent classifiers: 10 features occurred more than
twice are Entropies RGB (x1 ), LAB (x3 ), Variance LAB(x6 ), Kurtosis HSV (x14 ), Average pixel intensities a∗

Fig. 9 Heatmap of selected features(x24 ), b∗ (x25 ), Cb (x28 ) channels, Energy (x32 ), Entropy(x38 ), Homogeneity (x40 ), and 3 features occurred twice
are Kurtosises LAB (x15 ), YCr Cb (x16 ), Contrast (x30 ).
These 13 features indicate the suitability for classificationand can be selected to obtain favorable results.
We further examined the relationship between the selected features using the heat map shown in Fig.9. In
the map the redder, the higher relationship means and incontrast, the bluer, the lower. As it can be seen, the texture features are more correlated and fewer compared tothe color features. Among the texture features Contrast(x30 ) and Energy (x32 ) are highly correlated but in our
set of 10 features Contrast (x30 ) does not occur. For thecolor features, Kurtosis LAB (x15 ) and Kurtosis YCr Cb(x16 ) are highly correlated with each other and both occur
in the set of 13 features but not in the set of 10 features.
Entropy RGB (x1 ) and Entropy LAB (x3 ) are also highly
correlated with each other and both occur in the set of
10 features. Based on the above observations we drop

31

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 5 Three ensemble combinations we tested
Combination
1st
2nd
3rd
4th
5th
6th
7th
8th
9th
10th

3.5

Classifiers
RF, AB, SVM
RF, AB, LDA
RF, SVM, LDA
RF, AB, BDT
RF, SVM, BDT
RF, LDA, BDT
AB, SVM, LDA
AB, SVM, BDT
AB, LDA, BDT
SVM, LDA, BDT

Statistical analysis

We also performed statistical analysis for evaluation of
our best feature combination. The analysis are specificity
which expresses the ratio of correct positive observations,and sensitivity which is the ratio of correctly predicted
positive events. They were respectively calculated as below:
Specificity = TP/(TP + FP) × 100 [%],
Sensitivity = TP/(TP + FN) × 100 [%].

(6)

Further, we calculated the F1 score which is the weighted
average of specificity and sensitivity. The formula for F1
score is:()
Specificity × Sensitivity
F1 = 2 ×,(7)
Specificity + Sensitivity
it can give further insight into the accuracy of our model
because it takes both false positives and false negatives
into account.
We summarized all the calculated results for 10 featuresand 13 features in Table 6 and Table 7, respectively.
Since we did cross-validation 10 times with each classifier
as already stated, the average and standard deviation isshown in each cell of the Tables. On the whole we cansee from the tables that there is a tendency to show superiority of 3 and 5 ensemble classifiers compared withthe other single ones.
Furthermore, we also calculated the receiver operating
characteristic (ROC) curves for 3 and 5 ensemble classifier with 10 and 13 best features, and plotted them in
Fig.11 – Fig.14, respectively. ROC is known as a common and widely accepted method for analyzing performance of machine learning classification, and the area
under the ROC curve (AUC) can be used to measure how
well the method can distinguish two classes. The calculated AUC values are also shown in the above Tables 6
and 7. We can see from the figures and tables that 0.840
and 0.841 of the AUC is achieved using 10 features for3 ensemble and 5 ensemble respectively, and 0.852 and
0.851 is achieved with 13 features respectively. Althoughusing 13 features has slightly improved scores compared
to 10 features, however, in general, there is very little
statistical diﬀerence between the two approaches.

Fig. 10 Comparison for 3 classifier combinations

Entropy RGB (x1 ) for the set of 10 features and include
Kurtosis LAB (x15 ) instead, and then the new set of 10
features becomes Entropy LAB (x3 ), Variance HSV(x6 ), Kurtosis HSV (x14 ), Kurtosis LAB (x15 ), Pixel
intensity a∗ channel (x24 ), Pixel intensity b∗ channel(x25 ), Pixel intensity Cb channel (x28 ), Energy (x32 ),Entropy (x38 ), Homogeneity (x40 ). We decided to makeuse of the 10 (or 13) best features mainly in the following
evaluation experiments.
3.4

Evaluation of ensemble approach using 3classifier combination

As described in section 2.5, we adopt an ensemble approach for improvement of cervix cancer classification. To
determine which is the best three-ensemble combination,
we calculated the accuracy scores with 10 and 13 features
for all 10 possible diﬀerent combinations shown in Table
5. Obtained results are shown in Fig.10.
From the graph in Fig.10 we can see that 1st combination of 3 classifiers, i.e., random forest (RF), AdaBoost (AB) and support vector machine with RBF kernel (SVM) have the highest average accuracy among ourtested combinations of 3 classifiers. We use this set ofclassifiers for further analysis.

4.

Conclusion

In this paper, we proposed a method for detecting cervical cancer from cervigraph images of aﬀected cervix regions. Our method consist of typical image processing

32

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Table 6 Evaluation of classifiers using 10 best features
Classifier
SVM RBF kernel
Random Forest
Ada BoostThree Ensemble
LDA classifier
Bagging Tree
Five Ensemble

Sensitivity (%)
85.0±1.882.2±4.0
80.6±8.8
83.1±1.586.3±2.081.6±2.0
83.7±1.5

Specificity (%)
81.4±2.782.3±2.1
81.2±2.782.5±1.581.6±2.682.1±2.082.6±1.4

F1-Score
0.824±0.024
0.822±0.021
0.808±0.028
0.827±0.015
0.828±0.029
0.818±0.020
0.830±0.014

AUC
0.841±0.027
0.840±0.021
0.822±0.027
0.840±0.015
0.840±0.026
0.830±0.020
0.841±0.014

Table 7 Evaluation of classifiers using 13 best features
Classifier
SVM RBF kernel
Random Forest
Ada BoostThree Ensemble
LDA classifier
Bagging Tree
Five Ensemble

Sensitivity (%)
83.6±2.3
82.4±1.981.9±1.6
83.4±1.685.9±2.382.8±1.2
83.9±1.8

Specificity (%)
81.4±3.1
82.7±2.082.5±1.5
83.1±1.981.4±2.783.3±1.2
83.2±1.8

F1-Score
0.827±0.024
0.825±0.019
0.820±0.016
0.838±0.018
0.825±0.02
0.829±0.02
0.834±0.01

AUC
0.848±0.031
0.851±0.023
0.823±0.020
0.852±0.015
0.836±0.028
0.844±0.013
0.851±0.018

Fig. 11 ROC of 3 ensemble classifier using 10 features(AUC=0.840)

Fig. 13 ROC of 3 ensemble classifier using 13 features(AUC=0.852)

Fig. 12 ROC of 5 ensemble classifier using 10 features(AUC=0.841)

Fig. 14 ROC of 5 ensemble classifier using 13 features(AUC=0.851)

techniques (GrabCut segmentation and features extraction) and ensemble classification model. Our proposed

segmentation method can covers the aﬀected area aroundthe cervix and our classification experiments showed that33

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）high accuracy can be achieved with a set of few usefulfeatures, comparative to visual inspection based on subjective opinion. The results demonstrated that better
performance over 80% scores compared to VIA can beachieved through computer aided diagnosis. Particularly
only 10–13 features yields satisfactory results suﬃcient todetermine most cases and provide consistent information
for determining CIN.
The use of ensembles classifier can improve the accuracy and reliability of classification. Further, the combination of 3 classifiers (random forest, AdaBoost ans support vector machine) gives the best result with very little statistical diﬀerence from using all 5 classifiers. This
work also demonstrates computer aided diagnosis can potentially assist in cervical cancer diagnosis by requiring
less training, fewer equipment and infrastructure, as well
as fewer specialized medical personnel in remote areas
around the world with high rates of cervical cancer.
The overall system can aid the doctor for making diagnosis decisions, providing quick second opinions and inprioritising patients to be attended to early due to theseverity of their case. Once cancer is suspected referral
to the Oncologist as early as possible is critical since early
detection is important for treatment.
Future work could involve experimenting on other image data-set of varying features and qualities and improving the cervix lesion detection algorithm for eliminating
the cervix boundary. Moreover, we need to examine how
the segmented regions aﬀect the classification accuracy.
Also, feature selection such as using PCA (principal component analysis) for dimensional reduction of features andtesting with other classification algorithms.

pp.187–191 (2014).
4) A. Sarwar, V. Sharma, R. Gupta: “Hybrid Ensemble Learning
Technique for Screening of Cervical Cancer Using Papanicolaou
Smear Image Analysis”, Personalized Medicine Universe, Vol.4,
pp.54–62 (2015).
5) K. Bora, M. Chowdhury, L. B. Mahanta, M. K. Kundu, A.
K. Das: “Automated Classification of Pap Smear Images to
Detect Cervical Dysplasia”, Computer Methods and Programsin Biomedicine, Vol.138, pp.31–47 (2017).
6) T. Chankong, N. Theera-Umpon, S. Auephanwiriyakul: “Automatic Cervical Cell Segmentation and Classification in Pap
Smears”, Computer Methods and Programs in Biomedicine,Vol.113, No.2, pp.539–556 (2014).
7) H. A. Phoulady, M. Zhou, D. B. Goldgof, L. O. Hall, P. R. Mouton: “Automatic Quantification and Classification of Cervical
Cancer via Adaptive Nucleus Shape Modeling”, Proc. of theIEEE International Conference on Image Processing, pp.2658–
2662 (2016).
8) R. Sankaranarayanan, L. Gaﬃkin, M. Jacob, J. Sellors, S. Robles: “A Critical Assessment of Screening Methods for Cervical
Neoplasia”, International Journal of Gynecology and Obstetrics, Vol.89, pp.S4–S12 (2005).
9) D. Song, E. Kim, X. Huang, J. Patruno, H. Munoz-Avila, J.
Heflin, L. R. Long, S. K. Antani: “Multi-Modal Entity Coreference for Cervical Dysplasia Diagnosis”, IEEE Trans. in Medical
Imaging, Vol.34, No.1, pp.229–245 (2015).
10) Z. Xue, L. R. Long, S. Antani, J. Jeronimo, G. R. Thoma: “A
Unified Set of Analysis Tools for Uterine Cervix Image Segmentation”, Computerized Medical Imaging and Graphics, Vol.34,
No.8, pp.593–604 (2010).
11) H. Acosta-Mes, N. Cruz-Ramirez, R. Hernandez-Jimenez:“Aceto-White Temporal Pattern Classification Using K-NN
to Identify Precancerous Cervical Lesion in Colposcopic Images”, Computers in Biology and Medicine, Vol.39, pp.778–784(2009).
12) G. Zimmerman, H. Greenspan: “Automatic Detection of Specular Reflections in Uterine Cervix Image”, Proc. of the International Society for Optical Engineering (SPIE 6144), Medical
Imaging 2006: Image Processing, 61446E (2006).
13) T. Xu, H. Zhang, C. Xin, E. Kim, L. R. Long, Z. Xue, S. Antani, X. Huang: “Multi-Feature Based Benchmark for Cervical Dysplasia Classification Evaluation”, Pattern Recognition,
Vol.63, pp.468–475 (2017).
14) T. Xu, H. Zhang, X. Huang, S. Zhang, D. Metaxas: “Multimodal Deep Learning for Cervical Dysplasia Diagnosis”, Proc.
of the Medical Image Computing and Computer-Assisted Intervention (MICCAI 2016), pp.115–123 (2016).

Acknowledgments
We would like to acknowledge the support and advice
from the National Cancer Institute (NCI) of the UnitedStates.

15) J. Lui, L. Li, L. Wang: “Acetowhite Region Segmentation in
Uterine Cervix Images Using Registered Ratio Image”, Computers in Biology and Medicine, Vol.93, pp.47–55 (2018).

References

16) W. Li, J. Gru, D. Ferris, A. Poirson: “Automated Image Analysis of Uterine Cervical Images”, Proc. of the International
Society for Optical Engineering (SPIE 6514), Medical Imaging
2007: Computer-Aided Diagnosis, 65142P (2007).

1) J. Ferlay, I. Soerjomataram, M. Ervik, R. Dikshit, S. Eser, C.
Mathers, M. Rebelo, D.M. Parkin, D Forman, F. Bray: “Cancer Incidence and Mortality Worldwide: Sources, Methods andMajor Patterns in GLOBOCAN 2012”, International Journalof Cancer, Vol.136, pp.E359–E386 (2015).

17) S. Zhang, J. Huang, D. Metaxas, W. Wang, X. Haung: “Discriminative Sparse Representations for Rervigram Image Segmentation”, Proc. of the IEEE International Conference on
Biomedical Imaging, pp.133–136 (2010).

2) J. Jeronimo, P.E. Castle, R. Herrero, R.D. Burk, M. Schiﬀman: “HPV Testing and Visual Inspection for Cervical Cancer
Screening in Resource-Poor Regions”, International Journal of
Gynecology and Obstetrics, Vol.83, pp.311–313 (2003).

18) S. Gordon, S. Lotenberg, R. Long, S. Antani, J. Jeronimo, H.
Greenspan: “Evaluation of Uterix Segmentation Using Ground
Truth from Multiple Experts”, Australasian Physical and Engineering Sciences in Medicine, Vol.33, pp.205–216 (2009).

3) H. S. Saleh: “Can Visual Inspection with Acetic Acid be
Used as An Alternative to Pap Smear in Screening Cervical
Cancer?”, Middle East Fertility Society Journal, Vol.19, No.3,

19) B. Bai, P. Liu, Y. Du, Y. Luo: “Automatic Segmentation

34

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Suleiman MUSTAFA

of Cervical Region in Colposcopic Images Using K-Means”,
Australasian Physical and Engineering Sciences in Medicine,Vol.41, No.4, pp.1077–1085 (2018).

received both B.Sc. and M.Sc. degrees
in Computer Science from the University of
Hertfordshire in 2004 and 2006, respectively.
He works with NASENI, Ministry of Scienceand Technology of Nigeria as a program Analyst. He is now a Ph.D Student at Iwate
University, Japan.

20) S. Mustafa, A. Kimura:“A SVM-Based Diagnosis ofMelanoma Using Only Useful Image Features”, Proc. of
the International Workshop on Advanced Image Technology(IWAIT2018), pp.1–4 (2018).
21) C. Rother, V. Kolmogorov, A. Blake: “GrabCut: Interactive Foreground Extraction Using Iterated Graph Cuts”, Proc.
of the ACM Transactions on Graphics (SIGGRAPH), Vol.23,
pp.309–314 (2004).

Akio KIMURA (Member)completed the master’s program of Computerand Information Sciences, Graduate School ofEngineering of Iwate University in 1993, andjoined Sony Corporation. While at Sony, he
was engaged in research and development ofmagnetic recording fields. In 1995, he joined
Iwate University, and is now an associate professor of Faculty of Science and Engineering.
He is engaged in research related to image
processing and computer vision. He holds aD.Eng. degree, and is a member of IEICE,IPSJ, and the Society for Art and Science.

22) Y. Boykov, G. Funka-Lea: “Graph Cuts and Eﬃcient N-D Image Segmentation”, International Journal on Computer Vision,
Vol.70, pp.109–131 (2006).
23) A. Alush, H. Greenspan and J. Goldberger: “Automatedand Interactive Lesion Detection and Segmentation in Uterine Cervix Images”, IEEE Trans. on Medical Imaging, Vol.29,
No.2, pp.488–501 (2010).
24) Q. Ji, J. Engel, E. Craine: “Texture Analysis for Classificationof Cervix Lesions”, IEEE Trans. on Medical Imaging, Vol.19,
No.11, pp.1144–1149 (2009).
25) O. Verevka, J. Buchanan: “Local K-Means Algorithm for
Colour Image Quantization”, Proc. of the Graphics/Vision Interface Conference, pp.128–135 (1995).
26) R.M. Haralick, K. Shanmugam, I. Dinstein: “Textural Featuresof Image Classification”, IEEE Trans. on Systems, Man andCybernetics, Vol.SMC-3, No.6, pp.610–621 (1973).
27) S. Dzeroski, B. Zenko: “Is Combining Classifiers Better than
Selecting the Best One”, Machine Learning, Vol.54, No.3,
pp.255–273 (2004).
28) T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning: Data Mining, Inference and Prediction (2nd
Edition), Springer, Section 4.3, pp.106–119 (2008).
29) C. Hsu, C. Chung, C. Lin: “A Practical Guide to Support
Vector Classification”, National Taiwan University Technical
Report, pp.1–16 (2003).
30) J. Zhu, H. Zou, S. Rosset, T. Hastie: “Multi-class AdaBoost”,
Statistics and Its Interface, Vol.2, pp.349–360 (2009).
31) L. Breiman: “Random Forests”, Machine Learning, Vol.45,
No.1, pp.5–32 (2001).
32) L. Breiman: “Bagging Predictors”, Machine Learning, Vol.24,
No.2 (1996).
33) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B.
Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M.
Brucher, M. Perrot, E. Duchesnay: “Scikit-Learn: Machine
Learning in Python”, Journal of Machine Learning Research,
Vol.12, pp.2825–2830 (2011).
34) S. Raschka: Python Machine Learning (1st Edition), Birmingham UK: Packt Publishing Ltd. (2016).

(Received October 24, 2018)(Revised February 21, 2019)35

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
IIEEJ

Paper
Region Control in Stylized Shading Using Radial Transformation
within Texture Projection
Muhammad ARIEF† , Hideki TODO†† (Member), Koji MIKAMI† , Kunio KONDO† (Honorary Member)† Tokyo University of Technology , †† Chuo Gakuin University<Summary> Recent works in Non-Photorealistic Rendering (NPR) are capable of stylizing global appearance to ﬁt shading designs by artists. Meanwhile, local appearance control is still required for theartists to depict geometric and emotional features. We propose a simple region control algorithm for an
existing light-based texture projection technique. Since the shading eﬀects of the texture projection cannot
be overlayed, we consider to deform the original projection result by modifying texture coordinates. To
enable artists to use familiar interface, we incorporate multiple lighting eﬀects and painting control with theoriginal light-based texture projection process. Our shading pipeline is implemented on GPU, which enables
a real-time preview for the shading design at interactive rates. We also conducted user studies to evaluate
the usability of our system.
Keywords: non-photorealistic rendering, stylized shading, texture projection

1.

for more detailed region controls.
Figure 1 illustrates a typical multiple lighting eﬀect
designed in a 2D artwork. In Fig.1, multiple lighting effect is used for emphasizing muscle through the character
body. The character is lit by several lights in order toemphasize features such as muscle in the chest and upperarm. For creating such a multiple lighting eﬀect, we canconsider to combine two texture projection of the Litsphere extension4). However, Fig. 2 shows that simple

Introduction

Non-Photorealistic Rendering (NPR) could accent theimaginative impression to the viewers especially in cartoon animation due to the ﬁctive story that is createdfor it. Artists can give imaginative impression by using
stylized shading for the target 3D object. Most of 3D software provide lighting controls to change the impressionof shading, which is a key factor for cinematic lighting.
A lot of decorative patterns appears in stylized shading
which is resulted from artist expression for portraying object features also emotions1),2). The ﬂexibility in the 2D
environment gives artist to create more expressive artworks rather in 3D environment because artist does notneed to consider many parameters especially for shadingcolor overlay cannot work in this case since color transforms are applied via texture projection in advance.
Figure 3 shows the imaginative shading shape on thecharacter face where this shape is designed for accentuating the mysterious emotion and emphasizing shape fea-

creation. This circumstances also need to be considered
in term of creating the supporting tool for 3D stylizedshading design.
Texture mapping is one of easiest way for creating
hand-drawn appearance. Its simple process can also provide real-time interactions for shading design. In thispaper, we focus on texture projection strategy of Litsphere3) and its extension4). Thanks to the texture-based
shading representation, the resulted shading includes a
lot of unique features which the artist wants to preserve.
While the global shading appearance is well controlled by
the Lit-sphere approach, there are still signiﬁcant needs

Fig. 1 Multiple lights in 2D artwork

36

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

tures. In Fig. 3, the left and middle picture has shading over the eyes and the nose give mysterious feelingto the character. Also round shape shading emphasizing
the roundness of the head in the right picture. The artist
may also need an ability to design such unrealistic shadingin 3D stylized shading. Painting systems1),5) well handle this situation by modifying the original shading result
produced from photorealistic illumination. Therefore, theintegration of painting control with texture-based shading
process has a potential ability to provide a more ﬂexible
design framework for the artists.
In this paper, we propose a simple shading design
framework where artistic region controls can be applied tothe original texture projection result. We start from considering the texture projection process of the Lit-sphere
extension4). Since the ﬁnal shading color is aﬀected by

Fig. 3 Artistic shading shape in 2D artwork(a) original shading

improved style transfer results by using more strict correspondence between rendering elements from light pathexpressions. Similar results can also be obtained by neural network9). While these methods are capable of easyand quick stylization from a style example image, there
are still signiﬁcant needs for controlling the lighting shape
to fulﬁl artistic requirements.
2.2

Stylized shading model

To consider lighting shape control, we review simpler
shading models than example-based stylization. The
most fundamental shading model is cartoon shading
based on a simple 1D texture lookup process for style
conversion. This 1D texture reference brought many styleextensions such as illustrative rendering10) or dynamic
stylized shading primitives11).
X-Toon12) extended the dimension of the texture ref-• Simple region control mechanisms that can be applied to the texture projection method.
• User interaction examples of multiple lighting andpainting operation based on the region control mechanism.

erence by storing the secondary attribute eﬀects in 2D
texture. As for brush stroke style, early example-based
stylization techniques for 2D texture representation13),14)• Simple but eﬀective algorithms to fulﬁll both of stylized design and real-time interactions.

2.1(c) our method

Fig. 4 Comparison of multiple lighting eﬀects

the relation between the input light direction and surface normal, the normal transform operator can change
the underlying illumination. We will introduce a set of
region control mechanisms based on the normal transform functions, while the texture-based shading style canbe preserved. Comparison of three images are shownin Fig. 4. In Fig. 4, our method successfully create
smmothly blended for creating multiple lighting eﬀect.
Comparing to previous method that only capable overlay
texture thus produce unsmoothly blended texture. Thus,
our contributions will be summarized as follows:2.

(b) texture blending

are proposed to model brush stroke styles using samples
from a painted image.
All of these stylized shading models mainly focus on

Related Work

Example-based stylization

Recent NPR researches focus on example-based stylization to synthesize stylized shading appearance fromphotorealistic illumination. Bénard et al. 20136) used

global appearance derived from a 3D lighting result.
Therefore, we need additional techniques for lighting
shape control to fulﬁll artistic demands.

Image Analogies7) for automatic style transfer. Stylit8)

2.3

Painting system

Painting systems can provide eﬀective user interfaces
for artistic stylizations. Inverse design from user inputs
is a common approach for painting systems. There areseveral inverse design systems to obtain the optimal illumination15)–17) or surface normal vector ﬁeld18). MatCap

Fig. 2 Multiple texture mapping limitation using extended Lit-sphere4)

decomposition19) can also recovers expressive material

37

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）representation of the Lit-sphere3).

the texture blending problem on multiple lighting ef-

Another research direction on painting system is to design strokes directly on the 3D model20). However, thisfects, where two projected shading eﬀects are overlappedstrategy lacks the ability to animate the designed strokes
by 3D lighting scheme.
User interface evaluation for 3D lighting design21) con-achieving an intuitive lighting shape control mechanism
while utilizing the expressive shading style through texture projection.

through simple color overlay scheme. We will focus on

cluded that direct interface in common 3D software such
R and 3dsmax R perform faster to
as Autodesk Maya achieve the desired design. Our method focuses on thecreative side of the designed shading and also aim to integrate the painting scheme with the direct light source.
Local control through shading painting system1) en-

3.

In our approach, shading is designed as texture projection onto the target 3D model. To provide further ﬂexibility in our approach, we provide two types of controls:
global appearance and region control. Global appearance
is determined by the common 3D lighting with key lighting elements. Region control enable the user to change
the lighting area of the global appearance result throughmultiple lighting and painting operations.

ables the user to add or erase the shading by local lighting on the 3D object. This method demonstrates seamless tweaking for the detailed shape within key-framing
process. The highlight brush 5) can create various shapeof highlight. While these methods are well integrated
into the existing lighting scheme, the global appearance
through cartoon shading is much simpler than our target
style of hand-drawn shading style.
2.4

Shading Design via Texture Projection

3.1

Global appearance

We assume that a single key light source will generate the global appearance which consists of diﬀuse lighting and specular lighting. For the global appearance weconsider to apply the light space normal deﬁnition described in the Lit-sphere extension 4). With the light

Texture projection

To consider the integration of hand-drawn shading styleand lighting shape control, texture projection is a reasonable approach. The fundamental scheme is proposedby Lit-sphere3), where the view-dependent shading appearance is designed with 2D texture reference. Todo etal. 20134) extended the original Lit-sphere approach tovector L(p) ∈ R3 and the normal vector N(p) ∈ R3 at
the surface point p, light space normal NL (p) ∈ R3 isdeﬁned as follows:
NL (p) = (NLx (p), NLy (p), NLz (p)),(1)

where NLx (p) = N(p) · Lx (p), NLy (p) = N(p) · Ly (p), andNLz (p) = N(p) · L(p) are the light-space normal coordi-

general lighting control with a single light source.
Figure 5 demonstrates the comparison of stylizedshading appearance. In, Fig. 5, the simple abstractionof realistic illumination by cartoon shading lacks the capability of making artistic features such as the irregulargradation in hand-drawn shading styles. The Lit-sphere
extension can include irregular gradation which is commonly found in hand-drawn artworks, while the lighting

nates spanned by the light coordinate frames Lx , Ly , and
L. Figure 6 illustrates the light space normal representation.
The diﬀuse lighting eﬀect is obtained by simple 2D texture sampling with (u, v) = (NLx (p), NLy (p)). Figure 7
shows global lighting appearance. We used two diﬀerence
textures for producing each type of lighting eﬀects suchas diﬀuse lighting, specular lighting. We can encode 2D

control is still possible to design the lighting animation.
However, there are still technical challenges in lighting control for the Lit-sphere extension. Figure 2 shows

shading details for the lighting eﬀect. We also deﬁne thespecular lighting eﬀect eﬀect through the simple modiﬁcation of Eq. (1). Derived from phong reﬂectance term,
we used the reﬂected view vector V for the specular effect by replacing N in the deﬁnition of Eq. (1) to be in
Eq. (2).
Ns (p) = (NSx (p), NSy (p), NSz (p)),(a) cartoon shading(2)

where NSx (p) = V (p) · Lx (p), NSy (p) = V (p) · Ly (p),and NSz (p) = V (p) · L(p) . These additional eﬀects are(b) lit-sphere extension4)

Fig. 5 Comparison of stylized shading appearance

38

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 8 Surface normal vector transformation
Fig. 6 Light space normal

Fig. 9 Multiple lighting behavior

where the z component in the NL (p) is tweaked by thebasic form of the weight function wi (p) ∈ R. Each wi (p)aﬀects the texture sampling for the brightness axes (radial
direction).
With the transformed normal NL = f (NL ), we can ﬁ for texture sam
nally obtain the unit-length normal N
L

Fig. 7 Global lighting appearance

rendered as layered eﬀects. Final shading appearance isdemonstrated in Fig. 7.

pling process.
In our experiments, we apply this region control scheme
for the following detailed lighting eﬀects.

3.2 Region control
The main idea for region control is to modify the lightspace normal coordinates depending on the target userinteractions. Figure 8 illustrates how we can change theﬁnal shading result via surface normal modiﬁcations. In
Fig. 8, the modiﬁcation of surface normal vector ﬁeld willchange the illumination result. The green curve shows thesurface normal on the chest in the original lighting. The

3.2.1

From the general formula of the region control in
Eq. (3), we design the ﬁrst weight function w1 (p) =wml (p) as follows:
wml (p) =red curve shows the transformed surface normal throughour region control operation. Our region control modify

wi (p)(0, 0, 1),

ILk g(|x(p) − xLk |, Rk ),(4)

where ILk ∈ [−1, 1] is the intensity parameter for k-th
light, x(p) ∈ R3 is the surface position, and xLk ∈ R3
the light position. The distribution parameter Rk ∈ R
restricts the each lighting area. The light contributiong(d, Rk ) is computed as follows:

coordinate of the light space normal is the key factor forthe brightness which is computed as the usual 3D lighting models (e.g. Lambertian model for the diﬀuse lighting
eﬀect). Therefore, we consider to transform the z coordinate to control the 2D texture mapping result for moredetailed region controls. We incorporate the z coordinatetransformation f : R3 → R3 into the light space normal
NL as follows:kthe normal direction to become closer to the designed
light direction in this case, which leads the chest area
becomes brighter.
As we can see Eq. (1) with NLz (p) = N(p) · L(p), z

f (NL (p)) = NL (p) +Multiple lighting behavior

g(d, Rk ) =1 − d/Rk00 ≤ d ≤ Rk
Rk < d.

(5)

The function g(d, Rk ) is deﬁned as a clamped linear interpolation between [0, 1] inﬂuenced by the user-given lighting area Rk .
Figure 9 demonstrates the multiple lighting behavior
in Eq. (4). In Fig. 9, two additional lighting eﬀects areused to modify the shading result obtained by global appearance. We visualized the distribution parameter Rk

(3)

i

39

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 10 Painting operations

as 3D sphere with the center xLk for two additional lighting eﬀects. These guidance sphere locations are directly
controlled in 3D space to update the light positions. The
intensity parameter ILk will adjust the overall strength

Fig. 11 Requirement for 2D texture reference and its
variationsof the lighting eﬀect in the area of the guidance sphere.
Negative values for ILk are also possible for making theWith the implemented shading pipeline, our systemprovides a typical lighting design workﬂow summarized
as follows.
First, the user designs the key lighting for the prepared
input 3D scene and camera path. These settings willaﬀect the global appearance whose shading attributes are
the light vector, the normal vector, and the view vector
in section 3.1
Next, the user modiﬁes the global appearance throughmultiple lighting behavior in section 3.2.1 and painting
operations in section 3.2.2. In our experiments, we use
additional lighting eﬀects for multiple lighting behavior.
We also provide additional controls for specular by adding
the control parameters for them. Painting operations arealso extended to specular for more detailed controls.
In the following, we will discuss the characteristics ofour method.

eﬀect darker. All these parameter editing processes willaﬀect the weight function in Eq. (4). These parameters
can be freely adjusted until the desired shading result isachieved.
3.2.2 Painting operation
For the painting operation, we deﬁned the second
weight function w2 (p) = wp (p) with the following simple form:
wp (p) = Ip Cp (p),(6)

where Ip ∈ [0, 1] uniformly controls the power of the eﬀect
and Cp (p) ∈ [−1, 1] is the painted brightness value at thesurface point p.
As shown in Fig. 10, the painted brightness modiﬁes
more detailed lighting areas compared to the multiple
lighting behavior. In Fig. 10, painted brightness Cp (inthe middle) aﬀects the detailed shade in the particulararea. The direct 3d painting process is provided for theediting process. Artist can set oﬀset values which aﬀected
the painted map which internally updated to change the

4.2

In our shading pipeline through texture projection, we
assume a circular pattern in the color distribution of a
2D texture reference. As shown in Fig. 11, our z co-

ﬁnal shading result. Since this local lighting usage canbe darkened, we set the painted brightness range withnegative values.

4.

Requirement for 2D texture reference

ordinate transformation will change the texture lookup
coordinates along radial directions. Figures 11 (a)–(d)portray the impression of the center means the brightest
and the outside becomes darker.
With the requirement as a shading eﬀect, the artist candesign 2D color distributions for the global shading appearance. The specular lighting texture is also designedin the same manner as the diﬀuse lighting texture. To
composite them, the specular lighting texture is clipped

Results and Discussion

4.1 Implementation
Our shading pipeline is implemented in Autodesk
R as plug-in using CgFx shader (GPU shading).
Maya 
Interactive frame rates are either possible for manylights since the method operates a simple modiﬁcation of
texture coordinates in texture projection. The computer
speciﬁcation that used to be the experiment environmentTMR
i7 3.4 GHz, 32GB Memory GeForcewas Intel Core
GTX 770. We can provide real-time preview at 70 FPS(average) for the animation example (with 9201 vertices
and two additional lighting eﬀects).

with alpha regions.
4.3

Multiple lighting behaviour

Since the global appearance is designed by only one
key lighting, the artists want to lit some dark areas resulted in the key lighting. In the experiments on Fig. 9
and Fig. 12, we design multiple lighting eﬀects for static

40

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 13 Multiple lighting case in animated scene

which moving from left to right in the upper part of character, after that we add 2 additional lightings in the rightthigh and left thigh.
Figure 9 and Fig. 12 demonstrate the use case of themultiple lighting result to emphasizing muscle as the features of the character, thus the additional lighting canlit the other parts of character that lead to the stronger
impression. Our function can be acted as a local multiple
lighting for the character.
This kind of style is diﬃcult to achieve using simple multiple texture projection computation since thosemethod can occurs blending problem. Also the additional
light can also move to the other parts of the surface sothat give interactive interaction to the artist.
The multiple texture mapping blending problem asshown in Fig. 2 occurs because multiple 2D textures are
simply composed together. This limitation can be solved
using our proposed method to produce the multiple lighting by transforming the texture sampling for getting newcolor within radial direction with designated intensity parameter and distribution parameter.

Fig. 12 Multiple Lighting Result for difuse lighting with
Fig. 11 (b) as 2D texture reference

models and animated lighting. Figure 11, shows the additional 2 lights incorporate with the 1 key light. The ﬁrst
row shows the original shading with one key lighting. In
the second row, the right thigh is lit by the additional

Figure 13 shows how our method is applicable forthe dynamic scene including the character animation. In

light. In the third row, the left thigh is lit also by theadditional light.
The process of creating Fig. 9 start with inputting the
texture reference as shown in Fig. 11 (a) for the global
appearance as a single diﬀuse lighting which lit from theupper part of the character, then we put 2 additional
lightings in chest and upper arm. In making Fig. 12, thetexture reference in Fig. 11 (b) is used for the diﬀuse lighting eﬀect, which moves from left to right in the upper partFig. 13, the ﬁrst row use only one key lighting and the second row use one additional lighting for lit the leg. Artist
set Fig. 11 (a) as single static lighting for global lighting as diﬀuse lighting then add additional one lighting
in the leg in order to emphasize the leg area. The additional lighting can also be seamlessly blended over the
key lighting and make it easier for the animator to create
seamlessly multiple lighting eﬀects in the animation. The
lighting interface is created to be stick on the surface sothat easier for controlling in animated object rather thanhovering like usual 3D lighting editing operation.

of the character. We then design 2 additional lightings in
the right thigh and left thigh. Same process for creating
Fig. 12, Fig. 11 (b) as an animated single difuse lighting

41

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 14 Multiple lighting result for specular lightingwith Figs. 11 (d) and (f) as texture reference

To full ﬁll the artist intention for achieving multiple
lighting looks certain looks in particular frame, the additional lighting can be incorporated with key framing
technique with animating the light position with common interface and tweak the color with slider interface
using intensity parameter for each light also distribution
parameter.
Multiple lighting also can be applied into the specular
lighting in the animated lighting and object as shown in
Fig. 14 by inputting Figs. 11 (d) and (f) animated diﬀuse
lighting and specular lighting which move from bottom
to up, then the artist add 1 additional light to broaden
the specular lighting. In Fig. 14, the ﬁrst row shows theoriginal shading with one specular lighting. In the secondrow, one additonal lighting is added to make the specularligthting bigger. The result shows that the method can
expand the specular lighting for changing the charactermood into stronger impression while maintaining artistic
result.
4.4

Fig. 15 Painting operation results for specular lightingwith Fig. 11 (d) and Fig. 11 (g) as texture reference and the scene of animated light and animated object

around the nose area to create more mysterious feelingwhile result in Fig. 15 (second row - third column) shows
the capability of the operation to create particular shapefor giving accent. In Fig. 15, negative value is inserted
in the painting operation to erase unwanted shading andgetting certain shape.
Fig. 16 and Fig. 15 (second row - second column) showthat our system also applicable for editing the unwanted
shading in animated scene toward diﬀuse lighting andspecular lighting respectively. In Fig. 16, the ﬁrst row
shows the shade before the painting operation and thesecond row shows shading after painting operation. Artist
set the negative value in the painting operation so thatdarker area can be achieved to get certain shape.
Such modiﬁcations are diﬃcult to achieve by the multiple lighting behavior where the lighting eﬀects are affected by the model’s geometry. Our system provides dif-

Painting operation

Sometimes artists want to make imaginative shading tochange the mood of the character so that familiar paintinterface is also provided to modify the shading in arbitrary areas for making various shapes. These modiﬁcations act as local lighting eﬀects, which will be seamlessly
integrated with the texture projection result in global ap-

ferent levels of lighting controls depending on the artistic
requirement.
While our system provides a controllable single brightness map for the whole animation, we can observe thatthe painted lighting eﬀects are useful for cut shots. Depending on the demands of artists, we require key-framing
scheme like Todo et al. 20071) .

pearance.
Compared to the multiple lighting behavior, thepainted shading eﬀects can adjust more local lighting indetails. By using the same scene as used in Fig. 14,
we demonstrate the usability of painting operations in
Fig. 15 and Fig. 16 as shown in Fig. 10, our paint interface enables the artist to design the unrealistic shape

4.5

User study

The questionnaire was conducted in order to get theviewer’s impression to our result which seamlessly blends
multiple lighting eﬀects. 5-point Likert rating scales areused to gather the subjective opinions of the participant

42

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 1 Quistionnaire resultAB
1 2 3 4 5 1 2 3 4Naturalness
2 8 2
8 2 1
Beautifulness
3 6 3
3 3 3

5
1
3

Fig. 18 Artist’s results

result shows that many participants chose our result for
the questions about the naturalness and beautifulness.
We also tried to conduct a user study with an artist to
consider more practical scenarios and get opinion about
the usability of the proposed method. The artist is from

Fig. 16 Painting interface result for difuse lighting using
Fig. 11 (c) as 2D texture reference and the scene
contains animated light and animated object

the undergraduate student who has the best skill among
the others. As for the introduction to the method, we
brieﬂy explained to the artist about the use of the shading
also the user interface to operate.
Roughly the experiment done in 30 minutes. Firstly
the artist created one 2D texture for making the global
appearance. After applying the 2D texture, artist put
3 additional lighting prior to one global lighting, whichmainly placed on the head speciﬁcally on the mouth, eye,and ear. These additional lightings are intended to lit thehead that cannot be lit just from 1 global lighting. Thus,
these additional lightings can enhance the shape of thehead. In addition, the artist made more speciﬁc tweak
using painting interface for sparsely erase shading on thecharacter. This produces a stronger emotion impression
toward the viewers. The result of this experiment can

Fig. 17 Questionnaire for Fig. 4

towards the resulted shading in Fig. 4 and relation toeach question.
The questionnaire as shown in Fig. 17 consists of Fig. 4
picture and two questions with 5 point Likert rating scalesin order to collect subjective evaluation from each question. The participant chooses the scale according to themthough about the relation between the picture and eachquestion with 1 is the lowest mark and 5 is the highest
mark. We focus on the aesthetic perspective such naturalness and beautifulness. Thus, the questions are deﬁned
as follows:

be seen on Fig. 18. We asked the artist to evaluate
the usability of our system. The impression was positive:
the operation is enough intuitive to perform editing as
she expected. The slider interface in the method makes
artist easier to change the look of each additional lighting
also in the painting interface that uses for choosing thecolor.

• Which one is natural as shading eﬀect?
• Which one is beautiful?
We conducted the experiment with 12 participants,
mainly form the university students with various art
skills. Each participant was introduced to the problem
brieﬂy then give the freedom to decide their choices.

5.

Conclusion

In this paper, we have discussed region control for a
3D lighting scene with hand-drawn appearance in NPR
scenarios. We propose a simple z coordinate transformation in dynamic texture projection to provide the user
with lighting modiﬁcations such as multiple light eﬀects

From that, we collected the values and analyzed the result.
Our result can receive good impression about the aesthetic perspective. In Table 1, we can see the actual

43

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）and painting operation. Our implementation ﬁts the ex-7) A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, D. H.
Salesin: “Image Analogies”, Proc. of SIGGRAPH 2001,
pp.327–340 (2001).

isting pipeline with the familiar user interface and run atinteractive rates for real-time previewing.

6.

8) J. Fišer, O. Jamriška, M. Lukáč, E. Shechtman, P. Asente,
J. Lu, D. Sýkora: “StyLit: Illumination-guided Examplebased Stylization of 3D Renderings”, ACM Trans. on Graphics,
Vol.35, No.4, pp.92:1–92:11 (2016).

Limitation and Future Works

While we have shown how our method is applicable forseveral practical scenarios to satisfy the artistic requirements, there is still room for improvement.
Since we have focused on lighting behavior for the texture projection, our transformation only handles radial
directions in texture mapping process as shown in Fig. 11.
We need to discuss a more general transform along arbitrary directions with corresponding practical scenarios.
For example, several aﬃne transformations are discussed
in previous Lit-sphere extension4).

9) L. A. Gatys, A. S. Ecker, M. Bethge: “Image Style Transfer
Using Convolutional Neural Networks”, Proc. of Conference
on Computer Vision and Pattern Recognition (CVPR) 2016,
pp.2414–2423 (2016).
10) J. Mitchell, M. Francke, D. Eng: “Illustrative Rendering inTeam Fortress 2”, Proc. of Non-Photorealistic Animation and
Rendering (NPAR) 2007, pp.71–76 (2007).
11) D. Vanderhaeghe, R. Vergne, P. Barla, W. Baxter: “Dynamic
Stylized Shading Primitives”, Proc. of Non-Photorealistic Animation and Rendering (NPAR) 2011, pp.99–104 (2011).
12) P. Barla, J. Thollot, L. Markosian: “X-toon: An Extended
Toon Shader”, Proc. of Non-Photorealistic Animation and
Rendering (NPAR) 2006, pp.127–132 (2006).

Another limitation is that we assume only single 2D
texture reference for our lighting controls. As the future
works, we also artist want to explore the multiple texture
projection with various styles from a set of 2D texture
references so that these shading eﬀects can be seamlessly
blended and animated.
The current painting framework is limited to a single
brightness map for the whole animated scene. We may require the ability to modify the painted brightness map dynamically including dragging operations or key-framing
schemes. In addition, our scheme restrict the artist to
combine the diﬀerent levels of lighting controls. Inverse
decomposition schemes may be desirable to directly animate the painted shading from scratch.
In all future directions, we consider that providing intuitive shading design user interface is essential for animating hand-drawn shading in NPR.

13) C.-R. Yen, M.-T. Chi, T.-Y. Lee, W.-C. Lin: “Stylized Rendering Using Samples of a Painted Image”, IEEE Trans. on
Visualization and Computer Graphics, Vol.14, No.2, pp.468–
480 (2008).
14) C. D. Kulla, J. D. Tucek, R. J. Bailey, C. M. Grimm: “Using
Texture Synthesis for Non-Photorealistic Shading from Paint
Samples”, Proc. of Computer Graphics and Applications 2003,
pp.477–481 (2003).
15) J. Obert, J. Křivánek, F. Pellacini, D. Sykora, S. Pattanaik:“iCheat: A Representation for Artistic Control of Indirect Cinematic Lighting”, Proc. of Eurographics Symposium on Rendering (EGSR) 2008, pp.1217–1223 (2008).
16) Y. Kim, J. Noh: “LightShop: An Interactive Lighting System Incorporating the 2D Image Editing Paradigm”, Proc. ofInternational Symposium on Visual Computing (ISVC) 2009,
pp.59–70 (2009).
17) W.-C. Lin, T.-S. Huang, T.-C. Ho, Y.-T. Chen, J.-H. Chuang:“Interactive Lighting Design with Hierarchical Light Representation”, Proc. of Eurographics Symposium on Rendering(EGSR) 2013, pp.133–142 (2013).
18) M. T. Bui, J. Kim, Y. Lee: “3D-look Shading from Contours
and Hatching Strokes”, Computers & Graphics, Vol.51, No.C,
pp.167–176 (2015).

References
1) H. Todo, K. A’njyo, W. Baxter, T. Igarashi: “Locally Controllable Stylized Shading”, ACM Trans. on Graphics, Vol.26,
No.3, Article No.17 (2007).

19) C. J. Zubiaga, A. Muñoz, L. Belcour, C. Bosch, P. Barla: “MatCap Decomposition for Dynamic Appearance Manipulation”,
Proc. of Eurographics Symposium on Rendering (EGSR) 2015
(2015).

2) P. Wisessing, J. Dingliana, R. McDonnell: “Perception ofLighting and Shading for Animated Virtual Characters”, Proc.
of Applied Perception 2016, pp.25–29 (2016).

20) K. Bassett, I. Baran, J. Schmid, M. Gross, R. W. Sumner: “Authoring and Animating Painterly Characters”, ACM Trans. on
Graphics, Vol.32, No.5, pp.156:1–156:12 (2013).

3) P.-P. J. Sloan, W. Martin, A. Gooch, B. Gooch: “The Lit
Sphere: A Model for Capturing NPR Shading from Art”, Proc.
of Graphics Interface 2001, pp.143–150 (2001).

21) W. B. Kerr, F. Pellacini: “Toward Evaluating Lighting Design Interface Paradigms for Novice Users”, ACM Trans. on
Graphics, Vol.28, No.3, pp.26:1–26:9 (2009).

4) H. Todo, K. Anjyo, S. Yokoyama: “Lit-Sphere Extension for
Artistic Rendering”, The Visual Computer, Vol.29, Issue 6-8,
pp.473–480 (2013).

(Received November 29, 2018)5) R. Pacanowski, X. Granier, C. Schlick, P. Poulin: “Sketch andPaint-based Interface for Highlight Modeling”, Proc. of SketchBased Interfaces and Modelling (SBIM) 2008, pp.17–23 (2008).
6) P. Bénard, F. Cole, M. Kass, I. Mordatch, J. Hegarty, M. S.
Senn, K. Fleischer, D. Pesare, K. Breeden: “Stylizing Animation by Example”, ACM Trans. on Graphics, Vol.32, No.4,
pp.119:1–119:12 (2013).

44

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Muhammad ARIEF

Koji MIKAMI

He received the B.S. from Institut Teknologi
Sepuluh Nopember and M.S. from Tokyo University of Technology. His research interests include non-photorealistic rendering, andgame design.

He worked at Nissho Iwai Corporation and
MK Company as a producer. In 1998, he established “Creative Lab.” at the Katayanagi
Advanced Research Institute of Tokyo University of Technology (TUT), where research
on Animation, Game and CG production
technology are conducted. Currently, he isa professor at the School of Media Science,
TUT. He received his Ph.D. degree at KEIO
University in 2018. Now his research interests
lie in game design and production technology
for game and animation.

Hideki TODO (Member)
He is an assistant professor at Chuo Gakuin
University. His work experience includes
working as a researcher in animation industry at OLM Digital, Inc. from 2011 to 2013.
He received his Ph.D. degree of Information
Science and Technology from the Universityof Tokyo in 2013. His research interests lie
in computer graphics, image processing, andcomputer vision, including stylized rendering,
image stylization, inverse rendering, and image or video editing.

Kunio KONDO (Honorary Member)
He is a professor at the School of Media Science, Tokyo University of Technology. He received his Ph.D degree from the Universityof Tokyo in 1988. He was associate professorof Department of Information and Computer
Sciences, Saitama University, and Technicalstaﬀ of Nagoya University. He is the president of Asia Digital Art and Design Association. He was former President of TheInstituteof Image Electronics engineers of Japan, former President of The Society for Art and Science, and Chair of SIG on Computer Graphics
and CAD of Information Processing Societyof Japan.

45

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
IIEEJ

Short Paper
Region Mining of Fetal Head in Ultrasound Image Based on
Weakly Supervised Annotations and Deep Learning
Yan LI† (Student Member), Rong XU†† , Artus KROHN-GRIMBERGHE††† , Jun OHYA†, †† (Member),
Hiroyasu IWATA†† Department of Modern Mechanical Engineering, Waseda University ,†† Global Information and Telecommunication Institute, Waseda University ,††† Department of Business Information Systems, Paderborn University<Summary> To locate the fetal head in ultrasound (US) images, this paper proposes a deep learning
based method for weakly supervised learning from image-level annotation. We first modify and train fetal
head classification models based on existing backbone structures, then adopt the feature maps and learned
weights to visualize the high response areas of fetal head. In order to improve the localization accuracy,
this paper further optimizes completeness of the salient area of the fetal head by adopting multiple featuremaps from diﬀerent feature levels. The final bounding box of the fetal head is obtained from mined regions
through threshold. We evaluate both fetal head plane classification and weakly learned localization resultsin US images. In the experiments we compare several backbone structures and verify the eﬀectiveness ofthe proposed method.
Keywords: weakly supervised learning, fetal head localization, ultrasound, convolutional neural network

1.

into hieratical feature spaces, then the feature maps canbe adopted as salient image of the target object throughlinear combination. In case of medical image processing, recently, some of the publications directly use themethod to visualize the interesting area of the illness, forinstance, Xiaosong Wang et al’s work5). They mainly target at the classification tasks. The extracted response of
the last convolution layers have small size and high feature level, which cause the response map only reflects on
the regions which have the most discriminative information. Yunchao Wei et al.6) propose to solve the problem
by iteratively masking the pixels which have high confidence then re-classify the image from the rest of the
pixels. However the method cannot provide a reliable
termination method, the mining region has risk to beover-segmented. Therefore, the issue of the completenessof the weakly supervised region mining is still need to befurther solved.
The targets of the paper are illustrated in Fig.1. Given
US images, through learning from image level annotations, we aim at mining the region of the fetal head fromthe learned models. The region of the fetal head is represented as a tight bounding box of the fetal head area. We
first propose a fetal head classification framework basedon exist backbone networks, then use discriminative maps

Introduction

Ultrasound (US) imaging is widely used in fetal prenatal diagnosis due to its low cost and low harm to thehuman body. Recently, automatic fetal care systems aredesired by medical areas. Such a system requires a technology that locates the fetal head so as to infer the gestureand position of the fetus; then, the system can perform
subsequent processes such as guiding the US probe to thedesired positions for further measurement.
Classification of fetal head plane from US image has
been studied for a long time, e.g. by L Zhang et al.1) and
Hao Chen et al2). However, in many cases of automatic
medical treatments, classification of the category of US
plane is not enough. We need to locate the position of
the targets in the image in order to provide more reliablereferences. Normally, object detection model can be obtained by fully supervised learning, such as in the work
proposed by Albayrak et al3). However, these methods
rely on accurate manual annotation of the target position, such as pixel-wise classification mask or boundingboxes, which cost a large amount of human resources.
The hieratical feature can be used to visualize the discriminative region by the method proposed by Zhou etal4). The convolution neural network maps the input data

46

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

function is written by1∑[yn log pn + ω(1 − yn ) log(1 − pn )], (1)
n n=1
n

L=−

where yn , pn indicate binary values of the label and prediction of the model, respectively.
2.2

Localization of fetal head by multi-scale
discriminative maps
We adopt the feature maps’ responses of the last convolution layer and the weights of the output layer to obtain the discriminative area from the learned model. In
particular, the input blob of the output layer is calculated from global averaging over the feature maps f ofthe last convolution layer. Then the compressed feature
is directly mapped into the shape of the output vector bylinear combination with the weights corresponding to thetwo categories. The weights of the output layer w andfeature maps f are adopted for visualizing the discriminative maps of the desired categories.
The highest feature level in the deep learning modelis learned from the most discriminative areas of the input feature maps. However, the most discriminative areacannot be seen as the complete area of the target object.
The mined region lacks completeness due to presence ofsome of not so important areas. This makes finally located fetal head area is not good enough.
The pooling layers are used multiple times to narrow
down the original image through averaging or maximum
operations. In our research, the response maps for the discriminative localization are extracted from the last output of the hieratical layers. As the feature level becomesdeeper; the semantic features get higher. The less important feature areas are progressively ignored by the network. We think the ignorance could degrade the completeness of target shape. To deal with the issue, we propose to adopt the outputs from the multi-level features.
The response maps from diﬀerent feature levels represent
diﬀerent discriminative locations.
The proposed method replaces the output with multiple output branches which split from diﬀerent level ofbackbone network’s outputs and merge the multiple discriminative maps into one, as shown in Fig.2. In particular, for each output branch, we add specified numberof convolution layers and global average pooling operations. During the training, the output of each branch iscompared with the ground truth labels, and the weightsof each output branch are independently updated. By
merging multiple outputs, our models learn discrimina-

Fig. 1 Localization of fetal head by learning from image
level annotationsof fetal head which are merged from multi-scale featuremaps to achieve the target.

2.
2.1

Proposed Method

Fetal head plane classification from US
imagesThe modified CNN models are used to train classifier
from US slice and image level fetal head annotations,which are manually specified if each image includes a fetal head. To classify the input US image into diﬀerent
categories, the CNN model uses hieratical designed backbone network to extract the feature. In this paper wetreat oﬀ-the-shelf networks as the feature extractors andtheir base structures remain unchanged.
In order to visualize the discriminative area fromlearned classification models, the modifications comefrom the output of the network structure. In order tomap the output into one dimensional vector, which indicates whether a fetal head exists, the normal CNN classification model first needs to reshape the responses of the
convolution layers into a vector and then connects thefeature vectors to inner production layers. However, theinner production changes the geometric feature, because
all of the pixels are mapped into one through linear combination. We need to use discriminative feature maps to
localize the fetal head area. Therefore, we replace the reshape and inner product operations of the selected backbone structures by a single global average pooling layer.
During the training phase, we use typical cross entropy
loss to update the weights of our binary classification
model. In the sample US video sequence, fetal heads
exist only in a small number of the sequences. This leads
to the situation in which the data distribution has severe
imbalances. Therefore, we modify the loss function withweights of negative category. The weight ω is calculatedby ω = P/N , where P and N are number of positive andnegative samples. For each batch with n samples, the loss

47

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Fig. 2 Merging of multi-scale discriminative maps

23 are used for the experiments. The 2-folder cross validation is adopted by alternatively training and testingon randomly selected groups of US clips (each group has
two clips) with diﬀerent subjects. We select the clips thathave fetal body perpendicular to the US scan plane. After
pruning and interval sampling operations, the US video
clips are stored in sequential image format. The framesthat with observe-able fetal head are manually identified
from the image sequences. Then, we assign pixel-wise labels to each of the images containing fetal heads. Notethat the pixel-wise annotations are only used for evaluation purpose. The bounding boxes of the fetal head are
calculated from thus obtained segmentation masks.

Fig. 3 Detailed structure of output branches added on
VGG19tive response maps from layers with diﬀerent feature levels. The branch structure with detailed parameters on
VGG19 structure can be found in Fig.3. During the
localization, we extract the response maps of the lastconvolution layers from each output branch and use theweights of each branch separately to linear combine themas multiple discriminative maps. At last we use a sum
calculation of each pixel to obtain the final discriminative image M of fetal head. The optimized approach canbe presented as
M=

∑ 1 ∑
i

ki

wF i fi ,3.1.2

Our dataset has severe data imbalance and lacks of diversity issues. Therefore, we use related large scaled data
augmentation operations on the training set. We add random crop, rotation, scale transform, and horizontal flip
to the raw US image to obtain disturbance to each of the
training samples. The augmented samples are resized to
the fixed size in order to fit the input of the backbone
network.

(2)

kiwhere wF i and fi indicate the weights that correspond to
the fetal head and feature maps of the ith output branch,respectively; ki is the number of kernels of the ith output
branch.

3.
3.1
3.1.1

Data augmentation

3.1.3

Domain transferred learning

Compared with learning from random initialed weights,
better initializations can be obtained from pre-trained
low-level representations that learned from large scalecross domain images. Most of weights of the convolution
layers are updated from imagenet7) pre-trained models
while the additional layers with learn-able weights arelearned from scratch.

Experiment

Dataset and training details
Data collection

We conduct preliminary experiments on clinical US
dataset to verify the proposed method. The research acquires four clips of US pregnant examinations from a hospital as the metadata. For each US video clip, anonymous
patient with diﬀerent fetal weeks that arranged from 19 to3.1.4

Threshold

The discriminative map is normalized by rescaling thevalues from 0 to 1. In our experiments we use a fixed value(0.8) to obtain binary masks. To determine the threshold

48

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）
Table 1 Classification results of diﬀerent backbone networks (%)

value, we quantitative evaluate the IoU of region mining
results on the training sets and roughly select the values
which have the best performance on each sub folder. The
bounding box of the fetal head is obtained from connecteddomain. We only keep the bounding box which has thelargest area if multiple areas are obtained.

Network AlexnetGAP Resnet50GAP VGG19GAP
Recall
87.79
86.92
90.92
Precision
88.65
90.73
90.26
F1 score
87.62
88.39
90.35
AUC
95.87
96.34
96.80
Table 2 Localization results (IOU) with diﬀerent backbone networks in VGG19GAP OutputMerge(%)

3.2 Results and discussions
3.2.1 Metrics
First, to evaluate the model for fetal head plane classification, we adopt precision, recall and f1 score. They
can be defined as
Recall × P recision
f1 = 2 ×,(Recall + P recision)

Network Alexnet
GAP
Bbox
Pwise

(S ∩ SGT ),(S ∪ SGT )VGG19GAP
OutputMerge
41.4(37.0) 40.3(36.1) 61.2(54.8) 65.3(58.0)
60.7(58.8) 57.7(55.0) 72.0(70.8) 76.5(73.1)(3)
Table 3 Localization results (IOU) with diﬀerent output layers (%)

where Recall = NT P /(NT P + NF N ), P recision =NT P /(NT P + NF P ), the NT P , NF N and NF P are numbers of true positive, false negative and false positive
predictions, respectively. We also provide the Area Under Curve (AUC) of Receiver Operating Characteristics(ROC). Regarding the localization accuracy, we obtain
the bounding box from manually annotated segmentation masks of the fetal head. We adopt Intersection Of
Union (IOU) to evaluate the results. The IOU is defined
as
IOU =

Resnet50 VGG19
GAP
GAP

OutputConv4 4
layer
Pwise
19.83
FetalHead
Pwise
92.80
Bkg
Bbox
53.20
Pwise
56.32

Conv5 4

Conv6 3

51.51

54.95

Output
Merge
57.94

95.40

94.83

95.13

62.24
73.46

55.71
74.89

65.25
76.53

following experiments we build our optimizations basedon VGG19 structure.
The comparison of the IOU of the bounding box ofthe three backbone structures can be seen in Table 2.
Here, values not in bracket show w/o false negative classifications, and values in bracket show w/ false negative classifications in Table 2. In Table 2, to decouple the localization and classification performance, thescores (not in brackets) do not count false negative classification samples, while the scores in brackets are the
IoU results with false negative classifications. The results of resnet50GAP are worse than alexnetGAP probably because it has too small feature map size. Same as
the classification results, the VGG19 achieves the best
localization results. By adding our proposed optimization method (VGG19GAP OutputMerge), the localization performance is further improved. We compared theresult of only using the last feature map output and thetwo optimization methods in the Table 3. The nameof each result indicates the output of diﬀerent layers
in VGG19GAP. In Table 3, pixel-wise accuracy of fetal head (“ FetalHead”) and other areas (“ Bkg”) aredemonstrated separately. Regarding the proposed multiple discriminative outputs, we adopt the conv4 3, conv5 3and conv6 3 from VGG19 structure (the detail is shownin Fig.3). The result obtained by merging the above-(4)

where S and SGT are obtained area of the predictions andthe ground truth. Besides IOU of bounding box (Bbox),
we also provide the pixel-wise IOU (Pwise). The calculation of pixel-wise IOU is similar as bounding box IOU,
except we judge the classification accuracy at each of the
pixels of the input image.
3.2.2 Results
Regarding the classification model, we compare several popular backbone networks, which are alexnet8),
resnet509) and VGG1910). For both Alexnet and VGG19
structures we add batch normalization11) after each convolution layer. As we mention in section 2.1, we replace
all of the fully connecting layers with the convolution
layer and global average pooling operations.
The comparison results of the three models for the fetal
head plane classification task can be found in Table 1.
From the result we can see that, among the selected
three models, the VGG19 with batch normalize structure
achieves the best performance. Although the alexnetGAP has simple structure and the smallest network scale,it still can get acceptable classification rate because of thebatch normalization and the pre-trained weights. In the49IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）tages over learning only from image level annotations, inthat the weakly supervised method can avoid complex
and time-consuming medical image annotation tasks.

4.

This paper has proposed a method for training fetal
head plane classification models for US pregnant images.
The weakly supervised method is adopted to obtain the
fetal head area. This paper discusses the insuﬃcient of
the current method, and tries to inspire the future research to merge multi-scaled discriminative maps with
diﬀerent feature levels to get more complete salient areas. The eﬀectiveness of proposed method is verified
through preliminary experiments. More experiments onlarger scaled clinical US dataset need to be conducted for
future works.

obtainedbyFig. 4 Visualized
results
VGG19GAP OutputMerge;a) merged discriminative map, b) predicted fetal head region
on input image, c) obtained bounding box (yellow) and ground truth (red) of fetal head region,and d) classification probability of positive
category for each sample

References

Table 4 Localization results (IOU) of weakly and fully
supervised methods (%)
MethodVGG19GAPVGG19GAP
OutputMerge
F-RCNN12)

Weaklysupervised
√
√
×

Fully
supervised
××√

Conclusion

1) L. Zhang, S. Chen, C. T. Chin, T. Wang, S. Li: “Intelligent
Scanning: Automated Standard Plane Selection and BiometricMeasurement of Early Gestational Sac in Routine UltrasoundExamination”, Medical Physics, Vol. 39, No. 8, pp. 5015–5027(2012).

Bbox
(IOU)
61.2
65.3

2) H. Chen, D. Ni, J. Qin, S. Li, X. Yang, T. Wang, P. Heng:“Standard Plane Localization in Fetal Ultrasound via Domain
Transferred Deep Neural Networks”, IEEE Journal of Biomedical & Health Informatics, Vol. 19, No. 5, pp. 1627–1636 (2015).

72.2

mentioned three outputs is indicated by “OutputMerge”.
From the quantitative results we can see that the merged
feature maps bring significant improvements by enriching
the integrity of the fetal head area. Some of the visualized
results are shown in Fig.4.
In addition, to evaluate the accuracy diﬀerence with the
fully supervised approach, we further demonstrate the results which are obtained from Faster RCNN12)(F-RCNN).
Note that, the model of F-RCNN is learned from bounding box annotations. Compared with weakly supervised
methods, it is impossible to learn from image level annotations. To train this model, we adopt the manuallylabeled bounding box annotations, which are originallyused for evaluation. In particular, to the structure of
F-RCNN model we embed the original VGG16 as the
backbone structure and use 600 × 600 pixels as the input
resolution for both training and testing. The comparison results are shown in Table 4. From the results wecan see that, comparing with the baseline weakly supervised method (first row), the method with our proposed
optimizations (second row) shorten the accuracy gaps between the models which learn from image level annotations and from object position annotations. On the other
hand, the weakly supervised method has strong advan-

3) B. A. Nur, B. O. Ayse, S. A. Yusuf: “Prostate Detection
from Abdominal Ultrasound Images: A Part Based Approach”,
Proc. of IEEE Conference on International Conference on Image Processing, pp. 1955–1959 (2015).
4) B. -L. Zhou, B. , A. Khosla, A. Lapedriza, A. Oliva, A. Torralba: “Learning Deep Features for Discriminative Localization”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016).
5) X. -S. Wang, Y. -F. Peng, L. Lu, Z. -Y. Lu, M. Bagheri, R. M.
Summers: “ChestX-ray8: Hospital-scale Chest X-ray Databaseand Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3462–
3471 (2017).
6) Y. -C. Wei,J. -S. Feng, X. -D. Liang, M. -M. Cheng, Y.
Zhao:“Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach”, Proc.
of IEEE Conference on Computer Vision and Pattern Recognition, pp. 6488–6496 (2017).
7) J. Deng, W. Dong, R. Socher, L. -J. Li,K. Li, F. -F.
Li:“ImageNet: A Large-scale Hierarchical Image Database”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition, pp. 248–255 (2009).
8) A. Krizhevsky I. Sutskever, G. Hinton: “ImageNet Classification with Deep ConvolutionalNeural Networks”, Advancesin Neural Information Processing Systems 25, pp. 1097–1105(2012).
9) K. -M. He, X. -Y. Zhang, S.-Q. Ren, J. Sun: “Deep Residual
Learning for Image Recognition”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition, pp. 770–778 (2016).
10) K. Simonyan, A. Zisserman: “Very Deep Convolutional Net-

50

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.1 （2019）

Jun

OHYA (Member)
He earned B.S., M.S and Ph.D degrees in precision machinery engineering from the University of Tokyo, Japan, in 1977, 1979 and1988, respectively. He joined NTT ResearchLaboratories in 1979. He was a visiting research associate at the University of Maryland, USA, from 1988 to 1989. He transferredto ATR, Kyoto, Japan, in 1992. Since 2000,
he has been a professor at Waseda University,
Japan. He was a guest professor at the University of Karlsruhe, Germany, in 2005. His
research areas include computer vision andmachine learning. He is a member of IEEE,IIEEJ, IEICE, IPSJ and VRSJ.

works for Large-Scale Image Recognition”, arXiv:1409.1556(2014).
11) S. Loﬀe and C. Szegedy: “Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate Shift”,
Proc. of the 32nd International Conference on Machine Learning, pp. 448–456 (2015).
12) S. -Q. Ren, K. -M, He, R. Girshick, J. Sun: “Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks”, Proc. of International Conference on Neural Information Processing Systems, Vol.1, pp. 91–99 (2015).

(Received June 22, 2018)(Revised December 4, 2018)

Yan

LI (Student Member)
He earned B.S. degree in Communication
University of China. He obtained M.S. degree in Graduate School of Global Information and Tele-communication Studies (GITS),
Waseda University. He is currently a doctoral
program student in Graduate School of Creative Science and Engineering, Waseda University. His research areas include computer
vision and machine learning.

Hiroyasu IWATA
He earned the B.S., M.S. and Dr. of Engineering degrees in Mechanical Engineeringfrom Waseda University in 1997, 1999 and2003, respectively. He is currently a Professorof Department of Modern Mechanical Engineering, School of Creative Science and Engineering, Waseda University and appointed
as a Next Generation Core Researcher in Key
Researchers Development Program by Boardof directors, Waseda University (since 2016)as well. His research fields include humanassistive and augmentation robotics, skinadhesive bio-sensors, supernumerary robotic
limb “Third Arm”, echography-based prenatal checkup assistive robot, perceptionassistive rehabilitation RT and artificial intelligence for construction machinery. He isa member of RSJ, JSME, SICE, EAJ andIEEE.

Rong XU
He earned B.S. degree in School of Information Science and Engineering, and M.S. degree in School of Computer Science and Engineering from Southeast University, China
in 2003 and 2009, respectively. He obtained
his Ph.D degree in Graduate School of Global
Information and Tele-communication Studies(GITS), Waseda University in 2014. Currently, he is a senior researcher in Datasection
Inc., a Japanese listed company. His research
focuses on image processing, video analysis,
pattern recognition, deep learning, surgicalimage-guidance, etc.

Artus KROHN-GRIMBERGHE
He earned his MSc equivalent in informationengineering and management from KIT, Karlsruhe Institute of Technology in 2006. Afterroughly a year in the industry he went backto university and obtained his Ph.D. in Machine Learning from University of Hildesheim
in 2012. Afterwards he joined University ofPaderborn as an assistant professor in Applied Machine Learning in 2012 for six years
till 2018. Currently, he is founder and CEO
of LYTIQ GmbH, a Germany-based startup
focused on consulting in AI (www.lytiq.de).

51

Call for Papers
Special Issue on CG & Image Processing Technologies
for Generation and Post-Processing of Image/Animation
IIEEJ Editorial Committee
In computer graphics (CG), not only realistic images but also animations can be generated in short time
and with low cost, thanks to the improvement of hardware performance in recent years. In addition,
creation/post-processing of animations can easily be achieved using image/video editing software and gameengine under integrated environments for its development. Furthermore, production companies equippedwith studios for motion capture have been increasing, so natural motion of human and animal can easily bereflected to 3D characters.
Yet, time and task for generation and editing of images/animations are still quite large and expensive. To
speed-up and facilitate these processes, various methods have been proposed in worldwide. Especially,learning-based techniques (e.g. AI and deep learning, etc) have been often used just like other research areas.
In the image processing field, image completion and colorization, pencil drawing generation are achieved
using these learning-based techniques. The techniques are also used for accelerating physics-based renderingand simulation, in computer graphics. On the other hand, several methods using crowdsourcing are proposed,
which can easily create the user-desired image/animation. As above, various techniques are being applied tofacilitate/accelerate generation and editing of images and animations, and further development is expected.
Based on this background, we look forward to receiving your papers, system development papers, andmaterial papers in this special issue.
1.

Topics covered include but not limited toComputer graphics, Modeling, Rendering, Simulation, Stereoscopic, Visualization, AR/MR/VR, Game,
Anime, Manga, Entertainment, Motion capture, Wearable sensor, Image processing, Image analysis,
Object detection, Image recognition, Motion analysis, Image encoding, 3D shape reconstruction,
Computer vision, Big data, Database, Crowdsourcing, Image/video retrieval, Usability, Interface,
Interaction, AI, Deep learning, Machine learning, Other related fundamental / application / systemized
technologies

2.

Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary
contributed paper. If the number of accepted papers is less than the minimum number for the special
issue, the acceptance paper will be published as an ordinary contributed paper. We ask for yourunderstanding and cooperation.

3.

Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.8, No.1 (June, 2020)

4.

Submission Deadline:

Friday, November 29, 20195.

Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org

6.

Online Submission URL: http://www.editorialmanager.com/iieej/

52

Guidance for Paper Submission
1. Submission of Papers(1) Preparation before submission
There are three categories of manuscripts as follows:-

Full Paper: It should be a scholarly thesis on a unique study, development or investigation
concerning image electronics engineering. This type of paper is classified as an ordinary
thesis or a system development thesis for a proposed new system. As a general rule, you arerequested to summarize a paper within eight pages.

-

Short Paper: It is not yet a completed full paper, but instead a quick report of the result
obtained at the preliminary stage as well as the knowledge obtained from said result.

-

Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though its
novelty is not high. The length of this paper shall be summarized within four printed pages.



We prohibit duplicate submission of a paper. If a paper with the same content has been
published or submitted to other open publishing forums by the same author or, at least, one
co-author, it will not be accepted as a rule. Open publishing implies internal or externalbooks, magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). The regulation is not applicable to patent gazette,dissertation and specified publications that the editing committee of our society has
approved. But the announcement of such a disclosure must be clarified in the document. A
paper that was once approved as a short paper and being submitted again as the full paper
after completion is not regarded as a duplicate submission.

(2) Submission of a paperTo prepare of a paper for submission, an author can download Guidance for Paper
Submission or Style Format from our homepage.
(Note: Delete all author information at the time of submission. But, deletion of reference
information is author’s discretion. )At first, please register your name on the paper submission page of the following URL:
http://www.editorialmanager.com/iieej/

53

Then, log in again and fill in the necessary information. Use Style Format to upload yourmanuscript.


There are TeX and MS Word versions in the Style Format. An applicant should use PDF
format (converted dvi of Tex) or MS Word format for the manuscript. In principle, charts
shall be inserted into the manuscript.
(A different type data file, such as audio and video, can be uploaded at the same time forreference.)

For those who want to submit a paper by E-mail or do not use the Style format, or want topost-mail a paper by hard copy, please consult the editor at our office.

Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2. Review of Papers and Procedures(1) Review of a paperA manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “approved”, “conditionally approved” or “returned”. The applicant is notifiedof the result of the review by E-mail.
In the review, novelty, reliability and utility are discussed and evaluated.

(2) Procedure after a review

In case a paper is accepted, the author prepares a final manuscript (as mentioned in 3. ).



In the case of a paper that is accepted with comment by the reviewer, the author may revise
the paper in consideration of reviewer’s opinion and proceeds to prepare the final manuscript(as mentioned in 3).

A paper that is not worthy of publishing as it is, but may be published after modification isclassified as conditionally accepted. In case of conditional acceptance, the author shallmodify a paper based on the reviewer’s requirements by a specified date (within 60 days),54

and submits the modified paper for approval.
In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.

(3) Review request for a revised manuscript

If you want to submit your paper after it has been conditionally approved, please submit the
reply letter, the revised clean manuscript and the reviewed manuscript to the submission
page of our site. Please observe the designated date for submission. Outdated manuscripts
may be treated as new applications.



In principle, a revised manuscript shall be reviewed by the same reviewer. It is judged either
approved or returned.

After the judgment, please take the same procedure as (2).

3. Submission of final manuscript for publication(1) Submission of a final manuscript

An author who received our adoption notice shall prepare a final manuscript and submit it
by attaching it to E-mail or by way of CD-ROM. (The contact is the same as mentioned in 1.
Paper submission (2))



The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings, an eps file for author’s photograph (L:W ratio 2:3, more than
300dpi, upper part of the body) for introduction. Please submit these in a compressed format,
such as a zip file.



Before submission, please fully confirm the attachment and check the items being sent.

Particularly, please check that the description method of a chart and writing method of citedreferences meet our society’s designated style.

In the final manuscript, write the name of the author, name of an organization, introduction of
an author, and if necessary, an acknowledgment of appreciation. (In the case of TeX, cancel
macros in the Style file.)

An author whose paper is accepted shall pay a page charge before publishing. It is the author’s

55decision whether to purchase offprints. (ref. Page charge and offprint price information) At least
one member or one student member of our society must be included as an author, before
publishing.

(2) Galley print proof

The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and return it to our office in PDF file form. At
that time, fill in the Offprint Purchase Slip and Copyright Form and return the scanned
data to our office in PDF file form.



In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.



You can download the Offprint Purchase Slip and the Copyright Form from the journal on
our homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication

After final proofreading, a paper is published in the online journal (in the electronic
database). It is also posted on our homepage.

56

IIEEJ Trans. on Image Electronics and Visual Computing©2019 IIEEJ

Vol. 7, No. 1 2019

Editor in Chief: Mei Kodama
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan