IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on CG & Image Processing Technologies,for Automation,
Labor Saving and Empowerment PartⅡ

Vol. 9, No. 1 2021

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief
Mei KODAMA (Hiroshima University)

Vice Editors in Chief
Osamu UCHIDA (Tokai University)
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)

Advisory Board
Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)

Editors
Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (Fuji Xerox Co., Ltd.)
Tomokazu ISHIKAWA（Toyo University）
Masahiro ISHIKAWA （Saitama Medical University）
Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (Fuji Xerox Co., Ltd.)
Shoji KURAKAKE (NTT DOCOMO)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Koji MAKITA (Canon Inc.)
Junichi MATSUNOSHITA (Fuji Xerox Co., Ltd.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Syuhei SATO（University of Toyama）
Masanori SEKINO (Fuji Xerox Co., Ltd.)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer
Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)
Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)

Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)
Munetoshi IWAKIRI (National Defense Academy of Japan)
Yuki IGARASHI (Meiji University)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya) F
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ OfficeOsamu UKIGAYA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information
The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.9 No.1 June 2021
CONTENTS

Special Issue on CG & Image Processing Technologies,for Automation, Labor Saving and Empowerment
1

Upon the Special Issue on CG & Image Processing Technologies,for
Automation, Labor Saving and Empowerment

Masanori SEKINO

Contributed Paper
2

Robust Defect Detection for Logotype Printed on Embossed Surfaces byMultiple Paired Pixel Consistency

12

Weakly Supervised Logo Detection Using a Dual-Attention Dilated Residual
Network

Sheng XIANG, Shun'ichi KANEKO

Short Paper
Rahul KUMAR JAIN, Yutaro IWAMOTO, Taro WATASUE,
Tomohiro NAKAGAWA,Takahiro SATO, Xiang RUAN,
Yen-Wei CHEN

Regular Section
Contributed Paper
20

Interactive Control of Fire Color based on Physicochemical Combustion

Ryosuke UCHIKAWA, Hiroki ISHIZUKA, Sei IKEDA,
Yoshihiro KURODA,Shunsuke YOSHIMOTO, Osamu OSHIRO

Announcements
33

3rd Call for Papers (Call for Late Breaking Papers)

34

Call for Papers : Special Issue on Image-related Technology for Realizing
Immersive Media

35

Call for Papers : Special Issue on CG & Image Processing Technologies
Supporting and Expanding Human Creativities

Guide for Authors
36

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan （IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Upon the Special Issue on CG & Image Processing Technologies
for Automation, Labor Saving and Empowerment
Editor: Masanori SEKINO(FUJIFILM Business Innovation Corp.)
A declining birthrate and an aging population are a common problem in developed
countries. In order to solve the labor shortage, it is necessary not only to expand theworkforce but also to improve labor productivity. Therefore, improvement in productivity is
expected through automation and labor saving technologies using evolving image
recognition technology, increasing information from IoT devices, and cheaper robots.
At the call for submission, we received 12 papers, 8 in Japanese and 4 in English.
Concerning English papers, one paper was adopted in December 2020 issue, and two additional
papers were adopted and included in this issue. By the way, three Japanese papers ware
adopted in January 2021 issue and one additional Japanese paper was adopted (not cited as
special issue paper) in April 2021 issue. The two papers included in this issue were both related
to logo detection though the discussion points are rather different.
In our journal, we are soliciting “system development papers” and “practical papers”,
which are the categories of the papers that make it easy to present the outcome of research
focusing on the practical and applied aspects of technology. As the technology recruited in thisspecial issue will well match with these paper categories, we hope your further submission of
the papers related to this special issue especially in these paper categories.
Finally, I would like to appreciate the reviewers and editors for careful review and all theefforts to improve the quality of papers. Also, I would to thank to the editorial committee
members of IIEEJ and the staff at IIEEJ office for various kinds of support.

1

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）
IIEEJ

Paper

-- Special Issue on CG & Image Processing Technologies for Automation,
Labor Saving and Empowerment --Robust Defect Detection for Logotype Printed on Embossed Surfacesby Multiple Paired Pixel Consistency
Sheng XIANG† (Student Member) , Shun’ichi KANEKO†† Graduate school of information science and technology, Hokkaido University<Summary> Embossed surfaces are widely used in many electronic products to enhance the appearance
and user experience. However, there are inevitably some defects in the production process. They are diﬃcult
to detect by traditional detection algorithms because of the irregular textures on the surface, which are easily
aﬀected by illumination conditions. To achieve robust defect detection for logotype printed on the embossed
surface, we proposed a novel defect detection method called multiple paired pixel consistency (MPPC).
Firstly, we propose a consistency measure for high consistency pixel pairs to realize a robust defect-freemodel. Secondly, based on this model, we design a measure for judging defective pixel or defect-free pixel.
Experimental results with some real-world defective images demonstrate that our approach can achieve
state-of-the-art accuracy in real industrial applications.
Keywords: defect detection, embossed surface, multiple paired pixel consistency, orientation code1.

tural approaches, which are usually consider texture to
be characterized by texture primitives, and the spatial
arrangement of these primitives7), are eﬀective for regular textures. However, utilizing structural methods todetect defects has certain drawbacks, i.e., they are notapplicable to structures with a low degree of regularity or small defects. Filter-based approaches, which canbe divided into spatial-domain, frequency-domain, andjoint-domain techniques, apply ﬁlter banks to an image
for computing the energy of the ﬁlter responses, which
include the Fourier transform8), Gabor transform9), andorthogonal wavelet transform10). They are eﬀective andsimple to implement, but are ineﬀective with images witha complex texture. Model-based approaches, which detect defect by designing or learning a separation model,take into account both spatial texture and statistic information. Such approaches include the Gaussian-Markov
random ﬁeld model11)and the Gaussian mixture model12)
as representatives. Their results have been satisfactory;
however, the computational complexity is high, and they
are not good at detecting small defects. And recently,
deep learning based method have attracted signiﬁcant attentions and obtained highly satisfactory results13). However, it takes a lot of time for training. In addition, it
is diﬃcult and impractical to collect and label plenty ofsamples.
We focus on printed logotypes on a surface embossed

Introduction

The appearance of product is very important for customers to make a good expression. However, in the production and assembly process, various defects are sometimes generated on the surface of the product. Therefore,
defect detection plays an important role in the quality
control (QC) of the manufacturing industry. At present,
defect detection is often performed by human inspectors.
Manual inspection is reliable for a small number of products. However, today, a large number of products, suchas mobile phones and personal computers, are being massively produced and delivered to the market. It is impossible for human beings to inspect each product one byone, and it requires considerable time to perform suchhuge amount of work. In addition, it is very expensive
to hire workers for inspecting a huge number of products. Therefore, human inspection is being replaced byautomatic visual inspection systems1).
There are many defect detection methods have been
proposed in the past several years. From the viewpointof the algorithm, existing visual inspection methods can
be classiﬁed to four categories2). Statistical approaches,
which based the spatial information to evaluate textures,are general applied to irregularly texture case. They including histogram properties3), co-occurrence matrices4), autocorrelation5), and local binary patterns6). Struc-2

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

with 3D micro-textures distributed randomly. This typeof surface is formed by embossing processes that create
mixed miniature convex and concave patterns on the surfaces. And these surfaces have been widely used in many
products, because they have an attractive appearance,
good feel, and excellent slip resistance. Furthermore, thechanges in illumination on such surfaces have signiﬁcant
inﬂuence on the appearance of the surface, which makes
it diﬃcult to perform defect detection. To overcome thisproblem, we use the orientation code (OC) [14] as the basis for the algorithm proposed in this study. OC was previously proposed to be used for robust image matching,and its robustness to illumination ﬂuctuations was veriﬁed in diﬀerent scenes. Furthermore, based on this modeling, we propose an algorithm for defect detection mainlyin the logotype regions, which any customer maybe sensitive to check the quality of their printing.
The proposed model called multiple paired pixel consistency (MPPC) for defect detection, was brieﬂy introduced in15). In this paper, we introduce MPPC with somenew contents and give more detailed explanations for ourwork with new experiment results. The contribution ofthis paper are as follows: (1) this work propose the MPPC
model of defect-free to realize the robustness of illumination ﬂuctuations for detecting the defect in logotypes
printed on embossed surfaces; (2) a new deﬁnition of OC
diﬀerence is proposed, which is better suited for the detailed statistical design; (3) the parameter setting in practical application is discussed, which can make parametersetting easier for readers to achieve good detection results.
The rest of this paper is organized as follows. Section 2.
introduces the working mechanism of the MPPC defectfree model. Section 3. presents defect detection using
the MPPC model, and to show the eﬀectiveness of the
proposed method some experimental results are given in
Section 4.. Finally, some concluding remarks are given in
Section 5..

2.

2.1 Orientation codes
The orientation codes were proposed for applying a ﬁltering to extract the robust features based on only theorientation information involved in gradient vectors fromany images. This approach achieves strong robustness toan image representation and can resist the ﬂuctuations in
the illumination in the real world because it utilizes only
a gradient orientation as a discriminative feature, which
represents the orientation angle rather than the strengthof the gradient.
Let I (i, j) be the brightness of pixel (i, j). Its partial derivatives in the horizontal and vertical directions
are written as ∇Ix = ∂I/∂x and ∇Iy = ∂I/∂y, respectively. The orientation angle θ can be computed usingθ = tan−1 (∇Iy /∇Ix ) of which the actual orientation isdetermined after checking the signs of the derivatives,
thus making the range of θ be [0, 2π). The OC is obtained by quantifying the orientation angle θ into N levels with a constant width ∆θ (= 2π/N ). The OC can be
expressed as follows:[] θi,j|▽Ix | + |▽Iy | ≥ Γ∆θCi,j =(1)Notherwise
where Γ is a threshold level for ignoring pixels with
low-contrast neighborhoods. The pixels with neighborhoods of suﬃcient contrast are assigned an OC of{0, 1, . . . , N − 1}, whereas the ignored pixels are assigned
the code N . Figure 1 shows the code system used in thispaper, where Γ = 10 and N = 16.
Here, a new deﬁnition of OC diﬀerence, which is better suited for the detailed statistical design, is proposed.
Compared with the previous deﬁnition14), the deﬁnition
used here has both positive and negative diﬀerences. It
can provide a more complete and precise distribution of

MPPC Model for Defect-Free Image

In this section, we ﬁrst introduce the original version of
the orientation code, and then, propose the signed diﬀerence between any two codes in preparation for developinga more precise statistical model to represent their diﬀerences. In addition, we provide a precise spatial diﬀerentiation for calculating the codes with a higher resolution.
Finally, we introduce how to construct the MPPC modelfrom defect-free logos.

Fig. 1 Sixteen-OC

3澥

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

the diﬀerences in OC. In this way, we will obtain a symmetric distribution of the OC diﬀerence, which facilitates
statistical handling. This is expressed as follows: min {|a − b| , N − |a − b|}∆ (a, b) =−min {|a − b| , N − |a − b|}澧澦澥(a − b) < − N2 ,0 ≤ (a − b) < N2澧otheriwse澦(2)

where a and b represent the OC to be compared or subtracted, for澥instance, from the target and reference images, respectively,
and N shows the invalid-pixel code.
澧
Since we are dealing with objects having very ﬁne tex澦tures, and furthermore,
we aim to detect extremely small
defects during the real factory production in this study,
the manufacturing processes continue to advance, and thedefects occurring in actual production lines continue todecrease. Yet, the existing Sobel operator cannot obtainenough texture information for ﬁne textures. Therefore,
to realize a better spatial sensitivity, rather than greater濄濆stability, we proposed a new gradient濅 or diﬀerentiation
operator for obtaining a higher resolution or deﬁnition.
In the next equation 3, we deﬁne two convolution matrices for the gradient operator in a 2 × 2 region.
[
]
[
]
−1 1
2
1
濆
Gx =
, Gy =
(3)
−2 2
−2 −1

濄

(a)
濅

濆

(b)

(c)

A pixel of interest in a convolution matrix is deﬁned at
the lower-left position in this 2 × 2 region and may have aweight of -2; the neighboring pixels along the x and y axes
then have a weight of +2 whereas the neighboring pairsof pixels have a weight of -1 or +1, allowing a smoothingeﬀect for de-noising.
Figure 2 shows three typical examples used to evaluate the sensitivity by comparing the proposed operator
with a Sobel operator using a standard Lena image. On
comparing the Figs. 2 (c) and (d), we can see that applying proposed operator results in a thinner edge without
increasing noise levels. Then, by observing the partial
enlargement image, the proposed operator can extracta more complete texture. For instance, when the Sobel
operator is applied in the upper part of the hat, the extracted texture is partially missing. On the other hand,the proposed operator can extract a more complete texture. This shows that the proposed operator is more sensitive to edges.

(d)Fig. 2 Examples of applying the proposed and Sobel operators; (a) Lena, (b) magniﬁed parts, (c) Sobel
operator, and (d) proposed operator

consider the relationship between any pair of two pixels
on the logotype. Here, we aim to investigate what typeof relation any pair of pixels in diﬀerent locations has in
the OC diﬀerence. In one pair, both pixels are on the
logotype, whereas in the other pair one pixel is on the
logotype and the other is on the background. Figure 3
shows two typical examples of OC diﬀerence histograms
made from a dataset of 160 logotype images. As an example of a typical logotype defect, such as a lack of printing,
we can see that one of the logo-background pairs, shownin Fig. 3(b), has a wide variance compared with the logologo pairs shown in Fig. 3(a), where the proﬁle has a clear
peak as another prominent feature in comparison with thelatter; we can also see a symmetric property around themean or center value.
From this observation and the rareness of the defects
occurring in actual factories, we assume that they can ba-

2.2 MPPC defect-free model
2.2.1 Preparation
As the ﬁrst step toward modeling the statistical behavior of pixels on a logotype on an embossed surface, we

4IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）
6&9:&78; +0

ﬁned between each target pixel P on a logotype and theset of supporting pixels, also on the same logotype, whichshould be selected to achieve a higher consistency or correlation with the target pixel. In other words, similar
trends of change as the target pixel are shown, for which
we can make a statistical model by ﬁtting a single Gaussian distribution to the OC diﬀerence histogram of thesepairs with high consistency.
2.2.2 Consistency based supporting pixel selection
We now consider how to select the supporting pixels
from all candidate pixels for a target pixel. For formalization, in this paper, we use capital letters in bold face,such as Q, to represent any set, capital letters in normal
font to represent any pixel, and lowercase letters to show
any OCs of the pixels.
For an arbitrary logo-logo pixel pair (P, Q), we have
two sets of OC sequences at the same positions in all K
training images as follows:

0#. <&7=!>;() (* ($ (+ (, (- (. (/ 0 / . - , + $ * )1234!55&6&78&(a)
6&9:&78; +0

0#. <&7=!>;() (* ($ (+ (, (- (. (/ 0 / . - , + $ * )1234!55&6&78&(b)Fig. 3 Two representative types of OC diﬀerence in apair of pixels; (a) logo-logo pair and (b) logobackground pairsically be ﬁtted by a single Gaussian distribution. Their
kurtoses are calculated to ﬁnd their potential distributions. We found that the kurtosis of the histogram in
Fig. 3(a) was 0.01, which is close to 0, making it reasonable to ﬁt or model using a single Gaussian distribution.
The histogram can be modeled by the single Gaussian distribution with a mean of -0.56 and a standard deviationof 2.3. By contrast, for a histogram of logo-backgroundpairs, the calculated kurtosis was -1.2, which shows thatit can be modeled using a uniform distribution. Fig. 3(b)can be ﬁtted with a uniform distribution, and their mean
and variance are -1 and 17.6, respectively. Based on theseobservations, we ascertain the following signiﬁcant diﬀerences in the behaviors of these pairs: logo-logo pixel pairs
may generally have an extremely high correlation owing
to the logotype printing, whereas logo-background pairs
have no clear correlative relation. We expect the latter
type of pair to possibly be used to represent typical defects, such as a lack of printing.
Figure 4 (a) shows a schematic structure of the proposed MPPC model, which can represent one statisticalcharacteristic in the OC diﬀerence between two elemental
pixels among the pairs. The main idea of this statisticalmodeling of images is called ‘CP3’, which was previously
proposed by Liang16)for robust background subtraction.
We propose an extension of this scheme by introducing
the cohesive relationship of OCs in logo-logo pairs de-

P = {p1 , p2 , · · · , pK }(4)

Q = {q1 , q2 , · · · , qK } ,

(5)

and

where K is the total number of training sample images,
as shown in Fig. 4(a). And the expected values and variances over P and Q are deﬁned as follows:1 ∑1 ∑pk , q̄ =
qk
K
K

(6)

1 ∑
1 ∑
2
2
2
(pk − p̄) , σQ=
(qk − q̄)
K
K

(7)

p̄ =

2
σP
=

K

K

k=1

k=1

K

K

k=1

k=1

The covariance between P and Q is deﬁned as follows:
1 ∑(pk − p̄) (qk − q̄)K
KCP,Q =(8)k=1

If CP,Q > 0, a consistency or co-occurrence probability
occurs, and to measure the consistency quantitatively, we
use the following Pearson’s product-moment correlationcoeﬃcient:γP,Q =CP,Q
σP · σQ(9)where σP and σQ are the standard deviations of P and
Q, respectively.
For all pixels P (u, v) at the position (u, v) over thefull regions covering the logotype, we may have (M − 1)
candidate pixels in the same logotype, where M properly

5IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

MPPC model

Frequency 50 0.3 Density

Frequency 50 0.3 Density

Defect free: DF

Pair #1Q2 … QN
݇ൌͳ

݇ൌʹ

‫گ‬

Q2
-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference

…

Q1

DF

Q1
QN

-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference

Defective: D
P

…

݇ൌ‫ܭ‬

QN

Pair #2
D

Pair #2

Q1
Q2

PFrequency 50 0.2 DensityP-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference
Frequency 50 0.2 Density

Logotype ‘H’-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference

P

D

DF

Pair #1

Frequency 50 0.2 Density

Q1

Frequency 50 0.2 Density

Pair #ND

DF

Pair #N

Q2

QN-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference-8-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8OC difference

(a)

(b)

Fig. 4 Scheme of MPPC; (a) MPPC modeling and (b) defect detection based on MPPC model

Because the MPPC model can represent the statisticalbehavior of the relation of an individual target pixel tothe supporting pixels around it, we utilize it to statistically test whether a target pixel is recognized as a reasonable sample from the distribution registered in the LUT.
The scheme for the defect detection algorithm applied isshown in Fig. 4(b). This include two main procedures:(1) evaluate the normal or abnormal state of each pixel
pair (P, Qi ); (2) estimate target pixel P is a defect-freeor defective pixel.

deﬁnes the total size of the logotype in pixels. From thesecandidates, we can select N (< M ) supporting pixels indescending order of the value of γP,Q . The set of N
supporting pixels is as follows:{}
Q = Qi (ui , vi ) |γP,Qi ≥ γP,Qi+1(10)
i=1,2,··· ,N

And then record the position ui , vi of supporting pixels.
2.2.3 Building defect-free model
In this subsection, we discuss how to build defect-freemodel based on single Gaussian model. We assume thateach supporting pixel Qi maintains a bivariate OC diﬀerence with the target pixel P as follows:∆ (p, qi ) ∼ N (µi , σi )3.1

From these considerations and the investigation described in Section 2., the two features of our MPPC
model, spatial sparseness and high consistency in a correlative relation, can be utilized to handle such defects. The
former may prevent any defects from occupying P andsome supporting pixels simultaneously, and through theuse of the latter feature we then expect to have an evaluation scheme to recognize whether P is occupied withany defects.
The next task must be to design a measure for judging defective pixels or defect-free pixels using the MPPC
model. We ﬁrst test each OC diﬀerence between the target pixel P and a supporting pixel Q in the MPPC modelor the LUT by applying Q for P in the target image. We
deﬁne the following measure,(11)

where N (µi , σi ) is the Gaussian distribution with a mean
µi and variance σi2 , which are calculated from the corresponding pixel sets P and Q.
After modeling for one target pixel P , the above setof N pairs of four parameters for the position ui , vi andthe two statistical parameters µi and σi are recorded ina row of the look-up table (LUT). Through repetitivemodeling, the LUT is ﬁlled in to include the total set of
MPPC models for all pixels in an elemental logotype.

3.

Evaluation of the state of pixel pair

Defect Detection by MPPC

We now discuss how to utilize the proposed MPPC
model of the relationship between pixel pairs in a defectfree logotype for detecting many types of logotype defects.

6

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 6 Representative defect types; From left to right:
micro defect, spot, low contrast defect, and missing print

are generally continuous, blocky, and close to circular inshape; (c) low contrast defects are close to logotypes in intensity and have no obvious boundaries; (d) missing print
usually occurs at the boundary of the logotype, and theboundary is fuzzy. Through we set the two thresholds ofthe detection stage as C = 2.0 and T = 0.5, respectively.
Basically, because the proposed method does not consider the translation or rotation changes of the target
image, any original data image should be aligned at thesame position and should not have any rotation. In areal factory, it is not very diﬃcult to achieve this requirement due to quality control. In this study, we achieved
it through three inspection procedures. The ﬁrst is thatthe operator checks all the data. Use a dedicated ﬁxing tool to correct it so that it has a constant positionand orientation relative to the camera. Next, we checked
their rotation constancy. Finally, the position correlationof all elemental logotype images corresponding to eachcharacter is corrected by aligning them with the normalized cross correlation.
To evaluate the methods in pixel level, we utilize threecommon analysis measurements17): P recision can beseen as a measure of exactness, Recall can be seen asa measure of completeness. and F − measure as a harmonic average of the P recision and Recall.
For image-based performance evaluation, the detection
rate used was as follows:

Fig. 5 Some examples of training data
1 ∥∆ (p, q ) − µ ∥ ≥ C · σ
i
i
i
βi =
,
0 otherwise(12)

to identify the normal (βi = 0) or abnormal (βi = 1)
states at the corresponding position deﬁned by the elemental MPPC model, where the constant C is a parameter that can be set from 1 to 3 to deﬁne an area for an
acceptance probability of 68% through 99.7%.
3.2 Judgement of pixel state
Here, we introduce how to judge the target pixel P is
defective or defect free pixel. We use the total sum
1 ∑
βi
N i=1
N

ξ=

(13)

to construct a decision rule for the occupation of P by any
defect: ξ ≥ T , where T = (⌊N/2⌋ + 1) /N is a threshold isdetermined from the general majority rule, ⌊·⌋ shows theﬂoor function, and N is the total number of supporting
pixels.

4.

Experiments

4.1 Dataset and evaluation indicators
Our method is tested on a dataset collected from factories. This dataset consists of four sets of real images
including four characters ‘A’, ‘E’, ‘I’, and ‘H’, each ofwhich includes 100 defect-free images under three diﬀerent illumination conditions: dark, normal, and bright.
Figure 5 shows some examples of the training data.
We utilize these defect-free images to train the MPPC
model. All the images have a size of approximately
100 × 100 pixels and a resolution of 15 pixels/mm. The
physical size of each character is approximately 7mm.
Usually, a defect is less than 10 pixels in length (or lessthan 1mm) and is usually a hard target for human inspectors to perform visual inspection with their nakedeyes. Figure 6 shows some representative examples: (a)micro defects are very small in size and can easily be confused with the texture of logotypes themselves; (b) spots

Detection Rate =

NT
NT D

(14)

where NT and NT D are the numbers of defective logotypes or images correctly detected and the total numberof defective samples, respectively.
4.2

Discussion on parameter setting

In this section, we discuss the inﬂuence of some parameters on the defect detection results. They include theparameter C in Eq. (12) and N which is the number of
supporting pixels.
The parameter C in Eq. (12) has a direct and important inﬂuence on the detection results. As the value of C

7

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

(a)

(b)

(c)

(d)

caused insuﬃcient training models. When the number isincreased, the noise level in the detection result is obviously decreased; however, the extracted part of the defectis not complete. This may be due to the over-ﬁtting of the
training model. Figure 8 shows the quantitative analysisof character ‘H’. From Fig. 8, we can see that the parameter N mainly aﬀects Recall. As the number of supporting
pixels increases, there is a slight decrease in Recall, anda slight increase in Precision. When N = 2, the Precision is approximately 0.5. According to the deﬁnition ofPrecision, this result means that the number of false positive pixels (FP) and true positive pixels (TP) are similar,
which indicates that a very small number of supporting
pixel cannot enable robust detection. When the value of
N is increased up to 10, Recall can maintain a relatively
high level, while Precision gradually increases. When N
is increased beyond the value of 10, as N increases, there
is a signiﬁcant drop in Recall, while Precision is almost
unchanged. Therefore, we think that 10 is the optimalvalue of N in this study.

(e)Fig. 7 Some examples of defect detection result; (a)Ground truth, (b)-(e) detection result under different number of supporting pixels 4, 8,12 and16)*+,--8>8./*+01023!%)*+,--./*+01023

(&

(#

!$"
!$
!#"
!#
!""
!"
&

'

#

%

(

('

(%

&

4567*/8298815::2/;03<8:0=*-8
Fig. 8 Quantitative analysis of defect detection

4.3

decreases, the integrity of the detection increases; however, the noise will also increase, which reduces precision.
In defect detection, the integrity of the detection results
is very important, even at the expense of a certain precision. Therefore, through experiments, we set C as 2, andto improve the precision, we utilized a closing morphological operator of size of 5 × 5.
The number of supporting pixels N is important for establishing a reliable model. A larger number of supporting pixels can provide more information, which makes
subsequent defect extraction more eﬃcient. However, using too many supporting pixels cause two problems. One
is that it can result in over-ﬁtting of the extracted information, which results in more noise in the experimental
results, and the other is that it will require more computational time.
A basic experiment is designed to examine the defectdetection performance using diﬀerent numbers N, and toﬁnd the optimal value of them. Here we set the values in
the range of [2, 4, · · · , 20].
Figure 7 shows some the results.From Fig. 7 , it canbe seen that as the number increases, the defect extraction integrity decreases to some extent, along with a decline of noise level, and furthermore, when the number
is very small, the extraction of the defect performance israther high, while there is considerable noise such as a
lot of false positive pixels have appeared. This may be

Robust to ﬂuctuations in the illumination

Precise control of the actual factory illumination conditions is not easy. Furthermore, it is common in factories
to have plural sets or stages of visual inspection, which
may undergo changes in the conditions over time. Therefore, the proposed algorithm should be robust to ﬂuctuations in the illumination, for which we have introduced
OCs to build the MPPC model. Here, we utilize the images of the same objects under ﬁve diﬀerent illumination
conditions for the experiments, as shown in Fig. 9, whichwere collected from real factories under the considerationof real illumination conditions. There are 60 real defectimages.
Figure 9 shows some of the detection results. Despite asevere ﬂuctuation in illumination, we found that the proposed method can detect real defects similarly in size andshape from this ﬁgure, which shows the strong robustness of the OCs in the MPPC models. Table 1 shows
the pixel-level performance evaluation. From Table 1, we
can see that under diﬀerent illumination conditions, theperformance of the defect detection is extremely high andachieves similar results. And even under three diﬀerent
lighting conditions, the detection rate reaches 100%. This
experimental result shows that our proposed algorithm isvery robust to illumination ﬂuctuations. In addition, bycomparing the results based on the Sobel operator andthe proposed operator, it can be seen that the proposed

8

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 9 Some examples of defect detection under diﬀerent illumination conditions; From left to right:
the test images, their ground truth images, theresults based on Sobel operator, and the results
based on the proposed operator; From top tobottom: images from dark to bright illumination
conditions

Fig. 10 Some examples of defect detection results fordiﬀerent characters; From the ﬁrst row: test images, their ground truth images, the detected
results, and the ﬁltered results

Table 1 Pixel-level performance of defect detection
under diﬀerent illumination conditions
Illumination
Operator Recall Precision F-measurecondition
Sobel
0.73
0.97
0.83
Dark
Proposed 0.86
0.97
0.91
Sobel
0.72
0.97
0.83
NormalProposed 0.88
0.98
0.93
Sobel
0.0.720.97
0.83
BrightProposed 0.88
0.97
0.92

Table 2 Pixel-level based performance of defect detection for diﬀerent characters
Character Filter Recall Precision F-measure
OFF
0.65
0.62
0.64
A
ON
0.88
0.96
0.92
OFF
0.66
0.62
0.64
E
ON
0.87
0.89
0.88
OFF
0.58
0.6
0.59
I
ON
0.73
0.95
0.83
OFF
0.62
0.65
0.63
H
ON
0.83
0.97
0.90

operator is capable of detecting more complete defects
and is more sensitive to defects.
4.4

excluded the background area (unprinted area). However, in Fig. 10, the ﬁrst and third columns contain some
defects in the background area. To solve these problems,
we will extend our algorithm to include defect detectionin the background area in the future.
Table 2 shows a pixel-level based performance evaluation. After ﬁltering, the recall and precision improved
substantially, and the F-measure reached approximately
0.85. This shows that our method has achieved good results. For the image-based evaluation, all images with defects were successfully identiﬁed, demonstrating the stability of our method.

Diﬀerent characters

In this section, we consider the performance of defectdetection for diﬀerent printed characters. We used fourcases of real defects in diﬀerent positions for ‘A’, four for‘E’, three for ‘I’, and four for ‘H’, respectively. For the
logotype ‘I’, we need an additive explanation because thesize is relatively small. On the one hand, we cannot ﬁndan adequate number of stable supporting pixels, which
will impact the eﬀectiveness of our algorithm. On theother hand, the small size may cause the selected supporting pixels to become too clustered, and thus the supporting pixels are easily aﬀected by each other. Therefore,
we extend the area to include a part of the neighboring‘E’, as shown in the third column of Fig. 10. This ﬁgure shows some examples of defect detection results foreach character, We can see that the proposed method canlocate most parts of the defect area. After ﬁltering, theover-detection noise is mostly removed and some holes are
ﬁlled in. In this paper, we focus on the logotype area and

4.5

Comparison experiment

Many researchers carrying out defect detection of surfaces have commonly examined steel18),19), textile20),21),
and wood22). For the visual inspection of printed patterns, examinations have generally been performed using
paper materials and pharmaceutical capsules1). However,to the best of our knowledge, few studies have inves-

9IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

70% of the defects, whereas the proposed method cansuccessfully detect all defects.

5.

In this paper, we proposed a novel defect-free model
named as MPPC, for defect detection against illumination ﬂuctuations. Firstly, we deﬁned a consistency measure for high consistency pixel pair to realize a robust
defect-free model. In this model, OC diﬀerence deﬁnition could be combined for more precise modeling. Then,based on this model, a new defect detection scheme wasproposed for ﬁnding logotype defects. The experimental
results for real defect images demonstrate the eﬀectiveness of proposed method. In this study, we focused on theprinted logotype, however, the extension of algorithm toperform defect detection in both logotype area and background must be planned. Furthermore, we will attempt
to design schemes to identify and classify diﬀerent defecttypes to improve the eﬀectiveness of QC in manufacturing
production lines.

Fig. 11 Some examples of detection results; From top
to bottom: test images, their ground truth images, and the detected results by PHOT and the
proposed method
Table 3 Pixel-level based performance of defect detection for comparison
PHOT
Proposed

Recall0.53
0.83

Precision
0.87
0.97

Conclusions

F-measure
0.66
0.90

References
1) A. Mehle, M. Bukovec, B. Likar, D. Tomaževič: “Print
Registration for Automated Visual Inspection of Transparent
Pharmaceutical Capsules”, Machine Vision and Applications,Vol.27, No.7, pp.1087–1102 (2016).

tigated defect detection for logotypes on embossed surfaces. To verify the eﬀectiveness of the proposed method,
we compared it with a phase only transform (PHOT) for
surface defect detection23). It has high eﬀectiveness in detecting defects on irregularly textured surfaces, and theobjects they deal with are somewhat comparable to thematerials mentioned in this paper. Furthermore, the authors disclosed their source codes and the settings of theparameters. Herein, we utilize 40 real defect images forcomparison experiments. The implementation was conducted using the source code disclosed, and the parameters selected were those recommended by the respective
authors.
Figure 11 shows some representative results. We
can see that around half of the defects were detectedby PHOT, which is roughly consistent with the groundtruth, whereas the proposed method could detect all of
the defects. For high-contrast defects, PHOT could obtain a rather good detection performance. However, forlow-contrast defects, it did not achieve a good performance. Table 3 shows the pixel-level based performance
evaluation. From this, we can see that our method canachieve good performance. And for image-based evaluation, the PHOT can successfully detect approximately

2) X. Xie: “A Review of Recent Advances in Surface Defect Detection Using Texture Analysis Techniques”, ELCVIA: Electronic
Letters on Computer Vision and Image Analysis, Vol.7, No.3,
pp.1–22 (2008).
3) Z. Wakaf, H. A. Jalab: “Defect Detection Based on Extreme
Edge of Defective Region Histogram”, Journal of King Saud
University-Computer and Information Sciences, Vol.30, No.1,
pp.33–40 (2018).
4) A. Latif-Amet, A. Ertüzün, A. Erçil: “An Eﬃcient Method for
Texture Defect Detection: Sub-Band Domain Co-Occurrence
Matrices”, Image and Vision Computing, Vol.18, No.6, pp.543–
553 (2000).
5) D. Zhu, R. Pan, W. Gao, J. Zhang: “Yarn-Dyed Fabric DefectDetection Based on Autocorrelation Function and GLCM”,
Autex Research Journal, Vol.15, No.3, pp.226–232 (2015).
6) S. Fekri-Ershad, F. Tajeripour: “Multi-Resolution and NoiseResistant Surface Defect Detection Approach Using New Version of Local Binary Patterns”, Applied Artiﬁcial Intelligence,
Vol.31, No.5, pp 395–410 (2017).
7) F. M. Vilnrotter, R. Nevatia, K. E. Price: “Structural Analysis
of Natural Textures”, IEEE Trans. on Pattern Analysis andMachine Intelligence, Vol.1, pp.76–89 (1986).
8) K. Sakhare, A. Kulkarni, M. Kumbhakarn, N. Kare: “Spectral
and Spatial Domain Approach for Fabric Defect Detection andClassiﬁcation”, Proc. of IEEE 2015 International Conferenceon Industrial Instrumentation and Control (ICIC), pp.640–644(2015)
9) J. Jing, S. Chen, P. Li: “Automatic Defect Detection of Patterned Fabric via Combining the Optimal Gabor Filter andGolden Image Subtraction”, Journal of Fiber Bioengineering
and Informatics, Vol.8, No.2, pp.229–239 (2015).

10

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Sheng XIANG (Student Member)10) Z. Wen, J. Cao, X. Liu, S. Ying: “Fabric Defects Detection
Using Adaptive Wavelets”, International Journal of Clothing
Science and Technology (2014).

He received M.S. degree and Ph.D. degree in system science and informatics from
Hokkaido University, Sapporo, Japan, in 2018and 2021, respectively. His research interests
include machine vision and its industrial applications.

11) Y. Xiaobo: “Fabric Defect Detection of Statistic Aberration
Feature Based on GMRF Model”, Journal of Textile Research,
Vol.4, p.26 (2013).
12) M. Li, S. Cui, Z。Xie: “Application of Gaussian Mixture Modelon Defect Detection of Print Fabric”, Journal of Textile Research, Vol.36, No.8, pp.94–98 (2015).
13) Y. Li, D. Zhang, D. Lee: “Automatic Fabric Defect Detection with a Wide-and-Compact Network”, Neurocomputing,
Vol.329, pp.329–338 (2019).

Shun’ichi KANEKOHe received the B.S. degree in precision engineering and the M.S. degree in information engineering from Hokkaido University,
Japan, in 1978 and 1980, respectively, andthen the Ph.D degree in systems engineeringfrom the University of Tokyo, Japan, in 1990.
He had been a research assistant of the Department of Computer Science since 1980 to
1991, an associate professor of the Department of Electronic Engineering since 1991 to
1995, and an associate professor of the Department of Bio-application and Systems Engineering since 1995 to 1996, in Tokyo University of Agriculture and Technology, Japan.
He had been a professor of the Laboratory of
System Sensing and Control in the Graduate
School of Information Science and Technology in Hokkaido University, and then retired
in March 2020. He received many awards including the Best Paper Award 2014 from the
British Physics Society: IOP. His research interest includes machine vision, image sensing
and understanding, robust image registration.
He is a member of IEICE, JSPE, IEEJ, SICE
and the IEEE computer society.

14) F. Ullah, S. Kaneko: “Using Orientation Codes for RotationInvariant Template Matching”, Pattern Recognition, Vol.37,
No.2, pp.201–209 (2004).
15) S. Xiang, Y. Yan, H. Asano, S. Kaneko: “Robust Printing
Defect Detection on 3D Textured Surfaces by Multiple Paired
Pixel Consistency of Orientation Codes”, Proc. of IEEE 2018
12th France-Japan and 10th Europe-Asia Congress on Mechatronics, pp.373–378 (2018).
16) D. Liang, M. Hashimoto, K. Iwata, X. Zhao: “Co-Occurrence
Probability-Based Pixel Pairs Background Model for Robust
Object Detection in Dynamic Scenes”, Pattern Recognition,
Vol.48, No.4, pp.1374–1390 (2015).
17) H. Y. Ngan, G. K. Pang, N. H. Yung:“Automated Fabric DefectDetection a Review”, Image and Vision Computing , Vol.29,
No.7, pp.442–458 (2011).
18) M. Selvi, D. Jenefa: “Automated Defect Detection of Steel
Surface Using Neural Network Classiﬁer with Co-occurrence
Features”, International Journal, Vol.4, No.3 (2014).
19) S. Zhou, Y. Chen, D. Zhang, J. Xie, Y. Zhou:“Classiﬁcationof Surface Defects on Steel Sheet Using Convolutional Neural
Networks”, Materials and Technology, Vol.51, No.1, pp.123–
131 (2017).
20) P. Li, J. Liang, X. Shen, M. Zhao, L. Sui:“Textile Fabric DefectDetection Based on Low-Rank Representation”, Multimedia
Tools and Applications, Vol.78, No.1, pp.99–124 (2019).
21) H. Abdellah, R. Ahmed, O. Slimane: “Defect Detection and
Identiﬁcation in Textile Fabric by SVM Method”, IOSR Journal of Engineering, Vol.4, No.12, pp.69–77 (2014).
22) Y. Zhang, C. Xu, C. Li, H. Yu, J. Cao:“Wood Defect Detection
Method with PCA Feature Fusion and Compressed Sensing”,
Journal of Forestry Research, Vol.26, No.3, pp.745–751 (2015).
23) D. Aiger, T. Hugues: “The Phase Only Transform for Unsupervised Surface Defect Detection”, Proc. of 2010 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, pp.295–302 (2010).

(Received May 27, 2020)(Revised February 25, 2021)

11

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）
IIEEJ

Short Paper-- Special Issue on CG & Image Processing Technologies for Automation,
Labor Saving and Empowerment --Weakly Supervised Logo Detection Using a Dual-Attention Dilated
Residual Network
Rahul Kumar JAIN† (Member), Yutaro IWAMOTO† , Taro WATASUE†† , Tomohiro NAKAGAWA†† ,Takahiro SATO†† , Xiang RUAN†† , Yen-Wei CHEN†† College of Information Science and Engineering, Ritsumeikan University , †† tiwaki Co. Ltd., Kusatsu<Summary> Automatic logo detection is a key function for many applications. Most existing logo
detection methods are based on strong object level bounding box annotations, annotated either manuallyor automatically in synthesized images. However, in real applications, there are thousands of diﬀerent typesof logos with clutter background and varying sizes in images, that makes supervised detection methods
less applicable because it is a labor-intensive task to do bounding box annotations. On the other hand,some weakly supervised learning-based methods have been proposed, their experimental results are notpromising. In this paper, we propose weakly supervised methods for logo detection based on a dual attention
dilated residual network (DRN) containing spatial and channel attention mechanisms using image-level labels
instead of bounding box annotation for training data. The incorporated spatial attention module computes
attention weights which are useful in predicting the spatial location in an image. Channel wise feature maps
are used to emphasize the dependency of the channel in the global feature map and help to classify thelogos into diﬀerent categories. The use of attention-based mechanisms with DRN improves classiﬁcation
accuracy (around 4%) and considerably increases localization accuracy by more than 4% over a regulardilated residual network architecture.
Keywords: deep learning, weakly supervised localization, channel attention, spatial attention, dilated residual network

1.

only time-consuming and labor-intensive but also not affordable in some cases. To address this problem, in someapproaches, the models have been trained with dataset
consisting of synthesized training logo images to avoid
manual annotation work2),3). But the performance was
not suﬃcient when these models were tested on real images. On the other hand, the small number of logo images
and classes in logo datasets aﬀects the scalability of themodel3).
From the application perspective, learning from image
level annotated data is more applicable and scalable thanthe object level annotated data. To address the bounding box annotation problem, in this paper, we propose
a weakly supervised method using attention mechanisms
based deep learning framework.

Introduction

Logo detection, which localizes and identiﬁes logos inimages, has become a very important functionality inmany applications such as brand promotion, social media monitoring, auto-driving, intelligent transportation,market research and illegal logo detection, to name someof many.
However, logo detection is a challenging and diﬃculttask in a complex scenario as there are many existing
brand and logo images with diverse context. Logo isa proportionately small entity to detect in a large unconstrained image. Logos may contain unknown fonts(stylizes), diﬀerent colors and logos can appear in varying sizes. Logos may also have inter-class similarity andintra-class diﬀerences, which makes this problem morediﬃcult1).
Nowadays deep learning is being used in logo recognition because of superior performance in object detection.
Deep learning requires a large training data with object
level annotations to train a detector-model, which is not2.

Related Work

Most of the recently proposed deep learning based logo
recognition methods deﬁne the problem as a sub-task ofgeneral object detection, these methods use the datasets
created manually or synthetically with object level anno-

12

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

many existing algorithms5),10),11),13)–15), our weakly supervised logo detection algorithm is based on a dualattention dilated residual network (DADRN), which was
proposed in our previous work12). In our previous study,
we have seen that DADRN has proved to be a more efﬁcient feature extractor for weakly supervised localization than other networks. This network produces a large
spatial resolution feature map which is more suitable formodeling small logo objects and helps to identify them.
The proposed spatial and channel-wise attention mechanisms to generate and reﬁne the robust ﬁnal output feature map to activate the most probable candidate logo
regions.
This approach can be summarized as: (I) We propose
a weakly supervised logo detection algorithm which is capable to learn logo recognition with multiple logo classes
without bounding box annotated training data. To ourbest knowledge this is the ﬁrst weakly supervised training scheme for logo detection. We neither used boundingbox annotations nor synthesized data for training, hence
it is highly scalable to large scale real-world applications.
(II) Logo image classiﬁcation and logo detection are improved by implementing dual-attention based mechanismwith the DRN Network.

tations for training. István et al1) trained a class-agnostic
detector4) using logo and non-logo/background classes to
identify logo object proposals. To retrieve logo images,
they trained a separate network5) by triplet loss using
proxies6). Su et al.2) described the eﬀect of the synthesized training data to deal with training data issue.
In Ref.3), Su et al. proposed to use GANs methods forsynthesizing coherent training logo images. Model selflearning principle using webly collected data has been
implemented by Su et al7). They automatically collectedmost compatible training images from a webly noisy created data and trained a model iteratively by updating
training dataset. Authors introduced model co-learningwith last method in8). They used two deep learning models4),9) to identify the compatible training images. These
identiﬁed training images have been fed as an input incross-model manner.
Although in some methods, training images are synthetically or automatically created, the results do notshow satisfactory performance on real images. A possible reason is that synthetically generated training images
are not “real” enough to simulate their real-world counterparts, whereas automatically generated training data
is very noisy. These methods are not much scalable. In
this scenario we need more scalable deep learning based
methods to address the issue of training the model with
insuﬃcient data. In real world, there are large number ofexisting logos. To train a detector we must annotate thelogo images due to the data scarcity problem. To avoid
such issues, weakly supervised training schemes can beused to build a more scalable model. In addition, attention mechanisms are very useful to locate and identify
target regions in a weakly supervised manner.
Meanwhile spatial and channel attention mechanisms
have been proposed and applied for various tasks suchas image classiﬁcation, object localization. Wang et al.
10)
proposed a mask module implemented with a trunk
branch that embeds bottom-up top-down feedforward
structure to enhance spatial characteristics. In Ref.5), Hu
et al. proposed to calculate channels statistical weights
to enhance channel-wise feature responses. Woo et al.11)
proposed to use a sequential combination of channel andspatial attention mechanisms. Channel and pixel-wiseweights are determined using max-pooling and averagepooling operations. Chen et al.12) incorporated a spatial
and channel dual-attention mechanism for classiﬁcationand localization of liver lesions on CT images. Among

3.

Method

3.1 Overview
We present dilated residual network (DRN)16) basedarchitecture with attention mechanisms as feature extractor to generate a high resolution feature map. We employ spatial and channel attention mechanisms in diﬀerent combinations to reﬁne the ﬁnal output feature maps.
For weakly supervised localization task, class activation
maps are generated using the Gradient-weighted method(Grad-CAM)17) to visualize the possible logo regions.
3.2 Dilated Residual Network (DRN)
The architecture of DRN16) follows the residual skip
connections learning scheme with dilated convolution inhigher layers. Dilated convolution operations cover alarger area than normal convolution operations and increase the receptive ﬁelds in higher layers. The last blockof DRN network produces a high resolution global feature map (28×28), which provides better deﬁnition ofimage and improves the extracted features. Figure 1
shows dilated convolution operation and DRN network.
As depicted in the ﬁgure, ResNet network uses normal
convolution operation (d = 1) while DRN uses an in-

13

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 1 DRN Network architecturefeature map (γji ) are the normalized values calculatedusing the sigmoid of the matrix multiplication between
the feature map A and its transpose AT . The channel
attention map (Cj ) is the element-wise sum of the original
feature map and multiplication of the attention feature
map with the original feature map.
C∑
Cj = ρ
(γji Ai ) + Aj .
(3)
i=1
)
(
T
− A×A
}, where γ ∈ Rc×c andHere γji = 1/{1 + e
ρ is a parameter which is ﬁrst initialed with 0 and thengradually updated via back-propagation. The output feature maps of spatial (Sj ) and chanel (Cj ) attention modules are concatenated [Sj , Cj ] into X ∈ R(C×2)×H×W andpassed as input to the classiﬁcation branch. Where C × 2
is the number of features of the concatenated attention
maps and (H × W ) is the spatial dimensions. Classiﬁcation branch employs a global average pooling operation upon the concatenated attention feature maps. Thisbranch has a 256-way fully-connected layer with Softmax
to classify the logo images.

creasing factor of dilation (d = 1, 2, 4) in higher layers.
The architecture of DRN and ResNet networks consistsof ﬁve groups or blocks of convolutional layers. In Fig. 1,
Group4 and Group5 represent the fourth and ﬁfth convolutional blocks of the respective networks. Note that the
DRN always shows better performance than ResNet inour previous work12). So, we use DRN as our backbone
network in this paper.
3.3

Spatial attention mechanism

To generate a spatial attention map, the global feature
map A ∈ RC×H×W produced by DRN is fed into a 1×1
convolutional layer. The feature map is now transformed
into two feature spaces B1 and B2 ∈ RC1×(H×W) .
Where C1 shows the number of features and (H × W)
is the size of ﬂatten tensor. The value for each spatial
position vector (Pj ) is calculated using all other spatial
position values in Eq.(1).
N∑Pj =αj,i Di ,(1)
i=1
where αj,i is the sigmoid normalization of BT1 B2 . Di is thetransformed feature map after feeding into the 1×1 convolutional layer. αj,i uses the importance of ith position,
while synthesising jth position. The result of elementwise sum between the original feature map and attention
feature map was the ﬁnal output of the spatial attention
mechanism, i.e.,
Sj = βPj + Aj ,3.5 Gradient-weighted class activation map
The gradient-weighted class activation map (GradCAM) method17) has been used to localize the logo regions. Grad-CAM generates the class activation map of
the discriminative regions using the class speciﬁc prediction scores of channels in a feature map. In the GradCAM acactivation map, the gradient of the prediction
score y c is computed for class c with respect to the feature
map activation An (n ∈ [1, N ], where N is the number ofchannels) of a convolutional layer, the gradient can beshown as ∂y c /∂Anij . ReLU function is applied to generated weighted combination of activation maps to obtainthe features with positive inﬂuence for any class c.

(2)where β is a scale parameter, which is ﬁrst initialized to
0 and assigned weight gradually during training process.
3.4 Channel attention mechanism
Channel based attention weights are calculated withrespect to the global feature map. The values of attention
14

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 2 DRN Network architecture with parallel attention mechanisms

Fig. 3 DRN Network architecture with sequential attention mechanisms (CH=>SP)∑ 1 ∑ ∑ ∂y cn
LcGradCAM = ReLU 
n A
z
∂A
ij
n
i
j

to investigate and explore the signiﬁcance and applicability of the attention modules. Figure 2 depicts the network architecture with attention mechanisms in parallel
fashion, where both attention modules are implemented
simultaneously and the attention weighted outputs areconcatenated. Figure 3 shows the sequential implementation of the channel (CH) and spatial (SP) attention
modules with DRN network (i.e. CH => SP). The output of DRN feature extractor is ﬁrst fed to the channel
attention module and the ﬁned output is sent throughspatial attention module to generate the spatial attention maps.

(4)

where LcGradCAM is the generated activation map, z is thespatial dimension-size (height×width) of feature maps.
We implement channel and spatial attention mechanisms in various combinations. We employ these modules
separately, parallelly and sequentially along the network.
Instead of only parallel combination of both attention
modules as studied in previous work12), in this study weintroduce sequential combinations of attention modules
as an extension of the method. The main objective is

15

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

4.
4.1

Table 1 Correctly classiﬁed logo images and classiﬁcation accuracy

Experiments and Results

Implementation

MethodWe implemented DRN50 framework in PyTorch witha mini-batch size of 16 and performed training for 200
epochs. For parameters optimization, we used Adam19)
optimizer with an initial learning rate 0.0001 and a weightdecay rate 0.1.
4.2

ResNet50 CBAM15)
SEResNet16)
DRN50 (w/o Attention)
DRN50 Channel Attention (CH)DRN50 (Sequential: CH=>SP)
DRN50 (Sequential: SP=>CH)
DRN50 Dual Attention (Parallel)
DRN50 Spatial Attention (SP)

DatasetWe performed experiments on the logo images ofFlickrLogos-32 dataset20). FlickrLogos-32 dataset consists of 32 logo classes. There are 70 logo images in eachclass, where 40 images are for training-validation and 30
images are for test. In these 960 test images, there aretotal 1602 logo objects.
4.3

MethodResNet50 CBAM15)
SEResNet16)
DRN50 (w/o Attention)
DRN50 Spatial Attention (SP)DRN50 Channel Attention (CH)DRN50 (Sequential: CH=>SP)
DRN50 (Sequential: SP=>CH)
DRN50 Dual Attention (Parallel)

We calculated classiﬁcation accuracy for each networkby determining the ratio of the number of total true positive cases to the total input test images in Eq.(5).

4.4

81.56384.79284.89586.87588.12588.64688.85489.271

True
Positives
241
231
334
389
388
346
311
401

Accuracy
15.04
14.41
20.84
24.28
24.21
21.54
19.41
25.03

using the channel attention mechanism. However, othermethods achieve lower accuracy rate than our methods.
Table 2 depicts the mean localization accuracy andnumber of correctly located logo objects. This method
focuses on a weakly-supervised logo localization training
scheme based on attention mechanisms where the modelis trained only with image-level annotations. Previously
proposed methods in the ﬁeld of logo detection use object level bounding box annotations for model training,
so we did not ﬁnd any weakly supervised based method
for comparison.
Logo-object localization results are signiﬁcantly betterin comparison to the conventional DRN network. The
accuracy increases with all combinations of the attention modules except for the sequential implementation of
spatial and channel attention modules. We obtain morethan 3% higher accuracy compared to conventional network when the attention modules are implemented separately. Parallel implementation of both attention moduleachieves the highest accuracy rate. It increases accuracyof the DRN network from 20.84% to 25.03%. While sequential implementation of channel and spatial attention
mechanism enhances the accuracy by more than 1.5%.
The results indicate, the logo regions can be eﬃciently
located using the dual-attention method. The parallel
approach outperforms sequential and conventional approaches because: 1) parallel approach simultaneously
reﬁnes the output feature maps in spatial and channel di-true positive predictions(5)
total predictions

For weakly supervised localization, a predicted boundingbox was considered correct when Intersection over Union(IoU) exceeds 0.5 with the ground truth bounding box.
We determined the localization accuracy using Eq.(6).
Localizationacc =Accuracy

Table 2 Correctly localized logo and localized accuracy

Evolution measures

Classif icationacc =True
Positives
783
814
815
834
846
851
853
857

true positives(6)
total number of ground truths

Comparison with diﬀerent attention modules

Table 1 shows classiﬁcation accuracy for various networks: dilated residual network (DRN50), DRN50 network with parallel dual attention, DRN50 with separate spatial (SP) attention and channel (CH) attention
module, DRN50 with sequential combinations of attention modules (i.e. channel and spatial attention module (CH=>SP), apatial and channel attention module(SP=>CH)), other existing methods SEResNet505) andResNet50 with CBAM11).
The results show that DRN network performance is always better when attention modules are implemented inany combination. DRN50 network achieved around 84%
accuracy rate. The classiﬁcation accuracy of DRN network signiﬁcantly increases by about 4% for parallel andsequential combinations of spatial and channel attention
modules. We obtain more than 4% accuracy with spatial attention. While the accuracy rate improves by 2%

16

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

logo regions. This study can be useful to assign morebiased weights to these speciﬁc channels to improve theperformance.
Figure 4 visualizes the channel-speciﬁc response of top
ﬁve channels (left to right - 211, 231, 229, 213, 217) andthe ﬁnal heatmaps, respectively. These binary images illustrate diﬀerent attention-weights having diﬀerent corresponding response. The ﬁnal heatmaps are generated by
calculating Grad-CAM visualization for the last convolutional outputs. Grad-CAM visualization uses the positive inﬂuence of all channels to locate the possible logo
regions.
Figure 5 depicts the class activation regions of various
methods generated through Grad-CAM17). These methods are DRN50, DRN50 with parallel attention modules,
DRN50 with spatial attention, ResNet50 with CBAM
attention block. Our proposed dual-attention modules
in parallel combination generates qualitative results withmore precise target speciﬁc class activation and enhances
object localization capability of feature extractor.
In practice, all bounding box-based object detectors
face the major problem of lack of suﬃcient object-levelannotated training data. Creating datasets with manually drawn object level bounding boxes is time and laborconsuming. Training data with only image level annotations is more useful in this scenario. Although the performance of the logo localization is not compatible withmethods that use object-level bounding boxes for training, our weakly supervised method is considerable andeasily scalable to a large number of logo classes.

mensions. 2) concatenation of feature maps provides theimportant attention weights to learn classiﬁcation and localization tasks more accurately. Whereas, in sequential
approaches, attention-weighted feature maps are reﬁned
independently one after the other, reducing the eﬀectiveness of important weights during learning.
In the parallel implementation, the ﬁnal output wasa concatenated feature map of those generated by thechannel and spatial attention module. Out of 256 channels, the ﬁrst 128 channels were generated by the spatial attention module and the channel numbers 129 to
256 were generated by the channel attention module. We
computed channel speciﬁc response of channel attention
module for each category. The top 5 channels with higher
weights for each category are given in Table 3. The channel speciﬁc response indicates the importance of channels
with the corresponding category. After analyzing the table, we found that ﬁve channels with corresponding numbers [211, 231, 229, 213, 217] occur more frequently inall categories. Analysis of the occurrence of the channels
suggests these channels play a major role in identifying
Table 3 Channel speciﬁc response for DRN50 Parallel
Dual-Attention network
Adidas211, 231, 238, 251, 229,
Aldi
231, 169, 197, 199, 129,Apple
231, 233, 238, 132, 251,
Becks
229, 217, 211, 190, 227
Bmw
213, 149, 221, 231, 233
Carlsberg
146, 211, 147, 213, 245
Chimay229, 147, 248, 216, 132
Cocacola
231, 129, 217, 190, 211
Corona211, 229, 169, 245, 248
Dhl
213, 129, 197, 251, 221
Erdinger
211, 169, 251, 232, 213
Esso
217, 202, 211, 214, 233
Fedex
217, 213, 233, 241, 229
Ferrari
216, 231, 238, 215, 207
Ford211, 245, 190, 216, 213
Fosters
211, 189, 221, 216, 213
Goolge229, 231, 190, 241, 254
Guiness
211, 219, 146, 248, 245,
Heineken
217, 146, 213, 238, 229
HP231, 229, 240, 190, 197
Milka
211, 241, 233, 151, 231
Nvidia229, 153, 238, 145, 146
Paulaner
169, 213, 221, 135, 236
Pepsi
213, 233, 217, 189, 251
Rittersport 243, 145, 229, 185, 238
Shell231, 251, 197, 229, 129
Singha232, 211, 189, 169, 238
Starbucks
238, 227, 221, 169, 189
Stellaartois 227, 211, 189, 129, 147
Texaco217, 197, 227, 190, 151
Tsingtao
189, 241, 177, 217, 213
Ups
233, 211, 145, 251, 245

5.

Conclusion

In this paper, we proposed a spatial and channel attention mechanism based network for logo classiﬁcation andlocalization. This method is scalable to a large numberof logo images with diﬀerent classes and it needs imagelevel annotated data for training the model. The results
show better classiﬁcation and localization accuracy overconventional architecture. This method is highly scalable
to deal with real-world problems.

References
1) I. Fehérvári, S. Appalaraju: “Scalable Logo Recognition Using
Proxies”, Proc. of IEEE Winter Conference on Applications ofComputer Vision (WACV), pp. 715–725 (2019).
2) H. Su, X. Zhu, S. Gong: “Deep Learning Logo Detection with
Data Expansion by Synthesising Context”, Proc. of IEEE Winter Conference on Applications of Computer Vision (WACV),
pp. 530–539 (2017).
3) H. Su, X. Zhu, S. Gong: “Open Logo Detection Challenge”,

17

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 4 Channel speciﬁc response for diﬀerent logo objects

Fig. 5 Class activation map visualization using Grad-CAM
Pattern Recognition (CVPR), pp. 6450–6458 (2017).

Proc. of British Machine Vision Conference (BMVC) (2018).

11) S. Woo, J. Park, J. Y. Lee, I. S. Kweon: “CBAM: Convolutional Block Attention Module”, Proc. of European Conference
on Computer Vision (ECCV), pp. 3–19 (2018).

4) S. Ren, K. He, R. Girshick, J. Sun: “Fatser R-CNN: Towards
Real-Time Object Detection with Region Propsal Networks”,
IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.
39, pp. 1137–1149 (2017).

12) X. Chen, L. Lin, D. Liang, H. Hu, Q. Zhang, Y. Iwamoto, X.H.
Han, Y.W. Chen, R. Tong, J. Wu: “A Dual-Attention Dilated
Residual Network for Liver Lesion Classiﬁcation and Localization on CT Images”, Proc. of IEEE International Conference
on Image Processing (ICIP), pp. 235–239 (2019).

5) J. Hu, L Shen, G Sun: “Squeeze and Excitation Networks”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 7132–7142 (2017).
6) Y. Movshovitz-Attias, A. Toshev, T. K. Leung, S. Ioﬀe, S.
Singh: “No Fuss Distance Metric Learning Using Proxies”,
Proc. of IEEE International Conference on Computer Vision(ICCV), pp. 360–368 (2017).

13) J. Choe, H. Shim: “Attention-Based Dropout Layer for Weakly
Supervised Object Localization”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.2219–
2228 (2019).

7) H. Su, X. Zhu, S. Gong: “Weblogo-2M: Scalable Logo Detection by Deep Learning from the Web”, Proc. of IEEE International Conference on Computer Vision Workshops (ICCVW),
pp. 270–279 (2017).

14) K. Yang, D. Li, Y. Dou: “Towards Precise End-to-end Weakly
Supervised Object Detection Network”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 8372–8381(2019).

8) H. Su, X. Zhu, S. Gong: “Scalable Logo Detection by Self CoLearning”, Pattern Recognition, Vol.97, 107003 (2020).
9) J. Redmon, A. Farhadi: “Yolo9000: Better, Faster, Stronger”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 6517–6525 (2017).

15) H. Xue, C. Liu, F. Wan, J. Jiao, X. Ji, Q. Ye: “DANet: Divergent Activation for Weakly Supervised Object Localization”,
Proc. of IEEE International Conference on Computer Vision(ICCV), pp. 6589–6598 (2019).

10) F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X.
Wang, X. Tang: “Residual Attention Network for Image Classiﬁcation”, Proc. of IEEE Conference on Computer Vision and16) F. Yu, V. Koltun, T. Funkhouser: “Dilated Residual Networks”, Proc. of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 472–480 (2017).

18

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Takahiro SATO

17) R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh,
D. Batra: “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 618–626(2017).

He received the B.E., M.E, and Ph.D. degrees
from Hiroshima University, Hiroshima, Japan
in 2006, 2008 and 2011, respectively. From2011 to 2016, he was an Expert Engineer withTOME R & C Inc., Kyoto, Japan. He is currently an Expert Engineer with tiwaki Co.,
Ltd., Kusatsu, Shiga, Japan. His current research interests include computer vision, machine learning, and image processing.

18) K. He, X. Zhang, S. Ren, J. Sun: “Deep Residual Learning for
Image Recognition”, Proc. of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 770–778 (2016).
19) Y. Bengio, Y. LeCun: “Adam: A Method for Stochastic Optimization”, Proc. of 3rd International Conference on Learning
Representations (ICLR) (2015)20) S. Romberg, L. G. Pueyo, R. Lienhart, R. Van Zwol: “Scalable Logo Recognition in Real-World Images”, Proc. of the 1st
ACM International Conference on Multimedia Retrieval, pp.
1–8 (2011).

Xiang RUAN
He received the B.E. degree from Shanghai
Jiao Tong University, Shanghai, China, in
1997, and the M.E and Ph.D. degrees from
Osaka City University, Osaka, Japan, in 2001
and 2004, respectively. From 2007 to 2016,he was an Expert Engineer with OMRONCorporation, Kyoto, Japan. He is currently
the Founder and CEO of tiwaki Co., Ltd.,
Kusatsu, Japan. His current research interests include computer vision, machine learning, and image processing.

(Received June 12, 2020)(Revised January 6, 2021)

Rahul Kumar JAIN (Member)He received BCA degree from Dr. H. S. Gour
Central University, Sagar, India, and MCA
degree from RGPV University, Bhopal, India in 2011 and 2015, respectively. He is
now pursuing PhD at Ritsumeikan University,
Kusatsu, Japan.

Yen-Wei CHEN
He received his B.E. degree in 1985 from
Kobe University, Kobe, Japan. He received
his M.E. degree in 1987, and his D.E. degreein 1990, both from Osaka University, Osaka,
Japan. He is currently a professor at the college of Information Science and Engineering,
Ritsumeikan University, Japan. His researchinterests include computer vision, medical image analysis.

Yutaro IWAMOTO
He received the B.E. and M.E., and D.E. degrees from Ritsumeikan University, Kusatsu,
Japan in 2011 and 2013, and 2017, respectively. He is currently an Assistant Professor at Ritsumeikan University, Kusatsu,
Japan. His current research interests include
image restoration, segmentation, classiﬁcation of medical imaging and deep learning.

Taro WATASUEHe received the B.E. and M.E. degrees from
Osaka University, Osaka, Japan, in 1998 and
2002, respectively. From 2004 to 2007, he
was an assistant professor in Kobe University.
From 2007 to 2016, he was an Expert Engineer with TOME R&C Inc., Kyoto, Japan.
He is currently the co-founder of tiwaki Co.,
Ltd., Kusatsu, Japan. His current researchinterests include computer vision, machinelearning, and image processing.

Tomohiro NAKAGAWA
He has experience in R & D of machine learning and computer vision since 1998. He iscurrently the co-founder of tiwaki Co., Ltd.,
Kusatsu, Japan. Since 2019, he has been involved in developing pose estimation technology that runs over 50fps on Jetson nano.

19

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）
IIEEJ

Paper
Interactive Control of Fire Color based on Physicochemical Combustion
Ryosuke UCHIKAWA† (Student Member), Hiroki ISHIZUKA† , Sei IKEDA† , Yoshihiro KURODA†† ,Shunsuke YOSHIMOTO††† , Osamu OSHIRO†† Osaka University , †† University of Tsukuba , ††† The University of Tokyo<Summary> The expression of ﬂames using computer graphics is frequently required as a familiar phenomenon. Various colors, including red and blue, can be seen in real ﬂames. However, the combustion
reaction in a real ﬂame is too complex, involving thousands of chemical reactions to render interactive simulations. In this study, the authors constructed a novel combustion reaction model for expressing realistic
ﬂames with simpliﬁed chemical reactions of complex combustion. To achieve interactive physicochemical
simulations of ﬂames, ﬂuid physics simulations were applied using the position-based ﬂuids method. In the
proposed combustion model, the combustion reaction is represented as ﬁve dominant chemical reactions.
Opposed to the conventional methods, the proposed model can reproduce the blue and red regions of theﬂame. The proposed ﬂame simulation method can be operated at approximately 20 frames per second or
more, making interactive rendering possible.
Keywords: ﬂame, combustion, interactive, position-based ﬂuids (PBF), position-based dynamics (PBD),
computer graphics (CG)1.

tion model and simulated a ﬂame based on the hydrogen
combustion mechanism, which can be represented by ﬁvechemical reaction equations. However, this model does
not consider the generation of the chemical species thatare important in determining the ﬂame’s color or reproduce the ﬂame’s blue color.
In this study, we aim to establish an interactive method
for simulating a realistic ﬂame having various colors,
including blue and red, originating from the luminescence. To achieve this, we propose a novel physicochemical combustion reaction model suitable for reproducing
the ﬂame’s color. We also propose a method for interactive physicochemical simulation of ﬂames. For thissimulation, we combined the proposed combustion reaction model and the position-based ﬂuids (PBF) model 9),
which is a high-speed ﬂuid calculation method.
Our simulation of the combustion reaction can be divided into two stages: decomposition and synthesis. In
the decomposition stage, fuel molecules are decomposedto the atomic level in one chemical reaction. In the synthesis stage, the ﬁnal products of combustion, that is,carbon monoxide, carbon dioxide, and soot, are determined by referring to the concentration and temperatureof each species. Using this method, the calculation time
for obtaining the information necessary for reproducing

Introduction

In most conventional computer graphics (CG) methods
expressing ﬂames, ﬂames are represented only by the colors around red and yellow spectral bands derived from theblack-body radiation of the soot produced by the combustion reaction 1)–5). However, real ﬂames apparently contain colors on diﬀerent bands, such as blue. For example,
the ﬂames from gas cooking burners are turquoise blue,
the ﬂame of burning alcohol is pale blue, and the base of
a lighter ﬂame is blue, whereas its tip is yellow or red.
Blue ﬂames originate as the luminescence of intermediate components produced when the fuel decomposes andluminescence of carbon monoxide combustion. Nguyen
et al. 6) proposed a simulation method that considers theblue region existing at the base of a ﬂame. However, thismethod only allows a single intensity of the ﬂame’s blue
color; thus, it is expressed by only a single color.
The luminescence intensity of the generated chemical
species must be calculated based on the computation of
the combustion reaction to reproduce a more plausible
color of the ﬂame. However, the combustion reaction consists of thousands of chemical sub-reactions and cannot
be accurately reproduced in real-time simulation. Ihm etal. 7) and Kang et al. 8) constructed a combustion reac-

20

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

the ﬂame color, such as the fuel decomposition rate andsoot concentration can be shortened. The PBF modelis used to calculate the air ﬂow. The ﬂuid simulation
was performed with the constraint of constant density oneach particle, rather than calculating the pressure term of
the Navier–Stokes equation. This method increases thesimulation time step width, decreases the computation
time, and enables interactive calculation. Furthermore,
to reproduce the ﬂames produced by the combustion of
gaseous and solid fuels, such as matchsticks, the solid isexpressed as a mass of particles, and the thermal decomposition reaction inside the solid particles is considered.
An interactive ﬂame simulation method is presented
in section 3 and section 4. In particular, a method forachieving a physicochemically accurate reproduction of
ﬂame colors is described. The representation of solid fuelcombustion is also explained. After the implementationof the proposed model is shown in section 4, the examples
of solide fuel combustion are shown in section 5.

2.

Fluidsmovement

Combustion

Diffusion

Rendering

Particle

Particle

Temperature
C

Flame

Move
Simulation area

Chemical species

Particle

Fig. 1 Overview of the proposed simulation method

Studies in which the combustion reaction was modeled
directly, and the ﬂame was simulated accordingly havebeen conducted. Min et al. 13) and Stam et al. 14) simulated ﬂames by modeling the complex combustion reactions using one chemical reaction equation. However,
because this method treats only one chemical reaction,
it cannot eﬀectively include ﬂame characteristics, suchas temperature and soot concentration distributions, andthe resulting CG reproduction cannot be considered realistic. Ihm et al. 7) and Kang et al. 8) modeled combustion based on the simplest hydrogen combustion reaction
mechanism, which consists of ﬁve chain reactions. However, even in this model, factors such as soot concentration and the burning rate of carbon monoxide, which areessential for rendering a ﬂame in CG, were not addressed.

Related WorkThe red and yellow colors of the ﬂame are due to sootblackbody radiation, and the blue of the ﬂame is due tolight emission when the fuel decomposes or when carbon monoxide burns 10). If the combustion reaction simulation can reproduce this phenomenon, it is possible todraw a plausible ﬂame. However, the combustion reaction is a complex reaction in which several thousand elementary reactions occur serially. Many researchers haveattempted to model combustion reactions. One of themost interesting methods is the ﬂame front model, whichhandles the combustion reaction by assuming that the
combustion reaction rate is suﬃciently fast compared tothe diﬀusion of chemical species and heat, so that the
combustion reaction occurs only on a thin surface (ﬂamefront) 11). Nguyen et al. 6) combined a ﬂame front modelwith a level set method to produce a realistic ﬂame without considering the details of the combustion reaction. In
addition, Hong et al. 12) applied explosion theory to themethod of Nguen et al. 6) to allow the representation of
wrinkled ﬂames by using simulations. However, in theseﬂame front models, although the location of the combustion reaction can be speciﬁed, the reaction speed and thetype of reaction cannot be considered, and thus, the colorof the ﬂame cannot be accurately reproduced. A realistic
ﬂame can be interactively generated if a new combustion
reaction model could reproduce the color of the ﬂame athigh speed.

3.

MethodTo render a ﬂame using simulation, it is necessary to
calculate the ﬂow of the atmosphere around the ﬂame andthe combustion reaction inside the ﬂame. We simulate
the ﬂames by discretizing air with particles and tracking their behavior. Our system consists of four parts,
as shown in Fig. 1. The ﬁrst part is a numerical ﬂuid
calculation unit that calculates the gas ﬂow. The second is the diﬀusion calculation unit that calculates thediﬀusion of each species and temperature, The third is acombustion reaction calculation unit that calculates the
combustion reaction in each particle, and the fourth is arendering unit for expressing a ﬂame using the numericalvalues obtained by calculation. In the following sections,
we ﬁrst describe the calculation of ﬂuid motion and diffusion. Then, we discuss the modeling and calculationof the combustion reaction. Finally, the ﬂame rendering
method is explained.

21

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

3.1 Fluid motion computation
In this section, we describe an interactive ﬂuid motion
calculation method. We ﬁrst describe position-based dynamics (PBD), which is a fast and stable physical simulation method, and then PBF, which applies PBD to ﬂuid
motion computation. Finally, we discuss the treatmentof buoyancy in the PBF method.
3.1.1 Position-based dynamics algorithm
PBD is a method of obtaining the position of a calculation point directly using a constraint condition. Let
xi , vi , mi and fi be the the position, velocity, mass andexternal force of the i-th particle. The total number ofparticles is n. x is (x0 , ..., xn−1 ), v is (v0 , ..., vn−1 ), andf is (f0 , ..., fn−1 ).
The process at each step in the basic PBD is asfollows. The predicted velocity v∗ is determined asv∗ = v + f /m∆t, where ∆t is the time step, f /m =(f0 /m0 , ..., fn−1 /mn−1 ), and the predicted position x∗ isdetermined by x∗ = x + v∗ ∆t. Then, to ﬁnd the corrected position x̃ that satisﬁes the constraint condition
C(·), as explained below, ∆x = x̃ − x∗ is obtained bysolvingC(x∗ + ∆x) = C(x∗ ) +∑∇xk C(x∗ ) · ∆xk = 0,

ρi =

∆xi =

∗ 2
m−1
k |∇xk C(x )|and chemical species diﬀusion.
 Nonuniform temperature distribution due to com-bustion reaction and temperature diﬀusion.
In the PBF method, the buoyancy generated between
ﬂuids with diﬀerent densities is expressed by the diﬀerence in the rest density ρ0 and the particle mass mi 15).
Therefore, we update mi and ρ0i for each step:∑mi =Dℓ di,ℓ ,
(7)
ℓ<Θ

∑

(1)

ρ0i = ∑

di,ℓ

p0
,
R0 T0

(8)

ℓ<Θ

where di,ℓ is the molar amount of the chemical species ℓin particle i. Θ is the total number of chemical species
used in the simulation, Dℓ is the mass per mole of thechemical species ℓ, p0 is the reference pressure, R0 is thegas constant, and T0 is the initial temperature.
The buoyancy due to the temperature distribution
should be expressed as the force generated by the density change; however, if a large diﬀerence exists in thedensity ρ0 of the particles, the calculation diverges andthe value becomes extremely large. Therefore, we implemented buoyancy due to temperature, not as densitychange, but as an external force acting on the particles 6).
Let Tiamb be the temperature around particle i. Then,we express the buoyancy Fibuo applied to particle i as

(3)

Finally, the particle velocity from the corrected positionx̃ and initial position x is updated. In general, PBD
methods use Jacobi or Gauss-Seidel iterations to converge
the constraints.
3.1.2 Position-based ﬂuids
In the PBF method, ﬂuid simulation is achieved by imposing density constraints on all particles 15). The density
constraint equation for particle i in the PBF method isexpressed as
ρi− 1 ≤ 0,
ρ0i

Dℓ di,ℓℓ<Θk<n

Ci (x) =(6)

 Nonuniform density distribution due to fuel supply

(2)

.

)
∑ ( λiλj+mj ∇xi Wij .
mi ρ0imj ρ0jj<n3.1.3 Buoyancy
The following two factors are considered as the main
factors of buoyancy associated with gas combustion.

where λ is a variable. From Eqs. (1) and (2), λ can beobtained as follows:C(x∗ )(5)

Wij = W (xi − xj , ξ) is the kernel function, where ξ isthe eﬀective particle radius. In the original PBF 9), density constraint Ci (x) = 0 was used, but a non-negative
clamping density constraint is suﬃcient for representing
ﬂuid motion 15). From Eqs. (1) to (5), the position correction vector ∆xi of particle i can be obtained aswhere ∇xk = (0, ..., ∂/∂xk , ∂/∂yk , ∂/∂zk , ..., 0). To satisfy
the conservation of momentum and angular momentum,∆xk is expressed as

λ = −∑

mj Wij .

j<n

k<n

∗
∆xk = λm−1k ∇xk C(x ),∑Fibuo = ϑ(Ti − Tiamb )g,(9)where ϑ is the buoyancy constant. The ambient temperature Tiamb is given by∑(4)

Tj Wijj̸=iTiamb = ∑where ρ0i is the rest density of the target ﬂuid, and ρi isthe density of particle i, calculated as

j̸=i

22

.
Wij

(10)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

3.1.4 Viscosity
The contribution of viscosity to the shape of the ﬂame
is considered non-negligible. We applied the artiﬁcial
method 16) used in SPH to consider changes in motion
due to viscosity. Although this is not the physical viscosity, it is enough to represent the viscosity eﬀect and iseasier to tune.
vinew = vi + ζ∑(vj − vi )Wij ,each ﬂame. Moreover, we describe the application of the
proposed combustion reaction model to solid fuel combustion.
The reaction in which methane gas burns to form carbon dioxide is represented as
CH4 + 2O2 → CO2 + 2H2 O.

However, this represents only the beginning and the endof the combustion reaction, and does not represent thereaction process. The combustion reaction occurring inan actual ﬂame is expressed as a set of reactions, for example:

(11)

j<n

where vinew is the new velocity of particle i, and ζ is thetunable parameter set as ζ = 0.0002 in our simulations.
3.2 Diﬀusion
This section describes the diﬀusion of the chemical
species and temperature. The diﬀusion equation of a generalized physical quantity ϕ can be expressed as∂ϕ= ∇ · (D∇ϕ),∂t
CH4 + OH
CH3 + HO2
CH3 O + O2
 HCHO + OH



CHO



CO + OH(12)∂ϕ=∇D · ∇ϕ + D∇2 ϕ∂t

1T∇T · ∇ϕ + D0 ∇2 ϕ.
T0
T0

(13)

In our simulation, ∇T ≈ ∇ϕ and 1 ≪ T , we can assume
that D0 /T0 ∇T ≪ D0 T /T0 ∇ϕ and ignore the ﬁrst termof Eq.(13) and Eq.(14) is derived.
∂ϕT= D0 ∇2 ϕ
∂t
T0

(14)

Assuming that the temperature diﬀusion and concentration diﬀusion are caused by the gradient to maintain symmetry, Eq.(15) is obtained by discretizing Eq.(14) using
the kernel function and forward Eulerian form.
ϕnew= ϕi + ∆t
i

∑
j<n

mj D0

Ti + Tj ϕj − ϕi 2
∇ Wij .
2T0
ρj

→
→
→
→
→
→

CH3 + H2 O,
CH3 O + OH,
HCHO + HO2 ,CHO + H2 O,
CO + H,
CO2 + H.

(17)

Formula (17) shows one of the pathways by which one
methane molecule is oxidized to carbon dioxide. The speciﬁc processes that occur in each chemical species thatmake up Fm.(17) are called elementary reactions.
Combustion is a complex chemical phenomenon involving many elementary reactions; approximately 400 elementary reactions are involved even in the case of lowcarbon lower alcohols, such as ethanol 17). The number of
elementary reactions increases exponentially as the carbon number increases. Thus, it is diﬃcult to simulate all
the elementary reactions involved in the combustion ofeach particle. However, it is known that the blue colorof a ﬂame is due to the light emission of the chemical
species generated when the fuel is decomposed and thelight emission when carbon monoxide is burned to form
carbon dioxide. Similarly, the red color of a ﬂame is dueto the black body radiation of the soot resulting from the
combustion reaction. Therefore, the important information required to render a ﬂame is the soot concentration,
decomposition rate of the fuel, and burning rate of carbon
monoxide. To obtain this information without considering the entire reaction, we consider a simpliﬁed model ofthe combustion reaction, as shown in Fig. 2. In the next
subsections, we explain our combustion model, which iscomposed of these chemical reactions.

where D is the diﬀusion coeﬃcient of ϕ, and t is the time.
In our simulation, ϕ is the temperature T or amount ofchemical species d.
For gas concentration diﬀusion and temperature diﬀusion, it is assumed that the diﬀusion coeﬃcient D canbe expressed as D = D0 T /T0 using the diﬀusion coefﬁcient D0 at the reference temperature. Equation (12)then becomes

=D0

(16)

(15)

We use Eq.(15) to handle the diﬀusion phenomena of
the gas temperature and concentration of each chemical
species.

3.3.1

Fuel decomposition

Figure 2(a) shows the decomposition of gaseous fuel.
The decomposition reaction is generated by the collisionof fuel molecules in high heat with other molecules; it isassumed that the fuel molecules are decomposed to the

3.3 Combustion reaction modelingThis section describes the combustion reaction and proposes a novel combustion reaction model for rendering

23

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

CO

C

Fuel Gas

H
(a)

(b)

OH
Soot
(c)

(d)

Fig. 2 Proposed combustion reaction model; (a)Decomposition of gaseous fuel, (b)Combustion mechanism of
the generated carbon atom, (c)Process of the generated hydrogen atoms to become water, and (d)Soot
generation model

atomic level in one collision. Assuming that the fuel isrepresented as Ca Hb Oc , the reaction formula is
Ca Hb Oc + M → aC + bH +c
O2 + M,2

which carbon monoxide and carbon dioxide are generated
from carbon atoms and oxygen molecules as follows(18)

where M denotes a third body. To simplify the decomposition of fuel and the following chemical reactions,
we consider the generated oxygen atoms become oxygen molecules immediately and do not react with othermolecules. We consider that this simpliﬁed model is more
complicated than the conventional models and can reproduce complex phenomena. Assuming that Eq.(18) isa second-order reaction, the reaction rate ufuel is determined asufuel = κfuel [fuel][M],Eaκ = Aexp −
R0 T

(19)

(20)

(22)

uCO =κCO [C][O2 ],(23)

uCO2 =κCO2 [CO][O2 ].

(24)

H + O2 →OH + O.

(25)

H + OH→H2 O.

(26)

Furthermore, it is assumed that the reaction in Eq.(26) is
suﬃciently fast compared to the reaction in Eq.(25) andthat the generated OH is consumed instantaneously. By
applying the steady-state approximation to OH, we candetermine a series of reaction rates uH2 O as

where A is a frequency factor and Ea is the activationenergy, which varies depending on the reaction equationof interest.
3.3.2

CO + O2 →CO2 + O.

3.3.3 Hydrogen combustion
Figure 2(c) shows the process by which the generatedhydrogen atoms become water, which is one of the ﬁnal
products. It is assumed that the mechanism of water formation from hydrogen atoms and oxygen molecules consists of two elementary reactions:

)
,

(21)

We assume that the generated oxygen atoms become oxygen molecules immediately. Both Fms.(21) and (22) represent secondary reactions. The reaction rates uCO anduCO2 were determined bywhere [fuel] is the concentrations of the fuel. κfuel isthe reaction rate constant, obtained by the Arrhenius
Eq.(20).
(C + O2 →CO + O,

Carbon combustion

Figure 2(b) shows the combustion mechanism of thegenerated carbon atom. We model the mechanism by

uH2 O = κH2 O [H][O2 ].

24

(27)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

3.3.4

Soot generation

for the formation of chemical species from the standard
state. Let cp [J/(mol · K)] be the constant pressure speciﬁc heat of the particles. We use Eq.(38) to estimate
cp,ℓ , and Eq.(39) to calculate cp .

Figure 2(d) shows the soot generation model. In our
combustion model, soot is formed by the concentrationof carbon that does not react with oxygen. The formation process of this soot is not described by a reaction
equation, and it is assumed that the soot concentration
is proportional to the carbon concentration.
3.4

cp,ℓ =a + bθ + cθ 2 + dθ 3 + eθ 4 +∑Chemical species concentrationcp =We handle combustion reactions by dividing them into
decomposition and synthesis. Therefore, the chemical
species concentration is updated in two steps. First, therate of the decomposition reaction shown in Eq.(18) is
calculated inside each particle, and the amount of eachchemical species is updated:

f
,
θ2

(38)

cp,ℓ di,ℓ

ℓ<Θ

∑

.

(39)

di,ℓ

ℓ<Θ

a to f is the constant coeﬃcient. θ is T /1000. The temperature change in the particles dT /dt is determined as
dT
1
dQ
= ∑
.
dt
cp
di,ℓ dt(40)ℓ<Θ

d[fuel]=−ufuel ,dt
d[C]=aufuel ,dt
d[H]=bufuel ,dt
d[O2 ] c= ufuel .
dt
2

(28)

3.5 Solid fuel heat decomposition
This section describes the extension of our combustion
reaction model to solid fuel combustion. When solid fuelis burned, its decomposition generates fuel gas, which
in turn reacts with oxygen and the ﬂame appears. We
model this phenomenon and consider the combustion ofsolid fuel. Note that to treat solid fuel consumption, solid
deformation simulation need to be performed. However,this study focuses on ﬂame simulation and does not treat
solid fuel consumption.
Let sFuel be the solid fuel, gFuel be the fuel gas that is
generated, and bFuel be the other products that are generated. The chemical reaction for thermal decompositionis assumed to be

(29)
(30)
(31)

Next, after updating the concentration of each species
using Eq.(28) to Eq.(31), we obtain the rate of the synthetic reaction shown in Eqs.(21), (22) and (25), and theconcentration of the chemical species is next calculated:d[C]=−uCO ,
dt
d[H]=−2uH2 O ,dtd[O2 ]=−0.5 (uCO + uCO2 + uH2 O ) ,dt
d[CO2 ]
=uCO2 ,dt
d[H2 O]=uH2 O .
dt

3.4.1

(32)
(33)
(34)

sFuel → gFuel + bFuel.

(35)

The heat per unit mole generated by the Eq.(41) is∆Hdc [J/mol]. The reaction speed udc of Eq.(41) is determined as:(36)

Heat of total combustion

Finally, we describe the temperature change due to the
combustion reaction. The time change of the heat Q generated by the combustion reaction is

udc = κdc [sFuel].

(42)

In this study, [sFuel] is a constant. Let ς g be the amountof fuel gas generated, T chS be the temperature of the solid
particle.

dQ= ∆f Gfuel ufueldt−∆f GC (aufuel − uCO )dς g=udc ,
dt
chS∆Qdc∆Hdc udc
dT
=
=
,
dt
Cp
dsFuel cp−∆f GH (bufuel − uH2 O )−∆f GCO (uCO − uCO2 )−∆f GCO2 uCO2−∆f GH2 O uH2 O ,

(41)

(43)
(44)

where ∆Qdc is the heat, Cp [J/K] is the heat capacity anddsFuel [mol] is molar amount of sFuel. gFuel is assumed
to be evenly distributed around the solid particles. The
amount of fuel gas dς¯g distributed to one gas particle is(37)

where ∆f G is the standard Gibbs free energy of formationof the chemical species, which denotes the required energy

25

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）dς¯g = dς g /namb . namb is the number of gas particles inthe vicinity of a solid particle. The combustion reaction
model described in the previous section was applied forthe combustion of fuel gas distributed in the gas particles.

hc(δ(430 nm) + δ(510 nm))λhc
+γuCO2 NA δ(460 nm).
λIr =βufuel NA

where δ is the Dirac delta function and NA is the Avogadro number. β and γ are user-deﬁned parameters that
determine the intensity of the light emission due to fueldecomposition and carbon monoxide combustion, respectively. As in the case of black-body radiation, color
matching functions w̄(·) are used to obtain the color
values Wr in the XYZ color system as

3.6 Rendering
3.6.1 Flame color
In this section, we describe the determination of thecolor of the particles from the values obtained by the simulation. The color of the ﬂame is determined using black
body radiation of the soot and the color emitted by thereaction intermediate generated when the fuel molecules
are decomposed.
First, we describe the color of the black body radiation of the soot. The surface area of the soot is assumed
proportional to [C]. Let the surface area of a coﬃn be
Ssoot = φ[C], where φ is the proportional constant. The
spectral radiant ﬂux IT (λ, T ) due to the black body radiation of soot is given by the Planck’s law:
8πhc
IT (λ, T )=αSsoot 5
λ

1
(
),
hc
exp
−1
λKb T

=αSsoot I`T ,Wr =βufuel NA(49)

The particle color value W = (X, Y, Z) is:
W=

(45)

WT + Wr,∥WT + Wr ∥1 + H(50)

where ∥ · ∥1 denotes the L1 norm. Because W is a normalized value, the synthesis of the two lights is given by
WT + Wr /∥WT + Wr ∥1 . In addition, the eﬀect of ambient light on ﬁre color is expressed by adding H to thedenominator. H = ϱε is the intensity of the ambient light.
ϱ is the proportional constant, and ε is the user-deﬁned
parameter that determines the intensity of the ambient
light. Finally, we convert the particle color W into thesRGB color system U = (R, G, B). The user-deﬁned parameter was made changeable during the simulation as alight intensity L = (α, β, γ, ε). The user can change thecolor of the ﬂame while watching the simulation result by
adjusting L.
3.6.2 Visual particles
To improve the quality of the image obtained by thesimulation, it is necessary to reduce the radius of theparticle that is to be positioned and to perform a moredetailed simulation. However, a decrease in the particle radius increases the number of particles required for
simulation, increasing the calculation time and decreasing the interactivity. Therefore, instead of increasing thenumber of particles used in the simulation, the visual particles were placed in the area where combustion reactions
are likely to occur in the simulation space. This is themethod used by Macklin et al. 15) to render smoke in CG.
To locate the drawn particles only in the area where
the combustion reaction occurs, the visual particles are
positioned in the area where fuel is supplied. The velocity

(46)

∫

I`T (λ, T )w̄(λ)dλ.

hcw̄(430 nm)
λ430 nmhc+βufuel NAw̄(510 nm)
λ510 nmhcw̄(460 nm).
+γuCO2 NA
λ460 nmwhere λ, c, Kb , and h are the wavelength, speed of light,
Boltzmann’s constant, and Planck’s constant, respectively. α is the user-deﬁned parameter that determines
the blackbody radiation. Equation (46) provides the intensity of each wavelength at a given temperature. Using
the color matching function w̄(λ) = (x̄(λ), ȳ(λ), z̄(λ)), therelationship between the color values WT = (XT , YT , ZT )in the XYZ color system and spectral radiant ﬂux is
WT = αSsoot

(48)

(47)

Because a large amount of integration calculation is required, we create a table in advance and refer to the temperature in the table to obtain the values for the simulation.
Next, we describe the color due to the luminescenceof the reaction intermediate. The luminescence of thereaction intermediate is composed mainly of the 430 nm
emission spectrum of methylidene radical (CH), 510 nm
emission spectrum of diatomic carbon (C2 ), and 460 nm
emission spectrum of carbon monoxide. The emission of
methylidene radicals and diatomic carbon is conﬁrmed
when the fuel decomposes, and the emission of carbon
monoxide is conﬁrmed when it burns. Accordingly, we
model the spectral ﬂux Ir owing to the emission of thereaction intermediate as26

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Algorithm 1 Flame simulation
1: for all particle i do
2:
update velocity vi and position xi by PBF method
3: end for
4: for all particle i do
5:calculate viscosity eﬀect
6: end for
7: for all particle i do
8:solve diﬀusion equation using Eq.(15)9:update each chemical species concentration and tem-

perature(a)

10: end for
11: for all particle i do
12:compute decomposition reaction speed using Eq.(19)(b)

speed uupdate each chemical species concentration
compute combustion reaction speed u using Eq.(23),
Eq.(24), and Eq.(27)
15:
update each chemical species concentration
16:
update temperature
17:
calculate ambient temperature
18: end for
19: for all particle i do
20:
calculate color and update
21:
update mass and default density
22: end for
23: for all visual particle i do
24:
calculate velocity and color vi and ci using Eq.(51) andEq.(52)25:
update position xi .
26: end forFig. 3 Rendering results; (a) Visual particles, (b) Computational particles

13:
14:

Table 1 Particle i’s physical quantity
Quantity
Notationmass
midensityρirest density
ρ0i
amount of species ℓdi,ℓtemperatureTi
ambient temperature
Tiamb
PBD variable
λi
reaction rate of Eq.(22)uCO2reaction rate of Eq.(18)
ufuelvi of the visual particle Ψi can be obtained from thatof the computational particles around the visual particle
using the equation∑vi =and amount of chemical species contained in the particles are updated. Next, the velocity of the particles is
modiﬁed using the updated temperature. Furthermore,
the color of the particle is determined from the obtained
particle temperature, amount of chemical species, and reaction rate. Finally, the velocity and color of the particles
are determined for drawing.
We use the Poly6 kernel for Wij and the Spiky kernelfor ∇Wij , as in Müller et al. 18). The following boundary condition is applied: particles that collide with thewall reverse their velocity in the normal direction. At
the same time, the temperature and amount of chemical
species are returned to the initial state. In this simulation, we used eight chemical species: oxygen, nitrogen,carbon dioxide, carbon monoxide, carbon atoms, hydrogen atoms, water, and fuels. The parameters of particle iare given in Table 1. The parameters used in this simulation are given in Tables 2, 3, 4 and 5. In Table 4, weshow the Arrhenius parameter A and Ea of each chemical
reactions. Decomposition reaction (Eqs. (18) and (41))
is not real reaction, thus there is no referenceable data.
We implement Arrhenius parameter of Eqs. (18) and (41)
as a tunable parameter. In Table 5, all coeﬃcients (a–f)
in Eq.(38) refer to Ref.19). In Table 5, D0 of the chemical species without reference data was obtained using the

vj W (xi − xj , ξ)j<n∑W (xi − xj , ξ)

,

(51)

j<n

Using the same method, the color ci of the visual particle
Ψi is obtained as∑

cj W (xi − xj , ξ)j<nci = ∑

W (xi − xj , ξ)

,

(52)

j<n

Figure 3 compares the rendering result of visual particles with that of computational particles, where the number of particles is the same. By arranging visual particles
between computational particles, it is possible to express
a ﬁner ﬂame while maintaining interactivity. In Fig. 3,
the radius of visual particle is 0.35 times the radius of thecomputational particles.

4.

Implementation

For ensuring interactivity, the proposed model is implemented on a GPU using CUDA. The pseudo code isshown in Algorithm 1.
First, the particle position and velocity are updated by
the PBF method. Subsequently, the diﬀusion and chemical reaction equations are solved, and the temperature

27

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）
Table 2 Parameters
Parameter
Notationcomputational particle radiusrtime step∆t
gravitygreference pressurep0
initial temperatureT0
buoyancy constantϑsoot constantφambient light constantϱheat absorbed
Qdc

Value
0.20 m0.10 ms
9.81 m/s2
100000 Pa
20 °C
0.0051
1.0 ×1012
1.0 ×107-100 kJ/mol(a)

Table 3 Standard Gibbs free energy of formation
Species Notation
Value [kJ/mol]
CO2∆f GCO2-394.39
CO∆f GCO-137.16
C∆f GC
671.20H∆f GH
203.28
H2 O∆f GH2 O-228.14C2 H6 O ∆f GC2 H6 O
-135.8
6.8
C8 H18 ∆f GC8 H18
Table 4 Arrhenius parameters
Chemical reaction
A
Ea
Eq.(18)
1.0 ×1010
130 ×103
Eq.(21)3.745684 ×107
0.1 ×103
6
Eq.(22)4.504 ×10
201 ×1038
Eq.(25)1.830688 ×10 67.55 ×103
Eq.(41)
1.0 ×105
150 ×103

(b)

(c)

(d)

Fig. 5 Split display of light source; (a) Original ﬂame,(b) Highlighting the light derived from fuel decomposition, (c) Highlighting the light derived
from the combustion of carbon monoxide, and(d) Highlighting the light from the black bodyradiation of the soot

Ref
19)
19)
19)

-

(a)

(b)

(c)

(d)

Fig. 6 The color of the ﬂame changed by manipulating
the light intensity L; (a) L = (20, 0.4, 0.1, 0.1) ,(b) L = (1, 1, 0.02, 0.1) , (c) L = (1, 1, 1, 1) , and(d) L = (50, 1.25, 0, 0.1)

(a)

(b)

(c)

(C2 H6 O) as the fuel and L set to (20.0, 0.40, 0.10, 0.10).
Figure 4(a) to Figure 4(d) shows the images after the fuelhas been supplied and ignited.
Figure 5 shows the result when each light source is displayed separately. Figure 5(b) shows the turquoise blue
light from the fuel decomposition and is found to occur
mainly near the base of the ﬂame. This result is consistent with the characteristics of the actual ﬂame combustion phenomena. Figure 5(c) shows the pale blue lightfrom the combustion of carbon monoxide that is particularly strong at the outer edge of the ﬂame. Figure 5(d)shows the red and yellow light from the black body radiation of soot that is particularly strong at the tip of theﬂame. These results capture the characteristics of a real
ﬂame when the combustion is complete, which is mainly
due to a large amount of oxygen at the outer edge of theﬂame; incomplete combustion occurs because of a lack ofoxygen inside the ﬂame, and a large amount of soot isgenerated.
As shown in Fig. 5, our simulation captures the ba-(d)Fig. 4 Ignition experiment of ethanol by the proposed
method; (a) A sample image immediately after
ignition, and (b)–(d) Sample images after thepassage of time

data of chemical species with similar atomic weights.

5.

Results

In this section, we describe the results achieved by implementing and operating the proposed method. For performing the simulations, we used an NVIDIA GeForceGTX 1060 GPU, Intel Core i7 7700k CPU, and 16GB
RAM. C++ was used as the development language,
OpenGL 4.6 as the graphic library, and CUDA as the
GPGPU environment.
Figure 4 shows the reproduction of the ﬂame generation using the proposed method We speciﬁed ethanol

28

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Species
O2
N2
CO2
CO
C
H
H2 O
C2 H6 O
C8 H18CH4(a)D0
0.21 ×10−4
0.168 ×10−4
0.113 ×10−4
0.15 ×10−4
0.181 ×10−4
0.257 ×10−4
0.296 ×10−4
0.12×10−4
0.109×10−4
0.10×10−4(b)Table 5 Coeﬃcients of diﬀusion and speciﬁc heat
Ref
a
b
c
d
20)
30.03235
8.772972-3.988133 0.788313
20)19.50583
19.88705-8.598535 1.369784
20)
58.16639
2.720074-0.492289 0.038844
25.56759
6.0961304.054656 -2.671301
21.17510-0.812428
0.448537 -0.043256
33.066178 -11.363417 11.432816 -2.772874
20)
30.09200
6.832514
6.793435 -2.534480
4.9807239.94-135.34
37.346-16.776813.28-435.193.657-0.703029
108.4774-42.52157 5.862788

(c)

e
0
0
0
0
0
0
0
-4.029100f-0.741599
0.527601-6.447293
0.131021-0.013103-0.158558
0.082139000.678565

the simulation results. For comparison, we chose the bunsen burner ﬁre. For the simulation, methane, which is
the main fuel in the bunsen burner, was used. We lit theburner without supplying oxygen, and then increased theoxygen supply until the ﬁre became completely blue. A
video of the entire process was captured using a SONY
FDR-AX700. L = (2, 1, 2, 0.1), and we add 1.2 m/s speed
towards vertical upward in the fuel supply area. Figure8(a) and Fig. 8(d) show that the images when only gas
is supplied. Figure 8(b) and Fig. 8(e) capture the image
when oxygen is supplied. Figure 8(c) and Fig. 8(f) are
the cases where oxygen and gas are supplied and complete
combustion occurs. In the real images, ﬁre color changes
with increasing supply of oxygen. Although the supply ofoxygen varied, our simulation successfully captured thisphenomenon by capturing the changes in the characteristics of real ﬁre color, which is not possible using theconventional methods 3),6),12).
In contrast, there is a diﬀerence between the real
footage and simulation results in Fig. 8(c) and Fig. 8(f).
In the real situation, oxygen was supplied at the center ofthe burner, but oxygen was supplied equally in the fuelsupply area during the simulation. It is assumed thatthe diﬀerence in the distribution of oxygen supply caused
a diﬀerence in the ﬁre color, especially when the oxygen
supply was high. By aligning the distribution of oxygen
supply, more plausible simulation results can be achieved.
Figure 9 is an example of solid fuel combustion. Solidparticles were placed near the tip of a match. The appearance of a ﬁre from a lighter igniting the match is
reproduced.
Figure 10 demonstrates changes in the ﬂame causedby adding oxygen simultaneously with the fuel. Figure 10(a) shows the case where only fuel is used, Fig.
10(b) shows the case where a small amount of oxygenis supplied, and Fig. 10(c) shows the case where alarge amount of oxygen is supplied. In all three ﬁgures,(d)Fig. 7 Combustion simulation of octane; (a) L =(20, 0.4, 0.1, 0.1) , (b) L = (1, 1, 0.02, 0.1) , (c)L = (1, 1, 1, 1) , and (d) L = (50, 1.25, 0, 0.1)

sic characteristics of the ﬂame combustion reaction andcan easily create diﬀerent types of ﬂames by changing the light intensity L(Fig. 6). In each column,
the light intensity L is equal.
From the left column, L is (20, 0.4, 0.1, 0.1), (1, 1, 0.02, 0.1), (1, 1, 1, 1),
and (50, 1.25, 0, 0.1). Figure 6(a) shows the result for
L = (20, 0.4, 0.1, 0.1). A ﬂame with a blue base and redtip, as seen in candles, is generated. Figure 6(b) shows
the result for L = (1, 1, 0.02, 0.1). A faint blue ﬂame is
produced, which is seen when alcohol is burned. Figure6(c) is the result for L = (1, 1, 1, 1). A red ﬂame is generated, which is common in bonﬁres. Figure 6(d) is theresult for L = (50, 1.25, 0, 0.1). An intense blue-coloredﬂame, seen mainly in gas burners, is produced.
Figure 7 shows the results when the fuel was changed
from ethanol to octane (C8 H18 ). The light intensity L
is the same as in Fig. 6. Although the ﬂame is expanded sideways compared to the shape before the fuelwas changed, there is no signiﬁcant change in the colorof the ﬂame. The proposed combustion reaction model
could not reproduce the changes in the ﬂame color due
to diﬀerences in the fuel.
To check whether our simulation can capture the features of real ﬁre color, we compared the real images with29

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Fig. 8 Comparison of real iamges (upper) and corresponding simulation results(lower); (a)(d) Only gas issupplied, (b)(e) Oxgen is supplied, and (c)(f) Gas and oxygen are supplied

(a)

(b)

(c)

Fig. 9 Simulation of a burning match

(a)

(b)

soot generated is clearly greater than when oxygen is not
added. The addition of oxygen is considered to increase
the likelihood that the reaction will proceed and cause alarge amount of fuel decomposition, while the generationof water consumes most of the oxygen.
Figure 11 shows an example of a dragon’s breath reproduced using the proposed method. The fuel is supplied and the initial velocity is added. Figure 11 (a)–(d)show the results of changing the light intensity L for various ﬂame colors.
The simulation time was used to evaluate the system.
The relationship between the simulation time and thenumber of particles is summarized in Table 6. Our
simulation operates at approximately 20 fps with 50,000
computational particles and 100,000 visual particles. According to Table 6, the computation time when using the
proposed combustion model is reduced by approximately
half to one-third than that by the constraint method.

(c)Fig. 10 Results of diﬀerent supplying of oxygen to fuel;(a) When oxygen is not supplied, (b) When asmall amount of oxygen is supplied, and (c)When a large amount of oxygen is supplied

L = (10, 0.5, 0.1, 0.1). The case where only fuel is used issimilar to the real situation where lighter gas or a candle
is burning. The supplied fuel diﬀuses and reacts with theoxygen originally present in the air. However, the case inwhich fuel and oxygen are added simultaneously is similar
to the situation of combustion in a burner. By supplying
a large amount of oxygen, the generation of soot can beeliminated completely as the fuel is burned completely.
When a small amount of oxygen is added, the amount of

6.

Limitation

In our method, the combustion reaction is represented
by the same reaction path as in Fms. (18), (21), (22),(25), (26) of the generalized fuel Ca Hb Oc . The values of30

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

(a)

(b)

(c)

(d)

Fig. 11 Reproduction of a dragon’s breath
Table 6 Relationship between computation time and number of particles
Scene
Fig. 9Fig. 4
Fig. 11

Simulationspace
3m × 6m × 3m
3m × 6m × 3m
5m × 8m × 1m

Computational
particles
49,07249,011
37,576Visualparticles
56,250100,000
120,000

Constraint
Solver
18.9 ms
16.3 ms
15.7 ms

Combustion andDiﬀusion
11.3 ms
7.01 ms
5.10 ms

evaporation.

a, b, and c are changed by replacing the fuel, and the ﬁrecolor becomes slightly diﬀerent. In contrast, the combustion reaction path and the ﬁre color should vary depending on the type of fuel. However, the proposed method
does not cover the diﬀerences in the reaction paths. In
addition, to handle large ﬂames, a ﬂuid simulation using many particles is required; however, interactivity islost in this case. One solution is to position the particles
along the vicinity of the ﬂame, to simulate the ﬂame inreal time and shorten the calculation time.

7.

Total
time
45.7 ms
42.7 ms
49.8 ms

Acknowledgement
This research was supported in part by JSPS KAKENHI under Grant 18H05010.

References
1) S. Sato, K. Mizutani, Y. Dobashi, T. Nishita, T. Yamamoto:“Feedback Control of Fire Simulation Based on Computational
Fluid Dynamics”, Computer Animation and Virtual Worlds,
Vol. 28, No. 3-4, p. e1766 (2017).
2) J. Zhu, Y. Chang, E. Wu: “Realistic, Fast, and Controllable
Simulation of Solid Combustion”, Computer Animation andVirtual Worlds, Vol. 22, No. 2‐3, pp. 125–132 (2011).

Conclusion

3) C. Horvath, W. Geiger: “Directable, High-Resolution Simulation of Fire on the GPU”, ACM Trans. on Graphics, Vol. 28,
No. 3, pp. 41:1–41:8 (2009).

In this paper, we proposed a simple combustion reaction model that can reproduce complex ﬂame colors
interactively. The combustion reaction, which involves
thousands of chemical reactions, is modeled using ﬁve reactions. The blue and red colors of the ﬂame can be
simulated while maintaining interactivity. The calculation time of the simulation was measured, and it was
conﬁrmed that this simulation could operate at 20 fps
or more if the number of calculated particles is approximately within a 50,000 limit. Thus, we can interactively
change the color of the ﬂame during the simulation. In
contrast, by increasing the particle number, we can decrease the artifact. In addition, our proposed method
adds the simulation of the combustion reaction and reactions to the ﬂuid simulation of PBF, and therefore, it
can be easily combined with physical simulation using
other PBD models. For example, it is also possible tosimulate the burning of a rigid body by combining thesimulation with a rigid-body simulation produced by the
PBD method. In the future, this study can be extended
to simulate the combustion of a liquid by modeling its

4) Z. Melek: “Interactive Simulation of Fire, Burn and Decomposition”, Ph.D. Thesis, Texas A&M University (2007).
5) A. Lamorlette, N. Foster: “Structural Modeling of Flames for a
Production Environment”, ACM Trans. on Graphics, Vol. 21,
No. 3, pp. 729–735 (2002).
6) D. Q. Nguyen, R. Fedkiw, H. W. Jensen: “Physically Based
Modeling and Animation of Fire”, ACM Trans. on Graphics,
Vol. 21, No. 3, pp. 721–728 (2002).
7) I. Ihm, B. Kang, D. Cha: “Animation of Reactive Gaseous
Fluids through Chemical Kinetics”, Proc. of the 2004 ACM
SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’04, pp. 203–212 (2004).
8) B. Kang, Y. Jang, I. Ihm: “Animation of Chemically Reactive Fluids Using a Hybrid Simulation Method”, Proc. of the2007 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’07, pp. 199–208 (2007).
9) M. Macklin, M. Müller: “Position Based Fluids”, ACM Trans.
on Graphics, Vol. 32, No. 4, pp. 104:1–104:12 (2013).
10) M. Faraday, The Chemical History of a Candle, Crowell (1957).
11) G. H. Markstein, Nonsteady Flame Propagation, Pergamon(1964).
12) J.-M. Hong, T. Shinar, R. Fedkiw: “Wrinkled Flames andCellular Patterns”, ACM Trans. on Graphics, Vol. 26, No. 3(2007).
13) K. Min, D. Metaxas: “A Combustion-Based Technique for Fire31

IIEEJ Transactions on Image Electronics and Visual Computing Vol.9 No.1 （2021）

Sei

Animation and Visualization”, The Visual Computer, Vol. 23,
No. 9, pp. 679–687 (2007).

IKEDA

He received his Ph.D. degree from Nara Institute of Science and Technology, Ikoma, Japan
in 2006. He is an Associate Professor with
the Graduate School of Engineering Scienceof Osaka University. His research interests
include human-computer interaction, mixed
reality, and computer vision.

14) J. Stam, E. Fiume: “Depicting Fire and Other Gaseous Phenomena Using Diﬀusion Processes”, Proc. of ACM the 22nd
Annual Conference on Computer Graphics and Interactive
Techniques, SIGGRAPH ’95, pp. 129–136, (1995).
15) M. Macklin, M. Müller, N. Chentanez, T. Kim: “Uniﬁed Particle Physics for Real-Time Applications”, ACM Trans. on
Graphics, Vol. 33, No. 4, pp. 153:1–153:12 (2014).
16) H. Schechter, R. Bridson: “Ghost SPH for Animating Water”,
ACM Trans. on Graphics, Vol. 31, No. 4, pp. 61:1–61:8 (2012).

Yoshihiro KURODA
He received the Ph.D. degree in informatics from Kyoto University, Kyoto, Japan, in
2005. He has been a Professor with the Faculty of Engineering, Information and Systems
at University of Tsukuba. His research interests include haptic interaction technologies
and biomedical engineering.

17) N. M. Marinov: “A Detailed Chemical Kinetic Model forHigh Temperature Ethanol Oxidation”, International Journalof Chemical Kinetics, Vol. 31, No. 3, pp. 183–220 (1999).
18) M. Müller, D. Charypar, M. Gross: “Particle-Based Fluid Simulation for Interactive Applications”, Proc. of the 2003 ACM
SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’03, pp. 154–159, Eurographics Association (2003).
19) W. M. P.J. Linstrom, Nist Chemistry Webbook, Nist Standard
Reference Database Number 69, https://doi.org/10.18434/
T4D303 (2020).

Shunsuke YOSHIMOTO

20) Thermopedia, www.thermopedia.com/content/696/ (2020).

(Received July 2, 2020)(Revised December 27, 2020)

Ryosuke UCHIKAWA(Student Member)He received the B.E. degrees from Osaka University, Toyonaka, Japan in 2019. His research intersts include physically based simulation.

Hiroki ISHIZUKAHe received the Ph.D. degree in Integrated
De-sign Engineering from Keio University
in2016. He is an Assistant Professor with theGraduate School of Engineering Science ofOsaka University. His research interests in-clude
tactile displays and soft robotics.

32

He received the Ph.D. degree in engineeringfrom Osaka University, Toyonaka, Japan, in
2012. From 2020, he is a Lecturer with the
Graduate School of Frontier Sciences, The
University of Tokyo. His research interests
include haptic engineering and biomedical instrumentation.

Osamu OSHIROHe received the Ph.D. degree in engineering from Osaka University, Toyonaka, Japan,
in 1990. He was a Research Engineer with
the Sumitomo Metal Industries from 1990 to
1993 and was an Assistant/Associate Professor with the Nara Institute of Science andTechnology from 1993 to 2003. Since 2003,
he has been a Professor with the Graduate
School of Engineering Science, Osaka University. His research interests include PBR, facing technology, modeling of distributed human system, etc.

3rd Call for Papers (Call for Late Breaking Papers)
The 7th IIEEJ International Conference on
Image Electronics and Visual Computing 2021
(IEVC2021)
Shiretoko (Shari), Hokkaido, Japan / Sept. 8-11, 2021

https://www.iieej.org/en/ievc2021/
Purpose:The International Conference on Image Electronics and Visual Computing
2021 (IEVC2021) will be held in Shiretoko (Shari), Hokkaido, Japan, on Sept.
8-11, 2021, as the 7th international academic event of the Institute of
Image Electronics Engineers of Japan (IIEEJ) based on the great success of
IEVC2007 held in Cairns, Australia, IEVC2010 held in Nice, France, IEVC2012
held in Kuching, Malaysia, IEVC2014 held in Koh Samui, Thailand, IEVC2017
in Danang, Vietnam, and IEVC2019 held in Bali, Indonesia. The conference
aims to bring together researchers, engineers, developers, and students
from various fields in both academia and industry for discussing the latest
researches, standards, developments, implementations and application
systems in all areas of image electronics and visual computing.

Conference venue:
The conference venue is Yumehall Shiretoko, Shari Town, Hokkaido, JAPAN.

Paper submission:

The official language is English, and authors should submit their papers as PDF through the online submission system.
For details, visit the IEVC2021 official website.
(General papers submissions have already closed.)

Late Breaking Papers:

All suitably submitted papers for this category will be accepted for the conference. The authors must submit an
abstract of which length is 1-2 pages, and select one from the following two types: 1) Technical papers or 2)
Art/Demo papers. All the registered papers as late breaking papers will be published only in the USB proceedings of
IEVC2021.
Important Dates- Pre-Entry Submission (title, authors):
June 28, Monday, 2021 (extended)- Abstract Submission (1-2 pages):
July 1, Thursday, 2021 (extended)- Notification of Acceptance:
July 5, Monday, 2021
- Camera-Ready Paper (1-2 pages):
July 9, Friday, 2021 (extended)

Important information:

Online participation/presentation will beallowed for those who cannot come to thevenue and make a presentation due to the
COVID-19.

33

Call for Papers
Special Issue on
Image-related Technology for Realizing Immersive MediaIIEEJ Editorial Committee
Research on immersive media, which is an advanced system with such as all-sky video (360-degree video), VR,
AR, and MR, is being actively conducted. In addition, the development of a head-mounted display (HMD) and thestudy of specific user interfaces are underway to fully demonstrate the appeal and the benefit of immersive media.
In ISO / IEC, Coded Representation of Immersive Media (MPEG-I) has been studied for realizing immersive media
in MPEG team, and though its standardization is being completed, further technologies are expected to be
standardized. ITU-T SG16 is also promoting the recommendation of ultra-high presence live experience (ILE:
Immersive Live Experience), and the momentum for systematization is increasing.
In this special issue, we are looking for a wide range of papers on elemental technologies that realize immersive
media and system development papers on research to realize immersive high-presence systems by applying thesetechnologies.
1. Topics covered include but not limited toAll-sky video (360-degree video), Free viewpoint Image, Point cloud, Light field, Holography, Head-mounted
display (HMD), User interface, User experience, Usability, Interaction, Immersive content, VR, AR, MR, SR,
xR, CG, Image processing, Image coding, Computer vision, Deep learning, MPEG-I, ILE
2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary contributed
paper. If the number of accepted papers is less than the minimum number for the special issue, the acceptancepaper will be published as an ordinary contributed paper. We ask for your understanding and cooperation.
3. Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.9, No.2 (December 2021)
4. Submission Deadline（Extended）
Monday, May 31, 2021 ⇒ Wednesday, June 30, 2021
5. Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL: http://www.editorialmanager.com/iieej/

34

Call for Papers
Special Issue on CG & Image Processing Technologies
Supporting and Expanding Human Creativities
IIEEJ Editorial Committee
The AI technology is expected to become a key technology for solving social problems and SDGs
(Sustainable Development Goals), such as declining birthrate, aging population, shortage of labor,
depopulation area, and so on, to which mature society, especially Japan is facing. The application scope ofthe technology in the field of image processing is expanding beyond the image generation and object
recognition to include areas related to creativity, such as attribute conditioned image generation,
super-resolution, image colorization, and line art coloring.
On the other hand, deep learning techniques don’t always have good nature at interpretability andexplainability of results required for creativity processing. Also, there is still issues to be addressed in AI
technology, such as the ineffectiveness to the human-computer interface field by a simple application. So,
technologies to support and expand the human creativity are gathering much concern, and quite promising inwide variety of applications.
This special issue of the paper targets various image-related technologies that support and expand
creativities, and calls for papers and system development papers that cover not only applications of deeplearning but also other technologies.
1.

Topics covered include but not limited to
Image Processing, Image Recognition, Image Detection, Pattern Recognition,
Computer Graphics, Visualization, Binocular Vision, 3D image processing
Computer Vision, Big Data, Image Data Bases,
Machine Learning, Deep Learning, Understandability, Explainability,
Creativity, Usability, Interpretability, Human Interface and Interaction, User Experience, Ubiquitous,
Other related fundamental / application / systemized technologies.

2.

Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary
contributed paper. If the number of accepted papers is less than the minimum number for the special
issue, the acceptance paper will be published as an ordinary contributed paper. We ask for yourunderstanding and cooperation.

3.

Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.10, No.1 (June 2022)

4.

Submission Deadline:

Tuesday, November 30, 2021
5.

Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org

6.

Online Submission URL: http://www.editorialmanager.com/iieej/

35

Revised: January 6, 2017
Revised: July 6, 2018

Guidance for Paper Submission
1.

Submission of Papers(1) Preparation before submission
・ The authors should download “Guidance for Paper Submission” and “Style Format” from the“Academic Journals”, “English Journals” section of the Society website and prepare the paperfor submission.
・ Two versions of “Style Format” are available, TeX and MS Word. To reduce publishing costs
and effort, use of TeX version is recommended.
・ There are four categories of manuscripts as follows: Ordinary paper: It should be a scholarly thesis on a unique study, development orinvestigation concerning image electronics engineering. This is an ordinary paper topropose new ideas and will be evaluated for novelty, utility, reliability andcomprehensibility. As a general rule, the authors are requested to summarize a paper
within eight pages.
 Short paper: It is not yet a completed full paper, but instead a quick report of the partial
result obtained at the preliminary stage as well as the knowledge obtained from the said
result. As a general rule, the authors are requested to summarize a paper within four
pages.
 System development paper: It is a paper that is a combination of existing technology or it
has its own novelty in addition to the novelty and utility of an ordinary paper, and thedevelopment results are superior to conventional methods or can be applied to othersystems and demonstrates new knowledge. As a general rule, the authors are requested
to summarize a paper within eight pages.
 Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though
its novelty is not high. As a general rule, the authors are requested to summarize a paper
within eight pages.
・ To submit the manuscript for ordinsry paper, short paper, system development paper, or data
paper, at least one of the authors must be a member or a student member of the society.
・ We prohibit the duplicate submission of a paper. If a full paper, short paper, systemdevelopment paper, or data paper with the same content has been published or submitted toother open publishing forums by the same author, or at least one of the co-authors, it shallnot be accepted as a rule. Open publishing forum implies internal or external books,
magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). A paper that was once approved as a short paper andbeing submitted again as the full paper after completion is not regarded as a duplicate
submission.
(2) Submission stage of a paper・ Delete all author information at the time of submission. However, deletion of reference
information is the author’s discretion.
・ At first, please register your name on the paper submission page of the following URL, andthen log in again and fill in the necessary information. Use the “Style Format” to upload yourmanuscript. An applicant should use PDF format (converted from dvi of TeX or MS Word36

format) for the manuscript. As a rule, charts (figures and tables) shall be inserted into themanuscript to use the “Style Format”. (a different type of data file, such as audio and video,
can be uploaded at the same time for reference.)
http://www.editorialmanager.com/iieej/
・ If you have any questions regarding the submission, please consult the editor at our office.
Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-Ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2.

Review of Papers and Procedures(1) Review of a paper・ A manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “acceptance”, “conditionally acceptance” or “returned”. The applicant is
notified of the result of the review by E-mail.
・ Evaluation method
Ordinary papers are usually evaluated on the following criteria:
 Novelty: The contents of the paper are novel.
 Utility: The contents are useful for academic and industrial development.
 Reliability: The contents are considered trustworthy by the reviewer.
 Comprehensibility: The contents of the paper are clearly described and understood by
the reviewer without misunderstanding.
Apart from the novelty and utility of an ordinary paper, a short paper can be evaluated byhaving a quickness on the research content and evaluated to have new knowledge withresults even if that is partial or for specific use.
System development papers are evaluated based on the following criteria, apart from thenovelty and utility of an ordinary paper.
 Novelty of system development: Even when integrated with existing technologies, thenovelty of the combination, novelty of the system, novelty of knowledge obtained fromthe developed system, etc. are recognized as the novelty of the system.
 Utility of system development: It is comprehensively or partially superior compared tosimilar systems. Demonstrates a pioneering new application concept as a system. The
combination has appropriate optimality for practical use. Demonstrates performance
limitations and examples of performance of the system when put to practical use.
Apart from the novelty and utility of an ordinary paper, a data paper is considered novel if
new deliverables of test, application and manufacturing, the introduction of new technology
and proposals in the worksite have any priority, even though they are not necessarily
original. Also, if the new deliverables are superior compared to the existing technology andare useful for academic and industrial development, they should be evaluated.
(2) Procedure after a review
・ In case of acceptance, the author prepares a final manuscript (as mentioned in 3.).
・ In the case of acceptance with comments by the reviewer, the author may revise the paper inconsideration of the reviewer’s opinion and proceed to prepare the final manuscript (as37mentioned in 3.).
・ In case of conditional acceptance, the author shall modify a paper based on the reviewer’s
requirements by a specified date (within 60 days), and submit the modified paper for
approval. The corrected parts must be colored or underlined. A reply letter must be attachedthat carefully explains the corrections, assertions and future issues, etc., for all of theacceptance conditions.
・ In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.
(3) Review request for a revised manuscript
・ If you want to submit your paper after conditional acceptance, please submit the reply letterto the comments of the reviewers, and the revised manuscript with revision history to thesubmission site. Please note the designated date for submission. Revised manuscripts
delayed more than the designated date be treated as new applications.
・ In principle, a revised manuscript willl be reviewed by the same reviewer. It is judged either
accceptance or returned.
・ After the judgment, please follow the same procedure as (2).

3.

Submission of final manuscript for publication(1) Submission of a final manuscript
・ An author, who has received the notice of “Acceptance”, will receive an email regarding thecreation of the final manuscript. The author shall prepare a complete set of the final
manuscript (electronic data) following the instructions given and send it to the office by thedesignated date.
・ The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings (including bmp, jpg, png), an eps file for author’s photograph(eps or jpg file of more than 300 dpi with length and breadth ratio 3:2, upper part of the body)for authors’ introduction. Please submit these in a compressed format, such as a zip file.
・ In the final manuscript, write the name of the authors, name of an organizations,introduction of authors, and if necessary, an appreciation acknowledgment. (cancel macros in
the Style file)・ An author whose paper is accepted shall pay a page charge before publishing. It is theauthor’s decision to purchase offprints. (ref. page charge and offprint price information)(2) Galley print proof
・ The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and prepare a PDF file, and send it to our officeby email. At that time, fill in the Offprint Purchase Slip and Copyright Form and return thescanned data to our office in PDF file form.
・ In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.
・ You can download the Offprint Purchase Slip and the Copyright Form from the journal on our
homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication
・ After final proofreading, a paper is published in the Academic journal or English transaction(both in electronic format) and will also be posted on our homepage.

38

IIEEJ Trans. on Image Electronics and Visual Computing©2021 IIEEJ

Vol. 9, No. 1 2021

Editor in Chief: Mei Kodama
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan