Journal Editorial Board Editor-in-ChiefProf. Thomas Byeongnam YOON, PhD.
Founding Editor-in-Chief
ICACT Transactions on the Advanced Communications Technology (TACT) Editors
Prof. Jun-Chul Chun, Kyonggi University, Korea
Dr. JongWon Kim, GIST (Gwangju Institute of Science & Technology), Korea
Dr. Xi Chen, State Grid Corparation of China, China
Prof. Arash Dana, Islamic Azad university , Central Tehran Branch, Iran
Dr. Pasquale Pace, University of Calabria - DEIS - Italy, Italy
Dr. Mitch Haspel, Stochastikos Solutions R&D, Israel
Prof. Shintaro Uno, Aichi University of Technology, Japan
Dr. Tony Tsang, Hong Kong Polytechnic UNiversity, Hong Kong
Prof. Kwang-Hoon Kim, Kyonggi University, Korea
Prof. Rosilah Hassan, Universiti Kebangsaan Malaysia(UKM), Malaysia
Dr. Sung Moon Shin, ETRI, Korea
Dr. Takahiro Matsumoto, Yamaguchi University, Japan
Dr. Christian Esteve Rothenberg, CPqD - R&D Center for. Telecommunications, Brazil
Prof. Lakshmi Prasad Saikia, Assam down town University, India
Prof. Moo Wan Kim, Tokyo University of Information Sciences, Japan
Prof. Yong-Hee Jeon, Catholic Univ. of Daegu, Korea
Dr. E.A.Mary Anita, Prathyusha Institute of Technology and Management, India
Dr. Chun-Hsin Wang, Chung Hua University, Taiwan
Prof. Wilaiporn Lee, King Mongkut’s University of Technology North, Thailand
Dr. Zhi-Qiang Yao, XiangTan University, China
Prof. Bin Shen, Chongqing Univ. of Posts and Telecommunications (CQUPT), China
Prof. Vishal Bharti, Dronacharya College of Engineering, India
Dr. Marsono, Muhammad Nadzir , Universiti Teknologi Malaysia, Malaysia
Mr. Muhammad Yasir Malik, Samsung Electronics, Korea
Prof. Yeonseung Ryu, Myongji University, Korea
Dr. Kyuchang Kang, ETRI, Korea
Prof. Plamena Zlateva, BAS(Bulgarian Academy of Sciences), Bulgaria
Dr. Pasi Ojala, University of Oulu, Finland
Prof. CheonShik Kim, Sejong University, Korea
Dr. Anna bruno, University of Salento, Italy
Prof. Jesuk Ko, Gwangju University, Korea
Dr. Saba Mahmood, Air University Islamabad Pakistan, Pakistan
Prof. Zhiming Cai, Macao University of Science and Technology, Macau
Prof. Man Soo Han, Mokpo National Univ., Korea
Mr. Jose Gutierrez, Aalborg University, Denmark

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Dr. Youssef SAID, Tunisie Telecom, Tunisia
Dr. Noor Zaman, King Faisal University, Al Ahsa Hofuf, Saudi Arabia
Dr. Srinivas Mantha, SASTRA University, Thanjavur, India
Dr. Shahriar Mohammadi, KNTU University, IranProf. Beonsku An, Hongik University, korea
Dr. Guanbo Zheng, University of Houston, USA
Prof. Sangho Choe, The Catholic University of Korea, korea
Dr. Gyanendra Prasad Joshi, Yeungnam University, korea
Dr. Tae-Gyu Lee, Korea Institue of Industrial Technology(KITECH), korea
Prof. Ilkyeun Ra, University of Colorado Denver, USA
Dr. Yong Sun, Beijing University of Posts and Telecommunications, China
Dr. Yulei Wu, Chinese Academy of Sciences, China
Mr. Anup Thapa, Chosun University, korea
Dr. Vo Nguyen Quoc Bao, Posts and Telecommunications Institute of Technology, Vietnam
Dr. Harish Kumar, Bhagwant institute of technology, India
Dr. Jin REN, North china university of technology, China
Dr. Joseph Kandath, Electronics & Commn Engg, India
Dr. Mohamed M. A. Moustafa, Arab Information Union (AIU), Egypt
Dr. Mostafa Zaman Chowdhury, Kookmin University, Korea
Prof. Francis C.M. Lau, Hong Kong Polytechnic University, Hong Kong
Prof. Ju Bin Song, Kyung Hee University, korea
Prof. KyungHi Chang, Inha University, Korea
Prof. Sherif Welsen Shaker, Kuang-Chi Institute of Advanced Technology, China
Prof. Seung-Hoon Hwang, Dongguk University, Korea
Prof. Dal-Hwan Yoon, Semyung University, korea
Prof. Chongyang ZHANG, Shanghai Jiao Tong University, China
Dr. H K Lau, The Open University of Hong Kong, Honh Kong
Prof. Ying-Ren Chien, Department of Electrical Engineering, National Ilan University, Taiwan
Prof. Mai Yi-Ting, Hsiuping University of Science and Technology, Taiwan
Dr. Sang-Hwan Ryu, Korea Railroad Research Institute, Korea
Dr. Yung-Chien Shih, MediaTek Inc., Taiwan
Dr. Kuan Hoong Poo, Multimedia University, Malaysia
Dr. Michael Leung, CEng MIET SMIEEE, Hong Kong
Dr. Abu sahman Bin mohd Supa'at, Universiti Teknologi Malaysia, Malaysia
Prof. Amit Kumar Garg, Deenbandhu Chhotu Ram University of Science & Technology, India
Dr. Jens Myrup Pedersen, Aalborg University, Denmark
Dr. Augustine Ikechi Ukaegbu, KAIST, Korea
Dr. Jamshid Sangirov, KAIST, Korea
Prof. Ahmed Dooguy KORA, Ecole Sup. Multinationale des Telecommunications, Senegal
Dr. Se-Jin Oh, Korea Astronomy & Space Science Institute, Korea
Dr. Rajendra Prasad Mahajan, RGPV Bhopal, India
Dr. Woo-Jin Byun, ETRI, Korea
Dr. Mohammed M. Kadhum, School of Computing, Goodwin Hall, Queen's University , Canada
Prof. Seong Gon Choi, Chungbuk National University, Korea
Prof. Yao-Chung Chang, National Taitung University, Taiwan
Dr. Abdallah Handoura, Engineering school of Gabes - Tunisia, Tunisia
Dr. Gopal Chandra Manna, BSNL, India

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Dr. Il Kwon Cho, National Information Society Agency, Korea
Prof. Jiann-Liang Chen, National Taiwan University of Science and Technology, Taiwan
Prof. Ruay-Shiung Chang, National Dong Hwa University, Taiwan
Dr. Vasaka Visoottiviseth, Mahidol University, Thailand
Prof. Dae-Ki Kang, Dongseo University, Korea
Dr. Yong-Sik Choi, Research Institute, IDLE co., ltd, Korea
Dr. Xuena Peng, Northeastern University, China
Dr. Ming-Shen Jian, National Formosa University, Taiwan
Dr. Soobin Lee, KAIST Institute for IT Convergence, Korea
Prof. Yongpan Liu, Tsinghua University, China
Prof. Chih-Lin HU, National Central University, Taiwan
Prof. Chen-Shie Ho, Oriental Institute of Technology, Taiwan
Dr. Hyoung-Jun Kim, ETRI, Korea
Prof. Bernard Cousin, IRISA/Universite de Rennes 1, France
Prof. Eun-young Lee, Dongduk Woman s University, Korea
Dr. Porkumaran K, NGP institute of technology India, India
Dr. Feng CHENG, Hasso Plattner Institute at University of Potsdam, Germany
Prof. El-Sayed M. El-Alfy, King Fahd University of Petroleum and Minerals, Saudi Arabia
Prof. Lin You, Hangzhou Dianzi Univ, China
Mr. Nicolai Kuntze, Fraunhofer Institute for Secure Information Technology, Germany
Dr. Min-Hong Yun, ETRI, Korea
Dr. Seong Joon Lee, Korea Electrotechnology Research Institute, korea
Dr. Kwihoon Kim, ETRI, Korea
Dr. Jin Woo HONG, Electronics and Telecommunications Research Inst., Korea
Dr. Heeseok Choi, KISTI(Korea Institute of Science and Technology Information), korea
Dr. Somkiat Kitjongthawonkul, Australian Catholic University, St Patrick's Campus, Australia
Dr. Dae Won Kim, ETRI, Korea
Dr. Ho-Jin CHOI, KAIST(Univ), Korea
Dr. Su-Cheng HAW, Multimedia University, Faculty of Information Technology, Malaysia
Dr. Myoung-Jin Kim, Soongsil University, Korea
Dr. Gyu Myoung Lee, Institut Mines-Telecom, Telecom SudParis, France
Dr. Dongkyun Kim, KISTI(Korea Institute of Science and Technology Information), Korea
Prof. Yoonhee Kim, Sookmyung Women s University, Korea
Prof. Li-Der Chou, National Central University, Taiwan
Prof. Young Woong Ko, Hallym University, Korea
Prof. Dimiter G. Velev, UNWE(University of National and World Economy), Bulgaria
Dr. Tadasuke Minagawa, Meiji University, Japan
Prof. Jun-Kyun Choi, KAIST (Univ.), Korea
Dr. Brownson ObaridoaObele, Hyundai Mobis Multimedia R&D Lab , Korea
Prof. Anisha Lal, VIT university, India
Dr. kyeong kang, University of technology sydney, faculty of engineering and IT , Australia
Prof. Chwen-Yea Lin, Tatung Institute of Commerce and Technology, Taiwan
Dr. Ting Peng, Chang'an University, China
Prof. ChaeSoo Kim, Donga University in Korea, Korea
Prof. kirankumar M. joshi, m.s.uni.of baroda, India
Dr. Chin-Feng Lin, National Taiwan Ocean University, Taiwan
Dr. Chang-shin Chung, TTA(Telecommunications Technology Association), Korea

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Dr. Che-Sheng Chiu, Chunghwa Telecom Laboratories, Taiwan
Dr. Chirawat Kotchasarn, RMUTT, Thailand
Dr. Fateme Khalili, K.N.Toosi. University of Technology, IranDr. Izzeldin Ibrahim Mohamed Abdelaziz, Universiti Teknologi Malaysia , Malaysia
Dr. Kamrul Hasan Talukder, Khulna University, Bangladesh
Prof. HwaSung Kim, Kwangwoon University, Korea
Prof. Jongsub Moon, CIST, Korea University, Korea
Prof. Juinn-Horng Deng, Yuan Ze University, Taiwan
Dr. Yen-Wen Lin, National Taichung University, Taiwan
Prof. Junhui Zhao, Beijing Jiaotong University, China
Dr. JaeGwan Kim, SamsungThales co, Korea
Prof. Davar PISHVA, Ph.D., Asia Pacific University, Japan
Ms. Hela Mliki, National School of Engineers of Sfax, Tunisia
Prof. Amirmansour Nabavinejad, Ph.D., Sepahan Institute of Higher Education, Iran

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

Editor Guide
 Introduction for Editor or Reviewer
All the editor group members are to be assigned as a evaluator(editor or reviewer) to submitted journal
papers at the discretion of the Editor-in-Chief. It will be informed by eMail with a Member Login ID and
Password.
Once logined the Website via the Member Login menu in left as a evaluator, you can find out the paper
assigned to you. You can evaluate it there. All the results of the evaluation are supposed to be shown in
the Author Homepage in the real time manner. You can also enter the Author Homepage assigned to
you by the Paper ID and the author's eMail address shown in your Evaluation Webpage. In the Author
Homepage, you can communicate each other efficiently under the peer review policy. Please don't missit!
All the editor group members are supposed to be candidates of a part of the editorial board, dependingon their contribution which comes from history of ICACT TACT as an active evaluator. Because the main
contribution comes from sincere paper reviewing role.

 Role of the Editor
The editor's primary responsibilities are to conduct the peer review process, and check the final cameraready manuscripts for any technical, grammatical or typographical errors.
As a member of the editorial board of the publication, the editor is responsible for ensuring that the
publication maintains the highest quality while adhering to the publication policies and procedures of
the ICACT TACT(Transactions on the Advanced Communications Technology).
For each paper that the editor-in-chief gets assigned, the Secretariat of ICACT Journal will send theeditor an eMail requesting the review process of the paper.
The editor is responsible to make a decision on an "accept", "reject", or "revision" to the Editor-in-Chief
via the Evaluation Webpage that can be shown in the Author Homepage also.

 Deadlines for Regular Review
Editor-in-Chief will assign a evalaution group( a Editor and 2 reviewers) in a week upon receiving acompleted Journal paper submission. Evaluators are given 2 weeks to review the paper. Editors are
given a week to submit a recommendation to the Editor-in-Chief via the evaluation Webpage, once all or
enough of the reviews have come in. In revision case, authors have a maximum of a month to submit
their revised manuscripts. The deadlines for the regular review process are as follows:

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Evalution Procedure

Deadline

Selection of Evaluation Group
Review processing
Editor's recommendation
Final Decision Noticing

1 week2 weeks
1 week
1 week Making Decisions on Manuscript
Editor will make a decision on the disposition of the manuscript, based on remarks of the reviewers. The
editor's recommendation must be well justified and explained in detail. In cases where the revision isrequested, these should be clearly indicated and explained. The editor must then promptly convey thisdecision to the author. The author may contact the editor if instructions regarding amendments to themanuscript are unclear. All these actions could be done via the evaluation system in this Website. The
guidelines of decisions for publication are as follows:

Decision

Description

AcceptAn accept decision means that an editor is accepting the paper withno further modifications. The paper will not be seen again by theeditor or by the reviewers.
The manuscript is not suitable for the ICACT TACT publication.
The paper is conditionally accepted with some requirements. A
revision means that the paper should go back to the original
reviewers for a second round of reviews. We strongly discourage
editors from making a decision based on their own review of themanuscript if a revision had been previously required.

RejectRevision Role of the Reviewer
Reviewer Webpage:
Once logined the Member Login menu in left, you can find out papers assigned to you. You can alsologin the Author Homepage assigned to you with the paper ID and author's eMail address. In there you
can communicate each other via a Communication Channel Box.

Quick Review Required:
You are given 2 weeks for the first round of review and 1 week for the second round of review. You must
agree that time is so important for the rapidly changing IT technologies and applications trend. Pleaserespect the deadline. Authors undoubtedly appreciate your quick review.

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Anonymity:
Do not identify yourself or your organization within the review text.

Review:
Reviewer will perform the paper review based on the main criteria provided below. Please provide
detailed public comments for each criterion, also available to the author.
 How this manuscript advances this field of research and/or contributes something new to theliterature?
 Relevance of this manuscript to the readers of TACT?
 Is the manuscript technically sound?
 Is the paper clearly written and well organized?
 Are all figures and tables appropriately provided and are their resolution good quality?
 Does the introduction state the objectives of the manuscript encouraging the reader to read on?
 Are the references relevant and complete?

Supply missing references:
Please supply any information that you think will be useful to the author in revision for enhancing
quality of the paperor for convincing him/her of the mistakes.

Review Comments:
If you find any already known results related to the manuscript, please give references to earlier papers
which contain these or similar results. If the reasoning is incorrect or ambiguous, please indicate
specifically where and why. If you would like to suggest that the paper be rewritten, give specificsuggestions regarding which parts of the paper should be deleted, added or modified, and pleaseindicate how.

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Journal Procedure
Dear Author,

 You can see all your paper information & progress.
 Step 1. Journal Full Paper Submission
Using the Submit button, submit your journal paper through ICACT Website, then you will get newpaper ID of your journal, and send your journal Paper ID to the Secretariat@icact.org for the review andeditorial processing. Once you got your Journal paper ID, never submit again! Journal Paper/CRF
Template Step 2. Full Paper Review
Using the evaluation system in the ICACT Website, the editor, reviewer and author can communicate
each other for the good quality publication. It may take about 1 month.

 Step 3. Acceptance Notification
It officially informs acceptance, revision, or reject of submitted full paper after the full paper review
process.

Status
Acceptance
Revision
RejectActionGo to next Step.
Re-submit Full Paper within 1 month after Revision Notification.
Drop everything.

 Step 4. Payment Registration
So far it's free of charge in case of the journal promotion paper from the registered ICACT conference
paper! But you have to regist it, because you need your Journal Paper Registration ID for submission of
the final CRF manuscripts in the next step's process. Once you get your Registration ID, send it to
Secretariat@icact.org for further process.

 Step 5. Camera Ready Form (CRF) Manuscripts Submission
After you have received the confirmation notice from secretariat of ICACT, and then you are allowed tosubmit the final CRF manuscripts in PDF file form, the full paper and the Copyright Transfer Agreement.
Journal Paper Template, Copyright Form Template, BioAbstract Template,

Copyright ⓒ 2019 GiRI (Global IT Research Institute)Journal Submission Guide
All the Out-Standing ICACT conference papers have been invited to this "ICACT Transactions on the
Advanced Communications Technology" Journal, and also welcome all the authors whose conference
paper has been accepted by the ICACT Technical Program Committee, if you could extend new contents
at least 30% more than pure content of your conference paper. Journal paper must be followed toensure full compliance with the IEEE Journal Template Form attached on this page.

 How to submit your Journal paper and check the progress?
Step 1. Submit Using the Submit button, submit your journal paper through ICACT

Step 2. Confirm

Step 3. Review

Website, then you will get new paper ID of your journal, and send yourjournal Paper ID to the Secretariat@icact.org for the review and editorial
processing. Once you got your Journal paper ID, never submit again! Using
the Update button, you can change any information of journal paper
related or upload new full journal paper.
Secretariat is supposed to confirm all the necessary conditions of yourjournal paper to make it ready to review. In case of promotion from theconference paper to Journal paper, send us all the .DOC(or Latex) files ofyour ICACT conference paper and journal paper to evaluate the differenceof the pure contents in between at least 30% more to avoid the self
replication violation under scrutiny. The pure content does not include any
reference list, acknowledgement, Appendix and author biography
information.
Upon completing the confirmation, it gets started the review process thru
the Editor & Reviewer Guideline. Whenever you visit the Author
Homepage, you can check the progress status of your paper there fromstart to end like this, " Confirm OK! -> Gets started the review process > ...", in the Review Status column. Please don't miss it!

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

Volume. 8 Issue. 6

6J

1 Robust Semantic Segmentation for Street Fashion Photos

1248

Anh H. Dang*, Wataru Kameyama***GITS, Waseda University, Tokyo, Japan** Faculty of Science and Engineering, Waseda University, Tokyo, Japan

2 Wearable Passive biosensing interface method for gathering bioinformation

1258

Tae-gyu Lee ** Smart Contents Major, Division of ICT Convergence, Pyeongtaek University, 3825 Sedong-daero Pyeongtaek-si,
Gyeonggi,17869, Korea

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

1248

Robust Semantic Segmentationfor Street Fashion Photos
Anh H. Dang∗ , Wataru Kameyama†∗ GITI, Waseda University, Tokyo, Japan† Faculty of Science and Engineering, Waseda University, Tokyo, Japan

anh@aoni.waseda.jp, wataru@waseda.jp

Abstract—In this paper, we aim to produce the state-ofthe-art semantic segmentation for street fashion photos withthree contributions. Firstly, we propose a high-performance
semantic segmentation network that follows the encoder-decoder
structure. Secondly, we propose a guided training process using
multiple auxiliary losses. And thirdly, the 2D max-pooling-based
scaling operation to produce segmentation feature maps forthe aforementioned guided training process. We also propose
mIoU+ metric taking noise into account for better evaluation.
Evaluations with the ModaNet data set show that the proposed
network achieves high benchmark results with less computational
cost compared to ever-proposed methods.
Index Terms—semantic segmentation, street fashion photos,
label pooling, mIoU+

I. I NTRODUCTIONIFFERENT from the classic object detection and classiﬁcation problem, semantic segmentation requires eachpixel in the input image to be assigned to a class of objects.
Fig. 1 shows examples of inputs and corresponding groundtruth labels in semantic segmentation problem for street fashion photos.
Fully convolutional neural network (FCN) for semantic
segmentation [1] has laid the foundation for applying CNN
into dense segmentation. Recently proposed models such as
SegNet [2], DeepLabv3+ [3], and PSPNet [4] have achieved
high benchmark results on data sets such as MSCOCO [5],CityScapes [6], and ADE20K [7].
ModaNet [8] is the ﬁrst large-scale street fashion data set
with pixel-level annotation published by S. Zheng et al.. Thisdata set consists of 55,176 fully annotated images, where
52,377 images are for training and the remaining 2,799 images
are used for validation.
In this paper, we aim to produce the state-of-the-art semantic
segmentation for street fashion photos with three contributions.
First, we propose a lightweight asymmetric network that
follows the encoder-decoder structure. Secondly, we propose
a guided training process with auxiliary training objectives.
And thirdly, the 2D max-pooling-based scaling operation is
proposed to produce labels to be used in one of the auxiliary training objectives. For a better evaluating segmentation

D

Fig. 1. Samples from our custom street fashion data set. In the top row,
original images are shown, and in the bottom row, corresponding segmentation
ground truths are shown. The class names for each color are shown in Table I.
Photos are public domain works downloaded from Pexels.com, and labels aremanually annotated by the authors.

result, we also propose the mIoU+ metric. Different from theconventional mIoU metric, which only counts the classiﬁcation
accuracy of individual pixels, segmentation noise is also taken
into account in mIoU+. By both mIoU and mIoU+, theproposed network achieves the highest benchmark result in
ModaNet while keeping less computational cost comparedwith ever-proposed methods.
The rest of the paper is organized as follows. In Section II,
we introduce previous works on semantic segmentation and theuse of auxiliary loss functions. In Section III, we describe our
network design, auxiliary loss functions that we use to train the
network, including image pyramid loss, segmentation pyramid
loss, and label pooling loss. In Section IV, we describe theexperimental setting, the mIoU+ metric, and the evaluation
result. The paper is concluded in Section V.
II. R ELATED W ORKS
A. Pre-Deep Learning Era

Manuscript received October 19, 2019. This research is supported byfunding from leaftnet Co., Ltd. in Japan.
Anh H. Dang is with GITI, Waseda University, Tokyo, Japan. (corresponding author, phone: +81-80-1367-9637, email: anh@aoni.waseda.jp)
Wataru Kameyama is with Faculty of Science and Engineering, Waseda
University, Tokyo, Japan (email: wataru@waseda.jp)

Semantic segmentation has been a challenge in the ﬁeldof computer vision. Before the deep learning era, the stateof-the-art works have been based on Texton Forest [9] andconditional random ﬁeld (CRF) [10]. CRF is still being used

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019as a post-process method to reﬁne the segmentation output [1],[4], [11]–[14].
B. Fully Convolutional Neural Network
Early works on this topic mostly adopt the straight network
design. The ﬁrst proposed model is FCN [1]. The most important contribution of FCN is converting the fully connectedclassiﬁcation layers of image classiﬁcation networks into a
1 × 1 (i.e., pointwise) convolutional layers to produce pixellevel segmentation prediction. Hence, it can be implemented
on top of the ever-proposed classiﬁcation models such as
GoogLeNet [15], VGG [16], and ResNet [17]. The authors
of FCN have found that their proposal works best by using
VGG-16 as the network base.
PSPNet [4] introduces the spatial pyramid pooling scheme,
which results in better context-awareness in the ﬁnal result.
In this pyramid pooling scheme, features maps from different
layers of the base network are resized and concatenated. The
concatenated feature map is then used as input for a pointwise
CNN to produce segmentation results.
C. Encoder-Decoder Based
Later works on the topic mostly utilize the encoder-decoder
structure. Models following this approach usually yield better
performance. Popular models in this category include SegNet [2] and U-Net [18]. SegNet is a CNN based autoencoder.
It utilizes the indices from 2D max-pooling layers in theencoder to upscale the feature map using unpooling layers
in the decoder. U-Net implements skip connections between
the corresponding encoder and decoder blocks.
D. Dilated Convolutional Neural Network
In [11], F. Yu et al. propose both dilated convolutional
neural network (DCNN) for semantic segmentation and areference network design. DCNN allows the deeper layers of
the network to capture the context without losing resolution.
The main drawback of this design is the great demand forcomputational resources because the feature map is rarely
down-sampled.
DeepLabv3+ [3] combines all of the above approaches andachieves state-of-the-art performance in many benchmarks.
E. Auxiliary Losses
As networks become deeper, new challenges arise. One of
the most challenging problems is the vanishing gradient [19].
In this problem, the gradient becomes too small in the layers
being far away from the training loss function.
At ﬁrst, auxiliary losses are commonly used to overcome
the problem. For example, in GoogLeNet [15], besides themain softmax classiﬁcation loss at the end of the network,
another two similar classiﬁcation losses are added into the
middle of the network. Thus, the weights of early blocks arelearned mostly by gradient propagated from auxiliary losses.
In the research on GANs [20], besides the usual real or
fake discrimination, Chen et al. propose an auxiliary loss todiscriminate the orientation of the input and output pairs to1249

TABLE IDATA S ET S TATISTIC

Id.
0
1
2
3
4
5
6
7
8
9
10
11
12
13

Color

Class

Inst. Count
Train
Val

Background
Bag
Belt
Boots
Footwear
Outer
Dress
Sunglasses
Pants
Top
Shorts
SkirtsHeadwear
Scarf & Tie–
19,60313,081
6,719
37,46822,597
13,764
8,340
21,95033,131
6,709
12,953
5,164
4,711

–
948
636
365
1,753
1,093
662
411
1,064
1,544
322
622
281
284

Avg Inst. Size
Train
Val
–
2.46%
0.46%
2.40%
0.94%
7.43%
10.46%
0.30%
5.65%
4.79%
2.75%
6.37%
1.22%
2.85%

–
2.53%
0.44%
2.36%
0.93%
7.42%
10.52%
0.30%
5.47%
5.04%
2.83%
6.23%
1.21%
3.20%

produce a more robust model. Undoubtedly, selecting the typeof auxiliary objectives and their position greatly inﬂuences theperformance of the network. The auxiliary training objectives
also depict the type of features learned by the network. Thus,
it does not guarantee that the best feature would be learned.
Another solution to the gradient vanishing problem is using
skip connections [21], [22]. In [23], skip connections are used
to patch feature maps from early blocks to deeper blocksof the encoder. In [24], ResBlocks [17] are used to replace
the conventional CNN blocks in both encoder and decoder,resulting in a very deep encoder-decoder based network.
Even though skip connection has become more popular
due to its simplicity, it is not the replacement for auxiliary
loss. Perhaps, they can be complements to each other. In [4],Zhao et al. conduct an ablation study for auxiliary loss on
ResNet [17] based FCN [1]. By adding an auxiliary loss after
the res4b22 residue block and weighted it appropriately, thenetwork performance is gained by 0.94% on pixel accuracy.
In [25], multiple spatially scaled versions of training labels areused as auxiliary training objectives.
III. P ROPOSAL
A. Motivation
1) Problems: Two common problems of semantic segmentation are category confusion and inconspicuous segmentation [4]. Despite efforts to tackle the problems in previous
researches such as [3] and [4], the problems still occur onstreet fashion photos, as shown in Fig. 5 in Appendix A.
In category confusion, the models fail to identify the correctclass of the whole segment. For example, PSPNet fails to
identify the outerwear in the image (n). And with the image(a) and (b), all the models recognize boots as an ordinary
footwear. Another example of this problem is the segmentationof the image (k) by DeepLabv3+. We observe that thisproblem usually happens with networks that have high contextawareness.
When the above-mentioned problem is limited to local areas, it creates inconspicuous segmentation. For example, with

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

Segmentation Pyramid Loss

Input

Image Pyramid Loss

1250

Ground truth

OutputGround truth

A

B1

E0

B3

B2

E1

E2

B4

E3

L3

L4

E4

E5

E6

D6
C6

L0

D5

D4

D3

D2

C5

C4

C3

C2

C1

C0

P2

P1

P0

P5

Constrain

L1

L2

P4

P3

D1

D0

Network Connection
Feature Map
Label Pooling Loss

Fig. 2. Overview of the Proposed Network Structure with all three auxiliary losses. The three auxiliary losses are explained in detail in Section III-D. Groundtruth for Image Pyramid Loss and Segmentation Pyramid Loss are scaled versions of the image and the ground truth segmentation. However, in Label Pooling
Loss, the initial ground truth on the bottom right of the ﬁgure is one-hot vector version of the ground truth segmentation. This one-hot version of segmentationis then progressively scaled-down by P[0..5] . This label pool feature is explained in detail in Section III-C. Moreover, in Label Pooling Lost, constrains ()
are made only between label pool feature maps ( ) and output of decoders ( ). Network connections that are not necessary to generate output
from input
are ignored during inference. The detailed conﬁguration of the whole network is described in Table II.

input image (i), SegNet detects parts of the dress as top-wear
and outer-wear. With image (f), PSPNet frequently confuses
between pants and skirts. Thus, it results in segmentation witha considerable amount of noise.
PSPNet [4] deliberately addresses this problem by proposing the spatial pyramid pooling module (PPM). This module
is expected to increase the size of the receptive ﬁeld of the
network. PPM is also adopted in [3]. However, it appears thatthe receptive ﬁeld is still limited for the street fashion problem.
A possible reason is that the PPM operates on the feature map
produced by a CNN head. Thus, information is already lost
during the process, and the important information may not be
produced simply by pooling the feature map.
2) Direction: We observe that, for street fashion photo,
the above-mentioned problems can be eliminated by knowing
whatever a type of apparels is presented in the whole image.
For example, in the image (d) of Fig. 5, there are only two
types of apparels presented in the image that are dress andpants (a small dark gray area under the model’s left arm as
in the ground truth image). Thus, if a network only considers
dress and pants for the segmentation result, the problem ofclass confusion and inconspicuous would greatly be reduced.
On the one hand, it is uncomplicated to create a separated
model to detect whatever the type of clothes is presented in
an image. On the other hand, it is not efﬁcient to create andtrain separated networks to solve a single problem. Therefore,
we merge two types of networks into one and further extend
the concept of apparel detector to all of the scales.

3) Implementation: Based on the encoder-decoder structure, we ﬁrst set the length of the network so that the feature
map at the end of the decoder is 1 × 1. It is to ensure the highcontext awareness of the network. Secondly, at every scale ofthe decoder, we expect the network to produce a prediction
to indicate whatever the type of clothes is presented in thereceptive ﬁeld of the corresponding pixel of the feature map,i.e., the network ﬁrst detects the presence of apparel type over
the whole image, and then reﬁnes it until reaching the required
resolution. Ground truth for such prediction can be produced
by applying 2D max-pooling on a one-hot vector form of theoriginal ground truth. This process is explained in detail in
Section III-C.
B. Network Structure
Fig. 2 shows the structure of our proposed network. It
comprises two main parts: encoder and decoder. Both theencoder and decoder parts consist of 7 CNN blocks (Ei andDi blocks, where i ∈ [0..6]). Feature map is downscaled every
time it is processed by an encoder block, and correspondingly
upscaled every time it is processed by a decoder block.
We organize this network into 7 different levels based on 7
different scales of the feature maps.
Similar to U-Net, skip connections with identity function
are implemented between encoder and decoder blocks of theas in Fig. 2). However, in oursame level (black arrows
proposal, the feature map produced by an encoder also leaks

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

TABLE IIN ETWORK PARAMETERS
Block

OutputFilters

Kernel

St.a

Pd.b

Ac./Op.cA

224 × 224

32
64
32

5×5
3×3
1×1

1
1
1

2
1
0

ReLu
ReLu
ReLu

B1

112 × 112

64
3

1
1

1
1

B2
B3
B4

56 × 56
28 × 28
14 × 14

–”–
–”–
–”–

3×3
3×3
–”–
–”–
–”–

–”–
–”–
–”–

–”–
–”–
–”–

ReLu
Sigmoid
–”–
–”–
–”–

E0

112 × 1121
1
0

56 × 56
28 × 28
14 × 14
7×7

E5

3×3

E6

1×1

4×4
3×3
1×1
–”–
–”–
–”–
–”–
3×3
3×3
1×1
3×3
1×1
1×1

2
1
1

E1
E2
E3
E4

64
128
64
128
256
512
1024
1024
2048
1024
1024
2048
1024

–”–
–”–
–”–
–”–
3
1
1
1
1
1

–”–
–”–
–”–
–”–
1
1
0
0
0
0

112 × 112224 × 224
112 × 112
56 × 56
28 × 28
14 × 14
3×3
7×7
1×1
3×3

128
1
–”–
–”–
–”–
–”–
128
1
128
1

3×3
2×2
–”–
–”–
–”–
–”–
3×3
3×3
1×1
3×3

1
2

1
0

–”–
–”–
–”–
–”–
1
3
1
3

–”–
–”–
–”–
–”–
1
1
0
0

112 × 112

3×3
3×3
2×2
–”–
–”–
–”–
–”–
3×3
3×3
3×3
1×1
1×1
3×3

1
1
2

1
1
0

3×3

128
14
1
–”–
–”–
–”–
–”–
128
14
1
128
14
1

–”–
–”–
–”–
–”–
1
1
3
1
1
1

–”–
–”–
–”–
–”–
1
1
1
0
0
0

ReLu
Sigmoid
UnPool
–”–
–”–
–”–
–”–
ReLu
Sigmoid
UnPoolReLu
Sigmoid
UnPool

P0
P1
P2
P3
P4
P5

56 × 56
28 × 28
14 × 14
7×7
3×3
1×1

1
–”–
–”–
–”–
–”–
–”–

2×2
–”–
–”–
–”–
3×3
3×3

2
–”–
–”–
–”–
3
1

0
–”–
–”–
–”–
1
1

MaxPool
–”–
–”–
–”–
–”–
–”–

L0

224 × 224

1
1

112 × 112
56 × 56
28 × 28
14 × 14

3×3
3×3
–”–
–”–
–”–
–”–

1
1

L1
L2
L3
L4

128
14
–”–
–”–
–”–
–”–

–”–
–”–
–”–
–”–

–”–
–”–
–”–
–”–

ReLu
Softmax
–”–
–”–
–”–
–”–

C0
C1
C2
C3
C4
C5
C6

D0
D1
D2
D3
D4
D5

224 × 224
112 × 112
56 × 56
28 × 28
14 × 14
3×3
7×7

D6

1×1

a Stride, b Padding, c Activation/Operation

ReLu
ReLu
ReLu
–”–
–”–
–”–
–”–
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
UnPool
–”–
–”–
–”–
–”–
ReLu
UnPool
ReLu
UnPool1251

into the next level of the decoder. To adapt the feature map into
the larger scale, we employ CNN - 2D unpooling blocks Ci .
In our network, encoder blocks scale down the feature map byutilizing CNN with stride 2 instead of using 2D max-pooling
operation. Thus, different from SegNet [2], the 2D unpoolinglayer in our network doesn’t utilize the pooling indices.
As mentioned, in this network, we expect decoder blocks to
produce the prediction on the presence of a class within thewhole image and then gradually reﬁne the prediction result.
Therefore, all the decoder blocks have the same design that
output only 14 channels feature map, which is the number ofsegmentation classes of ModaNet (13 classes plus background,as shown in Table II).
Element-wise sigmoid function is used as the activation
function for Di blocks as follows.
di =1
1 + exp(−d′i )(1)

Where di is the output of Di block, and d′i is the preactivation value of Di .
Segmentation prediction is produced by L0 block. In thisnetwork, besides L0 , there are another 4 Li blocks wherei ∈ [1..4]. These blocks produce smaller-scale versions of thesegmentation prediction. In general, the scale of the prediction
produced by Li block is 2−i . The input of Li block is theconcatenation of feature maps output from Ci , Di and Ei
blocks. Pixel-wise softmax is used to produce the output ofLi blocks as follows.
(′ )exp lij
lij = ∑K(2)′
k=1 exp (lik )Where lij denotes the value of channel j of the feature′
map produced by Li , lijdenotes the pre-activation value oflij , and K denotes the total number of channels which also is
the number of segmentation classes.
From level 1 to level 4, different scales of the input image
are reconstructed by Bi blocks where i is the level number.
The input of Bi block is the feature map ei−1 produced byEi−1 block. All the Bi blocks reconstruct the input at the scaleof 2−i . Thus, all the outputs from Bi blocks create a spatial
scale pyramid of the input image. Element-wise sigmoid, asin (1) is used as the activation function for Bi layers. Thus,
different from works such as [25] and [26], we are not using
the image pyramid as input but as auxiliary training objective.
C. Label Pooling
Previous works involving multiple-scale inputs or outputs
only consider spatial scaling. In [25], they are used as an
auxiliary training objective. In [26], they are used to create
multi-scale fusion features. In [4] and [3], the network is
trained with different spatial scaled versions of input andoutput to produce more robust features.
However, with spatial scaling, details from the original input
eventually are lost at smaller scales. To avoid such problems,
instead of spatially scaling the label, we use max-pooling
operation on the one-hot label vector to produce multiple
scales of labels. As such, the existence of a segment is

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ts
or
sh

to

p

se
s
gl
as
su
n

ot
we
ar
fo

or

p
to

gl
as

ts

se
s

1252

sh

fo

su
n

ot
we
a

ts
or
sh

to

p

r

se
s
gl
as
su
n

ot
we
a
fo

ts
or
sh

to

p

r

se
s

r

gl
as
su
n

ot
we
a
fo

ts
or
sh

p
to

gl
as
su
n

fo

Proposal

ot
we
a

r

se
s

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

Spatial Scaling

Convert to one-hot(16 × 16)

(8 × 8)(4 × 4)(2 × 2)(1 × 1)

Fig. 3. Comparison between proposed 2D max-pooling-based label scaling and conventional label scaling. With conventional label scaling, the label is). In our proposal, the original label (bottom left) is ﬁrst converted to one-hot
progressively scaled-down using nearest-neighbor interpolation (blue arrows
vectors (top left) and then progressively scaled-down by 2D max-pooling operation (red arrows). The segmentation color codes in the label are describedin Table I. On the top row, classes rather than footwear, sunglasses, top, and shorts are ignored.

preserved even in the smallest scale. This process is illustrated
in Fig. 3.
In Fig. 3, the spatial scaling operation makes the existenceof segmentation vanished. After the ﬁrst scale down operation,
the segmentation of sunglass class has vanished. From thescale of 4 × 4 to 2 × 2, the top segment has vanished. By
the time of scaling down to 1 × 1 pixel, all the segmentations
vanished. On the other hand, the proposed label pool features
retain all of the segmentation even at the smallest scale.
Shown in Fig. 2, Di blocks are guide-trained by the resultof Pi blocks where i is the level number, and Pi blocks are 2D
max-pooling operation on the input label. Their conﬁgurationis shown in Table II. This is to avoid the detail loss when
scaling down the label. Also shown in Table II, the stridesof Pi are matched with the strides of Di and Ei blocks.
Furthermore, the kernel size of Pi is also matched with thekernel size of Di .

1 ∑(xij · log (bij ) +
N j=0
N −1

H (xi , bi ) = −(1 − xij ) · log (1 − bij ))(4)

Where xi and bi are input image and reconstructed image
at scale 2−i with i ∈ [1..4], xij and bij are the j-th elementof xi and bi , N is the total number of elements in xi and bi(i.e. number of pixel × number of color channels). We use
binary cross entropy (i.e. log loss) as the error function. Then,the image pyramid loss is calculated by:
1∑
H(xi , bi )4 i=1
4IP L =(5)2) Segmentation Pyramid Loss (SPL): It is the averageof the cross-entropy between segmentation and ground truth
across different scales.

D. Training Objectives

1∑
H(ti , li )4 i=1
4With the segmentation prediction l0 coming from L0 , weutilize pixel-wise cross-entropy as a training objective.
N −1 K−1

1 ∑ ∑H (t, l0 ) =tij log (l0ij )N i=0 j=0(3)

Where t is the ground-truth, tij is the j-th channel of the ith pixel of t, l0ij is the j-th channel of the i-th pixel of l0 , K is
the number of segmentation class, and N is the total number ofpixel in the output. Besides this conventional training criteria,
we introduce the three auxiliary training objectives as follows.
1) Image Pyramid Loss (IPL): Different from popular
works, we do not utilize multi-scaled input to reinforce thetraining process. Instead, we expect the network to reconstructscaled versions of the input using feature maps produced byencoder blocks. Thus, additional scaled inputs and processing
are not required during the inference process. We penalize
the difference between output bi from Bi block and the input
image xi by binary cross-entropy loss as follows.

SP L =(6)

Where H(·) is binary cross-entropy loss similar to (4), ti
and li are ground truth and predicted segmentation at scale
2−i with i ∈ [1..4].
3) Label Pooling Loss (LPL): It is the average of binary
cross-entropy loss between label pool features and output of
decoders across different scales as follows.
1∑
H(pi , di )6 i=1
6LP L =(7)

Where H(·) is binary cross-entropy loss similar to (4), pi
and di are ground truth and prediction of label pool feature atscale 2−i .
The ﬁnal loss is calculated by averaging all the above
mentioned losses as follows:
loss =1(H (t, l0 ) + IP L + SP L + LP L)
4(8)

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

1253

Proposal

PSPNetGround Truth

Input

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Fig. 4. Illustration of segmentation results. Photos are public domain works downloaded from Pexels.com. Label are manually annotated by the authors.

IV. E VALUATION
Using the ModaNet data set, we compare our model with
U-Net [18], PSPNet [4], SegNet [2], and DeepLabv3+ [3].
A. Data Set
We split the original training set into new training andevaluation sets. The new evaluation set consists of 2,400
images, and the new training set consists of the remaining
49,977 images.
The randomized splitting process is constrained so that
there are at least 280 instances of each class available in theevaluation set to ensure the quality of evaluation. The statisticof training and validation data sets are shown in Table I.
B. Data Augmentation
We train and evaluate all the networks with input and output
sizes of 224 × 224. To make the network more robust, thefollowing pipeline is used for data augmentation:1) Random horizontal ﬂipping2) Random expanding with a max expansion ratio of 1.5.
In this step, black bars of random size t, b, l and r are
padded into the original image so that l + r ≤ 0.5 × w
and t+b ≤ 0.5×h, where w and h are width and heightof the input of this step, t and b are the sizes of blackbars padded on the top and bottom of the image, and l
and r are the sizes of black bars padded on the left andright side of the image.
3) Randomly cropping the image with the scale ratio range(0.5, 1] and aspect ratio range [ 43 , 43 ]. Thus, width w andheight h of the cropping window are randomized so that:w ≤ w0 and h ≤ h0w
4
3
4 ≤ h ≤ 3
• 0.5 × (w0 × h0 ) < w × h ≤ w0 × h0
Where w0 and h0 are the width and height of the inputof this step. Hence, the top left corner of the cropping
window (x, y) must satisfy the following conditions:• 0 ≤ x ≤ w0 − w• 0 ≤ y ≤ h0 − h4) Adding Gaussian noise with mean µ = 0 and standard deviation σ = 25.5 . Thus, the output image is
x = min (255, max (0, x + G (µ, σ) ) where G(·) is theGaussian function.
5) Resize to 224 × 224 × 3
We then normalize the input image by scaling pixel value
into [0, 1] range. Since the label is an image containing pixellevel segmentation of the input image, it also needs to be augmented correspondingly, except for the step 4. Furthermore,nearest-neighbor sampling must be used in all the steps thatinvolve interpolation to preserve class information.
••

C. Metrics
1) mIoU: We utilize intersection over union (IoU, i.e.,
Jaccard distance) as the performance metric. We ﬁrst computethe IoU of individual class as follows.
N −1

IoUi =

1 ∑ |Tij ∩ Lij |N j=0 |Tij ∪ Lij |(9)

Where IoUi is the IoU score of class i, N is the total
number of photos in the data set, Tij is the set of all the

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

1254

TABLE IIII O U OF I NDIVIDUAL C LASSES
A

B

C

D

E

F

G

H

Unet

DLv3+aPSPNetSegNetBackground
Bag
Belt
Boots
Footwear
Outer
Dress
Sunglasses
Pants
Top
Shorts
SkirtsHeadwear
Scarf & Tie

0.979
0.690
0.465
0.556
0.630
0.642
0.669
0.634
0.805
0.647
0.686
0.661
0.608
0.393

0.979
0.692
0.462
0.577
0.624
0.644
0.668
0.611
0.810
0.650
0.718
0.674
0.586
0.429

0.978
0.691
0.467
0.570
0.629
0.623
0.633
0.652
0.793
0.625
0.686
0.659
0.606
0.396

0.979
0.694
0.442
0.543
0.628
0.629
0.663
0.606
0.790
0.641
0.697
0.671
0.584
0.409

0.979
0.693
0.456
0.575
0.638
0.639
0.651
0.650
0.802
0.653
0.697
0.646
0.582
0.439

0.980
0.689
0.454
0.557
0.638
0.651
0.657
0.625
0.808
0.660
0.692
0.673
0.590
0.426

0.978
0.707
0.454
0.569
0.631
0.627
0.651
0.646
0.801
0.641
0.688
0.666
0.606
0.397

0.979
0.694
0.440
0.559
0.622
0.638
0.649
0.621
0.792
0.652
0.708
0.652
0.590
0.395

0.979
0.701
0.479
0.561
0.637
0.623
0.594
0.675
0.787
0.624
0.662
0.618
0.618
0.420

0.975
0.674
0.394
0.535
0.586
0.625
0.660
0.531
0.761
0.610
0.715
0.683
0.545
0.370

0.977
0.708
0.415
0.567
0.555
0.665
0.689
0.534
0.800
0.679
0.711
0.709
0.594
0.473

0.955
0.429
0.205
0.369
0.452
0.438
0.462
0.321
0.683
0.478
0.487
0.503
0.258
0.129

mIoU
inference time (ms)training time (h)0.648
100.9619.5680.652
78.99
19.3280.643
68.81
17.8250.641
68.80
17.6250.650
75.11
19.3590.650
75.23
19.2970.647
66.16
17.831

0.642
66.05
17.4830.641
116.2430.84

0.619
137.8333.64

0.648
115.98
23.68

0.441
29.61
9.04

a DeepLabv3+

pixels belongs to the i-th class in j-th ground truth, Lij is theset of all pixels predicted as i-th class in the j-th prediction,and | · | is the cardinality of a set. Thus, the mIoU metric is
calculated as follows.

Where SSi (U, V ) is the segmentation score of the i-thclass between two segmentations U and V . Similar to theconventional mIoU, IoU+ of each class is computed as follows.
N −1

IoU +i =M −1

1 ∑mIoU =
IoUi
M i=0

(10)

Where M is the total number of segmentation classes.
2) mIoU+: Because the mIoU metric favors the total number of accurately classiﬁed pixels, a prediction with noise frequently results in higher mIoU compared to a prediction withno noise. Depending on the application, prediction with low
noise may be favored over absolutely high mIoU prediction.
Therefore, we propose mIoU+ (mIoU-plus) metric in which
noise is taken into account. This metric is not based on
individual pixel but connected components (i.e. individual segments). Given a prediction and a ground truth segmentation,
the connected component based segmentation score of a classis calculated as follows.
CCSSi (U, V ) =1 ∑max IoU (u, v)
v∈Vi
|Ui |

(11)

u∈Ui

Where CCSSi is the segmentation score of the i-th class
between predicted segmentation U and ground truth V , Ui
is the set of all connected components of i-th class in theprediction, Vi is the set of all connected components of ith class in the ground truth. However, because this scoreis not symmetric (i.e., CCSSi (U, V ) ̸= CCSSi (V, U )), thesegmentation score is calculated as follows.
SSi (U, V ) = CCSSi (U, V ) ∧ CCSSi (V, U )

(12)

1 ∑
SSi (Uj , Vj )
N j=0

(13)

Where IoU +i is IoU+ score of the i-th class, N is the total
number of samples, Uj is the set of connected components
from the j-th prediction, and Vj is the set of connectedcomponents from the j-th ground truth. The score for awhole segmentation with multiple connected components is
calculated as follows.
1 ∑SSi (U, V )
K i=0
K−1

mIoU + =(14)

Where K is the total number of segmentation classes.
D. Ablation Study on Effect of Auxiliary Training Objectives
We investigate more into the effect of auxiliary training objective on the model performance. We retrain our network withdifferent training objective conﬁgurations. The loss functionused in this experiment is as follows.
loss =

H (t0 , l0 ) + αIP L + βSP L + γLP L
1+α+β+γ(15)

Where α, β, γ ∈ {0, 1}. In practice, when α = 0, bi
computations are ignored. Similarly, when β = 0, l[1..4]
computations are ignored. However, when γ = 0, di still need
to be computed because it is an integrated part of the model.
There are 8 different conﬁgurations of the loss function. We
annotate them as conﬁguration A to conﬁguration H, as shownin Table V.

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

1255

TABLE IVI O U+ OF I NDIVIDUAL C LASSES
A

B

C

D

E

F

G

H

Unet

DLv3+aPSPNetSegNetBackground
Bag
Belt
Boots
Footwear
Outer
Dress
Sunglasses
Pants
Top
Shorts
SkirtsHeadwear
Scarf & Tie

0.373
0.441
0.290
0.347
0.502
0.412
0.425
0.501
0.629
0.432
0.488
0.459
0.479
0.278

0.344
0.398
0.257
0.303
0.474
0.379
0.393
0.471
0.613
0.439
0.426
0.413
0.399
0.230

0.369
0.434
0.312
0.358
0.503
0.383
0.370
0.541
0.642
0.438
0.490
0.438
0.468
0.239

0.336
0.361
0.245
0.269
0.483
0.356
0.336
0.433
0.568
0.394
0.393
0.370
0.362
0.185

0.359
0.429
0.316
0.335
0.503
0.409
0.398
0.531
0.635
0.452
0.527
0.465
0.452
0.270

0.347
0.392
0.257
0.297
0.489
0.375
0.382
0.463
0.621
0.428
0.454
0.426
0.379
0.186

0.339
0.424
0.297
0.360
0.504
0.408
0.393
0.514
0.623
0.401
0.449
0.454
0.467
0.238

0.338
0.385
0.240
0.280
0.475
0.329
0.360
0.402
0.567
0.391
0.374
0.390
0.310
0.172

0.345
0.373
0.287
0.297
0.500
0.286
0.208
0.502
0.557
0.347
0.364
0.297
0.441
0.184

0.379
0.364
0.204
0.299
0.434
0.381
0.366
0.355
0.591
0.406
0.453
0.436
0.336
0.211

0.404
0.427
0.211
0.301
0.418
0.425
0.434
0.392
0.641
0.465
0.497
0.497
0.400
0.263

0.233
0.124
0.082
0.100
0.247
0.104
0.082
0.157
0.316
0.173
0.203
0.163
0.083
0.049

mIoU+
inference time (ms)training time (h)0.433
100.9619.5680.396
78.99
19.3280.428
68.81
17.8250.363
68.80
17.6250.434
75.11
19.3590.392
75.23
19.2970.419
66.16
17.8310.358
66.05
17.4830.356
116.2430.84

0.373
137.8333.64

0.412
115.98
23.68

0.151
29.61
9.04

a DeepLabv3+

E. Settings
We implement all the networks using Chainer deep learning
framework [27]. LeCun normal weight initializer [28] is used.
The models are trained by Adam optimizer [29] with thelearning rate of 1 × 10−3 and the decay rate of 0.99. The
machine used to carry out the experiment is a Linux box
equipped with three Nvidia Pascal GPUs.
We train each conﬁguration for three times with 100 epochs
each and take averages of the best mIoU and mIoU+.
F. Result
We report the experimental results using mIoU metricin Table III. Our proposed network conﬁgured with threedifferent auxiliary losses outperforms all the ever-proposedmodels in terms of performance. Among all the auxiliary
loss conﬁgurations, conﬁguration B achieves the highest mIoU
score. This conﬁguration consists of only image pyramid loss
and segmentation pyramid loss. Among the ever-proposed
networks, PSPNet achieves the highest mIoU score. Moreover,
our top proposed network takes only 2/3 of the time fortraining as well as inference compared to PSPNet.
We realize that the mIoU performance of the model isworsened when combining label pooling loss with the othertwo auxiliary training losses. In fact, conﬁguration B, D, andTABLE VD IFFERENT AUXILIARY C ONFIGURATIONS

α
β
γ

A

B

C

D

E

F

G

H

1
1
1

1
1
0

1
0
1

1
0
0

0
1
1

0
1
0

0
0
1

0
0
0

F achieve higher mIoU compared to conﬁguration A, C, andE.
We report experimental results using mIoU+ metric in
Table IV. The training and inference time in Table IV are
carried over from Table III. We observe that all the models
achieve their best mIoU and mIoU+ in the same epoch.
Furthermore, mIoU and mIoU+ are loosely proportional toeach other during the training process.
As shown in Table IV, the proposed model with conﬁguration E achieves the highest mIoU+ score. Conﬁguration
E consists of segmentation pyramid loss and label poolingloss. Conﬁguration A with all the auxiliary loss functions is
the runner-up. Among the ever-proposed model, PSPNet alsoachieves the highest mIoU+ score.
The proposed model used to generate samples in Fig. 4 andFig. 5 is trained with conﬁguration A.
V. C ONCLUSION
In this paper, we propose a high-performance semantic
segmentation model for street fashion photos. This network
infers the existence of the classes over the whole image, andprogressively reﬁnes the result up to the desired resolution.
We also propose a new label pool feature that can be used
to improve the performance of the proposed network. And
ﬁnally, we provide benchmarking results of the network withand without 3 different types of auxiliary loss. For better
evaluation, we propose mIoU+ metric in which noises aretaken into account.
We compare the performance of our network to the otherstate-of-the-art networks, including U-Net [18], PSPNet [4],SegNet [2], and DeepLabv3+ [3]. We report the evaluation result using both conventional mIoU and newly proposed mIoU+
metrics. The experiment shows that our network requires lesstime to train and infer while achieves the highest segmentation
performance in both mIoU and mIoU+ metrics.

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

For future work, we will extend the evaluation on thescene parsing problem using data sets such as MSCOCO [5],CityScapes [6], and ADE20K [7].
R EFERENCES[1] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proceedings of the IEEE conference on
computer vision and pattern recognition, 2015, pp. 3431–3440.
[2] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE
transactions on pattern analysis and machine intelligence, vol. 39,
no. 12, pp. 2481–2495, 2017.
[3] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoderdecoder with atrous separable convolution for semantic image segmentation,” in Proceedings of the European Conference on Computer Vision(ECCV), 2018, pp. 801–818.
[4] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing
network,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2017, pp. 2881–2890.
[5] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects incontext,” in European conference on computer vision. Springer, 2014,
pp. 740–755.
[6] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for
semantic urban scene understanding,” in Proc. of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2016.
[7] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba,“Scene parsing through ade20k dataset,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2017.
[8] S. Zheng, F. Yang, M. H. Kiapour, and R. Piramuthu, “Modanet: A largescale street fashion dataset with polygon annotations,” arXiv preprint
arXiv:1807.01394, 2018.
[9] J. Shotton, M. Johnson, and R. Cipolla, “Semantic texton forests forimage categorization and segmentation,” in 2008 IEEE Conference on
Computer Vision and Pattern Recognition. IEEE, 2008, pp. 1–8.
[10] J. Shotton, A. W. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio,
R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognitionin parts from single depth images.” in Cvpr, vol. 2, 2011, p. 3.
[11] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated
convolutions,” arXiv preprint arXiv:1511.07122, 2015.
[12] G. Lin, A. Milan, C. Shen, and I. Reid, “Reﬁnenet: Multi-path reﬁnement
networks for high-resolution semantic segmentation,” in Proceedings ofthe IEEE conference on computer vision and pattern recognition, 2017,
pp. 1925–1934.
[13] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille,“Semantic image segmentation with deep convolutional nets and fully
connected crfs,” arXiv preprint arXiv:1412.7062, 2014.
[14] ——, “Deeplab: Semantic image segmentation with deep convolutional
nets, atrous convolution, and fully connected crfs,” IEEE transactions on
pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848,
2017.
[15] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”in Proceedings of the IEEE conference on computer vision and patternrecognition, 2015, pp. 1–9.
[16] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778.
[18] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in International Conference on
Medical image computing and computer-assisted intervention. Springer,
2015, pp. 234–241.
[19] S. Hochreiter, “The vanishing gradient problem during learning recurrent
neural nets and problem solutions,” International Journal of Uncertainty,
Fuzziness and Knowledge-Based Systems, vol. 6, no. 02, pp. 107–116,
1998.
[20] T. Chen, X. Zhai, M. Ritter, M. Lucic, and N. Houlsby, “Self-supervisedgans via auxiliary rotation loss,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2019, pp. 12 154–12 163.
[21] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual
networks,” in European conference on computer vision. Springer, 2016,
pp. 630–645.

1256[22] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely
connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
[23] M. Drozdzal, E. Vorontsov, G. Chartrand, S. Kadoury, and C. Pal, “Theimportance of skip connections in biomedical image segmentation,” inDeep Learning and Data Labeling for Medical Applications. Springer,
2016, pp. 179–187.
[24] S. Jégou, M. Drozdzal, D. Vazquez, A. Romero, and Y. Bengio, “Theone hundred layers tiramisu: Fully convolutional densenets for semantic
segmentation,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition Workshops, 2017, pp. 11–19.
[25] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia, “Icnet for real-time
semantic segmentation on high-resolution images,” in Proceedings ofthe European Conference on Computer Vision (ECCV), 2018, pp. 405–
420.
[26] J. Li, W. Speier, K. C. Ho, K. V. Sarma, A. Gertych, B. S. Knudsen, andC. W. Arnold, “An em-based semi-supervised deep learning approachfor semantic segmentation of histopathological images from radical
prostatectomies,” Computerized Medical Imaging and Graphics, vol. 69,
pp. 125–133, 2018.
[27] S. Tokui, K. Oono, S. Hido, and J. Clayton, “Chainer: a next-generation
open source framework for deep learning,” in Proceedings of workshop
on machine learning systems (LearningSys) in the twenty-ninth annual
conference on neural information processing systems (NIPS), vol. 5,
2015, pp. 1–6.
[28] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efﬁcient
backprop,” in Neural networks: Tricks of the trade. Springer, 2012,
pp. 9–48.
[29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.

Anh H. Dang (S’09) received his bachelor degree
in business administration, information & communication technology from Ritsumeikan Asia Paciﬁc
University (Beppu, Oita, Japan) in 2010. He thenreceived the master degree in computer sciencefrom Waseda University (Shinjuku, Tokyo, Japan) in
2012. Since 2012, he is a Ph.D. candidate at Waseda
University. He is a member of IEEE, ACM, and
IEICE. His research interests are machine learning,
artiﬁcial intelligence, and computer vision.

Prof. Wataru Kameyama (M’86) received thebachelor’s, master’s, and D.Eng. degrees from the
School of Science and Engineering, Waseda University, in 1985, 1987, and 1990, respectively. He joined
ASCII Corporation in 1992, and was transferred toFrance Telecom CCETT from 1994 to 1996 forhis secondment. After joining Waseda University
as an Associate Professor in 1999, he has been a
Professor with the Department of Communications
and Computer Engineering, School of Fundamental
Science and Engineering, Waseda University, since2014. He has been involved in MPEG, MHEG, DAVIC, and the TV-Anytime
Forum activities. He was a Chairman of ISO/IEC JTC1/SC29/WG12, and a
Secretariat and Vice Chairman of the TV-Anytime Forum. He is a member ofIEEE, IEICE, IPSJ, ITE, IIEEJ, and ACM. He received the Best Paper Awardof Niwa-Takayanagi in 2006, the Best Author Award of Niwa-Takayanagi in2009 from the Institute of Image Information and Television Engineers, andthe International Cooperation Award from the ITU Association of Japan in
2012.

A PPENDIX AS EGMENTATION R ESULTS

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Fig. 5. Illustration of segmentation results coming from the models. Photos are public domain works downloaded from Pexels.com. Label are manually annotated by the authors.

(Model A)

Proposal

DeepLabv3+

U-Net

SegNetPSPNetGround Truth

Input

(a)

(m)

(n)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019
1257

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

1258

Wearable Passive biosensing interface method
for gathering bioinformation
Tae-gyu Lee ** Smart Contents Major, Division of ICT Convergence, Pyeongtaek University, 3825 Sedong-daero Pyeongtaek-si,
Gyeonggi,17869, Korea
tglee@ptu.ac.kr
Abstract— Recently, a variety of biometric information systems
have been tried and developed to provide IoT-based healthcare
and medical information services. Existing biometric
information systems actively collect and analyze biometric
information around user terminals or smart devices. On thecontrary, this study proposes a passive biosensing system and apassive sensing method for collecting and analyzing biometric
information using wearable passive patches. In particular, apassive sensing system and method using a passive patch thatcan be detected by activation of an external power source
without an internal battery is proposed. This paper focuses on
the following elements of passive biometric information system.
The first describes the structure and components of a passive
patch system that can be operated from an external source
without an internal battery. The second describes passive
biometric information algorithms that passively detectbiometric information in real time. Finally, we propose apassive sensing analysis modeling that analyzes and evaluates
the signal sensitivity of biometric information, and describes ananalysis example of bioinformation sensitivity.
Keyword—Passive sensing, Patch interface, IoT, Wearable
system, Bioinformation

I. INTRODUCTIONW

ITH the development of various wearable systems, themethods for monitoring a human body, analyzing a
bio-signal, evaluating a health condition of a user, andproviding a healthcare service based on the evaluation havebeen continuously attempted. These efforts have had somefruitful consequences, and have become a future vision
strategy that is important for the healthcare sector. This paper
describes a sensing system and a sensing method based on
bio-information collection interface using passive patches.
Especially, it is a research on sensing system and sensing
method based on information gathering interface using
passive patch which can be sensed by the activation of
external power without internal battery [1].
When a drug is administered to treat a patient, it is largely
divided into oral administration and parenteral administration.
In this case, the parenteral administration may be divided into———————————————————————
Manuscript received on June 21, 2019. This work is a follow-up theinvited journal to the accepted & presented paper of the 20th International
Conference on Advanced Communication Technology (ICACT2018), andGrant ID is 2020001.
T. L. Author is with the Division of ICT Convergence (Smart Contents
Major), Pyeongtaek University, Pyeongtaek, Gyeonggi 17869 Korea(corresponding author to provide phone: +82-31-659-8370; fax:
+82-31-659-8011; e-mail: tglee@ptu.ac.kr).

mucosal administration methods such as ocular mucosal
administration,
nasal
administration,
sublingual
administration and rectal administration, and injection
administration methods such as intravenous injection andsubcutaneous injection, and skin administration method [2-6].
Among them, the method of administering the drug throughthe skin is as shown in Fig. 1, using a paste, a patch, or a
wetting agent.
Until now, digital patches have been used as means andmethods for collecting human body information. These
efforts contributed to the collection of various biometric
information such as body temperature, sweat, humidity,
respiration, pulse, and heart rate. Nevertheless, the types of
sensors available for patching are limited and the amount of
information that can be collected is limited. These attempts
should continue to develop new sensors, patch structures, andbio-signal analysis.
This paper considers the following issues in order torealize a passive patch system that maximizes applicability.
First, there is a need for an alternative to overcome the mobile
support and user convenience constraints of a typical powerbuilt-in active sensing patch. Next, a power supply
alternative is needed with a no-power biometric information
system configuration for semi-permanent product and service
configuration. Finally, by providing a consumable patch
sensing module according to the no-power patch system
configuration, the patch production cost is minimized tomaximize the applicability.
The subjects of this study are as follows [7]. The first is toprovide an organization of a passive patch system based onan bioinformation gathering interface that can operate withexternal power without an internal battery. The second is toprovide a passive patch based on an information collection
interface that can sense the degree of action of the applieddrug and display it to the patient or the user. The third is toprovide a passive patch based on an information gatheringinterface that can inform the user or the patient when the
degree of pharmacological action is out of the threshold andcan lead the user or the patient to remove the patch attached.
Finally, this study is to provide a sensing system and asensing method using a passive patch based on an
information collection interface capable of acquiring
biometric information and providing it to an external
network.
This study aims to classify and define biometric
information sensing method in systematically developing andimplementing biometric information sensing and biometric
information system [8] [9] [10].
Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

First, there is an active bio-sensing method for sensing andcollecting biometric information of a target object inaccordance with its own bio-logic execution while
continuously supplying power and electric power to thesensing module.
Second, the sensing module can be configured with
passive sensing logic without its own built-in power orwithout external power supply. In order to sense and collectbiometric information of a target object, a sensing module istransferred to a wireless local power and activation (orsynchronization) signal, and necessary biometric informationis sensed through a sensing circuit and a wirelesstransmission circuit module that are passively operated It is apassive bio-sensing method to collect.
This study applies passive bio-sensing method without
built-in power in the patch module as bio-information
sensing module and system configuration method. The main
issues to build processes and systems for collecting, storing,and processing biometric information are as follows. An
active agent and a wireless interworking interface
configuration for activating the passive sensing module areneeded. In particular, consider how to design the signal
activation, the available frequency band and the signal
strength. As a layered structure of the passive sensing module,
process support such as activation signal reception,individual bio-information sensing, biometric informationtransmission and power signal amplification are important
issues. Finally, the real-time signal analysis capability of theactive agent and the non-real-time bio-information
application data model of the background server are
recognized as major issues.
The application fields of biometric information technology
and system of this study are as follows. Currently, it is activein the medical and healthcare fields that utilize mobile
platforms such as smart phones and smart clocks [11]. It has
been steadily expanding in the fields of sports and clothing[12], beauty sector [13], agricultural biology sector [14],automobile and transportation sector [15] and security
certification sector [16].

1259patch can measure the degree of action of the drug and induce
removal of the patch when out of the threshold range.
Additionally, it can transmit the biometric information of thepatient to the outside and transmit the action information of
the internal medicine of the patch used for the treatment to theoutside, so that the transmitted data can be easily browsed.
Accordingly, the patient, the caregiver or physician canconveniently check the patient status, and can transmit apatch removal message or the like when the medicinal
threshold of the passive patch attached for the patient
treatment is out of the range. Finally, the sensing systemusing the passive patch based on the proposed
bio-information collecting interface receives the biometric
information transmitted from the passive patch and transmits
the biometric information to the registered communication
terminal when the biometric information is out of thethreshold range.
II. ARCHITECTURE OF PASSIVE SENSING SYSTEM
A. Component and functions
In this study, a passive patch attached to a patient andmeasuring the biometric information signal of a patient andtransmitting it to the outside, and an active terminal
connected to the passive patch, transmits an activation signal
to the passive patch, and receives the biometric information
signal as shown in Fig. 2 [7]. This passive patch constitutes
an information collecting interface based passive patch
sensing system characterized in that external power is
temporarily supplied without operation of an internal battery.

Fig. 2. The Wearable Bioinformation Computing Systemwith a Passive Patch Interface

Fig. 1. A Computing Domain Configuration for Wearable
Passive Patch Bioinformation System
This study has the following contributions [7]. The passive
patch based on the proposed biometric information collection
interface can operate with external power without built-in
battery, it is easy to miniaturize, shorten manufacturing
process time, and reduce production cost. And, such a passive

In addition, the expansion system may further include anexternal server connected to the active terminal in a wirelessor wired manner, and the external server may include apassive information collection interface-based patch sensing
system capable of receiving a biometric information signal ofa patient from an active terminal acting as a mobile agent.
Here, as shown in Fig. 3, the passive patch constitutes
several layers of partial modules from the bottom to the top as
follows. The first lower layer constitutes the drug application
layer on which the drug is formed on the lower surface whichis in close contact with the skin of the patient. The second
layer is a supporting module formed on the upper part of thelower drug coating layer, and includes a biosensor module formeasuring a biometric information signal of a patient and adrug action detection sensor module for generating a drug
action signal by measuring the degree of the drug acting on
the patient do. And a power supply module for operating thebuilt-in circuit by receiving power from the outside. The thirdlayer includes a display module formed on the upper part and
Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 20191260

a control module for generating a patch removal inducing
signal corresponding to the drug operation signal when the
drug action signal is out of a predetermined threshold value
range and transmitting the signal to the display module.
The sensing data transmission structure of the passive
patch system is as follows. First, the active terminal performs
an activation process to send an activation signal to thepassive patch attached to the patient. Second, a passive patchthat transmits an activation signal from the active terminal
proceeds with a biological information sensing step ofsensing drug and human body information. Third, the passive
patch consists of a feedback step of feeding the sensed drug
and human body information to the active terminal. Finally,
the active terminal constructs a storage process for storing thesensed drug and human body information in the internal
memory. At this time, the passive patch can be operated bytemporarily supplying external power without an internal
battery.
Fig. 4. Data Flow Processes on Bioinformation System
In addition, the information collected in the smart agent
terminal is transmitted to a PC or a server located in the
background, continuously analyzed and evaluated, andupdates various bio-information.
III. ALGORITHM FOR PASSIVE SENSING PROCESSES

Fig. 3. The Layered Architecture and Components ofPassive Patch on the Client-side
Further, the drug information and the human bodyinformation stored in the active terminal can be further
expanded to an external transfer process for transferring andstoring the drug and the human body information to a
database of the external server.
B. Data and control flows
Data flow from data collection to data analysis andevaluation service of bio-information sensing system isshown in Fig. 4.
The data flow of the passive biometric information systemstarts data input and sensing at the passive object by theactivation signal of the active agent.
Then, data processing such as data conversion, datatransmission and storage based on data compression,
real-time data pattern and semantic analysis, non-real-timebackground data analysis, and data generation of biometric
information application services are performed.
The passive-based sensing data flow mainly collects asmall amount of data on the smart agent on a non-periodic
basis, and provides a real-time alarm and a status notification
service of the biometric information to the user or theterminal according to the occurrence of the event.

The detailed components for implementing the passive
patch biometric information system include a drug action
detection sensing module, a biosensor module, a power
supply module, a control module, a communication module,
and a display module as follows.
The drug action detection sensing module can detect the
drug application layer, or the concentration of the drug
detected in the skin of the patient. Further, the drug action
detection sensing module may generate a drug action signal
through the detected signal and transmit the drug action
signal to the control module. Here, the drug action detection
sensing module may be formed through at least one probe tocontact the drug application layer or the skin of the patient.
The biosensor module can measure a bio-information
signal including at least one of body temperature, pulse,
blood sugar, skin humidity, electrocardiogram signal, andbrain wave. In addition, since the biometric information
signal measured according to the attachment position of thepassive patch differs, the bio-sensing module can select a
different type of sensor depending on the attachment
position.
The power supply module may be formed to be able toreceive power temporarily from the outside. To this end, thepower supply module may configure a USB terminal or
configure an electric wire or a pin to receive external power.
At this time, the power supply module may be connected tothe built-in circuit formed in the control module. That is, thepower supply module can receive the power temporarily
from the outside to operate the built-in circuit, whereby thepassive patch can be operated. Meanwhile, the power supply
module may receive power wirelessly. At this time, thewireless power supply may be an electromagnetic induction
type, the power supply module may constitute a primary coil,and electromagnetic can be guided through a wireless supply
device having a secondary coil to supply power to an internal
Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

circuit.
The control module may receive the drug action signal
input from the drug action detection sensing module andcompare the received drug action signal with a pre-set
threshold value. Also, the control module may generate apatch removal inducing signal and display the patch removal
inducing signal on the display module when the value
included in the drug action signal is out of the threshold value
range. For example, if the concentration of the drug containedin the received drug action signal is lower than the thresholdvalue, the control module can generate a notification that thepatient can remove the passive patch, that is, a patch removal
inducing signal because the drug treatment effect is low. In
addition, the control module can generate a patch removal
inducing signal so that the patient removes the passive patch
because the drug may cause side effects when theconcentration of the drug contained in the received drug
action signal is higher than the threshold value.
The communication module may transmit a biometric
information signal or a patch removal inducing signal or adrug action signal to the outside of the passive patch. Further,
the communication module may include wirelesscommunication means capable of communicating at a shortdistance, such as Bluetooth. For example, the communication
module can transmit a biometric information signal or a patch
removal inducing signal or a pharmacological action signal toa nearby electronic communication terminal (for example, asmart phone) in a Bluetooth manner. At this time, thecommunication module may use a module to which the
ZigBee communication method or the NFC (Near Field
Communication) communication method other than the blue
pitching method is applied.
The display module may be formed on the upper surface todisplay a signal informing of the removal of the patch. The
display module also receives the patch removal induction
signal and displays the set colour or message to inform thepatient or user of the patch removal.
As described above, the passive patch described in thisstudy can be powered by external power supply without
built-in battery.
In addition, biometric information of the patient can betransmitted to the outside, and action information of theinternal medicine of the patch used for the treatment can betransmitted to the outside, so that the transmitted data can beeasily browsed. In this study, an active terminal as an agent
terminal can use a WIFI or Bluetooth network or the like, andthus can connect to an external server using such a
communication network.
In addition, the active terminal and the external server canstore the subscribed patient information. At this time, thepatient information may be stored through one of the active
terminal or the external server and shared with each other. In
other words, when stored in the active terminal, the stored
information is transmitted to the external server and stored in
the external server, transferred from the external server to theactive terminal upon storage, and transmitted to the active
terminal.
Here, the reference health information may be reference
health information classified according to the patient's age,sex, body shape or the like or pre-set reference healthinformation (for example, blood pressure, pulse, blood1261

glucose level, etc.).
This standard health information can be used by backing
up the data stored in the existing hospital database for the
recent health condition information and medical history
information. If necessary, the patient can receive the datafrom the treated hospitals and store it on the active terminal or
the external server is.
This is an extended feature of the sensing system using thepassive patch based on the information gathering interface sothat it is possible to manage the health and medical
information according to each user, thereby enabling more
efficient additional service support.
The proposed passive biometric information sensing
system is divided into two parts. One is the passive patch
portion of Fig. 5, and the other is the mobile active agent
portion shown in Fig. 6. The sensing module of the passive
patch waits for the activation signal from the mobile active
agent, and the mobile active agent transmits the activation
signal to the passive patch whenever biometric informationcollection is required. The details of each process are shown
below.

Fig. 5. The Embedded Logic Sequences of Passive Patch
Fig. 5 is a flow chart showing a passive sensing method
using passive patches based on the bio-information collection
interface. The bio-sensing process using the passive patch is
as follows. First, a process in which the active terminal of aclient transmits an activation signal to a passive patch
attached to the patient. Then, a passive patch that receives anactivation signal from an active terminal and wakeups itself
senses drug or human body bio-information such as bodytemperature, pulse, respiration, etc. At this time, if thebio-sensing information exceeds the threshold or an
emergency occurs, the alert message is notified to the patch
display interface. Third, a reply process returns the sensed
drug and human body information to the active terminal. At

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

this time, feedback control can be performed on the passive
patch if necessary. Finally, the active terminal carries out aprocess of receiving the sensed drug and the human bodyinformation, and storing it therein.
Fig. 6 shows the processes of collecting sensing
information received from a user mobile terminal andtransmitting information to an external server as a mobile
agent.

Fig. 6. The Active Agent Logic Sequences of Mobile
Terminal
In Fig. 6, the user terminal initially activates the passive
patch with an active signal. Next, it waits for receiving thebioinformation and the user's biometric information from thepassive patch. If a reply message is detected, it stores thereceived bio-information in its storage space, and displays the
bio-information state of passive patch at real-time. Then, if
there is an exceptional state, it can show the alarm services on
the mobile terminal. Finally, when the external DB server isexisted and interconnected, the information collected throughthe external communication channel can be relayed andconstructed as a database.
IV. ANALYSIS MODEL OF PASSIVE SENSING
This study first proposed a passive bioinformatics system.
And, this configuration can be a good alternative to overcome
the power consumption limitation of the digital patch. In
addition, the cost of producing a digital patch is minimized,
thereby maximizing the economic efficiency of the product.
Ultimately, we will expect the availability of passive digital
patches to vary.

1262

A. Experimental environments
Passive RFID systems use tags with no internal powersource and instead are powered by the electromagnetic
energy transmitted from an RFID reader. The wirelesscommunication configuration of the passive sensing datatransmission system for passive patch sensing is as follows.
As the name implies, passive tags wait for a signal from an
RFID reader. The reader sends energy to an antenna whichconverts that energy into an RF wave that is sent into the readzone. Once the tag is read within the read zone, the RFID
tag’s internal antenna draws in energy from the RF waves.
The energy moves from the tag’s antenna to the IC andpowers the chip which generates a signal back to the RF
system. This is called backscatter. The backscatter, or change
in the electromagnetic or RF wave, is detected by the reader(with the antenna), which interprets the information.
Combining sensor monitoring with RFID allows for theobservation of uniquely identifiable items at short and longranges. Whether using battery power or collected RF energy,
sensors can collect relevant data pertaining to temperature,humidity, moisture, and motion/movement of human body.
With advancements in technology, especially in energy
collection and impedance, sensor monitoring can now be
accomplished using passive RFID tags without a decrease inread range. Recently, manufacturers have figured out a way
to incorporate sensors into passive RFID tags without greatly
affecting read range. Because passive tags do not have aninternal battery to power the sensors, passive tags use RFenergy to power the IC/sensor and to send the relevant
information back to the reader. Currently, passive RFID
sensor tags can be used to detect body states such astemperature and humidity or moisture because of a newly
wearable integrated circuit (IC). The IC is designed with abank of capacitors that is able to detect the presence ofmoisture based on how the moisture affects the tuning of theantenna. That information is then backscattered back to the
RFID reader for observation.
RFID systems, readers and tags communicate mostly
through the method of electromagnetic coupling. In order for
an RFID tag to communicate with an RFID reader/antenna,
the tag circuit and reader circuit generally must couple insome way. Coupling is a transfer of energy between twoelectronic items or two circuits. For example, systems thatuse capacitive coupling use electric currents instead of themagnetic field in order to couple. Contact to a few
centimetres of read range is normal for (LF) or Low
Frequency communication because of the need to produce anelectric field using electrodes.
Passive RFID tags do not all operate at the same frequency.
There are three main frequencies within which passive RFID
tags operate. The frequency range, along with other factors,
strongly determines the read range, attachment materials, andapplication options.
As the first frequency model, Low Frequency (LF, 125~134 KHz) has an extremely long wavelength with usually ashort read range of about 1~10 centimetres. This frequency is
typically used with animal tracking because it is not affectedmuch by water or metal.
The second frequency model with High Frequency (HF,
13.56MHz) & Near-Field Communication (NFC) has amedium wavelength with a typical read range of about 1
centimetre up to 1 meter. This frequency is used with datatransmissions, access control applications, DVD kiosks, andCopyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

passport security – applications that do not require a longread range.
The third model with Ultra High Frequency (UHF,865~960 MHz) has a short, high-energy wavelength of abouta one meter which translates to long read range. Passive UHFtags can be read from an average distance of about 5 – 6
meters, but larger UHF tags can achieve up to 30+ meters ofread range in ideal conditions. This frequency is typicallyused with race timing, IT asset tracking, file tracking, andlaundry management as all these applications typically needmore than a meter of read range.
As a general rule, higher frequencies will have shorter,
higher-energy wavelengths and, in turn, longer read ranges.
Moreover, the higher the frequency, generally speaking, themore issues an RFID system will have aroundnon-RFID-friendly materials like water and metal.
B. Analysis and evaluation Model
This study starts by collecting human body and patch
application data based on a sensing patch module byconstructing a human - based wearable patch.
The factors affecting the wearable sensing system aresensing sensitivity and sensing success rate, sensing data
rate, power consumption, production and component cost,sensing scalability, mobility, lightweightness, and manyother environmental variables.
Nevertheless, this study focuses primarily on the sensing
sensitivity and the ability to transmit sensing data as themain environmental variables for analysing and evaluating
the performance of passive sensing patches.
Let us assume that the power transmitted by the reader
has a uniform power density in all directions over aspherical surface at any given distance d. Some of thispower is received by the tag antenna and is proportional tothe eﬀective aperture of the tag antenna and the power
impinging on the tag [17].
In RFID, the distance between the reader and passive tag
has a very signiﬁcant eﬀect on the power loss which
decreases as the inverse square of the distance between thereader and passive tag. The power density received at a
distance d is given by the free space transmission formula,
𝑃𝑟
𝑃𝑑 =
4𝜋𝑑 2
where 𝑃𝑑 is the received power density, 𝑃𝑟 is the powerradiated by the reader antenna and d is the distance between
the reader and passive tag.
First, the sensing sensitivity represents the reception
sensitivity for each radius according to power and active
signal strength as shown in Fig. 7.

Second, the receiving sensitivity is expressed according tochanging in transmitter power of the received signal versus
the signal transmission distance between the active terminal
and the passive patch as shown in Fig. 8.

Fig. 8. Receiving Signal Sensitivity over Distances

V. CONCLUSION
In this paper, we first proposed the architecture andcomponents of passive biometric information system. This
proposed system can be a good alternative to overcome thelimitation of power consumption of digital patches. In
addition, the cost of manufacturing a digital patch isminimized, maximizing the economic efficiency of the
product and maximizing market usability.
Then we proposed biometric information collection andfeedback algorithms, proposed an analytical model of passive
sensing system, and described analysis evaluation examples.
This suggests a future direction for the design of advanced
biometric information system.
In the future, this study intends to expand the scope of
research by constructing multi passive patch biometric
information system. We intend to build an embedded
bio-sensing platform for the development of embedded
bio-sensing PCB / Chip.
REFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

[7]

Fig. 7. Passive Signal Sensitivity over Power Radiation

1263

T. G. Lee and S. H. Lee, “Dynamic Bio-sensing Process Design inMobile Wellness Information System for Smart Healthcare s”,
Wireless Personal Communications, Vol. 86, No. 1, p. 201-215, 2016.
P. Celka, R. Vetter, P. Renevey, C. Verjus, V. Neuman, J. Luprano, J.
D. Decotignie and C. Piguet, “Wearable biosensing: signal processing
andcommunicationarchitectures
issues”,
Journalof
Telecommunications and Information Technology, p. 90-104, 2005.
T. G. Lee and S. H. Lee, “Design of Wearable Bio-patch System
Platform in Human Healthcare Environment”, Indian Journal of
Science and Technology, Vol. 8, p. 1-7, 2015.
S. Patel, H. Park, P. Bonato, L. Chan and M. Rodgers, “A review of
wearable sensors and systems with application in rehabilitation”,
Journal of NeuroEngineering and Rehabilitation, Vol. 9, No. 21, 2012.
T. G. Lee, M. S. Ko and S. H. Lee, “Wearable Multiple Bio-sensing
Process Architecture in Human Healthcare Environments”,
International Journal of Bio-Science and Bio-Technology, Vol. 6, No.
5, p. 177-184, 2014.
Tae-Gyu Lee, Chapter 15. Mobile Healthcare Computing in the Cloud,Mobile Networks and Cloud Computing Convergence for Progressive
Services and Applications, IGI Global, p. 275-294, 2014.
T. Lee, "Sensing system and sensing method of passive patch interface
for gathering bioinformation," The 20th International Conference on
Advanced Communication Technology (ICACT), Chuncheon-si
Gangwon-do, Korea (South), 2018, pp. 652-655. doi:
10.23919/ICACT.2018.8323869

Copyright ⓒ 2019 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 8, Issue 6, November 2019

1264

[8]

Ki-hwan Chon and Hyung-jin Choi, "A Study on Ubiquitous
Psychological State Recognition Model Using Bio-Signals," The
Journal of Korean Institute of Communications and Information
Sciences, Vol. 35, No. 2, pp. 232~243, 2010.
[9] Jin Seob Shin, Woo Young Ahn and Il Yong Oh, "Development of
Diagnosis System through Human-body Information Measurement,"
Journal of the Korea Society of Computer and Information , Vol. 13,
No. 1, pp. 219~226, 2008.
[10] Jee Hyun Lee, Eun Ji Lee, Ji Eun Kim, Yoo Lee Kim and Sin Won Cho,
"Wearable Sensing Device Design for Biological Monitoring," Journalof the Korean Society of Costume, Vol. 65, No. 1, pp. 118~135, 2015.
[11] Ju-Hyun Choi, Seung-Man Chun, Dong-Hyun Jang and Jong-Tae Park,"Design and Implementation of Bio-data Monitering System Based on
ISO/IEEE 11073 DIM/REST for IoT Healthcare Service," Journal of
the Institute of Electronics and Information Engineers, Vol. 52, No. 3,
pp. 3~12, 2015.
[12] Tae-Gyu Lee, Gi-Soo Chung and Seong-Hoon Lee, "Textile IT
Convergence: Digital Clothing and Wearable Computing Status,"
Korea Institute of Information Technology Magazine, Vol. 10, No. 2,
pp. 77~86, 2012.
[13] Eun Hye Park and Seung Jin Moon, "U-Healthcare offers and designed
wired and wireless integrated Hair care System," Proceedings ofSymposium of the Korean Institute of communications andInformation Sciences, pp. 698~699, 2010.
[14] Y. S. Kwon, "Greenhouse environmental management byphytomonitoring," Protected Horticulture and Plant Factory, Vol. 8, No.
1, pp. 127~130, 1995.
[15] Tae-hwan Park, Hwa-joeng Seo, Ji-hwan Lim and Ho-won Kim,
"Suggestion of Secure Driver Authentication and Vehicle Control
System based on NFC Communication and Biometric Information,"
Journal of the Korea Institute of Information and Communication
Engineering, Vol. 22, No. 4, pp. 700~707, 2018.
[16] Myung Geun Chun and Ki-young Moon, "Use of biometric
information and protection of privacy," REVIEW OF KIISC, Vol. 15,
No. 6, pp. 11~18, 2005.
[17] R. B. Green, “The general theory of antenna scattering,” in Ph.D.
Thesis, The Ohio State University, Columbus, OH, 1989.
Tae-Gyu Lee (BSc’92–MSc’96–PhD’06) received
the B.Sc. degree from Kunsan National University,
Kunsan, Korea in 1992, the M.Sc. degree fromSoongsil University, Seoul, Korea in 1996, and the
Ph.D. degree from Korea University in 2006. He iscurrently a Professor in the Smart Contents Major,
Division of ICT Convergence, Pyeongtaek
University, Gyeonggi, Korea since 2018. He has beena Professor in the Support center for Field Practice
Education, Wonkwang University, Jeonbuk, Korea for 2014-2018. He has
been a Professional Researcher in Advanced Convergent Technology R&D
Group, Korea Institute of Industrial Technology (KITECH), Ansan, Korea
for 2009-2013. He has also been a President in the JIGUNET Corporation,
Seoul, Korea, from 1999. His research interests are in distributed systems,
ubiquitous computing, middleware, networks, bioinformatics, wearable androbot computing. Prof. Lee is currently a lifetime member of Korea
Information Processing Society, IEEE ICACT reviewer, IEEE TPDS
reviewer.

Copyright ⓒ 2019 GiRI (Global IT Research Institute)