Journal Editorial Board Editor-in-ChiefProf. Thomas Byeongnam YOON, PhD.
Founding Editor-in-Chief
ICACT Transactions on the Advanced Communications Technology (TACT) Editors
Prof. Jun-Chul Chun, Kyonggi University, Korea
Dr. JongWon Kim, GIST (Gwangju Institute of Science & Technology), Korea
Dr. Xi Chen, State Grid Corparation of China, China
Prof. Arash Dana, Islamic Azad university , Central Tehran Branch, Iran
Dr. Pasquale Pace, University of Calabria - DEIS - Italy, Italy
Dr. Mitch Haspel, Stochastikos Solutions R&D, Israel
Prof. Shintaro Uno, Aichi University of Technology, Japan
Dr. Tony Tsang, Hong Kong Polytechnic UNiversity, Hong Kong
Prof. Kwang-Hoon Kim, Kyonggi University, Korea
Prof. Rosilah Hassan, Universiti Kebangsaan Malaysia(UKM), Malaysia
Dr. Sung Moon Shin, ETRI, Korea
Dr. Takahiro Matsumoto, Yamaguchi University, Japan
Dr. Christian Esteve Rothenberg, CPqD - R&D Center for. Telecommunications, Brazil
Prof. Lakshmi Prasad Saikia, Assam down town University, India
Prof. Moo Wan Kim, Tokyo University of Information Sciences, Japan
Prof. Yong-Hee Jeon, Catholic Univ. of Daegu, Korea
Dr. E.A.Mary Anita, Prathyusha Institute of Technology and Management, India
Dr. Chun-Hsin Wang, Chung Hua University, Taiwan
Prof. Wilaiporn Lee, King Mongkut’s University of Technology North, Thailand
Dr. Zhi-Qiang Yao, XiangTan University, China
Prof. Bin Shen, Chongqing Univ. of Posts and Telecommunications (CQUPT), China
Prof. Vishal Bharti, Dronacharya College of Engineering, India
Dr. Marsono, Muhammad Nadzir , Universiti Teknologi Malaysia, Malaysia
Mr. Muhammad Yasir Malik, Samsung Electronics, Korea
Prof. Yeonseung Ryu, Myongji University, Korea
Dr. Kyuchang Kang, ETRI, Korea
Prof. Plamena Zlateva, BAS(Bulgarian Academy of Sciences), Bulgaria
Dr. Pasi Ojala, University of Oulu, Finland
Prof. CheonShik Kim, Sejong University, Korea
Dr. Anna bruno, University of Salento, Italy
Prof. Jesuk Ko, Gwangju University, Korea
Dr. Saba Mahmood, Air University Islamabad Pakistan, Pakistan
Prof. Zhiming Cai, Macao University of Science and Technology, Macau
Prof. Man Soo Han, Mokpo National Univ., Korea
Mr. Jose Gutierrez, Aalborg University, Denmark

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Dr. Youssef SAID, Tunisie Telecom, Tunisia
Dr. Noor Zaman, King Faisal University, Al Ahsa Hofuf, Saudi Arabia
Dr. Srinivas Mantha, SASTRA University, Thanjavur, India
Dr. Shahriar Mohammadi, KNTU University, IranProf. Beonsku An, Hongik University, korea
Dr. Guanbo Zheng, University of Houston, USA
Prof. Sangho Choe, The Catholic University of Korea, korea
Dr. Gyanendra Prasad Joshi, Yeungnam University, korea
Dr. Tae-Gyu Lee, Korea Institue of Industrial Technology(KITECH), korea
Prof. Ilkyeun Ra, University of Colorado Denver, USA
Dr. Yong Sun, Beijing University of Posts and Telecommunications, China
Dr. Yulei Wu, Chinese Academy of Sciences, China
Mr. Anup Thapa, Chosun University, korea
Dr. Vo Nguyen Quoc Bao, Posts and Telecommunications Institute of Technology, Vietnam
Dr. Harish Kumar, Bhagwant institute of technology, India
Dr. Jin REN, North china university of technology, China
Dr. Joseph Kandath, Electronics & Commn Engg, India
Dr. Mohamed M. A. Moustafa, Arab Information Union (AIU), Egypt
Dr. Mostafa Zaman Chowdhury, Kookmin University, Korea
Prof. Francis C.M. Lau, Hong Kong Polytechnic University, Hong Kong
Prof. Ju Bin Song, Kyung Hee University, korea
Prof. KyungHi Chang, Inha University, Korea
Prof. Sherif Welsen Shaker, Kuang-Chi Institute of Advanced Technology, China
Prof. Seung-Hoon Hwang, Dongguk University, Korea
Prof. Dal-Hwan Yoon, Semyung University, korea
Prof. Chongyang ZHANG, Shanghai Jiao Tong University, China
Dr. H K Lau, The Open University of Hong Kong, Honh Kong
Prof. Ying-Ren Chien, Department of Electrical Engineering, National Ilan University, Taiwan
Prof. Mai Yi-Ting, Hsiuping University of Science and Technology, Taiwan
Dr. Sang-Hwan Ryu, Korea Railroad Research Institute, Korea
Dr. Yung-Chien Shih, MediaTek Inc., Taiwan
Dr. Kuan Hoong Poo, Multimedia University, Malaysia
Dr. Michael Leung, CEng MIET SMIEEE, Hong Kong
Dr. Abu sahman Bin mohd Supa'at, Universiti Teknologi Malaysia, Malaysia
Prof. Amit Kumar Garg, Deenbandhu Chhotu Ram University of Science & Technology, India
Dr. Jens Myrup Pedersen, Aalborg University, Denmark
Dr. Augustine Ikechi Ukaegbu, KAIST, Korea
Dr. Jamshid Sangirov, KAIST, Korea
Prof. Ahmed Dooguy KORA, Ecole Sup. Multinationale des Telecommunications, Senegal
Dr. Se-Jin Oh, Korea Astronomy & Space Science Institute, Korea
Dr. Rajendra Prasad Mahajan, RGPV Bhopal, India
Dr. Woo-Jin Byun, ETRI, Korea
Dr. Mohammed M. Kadhum, School of Computing, Goodwin Hall, Queen's University , Canada
Prof. Seong Gon Choi, Chungbuk National University, Korea
Prof. Yao-Chung Chang, National Taitung University, Taiwan
Dr. Abdallah Handoura, Engineering school of Gabes - Tunisia, Tunisia
Dr. Gopal Chandra Manna, BSNL, India

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Dr. Il Kwon Cho, National Information Society Agency, Korea
Prof. Jiann-Liang Chen, National Taiwan University of Science and Technology, Taiwan
Prof. Ruay-Shiung Chang, National Dong Hwa University, Taiwan
Dr. Vasaka Visoottiviseth, Mahidol University, Thailand
Prof. Dae-Ki Kang, Dongseo University, Korea
Dr. Yong-Sik Choi, Research Institute, IDLE co., ltd, Korea
Dr. Xuena Peng, Northeastern University, China
Dr. Ming-Shen Jian, National Formosa University, Taiwan
Dr. Soobin Lee, KAIST Institute for IT Convergence, Korea
Prof. Yongpan Liu, Tsinghua University, China
Prof. Chih-Lin HU, National Central University, Taiwan
Prof. Chen-Shie Ho, Oriental Institute of Technology, Taiwan
Dr. Hyoung-Jun Kim, ETRI, Korea
Prof. Bernard Cousin, IRISA/Universite de Rennes 1, France
Prof. Eun-young Lee, Dongduk Woman s University, Korea
Dr. Porkumaran K, NGP institute of technology India, India
Dr. Feng CHENG, Hasso Plattner Institute at University of Potsdam, Germany
Prof. El-Sayed M. El-Alfy, King Fahd University of Petroleum and Minerals, Saudi Arabia
Prof. Lin You, Hangzhou Dianzi Univ, China
Mr. Nicolai Kuntze, Fraunhofer Institute for Secure Information Technology, Germany
Dr. Min-Hong Yun, ETRI, Korea
Dr. Seong Joon Lee, Korea Electrotechnology Research Institute, korea
Dr. Kwihoon Kim, ETRI, Korea
Dr. Jin Woo HONG, Electronics and Telecommunications Research Inst., Korea
Dr. Heeseok Choi, KISTI(Korea Institute of Science and Technology Information), korea
Dr. Somkiat Kitjongthawonkul, Australian Catholic University, St Patrick's Campus, Australia
Dr. Dae Won Kim, ETRI, Korea
Dr. Ho-Jin CHOI, KAIST(Univ), Korea
Dr. Su-Cheng HAW, Multimedia University, Faculty of Information Technology, Malaysia
Dr. Myoung-Jin Kim, Soongsil University, Korea
Dr. Gyu Myoung Lee, Institut Mines-Telecom, Telecom SudParis, France
Dr. Dongkyun Kim, KISTI(Korea Institute of Science and Technology Information), Korea
Prof. Yoonhee Kim, Sookmyung Women s University, Korea
Prof. Li-Der Chou, National Central University, Taiwan
Prof. Young Woong Ko, Hallym University, Korea
Prof. Dimiter G. Velev, UNWE(University of National and World Economy), Bulgaria
Dr. Tadasuke Minagawa, Meiji University, Japan
Prof. Jun-Kyun Choi, KAIST (Univ.), Korea
Dr. Brownson ObaridoaObele, Hyundai Mobis Multimedia R&D Lab , Korea
Prof. Anisha Lal, VIT university, India
Dr. kyeong kang, University of technology sydney, faculty of engineering and IT , Australia
Prof. Chwen-Yea Lin, Tatung Institute of Commerce and Technology, Taiwan
Dr. Ting Peng, Chang'an University, China
Prof. ChaeSoo Kim, Donga University in Korea, Korea
Prof. kirankumar M. joshi, m.s.uni.of baroda, India
Dr. Chin-Feng Lin, National Taiwan Ocean University, Taiwan
Dr. Chang-shin Chung, TTA(Telecommunications Technology Association), Korea

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Dr. Che-Sheng Chiu, Chunghwa Telecom Laboratories, Taiwan
Dr. Chirawat Kotchasarn, RMUTT, Thailand
Dr. Fateme Khalili, K.N.Toosi. University of Technology, IranDr. Izzeldin Ibrahim Mohamed Abdelaziz, Universiti Teknologi Malaysia , Malaysia
Dr. Kamrul Hasan Talukder, Khulna University, Bangladesh
Prof. HwaSung Kim, Kwangwoon University, Korea
Prof. Jongsub Moon, CIST, Korea University, Korea
Prof. Juinn-Horng Deng, Yuan Ze University, Taiwan
Dr. Yen-Wen Lin, National Taichung University, Taiwan
Prof. Junhui Zhao, Beijing Jiaotong University, China
Dr. JaeGwan Kim, SamsungThales co, Korea
Prof. Davar PISHVA, Ph.D., Asia Pacific University, Japan
Ms. Hela Mliki, National School of Engineers of Sfax, Tunisia
Prof. Amirmansour Nabavinejad, Ph.D., Sepahan Institute of Higher Education, Iran

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

Editor Guide
 Introduction for Editor or Reviewer
All the editor group members are to be assigned as a evaluator(editor or reviewer) to submitted journal
papers at the discretion of the Editor-in-Chief. It will be informed by eMail with a Member Login ID and
Password.
Once logined the Website via the Member Login menu in left as a evaluator, you can find out the paper
assigned to you. You can evaluate it there. All the results of the evaluation are supposed to be shown in
the Author Homepage in the real time manner. You can also enter the Author Homepage assigned to
you by the Paper ID and the author's eMail address shown in your Evaluation Webpage. In the Author
Homepage, you can communicate each other efficiently under the peer review policy. Please don't missit!
All the editor group members are supposed to be candidates of a part of the editorial board, dependingon their contribution which comes from history of ICACT TACT as an active evaluator. Because the main
contribution comes from sincere paper reviewing role.

 Role of the Editor
The editor's primary responsibilities are to conduct the peer review process, and check the final cameraready manuscripts for any technical, grammatical or typographical errors.
As a member of the editorial board of the publication, the editor is responsible for ensuring that the
publication maintains the highest quality while adhering to the publication policies and procedures of
the ICACT TACT(Transactions on the Advanced Communications Technology).
For each paper that the editor-in-chief gets assigned, the Secretariat of ICACT Journal will send theeditor an eMail requesting the review process of the paper.
The editor is responsible to make a decision on an "accept", "reject", or "revision" to the Editor-in-Chief
via the Evaluation Webpage that can be shown in the Author Homepage also.

 Deadlines for Regular Review
Editor-in-Chief will assign a evalaution group( a Editor and 2 reviewers) in a week upon receiving acompleted Journal paper submission. Evaluators are given 2 weeks to review the paper. Editors are
given a week to submit a recommendation to the Editor-in-Chief via the evaluation Webpage, once all or
enough of the reviews have come in. In revision case, authors have a maximum of a month to submit
their revised manuscripts. The deadlines for the regular review process are as follows:

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Evalution Procedure

Deadline

Selection of Evaluation Group
Review processing
Editor's recommendation
Final Decision Noticing

1 week2 weeks
1 week
1 week Making Decisions on Manuscript
Editor will make a decision on the disposition of the manuscript, based on remarks of the reviewers. The
editor's recommendation must be well justified and explained in detail. In cases where the revision isrequested, these should be clearly indicated and explained. The editor must then promptly convey thisdecision to the author. The author may contact the editor if instructions regarding amendments to themanuscript are unclear. All these actions could be done via the evaluation system in this Website. The
guidelines of decisions for publication are as follows:

Decision

Description

AcceptAn accept decision means that an editor is accepting the paper withno further modifications. The paper will not be seen again by theeditor or by the reviewers.
The manuscript is not suitable for the ICACT TACT publication.
The paper is conditionally accepted with some requirements. A
revision means that the paper should go back to the original
reviewers for a second round of reviews. We strongly discourage
editors from making a decision based on their own review of themanuscript if a revision had been previously required.

RejectRevision Role of the Reviewer
Reviewer Webpage:
Once logined the Member Login menu in left, you can find out papers assigned to you. You can alsologin the Author Homepage assigned to you with the paper ID and author's eMail address. In there you
can communicate each other via a Communication Channel Box.

Quick Review Required:
You are given 2 weeks for the first round of review and 1 week for the second round of review. You must
agree that time is so important for the rapidly changing IT technologies and applications trend. Pleaserespect the deadline. Authors undoubtedly appreciate your quick review.

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Anonymity:
Do not identify yourself or your organization within the review text.

Review:
Reviewer will perform the paper review based on the main criteria provided below. Please provide
detailed public comments for each criterion, also available to the author.
 How this manuscript advances this field of research and/or contributes something new to theliterature?
 Relevance of this manuscript to the readers of TACT?
 Is the manuscript technically sound?
 Is the paper clearly written and well organized?
 Are all figures and tables appropriately provided and are their resolution good quality?
 Does the introduction state the objectives of the manuscript encouraging the reader to read on?
 Are the references relevant and complete?

Supply missing references:
Please supply any information that you think will be useful to the author in revision for enhancing
quality of the paperor for convincing him/her of the mistakes.

Review Comments:
If you find any already known results related to the manuscript, please give references to earlier papers
which contain these or similar results. If the reasoning is incorrect or ambiguous, please indicate
specifically where and why. If you would like to suggest that the paper be rewritten, give specificsuggestions regarding which parts of the paper should be deleted, added or modified, and pleaseindicate how.

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Journal Procedure
Dear Author,

 You can see all your paper information & progress.
 Step 1. Journal Full Paper Submission
Using the Submit button, submit your journal paper through ICACT Website, then you will get newpaper ID of your journal, and send your journal Paper ID to the Secretariat@icact.org for the review andeditorial processing. Once you got your Journal paper ID, never submit again! Journal Paper/CRF
Template Step 2. Full Paper Review
Using the evaluation system in the ICACT Website, the editor, reviewer and author can communicate
each other for the good quality publication. It may take about 1 month.

 Step 3. Acceptance Notification
It officially informs acceptance, revision, or reject of submitted full paper after the full paper review
process.

Status
Acceptance
Revision
RejectActionGo to next Step.
Re-submit Full Paper within 1 month after Revision Notification.
Drop everything.

 Step 4. Payment Registration
So far it's free of charge in case of the journal promotion paper from the registered ICACT conference
paper! But you have to regist it, because you need your Journal Paper Registration ID for submission of
the final CRF manuscripts in the next step's process. Once you get your Registration ID, send it to
Secretariat@icact.org for further process.

 Step 5. Camera Ready Form (CRF) Manuscripts Submission
After you have received the confirmation notice from secretariat of ICACT, and then you are allowed tosubmit the final CRF manuscripts in PDF file form, the full paper and the Copyright Transfer Agreement.
Journal Paper Template, Copyright Form Template, BioAbstract Template,

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Journal Submission Guide
All the Out-Standing ICACT conference papers have been invited to this "ICACT Transactions on the
Advanced Communications Technology" Journal, and also welcome all the authors whose conference
paper has been accepted by the ICACT Technical Program Committee, if you could extend new contents
at least 30% more than pure content of your conference paper. Journal paper must be followed toensure full compliance with the IEEE Journal Template Form attached on this page.

 How to submit your Journal paper and check the progress?
Step 1. Submit Using the Submit button, submit your journal paper through ICACT

Step 2. Confirm

Step 3. Review

Website, then you will get new paper ID of your journal, and send yourjournal Paper ID to the Secretariat@icact.org for the review and editorial
processing. Once you got your Journal paper ID, never submit again! Using
the Update button, you can change any information of journal paper
related or upload new full journal paper.
Secretariat is supposed to confirm all the necessary conditions of yourjournal paper to make it ready to review. In case of promotion from theconference paper to Journal paper, send us all the .DOC(or Latex) files ofyour ICACT conference paper and journal paper to evaluate the differenceof the pure contents in between at least 30% more to avoid the self
replication violation under scrutiny. The pure content does not include any
reference list, acknowledgement, Appendix and author biography
information.
Upon completing the confirmation, it gets started the review process thru
the Editor & Reviewer Guideline. Whenever you visit the Author
Homepage, you can check the progress status of your paper there fromstart to end like this, " Confirm OK! -> Gets started the review process > ...", in the Review Status column. Please don't miss it!

Copyright ⓒ 2018 GiRI (Global IT Research Institute)Volume. 7 Issue. 6

6J

1 SvgAI – Training Methods Analysis of Artificial Intelligent Agent to use SVG Editor

1159

Anh H. Dang*, Wataru Kameyama***GITS, Waseda University, Tokyo, Japan** Faculty of Science and Engineering, Waseda University, Tokyo, Japan

2 Cognitive Workload Detection from Raw EEGSignals of Vehicle Driver using Deep Learning

1167

Mohammad A. Almogbel*, Anh H. Dang**, Wataru Kameyama****Department. of Computer Science and Communications Engineering,
Graduate School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan.
**GITS, Waseda University, Tokyo, Japan***Faculty of Science and Engineering, Waseda University, Tokyo, Japan

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

1159

SvgAI – Training Methods Analysis ofArtificial Intelligent Agent to use SVG
EditorAnh H. Dang*, Wataru Kameyama***GITS, Waseda University, Tokyo, Japan** Faculty of Science and Engineering, Waseda University, Tokyo, Japan
danghoanganh@akane.waseda.jp, wataru@waseda.jp
Abstract— Deep reinforcement learning has been successfully
used to train artificial intelligent (AI) agents, which outperforms
humans in many tasks. The objective of this research is to train
an AI agent to draw SVG images by using scalable vector
graphic (SVG) editor with deep reinforcement learning, where
the AI agent is to draw SVG images that are similar as much as
possible to the given target raster images. In this paper, we
propose framework to train the AI agent by value-function
based Q-learning and policy-gradient based learning methods.
With Q-learning based method, we find that it is crucial todistinguish the action space into two sets to apply a different
exploration policy on each set during the training process.
Evaluations show that our proposed dual ϵ-greedy exploration
policy greatly stabilizes the training process and increases theaccuracy of the AI agent. On the other hand, policy-gradient
based training does not depend on external reward function.
However, it is hard to implement especially in the environment
with a large action space. To overcome this difficulty, we propose
a strategy similar to the dynamic programming method to allow
the agent to generate training samples by itself. In ourevaluation, the highest score is archived by the agent trained bythis proposed method. SVG images produced by the proposed
AI agent have also superior quality compared to popular rasterto-SVG conversion softwares.
Keywords—Reinforcement
Policy, Q-learning

Learning,

SVG,

Exploration

I. INTRODUCTION
Besides defeating the world best human player in Go [1],AI agent (hereafter referred as agent) trained by deep
reinforcement learning (RL) [2] has achieved human-level ina wide variety of tasks like playing 3D first-person shooter
game [3], and enhances the capability of robotic automation[4]. For example, Mnih et. al. have introduced deep Qnetwork (DQN) [5] that plays Atari 2600 games well above
the skill of human players and any other linear models.
Subsequently, the works on prioritized experience replay [6],double Q-network [7], duel Q-network [8] and asynchronous
actor-critic method [9] further enhance the efficiency of the
training process.

Manuscript received on Dec. 31, 2017. This work is a follow-up of theinvited journal to the accepted & presented paper of the 20 th International
Conference on Advanced Communication Technology (ICACT2018).
Anh H. Dang is with Graduate School of Global Information andTelecommunication Studies, Waseda University, Tokyo, Japan(corresponding
author,phone:+81-80-1367-9637,
email:
danghoanganh@akane.waseda.jp)
Wataru Kameyama is with School of Science and Engineering, Waseda
University, Tokyo, Japan (email: wataru@waseda.jp)

On the subject of image understanding and raster-to-vector(R2V) conversion, Karpathy et. al. present a breakthroughwork [10] on training a deep neural network (DNN) [11] for
automatic image description. Beltramelli proposes Pix2CodeDNN [12] that generates Extensible Markup Language (XML)based code from raster screenshot of graphical user interface(GUI).
Despite being a mature branch of research, imageprocessing based R2V conversion is not yet reliable [13].
Therefore, we propose a framework to train an agent to use
SVG editor (hereafter referred as editor) with RL. The
objective of this agent is to draw an SVG image that is similar
as much as possible to a given target raster image. It can beconsidered as a new paradigm to solve the R2V problem.
In this paper, we concentrate on exploring the feasibility ofthis new paradigm by training the agent to work on randomly
generated target images. A custom editor is created forcarrying out the research, which has modeled after OpenAI
Gym [14] environment due to its robustness in interface
design.
We train the agent by using both Q-leaning based andpolicy-gradient based methods. We evaluate the agent
performance by comparing the similarity between generated
SVG images and target images. Finally, we compare thequality between the SVG image produced by the proposed
agent with that produced by popular R2V softwares.
This paper is organized as follows: Section II describes
related works on R2V and RL. Section III describes our
proposal for both the agent and the environment design.
Section IV describes the agent model. Then, Section V
describes training and evaluation process in detail. Section VI
concludes the paper with possible improvement and future
works.
II. RELATED WORKS
The agent needs to be trained to ultimately convert a raster
image to an equivalent vector representation by using theeditor. Therefore, this research is related to not only RL but
also a series of visual-vector cross model works. There are
two categories of works related to this paper: imageprocessing based R2V and deep learning based R2V
conversions.
A. Image-Processing Based R2V conversion
As mentioned in Section I, image-processing based R2V
conversion is not yet reliable [13]. Major problems includedifficulties of color quantization, aliasing effects, shift,

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018superposition effects, and miss-identification of texture andtext [15].
There are numerous works to try to solve the abovementioned problems. For example, Kansal et. al. propose aframework to reproduce linear filled gradient [16]. Vector
representation of halftone dots in binary images is presented
by Kawamura et. al. [17]. However, they leave the work of
identifying the type of problems to the human operator. Thus,
for example, well-known conversion tools such as Potrace [18]still require human’s intervention to achieve desirable results[19].
B. Deep Learning Based R2V conversion
Image annotation has been an active research area. It
becomes overly crowded recently due to the advancement of
deep learning based natural language processing andcomputer vision. Karpathy et. al. achieve a breakthrough withan end-to-end DNN model [10]. Learning only from images
and corresponding annotations, this model not only
recognizes and locates objects, but also annotates images atdifferent hierarchical levels. The model is realized by theembedding visual model using RCNN (Regional
Convolutional Neural Network) [20] and the language modelusing BRNN (Bi-directional Recurrent Neural Network) [21]into the same multimodal space.
Beltramelli proposes Pix2Code [12], an end-to-end DNN
that generates XML based GUI code from mock-up images.
Even though similar to [10] in general design, the visualmodel used in Pix2Code is a plain CNN block while language
model is handled by a Long-Short Term Memory (LSTM)
network [22] block. Another LSTM block is used to decode
the network’s output into code tokens. This work can beunderstood as a rigid version of automated image annotation.
However, the model is not flexible because the visualpresentations of all the GUI elements in this work aretemplated. Thus, it only works with GUI images based on the
fixed templates.
C. Reinforcement Learning
The agent in deep RL holds a policy set that ultimately
decides which action to be performed in the next step. At eachstate of time step 𝑡, the agent observes the state 𝑠𝑡 of theenvironment, and decides action 𝑎𝑡 based on its current policy
𝜋. Reward 𝑟𝑡 is then given to the agent as the feedback fromthe environment. The objective of the training process is totrain the agent so that its policy will result in maximized
reward in the future. The expected reward is calculated asfollows.
(1)

𝑖=𝑡

Where 𝑅𝑡 is the sum of expected discounted reward at time
step 𝑡, 𝑇 is the time when the episode is terminated, and 𝛾 isthe discount factor which lies within the range of [0 … 1]. The
higher the value is, the more important the future reward is.
1) Q-learning: In Q-learning, 𝑅𝑡 is approximated by the
𝑄 function. This function returns an action-state value
according to policy 𝜋 as follows:𝑄𝜋 (𝑠, 𝑎) = 𝔼[𝑅𝑡 |𝑠𝑡 = 𝑠, 𝑎𝑡 = 𝑎]

the action that gives the highest 𝑄 value is chosen to be
executed by the agent. The optimal value 𝑄 ∗ (𝑠, 𝑎) is defined
as follows:𝑄 ∗ (𝑠, 𝑎) = max 𝑄𝜋 (𝑠, 𝑎) = max 𝔼[𝑅𝑡 |𝑠𝑡 = 𝑠, 𝑎𝑡 = 𝑎] (3)𝜋𝜋Hence, using (1) and (2), equation (3) can be written as
follows:〖𝑄 ∗ (𝑠 ′ , 𝑎′ )|𝑠, 𝑎〗]𝑄∗ (𝑠, 𝑎) = 𝔼 [𝑟 + 𝛾 max
′
𝑎

(4)

Where 𝑠 ′ and 𝑎′ are the state and the action of thesubsequent time step, respectively, and 𝑟 is the immediate
reward for action 𝑎 given in state 𝑠. In deep 𝑄-learning, thisvalue is approximated by the neural network parameterizedby 𝜃:𝑄𝜃 (𝑠, 𝑎) ≈ 𝑄 ∗ (𝑠, 𝑎)(5)Thus, with a trained agent, the 𝑄 value of an action 𝑎 canbe estimated as follows:𝑄𝜃 (𝑠, 𝑎) ≈ 𝑟𝑡 + 𝛾 max
𝑄𝜃 (𝑠 ′ , 𝑎′ )
′
𝑎

(6)

Given that 𝑦𝑡 = 𝑟𝑡 + 𝛾 max
𝑄𝜃 (𝑠 ′ , 𝑎′ ), the loss function is′𝑎defined as follows:2

𝐿𝑡 (𝜃𝑡 ) = 𝔼𝑠,𝑎,𝑟,𝑠′ [|𝑦𝑡 − 𝑄𝜃𝑡 (𝑠, 𝑎)| ](7)

In practice, instead of squared difference loss, Huber loss
is usually used to stabilize the training process:𝐿(𝛼) = {1 2𝛼2if |𝛼| ≤ 𝛿

1𝜎 (|𝛼| − 𝛿) otherwise
2(8)

Where 𝛼 is the difference between 𝑦𝑡 and 𝑄𝜃𝑡 (𝑠, 𝑎), and 𝛿
is the point where the loss function change from quadratic tolinear.
2) Policy-Gradient: Agents trained by the Q-learning
method predict the state-action value as in formula (6), thenthe action is chosen deterministically based on this value.
Thus, it heavily depends on the value function to result inbetter policy approximation. On the contrary, with policygradient based training, the agent is trained to output theaction probability directly from a given state. Formally,
policy-gradient optimizes policy 𝜃 to maximize the expecteddiscount return 𝑅𝑡 :
𝜃 = argmax 𝔼[𝑅𝑡 ]

(9)

𝜃

𝑇

𝑅𝑡 = ∑ 𝛾 𝑖−𝑡 𝑟i1160

To optimize the policy 𝜃, the gradient of policy 𝜃 is givenby:∇𝜃 𝔼[𝑅𝑡 ] = 𝔼[∇𝜃 log 𝑃(𝑎𝑡 )𝑅𝑡 ](10)

Where 𝑃(𝑎𝑡 ) is the probability of action 𝑎 at time step 𝑡.
Thus, actions that lead to better expected reward 𝑅𝑡 areencouraged. In order to train an agent using policy-gradient,
𝑅𝑡 must be known or has to be approximated.
3) Experience Memory Replay: One of the mostsignificant difficulties in training an agent is the strong(2)

Hence, 𝑄𝜋 (𝑠, 𝑎) is essentially an expected future reward if
the agent performs action 𝑎 in the given state 𝑠. In practice,

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

correlation between the network policy and the action
outcome of subsequent time steps. This difficulty makes
online training impossible. To break the strong correlation,
experience memory replay [6] is used.
In the experience memory replay, at every time step 𝑡, theexperience (𝑠𝑡 , 𝑎𝑡 , 𝑟𝑡 , 𝑠𝑡+1 ) of the agent is stored in replay
memory which is a large capacity queue. In popular works
like DQN, it is usually set to store one million experiences.
When it’s full, the oldest experience in the queue is removed
to make place for a new experience. The network is trained byusing mini batches randomly drawn from this memory.
4) Exploration in RL: Training an agent using RL
requires a right balance between exploitation and exploration.
Exploitation is relying on the learned policy to improve theprediction accuracy while exploration allows the agent to seek
for better potential solutions (i.e. avoiding sub-optimal trap).
A popular exploration policy being used in RL is 𝜖-greedy [2].
Under this policy, the output of the agent has an 𝜖 chance ofbeing random.
There are other exploration policies based on
randomization, such as Thompson sampling [23] andBayesian sampling [24]. However, a variance of ϵ-greedy
policy named reducing ϵ -greedy is most commonly used.
With this policy, the agent starts with high exploration rate,
and gradually reduces it throughout the training process. Thus,
it allows the agent to explore in the beginning, and to focus
more on exploitation in the later phase of the training process.
In this paper, we evaluate the reducing 𝜖-greedy variances
only. All 𝜖 -greedy policies mentioned hereafter refer toreducing 𝜖-greedy exploration policy.
III. SVG EDITOR
As seen in common in AI settings, the proposed framework
consists of two parts: the agent and the environment. As
shown in Fig. 1, the editor is playing the role of theenvironment in this research. For every time step, the agent
observes the state of the editor (step 1). Then, the agent
processes the observed state and sends a new action to theeditor (step 2). So, the editor executes the action as requested
and sends reward back to the agent (step 3). And the process
goes back to step 1.
SVG
DOCUMENTACTIONSVG EDITORAI AGENTOBSERVATIONREWARD

Fig. 1. The proposed framework of SvgAI. The editor is playing the role ofthe environment.

A. SVG Construction
SVG is an XML based vector image format. An SVG
document is composed of different types of elements, such as
path, circle, and rectangle. Previous works on R2V
conversion usually produce SVG image using path element
exclusively because this element is flexible and can be used
to form any type of shape. Furthermore, using fundamental
shape elements, such as rectangle, circle and arc, not only

Add New
Element

1161

Element Editing

START

a1

a2

a3

a4

a j4

a j 3

aj

ak 1

ak

aT  2

aT 1

a j2

a j 1

aT

aT 3

END

Fig. 2. The process of SVG composition: the agent keeps adding andediting new elements until the desired result is achieved. 𝑎𝑖 is action givenby the agent at time step 𝑡. 𝑇 is the last time step in the episode.

requires the higher level of visual understanding but alsocomes with none trivial challenges.
On the other hand, using a large number of elements andpaths significantly increases the SVG image size. Thus, theoutput SVG requires much more computational resource to be
displayed. In this research, we utilize two fundamental shapeelements (and with their transformations) that are square
element and circle element for SVG image construction.
Fig. 2 shows the process of SVG composition, where every
episode starts with a blank canvas. During the process, theagent is either adding a new element into the working image
or editing the most recently added element.
The newly added element has a default presentation whenit’s firstly added. The default presentations of the circle andsquare elements are shown in sub-figure (a) and (g) of Fig. 5.
As the agent keeps editing, the presentation of the element is
to be updated. For example, subfigure (l) in Fig. 5 is thepresentation of a circle element after 400 steps of editing.
B. Action SpaceWith the above-explained process, once a new element is
added to the working SVG document, the old element is nolonger editable. Thus, the consequence of adding newelements is more significant compared to editing them.
Therefore, for Q-learning based training, it is crucial todistinguish the two sets of actions, set A and set B, and to
apply separated exploration policies during the training
process. Otherwise, it is impossible for the model to converge.
Table I lists all the actions of set A and B supported by theeditor. Set A consists of actions that add new element into theworking SVG document, while set B consists of editing
actions, i.e. element-shape manipulation actions.
TABLE I
LIST OF ACTIONS SUPPORTED BY THE EDITOR
Set

Action Id.

Description

A0, 1
2, 3, 4, 5
6, 7, 8, 9
10, 11
12, 1314, .., 21
22, .., 29

Add circle/square element
Move element left/right/up/downCompress/expand element horizontally/vertically
Rotate the element clockwise/counter clockwise
Reduce/increase element’s line thickness
Adjusta. element’s line color (RGB𝛼)Adjusta. element’s fill color (RGB𝛼)B

a.

Increase/decrease value of each channel

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018
Observation[128 × 128 × 12]1162

Flattenedfeature map 3

Conv 2[4 × 4 × 96 × 192] – Stride 2

FC 1[2048]

Conv 3[3 × 3 × 192 × 192] – Stride 1

Feature map 1[31 × 31 × 96]

Feature map 2[14 × 14 × 192]
Feature map 3[12 × 12 × 192]

Conv 1[8 × 8 × 12 × 96] – Stride 4

FC 2[2048]

Action scores[number_of_actions]

Auxiliary
vectorFig. 3. The architechture of the agent. The stacked images explained in Secion III-C is processed by the convolution layers. Auxilary vector is concatenatedwith the output of the convolution layers before feeding to fully connected blocks at the end of the network.

C. State Observation
The state of the editor at any time step 𝑡 consists of four
components:
1. Raster version of the image being edited 𝐼𝑡 .
2. Raster target image 𝑌, i.e. the image which the agent
attempts to compose by using the editor.
3. Raster image of the element being editted in 𝐼𝑡(component 1). Thus, , this component is exactly thesame to 𝐼𝑡 when only one element is in 𝐼𝑡 .
4. An auxiliary vector that describes the state of the SVG
element being edited (component 3). This auxiliary
vector consist of paremeters such as orientation,
position, line thickness, line color, and fill color.
All raster images are in the RGB 𝛼 format with theresolution of 128 by 128 pixels. The first three component arestacked in channels dimention to form the image stack.

A. Network Architecture
Fig. 3 shows the architecture of the proposed agent. The
above mentioned image stack is processed by the
convolutional neural network (CNN) blocks. These blocks
produce a feature map with around 27k parameters. The
combination of this feature map and the auxiliary vector isthen processed by a fully connected block to produce a scoreor probability for each action supported by the environment.
B. Error and Reward
With every action received from the agent, the score is
calculated as follows:|𝐼𝑡 − 𝑌|2)
𝜎2

𝑖𝑓 𝑔𝑡 > 𝑔𝑡−1
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒(12)

Where 𝑟𝑡 is the immediate reward at time step 𝑡. Thus, theenvironment returns 0 reward when the action results in no
improvement, and returns a small reward ranging from 1 to 2
depending on the similarity between the result and the target
image.
Given 𝑣𝑡 which is the number of SVG elements at time
step 𝑡, the penalty 𝑝𝑡 is given as follows:
𝑝𝑡 = {−1
0

𝑖𝑓 𝑣𝑡 > 𝑣𝑡−1
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒(13)

Hence, the discounted reward 𝑅𝑡 at time step 𝑡 is calculated
as follows:𝑇

𝑅𝑡 = ∑(𝑟𝑖 + 𝑝𝑖 )𝛾 𝑖−𝑡

(14)

𝑖=𝑡

IV. MODEL

𝑔𝑡 = exp (−1 + 𝑔𝑡𝑟𝑡 = {
0(11)

Where 𝑔𝑡 is the score at time step 𝑡, which describes thesimilarity between the rasterized version 𝐼𝑡 of the working
SVG document at time step 𝑡 and the target image 𝑌, and 𝜎 isthe scaling factor. Thus, the domain of the score 𝑔𝑡 is from 0
to 1 where 1 means perfect matching.
Base on the score 𝑔𝑡 , the reward is given to the agent asfollows:

Where 𝛾 is the discount factor and 𝑇 is the length of theepisode. By giving a penalty for adding elements more thannecessary, the agent is discouraged to perform actions in set 𝐴.
However, this technique is only feasible with a controlledenvironment in which the structure of a state is known. The
training process is expected to be slower without this penalty.
C. Q-learning and Exploration Policy
In our early attempts, the conventional 𝜖-greedy policy wasused. On average, with the action sets exlained in Section IIIB, there was a new element added into the image for every 14
adjustments during the exploration process. It means,regardless the value set to the exploration rate 𝜖, actions in set
A were explored more than necessary. This bad exploration ismost likely making the network to be trapped in sub-optimal
solution [25].
The simplest solution is to apply lower weight for theactions in set A during the random exploration process. The
probability for an action being performed randomly by the 𝜖greedy policy is:𝜖𝜔𝑝(𝑎𝑖 ) = { 𝑎𝜖𝜔𝑏if 𝑎𝑖 ∈ A
if 𝑎𝑖 ∈ B

(15)

Where 𝜔𝐴 and 𝜔𝐵 are weights for the actions of set A andB. However, we observe that the agent trained by using thisCopyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018weighted 𝜖-greedy policy often inserts incorrect element into
the working image. A possible explanation for this problem isthat the 𝜖 value is already saturated when the agent does notlearn yet the long-term reward of adding the right element.
Prolonging the exploration phase does not improve theresult because the agent has to keep exploring more on
element editing (i.e. action set B) in order to discover better
solution. Our approach to solve this problem is to apply thedifferent reducing 𝜖 -greedy policies (hereafter referred asdual 𝜖-greedy policy) on each action set. In this setting, there
are two independent reducing 𝜖-greedy polices applied foreach action set:𝜖 𝜔𝑝(𝑎𝑖 ) = { 𝑎 𝑎𝜖𝑏 𝜔𝑏if 𝑎𝑖 ∈ A
if 𝑎𝑖 ∈ B

Maximum time step 𝑡𝑚𝑎𝑥
Number of actions 𝑛
output: Optimized policy 𝜃variables: 𝑀 experience memory
𝐿 training interval
𝑡 time step
𝑔𝑡 global counter
while not converged𝑌 ← random target image 𝑌 with random 𝑘
for 𝑡 = 0 to 𝑡𝑚𝑎𝑥
𝑔𝑡 ← 𝑔𝑡 + 1
𝑎𝐴 ← Rnd(A) ; 𝑎𝐵 ← Rnd(B)𝑎𝑄 ← max(𝑄𝜃 (𝑠𝑡 , 𝑎))𝑎𝑎𝑡 ← Rnd (𝑎𝐴 , 𝑎𝐵 , 𝑎𝑄 ∣ 𝑝(𝑎𝐴 ), 𝑝(𝑎𝐵 ), 𝑝(𝑎𝑄 ))unroll episode with action 𝑎𝑡
add 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 to 𝑀
if 𝑔𝑡 mod 𝐿 = 0
train mini batch taken from 𝑀
if 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 end
break𝑃(𝑎𝑖 ∣ 𝑠𝑡−1 ) =[𝐷(𝑠𝑡∣𝑎𝑖 , 𝑌) = d](17)

∑𝑗 [𝐷 (𝑠𝑡∣𝑎𝑗 , 𝑌) = d]Where 𝑃(𝑎𝑖 ∣ 𝑠𝑡−1 ) is the probability of action 𝑎𝑖 given thestate 𝑠𝑡−1 at time step 𝑡 − 1 , 𝐷(𝑠𝑡∣𝑎𝑖 , 𝑌) is the minimum
number of steps for the agent to finish the episode given in thestate 𝑠𝑡∣𝑎𝑖 at time step 𝑡 which is the result from action 𝑎𝑖
from the last time step.
Blank canvas

D. Policy-Gradient
As mentioned in Section II-C-2, in order to train the agent
using policy-gradient, 𝑅𝑡 has to be known. This value can beapproximated for training. Thus, it makes the efficiency of the
training process again depend on an external function.
Another solution is to fully unroll the episodes. Monte Carlo
method is commonly used for this purpose. However, thismethod is intractable in the environment with a large action
space, as it requires an immerse computational resource.
To overcome this difficulty, we unroll the episode and train
the agent reversively. As shown in Fig. 4, supposing that 𝑘 is
the minimum number of steps to compose the target image
from a blank canvas, if an agent is trained to perform last 𝑑

inputs:step from time step 𝑡, action at time step 𝑡 − 1 can be unrolled
in a brute force way:

Target image(16)

Where 𝜖𝐴 and 𝜖𝐵 are weight values of two 𝜖 -greedy
policies for the action set A and B, respectively. In this way,
the action set A and B can be independently explored.
Algorithm 1 shows the pseudo code to train the agent using
Q-learning paradigm with dual 𝜖-greedy policy, where therandom function has several forms (analogous to C++’sfunction overloading). The function Rnd(A) and Rnd(B)randomly select an action in set 𝐴 and 𝐵. Rnd(X ∣ W) picks
element from X with weight W . The difficulty 𝑘 is theminimum number of steps to compose the target image froma blank canvas as explained in Section IV-D.

ALGORITHM 1. Q-LEARNING BASED TRAINING ALGORITHM

1163

t
Current image

d

Steps

k

Fig. 4. The minimum number of steps to compose target image from a
blank canvas is 𝑘. The minimum number of steps to compose target image
from the current image is 𝑑.

Algorithm 2 shows the pseudo code of the policy-gradient
based training. The agent is trained to work on theincrementally difficult states. The difficulty of a state ismeasured by distance 𝑑 . Once the policy network 𝜃𝑑 for
distance 𝑑 is converged, it is then trained with more difficult
distance 𝑑 + 1. 𝑈(⋅) is an uniform sampling function.
The primary disadvantage of this method is that it is only
feasible with a controlled environment where the difficulty ofthe state can be calculated. However, it is useful in
combination with techniques such as transfer learning when
applying it for more complex data.
V. EVALUATION
A. Dataset
Works on image-processing based R2V conversion have
many diverse objectives. As the result, they mostly use
relatively plain and small datasets. The diversity in research
objectives also leads to the lack of unified evaluation dataset[15]. While the complexity of these data sets is suitable forthis research, their modest size and heterogeneous properties
are not adequate to be used to train DNN.
ALGORITHM 2. POLICY-GRADIENT BASED TRAINING ALGORITHM

inputs:Upper limit 𝑘𝑚𝑎𝑥
Number of actions 𝑛
output: Optimized policy 𝜃variables: 𝑀 experience memory
𝑔𝑡 global counter
𝐿 training interval
for 𝑑 = 1 to 𝑘𝑚𝑎𝑥
while θ𝑑 is not converged:𝑘 ← 𝑈([𝑑. . 𝑘𝑚𝑎𝑥 ])𝑡 ←𝑘−𝑑with random 𝑒𝑝𝑖𝑠𝑜𝑑𝑒 at difficulty 𝑘
𝑌 ← last state of 𝑒𝑝𝑖𝑠𝑜𝑑𝑒𝑠𝑡−1 ← state at time step 𝑡 − 1 of 𝑒𝑝𝑖𝑠𝑜𝑑𝑒for 𝑖 = 0 to 𝑛
if 𝐷(𝑠𝑡∣𝑎𝑖 , 𝑌) = 𝑑:add (𝑠𝑡−1 , 𝑎𝑖 ) to 𝑀
𝑔𝑡 ← 𝑔𝑡 + 1
if 𝑔𝑡 mod 𝐿 = 0
train mini batch taken from 𝑀
Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018(a) Circle-0(e) Circle-300(i) Square-100(b) Circle-50(f) Circle-400(g) Square-200(c) Circle-100(g) Square-0(k) Square-300(d) Circle-200(h) Square-50(l) Square-400

Fig. 5. Examples of images generated by training and evaluation
processes. The caption under each image shows the name of the element
and the minimum number of steps (𝑘 as in Fig. 4) required to compose thatimage from a blank canvas by using the editor.

With the above reasons, the training and the evaluation of
the agent have been done on a randomly generated data set. It
not only helps in avoiding the above-mentioned problems, butalso provides a controlled level of difficulty and theuniformity of the dataset. Target images are created byrasterizing the randomly generated SVG documents. These
documents contain a single shape element with difficulty 𝑘.
Fig. 5 shows examples of target images in format <element><k>.

1164

only interacts with the environment in every 𝑟 steps.
Therefore, once an action is decided by agent, it repeats 𝑟time steps. A popular value of 𝑟 in many tasks is 4 as it isusually a good tradeoff between the performance and thetraining speed.
In this research, dismissing frame skip helps to improve theperformance of the agent because one action repeated several
times makes the agent miss its target due to overshooting.
D. Parameter Update
The agent does not update its parameter at every time step,
but once for every 𝐿 experiences added in the replay memory.
Thus, given the batch size of 𝑁, one experience is learned by
the agent 𝑁/𝐿 times in average.
E. Settings
We have implemented the agent using the model proposed
in Section IV. We evaluate the performance of the agent
trained by different training schemes:• Policy-gradient• Q-learning under different exploration policies
o Conventional ϵ-greedy policy.
o Weighted ϵ-greedy policy.
o Dual 𝜖-greedy policy as.
Table II shows the hyperparameters used for theexperiments. Policy-gradient training experiments share thefirst 5 parameters with other experiments.

F. PerformanceFig. 6 shows the distribution of the evaluation scores of theagent under different training schemes. Similar to the training
configuration, the target images used for evaluation are
generated with random difficulty 𝑘 given that 𝑘 ≤ 400. The
box plot describes the distribution of evaluation scores by
B. Episode Termination
4
4
The episode is terminated if one of the following conditions 5 × 10 iterations interval from 0 to 40 × 10 . For eachscheme, we train the agent 5 times. Each time, we evaluate
is met:the agent after every 100 training iteration and collect• After a fixed time step 𝑡𝑚𝑎𝑥 . Since there is no further
step after the termination, for Q-learning, the 𝑄 value evaluation score calculated using equation (11). Thus, eachas mentioned in (6) at the final step 𝑡𝑚𝑎𝑥 is: box describes the distribution of 500 evaluation scores. The
colored box indicates IQR (interquartile range). The
𝑄𝜃 (𝑠𝑡𝑚𝑎𝑥 , 𝑎𝑡𝑚𝑎𝑥 ) = 𝑟𝑡𝑚𝑎𝑥 .
horizontal line within the box indicates the median score, and• When 𝑔𝑡 in (11) is greater than a certain threshold.
the extended bar indicates the maximum and minimum scores.
Dots indicate outliers. Our proposed dual 𝜖-greedy policy not
C. Frame Skip
only shows significant performance gain but also highly
Due to the computational demand of the agent, it isstable compared to conventional 𝜖 -greedy policy andinefficient to let the agent perform an action at every time step.
weighted 𝜖 -greedy policy during the training process. The
The popular solution is the frame skip [14] where the agent
agent trained by policy-gradient achieves the best result andthe high stability when the number of iterations is more than
TABLE IIor equal to 15 × 104 .
HYPER-PARAMETER SETTING FOR EXPERIMENTS
Minibatch size
Upper limit 𝑘𝑚𝑎𝑥
Train Interval 𝐿
Discount factor 𝛾
Memory replay size
max - min𝜖-greedy
start - end
max - min𝜖𝐴 -greedy
start - end
max - min𝜖𝐵 -greedy
start - end
𝑊𝐴
𝑊𝐵

Conventional
32
400
16
0.999
1e6
1 – 0.1
0 – 1e611

Weighted
Dual
32
32
400
400
16
16
0.999
0.999
1e6
1e6
1 – 0.1
0 – 1e6
1e-3 – 0
5e6 – 10e6
0.999 – 0.1
0 – 1e6
1e-3
1999e-3
1

G. Accuracy
With dual 𝜖-greedy policy, our trained agent favors adding
circle element and achieves higher score by editing circle
element in general. To further analyze this observation, we
evaluate each trained agents for 500 episodes with the samesetting used in Section V-F and drill down the evaluation
results. In general, evaluation results can be divided into twosets:• Circle set: consists of episodes with target images that
contain circle elements only (250 episodes for eachagent).
• Square set: consists of episodes with target images that
contain the square elements only (250 episodes for eachagent).
Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 20181165

inserted, the performance of all the trained agents are over 0.5.
This reflects the fact that all the agents are trained with asustainable amount of episodes in which the wrong element isinserted. This result is also correlated to action set A
performance shown in Table II: Because the agents trained bypolicy-gradient have high accuracy for action set A, theyrarely experience the training episodes where the wrong
element is added. Thus, on set 2, their performance is evenlower than the performance of agents trained by dual ϵ-greedy
policy (bolded italic v.s. italic on Table IV).

Policy-gradientQ-learning
Dual 𝜖-greedy
Weighted 𝜖-greedy
Conventional 𝜖-greedy

Circle set
0.99

Square set
0.95

0.94
0.67
0.53

0.76
0.58
0.61

To evaluate the agent’s performance on action set B. We
again evaluate each agent for 500 episodes with an element
already inserted. The evaluation results are divided into twosets:• Set 1: consists of episodes with target images that
contain the same element with the pre-inserted element(250 episodes each agent).
• Set 2: consists of episodes with target images that
contain an element that is different from the preinserted element (250 episodes for each agent).
Table IV shows the agent’s performance on action set B.
The evaluation scores of set 1 are in the grey background. For
this set, the performance of agents trained by policy-gradientare better than the agents trained by Q-learning.
The evaluation scores of set 2 are in white background.
Interestingly, for set 2, even with the wrong element preTABLE IV
AGENT PERFORMANCE ON ACTION SET B

Agent

Q-learning

Policy-Gradient
Dual 𝜖-greedy
Weighted 𝜖-greedy
Conventional 𝜖-greedy

Pre-insterted
CircleSquareCircleSquareCircleSquareCircleSquareTarget Image
CircleSquare
0.97
0.65
0.71
0.94
0.93
0.78
0.72
0.89
0.81
0.76
0.67
0.79
0.59
0.66
0.61
0.73

Potrace
AutoTrace

TABLE IIIAGENT PERFORMANCE ON ACTION SET A

TargetWith each set, we analyze the performance of the agent on
action set A and set B separately. The agent’s performance on
action set A is measured by the number of episodes where acorrect shape element inserted into the working SVG
document over the total number of episodes in the set.
Correctly insterting a shape is important because further
editing an incorrect element results in a large number of bad
experience that negatively affects the agent’s policy network.
Table III shows the performance on action set A of the agentstrained under different training schemes.

SvgAI

Fig. 6. Evaluation score distribution of the agent throughout the training
process under different training scheme.

H. SVG Quality
We visually compare the SVG image produced by ourtrained agent with two popular opensource and commercial
R2V solutions: Potrace and AutoTrace [26]. Fig. 7 shows thecomparison between the outputs. As shown in the figure,
Potrace not only has a problem on color quantization but also
results in distorted circle. On the other hand, AutoTrace
produces a much better result, however, the linear gradient fillhas been converted into color blobs. Without any manual
configurations prior to the conversion/drawing process, ouragent produces significantly better result.

Fig. 7. Comparision between SVG images produced by our agent (SvgAI),
Portrace and AutoTrace. SvgAI trained by Q-learning and policy-gradientproduces identical result for this example.

Not only visually better, but the SVG images produced by
our agent are also smaller both in file size and node counts.
As shown in Table V, SVG images produced by our agent are
40% smaller in size and 63% smaller in node counts than thesecond best solution in average over 100 images where each
one contains a single element, i.e. a circle or a square.
TABLE VAVERAGE SVG SIZE COMPARISON

File Size
Node Count
Color

Target
5.8KB≥ 16mil

SvgAI
852B
1.4
≥ 16mil

Potrace
1.4KB
2.7
2

AutoTrace
15.5KB174
10

VI. CONCLUSION
In this paper, we introduce a new paradigm to solve theR2V conversion problem. We propose an agent model and aframework to train the agent to use SVG editor by using Qlearning and policy-gradient. In order to successfully train theagent by using Q-learning, we divide the action space into two
sets and apply independent exploration policies on each

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

action set. Evaluation results show the efficiency of the
proposed dual 𝜖-greedy policy and policy-gradient. The SVG
image quality produced by our agent is also superior
compared to the popular software solutions. The problem ofincorrect shape detection as explained in Section V-G showsa weakness of Q-learning in applying it to this problem. While
policy-gradient produces the most efficient agent, it can only
be used for managed environments. For future works, we will
investigate and improve the training process for better reward
back-propagation and for better shape detection accuracy.
REFERENCES
[1]

[2]
[3]
[4]
[5]
[6]
[7]

[8]

[9]

[10]

[11]

[12]
[13]

[14]
[15]

[16]

[17]

[18]
[19]

[20]
[21]

[22]
[23]

[24]

D. Silver et al., “Mastering the game of Go with deep neuralnetworks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489,
Jan. 2016.
R. S. Sutton and A. G. Barto, Reinforcement learning: An
introduction, vol. 1, no. 1. MIT press Cambridge, 1998.
G. Lample and D. S. Chaplot, “Playing FPS Games with Deep
Reinforcement Learning.,” in AAAI, 2017, pp. 2140–2146.
Lin and Long-Ji, “Reinforcement learning for robots using neuralnetworks.” Carnegie Mellon University, 1992.
V. Mnih et al., “Human-level control through deep reinforcement
learning.,” Nature, vol. 518, no. 7540, pp. 529–33, Feb. 2015.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “PrioritizedExperience Replay,” Nov. 2015.
H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learningwith double Q-Learning,” Proceedings of the Thirtieth AAAI
Conference on Artificial Intelligence. AAAI Press, pp. 2094–2100,
2016.
Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N.
De Freitas, “Dueling network architectures for deep reinforcement
learning,” Proceedings of the 33rd International Conference on
International Conference on Machine Learning - Volume 48.
JMLR.org, pp. 1995–2003, 2016.
V. Mnih et al., “Asynchronous Methods for Deep Reinforcement
Learning,” in Proceedings of The 33rd International Conference on
Machine Learning, 2016, vol. 48, pp. 1928–1937.
A. Karpathy and L. Fei-Fei, “Deep Visual-Semantic Alignments for
Generating Image Descriptions,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 39, no. 4, pp. 664–676, Apr. 2017.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNetclassification with deep convolutional neural networks,” Commun.
ACM, vol. 60, no. 6, pp. 84–90, May 2017.
T. Beltramelli, “pix2code: Generating Code from a Graphical User
Interface Screenshot,” May 2017.
H. S. M. Al-Khaffaf, A. Z. Talib, and R. A. Salam, “Empiricalperformance evaluation of raster-to-vector conversion methods: A
study on multi-level interactions between different factors,” IEICE
Trans. Inf. Syst., vol. 94, no. 6, pp. 1278–1288, 2011.
G. Brockman et al., “OpenAI Gym,” Jun. 2016.
V. Lacroix, “Raster-to-Vector Conversion: Problems and Tools
Towards a Solution A Map Segmentation Application,” in 2009
Seventh International Conference on Advances in Pattern
Recognition, 2009, pp. 318–321.
R. Kansal and S. Kumar, “A framework for detection of linear
gradient filled regions and their reconstruction for vector graphics,”
2013.
K. Kawamura, H. Watanabe, and H. Tominaga, “Vector
representation of binary images containing halftone dots,” in 2004
IEEE International Conference on Multimedia and Expo (ICME)(IEEE Cat. No.04TH8763), pp. 335–338.
P. Selinger and P. Selinger, “Potrace: a polygon-based tracing
algorithm,” IN HTTP://POTRACE.SOURCEFORGE.NET, 2003.
X. Hilaire and K. Tombre, “Robust and accurate vectorization of line
drawings,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 6,
pp. 890–904, Jun. 2006.
R. Girshick, “Fast R-CNN,” in 2015 IEEE International Conference
on Computer Vision (ICCV), 2015, pp. 1440–1448.
M. Schuster and K. K. Paliwal, “Bidirectional recurrent neuralnetworks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–
2681, 1997.
S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,”
Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
W. R. Thompson, “On the likelihood that one unknown probability
exceeds another in view of the evidence of two samples,” Biometrika,
vol. 25, no. 3/4, pp. 285–294, 1933.
J. Asmuth, L. Li, M. L. Littman, A. Nouri, and D. Wingate, “A

1166

Bayesian sampling approach to exploration in reinforcement
learning,” in Proceedings of the Twenty-Fifth Conference on
Uncertainty in Artificial Intelligence, 2009, pp. 19–26.
[25] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement
learning: A survey,” J. Artif. Intell. Res., vol. 4, pp. 237–285, 1996.
[26] M. Weber, “Autotrace-converts bitmap to vector graphics.” 2004.
Anh H. Dang (S’09) received his bachelor degree inbusiness
administration,
information&
communication technology from Ritsumeikan Asia
Pacific University (Beppu, Oita, Japan) in 2010. He
then received the master degree in computer sciencefrom Waseda University (Shinjuku, Tokyo, Japan) in
2012. Since 2012, he is a Ph.D. candidate at Waseda
University. He is a member of IEEE, ACM, and
IEICE. His research interests are machine learning,
artificial intelligence, and computer vision.
Prof. Wataru Kameyama (M’86) received thebachelor’s, master’s, and D.Eng. degrees from the
School of Science and Engineering, Waseda
University, in 1985, 1987, and 1990, respectively. He
joined ASCII Corporation in 1992, and wastransferred to France Telecom CCETT from 1994 to
1996 for his secondment. After joining Waseda
University as an Associate Professor in 1999, he has
been a Professor with the Department ofCommunications and Computer Engineering, School of Fundamental
Science and Engineering, Waseda University, since 2014. He has been
involved in MPEG, MHEG, DAVIC, and the TV-Anytime Forum activities.
He was a Chairman of ISO/IEC JTC1/SC29/WG12, and a Secretariat andVice Chairman of the TV-Anytime Forum. He is a member of IEICE, IPSJ,
ITE, IIEEJ, and ACM. He received the Best Paper Award of NiwaTakayanagi in 2006, the Best Author Award of Niwa-Takayanagi in 2009
from the Institute of Image Information and Television Engineers, and theInternational Cooperation Award from the ITU Association of Japan in 2012.

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

1167

Cognitive Workload Detection from Raw EEGSignals of Vehicle Driver using Deep Learning
Mohammad A. Almogbel*, Anh H. Dang**, Wataru Kameyama****Department. of Computer Science and Communications Engineering,
Graduate School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan.
**GITS, Waseda University, Tokyo, Japan***Faculty of Science and Engineering, Waseda University, Tokyo, Japan
almogbela@ruri.waseda.jp, danghoanganh@akane.waseda.jp, wataru@waseda.jp
Abstract—Electroencephalography (EEG) signals have been
proven to be effective in evaluating human’s cognitive state
under specific tasks. Conventional classification models utilized
for EEG classification heavily rely on signal pre-processing andhand-designed features. In this paper, we propose an end-to-end
deep neural network which is capable of classifying multiple
types of cognitive workload of a vehicle driver and the context
of driving using only raw EEG signals as its input without any
pre-processing nor the need for conventional hand-designed
features. Data used in this study are collected throughout
multiple driving sessions conducted on a high-fidelity driving
simulator. Experimental results conducted on 4 channels of raw
EEG data show that the proposed model is capable of accurately
detecting the cognitive workload of a driver and the context ofdriving.
Keywords— Deep Learning, EEG, Neural Networks, Cognitive
Workload, Driving, Stress

I. INTRODUCTION
Maintaining safe levels of cognitive workload is extremely
crucial to ensure optimal performance and attention whilst
driving automobiles. Different methods have been widely
investigated to monitor driver’s cognitive workload including
heart rate monitoring [1][2], galvanic skin response [3][4],facial expression [5][6] and so on [7][8][9][10]. One of themost popular measures for assessing the mental state ofhumans is electroencephalography (EEG) signals. EEG has
reportedly shown success in evaluating the mental state suchas drowsiness [11], mind wandering [12] and alertness [13].
Despite being an active research area, understanding andapplying EEG signals are still limited due to the lack of true
understanding of the brain’s activities which prevents use ofgood EEG features from being engineered. Besides, EEG
signals are usually strongly affected by noise andinterference. Accurate reading requires a delicate equipment
and sealing.
Conventional analyzing approaches make extensive use of
Fourier transform in order to decompose EEG signal into

Manuscript received on Dec 31, 2017.
Mohammad A. Almogbel is with Graduate School of Fundamental Scienceand Engineering, Waseda University, Tokyo, Japan (Corresponding author,
email: almogbela@ruri.waseda.jp)
Anh H. Dang is with GITS, Waseda University, Tokyo, Japan. (email:danghoanganh@akane.waseda.jp)
Wataru Kameyama is with Faculty of Science and Engineering, Waseda
University, Tokyo, Japan (email: wataru@waseda.jp)

multiple frequencies. Only well-known frequency is thenused for feature design process. For example, the alpha band
(8 to 15[Hz]) is correlated with relaxing, and the beta band
(16 to 31[Hz]) associates with mental stress. Pre-processing
methods such as Butterworth bandpass filter and stationary
wavelet transform filters [14] are used to remove high andlow frequency noises [13]. Several efforts in applying deeplearning on EEG have been carried out [11][19][20][21].
However, all the previous researches still rely on signal preprocessing pipelines, and none come up with a system that
can handle raw data directly. Thus, valuable information
might be discarded during the pre-processing.
The purpose of this paper is to introduce an end-to-end
deep neural network model that can directly infer cognitive
workload and context of driving from raw EEG, where signal
pre-processing as well as conventional hand-designed
features are not required.
To evaluate the proposed model, EEG signal recordings
have been carried out on a subject driving a vehicle in a
relatively realistic simulated environment under several
different types of cognitive workloads and contexts ofdriving.
Our experiment shows that the proposed model canaccurately classify different labels of cognitive workload andthe driving context from raw EEG signals of vehicle driver ina simulated environment without the use of any conventional
hand-designed features.
This paper is organized as follows: Section II previews therelated researches. Section III explains the proposed model
used to classify the data. Section IV details the data collection
methodology, and the conducted experiments for evaluating
the proposed model. Section V summarizes the acquired
results, and finally the conclusion is given in Section VI.
II. RELATED WORKS
Monitoring electrical brain activity in a non-invasivemeans is referred to as EEG in clinical context. Brain
activities produce electrical charges caused by the neurons
inside the brain, and voltage fluctuations are measured using
conductive material called electrodes and a referenceelectrode attached to the head and scalp [15]. These voltages
pass through an amplifier for analysis. Thanks to recent
advancements, compact lightweight devices such as
MindWave Mobile Headset [16], Emotiv Insight 5 [17] andMuse [18] which is used in this study, have been introduced

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

1168

EEG
Conv1

Conv2

Conv3
Conv4
Conv5

N

4

4

4

Conv6

4
4

4

8
4
16

8

32

32

4

64

32
4
64

32

32

64

8

8

16

4

4

Conv7
Conv8

16

4

Fig. 1 Network Diagram. The blue cuboid denotes the four raw EEG channels while the gray cuboid inside denotes the slicing window. The white cuboid
denotes the kernel while the red cubes denotes the filters applied within each convolutional layer. All the configuration are detailed in Table II.

into the market to allow a more consumer friendly means tomonitor EEG signals.
Conventional approaches to classify EEG signals consistof decomposing the signals to extract features to be used forclassification. Recent studies of EEG analysis to utilize deeplearning with different pre-processing methodologies areshown in Table I.
In [11], FFT is used before classifying the signals to detect
driver’s drowsiness. [19] evaluates drivers’ cognitive
performance in a simulated environment using filteredfrequency EEG. Similarly, [20] also proposes a model thatcan predict right and left hand movements with frequency
filtered EEG signals. [21] applies a spatial filter before
classifying pathological from normal EEG recordings.
TABLE IRECENT STUDIES ON EEG ANALYSIS WITH DEEP LEARNING
Ref.
[19]
[11]
[21]
[20]

Pre-processing Feature-Extraction Classifier
Frequency filtering
FFT
Spatial Filter
Frequency filtering

CNN
N/A
CNN
CNN

1-layer ANN
1-layer ANN
1-layer ANN
1-layer ANN

Accuracy
86.06%
86.50%
84.80%
86.41%

III. PROPOSED MODEL
One of the major advantages of deep neural networks is theability to extract, learn and generalize features without preprocessing [22]. In this study, we propose the use of deep
convolutional networks to predict the cognitive workload ofa driver and the context of driving by using raw EEG signals
with no additional pre-processing.
CNN is inspired by biological visual system [23] where
different levels of visual hierarchy are handled by different
parts of the brain. Analogically, multi-layers of CNN
generalize visual features in multiple levels. Since each layerof CNN acts as a filter, it is expected that CNN can be also
used to extract high level features even in the case of EEG
signals.
The architecture of the proposed network is visualized inFig. 1. The network takes a vector of 4 × N as its input where
4 is the number of raw EEG channels collected from Musedevice and N is the length of the signal. Throughout the paper
we test the network with different lengths which is explained
in detail in section IV-A.
The model consists of several convolutional layers thattransform raw EEG data to feature vectors. These features arethen flattened and processed by 3 layers of fully connected(FC) classifier blocks.

The FC block outputs a two-, three- or six-dimensional
vector depending on the type of classification. Softmax
function is used to interpret this vector into the probability ofeach classification class. Except the final layer of FC block,all layers in this deep network utilize rectified linear unit
(RELU) [24] activation function. Batch normalization [25] isalso used after every RELU activation function.
Our CNN filters are applied to all EEG channels in thesame time. This method ensures that all possiblecombinations of features are captured. Small filter size andstride together with the high number of CNN layers also
ensure that features from multiple hierarchical levels are
extracted.
Different from images, EEG comprises multiple timeseries from each channel. For that reason, it is essential todesign CNN filters to stride along time direction only. Thus,
it must fully cover all other dimensions of input data.
Table II describes the network configuration for the
proposed model. The output shape of each layer is omitted
from the table since it is tested on different input sizes. Input
sizes are explained in Table III in Section IV-A.

Conv1
Conv2
Conv3
Conv4
Conv5
Conv6
Conv7
Conv8

FC1
FC2
FC3

TABLE IINETWORK CONFIGURATION
Convolutional Block
Filter Size
Strides
8×4×4
2
8×4×4
2
8×4×4
2
16×4×82
32×4×162
32×4×162
64×4×32
4
64×4×32
4
Flat-out
Fully-Connected BlockNo. units
Dropout
64
0.5
32
0.5
6/3/2
-

Activation
RELU → BN
RELU → BN
RELU → BN
RELU → BN
RELU → BN
RELU → BN
RELU → BN
RELU → BN

Activation
RELU → BN
RELU → BN-

IV. EXPERIMENT
A. Data Collection
1) EEGEEG recording has been done using Muse [18] headband.
Fig. 2 shows the international 10-20 system that is used todescribe the location of the electrodes on the head. The figure
also explains the electrodes used in this device where 4-

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

channel configuration has been utilized; two dry electrodeson the forehead AF7 (Left) and AF8 (Right), two behind theears TP9 (Left) and TP10 (Right), and a reference electrode(Fpz) in the middle of the forehead above the nasion. Muse iscapable of recording EEG signals at the sampling rate of
256[Hz]. Moreover, it is very portable, lightweight, easy touse, and it utilizes low-energy Bluetooth technology totransmit data making it feasible for real-life usages.

three levels; zero traffic (light), moderately dense traffic(medium) and high dense traffic (high). Table IV shows the
traffic flow and density for each level where traffic flow
denotes the average number of cars passing a single point perhour and traffic density denotes the average number of cars
within 1 kilometer of a single point.

Light
MediumHigh

Fig 2. Locations of Scalp Electrodes in the International 10-20 System [18]Raw EEG data are normalized by z-score for each of the 4channels. Slicing window approach is utilized to prepare thedata for training, where each slicing window is denoted by
the time segments of the EEG data. Table III explains thedifferent slicing window size used for training the model aswell as the shape of the input data for the network. The slicing
windows are overlapped with 1/256[sec] step.
TABLE IIIINPUT SHAPES
Raw EEG Slicing Window Size
256Hz × 180 sec × 4 channels
256Hz × 150 sec × 4 channels
256Hz × 120 sec × 4 channels
256Hz × 90 sec × 4 channels
256Hz × 60 sec × 4 channels
256Hz × 30 sec × 4 channels

Input Shape46080 × 4
38400 × 430720 × 4
23040 × 4
15360 × 4
7680 × 4

2) Driving Simulator
Data used in this study are collected through recordings
conducted on a popular driving simulation video game (GTAV free driving mode) [26] that is highly capable ofrepresenting the real life driving situations. GTA-V has
achieved an increasing popularity amongst recent studies thatutilizes driving simulators [27][28][29][30]. Other usedequipment consists of Logitech G27 driving wheel and an
ultrawide (21:9) curved monitor to better simulate naturalvision inside a real car. The experiment environment is shownin Fig. 3.

Fig 3. Representation of the used Simulation Environment with Examplesof City (Left) and Highway (Right).

3) Driving Sessions
The driving sessions were designed to accommodate broad
scenarios of multiple cognitive workloads and contexts ofdriving. The data collection was conducted by breaking down
the traffic flow/density inside the simulator environment into

1169

TABLE IV
EXPERIMENT CONDITIONS AND COMBINATIONS
Traffic
Type Combinations
Flow
Density
City
Highway
0 cars/min
0 cars/km
LC
LH
30 cars/min 22.5 cars/km
MCMH
36 cars/min 33.23 cars/km
HC
HH

For each traffic flow/density, city and highway recordings
were conducted, which means that the subject was advised todrive exclusively in both city and highway for each traffic
flow/density. By doing so, six different types of driving
scenarios are accumulated to evaluate the proposed model.
Table IV also shows the experiment conditions andcombinations with their abbreviations which are usedhereafter, such as LC for light traffic in city.
Twelve sessions were collected for each of the six types,
where one session was recorded for 15 minutes, which sums
up to 72 sessions and 18 hours of raw EEG signals sampled
at 256[Hz]. A total of 16,588,800 data points for each of thefour raw channels was collected.
The recordings were conducted on a span of two months
with random intervals and different times throughout the day.
In this experiment, one subject of male with age 29 wasjoined, who drives frequently with 12 years of driving
experience.
B. Types of Classification
To evaluate our proposed model, we considered thefollowing classification problems:
1) Cognitive Workload Classification:
In this type of classification, we are interested inclassifying the cognitive workload of the driver regardless ofthe driving context. In order to carry out this experiment, wegroup 6 types of driving sessions as explained in section IVA into 3 classes with regards to the traffic flow/density,- Low workload: LC, LH- Medium workload: MC, MH- High workload: HC, HH.
Traffic congestion is reportedly highly correlated withstress [31], and by grouping the sessions in such manner weinvestigate the correlation of EEG signals with thetraffic/flow density.
Each class in this experiment consists of 24 merged
sessions upon each traffic flow/density (5,529,600 × 4 data
points). The last session of each merged class is used forevaluation. By doing so, the real-life situation, where themodel requires to estimate the workload from future data bylearning previous data, can be simulated.
Slight modifications in the network are as follows; thenumber of the units in FC1 is decreased to 32 in slicing
window sizes of 90, 60 and 30 [sec]; it remains as the default
configuration shown in Table II in slicing windows of size
180, 150 and 120 [sec].
2) Context Classification:
In this type of classification, we focus on classifying the

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

1170

ccuracICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

raining
Evaluation

terationsFig. 4 Learning curve for training and evaluation of the model with the best performing slicing window size (150 [sec]) for workload (left), context(middle) and workload/context (right)

context of driving regardless of the cognitive workload of thedriver. Inferring the context of driving directly from EEG
signals can be helpful in analyzing the brain signals activity
during different context of driving. Therefore, we group ourEEG data into 2 classes: city driving and highway driving;- City: LC, MC, HC- Highway: LH, MH, HH
Each of the two classes yields 36 sessions (8,294,400 × 4
data points) for each of city and highway sessions. The lastsession of each merged class is used for evaluation. In thistype of classification, the network configuration is unchanged
from the default parameters as in Table II.
3) Context/Workload Classification:
Finally, we are interested in classifying the cognitive
workload of the driver as well as the driving context at thesame time. We assign a unique label to every cognitive
workload and context mentioned above to perform a six-way
classification for detecting both the cognitive workload of thedriver and the context of driving.
Each of the six types yields 2,764,800 data points for every
raw EEG channel. The last session for each type is used forevaluation. Slight modifications to the model are as follows;
for the slicing window size of 180[sec], the number of unitsin FC1 is increased to 128. And for the slicing window sizesof 60/30 [sec], the number of units in the FC1 is decreased to
32 instead of 64, and the strides for the 7th convolutional layer
is lowered to a stride of 2 down from a stride of 4. The rest
remains the same as the default configuration shown in TableII.

the training of the model throughout all the data. Input data is
divided into batches of 64. During the training the process
50% dropout is applied to the first and second FC layers.
To evaluate the model, a random batch of 64 and its
corresponding label is sampled from the evaluation set to be
used for cross-validation, and for every batch; recall,precision and accuracy scores are calculated.
V. RESULTS & DISCUSSION
Fig. 4 shows the learning curve for training and evaluationof the model for the best performing slicing window size of
150[sec] for all three types of classification.
1) Workload Classification
We train our model with data prepared using different sizesof slicing windows. Table V shows the evaluation result ofour proposed model for each window after the training
process. The bold type denotes the highest score among
different slicing window sizes. As shown in the table, themodel achieves the highest performance with larger slicing
window sizes. By average, 120[sec] is the optimal window
sizes for this experiment.
Furthermore, the network is capable of accurately
classifying low workload sessions, whereas high workload
data are sometimes misclassified as medium workload.
Session types are grouped together with regards to the traffic
flow/density, however the actual cognitive workload could
not be correlated in such manner in real life situations.
2) Context Classification
Similar to the workload classification, Table VI shows theevaluation results of our model on context classification. This
type of classification achieves the best overall results, and by
average, 150[sec] is the optimal window sizes for thisC. Training and Evaluation Parameters
The model is trained for 10 epochs using RMSProp
optimizer with a learning rate of 0.0001. An epoch represents

TABLE VWORKLOAD CLASSIFICATION METRICS FOR DIFFERENT WINDOW SIZES

Light

MediumHigh

Average

RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score

180

150

120

90

60

30

Avg.

0.997
1.000
0.999
0.999
0.833
0.997
0.950
0.908
1.000
0.884
0.951
0.938
0.943
0.960
0.967
0.948

0.979
0.996
0.991
0.987
0.921
0.878
0.923
0.899
0.871
0.926
0.932
0.898
0.923
0.933
0.949
0.928

0.960
1.000
0.984
0.980
0.905
0.950
0.953
0.927
1.000
0.900
0.969
0.947
0.955
0.950
0.969
0.951

0.928
0.982
0.970
0.954
0.915
0.673
0.834
0.776
0.627
0.912
0.855
0.743
0.823
0.856
0.886
0.824

0.936
0.987
0.976
0.961
0.890
0.867
0.912
0.878
0.912
0.900
0.933
0.906
0.912
0.918
0.940
0.915

0.791
0.982
0.926
0.876
0.974
0.673
0.838
0.796
0.749
0.986
0.908
0.851
0.838
0.880
0.891
0.841

0.932
0.991
0.974
0.959
0.906
0.840
0.902
0.864
0.860
0.918
0.925
0.881
0.899
0.916
0.934
0.901

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018

1171

TABLE VI
CONTEXT CLASSIFICATION METRICS FOR DIFFERENT WINDOW SIZESCity

Highway

Average

RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score

180

150

120

90

60

30

Avg.

0.885
0.926
0.996
0.905
0.927
0.890
0.996
0.908
0.906
0.908
0.996
0.906

0.954
0.939
0.998
0.947
0.937
0.957
0.998
0.947
0.946
0.948
0.998
0.947

0.862
0.862
0.985
0.862
0.886
0.886
0.985
0.886
0.874
0.874
0.985
0.874

0.918
0.887
1.000
0.903
0.886
0.921
1.000
0.903
0.902
0.904
1.000
0.903

0.886
0.935
0.995
0.910
0.946
0.901
0.995
0.923
0.916
0.918
0.995
0.916

0.903
0.901
0.981
0.902
0.890
0.896
0.981
0.893
0.896
0.898
0.981
0.897

0.901
0.908
0.976
0.905
0.912
0.908
0.976
0.910
0.907
0.908
0.976
0.907

experiment. The network can accurately distinguish between
city and highway driving which shows a high correlation
between the context of driving and the EEG signals.
3) Workload/Context Classification
Table VII shows the workload/context classification
results. Similar to context classification, the window size of
150[sec] achieves the best performance by average. However,in general the network performance slightly drops in this typeof classification. Since we train the network each classindividually, each class yields smaller number of samples totrain per class and the potential similarities between specifictypes of sessions such as MC and HH.
Low recall scores in LC and MH can be examined insmaller slicing window sizes due to the network
misclassifying them for MC or HC.
Discussion
In Table I in section II we show recent reseach that utilize
deep learning on EEG signals, however study does not
impose in any way a direct comparison with the distinguishedprevious works because the used data, experimental
conditions and classification targets are different in each, butrather explores and introduces the potential of using deep

CNN architecture to eliminate the need for pre-processing
and feature-extracion.
Nevertheless, our proposed model achieves high accuracy,
recall and precision scores on raw EEG signals without
applying any conventional pre-processing methodologies.
VI. CONCLUSION
Ever-proposed and conventional EEG analysis andclassification studies are heavily reliant on signal preprocessing and hand-designed features. Such methodologies
can be time consuming and potentially cause loss for viable
information during the process. In this paper, we propose an
end-to-end deep neural network which eliminates thenecessity for such pre-processing pipelines, whilst achieving
high classification performance.
Using only raw EEG signals from 4 channels as its input,
the proposed model performs highly robust and accurateclassification. The model is able to achieve an accuracy of
0.960 on average for the vehicle driver’s cognitive wor load
and the context of driving.
Future works include testing the proposed model withpublic available data sets. Further investigation will be

TABLE VII
CONTEXT/WORKLOAD CLASSIFICATION METRICS FOR DIFFERENT WINDOW SIZES
`
LC

LH

MC

MH

HC

HH

Avg.

RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score
RecallPrecision
Accuracy
F1 Score

180

150

120

90

60

30

Avg.

0.745
1.000
0.963
0.854
1.000
1.000
1.000
1.000
0.722
0.739
0.898
0.731
0.725
1.000
0.955
0.840
0.932
0.770
0.939
0.844
0.932
0.804
0.954
0.863
0.843
0.886
0.952
0.855

0.773
1.000
0.970
0.872
0.994
1.000
1.000
0.997
0.682
0.758
0.905
0.718
0.910
0.965
0.980
0.937
0.918
0.765
0.927
0.835
0.918
0.904
0.977
0.911
0.866
0.899
0.960
0.878

0.744
0.983
0.951
0.847
0.972
0.938
0.984
0.955
0.693
0.771
0.905
0.730
0.809
0.983
0.973
0.888
1.000
0.738
0.928
0.849
1.000
0.921
0.983
0.959
0.870
0.889
0.954
0.871

0.796
0.958
0.958
0.870
0.889
0.955
0.977
0.921
0.892
0.701
0.913
0.786
0.898
0.887
0.960
0.892
0.635
0.811
0.913
0.712
0.635
0.841
0.953
0.724
0.791
0.859
0.946
0.817

0.685
0.935
0.942
0.790
0.995
0.954
0.992
0.974
0.679
0.696
0.895
0.688
0.517
0.886
0.914
0.653
0.867
0.741
0.919
0.799
0.867
0.682
0.905
0.763
0.768
0.816
0.928
0.778

0.704
0.789
0.911
0.744
0.711
0.806
0.926
0.756
0.918
0.693
0.909
0.789
0.807
0.826
0.945
0.816
0.784
0.914
0.948
0.844
0.784
0.915
0.964
0.845
0.785
0.824
0.934
0.799

0.748
0.944
0.949
0.830
0.928
0.942
0.980
0.934
0.757
0.727
0.904
0.740
0.777
0.927
0.954
0.838
0.857
0.786
0.929
0.814
0.951
0.843
0.956
0.844
0.836
0.862
0.945
0.833

Copyright ⓒ 2018 GiRI (Global IT Research Institute)

ICACT Transactions on Advanced Communications Technology (TACT) Vol. 7, Issue 6, November 2018carried out by collecting more data from more subjects withdifferent driving experiences.
REFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]
[17]

[18]
[19]

[20]

[21]

[22]

N Munla, M Khalil, A Shahin, and A Mourad, “Driver stresslevel detection using HRV analysis,” 2015 Int. Conf. Adv.
Biomed. Eng. ICABME 2015, pp. 61–64, 2015.
B. Eilebrecht et al., “ he relevance of HRV parameters for driver
wor load detection in real world driving,” Comput. Cardiol.
(2010)., vol. 39, pp. 409–412, 2012.
L. Boon-Leng, L. Dae-Seok, and L. Boon-Giin, “Mobile-based
wearable-type of driver fatigue detection by GSR and EMG,”IEEE Reg. 10 Annu. Int. Conf. Proceedings/TENCON, vol. 2016–
Janua, pp. 1–4, 2016.
V Rajendra and O Dehzangi, “Detection of distraction undernaturalistic driving using Galvanic S in Responses,” 2017 IEEE
14th Int. Conf. Wearable Implant. Body Sens. Networks, BSN
2017, no. 2, pp. 157–160, 2017.
M Venturelli, G Borghi, R Vezzani, and R Cucchiara, “Deephead pose estimation from depth data for in-car automotive
applications,” Lect. Notes Comput. Sci. (including Subser. Lect.
Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 10188 LNCS,
pp. 74–85, 2018.
H Gao, A Yuce, and J P hiran, “Detecting emotional stressfrom facial expressions for driving safety,” 2014 IEEE Int. Conf.
Image Process. ICIP 2014, no. May 2015, pp. 5961–5965, 2014.
A Sahayadhas, K Sundaraj, and M Murugappan, “Detecting
driver drowsiness based on sensors: A review,” Sensors(Switzerland), vol. 12, no. 12, pp. 16937–16953, 2012.
S. F. Liang, C. T. Lin, R. C. Wu, Y. C. Chen, T. Y. Huang, and T.
P Jung, “Monitoring driver’s alertness based on the driving
performance estimation and the EEG power spectrum analysis ,”Conf. Proc. IEEE Eng. Med. Biol. Soc., vol. 6, pp. 5738–5741,
2005.
A. Palazzi, D. Abati, S. Calderara, F. Solera, and R. Cucchiara,“Predicting the Driver’s Focus of Attention: the DR(eye)VE
Project,” pp –25, 2017.
X Li, H Huang, and Y Sun, “Dri ri: An in-vehicle wirelesssensor network platform for daily health monitoring,” Proc. IEEE
Sensors, pp. 3–5, 2017.
I Bela hdar, W Kaaniche, R Djmel, and B Ouni, “Detecting
driver drowsiness based on single electroencephalography
channel,” 13th Int. Multi-Conference Syst. Signals Devices, SSD
2016, pp. 16–21, 2016.
C. L. Baldwin, D. M. Roberts, D. Barragan, J. D. Lee, N. Lerner,
and J S Higgins, “Detecting and Quantifying Mind Wandering
during Simulated Driving,” Front. Hum. Neurosci., vol. 11, no.
August, pp. 1–15, 2017.
L. Bi, R. Zhang, and Z. Chen, “Study on Real-time Detection of
Alertness Based on EEG,” 2007 IEEE/ICME Int. Conf. Complex
Med. Eng., pp. 1490–1493, 2007.
S S Daud and R Sudirman, “Butterworth Bandpass andStationary Wavelet Transform Filter Comparison forElectroencephalography Signal,” Proc. - Int. Conf. Intell. Syst.
Model. Simulation, ISMS, vol. 2015–Octob, pp. 123–126, 2015.
S. Donald L. and F. H. L. da Silva, Electroencephalography:
Basic Principles, Clinical Applications, and Related Fields.
Lippincott Williams & Wilkins, 2011.
NeuroS y, “MindWave Mobile Headset ,”5 [Online]
Available: https://store.neurosky.com/pages/mindwave.
EMO IV, “EMO IV Insight 5 Channel Mobile EEG - Emotiv,”
Emotive, 2018. [Online]. Available:
https://www.emotiv.com/product/emotiv-insight-5-channelmobile-eeg/.
Interaxon, “Muse: the brain sensing headband,” Tech. Specif.
Valid. Res. use, pp. 4–9, 2017.
M. Hajinoroozi, Z. Mao, T. P. Jung, C. T. Lin, and Y. Huang,“EEG-based prediction of driver’s cognitive performance by deep
convolutional neural networ ,” Signal Process. Image Commun.,vol. 47, pp. 549–555, 2016.
Z. Tang, C. Li, and S. Sun, “Single-trial EEG classification of
motor imagery using deep convolutional neural networ s,” Optik(Stuttg)., vol. 130, pp. 11–18, 2017.
R. T. Schirrmeister, L. Gemein, K. Eggensperger, F. Hutter, andBall, “Deep learning with convolutional neural networks fordecoding and visualization of EEG pathology,”
7
Y LeCun, K Kavu cuoglu, and C Farabet, “Convolutional
networ s and applications in vision,” IEEE Int. Symp. Circuits
Syst. Nano-Bio Circuit Fabr. Syst., pp. 253–256, 2010.

[23]

[24]

[25]

[26]
[27]
[28]
[29]

[30]
[31]

1172

K. Fukushima, “Neocognitron: A self-organizing neural networkmodel for a mechanism of pattern recognition unaffected by shift
in position,” Biol. Cybern., vol. 36, no. 4, pp. 193–202, 1980.
V Nair and G E Hinton, “Rectified Linear Units Improve
Restricted Boltzmann Machines,” Proc. 27th Int. Conf. Mach.
Learn., no. 3, pp. 807–814, 2010.
S Ioffe and C Szegedy, “Batch Normalization: Accelerating
Deep Networ raining by Reducing Internal Covariate Shift,”Feb. 2015.
“Gta V,”
3 [Online] Available:
https://www.rockstargames.com/V/.
Y Chen, W Li, and L Van Gool, “ROAD: Reality Oriented
Adaptation for Semantic Segmentation of Urban Scenes,”
7
P. Bazilinskyy et al., “Eye movements while cycling in G A V,”
no. January, pp. 7–11, 2018.
M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K.
Rosaen, and R Vasudevan, “Driving in the Matrix: Can virtualworlds replace human-generated annotations for real world
tas s?,” Proc. - IEEE Int. Conf. Robot. Autom., pp. 746–753,
2017.
M. Angus et al., “Unlimited Road-scene Synthetic Annotation(URSA) Dataset,”
D A Hennessy, D L Wiesenthal, and P M Kohn, “ heInfluence of Traffic Congestion, Daily Hassles, and Trait Stress
Susceptibility on State Driver Stress: An Interactive Perspective,”
Mohammad A. Almogbel (S’ ) received hisbachelor’s degree in Information Systems from King
Saud University, Riyadh, Saudi Arabia in 2009. He
joined King Abdul-Aziz City for Science andTechnology in Saudi Arabia as a researcher in the same
year and received a scholarship to complete his
graduate school in
He then received master’s
degree in computer science from Waseda University in2014 and he continued to pursue his Ph.D. since then.
He is a member of IEEE, ITS and JSAE.
Anh H. Dang (S’ 9) received his bachelor’s degree inbusiness
administration,
information&
communication technology from Ritsumeikan Asia
Pacific University (Beppu, Oita, Japan) in 2010. He
then received the master’s degree in computer sciencefrom Waseda University (Shinjuku, Tokyo, Japan) in
2012. Since 2012, he is a Ph.D. candidate at Waseda
University. He is a member of IEEE, ACM, and IEICE.
His research interests are machine learning, artificial
intelligence, and computer vision.

Wataru Kameyama (M’ ) received the bachelor’s,
master’s, and D.Eng. degrees from the School of
Science and Engineering, Waseda University, in 1985,
1987, and 1990, respectively. He joined ASCII
Corporation in 1992 and was transferred to France
Telecom CCETT from 1994 to 1996 for his
secondment. After joining Waseda University as an
Associate Professor in 1999, he has been a Professor
with the Department of Communications andComputer Engineering, School of Fundamental Science and Engineering,
Waseda University, since 2014. He has been involved in MPEG, MHEG,DAVIC, and the TV-Anytime Forum activities. He was a Chairman of
ISO/IECTC1/SC29/WG12, and a Secretariat and Vice Chairman of the TVAnytime Forum. He is a member of IEICE, IPSJ, ITE, IIEEJ, and ACM. He
received the Best Paper Award of Niwa-Takayanagi in 2006, the Best AuthorAward of Niwa-Takayanagi in 2009 from the Institute of Image Information
and Television Engineers, and the International Cooperation Award from the
ITU Association of Japan in 2012.

Copyright ⓒ 2018 GiRI (Global IT Research Institute)