Journal of Robotics, Networking and Artificial Life
Vol. 8(4); March (2022), pp. 293–297
DOI: https://doi.org/10.2991/jrnal.k.211108.013; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

A Fundamental Study on Control of CG Character
using Brain Wave Discrimination
Makoto Sakamoto1,*, Kenji Sakoma1, Taketo Kamasaka1, Kodai Miyamoto1, Amane Takei1, Tsutomu Ito2, Takao Ito3
Faculty of Engineering, University of Miyazaki, Miyazaki-City, Miyazaki 889-2192, Japan
National Institute of Technology, Ube College, Ube-City, Yamaguchi 755-8555, Japan
3Graduate School of Advanced Science and Engineering, Hiroshima University, Higashi-Hiroshima, Hiroshima 739-8527, Japan
12ARTICLE INFOABSTRACT

Article History

Virtual Reality (VR) has grown with the development of computer Graphics (CG). In recent years, VR technology is expected todevelop in various fields such as medical, welfare, education, training, tourism, business, entertainment, and so on. This paper
aims at intuitive operation by focusing on troublesome mounting in VR. When using VR equipment, it is required to set up manycables and sensors. In addition, when using it, there must always have a certain space around the user, because we always use
the controller when operating in the world of VR, for reasons of care or for the safety of the user. Therefore, the location must
be selected when the VR device is used. Furthermore, we also verified whether the VR space can be operated just by thinking
in the head of a device that is economic than the device used in the existing study. As a method to solve this issue, we propose
manipulation method of CG character by Electroencephalography (EEG). We use the brain–machine interface to measure brainwaves, the fast Fourier transform and the support vector machine as the methods of EEG analysis in this paper. In the future, wewill continue our study using different approaches of EEG analysis and new conditions by changing the method of EEG analysis
and other conditions.

Received 13 November 2020
Accepted 06 May 2021
Keywords
Brain–machine interface (BMI)
computer graphics (CG)electroencephalography (EEG)
the fast Fourier transform (FFT)
support vector machine (SVM)
virtual reality (VR)© 2021 The Authors. Published by Atlantis Press International B.V.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Until now, Computer Graphics (CG) has only been to create andview works. On the other hand, CG in the future will be able tocreate more realistic images. Furthermore, we will be able to enter in
the images, and touch, smell, and taste the images, and will become
more familiar. Therefore, CG has become more and more important
to cooperate with many fields such as physics, chemistry, earth science, biology, life science, medicine, psychology, cognitive science,electrical and electronic engineering, information and communication technology, computer, civil engineering, architectural engineering, mechanical engineering, robotics, marine engineering, spaceengineering, art, entertainment, and so on. And, the bridge between
them is the concept of artificial reality, Virtual Reality (VR), andtele-existence. In particular, VR is a technology that enables humans
to experience a virtual environment other than the one in which they
actually exist, as if it were a real environment, and to act in that virtual world. Artificial reality has the same meaning.
Virtual reality has been studied since the 1960s. However, since2016, the commercialization of more inexpensive and highperformance VR equipment has rapidly increased, so 2016 is called
the “first year of VR”, and VR technology has been studied anddeveloped all over the world and has made remarkable progress.
Corresponding author. Email: sakamoto@cs.miyazaki-u.ac.jp*

Various companies have begun to develop and sell VR equipment,
and although it is a little expensive, even ordinary people can easilypurchase it. VR technology has become widespread, as PCs andother devices that can use VR devices have become affordable evenfor individuals. As a result, VR technology is expected to develop
in various fields such as medical, welfare, education, training, business, entertainment, and so forth. However, there are some major
challenges in the development of VR technology.
We paid attention to the troublesomeness of wearing in this. Whenusing VR equipment, it is often necessary to set up cables and sensors
in advance. Also, when using a VR device, it is necessary to have acertain amount of space around the user. This is to ensure the safetyof the surroundings and the user himself/herself because the controller is always used when operating/moving in the world inside VR.
Therefore, there is a problem that the place where the VR device canbe used is limited. It tends to be more expensive than models that
require cables, but recently, integrated (standalone) models that do
not require cables have also been released. Therefore, the problem
of setting up the cable or the like can be solved. However, the spatialproblem when using VR equipment has not been solved.
Therefore, to try to solve the spatial problem, the operation bybrain waves—Brain–Machine Interface (BMI) is used. Since thefrequency band of human brain waves, such as “a wave” and “bwave”, fluctuates depending on the activity state, the purpose is touse them for controlling CG characters and robots by reading themM. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 8(4) 293–297

294with a device and discriminating them. It is a study. I think that it
may be possible to solve the spatial problem when using VR equipment by using this method [1–4].

2.4. Library•• libStreamSDKUsed to receive brainwave data from Mind Wave Mobile in
Visual Studio 2019.

2. METHOD OF STUDY•• MathNet.Numerics

2.1. Virtual Reality•• Accord

Virtual reality is the technology and system that creates an environment that is not the actual thing or the real thing but has thesame essence as a function by stimulating the senses including thefive senses of the user. In the olden days, novels, paintings, plays,television, etc. also had more or less VR functions.
Several problems have been identified with VR technology, andthese are problems that must be resolved for future development.
(1) Troublesome installation problem(2) Spec problem(3) Social and market issues(4) Problems at the production site(5) Health issues
The above-mentioned “installation troublesome problem” includes
a safe space problem that must be ensured in order to use the VR
product that occurs when the VR product is used. This problem
means that you can’t play anywhere when using a VR product.
This study focused on this problem.

•• Accord.MachinerLearning•• Accord.Math•• Accord.Statistics
Used to process data received from Mind Wave Mobile in Visual
Studio 2019.

2.5. Brainwave Measurement
Use NeuroSky’s Mind Wave Mobile brain wave sensor to measurebrain waves. In addition, to receive brain wave data in Visual Studio
2019, we used a library called libStreamSDK included in the development tool provided by NeuroSky.

2.6. EEG Processing Method
2.6.1. Fast Fourier Transform
It is a type of Discrete Fourier Transform (DFT) and is an algorithm
devised so that DFT can be executed at high speed on a computer[see Equation (1)]. When using this analysis method, the MathNet.
Numerics library was used [5].

2.2. Used Equipment
Equipment is sold by NeuroSky (see Figure 1). In this study, we use
this device to measure brain waves.

2p

N -1-i		  
X k = åxne N  (1)n =02.3. Development Environment

2.6.2. EEG feature discrimination method

Development environment is as shown in Table 1.

For EEG characterization, Support Vector Machine (SVM),
which is generally used for EEG characterization, was used.
SVM is one of the pattern recognition models that uses supervised learning [see Equation (2)]. SVM has high discrimination
performance for untrained data. In this study, a Gaussian kernel[see Equations (3) and (4)] was used as the kernel function. Inaddition, five libraries of MathNet.Numerics, Accord, Accord.
MachinerLearning, Accord.Math, and Accord.Statistics wereused to use SVM.

Figure 1 | Mind wave mobile.
Table 1 | Development environment
OSWindows 10

Programming language
Measuring equipment
Software

C#Mind Wave Mobile
Visual Studio 2019
Unity 2019.2.18f		 y(x ) = sgn  ∑ wn K (xn , x ) + b n=SV(2)x ∈ class A
 (3)x ∈ class B

0=		 12

	  K (x1 , x2 ) = exp  − || x1 −2x2 ||   (4)2s

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 8(4) 293–297

2.6.3. Visual Studio 2019
Visual Studio 2019 processes the brain wave data read by Mind
Wave Mobile and performs feature detection. At this time, FFT
is used to detect the feature amount of the electroencephalogram
data. Then, using SVM, the processed brain wave data is discriminated so that 0 is output when the brain wave is relaxed and 1 isoutput when the brain wave is conscious of moving.

2.6.4. Unity 2019.2.18f
We created CG characters and programs that act according to thecharacteristics of the read brain waves. The brain wave data wasprocessed by Visual Studio 2019, the result of discrimination by
SVM was received, and the CG character was created to act according to the discrimination result. We use “Unity-Chan!” as the CG
character, which is provided free of charge in Unity’s Asset Store(see Figure 2).

3. EXPERIMENTAL METHOD
We have the subject sit in a chair, wear Mind Wave Mobile on thehead, and then start measuring brain waves. During the experiment, the subjects are made aware of relaxing and moving, andevaluate how the CG character in Unity moves accordingly. In
this study, there are only two types of CG character movements:
stop when you are relaxed and keep moving forward while you arethinking of moving your body.

4. EVALUATION EXPERIMENTIn this experiment, three university students will be the subjects.
An evaluation experiment was conducted and a questionnaire
was conducted. The evaluation contents were “good points”, “badpoints”, and “others” in a free description format.

5. EVALUATION RESULTThe results of the questionnaire are as follows.
In “Good points,” “It is fresh and interesting to move just by thinking” was mentioned. On the other hand, “Bad points” are “I wasworried because there was a little time from thinking until theoperation was reflected in the CG character,” “Mind Wave Mobile
is uncomfortable to wear,” and “The action that the CG character is thinking There were times when it did the opposite of whatFigure 2 | The CG character which used in Unity.

295it was.” Moreover, “Others” included “Can I do anything other thanmove forward?” and “Can I use running properly when walking?”“Other” were that “is it not compatible with other movements?”
and “is there no other CG character?”, etc.

6. CONSIDERATION
From the “good points” of the evaluation results, it was foundthat the control of the CG character by brain waves, which is thepurpose of this study, can obtain a certain result in distinguishing
only two patterns of forward and stop.
From the “bad points”, it was found that there was a problem withthe accuracy and speed of EEG discrimination, and it was foundthat it was necessary to improve the EEG feature detection methodand discrimination method, and to improve the performance ofthe processing device. It was also found that there was a problem
with the installation of Mind Wave Mobile, but it is thought thatthe problem with this is that the size of the product used this timecannot be adjusted. From the above “bad points”, it was found that
there are points that should be further improved in this study.
From the opinion of “Other”, it was found that actions other thanadvancing and stopping are necessary in this study.
Moreover, we would like to increase the number of subjects of theexperiment in the future.

7. CONCLUSION
In this study, we tried to solve the space problem around the user,among the “troublesomeness of wearing” which has become anissue in the development of VR technology. The brain wave data
was measured by Mind Wave Mobile, which is superior to conventional equipment concerning the compactness and the ease of handling. The brain wave data from Mind Wave Mobile was received
and processed by Visual Studio 2019, and the characteristics of thebrain wave data processed by SVM were determined. We sent thediscrimination result to Unity and verified the method in the field
called BMI, which manipulates the characters in Unity based on thediscrimination result.
From the results of the evaluation experiment, it was found that
although the operation of the CG character by brain waves canobtain certain results, the accuracy is not sufficient and there aremany points to be improved. Future tasks include improving theaccuracy of brain wave feature detection and brain wave feature
discrimination, increasing character movements by discriminating the ratio of brain wave frequency bands, and improving theuncomfortable wearing of Mind Wave Mobile. It is thought that
there is, and we plan to work on it in the future.
If this study progresses, it is expected that it will play a major rolein the medical field. By using this technology for people with physical disabilities who are physically handicapped or who are using
artificial limbs, it will be possible to operate the body only withbrain waves, which will be much more convenient than it is now. Inaddition, the field of e-sports is currently developing in the world,and VR technology is attracting attention in that field as well, so
it is expected that if this study progresses, it will be able to play anactive role in the field of e-sports.

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 8(4) 293–297

296

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

[3]

REFERENCES

[4]

[1]
[2]

T. Ishizu, K. Sakoma, M. Sakamoto, Fundamental study oncontrol of CG characters by electroencephalography (EEG) analysis, Mem. Faculty Eng. Univ. Miyazaki 48 (2019), 159–162.
Visual Industry Promotion Organization, 2017 Advanced ContentTechnologies, etc. Distribution Promotion Business Report,[5]Part I, guidelines for using content production technology suchas VR, 2018 [Online].
T. Hidaka, Y. Shiraishi, Y. Kawai, J. Okuda, Information control
system based on EEG using smartphone, B4-3, Information
Processing Society of Japan, Japan, 2014, pp. 480–485.
H. Uchida, K. Fujimura, Y. Maegaki, A Study on Digitizing
Process of Brain Wave Data for the EEG Analysis, 30th Fuzzy
System Symposium, SOFT (Japan Society for Fuzzy Theory andIntelligent Informatics), Kochi, 2014, pp. 246–251 (in Japanese).
J. Sakamoto, Y. Mori, T. Sekioka, Probability analysis method bydiscreate fast Fourier transform, J. Struct. Constr. Eng. Trans. AIJ
60 (1995), 39−45.

AUTHORS INTRODUCTION
Prof. Makoto Sakamoto

Prof. Amane Takei

He received the PhD degree in computer science and systems engineering from Yamaguchi
University. He is presently an Associate
Professor in the Faculty of Engineering,
University of Miyazaki. He is a theoretical computer scientist, and his current main researchinterests are automata theory, languages and computation. Heis also interested in digital geometry, digital image processing,computer vision, computer graphics, virtual reality, augmentedreality, entertainment computing, complex systems, data science, and so forth.

He is working as Associate Professor for
Department of Electrical and Systems
Engineering, University of Miyazaki, Japan. His
research interest includes high performancecomputing for computational electromagnetism, iterative methods for the solution ofsparse linear systems, domain decomposition methods for largescale problems. He is a member of IEEE, an expert advisor of
The Institute of Electronics, Information and Communication
Engineers (IEICE), a delegate of the Kyushu branch of Instituteof Electrical Engineers of Japan (IEEJ), a director of Japan
Society for Simulation Technology (JSST).

Mr. Kenji Sakoma
He is a master student at Department ofComputer Science and System Engineering,
University of Miyazaki. His current researchinterests are computer graphics, image processing, and VR technology.

Mr. Taketo Kamasaka
He is a master student at Department ofComputer Science and System Engineering,
University of Miyazaki. His current researchinterests are computer graphics, entertainment
computing, and so on.

Mr. Kodai Miyamoto
He is a master student at Department ofComputer Science and System Engineering,
University of Miyazaki. His current researchtopic is a fundamental study on educational
support using VR technology.

Prof. Tsutomu Ito
He is an Assistant Professor in Department of
Business Administration, Ube National collegeof technology. He has published many papers
in refereed journals and proceedings, particularly in the area of industrial management, andcomputer science. His current research interests
include internet of things (IoT), mechanical engineering, artificial intelligence (AI), automata theory, quantitative analysis ofJapanese Keiretsu. He was one of the winners of the Best Paper
Award in the International Conference on Artificial Life andRobotics (ICAROB) in 2015 and 2016, respectively. He earned
his Doctor degree of Engineering from Hiroshima University,Japan in 2018.

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 8(4) 293–297

Prof. Takao Ito
He is Professor of Management of Technology(MOT) in Graduate School of Advanced Scienceand Engineering at Hiroshima University. He
is serving concurrently as Professor of Harbin
Institute of Technology (Weihai) China. He has
published numerous papers in refereed journalsand proceedings, particularly in the area of management science, and computer science. He has published more than eight
academic books including a book on Network Organizations
and Information (Japanese Edition). His current research interests include automata theory, artificial intelligence, systems control, quantitative analysis of inter-firm relationships using graphtheory, and engineering approach of organizational structuresusing complex systems theory.

297