1B2-03
This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere.

第 28 回日本バーチャルリアリティ学会大会論文集（2023 年 9 月）

Enhancing Social and Physical Interaction of an ALS
Patient using extra robotic arms controlled by BMI周 嵩晨 1) ， 安藤 良一 1) , 川口 碧 1) ，Mark Armstrong1) ，Giulia Barbareschi１) , 荻野 幹人 2) , 武藤 将胤 3) , 南澤 孝太 1)Zhou Songchen, Ryoichi Ando, Midori Kawaguchi, Mark Armstrong, Giulia Barbareschi, Mikito Ogino,
Masatane Muto, Kouta Minamizawa
1) 慶應義塾大学大学院メディアデザイン研究科（〒 223-8526 横浜市港北区日吉 4-1-1，zhousongchen，kouta@kmd.keio.ac.jp）

2) 株式会社電通サイエンスジャム
3) 一般社団法人 WITH ALS
概要:Patients with ALS have diﬃculty expressing themselves through body language, which is

the main means of communication during social participation. In this study, we have developed
a human augmentation technology for ALS patients to control alternative arms using BMI and atouch sensor to have social interaction with others.
キーワード： Human Augmentation, Disability, Art& Entertainment, Communication

1.

Introduction

The integration of touch sensors to initiate mechanicalCommunication, both verbal and nonverbal, is crucial

arm movements addresses the issue of the non-real-time

for human interaction. However, individuals with Amy-nature of BMI signals and the inherent latency in ob-otrophic Lateral Sclerosis (ALS) progressively lose theirtaining these signals. This latency presents a challenge

ability to speak and control their body movements[1],for users in accurately timing the robotic arm’s actions.

greatly aﬀecting their communication abilities. StevenHence, our system emulates the natural progression of[2] has shown that communication plays a critical role inhuman movement, from the brain’s conception of action

improving the mental health of ALS patients.

to the neural signals controlling muscle contractions toBrain-Machine Interface (BMI) has emerged as an es-

execute it. In this model, the BMI simulates the brain

sential communication tool for these individuals, espe-

conceiving an action, while the touch sensors mimic thecially when ALS progresses to a state known as total

neural signals triggering muscle contractions to initiate

locked-in syndrome (TLS). Notably, many ALS patientsmovements.
This design not only enhances the users’ sense of agency

hope for BMI to aid in communication and emergency

but also allows more precise control over the mechanicalalerting[3].
Several studies[4][5][6] have utilized BMI technology

arm’s actions. As the disease progresses, the system isto enhance the communication abilities and control overcapable of adapting to changing needs, transitioning torobotic arms and wheelchairs of ALS patients. However,full BMI control. Thus, our research aims to elevate thethere is a research gap in addressing the crucial role ofquality of life for ALS patients through this novel com-body language in communication, highlighted by the 7%-munication method.

38%-55% rule[7].
In our research, we devised an innovative robotic avatar
control system combining BMI and touch sensor to com-

2.

Methods
This section describes the overall structure and oper-

mand the movements of a robotic arm, enabling commu-

ational workﬂow of the system.

nication through body language for ALS patients. This

2.1

system oﬀers a viable solution, particularly for patientsin the early to mid-stages of ALS, where some degree ofﬁnger movement control is still retained.

Ⓒ 2023 日本バーチャルリアリティ学会

Movements DesignMovements Recording:Method 1: By recording the trajectory of the top position of the robotic arm and using the humanoid arm-1B2-03-1B2-03

Inverse Kinematics (IK), the entire arm trajectory canbe deduced through reverse inference.
Method 2: By utilizing the animation system, the positions and rotation data of the seven joints of the roboticarm are recorded to capture the arm’s motion trajectory.
Considering the diﬃculty of accurately capturing thearm’s motion trajectory using IK and ensuring user safety,
we adopt the second method.

Fig.1: Illustrations of Performance and Daily Life
Movements

Fig.3: Fusion 4 Robot Arm

In this project, taking into account the ALS patient’s
profession as a DJ, in order to enhance his social interaction, the designed actions include performance gestures
such as waving, clapping, and DJ movements. Additionally, everyday actions like handshaking, taking photos
gestures, and greeting gestures are also included. (Fig. 1shows)
2.2

System Components
Fig.4: Hackberry Robot Handto Fusion 4 and controlled using the pins on Fusion 4. In
this project, we use Touch Designer to send commands toUnity, and Unity scripts relay the information to the pinsof Fusion 4, thereby controlling the hand movements.

Fig.2: BMI Device
The system utilizes an OpenBCI BMI device (Fig.2
shows), programmed by Dentsu ScienceJam Inc., to enable user to perceive sounds from the left, center, andright directions. During the sound presentation, the user
focuses on the left half of the body, the center, and theright half of the body, respectively. The BMI device thenoutputs signals of 1, 2, and 3, corresponding to the left,center, and right directions.
The system employs the Fusion 4 robotic arm (Fig.3shows) from avatarin Company, which can be controlled
Fig.5: Touch Sensor

using the Unity animation system. In this project, we
utilize Touch Designer to send commands and switch animations in Unity, thereby controlling the robotic arm.

To detect touch, an ESP32 Arduino board and a touch

Additionally, the system incorporates the Blackberry

sensor are used as a touch system (Fig.5 shows), and it

Hands robotic arm (Fig.4 shows), which can be connectedsends information to Touch Designer through the OSCⒸ 2023 日本バーチャルリアリティ学会-1B2-03-1B2-03

signal. When the user’s ﬁnger leaves the touch sensor, it

3.

triggers the movement of the robotic arm.
2.3

ResultAs of the submission of this paper, due to the under-

Usage Workﬂow

developed state of the Unity animation control system,
we did not allow ALS patients to use the system to control the Fusion 4 robot for limb language communication
experiments to ensure the safety of the experimenters.

Fig.6: Workﬂow
Fig.7: Robot Control TestFirstly, based on user requirements, a variety of motion trajectories for the robotic arm are designed andHowever, we utilized the Oculus Rift and controllers,

recorded, encompassing actions such as waving, clapping,and under controllable safety conditions, we operated theDJ-ing, handshaking, photo-taking gestures, and greet-Fusion[8] robot, allowing ALS users to experience en-ing.

hanced human interaction using the robotic arm (Fig.7

A patient with ALS is selected as the experimentalshows). The users found this experience intriguing.

subject, who operates the robotic arm via the BMI device
and touch sensor.
When the ALS patient hears sounds from the left, center, or right, they choose the action they want to activate
and transmit signals to Touch Designer via the BMI device.
Upon receiving the data, Touch Designer selects theinformation to be sent to Unity through OSC signal andwaits for the activation signal from the touch sensor.

Fig.8: Touch Sensor Test

When the user wants to trigger the action, the user removes their ﬁnger from the touch sensor. The touch sen-Furthermore, we also enabled users to utilize a touchsor then sends activation information to Touch Designersensor (Fig.8 shows). Once an action was selected us-via the OSC. Touch Designer subsequently transmits theing BMI signals, a slight removal of the ﬁnger from theexecution information, such as action instructions andtouch sensor would activate the mini robot arm. This

execution speed parameters, hands gesture command, toexperiment demonstrated the feasibility of the system.

Unity via OSC.

Compared to triggering actions without the touch sensor,

Unity, upon receiving this, switches actions and ad-

where the robotic arm would immediately move upon re-justs the speed of action execution through its animation

ceiving the BMI signal, users reported improved control

system, and uses Fusion 4 SDK to send signals to pins toover the robotic arm.

move Hackberry Hands. Therefore, achieving the eﬀectof the user using the BMI to control the robotic arm for4.

Discussion
We acknowledge several limitations in our research.

nonverbal communication.
The aforementioned process is illustrated in the Fig.6.

Firstly, real-time recording of actions is not supported
by our system, requiring users to pre-program each gesture in our experiments. This poses a signiﬁcant challenge, particularly for ALS patients. Secondly, we have

Ⓒ 2023 日本バーチャルリアリティ学会-1B2-03-1B2-03

not conducted experiments involving user interaction forreceiving BMI signals without the touch sensor, users re-non-verbal communication with others, limiting our un-

ported a greater sense of control when utilizing the touch

derstanding of the system’s practical application in socialsensor.

contexts.

This system held signiﬁcant potential in enhancing com-To address these limitations and enhance our system,
we propose the following improvements:

munication abilities, particularly for individuals aﬄictedwith ALS and similar conditions.

Developing a customizable K-frame system: We areAcknowledgment

considering the development of a K-frame system that al-This work supported by JST Moonshot R&D Programlows users to customize the motion trajectory and modes”Cyberneticbeing” Project. (Grant numberJPMJMS2013)of the robotic arm using eye-tracking technology. ThisReference

customization will enhance the ﬂexibility and personal-[1] Lewis P Rowland and Neil A Shneider. Amyotrophicization of the system, better meeting the individual needs

lateral sclerosis. New England Journal of Medicine,of ALS patients.
Conducting experiment: We plan to involve volunteers

344(22):1688–1700, 2001.
[2] Steven Laureys, Frédéric Pellas, Philippe Van Eeck-in interacting with the robotic arm controlled by users.

hout, Soﬁane Ghorbel, Caroline Schnakers, Fabien

Through visits, interviews, surveys, and questionnaires,Perrin, Jacques Berre, Marie-Elisabeth Faymonville,

we will evaluate the system’s potential to enhance users’

Karl-Heinz Pantke, Francois Damas, et al.

communication abilities. We will also analyze the dif-locked-in syndrome: what is it like to be conscious

ferences between robotic arm non-verbal communication

but paralyzed and voiceless?

and natural human non-verbal communication from an
observer’s perspective, seeking ways to improve it.

TheProgress in brain re-search, 150:495–611, 2005.
[3] Yu Kageyama, Xin He, Toshio Shimokawa, Jinichi

Making movements based on text and tone: With ad-Sawada, Takufumi Yanagisawa, Morris Shayne, Os-vancements in AI and eye-tracking technology, patients

amu Sakura, Haruhiko Kishima, Hideki Mochizuki,

can now input text using their eyes and generate speech

Toshiki Yoshimine, et al. Nationwide survey of 780

with voice tone and emotions based on AI-trained voicejapanese patients with amyotrophic lateral sclerosis:

data. However, control over physical gestures is lacking.

their status and expectations from brain–machine in-To address this, we aim to enhance patients’ expressive

terfaces. Journal of Neurology, 267:2932–2940, 2020.
[4] Mikito Ogino and Yasue Mitsukura. An eeg-based

capabilities by integrating text input, selected voice tone,

robot arm control to express human emotions. Inand emotions to control the robotic arm.
Through these improvements, our goal is to enhance2018 IEEE 15th International Workshop on Ad-

the functionality, usability, and practical applicability ofvanced Motion Control (AMC), pages 322–327. IEEE,

our system, ultimately beneﬁting ALS patients in real-

2018.
[5] 加納尚之. 完全閉じ込め症候群 (tls) となった als 患者

world scenarios.のコミュニケーション-脳波 (erp) を用いた android ス

5.

マートフォンアプリの開発. 看護理工学会誌, 6(2):63–69,

Conclusion

signals and the Fusion 4 robotic arm, our study incorpo-

2019.
[6] Sanjaya Mallikarachchi, Dulith Chinthaka, Janith

rated a BMI device to capture users’ brainwave signals.

Sandaruwan, Isuru Ruhunage, and Thilina Dulan-

These signals were processed using Touch Designer andtha Lalitharatne.

transmitted to the Unity system to determine the appro-

based brain machine interface (bmi) for a mobile

priate actions for the robotic arm and hand. Addition-

robotic assistant (mra). In 2019 IEEE 19th Inter-

ally, the system awaited input from a touch sensor, which,national Conference on Bioinformatics and Bioengi-

upon activation, prompted the robotic arm and hand toneering (BIBE), pages 812–816, 2019.
[7] Dipika S Patel. Body language: An eﬀective commu-Building upon previous advancements in decoding BMI

perform physical gestures, facilitating bodily interaction

nication tool. IUP Journal of English Studies, 9(2),with others.
During the experiments, a patient with ALS expressed
novelty and anticipation towards this novel form of physi-

2014.
[8] MHD Yamen Saraiji, Tomoya Sasaki, Reo Mat-cal interaction. The patient was able to successfully operate our system, demonstrating its feasibility. Comparedto triggering the robotic arm motion immediately upon

Ⓒ 2023 日本バーチャルリアリティ学会

Motor imagery eeg-eog signals-1B2-03-sumura, Kouta Minamizawa, and Masahiko Inami.
Fusion: full body surrogacy for collaborative communication. In ACM SIGGRAPH 2018 Emerging Technologies, pages 1–2. 2018.