Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

MetaArms: Body Remapping Using Feet-Controlled
Artificial Arms
MHD Yamen Saraiji∗ Tomoya Sasaki† Kai Kunze∗ Kouta Minamizawa∗ Masahiko Inami†
Keio University Graduate School of Media Design ∗Research Center for Advanced Science and Technology, The University of Tokyo †{yamen, kai, kouta}@kmd.keio.ac.jp∗ {sasaki,inami}@star.rcast.u-tokyo.ac.jp†
The first two authors contributed equally to this work.

Figure 1. MetaArms allows expanding the number of upper limbs by remapping the user’s legs to new artificial limbs: A) Users can experience four
fully-independent arms that can facilitate tasks difficult to be achieved without an extra pair of hands; B) User’s attention is directed towards the newarms rather than the operating limbs (i.e. legs); and C) The artificial limbs can be customized to perform new functions.

ABSTRACT

Author Keywords

We introduce MetaArms, wearable anthropomorphic roboticarms and hands with six degrees of freedom operated by theuser’s legs and feet. Our overall research goal is to re-imagine
what our bodies can do with the aid of wearable robotics using a body-remapping approach. To this end, we present an
initial exploratory case study. MetaArms’ two robotic arms
are controlled by the user’s feet motion, and the robotic hands
can grip objects according to the user’s toes bending. Haptic
feedback is also presented on the user’s feet that correlate withthe touched objects on the robotic hands, creating a closedloop system. We present formal and informal evaluations of
the system, the former using a 2D pointing task according toFitts’ Law. The overall throughput for 12 users of the system
is reported as 1.01 bits/s (std 0.39). We also present informal
feedback from over 230 users. We find that MetaArms demonstrate the feasibility of body-remapping approach in designing
robotic limbs that may help us re-imagine what the humanbody could do.

Artificial Limbs; Feet Interactions;Body Remapping; Body
Schema; Fitts’ Law; Augmented Arms; Human Enhancement.
INTRODUCTION

The human body is inherently constrained by its physical structure that imposes some limits on the functions the body can
perform. Such physical structure is reflected in the mental
model of our body (i.e. body schema) that can be temporarilychanged by using tools and devices [16, 26, 3], while enhancing our body functions beyond the inherent limits. Bothwearable computing and wearable robotics have been experimenting with ways to go beyond such human limits withdifferent emphases.
Today’s wearable computing devices are mostly staticallyworn on a particular body part with relatively conservative
input and output modalities (e.g. visual, audio, and limited
haptics). Wearable robotics, on the other hand, has largely
focused on assistive or rehabilitative functions, such as an exoskeleton that helps a user lift heavy items [47] or specialized
robotic limbs that support humans in performing specific tasks[31, 30]. In both areas, however, the ease and intuitivenessof system operation and subjective sense of embodiment remain a challenge. In our research, we endeavor to expand onsuch “wearable” technology to try and create a system thatfeels more like an integral part of our body and augments ourhuman capabilities, moving from human-computer interaction
towards human-computer integration.

CCS Concepts•Human-centered computing → Interaction devices;
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.

Towards this end, we adopt a “body-remapping” approach.
This is in part inspired by the neurology research that shows

UIST ’18, October 14–17, 2018, Berlin, Germany© 2018 ACM. ISBN 978-1-4503-5948-1/18/10. . . $15.00
DOI: https://doi.org/10.1145/3242587.3242665

65

Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

we can transfer learned motor skills between different bodyparts [37]. We present a system that demonstrates how technology can be used to remap parts of our body with relative easeto provide not simply augmenting functions but also create aroom for re-imagining how we may use our body. We present
a first exploratory case study in which we propose mapping the
movements of the lower limbs to the upper limbs. MetaArms
are additional anthropomorphic robotic arms worn on the backand controlled by the user’s foot motion. Figure 1 shows
an overview of the proposed system and how these arms areoperated in coordination with the user’s leg and foot motion.
Further details of the proposed interaction are discussed in the
Approach section.

research on robotic limbs specialized for predetermined tasks,
there are also more exploratory works that provide modular
robotic platforms to build customizable robotic limbs. Leigh et
al. introduced a system of programmable robotic fingers that
could provide synergistic interaction through programmablejoints [21, 22].
Sasaki et al. provided a similar setup, though their original
work was merely conceptual and did not provide an evaluationof their system (neither technical, qualitative nor quantitative) [34].
We believe our work provides a useful and hopefully thoughtprovoking addition to the wearable robotic limbs researchwith our focus on “remapping” limbs, which can offer not
only possible substitution and augmentation functions but also
could create an exploratory space for re-imagining what thehuman body can do.

This work’s main contribution is the introduction of a new
approach to robotic limbs research that remaps lower limbs
onto the artificial arms/hands, using feet both for motion input
and feedback output. We have conducted a preliminary testingof the interaction’s validity, using a 2D pointing task following
Fitts’ Law (ISO 9241-9), as well as through informal observation of more than 230 users as they tried out the system at threedifferent events. In the Fitts’ Law test, the system achieved amean throughput of 1.01bits/s (std 0.39) among 12 users.

Feet Interaction in HCIThere are many traditional studies that use hands for input
in HCI to manipulate virtual objects or control the motion ofrobots with a glove device, etc. [39]. In contrast, there arefewer approaches that use feet for input.

RELATED WORKS

Velloso et al. provide a good summary of interactions withfeet [42]. Previous work in feet interaction mainly used conventional pedals to control graphical user interfaces (GUI), but
it has more recently expanded to include physical applications
such as walking detection or dance performance feedback [12,36, 45, 7, 11, 29]. In feet interaction research, sensing techniques for detecting foot conditions, acceleration, bending,and pressure are commonly employed. These works havemainly focused on detecting the motion of the foot, and toe
movements or bending have not been typically considered.

Our work draws from and is inspired by a diverse research
corpus. In this section, we focus on these most relevant areas:
augmented limbs (both for substitution and expansion) andfeet interactions in HCI.
Augmented Limbs

Robotic limb research has, for a large part, focused on theprosthesis or industrial applications to increase strength orsimilar human parameters [17, 9, 20, 6]. There is a large bodyof studies that covers augmented arms, hands, and fingers, andamong them are some notable studies on enhancing humancapability by adding extra limbs to the human body. Stelarc’s
Third Hand [18, 38] was one of the earliest art performancesthat investigated the effect of adding a third hand to augment
bodily functions. The hand was controlled by a set of electromyography (EMG) sensors attached to his body, requiring
him years of training to adapt using the hand effectively [8].

As for feedback, tactile stimulation to the feet has been already
employed, but for relatively limited kind of feedbacks, mainly
focused either on notification scenarios or on simulating awalking effect [33, 35]. In order to create a closed-loop interaction between the feet and the user, feedback is necessary to
be actively and passively perceived by the user. As the rubber
hand illusion [5, 1] shows, coupling haptic feedback with visual feedback can induce a strong sense of ownership toward
the artificial hand, which could contribute to more intuitive
control.

More recently, Parietti et al. proposed Supernumerary Robotic
Limbs (SRL): wearable robotic arms that supported the user inperforming specific tasks, such as machine assembly [31, 30].
Non-actuated robotic arms were installed on the user’s hip andassisted the user by holding objects or fixing the user’s posture.
In another example of robotic arms worn on the body, Bonilla
and Asada presented a design concept of augmented arms on
the shoulders [4]. Vatsal et al. proposed another wearable limb
with a twist, implementing an additional hand attached to theforearm [41].

To our knowledge, no study has been conducted to evaluate theeffectiveness of feet control for manipulating a tightly coupledrobotic arms system with haptic feedback applied to the feet.
APPROACH

As seen in the previous section, there are already wellestablished research paradigms for augmenting the humanbody using external limbs that explore a variety of methods tocontrol the additional limbs. We believe our approach offers
a new perspective to the robotic limbs research by focusing
on body remapping. MetaArms remap the user’s foot-and-toe
movements on the robotic arms and hands, and this approach
has a few advantages. Chief among them is the relative easeof mastering control of the robotic arms. Although the idea ofusing the lower limbs to perform the functions of the upper

There are some notable works that explored the possibility ofadding extra fingers as well. Prattichizzo et al. proposed The
Sixth-Finger, a modular robotic sixth finger to increase hand
dexterity and to augment gripping [32]. In a similar vein, Wu
and Asada [43, 44] offered Supernumerary Robotic Fingers(SR Fingers), adding two robotic fingers to support tasks thatusually require two hands. In addition to the wide body of66

Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

Figure 2. MetaArms design approach: a closely situated anthropomorphic arms system driven by leg and feet motion, with haptic feedback loop.

limbs may not seem "easy" at first, we believe it can potentiallyallow for more intuitive control, compared to other modalities
such as eye gaze and EMG signals. In fact, Spampinato etal. showed that it is possible to transfer motor skills between
hand and legs [37]. MetaArms is an attempt to apply theconcept of motor-skills transfer to robotic limbs. We found thefollowing design considerations necessary in our first designof the exploratory MetaArms prototype. Figure 2 shows an
overview of the designed approach, postural mapping, andfeedback loop.

The above consideration led us to decide on a wearable solution: a backpack with the two anthropomorphic arms to be
mounted on the user’s back.
Feedback Loop

For robotic body extensions, it is not only tracking and translation of motion that are essential. Creating some sort of
feedback loop is also crucial, even if it is fairly limited [14].
For our first prototype, we decided on providing force feedback for gripping and touching on the soles of the feet in thehope that this would contribute to the ease and intuitiveness ofcontrol.

Direct Manipulation

The first design consideration is to give the user control of
two additional independent arms/hands for interactions. We
considered and prototyped different interaction modalities,
ranging from electroencephalography (EEG), eye gaze, and
EMG to game controller input. However, each came with
substantial limitations (user-dependent training, noise issues,etc.). In addition, we did not want to hinder free movement of
the user’s own hands, as our aim was not just to substitute thehuman arms/hands but to extend their capabilities by giving theuser additional arms. Therefore, we chose to map the robotic
arm/hand movements directly with the leg/toe movements of
the user. In addition to legs being the closest extremities toarms in terms of degrees of freedom and reach, there is anadded benefit for mastering control as we are already familiar
with how to use our feet in 3D space.

In addition to these design considerations, we also believe inmaking our system prototype as accessible as possible so thateverybody can experiment with body remapping. Therefore
we provide detailed information about the current prototype
implementation in this paper and will work towards an opensourced platform for MetaArms and similar systems.
METAARMS PROTOTYPE

In this section, we describe our exploratory prototype implementation of MetaArms based on the design considerations
presented above. Our prototype consists of three basic components: a mobile backpack equipped with two robotic arms,
feet tracking devices, and a control software.
Robotic Limbs Backpack

The backpack includes two robotic limbs (arm/hands manipulators), control boards, and a 12V battery. The purpose of
making the system as a backpack is to maintain the relationship between body motion on the torso and the arms, so thatboth are in the same frame. Such design would help maintainconsistent coordination and thus consistent operation.

Close Proximity to The Body

In addition to body schema, our brain also keeps a representation of anything in close proximity, or in peripersonal space.
Although this space and its encoding are not well understood,
research suggests that the boundaries between the peripersonal
space and our body schema are fluid [15]. Proximity to thebody seems key to having a sense that something is (or isalmost) part of one’s body. Taking this into consideration, we
wanted to place the robotic arms as close to the user’s own
arms as possible without interfering with them in order toprovide the user an egocentric point of view when using therobotic arms, as well as to make it easier for the user to have asense that the artificial arms are part of his body.

The total weight of the system is about 9 kg, and the dimensions excluding the robotic limbs are shown in Figure 3 . An
external PC is used for controlling the system and calculating
the coordination of the joints.
Robotic Arms

The arms of this prototype are designed in human scale with
6 degrees of freedom (DoF) to perform free spatial manipu-

67

Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

Figure 4. Overview of foot tracking and feedback device.

Figure 3. Schematic design of the prototyped backpack arms system.

lations. The end effectors of the arms are designed to accept
different types of manipulators (e.g. hand, pointer, tool, etc.),
thus allowing for modular design. Figure 3 provides theschematic design of the developed arms, along with the joint
placements and the link dimensions.
For driving the arms, servo motors (model Kondo Kagaku
B3M-SC-1070-A and B3M-SC-1170-A) were used for the six
joints on each arm. The first two joints (J1, J2) are driven using
twin motors since these joints would require the highest torque.
The rest of the joints (J3-J6) are driven by single motors.
Robotic Hand & Manipulators

On the tip of the robotic arms, where hands would be on human arms, we can attach various manipulators (e.g. robotic
hands or more specialized tools). Since our primary focus
was on remapping limbs rather than on specific task performance, we first opted for functional robotic hands that can grip
and provide tactile feedback. We chose to implement robotic
hands based on an open-source artificial prosthetic limb (Exiii
HACKberry1 ), a human scale hand with five fingers operated
by 3 DoF. We modified the design to add touch sensors for
haptic feedback. A pressure sensor (Interlink Electronics FSR
400, FSR 402) is attached to each thumb cushion and palm.
These sensors detect the pressure levels when touching an
object that are reflected in the feedback.

Figure 5. Mechanical time lag and limits between each of user’s motion
and the corresponding robot limb’s motion for each of the three axes
(X,Y,Z).

The toes are used to drive the end-effectors, and we employ
resistance-based bending sensors to detect toe postures (Spectra Symbol Flex Sensor 2.2"). The tip of the sensor is attachedon the upper part of the toes on a sock, and the root of thesensor is connected with the device unit on the arch of the feet.
Based on the posture of the toes, the sensor bends accordingly.
We deployed an initial calibration to detect the minimum andmaximum bending levels of the user’s toes.
Force feedback to the sole is provided by a motor-driven beltmechanism [28]. Two motors are placed inside of the device
unit (MAXON DC 134849, 0.75 W). A belt is wrapped aroundthe user’s foot, and each of the belt’s ends is attached to themotor shaft. The user can feel force feedback on the sole oftheir foot as the belt grips on the foot according to the pressure
value reported by the associated robotic hand’s touch sensors.

Foot Tracking & Feedback

We developed a wearable device unit shown in Figure 4 totrack the foot position/rotation and the toe posture of the user
as well as to provide force feedback to the sole.
The position and rotation of the foot are tracked using an
optical-based motion tracking system, HTC VIVE. The tracking unit is attached to the top of the device, providing an
absolute measurement of foot postures. We initially investigated the use of IMU-based tracking, but due to tracking
latency, accumulated error, and drifting of the measurements,
we decided on optical-based sensing for reliable operation.

Technical Analysis

The system’s mechanical design was evaluated by measuring
the mechanical latency during an arbitrary operation. Usingan optical tracking system (HTC VIVE trackers), the motionof the user and that of the corresponding robotic end effector
were captured. Figure 5 shows the motion signals for each of

1 http://exiii-hackberry.com

68Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

the three axes (X, Y, Z) for both the user and the robot. The
mechanical time lag was measured through cross-correlation
between the control position (User), and the measured position(Robot). Mechanical motion limits and time-lag results arereported in Table 1 .
Axis
X
Y
Z

Motion Limit(mm)Min Max-281 325-250 250-163 180

TP =Time Lag(ms)Mean STD
195
40
375
400
140
62.5

Experimental Setup

As shown in Figure 6 , the experimental setup consisted ofa chair in which the participants sat, facing a display that
prompted the pointing task. The display measured 930 mm ×
485 mm (W × H), which was enough to cover the arm reach.

As is shown, Y Axis suffered the highest time lag, whichsuggests an effect by gravitational force.

The chair was adjusted so the arm’s initial position pointed
at the center of the screen before starting the experiment inorder to normalize the initial condition of the study for allparticipants.

ONE ARM POINTING TASK EVALUATION

To evaluate the performance of our system and get some insights into potential learning, fatigue and embodiment effects,
we conducted a two-directional Fitts’ Law study according
to ISO 9241-9 [40, 19]. The tests were conducted with one
robotic arm, as most of the participants were not accustomed
to manipulating the system.

For this experiment, the robotic hand was replaced with apointer module to facilitate the clicking task on the screen.
This module contained a pressure sensor that measured when
the pointer touched the screen. An optical marker was attachedto the pointer to track its spatial position and orientation as wellas to identify its projected location on the screen coordinates.

Apparatus

Fitts’ Law [10, 25] states that the index of difficulty (ID) is
influenced by the distance from the center point to any of thetargets and the width of the said targets:
A
+ 1)
W

The experiment was conducted in a well-lit room and sufficientspace for the user’s feet to move without special constraint.

(1)

A is the amplitude, or distance where W is the width of target
in unity units. The targets in this experiment were modeled as
white circles on a 2D screen. At the start of the experiment,
one of the targets is rendered red and the participant is asked
to touch it with the robotic arm/hand. Then another target,
located on the opposite of the previously selected target, turns
red, until each of the targets are activated. The session ends
when the latest activated circle comes back to the first position.
During the experiment, we also recorded the movement time(MT ) of the whole trial for each input. At the beginning ofeach trial round (i.e. when a new circle turned red), the position of the robot arm was reset to face the center of the screen.
At the point of selection, we also measured the standard deviation (SD) for over-shoots and under-shoots from the centerof the target circle. We used the SD values to calculate theeffective width (We) shown below:
We = 4.133 ∗ SD

(4)

For the duration of the experimental tests, we employed athink-aloud protocol: the participants were free to express
their opinions and provide feedbacks at any time, which werethen recorded.

Table 1. Mechanical evaluation of the system.

ID = log2 (IDeMTFigure 6. Experimental setup overview.

Participants

Twelve participants (10 male, 2 female) voluntarily joined thestudy at the local university. Participants’ age ranged from 22
to 39 year-old (Mean=26.5, STD=5.0). No participants experienced any physical disabilities and all had fully functionalright legs. Only one participant (female) was left-leg dominant
as reported in the pre-test questionnaire (See next section for
more detail). All participants provided their informed consent prior to starting the study and were informed about theprocedure and the approximated length of the study.

(2)This allows us to compute the effective index of difficulty(IDe) and the final throughput (T P).

Procedure

IDe = log2 (

A
+ 1)
We

Prior to starting the experiment, participants provided basic information, including age, gender, height, and answered some

(3)

69

Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

questions on physical exercise. They were also asked to describe if they were currently suffering from any physical injuries. The evaluative study consisted of four sessions for eachparticipant. Each session lasted for approximately 10 minutes.
Between pointing-task sessions, the participant confirmed his
readiness and that he had had enough rest. During each session, a participant performed a Fitts’ Law pointing task withdifferent index of difficulties (ID) and a total of 13 targets for
Number of Targets (FittsNoT ). The order of the sessions (interms of difficulty) was randomized, as well as the order ofthe first target for each Fitts trial. Fitts parameter (A/W ) values used in the experiment, along with the corresponding IDs(calculated using Eq. 1) of their combinations, are reported asfollows:
Figure 7. Throughput of pointing task over four sessions.

A = 250mm, 350mm
W = 40mm, 75mm
ID = 2.12 bits, 2.50 bits, 2.86 bits, 3.29 bits
After each trial, a random white circle appeared on the monitor in one of the four corners (X : ±200mm,Y = ±100mm
from the center of the screen) for two seconds. The user was
asked to remember its location, and then point at its memorized location with their eyes closed. The purpose of this step
was to measure body coordination without visual feedback(Haptic feedback was still presented when the arm touched thescreen). The distance to the target from the point of contact
was measured and stored for each trial.
The total number of pointing and clicking tasks per session
were calculated as follows:

Figure 8. NASA Task Load Index of the 12 participants.

Nclicks = (FittsNoT + 1) × Ntrials= (13 + 1) × 4= 56 clicks

RESULTS AND DISCUSSION
Throughput

At the end of each session, the user answered the NASA Task
Load Index for perceived task load [13], and rated four statements adopted from the rubber hand illusion experiment [5].
The statements are related to the subjective feeling of theparticipants:

The mean throughput for this system for all participants
over all sessions was 1.01bits/s with a standard deviationof 0.39bits/s. Figure 7 shows the throughput for each session across all participants. Movement times between targets
ranged between 0.42s and 9.66s (mean=3.23s STD=1.35s)
across all sessions. For comparison, Young et al. [46] reported
1.08bits/s with a movement speed varying between 1.0s to 2.4s
for a 6 DoF input device that used arm motion. Although thethroughput of our system was a bit lower, this is perhaps not
surprising given the novelty of carrying out a pointing taskwith foot motion• How much the robotic arm feels like a part of their own
body.
• How much it looks like their own arm.
• How much they feel they control it as part of their own
body.
• If the haptic feedback feels like it it is caused by the roboticarm.

Task Load Index

The standard deviation decreased noticeably in the second
and third session, but later increased for the fourth session.
We assume this is due to the increase in perceived physicalfatigue of the participants (later discussed in Task Load Index
section).

For each of the statements, the participants were asked to rate
how much it held true for them on a 7-point Likert scale from 1(“not at all true for me”) to 7 (“very true for me”). The participants were also asked to fill a questionnaire to assess their subjective Embodied Sense of Self Scale (ESSS) [2]. The ESSS
is a summary measure of several subjective embodiment assessments including Ownership, Agency, and Narrative tested
on a large cohort (700+ users) [23, 27].

Although the throughput increased over the four sessions, we
did not find a significant learning effect between the first andfourth session. Applying a two-tailed t-test between the first
and the last session results in a p-value of 0.0909. This result

70Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

Figure 9. Measured error for all participants over four sessions with eyes closed pointing task

alone, however, cannot suggest anything concrete on learnability as the four sessions took place in a fairly limited period oftime.

(M=5.0, SD =1.5) results in p < 0.005 (DF = 12). Given thevery small sample size (12 participants), we also calculated theeffect size E = E/S(~∆) ∗ S(~∆) (T-statistic and non-centrality
parameter) to be 1.072, which is medium also indicating a significant result. Additionally, we performed the Mann-Whitney
U test and obtained results indicating moderate significance:
The U-value is 45. The critical value of U at p < .05 is 45. The
Z-Score is -2. The p-value is -0455.

Figure 8 shows the results of NASA Task Load Index of
the 12 participants across four sessions. Strong increases
in mental and physical demand through the course of theexperiment can be seen. We believe this is partly due to thestandard Fitts’ Law experimental setup: the users could not put
down their controlling leg during the task and had to continue
pointing for the entire session. They only rested in between
sessions. This is quite different from what we consider to be
more normal usage, such as we observed during demonstration
sessions (The demo observation will be discussed more later).
Some participants complained about slight muscle ache in
the right hip (most likely Iliacus muscle) after performing
the 4th iteration. Despite the reported physical fatigue by theparticipants, the perceived performance, effort, and frustration
improved over the four sessions. This may indicate that theparticipants started to learn how to map their feet motion to the
new artificial limb, even though such perceived improvements
were not reflected in the objective measurements.

Body Coordination

After the standard Fitts’ Law experiment, we briefly displayeda white circle in a randomly selected corner on the screen. Our
aim was to measure the adaptation of the new body coordination without continuous visual feedback. Figure 9 shows
the projected error over the four sessions for all participants.
Although the mean error of the distance to the center of thecircle remains, it decreased over the course of four sessions
along with the standard deviation, providing indications ofenhanced temporal proprioceptive adaptation.
The most dominant error was caused by the vertical axis (Y),which can be correlated with the mechanical latency of thearms (reported in the Technical Analysis), as well as the difficulty to move the feet upward for tall people due to the setup.

Self-reported Embodiment

The statements related to the subjective look, feel and embodiment did not show any significant results. However, theanswers to the statement about control “I felt like I controlled
the robot arm as if it was part of my own body.” showed asignificant change over the four sessions. Comparing all participants’ answers given after the first session with the answers
after the last session, the participants reported a significant
increase in feeling that the robot arm is part of their body. As
expected, the users found it hard to associate the robotic armwith their own as we installed a pointer rather than a robotic
had to perform the Fitts’Law experiments (see Figure 6 ). Also,
the statement about the touch was not significant.

APPLICATIONS

During the course of the prototyping, development, user testing, and exhibitions of our system, we collected a variety of
application scenarios for MetaArms . The possibility of coaction between the four arms can help facilitate complex tasks
that are difficult to be achieved by a single person. Figure
1 (A) shows a user exchanging the contents of four cups inparallel without any second user support. We imagine thepossibility of using such an approach for industrial or professional applications. For example, Figure 1 (C) shows ascenario which the end effector of one arm is replaced with asoldering tool, allowing the user to complete more complex
soldering tasks without assistance.

The answers to the statement about control show a significant
change over the four sessions “I felt like I controlled the robot
arm as if it was part of my own body.”. Comparing all answers
from the 12 participants from the first session with the answers
from the last, the participants reported a significant increase infeeling that the robot arm is part of their body.

So far, the envisioned application scenarios fall into three
categories: substitution, augmentation, and experimentation.
Substitution includes applications for amputees or people
with arm/hand disabilities. For example, Figure 1 (B)shows how MetaArms can be used to hold and manipulate
objects. We had semi-formal interviews with stroke patients

The paired t-test between the Session 1 answers for the control
statement (M=3.5, SD =1.43) and the fourth Session 4 answers

71Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

and hand/arm amputees who showed great interest in using
this system.

want to improve the lag between tracking and actuation to get
below 100 ms for all three axes to induce the feeling of instantaneous movement. This might be especially difficult for they-axis (which is affected by gravity). Also, we are planningto improve the haptic feedback, moving from simple force
feedback to more complex haptic interactions, such as using
electric muscle stimulation on the leg [24]. Even with the limitations of our study and our exploratory prototype, we hopeand believe that MetaArms with its body remapping approach
would create a space of play for exploring and reimaging what
the human body could do.

Augmentation focuses on use cases where additionalhands/arms obviously are needed. For example holding acomponent part and soldering (see Figure 1 (C)). This is suitable in situations and applications where the legs are not used(for example, in seated position).
Experimentation explores what other use cases may be madepossible by remapping limbs. This exploration may be best
pursued in the realm of art and performance, such as playing
multiple instruments at the same time, or juggling with fourarms.

Acknowledgements

This project is supported by JSPS KAKENHI Grant Number
15H01701.

Remapping could let us reimagine what our bodies can do,rather than simply filling an existing need or augmenting what
we already do.

REFERENCES

1. K Carrie Armel and Vilayanur S Ramachandran. 2003.
Projecting sensations to external objects: evidence fromskin conductance response. Proceedings of the Royal
Society of London B: Biological Sciences 270, 1523
(2003), 1499–1506.

INFORMAL EVALUATION

For informal evaluation, we showcased the MetaArms system at three events (two domestic, one international) for a
total of 12 days, around 7 hours a day with over 230 users.
They were conferences and laboratory events attracting mostly
researchers, professionals and tech enthusiasts. The user demographic was unexpectedly diverse, however, from a 13-year
old child to a 67-year-old professor emeritus. Such diverse
demographic helped us find some height limitations of thecurrent system where users smaller than 1.30 m or taller than
1.90 m have trouble operating or wearing it.

2. Tomohisa Asai, Noriaki Kanayama, Shu Imaizumi,
Shinichi Koyama, and Seiji Kaganoi. 2016. Developmentof embodied sense of self scale (ESSS): exploring
everyday experiences induced by anomalous
self-representation. Frontiers in psychology 7 (2016),
1005.

Most demonstration interactions lasted for approximately 1015 minutes. Most users could successfully operate the system.
Typical demonstration interactions included gripping gestures
to hold a cup or shake a hand, playing with tennis balls, andinteracting with other people. A user succeeded in throwing aball with their normal hand and catching it with their robotic
limb after a few trials. We think that the adaptation timeof operating the robotic arms is highly relevant to system’s
operation speed and response (current system performance isreported in the Technical Analysis). Many users spontaneously
tried to touch their nose or make drinking gestures with thecup, though these gestures did not work as we restricted theartificial limb movement to prevent injuries.

3. Anna Berti and Francesca Frassinetti. 2000. When far
becomes near: Remapping of space by tool use. Journalof cognitive neuroscience 12, 3 (2000), 415–420.
4. Baldin Llorens Bonilla and H Harry Asada. 2014. A
robot on the shoulder: Coordinated human-wearable
robot control using coloured petri nets and partial leastsquares predictions. In Robotics and Automation (ICRA),
2014 IEEE International Conference on. IEEE, 119–125.
5. Matthew Botvinick and Jonathan Cohen. 1998. Rubber
hands ’feel’ touch that eyes see. Nature 391, 6669 (1998),
756–756.
6. MC Carrozza, B Massa, S Micera, R Lazzarini, M Zecca,
and P Dario. 2002. The development of a novel prosthetic
hand-ongoing research and preliminary results.
IEEE/Asme Transactions on Mechatronics 7, 2 (2002),
108–114.

CONCLUSION AND FUTURE WORKIn this paper we have presented MetaArms, an exploratory feetcontrolled robotic arms prototype, to examine the feasibilityof using limb remapping for operating wearable extra roboticarms with haptic feedback. We introduced a new framework
for mapping foot motions to new artificial limbs based on ourdesign considerations. We presented a first prototype implementation of MetaArms as well as technical analysis, the results from a standardized Fitts’ Law experiment and subjective
feedbacks from the user tests and large-scale demonstration
sessions. Overall, the system worked reasonably well within
the set parameters, and in some cases seemed to succeed in
temporarily eliciting the feeling of having two additional arms.

7. Meng Chen, Bufu Huang, and Yangsheng Xu. 2008.
Intelligent shoes for abnormal gait detection. In Robotics
and Automation, 2008. ICRA 2008. IEEE International
Conference on. IEEE, 2019–2024.
8. Andy Clark. 2003. Natural-Born Cyborgs: Minds,
Technologies, and the Future of Human Intelligence.
(2003), 116.
9. R Fearing. 1986. Implementing a force strategy for object
re-orientation. In Robotics and Automation. Proceedings.
1986 IEEE International Conference on, Vol. 3. IEEE,
96–102.

Regarding future work, we will need to address a number oftechnical limitations of our current MetaArms system as wellas limitations of haptic feedback in general. For instance, we72Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

10. Paul M Fitts. 1954. The information capacity of thehuman motor system in controlling the amplitude of
movement. Journal of experimental psychology 47, 6(1954), 381.

22. Sang-won Leigh, Harpreet Sareen, Hsin-Liu Cindy Kao,
Xin Liu, and Pattie Maes. 2017b. Body-Borne Computers
as Extensions of Self. Computers 6, 1 (2017), 12.
23. Matthew R Longo, Friederike Schüür, Marjolein PM
Kammers, Manos Tsakiris, and Patrick Haggard. 2008.
What is embodiment? A psychometric approach.
Cognition 107, 3 (2008), 978–998.

11. Koumei Fukahori, Daisuke Sakamoto, and Takeo Igarashi.
2015. Exploring Subtle Foot Plantar-based Gestures with
Sock-placed Pressure Sensors. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in
Computing Systems. ACM, 3019–3028.

24. Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015.
Impacto: Simulating physical impact by combining
tactile stimulation with electrical muscle stimulation. In
Proceedings of the 28th Annual ACM Symposium on User
Interface Software & Technology. ACM, 11–19.

12. Fabian Göbel, Konstantin Klamka, Andreas Siegel,
Stefan Vogt, Sophie Stellmach, and Raimund Dachselt.
2013. Gaze-supported foot interaction in zoomable
information spaces. In CHI’13 Extended Abstracts on
Human Factors in Computing Systems. ACM,
3059–3062.

25. I Scott MacKenzie. 1992. Fitts’ law as a research anddesign tool in human-computer interaction.
Human-computer interaction 7, 1 (1992), 91–139.

13. Sandra G Hart. 2006. NASA-task load index(NASA-TLX); 20 years later. In Proceedings of thehuman factors and ergonomics society annual meeting,Vol. 50. Sage Publications Sage CA: Los Angeles, CA,
904–908.

26. Angelo Maravita and Atsushi Iriki. 2004. Tools for the
body (schema). Trends in cognitive sciences 8, 2 (2004),
79–86.
27. Jessie E Menzel. 2010. The psychometric validation of
the physical body experiences questionnaire. (2010).

14. Matej Hoffmann, Hugo Marques, Alejandro Arieta,
Hidenobu Sumioka, Max Lungarella, and Rolf Pfeifer.
2010. Body schema in robotics: a review. IEEE
Transactions on Autonomous Mental Development 2, 4(2010), 304–324.

28. Kouta Minamizawa, Souichiro Fukamachi, Hiroyuki
Kajimoto, Naoki Kawakami, and Susumu Tachi. 2007.
Gravity grabber: wearable haptic display to present
virtual mass sensation. In ACM SIGGRAPH 2007
emerging technologies. ACM, 8.

15. Nicholas P Holmes and Charles Spence. 2004. The bodyschema and multisensory representation (s) ofperipersonal space. Cognitive processing 5, 2 (2004),
94–105.

29. Joseph A Paradiso, Kai-Yuh Hsiao, Ari Y Benbasat, and
Zoe Teegarden. 2000. Design and implementation of
expressive footwear. IBM systems journal 39, 3.4 (2000),
511–529.

16. Atsushi Iriki, Michio Tanaka, and Yoshiaki Iwamura.
1996. Coding of modified body schema during tool use
by macaque postcentral neurones. Neuroreport 7, 14(1996), 2325–2330.

30. Federico Parietti and H Harry Asada. 2014.
Supernumerary robotic limbs for aircraft fuselage
assembly: body stabilization and guidance by bracing. In
2014 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 1176–1183.

17. Steve Jacobsen, E Iversen, D Knutti, R Johnson, and K
Biggers. 1986. Design of the Utah/MIT dextrous hand. In
Robotics and Automation. Proceedings. 1986 IEEE
International Conference on, Vol. 3. IEEE, 1520–1532.

31. Federico Parietti, Kameron Chan, and H Harry Asada.
2014. Bracing the human body with supernumeraryrobotic limbs for physical assistance and load reduction.
In 2014 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 141–148.

18. Eduardo Kac. 1997. Foundation and development ofrobotic art. Art Journal 56, 3 (1997), 60–67.
19. Luv Kohli, Mary C Whitton, and Frederick P Brooks.
2013. Redirected Touching: Training and adaptation in
warped virtual spaces. In 3D User Interfaces (3DUI),
2013 IEEE Symposium on. IEEE, 79–86.

32. Domenico Prattichizzo, Monica Malvezzi, Irfan Hussain,
and Gionata Salvietti. 2014. The sixth-finger: a modular
extra-finger to enhance human hand capabilities. In The
23rd IEEE International Symposium on Robot andHuman Interactive Communication. IEEE, 993–998.

20. Peter J Kyberd and Paul H Chappell. 1994. The
Southampton Hand: an intelligent myoelectric prosthesis.
Journal of rehabilitation Research and Development 31,
4 (1994), 326.

33. AF Rovers and HA Van Essen. 2005. FootIO-design andevaluation of a device to enable foot interaction over a
computer network. In Eurohaptics Conference, 2005 andSymposium on Haptic Interfaces for Virtual Environment
and Teleoperator Systems, 2005. World Haptics 2005.
First Joint. IEEE, 521–522.

21. Sang-won Leigh, Kush Parekh, Timothy Denton,
William S Peebles, Magnus H Johnson, and Pattie Maes.
2017a. Morphology Extension Kit: A Modular Robotic
Platform for Customizable and Physically Capable
Wearables. In Proceedings of the 2017 CHI Conference
Extended Abstracts on Human Factors in Computing
Systems. ACM, 397–400.

34. Tomoya Sasaki, MHD Yamen Saraiji, Charith Lasantha
Fernando, Kouta Minamizawa, and Masahiko Inami.
MetaLimbs: Multiple Arms Interaction Metamorphism.
In ACM SIGGRAPH 2017 Emerging Technologies(SIGGRAPH ’17). ACM (New York, NY, USA).
73

Session 2: Human-Robot Symbiosis

UIST 2018, October 14–17, 2018, Berlin, Germany

35. Henning Schmidt, Stefan Hesse, Rolf Bernhardt, and Jörg
Krüger. 2005. HapticWalker—a novel haptic foot device.
ACM Transactions on Applied Perception (TAP) 2, 2
(2005), 166–180.

42. Eduardo Velloso, Dominik Schmidt, Jason Alexander,
Hans Gellersen, and Andreas Bulling. 2015. The feet inhuman–computer interaction: a survey of foot-based
interaction. ACM Computing Surveys (CSUR) 48, 2(2015), 21.

36. Adalberto L Simeone, Eduardo Velloso, Jason Alexander,and Hans Gellersen. 2014. Feet movement in desktop 3Dinteraction. In 3D User Interfaces (3DUI), 2014 IEEE
Symposium on. IEEE, 71–74.

43. Faye Wu and Harry Asada. 2014. Bio-Artificial Synergies
for Grasp Posture Control of Supernumerary Robotic
Fingers. In Proceedings of Robotics: Science andSystems. Berkeley, USA.

37. D Spampinato, Hannah J Block, and P Celnik. 2017.
Cerebellar-M1 connectivity changes associated to motor
learning are somatotopic specific. Journal of
Neuroscience (2017), 2511–16.

44. Faye Y Wu and H Harry Asada. 2015.
"Hold-and-manipulate” with a single hand being assisted
by wearable extra fingers. In Robotics and Automation(ICRA), 2015 IEEE International Conference on. IEEE,
6205–6212.

38. Stelarc. 1980. Third Hand. (1980).
http://stelarc.org/?catID=20265 (Last accessed July2018).

45. Weizhong Ye, Yangsheng Xu, and Ka Keung Lee. 2005.
Shoe-Mouse: An integrated intelligent shoe. In Intelligent
Robots and Systems, 2005.(IROS 2005). 2005 IEEE/RSJ
International Conference on. IEEE, 1163–1167.

39. David J Sturman and David Zeltzer. 1994. A survey ofglove-based input. IEEE Computer graphics andApplications 1 (1994), 30–39.
40. Robert J Teather and Wolfgang Stuerzlinger. 2011.
Pointing at 3D targets in a stereo head-tracked virtualenvironment. In 3D User Interfaces (3DUI), 2011 IEEE
Symposium on. IEEE, 87–94.

46. Thomas S Young, Robert J Teather, and I Scott
MacKenzie. 2017. An arm-mounted inertial controller for
6DOF input: Design and evaluation. In 3D User
Interfaces (3DUI), 2017 IEEE Symposium on. IEEE,
26–35.

41. Vighnesh Vatsal and Guy Hoffman. 2017. Wearing yourarm on your sleeve: Studying usage contexts for awearable robotic forearm. In Robot and Human
Interactive Communication (RO-MAN), 2017 26th IEEE
International Symposium on. IEEE, 974–980.

47. Adam B Zoss, Hami Kazerooni, and Andrew Chu. 2006.
Biomechanical design of the Berkeley lower extremity
exoskeleton (BLEEX). IEEE/ASME Transactions On
Mechatronics 11, 2 (2006), 128–138.

74