CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

Dynamic Motor Skill Synthesis
with Human-Machine Mutual Actuation
Azumi Maekawa∗, Seito Matsubara∗, Sohei Wakisaka
Daisuke Uriu, Atsushi Hiyama, Masahiko Inami
The University of Tokyo
Tokyo, Japan{azumi, matsubara, wakisaka, uriu, hiyama, inami}@star.rcast.u-tokyo.ac.jp
ABSTRACTThis paper presents an approach for coupling robotic capability
with human ability in dynamic motor skills, called “HumanMachine Mutual Actuation (HMMA).” We focus speciﬁcallyon throwing motions and propose a method to control the release timing computationally. A system we developed achieves
our concept, HMMA, by a robotic handheld device that acts
as a release controller. We conducted user studies to validate the feasibility of the concept and clarify related technicalissues to be tackled. We recognized that the system successfully performs on throwing according to the target while it
exploits human ability. These empirical experiments suggestthat robotic capability can be embedded into the users’ motions without losing their senses of control. Throughout theuser study, we also revealed several issues to be tackled infurther research contributing to HMMA.
Author Keywords

Robotic device; Motor skill; Motion sensing; Human
augmentation; Human-machine mutual actuation
CCS Concepts•Human-centered computing → Human computer interaction (HCI); Interaction devices;
INTRODUCTION

Humans have extraordinary abilities for physical motions anddynamic movements. Athletes perform sophisticated motions,
utilizing even slight information from sensory organs. Surprisingly, human motor skills can be controlled with millisecond
order precision in some situations [6].
Recent improvements in computing powers, sensors, and actuators have allowed humans to perform closed-loop controlof robotic systems at a sampling rate exceeding the nature of* Both authors contributed equally to this research
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proﬁt or commercial advantage and that copies bear this notice and the full citationon the ﬁrst page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee. Request permissions from permissions@acm.org.
CHI ’20, April 25–30, 2020, Honolulu, HI, USA.
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-6708-0/20/04 ...$15.00.
http://dx.doi.org/10.1145/3313831.3376705

Paper 576

the human body. Thanks to the advancement of digital fabrication technologies, robotic systems have the feasibility toreplace parts of a human’s physical abilities and have started
to provide physical augmentations not only for coping withdisabilities [12, 28].
Technically, robots can perform motor skills that are comparable or superior to those of humans, although this is only insimple tasks or limited environments. However, when a robot
needs higher power, its size and weight must be bigger. Therefore, it is almost impossible to keep its mobility; moreover, its
costs should also be expensive. Overcoming these limitations,this research challenges to interweave robotic capability withhuman dynamic motions and offer an intuitive operation.
Human-machine mutual actuation

Human actuation is a concept of using people as substitutions
to motors and mechanical components, as proposed in previous research [2, 3, 4]. It is generally applied for large-scale
force feedback to the users in immersive experiences. Similar
approaches have also been proposed for large-scale objects
in the digital fabrication ﬁeld. In the approaches, humans
provide mobility, and active tools perform cutting or additive
manufacturing within a short-range [31, 44].
Inspired by previous works, we propose an approach called
Human-machine mutual actuation (HMMA), where robots
act with human body actuation synchronously. Our key idea
is applying the concept of human actuation to the robotics
and human augmentation domain; a mobile (wearable) robot
with a human body operates in its local coordinate system
according to his/her motions that synthesize dynamic motor
skills. HMMA couples human body actuation with dynamic
motor skills instead of the conventional stand-alone robots. In
our approach, a robot for completing a task exploits humanactuator output power and synthesizes a motor skill. On theother hand, the human body exploits the robotic capability of
performing motions. HMMA comprehensively allows humans
to exploit robotic capabilities.
Throwing motion as a case study

As a case study of HMMA, we propose a system illustrated in
Figure 1. The system focuses on throwing motions, particularly throwing underhand, because it is a basic physical skill
and requires dynamic movement with muscle force.

Page 1CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

Figure 1. (a) Framework of a system developed for a case study of HMMA. (b) A handheld device acting as a release controller. (c) The device iscontrolled by the computational process according to a user’s motion.

With our motivation in coupling robotic capabilities and human abilities, we present HMMA with the system focusing
on the motion of throwing, leveraging human actuation concept. Our long-term goal is to develop the design guidelines
for HMMA systems. This research investigates the feasibilityof HMMA and explores how we can design human-machine
relationships in the HMMA situation. Our computational
processing system for throwing controls release timings according to human motions. It illustrates a concrete exampleof HMMA, preparing a robotic handheld device that acts as arelease controller. We conducted user studies to verify our concept and investigate the users’ subjective experiences. We also
demonstrated the system at a public exhibition and observed
the users’ behaviors.
In summary, our contributions are as follows: 1) we propose aconcept (human-machine mutual actuation: HMMA) utilizing
the human body as an actuator for robotic motor skills, 2) wedeveloped a system that consists of a simple handheld device
and the framework tracking and predicting the device motions,
3) we conducted user studies to verify HMMA, revealing
several key challenges throughout this research.
RELATED WORKThis research is related to multiple ﬁelds. It mainly contributes
to the human augmentation research community with our proposed system utilizing knowledge in the computational tooland device researches.
Computational tool and device
Handheld device

Our system refers to previous works about human-machine
hybrid systems using wearable devices. A handheld device
is easy to install and is a familiar interface to users. For 2D
digital fabrication, Rivers et al. proposed a human-machine
hybrid approach where a human provides a wide range ofmobility, and a tool adjusts the precise position [31]. Yoshida
et al. also proposed an approach that utilizes human actuation
for architecture-scale 3D printing with a handheld device [44].
Lafreniere et al. proposed a method using crowdsourced power
for large-scale fabrication [13]. Yamaoka & Kakehi presenteda system that provides the user with an augmented sketchingexperience by controlling the movements of a pen computationally [41]. FreeD is also a pen-like device to augment
freehand sculpting of 3D shapes in a human-in-the-loop manner [47, 48]. These works focus on augmenting creativity with

Paper 576

digital technologies. We also developed a similar device for
performing dynamic motor skills.
In the areas of Augmented Reality and Virtual Reality, mechanical devices have been used as haptic displays in a humanin-the-loop manner. For example, handheld devices have been
proposed for rotor thrust [42] and the gyro effect [5]. These
devices are controlled according to human motion, whereas,
their goal is to generate haptic sensations in AR or VR environments. Nojima et al. developed a system augmenting
interactions connected with the environment via a mechanical device [24]. Referring to these works, we integrated
computational fast-loops with slow-loops between human andmachine.

Digital technologies for human throwing

Related with the throwing motion, researchers have applied
technologies to sports including the throwing motion. Nitta etal. proposed a ball device that enhances the ﬂying ability witha small quadcopter’s thrust [22, 23]. Ohta et al. also developed
a ball device that can change its trajectory with the emissionof a gas-jet [25]. As another example, there is a moving target
approach that allows a users to get a bulls-eye every time [32].
These works focus on applying the computational approaches
to each projectile or target.

Robotic exoskeleton and active prostheses

Robotic exoskeleton is one of the ways of integrating robotic
systems with human ability, e.g., by wearing on the hand [1,43] or arm [8, 10]. These studies focus on rehabilitation or
power assistance to increase load-bearing capabilities. Asa case using robotic exoskeletons for dynamic motor skills,
methods of minimizing human energy cost during walking
or running have been proposed [46, 11]. These works deal
with performing dynamic motions in lower body movementswith robotic systems. In recent years, with the development
of digital fabrication technologies, active prosthetic hands
have become popular and some of them are available as open
sources [12]. Humans have a remarkable ability to use thesehands with dexterity, for instance, as shown in Cybathlon [30].
Our concept HMMA might fully exploit the ability of thesehands and the abilities of the human body, and assist humans
to complete tasks.

Page 2

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

Human augmentation

Human augmentation is an approach to enhancing and empowering human functions with information technologies utilizing
robotics and sensing devices.
Robotics

Robotic systems commonly utilize sensors installed in theenvironment for a robot’s motor skills. Senoo et al. proposed amethod for robotic throwing [36, 39] and batting [37, 38] using
high-speed cameras installed in the environment. Murakami etal. proposed a ball catching strategy with a robotic hand using
high-speed visual feedback [20]. Our basic idea is designingthese robotic motor skill techniques with human actuators.
In the robotics ﬁeld, the human-in-the-loop method has been
investigated to deal with a dynamic task [27, 40]. Robots inthese works are usually ﬁxed to the environment while ourscope of this paper is a wearable robot and the interaction
between it and the user. Recently, extra robotic limbs havebeen proposed as approaches to enhance human manipulation
ability or assist human works [15, 33, 34, 18]. For example,
the Sixth-Finger is an extra-ﬁnger enhancing the wearer’s
manipulation dexterity [28]. Supernumerary Robotic Limbs
were proposed for reducing the wearer’s workload [26].
Sensing and actuation

Itoh et al. [9] and Sato et al. [35] developed systems visualizing
the projectile trajectory or the landing position in the physicalworld. These works aim to augment a human’s prediction
skills and support appropriate responses using visualization ofprojectile trajectory. Our developed system includes a motion
prediction to minimize effects by the system latency. Nishida
et al. proposed a wired muscle system accelerating the humancatching motion with electrical muscle stimulation [21]. It
indicates that its actuation by a computational process canbe perceived as a voluntary action. We also aim to evaluate
not only the quantitative performances but also the qualitative
aspects from the user’s feedback.
Our contribution

Robotic tools have been used as the devices for fabricationand the tools providing haptics in the HCI community. Recently, interactions with robotic devices and augmented bodies
have been explored in the human augmentation ﬁeld. However, we believe there is a frontier integrating the interaction
with robotic devices with dynamic motor skills or high-speed
movements. Based on the HMMA concept, we explore key
elements of the interaction by the developed system. This research offers new insight into human augmentation researches.
It also contributes to robotic technology researches, providing
a direction for the human-robot interaction.
SYSTEM DESIGN
Overview of the system design

In this section, we describe the details of the developed system
as a case study of HMMA. Figure 1 (a) shows an overview ofthe developed system framework. Our system consists of thehandheld device holding and releasing an object, and the control architecture tracking the device’s motion in real time andcontrolling the release timing. The handheld device has a gripper for holding and releasing a projectile with computational

Paper 576

Figure 2. The developed device employs one actuated degree of freedom (DoF) and acts as a release controller with an external trigger. This
device has an onboard IMU sensor, and a button interface is installed tocontrol the gripper’s opening and closing manually.

triggers. The device is tracked by external motion capture
cameras. An onboard inertial measurement unit (IMU) sensor
on the device consists of an accelerometer sensor and a gyro
sensor. While the user is performing the throwing motion, thecomputation of the projectile trajectory is running based on thedevice motion in real time. At the timing when the computed
trajectory passes through a deﬁned target area, the gripper isopened with the trigger signal, and the projectile is throwninto the air. In other words, the object is not released while
doing a motion that misses the target area. In this manner, thesystem allows the user to throw a projectile toward the deﬁned
target.
In the following section, we describe the handheld device andthe motion tracking architecture in detail.
Handheld device

Enabling the system with a human to throw in the arbitrary
trajectory by only controlling the release timing according to
the user’s motion, we designed a robotic handheld device that
can hold and release an object with a computational trigger(Figure 2). There were two key design criteria for this device:1) holding an object without it slipping out, 2) releasing theobject quickly from receiving the trigger to minimize the system latency. The device employs a gripper driven by a DC
motor via a belt and pulley mechanism. The gripper has one
actuated DoF. The gripper opens and closes in parallel and canhold an object by applying a force to both sides of the object.

Page 3

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

The motor spec and gear ratio need to be selected carefully.
For example, a high gear ratio can provide sufﬁcient torquefor holding an object while it usually gets high inertia and friction. The high inertia prevents the gripper from accelerating
quickly. We chose a 10 W class coreless DC motor, MAXON
DCX series, with a 21:1 ratio. The motor was controlled by
the motor driver, ESCON 50/5, from MAXON. To control
the force applied to the object, the motor driver controls thecurrent going through the motor. An IMU sensor is installedon the device for using the position tracking described in thefollowing section. The device also employs a button interface
on the handle to control the gripper’s open and close manually.
The onboard IMU sensor and DC motor are connected to theexternal main controller and power supply by wire.
The total weight of the device is 1.0 kg, and the user grabs
the handle and swings it. We attached a soft material to thegripper to adapt to the object to be grasped.
Control architecture

To release an object at the optimal timing, we must detect theposition of the device in real time. Since human throwing
motion is a dynamic movement, the precision of the projectile
trajectory can be affected by the difference of the releasetiming in a millisecond order. To this end, we developed areal-time position tracking workﬂow.
Real-time position tracking

Figure 3 shows the release control workﬂow of the developed
system. We aim to track the position of the device by combining an external optical motion capture system and the onboard
IMU sensor installed on the device. The motion capture cantrack the registered object accurately and precisely. However,
we found that the frame rate was not fast enough for controlling the release timing in the human throwing movement. On
the other hand, the onboard sensor can run at a high frame rate,
although it is not suitable for estimating the absolute position
accurately. Therefore, we designed the tracking workﬂow sothat the current position of the device is estimated by sequentially integrating the data from the onboard sensor base on theposition data obtained from the motion capture. We used a
Madgwick ﬁlter [16] to estimate the orientation of the device
from the raw values of the sensor.
The system computes a projectile trajectory in each time step
based on the obtained current position data. This workﬂow
includes the latency compensation process described in thefollowing section. If the trajectory passes through the target
area within the deﬁned threshold, the trigger signal is sent to
the motor driver and the object is released. The trajectory is
computed using Newton’s second law. We assumed that theforce acting on the object was only gravity and the inﬂuenceof air resistance was negligible.
System architecture

Figure 4 shows the system architecture. We used MAC3D
systems including 8 cameras (Kestral 2200) for the opticalmotion capture systems, and it was run with a 300 Hz frame
rate. For the onboard sensor, MPU 6050 was used, and it
includes an acceleration and gyroscope sensor. We selected

Paper 576

Figure 3. Release control workﬂow (the X-Y plane target with the ZUP world coordinate system example). xx, vv, aa, and R are the device’s
position, velocity, acceleration, and orientation in the world coordinate
system respectively.

the STM32 Nucleo-767ZI board, (which houses a CortexM7 clocked at 216 MHz) from STMicroelectronics, as themain controller. The controller and the motion capture systemwere connected through a Windows 10 PC with Intel Corei9-8950HK 2.90G Hz CPU and an NVIDIA Quadro P2000
GPU.
Latency compensation

Since human throwing motion is a dynamic movement, thedevice can move by an amount sufﬁcient for the projectile

Page 4

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

formed in accordance with the ethical standards provided in
the Declaration of Helsinki.
User study 1: With aiming at the target

The goal of this experiment is to evaluate the performanceof the developed system in a situation where the user aims
at the target, and investigate how the proposed system assists
the user’s performance. We also aimed at understanding andvalidating the user experience by analyzing the subjective
reports from the users.
Figure 4. The schematic diagram of the system architecture

trajectory to change drastically within the system latency. In apreliminary study, we identiﬁed that the discrepancy of the position caused by the latency signiﬁcantly degrades the system
performance. To suppress the effect of the latency, we need to
predict the user’s motion.
We measured the duration from when the motion capture system acquired the position data of the object held by the gripper
until the object was released. We deﬁned this duration as thesystem latency. This latency was about 40–50 milliseconds
with our setup. We concluded that this latency is composed
mainly of two factors: 1) the duration required for the communication process between the microcontroller via the PC fromthe motion capture cameras and 2) the duration from when
the trigger is received until the motor is accelerated and theobject is released. In our setup, we compensated the device’s
movement in the former duration using the integrated value ofthe onboard acceleration sensor. The movement in the latter
duration needs to be compensated by predicting future motionthat has not yet occurred. We decided to predict the future
motion based on the assumption that the user’s arm movement
can be regarded as constant acceleration motion in a shorttime. We tuned and determined the ratio of these two factors
empirically.
USER STUDY AND RESULTS

We conducted user studies to validate the feasibility of the
proposed HMMA approach and to understand the users’ subjective experiences. We evaluated the system by measuring
the landing position of the projectile and collecting the users’
feedback. First, in the situation where the user was aiming
at the target, we compared the case where the user manuallycontrolled the release timing and the case where the system
controlled the release. We also analyzed the qualitative aspects
based on the user’s feedback in this study. In the second study,
system performance was investigated in a situation where theuser performed throwing without aiming at the target. We also
demonstrated the system at an exhibition, observed the users’
behaviors, and analyzed the users’ feedback and experience.
All experiments complied with the safety standards approvedby the Local Ethics Research Committee at the Universityof Tokyo, Japan. Moreover, all the participants signed a letter of consent after they were provided with an overview ofthe experiment and instructions. The study protocol was per-

Paper 576

Figure 5. The experimental setup for the system evaluation: we predeﬁned three targets line depicted as three dots (red: 2.5 m, blue: 3.5 m
and green: 4.5 m), and installed a marker for a guide in arm swing.

Setup

The experimental setup is shown in Figure 5. The system described in the system design section was installed in an indoor
room. We predeﬁned three targets as shown in Figure 5. The
threshold on the x-axis was set to 40 mm. In this study, since
the accuracy in the y-axis direction was not considered, thethreshold on the y-axis was set to 1000 mm. The system was
set to select one of these targets for each trial in a randomizedorder. The projectile was a sphere ball with retro-reﬂective
markers for the motion capture weighing about 170 g andabout 68 mm in diameter.
The system has two modes: 1) auto mode and 2) manual mode.
In the auto mode, the projectile was released automaticallywith the computational control according to the user’s motion.
In the manual mode, the projectile was released by pushing thedevice’s button. In this study, we aim to investigate not only
the performance but also the effect of adjusting the releasetiming by the system on the user experience. To this end, we
asked the user to push the button in both modes and investigated how much the user could recognize the intervention of
the computational process. In this study, 1) the release was
executed after the computed optimal time if the user pushed abutton in the auto mode, and 2) the release was executed after
a ﬁxed time if the user pushed a button in the manual mode.
We introduced this ﬁxed time aiming to make the conditions
closer in the auto and manual modes. We set this time to 75 ms
empirically and concluded that this delay was short enoughsuch that the users were unaware of it and that it would notaffect their sense and performance. In addition, we designed
the system to not execute the release if the button was pushed
at a timing when the projectile did not reach 1.5 m in the computation, or the button was pushed at a timing later than thecomputed optimal timing. In other words, it was designed not
to be released if the button was pushed too early or too late

Page 5

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

during motion in both modes. This was designed to preventthe user from recognizing the current mode of the system due
to a mistaken button push.
Participants

We recruited nine participants (6 male, 3 female, mean=20.9
years old, SD=2.0). There were eight right-handed participants and one left-handed participant. No participants had any
physical disability.
Procedure

Before the experiment, we set up a training session to help
participants become familiar with the system. The participants
were asked to operate the device with their dominant hand.
First, the experimenter demonstrated how to use the systemand swing the arm. The participants were instructed to remain
standing and to swing their arm holding the device without
bending their elbow. Second, the participant swung the device
repeatedly without setting the projectile in the gripper until
the gripper was open in the auto mode. At this time, a marker
position depicted in Figure 5 was set as the target. Finally, theparticipants actually performed throwing with the device ﬁvetimes each in the auto and manual modes. To ensure that the
projectile obtained sufﬁcient kinetic energy, the participants
were instructed to follow through with the swing. In this actual
throwing step, the target was selected in a randomized order.
At these trials, participants were informed in which mode
(auto mode or manual mode) the system was running.
We deﬁned that one throwing action was one trial, and a setof 18 trials was one set. The participants performed three sets
in manual mode and three sets in auto mode in a randomizedorder. For each trial, the participants were assigned one ofthree targets and instructed to aim at the target. In all trials,
the participants were instructed to push the button aiming at
the assigned target, whether in the auto or manual mode. The
participants were instructed to push the device’s button torelease the projectile in all trials but they were not informed
whether the current system was in the auto or manual mode.
The landing position of the projectile was measured for eachtrial. To alleviate fatigue, the participants were allowed totake a break between each set. After each set, the participants
were asked if they were able to recognize the mode in which
the system was running in the ﬁnal set. After all sets, weinterviewed the participants about the experience.
Results of User study 1

We collected 486 trials in each auto mode and manual mode
from User study 1. The collected data are labeled into three
types according to the assigned target. Figure 6 and Table 1 depict the collected results in User study 1. There are three peaksin the histograms of the trials in the auto mode (Figure 6 (a).
The statistical signiﬁcance α was determined at a two-sided pvalue of ≤ 0.05. The distances were signiﬁcantly different according to the assigned target on both modes (Kruskal–Wallis
test, p < 0.01). Signiﬁcant differences are found in all combination of targets (p < 0.001) except between 3.5 and 4.5 m
targets in manual mode (p = 0.076). We found that the system performs throwing selectively according to the assigned
target. Compared with the auto mode result, the manual mode
result has a larger variance, as shown in Figure 6 (b). The

Paper 576

Figure 6. Histogram and ﬁtted Gaussian curves of the projectile landing
position under conditions where the experiment participants aimed at
the target (a) in the auto mode and (b) in the manual mode.

difference in variance between the auto and manual modes
was analyzed. The variances were lower with the auto mode
than with the manual mode in all combinations of participants
and targets. A signiﬁcant difference in the variance between
the auto and manual modes was found (Wilcoxon signed-rank
test, p < 0.001). Although this result is not surprising given
the fact that, in the manual mode, the users must perform all
operations manually with the novel system. It indicates thatthe users may get used to the system immediately, and our
method may assist the user’s manual operation.
In addition, we observed that the projectile tends to land on
the near side of the target positions. We concluded that one of
the reasons for this can be errors of the model adopted in oursystem and hyperparameters in the prediction process. The
results also indicate that the variance increases as the distanceto the target increases. The cause for this is the fact that theinﬂuence of the initial state deviation increased in addition to
the above errors due to the longer ﬂight duration. As Table 1
shows, the variances of the results for each participant tends
to be smaller than that from the total trials of all participants.
We observe that the total variance becomes larger due to thedifferences among the means of each participant’s result. It canbe caused by the fact that the system developed in this paper
was not optimized for each participant. We have identiﬁedthat by introducing a calibration process for each individual,

Page 6

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

ID

Mode

2.5 m

3.5 m

4.5 m

1

A
M

2.25 ± 0.12
2.20 ± 0.97

2.89 ± 0.15
3.55 ± 1.23

3.48 ± 0.45
3.13 ± 1.15

2A
M

2.46 ± 0.18
2.40 ± 0.65

3.33 ± 0.45
3.11 ± 0.78

3.96 ± 0.58
3.79 ± 1.25

3A
M

2.65 ± 0.32
2.83 ± 0.70

3.55 ± 0.36
3.22 ± 0.70

4.20 ± 0.38
4.33 ± 1.37

4A
M

2.41 ± 0.18
2.69 ± 0.91

3.12 ± 0.21
2.98 ± 1.15

3.86 ± 0.34
3.19 ± 1.23

5A
M

3.68 ± 0.42
2.54 ± 0.68

3.35 ± 0.54
3.15 ± 0.76

4.32 ± 0.28
4.11 ± 0.89

6A
M

2.70 ± 0.40
2.30 ± 0.53

3.51 ± 0.59
3.46 ± 1.32

4.02 ± 0.94
3.27 ± 1.56

7

AM

2.52 ± 0.18
1.86 ± 0.29

3.44 ± 0.16
2.13 ± 0.80

4.07 ± 0.14
2.24 ± 0.87

8AM

2.97 ± 0.38
2.26 ± 0.42

3.65 ± 0.33
2.68 ± 0.65

4.38 ± 0.26
3.01 ± 1.03

9A
M

2.67 ± 0.24
2.23 ± 0.28

3.46 ± 0.47
3.09 ± 0.53

4.12 ± 0.50
3.41 ± 0.84

All

A
M

2.59 ± 0.34
2.38 ± 0.69

3.37 ± 0.44
3.04 ± 0.98

4.05 ± 0.53
3.39 ± 1.27

Table 1. Means and standard deviations of the results in each participant and the total result of all participants in User study 1 (in the condition where the participants were aiming at the assigned target). A: automode, M: manual mode.

the target in front! Amazing.” From this example, we foundthat the system might correct human errors.
User study 2: Without aiming at the target

The goal of this study is to evaluate the feasibility of thedeveloped system in a situation where the user does not aim at
the target.
Setup

The study setup is the same as User study 1. The system was
executed in the auto mode in all trials.
Participants

We recruited seven participants (7 male, mean=24.1 years
old, SD=1.1). There were six right-handed participants andone left-handed participant. All participants did not have aphysical disability.
Procedure

The participants were instructed to operate the device as inUser study 1. Each participant performed 3 sets, with 18
throwing actions as one set. To ensure that the projectile gets
sufﬁcient kinetic energy, the participants were instructed toswing their arm aiming toward a marker depicted in Figure 5
in all trials. The participants were not informed of the target
position of the system aiming for each trial. As in User study
1, the projectile’s landing position was measured, and theparticipants were allowed to take a break between each set.

this bias can be reduced, and the system performance can be
improved.
Qualitative ﬁndings

We describe the qualitative results based on the interview andthe user’s feedback, in User study 1. First, we asked theparticipants if they were able to recognize the mode (auto or
manual mode) in which the system was running in each set.
In most cases, the participants were aware of which modethe system was running in. We found that the users wereable to judge the mode from the results of throwing. On
the other hand, they could not judge from the release timing
difference. For example, in the interview, User 6 reported, “Ithought it would be computer-controlled if the accuracy was
good.” However, interestingly, there was a case where theuser incorrectly answered the question about the mode. User
4 reported, “I can’t recognize it in the ﬁrst set, but when I
did the second set, it was clearly less accurate than the ﬁrst
set. So I thought the current mode was manual and the ﬁrst
set was in the auto mode.” This suggests that the user cannot
perceive the computational process intervention on each trial
alone. In other words, we found that the system can createa situation where the operation is performed by a computer
process even though the users think that they are performingby themselves. In addition, unexpectedly, there was a trial
where a user aimed at a position different from the assigned
target. In the interview, User 3 reported, “I tried to throw it to
the target behind by mistake, but it was controlled and fell on

Paper 576

Figure 7. Histogram and ﬁtted Gaussian curves of the projectile landing
position under the auto mode condition where the experiment participants did not aim at the target.

Results of User study 2

We collected 378 trials and labeled them into three types according to the assigned target. Figure 7 depicts the collected
results in User study 2. There are three peaks similar to theresult of the auto mode in User study 1. Signiﬁcant differences
are found in all combinations of targets (p < 0.001) with Steel–
Dwass test. The results suggest that the system can perform
throwing even if the user does not pay attention to the target
or is in a blind condition. We observe a difference between
the two results in auto mode. Unexpectedly, the group where
the users did not aim at the target threw more precisely. Thisis probably because the constant acceleration model used inPage 7

CHI 2020 Paper

CHI 2020, April 25–30, 2020, Honolulu, HI, USA
Lessons learned

Figure 8. Snapshot of the demonstration at an public exhibition. Twotypes of targets were installed (a box and nine numbers panels).

this setup is better suited to swing aiming toward the far sideof the target.
In the current system, the computed projectile trajectory andthe actual trajectory do not match in the order of a centimeter accuracy. This can be caused by various factors such asmeasurement error, unexpected user movement, variations of
the contact condition between the projectile and the gripper,and so on. For example, the system adopted a dynamic model
where air resistance is ignored. Therefore, depending on the
shape of an object and the ﬂight duration, it will deviate signiﬁcantly from the computed trajectory. It will be possible
to improve accuracy by updating the current model to moredetailed models with an object’s physical properties.
Demonstration at a public exhibition

For an informal validation study, we demonstrated the developed system at a public exhibition [17] to observe the users’
behavior and obtain feedback.
As shown in Figure 8, two types of targets were set. All
participants were given a brief overview of the system andwere instructed on how to use the device. The participants
were also instructed to hold the device with their dominant
hand and to swing their arms like throwing underhand. In alltrials, the system was executed in auto mode. The object to be
thrown was a sphere ball with a diameter of about 65 mm and aweight of about 130 g. The target was chosen by the participant
for each throwing, and they were allowed to interact with thesystem for as long as they liked. After the experience, withthe consent of the participant, the following questionnaire in a
7-item Likert scale (from 1: "strongly disagree" to 7: "strongly
agree"): Q1) I felt throwing ball with the handheld device wasfun. Q2) I felt accomplishment by throwing the ball using thehandheld device.
Findings from the demonstration

One-hundred thirty participants aged from under 20 to over
60 years completed the questionnaire. Most participants gave
positive answers to the questions (mean=6.02, SD=1.24 forQ1, mean=5.58, SD=1.34 for Q2). Some participants reported
that they felt as if they controlled the release timing while
the system was run in auto mode. This fact indicates that thesystem can not only perform dynamic motor skills but can alsoenhance the subjective experience with the HMMA approach.

Paper 576

When the user swung the device very quickly, we sometimes
observed a case when the projectile was not released. We
found that this could be caused by a spatial resolution of the
trajectory step size. In the implemented system, the computed
projectile trajectories were discretized with the sampling rate.
When the device moves faster, the spatial step size between the
trajectories become larger. As a result, the trajectory can jumpover the target area in one timestep. To address this issue, we
could enlarge the threshold; however, it has a trade-off for system accuracy. In other words, a signiﬁcantly small threshold
can cause the rare release while the system accuracy can belower with a larger threshold. We consider that employing ahigher sampling rate or a dynamic tunable threshold according
to the step size will be a useful approach for this issue.
We observed several cases where the ball fell before the target
at the early stage of the user’s experience. We found that one
reason for this can be that the kinetic energy of the users’ motion was overestimated by the prediction process even though
it did not have enough energy to obtain the desired ﬂight distance. However, the number of such cases decreased after
several trials, suggesting the possibility that the user can adapt
to the system in a short time.
DISCUSSION AND LIMITATION

Throughout the results, we revealed three key challenges to be
tackled for designing HMMA systems.
First, we must consider how the HMMA system suppresses
disturbances on computational processing caused by unintentional human motion (errors). For example, the developed
device employs a gripper that moves only laterally in the local
coordinate. When the user performed an arm-twisting motion
to throw, we found that the gripper affected the behavior of theobject being thrown out. This is because the object remains
in contact with the gripper, although the system has released
the object. The users can unintentionally disturb the system
performance, even if the users recognize this phenomenon.
For dealing with such human disturbance, active stabilization
techniques will be effective (e.g., gimbal mehcanisms [29]).
Second, the HMMA system needs to particularly construct amodel of each user’s individual motions with an appropriate
latency setting, which is the key element improving the system performance. With the current system, we struggled tosuppress the effect of system latency. Through developing thesystem, we identiﬁed that humans can accurately perform theirdynamic motor skills with their self-motion prediction models
as much as possible. Therefore, a user-speciﬁc calibration
will be highly required, utilizing advanced motion prediction
models or machine learning-based techniques as [7, 19].
Third, we need to develop a method of inducing humans tomove “correctly” as deﬁned by the HMMA system. Our observations suggest that a user’s undesired behaviors might leadto a signiﬁcant decrease in system performance. We consider
that one way to address this issue can be to design an incentivethat occurs in the user experience. Our results indicate that theuser’s performance can be controlled without losing his/her
sense of control. This suggests that a system (designed appro-

Page 8

CHI 2020 Paper

priately) can provide a sense of achievement or motivation to
the users and induce them into doing desired behaviors deﬁned
by the system.
In this research, we developed a prototype system to perform
throwing based on the HMMA concept. Through the development and user studies using the system, we have obtainedsome ﬁndings about interactions and challenges to address.
Our research complements the ﬁndings for the interaction inhuman augmentation researches utilizing robotics techniques,
which mainly have focused on static situations [28, 33]. We
found that our proposed approach can also contribute to therobotics domain by compensating for the disadvantages ofrobots using human actuation. For example, object graspingis one of the challenging issues in robotics [14, 45] whereas,
humans can recognize and grasp unfamiliar objects without
prior knowledge. We observed that the users did not struggle
to grab objects using the developed device. On the other hand,a robotic hand could manipulate objects that are dangerous fora human hand to touch directly.
FUTURE WORKCHI 2020, April 25–30, 2020, Honolulu, HI, USA

investigation for the human-robot hybrid system including
overcoming disabilities and human physical augmentation.
ACKNOWLEDGMENTS

We would like to thank Hiroto Saito, Ken Arai, and Riku
Arakawa for useful advice and their help with the systemdemonstration. This project was supported by JST ERATO
Grant Number JPMJER1701, Japan.
REFERENCES[1] Ismail Ben Abdallah, Yassine Bouteraa, and Chokri
Rekik. 2017. Design and development of 3D printed
myoelectric robotic exoskeleton for hand rehabilitation.
International Journal on Smart Sensing & Intelligent
Systems 10, 2 (2017).
[2] Lung-Pan Cheng, Patrick Lühne, Pedro Lopes,
Christoph Sterz, and Patrick Baudisch. 2014. Haptic
turk: a motion platform based on people. In Proceedingsof the SIGCHI Conference on Human Factors in
Computing Systems. ACM, 3463–3472. DOI:
http://dx.doi.org/10.1145/2556288.2557101

Although we only focus on the throwing skill in this study, ouridea can be applied to other motor skills, such as catching andkicking motions, and to using static tools as well. Furthermore,
the idea will be expanded to situations that require more power
and force by using multi-human actuation. Inducing multiple
people’s movements cooperatively will offer a new challenge.

[3] Lung-Pan Cheng, Sebastian Marwecki, and Patrick
Baudisch. 2017. Mutual human actuation. In
Proceedings of the 30th Annual ACM Symposium on
User Interface Software and Technology. ACM,
797–805. DOI:

Application to the robotic prosthesis will be one of the promising avenues. The experimental results indicate the possibility
that the motor skill performance can be assisted via our proposed approach without losing the sense that the users arecontrolling the robotic hand themselves. By installing a computational environment, our approach can be used to augment
the existing robotic limbs and provide an enhanced user experience.

[4] Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch,
Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes
Jasper, Jonas Kemper, and Patrick Baudisch. 2015.
Turkdeck: Physical virtual reality based on people. In
Proceedings of the 28th Annual ACM Symposium on
User Interface Software & Technology. ACM, 417–426.
DOI:http://dx.doi.org/10.1145/2807442.2807463

CONCLUSION

We proposed a new concept, HMMA coupling human actuation with robotic capability. Concretely, we investigated thefeasibility of our concept and designing of particular interaction settings. Particularly focusing on the throwing underhand
motion as a case study, we proposed a method to control therelease timing in throwing using the computational processaccording to human motion. The developed system consistsof a robotic handheld device that acts as a release controller
and the control architecture to perform throwing. The handheld device has one actuated DoF and its gripper is controlled
based on the computational process. The device position istracked and estimated with the combination of the data fromthe motion capture system and the onboard IMU sensor. To
validate the feasibility of the proposed concept and clarify theissues to be tackled, we conducted user studies. We foundthat the proposed system successfully performs throwing according to the assigned target with exploiting human ability.
The results suggest that, in our approach, the robotic capability can be embedded into the user’s motion without losing a
sense of control. We hope that our work will lead to additional

Paper 576

http://dx.doi.org/10.1145/3126594.3126667[5] Seongkook Heo, Christina Chung, Geehyuk Lee, andDaniel Wigdor. 2018. Thor’s hammer: An ungrounded
force feedback device utilizing propeller-induced
propulsive force. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems.
ACM, 525. DOI:
http://dx.doi.org/10.1145/3173574.3174099[6] Jon Hore and Sherry Watts. 2011. Skilled throwers use
physics to time ball release to the nearest millisecond.
Journal of Neurophysiology 106, 4 (2011), 2024–2033.
DOI:http://dx.doi.org/10.1152/jn.00059.2011[7] Yuuki Horiuchi, Yasutoshi Makino, and Hiroyuki
Shinoda. 2017. Computational foresight: forecasting
human body motion in real-time for reducing delays ininteractive system. In Proceedings of the 2017 ACM
International Conference on Interactive Surfaces andSpaces. ACM, 312–317. DOI:
http://dx.doi.org/10.1145/3132272.3135076[8] Jian Huang, Weiguang Huo, Wenxia Xu, Samer
Mohammed, and Yacine Amirat. 2015. Control ofupper-limb power-assist exoskeleton using ahuman-robot interface based on motion intention

Page 9

CHI 2020 Paper

recognition. IEEE transactions on automation scienceand engineering 12, 4 (2015), 1257–1270. DOI:
http://dx.doi.org/10.1109/TASE.2015.2466634[9] Yuta Itoh, Jason Orlosky, Kiyoshi Kiyokawa, andGudrun Klinker. 2016. Laplacian vision: Augmenting
motion prediction via optical see-through head-mounted
displays. In Proceedings of the 7th Augmented Human
International Conference 2016. ACM, 16. DOI:
http://dx.doi.org/10.1145/2875194.2875227[10] Urs Keller, Hubertus JA van Hedel, Verena
Klamroth-Marganska, and Robert Riener. 2016.
ChARMin: The ﬁrst actuated exoskeleton robot forpediatric arm rehabilitation. IEEE/ASME Transactions
on Mechatronics 21, 5 (2016), 2201–2213. DOI:
http://dx.doi.org/10.1109/TMECH.2016.2559799[11] Jinsoo Kim, Giuk Lee, Roman Heimgartner,
Dheepak Arumukhom Revi, Nikos Karavas, Danielle
Nathanson, Ignacio Galiana, Asa Eckert-Erdheim,
Patrick Murphy, David Perry, and others. 2019.
Reducing the metabolic rate of walking and runningwith a versatile, portable exosuit. Science 365, 6454
(2019), 668–672. DOI:
http://dx.doi.org/10.1126/science.aav7536[12] Jan Koprnickỳ, Petr Najman, and Jiří Šafka. 2017. 3D
printed bionic prosthetic hands. In 2017 IEEE
International Workshop of Electronics, Control,
Measurement, Signals and their Application to
Mechatronics (ECMSM). IEEE, 1–6. DOI:
http://dx.doi.org/10.1109/ECMSM.2017.7945898[13] Benjamin Lafreniere, Tovi Grossman, Fraser Anderson,
Justin Matejka, Heather Kerrick, Danil Nagy, Lauren
Vasey, Evan Atherton, Nicholas Beirne, Marcelo H
Coelho, and others. 2016. Crowdsourced fabrication. In
Proceedings of the 29th Annual Symposium on User
Interface Software and Technology. ACM, 15–28. DOI:
http://dx.doi.org/10.1145/2984511.2984553[14] Ian Lenz, Honglak Lee, and Ashutosh Saxena. 2015.
Deep learning for detecting robotic grasps. The
International Journal of Robotics Research 34, 4-5(2015), 705–724. DOI:
http://dx.doi.org/10.1177/0278364914549607

CHI 2020, April 25–30, 2020, Honolulu, HI, USA

Emerging Technologies. ACM, 20. DOI:
http://dx.doi.org/10.1145/3305367.3327996[18] Azumi Maekawa, Shota Takahashi, MHD Saraiji, Sohei
Wakisaka, Hiroyasu Iwata, and Masahiko Inami. 2019b.
Naviarm: Augmenting the Learning of Motor Skills
using a Backpack-type Robotic Arm System. In
Proceedings of the 10th Augmented Human
International Conference 2019. ACM, 38.
[19] Julieta Martinez, Michael J Black, and Javier Romero.
2017. On Human Motion Prediction Using Recurrent
Neural Networks. In 2017 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR).
IEEE, 4674–4683. DOI:
http://dx.doi.org/10.1109/CVPR.2017.497[20] Kenichi Murakami, Yuji Yamakawa, Taku Senoo, andMasatoshi Ishikawa. 2015. Motion planning for catchinga light-weight ball with high-speed visual feedback. In
2015 IEEE International Conference on Robotics andBiomimetics (ROBIO). IEEE, 339–344. DOI:
http://dx.doi.org/10.1109/ROBIO.2015.7418790[21] Jun Nishida, Shunichi Kasahara, and Kenji Suzuki. 2017.
Wired muscle: generating faster kinesthetic reaction byinter-personally connecting muscles. In ACM
SIGGRAPH 2017 Emerging Technologies. ACM, 26.
DOI:http://dx.doi.org/10.1145/3084822.3084844[22] Kei Nitta, Keita Higuchi, and Jun Rekimoto. 2014.
HoverBall: augmented sports with a ﬂying ball. In
Proceedings of the 5th Augmented Human International
Conference. ACM, 13. DOI:
http://dx.doi.org/10.1145/2582051.2582064[23] Kei Nitta, Keita Higuchi, Yuichi Tadokoro, and Jun
Rekimoto. 2015. Shepherd pass: ability tuning foraugmented sports using ball-shaped quadcopter. In
Proceedings of the 12th International Conference on
Advances in Computer Entertainment Technology. ACM,
11. DOI:http://dx.doi.org/10.1145/2832932.2832950[24] Takuya Nojima, Dairoku Sekiguchi, Masahiko Inami,
and Susumu Tachi. 2002. The SmartTool: A system foraugmented reality of haptics. In Proceedings IEEE
Virtual Reality 2002. IEEE, 67–72. DOI:
http://dx.doi.org/10.1109/VR.2002.996506[15] Baldin Llorens-Bonilla, Federico Parietti, and H Harry
Asada. 2012. Demonstration-based control ofsupernumerary robotic limbs. In 2012 IEEE/RSJ
International Conference on Intelligent Robots andSystems. IEEE, 3936–3942. DOI:
http://dx.doi.org/10.1109/IROS.2012.6386055[25] Tomoya Ohta, Shumpei Yamakawa, Takashi Ichikawa,and Takuya Nojima. 2014. TAMA: development oftrajectory changeable ball for future entertainment. In
Proceedings of the 5th Augmented Human International
Conference. ACM, 50. DOI:
http://dx.doi.org/10.1145/2582051.2582101[16] Sebastian Madgwick. 2010. An efﬁcient orientation ﬁlter
for inertial and inertial/magnetic sensor arrays. Report
x-io and University of Bristol (UK) 25 (2010), 113–118.
[17] Azumi Maekawa, Seito Matsubara, Atsushi Hiyama,and Masahiko Inami. 2019a. PickHits: hitting
experience generation with throwing motion via a
handheld mechanical device. In ACM SIGGRAPH 2019

Paper 576

[26] Federico Parietti, Kameron Chan, and H Harry Asada.
2014. Bracing the human body with supernumeraryrobotic limbs for physical assistance and load reduction.
In 2014 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 141–148. DOI:
http://dx.doi.org/10.1109/ICRA.2014.6906601

Page 10

CHI 2020 Paper[27] Luka Peternel, Tadej Petriˇc, Erhan Oztop, and Jan Babiˇc.
2014. Teaching robots to cooperate with humans indynamic manipulation tasks based on multi-modal
human-in-the-loop approach. Autonomous robots 36, 1-2(2014), 123–136. DOI:
http://dx.doi.org/10.1007/s10514-013-9361-0

CHI 2020, April 25–30, 2020, Honolulu, HI, USA[37] Taku Senoo, Akio Namiki, and Masatoshi Ishikawa.
2004. High-speed batting using a multi-jointed
manipulator. In IEEE International Conference on
Robotics and Automation, 2004. Proceedings. ICRA’04.
2004, Vol. 2. IEEE, 1191–1196. DOI:
http://dx.doi.org/10.1109/ROBOT.2004.1307986[28] Domenico Prattichizzo, Monica Malvezzi, Irfan Hussain,
and Gionata Salvietti. 2014. The sixth-ﬁnger: a modular
extra-ﬁnger to enhance human hand capabilities. In The
23rd IEEE International Symposium on Robot andHuman Interactive Communication. IEEE, 993–998.
DOI:http://dx.doi.org/10.1109/ROMAN.2014.6926382[38] Taku Senoo, Akio Namiki, and Masatoshi Ishikawa.
2006. Ball control in high-speed batting motion using
hybrid trajectory generator. In Proceedings 2006 IEEE
International Conference on Robotics and Automation,
2006. ICRA 2006. IEEE, 1762–1767. DOI:

[29] RJ Rajesh and P Kavitha. 2015. Camera gimbal
stabilization using conventional PID controller andevolutionary algorithms. In 2015 International
Conference on Computer, Communication and Control(IC4). IEEE, 1–6. DOI:

[39] Taku Senoo, Akio Namiki, and Masatoshi Ishikawa.
2008. High-speed throwing motion based on kinetic
chain approach. In 2008 IEEE/RSJ International
Conference on Intelligent Robots and Systems. IEEE,
3206–3211. DOI:

http://dx.doi.org/10.1109/IC4.2015.7375580[30] Robert Riener. 2016. The Cybathlon promotes thedevelopment of assistive technology for people withphysical disabilities. Journal of neuroengineering andrehabilitation 13, 1 (2016), 49. DOI:
http://dx.doi.org/10.1186/s12984-016-0157-2[31] Alec Rivers, Ilan E Moyer, and Frédo Durand. 2012.
Position-correcting tools for 2D digital fabrication. ACM
Transactions on Graphics (TOG) 31, 4 (2012), 88. DOI:
http://dx.doi.org/10.1145/2185520.2185584[32] Mark Rober. 2017. Automatic Bullseye, MOVINGDARTBOARD.
https://www.youtube.com/watch?v=MHTizZ_XcUM. (2017).
Accessed: 2019-09-10.
[33] MHD Saraiji, Tomoya Sasaki, Kai Kunze, Kouta
Minamizawa, and Masahiko Inami. 2018a. MetaArmS:
Body remapping using feet-controlled artiﬁcial arms. In
The 31st Annual ACM Symposium on User Interface
Software and Technology. ACM, 65–74. DOI:
http://dx.doi.org/10.1145/3242587.3242665[34] MHD Saraiji, Tomoya Sasaki, Reo Matsumura, Kouta
Minamizawa, and Masahiko Inami. 2018b. Fusion: fullbody surrogacy for collaborative communication. In
ACM SIGGRAPH 2018 Emerging Technologies. ACM,
7. DOI:http://dx.doi.org/10.1145/3214907.3214912[35] Koya Sato, Yuji Sano, Mai Otsuki, Mizuki Oka, andKazuhiko Kato. 2019. Augmented Recreational
Volleyball Court: Supporting the Beginners’ Landing
Position Prediction Skill by Providing Peripheral VisualFeedback. In Proceedings of the 10th Augmented
Human International Conference 2019. ACM, 15. DOI:
http://dx.doi.org/10.1145/3311823.3311843[36] Taku Senoo, Yuuki Horiuchi, Yoshinobu Nakanishi,
Kenichi Murakami, and Masatoshi Ishikawa. 2016.
Robotic pitching by rolling ball on ﬁngers for arandomly located target. In 2016 IEEE International
Conference on Robotics and Biomimetics (ROBIO).
IEEE, 325–330. DOI:

http://dx.doi.org/10.1109/ROBOT.2006.1641961

http://dx.doi.org/10.1109/IROS.2008.4651142[40] Tatsuya Teramae, Koji Ishihara, Jan Babič, Jun
Morimoto, and Erhan Oztop. 2018. Human-in-the-loop
control and task learning for pneumatically actuatedmuscle based robots. Frontiers in neurorobotics 12(2018). DOI:
http://dx.doi.org/10.3389/fnbot.2018.00071[41] Junichi Yamaoka and Yasuaki Kakehi. 2013. dePENd:
augmented handwriting system using ferromagnetism of
a ballpoint pen. In Proceedings of the 26th annual ACM
symposium on User interface software and technology.
ACM, 203–210. DOI:
http://dx.doi.org/10.1145/2501988.2502017[42] Hiroaki Yano, Masayuki Yoshie, and Hiroo Iwata. 2003.
Development of a non-grounded haptic interface using
the gyro effect. In 11th Symposium on Haptic Interfaces
for Virtual Environment and Teleoperator Systems, 2003.
HAPTICS 2003. Proceedings. IEEE, 32–39. DOI:
http://dx.doi.org/10.1109/HAPTIC.2003.1191223[43] Hong Kai Yap, Jeong Hoon Lim, Fatima Nasrallah,
James CH Goh, and Raye CH Yeow. 2015. A soft
exoskeleton for hand assistive and rehabilitation
application using pneumatic actuators with variable
stiffness. In 2015 IEEE international conference on
robotics and automation (ICRA). IEEE, 4967–4972.
DOI:http://dx.doi.org/10.1109/ICRA.2015.7139889[44] Hironori Yoshida, Takeo Igarashi, Yusuke Obuchi,
Yosuke Takami, Jun Sato, Mika Araki, Masaaki Miki,
Kosuke Nagata, Kazuhide Sakai, and Syunsuke Igarashi.
2015. Architecture-scale human-assisted additive
manufacturing. ACM Transactions on Graphics (TOG)
34, 4 (2015), 88. DOI:
http://dx.doi.org/10.1145/2766951[45] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon,
Francois R Hogan, Maria Bauza, Daolin Ma, Orion
Taylor, Melody Liu, Eudald Romo, and others. 2018.
Robotic pick-and-place of novel objects in clutter with

http://dx.doi.org/10.1109/ROBIO.2016.7866343

Paper 576

Page 11

CHI 2020 Paper

multi-affordance grasping and cross-domain image
matching. In 2018 IEEE International Conference on
Robotics and Automation (ICRA). IEEE, 1–8. DOI:
http://dx.doi.org/10.1109/ICRA.2018.8461044[46] Juanjuan Zhang, Pieter Fiers, Kirby A Witte, Rachel W
Jackson, Katherine L Poggensee, Christopher G
Atkeson, and Steven H Collins. 2017.
Human-in-the-loop optimization of exoskeleton
assistance during walking. Science 356, 6344 (2017),
1280–1284. DOI:

CHI 2020, April 25–30, 2020, Honolulu, HI, USA[47] Amit Zoran and Joseph A Paradiso. 2013. FreeD: afreehand digital sculpting tool. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems. ACM, 2613–2616. DOI:
http://dx.doi.org/10.1145/2470654.2481361[48] Amit Zoran, Roy Shilkrot, and Joseph Paradiso. 2013.
Human-computer interaction for hybrid carving. In
Proceedings of the 26th annual ACM symposium on
User interface software and technology. ACM, 433–440.
DOI:http://dx.doi.org/10.1145/2501988.2502023

http://dx.doi.org/10.1126/science.aal5054

Paper 576

Page 12