1
2
3

Whose touch is this?: Understanding the Agency Trade-off Between
User-driven touch vs. Computer-driven Touch

4
5
6
7
8
9
10

DAISUKE TAJIMA, Sony Group Corporation, Sony Computer Science Laboratories, Inc., Japan
JUN NISHIDA, University of Chicago, United States
PEDRO LOPES, University of Chicago, United States
SHUNICHI KASAHARA, Sony Computer Science Laboratories, Inc., Japan11

I did it!

14
15

L or R

16
17
18
19
20

Human AgentEMS correct

Machine Agent

13

user correctuser incorrect

Joint Success

Forced SuccessagencyEMS
fasteragencyUser
fasterEMS
faster

EMS timing

EMS
Interface

21
22

Electrical muscle
stimulation(EMS) actuates
the user's muscles to press
gamepad buttons, even
before the user presses

EMS incorrect

12

Forced Failure
agencyEMS
fasterUser
faster
EMS timing

Joint Failure
agencyUser
fasterEMS
faster

EMS timing

User
faster
EMS timing

23
24
25
26
27
28
29
30
31
32
33
34

Fig. 1. The paradox of adding force-feedback to digital touch systems is that the haptic device needs to force the user to touch, fasterthan they normally would—so that the resulting touch is computer-driven, rather than user-driven—while this can have a range ofbenefits, it unfortunately also causes users to lose their sense of agency. To contribute to the improvement of digital touch systems, weinvestigated how the user’s sense of agency changes when their own touches are assisted by electrical muscle stimulation (EMS). We
went deeper than previous work by investigating situations where the user has a choice on how to touch (see matrix). For instance, weinvestigated what happens not only when the user-driven touch and the computer-driven touch are aligned (joint success), but, more
importantly, explored when the computer-driven touch forces the user into an outcome they did not intend (forced failure or a joint
failure), or conversely, when the computer-driven touch forces the user in an outcome they would not achieved alone (forced success).
We found different agency-profiles depending on whether the user-driven touch or the computer-driven touch is congruent or not; assuch, does the timing of the computer-driven touch influence the sense of agency, but the outcomes affect agency too.

35

Abstract: Force-feedback enhances digital touch by enabling users to share non-verbal aspects such as rhythm, poses, etc. To achieve

36

this, interfaces actuate the user’s to touch involuntarily (using exoskeletons or electrical-muscle-stimulation); we refer to this as

37
38
39
40

computer-driven touch. Unfortunately, forcing users to touch causes a loss of their sense of agency. While researchers found thatdelaying the timing of computer-driven touch preserves agency, they only considered the naïve case when user-driven touch is alignedwith computer-driven touch. We argue this is unlikely as it assumes we can perfectly predict user-touches. But, what about all the41

remainder situations: when the haptics forces the user into an outcome they did not intend or assists the user in an outcome they42

would not achieve alone? We unveil, via an experiment, what happens in these novel situations. From our findings, we synthesize a43framework that enables researchers of digital-touch systems to trade-off between haptic-assistance vs. sense-of-agency.

44
45
46
47
48
49
50

CCS Concepts: • Human-centered computing → Human computer interaction (HCI); Haptic devices; User studies.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party
components of this work must be honored. For all other uses, contact the owner/author(s).
© 2021 Copyright held by the owner/author(s).
Manuscript submitted to ACM

51
52

1

Sub., TOCHI, Digital Touch

53

Tajima, Nishida, Lopes and Kasahara

Additional Key Words and Phrases: touch, haptics, proprioception, agency, electrical muscle stimulation, automation, tactile

54
55

ACM Reference Format:

56

Daisuke Tajima, Jun Nishida, Pedro Lopes, and Shunichi Kasahara. 2021. Whose touch is this?: Understanding the Agency Trade-off

57

Between User-driven touch vs. Computer-driven Touch. In . ACM, New York, NY, USA, Article 1, 27 pages. https://doi.org/10.1145/

58

3489608

59
60

1

INTRODUCTION

61
62

While we typically think of touch as a depiction of haptic qualities such as contact & texture, recent advances in

63

force-feedback compel us to also consider the challenge of assisted-touch, i.e., a touch that is driven by the computer

64

interface via force feedback, which causes the users to touch their user interface involuntarily—we refer to this emergent

65
66

type of haptics as computer-driven touch.

67

On one hand, the addition of computer-driven touch to digital touch systems greatly enhances not only the sense68

of immersion but also interpersonal communicative capacity by enabling users to communicate non-verbal aspects

69

such as touching/tapping to a rhythm [60], speeding up users reaction times in a touch-based interaction [31], sharing

70
71

poses with a remote user [59], and even sharing emotions [25]. To deliver this type of proprioceptive information to72

their users, haptic interfaces actuate the user’s body, typically via exoskeletons [27, 46, 63, 78] or electrical muscle

73

stimulation (EMS) [42].

74
75
76

On the other hand, to realize computer-driven touch, these haptic interfaces need to actuate the user’s body bymoving it, even against the user’s intention. In fact, to provide users with these benefits (e.g., speeding up users’77

reaction times in touch-based interactions [31]), these devices need to actuate the user’s to touch faster than they normally

78

would—they force users to touch preemptively (otherwise, the user would finish the touch and not benefit from the79

improved outcome via the haptic assistance) [31].

80
81
82

Unfortunately, forcing the user to move causes users loose their sense of agency ( i.e, the sense of control of one’sbody) [4, 31, 51], which is particularly detrimental to both digital touch and interpersonal communication systems.

83

Moreover, while there are efforts designing and engineering haptic devices capable of actuating the user’s body, the84efforts focused on understanding this loss of agency and even engineering devices that mitigate this loss of agency are

85
86virtually nonexistent. As such, there is a critical, yet unanswered, challenge associated with the advent of computer-87

driven touch: Whose touch is this? or, in other words, to whom will users attribute these computed-driven touches? To

88

themselves or to the UI?

89
90
91

One promising approach to improve the lack of agency has been revealed recently, as researchers found out that
adjusting the timing of the computer-driven touch allows users to feel more in control [31]. In other words, delaying

92

the computer-driven touch timing to be closer to that of user-driven touch has been shown to preserve some of the93

user’s sense of agency, even when externally moved by means of a haptic device, such as an EMS-based interactive

94
95
96

device [31]. However, this prior work only considered the naive case in which the computer-driven touch is alignedwith the user-driven touch (joint success).

97

We argue that this is actually a rather unlikely case because it only works when we can predict what the user’s

98

touch will be, which is notoriously difficult and has eluded decades of HCI/Neuroscience research [48, 66]. So, what99

about all the remaining and more frequent situations depicted in Figure 1? For example, when the computer-driven

100
101

touch forces the user into an outcome they did not intended (forced failure or joint failure), or, conversely, when the102

computer-driven touch assists the user when they would have made a mistake by themselves (forced success). To

103

reveal what happens at the intersection of the user’s agency and computer-driven touch, we explore, by means of an104

2

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

105

experiment, what happens precisely in these novel situations in which the user actually has a choice and thus

106

the UI might correctly or incorrectly provide adequate or inadequate haptic assistance.

107

As such, in this article, we make a systematic attempt to understand the loss of agency in haptic interfaces and

108
109

propose methods designers can employ to mitigate these effects, allowing designers to effectively build better haptic

110

systems that can prioritize and/or trade-off between the user’s sense of agency vs. haptic assistance. We conduct our111

investigation of this loss of agency by using electrical muscle stimulation as the ideal case-study since its loss of agency

112
113
114

in computer-driven touch is well-known and, simultaneously, many see this stimulation technique as a promising
approach to wearable haptics.

115

To understand the trade-offs between the user’s sense of agency and computer’s haptic-assistance, we engineered116

a novel variation on the Stroop test [26, 50, 77], in which we introduced EMS as the computer-driven touch. In our

117
118
119

experiment, participants answered two-choice questions by tapping with either hand. Not only could they always try
to choose the correct answer voluntarily (user-driven touch), but an EMS device was engineered to also simultaneously

120

actuate their hand to answer (computer-driven touch) slower and/or faster than the user’s own reaction. As such, we

121

were able to make the haptics play two distinct roles: (1) assistive-touch (which will result in joint success or forced

122
123

success) or (2) adversarial-touch (forced failure or joint failure). Then, our Stroop experiment allowed us to investigate

124

the sense of agency in the four quadrants of computer-driven and user-driven touch. We found that the participant’s

125

attribution of action was more biased by the positive outcome, in which computer-driven touch and user-driven

126

touch were congruent. In contrast, in incongruent-touches, the attribution of action was clearly distinguished depending

127
128

simply on whether user-driven touch or computer-driven touch was faster to complete the task.

129

From our findings, we ultimately synthesize a framework that we then illustrate using a set of exemplary applications.

130

We expect that these examples, together with our underlying framework, will enable researchers, designers, and131

practitioners of future digital touch systems to prioritize or trade-off between the user’s sense of agency and the

132
133

computer’s haptic-assistance.

134
135
136
137
138
139

2

RELATED WORK

Our work builds primarily on the areas of digital touch, with particular emphasis on proprioceptive haptics using
actuators such as exoskeletons or electrical muscle stimulation (EMS). Furthermore, our investigative focus on agency
has its root in contemporary theories of the sense of agency originating from neuroscience and cognitive psychology.

140
141
142
143

2.1

Sense of agency

The sense of agency, which we define as the user’s sense of control over their own body, is a neural mechanism that144

drives one’s awareness of initiating, executing, and controlling our own voluntary actions in our surroundings [29].

145

The sense of agency allows us to recognize ourselves as the agent of a particular behavior, enabling us to build a self146

that is distinct from the external world [29]; it is thus considered one of the most primal mechanisms of the human

147
148
149

experience. Furthermore, while some conceptual HCI frameworks (such as [19, 53]) also consider the agency of aninteractive computer system, we focus solely on the agency of the user in our investigation.

150

In the field of neuroscience there are different views that suggest a neural mechanism for handling the sense of151

agency. One view, which is of relevance to our research, places the motor system (i.e., the neural apparatus that controls

152
153

voluntary movements) at the center of the question of agency [21]. Support for this comes from, for example, the154experiments conducted by Frith et al., where it was revealed that participants with lesions on the motor cortex (but no155

lesions on the sensory organs or pathways that allow to feel touch, etc.) exhibited a decreased sense of agency [21].

156

3

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

157

This theory states that the agency is dependent on the comparison between the prediction of a movement and its

158

outcome–as we will later see, this plays a key role in our investigation as haptic devices can change the outcome of

159
160

the action in ways the user did not intend, e.g., in false-positives cases, where the device incorrectly predicts the user’s

161

intention. Through our studies, we validate an interaction between outcome and the sense of agency, which we then162

use in our techniques for preserving agency even for systems that move users by means of EMS.

163
164
165

Moreover, while much is still not understood regarding the inner workings of our sense of agency, we do know thatthe sense of agency is not a static and immutable state, as many experiments have shown such as in the rubber

166

hand illusion [7, 36, 76], randomly generated feedback [17, 30], inversion of user’s movements [30], distortion of the167

haptic feedback [74], deception [61], and delays or speed-ups in feedback [17, 30].

168
169
170

As such, by drawing from neuroscience and cognitive psychology, we were able to leverage two interactions between the user’s sense of agency and two properties of the haptic interface that we postulated would

171

assist HCI designers in preserving the user’s sense of agency, even when users are moved so as to touch involuntarily

172

by a haptic interface (e.g., EMS). These were (1) haptic/touch timing (when does the haptic device start to move the173user); and (2) expected outcome of an action, such as a touch on a computer-interface–we succinctly present

174
175

the neuroscientific basis of each below.

176
177

2.1.1 Interaction between touch/movement timing and the sense of agency. A series of studies has shown that the178

perception of time is one of the elements that create the degree of association between a voluntary action (our intention)179

and outcome (the result of our intention); this has been shown to drive the sense of agency by Libet et al. [35] and

180
181

Haggard et al. [23]. These findings (e.g., Libet’s clock experiment) found the existence of a time delay between a

182

volitional action and our sense of agency for this action [47]. One key conclusion of all these studies in neuroscience is183

that "temporal sensorimotor discrepancies reduce the sense of agency" [14]. This has been shown by offsetting the184

onset of the expected outcome in visual, tactile, or auditory modalities [17]. Thus, time is a key in preserving the sense

185
186

of agency. We argue that this is the parameter that preemptive interactive systems must tune in order to provide more187

agency to their users.

188
189
190
191

2.1.2 Interaction between action outcome and the sense of agency. Secondly, not only does the sense of agency interact
with the timing of one’s action but, moreover, researchers in neuroscience have found it to also interact with the192

expected outcomes of one’s action. For instance, Yoshie et al. found that the emotional beep sound (e.g., negative like193

fear sound and positive like achievement sound) caused by the participant action could modify the sense of agency

194[79]. Barlas et al. found that the ease of expectation of the outcome boosts the sense of agency [3]. This interaction

195
196

between our sense of agency and our expected outcomes of a motor action is especially important for real usage of an

197

interactive system because users are free to act out their intentions (e.g., one can choose where to tap, which button to198

click, etc). However, when haptic systems can move the user’s body so as to generate touches for the user (such as what199

has been achieved with exoskeletons or EMS), this matter is further complicated as the computer system is now able to

200
201

actuate the user into outcomes that the user might not be able to predict or be interested in.

202

We argue that this is the crux of the agency question because UIs cannot predict the user’s intention accurately; in

203

fact, this is known to be notoriously difficult and has eluded decades of research. For instance, in neuroscience, many

204

argue this is a hard challenge for the field (e.g., "there is no evidence that the outcome of more complex free decisions

205
206
207
208

can be predicted from prior brain signals" [66]); similarly, some consider predicting outcomes the "holy grail" of sports
medicine [48]—still, no interactive system has shown how to accurately predict complex user intentions.
4Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

209
210
211

2.2

Sub., TOCHI, Digital Touch

Efforts in HCI to understand the user’s sense of agency

When designing interactive systems, the sense of agency is key to achieve a user experience that grants a sense of212

control to the user. Unfortunately, achieving this becomes more challenging with assistive systems that automate

213

the user’s behaviour in any way (haptic or otherwise). This has been observed for UIs that automatically change the214system to facilitate interactions [49, 51]. For instance, Coyle et al. found that changing the amount of assistance the

215
216
217

interactive system provides to the mouse cursor that the user is manipulating (e.g., via predictive mouse acceleration)
had a significant impact on the user’s sense of agency [13].

218

In fact, understanding the user’s sense of agency in modern UIs is so crucial to designing interactive systems that219

the HCI community has been deepening its understanding of it: McEneaney et al. found that machine-induced mouse

220
221

clicks affect the user’s perceived agency [49]; Coyle et al. found that the mouse cursor speed manipulations can affect222

agency [13]; Bergstrom-Lehtovirta et al. found that on-skin interaction improves the sense agency vs. a traditional

223

button press [5]; Martinez et al. found that the sense of agency varied during a mid-air touching task, across different

224

feedback modalities, such as visual, audio, or vibration [12]–these authors found that both auditory and haptic feedback

225
226

modalities were superior to the visual-only baseline in terms of increased sense of agency; along similar lines, Limerick

227

et al. showed that, in speech-based interfaces, the voice input modality reduces agency when compared to keyboard

228

input [38].

229
230
231

However, while researchers in HCI already started exploring the sense of agency for a number of aforementioned
systems, including machine-induced mouse clicks, button presses, and mid-air haptics, the understanding of the user’s

232

sense of agency in haptic systems is at its infancy–this is precisely what we advance. As such, we first cover the types of233

haptic devices our findings will apply to, including force-feedback devices, especially those designed for interpersonal234

communication of non-visual cues (proprioceptive aspects such as poses, gestures, rhythms, etc.).

235
236
237
238
239
240

2.3

Haptic devices that can generate computer-driven touch by actuating the user’s body

The type of haptics we explore, with regards to their implications on the user’s sense of agency, are strong haptic

241

systems capable of moving a user’s body, even against their own volitional force (this property of a haptic device is most242

often referred to as force-feedback). These types of devices can move the user’s body, for instance, to touch a computer

243interface, which we refer to as computer-driven touch—since this type of touch is involuntary, i.e., not generated by the

244
245

user but, instead, by the computer via the haptic device.

246

Broadly speaking, there are two types of actuators that provide enough output force to actuate human limbs, such247

as the force required to move a user’s finger and involuntarily drive touch actions: (1) mechanical actuators (e.g.,248large robotic arms or exoskeletons), and (2) electrical muscle stimulation (EMS). The former is the traditional type of

249
250

mechanical force feedback device, constructed by having users hold on to a handle, which is then actuated by a robotic251

arm or by pulley system (e.g., SPIDAR [54]). While the majority of these devices are stationary, researchers have long252

been exploring mechanisms to achieve ungrounded force-feedback: this is typically realized by mechanically actuated253

devices that provide forces by pulling the user’s limbs against a mounting point on the user’s body. Exoskeletons are

254
255

the canonical example of a wearable ungrounded haptic device: they are often employed to actuate the user’s arms [72],256

legs [68], or fingers [22]. Unfortunately, mechanical-based force-feedback devices are also rather impractical due to their257bulky weight and size, which might explain why few such commercial devices exist despite the decades of research [40].

258This led many researchers to explore alternative actuation approaches that afford smaller form-factors when compared

259
260

5

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

261

to mechanical devices. The most promising approach to miniaturizing mechanical force feedback is precisely electrical262

muscle stimulation, which we cover next.

263
264
265

2.4

Electrical Muscle Stimulation

266

Electrical Muscle Stimulation (EMS) is an actuation technique with roots in rehabilitation medicine, pioneered as a267

means to restore lost motor functions, e.g., allow a patient to regain muscle movements even after suffering spinal

268
269

cord injuries [67]. At its core, EMS uses electrical impulses to the user’s muscles, delivered via electrodes attached to270

the skin. When these electrical impulses reach the user’s muscles, they cause them to contract, which results in an

271

involuntary movement.

272
273
274

A decade ago, researchers in HCI started to explor EMS as a component of interactive devices that can realize force
feedback. It has been successfully used to create force-feedback in virtual reality [18, 39, 41, 44], mobile information

275

systems [42], haptic training systems [16, 43, 58, 62, 72], and especially in interpersonal communication systems that276

allow users to synchronize their movements over distance (e.g., [24, 25, 58–60]–because these interactive applications

277

of EMS are of particular relevance to our proposal, we explore them in detail next.

278
279
280

2.5

Interpersonal Haptic Communication

281

Even beyond EMS, a number of haptic devices have been developed specifically to convey one’s haptic or kinetic

282

experience to another person, with the goal of enriching communication with non-verbal aspects such as force, poses,

283
284

gestures, rhythms, etc. For instance, InTouch [9] allows remote users to feel virtual sensations of a shared object through285

somatosensory interactions by using connected mechanical rollers. Similarly, a paired shape-shifting stick [55] allows

286

users to feel the movements and presence of a performer by mimicking the remote user’s stick movements. These haptic

287

cues are capable of even conveying or complementing emotions, such as anger, disgust, fear, and joy [2]. Interpersonal

288
289

haptic communication makes it possible to transfer not only physical skills between users [46, 63] but also improves290

the users’ sense of presence and users’ ratings of trust and togetherness [10, 34, 56]. Recently, EMS-based interpersonal291

communication systems are also employed for conveying non-verbal aspects, such as rhythms or force [59], joint292

rigidity [60], poses and presence [24], and even emotions via body gestures [25]).

293
294
295

2.6

Loss of agency in actuated haptic systems

296

Unfortunately, while researchers are excited about the prospects of haptic devices such as EMS, very little is known

297

about their impact on the user’s sense of agency. This lack of a deep understanding is where we draw inspiration for

298
299

our present investigation. Specifically, we identified several instances of previous HCI studies conducted on these EMS-

300

based interactive systems that point to some degree of the participants’ loss in agency. For example, in Affordance++,301

participants attributed agency to the object they manipulated (e.g., a spray can that shakes) rather than to themselves [43].

302

Similarly, in Proprioceptive Interaction, participants often voiced not feeling in control, e.g., "it was so remarkable to

303
304

see my hand moving without my intention" [42]. Moreover, Nishida et al. found in their demonstrations that the use

305

of EMS used to speed-up a participant in a collaborative pen-drop task caused users to have a decreased sense of306

agency [58].

307
308

2.7

Attempts to preserve the user’s sense of agency in haptic systems: the case of computer-driven touch

309
310

This last example led to the only known attempt to preserve the user’s sense of agency in haptic actuated systems,

311

such as those powered by EMS. Researchers found that, for a task exclusively involving touch-interactions, adjusting312

6

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch313

the timing of the preemptive assistance provided by the haptic device allows the user’s to feel more in control [31].

314

In other words, delaying the computer-driven touch timing to be closer to the user’s own movement timing

315
316
317

helped in preserving the user’s sense of agency, even when the touch was externally created by means of a haptic
device, such as an EMS-based interactive device [31].

318

However, this prior work only considered the naive case in which the user-driven touch is aligned with the machine-319

driven touch [31], which we call a joint success. This literally assumes that the user has no choice in their touch actions.

320
321

In other words, the only prior research assumes the extremely unlikely scenario in which the user can only touch in the322same way as the machine touches (and cannot touch or do anything else, including simply not touching and remaining

323idle); ironically, this former investigation shows how to preserve the user’s agency in a heavily-constrained scenario324

where the user has no real choice over their touches, which, as we saw, is a component of the sense of agency.

325

We argue that this is actually a rather unlikely case because it only works when we can predict the user’s intention,

326
327

which, as aforementioned, is typically seen as a computationally hard problem that has eluded decades of research. As

328such, this paper contributes an understanding of what happens to the user’s sense of agency in more nuanced and329

complex situations where the system and the user’s touch might not always be perfectly aligned. This understanding

330
331
332

will allow researchers to effectively build better digital touch systems that can prioritize and/or trade-off between theuser’s sense of agency vs. the amount of haptic assistance.

333
334
335

3

OVERVIEW OF OUR USER STUDIES

336

Our experiments were designed to understand to whom users attribute a touch is made when their own body is actuated337

by a haptic device. As previously explained, we opted for electrical muscle stimulation since it presents itself as the

338
339
340

ideal case-study as its loss of agency in EMS-driven touch is already well-known [31], while, simultaneously, EMS isseen by researchers as a very promising approach to wearable haptics.

341

As such, our experiments investigate the four scenarios depicted in Figure 1–allowing us to go beyond the insights

342

from prior work [31], which only investigated the narrow situation in which the EMS and the user perform the exact

343
344

same touch (no-choice). Moreover, we were interested in understanding whether this attribution of agency is modulated345

by the outcome of the touch action, i.e., do we tend to attribute correct actions to ourselves, while we attribute incorrect346

actions to computer systems? To investigate these key questions, we designed a novel variation of the Stroop test, a347

well-known cognitively demanding task [26, 50, 77], by adding electrical muscle stimulation (EMS) as our haptic device

348
349
350

that can generate computer-driven touches; the EMS can do this because it can touch a UI faster than a participant andthus the computer-driven touch is the one generates the outcome.

351
352
353

3.1

Stroop test

354

The Stroop effect is a well-known phenomenon of cognitive interference [26, 50, 77], in which we can measure a delay355

in a reaction time due to a mismatch in stimuli. A popular Stroop effect example is: "touch the word red" with options

356
357

"BLUE" vs. "RED". The correct answer in this example is to touch "RED", despite the fact that it is shown in a blue color.

358

The cognitive mismatch happens between the "name of a color" (e.g., "BLUE" or "RED") and the color that the word is359

rendered with (i.e., the word "RED" shown in blue color instead of red color).

360
361
362

This cognitive interference happens robustly, even when someone is fully cognizant of the so-called Stroop effect [26,50, 77]. Therefore, the Stroop task is extremely useful and widely used for simulating a cognitively loaded task, i.e.,363

a task where the participant really has to ponder their choice before they respond with a touch on the screen. In this364

7

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

365

sense, the Stroop task is cognitively hard because it asks participants to choose the correct answer over two cognitively-366

hard-to-process items, such as our previous example: "touch the word red" with options "BLUE" vs. "RED". What often

367
368

happens in this example is that participants erroneously choose the word by color, picking out "BLUE" since its color369

is red. There are countless variations on this well-known effect, even in non-visual modalities, such as the auditory

370

Stroop effect [52].

371
372
373
374
375
376
377
378

3.2

Our novel variation: EMS-based Stroop test

We designed a novel variation on the Stroop test with the addition of EMS that allowed us to investigate our main
research question. In our study, participants were asked to answer typical two-choice Stroop questions by touching,with either hand, a touchscreen UI. The items from which to choose were placed in either sides of the screen, therefore,
to choose the left item, the participant is required to touch with their ring-finger of their left-hand, and vice versa.

379

Simultaneously, our EMS device actuated participants’ fingers as well, therefore our EMS device can automatically380

touch for the participant—i.e., computer-driven touch. Moreover, it can do that either faster or slower than the participant’s

381
382
383

own reaction time; in other words, we can adjust the preemptive timing of the haptic-assistance to reveal potential
effects with agency. Using this setup, we made the EMS-based haptic assistance play one of two roles: assistive-touch

384(actuated the finger to touch the correct answer) or adversarial-touch (actuated the finger to touch an incorrect answer).

385

As such our unique study design allows us to simulate the situation where the computer directly assists or interferes

386
387

with one’s body as we are also performing a touch. Since our interest was in how participants attribute the success388

or failure of each touch action to themselves or to the computer, we asked participants to rate their sense of agency

389

explicitly after each touch (as in [31]).

390
391
392

3.3

Apparatus

393

Our experimental setup is depicted in Figure 2. To assist readers in replicating our experiment, we now provide the394

necessary technical details and, moreover, the complete source code and EMS-stimulator design1 . Participants sat in

395
396

front of the experimental system consisting of: a touchscreen computer (Microsoft Surface Pro Model 1796, with a 60fps

397

display). Then, in our main study, we added also two pairs of electrodes (applied on each arm, calibrated to actuate each398

hand’s ring finger), and an EMS stimulator (bioSync [59]) connected to the computer. We also designed two 3d printed

399

hand supports; these supports leave the participants’ ring fingers free to flex and touch the screen so as to answer the

400
401

Stroop test. Furthermore, this prevents participants muscles from fatiguing quickly.

402
403
404
405

3.4

EMS Stimulation (only in main study)

Participants were actuated using the aforementioned EMS stimulator. This study was approved by our local ethics

406

committee (Sony Group Corporation, application 19F0010). Furthermore, during recruitment, we confirmed that no407

participant had any medical condition that discouraged the use of EMS.

408
409
410

Electrode placement: We attached two pairs of electrodes to both hands: each pair stimulated the forearm muscle
responsible for flexing the ring finger (flexor digitorum profundus). We chose to actuate the ring finger, rather than the411

index finger, since previous research has confirmed that one can robustly actuate it so as to touch the screen without412

any parasitical motion of neighboring muscles [31, 69].

413
414
415
416

1 https://lab.plopes.org/#assistedEMStouch8Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

417
418
419
420

EMS electrode

421
422
423
424
425

EMS electrode

426
427
428
429
430
431
432

Display for Stroop test

433
434

Finger baseFig. 2. The setup for our Stroop test studies. The EMS was added only for the main study.

435
436
437
438
439
440
441

Calibrating the EMS stimulation: We interactively adjusted the stimulation parameters to actuate both ring
fingers robustly. We fixed the output of our voltage-controlled muscle stimulator at 38V [59]. Then, in order to calibrate,
we adjusted the electrical impulses’ pulse-width (ranging from 100-300 microseconds), the number of pulses in a442

stimulation (ranging from 1 to 4 pulses) and the interval between consecutive pulses (ranging from 30-100 ms, in443

case more than one pulse was used). We chose this since prior research suggests that short stimulations are less444

noticeable [41]. Prior to starting the trials, we confirmed that participants’ finger was actuated via the EMS to tap the

445
446

surface and that each touch was consistently recognized by the touchscreen.

447
448
449
450
451
452
453
454
455

3.5

Trial design

Participants performed a touch-based forced-choice task in a Stroop test setting. The design of our task, which is
depicted in Figure 3 with the example of a single trial, was as follows:(1) Fixation: The fixation cross was displayed with a randomized duration (between 1 and 4s) to keep participants
attentive.
(2) Instruction for touch task: Next, participants were presented with the instruction for this trial. The choice-task

456

was one of four possible combinations of [’RED’ or ’BLUE’] and [’COLOR’ and ’TEXT’], e.g. ’TOUCH RED COLOR’

457(touch the word colored in red), ’TOUCH BLUE TEXT’ (touch the word ’blue’ not matter the color in which it is printed).

458

This instruction is shown for 300ms, which we determined via pilot testing to be sufficiently cognitively demanding.

459
460

(3) Choice & touch: Then, after being presented with the instruction, the Stroop visual stimuli appeared on the461screen. The Stroop choice visual stimuli consisted of two colored words located in right and left in the screen as depicted

462

in Figure 2. Those two words are colored contrary to the meaning of the word: for example, the word “BLUE” appeared463

in a red colored font, like this BLUE. Here, participants were asked to touch the correct side of the screen with the ring

464
465
466
467
468

finger of the corresponding hand (right or left) as fast and accurate as possible.
In addition to the Stroop visual stimuli, we also presented participants with an on-screen time indicator to limit the
maximum time available for each trial (a horizontal bar that decreased over time, depicted in Figure 2).
9Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

5. Agency Rate

469

471
472

2. Cue473
474

6. End of Trial

4. Feedback470

3. Task

1.Fixation cross

475
476
477
478
479
480

Visual Stimuli

481
482
483

Reaction

484

EMS

485
486

et
EMS Offs

Time

Time

487
488

Human Response

489
490
491

Fig. 3. Our novel variation of the Stroop test using electric muscle stimulation (EMS).

492
493
494
495

EMS-conditions: assistive-touch and adversarial-touch (main study) The EMS actuated participants’ finger

496

to answer the Stroop test by touching on behalf of the participants (involuntarily). It played two different roles in

497
498

our study: assistive-touch, in which the EMS forced the participant’s finger to touch the correct answer (success);499

or, adversarial-touch, in which the EMS forced the participant’s finger to touch the incorrect answer (failure). In

500

addition, the EMS impulse that drove the computer-driven touch was delivered with seven different timings (denoted as

501

EMS offset): 100ms, 200ms, 275ms, 350ms, 425ms, 500ms, and 600ms, all of these are relative to the time of the visual

502
503

stimulus.

504(4) Outcome visual feedback: As soon as a touch was detected (by either the user’s voluntary movement or EMS’

505

involuntary action), the visual feedback showing the outcome was displayed with an "OK" or "NG" (not good) alongside506

this trial’s instruction. While the message was displayed immediately after the first touch was detected (first touch is

507
508

the one that drives the outcome), our study system also recorded any subsequent touches for our analysis.

509(5) Questionnaire for sense of agency: After each touch (a Stroop question), participants were presented with a510

questionnaire regarding their perceived sense of agency for this trial. We follow the typical agency questionnaire, i.e., a

511

Likert scale question with 1 = "I did not do it" and 7 = "I did it" (as in [31]).

512
513
514
515
516
517

4

PRE-STUDY: STANDARD STROOP TESTThe objective of our pre-study was to determine the typical performance participants’ exhibit in an unmodified Stroop
test with participants, especially: what is the average time needed to cognitively process the stimuli and touch the518

correct answer? We required determining this prior to advancing to our novel variation of the Stroop test because we

519

need to know when to activate the EMS to create the computer-driven touch. Moreover, this pre-study also served as a

520

10

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

521

screening process for determining which participants of our sample are eligible for the main study (more than 55% of522

correct answers).

523
524
525
526

4.1

Apparatus, experimental design and procedure

527

This study was conducted as described above, except no haptic assistance was used; in other words, participants

528

performed a standard Stroop test using only user-driven touches. We measured the correct answer-rate and the reaction

529
530

time. Before engaging in our pre-study task, participants performed a training session using congruent color and531

text (e.g., BLUE) for at least 10 min (simply tapping on screen to advance), to understand the task. Then, we asked

532

participants to perform 32 Stroop trials (again, without any haptic assistance).

533
534
535

4.2

Participants

536
537

We recruited 18 participants (three self-identified as female, 15 as male; M= 32.8 years old, SD= 9.03) from our local

538

institution. With their prior written consent, we transcribed their comments.

539
540
541
542

4.3

Result of pre-study543

We collected 32 trials from each participant (for a total of 576 samples), with two data points each: the reaction time per

544

touch and the correct/incorrect answer. Data was assessed for normality using the Shapiro-Wilk test (p > .05).

545
546

Analyzing accuracy of responses: Before analyzing between participants, we checked the accuracy of the task

547

for each participant to exclude results around chance-level (approx. 50%), indicating the participant may not have548

performed the task correctly, most likely the difficulty of the Stroop test was still too high for these individuals with549

the given time limit. As a result, we excluded three participants whose accuracy was around chance-level (M=53.5%,

550
551
552
553
554
555
556

SD=0.7), these did not qualify for the main study as they did not reach our eligibility criteria (more than 55% of correct
answers). After excluding these three participants, the average accuracy was 77.1 ± 14.4% (of correct answers).
Analyzing average touch time: the average reaction time was 516.4 ± 54.8 milliseconds.
Analyzing touch time based on accuracy: In order to check participants’ behavior according to success or failureof the task, the data separation based on the accuracy of their response was conducted (separating correct vs. incorrect557

trials). Averages of touch time were 516.0 ± 54.4 milliseconds for a successful response and 512.4 ± 73.3 milliseconds for

558

failed response. Type III ANOVA analysis did not show significance in the average of reaction time between success559

and failure responses (F 1,27 =.022, p = .883). This result indicates that the participant’s speed did not differ based on the

560
561

touch’s outcome (success and failure).

562

Validating the added cognitive load: Knowing from the psychophysics literature that the typical reaction time563

involving a touch action is around 250ms [28, 65], our resulting average time of 516.4ms (SE = 14.2 ms) validated that564

our Stroop test successfully provided a significant cognitive load, which was a critical aspect for our main study. In

565
566

addition, the analysis of correct answer rate revealed that participants often missed due to incongruency between the567

color and semantic information of the presented text. Based on the result of this pre-study, a participant might make a

568

mistake once every five touches. As such, moving forward to our main study, we expected that participants would also569

make mistakes in the task, which will allow us to investigate our hypotheses. In this light, it once more validates the

570
571
572

current design of our Stroop task as it successfully induced enough mistakes.
11Sub., TOCHI, Digital Touch

573
574
575

5

Tajima, Nishida, Lopes and Kasahara

MAIN STUDY: EMS-BASED STROOP TESTThe objective of our main study was to investigate how participants attribute their sense of agency for the outcome of a576

touch, while, simultaneously, being actuated by a haptic device (EMS) that played one of two roles: assistive-touch577(forced success) or adversarial-touch (forced failure); this investigation of the influence of both the role of the578

touch-assistance and timing over agency is the novel aspect of this study, which was not realized in previous work. To

579
580

explore the influence of the touch timing over agency, the EMS impulse that generated the computer-driven touch was

581

delivered with seven different timing intervals (100ms - 600ms after the Stroop task is displayed), this yields two types

582

of experience a participant is expected to feel:

583
584
585

Human-computer touch-congruence: when a participant and haptic device (EMS) touch with the same finger
and thus touch the same answer. In this pattern, there are possible sub-patterns based on our two factors: (1) outcome

586(correct or incorrect answer) and (2) whether computer-driven touch was faster than the user-driven touch or slower.

587

Human-computer touch-incongruence: when a participant and the haptic device (EMS) touch different using588

fingers and thus touch different answers. The same factors apply, however, the correct outcome to the answer in this

589
590

case depends on who touched first, the user-driven touch or the computer-driven touch.

591
592

5.1

Hypotheses

593
594

Our two main hypotheses were as follows:595

H1: In the event that the user and the haptic device touch in the same way (i.e., the touch outcomes are congruent),596

we expect that the resulting outcome (i.e., whether it is a success or a failure) will bias the attribution of the touch, even

597
598

when the haptics actuates participant to touch faster than their voluntary touch; in other words, we hypothesized the599

existence of an outcome bias for assistive-touches, i.e., users would feel more agency if the outcome is favorable600

to them and vice-versa. As such, we expected the distribution of agency with regards to the EMS timing would follow a601smooth curve similar to [31].

602
603

H2: In the event that the user and the haptic device touch in different ways (i.e., the touches are incongruent), we604

expected participants would be able to discriminate who initiated the touch, simply dictated by who (them or system)

605

was faster to touch. Thus, we expected that, when compared to the congruent case (from H1), the agency scores would606

be separated clearly (which we expected, based on prior work, to be most visible halfway in the agency distribution

607
608
609

curve), rather than following a smooth curve–in other words, we expected the distribution of agency scores to differ
from that of [31]).

610
611

5.2

Study Procedure

612
613

To eliminate any confounding variables from the Stroop design itself, we varied all possible instructions (color or text),

614

the position of the answer (right or left) and colors (red or blue). Furthermore, we also randomized the task sequence615

for all 14 conditions (seven EMS timings, two EMS roles), which in total produced 112 trials per participant.

616
617
618

To prevent participants’ fatigue or loss of concentration, we divided our task sequence into three sessions separated
by five minute breaks in between, the first and the second session had 32 trials, while the third session had 54 trials.

619

In addition to this, each session had an extra six trials at the start of the session. These starting trials were comprised

620

of very fast and slow EMS offsets to allow participants to create their own consistent subjective criteria for the

621
622
623
624

assessment of agency while under the EMS stimulation (fast: 100ms, slow: 600ms). The order of the task sequence wascounter-balanced between participants.
12

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

625
626
627

5.3

Sub., TOCHI, Digital Touch

Data Analysis

Data analysis pipeline: Data was assessed for normality using the Shapiro-Wilk test (p > .05). Results of the question-

628

naires, and part of reaction time results of the task were not normally distributed; thus, non-parametric analysis were629

employed for these data sets to check significance of the factors. However, after data separation based on congruence630

between EMS actuation and human response, the remainder sample sizes often become unequal. Therefore, a type III

631
632

ANOVA was employed for main effect analysis in unequal sample size data, which is effective for unbalanced samples.

633

In addition, data screening was conducted to exclude outliers, such as accidental touches from an extremely fast or

634

slower than normal response (these can be typically caused by a distracted participant). The trials in which the reaction

635

time was over ± three SD were excluded, which amounted to 0.3% from all data points.

636
637

Classifying trials as human-computer in/congruent: Our main hypotheses implies the analysis of the human-

638

computer congruent cases vs. human-computer incongruent cases. As such, we performed a straightforward case639

separation into two cases based on the number of touches and the response (correct vs. incorrect). Since our Stroop task640

was performed with two hands, human-computer in/congruent case can be separated based on the number touches on

641
642

the two sides of the screen. In Figure 4, we describe how these touch sequences allow us to separate these two cases.

643
644

(a)

646
647
648
649

Correct
Response

650
651
652
653

EMS

(b)

Incorrect
Response

human-computer incongruent-touch

human-computer congruent-touch

645

EMS

Human

Human

EMS/Human

EMS/Human

Human

Human

EMS

EMS

(c)-1

(d)-1

EMS

EMS

Human

Human

(c)-2

Human

EMS

Human

EMS

(d)-2

654
EMS is faster than human

655

Human is faster than EMS

EMS is faster than human

Human is faster than EMS

656
657
658

Fig. 4. Analysis cases: human-computer congruent-touch and human-computer incongruent-touch. In both cases, the response forthe task can be correct or incorrect.

659
660
661
662

In the case of a human-computer congruent-touch, the result of user response and EMS actuation can be the same663

answer. In this case, touches are only detected on only the right- or left-side, regardless of whether the user or EMS is664

faster. The number of touches can be one or two, depending on the timing of the EMS and the user’s touch, but, in

665
666

either situation, this is classified as a congruent case.

667

In the case of human-computer incongruent-touch, touches are detected as a mix of right- and left-side, as EMS

668

and the user give a different response. Therefore, regardless of the touch order, if touches are detected on both sides, it669

is classified as an incongruent case.

670
671
672
673
674
675
676

5.4

Participants

We recruited the aforementioned 15 participants that passed the eligibility criteria from the pre-study (more than 55%
of correct answers). With their prior written consent, we transcribed their comments.
13Sub., TOCHI, Digital Touch

677
678
679

6

Tajima, Nishida, Lopes and Kasahara

RESULTS

We collected participant responses for each trial. The total number of trials per participants was 112 (text color: red or

680

blue; response type: text or color; correct position: right or left; EMS role: assistive or adversarial; and, EMS Offset: 7681

conditions, which yields 2 x 2 x 2 x 2 x 7 = 112 trials). Across all participants we gathered 1680 trials, each containing a

682

perceived agency score (1-7). We now analyze our hypotheses by analyzing each case: (H1) human-computer congruent-

683
684

touch preserves agency and is biased towards the touch’s outcome; (H2) human-computer incongruent-touch will allow

685

participants to discriminate who delivered the touch that drove the outcome, leading to a clear distinction of agency

686

scores (which we expected, based on prior work, to be most visible halfway in the agency distribution curve).

687
688

6.1

Human-computer congruent-touches: validating our H1

689
690

To assess our first hypothesis, we analyzed the case of human-computer congruent-touches. In this case, there are two

691

possible outcomes: joint success (the user and EMS were both correct with assistive EMS) and joint failure (the user and692

EMS were both incorrect).

693
694
695

Perceived agency: We observed that the agency distribution across different computer-driven touch timings (howearly the EMS causes the user to involuntarily touch) is different for different outcomes. In other words, adversarial-

696

touch and assistive-touch induce a different feeling of agency, even for the same EMS timing— this is one of697

our key results, which is depicted in Figure 5. To confirm this, we used a two-way repeated ANOVA analysis, which

698
699

revealed a significant main effect for the role of the computer-driven touch (assistive-touch vs. adversarial-touch) and700

the EMS offset on the agency score (EMS role: F 1,12 =13.3, p =.003; EMS offset: F 6,79 = 77.4, p < .001). This statistical701

difference confirms our first hypothesis: participants’ agency was biased towards the outcome, i.e., for favorable

702

outcomes (assistive), participants reported higher agency than for unfavorable outcomes (adversarial EMS). Lastly, our

703
704

analysis did not find an interaction between EMS role and EMS offset (F 6,79 = .92, p = .485).

705

Outcome: Correct
Outcome: Incorrect

706
707

p < .01

708

710
711
712
713
714

Agency Score

709

715
716
717
718
719
720
721

EMS offset (ms)

722
723
724

Fig. 5. Agency score with different EMS offset in the condition where computer-driven and user-driven touch were congruent (color
depicts outcome, blue for positive outcome, red for negative outcome).

725
726

Furthermore, we also observed that our measured agency distributions followed a similar pattern as previously

727

revealed by Kasahara et al. [31], as their singular case that did not involve any choice during touches. Our objective is

728

14

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch729

to depict a single model that aggregates the observed agency scores across all participants. To do this, we follow the730

method of [31]: Per participant, we plot the logistic regression that computes the relationship between the perceived731

agency and the EMS offset. Then, from all these curve-fits, we computed a mean fit of R 2 = 0.60 (with SD=0.17, MIN=0.17,

732
733

MAX=0.82) for our overall logistic model, which is depicted in Figure 6 (a) as a bold curved line with the shaded area734

around the curves depicting the range of its standard error. This demonstrates that the curves are in most cases very735

similar and relatively consistent among all users (as suggested by the low standard deviation).

736
737

p < .05

739

741
742
743
744
745
746
747
748
749

Agency score (normalized)

740

Halfway agency point

Outcome: Correct
Outcome: Incorrect

750
751

EMS offset (ms) at the halfway ageny point

738

752

EMS offset (ms)753

Correct

Incorrect

754
755
756

Fig. 6. (a) Result of curve fitting for each condition. (b) Comparison of estimated EMS offset, at the indicated halfway agency point(color depicts outcome, blue for positive outcome, red for negative outcome).

757
758
759
760
761

Impact of haptic role (assistive-touch vs. adversarial-touch) in the design of the haptic timing (EMS offset): to give readers a sense of how the haptic role (assistive-touch vs. adversarial-touch) impacts the resulting

762

agency/touch-timing trade-off, we pick the same halfway agency point as chosen by [31], i.e., a normalized agency score763

of 0.5. As we can observe in Figure 6 (a), this point is reached differently depending on the haptic role (assistive-touch vs.

764

adversarial-touch). We confirmed this via a one-sided paired T-test 2 , which revealed a statistical difference between the

765
766

two roles at this halfway agency point (t(13) = -3.15, p = .001.), which we depict in Figure 6 (b). As our results confirmed,

767

the adversarial-touch role yielded a negative agency bias, and, as such, the halfway agency point is triggered by an

768

average EMS offset of 327 ms (SE=23.8 ms). Conversely, the assistive-touch role yielded a positive agency bias, and, as769

such, the halfway agency point is triggered with an earlier stimulus with an average EMS offset of 379 ms (SE= 25.0

770
771

ms), meaning, when the outcomes are positive and the situation is touch-congruent, the system can accelerate the users

772

more and, simultaneously, preserve some degree of their sense of agency—this is an implication of our findings, we

773

will present how this affects the design of digital touch systems later. Moreover, we will depict average agency-timing

774

trade-off curves in our resulting framework.

775
776
777

It is worth noting that our above insight takes into account the design of the end-to-end latency experimental
apparatus (including EMS latency, touch latency, etc.), which was measured to be around 100 ms. As such, considering778

2 Because the sample size of data of each condition was unequal (EMS assistive-touch: 15, EMS adversarial-touch: 14), paired data were employed for this779

statistical analysis.

780

15

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

781

the two aforementioned average EMS offsets at the halfway agency point (327 ms and 379 ms), we can now add in782

the 100 ms to obtain a final reaction time of 427 ms and 479 ms. This value is below the nominal reaction time of

783
784

the standard Stroop test, which we found in our Pre-study to be around 500 ms. As such, we confirm that, indeed,

785

participants exposed to these EMS offsets can be accelerated beyond their normal reaction time while simultaneously

786

preserving some degree of their sense of agency.

787
788
789
790
791

6.2

Human-computer incongruent-touch case: validating our H2Now, we turn our attention to the extreme case, where the user-driven touch and computer-driven touch give rise
to different answers. These competing touches generate competing outcomes, which we postulated in our second

792

hypothesis: we expected participants would be able to discriminate who initiated the touch, simply dictated by who

793(user or computer) was faster to touch. To assess if indeed, as we expected, the distribution of agency scores differs

794
795

from the congruent case, we conducted the same type of analysis as for the congruent-touch case, but also took into

796

account which touch was faster (user or computer), because the fastest touch is always the one that drives the final

797

outcome in this incongruent-touch case.

798
799
800

Perceived agency: We observed that the agency distribution is distinct from the congruent case and does not tend
to follow the same smooth distribution (as in both our H1 or in [31]). In fact, we observed that the agency distribution

801

is distinct when we consider both the role and the outcome (which yields four classes), as is depicted in Figure 7. We

802found that agency falls into two sharp and distinct regions: low-agency and high-agency (again, rather than the smooth803

curve from [31]), only dictated by whether the participant touched faster (higher-agency) or the computer caused the

804
805

participant to tap faster (lower-agency)— this is one of our key results. In other words, when touches are at conflict

806

between user and computer, the user’s sense of agency is much easier to discriminate than in a shared touch. This

807

insight was confirmed by a Three-way ANOVA analysis that revealed a main effect of EMS offset (F 6,78 = 45.5, p < .001)

808

and outcome (F 1,14 = 16.7, p = .001) on the participants’ perceived agency. Moreover, we did not find a main effect of the

809
810

EMS role (assistive vs. adversarial) on the sense of agency (F 1,7 =.719, p =.424)). Taken together, these two significant

811

effects confirm our second hypothesis. Moving forward, we will visually illustrate, in our final design implications, this812

clear separation of the sense of agency for incongruent-touch cases as a sharp agency curve (almost like a square wave

813

step function, from 0 to 1, i.e., from very-low to very-high agency).

814
815
816
817

7

DISCUSSION

818

We now discuss how our results can both validate existing research and help shape the current theories of how haptic

819

devices, with a particular emphasis on EMS, interface with the user’s sense of agency.

820
821
822
823
824
825
826

7.1

Outcome bias

The most intriguing and exciting of our findings is that the outcome bias has been observed even when the users’ bodies
were haptically actuated to induce a touch that they did not generate. Here, we ground our outcome bias finding inexisting research.

827

Outcome bias (often dubbed as self-serving bias, as it serves one’s own interests) has been reported in various

828

experiments both in cognitive science [20] as well as in behavioral science [75]. Outcome bias is a well-known human

829
830

tendency that individuals ascribe success to their own abilities and efforts, but ascribe failure to external factors [8]. The

831

origins of outcome bias have been often pointed to the sense of agency [3, 71, 79]. For instance, correlation between

832

16

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

User correct

833

Sub., TOCHI, Digital Touch

User incorrect

834

837
838
839
840
841

EMS
fasterEMS
fasterUser
fasterOutcome: Correct
EMS: Assistive

User
fasterOutcome: Incorrect
EMS: Assistve

Agency Score

836

Agency Score

EMS correct

835

Outcome: Correct
EMS: Assistive842
EMS Offset (ms)EMS Offset (ms)

843
844

847
848
849
850
851
852

EMS
fasterUser
fasterEMS
fasterOutcome: Correct
EMS: Adversarial
Agency Score

846

Agency Score

EMS incorrect

845

EMS:
EMS:Adversarial
AdversarialOutcome:
Outcome:Incorrect
Correct

EMS: Adversarial Outcome: Incorrect

Outcome: Incorrect
EMS: Adversarial

Outcome: Incorrect
EMS: Adversarial

EMS Offset (ms)EMS Offset (ms)

853

User
faster

854
855
856
857

Fig. 7. Distribution of agency for incongruent-touch cases, separated into four sub-cases based on both the haptic role and the touchoutcome (color depicts the outcome, blue for positive outcome, red for negative outcome).

858
859
860
861
862

agency measurements and positive vs. negative outcomes have been shown in several aspects such as emotional ormonetary gain [71, 79].

863

While previous findings in sense of agency and outcome bias focused on the relationship between an action and its

864

outcome, our findings shed light on a novel direction between voluntary touches and haptic-induced touches, which is865

of particular relevance to interactive systems at large, but especially to digital touch systems.

866
867
868

7.2

Postdictive attribution in human-computer joint actions

869

One theory postulates that outcome bias also happens at a higher cognitive level, on a reflective level; in other words,

870

when we look back at our action (in retrospect), we will more likely favor attributing it to us if the outcome was

871
872

positive [45, 73]. Such contribution of a retrospective process to conscious experience is often called postdiction, which873

has been shown to be an important component in the subjective experience of sequential events such as visual motion874

perception [15, 64] and even causality judgment [11].

875
876
877

7.3

Clear discrimination in incongruent-touches878

Our results suggest that participants felt a clear distinction of agency for incongruent touches. So we ask: what cues do879

users tend to rely on to understand if the system is forcing them to touch?

880
881

Haptic cue#1 the EMS tingling sensation: One of the possible cues that participants might use to make judgments882

of their sense of agency during a computer-driven touch is the so called "tingling sensation" that EMS induces in883

the user’s skin–a tactile cue. This haptic cue allows a user of a haptic system to reason/discriminate between their

884

17

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

885

voluntary touch and computer-induced touch. The first implication this has for HCI research is: the tingling sensation886

is a limitation of EMS and needs to be addressed, but this problem is non-trivial and has escaped a solution in both HCI

887
888

as well as in medical rehabilitation, the field where EMS originated from decades ago. Moreover, the tingling sensation889

is present in all cases of our study, but in some cases participants did indeed feel agency, even when they were still able890

to feel this haptic cue. As such, the cue alone is not what drives the distinction between "I did it" vs. "the UI did it", but

891
892
893

likely provides one more piece of information available to users. In fact, one expects that this cue is more noticeable
when it strikes the arm that did not touch (e.g., in the cases where user and system touched differently). Conversely, in894

the case where this cue is felt at the arm the user is intending to touch with, it is downplayed by the user’s own sensory895

attenuation [6] –this is a well-known effect in which the sensory consequences of an user’s actions are perceived less896

intensely than sensory stimuli that are not caused and thus not predicted. Precisely this type of sensory attenuation has

897
898

also been measured, using fMRI, for users subjected to EMS movements [37].

899

Haptic cue#2 the sensation of the computer-induced touch itself: Another possible cue that users might use

900

to reason about their sense of agency is the unexpected somatosensory feedback induced by the EMS as it contracts

901

their muscles to touch with their finger (this is a proprioceptive cue, not a tactile cue as the previous one). Again,

902
903

due to sensory attenuation [6, 37] we expect this cue to be more easily discernible for incongruent situations, i.e.,904

participants would feel that both hands were moving, but their intention should have led only to one contraction;905

allowing participants to reason the cause of the touch.

906
907

7.4

Study limitations

908
909

As with any experimental design, our study has limitations that we urge readers to be aware of, prior to generalizing910

any of our insights. The first one is that any haptics study of this nature needs to choose a haptic-device; we chose

911

electrical muscle stimulation since the loss of agency during EMS-driven touch is well-known from prior art, such as

912
913
914

from [31, 42], just to cite a few. While we do believe these findings might generalize well beyond EMS, future work isneeded to apply our novel study design to more and more haptic actuators.

915

Finally, note that while we provide one possible interpretation of our results, we argue for exploring more inter-916

pretations by designing upcoming experiments that can narrow down on the exact causes of the observed effects. For

917
918
919

instance, the perception of who finalized the touch action can also be driving the agency perception in retrospect, which
could also explain how different EMS timings causes different feelings of agency.

920
921
922
923

8

SYNTHESIS OF OUR FINDINGS: AGENCY-ASSISTANCE TRADE-OFF MATRIXFrom our findings, we synthesized a matrix that generalizes the use of touch assistance beyond the simple case covered924

by prior art [31]. This resulted in what we call the agency-assistance trade-off matrix, which effectively illustrates the925

design implication for haptic systems that proactively actuate the user.

926
927
928

Our agency-assistance trade-off matrix, depicted in Figure 8 consists of four quadrants, guided by two dimensions:
whether the user-driven touch is correct or incorrect, and whether the computer-driven touch is correct or incorrect.

929

More interestingly, in each quadrant we depict an agency curve that demonstrates how likely it is that the prospective930

UI designer can preserve the user’s agency by adjusting the timing of the computer-driven touch stimulus. To simplify

931

the reading of these curves, we depict each by using a simple pictograph of either a smooth curve or a sharp curve. A

932
933

smooth curve indicates that we found that it is possible to preserve the user’s agency to some degree, while a sharp934

curve indicates that the haptic assistance will create a trade-off with agency. In the most immediate future, we believe935

these curves might directly empower researchers in this area who are building haptic interfaces with assistive-touch.

936

18

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

user correctuser incorrect(a) Joint Success(b) Forced Success

937
938

EMS correct

939
940
941
942
943
944
945

agencyEMS
fasterSub., TOCHI, Digital TouchagencyUser
fasterEMS
faster

EMS timing

946

User
faster
EMS timing

947

EMS incorrect

948
949
950
951
952
953
954
955

(c) Forced Failure(d) Joint Failure

agencyEMS
fasteragencyUser
fasterEMS
faster

EMS timing

User
faster
EMS timing

956
957
958
959

Fig. 8. Agency-assistance trade-off matrix: Four quadrants, each depicting the general trend we observed regarding the trade-off
between agency and haptic assistance. Our matrix is guided by two dimensions: whether the user-driven touch is correct or incorrect,
and whether the computer-driven touch is correct or incorrect.

960
961
962

Furthermore, we expect that, as more research follows and refines our findings and recommendations, these curves

963

might also prove beneficial as aids for prospective designers in this area.

964
965

To better illustrate our recommendations for digital touch systems, we describe the details of each quadrant.

966

Furthermore, we also demonstrate examples of how a prospective researcher or designer might apply this matrix for

967

digital touch or, generally speaking, interactive haptic systems in which either the UI or a remote user assists the local

968

user via computer-driven touches.

969
970
971

8.1

Quadrant (a): Joint Success972

Our first quadrant depicts the case of a joint success (Figure 8 - a), depicting a congruent situation in which both the973

user-driven touch and the computer-driven touch result in a positive (correct) outcome for the user. This is naturally a

974
975

best-case scenario as it is possible to preserve some level of sense of agency even when EMS-driven touch is faster976

than the user’s own touch. The result is that the haptic system accelerates the user’s touch, but note that the user

977

would respond correctly anyway; in other words, the benefit is an assistive touch speed-up. This situation aligns with978prior literature on simple touch interactions accelerated by haptics [31]. However, one must also denote the extreme

979
980
981

assumption for the HCI designer, because this quadrant is unlikely: it assumes one can both predict the user’s intention
and with sufficient time to preemptively touch.

982

Typical application examples of this quadrant include relatively simple reaction tasks such as haptic assistance to983

catch a falling object between two users, such as the pen-drop shown in Figure 9(a) (from [31]) or touch-assistance that

984
985

lets users press the shutter button of a camera to take a picture of a high-speed moving object Figure 9(b) (from [31]).

986

Moreover, these scenarios are not limited to computer-to-user assistance, but lend themselves well to inter-personal987

communication scenarios too. For instance, a skilled video-game player can provide touch-assistance so that the novice

988

19

Sub., TOCHI, Digital Touch989

aTajima, Nishida, Lopes and Kasahara

c

b

990

!

991
992
993
994
995
Electrodes

996
997

d

998
999

!

e

f

1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011

Fig. 9. Exemplary interactive applications from the HCI community where one might apply our findings: (a) a two-user situationin which they collaborate to catch a falling object (pen drop) [58], (b) accelerating a user for high-speed photography [57] and (c)speeding-up reaction time for sports. As such, (a) - (c) are exemplary cases of Joint Success. (d) EMS haptics actuate the user’s hand toprevent touching a hot object [43], (e) EMS actuates the user’ legs to direct their walking, including avoiding hitting a lamp post [62].
As such, (d) and (e) are example case where forced success will benefit users. Lastly, (f) Game with EMS assistance from other agents(remote user or a UI). This depicts a case that should be designed with agency priority where both joint success or joint failure will be
expected.

1012
1013
1014
1015
1016

user can perform touches faster in time-demanding games [32], such as the fast-paced video game depicted in Figure 9

1017

(f).

1018
1019

Similarly, this touch-assistance might lend itself well to many other motor skill transfers from skilled user to1020

novices [70]. In these examples the interactive system can clearly predict the user’s intention from context and a1021

plethora of additional sensors (e.g., EMG to detect the falling pen and a ball-toss sensor to detect the ball launch).

1022

We recommend that this type of preemptive touch is used in situations where user’s touch intention is relatively

1023
1024

easy to predict at a high degree of confidence; as such, the result is that the agency curve will follow our smooth

1025

distribution, meaning it is possible to still accelerate users to perform these touches or even more general movements

1026

with haptic assistance while preserving some sense of agency. We also recommend that this is the best case scenario, in

1027

which designers can opt for the most extreme accelerations, making it suitable for these types of impressive physical

1028
1029

augmentations that depict faster-than-human touch ability.

1030
1031
1032
1033

8.2

Quadrant (b): Forced Success

Our second quadrant depicts the case of a forced success, which we depict in Figure 8 (b). Here, despite the computer-

1034

driven touch is incongruent to that of the user, the touch results in a positive outcome; in contrast, the user-driven

1035

touch would have resulted in an negative outcome. In other words, this is a corrective type of touch-assistance that1036

dramatically helps the user obtain the right outcome. While this provides positive outcomes, it comes as a trade-off to

1037
1038
1039
1040

the user’s sense of agency. We depict this via the sharp agency trade-off curve, suggesting users will be sensitive to thetouch-assistance and decrease their sense of agency.
20

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

1041

As our studies revealed for this quadrant, both the outcome and agencysimply depend on whether the user or

1042

computer was faster to touch. If computer-driven touch is faster than the user-driven touch, the outcome of the action

1043
1044

will be positive (correct) with a faster involuntary movement by the EMS, but the user is not likely to feel agency.

1045

While this pattern eventually diminishes the user’s sense of agency, it is still important to consider this case as it can

1046

assist users. This is especially beneficial in safety situations or guidance/training environments. In the case of critical

1047

situations (e.g., avoiding accidents), the touch-assistance should be applied regardless of the user’s touch and negative

1048
1049

impact on agency. We can find exemplary EMS-based haptic applications from HCI literature that explore this quadrant,

1050

such as the Affordance++ system that uses EMS to actuate the user’s arm to prevent touching hot objects that might

1051

harm the user [43]. In this example, the EMS actuation is not always aligned with the user’s action, but it is beneficial

1052

to prioritize the assistance as it brings users to safety. Similarly, for the case of guidance/training systems, one could

1053
1054

envision a variation on Pedestrian Cruise Control, a system that uses EMS to control the user’s walking direction so as1055

to avoid colliding with incoming obstacles [62], that would allow, for instance, guiding users into choosing a better

1056

path (e.g., a shorter path or even a safer path) and thus creating incongruent situations where the EMS overrides the1057

user’s walking direction.

1058
1059
1060

8.3

Quadrant (c): Forced Failure1061

Our third quadrant depicts the case of a forced failure, which is depicted in Figure 8(c). This is the situation that1062

prospective designers want to most likely avoid. In this case, the computer-driven touch results in an incongruent and

1063
1064

incorrect outcome (e.g., this could be due to tracking errors, prediction errors, false-positives, or over-fitting in gesture1065

recognizers, etc.), but the user-driven touch results in the correct response. As such, the prospective designer should1066

use all the available means to prevent this case. For instance, most tracking systems and predictive UIs report a level1067

of confidence with their prediction. We strongly recommend that, if this prediction is low (i.e., the system is unlikely

1068
1069
1070

to be making a correct guess), the haptic assistance should be dismissed entirely as it will force only negative results
and most likely decrease the sense of agency.

1071
1072
1073
1074

8.4

Quadrant (d): Joint Failure

Our fourth and last quadrant depicts the case of a joint failure, as shown in Figure 8(d). It is an congruent situation in1075

which the computer-driven touch and the user-driven touch result in an incorrect outcome. This is a case that may

1076

occur when the system can estimate users’ intention with high accuracy, but does not know what could be a positive1077

outcome (e.g., task is too computationally complex to be solved in run-time, etc.). While this initially seems like a

1078
1079
1080

dead-end for designers, we would like to show how our results provide insights and even opportunities for this extreme
case.

1081

First, the prospective designer might refrain from resolving this situation by claiming that "the result of this touch is1082

failure anyway, so why bother?" This is a fallacy. As our results reveal, if the touch-assistance accelerates the user’s

1083
1084

touch towards an imminent failure, this will not only enforce the early negative result, but also further exacerbate the1085loss of the sense of agency due to the negative outcome bias we observed in our studies (see Figure 5). In general, this is1086

likely to be avoided.

1087
1088
1089

While accelerating users’ touch for incorrect answers is not in itself very beneficial from a practical point of view, thisapproach can be an intriguing and novel design opportunity in that one can create interfaces where users do not blame

1090

themselves too much when they fail a task. In other words, we can construct interfaces that sees the imminent failure

1091

and takes the blame for it, which might be especially relevant for preserving the user’s motivation in arduous training

1092

21

Sub., TOCHI, Digital Touch

Tajima, Nishida, Lopes and Kasahara

1093

tasks or even inter-personal communication systems for multiple users leveraging haptic-assistance via computer-driven

1094

touch.

1095
1096
1097

8.5

Designing haptic interfaces mindfully regarding the agency vs. assistance trade-off

1098

By taking insights from these our different quadrants into account, our agency-assistance trade-off matrix assists

1099

prospective designers in improving their digital touch systems, especially applicable to those based on EMS. Here, we

1100
1101
1102

describe design implications using two trade-off types: (1) prioritizing the sense of agency during assisted-touch or (2)trading agency for touch-assistance that ensures positive outcomes.
If a prospective designer wants to prioritize the sense of agency of a user who experiences computer-driven touch,

1103
1104
1105
1106

they should design their interactive system in such a way that promotes (a) joint success and/or (d) joint failure cases.
Technically speaking, the target interface should make use of reliable and robust ways to estimate the user’s intentions

1107

to ensure that the computer-driven touch and the user-driven touch are aligned. For instance, designers should make1108

use of the confidence intervals provided by the recognizers in real-time to regulate the amount of touch-assistance,1109

following the agency curves we depicted in these two quadrants. As we have demonstrated, these cases will enable

1110
1111

situations where the user feels that their touch is haptically accelerated and still feels agency.

1112

On the other hand, if the prospective designer wants to prioritize the touch-assistance for positive outcomes,1113

the interactive system should be designed in such a way that promotes (a) joint success and (b) forced success. Since

1114

EMS will create computer-driven touches with correct outcomes, we can expect either speeding up the touch that

1115
1116

drives the correct response or correcting the user’s touch into the right outcome. None of these will guarantee that1117

agency is preserved but there are some extra considerations designers should keep in mind. Especially in the case1118

of interfaces that deal with critical or emergency scenarios (health, etc.), the system should prevent injury to the1119

user. As such, touch-assistance with positive outcome is prioritized over the sense of agency. Even if this kind

1120
1121

of touch-assistance removes the user’s agency entirely, the outcome is tremendously beneficial to the user. As such,

1122

if the interactive system can predict risky scenarios (e.g., such as touching the hot cup in [43] or hitting the lamp

1123

post while walking in [62]), the designers should actuate the haptic-assistance regardless of the role (forced success or

1124

joint success). Again, we believe that this is not only relevant for the design of single-user applications but also for

1125
1126
1127

multi-user applications, such as interpersonal communication systems that use haptic actuation, for instance, in remotecommunication between users for telepresence and/or skill-transfer over the network [1, 33, 70].

1128
1129
1130
1131

9

CONCLUSION

We believe that this paper empowers HCI researchers to consider not only touch as a depiction of haptic qualities (e.g.,

1132

texture) but also consider the challenge of assisted-touch, i.e., a touch that is driven by the computer interface via force1133

feedback.

1134
1135
1136

As such, we advanced our understanding of digital touch systems by contributing to the study and design of what
happens when digital touch systems can make their user touch the UI involuntarily. While this type of force-feedback is1137

known to enhance the interpersonal-communication by empowering remote users to share non-verbal aspects such as

1138

rhythm, poses, etc., it also requires forcing the user’s body to move against their intention–this computer-driven touch,1139

unfortunately, causes users to lose their sense-of-agency. While researchers have proposed that delaying the timing of

1140
1141

the computer-driven touch closer to the user’s own voluntary touch preserves some sense of agency, this was only

1142

considered for the simplest case, in which the users have no choice as to what they will touch. We argued that this1143

is an unlikely case, as it assumes we can perfectly predict the user’s own touches. But, what about all the remainder

1144

22

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

1145

situations: when the haptics forces the user into an outcome they did not intend, or when the haptics assists the user in1146

an outcome they would not achieve alone?

1147

We unveiled, via our experiments, what happens in these novel situations. We found out that participant’s attribution

1148
1149

of agency was biased (positively and negatively) by the outcomes, when the computer-driven touch and the user-

1150

driven touch are aligned. On the other hand, when the computer-driven touch created an outcome not intended

1151

by a participant, the attribution of the agency appeared easier to distinguish. Finally, from these study findings, we

1152
1153

synthesized a framework that enables researchers/practitioners of digital-touch systems to prioritize/trade-off between

1154

haptic-assistance and sense-of-agency. As of now, our findings originate from using EMS as the case study for this loss

1155

of agency in haptic systems. Moving forward, we would expect future researchers to apply and extend our novel study

1156

to other force-feedback devices (e.g., magnetic actuation of the finger, mechanical force-feedback joysticks, exoskeletons,

1157
1158

etc.), ultimately contributing to a larger understanding that might generalize better.

1159

Finally, we believe that, as we design new types of haptic systems with the capability to perform actions in behalf of1160

users, we start to encounter all these cases of complex/shared agencies. We believe our study systematically explored1161

many of these cases, in particular the situations in which the computer’s and user’s intentions are in conflict. While

1162
1163
1164

these are not yet commonplace, that is precisely our job as researchers to anticipate the future and design it mindfullyof the user’s agency.

1165
1166

10

ACKNOWLEDGMENTS

1167
1168

Firstly, we would like to thank the TOCHI reviewers and editors for their valuable feedback. Secondly, we would

1169

like to thank our colleague Jas Brooks for assisting us in proof-reading this manuscript. This work was supported by

1170

Grant-in-Aid for JSPS Research Fellow (JP16J03777) and in part by NSF grants 2047189 and 2024923. This work was also

1171
1172

supported by JST Moonshot R&D Program (Grant Number JPMJMS2013). Any opinions, findings, and conclusions or

1173

recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any

1174

funding agencies.

1175
1176
1177

REFERENCES

1195[1] E J Avila Mireles, J Zenzeri, V Squeri, P Morasso, and D De Santis. 2017. Skill Learning and Skill Transfer Mediated by Cooperative Haptic Interaction.
IEEE Trans. Neural Syst. Rehabil. Eng. 25, 7 (July 2017), 832–843.
[2] Jeremy N. Bailenson, Nick Yee Ph.D., Scott Brave Ph.D., Dan Merget, and David Koslow. 2007. Virtual Interpersonal Touch: Expressing andRecognizing Emotions Through Haptic Devices. Human-Computer Interaction 22, 3 (2007), 325–353. https://doi.org/10.1080/07370020701493509
arXiv:https://www.tandfonline.com/doi/pdf/10.1080/07370020701493509[3] Zeynep Barlas, William E. Hockley, and Sukhvinder S. Obhi. 2018. Effects of free choice and outcome valence on the sense of agency: evidence frommeasures of intentional binding and feelings of control. Experimental Brain Research 236, 1 (Jan. 2018), 129–139. https://doi.org/10.1007/s00221-0175112-3[4] Bruno Berberian, Jean-Christophe Sarrazin, Patrick Le Blaye, and Patrick Haggard. 2012. Automation Technology and Sense of Control: A Window
on Human Agency. PLOS ONE 7, 3 (March 2012), e34075. https://doi.org/10.1371/journal.pone.0034075[5] Joanna Bergstrom-Lehtovirta, David Coyle, Jarrod Knibbe, and Kasper HornbÃęk. 2018. I Really did That: Sense of Agency with Touchpad, Keyboard,and On-skin Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, Montreal QC,
Canada, 1–8. https://doi.org/10.1145/3173574.3173952[6] S J Blakemore, D Wolpert, and C Frith. 2000. Why can’t you tickle yourself? Neuroreport 11, 11 (aug 2000), R11–6. http://www.ncbi.nlm.nih.gov/
pubmed/10943682[7] M. Botvinick and J. Cohen. 1998. Rubber hands ’feel’ touch that eyes see. Nature 391, 6669 (Feb. 1998), 756. https://doi.org/10.1038/35784[8] Gifford W. Bradley. 1978. Self-serving biases in the attribution process: A reexamination of the fact or fiction question. Journal of Personality andSocial Psychology 36, 1 (1978), 56–71. https://doi.org/10.1037/0022-3514.36.1.56[9] Scott Brave and Andrew Dahley. 1997. InTouch: A Medium for Haptic Interpersonal Communication. In CHI ’97 Extended Abstracts on Human
Factors in Computing Systems (Atlanta, Georgia) (CHI EA ’97). Association for Computing Machinery, New York, NY, USA, 363–364. https:

1196

23

1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194

Sub., TOCHI, Digital Touch

1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246

Tajima, Nishida, Lopes and Kasahara//doi.org/10.1145/1120212.1120435[10] Scott Brave, Clifford Nass, and Erenee Sirinian. 2001. Force-Feedback in computer-mediated communication. In Universal Access In HCI: Towards an
Information Society for All, Proceedings of HCI International ’2001 (the 9th International Conference on Human-Computer Interaction), New Orleans,
USA, August 5-10, 2001, Volume 3, Constantine Stephanidis (Ed.). Lawrence Erlbaum, 145–149.
[11] Hoon Choi and Brian J. Scholl. 2006. Perceiving causality after the fact: postdiction in the temporal dynamics of causal perception. Perception 35, 3(2006), 385–399. https://doi.org/10.1068/p5462[12] Patricia Ivette Cornelio Martinez, Silvana De Pirro, Chi Thanh Vi, and Sriram Subramanian. 2017. Agency in Mid-air Interfaces. In Proceedings of the2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 2426–2439. https://doi.org/10.1145/3025453.3025457[13] David Coyle, James Moore, Per Ola Kristensson, Paul Fletcher, and Alan Blackwell. 2012. I did that!: measuring users’ experience of agency in theirown actions. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI ’12. ACM Press, Austin, Texas, USA,2025. https://doi.org/10.1145/2207676.2208350[14] Nicole David, Stefan Skoruppa, Alessandro Gulberti, Johannes Schultz, and Andreas K. Engel. 2016. The Sense of Agency Is More Sensitive
to Manipulations of Outcome than Movement-Related Feedback Irrespective of Sensory Modality. PLOS ONE 11, 8 (Aug. 2016), e0161156.
https://doi.org/10.1371/journal.pone.0161156[15] D. M. Eagleman and T. J. Sejnowski. 2000. Motion integration and postdiction in visual awareness. Science (New York, N.Y.) 287, 5460 (March 2000),
2036–2038. https://doi.org/10.1126/science.287.5460.2036[16] Ayaka Ebisu, Satoshi Hashizume, Kenta Suzuki, Akira Ishii, Mose Sakashita, and Yoichi Ochiai. 2016. Stimulated Percussions: Techniques forControlling Human As Percussive Musical Instrument by Using Electrical Muscle Stimulation. In SIGGRAPH ASIA 2016 Posters (Macau) (SA ’16).
ACM, New York, NY, USA, Article 37, 2 pages. https://doi.org/10.1145/3005274.3005324[17] Kai Engbert, Andreas Wohlschlager, and Patrick Haggard. 2008. Who is causing what? The sense of agency is relational and efferent-triggered.
Cognition 107, 2 (May 2008), 693–704. https://doi.org/10.1016/j.cognition.2007.07.021[18] Farzam Farbiz, Zhou Hao Yu, Corey Manders, and Waqas Ahmad. 2007. An Electrical Muscle Stimulation Haptic Feedback for Mixed Reality Tennis
Game. In ACM SIGGRAPH 2007 Posters (San Diego, California) (SIGGRAPH ’07). ACM, New York, NY, USA, Article 140. https://doi.org/10.1145/
1280720.1280873[19] Umer Farooq and Jonathan Grudin. 2016. Human-computer integration. interactions 23, 6 (2016), 26–32.
[20] Leon Festinger. 1957. A theory of cognitive dissonance. Stanford University Press.
[21] C. D. Frith, S. J. Blakemore, and D. M. Wolpert. 2000. Abnormalities in the awareness and control of action. Philosophical Transactions of the Royal
Society of London. Series B, Biological Sciences 355, 1404 (Dec. 2000), 1771–1788. https://doi.org/10.1098/rstb.2000.0734[22] Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical
Exoskeleton for Motion Capture and Force Feedback in VR. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (SanJose, California, USA) (CHI ’16). ACM, New York, NY, USA, 1991–1995. https://doi.org/10.1145/2858036.2858487[23] Patrick Haggard, Sam Clark, and Jeri Kalogeras. 2002. Voluntary action and conscious awareness. Nature Neuroscience 5, 4 (April 2002), 382–385.
https://doi.org/10.1038/nn827[24] Shin Hanagata and Yasuaki Kakehi. 2018. Paralogue: A Remote Conversation System Using a Hand Avatar Which Postures Are Controlled with
Electrical Muscle Stimulation. In Proceedings of the 9th Augmented Human International Conference (Seoul, Republic of Korea) (AH ’18). Association
for Computing Machinery, New York, NY, USA, Article 35, 3 pages. https://doi.org/10.1145/3174910.3174951[25] Mariam Hassib, Max Pfeiffer, Stefan Schneegass, Michael Rohs, and Florian Alt. 2017. Emotion Actuator: Embodied Emotional Feedback throughElectroencephalography and Electrical Muscle Stimulation. Association for Computing Machinery, New York, NY, USA, 6133–6146. https://doi.org/
10.1145/3025453.3025953[26] Andrew Heathcote, Stephen J. Popiel, and D. J.K. Mewhort. 1991. Analysis of response time distributions: an example using the stroop task.
Psychological Bulletin 109, 2 (1991), 340–347. https://doi.org/10.1037/0033-2909.109.2.340[27] Pilwon Heo, Gwang Min Gu, Soo-jin Lee, Kyehan Rhee, and Jung Kim. 2012. Current hand exoskeleton technologies for rehabilitation and assistiveengineering. International Journal of Precision Engineering and Manufacturing 13, 5 (May 2012), 807–824. https://doi.org/10.1007/s12541-012-0107-2[28] R Hyman. 1953. Stimulus information as a determinant of reaction time. J. Exp. Psychol. 45, 3 (March 1953), 188–196.
[29] Marc Jeannerod. 2003. The mechanism of self-recognition in humans. Behavioural Brain Research 142, 1-2 (June 2003), 1–15. https://doi.org/10.
1016/S0166-4328(02)00384-4[30] C. Jeunet, L. Albert, F. Argelaguet, and A. Lecuyer. 2018. "Do You Feel in Control?": Towards Novel Approaches to Characterise, Manipulate andMeasure the Sense of Agency in Virtual Environments. IEEE Transactions on Visualization and Computer Graphics 24, 4 (April 2018), 1486–1495.
https://doi.org/10.1109/TVCG.2018.2794598[31] Shunichi Kasahara, Jun Nishida, and Pedro Lopes. 2019. Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation
Without Compromising Agency. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow,
Scotland Uk, 1–15. https://doi.org/10.1145/3290605.3300873[32] Shunichi Kasahara, Kazuma Takada, Jun Nishida, Kazuhisa Shibata, Shinsuke Shimojo, and Pedro Lopes. 2021. Preserving Agency During Electrical
Muscle Stimulation Training Speeds up Reaction Time Directly After Removing EMS. In Proceedings of the 2021 CHI Conference on Human Factors in
Computing Systems. 1–9.

1247
1248

24

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

Sub., TOCHI, Digital Touch

1299[33] T Kurogi, M Nakayama, K Sato, S Kamuro, C L Fernando, M Furukawa, K Minamizawa, and S Tachi. 2013. Haptic transmission system to recognize
differences in surface textures of objects for telexistence. In 2013 IEEE Virtual Reality (VR). 137–138.
[34] Daniel Leithinger, Sean Follmer, Alex Olwal, and Hiroshi Ishii. 2014. Physical Telepresence: Shape Capture and Display for Embodied, ComputerMediated Remote Collaboration. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii,USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 461–470. https://doi.org/10.1145/2642918.2647377[35] BENJAMIN LIBET, CURTIS A GLEASON, and ELWOOD W WRIGHT. [n.d.]. TIME OF CONSCIOUS INTENTION TO ACT IN RELATION TO
ONSET OF CEREBRAL ACTIVITY (READINESS-POTENTIAL). ([n. d.]), 20.
[36] Jakub Limanowski and Felix Blankenburg. 2016. That’s not quite me: limb ownership encoding in the brain. Social Cognitive and Affective
Neuroscience 11, 7 (July 2016), 1130–1140. https://doi.org/10.1093/scan/nsv079[37] Jakub Limanowski, Pedro Lopes, Janis Keck, Patrick Baudisch, Karl Friston, and Felix Blankenburg. 2020. Action-dependent processing of touch in
the human parietal operculum and posterior insula. Cerebral Cortex 30, 2 (2020), 607–617.
[38] Hannah Limerick, James W. Moore, and David Coyle. 2015. Empirical Evidence for a Diminished Sense of Agency in Speech Interfaces. In
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 3967–3970. https://doi.org/10.1145/2702123.2702379[39] Pedro Lopes and Patrick Baudisch. 2013. Muscle-propelled Force Feedback: Bringing Force Feedback to Mobile Devices. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). ACM, New York, NY, USA, 2577–2580. https://doi.org/10.1145/2470654.
2481355[40] P. Lopes and P. Baudisch. 2017. Immense Power in a Tiny Package: Wearables Based on Electrical Muscle Stimulation. IEEE Pervasive Computing 16,
03 (jul 2017), 12–16. https://doi.org/10.1109/MPRV.2017.2940953[41] Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical
Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software; Technology (Charlotte, NC, USA) (UIST ’15). ACM,
New York, NY, USA, 11–19. https://doi.org/10.1145/2807442.2807443[42] Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. 2015. Proprioceptive Interaction. In Proceedings ofthe 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 939–948.
https://doi.org/10.1145/2702123.2702461[43] Pedro Lopes, Patrik Jonell, and Patrick Baudisch. 2015. Affordance++: Allowing Objects to Communicate Dynamic Use. In Proceedings of the
33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 2515–2524.
https://doi.org/10.1145/2702123.2702128[44] Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Providing Haptics to Walls &#38; Heavy Objects in
Virtual Reality by Means of Electrical Muscle Stimulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver,
Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 1471–1482. https://doi.org/10.1145/3025453.3025600[45] Takaki Maeda, Motoichiro Kato, Taro Muramatsu, Satoru Iwashita, Masaru Mimura, and Haruo Kashima. 2012. Aberrant sense of agency in patientswith schizophrenia: forward and backward over-attribution of temporal causality during intentional action. Psychiatry Research 198, 1 (June 2012),
1–6. https://doi.org/10.1016/j.psychres.2011.10.021[46] Azumi Maekawa, Shota Takahashi, MHD Yamen Saraiji, Sohei Wakisaka, Hiroyasu Iwata, and Masahiko Inami. 2019. Naviarm: Augmenting the
Learning of Motor Skills Using a Backpack-Type Robotic Arm System. In Proceedings of the 10th Augmented Human International Conference 2019
(Reims, France) (AH2019). Association for Computing Machinery, New York, NY, USA, Article 38, 8 pages. https://doi.org/10.1145/3311823.3311849[47] Masao Matsuhashi and Mark Hallett. 2008. The timing of the conscious intention to move. The European journal of neuroscience 28, 11 (Dec. 2008),
2344–2351. https://doi.org/10.1111/j.1460-9568.2008.06525.x[48] Alan McCall, Maurizio Fanchini, and Aaron J Coutts. 2017. Prediction: the modern-day sport-science and sports-medicine âĂĲquest for the holygrailâĂİ. International journal of sports physiology and performance 12, 5 (2017), 704–706.
[49] John McEneaney. 2013. Agency Effects in Human-Computer Interaction. International Journal of Human-Computer Interaction 29 (Sept. 2013),
798–813. https://doi.org/10.1080/10447318.2013.777826[50] D. J.K. Mewhort, J. G. Braun, and Andrew Heathcote. 1992. Response Time Distributions and the Stroop Task: A Test of the Cohen, Dunbar, andMcClelland (1990) Model. Journal of Experimental Psychology: Human Perception and Performance 18, 3 (1992), 872–882. https://doi.org/10.1037/00961523.18.3.872[51] Christopher A. Miller and Raja Parasuraman. 2007. Designing for Flexible Interaction Between Humans and Automation: Delegation Interfaces forSupervisory Control. Human Factors 49, 1 (Feb. 2007), 57–75. https://doi.org/10.1518/001872007779598037[52] A L Morgan and J F Brandt. 1989. An auditory Stroop effect for pitch, loudness, and time. Brain Lang. 36, 4 (May 1989), 592–603.
[53] Florian Floyd Mueller, Pedro Lopes, Paul Strohmeier, Wendy Ju, Caitlyn Seim, Martin Weigel, Suranga Nanayakkara, Marianna Obrist, Zhuying Li,
Joseph Delfa, et al. 2020. Next Steps for Human-Computer Integration. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems. 1–15.
[54] Jun Murayama, Laroussi Bougrila, Yanlinluo Katsuhito Akahane, Shoichi Hasegawa, Béat Hirsbrunner, and Makoto Sato. 2004. SPIDAR G&G: A
two-handed haptic interface for bimanual VR interaction. In Proceedings of EuroHaptics 2004.
[55] Ken Nakagaki, Chikara Inamura, Pasquale Totaro, Thariq Shihipar, Chantine Akikyama, Yin Shuang, and Hiroshi Ishii. 2015. Linked-Stick:
Conveying a Physical Experience Using a Shape-Shifting Stick. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human

1300

25

1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298

Sub., TOCHI, Digital Touch

1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350

Tajima, Nishida, Lopes and Kasahara

Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA ’15). Association for Computing Machinery, New York, NY, USA, 1609–1614.
https://doi.org/10.1145/2702613.2732712[56] Hideyuki Nakanishi, Kazuaki Tanaka, and Yuya Wada. 2014. Remote Handshaking: Touch Enhances Video-Mediated Social Telepresence. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing
Machinery, New York, NY, USA, 2143–2152. https://doi.org/10.1145/2556288.2557169[57] Jun Nishida, Shunichi Kasahara, and Pedro Lopes. 2019. Demonstrating Preemptive Reaction: Accelerating Human Reaction Using Electrical Muscle
Stimulation Without Compromising Agency. In ACM SIGGRAPH 2019 Emerging Technologies (SIGGRAPH ’19). ACM, New York, NY, USA, 10:1–10:2.
https://doi.org/10.1145/3305367.3327997[58] Jun Nishida, Shunichi Kasahara, and Kenji Suzuki. 2017. Wired Muscle: Generating Faster Kinesthetic Reaction by Inter-personally Connecting
Muscles. In ACM SIGGRAPH 2017 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’17). ACM, New York, NY, USA, Article 26, 2 pages.
https://doi.org/10.1145/3084822.3084844[59] Jun Nishida and Kenji Suzuki. 2017. BioSync: A Paired Wearable Device for Blending Kinesthetic Experience. Association for Computing Machinery,
New York, NY, USA, 3316–3327. https://doi.org/10.1145/3025453.3025829[60] J. Nishida, K. Yagi, M. Hassan, and K. Suzuki. 2019. Wearable Kinesthetic I/O Device for Sharing Wrist Joint Stiffness. In 2019 41st Annual International
Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 3306–3310. https://doi.org/10.1109/EMBC.2019.8857538[61] Jay A. Olson, Mathieu Landry, KrystÃĺle Appourchaux, and Amir Raz. 2016. Simulated thought insertion: Influencing the sense of agency using
deception and magic. Consciousness and Cognition 43 (July 2016), 11–26. https://doi.org/10.1016/j.concog.2016.04.010[62] Max Pfeiffer, Tim Dünte, Stefan Schneegass, Florian Alt, and Michael Rohs. 2015. Cruise Control for Pedestrians: Controlling Walking Direction
Using Electrical Muscle Stimulation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic ofKorea) (CHI ’15). ACM, New York, NY, USA, 2505–2514. https://doi.org/10.1145/2702123.2702190[63] MHD Yamen Saraiji, Tomoya Sasaki, Reo Matsumura, Kouta Minamizawa, and Masahiko Inami. 2018. Fusion: Full Body Surrogacy for Collaborative
Communication. In ACM SIGGRAPH 2018 Emerging Technologies (Vancouver, British Columbia, Canada) (SIGGRAPH ’18). Association for Computing
Machinery, New York, NY, USA, Article 7, 2 pages. https://doi.org/10.1145/3214907.3214912[64] Shinsuke Shimojo. 2014. Postdiction: its implications on visual awareness, hindsight, and sense of agency. Frontiers in Psychology 5 (2014).
https://doi.org/10.3389/fpsyg.2014.00196[65] Irwin W Silverman. 2010. Simple reaction time: it is not what it used to be. Am. J. Psychol. 123, 1 (2010), 39–50.
[66] Chun Siong Soon, Anna Hanxi He, Stefan Bode, and John-Dylan Haynes. 2013. Predicting free choices for abstract intentions. Proceedings of the
National Academy of Sciences 110, 15 (2013), 6217–6222.
[67] P. Strojnik, A. Kralj, and I. Ursic. 1979. Programmed Six-Channel Electrical Stimulator for Complex Stimulation of Leg Muscles During Walking.
IEEE Transactions on Biomedical Engineering BME-26, 2 (Feb 1979), 112–116. https://doi.org/10.1109/TBME.1979.326520[68] Kenta Suzuki, Gouji Mito, Hiroaki Kawamoto, Yasuhisa Hasegawa, and Yoshiyuki Sankai. 2007. Intention-based walking support forparaplegia patients with Robot Suit HAL. Advanced Robotics 21, 12 (2007), 1441–1469.
https://doi.org/10.1163/156855307781746061
arXiv:https://www.tandfonline.com/doi/pdf/10.1163/156855307781746061[69] Akifumi Takahashi, Jas Brooks, Hiroyuki Kajimoto, and Pedro Lopes. 2021. Increasing Electrical Muscle Stimulation’s Dexterity by Means of Back of
the Hand Actuation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for
Computing Machinery, New York, NY, USA, Article 216, 12 pages. https://doi.org/10.1145/3411764.3445761[70] Nobuhiro Takahashi, Hayato Takahashi, and Hideki Koike. 2019. Soft Exoskeleton Glove Enabling Force Feedback for Human-Like Finger PostureControl with 20 Degrees of Freedom. In 2019 IEEE World Haptics Conference (WHC). IEEE, Tokyo, Japan, 217–222. https://doi.org/10.1109/WHC.
2019.8816142[71] Keisuke Takahata, Hidehiko Takahashi, Takaki Maeda, Satoshi Umeda, Tetsuya Suhara, Masaru Mimura, and Motoichiro Kato. 2012. ItâĂŹs Not My
Fault: Postdictive Modulation of Intentional Binding by Monetary Gains and Losses. PloS one 7 (Dec. 2012), e53421. https://doi.org/10.1371/journal.
pone.0053421[72] Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011. PossessedHand: Techniques for Controlling Human Hands Using Electrical Muscles Stimuli.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). ACM, New York, NY, USA,
543–552. https://doi.org/10.1145/1978942.1979018[73] Daniel M. Wegner. 2003. The mind’s best trick: how we experience conscious will. Trends in Cognitive Sciences 7, 2 (Feb. 2003), 65–69.
[74] Sebastien Weibel, Patrick Eric Poncelet, Yvonne Delevoye-Turrell, Antonio Capobianco, Andre Dufour, Renaud Brochard, Laurent Ott, and Anne
Giersch. 2015. Feeling of control of an action after supra and subliminal haptic distortions. Consciousness and Cognition 35 (Sept. 2015), 16–29.
https://doi.org/10.1016/j.concog.2015.04.011[75] Bernard Weiner. 2006. Social motivation, justice, and the moral emotions: An attributional approach. Lawrence Erlbaum Associates Publishers,
Mahwah, NJ, US.
[76] Andrew Wold, Jakub Limanowski, Henrik Walter, and Felix Blankenburg. 2014. Proprioceptive drift in the rubber hand illusion is intensified
following 1 Hz TMS of the left EBA. Frontiers in Human Neuroscience 8 (2014). https://doi.org/10.3389/fnhum.2014.00390[77] Motonori Yamaguchi, Emma L. Clarke, and Danny L. Egan. 2018. Is Your Color My Color? Dividing the Labor of the Stroop Task Between Co-actors.
Frontiers in Psychology 9, August (2018), 1–11. https://doi.org/10.3389/fpsyg.2018.01407

1351
1352

26

Whose touch is this?: Understanding the Agency Trade-off Between User-driven touch vs. Computer-driven Touch

1353
1354
1355
1356

[78] Yasuhisa Hasegawa, Yasuyuki Mikami, Kosuke Watanabe, and Yoshiyuki Sankai. 2008. Five-fingered assistive hand with mechanical compliance ofhuman finger. In 2008 IEEE International Conference on Robotics and Automation. 718–724. https://doi.org/10.1109/ROBOT.2008.4543290[79] Michiko Yoshie and Patrick Haggard. 2013. Negative Emotional Outcomes Attenuate Sense of Agency over Voluntary Actions. Current Biology 23,
20 (Oct. 2013), 2028–2032. https://doi.org/10.1016/j.cub.2013.08.034

1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404

Sub., TOCHI, Digital Touch27