CHI 2019 Paper

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Detecting Visuo-Haptic Mismatches in Virtual Reality
using the Prediction Error Negativity of
Event-Related Brain Potentials
Lukas Gehrke
lukas.gehrke@tu-berlin.deTechnical University of Berlin
Berlin, Germany

Sezen Akman
sezen.akman@campus.tu-berlin.deTechnical University of Berlin
Berlin, Germany

Pedro Lopes∗
pedrolopes@cs.uchicago.edu
University of Chicago
Chicago, Illinois, USA

Albert Chen
albertc1@uchicago.edu
University of Chicago
Chicago, Illinois, USA

Avinash Kumar Singhavinash.singh@uts.edu.au
National Chiao Tung University
Hsinchu, Taiwan

Hsiang-Ting Chen
Tim.Chen@uts.edu.au
University of Technology Sydney
Sidney, Australia

Chin Teng Lin
Chin-Teng.Lin@uts.edu.au
University of Technology Sydney
Sidney, Australia
ABSTRACT
Designing immersion is the key challenge in virtual reality;
this challenge has driven advancements in displays, rendering and recently, haptics. To increase our sense of physicalimmersion, for instance, vibrotactile gloves render the senseof touching, while electrical muscle stimulation (EMS) renders forces. Unfortunately, the established metric to assess
the effectiveness of haptic devices relies on the user’s subjective interpretation of unspecific, yet standardized, questions.
Here, we explore a new approach to detect a conflict invisuo-haptic integration (e.g., inadequate haptic feedback
based on poorly configured collision detection) using electroencephalography (EEG). We propose analyzing eventrelated potentials (ERPs) during interaction with virtual objects. In our study, participants touched virtual objects inthree conditions and received either no haptic feedback, vibration, or vibration and EMS feedback. To provoke a brain∗ Author affiliated also with Hasso Plattner Institute, Germany.
† Author affiliated also with University of California, San Diego.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
CHI 2019, May 4–9, 2019, Glasgow, Scotland Uk© 2019 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
ACM ISBN 978-1-4503-5970-2/19/05. . . $15.00
https://doi.org/10.1145/3290605.3300657

Paper 427

Klaus Gramann†klaus.gramann@tu-berlin.deTechnical University of Berlin
Berlin, Germany
response in unrealistic VR interaction, we also presented thefeedback prematurely in 25% of the trials.
We found that the early negativity component of the ERP(so called prediction error) was more pronounced in themismatch trials, indicating we successfully detected haptic
conflicts using our technique. Our results are a first step
towards using ERPs to automatically detect visuo-haptic
mismatches in VR, such as those that can cause a loss of theuser’s immersion.
CCS CONCEPTS• Human-centered computing → Haptic devices; Virtual reality.
KEYWORDS
force feedback; EEG; elecrical muscle stimulation; virtual
reality; ERP; prediction error
ACM Reference Format:
Lukas Gehrke, Sezen Akman, Pedro Lopes, Albert Chen, Avinash
Kumar Singh, Hsiang-Ting Chen, Chin Teng Lin, and Klaus Gramann. 2019. Detecting Visuo-Haptic Mismatches in Virtual Reality
using the Prediction Error Negativity of Event-Related Brain Potentials. In CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland Uk. ACM, New
York, NY, USA, 11 pages. https://doi.org/10.1145/3290605.3300657

1 INTROA key challenge in virtual reality is to create a user experience that mimics the natural experience as closely as possible.
This challenge has propelled advancements in display software and hardware (VR headsets and rendering), interaction

Page 1CHI 2019 Paper

techniques and, more recently, in haptic technology. In fact,many researchers argue that attaining haptic realism is thenext grand challenge in virtual reality [8, 51].
The addition of haptic feedback in VR has been shown
to dramatically increase the user’s sense of immersion [38].
For instance, vibrotactile gloves [22] stimulate the user’s
sense of touch, and force feedback and exoskeletons [7, 19]or electrical muscle stimulation [26, 27] stimulate the user’s
proprioceptive system.
To better understand how different interaction technologies support real world-like user experiences, questionnaires
are used that ask "how realistic is it? (from 1-7)" [42, 45, 50].
These questionnaires are also used as a metric to assess how
effective a haptic device is in rendering a realistic simulation(e.g., [2, 39, 56] just to mention a few). However, as Slater
pointed out in his critique, these metrics are subjective [44],i.e., they rely on the user’s own introspection and frameof reference. Furthermore, these metrics require breaking
the user’s immersion—literally, as they require the user tohalt the immersive experience—to collect the data about theprevious interaction.
Instead, in this paper, we propose analyzing the user’s
brain responses as a first step towards automatically detecting visuo-haptic mismatches, such as those that can cause
loss of immersion. In the future, such a technique can be seen
as a complimentary or even alternative metric of visuo-haptic
immersion that, unlike questionnaires, does not require anytask interruption or subjective reflection. Our approach, depicted in 1, works by analyzing the user’s brain dynamics
captured by an EEG worn under the VR headset. We foundthat we can use the user’s brain potentials to detect sensory
mismatches that occur in moments where the VR experience
is not immersive (e.g., due to a poorly configured collision
detection, inadequate or delayed haptic information, etc.).
2

TOWARDS ERPS AS A METRIC FOR HAPTICIMMERSION
The goal of haptic devices is to render realistic sensory feedback that mimics the sensory experience a user would normally perceive when interacting with the real world. The
simple case that we examine here is of touching an object.
Imagine grasping a cup of coffee on the breakfast table: as
we reach out for the cup, our visual system provides ongoing
feedback about the position of the arm and hand relative
to the cup on the table, while proprioceptive feedback frommuscles and joints provides information about the relative
position of the hand and the strength of our grasp. Combined with the motor plan, the sensory feedback is used tocompare what is effectively happening in the environment
with what was predicted to happen [11]. When making contact with the cup, the visual and proprioceptive feedback
are integrated with haptic feedback providing information

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Figure 1: We propose using the prediction error negativityof the brain’s event related potential (ERP) to detect visuohaptic conflicts arising from unrealistic VR feedback. In ourstudy, participants selected objects in VR. To provoke theirbrains to process an unrealistic interaction, we sometimes
provided the haptic feedback prematurely. When subtracting these ERPs to the ERPs from realistic interactions, wefound that the negative amplitude of the error prediction
increased, hinting at a loss in immersion.

about the contact with the object. In the case where all sensory information channels provide consistent feedback, theaction would be successful (and the coffee can be enjoyed).
However, in the case of a mismatch in the incoming information, attention has to be directed to this mismatch so that theaction can be corrected in real-time [41]. It is precisely thisidea – that the brain has evolved to optimize motor behavior
based on detected sensory mismatches – that inspired us
to investigate brain responses to sensory mismatches as apotential metric for haptic immersion.
To investigate these prediction errors in VR, an electroencehalogram (EEG) can be used. An EEG measures the electricalactivity of cortical neurons in the human brain with hightemporal resolution [32]. Transient sensory events (e.g., haptic feedback from touching a coffee mug) evoke event-related
potentials (ERPs) in the ongoing oscillatory activity of thebrain, reflecting sensory and cognitive processing of incoming stimuli [29]. An ERP is a stereotyped response comprisedof a series of positive and negative deflections. One specificcomponent of the ERP is the prediction error negativity(PEN)—a negative potential that occurs from 100 to 200 ms
whenever a deviation from the predicted state of the environment is detected [41]. We propose utilizing this prediction
error (highlighted in Figure1) as a metric for haptic immersion, a potential immersion indicator that does not require

Page 2

CHI 2019 Paper

subjective interpretation or interrupting the user, which results in breaking the immersive experience. Furthermore,
as we will discuss, this metric can be used in realtime tocontinuously adapt an environment depending on the users
prediction of and actual state of the environment.
To actualize this proposal, we conducted a user study inwhich we measured the brain activity of 11 participants using
a 64 channel EEG system. During the VR experience, participants touched different virtual objects with each touch being
accompanied by feedback via the incremental combinationof popular feedback modalities: (1) visual feedback, (2) tactile(via vibration) + visual feedback, and (3) force feedback (via
EMS) + tactile + visual feedback. To provoke the participant’s
brain into processing the experience of an unrealistic VR interaction, we provided the haptic feedback prematurely in
25% of the trials. When comparing these ERPs to the ERPs
from realistic interactions, we found that the amplitude of
the PEN increased, indicating that we can successfully detect
error processing hinting at a loss in immersion, without interrupting the VR experience. Furthermore, we found that thiserror prediction systematically covaried with the number of
feedback channels. Before detailing our experiment, we willleverage the HCI and neuroscience literature to ground our
ERP-based approach.
3 RELATED WORK
Our approach builds on the research done in the fields ofVirtual Reality (VR), haptics, neuroscience and cognitive
psychology.
Assessing immersion/presence in VR
One of the most commonly used questionnaires for evaluating how a user experiences presence in a virtual environmentis the Igroup Presence Questionnaire (IPQ) [42]. The questionnaire spans over 14 questions on four domains: general
presence, spatial presence, involvement and experienced realism. Its broad application scope makes it a widely adopted
metric used by [14, 16], just to cite a few. The authors of the
IPQ state that they interpret sense of presence as an individual experience and therefore a matter of subjective rating
scales. However, researchers have critiqued this approach
precisely due to its subjectivity. For instance, Slater elaborated a critique on these metrics in [44] and as Garvia-Valle etal. put it "presence is a subjective parameter, and that is why
the results depend on the participant opinion"; they evenadd "their answers depend on their level of expertise" [15].
Assessing haptic immersion
There are many ways to evaluate a haptic device, one of themore established is the Just-noticeable difference (JND) [46].
This study design allows researchers to measure the perceptual threshold by forcing the user to consider whether two

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

haptic events are dissimilar. While methods such as the JND
are very popular (e.g., [1, 37]), they target the user’s perceptual apparatus and are not a measure of haptic immersion in
a VR environment. Thus, many researchers rely the presencequestionnaire to assess haptic immersion.
The IPQ has been used by many researchers seeking tobetter understand their haptic devices. Just to exemplify afew: a vibrotactile cane for blind VR users [56]; a multihaptics interface that uses a combination of vibration, wind,
water spray and heat [39]; a wind feedback device basedon head-mounted fans [23]; vibration feedback for rollercoaster experiences [2]; an exoskeleton that provides forcefeedback [9]; and so forth.
In certain cases, the researchers selected a few questions of
the standard IPQ, but felt the need to append haptic specificquestions. For example, when Calvo et al. evaluate theiraforementioned exoskeleton device in a bow and arrow VR
simulator [9], they appended two haptic specific questions
to the IPQ, so they could evaluate the sensation of pulling on
the bowstring. Moreover, there seems to be an abundance ofrecent research in VR haptics that does not use the IPQ at
all and, instead, authors craft their own questions directly
targeted at their devices (e.g., [15, 21, 26]). These examples
propelled us to explore a complementary or perhaps evenalternative method to detect conflicts in the user’s sense of
haptic immersion.
The impact of realism in brain responses
The idea to analyze the user’s brain response as an indicatorof the user’s state has become increasingly popular at theintersection of neuroscience and HCI [5, 52, 54].
For instance, Zander et al. revealed that, as users observed
a cursor moving towards a target on a screen, any deviationfrom the user’s expectation about the cursor path was reflected in the amplitude of the prediction error negativity(PEN) [55]. Similarly, Holroyd et al. found out that participants exhibit a negative potential around 200ms after seeing
a visual stimulus that fell outside their expectations [20].
Along the same lines, Coles et al. found that the negative
evoked potentials are indeed sensitive to the processing ofincoming stimulus [12]. These studies utilized precisely thesame component of the ERP we propose for detecting visuohaptic conflicts.
More recently, Singh et al. demonstrated that, in an object
selection task in VR, the PEN component of the ERP was
more pronounced for incorrect feedback when the user’s
hand was represented by a realistic hand avatar as compared
to unrealistic representations [43]. Furthermore, they foundthe PEN amplitude correlated with the level of realism of theavatar hand, as suggested by the uncanny valley theory [30].
Also, Padrao et al. have shown, in a VR simulation, a differentiation of both the ERP amplitudes and latency between

Page 3

CHI 2019 Paper

self-generated and observed avatar behavioral errors in areaction time task prone designed to provoke errors in thetrials [34].
Moreover, Gonzalez-Franco and Lanier recently argued
that it is precisely the sensory prediction model that enables
all illusions in VR to take place [17]; thus, it is a key neuralmechanism for understanding immersion. Taken together,
these results show a link between the prediction error signal
and the level of immersion, suggesting that the increased
immersive experience of the user is reflected in an increased
sensitivity to deviations from the expected changes in the
VR environment during the interaction.
4 USER STUDY
The objective of our study is to explore whether ERPs havepotential in detecting visuo-haptic conflicts in VR. As such,
we designed a study in which participants perform a 3D object selection task in VR (modeled after [43]). As a participant
reaches out to touch an object, they are presented with three
sensory feedback modalities (a visual baseline, tactile andtactile with force-feedback). However, to provoke the participants’ brains into processing an unrealistic VR interaction,
we sometimes provide the feedback prematurely.
We hypothesized that the prediction error negativity (PEN)
component of the ERP would respond to this sensory conflictin a systematic manner.
Participants
We recruited 11 participants from our local institution (7female and 4 male; mean = 27.5 years old, sd = 2.8), allright-handed. No participant had experienced VR with either
vibrotactile feedback at the fingertip or any form of force
feedback, including EMS. Participants received 12 USD per
hour. The study design was approved by the local ethics
committee and all participants provided written informed
consent prior to their participation.
Apparatus
The experimental setup, depicted in Figure 2, comprised: (1)a VR headset and a wrist-mounted wearable VIVE tracker,(2) a 64-channel EEG system, (3) one vibrotactile actuatorworn on the fingertip, and (4) a medically-compliant EMS
device connected via two electrodes worn on the forearm. To
assist readers in replicating our experiment, we provide thenecessary technical details, the complete source code to the
VR experiment, the collected data, and the analysis scripts 1 .
(1) VR and hand tracking. We used an HTC Vive headset with the Vive Deluxe Audio Strap to ensure a good fit andless discomfort due to the EEG cap. We used a Vive Tracker,
attached to the participant’s wrist, to track their right hand.
1 https://osf.io/x7hnm/

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

64 channel EEG

vibration
motor
EMS

Figure 2: Our experimental setup (image with consent fromparticipant).

(2) Vibrotactile feedback. We used a vibration motor(Model 308-100 from Precision Microdrives), which generates
0.8g at 200Hz. This motor measures 8mm in diameter, making
it ideal for the fingertip. The vibration feedback was driven
at 70mA by a 2N7000 MOSFET, which was connected to an
Arduino output pin at 3V.
(3) Force feedback. We actuated the index finger via
electrical muscle stimulation (EMS), which was delivered via
two electrodes attached to the participants’ extensor digitorum muscle. We utilized the extensor digitorum since wefound that we can robustly actuate it without inducing parasitical motion of neighboring muscles; this was verified
during pilot studies. This finger actuation was achieved via
a medically-compliant battery powered muscle stimulator(Rehastim from Hasomed), which provides a maximum of
100mA and is controllable via USB. We chose this device
since it had been successfully used by researchers as a means
to generate force feedback in both VR [27] and AR [28]. The
EMS was pre-calibrated per participant to ensure a pain-freestimulation and robust actuation.
(4) EEG Setup. EEG data was recorded from 64 actively
amplified electrodes using BrainAmp DC amplifiers from
BrainProducts. Electrodes were placed according to the extended 10% system [33]. After fitting the cap, all electrodes
were filled with conductive gel to ensure proper conductivity
and electrode impedance was brought below 5kOhm for all
electrodes. EEG data was recorded with a sampling rate of1000 Hz. We synchronized tracking, EEG data, and an experiment marker stream that marked sections of the study
procedure using labstreaminglayer2 .
2 https://github.com/sccn/labstreaminglayer

Page 4

CHI 2019 Paper

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Figure 3: Interaction flow depicting one trial in our 3D object selection task.

Training phase
We asked participants to wear the HTC VIVE VR headset
for a maximum of 24 trials practice trials. Overall, the EEG
fitting, calibration, and practice trials took around 30 minutes(with two experimenters).
Task
Participants performed a 3D object selection task in VR. Theinteraction flow of our task, depicted in Figure 3, was asfollows: (1) participants moved their hands from the resting
position to the ready position, to indicate they were ready tostart the next trial; (2) participants waited for a new target
to appear (the time of a new target spawning was randomized between 1-2 s); (3) then, the target (a cube) would appear in one of three possible positions (center, left, right),
all equidistant from the participant’s ready position; (4) then,participants acquired the target by moving and touching thetarget with their index finger. (5) After a target was acquired,
participants moved back to the resting position. Here, they
could take a break before the next trial.
Interface conditions
Participants performed the task in three additive feedback
conditions:(1) visual-only (Visual): when participants touched the
cube, it changed its color from white to red (visual feedback);
our no-haptics baseline(2) tactile (Vibro): when participants touched the cube in
the vibro condition, they received a 100 ms vibroactile stimulus and the color change (visual + tactile feedback); this is
the only available haptic feedback in today’s VR experiences.
(3) force-feedback (EMS): in this condition, participantsalso received a 100 ms of EMS stimulation at the index finger
extensor in addition to the visual and vibrotactile feedback(visual + tactile + force feedback). As prior research showed
the EMS stimulation of the opposing muscle (in our case, theextensor) is perceived as the resisting force that arises frompushing against the cube (i.e., force feedback) [25–27].
We designed our three feedback conditions additively because additional haptic feedback is generally associated with

Paper 427

more haptic realism, therefore, we hypothesized that ERPs
would demonstrate some correlation to the ascending levelof the feedback’s realism.
Introducing Visuo-Haptic Mismatches
To allow us to compare the elicited ERPs in a realistic vs.
unrealistic interaction, we presented participants with two
different classes of trials: match trials (C) (75% of the trials) and mismatch trials (M) (25%). This procedure elicits
a prediction mismatch signal in 25% of the trials similar toprevious designs investigating the impact of target probabilities on ERP modulations [36]. In the matching trials, thefeedback stimuli were presented upon touching the object,
exactly when participants expected them to occur based on
the available visual information (finger touching the target).
In contrast, in the mismatch trials, the feedback stimuli
were triggered prematurely, which was accomplished by enlarging the invisible radius of touch detection by 350%. Whilein the match trials, we used a cube collider of the exact size of
the VR cube, in the mismatch trials, we used a larger spherecollider. Our collider enlargement was based on the study
design by Singh et al. [43], in which they showed that VR
users can detect a visual mismatch at around 200% of offset
from the target. In our pilot tests, we decided to extend theoffset to 350% to make the mismatch more obvious so as toprovoke more pronounced prediction errors.
Also, we used a match-to-mismatch ratio of 75%-25% of
the total trials by modeling our study after previous studies, which also ensure that participants are faced with a detectable unrealistic behavior of the virtual environment [13,24, 48]. For these unrealistic trials to occur, the participants
must first be able create a stable model of how the VR world
operates, thus the VR world cannot behave at a random
50%-50% match-mismatch ratio.
Finally, these match vs. mismatch trials were presented
in five randomly generated sequences, each with an equal
distribution of matches and mismatches.

Page 5

CHI 2019 Paper

Experimental design
The experiment consisted of five phases: (1) a setup phase;(2) a calibration phase; (3) a short training phase; (4) thetask itself, in all three possible interface conditions, eachfollowed by a subset of the IPQ questionnaire; and, lastly(5) participants were asked about their experience in the VR
and which condition they enjoyed the most.
We used a within-subjects design with 100 trials per feedback condition. The order of the Visual and Vibro conditions
was randomized across participants with the EMS condition
always being the last block. This was done to avoid potential overshadowing of the EMS stimulation (a very strong
sensation) on the two other stimulation conditions.
For completeness, at the end of each condition we presented the four most relevant questions from the standard
IPQ [42], in particular: G1, REAL2, SP4 and INV1. However,
our hypothesis was that the inclusion mismatch trials, whichwere presented in 25% of the cases, would lower the IPQ
ratings dramatically.
EEG data processing
We utilized the EEGLAB3 and MoBILAB4 toolboxes inside the
MATLAB environment for our analysis. To assist the reader
in replicating our analyses, we provide data and scripts5 .
The inherent delay of the EEG setup was corrected by subtracting 63ms to the timestamps. The raw EEG data wasthen re-sampled to 250Hz, high pass filtered at 1Hz and low
pass filtered at 125Hz. Finally, the data was re-referenced tothe average of all channels including the original reference
channel, the FCz electrode at the forehead.
To reject eye and line noise activity we computed independent component (IC) analysis on a dataset containing thecleanest 85% of the data. The IC is a robust and establishedmethod to separate the scalp EEG signals into independentsources, originating from different brain areas or artifacts(e.g., eye blinks) [47]. To perform the IC analysis, the original data was split into 1 second long epochs, and we calculated for each epoch the (1) mean absolute amplitude of
all channels, (2) standard deviation across all channel mean
amplitudes, and (3) the Mahalanobis distance of all channel
mean amplitudes. Then, we joined the results for each epochof all three methods and ranked all epochs highest to lowest.
We rejected the 15% highest ranking epochs [18].
On this cleaned data, we computed a single-model AMICA [35]. Lastly, we automatically assigned source descriptions to each independent component using the ICLabel
toolbox. We selected eye and line noise components (mean =3.1components, sd = 1.4) if they were assigned a probability
3 https://sccn.ucsd.edu/eeglab/index.php, last accessed 9/9/2018
4 https://sccn.ucsd.edu/wiki/MoBILAB, last accessed 9/9/2018
5 https://osf.io/x7hnm/

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UKhigher than 0.8 to belong to the eye or line noise category
and eliminated them from the data for further processing.
Extracting the ERPs
To obtain the ERPs (shown in Figure 4), we filtered the EEG
data with a 0.2Hz high pass and 35Hz low pass filter. Then
we sliced it between -0.3 seconds to 0.7 seconds around thestimulus onset, i.e. the moment of object selection. To guarantee robust data, we rejected 10% of the noisiest epochs
using the approach described before. We focused our analysis on one electrode, FCz, located on the forehead, which
has been shown in prior research to be the focal point of thisactivity [43].
Furthermore, we automatically extracted the ERP negativity peaks and their latencies by locating the minimum
peak in a 100 to 300 ms time window after object selection,using a 10Hz low pass filter. The time window was derived
from visual inspection of the mean difference ERP wave, see
Figure 1.
5 RESULTS
Our most important finding was an interaction effect of
the level of haptic feedback and feedback congruency on
ERP amplitude at the FCz electrode. As we will present, ourfindings suggest that this effect originates when the participants’ brain processed the mismatch trials. Other potential
candidate confounding effects that could explain this were
controlled by subtracting mismatch from match trials.
ERP results
We observed a strong amplitude modulation occurring at thesame instant as the participants selected the VR object; theseare our ERPs, depicted in Figure 4.
First, in the matched trials, we observed a consistent ERP
shape among participants, with a stereotypical main positive
component that occurred around 200 milliseconds after theobject was selected and matching feedback was provided,
depicted in Figure 4(A). We found that this positive deflection
was most pronounced in the EMS condition.
Secondly, in the mismatch trials (with premature feedback), we observed a consistent ERP shape among participants, but different from the match trials. In fact, as depicted
in Figure 4(B), we observed a negative deflection around
170ms after a participant had selected the object (i.e, theprediction error), and a subsequent positive peak reachinga maximum around 300ms. Similar to the match trials, the
ERPs in the EMS condition were more pronounced than
Visual or Vibro.
To validate our hypothesis we must compare the matchand mismatch ERPs. To compare these, we subtracted themean amplitude of all match trials from the mean amplitudeof all mismatch trial within each participant; this is what we

Page 6

CHI 2019 Paper

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Figure 5: Amplitude and standard error of the mean of theresulting ERPs obtained by subtracting the mean amplitudeof all match trials from the mean amplitude of all mismatchtrials for each participant (in µV ) at the forehead electrodeFCz; for all three feedback conditions (Visual, Vibro andEMS)

suitable approach to detect a visuo-haptic conflict, such as
the one we induced in the mismatch trials.
Figure 4: ERP amplitude (in µV ) and standard error of themean at the forehead electrode FCz across [A] all match trials and [B] mismatch trials in a -300ms to 700ms window
centered at the object selection time, for each of the threefeedback conditions (Visual, Vibro and EMS).

depict in 6. We found that the amplitude of a global minimum
after stimulus onset differs significantly when experiencing
EMS (mean = −6.2µV , sd = 2.1) compared to the Vibro(mean = −4.7µV , sd = 2.4) and Visual (mean = −4.0µV , sd =1.7) conditions (F (2, 30) = 3.31p = .05, η 2 = .18).
Post-hoc, we tested non-parametric pairwise differencesusing Wilcoxon signed-rank tests [49]. Post-hoc comparisonsfor latency and amplitude of the prediction error peaks are
depicted in Figure 6. First, as depicted in Figure 6(B), we
observed no significant differences for the peak latencies over
the three conditions. Secondly, we found that the negativity
peak amplitude was more pronounced while experiencing
EMS compared to Vibro (p = 0.1) as well as significantly
lower compared to Visual (p < 0.05)—this validated our main
hypothesis, showing that the prediction error amplitude is a

Paper 427

Questionnaire and users’ comments
First, as expected, we observed no significant differencesin the level of immersion between conditions for any of
the four IPQ questions we asked; this is likely caused by
the experiment design, which contains randomly presented
unrealistic trials that score very low on immersion.
Second, in the exit interviews, 8 participants voiced (that)they prefer the comfort and experience of the Vibro condition.
Two participants preferred the EMS condition, stating it was“more engaging”. One last participant stated that the visualcondition was the most realistic condition but added “it felt
easier to perform the task in the EMS condition” (likely due
to the extra force feedback that informs of collision).
6 CONTRIBUTION, BENEFITS & LIMITATIONSWith this study, we contributed a new approach to automatically detect conflicts in visuo-haptic sensory integration
based on analyzing event-related brain potentials. We demonstrated in eleven participants using EEG recordings that ourPage 7

CHI 2019 Paper

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

A

B

0.1

* 0.018

250

negative peak latency (ms)negative peak amplitude (µV)

0.0

via ERPs does not require interrupting the user’s immersive
experience to ask questions. (2) The latter will further enable
to conduct background evaluations of the user’s sense of
haptic immersion, enabling new paradigms for user studies
in VR (using implicit measures). (3) ERPs are not subject
to the same degree of introspection as the standard presence questionnaires. (4) This technique can be used as thebuilding-block for VR applications that want to automaticallyadjust to the user’s perception of conflicts, e.g., using ourapproach, an application could automatically adjust collision
detection volumes based on the user’s ERPs.

−2.5

−5.0

200

150

−7.5
100
visualvibro

ems

feedback condition

visualvibro

ems

feedback condition

Figure 6: Negative peak amplitudes (in µV ) and latencies (inms) 100 to 300ms post object selection event in difference
ERPs, see figure 5. Dots represent individual participants.
Uncorrected p-values of pairwise comparisons were computed with non-parametric rank-sum tests.

method is able to correctly detect prematurely given visuohaptic feedback (combinations of visual, vibration and EMS).
In the future, this approach might thus be used in combination with questionnaires such as IPQ or as an alternative
measure that does not require interrupting the user.
We argue in favor of the scalability of our findings to complex environments: Successfully performing intended motor
actions is the most fundamental way we interact with ourenvironment. The ERP negativity paradigm relies on the factthat during well-known tasks (e.g. motor tasks) prediction
errors elicit negative signals when the environment has auseful level of predictability. In fact, ERPs have been shown
to model prediction mismatches in fairly complex setups
such as: robotic arm control, driving, and gaming [3, 10, 40].
Furthermore, current trends in neuroadaptive technologies
emphasize the scalability of these signals beyond simple
tasks [55]. These findings suggest that this effect will be consistent in other, more complex, VR tasks that also require
motor commands such as touching objects.
Implications for the future of VR Research
We believe that this is a first step towards a potential metricfor visuo-haptic immersion based on detecting visuo-haptic
mismatches. If follow up studies replicate the reported patterns of ERP modulation based on sensory mismatch, VR research will benefit four-fold: (1) evaluating haptic immersion

Paper 427

Limitations
As with any system based on EEG, our approach has its inherent shortcomings: (1) ERP data is typically not taken per-trial
but averaged over many trials and thus require high numberof trial repetitions. In addition, (2) high-resolution EEG is
still cumbersome to apply and requires time and expertise.
However, researchers are working towards single-trial ERP
analysis approaches [6], and we do believe that new EEG
systems will be directly embedded in future VR headsets
allowing easy setup and recording of electrophysiologicalsignals 6 with new comfortable electrode types [31], including dry electrodes [53]. These EEG limitations put a cap on
using our approach for quickly iterating on the design of a
haptic VR scene. However, when designers want to develop
and validate haptic immersion in VR scenarios without interrupting the user, our approach could, in the future offer, apotential replacement for questionnaires.
7 CONCLUSIONSIn this paper, we propose a technique that allows us to detect
haptic conflicts in VR, which is based on event-related brain
potentials obtained using EEG during interaction with virtual
objects. We found out in our user study that the early negativity component of the ERP (the prediction error) is more
pronounced during situations with haptic conflicts, such as:
inadequate or delayed haptic feedback, poorly configured
collision detection, etc. This result suggests we can successfully detect haptic conflicts using our proposed technique.
In fact, we found out that, when the number of mismatchedfeedback channels increases, the prediction error increases.
Thus, we believe this is a first step to establish the potentialof ERPs as an indicator for visuo-haptic mismatches in VR.
We discussed the impact of our findings for VR researchand lay out two potential avenues to this future metric: acomplement to the traditional presence questionnaires oran alternative metric that does not require interrupting theuser.
6 http://looxidlabs.com/product/, last accessed on 18/09/2018.

Page 8

CHI 2019 Paper

As for future work, we plan two courses of action, a technical and an experimental angle. First, we plan to explore a realtime implementation of our analysis scripts, which would
enable real-time adaptation of the haptic devices based on
the user’s ERPs (e.g., inspired by recent work in EEG-based
adaptive VR [53]); to achieve this we will explore implementing our scripts into a real-time EEG-based cloud service, suchas intheon7 . Secondly, while we believe our work is a firststep, more research is required to solidify ERPs as a metricfor haptic immersion; for instance, one needs to explore how
sensitive the prediction error is to different sensory channels
beyond vibration and EMS.
Lastly, our work might fuel a new investigation into theuncanny valley of haptics [4]. Specifically, one might investigate at what perceptual threshold does haptic feedback
negatively impact the accuracy of user’s predictions of avirtual environment?
ACKNOWLEDGMENTS
We kindly thank our colleague Jas Brooks at the Universityof Chicago for helping us proofread this work.
REFERENCES[1] S. Allin, Y. Matsuoka, and R. Klatzky. 2002. Measuring just noticeable differences for haptic force feedback: implications for rehabilitation. In Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002. 299–302.
https://doi.org/10.1109/HAPTIC.2002.998972[2] Mortaja AlQassab, Adam Gomes, Maria Karam, David Wang, Zhechen
Du, Orion Bruckman, and Richard Bustos. 2016. The Impact of Tactile Sensations on Virtual Reality Impairment. In Universal Access in
Human-Computer Interaction. Interaction Techniques and Environments(Lecture Notes in Computer Science), Margherita Antona and Constantine Stephanidis (Eds.). Springer International Publishing, 231–240.
[3] Yazmir B. 2018. Neural Correlates of User-initiated Motor Success andFailure - A Brain-Computer Interface Perspective. Neuroscience (2018).
https://doi.org/10.1016/j.neuroscience.2016.10.060[4] Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek, and Ken
Hinckley. 2018. The uncanny valley of haptics. Science Robotics (2018).
https://doi.org/10.1126/scirobotics.aar7010[5] Chris Berka, Daniel J. Levendowski, Michelle N. Lumicao, Alan Yau,
Gene Davis, Vladimir T. Zivkovic, Richard E. Olmstead, Patrice D.
Tremoulet, and Patrick L. Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks.
Aviation, Space, and Environmental Medicine 78, 5 Suppl (May 2007),
B231–244.
[6] Benjamin Blankertz, Steven Lemm, Matthias Treder, Stefan Haufe, andKlaus-Robert Mueller. 2011. Single-trial analysis and classification of
ERP components – A tutorial. NeuroImage 56, 2 (May 2011), 814–825.
https://doi.org/10.1016/j.neuroimage.2010.06.048[7] Christoph W. Borst and Richard A. Volz. 2005. Evaluation of a Haptic
Mixed Reality System for Interactions with a Virtual Control Panel.
Presence: Teleoperators and Virtual Environments 14 (2005), 677–696.
https://doi.org/10.1162/105474605775196562
7 https://intheon.io, last accessed 17/09/2018

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK[8] Frederick P. Brooks. 1999. What’s Real About Virtual Reality? IEEE
Comput. Graph. Appl. 19, 6 (Nov. 1999), 16–27. https://doi.org/10.1109/
38.799723[9] Andres (Andres Alejandre) Calvo. 2017. A body-grounded kinesthetic
haptic device for virtual reality. Thesis. Massachusetts Institute ofTechnology. http://dspace.mit.edu/handle/1721.1/112394[10] R. Chavarriaga, Z. Khaliliardali, L. Gheorghe, I. Iturrate, and J. D.R.
Millán. 2015. EEG-based decoding of error-related brain activity in areal-world driving task. Journal of Neural Engineering (2015). https://doi.org/10.1088/1741-2560/12/6/066028[11] Andy Clark. 2013. Whatever next? Predictive brains, situated agents,
and the future of cognitive science. Behavioral and Brain Sciences 36, 3(June 2013), 181–204. https://doi.org/10.1017/S0140525X12000477[12] Michael G.H Coles, Marten K Scheffers, and Clay B Holroyd. 2001.
Why is there an ERN/Ne on correct trials? Response representations,stimulus-related components, and the theory of error-processing. Biological Psychology 56, 3 (June 2001), 173–189. https://doi.org/10.1016/
S0301-0511(01)00076-X[13] Emanuel Donchin and Michael G.H. Coles. 1988. Is the P300 component
a manifestation of context updating? Behavioral and Brain Sciences(1988). https://doi.org/10.1017/S0140525X00058027 arXiv:1511.04103[14] Katharina Emmerich and Maic Masuch. 2016. The Influence of Virtual
Agents on Player Experience and Performance. In Proceedings of the2016 Annual Symposium on Computer-Human Interaction in Play (CHI
PLAY ’16). ACM, New York, NY, USA, 10–21. https://doi.org/10.1145/
2967934.2968092[15] G. GarcÃŋa-Valle, M. Ferre, J. BreÃśosa, and D. Vargas. 2018. Evaluation of Presence in Virtual Environments: Haptic Vest and UserâĂŹs
Haptic Skills. IEEE Access 6 (2018), 7224–7233. https://doi.org/10.1109/
ACCESS.2017.2782254[16] Ceenu George, Manuel Demmler, and Heinrich Hussmann. 2018. Intelligent Interruptions for IVR: Investigating the Interplay Between
Presence, Workload and Attention. In Extended Abstracts of the 2018
CHI Conference on Human Factors in Computing Systems (CHI EA’18). ACM, New York, NY, USA, Article LBW511, 6 pages. https://doi.org/10.1145/3170427.3188686[17] Mar Gonzalez-Franco and Jaron Lanier. 2017. Model of Illusions andVirtual Reality. Frontiers in Psychology 8 (2017), 1125. https://doi.org/
10.3389/fpsyg.2017.01125[18] Klaus Gramann, Friederike U Hohlefeld, Lukas Gehrke,
and Marius Klug. 2018.
Heading computation in thehumanretrosplenial
complexduringfull-bodyrotation.
bioRxiv (2018).
https://doi.org/10.1101/417972
arXiv:https://www.biorxiv.org/content/early/2018/09/14/417972.full.pdf[19] Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and
Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight
Mechanical Exoskeleton for Motion Capture and Force Feedback in
VR. In Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems (CHI ’16). ACM, New York, NY, USA, 1991–1995.
https://doi.org/10.1145/2858036.2858487[20] Clay B. Holroyd, Kaivon L. PakzadâĂŘVaezi, and Olave E. Krigolson. 2008. The feedback correct-related positivity: Sensitivity of theevent-related brain potential to unexpected positive feedback. Psychophysiology 45, 5 (Sept. 2008), 688–697. https://doi.org/10.1111/j.
1469-8986.2008.00668.x[21] Dhruv Jain, Misha Sra, Jingru Guo, Rodrigo Marques, Raymond Wu,
Justin Chiu, and Chris Schmandt. 2016. Immersive Scuba Diving
Simulator Using Virtual Reality. In Proceedings of the 29th Annual
Symposium on User Interface Software and Technology (UIST ’16). ACM,
New York, NY, USA, 729–739. https://doi.org/10.1145/2984511.2984519[22] Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A
Wearable Haptic Device Towards Usable Virtual Touch Screen. In

Page 9

CHI 2019 Paper

Proceedings of the 2016 CHI Conference on Human Factors in Computing
Systems (CHI ’16). ACM, New York, NY, USA, 3694–3705. https://doi.
org/10.1145/2858036.2858196[23] A. Lehmann, C. Geiger, B. Woldecke, and J. Stocklein. 2009. Design andevaluation of 3D content with wind output. In 2009 IEEE Symposium on
3D User Interfaces. 151–152. https://doi.org/10.1109/3DUI.2009.4811231[24] Yu Liao, Klaus Gramann, Wenfeng Feng, Gedeon O. Deák, and Hong
Li. 2011. This ought to be good: Brain activity accompanying positive
and negative expectations and outcomes. Psychophysiology (2011).
https://doi.org/10.1111/j.1469-8986.2011.01205.x[25] Pedro Lopes and Patrick Baudisch. 2013. Muscle-propelled ForceFeedback: Bringing Force Feedback to Mobile Devices. In Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ACM, New York, NY, USA, 2577–2580. https://doi.org/10.1145/
2470654.2481355[26] Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto:
Simulating Physical Impact by Combining Tactile Stimulation with
Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM
Symposium on User Interface Software & Technology (UIST ’15). ACM,
New York, NY, USA, 11–19. https://doi.org/10.1145/2807442.2807443[27] Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, andPatrick Baudisch. 2017. Providing Haptics to Walls & Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation. In
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 1471–1482.
https://doi.org/10.1145/3025453.3025600[28] Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. 2018.
Adding Force Feedback to Mixed Reality Experiences and Games Using
Electrical Muscle Stimulation. In Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (CHI ’18). ACM, New York, NY,
USA, Article 446, 13 pages. https://doi.org/10.1145/3173574.3174020[29] Steven J. Luck. 2014. An Introduction to the Event-Related Potential
Technique (2nd edition edition ed.). A Bradford Book, Cambridge,
Massachusetts.
[30] Masahiro Mori. 1970. The uncanny valley. Energy 7, 4 (1970), 33–35.
[31] Vadim V. Nikulin, Jewgeni Kegeles, and Gabriel Curio. 2010. Miniaturized electroencephalographic scalp electrode for optimal wearing
comfort. Clinical Neurophysiology: Official Journal of the International
Federation of Clinical Neurophysiology 121, 7 (July 2010), 1007–1014.
https://doi.org/10.1016/j.clinph.2010.02.008[32] Paul L. Nunez and Ramesh Srinivasan. 2006. Electric Fields of the Brain:
The Neurophysics of EEG (2nd ed edition ed.). Oxford University Press,
Oxford ; New York.
[33] Robert Oostenveld and Peter Praamstra. 2001. The five percent electrode system for high-resolution EEG and ERP measurements. Clinical
Neurophysiology 112, 4 (April 2001), 713–719. https://doi.org/10.1016/
S1388-2457(00)00527-7[34] Gonçalo Padrao, Mar Gonzalez-Franco, Maria V. Sanchez-Vives, Mel
Slater, and Antoni Rodriguez-Fornells. 2016. Violating body movement
semantics: Neural signatures of self-generated and external-generated
errors. NeuroImage (2016). https://doi.org/10.1016/j.neuroimage.2015.
08.022 arXiv:1003.4074[35] Jason Palmer, Ken Kreutz-Delgado, and Scott Makeig. 2011. AMICA: An Adaptive Mixture of Independent Component Analyzers with
Shared Components. (01 2011).
[36] John Polich. 2007. Updating P300: An Integrative Theory of P3a andP3b. Clinical neurophysiology : official journal of the International
Federation of Clinical Neurophysiology 118, 10 (Oct. 2007), 2128–2148.
https://doi.org/10.1016/j.clinph.2007.04.019[37] Helena Pongrac. 2008. Vibrotactile perception: examining the coding ofvibrations and the just noticeable difference under various conditions.
Multimedia Systems 13, 4 (Jan. 2008), 297–307. https://doi.org/10.1007/

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

s00530-007-0105-x[38] Priscilla Ramsamy, Adrian Haffegee, Ronan Jamieson, and Vassil
Alexandrov. 2006. Using Haptics to Improve Immersion in Virtual
Environments. In Computational Science – ICCS 2006 (Lecture Notes
in Computer Science), Vassil N. Alexandrov, Geert Dick van Albada,Peter M. A. Sloot, and Jack Dongarra (Eds.). Springer Berlin Heidelberg,
603–609.
[39] H. Reckter, C. Geiger, J. Singer, and S. Streuber. 2009. Iterative design
and test of a multimodal experience. In 2009 IEEE Symposium on 3D
User Interfaces. 99–102. https://doi.org/10.1109/3DUI.2009.4811213[40] Andres F. Salazar-Gomez, Joseph Delpreto, Stephanie Gil, Frank H.
Guenther, and Daniela Rus. 2017. Correcting robot mistakes in real timeusing EEG signals. In Proceedings - IEEE International Conference on
Robotics and Automation. https://doi.org/10.1109/ICRA.2017.7989777[41] F.-A. Savoie, F. ThÃľnault, K. Whittingstall, and P.-M. Bernier. 2018.
Visuomotor Prediction Errors Modulate EEG Activity Over Parietal
Cortex. Scientific Reports 8, 1 (Aug. 2018), 12513. https://doi.org/10.
1038/s41598-018-30609-0[42] Thomas Schubert, Frank Friedmann, and Holger Regenbrecht. 2001.
The Experience of Presence: Factor Analytic Insights. Presence:
Teleoperators and Virtual Environments 10, 3 (June 2001), 266–281.
https://doi.org/10.1162/105474601300343603[43] Avinash Singh, Tim Chen, Yu-Feng Cheng, Jung-Tai King, Li-Wei
Ko, Klaus Gramann, and Chin-Teng Lin. 2018. Visual Appearance
Modulates Prediction Error in Virtual Reality. IEEE Access PP (May2018), 1–1. https://doi.org/10.1109/ACCESS.2018.2832089[44] Mel Slater. 1999. Measuring Presence: A Response to the Witmer
and Singer Presence Questionnaire. Presence: Teleoperators and Virtual Environments 8, 5 (Oct. 1999), 560–565. https://doi.org/10.1162/
105474699566477[45] Mel Slater and Martin Usoh. 1993. Representations Systems, Perceptual
Position, and Presence in Immersive Virtual Environments. Presence 2(Jan. 1993), 221–233. https://doi.org/10.1162/pres.1993.2.3.221[46] Melissa K. Stern and James H. Johnson. 2010. Just Noticeable Difference.
In The Corsini Encyclopedia of Psychology. American Cancer Society,1–2. https://doi.org/10.1002/9780470479216.corpsy0481[47] Yijun Wang and Tzyy ping Jung. 2013. Improving BrainâĂŞComputer
Interfaces Using Independent Component Analysis. In In Towards
Practical Brain-Computer Interfaces. 67–83.
[48] Jan R. Wiersema, Jacob J. van der Meere, and Herbert Roeyers. 2007.
Developmental changes in error monitoring: An event-related potential study. Neuropsychologia (2007). https://doi.org/10.1016/j.
neuropsychologia.2007.01.004[49] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods.
Biometrics Bulletin 1, 6 (1945), 80–83. http://www.jstor.org/stable/
3001968[50] Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in
Virtual Environments: A Presence Questionnaire. Presence: Teleoper.
Virtual Environ. 7, 3 (June 1998), 225–240. https://doi.org/10.1162/
105474698565686[51] Yasuyoshi Yokokohji, Ralph L. Hollis, and Takeo Kanade. 1996. "What
You can See Is What You can Feel." - Development of a Visual/Haptic
Interface to Virtual Environment. In In Proceedings of the IEEE 1996
Virtual Reality Annual International Symposium. 46–53.
[52] Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, DanielAfergan, Remco Chang, and Robert JK Jacob. 2016. Learn Piano with
BACh: An Adaptive Learning Interface That Adjusts Task Difficulty
Based on Brain State. In Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems (CHI ’16). ACM, New York, NY,
USA, 5372–5384. https://doi.org/10.1145/2858036.2858388[53] Thorsten O. Zander, Lena M. Andreessen, Angela Berg, Maurice Bleuel,Juliane Pawlitzki, Lars Zawallich, Laurens R. Krol, and Klaus Gramann.

Page 10

CHI 2019 Paper2017. Evaluation of a Dry EEG System for Application of Passive BrainComputer Interfaces in Autonomous Driving. Frontiers in Human
Neuroscience 11 (Feb. 2017). https://doi.org/10.3389/fnhum.2017.00078[54] Thorsten O. Zander, Christian Kothe, Sabine Jatzev, and Matti Gaertner.
2010. Enhancing Human-Computer Interaction with Input from Active
and Passive Brain-Computer Interfaces. In Brain-Computer Interfaces,
Desney S. Tan and Anton Nijholt (Eds.). Springer London, London,
181–199. https://doi.org/10.1007/978-1-84996-272-8_11[55] Thorsten O. Zander, Laurens R. Krol, Niels P. Birbaumer, and Klaus
Gramann. 2016. Neuroadaptive technology enables implicit cursor

Paper 427

CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

control based on medial prefrontal cortex activity. Proceedings of
the National Academy of Sciences 113, 52 (Dec. 2016), 14898–14903.
https://doi.org/10.1073/pnas.1605155114[56] Yuhang Zhao, Cynthia L. Bennett, Hrvoje Benko, Edward Cutrell,
Christian Holz, Meredith Ringel Morris, and Mike Sinclair. 2018.
Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. In Proceedingsof the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA, Article 116, 14 pages.
https://doi.org/10.1145/3173574.3173690

Page 11