Late-Breaking Work: Designing Interactive Systems#chi4good, CHI 2016, San Jose, CA, USA

Embodied Reading:
A Multisensory Experience
Susana Sanchez
Keio University, Japansu saps@hotmail.com

Tilman Dingler
Stuttgart University, Germany
tilman.dingler@vis.uni-stuttgart.deHeng Gu
Keio University, Japanheng.gu@network.rca.ac.uk

Kai Kunze
Keio University, Japan
kai.kunze@gmail.com

Abstract
Reading fiction is a silent activity, where readers cometo know imaginary worlds and characters from the book’s
pages. However, we perceive the natural world with morethan our eyes, and literature should be no different. Thus,
an embodied reading experience is proposed, adding soundeffects and haptic feedback to allow readers to listen andfeel the narrative text. This paper presents a preliminary
prototype for multisensory narratives and an experimental
methodology to measure embodiment in literature. Results for the subjective assessment of immersion and user
experience from 15 participants in three modalities: haptic,
sound, both combined are discussed.

Author Keywords
Embodied cognition; haptics; sound effects; reading.

ACM Classification Keywords
H.5 Information Interfaces and presentation [H.5.2 User Interfaces]: Haptics; Auditory (non-speech feedback.)
Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHI’16 Extended Abstracts, May 07-12, 2016, San Jose, CA, USA
ACM 978-1-4503-4082-3/16/05.
http://dx.doi.org/10.1145/2851581.2892353

Introduction
The study of literature is a scholarship that corresponds
to literary theory. Different schools focus on different aspects of literature to discuss and write about writing. One
school is the reader-response criticism that views literature
as a performing art where readers create their own text-

1459

Late-Breaking Work: Designing Interactive Systems

related performance. However, literature does not allow
for an embodied experience. The phenomenologist Schütz
argues that literature provides us with a ‘monothetic’ meaning, since it relies on idealizable and objectifiable semantic
content that makes it time-transcendent, sign-oriented, andconceptual [30]. To truly conceive reading as a performing art as the reader-response criticism claims, we needto look for a ‘polythetic’ meaning. For instance, theatrical performances are a form of storytelling that do not
rely on semantic content. Folkerth finds that in theatre,
meaning is time-immanent, fully perceived in an embodied experience of the moment, with its stage, sound effects
and lighting [13]. In this way, plays transmit a polythetic
meaning, allowing audiences to experience within the embodied consciousness and not as an object of consciousness.
Nonetheless, we find that literature, as with theatre or music, is consumed throughout time. In order to perceive thefullness of the narrative text, one must follow its linearity.
As such, literature shares some characteristics with otherforms of narrative that allow for an embodied experience,but it is not yet capable of providing a polythetic meaning, as print has made this form of narrative silent and only
perceivable by the eye.
This paper proposes the use of sound effects and haptic
feedback as literary tools to convey an embodied experienceof the written word, presenting (1) the initial exploration
to create an embodied reading experience using sound andhaptic feedback and (2) an experimental setup to measureembodiment in a multisensory experience.

Embodied Reading
Cognition is embodied when it is dependent upon the physical body, when aspects beyond the brain play a significant
role in cognitive processing [10]. To achieve an embodied
experience we designed narrative texts with the addition#chi4good, CHI 2016, San Jose, CA, USAof sound effects and haptic feedback as multisensory interventions. The multisensory feedback was designed toprovide the reader with a second and third channel throughwhich to perceive and process the contents of the literary
narrative. The design process consisted on testing a mix ofauditory and haptic information to provide simultaneous information that supported the narrative text. Bertelson calls
this simultaneous information valid co-occurrences, where
the sensory information coming from one event is perceived
by different sensory systems [3]. In a multisensory reading
experience these co-occurrences could influence the interpretation of read actions by adding information available inother senses, resulting in an intermodal redundancy. During these co-occurrences, the sonic and haptic cues could
operate in what Tribble calls a cognitive ecology of multidimensional content, related to the reader-response criticism,
where literature can serve to remember, hear, feel, think,
and imagine. [31]We worked with the author Carla Novi, redesigning four ofher original short stories, making sure to select the appropriate haptic and sonic cues that supported the narration.
This was an artistic process, comparable to sound designfor theatre, and in line with the creative process of writing
fiction. For instance, some multisensory information was
designed to be inferential. In one of the four short stories,
for example, rain is not described in the text, but the author agreed to create an embodied experience with haptic
and sonic cues to support the emotional stance of the main
character, who is running for his life. In urban settings rain
is often used to symbolize misery and discomfort, and tocreate an embodied experience of this discomfort a tactile
perception of impulse, pops or clicks, was used to create
the illusion of short bursts that resembled drops of water,
mixed with sound recordings of a rainy day.

1460

Late-Breaking Work: Designing Interactive Systems

Haptics
A wide variety of haptic effects can be achieved using vibration patterns to simulate sensations and textures using
sound. The TECHTILE Toolkit allowed us to use soundfiles to simulate sensations and textures, using a surface
transducer [28]. This technology was incorporated into an
add-on to augment an e-Reader [15]. Besides the surface
transducer the add-on also incorporated a small speaker.
Beyond simulating existing textures, haptic stimulation canalso create compelling tactile experiences that do not naturally occur. Special haptic effects have great potential
applications as literary tools in certain genres of fiction.
For instance, authors write verbal cues to their readers toinfer certain situations and emotions from the text, such asa “blush” or a “sweaty hand”, to tell their readers a certain
character is in love, without writing it explicitly. Insteadof using textual cues, for example, readers can infer theemotion of love, if they could feel the heart palpitations of
the named character. This cross-modal cue could provide
an embodied experience of a positive emotion, where thereader can actually feel the body of the fictional character.
Sound Effects
We perceive our surroundings not only with our eyes, butalso our ears. Sound design is used to create artificial ambience, allowing audiences an embodied experience. The
quality of sound to convey information and emotion is wellexemplified in movies. Moreover, Lopez, a major in medieval theatre, is carrying out a pilot study to replace audio
description in films for the blind with acoustical information [20]. Focused on sound effects, surround sound, andsound layering, she has developed an audio film that can befollowed by a visually impaired audience, without the needof a narration track.

#chi4good, CHI 2016, San Jose, CA, USA

The field of ecological psychology proposes that the physical nature of a sounding object, the means by which it
has been set into vibration, and the function it serves tothe listener, is perceived directly without any intermediate processing [27]. Even though this approach seems tocontradict information processing, where recognition is amultistage process between perceptual qualities of soundsource, abstract representation in memory, meanings, andassociations, the ecological approach is concerned with theinvariant properties of the sound [26]. For instance, one isable to recognize someone’s voice even if this person has aterrible cold. Thus, the ecological approach might involvean understanding of why sound effects, even when they aresynthetic, can produce a natural perception.
Multisensory Book
Four original short stories were supplemented with haptic feedback and sound effects to design the multisensory
book.The haptic feedback and sound effects trigger whenthe eye gaze of the reader is over the text, marked as span
region where the multisensory feedback was embedded. To
do this we used the Gaze Track Plugin for the Processing
IDE developed by Augusto Esteves, along with an off the
shelf eye tracker mounted on the e-Reader (See Figure 1).
The reader’s gaze acts as the input of location in the text totrigger the multisensory feedback [5]. The Processing applet pulls prerecorded .wav files from a library, which were
specifically designed for the haptic or audio channels. Subsequently, the signals are split between two stereo channels: left channel for .wav files for haptic feedback, andright channel for .wav files for sound effects. The surface
transducer and speaker in the add-on, which allow the multisensory book to be possible, are connected to the e-Reader
through a stereo jack that splits the output into vibration(left) and sound (right).

1461

Late-Breaking Work: Designing Interactive Systems#chi4good, CHI 2016, San Jose, CA, USA

Figure 1: Augmented e-Reader with eye tracker, surface
transducer, and speaker.

Evaluation
The multisensory reading experience was evaluated to detect if the haptic feedback and sound effects created anembodied experience. This was done with an immersion
questionnaire and a user experience questionnaire. The experimental methodology was based on interoception, as it
links cognitive and affective mechanisms to embodiment[8]. The insula, a part of the brain within the interoceptive network balances cognitive, sensory and behavioral
processes [16, 9]. Emotion is represented in the anterior
insula, which deals with all subjective feelings, and constitutes the ‘sentient self’ in a ‘global emotion of the moment’[11]. Moreover, interoception awareness requires selective
attention to a particular stimulus in order to experience it[12, 21]. Attention is an important factor in immersion,
thoroughly studied in gaming, where the demand for attention is greater in immersive experiences [7, 19].

Figure 2: Experimental setup using the augmented e-Reader
for an embodied reading experience.

Procedure
For the evaluation 15 participants (7 women, 22-42 years)
were asked to read four short stories, one story for eachcondition. The short stories were assigned in a latin square
design, so none of the participants would read the same
short story in the same condition and sequence. Each participant was provided with the e-Reader augmented withthe transducer and speaker add-on, and eye tracker (SeeFigure 2). The eye-tracker had to be calibrated for eachparticipant once. There was an initial explanation to allparticipants about the procedure and embodied reading
experience. They were always reminded of which condition they would experience next. After reading each shortstory participants completed the immersion and UX questionnaires. After reading the four short stories participants
were debriefed.

1462

Late-Breaking Work: Designing Interactive Systems

Materials
Four short stories, of approximately 1,000 words each, weredivided into four conditions: 1) Normal reading; 2) Soundeffects; 3) Haptic feedback; 4) Sound effects and haptic
feedback combined. The first condition was used as baseline, while the second and third were designed with only one
extra sensory modality, sonic or tactile. The last condition
was designed for a richer multisensory reading experience.
The conditions with extra sensory modalities had between
two to four embedded interventions, either of sound, tactile
sensations or a mix of both. These stories were presented
on the augmented e-Reader with the surface transducer,
speaker, and eye tracker. The immersion questionnaire wasapplied following the methodology of Kunze et al. andJennet [24, 19]. Ten statements were taken from the IEQ
and modified to measure reading tasks, each statement wasevaluated on a scale from one to three for levels of: empathy, interest, imagery, attention, enjoyment, frustration,
time, difficulty, boredom, and disinterest. In the same survey Hassenzahl’s bipolar verbal scale anchor was added,
using fifteen anchors to evaluate, with positive or negative
adjectives user experience.
Results
Due to technical problems we had to exclude the data fromtwo participants. A One Way ANOVA test between conditions was conducted for immersion (F = 3.16 >Fcrit=2.79), and UX (F = 3.04 >Fcrit =2.79). There wasa significant effect between groups in immersion levels as[F (3, 48) = 3.17, p = 0.033], and UX [F (3, 48) = 3.04,
p = 0.038]. Post hoc comparisons using Tukey HSD test
indicated levels of immersion baseline condition (M = 0.46,
VAR =15.93) were significantly different from sound (M=5.08,
VAR=10.7), haptic (M=3.46, VAR=16.10) and sonic andhaptic (M=2.23, VAR =19.69) and UX baseline condition (M = 2.76, VAR =3.36) was significantly different#chi4good, CHI 2016, San Jose, CA, USAfrom sound (M=4.23, VAR=1.69) and haptic (M=4.23,
VAR=1.02) but not for sonic and haptic (M=2.92, VAR=2.91).
Discussion
The UX results indicate that the mix of haptic and sonicfeedback was more disruptive to the reading experience
than using only sound or tactile cues. We contend that thedisruptions of sound and haptic feedback can act as literary
tools that, if properly managed, can prove to be more empowering to the story than what was interrupted (Moreiras,
2001). However, the disruption seems to work better if
only done through one more channel instead of two. Thereis also an indication that sound results in higher immersion than haptic or the mix of sound and haptic feedback.
This might be related to the ability of sound to carry nonverbal information, contributing for a significant experienceby structuring perception when associated with images andmeanings. There was in indications of this in the debriefing
sessions, where some participants reported that even when
they could not immediately relate the sound to the text,
as they progressed in the story they could remember andassociate those sounds to what was read before. We suggest this is because both sound and literature are consumed
across time, to understand a piece of music or a narrative
text one needs to link all available information. Finally, if
the cues were triggered before the participant was reading
through the passage, because of erratic eye movement, theasynchronous cue caused expectation, where some readers
tried to read quicker to know what the sound meant.

Reading Behavior
The results presented in this work-in-progress are from thesubjective assessment of immersion and user experience.
However, this work does not finish here, as during the experiment the participants’ eye gaze was recorded using the1463

Late-Breaking Work: Designing Interactive Systems

eye tracker. Studies suggest engaged reading shows a stable progression through lines and paragraphs, whereas in
non-engaged reading gaze, is chaotic [24]. This pattern is
consistent with previous studies using garden paths, grammatically correct sentences that start in such a way thatthe reader’s interpretation turns out to be incorrect. Thesetypes of sentences show long eye gaze regressions to the beginning of the sentence, or ambiguous regions in the text,
to correctly extract meaning [14]. It is widely assumed thatas a text gets more difficult, readers make longer fixations,shorter saccades and more regressions. For instance, therate of regression is higher in young children, poor readers,and dyslexic readers [29].

#chi4good, CHI 2016, San Jose, CA, USAstorytelling, to note taking, and digital paper augmentations [6, 2, 32]. Very close to our approach is te work
done by Yannier et al., who used haptic feedback to improve children’s reading by making stories more memorable.
They focused on young children, early reading, and on improving the learning rate [33]. On the other hand Biedert
et al. studied ways to enhance reading experience. The
EyeBook [4], which uses the Text 2.0 framework [5], is areading interface that detects the area being read, generating sounds. However, until now, we have not found work
evaluated between suitable interventions to produce an embodied experience.

Conclusion
Additionally, participants were asked to wear a prototype of
J!NS MEME smart eyewear. With this wearable device weretrieved data of: fit status; eye movement up, down, left,
and right; blink strength and speed; roll; head movementin x, y, and z. The smart eyewear integrates Electrooculography, electrodes to detect eye movement, along with anaccelerometer and gyroscope to monitor head movement[17, 1]. In fact, the smart eyewear has been used to monitor reading habits [23, 22] and behaviors [18, 25]. With
data from the eye tracker and smart eyewear we will continue exploring applications for interactive reading, such as
embodiment. Future work consists in the data analysis ofeye activity reading in the four conditions: normal reading,sound, haptic, sound and haptic, in order to understandhow the experimental embodied reading experience affects
reading behavior. Eye gaze recordings will be comparedwith the data collected from the smart eyewear to find new
possibilities for reading interactions.

Related Work
The Human Computer Interaction community has put agreat amount of effort to enhance reading, from interactive

In this paper, we presented an experimental approach to anembodied experience of literature, introduced a prototype
for an augmented e-Reader for multisensory narratives, andan experimental methodology to measure embodiment inreading tasks. So far, we have taken the first steps to
explore how embodiment could enrich written language,
where mind and body perceive the narrative. We believe
the multisensory approach could lead to a more memorable
reading experience, transporting readers to fictional worlds
they can physically experience.

Acknowledgments
We thank Carla Novi, the author of the short stories , theparticipants who took part in the experiment, and the CHI
reviewers.

References[1] Oliver Amft, Florian Wahl, Shoya Ishimaru, and Kai
Kunze. 2015. Making regular eyeglasses smart. Pervasive Computing, IEEE 14, 3 (2015), 32–43.
[2] Maribeth Back, Jonathan Cohen, Rich Gold, Steve
Harrison, and Scott Minneman. 2001. Listen Reader:1464

Late-Breaking Work: Designing Interactive Systems

An Electronically Augmented Paper-based Book. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’01). ACM, New
York, NY, USA, 23–29. DOI:http://dx.doi.org/10.
1145/365024.365031[3] P Bertelson and B de Gelder. 2003. The psychology ofmultimodal perception. Crossmodal space and crossmodal attention (2003).
[4] Ralf Biedert, Georg Buscher, and Andreas Dengel.
2010a. The eyebook–using eye tracking to enhance
the reading experience. Informatik-Spektrum 33, 3(2010), 272–281.
[5] Ralf Biedert, Georg Buscher, Sven Schwarz, ManuelMöller, Andreas Dengel, and Thomas Lottermann.
2010b. The text 2.0 framework: writing web-basedgaze-controlled realtime applications quickly and easily. In Proceedings of the 2010 workshop on Eye gazein intelligent human machine interaction. ACM, 114–
117.
[6] Mark Billinghurst, Hirkazu Kato, and Ivan Poupyrev.
2001. The MagicBook&#151Moving Seamlessly Between Reality and Virtuality. IEEE Comput. Graph.
Appl. 21, 3 (May 2001), 6–8. DOI:http://dx.doi.
org/10.1109/38.920621[7] Emily Brown and Paul Cairns. 2004. A grounded investigation of game immersion. In CHI’04 extended abstracts on Human factors in computing systems. ACM,
1297–1300.
[8] Nitasha Buldeo. 2015. Interoception: A Measure of
Embodiment or Attention? International Body Psychotherapy Journal 14, 1 (2015).
[9] Oliver G Cameron. 2001. Interoception: the insidestory—a model for psychosomatic processes. Psychosomatic medicine 63, 5 (2001), 697–710.
[10] Andy Clark. 2008. Supersizing the mind: Embodiment, action, and cognitive extension: Embodiment,#chi4good, CHI 2016, San Jose, CA, USAaction, and cognitive extension. Oxford University
Press.
[11] Arthur D Craig. 2009. How do you feel—now? Theanterior insula and human awareness. (2009).
[12] Hugo D Critchley, Stefan Wiens, Pia Rotshtein, ArneÖhman, and Raymond J Dolan. 2004. Neural systems supporting interoceptive awareness. Nature neuroscience 7, 2 (2004), 189–195.
[13] Wes Folkerth. 2014. The sound of Shakespeare. routledge.
[14] Lyn Frazier and Keith Rayner. 1982. Making andcorrecting errors during sentence comprehension: Eyemovements in the analysis of structurally ambiguous
sentences. Cognitive psychology 14, 2 (1982), 178–
210.
[15] Kunze Inami Heng, Sanchez. 2015. An Augmented
e-Reader for Multimodal Literacy. Demo for the 13th
Annual International Conference on Mobile Systems,
Applications, and Services (2015).
[16] Beate M Herbert and Olga Pollatos. 2012. The body inthe mind: on the relationship between interoception
and embodiment. Topics in cognitive science 4, 4(2012), 692–704.
[17] Shoya Ishimaru, Kai Kunze, Katsuma Tanaka, Yuji
Uema, Koichi Kise, and Masahiko Inami. 2015. Smart
Eyewear for Interaction and Activity Recognition. In
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing
Systems. ACM, 307–310.
[18] Shoya Ishimaru, Yuji Uema, Kai Kunze, Koichi Kise,
Katsuma Tanaka, and Masahiko Inami. 2014. Smarter
eyewear: using commercial EOG glasses for activity
recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous
Computing: Adjunct Publication. ACM, 239–242.

1465

Late-Breaking Work: Designing Interactive Systems[19] Charlene Jennett, Anna L Cox, Paul Cairns, Samira
Dhoparee, Andrew Epps, Tim Tijs, and Alison Walton. 2008. Measuring and defining the experience of
immersion in games. International journal of humancomputer studies 66, 9 (2008), 641–661.
[20] Mariana Julieta Lopez and Sandra Pauletto. 2009.
The design of an audio film for the visually impaired.
(2009).
[21] Mika Koivisto, Pasi Kainulainen, and Antti Revonsuo.
2009. The relationship between awareness and attention: evidence from ERP responses. Neuropsychologia
47, 13 (2009), 2891–2899.
[22] Kai Kunze, Masai Katsutoshi, Yuji Uema, andMasahiko Inami. 2015a. How much do you read?:counting the number of words a user reads using electrooculography. In Proceedings of the 6th Augmented
Human International Conference. ACM, 125–128.
[23] Kai Kunze, Katsutoshi Masai, Masahiko Inami, Ömer
Sacakli, Marcus Liwicki, Andreas Dengel, Shoya Ishimaru, and Koichi Kise. 2015b. Quantifying reading
habits: counting how many words you read. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM,
87–96.
[24] Kai Kunze, Susana Sanchez, Tilman Dingler, Olivier
Augereau, Koichi Kise, Masahiko Inami, and Terada Tsutomu. 2015c. The augmented narrative: toward estimating reader engagement. In Proceedings of
the 6th Augmented Human International Conference.
ACM, 163–164.
[25] Kai Kunze, Katsuma Tanaka, Shoya Ishimaru, Yuji
Uema, Koichi Kise, and Masahiko Inami. 2015d.
MEME: eye wear computing to explore human behavior. In Proceedings of the 2015 ACM International
Joint Conference on Pervasive and Ubiquitous Com-#chi4good, CHI 2016, San Jose, CA, USA

puting and Proceedings of the 2015 ACM International
Symposium on Wearable Computers. ACM, 361–363.
[26] Peter H Lindsay and Donald A Norman. 2013. Human
information processing: An introduction to psychology. Academic Press.
[27] Claire F Michaels and Claudia Carello. 1981. Directperception. Prentice-Hall Englewood Cliffs, NJ.
[28] Kouta Minamizawa, Yasuaki Kakehi, Masashi
Nakatani, Soichiro Mihara, and Susumu Tachi. 2012.
TECHTILE toolkit: a prototyping tool for design andeducation of haptic media. In Proceedings of the 2012
Virtual Reality International Conference. ACM, 26.
[29] Keith Rayner, Kathryn H Chace, Timothy J Slattery,and Jane Ashby. 2006. Eye movements as reflectionsof comprehension processes in reading. Scientific Studies of Reading 10, 3 (2006), 241–255.
[30] Alfred Schütz. 1951. Making music together: A study
in social relationship. Social research (1951), 76–97.
[31] Evelyn Tribble, John Sutton, and others. 2011. Cognitive ecology as a framework for Shakespearean studies.
Shakespeare Studies 39, 94 (2011), 10.
[32] Hiroki Yamada. 2010. SequenceBook: Interactive Paper Book Capable of Changing the Storylines by Shuffling Pages. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’10).
ACM, New York, NY, USA, 4375–4380. DOI:http://dx.doi.org/10.1145/1753846.1754156[33] Nesra Yannier, Ali Israr, Jill Fain Lehman, andRoberta L. Klatzky. 2015. FeelSleeve: Haptic Feedback to Enhance Early Reading. In Proceedings of the
33rd Annual ACM Conference on Human Factors in
Computing Systems (CHI ’15). ACM, New York, NY,
USA, 1015–1024. DOI:http://dx.doi.org/10.1145/
2702123.2702396

1466