UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

Towards an Intelligent Textbook: Eye Gaze
Based Attention Extraction on Materials for
Learning and Instruction in Physics
Shoya Ishimaru
German Research Center for
Artificial Intelligence
Kaiserslautern, Germany
Shoya.Ishiamru@dfki.de

Jochen Kuhn
University of Kaiserslautern
Kaiserslautern, Germany
kuhn@physik.uni-kl.de

Syed Saqib Bukhari
German Research Center for
Artificial Intelligence
Kaiserslautern, Germany
Saqib.Bukhari@dfki.de

Andreas Dengel
German Research Center for
Artificial Intelligence
Kaiserslautern, Germany
Andreas.Dengel@dfki.deAbstract
In this paper, we propose the attention extraction method
on a textbook of physics by using eye tracking glasses. We
prepare a document including text and tasks in physics,
and record reading behavior on the document from 6-gradestudents. The result confirms that students pay attention toa different region depending on situations (using the text asa material to learn the content, or using it as hints to solvetasks) and their comprehension.

Author Keywords
Eye tracking; attention; education; document analysis

Carina Heisel
University of Kaiserslautern
Kaiserslautern, Germany
heisel@physik.uni-kl.deACM Classification Keywords
H.5.m [Information interfaces and presentation (e.g., HCI)]:
Miscellaneous

Introduction
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org.
Ubicomp/ISWC’16 Adjunct, September 12–16, 2016, Heidelberg, Germany.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-4462-3/16/09 $15.00.
DOI: http://dx.doi.org/10.1145/2968219.2968566

1041

Reading a textbook is an important way to obtain new knowledge. However, students often avert their eyes from reading a textbook because it is static and boring. One of thesolutions to this problem is to develop a digital textbook
which can make the materials for learning and instruction
adaptive and dynamic on a mobile tablet computer. Especially in physics education, it is highly efficient to show
phenomenon, experiments, representations (such as text,diagram, formulas, etc.), and 3D models as dynamic con-Applications

Definitions Introduction

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY(a) text(b) tasks
Figure 1: A document with textand tasks in physics. These twofigures are in one page on adisplay (text on the left and tasks
on the right) during the experiment.

tents. Furthermore, the combination of a digital textbook
and activity recognition (e.g. recognizing students’ interest,workload, and trouble with understanding) makes it possible
to provide dynamically-generated content individually optimized for each student and context. Our final goal is to create such an "anticipating" textbook and enhance students’learning performances.

well explored in the field of activity recognition and psychology [3]. Concerning reading behavior, there is a research focusing on English skill level estimation from eye
gaze [10]. As the authors employ general features from eye(e.g. the number of blinks, average distance of each eyemovements), their approach can be used on a physics textbook as well.

As an initial analysis, we investigate students’ reading behavior on a textbook in order to find specific patterns which
are related to the context including situations and comprehensions. For this purpose, we prepared a document on"Basic Phenomena in Acoustics" (cf. Figure1) including
both, text and related exercises. In addition, we recorded
their eye gaze during the period when only the text page isshown and subsequently when both, texts and exercises,
are shown on a display.

Therefore we can use preliminary research from Mozaffari
et al. [8], where students’ eye gaze is recorded while solving tasks of physics by using a remote eye tracker mounted
to a tablet computer. The authors have revealed that students prefer different representations (vector, table and diagram) depending on their skill level. We follow their basicidea. But compared to their work, we are interested in students’ natural reading behavior on a textbook. We do not
optimize the text and tasks so much for the recording.

The contributions of this paper are two-fold. 1) We findsome rough relations between levels of students’ expertiseand their reading behavior on a textbook. 2) We recorded
the data from 6-grade students at school. It realized some
limitations to using eye-tracking devices in a realistic educational scenario.

Recently eye tracking technologies became more and more
affordable so that it works on an unmodified tablet computer
without external hardware [1], but also as an inexpensivehead-mounted wearable device [9]. Therefore, in future applications using eye tracking will work on many devices andscenarios, which is not limited to reading a textbooks [4, 6].

Related work

The Attention Extracting Method

There is a whole bunch of research investigating efficientvisualizations and representations to improve students’skills of understanding and solving in Physics Education
Research [7]. However, most of them obtain students’ insights from only answering sheets afterwards and do notcare about their learning process while reading texts andsolving tasks.

We believe that extracting the part where students pay attention is the first step to investigate their reading behavior.
We propose a method to extract attention by using an eye
tracking device. The method consists of three steps.

One of the interesting approaches to get insights into cognitive behavior is employing gaze data. The strong relationship between cognitive states and eye movements is1042(1) Mapping eye gaze coordinates on a document
We utilize mobile eye tracking glasses and record a student’s reading behavior. Because the output of the device
are coordinates of eye gaze on a scene camera, we need tomap them to the document under consideration. As shownin Figure 2, we detect the position of the document in a

SESSION: WAHM

Figure 2: One of the results of gaze mappings. Red circles ofboth scene and document image represents feature points ofSIFT. The documents is extracted as white rectangle on a scene
image. The gaze point on document is estimated as a black circle.

frame of the camera by using SIFT features [5], and calculate a homograph to map the gaze point to the document.
(2) Detecting fixations
The raw gaze data on the document is classified into fixations and saccades. Fixations appear, when the gazepause in certain position - normally lasting between 200
and 400 ms. Saccades are the jumps of the gaze between
two fixations taking 10-20 ms. We apply the fixation-saccade
detection algorithm proposed by Buscher et al. [2]. Figure 3
shows input and output. The radius of each circle corresponds to the fixation, and the line between two circles represents a saccade. Noises and drifts on raw data are alsofiltered in this step.
(3) Calculating the features for each area of attention
We divide a text beforehand based on the roll (e.g. theintroduction, definitions, applications on the document
shown in Figure 1.) We focus on the period of time a student needs to reads the content to obtain knowledge. Thus,
for each area a sum of fixation durations is calculated,which is divided by the size of area to be normalized.

1043(a) raw gaze data(b) ﬁxations and saccades

Figure 3: Eye gaze on the document while a student read atextbook. Colors represents the order of eye gaze (red to blue). (a)is raw data and (b) is an output of fixation-saccade detection

Experimental Setup
We asked 8 participants to wear eye-tracking glasses, toread a physics textbook and to solve respective exercises.
The participants were 6-grade students at a German highschool (around 12 years old). The document we preparedis shown in Figure 1. It consists of three parts: the introduction, itemized definitions, and applications. To analyze thestudents’ natural behavior while acquiring knowledge, we
selected content that they had not yet learned in class.
Only an explanation of about the content (the top page inFigure 1) was displayed at first. After they understood thecontent, they could make tasks appear by pressing thespace-key on a keyboard. They could go back to read thecontent to help them in their solving tasks. In this paper, we
define these two steps as "reading" and "solving."

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

Figure 4: An overview of theexperiment. A participant is solving
questions on a display with wearing
Tobii Pro Glasses 2.
reading

intro.

def.

reading

intro.

def.

reading

intro.

def.

Noviceappl.

Intermediate

appl.

intro.

Expertappl.

Results and Discussion

solving

intro.

def.

appl.

solving

def.

appl.

solving

intro.

def.

Figure 4 shows the overview of the experiment. To evaluate whether our proposed method works with different eye
tracking devices, two types of eye tracking glasses wereused during the experiment. We used Tobii Pro Glasses 2
with five participants (a, b, d, e, f). The glasses record eyegaze at a sampling frequency 100 Hz and a scene video
at 25 Hz. We applied one-point calibration with a marker
before starting each recording. The data of the other threeparticipants (c, g, h) were recorded with SMI Eye Tracking Glasses 2. The glasses record eye gaze at a sampling
frequency 60 Hz and a scene video at 30 Hz. We apply
three-point calibration with this device.

appl.

Figure 5: Histograms of the timestudents paid attentions [%]. Error
bars represent standard deviations.

Table 1 shows the percentages of time students paid attentions for the introduction, definitions, and the applications
on the document. We calculated the percentage dependingon the each situation while reading text and solving tasks.
Note that the data in Table 1 is sorted by the number of correct answers. We categorized 8 participants to 3 comprehension levels according to the score: novice (the score is 4or less), intermediate (the score is 5), and expert (the scoreis 6 or more).
By calculating mean values for each comprehension levelas shown in Figure 5, it has become obvious that students
with high level comprehension do not pay attention to theapplications part while both reading and solving tasks compared to other levels. They understand that the applications
part is useful for understanding the content, yet there is
not much information that can be used as hints for solving
tasks. They preferred to read definitions part because there
are direct hints (principles, formulas, etc.). Intermediates
and novices spend much time to paying attention to the application part while both reading and solving.

1044

Table 1: Percentages of time students paid attentions
P.

Score

Expertise

while reading text [%]while solving tasks [%]Intro.

Def.

Appl.

Intro.

Def.

Appl.

a3

Novice

1449

37

1359

28

b4

Novice

1743

40

1748

35

c

5

Intermediate

751

42

444

52

d5

Intermediate

31

41

28

214930e5Intermediate

23

37

40

2740

33

f

6

Expert

1647

37

1260

28g

7

Expert3450

16255619h7Expert2864822708(out of 14)

While percentages of attention in the reading and solving
phase are similar for all levels, their reading behavior isdeferent for each situation. Figure 6 represents fixationbased heat maps from novice (Participant b), intermediate(Participant e), and expert (Participant h) for both, readingand solving. High-attention part during reading is almost
same for all three skill levels, while the participants spend
much time to read left side of the definitions part. Note that
novices and intermediates look graphics carefully whilesolving, and this is one of the reasons their attention forthis application is higher than for experts.
Compared to a remote eye tracker, mobile eye tracking
glasses have the advantage that the device can record eye
gaze not only on display but everything on a scene camera.
However the big issue during this experiment was that students sometimes touch glasses unconsciously, and lose an
accuracy of eye tracking adjusted during calibration. Therefore using remote eye tracker might be better than mobile
glasses for young participants.

Conclusion and Future WorkIn this paper, we present an initial method to extract students’ attention by using gaze data. By applying the ap-SESSION: WAHM

proach to activities including reading a text and solving
tasks, it is revealed that reading behavior is related to students’ comprehension. Especially experts do not pay attention where to find specific hints to answer questions. Oneof the future work is a detailed analysis of reading behavior
focusing not only on the total duration on separated parts(texts and exercises) but also time-series information (e.g.
re-reading contents, transition between representations).

Novice(a) reading(b) solving

Intermediate

Acknowledgement
This work is funded by the Federal Ministry of Educationand Research (BMBF) in the framework of the joint initiative"Qualitätsoffensive Lehrerbildung" of the Federal and theFederal States of Germany for the project "U.EDU: Unified
Education" (support code: 01JA1616). The authors areresponsible for the content of this contribution.

REFERENCES(c) reading(d) solving
Expert(e) reading

1. Oliver Amft, Florian Wahl, Shoya Ishimaru, and KaiKunze. 2015. Making regular eyeglasses smart.
Pervasive Computing, IEEE 14, 3 (2015), 32–43.
2. Georg Buscher, Andreas Dengel, and Ludger van Elst.
2008. Eye movements as implicit relevance feedback.
In CHI’08 extended abstracts on Human factors incomputing systems. ACM, 2991–2996.

(f) solving

Figure 6: Fixation duration based
heatmaps while a student isreading the text and solving tasks.

3. Reinhold Kliegl, Antje Nuthmann, and Ralf Engbert.
2006. Tracking the mind during reading: the influenceof past, present, and future words on fixation durations.
Journal of experimental psychology: General 135, 1(2006), 12.
4. Jochen Kuhn, Paul Lukowicz, Michael Hirth, Andreas
Poxrucker, Jens Weppner, and Junaid Younas. 2016.
gPhysics–Using Smart Glasses for Head-Centered,Context-Aware Learning in Physics Experiments.

1045

Transactions on Learning Technologies (2016).
accepted.
5. David G Lowe. 1999. Object recognition from local
scale-invariant features. In Proceedings of the seventhIEEE international conference on Computer vision,Vol. 2. Ieee, 1150–1157.
6. Paul Lukowicz, Andreas Poxrucker, Jens Weppner, B
Bischke, Jochen Kuhn, and Michael Hirth. 2015.
Glass-physics: using google glass to support highschool physics experiments. In Proceedings of the2015 ACM International Symposium on Wearable
Computers. ACM, 151–154.
7. David E Meltzer. 2005. Relation between students’problem-solving performance and representational
format. American Journal of Physics 73, 5 (2005),463–478.
8. Saleh Mozaffari, Saqib Bukhari, Andreas Dengel,Pascal Klein, and Jochen Kuhn. 2016. A Study on
Representational Competence in Physics Using Mobile
Remote Eye Tracking Systems, Proceedings. In
Proceedings of the 18th International Conference on
Human-Computer Interaction with Mobile Devices andServices. accepted.
9. Erroll Wood and Andreas Bulling. 2014. Eyetab:
Model-based gaze estimation on unmodified tablet
computers. In Proceedings of the Symposium on Eye
Tracking Research and Applications. ACM, 207–210.
10. Kazuyo Yoshimura, Kai Kunze, and Koichi Kise. 2015.
The Eye as the Window of the Language Ability:
Estimation of English Skills by Analyzing Eye
Movement While Reading Documents. In Proceedingsof the 13th International Conference on Document
Analysis and Recognition. 251–255.