Science of reading behavior:
how and what do you read?
Pascual Martı́nez-Gómez, Chen Chen, Kyohei Tomita
Tadayoshi Hara, Yoshinobu Kano, Akiko Aizawa
Readability Diagnosis

Introduction
Word consumption

Currently, discriminative models for readability prediction:
Text and Personal influence on Cognitive effort

Generative models allow to:r̂ = argmax Pr (r | l1, . . . , lm)

Reader• Marginal MAP (e.g. MAP(ℓ̂ | Lexhigh, rgood))rDocument• Sensitivity analysis

Generative models necessary for cause diagnosis:
eye-movements• Hypothesis testingPr(r, l1, . . . , lm)
Bayesian causal network:• 100.500 words for avg. person on avg. day.

• Technicality• Background knowledge• 35K words per day from text.

• Layout• Native language

Challenges:• Syntactic complexity• Emotional state• Combine linguistic and eye data.

• Semantic consistency, etc.

• Working memory, etc.

• Quantitative analysis for inference.

Image registration for text-gaze alignment
Motivation• Increase recognition accuracy beyond sensor capabilities.
• Variable and systematic errors.

Formalization• Define a linear transformation as:

Optimizing Comma Placement for Chinese Readability

ga,b : (x, y) → (x, y · b + a)

• No assumption on reading behavior.
Raw gaze data when readingwhere a is a translation and b is a scaling.

Motivation• Objective function:
XXf (Ga,b, W) =|Ga,b(i, j)−W(i, j)|• Massive informal writing with wrong punctuations

iMethodology• Segmentation cues influence reading Chinese
CRF model-based Comma Predictor

Experimental Resultsj

Comma Distribution in General Text

Human Annotation+Gaze Features
Text with/withoutCommasFuture work
Comma Distribution for Readability• Theoretical support for rules / personalization

Optimization strategies
Low-resolution gaze data

+ Linguistic Features

Rule-based Comma Filterj• Solution space is not convex.

Raw gaze data

Parse Tree

Treebank(â, b̂) = argmin f (Ga,b, W)a,bXX= argmin|Ga,b(i, j) − W(i, j)|i

CRF model• Gaze data proved that readers benefited frombetter comma distribution (given by our method →)• Optimization problem:

a,b

Input (Comma-less) Text

Text width optimization based on gaze data

Blurred gaze data

Motivation for document layout optimization
Objective function:• More opportunities to read texts on electronic devices.
• Easy-to-read document layout depends on person.

NX• Document layout on electronic device can be personalized.

tk = N ·k=1
Solution space

Text width optimization

Solution space

Solution space∝ h·• Minimize total return sweep time during sequential reading.
• t̄RS(w) can be computed from gaze data on text of width w.

Distance

Distance

k=1 tkNPN

k=1 tk

PNNtk
1
k=1
∝ ·w
N1
∝ · t̄RS(w)w• t̄RS(w) : average return sweep time when reading text of width w.
DistancePN• Estimate t̄RS(wi) for different wi and minimize objective function.
Return sweep is key for optimal text width

Scaling factor

Translation (in pixels)

Scaling factor

Translation (in pixels)

Scaling factor

Translation (in pixels)

Prediction of word fixations/skips by readers

Recognizing Personal Attributes
Combine linguistic features and eye data

Reading act

ReaderDocument

eye-movements• Clues: surface, POS, length, frequency, etc.

Previous label• Prediction with 0.95 similarity to observed data(for distribution across readers)Fixation
Ħ:label• Regardless of individuality / unstableness→ general reading strategy

Sentence
Ħ: input

Eye-movement data in reading text by English natives (Kennedy, 2003)
t h r e a t en i n g●t he i r●

v e r ye x i s t en c e ?

●●skipped“Subjects with high understanding can be distinguishedfrom subjects with low understanding.”●: fixation

Word :POS :Length :Surprisal
Trigram :Screen :Position-2-1

01

2The

manwillTheDT0.07
(2,3)

manNN
0.07
(2,3)

will
havetoMD
VBTO
0.06
0.06
0.08
0.38
0.48
0.61
Features of input sequence
(3,3)
(3,3)
(3,3)

havetoFuture Work and Challenges

Testing linear separability:

Future Work• Real-time text-gaze alignment.
• (Semi-)automatic readability optimization.
• Diagnose causes of understanding.
• Layout optimization.
• Eye data for adaptive interfaces.

NII Contact: Prof. Akiko Aizawa
Digital Content and Media Sciences Research Division: saccade/ skipChallenges• Text-gaze alignment with low-quality sensors.
• Formal framework to estimate cognitive effort using:– Fixation time and locations,– Saccades,– Regressions,– Pupil diameter size changes.

National Institute of Informatics
E-mail: gazeNLP@nii.ac.jp