The Reading-life Log − Technologies to
Recognize Texts That We Read
Takashi Kimura∗ , Rong Huang∗, Seiichi Uchida∗, Masakazu Iwamura† , Shinichiro Omachi‡ and Koichi Kise†∗ Kyushu University, Fukuoka, Japan† Osaka Prefecture University, Osaka, Japan‡ Tohoku University, Sendai, Japan
Abstract—Reading life log is a type of techniques to automatically and unconsciously record people’s reading intentions,interests and habits. Besides, it can also serve as various assistantsin our daily life. In this paper, a reading-life log system isimplemented by a head-mounted and unobtrusive video camera
with a high resolution and a high shutter speed. We utilize DP
matching, and propose a text-based frame mosaicing method
to integrate multiple frames in a clip. The developed systemis tested in the various environments indoor and outdoor. The
experimental results show that our system can provide reliable
outputs with respect to the most correct responses. The infrequentmisregistration between lines also indicates the feasibility andvalidity of the text-based frame mosaicing.

I. I NTRODUCTION
Texts expressing the explicit information surround us, andreading is one of the most essential activities in our daily
life. For example, we read books, documents, posters, menus,
goods packages, web pages, video captions, signboards, etc.,
for getting various types of knowledge and information fromthem. As one can imagine, we will be in a tight corner without
those texts. On the other hand, nowadays, the ubiquitous minielectronic devices with high resolution and high shutter-speed
enable us record the texts appearing in our daily life. Since thetext collection is conducted by first-person vision, it is possible
to analyze the personal intentions, interests and habits.
This paper develops a practical reading-life log system,
which enables us acquire the text information automatically byusing OCR technologies. Reading-life log, as the terminology
itself implies, is a type of life-log systems, which try to capture
various types of information automatically and unconsciously
from some sensor. Our reading-life log is specialized for acquiring text information from a head-mounted and unobtrusive
video camera. The developed device can work under various
realistic environments such as book, magazine, newspaper,
signboards in the wild world, LCD (Liquid Crystal Display)screen.
II. OVERVIEW OF R EADING -L IFE L OG
Reading-life log has initiated a lot of interests in the
document analysis community due to the additional attention
information. Campbell et al. [1] aimed at tracking the texts
which an user was reading, and further analyzed his/herinterests or goals by exploring the eye movement information.
Kienzle et al. [2] invented a nonparametric visual saliency
model based on the human eye movement data. Bulling et al.
[3] utilized a wearable electrooculography to detect the reading
activity under a variety of indoor and outdoor situations.
Buscher et al. [4] developed a precise user-oriented documentclassifier by introducing the personalized attention information.

Xu et al. [5] devised a document summarization algorithm
by taking the time consumption estimation of reading eachword as a clue. Judd et al. [6] considered the top-down
image semantics and collected eye tracking data for learning asaliency model. Nyström et al. [7] proposed a velocity-basedalgorithm which could robustly identify fixations, saccades andglissades in the eye tracking data for event detection. Loboda
et al. [8] suggested using eye movement information to infer
the word relevance. Carlos et al. [9] designed a mobile headmounted, and employed the Maximal Stable Extremal Regions(MSERs) algorithm for scene text segmentation. Duggan etal. [10] investigated the eye movement data, and verified
the fact that readers have the ability to explicitly catch themost important information in which they were interested.
Dimigen et al. [11] sampled electroencephalogram (EEG)and eye movement signals when subjects were conducting
text reading task, and demonstrated the feasibility of the
combination of EEG and eye tracking data. Buscher et al.
[12] revealed the relations between eye movement measuresand user-perceived relevance for personal information retrieval
application. To distinguish the reading or skimming behavior
in the eye tracking data, Biedert et al. [13] extracted average
forward speed and angularity features to train a classifier. A
fresh work submitted by Kai et al. [14] attempted to measurebrain electrical activity by an off-the-shelf EEG device for
distinguishing genres of documents.
Through the above survey, we find that most existing
works focus on the eye tracking data while seldom employ
the practical recognized data to refine the final results. The
proposed reading-life log system fulfills text-based mosaicing
with two-dimensional alignment of recognized data at twoconsecutive frames. A detailed system outline is given in thenext section.
III. S YSTEM O UTLINE
The proposed reading-life log system is simply realized
by a head-mounted and unobtrusive video camera connected
to a laptop via USB 3.0. Figure 1 shows a realization of thedevice. This commercial video camera is characterized by ahigh resolution and high shutter-speed. The high resolution isnecessary to sample the target text lines with high quality. For
our implementation of the reading-life log system, at least,
so-called “HDTV resolution” is necessary to capture a partof a document containing multiple text lines. Moreover, since
the subject who equips the video camera may introduce theunwanted motion blur which seriously degrades the subsequent
text recognition performance, the high shutter-speed is also
necessary to avoid this type of performance attenuation.

Fig. 1: The appearance of the video camera.
The camera should be equipped aiming at the target textlines, namely the direction in parallel with the sight line, inorder to acquire what the user reads. In this sense, our readinglife log implementation coincides with the “first-person vision”
systems so that the user activity can be reflected throughthe video images from a head-mounted camera. There exist
numerous research activities on scene text detection and recognition like [15–19], which generally target on all characters
appearing in the wild world. Different from them, the proposedsystem exactly focuses on the specific regions in which anuser is interested. The “first-person vision” fashion not only
performs practically in the sense of well gearing towards theuser’s attention, but also eases the subsequent detection andrecognition task by providing the prior knowledge of location.
At the early stage of the authors’ trial, a glass-type mobile
eye-tracker was used for acquiring “gazed” texts. Unfortunately, the trial revealed that even a state-of-the-art eye-tracker
cannot locate the gazing point with an enough accuracy.
Specifically, it was very difficult to track target text lines
along with eye movement. Furthermore, even through we canlocate the gazing point perfectly, it is still problematic toexactly understand the “reading” point since gazing point andreading point might be different. Consequently, we resort tothe simple head-mounted video camera which can be equipped
physically aiming at the target text lines without the need ofeye movement estimation. Most existing eye movement based
systems, as surveyed above, definitively rely on an additional
eye-tracker assembly which renders the appearance of thedevice obtrusive.
Video frames captured by the head-mounted camera are fed
to an Optical Character Recognition (OCR) system prepared in
the laptop for extracting text information. Then, the recognizedresults will be stored in a storage of the laptop. See details in
section V. Note that instead of OCR, we can also exploit adocument image retrieval technique to get the text information
under the condition that we can specify the key target texts.
More specifically, instead of directly recognizing the video
frames, the document image retrieval technique will searcha database to get its exact text information.

IV. A PPLICATIONS
The proposed reading-life log system can serve as a journeyassistant to bridge the language barriers. Besides, the visuallyhandicapped can use our system as a navigation assistant.
Moreover, as a manage assistant, the designed system helps
us locate or recall the contents we have read somewhere.
More generally, the portability allows us to collect the text
information appearing in our daily life as a recorder assistant.

Fig. 2: The diagram of text-based frame mosaicing.
V. T ECHNOLOGIES
A. Recognizing texts in a video frame
The video camera working under the shutter-speed at
1/2000[sec] acquires the target text lines. Each frame is thenprocessed by smoothing and Otsu’s method. The binary frames
are fed into a commercial OCR one by one. For the aim of
low power consumption, in the current trial, we just employ
the straightforward binarization operator as the preprocessing
stage of OCR. In addition, the specialized technology tosegment the scene text is out of our scope since it is feasible
and tractable to further integrate the existing methods [15–19]
with our system.
B. Text-based frame mosaicing
The raw outputs of OCR are far from actual use since
frame-independence recognized results may only cover a partial target text lines. We propose a text-based frame mosaicing
to integrate all frame-independence recognized results into
completed text information (see the diagram in Fig.2). Although it is a natural idea to integrate multiple frames in
the reading-life log research, the proposed mosaicing approacheffectively leverages the recognized texts, which differs fromthe conventional image-based mosaicing methods with theprohibitive computational complexity.
The frame mosaicing is fulfilled by the Dynamic Programming (DP) matching in an iteration way. Let P denotea mosaiced frame until the previous frame and S represent
currently processing frame. Our problem is to merge P and S
for creating a new mosaiced P . At first, P and S are initialized
as the first and second frame, respectively. Each row in P and
S is assigned by a unique label as illustrated in Fig.3. The line
registration between P and S is boiled down to an optimization
problem solved by DP matching iteratively. For ∀ pi and sj ,where i = 1, 2, · · · , I and j = 1, 2, · · · , J, we compute theedit distance d(i, j) which measures the difference between
two strings, and the cumulative distance D(i, j) as follows.
D(i − 1, j − 1)
D(i, j) = d(i, j) + min D(i − 2, j − 1) + α
D(i − 1, j − 2) + α
where α is the penalty controlling the slope constraint between
two steps. Then, we backtrack from (I, J) along each (i, j),and determine the line registration between P and S.
Note that we compute the edit distance d(i, j) by asimilar DP matching process. For ∀ pi and sj , let pi =pi,1 , · · · , pi,m , · · · , pi,M and si = sj,1 , · · · , sj,n , · · · , sj,Ndenote the specific elements of two text lines, respectively.
A single element pi,m or sj,n corresponds to a recognizedFig. 3: The illustration of row label assignment for the consecutive frames P and S.

Fig. 4: The real process of the text-based frame mosicing.
character. The calculation process can be formulated below.
g(m − 1, ) + βg(m, n) = min g(m − 1, n − 1) + δ(pi,m , sj,n )(1)g(m, n − 1) + β
Here, δ(pi,m , sj,n ) is the function to check whether pi,mand sj,n are identical. Specifically, if pi,m = sj,n holds,
δ(pi,m , sj,n ) = 0, otherwise δ(pi,m , sj,n ) = 1. Similarly, β
is the penalty, which is always fixed at one. The edit distanced(i, j) is obtained as g(M, N ).
In addition, the actual algorithm is a bit more complicatedto be a “starting-point and ending-point free” DP matchingto accommodate the fact that the first several lines in P ,namely p1 , p2 ,· · · , and last several lines in S, namely sJ , sJ−1 ,sJ−2 ,· · · , usually do not appear in the S and P , respectively.
This happens when the camera moves and captures the newlines from top to bottom. Similarly, for the left-to-right case,the edit distance calculation of Eq. (1) should be organized in
the “starting-point and ending-point free” mode.
In Fig.4, we display a real process for illustrating the textbased frame mosaicing. The arrows between frames indicate
the results of line registration.
VI. F EASIBILITY T ESTIn this section, we conducted experiments to confirm thefeasibility of the proposed system under various environments
and targets. The images (a) of Figs.5∼9 displayed the target

text lines which appeared in outdoor and indoor signboards,
printed-document, LCD screen, and book/magazine cover. For
each video, we manually prepared its groundtruth as shown in
the left column of Figs.5∼9 (b).
To clearly exhibit the performance, the correct recognized
and integrated results were printed in black, the false results
in red. Blue characters stood for the missing detections whileorange ones represented a misregistration between lines. See
the detailed results in the right column of Figs.5∼9 (b). As
expected, apart from the outdoor video frame, our systemprovided reliable recognized and integrated results. For theoutdoor environment, the scene text might severely suffer fromthe non-uniform lighting or specularities so that the global
binarization method failed to make a correct segmentation.
The specialized methods [15–19] can be further employed inour system to cope with the target text lines appearing in thenatural scene. Remarkably, it was infrequent for the misregistration between lines (only occurred in Fig.5) throughout
the experiments, which demonstrated that the proposed DP
matching based approach was feasible to integrate multiple
frames. Moreover, the results with high accuracy given in
Fig.9 implied that our system was available for the LCD
screen. Furthermore, this example also proved the robustnessof our method in the sense that it could work under partial
occlusion situation. This was because benefiting from theframe mosaicing stage, the hidden part would be recoveredfrom other frames.

(a) Original frame.
(a) Original frame.








 





      

           









        

$ 



   



 

    ! 





 "                   

  



 "

 

         

         

 "          





 

   "      

 5       






      

$ 



     




 

 

 6    ! 





 "  

2          





 "

 



   

 

 " 







)

&2  "      #("   "  #      ! /"    ! '

 

(



   

     

!

  "                       

 
 





#   

 



     "







  !

* 

"

R=K? @ BHDY`KAGZ PHBH A^?=]=K=HD @K a@B@=P QA \HSJH =D a@G=PN

R=K? @ BHDY`KAGZ PHBH A^?=]=K=HD @K a@B@=P Q@ \H

C=D =DKADQP KH HXXAG @ GA>GA@K=HD@B _B@>A =D @ >HZXHGK=DY

F=X= F==K=D@P bc HGGAG @ GA>GA@K=HD@B _B@>A =D @ >HZXHGK=DY_B@>AdP=KAN_=@>AdP=KAN
eHG IJFPJF [D=OAGP=KJ fKH g@Z_FP ADKG@D>@ ? h B C=D _GH_HPAQ

g@Z_FPU R?AGA ?A >HDP>=HFPBJ GA>HYD=iAQ K?A ]@B@D>=DY

g@Z_FPU R?AGA ?A >HDP>=HFPBJ GA>HPD=iAQ =D ]@B@D>=DY

ABAZADKP PFGGHFDQ=DY =KN C=D >?HPA YGAAD @P K?A >ADKG@B >HBHGABAZADKP PFGGHFDQ=DY =KN C=D >?HPA YGAAD @P K?A >ADKG@B >HBHG_@KKAGD R@P @__GH_G=@KAQ XGHZ @ X@]G=> ?A XFDQ QFG=DY_@KKAGD R@P @__GH_G=@KAQ XGHZ @ X@]G=> ?A =>F= QFG=DY
jF@B=KJ @P RABB @P =KP OAGD@>FB@G PHFG>ANC=D KG@DPXHGZAQ K?A ADKG@D>A ?@BB =DKH @ GAPK=DY _H=DK ]HK?

C=D KG@DPXHGZAQ K?A ADKG@D>A ?@BB =DKH @ GAPK=DY _H=DK ]HK?

]@B@D>=DY K?A RHGS fD K?A >B@PP GHHZP R=K? K?A GAXFYA]@B@D>=DY K?A RHGS fD K?A >B@PP GHHZP R=K? K?A GAXFYA:=>?@AB C=D:=>?@AB C=D

kGAAD lHFPA

kGAAD lHFPA

mccn

mccn

AZFBP=HD HD R@BB

AZFBP=HD HD R@BB

opqc w qsosZZU

opqc r qsosZZU

opqc wtmuZZN opqc w uccZZ

opqc r tmuZZN opqc v w uccZZ

Fig. 6: Indoor signboard.

   

    ! '

 

(

=G@@ZHD@= KA^K==A _@KKAGDPU HXKAD ADB@GYAQ HDKH >G>= B= G>` =D a@G=P

! 0    29 11
 





 

 ( 

 # 

!



         









 

          !   

 "

"



"   <9        !(

 / ,       " "        # 













   

KG@Q=K=HD@B KA^K=BA _@KKAGDPU HXKAD ADB@GYAQ HDKH @G>?=KA>KFG@B(b) Groundtruth and results   # 



 " 

;        !!!

# " &  " "   (         "   "  

 #

 "



WXKAG GAKFGD=DY XGHZ K?A [V KH \@=R@DU :=>?@AB C=D APK@]B=P?AQ 

 + #(*!  ! ! ! "

 



   !

  #(  

 

! 9



(

    "          !

"



     2         

8* 4     / ,    



! 



(    ! '.  "        "     3 "   # +! !



 

 (          " "      ! 0 "            

 # 

EFGD =D = HIJH =D = @KLMN C=OAP @DQ RHGSP =D TGFPPABPU V?A @DQ

WXKAG GAKFGD=DY XGHZ K?A [V KH \@=R@DU :=>?@AB C=D APK@]B=P?AQ

jF@B=KJ @P RABB @P =KP OAGD@>FB@G PHFG>AN "    #             !

  !   &       



 

    "    

(       " 

* 

  



 

      "  

  #     "         

 "        








 #   "    

:=>?@AB C=D

THGD =D \HSJH =D bptqN C=OAP @DQ RHGSP =D TGFPPABPU V?@DY?@= @DQ

eHG IJFPJF [D=OAGP=KJ fKH g@Z_FP ADKG@D>A ?@BBU C=D _GH_HPAQ  

7             

          

! '    11     "     (.  "   

 #            # +! !





 " 

 "    #              !

  !   &         "   -   "  "       





 



  

  %   "  !

% 





 

      "  !
 " )







:=>?@AB C=D5
 #

 + #(*!  ! ! ! "     !!!

# " &  " "   3



" 

 

       

;         

 
 







  # 









  

!

   "(b) Groundtruth and results
Fig. 5: Printed-document.
VII. C ONCLUSION
In this paper, we propose a reading-life log system implemented by a head-mounted and unobtrusive video camera,
which can serve as various assistants in our day life. Comparedwith the eye tracker-based system, our trial practically performs a text-based frame mosaicing procedure by DP matching. In the feasibility tests, we conduct the experiments by
taking various environments as targets. The proposed system
can give reasonable responses for most target text lines. Thereseldom exists the misregistration between lines, indicating thetext-based mosaicing can effectively integrate multiple frames.
Since the performance in the natural scene is still dubious, ourfuture work is to combine with the existing scene text detection
method under a low power consumption and real-time mode.
ACKNOWLEDGMENT
The authors thank the support of Creation of HumanHarmonized Information Technology for Convivial Society(CREST), which is a project organized by Japan Science andTechnology Agency (JST).

(a) Original frame.
xyz{|}~{y ~|~y ~z {|~y   xy|y~{y~ |{y y| {}y {|} y |~y ~ {~yy |}{y ¡¢
{y £¤¥ ~y y| ¦|{} {z |{y£ ¢
§|~ ¨ ©¥ª{y ª ~|«¥ ~} ¢
{y £¤¥ {ª~  {z ¥ |{y y| {«£ ¬­®
«{y ­¢¡
~y  ¤¥ 
~|«¥ }{} ¤« ~« ¤{|y ¯¥ °y| x{ ~} ± ²x° ¨ ¦² ~|«¥ }{}y|y~{y~ }{} {z |{y ¦{}| ¤~}
{{} xy~}{| y|  ~ ¢­®
 ± ²x° ¤|ª{{ }{}{{}  ³ § |{{}  ³®¢­³
{|¥{ {y ¤¥|}~ x z{| |{y y| }
{{}  xy~}{| y|  }y~| |{{} ¡³®¢´{y ± ²x° ~y xµ² xy|y~{y~ {|¥{ {y|{y¶·~|~ xy|~«{y
{{} xy~}{| y|  ~ ¡¢¸± ²x° ¨ xµ² {y ~|«¥ }{}  ~y |{y ¦|{«{y }{} ~y ¸ ·{y {{} xy~}{| y|  ~ ¬­
¹¤¥ }{} º ª {y {y ~y ¸¥ ¤ ~« ¤{| ³¢®¥ ª º ~  ~ |{y ~{yy |{y y| ¤ ~y ~|«¥ y| ¤¹¥ ª |« ªºy ¯¥ °y| x{ ~} ~y ¤ ºy ª |{

xyz{|}~{y  ~|~y¹ ~z {| ¶»
~y   xy|y~{y~ |{y y| {}y {|} y |~y ~ {~yy |}{y¡»½
¼{yx £¤¥ ~y y| ¦|{} {z |{y£½ ¾ »½
yy |}{y  ½ ¡¢
¶½
§|~ ¨ ©¥ª{y ª ~|«¥ ~}¿{y £¤¥ {ª~  {z ¥ |{y y| {«£ ¬­®
»¶»»«{y ­¢¡
~y  ¤¥ 
~|«¥ }{} ¤« ~« ¤{|y ¯¥ °y| x{ ~} ± ²x° ¨ ¦² ~|«¥ }{}y|y~{y~ }{} {z |{y ¦{}| ¤~}
{{} xy~}{| y|  ~ ¢­®
 ± ²x° ¤|ª{{ }{}{{}  ³ § |{{}  ³®­³
{|¥{ {y ¤¥|}~ x z{| |{y y| }
{{} y~}{| y|  }y~| |{{}
¡³®»½´¹
{y ± ²x° ~y x½² xy|y~{y~ {|¥{ {y|{y¶·~|~ xy|~«{y
{{}xy~}{| y| x ~¡¬¸± ²x° ¨ ¦² {y ~|«¥ }{} ± ²x° ¤|ª{{ }{}{|¥yy yy ¤¥|}y x««y{ zy| °º|¹ÀyÀy |¹ Á¹ ­ £~y  ¤¥ {{}  ³ §x |{{} ¶³®¬|³  ~y |{~y ¦|y||yy Â}y|«yº  ~y |{y ¦|{«{y }{} ~y ¸ ·{y {{} xy~}{| y|  ~ ¬­
¹¤¥ }{} º ª {y {y ~y ¸¥ ¤ ~« ¤{| ³¢®¥ ª º~  ~ |{y ~{yy|{y y| ¤ ~y ~|«¥ y| ¤¥ ª |« ªºy ¯¥ °y| x{ ~} ~y ¤ º ª |{
»»

»¶¾¾
y
}½»· Ã} xx } y}(b) Groundtruth and results
Fig. 7: Outdoor signboard.

(a) Original frame.
ÄÅÆÇÈÉÊÉËÌ ÍÈÎÍËÇÏÉÈ ÐÅÑÒÈÐ ÉÈ ÊÓÓÓÔÏÕ
ÖÈÏÅ×ÈÅÏ ÏÅÊÅÕÍÎÍÉÈ ËÅÏÎ Ò ÑÉØÍÊÅ ÑÒÙÅÉÕÅ×Ú ÑÉØÍÊÅ ÕÅ×ÎÍÉÈ ÉÛ ÊÓÓÓÔÏÕ ÍÎ ÈÉÜÒÕÒÍÊÒØÊÅÝ ÞÊßÎ Ò ÈÅÜ ÒÞÞ ÆÒÈ ÒÊÎÉØÅ ÛÉßÈÐ ÍÈ ÌÉß× ÒÞÞ ÎÏÉ×ÅÔ à×ÍÈËÒÈ ÅÈÏÍ×Å ÈÅÏÜÉ×Ù ÉÛ ÏÅÆÇÈÉÊÉËÌ
ÍÈÎÍËÇÏ ÜÍÏÇ ÌÉßáâÅÈÅ×ÒÏÍÉÈÎ ÉÛ ÍÈÐßÎÏ×Ì ÊÅÒÐÅ×ÎÔÄÇÅ ÈÅÜÅÎÏ ÍÈÈÉÕÒÏÍÉÈÎÔ
Ä×ÅÈÐÎ ÎÇÒÞÍÈË Éß× ÛßÏß×ÅÔ
ÚÆÆÅÎÎ ÒÜÒ×ÐãÜÍÈÈÍÈË Þ×ÉË×ÒÑÎ
ÒØÉßÏ ÏÇÅ ÜÇÒÏÝ ÜÇÉÝ ÒÈÐ ÇÉÜ ÉÛ ÏÅÆÇÈÉÊÉËÌ ÏÉÐÒÌÔâÉ ÑÉØÍÊÅ É× ËÅÏ ÏÇÅ ÒÞÞÔÜÜÜÔÊÅÅÅÔÏÕ
ÖÓÓÓÄÅÆÇÈÉÊÉËÌ ÍÈÎÍËÇÏÉÈ ÐÅÑÒÈÐ ÉÈ ÊÓÓÓÔÏÕ
ÖÈÏÅ×ÈÅÏ ÏÅÊÅÕÍÎÍÉÈ ËÅÏÎ Ò ÑÉØÍÊÅ ÑÒÙÅÉÕÅ×Ú ÑÉØÍÊÅ ÕÅ×ÎÍÉÈ ÉÛ ÊÓÓÓÔÏÕ ÍÎ ÈÉÜÒÕÒÍÊÒØÊÅÝ ÞÊßÎ Ò ÈÅÜ ÒÞÞ ÆÒÈ ÒÊÎÉßÅ ÄÉßÈÒ ÍÈ ÌÉß× ÒÞÞ ÎÏÉ×ÅÔ ä×ÍÈËÒÈ ÅÈÏÍ×Å ÈÅÏÜÉ×Ù ÉÛ ÏÅÆÇÈÉÊÉËÌ
ÍÈÎÍËÇÏ ÜÍÏÇ ÌÉßáâÅÈÅ×ÒÏÍÉÈÎ ÉÛ ÍÈÐßÎÏ×Ì ÊÅÒÐÅ×ÎÔÄÇÅ ÈÅÜÅÎÏ ÍÈÈÉÕÒÏÍÉÈÎÔ
Ä×ÅÈÐÎ ÎÇÒÞÍÈË Éß× ÛßÏß×ÅÔ
ÚÆÆÅÎÎ ÒÜÒ×ÐãÜÍÈÈÍÈË Þ×ÉË×ÒÑÎ
ÒØÉßÏ ÏÇÅ ÜÇÒÏÝ ÜÇÉÝ ÒÈÐ ÇÉÜ ÉÛ ÏÅÆÇÈÉÊÉËÌ ÏÉÐÒÌÔâÉ ÑÉØÍÊÅ É× ËÅÏ ÏÇÅ ÒÞÞÔÕÕÑÕåÅâÅÔÏÕ
ÑÕÜÔÍÅÅÅÔÏÕ

(b) Groundtruth and results
Fig. 8: Magazine cover.

(a) Original frame.
æç èéêëìé íîïðëñêìò óôêìîéî èé õêòòîéö
æïîì÷îøé ùì úëïëì
üýþÿý >ê îê æïîì÷êìò  øîêòì
öøêéöé êì úëïëì êé øîñîøêìò ëöîø ëïøðìòî÷ éðòòêéô éïîðð ñëéî÷ *øéîìêìò úëïëì óôêìë öêîé+éì ö íììêìò øú îë÷îøéôêï
üýþÿý >ê îê 1éôê +éì öôîî-ñøëöêñ ëøö  úëïëìûé ï ðêññôêî ôëé ÷îñê÷î÷ ìö ö øì êì öôîïëøö ûé îñ ðîë÷îøéôêï îðîñöêì üôî ê îêðîëøìî÷ ì î÷ìîé÷ëüöë çîêìò ýöï ö ý -ì  ìêöé ùì û'1îìîøêñ ÷ø ò ë îøé ñéêìò ýì óëìñîøøòéûêéôê êîé ùìö æôøê-ï ëø -êìò ùì üôëêðëì÷
èîì ü üë î ýîø ëøìîø õøé óêìîïðî;îìöøî
<øî ëøöêñðîé>óôëììîð úè è ê îê ëì÷ üõæ õøë÷ñëéöêìò øèéêë?öô ùþþçùñ- æî -êìëøøîî ëññîéé ø ðþþçùñ- ëö úè ìòî êìæç èéêëìé íîïðëñêìò óôêìîéî èé õêòòîéö
æïîì÷îøé ùì úëïëì
ûüýþÿý ê îê æïîì÷êìò  øîêòì
ú ' öøêéöé êì úëïëì êé øîñîøêìò ëöîø ë ú ïøðìòî÷ éðòòêéô éïîðð ñëéî÷ *øéîìêìò úëïëì óôêìë öêîé+éì ö íììêìò øú îë÷îøéôêï
üýþÿý ê îð01éôê +éì öôîî-ñøëöêñ ëøö  úëïëìûé ïðêññôêî ôëé ÷îñê÷î÷ ìö ö øì êì öôîïëøöûé îñ ðîë÷îøéôêï îðîñöêì üôî ê îêðîëøìî÷ ì î÷ìîé÷ëüöë çîêìò ý öïö ý -ì ìêöé ùì û'1îìîøêñ øò ë îøé ñéêìò ýì óëìñîøøòéûðéôê êîé ùìö æôøê-ï ëø -êìò ùì üôëêðëì÷
èîì ü üë î ýîø ëøìîø õøé óêìîïðî;îìöøî
<øî ëøöêñðîé>óôëììîð úè è ê îê ëì÷ üõæ õø ë÷ñëéöêìò øèéêë?- ðþþçùñ- æî-êìëøøîî ëññîéé ø ðþþçùñ- ëö úèìòî êì(b) Groundtruth and results
Fig. 9: LCD screen.

R EFERENCES[1] C. Campbell, P. Maglio, A Robust Algorithm for Reading
Detection, ACM Workshop on Perceptive User Interfaces,
2001.
[2] W.Kienzle, F. Wichmann, B. Schölkopf and M. Franz, A
Nonparametric Approach to Bottom-Up Visual Saliency,
NIPS, 2006.
[3] A. Bulling, J. Ward, H. Gellersen and G. Tröster, Roubst
Recognition of Reading Activity in Transit Using Wearable
Electrooculography, International Conference on Pervasive Computing, 2008
[4] G. Buscher and A. Dengel, Attention-Based Document
Classifier Learning, DAS, 2008.
[5] S. Xu, H. Jiang and F. Lau, User-Oriented Document
Summarization through Vision-Based Eye-Tracking, International Conference on Intelligent User Interfaces, 2009[6] T. Judd, K. Ehinger, F. Durand and A. Torralba, Learning
to Predict Where Humans Look, ICCV, 2009.
[7] M. Nyström and K. Holmqvist, An Adaptive Algorithm for
Fixation, Saccade, and Glissade Detection in Eyetracking
Data, Behavior Reserach Methods, vol.42, no.1, pp.188204, 2010.
[8] T. Loboda, P. Brusilovsky and J. Brunstein, Inferring Word
Relevance from Eye-Movements of Readers, International
Conference on Intelligent User Interfaces, 2011.
[9] M. Carlos, L. Karel and M. Majid, A Head-Mounted
Device for Recognizing Text in Natural Scenes, CBDAR,
2011.
[10] G. Duggan and S. Payne, Skim Reading by Satisficing:
Evidence form Eye Tracking, SIGCHI Conference on
Human Factors in Computing Systems, 2011.
[11] O. Dimigen, W. Sommer, A. Hohlfeld, A. Jacobs and
R. Kliegl, Coregistration of Eye Movements and EEG
in Natural Reading: Analyses and Review, Journal of
Experimental Psychology: General, vol.140, no.4, pp.552572, 2011.
[12] G. Buscher, A. Dengel, R. Biedert and L. Elst, Attentive
Documents: Eye Tracking as Implicit Feedback for Information Retrieval and Beyond, ACM Trans. on InteractiveIntelligent Systems, vol.1, no.2, article 9, 2012.
[13] R. Biedert, J. Hees, A. Dengel and G. Buscher, A Robust
Realtime Reading-Skimming Classifier, Symposium on Eye
Tracking Research and Applications, 2012.
[14] K. Kai etc., to be submitted to ICDAR 2013.
[15] K. Jung, K. Kim and A. Jain, Text Information Extraction
in Images and Video: A Survey, Pattern Recognition,
vol.37, no.5, pp.977-997, 2004.
[16] J. Liang, D. Doermann and H. Li, Camera-Based Analysis of Text and Documents: A Survey, IJDAR, vol.7, no.2,
pp.84-104, 2005.
[17] C. Yi, Y. Tian, Text String Detection From Natural Scenes
by Structure-Based Partition and Grouping, IEEE Trans.
on Image Processing, vol.20, no.9, pp.2594-2605, 2011.
[18] Y. Pan, X. Hou and C. Liu, A Hybrid Approach to Detectand Localize Texts in Natural Scene Images, IEEE Trans.
on Image Processing, vol.20, no.3, pp.800-813, 2011.
[19] L. Neumann and J. Matas, Real-Time Scene Text Localization and Recognition, CVPR, 2012.