Film Comic Generation with Eye Tracking
Tomoya Sawada

Masahiro Toyoura
University of Yamanashi

Xiaoyang Mao

Abstract. Automatic generation of film comic requires solving several
challenging problems such as selecting important frames well conveying thewhole story, trimming the frames to fit the shape of panels without corrupting
the composition of original image and arranging visually pleasing speechballoons without hiding important objects in the panel. We propose a novel
approach to the automatic generation of film comic. The key idea is toaggregate eye-tracking data and image features into a computational map,
called iMap, for quantitatively measuring the importance of frames in terms ofstory content and user attention. The transition of iMap in time sequences
provides the solution to frame selection. Word balloon arrangement and image
trimming are realized as the results of optimizing the energy functions derived
from the iMap.
Keywords: Visual attention, eye-tracking, film comic, video processing, frame
selection, image trimming, balloon arrangement.

1

Introduction

Film comic, also called cine-manga, is a kind of art medium created by editing theframes of a movie into a book in comic style. It is loved by a wide range of readersfrom little babies to comic manias around the world. Traditionally, film comics aremanually created by professional editors who are familiar with comic styles. The
frames of movies are manually selected, trimmed and arranged into the comic-likelayouts. Verbal information of the movie, such as dialogues and narrations, areinserted into the correspondent panels as word balloons. Since a movie usuallyconsists of a huge number of frames, those editing tasks can be very tedious and time
consuming.
Recently, several research works have been conducted trying to provide
computational support to the generation of film comic [1-4]. The work by R. Hong etal. [3] first succeeded in automating the whole procedure of film comic by employing
modern computer vision technologies for extracting key frames and for allocating theword balloons. However, their method replies on existing human face and speaking
lip detecting technologies for identifying speakers, and are not applicable to cartoon
animations, which is the main target of film comics, because the appearance ofcharacters in cartoon animation is usually very different from that of human face.
Actually, many issues of automatic film comic generation cannot be solved with naive
image/video processing or even by combining text analysis. For example, a subtle
change of character’s posture which causes no remarkable change to image featuresmay be very critical in conveying the story. When placing a word balloon, only
avoiding occluding the speaker is usually not enough since an object about which the
speaker is talking may also be very important. All these problems are story or evenuser dependent.
In this paper, we present a new approach of using eye-tracking data to automate thewhole process of film comic creation. It is known that eye movement and gazeinformation provides good cues to the interpretation of scenes by viewers [5].
Existing studies [6-8] showed the effectiveness of applying visual attention model formeasuring the significance of images in creating video summarization. Thosetechniques use the computation model of visual attention which is mainly based on
the low level features of images and motions. The real attention of viewers, however,is largely dependent on the content of a story as well as the personal background of
the viewers, such as their age, sex and cultural background. By using the eye-tracking
data, we can provide a better prediction to the attention of viewers. Furthermore, byusing subjects from the assumed population of readers, e.g. using the eye-tracking
data of children for creating a film comic mainly targeting at children, we can evencreate film comics adapted to a particular population.
The major contribution of this paper can be summarized as follows:
1.
2.
3.
4.

Proposal of a new framework for automatically converting a movie into a comic
based on eye-tracking data.
Proposal of iMap, a new computational map which combines eye-tracking dataand image feature for quantitatively measuring the local informativeness in amovie frame.
Proposal of a new algorithm for detecting critical frames based on iMap .
Proposal of the optimization scheme for achieving desirable image trimming andballoon arrangement.

The remainder of the paper is organized as follows: Section 2 reviews the related
work. Section 3 gives the overview of the proposed method and Section 4 introduces
iMap together with its construction algorithm. The detailed algorithms of frame
selection, trimming, balloon allocation are described in Section 5. Section 6 is about
implementation and experiment and Section 7 concludes paper.

2

Related works

A pilot work on film comic generation was done by W.I. Hwang et al. [1]. Assuming
the correspondence between a balloon and a speaker is known, their method can
automatically allocate balloons following the rules of comic, but the major tasks offilm comic generation such as the selection of frame images were done manually.
Preuß et al. [2] proposed an automatic system for converting movies into comics, but
their method assumes that the screenplay of the movie is available and hence limits its
application to a very special case. More recently, R. Hong et al. [3] presented a fullautomatic system which employs face detector, lip motion analysis, and motion
analysis to realize the automatic script-face mapping and key-scene extraction. But asmentioned in the previous section, their technique cannot be used for cartoon
animations. M. Toyoura et al.[4] proposed a technique to automatically detect typicalcamera works, such as zooming, panning, fade-in/fade-out, and represent them in aparticular comic style. The same function is supported in our system.
Video summarization [9] is the field most closely related to film comic generation.
Both video summarization and film comic generation share the two major techniqueissues: how to select appropriate images from a video/movie and how to arrange theimages to effectively depicting the contents of the video/movie. Video summarization
has been well studied in the past decade and researchers have challenged the problem
through various approaches ranging from computer vision [9-10], text mining [11], tofMRI [12]. Several researchers proposed to use human visual attention model forvideo segmentation and skimming [6-7]. Those methods were further enhanced
through combining multiple visual cues [8] as well as high level semantic andcognitive information [13]. W. Peng et al. proposed an approach using both eyetracking and facial expression detection for selecting important frames in home video
summarization [14]. Comic style was also explored in the field of video
summarization for achieving an effective representation of video summaries [15].
While a video summary is mainly for providing the overview of a video or serving as
the index enabling users to quickly browse the required information in the originalvideo, film comic is an alternative of movie for storytelling and hence should enable
readers to easily understand the whole story without refereeing to the original movie.
Therefore, many technologies of video summarization cannot be applied to film
comic generation directly. For example, W. Peng et al. selected the frame sequencewith long gaze fixation as the important frames to be included in the summary [14].
But from the viewpoint of story development, a frame causing the change of attention
should be more important. We succeeded in selecting such frames by detecting thesignificant transition of iMap.

3

Framework

Figure 1 depicts the framework of the proposed technique. Given a movie including
subtitle information, our technique converts it into a comic in following 10 steps. First,
sample viewers are invited to watch the movie and their eye positions are recordedwith an eye-tracker. At the same time SURF image feature detection is performed foreach frame of the movie (Step 2). Then an iMap is generated from the eye-tracking
data and SURF feature (Step 3). Based on iMap, the frame sequence of the movie issegmented into groups of frames called shots (Step 4). Those shorts are subjected tocamera-works detection (Step 5). Then for each shot, the first frame, the frames withspeech line and the frames for representing camera works are selected (Step 6). Basedon the number of selected frames and the information on camera works, panel layoutof the comic is generated (Step 7). The information required for trimming the frames
and for placing the balloons are computed by referring to iMap (Step 8&9). Finally acomic book is created by arranging the trimmed frames into the comic layout andballoons with speech lines are inserted to the frames (Step 10). We use existing
technologies for Step 5 and 7, and the other steps are realized with newly developed
techniques, whose details are described in the next section.

Figure 1 The framework of film comic generation with proposed technique.

4

Film comic generation with iMap

4.1 Construction of iMap
A good film comic should contain the frames that are important for understanding thestory and each of the frames should be represented in a way well preserving theoriginal information while being visually pleasing as a comic. Those requirements
give rise to the issue of measuring how informative a frame or a region in a frame isin conveying the story. We address the issue by combing eye-tracking data and image
features into iMap, a probability distribution map providing a quantitative measure to
the local informativeness in each frame. It is known that eye movement plays animportant role in event perception [5]. Our eyes are likely to be drawn to a region
which is most relevant to the story development and a big eye movement usuallyprovides a cue to the change of events. However, eye-tracking data usually consists ofnoise. It is also difficult to distinguish an unintentional or meaningless fixation from adeliberate or purposeful one. To solve the problem, we first convert the raw eyetracking data of multiple viewers into an attention map, a probability distribution of
attention estimated with Gaussian kernel. At the same time we generate a feature map
which is a probability distribution estimated from SURF (Speeded Up Robust
Feature) [16]. We adopt SURF for taking its advantage in speed and robustness intracking features across frames. Then we construct iMap as the aggregation of theattention map and the feature map. The significance of combing eye-tracking data andimage feature lies in the fact that a fixation at a position with little image feature suchas edges and corners is likely to be a meaningless one and the resulting iMap gives amore reliable prediction to how informative the position is.
A feature map Mtf representing the probability distribution of image features in tth
frame is calculated as follows:t( x, y ) ⊗ G (0,σ 2f )M tf = f SURF

if ( x, y ) is SURF
⎧1t
( x, y ) = ⎨f SURF⎩0 if ( x, y ) is not SURF

(1)

where G(0,σf2) is a 2D Gaussian kernel with average 0 and variance σf2. ⊗ isconvolution. If pixel (x,y) is SURF, its contribution to the probability distribution ofimage feature is estimated with the Gaussian kernel.
Similarly, an attention map Mt representing the distribution of n sample viewer’s
attention is calculated as follows:n

M t = ∑ f i EYE ( x, ty ) ⊗ G (0,σ a2 )ti =1if ( x, y ) is an eye position⎧1t
f i EYE ( x, y ) = ⎨⎩0 if ( x, y ) is not an eye position(2)

iMap Mt of tth frame is constructed from maps Mtf and Mta as follows:
M t = M tf ∗ M at .
(3)
Since Mt is the product of Mtf and Mta, a high value of Mt indicates an informative
region in terms of both image features and viewers’ attention. When there is asignificant change of Mt in time sequences, there should be a transition of either/bothof image features or/and viewers’ attention. By detecting such transition, the movie
can be segmented into the shots consisting of frames of the same contents.
Although Mt gives a good solution to the detection of content transition among frames,
it does not contain enough information for performing frame image trimming andballoon arrangement, which requires measuring the local informativeness in thecontext given by neighboring frames also. We further extend iMap to encounter the
probability distribution in time dimension. An iMap representing the spatial-temporal
informativeness is calculated from the neighboring frames of tth frame as follows:
M Nt = ∑ G (t , σ t2 )( s ) * M s ,(4)s∈N ( t )where N(t) is the set of neighboring frames of tth frame and the contribution of eachneighboring frames is given by Gaussian function G(t,σ t2) (s) with highest value at tth
frame.
4.2 Frame selection
To select frames which are important for conveying the story of the movie, we first
segment the movie into shots by detecting significant change of iMaps across adjacentframes. The set of transition frames Ω(t) are detected as(5)
Ω(t ) = { t | || M Nt +1 − M Nt || > T }T=|| G ( x, y ) − G ( x − x d , y − y d ) ||∫∫M

Where G(x,y) is the Gaussian filter used for computing iMap, with ||*|| denotes L2
norm, d = xd + yd is the possible distance between the eye positions at adjacentframes when a saccade occurs. A saccade is a fast movement of an eye between
fixations. The central part of our retina, known as the fovea, plays a critical role inresolving objects. We move our eyes quickly toward a target so as to sense it with

greater resolution. Therefore, if a saccade is detected, we can say that the region ofinterest has changed, which indicates a possible transition of story contents.
After transition frames are detected, the movie is segmented into the group of frames,
say shots, at the transition frames. Then the first frame of each shot, which is thetransition frame, is selected for being included in the film comic. All the frames withspeech lines are also selected. If a shot contains special camera-works, multiple
frames are selected depending on the type of camera-works.
4.3 Image trimming
If the shape and size of a selected frame image are different from those of thecorresponding panel, the frame image is trimmed and resized to fit into the panel. As
our current implementation uses rectangular panels only, trimming is necessary only
when the frame image and its corresponding panel have different aspect ratios. A
good trimming should meet two requirements: First, the area being cropped should beless informative. Second, the composition of original frame should be preserved as
possible. We achieve the best trimming by minimizing the following cost computedfrom iMap MNt:
E trim =∑mij( i ', j ')∈I

S (I )

+

⎛

i

⎝

F

i'

j

j'

⎞

∑ m ⎜⎜ M − i − M − i' + N − j − N − j ' ⎟⎟( i ', j ')∈I

ij

I

F

2 Max ( M F , N F ) S ( I )

I

⎠.

(6)

Figure 2
Image trimmingby referring to iMap MNt.
As shown in Figure 2, F denotes the original frame image, I the trimmed one and I
the set of the pixels being cropped away with the trimming. (i',j’) is the position inoriginal frame image F for the pixel in I or I . S (I ) and S (I ) are the total
number of pixels (i,j) in I and I , MF, MI, NF, NI are the width and height of the frame
image F and trimmed image I, respectively. The first term in Eq (6) measures how
less informative the region being cropped is and the second term evaluates whether
the relative location of a pixel in original frame is preserved in the trimmed image. By
multiplying the cost with the value in MNt, we give those informative pixels a higher
priority for preserving their relative positions.
4.4 Arranging balloons
A balloon is placed into the panel of a selected frame with corresponding speech lines.
Balloons should be placed in a way to meet 3 requirements: First, it should not
occlude an important region. Second, it should be in a visually pleasing shape. Third,
it should be spatially close to the speaker. For the first requirement, iMap provides theinformation about the importance of regions. In our current implementation, we use
oval shaped balloons, which is one of the most popular shapes found in comics. To
avoid generating long narrow shaped balloons, we use the ratio of the two axes of theellipse as the measure to the aesthetic quality of the balloon. To measure the distancefrom the balloon to the speaker, the position of the speaker should be given. We havetried out existing face detector and speaking lip detector, but found all of them failed
to produce good results. The main reason is that the appearance of characters incartoon animation are usually very unique, which makes it difficult to apply theexisting face detector. Another reason is that a cartoon animation usually contains
exaggerated motions throughout the movie and this makes it difficult to detect small
motions such as a speaking lip. Put all together, a desirable balloon arrangement canbe obtained by minimizing the following energy function:
Eballon = a∑mij( i , j )∈I balloon

S ( I balloon )+b

Pballoon − PI spea ker
M 2 + N2+c

Lshort ,
Llong

(7)

where Iballoon is the region occluded by a balloon, Pballoom, PIspeeker are the gravity centerof balloon and speaker region, Lshort, Llong are the short and long axes of balloon,
respectively. a, b, c are user given parameters for controlling the weight of each term.
The values of Lshort, Llong are dependent to the length of speech lines. Lshort and Llong
take discrete values to avoid breaking a line at the middle of a word. Given a speechline, we first build a look up table containing all the possible pair of Lshort, Llong andloop through all the entries of the table during optimization.
After the shape and location of a balloon is decided, a tail is attached to the balloon
and headed toward the speaker.

5

Experiments

We invited sample viewers (3 male university students in their 20s) to watch the
beginning part of “Pinocchio” and “Snow White and the Seven Dwarfs” by DisneyInc. and a part of “Roman Holiday”. The movies were displayed on a 1920x1200
monitor and EMR-AT VOXER of Nac Image Technology was used for tracking
subject’s eye positions at 60Hz. Frame rate of the movies was 29.97fps. Figure 8 aresome example pages of the resulting film comics and Table 1 shows some statistics ofthe results for the two cartoon movies.
Figure 3 and Figure 4 compare the results of frame selection by the existing color
histogram based video segmentation [10] and by our method. Figure 3 is the scene thecricket sneaks into the house in the freezing night trying to warm his hip up with theburnt stone from the fireplace. Since the color histogram does not change muchthrough the scene, the result by existing work as shown in Figure 3(a) failed to detectenough frames required for representing such mise-en-scene. Figure 4 is a part of“Roman Holiday” where Mr. Joe and Princess Ann try to insert their hands into themouth of truth. Princess Ann gets frightened and moves her hand toward the mouth
timidly. Such process could not be visualized with the result of existing technique in
Figure 4(a) because of the little change of color histogram. In Figure 4(b), ourproposed method succeeded in catching such subtle but important transition ofcontents.
Figure 5(a) shows an example of image trimming. In the scene, the Prince exchangesbows with Snow White. Both of the Prince and Snow White are considered to be
informative and should be included in the result of trimmed image. Figure 5(b) is theiMap of the frame, and the lines indicated the trimmed area by energy optimization.
The yellow regions indicate high values in the iMap. We can recognize that they are
the regions corresponding to the Prince and Snow White. Moreover, the highest
position is Prince’s face. It demonstrates that our proposed method could estimate theimportant positions with the use of multiple viewers’ eye movement data and SURF
feature tracking.
Table 1 Experimental Results
Pinocchio
Snow White
Roman Holiday

Length20m 00s
11m 10s
1m 24sPanels352
200
38

Ballons Position error
254
10 (3.9%)
94
6 (6.3%)19
3 (15.7%)

Tail error
32 (12.5%)
12 (12.7%)
1 (5.2%)(a) By existing color histogram based technique.

(b) By proposed technique. Images with red frames are corresponding to the ones of (a).
Figure 3 Comparison of frame selection results for “Pinocchio.”(a) By existing color histogram based technique.

(b) By proposed technique. Images with red frames are corresponding to the ones of (a).
Figure 4 Comparison of frame selection results for “Roman Holiday.”Figure 6(a) shows a result of balloon arrangement. The blue box in Figure 6(b) is theresulting balloon position given by energy minimization. By placing the balloon at
this position and directing the tail to the highest area of the iMap, the important areain the frame remains not occluded and the tail is directed to the speaker.
The number of position error in Table 1 is the number of balloons that partiallyocclude a character or an object which is related to the story development. The

number of tail error is the number of tails not heading to the speaker. Figure 7(a) and7(b) are the examples of position error and tail error, respectively. In Figure 7(a), theballoon has been placed on the body of Wicked Queen since the surrounding areas
have more SURF features than the body area. We expect to solve this kind of problem
by using other higher level image features such as those for measuring objectiveness.
In Figure 7(b), the tail of the balloon has a wrong direction since the speaker is notincluded in the scene. We expect to reduce such errors by combining eye-tracking
data with some advanced motion detection techniques.

(a) Original frame image.
(b) iMap corresponding to (a).
Figure 5 An example of image trimming by referring to iMap.

(a) Composited balloon.

(b) Optimizing the position ofa balloon on iMap.
Figure 6 Arranging a balloon by referring to iMap.

(a) Balloon arrangement failure.
(b) Tail attaching failure.
Figure 7 Example of balloon arrangement and tail attaching failure.

6

Conclusions and feature works

This paper presented a novel technique for generating a film comic from a movie
automatically based on the eye-tracking data of sample viewers. We believe that theconcept of iMap, together with the optimization scheme for obtaining desirable image
trimming and balloon arrangement, has a very high potential for being used in otherimage/video processing applications. Also we believe that with the advancement ofeye-tracking technology as well as the cost down of eye tracking devices in the future,
the proposed technique will enable casual users to easily enjoy movie in the style ofcomic.

Acknowledgement
This work was supported by KAKENHI 21300033 Grant-in-Aid for Scientific
Research (B), Japan Society for the Promotion of Science (JSPS).

References
1.

Hwang, W.I., Lee, P.J., Chun, B.K., Ryu, D.S., Cho, H.G.: Cinema comics: Cartoon
generation from video stream. In: International Conference on Computer Graphics
Theory and Applications. (2006) 299–304

2.

Preuß, J., Loviscach, J.: From movie to comics, informed by the screenplay. ACM
SIGGRAPH (Poster) (2007)

3.

Hong, R., Yuan, X.T., Xu, M., Wang, M., Yan, S., Chua, T.S.: Movie2comics: a feast ofmultimedia artwork. In: Proceedings of the International Conference on Multimedia.
(2010) 611–614

4.

Toyoura, M., Kunihiro, M., Mao, X.: Film comic reflecting camera-works. In:
International Conference on MultiMedia Modeling. (2012) 406–417

5.

Smith, T.J., Whitwell, M., Lee, J.: Eye movements and pupil dilation during event
perception. In: Proceedings of the Eye Tracking Research and Applications conference.
(2006)

6.

Ma, Y., Hua, X., Lu, L., Zhang, H.: A generic framework of user attention model and its
application in video summarization. IEEE Trans. on Multimedia 7(5) (2005) 907–919

7.

Li, K., Guo, L., Faraco, C., Zhu, D., Deng, F., Zhang, T., Jiang, X., Zhang, D., Chen, H.,
Hu, X., Miller, S., Liuothers, T.: Human-centered attention models for video
summarization. In: International Conference on Multimodal Interfaces and the Workshop
on Machine Learning for Multimodal Interaction, ACM (2010) 27

8.

You, J., Liu, G., Sun, L., Li, H.: A multiple visual models based perceptive analysis
framework for multilevel video summarization. IEEE Transactions on Circuits andSystems for Video Technology 17(3) (2007) 335–342

9.

Money, A.G., Agius, H.: Video summarisation: A conceptual framework and survey ofthe state of the art. Journal of Visual Communication and Image Representation 19(2)(2008) 121–143

10.

Porter, S.V.: Video Segmentation and Indexing using Motion Estimation. PhD thesis,
University of Bristol (2004)

11.

Chen, B., Wang, J., Wang, J.: A novel video summarization based on mining the story
structure and semantic relations among concept entities. IEEE Transactions on
Multimedia 11(2) (2009) 295–312

12.

Hu, X., Deng, F., Li, K., Zhang, T., Chen, H., Jiang, X., Lv, J., Zhu, D., Faraco, C.,
Zhang, D., et al.: Bridging low-level features and high-level semantics via fmri brain
imaging for video classification. In: Proceedings of the international conference on
Multimedia, ACM (2010) 451–460

13.

Liu, A., Yang, Z.: Watching, thinking, reacting: A human-centered framework for movie
content analysis. International Journal of Digital Content Technology and its Applications
4(5) (2010) 23–37

14.

Peng, W.T., Huang, W.J., Chu, W.T., Chou, C.N., Chang, W.Y., Chang, C.H., Hung,
Y.P.: A user experience model for home video summarization. In: International
Conference on Multimedia Modeling. (2009) 484–495

15.

Boreczky, J.S., Girgensohn, A., Golovchinsky, G., Uchihashi, S.: An interactive comicbook presentation for exploring video. In: ACM Conference on Computer-Human
Interaction (CHI). (2000) 185–192

16.

Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (surf). Computer
Vision and Image Understanding 110(3) (2008)(a) Pinocchio.

(b) Show White and the Seven Dwarfs.

(c) Roman Holidays.
Figure 8 Comic pages generated with Proposed technique.