A Browser for Summarized Multiple Videos
Saki Kasamatsu

Takayuki Itoh

Graduate school for Humanities and Sciences, Ochanomizu University
E-mail:{saki,itot}@itolab.is.ocha.ac.jp

1. Overview

keyframes, because contents of frames vary in those kinds ofvideos.

Recent digitalization of multimedia contents evolved video
recording and streaming technologies, and amount of videos
stored in personal computers have been therefore increasing. It isoften time-consuming to look for demanded video files, consult
the content of each file, or compare the contents of multiple
video files, from the large number of stored video files.
There have been many works on video summarization, andvisualization of such summarized videos. Some of the works
developed browsing tools to visualize the video summaries,including features such as assigning different sizes to each

Figure 1: Keyframes on a time axis.

keyframe by calculating importance of scenes, or enabling
control of summary lengths by developers. However, most ofThis paper is organized as follows. Section 2 introduces

visualization works did not focus on representation of timing ofrelated work. Section 3 describes the architecture of ourscenes and total lengths of the video files. Moreover, those works

technique. Lastly, Section 4 describes conclusion andare just available to consult the content of single video file; they

discusses future work.

are not suitable for visual comparison of multiple video files.

2. Related Work

The paper presents a technique for visualization ofsummarized multiple video files. The technique places keyframesVideo summarization has been an active research topic, dueof the summarized video on a horizontal time axis as shown into the saturation of video files. Shot boundary detection is one

Figure 1. Also, the technique vertically places the time axis ofof the most important technical issues to divide videos into

multiple video files. It makes easier to grasp the story of videos,shots in the video summarization techniques. Miyamoto et al.

and compare the contents of multiple videos. The techniquehave developed the shot boundary detection system using

provides a user interface for level of detail control, which

optical flow, HSV color data, and edge difference [1], in

adaptively adjusts the number of displaying keyframes per unit

addition to the typical methods based on pixel difference.

time. It displays more keyframes with zoom in operation, so that

Other approaches [2][3] have adopted the support vector

users can understand the detailed story of the video. On the othermachines to learn from recognition patterns.

hand, it displays less keyframes with zoom out operation, so that

Keyframe selection is another important technical issue toit can display more number of summaries of video files, or longer

create a concise and meaningful representation of video in thestories on one screen.

video summarization techniques. While typical keyframe

In our approach, we chiefly focus on quickly getting an

extraction algorithms select keyframes based on clustering ofoverview of video files which the users have seen once, orframes [4], Smith et al. have developed the automatic detection

comparing the contents of accumulated multiple video files. Here,system of important keyframes, by calculating importance weight

we need to save the screen space that each file takes up, toof each video shot based on caption analysis and face detection

visualize more video summaries on a screen. To solve theresults [5]. In addition, Hamada et al. have developed systemproblem, we present a visualization interface that allows us towhichinteractively control the number of displayed keyframes for eachautomatically extract the important scenes of a cooking video

detects

specific

motions

fromthevideo,

tovideo file. We think this approach is particularly effective for TV[6]. Another relevant work has analyzed construction of shots

dramas and films, because they tend to have story changes andto extract specific scenes from sports video [7]. These

events more frequently. Users may easily understand the story ofapproaches have focused on specific kinds of videos, tosuch video files by looking at the less number of impressive

efficiently obtain good summarization results.
1

There have been also several works on visualization oftechniques calculate the importance score based on thesummarized videos. Video Manga [8] is one of the works mostsemantics, our implementation considers the appearance ofrelevant to ours, which automatically resizes keyframes andthe images as well. We think that frames are important if itpacks them into one-page pictorial video summaries, resembling

can be clearly recognized even they are zoomed out.

comic books. Another work has accomplished to visualize

Following are the elements used to calculate importance scorelarge-scale news videos, which enables control of sizes andof the keyframes:

places of the keyframes according to their importance weight [9].

Face detection results. We treat that a frame is important if

These advanced techniques can be integrated into traditional

faces of persons appear in the frame. Our implementation

video browser tools [10] [11]. These visualization techniques

assigns higher scores when the number of the face detected inmainly focus on representation of single video file. They are

the frame increases. Furthermore, our implementation also

useful for understanding the story or finding the specific scenes

takes sizes of faces into account for importance calculation.

from the specific video file. However, they are not always

Close-ups are also important because they are comprehensive

sufficient to find the demanded scenes or compare multiple videoeven when the frames are zoomed out.

files in one display space. Also, users are unable to control theSaturation. We presume that a frame is important if

length of summary to look minutely or simply, since theirsaturation average of the image is higher. Such frames are stilltechniques do not feature interactive level of detail control.

comprehensive even when they are zoomed out. Our
implementation

calculates

averagesofsaturationof

keyframes from YC C color values used in Section 3.1.

3. Processing Flow
This paper presents a technique for visualization of

Length of a scene. Our implementation supposes that lengthsummarized videos, by placing keyframes on a horizontal timeof a scene is proportional to size of a cluster which theaxis. This section describes the architecture of our technique in

keyframe belongs to. We treat that a frame is important if thedetail.

scene of the frame is long. We do not want to miss such

Our current study applies typical keyframe selection algorithm,

scenes from the visualization results.

and we mainly focus on interactive visualization of multiple

Time Position of a frame. Importance score also variesvideo summaries. We assume our technique is especiallydepending on the time position of the keyframe in the whole

suitable for videos which have frequent scene changes, such asvideo. It is an application-specific matter: we need to adjust

dramas and films.

the importance calculation using the time position according
to the genre of videos. For example, we think users may want

3.1. Video Summarization

to focus on latter parts of drama videos because they often

The technique first summarizes the video data by keyframe

include climaxes of the stories. In this case users can adjust toselection. Our implementation is based on an existing work

make the importance score higher. In contrast, we thinkpresented in [8]. This process first divides a video into “shots”

former parts of videos tend to contain less important scenes.

that are consecutive series of frames in the video constituting

In this case we can decrease the importance score for thea unit of action. It divides the video into shots by detecting

frames which belong to former parts of a video.

shot boundaries, where the difference of pixel values between

3.3. Sizing of Keyframes

two adjacent frames is large enough. It then selects
representativeframesofeachshot.

Ourcurrent

The technique then calculates sizes of each keyframe

implementation abstracts the temporally central frame in aaccording to their importance scores, to emphasize important

shot as its representative frame. The technique then generates

keyframes by assigning larger sizes. Our implementation

clusters of similar representative frames of temporally

calculates the sizes of the keyframes from the proportions ofadjacent shots, by calculating averages of YC Ccolor

their importance score to the maximum score of the wholevalues for each frame. We call the clusters “scenes”, which

video, so we can normalize the importance score among

denotes larger semantic series of frames in a video. It finallymultiple video files. It then assigns one of several different

selects the representative frames for each cluster, as

sizes to a given keyframe from the calculated proportion. Our“keyframes” of the video, which have the closest YC C

implementation uses fixed proportion thresholds to determine

color value to the averages of each cluster.

the sizes.
Figure 2 shows an example of the visualization result of

3.2. Importance Score of Keyframes

multiple video files. Various sizes are assigned to theThe technique then calculates importance score of each

keyframes according to their importance score, so larger

keyframes. While several existing video summarization

frames take users’ attention to the important scenes. A set of2the keyframes lined up horizontally along a time axis

fixing it according to the length of file. If it selects larger

represents a summary of one video file. Since each one

number of keyframes from a specific video, it eliminates lesssummary takes smaller space comparing to the existing

important keyframes to unify the number of displayedbrowsers for video summaries, our implementation can line

keyframes with other videos.

up and show more than one summary of video file at the same

Horizontal scroll

Vertical scroll

time.

(a)

Figure 2: Size-controlled keyframes of multiple video
files.
(b)3.4. Level of Detail Control
One of the main features of the visualization interface on
our implementation is realizing level of detail control by users’zooming operations. Here we define the level of detail of(c)video summary as the number of displayed keyframes per unit
time. The technique adjusts the number of displayed
keyframes by horizontal mouse operation. The number ofdisplayed keyframes increases according to zoom in operation,

Figure 3: Level of Detail Control: Keyframes of eachas shown in Figure 3(b), so that users can look the detail ofvideo(a) increases as (b) when they are zoomed in

the video contents. On the other hand, the number ofhorizontally. Keyframes are rearranged as (c) whenkeyframes decreases as users zoom out the summaries, so thatthey are zoomed in vertically.

they can look the simplified summary of the video.
While zooming out, the technique reduces the displaying

4. Example

keyframes by eliminating less important frames. The
technique automatically deal with the process, by comparing

Figures 2 and 4 are the results of implementation of thetwo adjoined keyframes that are timely close, and eliminating

proposed technique described in Section 3. As a user has a

the frame which has the less importance score. While

vertical zoom in operation, in the condition of Figure 2,zooming in, the technique revives the eliminated keyframes.

keyframes are rearranged to form two rows per video, asAs well as horizontal zoom operation, the techniqueshown in Figure 4. This interactive interface enables users tosupports vertical zoom operation. As users zoom out theget a glimpse at contents of multiple videos, or to consult thesummaries vertically, a set of keyframes per each video file isdetailed content of a video, depending on users’ purpose.

alternately lined up in two rows, above and under the time5. Conclusion and Future Workaxes, as shown in Figure 3(c). Placing keyframes in two rows
achieves an efficient visualization even when the number ofIn this paper we have presented a visualization interface

keyframes increases. Also, vertical zoom out operationforsimultaneously controls the number of displaying video file.

interactive level of detail control. We think the technique cansummarized

multiple

video

files,

which

realizes

Summaries of various videos often yield different numbers

assist users to easily understand or explore the contents ofof keyframes. To avoid gaps of the numbers of displayed

accumulated video files. Our study differs from many ofkeyframes between different videos, the technique unifies theexisting

number of keyframes per one video among multiple files, bysummarization of single video file, by concentrating on
3visualization

techniques

which

focusedon

[4] Y. Zhuang, Y. Rui, T.S.Huang, and S. Mehrotra,“Adaptive Key Frame Extraction Using UnsupervisedClustering,” Proceedings of ICIP ’98, Vol. I, pp.
866-870, Oct 1998.
[5] M. Smith, and T. Kanade, “Video Skimming andCharacterization through the Combination of Image andLanguage Understanding Techniques”, IEEE Computer
Society Conference on Computer Vision and PatternRecognition (CVPR'97), pp. 775-781, 1997.
[6] R. Hamada, K. Miura, I. Ide, S. Satoh, S. Sakai, H.
Tanaka, “A Motion Based Automatic Abstraction ofCooking Videos”, Meeting on Image Recognition andUnderstanding 2002 (MIRU 2002), Vol. 2, pp.203-208,
2002.
[7] B. Li, and M. I. Sezan, “Event Detection andSummarization in Sports Video,” Content-Based Accessof Image and Video Libraries, 2001. (CBAIVL 2001).
IEEE Workshop, Dec. 2001, pp. 132-138.
[8] S. Uchihashi, J. Foote, A. Girgensohn, and J. Boreczky,“Video Manga,” Proceedings of the Seventh ACM
International Conference on Multimedia, pp.383-392,
Orlando, United States, 1999.
[9] H. Luo, J. Fan, J. Yang, W. Ribarsky, S. Satoh,“Exploring Large-Scale Video News via InteractiveVisualization”, IEEE Symposium On Visual Analytics
Science And Technology (VAST2006), pp. 75-82, 2006.
[10] H. Zhang, C. Y. Low and S. W. Smoliar, “Video Parsingand Browsing Using Compressed Data,” Multimedia
Tools and Applications, Vol. 1, Issue 1, pp. 89-111,
March 1995.
[11] B. Yeo, and M. M. Yeung, “Retrieving and visualizingvideo,” Communications of the ACM, Vol. 40, Issue 12,
pp. 43-52, Dec 1997.

visualization of multiple video files.
Following are our potential future works. Firstly, we would
like to evaluate and discuss effectiveness of our importancecalculation technique, and consider new elements. Ourcurrent implementation calculates importance score only fromthe elements of the video itself, but the system will get more
effective visualization result if it can calculate while
comparing with other videos to concern differences andsimilarities between them.
We are also working on improvement on visualizing
keyframes effectively with less space. Current our system
places keyframes simply on their corresponding positions on
the time axis; however, the space utilization will be better if
the system adaptively modifies the positions on the time axis.
Also, we would like to enhance the presented browser as avideo player. For example, it is useful to support click
operations so that users can start the streaming from theclicked position of the videos.
Another important issue of our implementation would beapplying other kinds of videos. We are now working onvisualizing

abiologicalsimulation

videoofprotein

movement.

Figure 4: Keyframes lined up in two rows per video.

References[1] Y. Miyamura, T. Nakamura, K. Shinoda, S. Furui, “ShotBoundary Detection Using Multi-stage SVM”, Meetingon Image Recognition and Understanding 2007 (MIRU2007), IS-2-19, pp.815-820, 2007.
[2] K. Matsumoto, M. Naito, K. Hoashi, and F. Sugaya,“SVM-Based Shot Boundary Detection with a NovelFeature”. ICME 2006, pp. 1837-1840, 2006.
[3] Arnon Amir. The IBM Shot Boundary Detection
System at TRECVID 2003.
http://www-nlpir.nist.gov/projects/tvpubs/tvpapers03/IB
M_TREC03_SBD.slides.pd
4