The Visual Computer manuscript No.
(will be inserted by the editor)

Yaron Caspi · Anat Axelrod · Yasuyuki Matsushita · Alon Gamliel

Dynamic Stills and Clip Trailers

methods: (i) Dynamic still - a single image conveying theactivity by combining information from diﬀerent temporal moments; (ii) Video clip trailer - a few seconds long
video that displays the clip’s highlights. We believe thatsuch a mechanism will greatly enhance many domains
where visual summaries are desired such as online video
libraries or news web pages (e.g., [11, 12,14]).
Current video libraries use text, a single image (usually the clip’s ﬁrst frame), or a set of key-frames for depicting the content of videos. Since the diﬀerentiating
factor of videos from images is motion, we believe thatthe expressive power of text and key-frames is limited.
Thus, our goal is embedding and visualizing dynamics
in a concise representation, while maintaining the clarKeywords Video Summaries · Key-Pose Selection
ity and sensation of motion.
This work focuses on clips of human subjects. Videoobjects are extracted from diﬀerent time-frames, and
1 Introduction
several poses that best represent the activity captured
in the video are concurrently displayed. Technically, weDiﬀerent types of media have associated promotion mech- treat a video clip as a three-dimensional (3D) volume
anisms. Books have their covers, and movies have their with the spatial axes X, Y and temporal axis T . To retrailers. In both examples, promotion is coupled with duce the inherent spatial redundancy in the video, we
content information. In this work, we develop a similar cut out the foreground objects from this volume [17, 25]
mechanism for an emerging type of medium - video clips. and compute alpha matting values [24]. We further reWe develop means for eﬃciently extracting and visual- duce temporal redundancy of the foreground data by
izing the content of short video clips using two display automatically selecting key-poses from the sequence of
extracted objects, utilizing a new algorithm which takes
Yaron Caspi
shape information as input. This data reduction allows
Faculty of Mathematics and Computer Science
The Weizmann Institute of Sciencehandling of previously neglected scenarios. Unlike synopE-mail: yaron.caspi@weizmann.ac.il
sis mosaics [15] that only treat videos of objects translating from side to side, we also consider self-occluding acAnat Axelrod
School of Computer Science
tivities (Figure 2). The resulting output from the above
Tel Aviv University
process is a set of key-poses selected from segmented obE-mail: anataxel@post.tau.ac.il
jects. These poses with associated alpha masks are deYasuyuki Matsushita
noted as pose slices and can be rendered in two diﬀerent
Visual Computing Groupways. First, they can be composed into a single static
Microsoft Research Asia
image that we call a dynamic still. Second, they can
E-mail: yasumat@microsoft.com
form a few second long video presenting the essence ofAlon Gamliel
the activity. We term this representation a clip trailer.
School of Computer Science
Tel Aviv University
We provide a set of manual and automatic tools that
E-mail: gamliela@post.tau.ac.il
enable non-professional video editors to generate dynamic
Abstract We propose a method for generating visualsummaries of video. It reduces browsing time, minimizes
screen-space utilization, while preserving the crux of thevideo content and the sensation of motion. The outputs
are images or short clips, denoted as dynamic stills or
clip trailers, respectively. The method selects informative
poses out of extracted video objects. Optimal rotations
and transparency supports visualization of an increased
number of poses, leading to concise activity visualization. Our method addresses previously avoided scenarios, e.g., activities occurring in one place, or scenes with
non-static background. We demonstrate and evaluate themethod for various types of videos.

2Fig. 1 “Ascending and descending stairs”. Visualizing anactivity Eadweard Muybridge, 1887.

stills and clip trailers. These include: rotation and zoomof the entire set of pose slices and/or each pose slice independently, and use of transparency for adding spatialor temporal context and for emphasizing particular activities. Our visual content summaries have a range ofapplication areas, such as web sites where users download video clips or cellular phone download centers. Summaries might also be used as static or “breathing” thumbnails on a PC, where several clip trailers are played automatically in an icon size display. We illustrate these applications using a variety of clips from home and sports
videos.
The rest of this paper is organized as follows. Section 2 reviews prior work. Section 3 discusses motivating
observations. Section 4 reviews the technical details of
our method. Results and applications are discussed in
Sections 5 and 6. Section 7 concludes the paper.

2 Related work

Yaron Caspi et al.

(a) Non self-occluding(b) Self-occluding

Fig. 2 Self-occluding vs. non self-occluding activities. While
synopsis mosaics work for non-self occluding activities (a),they face diﬃculties with self-occluding activities (b). Our
approach handles both cases (see Figure 12).

Methods for visualizing dynamic scenes have beenstudied not only for producing still images such. Agarwala et al. [3] and Rav-Acha et al. [22] manipulated
both space and time to generate a wide ﬁeld-of-view
panoramic video. Interactive artistic manipulations ofsuch video volumes are implemented in the Khronos Projector [9]. Unlike this work, the above methods do not
reduce the size of the represented data, and most of themeven increase it for producing the new representation.
Since our focus is on video objects rather than video
frames, we employ a video object cutout process. We
have many options in choosing a video object cutout algorithm. Interactive approaches such as [17, 28] are both
useful for a cluttered background or a non-stationary
camera. Alternatively, if the camera is static and thebackground is also static, recently proposed automatic
methods [13, 25] may be applied. Segmented objects werealso used for the purpose of interactive video browsing [6]. However, standard image and video outputs are
not supported there, and user interaction is required to
select highly informative key-poses.

Representing motion and activity has been a challenging
problem faced by many visual art masters and scientists.
Photography entrepreneurs such as Muybridge, Mareyand others developed multiple-exposure techniques forvisualizing human activities (Figure 1). Advances in computer graphics and computer vision allowed spatially stitchOur video processing pipeline involves an analysis ofing several images together to create a “panorama” (e.g., the pose motion for the purpose of key-pose selection.
[26]). Panoramas have been used for representing videos. This problem has been studied in the context of video
For example, Taniguchi et al. [27] combined shot de- summarization and retrieval, traditionally achieved bytection and panoramas to generate a mixed catalog of key-frame selection to approximate the task of key-pose
key frames and panoramas. Irani et al. [15] superim- selection. Recently, key-pose selection algorithms that foposed copies of foreground objects taken at diﬀerent time cus on video objects rather than video frames are gainpoints on a background panorama. To avoid overlaps be- ing more attention. Loy et al. [19] computed a frame-totween diﬀerent copies, the video is diluted by sampling frame distance based on contour matching, while Liu et al.
frames uniformly or manually [20]. Recently, an approach [18] used distance information obtained from motionof stroboscopic visualization of movement based on in- capture data. Both methods cluster frames and select theteractive digital photo-montages was proposed by Agar- cluster centers as key-frames. Assa et al. [4] suggested
wala et al. [2]. Similarly, Chiu et al. [10] suggested ar- that a concise and informative synopsis of a human acranging blobs extracted from key-frames in a collage rep- tivity is better estimated by selecting the extreme poses
resentation, as a way to improve screen utilization when of the motion. In this work, we follow Assa et al. ’s apvisually summarizing business meetings. However, the proach of selecting extreme key-poses and extend it to 2D
above processes are only eﬀective for non self-occluding video sequences. While Assa et al. ’s work mainly conactivities, and fail when objects appear at the same im- sidered tracking of skeleton joints usually obtained fromage location more than once (Figure 2). Our framework motion-capture data, we achieve a similar goal only fromhandles this temporal self-occlusion problem.
2D foreground images.

Dynamic Stills and Clip Trailers

3(a) Video volume(b) Key-frames(c) Object segmentation(d) Pose slices

Fig. 3 Diﬀerent video visualizations. (a) A video visualized
as space-time volume. (b) A key-frames representation. The
representation is highly redundant and lacks the sensation of
motion (c). A space-time volume of the moving object. This
representation does not suﬀer from background redundancy,
but still hides the activity details. Our dynamic still representation in (d) combines a single background frame with aset of key-poses to concisely depict the activity.

Fig. 4 Components of our framework. First, the input
video volume is decomposed into foreground and background.
Then, the motion of extracted objects is analyzed to produce
key-poses and rendering instructions. These are used to compose the ﬁnal visualization.

4 Technical Description
3 Approach
The main challenge in generating an informative visualsummary is dealing with the huge redundancies inherent
in video. This section discusses several fundamental observations regarding video clips. Based on these observations, we then brieﬂy describe the proposed framework.
(1) Background has a huge temporal redundancy. This is
illustrated by the complete video volume in Figure 3(a).
This observation leads to today’s approach of key-frame
representation, in which temporal redundancy is reduced(Figure 3(b)).
(2) Foreground is more attractive and informative thanthe background. In Figure 3 the walking person is clearly
the most important portion in the scene. Furthermore,
the background contributes mostly as an atmosphere,
and its details are not crucial for understanding the scene.
Consequently, we cut the background oﬀ from the volume (Figure 3(c)), and place it once as the backgroundof the ﬁnal output.
(3) Not all poses have the same importance. Poses atthe beginning and the end of each consecutive motion
are more important than intermediate poses.
(4) Background and foreground are not very sensitive
to small rotations from the standard “head-on view”.
Hence, we encourage manipulations of the viewing angle. A pleasing result is illustrated in Figure 3(d). Notethat our display size is on the same order of a single
video frame’s. Accordingly, the essence of the video clip
is depicted by decreasing the redundancy in both background and foreground, and by choosing important posesof foreground that we call pose slices.

The proposed processing steps are illustrated in Figure 4. First, the input video volume is separated into
foreground and background. Then, the motion and pose
information of extracted foreground objects is analyzed,
and highly informative key-poses are selected. Prior tocomposition, we compute an optimal viewing angle thatminimizes the mutual occlusion of diﬀerent poses. The
composition step allows the author to deﬁne a set of rendering instructions to control the visualization eﬀects.
For producing dynamic stills, the visualization parameters such as the viewing angle and key-pose opacity are
ﬁxed. For a clip trailer, these parameters are time dependent. The next sections describe the major steps used inthis process.

4.1 Video decomposition
The video decomposition begins with extracting foreground objects. Recently, we witnessed signiﬁcant progressin this ﬁeld. Li et al. [17] and Wang et al. [28] describedinteractive approaches, and showed that it is possible
to segment complicated scenes with a user’s guidance.
Sun et al. [25] and Criminisi et al. [13] described methods of foreground extraction in real time for scenes withstationary background. Our method is not constrained toa particular video segmentation method, and the choicedepends on the complexity of the scene. This is illustrated in Figure 5. For the simple stationary backgroundin (a) the method of [25] was used, while for the cluttered scene in (b) the method of [17] was used. Aftervideo object segmentation, we apply an alpha matting

4

Yaron Caspi et al.

(a)(a) Dissimilarity matrices(b)(b) Uniﬁed matrix(c) Motion curve

Fig. 6 Motion extremum analysis. Several dissimilarity matrices (a) are merged into a single uniﬁed matrix (b). Poses
are then embedded into a low-dimensional space. The cuspsof the resulting motion curve (c) correspond to extremum
poses of the activity (wide-open/closed legs).
(c)Fig. 5 Segmentation and scene complexity. The simple scenein (a) was segmented automatically in realtime, while thecluttered scene in (b) was segmented interactively. Final results are shown in (c).

algorithm, crucial for achieving a high-quality visualization. In our implementation, we use the Poisson matting
algorithm [24].
4.2 Motion extremum analysis
This section describes an automatic method for selecting key-poses. The input is a sequence of segmented instances of foreground objects from all the frames in thevideo, denoted here as poses. This deﬁnes a curve in thehuge space of all poses. We argue that important poses
are extremum points on this curve, since activity is best
illustrated by changes in motion. Such extremum points
simplify the interpolation process for completing the activity representation. Thus, we call our approach motion
extremum analysis. The output of this module is “poseslices”. Figuratively, we slice the 3D blob of the foreground object’s data and select poses with high importance for depicting activity.
Our method for motion extremum analysis is composed of the following steps: (i) Computing several poseto-pose dissimilarity measures organized in dissimilarity matrices. These matrices are denoted Mdi , where
di is the i-th dissimilarity measure. Each matrix is ofsize F × F where F is the number of poses extracted
in the segmentation step (Figure 6(a)). (ii) Combining
these measures to produce an F × F uniﬁed dissimilarity matrix (Figure 6(b)). (iii) Embedding poses on a
low-dimensional sphere, forming a motion curve (Figure 6(c)). The analysis of this motion curve enables theevaluation of the importance of poses. The correspondence between extremum poses and motion curve cusps
is illustrated in Figure 6(c).

Inter-pose dissimilarity measures. Pose-to-pose dissimilarity is measured using features computed for each pairof foreground silhouettes: (i) Orientation of the majoraxis, (ii) Elongation, (iii) Solidity, (iv) Shape overlap,
and (v) Centroid position. Other features were tested,e.g., the velocity of the silhouettes’ centroids, and theradii of the silhouettes’ circumcircles centered at the centroids. However, these were discarded due to the small
weights they were given during the uniﬁcation process(see below).
(i) Orientation of the silhouette’s main axis is computed
using Principal Components Analysis (PCA) on the set S
of 2D pixel coordinates inside the silhouette. The orientation dissimilarity Md1 (i, j) between the poses in frames
i and j can be deﬁned as:
Md1 (i, j) = 1 − |cos αij | ,where αij is the angle between the main axis of pose iand that of pose j.
(ii) Elongation ri of pose i, is measured by the ratio of
the pixel distribution’s variance along its primary andsecondary axes in the eigen-space. It is measured by
the ﬁrst and second eigenvalues of the silhouette covariance matrix. This property diﬀerentiates thin elongatedshapes from circular shapes. The elongation dissimilarity
between poses i and j can be deﬁned as:
Md2 (i, j) = |log(ri /rj )| .
(iii) The solidity of a foreground object is deﬁned as theratio between the area inside the silhouette and the areaof its bounding box. This measure is useful for detecting
the extension of limbs, the waving of hair or clothes, etc.
The solidity dissimilarity Md3 (i, j) is deﬁned similarly to
Md2 (i, j) by substituting ri and rj with solidity values.
(iv) Shape overlap is computed using the absolute correlation [8] between pairs of registered silhouettes. Registration is achieved by aligning the silhouettes’ centroids.
The absolute correlation is minimized over a small searchDynamic Stills and Clip Trailers

5

radius R to reduce the sensitivity to subtle segmentation
errors:|Si (x + dx, y + dy) − Sj (x, y)|,Md4 (i, j) = mindx,dy(x,y)∈B

where B is the bounding box surrounding both silhouettes.
(v) The position dissimilarity is deﬁned as the Euclidean
distance between silhouettes’ centroids. A few other measures (velocity, radius and area) were also tested. These,
however, did not contribute to the uniﬁed representation of all dissimilarity measures (next paragraph), andtherefore have been omitted.
Unified dissimilarity matrix. A uniﬁed dissimilarity matrix M is computed, aggregating the diﬀerent inter-pose
dissimilarity matrices Mdi . We start by converting eachdissimilarity matrix into a corresponding cross-productmatrix Pdi , as in multidimensional scaling (MDS), andnormalizing each matrix by its ﬁrst eigenvalue. An aggregation matrix is computed as a linear combination of
the normalized matrices, and is then converted back into
the uniﬁed dissimilarity matrix M . The weights for the
linear combination are derived from a non-centered PCAof the Pdi matrices [1]. This gives an “optimal” solution
in the sense that measures which agree the most witheach other contribute the most to the uniﬁed dissimilarity matrix.
A low-dimensional motion curve. Based on the uniﬁed
dissimilarity matrix, we embed the sequence of input
poses into a low-dimensional space. For this task, we
use the spectral clustering method [21], a method designed for segmentation. It applies eigen-decomposition
to a normalized version of a dissimilarity matrix, denoted
as an aﬃnity matrix. In our case, the aﬃnity between
poses is derived from their dissimilarity, as computed in
the previous step. A spectral clustering method is applied
to the temporally ordered sequence of points (poses). To
incorporate this additional information and to avoid theinﬂuence of temporally distant poses, we multiply theentries of the aﬃnity matrix M (i, j) by an exponential
decay factor e−|i−j|/δ , where δ is a constant which indicates that human motion segments are usually a few
seconds long, e.g. 3 seconds. The output of the eigendecomposition is a set of column eigenvectors orderedby their eigenvalues. The ﬁrst k eigenvectors deﬁne an
F ×k matrix, where each pose i is associated with the ith
row of this matrix. In theory, spectral gap may be used

Fig. 7 A dynamic still showing key-poses selected from a 2D
animated sequence.

Fig. 8 Global rotation (a) and (b) is used to address occlusions between poses. (c) The optimal viewing angle is recovered by aligning the global motion-plane (yellow) with the
major diagonal of the video volume, as indicated by an arrow.

to select k. In practice, a signiﬁcant gap rarely exists.
Therefore, global measures such as stress [16] should be
used to select k. This set of F row-vectors are normalized to unit length and placed on the S k unit sphere [21].
The advantage of this process is that it attempts to preserve local distance, thus similar poses are represented byadjacent points on the sphere. The advantages and relations of this approach to other embedding techniques(e.g., [23]) are discussed in [7].
Pose slicing. To select key-poses, we follow the approachof Assa et al. [4]. We iteratively choose poses which arelocally extreme (distant from their temporal neighborhood). This is illustrated in Figure 6(c), where the sharp
corners of the motion curve are selected and indeed correspond to extremum poses of the original motion. We
compared our approach with Assa et al. ’s method byrendering one of their animation sequences (Figure 7).
Although our method does not take the explicit 3D motion data as input, it is able to generate a virtually equivalent result only from 2D silhouette data. This completes
the video decomposition process.

4.3 Display composition
Once pose slices are selected, they are optimally visualized using their shape information and positions in thespace-time video volume. This section introduces our approach for eﬃciently visualizing multiple pose slices in alimited screen space.
Global viewing angle. When visualizing multiple pose slices
at the same time, self-occlusion of poses becomes one of
the major problems. In our approach, self-occlusion of
the pose slices is minimized by rotating the space-timevideo volume from the head-on view to a new viewing

6

Yaron Caspi et al.

(a) Object only(a)(b) Object with spatial context
Fig. 10 Spatial context. (a) Only pose slices are visible; (b)shows the table that the samurai jumped from.
(b)
Fig. 9 After global rotation (a) some pose slices might appear ﬂat. This is resolved by an automatic local rotation (b).

angle (Figure 8(a) and (b)). This rotation approach has
an advantage; the orientation of the background planegives a visual cue as to the direction of the time axis.
An optimal viewing angle is computed automatically according to the global direction of the motion observedin the video volume. It can be computed by collecting
3D positions of foreground pixels throughout the video
volume and ﬁtting a 2D plane that minimizes the sumof perpendicular distances from the points. We implement this approach using PCA. The resulting 2D plane
is denoted the global motion plane.
To minimize the deviation from the original viewing
angle, we align the global motion plane with the closest
diagonal of the 3D video volume. Figure 8(c) illustrates
the global motion plane, and the recovered global rotation (marked in yellow). Note that this rotation depends
on the spacing between poses. Namely, stretching thetime axis reduces inter-occlusion and results in a smaller
rotation.

secondary rotation angle is automatically computed andapplied (Figure 9 (b)).
Spatial context. In some video clips, a better understanding of the portrayed activity can be achieved by incorporating spatial context into the visualization. This isaccomplished by visualizing additional “secondary” objects that appear next to the foreground object. These
secondary objects can be extracted in the video object
segmentation phase. Alternatively, image portions aroundthe foreground object can be visualized by manipulating alpha values to create a gradual fade-out eﬀect (Figure 10). In our case, alpha values are assigned using asigmoidal function of the city-block distance d from theforeground object’s boundary:
1

d−η ,
1+e σ
where η determines the width of the spatial context, andσ controls the strength of the fade-out eﬀect. η and σ
can be tuned interactively.

α(d) =

Temporal context. Another option to enrich the visualexperience is to add pose slices. This, however, mightcause a self-occlusion problem. Therefore, we add the

Local viewing angle. The global viewing angle is optimized to provide the most informative visualization. However, a global rotation of the space-time volume results
in a non-uniform viewing angle for diﬀerent poses, andmay cause some pose slices to appear ﬂat (Figure 9(a)).
A secondary rotation, we denote as the local viewing angle, is used to compensate for this ﬂattening artifact. Our
approach is motivated by visualization methods that use
a cylindrical (or spherical) coordinate system (e.g., [26]).
In contrast to a planar display, a cylindrical surface gives
an orthogonal projection of surface points to a camera.
At the same time, we wish to place poses along the Zaxis, in order to maintain relative positions of key-poses. Fig. 11 Temporal context. The transparency of additional
Therefore, we only rotate each pose slice about its cen- poses is based on their importance. Most informative poses
tral axis to face toward the camera. In this way, the (i.e., motion extreme points) are completely opaque.

Dynamic Stills and Clip Trailers(a) Key frames

7(b) Synopsis mosaic(c) Our representation

Fig. 12 Comparison of diﬀerent video summaries. We compare the results of our visualization with previous methods using
three diﬀerent examples. The top row illustrates that our method can address more elegantly non static background, anda scene with multiple objects. The middle row illustrates how our method depicts a self-occluding activity. The third row
compares summaries of a long home video.

poses (a) selectively using the key-pose selection results;(b) using transparency. Figure 11 displays an example ofthe temporal context eﬀect. The number of poses to be
added can be controlled interactively.

5 Discussion and evaluation
5.1 Beneﬁts
The advantages of using our visualization are evident
from the comparison of diﬀerent methods shown in Figure 12. It is clear that our method captures the sensationof motion that is lost in a key-frames representation. Dueto its better screen utilization, it provides a comprehensible visualization that allows a quick acquaintance with
an eﬃcient orientation in video sequences. It is also evident that previously proposed methods (i.e., synopsis
mosaics) cannot address the above scenarios.
In the presence of large camera motions, our methodshould be applied after registration of the video sequence.
The resulting space-time video volume is not a rectangular box. However, all the previously described steps
apply in such cases as well. It is also interesting to notethat a regular superposition of objects (e.g., [2, 15]) isa special case of our method. Namely, when the object
translates from side to side, the global rotation is eﬀectively 0◦ , and the resulting dynamic still is essentially adynamic panorama.

5.2 Limitations
The eﬀective range of clips that our current implementation can address depends on the number of poses neededin order to describe a given activity. From our experience,the proposed visualization allows for about a dozen poses
to be viewed simultaneously (i.e, in the same screenspace as a single frame). Obviously, our visualization canbeneﬁt from panoramic display and can depict longer
activities. Other packing problems exist in the presenceof multiple objects. While multiple objects may be displayed (see for example, Figure 12(c) top), there may becases where rotations of distinct objects will occlude eachother. Finally, the trade-oﬀ for the better screen utilization is that the exact spatial information of an activity
is lost. The global rotation slightly shifts the locations ofpose slices. An example where this may cause confusion
is shown in Figure 13; although the girl was walking inplace, it is perceived as forward motion.

5.3 User Studies
To verify the contribution of our representations we ran
two user studies. One user study measured whether theclip trailers deliver accurate content information in ashorter time, and the other evaluates whether they wereattractive.

8

Yaron Caspi et al.

clip trailers is in the domain of video media centers, suchas video download web sites, Video-on-Demand interactive TV channels, or even video information and entertainment services featured on cellular phones. In Figure 12, we compare a key-frames representation of video
clips as shown on a commercial web site with our innovative approach.
Breathing thumbnails. Another straightforward application of clip trailer technology is breathing thumbnails.
Breathing thumbnails stands for a dynamic visual representation of home-users’ PC folders containing video
clips. This representation is built by concurrently displaying miniaturized clip trailers summarizing the video
clips contained in the folder. It can be constructed manually or automatically if video segmentations are provided. An example of breathing thumbnails is illustrated
in Figure 14.
Fig. 13 This dynamic still illustrates the limitations of depth
ordering. While the person in the ﬁgure is walking in place,
it is perceived as if she is moving forward.

Evaluating informativeness. Thirty users were asked toﬁnd unique events in video sequences or in the respective summaries. They are also asked to answer simple
questions about them (e.g., describe the boy’s emotions,who failed in returning a tennis serve). For each of thegiven 6 tasks, we tested the correctness of the answers
and measured the time needed to complete the task. The
results indicate that: (i) The fraction of correct answers
was similar using both visualizations; (ii) The responsetime using the proposed video summaries was a few seconds (less than ﬁve), while using regular videos it had
a linear dependency in the temporal length of the video
clip, and was generally longer; (iii) Training with a single
video summary signiﬁcantly improved the success rate.

Informative time line and indexing. Locating a speciﬁcclip from a clip archive is a diﬃcult task, especially forvideos with long and rich contents. One needs to tediously use the rewind/fast-forward buttons in order tounderstand what the clip is all about. On the other hand,
clip trailers can show the major events in a video clip ina very informative way. Furthermore, our motion analysis that captures the key-poses may be used to generate
an informative timeline. As illustrated in Figure 15, an
informative timeline is constructed by rotating the poseslices to a 90◦ view angle, with the secondary rotation
making them all nicely visible1 . This technique, com1
The data was obtained from mocap.cs.cmu.edu. The
database was created with funding from NSF EIA-0196217.

Evaluating attractiveness. One hundred and forty users
were asked to select the most attractive video clip out of
six options. Three of these were visualized by clip trailers
and the other three were still images extracted from thevideos. To overcome the inherent bias to particular video
clips (e.g., clips that consisted of attractive objects), theselection of clip trailer or image for each video was reversed for half of the participants. The results indicatethat in ﬁve out of six cases clip trailers were more attractive. Further details may be found in [5].

6 Applications and results
This section shows several results of our visual video
summaries and illustrates the applicability of our video
previewing techniques for diﬀerent applications.
Video clips promotion. Clip trailers contribute to promotion of video clips. Therefore, a major application for

Fig. 14 Breathing Thumbnails. Iconized clip trailers can replace conventional static thumbnails. Such moving (“breathing”) thumbnails are more informative than the clip’s ﬁrst
frame that is currently used on regular PC’s.

Dynamic Stills and Clip Trailers

Fig. 15 Informative timeline.

bined with breathing thumbnails, simpliﬁes the browsing
process in a video archive.

7 Summary and future work
We have described means for an eﬀective visualizationof video clips. Our visualizations, dynamic stills and clip
trailers, provide a better utilization of the display space,
and therefore also a better utilization of time. Furthermore, our framework allows us to address scenarios thatso far have been neglected (e.g., the presence of selfocclusion). The eﬀectiveness of our approach is illustrated qualitatively by a large number of examples. It
is also evaluated quantitatively with two user studies.
In the future, we plan to address some derived issues.
First, video object segmentation remains the most time
consuming part of our system. We would like to speed
up this part, by combining proximity relations between
coherent segments. Second, we wish to combine key-pose
selection with occlusion constraints. By coupling the keypose selection process with the resulting visualization,
the screen space to information ratio and visual quality
may be improved. This could be achieved by enforcing a
new constraint on the key-pose selection algorithm.

References
1. Abdi, H., Valentin, D., O’Toole, A.J., Edelman, B.: Distatis: The analysis of multiple distance matrices. In:
EEMCV Workshop, pp. 42–47. San Diego, CA, USA(2005)
2. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S.,
Colburn, A., Curless, B., Salesin, D., Cohen, M.: Interactive digital photomontage. ACM Transactions on Graphics (SIGGRAPH) 23(3), 294–302 (2004)
3. Agarwala, A., Zheng, C., Pal, C., Agrawala, M., Cohen,
M., Curless, B., Salesin, D., Szeliski, R.: Panoramic video
textures. ACM Transactions on Graphics (SIGGRAPH)
24(3), 821–827 (2005)
4. Assa, J., Caspi, Y., Cohen-Or, D.: Action synopsis: Pose
selection and illustration. ACM Transactions on Graphics (SIGGRAPH) 24(3), 667–676 (2005)
5. Axelrod, A.: Msc thesis. Dept. of Computer Science Tel
Aviv University (July, 2006)
6. Axelrod, A., Caspi, Y., Gamliel, A., Matsushita, Y.: Interactive video exploration with pose slices. Sketches,
SIGGRAPH (2006)

9

7. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15(6), 1373–1396 (2003)
8. BenAbdelkader, C., Cutler, R., Davis, L.: Gait recognition using image self-similarity. EURASIP Journal on
Applied Signal Processing 15(4), 572–585 (2004)
9. Cassinelli, A., Ito, T., Ishikawa, M.: Khronos projector.
Emerging Technologies, SIGGRAPH (2005)
10. Chiu, P., Girgensohn, A., Liu, Q.: Stained-glass visualization for highly condensed video summaries. In: Proceedings of the IEEE International Conference on Multimedia
and Expo (ICME), pp. 2059–2062. IEEE, Taipei, Taiwan(2004)
11. CMU: Motion capture database, cmu graphics lab.
http://mocap.cs.cmu.edu/ (2002)
12. CNN: Cable news network. http://www.cnn.com/video/(2004)
13. Criminisi, A., Cross, G., Blake, A., Kolmogorov, V.: Bilayer segmentation of live video. In: IEEE Conference on
Computer Vision and Pattern Recognition, pp. 53–61.
New York, NY, USA (2006)
14. Google:
Googlevideo
sharing
service.
http://video.google.com/ (2005)
15. Irani, M., Anandan, P., Hsu, S.: Mosaic based representations of video sequences and their applications. In: International Conference on Computer Vision, pp. 605–611.
Washington, DC, USA (1995)
16. Kruskal, J.: Multidimensional scaling by optimizing
goodness of ﬁt to a nonmetric hypothesis. Psychometrika 29, 1–27 (1966)
17. Li, Y., Sun, J., Shum, H.Y.: Video object cut and paste.
ACM Transactions on Graphics (SIGGRAPH) 24(3),595–600 (2005)
18. Liu, F., Zhuang, Y., Wu, F., Pan, Y.: 3D motion retrieval
with motion index tree. Computer Vision and Image
Understanding 92(2-3), 265–284 (2003)
19. Loy, G., Sullivan, J., Carlsson, S.: Pose-based clustering in action sequences. In: Workshop on Higher-Level
Knowledge in 3D Modeling & Motion Analysis, pp. 66–
72. Nice, France (2003)
20. Massey, M., Bender, W.: Salient stills: Process and practice. IBM Systems Journal 35(3/4), 557–573 (1996)
21. Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering:
Analysis and an algorithm. In: Advances in Neural Information Processing Systems, pp. 849–856. MIT Press,
Cambridge, MA (2001)
22. Rav-Acha, A., Pritch, Y., Lischinski, D., Peleg, S.: Dynamosaicing: Video mosaics with non-chronological time.
In: IEEE Conference on Computer Vision and Pattern
Recognition, pp. 58–65. San Diego, CA, USA (2005)
23. Roweis, S., Saul, L.: Nonlinear dimensionality reduction
by locally linear embedding. Science 290(5500), 2323–
2326 (2000)
24. Sun, J., Jia, J., Tang, C.K., Shum, H.Y.: Poisson matting.
ACM Transactions on Graphics (SIGGRAPH) 23(3),315–321 (2004)
25. Sun, J., Zhang, W., Tang, X., Shum, H.Y.: Backgroundcut. In: European Conference on Computer Vision, pp.
628–641. Graz, Austria (2006)
26. Szeliski, R., Shum, H.Y.: Creating full view panoramic
image mosaics and environment maps. ACM Transactions on Graphics (SIGGRAPH) pp. 251–258 (1997)
27. Taniguchi, Y., Akutsu, A., Tonomura, Y.: Panoramaexcerpts: extracting and packing panoramas for video
browsing. In: MULTIMEDIA: Proceedings of the ﬁfth
ACM international conference on Multimedia, pp. 427–
436. ACM Press, New York, NY, USA (1997)
28. Wang, J., Bhat, P., Colburn, R.A., Agrawala, M., Cohen,
M.F.: Interactive video cutout. ACM Transactions on
Graphics (SIGGRAPH) 24(3), 585–594 (2005)

10

Yaron Caspi et al.

Yaron Caspi Yaron Caspi received the BSc degree in mathematics and computer sciencefrom the Hebrew Universityof Jerusalem in 1991 and the
MSc degree in mathematics
and computer science from theWeizmann Institute of Sciencein 1993. From 1994-1999, heworked at several computervision companies. He received
the PhD degree in 2003 fromthe Weizmann Institute of Science, working on sequence-tosequence alignment. For his
PhD research, he received theIsraeli Knesset (Israeli parliament) outstanding student award. He received the best paper
award at ECCV 2002, and the honorable mention at ICCV2001. He is currently at Weizmann Institute of Science.

Anat Axelrod Anat Axelrod is an M.Sc. student at
the Computer Science schoolof Tel-Aviv University and ateaching assistant at the Interdisciplinary Center Herzliya.
She received her B.A. degree in
Computer Science from the Interdisciplinary Center Herzliya
in 2002. Her research interests
are computer vision and video
visualization. She is also interested in visual arts and had
created illustrations for books.

Yasuyuki Matsushita Yasuyuki Matsushita received theBEng, MEng, and PhD degrees in electrical engineeringfrom the University of Tokyoin 1998, 2000, and 2003, respectively. Currently, he is a researcher in Visual Computing
Group at Microsoft Research
Asia. His research interests include photometric methods incomputer vision and video improvement algorithms. He is amember of IEEE.

Alon Gamliel Alon Gamliel
received the BSc degree in
Computers Engineering fromthe Technion, Israel Instituteof Technology, in 2001. He iscurrently an MSc student anda teaching assistant at TelAviv University. His researchinterests are computer graphics, video segmentation andmatting techniques.