Motion Based Automatic Abstraction of Cooking Videos
Koichi MIURA
Reiko HAMADAThe University of Tokyo
7-3-1 Hongo, Bunkyo-ku,
Tokyo, 113-0033, Japan
TEL: +81-3-5841-7413
FAX: +81-3-5800-6922

Ichiro IDE
Nat’l Institute of Informatics
2-1-2 Hitotsubashi,
Chiyoda-ku,
Tokyo, 101-8430, Japan

ide@nii.ac.jp

miura,reiko@mtl.t.utokyo.ac.jp
ABSTRACTIn this paper, we propose a method to abstract cooking
videos. We deﬁne cooking video abstraction as shrinking
videos maintaining the understandability of general cooking
procedures visually and intuitively. Cooking motions andappearances of foods are considered as important segments
in a cooking video. A method to extract such importantsegments referring to the intensity of motion in an image is
proposed, and eﬀectiveness of the method is shown throughevaluation experiments. We also developed a cooking video
abstraction system that assembles important segments detected by the proposed method and repetitious motions thatis especially important among cooking motions. The resultant abstracted videos were about 1/8 to 1/12 of the originalvideos in time, maintaining the understandability. The validity of the abstraction method was checked by comparing
the abstracted videos with manually abstracted videos provided from a broadcasting station.

1.

INTRODUCTION

Following the advance in telecommunication technology,
large amount of multimedia data has become available frombroadcast video. Multimedia data analysis is becoming important to store and retrieve them eﬃciently. However, characteristics of videos and purposes of viewers vary among different kinds of videos. Thus, it is necessory to limit the domain of the target video and refer to domain speciﬁc knowledge, for high level content analysis.
We chose cooking video as a target, and aim for building
a practical system with high accuracy using domain speciﬁcknowledge[5, 8]. Cooking is familiar to daily life, and thedemand for semantically structured cooking videos shouldgrow in proportion to home (especially kitchen) automation.
In this paper, we propose a method to abstract cooking
videos. A cooking program is a kind of an instruction video
that people view from a practical point of view. On the other

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
Copyright 2002 ACM X-XXXXX-XX-X/XX/XX ...$5.00.

Shuichi SAKAI
Hidehiko TANAKAThe University of Tokyo
7-3-1 Hongo, Bunkyo-ku,
Tokyo, 113-0033, Japansakai,tanaka@mtl.t.utokyo.ac.jp

hand, cooking videos include many redundant segments suchas chatting, which requires the viewers a certain amount oftime to view. Therefore, to select recipes and to actuallycook in daily life, a cookbook tends to be easier to browse.
However, since videos contain visual information that text
cannot express suﬃciently, it is more eﬀective to understandthe cooking procedures.
Thus, we abstract cooking videos to easily browse throughrecipes. We deﬁne cooking video abstraction as shrinking
videos maintaining the understandability of general cooking
procedures visually and intuitively.
Various studies have been done on automatic video abstraction for news, documentary, and so on[4, 6]. These
kinds of videos are relatively not redundant, and abstracted
videos tend to be utilized for selecting a segment in theoriginal videos. Works similar to ours are in sports video
abstraction. A work on sports video abstraction[7] refers to
an external database corresponding to video contents. Thismethod is eﬀective for videos that could easily refer to external data corresponding directly to their contents, which
is diﬃcult in cooking videos.
On the other hand, it is reported that abstracted videos
are not so eﬀective since split audio accompanying the abstracted images is too choppy[4]. From this point, audio
naturalness is considered in abstracting videos[6, 9], andaudio is usually replaced with new narrations in abstracted
videos provided from a broadcasting station. However, sincemotion and preparation steps can be roughly understoodsolely from visual information in cooking videos, outline of
the cooking procedure could be understood to a certain extent without audio. Thus, our abstraction does not consider
audio continuity, and abstracted videos are created solely
referring to image features.
Considering the above mentioned issues, cooking video
could be considered as a good target for abstraction.

2.

FEATURES OF COOKING VIDEOS

2.1

Structure of cooking videos

As shown in Fig. 1, shots in cooking videos could be categorized into (a)face shot, and (b)hand shot, which appear
almost alternately, as shown in Fig. 2.
In face shots, almost the whole kitchen is shown. Thougha teacher or an assistant explains a cooking procedure, theirmotions and foods are too small to obtain visual informa-tion. On the other hand, hand shots are close-ups of tools
and/or hands while cooking something that have rich visual
information. However, each hand shot contains sub-shotsegments such as cooking motion and appearance of foods
that are important keys to understand the recipe, but also
redundant segments between them as well.

(a)Face shot

4. Sum up the length of optical ﬂow vectors detected in
all pixels (320×240) in each frame. (=S)
5. Take the average of the sum (S) of every 10 frames inorder to reduce mal-eﬀects from noise. (=S)
Cut detection is performed by applying the DCT clustering method[1], and shots are classiﬁed by detecting faces.
Faces are detected based on skin color information, shape,and so on, similar to the method described in [3].
Temporal transition of S of a part of an actual video
is shown in Fig. 3. From this graph, important segments;(a)cooking motion and (b)appearance of foods, are detected.

(b)Hand shot

Figure 1: Shot categories in cooking video.

Figure 2: Structure of cooking video.

2.2

Important segments in cooking videos

In abstracting cooking videos which have the structure asshown in Fig. 2, ﬁrst, face shots are excluded since they havelittle visual information. Secondly, redundant segments inhand shots are excluded. Here, we consider that (1)important visual information that text cannot express suﬃciently
and (2)essential information to understand cooking procedures visually, should be included.
Condition (1) appears as two kinds of video segments.
One is (a)cooking motion. Tips of cooking motions cannot
be easily understood without visual information. Anotheris (b)appearance of foods, such as color of prepared ingredients and state of a dish. Cooking videos include static
segments without motions in order to show the state. Whenan abstracted video includes these segments, it is consideredthat it also satisﬁes condition (2).
In this work, therefore, we extract (a)cooking motion and(b)appearance of foods, to abstract cooking videos. Referring to actual cooking programs, we have examined thatthese segments have the following motion-related features.
(a) cooking motion: big (intense)(b) appearance of food: almost none

3.

IMPORTANT SEGMENT DETECTION

3.1

Motion based detection

In order to extract important segments considering the
motion-related features, we need to detect motion in theimage. Here, optical ﬂow is employed to detect the motion since we need to refer to direction, speed, and so on.
Among many techniques proposed to detect optical ﬂow, we
employed Horn and Schunck’s method[2]. This relatively
simple method was employed since our purpose requires only
rough motion detection.
The following procedure is taken for motion-based important segment detection:
1. Detect cuts and classify shots into (a)face shot, and(b)hand shot.
2. Exclude (a)face shots.
3. Detect optical ﬂows from (b)hand shots.

Figure 3: Temporal transition of the sum of thelength of optical flow vectors (S) in each frame.
First, S is the sum of the length of optical ﬂow vectors ina frame, and S is the average of S of every 10 frames. Saveis the average of S in a shot. Smove , Sstate1 , and Sstate2 arethresholds used for detection.
A segment that satisﬁes S ≥ αSave in a shot that satisﬁes
Save > Smove is judged as cooking motion (α is a constant).
This means that a relatively active segment in a shot witha big motion is judged as cooking motion.
As for appearance of foods, a segment that satisﬁes S <Sstate1 for more than T sequential frames, or a segment thatsatisﬁes S < Sstate2 in a shot that satisﬁes Save < Sstate2
is detected. This means that continuous inactive segments
and an inactive segment in a shot that has almost no motion
are judged as appearance of foods.

3.2

Camera motion exclusion

The above method misdetects also camera motions as
cooking motions since only the sum of the length of optical ﬂow vectors is referred to. Therefore, camera motion
detection is necessary to exclude the misdetected segments.
Camera motions in cooking videos are categorized intopanning (a translational motion of a camera) and zooming(a zoom up and down motion of a camera). Panning isobserved when a camera moves from an object to an object,
the purpose of which is not to show cooking motions nor
appearances of foods. It is the most common camera motion
in cooking videos and also the main cause of misdetection
since there is usually no important visual information in
the image. On the other hand, when zooming occurs, an
important object usually appears in the center of the image.
As a result, it can be neglected.
Thus, we concentrate on detecting and excluding panning.
The following procedure is taken to detect panning simply
referring to optical ﬂow vectors:
1. Calculate directions (angles, 0 ≤ θ(i, j) < 2π) of optical ﬂow vectors in all pixels p(i, j) of frame f . Weightthem with the length of vectors v(i, j), and calculate
the frequency of vectors by directions. This forms an“angle histogram”(Hf = { hf (Θ) | 0 ≤ Θ < π }).
Here, if the angle of a vector is π ≤ θ(i, j) < 2π, it isweighted as −v(i, j) at Θ = θ(i, j) − π.
1 δΘ (i, j) · v(i, j)(1)hf (Θ) =S i j
 1 ( if θ(i, j) = Θ )−1 ( if θ(i, j) = Θ + π )where, δΘ (i, j) =
0 ( otherwise )T = 90 (3 seconds), Fth = 0.025. They were deﬁned basedon preliminary observations.
The result of the experiment is shown in Tab. 1. Correct segments were judged and deﬁned manually. The number of correct answers of automatically detected segments
is NC , misdetection is NM , and oversights is NO . Recall is
NC /(NC + NO ), and precision is NC /(NC + NM ).

2. Sum up and take the average of the histograms(H ={ h(Θ) | 0 ≤ Θ < π }) as long as the motion could beconsidered continuous (during frames f1 to f2 ). Continuous motion is detected by referring to S. Since
motions which have opposite directions are canceledmutually, random noise and cooking motions do notappear much in an “angle histogram”, and panning is
detected eﬀectively.
f21h(Θ) =
hf (Θ)(2)f2 − f 1 + 1

As shown in Tab. 1, the proposed simple method is effective to detect important segments. The main cause of
misdetections of cooking motions and oversights of appearances of foods was detecting an unimportant motion, suchas a motion not related to cooking.
As a result, approximately 40% (7 out of 17) of misdetections of cooking motions previously made without detecting camera motion (panning) were eliminated applying themethod in 3.2.

f =f1

As the result of this procedure, the angle histogram shows
one clear peak in the case of panning, as shown in Fig. 4(a),and it shows no clear peak when there is no panning, asshown in Fig. 4(b). Taking advantage of this feature, whenFp is the peak value of the angle histogram and Fth is acertain threshold, a segment that satisﬁes Fp > Fth with
only one peak is considered as panning, and excluded fromimportant segments.

(a) Panning.

(b) Without panning.

Figure 4: Angle histogram.

3.3

Figure 5: Repetitious motions.

Repetitious motion detection

Although the above proposed method detects cooking motion and appearance of foods as important segments, more
important segment detection is necessary for more eﬀective
abstraction. We have examined cooking motions referring toactual cooking videos, and found that repetitious motions asshown in Fig. 5 are more important than others. Therefore,
we detect repetitious motions following the method that focus on time periodicity[8] and reﬂect the result in the abstraction.

3.4

Table 1: Result of important segment detection.
Important segments N C NM NO Recall Precision
Cooking motion
117
10
2 98%
92%
Appearance of foods 39
2
7 85%
95%

4.

VIDEO ABSTRACTION

4.1

Motion based abstraction

We developed an application to abstract cooking videos
using the method described in 3.
In each hand shot, the ﬁrst segment of repetitious motions
is extracted if it includes any. If not, or in segments morethan 10 seconds apart from repetitious motions, the ﬁrst segment of cooking motions detected by the proposed method is
extracted. Additionally, the last segment of appearances offoods is extracted. These segments are assembled in chronological order. Note that duration of each segment is set to
2 seconds, and shot classiﬁcation is manually done.
An example of the resultant abstracted video is shownin Fig. 6. In Fig. 6, repetitious motions((4)∼(7), (11))
have rich visual information that text cannot express sufﬁciently, such as strength and speed of motion. Next, othercooking motions((2), (9), (10)) enable us to easily understand the cooking procedure visually. Finally, appearancesof foods((1), (3), (7), (9), (12)) include important visual
information and make cooking steps clear.

Evaluation: important segment detection

We made an evaluation experiment that detects importantsegments; cooking motions and appearances of foods. The
target cooking videos consist of 6 recipes, with the length ofapproximately 40 minutes in total, taken from a Japanesecooking program.
In this experiment, shot classiﬁcation was manually done.
Cooking motions and appearances of foods were detectedbased on the method in 3.1, and panning was excluded basedon the method in 3.2. Thresholds used in this experimentwere Smove = Sstate2 = 10, 000, Sstate1 = 7, 000, α = 1.0,

Figure 6: Video segments abstracted from a cooking video (Black frame: Repetitious motion, Grayframe: Cooking motion, White frame: Appearanceof foods).

The abstraction method was applied to 9 recipes of 3 television programs from diﬀerent broadcasting stations. The
number of extracted segments for abstraction in each program for abstraction is shown in Tab. 2. The number ofrepetitious motions is NR , other cooking motions is NM ,and appearances of foods is NA.
Table 2: Number of extracted segments for abstraction.
Program
Program 1
Program 2
Program 3
Total

# of
recipe
4
2
3
9

NR

NM

NA

23
8
8
39

23
23
70
116

18
7
2
27

Abstraction rate
(in time)∼ 1/11∼ 1/10∼ 1/9∼ 1/10

The abstracted videos were approximately 1/8 to 1/12 of
the original videos in time. Nonetheless, we observed thatthe abstracted videos surely provide important visual tips
as well as inevitable cooking steps that enable us to easily
understand the cooking procedure.

4.2

Evaluation: video abstraction

Program 3 in Tab. 2, accompanies a manually abstracted
video at the end of the program as “today’s review”. Thus,
we compared the 3 manually abstracted videos in program
3 with our result.
First, as for the duration of the abstracted videos, manually abstracted ones had a ﬁxed length (40 seconds), andthe average of ours was 53 seconds.
Next, contents of both videos were compared. The resultof comparison in number of extracted segments is shown in
Tab. 3. Since our videos contain no audio, only visual information was compared. The number of segments in manually abstracted videos is SegH , segments in our videos is
SegM , and commonly detected segments is SegC . Recall is
SegC /SegH , and precision is SegC /SegM . Although there
were some segments which represent the same visual information in our videos, they were regarded as one segment in
the comparison.
Table 3: Comparison in # of extracted segments.
RecipeRecipe 1
Recipe 2
Recipe 3
Total

SegH
12
13
11
36

SegM
20
24
16
60

SegC
11
13
10
34

Recall
92%
100%
91%
94%

Precision
55%
54%
63%
57%

As shown in Tab. 3, it turns out that segments in manuallyabstracted videos were mostly extracted by the proposed
method with high recall. Actually, segments which cannot
be extracted by our method were only 2.
On the other hand, precision was not so high. Although
importance of most additional segments by our method whichshowed appearances of foods and basic cooking motions islittle, they were necessary to represent cooking procedures
visually. At the same time, some of the contents of thesesegments were supplemented by newly inserted captions andnarrations in manually abstracted videos.
Tab. 4 shows the result of a rough comparison including
the contents represented by captions and narrations. From
this result, better precision is obtained.
However, manually abstracted videos are “reviews” for

Table 4: Comparison of extracted segments(including caption and narration contents).
RecipeRecipe 1
Recipe 2
Recipe 3
Total

SegH
17
18
12
47

SegM
20
24
16
60

SegC
16
18
11
45

Recall
94%
100%
92%
96%

Precision
80%
75%
69%
75%

viewers who have already watched the original videos. This
purpose is slightly diﬀerent from ours; shrinking videos suﬃcient to understand general cooking procedures visually andintuitively.
As for the duration of abstracted videos, we will improve it
by setting the most suitable time for each segment, adjusting
to the user’s skill, and so on, as a future work.

5.

CONCLUSIONSThis paper proposed a motion based abstraction method
for cooking videos. A method to detect important segments referring to the intensity of motion in the image wasproposed, and its eﬀectiveness was shown through evaluation experiments. Additionally, we also developed a cooking video abstraction system that assembles important segments detected by the proposed method and repetitious motions. The resultant abstracted videos were about 1/8 to
1/12 of the original videos in time, maintaining the understandability of cooking procedures. The validity of the abstraction method was checked by comparing some automatic
abstracted videos with manually abstracted videos provided
from the broadcasting station.
In the future, we will develop a more eﬀective abstraction
system. For example, if the abstraction rate is adjustable,
the application will be more useful. Therefore, we will further investigate on classifying and prioritizing the cooking
motions depending on their importance to understand thecooking procedure.

6.

REFERENCES[1] Y. Ariki and Y. Saito. Extraction of TV news articles basedon scene cut detection using DCT clustering. In Proc. Intl.
Conf. on Image Processing, pages 847–850, 1996.
[2] B. K. P. Horn and B. Schunck. Determining optical ﬂow.
Artif. Intel., 17:185–203, Aug. 1981.
[3] C. Wren, A. Azarbayajani, T. Darrell, and A. Pentland.
Pﬁnder: Real-time tracking of the human body. IEEE
Trans. PAMI, 18(7):780–785, July 1997.
[4] M. Christel, M. Smith, C. Taylor, and D. Winkler. Evolving
video skims into useful multimedia abstractions. In Proc.
CHI’98 Conf. Human Factors in Computing Systems, pages
171–178, 1998.
[5] R. Hamada, I. Ide, S. Sakai, and H. Tanaka. Associatingcooking video with related textbook. In Proc. ACM
Multimedia 2000 Workshops, pages 237–241, Nov. 2001.
[6] R. Lienhart, S. Pfeiﬀer, and W. Eﬀelsberg. Video
abstracting. Commun. ACM, 40:55–62, 1997.
[7] N. Babaguchi, Y. Kawai, and T. Kitahashi. Generation of
personalized abstract of sports video. In Proc. ICME2001,
pages 800–803, Aug. 2001.
[8] R. Hamada, S. Satoh, S. Sakai, and H. Tanaka. Detection ofimportant segments in cooking videos. In Proc. IEEE
Workshop on CBAIVL 2001, pages 118–123, Dec. 2001.
[9] Y. Gong, X. Liu, and W. Hua. Creating motion video
summaries with partial audio-visual alignment. In Proc.
ICME2002, pages 285–288, Aug. 2002.