Detection of Important Segments
in Cooking Videos
Reiko HAMADA† , Shin’ichi SATOH∗ , Shuichi SAKAI‡,
and Hidehiko TANAKA‡†
Graduate School of Engineering, The University of Tokyo∗
National Institute of Informatics‡
Graduate School of Information Science and Technology, The University of Tokyo

Abstract
This paper presents a method to delineate visually important segments from cooking videos, which is expected
to be a core technique for cooking video indexing. Visual
information is exceptionally important in cooking videos,
since it represents essential technical skills owing to properties of instruction type videos. Here, we reveal that ordinary
shot-based video indexing approach will not work for cooking videos, since visually important segments are absorbed
in a shot. Based on the observation of cooking videos, themethod detects repetitious motion segments as visually important segments by using relatively simple and robust techniques. The experimental evaluation shows that the method
detects more than 80% of manually selected segments, while
84% of detected segments were correct. We also developed
a cooking video abstraction system as a sample applicationof our method and confirmed that the proposed method isuseful for a real application.

1. Introduction
Following the increase of computational power and storage capacity, a large amount of TV broadcast video is transmitted and stored everyday. Recently, practical use of theseinformation has become an important subject. Now, manystudies are made on indexing broadcast video and multimedia data. Following this trend, we picked up the cooking
video as a target of video indexing by analysis[3].
In many cases, supplementary documents are available
along with cooking video. But, unlike other types of video,
such as news, visual information is exceptionally importantin cooking videos. In news video indexing, for example,
text information such as closed captions play a major role
to realize news-on-demand system, whereas visual information is used only for topic segmentation in most cases. On

the other hand, although cooking videos might also haverelated text information such as closed captions and corresponding cookbooks, visual information in cooking videos
is extremely important because it represents essential technical tips or skills of cooking procedures, due to properties
of instruction type videos.
Taking advantage of these visual information, automatedcooking video indexing will enable many practical applications, such as, cooking video retrieval system, cooking
video abstraction, or construction of multimedia cooking
contents through integration with cookbook. In the future,
the demand for such indexed cooking video will increase inproportion to the popularity of computers at home. For example, visual recipe on demand, multimedia cooking tips
catalogue, or visual cooking courseware will be very effective applications. In addition, applications in kitchen
environment, such as smart kitchen or automated cooking
system, would be examined using these digitalized cooking
recipes.
Typical approaches to video indexing generally separate
given videos into segments, i.e. shots. Shot boundaries(cuts) are the discontinuous points in an image sequence,
so they can be detected in relatively good accuracy. Then,in shot-based video indexing, image, audio and text analyses is applied to the videos to realize content-based accessto shots[1, 2].
Cooking videos, on the other hand, have peculiar features different from other videos. We will show that generalvideo indexing methods which segment videos into shots as
the smallest units for indexing are not suitable for cooking
video indexing. We will discuss about features of cooking
videos and introduce our method in the following sections.

2. Features of Cooking Videos
In this section, features of cooking videos are revealed
and indexing method suitable for cooking videos is dis-cussed.
As shown in Fig. 1, shots in a cooking video are categorized into (1)Face Shot, and (2)Hand Shot.

(a)Face Shot(b)Hand Shot

Figure 1. Shots categories in cooking video.

In a face shot, almost the whole kitchen is shown.
Though a teacher or an assistant explains about cooking
procedure, their motion and foods are too small in the picture. So, it is hard to obtain visual knowledge from faceshots.
On the other hand, hand shots are close-ups of tools
and/or hands while cooking something, and they have richvisual information. However, there are smaller segments ineach hand shot, such as, image of preparation steps, cooking
motion(s), and appearance of foods. A hand shot contains
important segments which are the keys of the recipe, andredundant segments between them as well.
Typical structure of a cooking program is shown in Fig.
2. Face shots and hand shots appear almost alternately.
Each hand shot has smaller structure, i.e. it contains smaller
important segments and relatively redundant segments.

video content analysis is essential for segmenting and indexing cooking videos.
A cooking program is a kind of an instruction video, andin most cases, there is a supplementary cookbook describing the same recipe as the video. Since users can always
refer to the recipe described in textbooks, unimportant steps
in the recipe are sometimes omitted in the video. However,cooking video is useful because it can provide visual information that is essential to instruct cooking procedure thattext can not express.
So, cooking motions and appearances of foods are particularly important in cooking video. Among them, we haveexamined cooking motions referring to the actual cooking
programs, and we found that most motions in the cooking
recipes are repetitious motions. Repetitious motions have
many corresponding verbs, namely, “cut”, “toss”, “mix”,“whip”, “grind”, and so on. Besides, there are some verbs
that can not be considered as repetitious motions, but are
repeated in actual cooking videos in most cases. It is because people tend to repeat important motion for emphasis.
Examples of repetitious motions are shown in Fig. 3. Basedon this idea, we detect important segments using time periodicity of repetitious motions.

Figure 3. Examples of repetitious motions.

Figure 2. Structure in cooking video.

3. Detection of Repetitious Motion
Due to the structure of cooking videos, shots are toocoarse units for indexing; i.e. face shot is not visually significant, and hand shot contains both significant segments andredundant segments. Considering this structure, it is crucial to delineate these important segments from hand shots.
So, shot-based methods are not suitable for cooking video
indexing.
In shot-based video indexing, shot detection is realizedby simple signal processing-based techniques, since shotboundaries are relatively abrupt changes in signal. However, important segments in hand shots do not have obvious
discontinuities as their boundaries. Although shot-based
video indexing is based on signal processing techniques,

In this section, our method to detect repetitious motions
is described.
Many studies on gesture recognition use skin color todetect hand regions. After that, hands are tracked and theirmotion is recognized[5, 6].
However, in cooking videos, hands are not always in the
picture, even in hand shots. Then, we need to detect repetitious motion of tools, such as spatula or chopsticks. Since
these tools have various color and shape, it is difficult to
identify their graphical features. Accordingly, methods using specific color such as skin color is not suitable for ourtarget.

As shown in Fig. 3, we assume that an object goesback and forth in a small region in the picture of important repetitious motion segments. In our method, we analyze time transition of brightness in small regions using
time frequency analysis, and detect repetitious motion by
testing existence of periodicity.
First, a frame is divided into small blocks, each of themconsist of 3 × 3 pixels (see Fig. 4). V x,y (t) is the mean
brightness of every pixel in a block, where x, y are the spatial coordinates of the block and t is the time coordinate of
the frame.

Figure 5. FFT graph.

First, the sum of power within the target frequency isgiven by:f
0 +N
P owerw,x,y =
F (f)
(2)
f =f0

Next, the maximum value of F (f) is given by Eq. 3.
The frequency that gives the maximum is defined as fp .
Figure 4. Image division and FFT method application.

As shown in Fig. 4, if x and y are fixed and t varies,
Vx,y (t) shows the time transition of brightness at x, y. Especially, if the object moves back and forth in the block,Vx,y (t) changes periodically. If there is such periodicaltransition at a certain number of blocks, the method regards
that a periodical motion exists in the segment at that time.
We applied FFT(Fast Fourier Transform) to Vx,y (t) at
every x, y, to examine the periodicity. The size of each window for FFT is T = 2n frames. We move this window byTstep frames and examine whether periodical motion exists
or not at each point of time. Equation of FFT is shown as
follows:
2−1
 1 T

Fw,x,y (f) =  √Vx,y (w × Tstep + t) · W f t  (1)
 T

t=0
2π

where W f t is e−j T , and Fw,x,y (f) = F (f) is the powerat frequency f. w is the window number, which shows thetime zone that the FFT was applied. w × Tstep is the timeoffset of the window.
If Vx,y (t) is periodical, the graph of FFT result shouldhave a clear peak at a particular f. To extract such a peak,
we introduce some statistical criteria. We observed that thetarget frequency is limited in proportion to the human motions. Suppose that the range of frequency is f 0 ≤ f <f0 + N (see Fig. 5).

F (fp ) =max

f0 ≤f <f0 +N

F (f)

(3)

Now F (fp ) is the maximum but not always the clear
peak of the graph. So, we define Fpeak (fp ), that shows
how F (fp ) stands out from the average. This is the ratio ofF (fp ) to the average of F (f) excluding F (fp ) (Eq. 4).
F (fp ) × (N − 1)
Fpeak (w, x, y) = f0 +N −1
f =f0 ,f =fp F (f)(4)

If F (fp ) is smaller than the power at low frequencies,
namely, F (1) and F (2), it is probably due to slow motion
which is not periodical. To exclude such cases, R1 and R2
are defined, which are the ratio of F (fp ) to F (1) and F (2)respectively. R1 and R2 must be large enough for periodical
motion.
When the periodicity of the motion is vague, the total
power becomes large but the peak in the graph does not
become so clear. So, we define the index of sharpness ofthe peak as shown in Eq. 5.
F (fp ) × 4
Rsharp = fp +2
f =fp −2,f =fp F (f)(5)

In the above, we introduced six parameters: P ower,
Fpeak , fp , R1 , R2 and Rsharp . They express the total powerand sharpness of the peak in the FFT graph. In this method,
existence of periodicity in the motion is examined using
these six parameters.
To put it concretely, the repetitious motion is detected in
the window w when every 6 parameters exceed thresholds
in more than 2 blocks. For example, if repetitious motion

is detected at w = w0 , the motion exists within the frame
range of Tstep × w0 ≤ t < Tstep × w0 + T .
If there is only one block that satisfies the above conditions, the segment is not regarded as a repetitious motion
because it is very likely due to noise. In addition, picturesare very noisy at the borders of the frame. So we excluded
15 pixels (5 blocks) from each edge from consideration.

4. Evaluation Experiment

method was compared with these correct answers, and theaccuracy was evaluated.
The result is shown in Tab. 3. The number of manuallydetected segments is AnsH , the number of automaticallydetected segments is AnsM and the number of commonly
detected segments is AnsC . Recall is AnsC /AnsH , andprecision is AnsC /AnsM .
Table 3. The result of evaluation experiment.
AnsH
AnsM AnsC
Recall Precision
62
59
50
80.6%
84.7%

4.1. Conditions
We made an evaluation experiment according to theprevious section. The target data was cooking video (34recipes, about 160 minutes in total) from a Japanese cooking program. Properties of the video is shown in Tab. 1.
Table 1. Properties of the target video.
total time160.1 minnumber of recipe
34
file format
mpeg2
size
720 × 480 pixels
frame rate
15 frame/sec.

The method detects repetitious motions that repeat morethan two times in a window. Then we empirically determined 32 frames (about 2 seconds) for the window size T .
The window is moved by T step = 16 frames (about 1 second). The limits of target frequency are f 0 = 3, N = 12.
Parameters on frequency analysis are given in Tab. 2(a).
As mentioned in the previous section, when six parameters shown in Tab. 2(b) exceed the thresholds at more than
2 blocks, the method regards the segment as repetitious motion.

As shown in Tab. 3, the number of false alarms is small,so our method can detect repetitious motions at more than
84% of precision. On the other hand, relatively large number of correct motions were not detected, and recall is about
80%. Examples of results of our method is shown in Fig. 6.

(a)cut cabbage(b)mix chicken with sauce
successful detection(c) roast meat(d) soup in a bowl
false detection

Table 2. Values and thresholds in the evaluation experiment.
(a)Values(b)Thresholds
T = 32frame
Tstep = 16frame
f0 = 3
N = 12

P ower > 500
fp > 5
R1 > 3

Fpeak > 50
Rsharp > 3
R2 > 34.2. Result and Analysis
Correct answers were given by detecting repetitious motion from the target video manually. The result of our(e) slap flours(f) fry leek
false rejection

Figure 6. Examples of results: detection ofrepetitious motions by our method.

First, Fig. 6(a), (b) show the typical examples of successful detection. Each of them is fast and periodical enough,and important motions which represent important tips ofcooking procedures. Examples of false alarms are shownin Fig. 6(c), (d). There are no human motions in Fig. 6(c),(d), but in Fig. 6(c), the periodical motion of chopsticks
swaying on the frying pan was detected. In Fig. 6(d), soupin the bowl might have been stirred just before this shot, sosoup is swaying periodically.
Fig. 6(e), (f) shows examples of false rejections. In Fig.
6(e), a woman tries to beat flour off a fish. The motion isperiodical, but too small. In Fig. 6(f), leek sticks are slowly
fried in a frying pan. Not only in this case, slow motions
tend to be relatively irregular, so sometimes whether thesemotions are repetitious motions or not was difficult to detect even for humans. Consequently, the result was unstable
when the motion seemed obscure.
It depends on applications whether or not the detected results are useful. Our observation shows that the results seemto fit with perception of humans. To show usefulness of theresults, we will provide an actual example of application in
the next section.

5. Application of Our Method
We developed a simple application to confirm our
method’s effectiveness. Generally, abstraction of the video
is not so effective because synchronization of audio and image is often lost and the splitted audio is too choppy[4].
However, audio in cooking videos can easily be compensated by recipe texts. In addition, since cooking videos
include visually important segments representing tips ofcooking procedures as described before, abstract cooking
videos can be composed with such segments.

Figure 7. Cooking video abstraction system.

The system includes the motion segments as well as thefirst and last segments in the resultant abstract video for ahand shot with motion segments. The middle of the shot is
included for a hand shot without motion segment.
We observed that the resultant abstracted cooking videos
will surely provide information on important visual tips aswell as necessary steps of cooking procedures.

6

Future WorksFirst, the accuracy of our method should be improved.
The current system can hardly detect slow motion, so adaptively stretching analysis window for slow motion segments
should be considered. It might be effective to combine ourmethod with object detection and tracking.
Next, important motions which are not repetitious should
also be detected for more comprehensive cooking video
indexing. Frequency of these motions is relatively small,but sometimes there are non-repetitious motions peculiar
to each recipe. These motions have no specific features incommon, so it is not easy to distinguish them from unimportant motions such as “put the dish on the table”. However,important motions are supposed to be shown in the middleof the picture for a relatively long time. In this case, location and velocity of the center of the motion balance areuseful.
Appearances of foods are important information forsome motions such as “dish up”. In cooking video, sometimes appearances of foods are shown in still images before
or after the cooking actions. If those segments are extracted
and put together with the important motions, it turns out that
almost all important segments are extracted.
The abstraction system introduced in this work can be
improved by detecting such important segments. In addition, finer indexing to the video would be realized by integrating the cookbook, in order to achieve a database ofdigital cooking recipes. By doing this, fine grain cooking
video retrieval can be made; i.e. retrieving not only each
recipe but also each step or motion.
In the future, these results can be applied to the actual
cooking environment. For example, we can realize a “smart
kitchen,” equipped with several sensors to trace the current
step within the cooking procedure, which can instruct theuser what to do next or how the food will appear.

7. Conclusion
A snapshot of the cooking video abstraction system isshown in Fig. 7.
In this case, the system applies cut detection to given
cooking videos first, then hand shots are selected by excluding face shots. In doing this, face regions are detected basedon skin color information. Details are omitted due to page
limit, but a similar approach can be found in [7].

The paper proposes a novel approach for cooking video
indexing. We revealed that shot-based methods do not fit to
the structure of cooking video, and we decided to employ
important segments in a shot for indexing.
In this paper, repetitious motions are regarded as typicalimportant segments, and the automatic detection method forthem was proposed. The effectiveness of our method wasshown by the experiments.
We constructed a cooking video abstraction system as an
example application to show that our approach is practicalfor actual applications.

References[1] Y. Ariki, “Multimedia Technologies for Structuring and Retrieval of TV News,” New Generation
Computing, Vol.18, No.4, pp.341-358, 2000.
[2] A. G. Hauptmann and H. D. Wactlar, “Indexing and Search of Multimodal Information,” Proc.
ICASSP’97, April 1997.
[3] R.Hamada, I. Ide, S. Sakai, and H. Tanaka, “Associating Cooking Video with Related Textbook”,
Proc. ACM Multimedia 2000 Workshops, pp.237241, Nov. 2000.

[4] M. Christel, M. Smith, C. Taylor, and D. Winkler, “Evolving Video Skims into Useful Multimedia Abstractions,” Proc. of ACM CHI’98 Conference on Human Factors in Computing System,
April 1998.
[5] T. Nishimura, T. Mukai, and R. Oka, “Spotting Recognition of Gestures Performed by People
from a Single Time-varying Image,” Proc. of HCIInternational ’97, pp. 33, Aug 1997.
[6] C. Wren, B. Clarkson, and A. Pentland, “Understanding Purposeful Human Motion,” IEEE Intl.
Conf. on Automatic Face and Gesture Recognition, April 2000.
[7] C. Wren, A. Azarbayajani, T. Darrell, and A. Pentland, “Pfinder: Real-Time Tracking of the Human
Body,” IEEE Trans. PAMI, Vol. 18, No.7, pp. 780785, July 1997.