NICOGRAPH International 2014, pp.

A Method of Automatic Video Synthesis for Comparing
the Player Movements in Ski Competition
Tatsuya OKUYAMA1)

Takafumi KOJIMA1)

Masanori KAKIMOTO1)
1) Tokyo University of Technology

Tomokazu ISHIKAWA1)

Tomoyuki NISHITA2)

2) UEI Research / Hiroshima Shudo University

G311400735@edu.teu.ac.jp

Abstract
Records of sporting event are being updated every year. We are sure that clarifying the difference between
other player movements can assist athletes who want to make a new record. However, it takes t ime and requires
plenty of user efforts to synthesize two video sequences to clarify differences of motions. We address this problem and propose a method to realize an automatic video synthesis. The proposed method provides a new video
representation for not only sport athletes but also spectators. Although our ultimate goal is to provide a tool for
live coverage of a broad range of sports, we focus, in this research, on broadcast videos of alpine ski competitions. In the proposed method, we first make a rough image adjustment using mask images corresponding to theterrain covered with snow. Then the system synthesizes the two input videos of competing skiers by detecting
the gates or flags on the course and aligning the image position precisely for each frame. For exact timeline
matching of the two videos, we used the lap time shown as a TV caption. As a result, we were able to compound
two videos with a desired timing and position for each frame. A future work is to cancel errors of the synthesizing positions caused by the difference of camera zooming ratios in shooting two competing players of a similar
lap time range.

1. Introduction

2. Previous work

Recently, video synthesis and image composition have been

Video synthesis has been actively researched. Rüegg et al. [1]widely researched in the field of Computer Graphics (CG),proposed a method for cutting out two videos in a seam in orderwhich has facilitated the production of interesting visual prod-

to synthesize the videos. This approach can be applied in manyucts. We predict that, by clarifying differences between thecases moving, there is no need for special movie shooting

movements of athletes, it will be possible to compare player’equipment such as chroma keying. However, rather than being

motion and optimize motion in each sport easily. Convention-

fully automatic, two videos are combined by a user’s stroke on

ally, images are processed frame by frame and composed man-an object that is shown in both videos. Because a synthesizedually. However, composing images is difficult because video

video is completely different through the boundary, the video issynthesis is a time-consuming process. In addition, there areadjusted by color blending near the boundary. While we set

physical limitations in implementing cameras and setting up

three restrictions that are described later in this research, theapparatuses, such as chroma key composition, because suchproposed method, i.e., brush stroke, is not introduced, rather,devices can obstruct players in sport competitions. In this re-the synthesis is performed automatically.

search, we focus on alpine skiing. We assume that players hop-In research involving separating moving objects from a back-ing to set a new record by improving their motion, and video

ground, Ohta et al. [2] separated a movie into foreground andproducers who need interesting visual effects in presenting thebackground by classifying the directions of movement calcu-sport will be the primary users of our tool. We aim to realize alated using optical flow [4]. Optical flow is widely used to trace

fully-automatic video synthesis, without physical limitations

objects [5, 6]. However, it has some problems. Estimation using

due to the camera during the shooting, for use in live coverage.

optical flow can yield false recognition in case of animation

In video synthesis, two videos overlap to form one

wherein intensity barely changes. Further, the method in whichsemi-transparent video.

images are divided into a grid and feature points are comparedexhibits a low processing speed. In the method proposed by

NICOGRAPH International 2014, pp.
tion alignment technique is described in Section 3.4.

Figure 1. Example of identical objects.
Asaoka et al. [3], which uses robust statistics [7, 8] that fix errors due to outliers using a novel weighted least squares method,moving objects were separated accurately, and the backgroundFigure 2. Processing flow.

was estimated from sequential images by determining correspondence among feature points dispersed in the sequential
images. However, in our research, corresponding feature points
in videos must be extracted because the composition is the samebut one is two different videos and they are not the sequentialimages.

Original image

3. Proposed method

Mask image

Figure 3. Example of mask image.

3.1 Target videos
We set the following three restrictions on videos rather thansetting camera limitation.
(a)Videos are filmed at the same angle.

(b)Videos have a time indicator.

(c)Videos capture the same objects.

(a) is necessary because measurement at different angles
causes emptiness. (b) is necessary to determine synthesis frames.
(c) is necessary to perform position adjustment using the sameobject (エラー! 参照元が見つかりません。). We selected

Figure 4. Example of lap time display as a caption.

3.2 Preprocessing of videos

alpine skiing because this sport fulfill all three conditions re-Sports videos are large file size. Therefore, we resize videos

quired for synthesizing video. The movement of alpine skiers,

to manage them with greater efficiency. We calculate a mask

and consequently, the panning speed of the camera, is consid-image (Figure 3) using binarization after applying a medianerably fast. Therefore, detecting the same objects is difficult.

filter to clear noise, excluding the object from the filter.

Track events are often filmed using a stationary camera and acamera with slow pan speed. Therefore, if we can synthesize

3.3 Timeline matching

alpine ski videos, our method should be applicable to otherIn order to achieve an exact comparison between two playerssports. In this research, we use videos of the 2010 Winterof a speed competition, we need to synchronize their lap times

Olympic in Vancouver available on YouTube [9].

in synthesizing the motion pictures of the players. Fortunately,

Figure 2 shows a flow chart of our method. The input data arein relay broadcast of an alpine ski competition, they always

two sets of video sequence (video A and B), each of which rep-

provide a lap time displayed as a caption at a corner. Our sys-resents a competing skier for a similar range of lap time. Thetem uses the lap time for the timeline matching between the twooutline of the flow is to adjust the frame timing of video B, toinput video sequences of competing skiers. Each number in thealign each frame of video B, and then to synthesize video A andcaption is recognized with a template matching method [10-12]

B with some translucency. The frame timeline matching is ful-using small rectangular templates for numbers 0 to 9.

filled using lap time recognition for each video, and described

Since the place for the caption is fixed in a relay broadcast,in Section 3.3. The image position of each frame in video B is

we focus on a specific region of interest to recognize the lapaligned using two steps of object matching between the frame B

time for each frame, as shown by a red ellipse in Figure 4. Our

and the corresponding frame from video A. A detail of the posi-image synthesis is then carried out for frames with the same lap

NICOGRAPH International 2014, pp.
time in the two input videos.

Video Athe poles used in slalom events. For giant slaloms, a couple ofVideo BFigure 5. Mask median points for images from two input videos.
(a)

(b)

Figure 7. Extracted feature points.
feature points are extracted using color template matching technique for the gates. As described previously, the search range islimited within the mask region, excluding out-of-course objects
such as trees or spectators. Then our system compares the twoFigure 6. Masks of two input images after a rough shifting operation.
The white area represents the difference of the two mask images. The red rectangle is the ROI used for the differenceevaluation for image shifting.

sets of the extracted feature points from the two video sequences. For each feature point, we find the nearest feature point in
the paired image, and evaluate the total distance accumulating
for each corresponding feature point pair. Finally, the precisealignment vector is found while minimizing the total distance.

3.4 Image position alignment
In our method, each pair of timeline-matched frames from the4. Results

two input videos is aligned by extracting and recognizing sever-To implement our method, we used C++ and OpenCV for CPU

al corresponding feature points. The alignment process is di-programing on a standard PC (CPU: Intel® i7-4770 CPU 3.40GHz,vided into two steps, which are a rough shifting operation using

RAM: 12.0 GB). All videos used in the experiment had a resolu-

a terrain region as a mask, and a more precise alignment using

tion of 1280 × 720 pixels, a frame rate of 30 fps, and duration ofobjects on the terrain.

10 s. The calculation time is 1 min 38 s, 80% of which involved

In order to exclude objects and spectators out of course, wethe calculation of feature point extraction. We show the com-use the mask images generated in preprocessing as described inparison of processing time in case of the proposed method and aSection3.3. In addition, the mask image is used to align the two

manual method using "Adobe After Effect" in Table1. Exami-frames to synthesize. First, for each image (A and B), a mediannees for synthesis in manual were 10 students studying video

point is computed for the mask or the snow terrain region asprocessing technique and they studied beforehand operating

represented in white in Figure 5. Then one of the compared

procedures to obtain a quality equivalent to our synthesis result.

images (Image B) is shifted such that the two median points

We have visually checked the quality of synthesis result pro-coincide with each other. The image is further shifted within a

duced in manual. We can see that the processing time of ourcertain limited bound, minimizing the difference of the mask

method is shorter than the average time of manual method.

images. To appropriately evaluate the difference, we focus on aTable 1. Processing time comparison

region of interest (ROI), which is simply the overlapping rectangular region after the shift operations. Within the ROI, therelative difference is computed by dividing the different area

Processing time
1min 38sec
2min 51sec

Our method
Manual average(number of pixels) by the ROI area. Figure 6 describes an example of the ROI area (red rectangle) and the different area
(white area in the ROI).

(a)

(b)

After roughly shifting the image, the system proceeds to aFigure 7 shows results of feature point extraction by template

more precise alignment step, which finds corresponding objectsmatching. In Fig. 7 (a), the red bounding square indicates a flagin the two images incorporating a simple feature extraction

extracted as feature point in case of input video A. Similarly, inmethod. For the feature extraction we use gates or poles on theFig. 7 (b), the blue bounding square is shown in case of input

course of the alpine ski competition. A set of gates used in giant

video B. We can see that the feature points are extracted accu-slalom events consists of red and blue ones. The same is true for

rately, and the size of the frame is also supported.

NICOGRAPH International 2014, pp.
Figure 8 shows the results of synthesis video based on the ex-among the frames, although the position adjustment was com-

tracted feature points. Figs. 8 (a) to (c) are three sequential im-

pleted successfully. It is difficult to apply the proposed method

ages from 48 frames to 50 frames, (d) to (f) are three sequential

in case where there is a significant difference in the zoom fac-

images from 148 frames to 150 frames. In Figs. 8 (a) to (c), wetor.

succeeded in clarifying the movements of the athletes using thesynthesis videos because the positions of the flag in each frame

5. Conclusion

match. The 3 frames shown in Figs. 8 (d) to (f) have some mi-

In this research, using sports videos, we realized fully automatic

nor deviations caused by the difference in the zoom factor

video synthesis by template matching. By combining the video of two

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8. Synthesized results.

players, it is possible to compare the players’ movements and suggest[6]

R. Okada, Y. Shirai, J. Miura, Y. Kuno. "Object Tracking

better motion or technique. The method is also beneficial for people

Based on Optical Flow and Depth" The IEICE Transac-

who watch sports because it allows for interesting video effects.

tions. vol.80, no.6, 99.1530-1538, 1997.

Future work will involve increasing the precision of matching and[7]Y. Sato. "Recent Progress on Using Statistical Models indecreasing the speed of calculation so that the proposed method can belmage Processing: Robust Estimation and Minimum De-

used in live broadcast. Furthermore, because the position of the feature

scription Length Criterion" Medical imaging technology.

point is shifted if the zoom rate is different among the frames being

vol.12, no.1, pp.48-58, 1994.

compared, it is necessary to correct the resulting distortion. We will

[8]

consider straight-line detection as a universal feature value using theOutlier Detection" Wiley Series in Probability andHough transform [13], because we plan to apply the method to competitions other than skiing.

P. J. Rousseeuw, A. M. Leroy. "Robust Regression andMathematical Statistics, 1987.

[9]"olympicvancouver2010" YouTube.
http://www.youtube.com/user/olympicvancouver2010.

References[1]J. Rüegg, O. Wang, A. Smolic, M. Gross. "DuctTake:[10] T. Kaneko, O. Hori. "Update Criterion of Image Template

Spatiotemporal Video Compositing" Computer Graphics

for Visual Tracking Using Template Matching" TheForum, vol.32, no.2, pp.51-61, 2013.
[2]

[3]

[4]

IEICE Transactions. vol.88, no8, pp.1378-1388, 2005.

N. Ohta. "Structure from Motion with Confidence Meas-[11] Y. Shinohara, N. Funabiki, J. Kawahima, J. Takeuchi, M.

ure and Its Application for Moving Object Detection" TheIshizaki. "A License Plate Recognition Algorithm Using

IEICE Transactions, vol.76, no.8, pp.1562-1571, 1993.

Cross Count and Template Matching Methods" IEICE

T. Asaoka, N. Yokoya, H. Takemura, K. Yamazawa.

Technical Report. vol.104, no.573, pp.39-44, 2005.

“Motion Detection from Image Sequences with a Mov-[12] K. Arai, K. Morimoto, H. Yamana. "Similar Object De-ing Camera Using Robust Statistics” IEICE Technicaltection Using Template Matching Focused on PositionalReport. vol.96, no.492, pp.33-40, 1997.

Relationship of Feature Regions" The Special Interest

J. L. Barron, D. J. Fleet, S. S. Beauchemin. "PerformanceGroup Technical Reports of IPSJ, CVIM. vol.172, no.4,of Optical Flow Techniques" International journal ofpp.1-8, 2010.

computer vision. vol.12, no.1, pp.43-77, 1994.
[5]

2013-01-14.

[13] D. H. Ballard. "Generalizing The Hough Transform toT. Iwasaki, T. Yokoyama, H. Koga, T. Watanabe. "Hu-Detect Arbitrary Shapes" Pattern recognition. vol.13, no.2,

man Extraction Using Optical Flow in Complex Back-pp.111-122, 1981.

ground" IEICE Technical Report. vol.104, no.669,
pp.73-77, 2005.