Summarization of EgocentricMoving Videos for Generating
Walking Route Guidance
Masaya Okamoto and Keiji Yanai
Department of Informatics,
The University of Electo-Communications,Tokyo, Japanⓒ 2013 UEC Tokyo.

Background• Spread of wearable camera– Easy to take egocentric video• New application of egocentric videos

Automatic generation of route guidance videoⓒ 2013 UEC Tokyo.

Objective
Generate walking route guidanceby summarizing egocentric moving videos
Raw Video taken by wearable camera
・Too long to watch ・Boring

Our System• summarizing

Route Guide Video・Easy to understand walking route ・Very helpfulⓒ 2013 UEC Tokyo.

Demo (Raw Video)

RAWⓒ 2013 UEC Tokyo.

Demo (Result Video)• Control playing speed dynamicallyⓒ 2013 UEC Tokyo.

Related Work: Summarization
egocentric video• Tancharoen et al. [ACM SIGMM 2005]– Cues: GPS and other sensors– Target: Life-log video ( everyday life )– Output: Set of important frames• Lee et al. [CVPR 2012]– Cues: Visual features– Target & Output: same as the above• Ours– Cues: Visual information– Target: Walking video– Output: Summarized videoⓒ 2013 UEC Tokyo.

Overview1. Ego-motion classification
2. Crosswalk detection
On-line3. Estimation of importance
4. Calculation of playing speedⓒ 2013 UEC Tokyo.

1. Ego-motion Classification• Classify video sections into four classes① Moving forward② Stopping③ Turning right④ Turning left

Video section is four seconds longⓒ 2013 UEC Tokyo.

1. Ego-motion Classification• Extract 48 frames from one video section• Calculate a feature vector① Compute optical flows for 47 intervals② Build 18-bin directional histograms for 4x4 grids③ Normalize them within a video sectionⓒ 2013 UEC Tokyo.

1. Ego-motion Classification• Train 4 SVM classifiers in one-vs-all– Prepare hand-labeled training data

• Use pseudo-probability values– To estimate section importance4s

Moving forward

0.1

Stopping

0.2

Turning right

0.7

Turning left

0.0

ⓒ 2013 UEC Tokyo.

2. Crosswalk Detection
Crosswalk is important and remarkable cue• Extract three frames every second• Estimate ground regions– Use Geometric context (Hoiem et al. [IJCV Vol.75 2007])ⓒ 2013 UEC Tokyo.

2. Crosswalk Detection• Extract SIFT feature from ground regions• Make BoF vector with extracted SIFT
• Use non-linear SVM as classifier– about 240 learning framesⓒ 2013 UEC Tokyo.

3. Estimation of Importance• Expression of estimation importance

Goforward

𝑐𝑓 = −2

Weighting factors
StopTurningright𝑐𝑠 = 1

𝑐𝑟 = 2ⓒ 2013 UEC Tokyo.

Turningleft

𝑐𝑙 = 2

3. Estimation of Importance• Normalize importance• Regard crosswalk section– Total output is over pre-defined threshold• Add bias to crosswalk section

𝑆′′𝑖 = min(𝑆′𝑖 + 0.5, 1.0)
ⓒ 2013 UEC Tokyo.

4. Calculation of Playing Speed• Calculation play speed from importance𝑠𝑝[𝑖] =1𝑆 ′′ 𝑖 1 − 1/ 𝑠𝑝𝑚𝑎𝑥+ (1/ 𝑠𝑝𝑚𝑎𝑥 − 1 )+1• 𝑠𝑝𝑚𝑎𝑥 is given by user when playing– User can adjust max playing speed on-line• Smoothing playing speed (for easy watching)𝑠𝑝′ 𝑖 = 0.1 𝑠𝑝 𝑖 − 1 + 𝑠𝑝 𝑖 + 1 + 0.8 𝑠𝑝[𝑖]ⓒ 2013 UEC Tokyo.

Viewing System• Implemented view system in HTML5– Be embed classifier outputs and numbers of detectionof each video section

Viewing Systemⓒ 2013 UEC Tokyo.

Experiments• Dataset– Taken at around our university (Tokyo)– 9 Videos (average 9min long)• Evaluation experiments– Ego-motion classification– Crosswalk Detection• User study– Vote best summarization method by usersⓒ 2013 UEC Tokyo.

Evaluation of Ego-motion
Classification

Classification rate ( (TP + TF) / N ) = 83.8 %ⓒ 2013 UEC Tokyo.

Evaluation of Crosswalk Detection• Experiment Setup– 250 learning frames from four videos– 200 testing frames from five videos• Compare w/ and w/o ground estimation– To evaluate the ground region estimationⓒ 2013 UEC Tokyo.

Evaluation of Crosswalk Detection
Recallw/ groundPrecision f-number

0.37

0.787

0.503

0.26

0.839

0.397

estimation

w/o groundestimation

Improvement of F-number 0.106ⓒ 2013 UEC Tokyo.

User Study Setup
Comparing methods as follows:① Proposed (Ego-motion + Crosswalk)② Ego-motion classification only
③ Fast-forwarding at a uniform speed④ Storyboard-style
Ask 10 subjects to evaluate the results forthree videos by aboveⓒ 2013 UEC Tokyo.

User Study Result• Most of the subjects voted to proposed method
VideoVideo A
Video B

Video C
Total

Egomotion

Ego. +crosswalk

431866719ⓒ 2013 UEC Tokyo.

FastStoryboard
forwarding01120011

Conclusion• Summarization of egocentric moving videos
for generating route guide videos• Experiment result– Achieve 83.8% in ego-motion classification– Ground estimation improve crosswalk detection• User study– Most subject voted proposed methodⓒ 2013 UEC Tokyo.

Future Works• Extending target videos– Focused on only walking videos now– Bike and car egocentric videos• Adding important objects– Use other object cues for deferent situationⓒ 2013 UEC Tokyo.

Contacts• Masaya Okamoto• e-mail: okamoto-m@mm.inf.uec.ac.jp

Gracias!
ⓒ 2013 UEC Tokyo.

VS Google Street Views• Taken from high view point• Limited to large cities

Ours

Street Viewsⓒ 2013 UEC Tokyo.

Expression of Playing Speed• Calculation play speed from importance𝑠𝑝[𝑖] =1
𝑆 ′′ 𝑖 1 − 1/ 𝑠𝑝𝑚𝑎𝑥+ (1/ 𝑠𝑝𝑚𝑎𝑥 − 1 )• 𝑠𝑚𝑎𝑥 is given by user when playing– User can adjust max playing speed on-lineⓒ 2013 UEC Tokyo.

+1

Target Video
We assume our target videos are① Walking video recorded from a starting
place to a destination② Recorded by a moving wearable
camera③ Recorded continuously (notinterrupted)ⓒ 2013 UEC Tokyo.

Optical Flows
from Moving Object• Optical flows from moving object (ex. Car)cause failure of ego-motion classification

classified as rightⓒ 2013 UEC Tokyo.

Calculation of Optical• We use improved LK module in OpenCV• For high accuracyⓒ 2013 UEC Tokyo.

Detail Setup of User study• Use 3 videos for User study• Taken by me at residential area Tokyo
Video
Video A
Video B
Video C

Duration After
duration

7:47
9:17
11:26

1:45
2:20
2:40

ⓒ 2013 UEC Tokyo.

Average Storyboard
speedsize

4.5
3.9
4.3

212832

Learning of weight factors• Decided by preliminary experiments• We will decide several parameters fromtraining data
– Need supervised signals in each video
sectionⓒ 2013 UEC Tokyo.

Why not a multi-class classifier• Some section contains complex motion– Two motions in one section, looking aside• One-vs-all classifiers can represent
complex motionⓒ 2013 UEC Tokyo.

Evaluation of Crosswalk Detection
Recall-precision curves

Recall-Precision Curve of Crosswalk detectionⓒ 2013 UEC Tokyo.

2. Crosswalk Detection• Extract SIFT feature from ground regions• Make BoF vector with extracted SIFT
• Use non-linear SVM as classifier– about 240 learning framesⓒ 2013 UEC Tokyo.

Our dataset• Collected videos contain somebody faces andcar numbers• It’s difficult to distributeⓒ 2013 UEC Tokyo.