NII-KAORITM
A Semantic Video Search Engine
Duy-Dinh LE, Thanh Duc NGO, and Shin’ichi SATOH
National Institute of Informatics

Motivation

Proposal Current video retrieval systems rely on meta data
(e.g. genre, date, producer, etc) and usergenerated tags.
 Advanced video search engines should rely on
the actual content of the video data such as
objects, motions, people and events.
 Building such systems requires to solve theproblem of bridging the semantic gap.

 We introduce a semantic video search engine forcontent-based video retrieval.
 The video content information is automaticallyextracted using semantic video analysis.
 The demo enables users to search and explore
the content of large video archives with several
thousand hours of video.

Feature Highlights
1. Large and diverse video datasets
 TRECVID 2004-2008: news and documentary
programs in languages such as English, Arabic,
Chinese and Dutch.
 NHKNews7 2001-2009: news in Japanese.
2. Automatic semantic concept extraction
 Airplane, Building, Animal, Sitting, etc.

3. People search Search by names.
4. Region search
5. Support both targeted search and exploratory search Search by keywords and names.
 Browse by similar images.

Technical Details
1. Video datasets
 TRECVID 2004-2008: 658 hours, 659,322 keyframes.
 NHKNews7 2001-2009: 1,413 hours, 747,529keyframes.
 JP News and Documentary Video, 2005-2009:
11,335 hours, 680,106 keyframes.
 Total: 13,406 hours, 2,086,957 keyframes.
2. Semantic concepts
 LSCOM (http://www.lscom.org/) : 374 concepts.
3. Faces ~50 million faces are clustered.

Query keyword: animal

Query region: “Chicken Noodle Soup”
Query name: BushFuture Goals
1. Core Services for Video Analysis
 Sharing and Collaboration.

2. Applications for Knowledge Discovery
 Video recommendation, video mining, social analysis.

Duy-Dinh LE, Thanh Duc NGO, and Shin’ichi SATOH, National Institute of Informatics
TEL : 03-4212-2583

FAX : 03-3556-1916

Email : {ledduy,ndthanh,satoh}@nii.ac.jp

Web Image Searchin TV Video Archive
Sebastien POULLOT and Shin’ichi SATOH
National Institute of Informatics

Motivation

Proposal Link Internet resources to Archives: The
Internet is a massive source of detailed
informations, by automatically linking
pictures (and videos) from web pages toprecise position in the videos, thesurrounding texts can be used forautomatic or semi automatic annotation(high granularity and precise descriptions).

 We mainly have to face the very large sizeof the existing video databases. The focus
is on a scalable solution. However it shouldbecapabletohandle
varioustransformations
betweenimagesandvideos. For now, only images that arecopies (screen shots for example) can belinked to a video source.

Framework Overview
Videos archives

Keyframes

Descriptions
0010101010000
0010101000101
1010000001100
11111110110000100001000011

Storage

Index

1010001010100

Searching

Image

Web Pages

Display answer

Experiments and perspectives
1. Method Keyframe extraction based on global luminancechanges (about 1KF/second) PoI detection and local description extraction(DoG and SIFT).
 Frame level description using quantization (similarto BoF approach).
 Indexing based on locality of PoI in the frames.

PoI associated in
keyframe space

Local descriptors quantified
in description spacepositions
1, 3 & 11

positions5, 6 & 14
positions5, 12 & 16

Glocal description: 1010110000110101

1-3-11

5-6-145-12-16
Buckets

2. Video Datasets and Server Video recordings on 7 channels for 6 weeks: 8000
hours, 27 millions of keyframes.
 Database size: 38Gb.
 Server: 228Gb of main memory.
 A 40,000 hours database is feasible.
3. Applications
 Fast answer for searching pictures or videos in thereference database.
 Mining based on occurences in a database.
 Auto propagation of annotations.
 Video clustering.
 Automatic and semi automatic processes.
4. Next steps
 Test with larger and larger databases, one yearof recordings: 60,000h.
 Parallel implementation.
 Automatic update of the database: investigate
the dynamic aspects.
 Link with an automatic annotation propagation
application → some precision improvements arenecessary.
5. 10 years after Instant browsing in video space.
 Dynamic behavior of the video space.
 New content based video links: people, objects,places...
 Fusion of multi modal views: real multimedia.

Sebastien POULLOT and Shin’ichi SATOH, National Institute of Informatics
TEL : 03-4212-2583

FAX : 03-3556-1916

Email : {spoullot,satoh}@nii.ac.jp

Celebrity Search in Large Video Archives
Duy-Dinh LE, Thanh Duc NGO, and Shin’ichi SATOH
National Institute of Informatics

Motivation

Proposal

 A large volume of search queries in image
search engines is involved to celebrities.
 Most of the current image and video retrieval
systems rely on text descriptions for searching
persons rather than identifying them.
 Few studies on organizing very large videoarchives using faces.

 We introduce a framework for person search inlarge video archives.
 Huge number of faces in videos are extracted,tracked, and grouped into clusters.
 Semi-supervised techniques are used to annotatedface clusters in efficient and effective ways.

Feature Highlights
2. Many Celebrities
118 persons: government leaders athletics
movie stars.

1. Large Scale Video Dataset
NHKNews7 2001-2008:
 ~ 1,500 hours
 ~ 35 million faces

People search by names and images

Future Goals
1. Person Search Engine
Search all appearances of all people

appearing in videos.

2. Applications
Robust face identification for security, videoconferencing, and human computer interaction.

Duy-Dinh LE, Thanh Duc NGO, and Shin’ichi SATOH, National Institute of Informatics
TEL : 03-4212-2583

FAX : 03-3556-1916

Email : {ledduy,ndthanh,satoh}@nii.ac.jp

Large-Scale News Topic Tracking and Key-scene
Ranking with Video Near-Duplicate Constraints
Xiaomeng

WU1

1

NationalIchiro
InstituteofIDE2

Informatics

SATOH1

Shin’ichi
2

NagoyaUniversity

Motivation

展望 Topic tracking is a fundamental step for news browsing,
retrieval, topic threading, and summarization.

 デジタル放送の普及につれて、意味的に構造化された映像・メタデータの効率的流通について考える必要があり，これらのコンテンツの制作や再利用を行うコンテンツ提供者，コンテンツを伝送若しくは視聴するための端末を開発する企業や大学，コンテンツを楽しむ視聴者，皆が有意義に参加できるプロジェクトを設定し，具体的目標を定めて連携して進めていく予定である．

 Providing summarization and visualization of large-scalenews videos is an important task, which aims at saving
time for general audiences in searching and reading newsof interest.

NEWSTOPIC

TRACKING WITH

 News Story Tracking Based on Textual Information

NEAR-DUPLICATE CONSTRAINTS

 Video Near-Duplicate Detection
 LIP-IS+OOS [9] Topic Segmentation
 Topic Threading

PROGRAM 1

FNN SPEAK
2006/12/27

FNN SPEAK
2006/11/06

 Children stories are defined as news stories related to aparent, under the condition that the time stamps of thechildren stories always chronologically succeed their parent.
 The link tree can also be considered a set of candidate news
stories that is similar to the search query.

FNN SPEAK
2006/12/28

SS

S

PROGRAM 4

NHK NEWS 7
2006/11/05

NHK NEWS 7
2006/12/27

NHK NEWS 7
2006/12/29

SS

S

S

S

S

S

: News Story

SS

SS Assumption 1: Most stories in the same story group depict
the same topic.
 Assumption 2: The largest story group depicts the sametopic as the query.

SSS Near-Duplicate Constraints

Succeeding stories

SSStory Group 12006/01/16-1

S

PROGRAM 3

Search Query

2006/01/17-1

S

PROGRAM 2

 This procedure forms a simple story link tree starting fromthe story of interest.

Chronological Order

Search Query

SSSS
S

Story Group 2

2006/01/18-2

2006/01/19-3

2006/01/20-1

2006/01/27-2

2006/01/22-1

2006/01/21-6

2006/01/28-2

 Based on these assumptions, the largest story group ischosen as the expanded query to represent thecharacteristics of the corresponding news topic that thequery depicts.
 Keyword Vector
 General / Personal / Locational (Organizational)

KEY-SCENE
 nPR
 number of programs across which the corresponding
scene is broadcasted
 nST
 number of stories across which the corresponding scene isbroadcasted
 nSH
 number of shots in the corresponding key-scene group

RANKINGAND TF-IDF Weighting Based on Expanded Query Different from traditional tf-idf weighting schemes, we consider one story
group as one document.
 TF-IDF Similarity Based on Expanded Query The similarity between each story and the search query is re-defined as thecosine similarity between each story and the expanded query.

EXPERIMENTS

 Observation 1: Important scenes arenormally broadcasted across different
news programs (e.g. nPR ≥ 2), while
scenes of reporters are only used by one
single program (nPR = 1).
 Observation 2: Within one story, scenesof reporters are normally used more
frequently than important scenes (i.e.
scenes of reporters tend to have lower
value of nST/nSH than important scenes).
 Observation 3: Key-scenes broadcasted
across important stories are also
importantnPR = 3
nST = 8nSH = 13nPR = 3nST = 6nSH = 8nPR = 3nST = 5nSH = 8

srPR (k i) nPR (k i)
NPRsrST (k i) nST (k i)nSH(k i)srR (k i) nPR = 1
nST = 3nSH = 8nPR = 1nST = 3nSH = 8nPR = 1nST = 2nSH = 8

XiaomengsΓ i

Rank: 3/10
sr = 0.29

Rank: 4/10
sr = 0.29

Rank: 5/10
sr = 0.28

Rank: 6/10
sr = 0.24

Rank: 7/10
sr = 0.23

Rank: 8/10
sr = 0.08

Rank: 9/10
sr = 0.04

Rank: 10/10
sr = 0.02r(Σ , s)sr  (w  srPR  (1  w)  srST )  srR

WU ／Rank: 2/10
sr = 0.31

nST (k i)Γ i  {s:k i  s}

NationalTEL：+81-3-4212-2662

Rank: 1/10
sr = 0.32

Instituteof

Informatics

Email：wxmeng@nii.ac.jp