# Presentation: 003372

## TRECVID 2004 Search Task by NUS PRIS

- Tat-Seng Chua, et al.
- National University of Singapore

## Introduction and Overview
Query Analysis
Multi-Modality Analysis
Fusion and Pseudo Relevance Feedback 
Evaluations
Conclusions

- Introduction and Overview
- Query Analysis
- Multi-Modality Analysis
- Fusion and Pseudo Relevance Feedback
- Evaluations
- Conclusions

## Our emphasis is three-fold: 
Fully automated pipeline through the use of a generic query analysis module
The use of of query-specific models
The fusion of multi-modality features like text, OCR, visual concepts, etc
Our technique is similar to that employed in text-based definition question-answering approaches

- Our emphasis is three-fold:
- Fully automated pipeline through the use of a generic query analysis module
- The use of of query-specific models
- The fusion of multi-modality features like text, OCR, visual concepts, etc
- Our technique is similar to that employed in text-based definition question-answering approaches

## Overview of our System

## ASR
Shot Classes
Video OCR
Speaker Identification
Face Detection and Recognition
Visual Concepts

- ASR
- Shot Classes
- Video OCR
- Speaker Identification
- Face Detection and Recognition
- Visual Concepts

## Introduction and Overview
Query Analysis
Multi-Modality Analysis
Fusion and Pseudo Relevance Feedback 
Evaluations
Conclusions

- Introduction and Overview
- Query Analysis
- Multi-Modality Analysis
- Fusion and Pseudo Relevance Feedback
- Evaluations
- Conclusions

##  Morphological analysis to extract:
Part-of-Speech (POS) 
Verb-phrase 
Noun-phrase 
Named entities
Extract main core-terms (NN and NP)

- Query

- NLP Analysis
- (pos, np, vp, ne)

- Query-class

- Key Core Query Terms

- Constraints

- WordNet,  keywords list

-  Morphological analysis to extract:
- Part-of-Speech (POS)
- Verb-phrase
- Noun-phrase
- Named entities
- Extract main core-terms (NN and NP)

## PERSON:  queries looking for a person. For example: “Find shots of Boris Yeltsin” 
SPORTS: queries looking for sports news scenes. For example: “Find more shots of a tennis player contacting the ball with his or her tennis racket.”
FINANCE: queries looking for financial related shots such as stocks, business Merger & Acquisitions etc.
WEATHER: queries looking for weather related shots. 
DISASTER: queries looking for disaster related shots. For example: “Find shots of one or more building with flood waters around it/them”
GENERAL: queries that do not belong to any of the above categories. For example: “Find one or more people and one or more dogs walking together”

**PERSON:  **queries looking for a person. For example: “Find shots of Boris Yeltsin”
**SPORTS: **queries looking for sports news scenes. For example: “Find more shots of a tennis player contacting the ball with his or her tennis racket.”
**FINANCE: **queries looking for financial related shots such as stocks, business Merger & Acquisitions etc.
**WEATHER: **queries looking for weather related shots.
**DISASTER: **queries looking for disaster related shots. For example: “Find shots of one or more building with flood waters around it/them”
**GENERAL: **queries that do not belong to any of the above categories. For example: “Find one or more people and one or more dogs walking together”

## Examples of Query Analysis

| Topic | Query-class | Constraints | Core terms | Class |
| --- | --- | --- | --- | --- |
| 0125 | Find shots of a street scene with multiple pedestrians 
in motion and multiple vehicles in motion somewhere
 in the shot. | in motion somewhere | street | GENERAL |
| 0126 | Find shots of one or more buildings with flood waters 
around it/them. | with flood waters around 
it/them | Buildings, 
flood | DISASTER |
| 0128 | Find shots of US Congressman Henry Hyde's face, 
whole or part, from any angle. | whole or part, from any 
angle | Henry Hyde | PERSON |
| 0130 | Find shots of a hockey rink with at least one of the 
nets fully visible from some point of view. | one of the nets fully visible | hockey | SPORTS |
| 0135 | Find shots of Sam Donaldson's face - whole or part, 
from any angle, but including both eyes. No other 
people visible with him | whole or part, from any 
angle, but including both 
eyes. No other people visible 
with him | Sam 
Donaldson | PERSON |

## Corresponding Target Shot Classfor each query class

| Query-class | Target Shot Categories |
| --- | --- |
| PERSON | General |
| SPORTS | Sports |
| FINANCE | Finance |
| WEATHER | Weather |
| DISASTER | General |
| GENERAL | General |

**Pre-defined Shot Classes:** General, Anchor-Person, Sports, Finance, Weather

## Query Model -- Determine the Fusion of Multi-modality Features

| 

Class | Weight of 
NE in 
Expanded
terms | Weight 
of 
OCR | Weight of 
Speaker
Identifica-
tion | Weight
 of Face 
Recogni
-zer | Weight of Visual Concepts (total of 10 visual concepts used) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | People | Basket-
ball | Hockey | water-
body | fire | Etc |
| PERSON | High | High | High | High | High | Low | Low | Low | Low | . |
| SPORTS | High | Low | Low | Low | Low | High | High | Low | Low | . |
| FINANCE | Low | High | Low | High | Low | Low | Low | Low | Low | . |
| WEATHER | Low | High | Low | High | Low | Low | Low | Low | Low | . |
| DISASTER | Low | Low | Low | Low | Low | Low | Low | High | High | . |
| GENERAL | Low | Low | Low | Low | High | Low | Low | Low | Low | . |

- Weights obtained from labeled training corpus

## Introduction and Overview
Query Analysis
Multi-Modality Analysis
Fusion and Pseudo Relevance Feedback 
Evaluations
Conclusions

- Introduction and Overview
- Query Analysis
- Multi-Modality Analysis
- Fusion and Pseudo Relevance Feedback
- Evaluations
- Conclusions

## K1  query terms expanded using its Synset (and/or glossary) from WordNet
K2  ASR (terms with high MI) from sample video clips
K3  Web expansion (terms with high MI) union K1 & K2 

*K**1*  query terms expanded using its Synset (and/or glossary) from WordNet
*K**2** * ASR (terms with high MI) from sample video clips
*K**3*  Web expansion (terms with high MI) union *K**1* & *K**2** *

## Video OCR
Based on featured donated by CMU, with error corrections using minimum edit distance during matching
Face Recognition
Based on 2DHMM
Speaker Identification
HMM model using MFCC and Log of Energy
Visual Concepts
Using our concept-annotation approach for feature extraction

- Video OCR
- Based on featured donated by CMU, with error corrections using minimum edit distance during matching
- Face Recognition
- Based on 2DHMM
- Speaker Identification
- HMM model using MFCC and Log of Energy
- Visual Concepts
- Using our concept-annotation approach for feature extraction

## Fusion of Features

- Pseudo Relevance Feedback
- Treat top 10 returned shots as positive instances
- Perform PRF using text features only to extract additional keywords *K**4*
- Similarity- based retrieval of shots using *K**3* U *K**4*
- Re-rank shots

- Note for those features that have low confidence values, their weights will be re-distributed to other features

## Introduction and Overview
Query Analysis
Multi-Modality Analysis
Fusion and Pseudo Relevance Feedback 
Evaluations
Conclusions

- Introduction and Overview
- Query Analysis
- Multi-Modality Analysis
- Fusion and Pseudo Relevance Feedback
- Evaluations
- Conclusions

## Evaluations

_**Run1**_ (MAP=0.038)
- Text only

**We Submitted 6 runs:**

_**Run2**_ (MAP=0.071)
- Run1 +
- External Resource (Web + WordNet)

_**Run3**_ (MAP=0.094)
- Run2 + OCR, Visual concepts, shot Classes and Speaker Detector

## Evaluations -2

_**Run4**_ (MAP=0.119)
- Run3 +
- Face Recognizer

_**Run5**_ (MAP=0.120)
- Run4 +
- More emphasis on OCR

_**Run6**_ (MAP=0.124)
- Run5 +
- Pseudo Relevance Feedback

## Overall Performance

- Run6: mean average precision (MAP) of 0.124

## Actually an automatic system – We focused on using general purpose query analysis to analyze queries 
Focused on the use of query classes to associate different retrieval models for different query classes
Observed successive improvements in performance with use of more useful features, and with pseudo relevance feedback

We did a further run (equivalent to Run 5) but use AQUANT (news of 1998) corpus to perform feature extraction, lead to some improvement in performance (MAP 0.120 -> 0.123)
Main findings:
text feature effective in finding the initial ranked list, other modality features help in re-ranking the relevant shots
Use of relevant external knowledge is worth exploring 

- Actually an automatic system – We focused on using general purpose query analysis to analyze queries
- Focused on the use of query classes to associate different retrieval models for different query classes
- Observed successive improvements in performance with use of more useful features, and with pseudo relevance feedback
- We did a further run (equivalent to Run 5) but use AQUANT (news of 1998) corpus to perform feature extraction, lead to some improvement in performance (MAP 0.120 -> 0.123)
- Main findings:
- text feature effective in finding the initial ranked list, other modality features help in re-ranking the relevant shots
- Use of relevant external knowledge is worth exploring

## Employ dynamic Baynesian and other GM models for perform fusion of multi-modality features, learning of query models, and relevance feedback
Explore contextual models for concept annotations and face  recognizer etc.

- Employ dynamic Baynesian and other GM models for perform fusion of multi-modality features, learning of query models, and relevance feedback
- Explore contextual models for concept annotations and face  recognizer etc.

## Participants of this project:
	Tat-Seng Chua, Shi-Yong Neo, Ke-Ya Li, Gang Wang, Rui Shi, Ming Zhao and Huaxin Xu

The authors would also like to thanks Institute for Infocomm Research (I2R) for the support of the research project “Intelligent Media and Information Processing” (R-252-000-157-593), under which this project is carried out.

- Participants of this project:
- 	Tat-Seng Chua, Shi-Yong Neo, Ke-Ya Li, Gang Wang, Rui Shi, Ming Zhao and Huaxin Xu
- The authors would also like to thanks Institute for Infocomm Research (I2R) for the support of the research project “Intelligent Media and Information Processing” (R-252-000-157-593), under which this project is carried out.

## Question-Answering