# Presentation: 034965

## Dennis Moellman,
 VACE Program Manager

** ****VACE Program Manager**

**V****ideo ****A****nalysis and ****C****ontent ****E****xtraction ****(VACE) ****R&D P****rogram: *****Overview of Phase 2 ***

## ARDA’S  Charter and Mission

_***Charter***_
- An Advanced Research and Development Activity focused on Information Technology that provides a world-class, multi-disciplinary capability available to the Intelligence Community and, subject to the concurrence of the Secretary of Defense, the Information Security Community.  This activity is the Intelligence Community's center for conducting advanced research and development related to extracting intelligence from, and providing security for, information transmitted or manipulated by electronic means.
_***Mission***_
- Identify important information technology problems of special or unique interest, focus multi-disciplinary expertise against these problems, support high risk/high payoff research, coordinate with other government entities, industry and academe and assist the transfer of solutions to the applied technology centers of other IC organizations.

**ARDA’S  Charter and Mission**

## We originate and manage R&D programs
With fundamental impact on future operational needs and strategies
That demand substantial, long-term venture investment to spur risk-taking
That progress measurably toward mid-term and final goals
That take many forms and employ many delivery vehicles

**With fundamental impact on future operational needs and strategies**
**That demand substantial, long-term venture investment to spur risk-taking**
**That progress measurably toward mid-term and final goals**
**That take many forms and employ many delivery vehicles**

**What ARDA Does**

## VACE Video Analysis and Content Extraction

**HIGHLIGHTS  -  Phase 1**
**Primarily Focused on Core-Level Research Objectives; some early investigations into “Video Events”**
**Devised novel techniques for training up general object detectors with minimal annotated video data; Applied technique to faces, vehicles and other common objects found in office environment**
**Significantly improved tracking of multiple objects within video streams; robust to the merging & separating of objects, to varying rates of motion, to temporarily stationary objects and to occlusions **
**Very encouraging, preliminary success in detecting and understanding single-thread video events as well as simple, multi-thread video events**

**GOALS**
**Robust person, vehicle, and text detection and recognition.**
**Cross media content analysis and extraction**
**Fully automatic video indexing based in image, text, and audio content.**
**Efficient methods for representing video content.**
**Event Detection, Recognition and Understanding.**
**Low cost video corpus marking and preparation**

**CHALLENGE PROBLEMS  -  Phase 2**
**Significantly enhanced detection, recognition and tracking of people, faces and broad classes of other selected objects**
**Object modeling, measurement and mensuration**
**Multi-modal fusion**
**Added emphasis on UAV Motion Imagery Problems**
**“****Video Event” Challenge Problems**
**Foreign Broadcast News**
**Formal and Informal Meetings**
**Abstraction & Inferencing from Surveillance Video**
**UAV Motion Imagery**
**Ground Reconnaissance Video**

***As of Nov 03 ***

## VACE Program Research Goals
Robust person, vehicle, and text detection and recognition.
Cross media content analysis and extraction
Fully automatic video indexing based in image, text, and audio content.
Efficient methods for representing video content.
Event Detection, Recognition and Understanding.
Low cost video corpus marking and preparation

**Robust person, vehicle, and text detection and recognition.**
**Cross media content analysis and extraction**
**Fully automatic video indexing based in image, text, and audio content.**
**Efficient methods for representing video content.**
**Event Detection, Recognition and Understanding.**
**Low cost video corpus marking and preparation**

**VACE**** ****V****ideo ****A****nalysis and ****C****ontent ****E****xtraction**

## VACE Video Analysis and Content Extraction

_**Primary**_
**Object Detection****	****	****-- Object Recognition**
**Object Tracking ****	****	****	****-- Event Understanding**
**Video Summary ****	****	****	****-- Multi-Modal Fusion**
**Video Query by Example****	**
_**Secondary**_
**Multi-modal Video Mining****	****-- Kinematics Analysis**
**Motion Analysis****	****	****	****-- Video Mensuration**
**Integrity Analysis****	****	****--  Model Reconstruction**

**VACE**** ****V****ideo ****A****nalysis and ****C****ontent ****E****xtraction**

## VACE Program Contractors

##  VACE Video Analysis & Content Extraction

**HIGHLIGHTS – Phase 1**
**Primarily Focused on Core-Level Research Objectives; some early investigations into “Video Events”**
**Devised novel techniques for training up general object detectors with minimal annotated video data; Applied technique to faces, vehicles and other common objects found in office environment**
**Significantly improved tracking of multiple objects within video streams; robust to the merging & separating of objects, to varying rates of motion, to temporarily stationary objects and to occlusions **
**Very encouraging, preliminary success in detecting and understanding single-thread video events as well as simple, multi-thread video events**

## VACE Phase II Video Research Environment

## VACE Phase II Challenge Problems

**Continue Investigation of Core Technology Issues:**
**Significantly enhanced detection, recognition and tracking of a broad classes of people, faces and objects**
**Detailed object modeling, measurement and mensuration**
**Multi-modal fusion**
** **_***NEW:***_** “Video Event” Challenge Problems:**
**Foreign Broadcast News**
**Formal and Informal Meetings**
**Abstraction & Inferencing from Surveillance Video**
**UAV Motion Imagery**
**Ground Reconnaissance Video**
**Military Exercises / Rocket Launches**
** **_***NEW:***_** UAV Video Specific Technical Challenges**

## Envisioned as a high risk, long term R&D Program:
Phase 1		Fall 2000 - Fall 2002
Phase 2		Winter 2003 - Winter 2005
Phase 3		Spring 2006 -  Spring 2008
VACE Program Executive Committee:
-- CIA    -- NSA    --  DIA    --  NIMA    -- DHS    -- NIST    --  ARDA
Continue Investigation of Core Technology Issues:
Significantly enhanced detection, recognition and tracking of a broad classes of people, faces and objects
Detailed object modeling, measurement and mensuration
Multi-modal fusion
 NEW: “Video Event” Challenge Problems:
Challenge 1: Foreign Broadcast News
Challenge 2: Formal and Informal Meetings
Challenge 3: Abstraction & Inferencing from Surveillance Video
Challenge 4: UAV Motion Imagery
Challenge 5: Ground Reconnaissance Video
 NEW: UAV Video Specific Technical Challenges

**Phase 1****	****	****Fall 2000 - Fall 2002**
**Phase 2****	****	****Winter 2003 - Winter 2005**
**Phase 3****	****	****Spring 2006 -  Spring 2008**
**VACE Program Executive Committee:**
**-- CIA    -- NSA    --  DIA    --  NIMA    -- DHS    -- NIST    --  ARDA**
**Continue Investigation of Core Technology Issues:**
**Significantly enhanced detection, recognition and tracking of a broad classes of people, faces and objects**
**Detailed object modeling, measurement and mensuration**
**Multi-modal fusion**
** **_***NEW:***_** ****“Video Event” Challenge Problems:**
**Challenge 1: Foreign Broadcast News**
**Challenge 2: Formal and Informal Meetings**
**Challenge 3: Abstraction & Inferencing from Surveillance Video**
**Challenge 4: UAV Motion Imagery**
**Challenge 5: Ground Reconnaissance Video**
** **_***NEW:***_** ****UAV Video Specific Technical Challenges**

**VACE – Phase 2:**** ****ARDA’s Plan of Attack**

## VACE Program Phase 2 Contractors

## VACE Program Phase 2 ContractorsLate Fall 2003 – Late Fall 2005

| Title | Organization | Principal Investigator | Contracting
Agent |
| --- | --- | --- | --- |
| Challenge Problem 1: Foreign Broadcast News Exploitation  |  |  |  |
| ENVIE: Extensible News Video Information Exploitation | Carnegie Mellon University | Howard Wactlar |  |
| Reconstructing and Mining of Semantic Threads Across Multiple Video Broadcast News Sources using Multi-Level Concept Modeling | IBM T.J. Watson Research Center / Columbia Univ.  | John Smith / 
Prof. Shih-Fu Chang  |  |
| Challenge Problem 2: Formal and Informal Meetings  |  |  |  |
| From Video to Information: Cross-Modal Analysis of Planning Meetings  | Wright State University AFIT / Univ. of Chicago / Purdue Univ. / Univ. of Illinois-Urbana-Champaign  | Francis Quek /           Ronald Tuttle /              David McNeill & Bennett Bertenthal / Thomas Huang / Mary Harper  |  |
| Event Recognition from Video of Formal and Informal Meetings Using Behavioral Models and Multi-modal Events  | Alphatech, Inc. / MIT / Univ. of Maryland / Virage  | Stephen Zabele / William Freeman & John Fisher / Yaser Yacoob & Larry Davis / Andy Merlino  |  |

## Slide 14

## VACE Program Phase 2 ContractorsLate Fall 2003 – Late Fall 2005

| Title | Organization | Principal Investigator | Contracting
Agent |
| --- | --- | --- | --- |
| Challenge Problem 3 Abstraction and Inference about Surveillance Activities  |  |  |  |
| Video Event Awareness  | Sarnoff Corporation / Telcordia Technologies  | Rafael Alonso /       Dimitrios Georgakopoulos  |  |
| Integrated Research on Visual Surveillance  | University of Maryland  | Larry Davis,               Yiannis Aloimonos &   Rama Chellappa |  |
| Challenge Problem 4:  UAV Motion Imagery  |  |  |  |
| Adaptive Video Processing for Enhanced Object and Event Recognition in UAV Imagery  | Boeing Phantom Works / Univ. of Washington  | Robert Higgins /          Linda Shapiro  |  |
| Challenge Problem 5:  Ground Reconnaissance Video  |  |  |  |
| Content and Event Extraction from Ground Reconnaissance Video  | TASC, Inc. / Univ. of Central Florida / Georgia Tech  | Sadiye Guler / Mubarak Shah / Ramesh Jain  |  |
| Special Technical Problem Areas for UAV  |  |  |  |
| Task and Event Driven Compression (TEDC) for UAV Video  | Sarnoff Corporation  | Hui Cheng  |  |

## VACE Program Phase 2 ContractorsLate Fall 2003 – Late Fall 2005

| Title | Organization | Principal Investigator | Contracting
Agent |
| --- | --- | --- | --- |
| Cross Cutting / Enabling Technologies  |  |  |  |
| Probabilistic Graphical Model Based Tools For Video Analysis  | University of Illinois, Urbana - Champaign  | Thomas Huang  |  |
| Automatic Video Resolution Enhancement  | Salient Stills, Inc.   | John Jeffrey Hunter  |  |
| Robust Coarse-to-Fine Object Recognition in Video  | Carnegie Mellon University  | Henry Schneiderman & Tsuhan Chen |  |
| Multi-Lingual Video OCR  | BBN Technologies / SRI International  | John Makhoul / Greg Myers  |  |
| Model-based Object and Video Event Recognition  | USC - Institute for Robotics and Intelligent Systems / USC - Information Sciences Institute  | Ram Nevatia, Gerard Medioni & Isaac Cohen / Jerry Hobbs  |  |