# Presentation: 040437

- Dennis Moellman, VACE Program Manager
- 20 January 2006

**Bidder Brief**
**VACE Phase III BAA**
** ****Discussion**

**Notes:**
(U) The objective of this Program Management Review is to review each Program’s current status:
Financial
Metrics
Technical

(U) The briefing will begin with a short overview of the Program

(U) The briefer will then drill down into the Projects that focuses on the “problem children” and what the PM is doing to assist/encourage/support/resolve project issues.

(U) The briefer will then provide a look at how the Program is doing against the metrics established/published for the Jun-05 PMR.

(U) The briefer will then have the opportunity to highlight technical and programmatic progress, plans and problems (but only those problems that the briefer wants Program Office help on).

(U) The following charts provide more detailed instructions and a suggested time budget for each slide/topic.

## Administration

- Web conference: (2 hours max.)
- Simultaneous oral broadcast and visual presentation
- On website there is a chat capability
- Presentation is being recorded
- For playback access see BAA website
- Participants:
- Myself, Dennis Moellman
- John Garofolo, NIST -- Evaluation

## Brief Outline

- Introduction
- Context
- Background
- Technology
- Programmatics
- Schedule
- Evaluation Overview
- Formal Q&A
- Ad Hoc Q&A

## Context

- ARDA – Advanced Research and Development Activity
- A high-risk/high-payoff R&D effort sponsored by US DoD/IC
- VACE – Video Analysis and Content Extraction
- ARDA taking a new identity
- Now under the DNI
- Report to: ADNI(S&T)
- Renamed: **Disruptive Technology Office**
*Exploit path-breaking scientific and research advances that will enable us to maintain and extend intelligence advantages...(DNI National Intelligence Strategy)*
- New location: Adelphi, MD – middle 2006
- Winding down Phase II
- Entering into Phase III
- Schedule on succeeding VU-graph and in BAA

**What’s ARDA/VACE/DTO?**

## Background

_***What is the problem?***_
**Analysts need robust, automated video tools**
_***What are the barriers to solving this problem?***_
**Effectively ingesting, indexing, managing, accessing and understanding a large video corpus from multiple heterogeneous sources**
_***How will you overcome those barriers?***_
**By automating manual video activities**
_***What is the capability you are developing?***_
**Video analysis systems**
**Video analysis system components**
**Basic technology breakthroughs**
**Formal evaluations: procedures, metrics and data sets**

## VACE Roadmap

**Motion Analysis**

**Evaluation**

**Summarization**

**Video Browsing**

**Content-based Routing**

**Advanced query/retrieval using Q&A technologies**

**Video Monitoring**

**Video Mining**

**Indexing**

**Change Detection**

**Object Recognition**

**Mensuration**

**Scene Modeling**

**Event Recognition**

**Object Modeling**

**Simple Event Detection**

**Event Understanding**

**Complex Event Detection**

**Object/Scene Classification**

**Object Detection & Tracking**

**Enabling**
**Technologies**

**Image Enhancement/Stabilization**

**Multi-modal fusion**

**Event Ontology**

**Event Expression Language**

**Camera Parameter Estimation**

**Automated Annotation Language**

**Integrity Analysis**

**Motion Analysis**

**VACE Roadmap**

## Development

- R&D Process
- Understand a requirement
- Do the science to develop a capability
- Test and validate (evaluation)
- Make operationally viable

_**Metric**_**	****	****	**_**Current**_**	****  ****	**_**Objective**_** ****	**
- Accuracy		<<Human		>Human
- Speed			>>Real time		<Real time

## Changing Paradigm

- Will be advertised as DTO initiative
- Contracting Agency: DOI, Ft. Huachuca, AZ
- A central focal point
- Enable Phase III schedule synchronized
- Less difficulty with foreign researchers
- Have DOI central COR
- Hope to retain DoD/IC agency COTR’s
- Emphasis on both technology and system approach
- Move up technology path where applicable
- Stress ubiquity

## New Paradigm

**Visualization**

**Enhancement**
**Filters**

**Source Video**

**Recognition****Engine**

**Understanding****Engine**

**Extraction****Engine**

**Intelligent**
**Content Services**

**Concept ****Applications**

## Programmatics

- Open BAA
- Released** **early CY2006
- Span 3 FY’s and 4 CY’s
- BAA remain open thru 6/30/08
- Funding objective: $30M over program life
- Anticipate to grow in FY07 and beyond
- Address the same data source domains as Phase II
- Will conduct formal evaluations
- Will conduct maturity evaluations and tech transfer

## Programmatics (cont)

- Divided into two tiers:
- Tier 1: One year base with option year
- Technology focus
- Open to all – US and international
- More awards for lesser funding
- Tier 2: Two year base with option year(s)
- Comprehensive component/system level initiative
- Must be US prime
- Fewer awards for greater funding

##  Tentative Schedule

- 12/01/05 – Notice of intent in FedBizOPS; register interest; build teaming base
- 12/15/05 – Draft BAA posted; register for Bidders Brief
- 1/013/06 – Comments/Questions Cutoff
- 1/20/06 – Bidders Brief; planning virtual meeting
- 2/1/06 – Final BAA Announcement
- 3/03/06 – Proposals due to Government
- 4/15/06 to 4/30/06 – Evaluation recommendations completed (depending on response)
- 6/30/06 – Contract Awards Completed
- 6/30/08 – BAA extension period
- 6/30/09 – BAA completion

## VACE Phase-III Evaluation

- John Garofolo, NIST

## Goals

- Objective assessment of key technologies
- summary of state-of-the-art for DTO management for planning
- summative and developmental feedback for researchers
- Evaluation includes
- necessary source data and gold-standard annotations
- metrics/evaluation software tools
- protocols/plans for formal evaluation
- informative primary and contrastive test conditions
- technical workshops
- Evaluation is a critical part of the research

## VACE Approach

- Government coordinates certain evaluations to:
- Address problems that are important to the Government and assess the state-of-the-art
- Develop critical mass in important research areas and focus the research community
- Maximally leverage data resources
- Provide common ground for exchange of knowledge

## Government-coordinated Evaluation Selection Process

- Assess DTO needs
- strategic interest and emphases
- via VACE PM and Government Advisory Panel
- Assess researcher needs and desires
- interest and probable participation levels
- required effort
- via Evaluation Forum and cross-program committees
- Assess impact
- effect on key technologies
- relationship to other programs
- Assess feasibility
- required/available data resources
- required annotation and evaluation tool development effort
- estimate of current state-of-the-art
- Prioritize, down-select, and refine
- Co-optimize above
- Refine tasks and metrics through arbitration process

## VACE-II Evaluations

- References included to provide example of breadth and process in Phase-II.  These are not intended to be prescriptive of evaluation in Phase-III.
- Tasks: person/face/hand/text/vehicle detection and tracking, text recognition, shot boundary detection, excerpt search, feature extraction
- Domains: broadcast news, meetings, UAV, surveillance (security cam)
- Venues: VACE Core Eval, TRECVID, CLEAR, RT
- New Government-supported evaluation tasks are likely to be developed after the Phase-III selection process in the selection process described above.

## ProposalRequirements

- Clearly specify key critical research elements and how progress should be evaluated
- Existing “VACE-sanctioned” evaluations and/or metrics
- “Outside” existing evaluations
- must describe or provide sufficiently informative references that address questions in Appendix B.
- New or non-sanctioned evaluation tasks/protocols must be detailed in an evaluation plan addressing the questions in Appendix B
- must provide proposal for evaluation, proposed metrics, goals and milestones, schedule, and necessary data infrastructure (including whether resources need to be created or acquired.)
- Elements must be mapped to VACE technical objectives
- Must demonstrate a clear understanding of how evaluation will be integrated into the research process

## Proposal Requirements contd.

- Must address both performance and efficiency (processing speed)
- generally-accepted performance metrics should be use if possible
- otherwise, new metrics should be suggested, should be minimal necessary to summarize performance
- objective metrics and repeatability should be emphasized
- processing speed determination should be on single CPU if possible
- should be times-realtime for automated systems
- near-realtime goals will be favored
- User-interaction-based technologies must report standard usability metrics
- Proposals with baselines will be favored
- estimates must be provided for proposals with no baseline, must be performed within 6 months of award
- Must include objective performance and efficiency goals and timeline for achieving these goals
- awarded contracts will be measured against these goals

## Evaluation Budgeting

- Minimum of 10% for evaluation
- planning and arbitration process described above – must devote at least one rep to this process
- adaptation of research algorithms for evaluation processing
- dry run evaluations
- developmental “intrinsic” evaluations
- formal “extrinsic” evaluations
- participation in evaluation workshops
- Should NOT include algorithm development or hardware/software for such development.

## Data Resources

- Effort will be made to provide pertinent source data resources
- however Government cannot guarantee delivery of such data
- acquisition is very difficult and time-consuming
- collection efforts in the meeting, surveillance, and UAV domains are ongoing
- international collaborations are helping to accelerate the immediate availability of data
- Consideration will be given to research that can work with existing or near-ready data resources

## Current VACE Source Data Resources

- TRECVID broadcast news and BBC rushes collections
- NIST Meeting Corpora
- Multi-site meeting corpora and NIST corpora
- CHIL/AMI meeting corpora
- I-LIDS surveillance data
- VISA surveillance data (coming soon)
- DARPA VIVID UAV data
- VACE UAV data (coming during Phase-III)
- LDC Broadcast News data

## VACE Evaluation Programs

- VACE Core Evaluation (ongoing)
- VACE-supported evaluations implemented internally to the VACE program (will be reduced in Phase-III)
- CLEAR – Classification of Events, Activities, and Relationships (Spring)
- Collaborative cross-program/international evaluation program focused on multi-modal spatial analyses
- TRECVID – TREC Video (Summer)
- Collaborative international evaluation program focussed on multi-modal search
- RT – Rich Transcription (Spring)
- Collaborative cross-program/international evaluation program focused on multi-modal extraction of content from language

## Evaluation Workshops

- Separate annual workshops will be held for CLEAR, TRECVID, and RT
- RT and CLEAR will rotate between the US and Europe
- Plan to send key technical personnel (primes and subs) to each of these workshops as appropriate
- must attend workshops for which you participated in the associated evaluations
- Expect to prepare technical presentations and papers for each workshop you participate in
- Evaluation results will be made public via the proceedings of these workshops

## Scope

| Question | Answer |
| --- | --- |
| Will existing concepts that need development for the benefit of the government be available to the developers for sale in the commercial market? 
 | The government owns the IP rights to all technologies previously developed on the VACE program.  These technologies will be made available to Phase 3 participants as appropriate and relevant to your project.  We encourage this type of technological exchange among participants in Phase 3 especially between Tier 1 and Tier 2 projects.  If these technologies have been incorporated into larger commercial products that you feel are more helpful to your approach, then you will need to proceed with a normal acquisition and/or teaming arrangement. |
| VACE II, alone, is the criteria to be used as the selection criteria for VACE III, then this issue needs to be addressed to more clearly lay out the acceptability of all of the relevant issues of image processing, listed in VACE III, not just those relative to face and vehicle recognition.  | There is no intent to use VACE II selection criteria for VACE III selection.  Only the VACE III stated selection criteria are to be used.  Any reference to VACE II technologies in the VACE Roadmap are merely to guide the offeror to ensure his/her technology proposal is one that forwards the state-of-the-art and not address problems that have been solved already or are well along the path to being solved for the government is looking for innovation and disruptive developments.   |

## Scope

| Question | Answer |
| --- | --- |
| The draft PIP indicates that this work must build on VACE techniques already developed in Phase II. Does this mean that you are not interested in new innovations?  | Although the VACE program is broad in scope, we do recognize the limitations of this solicitation.  We are always interested in new innovations that are relevant to both the VACE program and the interests of the Intelligence Community.  Please refer to section 4.4.6.  |
| If new innovations are to be considered what is the likelihood of an award of this nature?  | A large evaluation team that spans several government agencies will evaluate all proposals.  This evaluation will be implemented according to guidelines within the Source Selection Plan as approved by the NBC Contracting Office and the DTO PMO.  All proposals will be evaluated equally in accordance with this document.  |

## Scope

| Question | Answer |
| --- | --- |
| On Figure 4, could you provide additional description, such as a definition of the terms used?  | Diagram |
| Section 3.2 discusses real time processing.  In analysis of processing speeds, several different factors are important:  a) processing rate (e.g., frames per second); b) latency (e.g., time difference between reading the first frame of a video sequence and producing results relevant to the first frame); and c) batch processing v. streaming data.  Which of these factors are important to the VACE program?  | As the technologies depicted in figure 2 become robust the emphasis will move to expanding scale and reducing a decision cycle.  Both processing speed and latency are major issues regarding video processing technologies and must be addressed.
The issue of how processing will be performed (batch or streaming) is application dependent.  So, either one or both might be appropriate for the proposed technology, depending on the applications that it seeks to address.  |

## New Paradigm

**Visualization**

**Enhancement**
**Filters**

**Source Video**

**Recognition****Engine**

**Understanding****Engine**

**Extraction****Engine**

**Intelligent**
**Content Services**

**Concept ****Applications**

## Scope (final)

| Question | Answer |
| --- | --- |
| The term “video” is used in a number of places in the PIP. In Section 4.4.1 speaker identification is listed as a focus of video broadcast news exploitation. For the purposes of VACE III, can one assume that the term "video" refers to all of the signals or subsets of the signals, depending on the situation, in a video transmission (e.g. video, audio, captions, and related data in NTSC and PAL)?  | Where applicable video refers to all signals such as the example you cite.  But there are other data sources besides broadcast news that are of interest of VACE III as stated in sections 4.4.1 through 4.4.6.  |
| Based on Section 3.8 platform (operating systems and/or hardware) independence is clearly desirable.  Are there preferred demonstration platforms (operating systems and/or hardware) for VACE III?  | No.  Most of the research that has been accomplished to date has been primarily done on the MSWindows platform.  However, in Phase 3 there is more of an emphasis on performance (figure 3) and consideration of system architecture that would facilitate collaboration and integration of technologies may require other OS platforms be taken under consideration.  |

## Tier Structure

| Question | Answer |
| --- | --- |
| Is a Tier 2 solution must be processed data from two different data sources using the same system?  Or should two systems be developed using a number of components that are shared among the two systems? | As mentioned on Page 10, section 3.1 Structure, "Unlike the Phase II program, Tier 2 solutions must address applicability to more than one of the heterogeneous video data sources mentioned in Section 2 and described in more detail in Section 4." This was written to describe the objective, but avoid suggesting an implementation method to demonstrate.  The government wants a Tier 2 system to show the capability of working with heterogeneous data sources commensurate with the performance objectives stated figure 3.  |
| Can the requirement of addressing more than one video data sources for a Tier 2 system be reconsidered in order to maximize the usability and impact of such a system to operational scenarios for a chosen data domain? | A Tier 2 system should be able to demonstrate that it can provide a comprehensive suite of capabilities for one data domain with a high level of performance, and have a reasonable subset of capabilities within another domain(s) with the same level of performance.  However, the architecture should facilitate external component integration to achieve comparable capabilities in another domain(s) as discussed in section 3.2.  |

## Tier Structure

| Question | Answer |
| --- | --- |
| On pg 10 of Draft BAA under Tier 2 discussion, multiple heterogeneous video data sources for system level integration are mentioned.  Do you mean multiple heterogeneous data sources within a domain (e.g. Broadcast News) or actually heterogeneous across multiple domains such as Broadcast News and Meeting/Conferences Surveillance, Ground Reconnaissance, and UAVs?  | The latter: heterogeneous across multiple domains such as Broadcast News and Meeting/Conferences Surveillance, Ground Reconnaissance, and UAVs.  Also see answer to Sarnoff question for more info.  |
| As Tier2 solutions are intended to be end-to-end systems and different data sources have very different processing requirements, these seem to be incompatible requirements. A clarification on the motivation and goals would be most helpful.  | The government’s intent for Tier2 solutions is to develop and demonstrate a capability to support both multiple feeds from a single domain and multiple feeds from multiple domains.  We recognize there is a tradeoff of performance and versatility/functionality.  It is important to be able to demonstrate that given access to more resources (computational and component technologies) that a Tier 2 system can scale and adapt to support multiple homogenous and/or heterogeneous data sources with performance levels commensurate with Figure 3.  |

## Evaluations

| Question | Answer |
| --- | --- |
| VACE III states a large scope but the VACE II evaluation protocol is narrow.  | This is true.  It is not possible for the VACE evaluation infrastructure to support all of the research tasks in the program.  Rather, the intent is to focus on a few key tasks which are common across research efforts.  These tasks will evolve as progress is made and new areas of emphasis are realized by the Government and the research community.  |
| For costing purposes, what assumptions should we make for meeting locations, including evaluations (e.g., where in Europe might the evaluations occur), and Program Reviews?  | Program reviews will occur within the contiguous U.S.  Some evaluation workshop may occur in Europe, so consideration should be given to what evaluations are relevant to your technology and plan accordingly.  |

## Evaluations

| Question | Answer |
| --- | --- |
| The "Performance Evaluation Protocol for text, face, hand, person and vehicle detection & tracking video analysis and content extraction (VACE II)," is listed as the sole criteria given for evaluation of a VACE III proposal.  | The “Performance Evaluation Protocol” for VACE II is exactly that.  It was arrived at based upon the collective and individual evaluation needs for VACE II given data that was available or that could be collected.  This Protocol was not developed before VACE II but during VACE II once all selected research activities were reviewed, common threads identified and an effort was made to ensure all participants had an evaluation venue.  For VACE II TRECVID was an alternative means to participate in an evaluation.  In the VACE III era there are anticipated to be addition “outside” evaluation venues in which to participate. A similar Performance Evaluation Protocol will be developed early on in VACE III once all the selected research is reviewed to scope the VACE III activities.  Therefore the pointer to the VACE II should not be construed as a plan for evaluation in VACE III but merely an illustration that may assist the offeror to understand intent and assist in scoping offeror level of effort commitment.   |

## Evaluations

| Question | Answer |
| --- | --- |
| Section 4.5.1 requests a detailed description of our evaluation plan.  Section 4.5.4 describes the evaluation workshops.  In order to align our detailed evaluation plan with the VACE program, can you provide any details on the VACE program evaluation schedule, such as what tasks are evaluated when, and what data will be made available for these evaluations?  | While certain data is in the pipeline and will likely be employed in VACE III, additional sources may be explored if they are deemed important for the success of a project.  Current data resources include security camera recordings, meeting recordings, broadcast news recordings, and limited UAV recordings.  The program will seek to synchronize evaluations on a periodic basis – preferably annually – with ongoing evaluation efforts a la Rich Transcription and Classification of Events, Activities, and Relationships, and TRECVID.  Our goal is to have extraction evaluations take place in the spring and search evaluations in the late summer.  However, this is subject to change.
Dry run evaluations may be implemented prior to formal evaluations for new tasks.  |
| In Section 4.5 of the Proposal Information Pamphlet (PIP), there are references to DTO-sponsored evaluations. Could you please provide us with a list of when and where these evaluations are taking place?  | See answer to BAE Systems last question for DTO evaluations.  DTO-sanctioned, TRECVID (Search and Extraction) annually begin in April, finish in September, and are reported in November at NIST, Gaithersburg, MD. |

## Data Management

| Question | Answer |
| --- | --- |
| Would storage and information management of video files be GFE through something like RDEC or some other Government-sponsored or owned archive or repository?  | The program is working on a more formal arrangement to support this function.  It would be our intention to allow this type of access within the program to facilitate the distribution process and reduce overhead.  However, we do not have a formal contract in place at this time.  |
| Is there a plan for expanded data availability/accessibility to the research groups?  | The VACE program is working several initiatives to provide a large corpus of data in several domains.  This will be done to support both the evaluation process and the individual research projects.  The amount will vary, but it is the intent of the program to provide data in at least four data domains (BN, UAV, Surveillance, and Meeting Room)  |

## Tech Assessment

| Question | Answer |
| --- | --- |
| What is the level of detail of the technology transition plan?  | This is something that continues to evolve.  There is a process defined, but unfortunately we do not have the ability to fully implement it at this time.  However, the process is roughly based on the DoD TRL standards (DoD 5000.2R), which defines a maturation path beginning with an idea to demonstratable prototype to various test environments and finally into production/operations.  |
| What is the format of interaction with the TRAA?  | There will be a POC assigned to your project from the TRA organization.  This person will become knowledgeable of your technologies and will work with you on refinements and enhancements in preparation for migration to a larger and more formal test environment.  |

## Tech Assessment

| Question | Answer |
| --- | --- |
| Will awardees have access to RDEC or be authorized to get a drop at their facility in order to work on this Phase III effort?  | RDEC is an environment used to test, evaluate and demonstrate a capability once it has achieved a certain readiness level.  Therefore no RDEC access can be guaranteed.  Any capability that will migrate to RDEC will be physically installed in the RDEC facility.  The TRA is viewed as an intermediate step between a laboratory and the RDEC.  |
| Would we have access to future video data stored within RDEC?  | Only when a capability has migrated to RDEC.  There may be exceptions but those would be reviewed on a case by case basis.  |

## Adhoc Q&A