Skip to main content    

An official website of the United States government

Here’s how you know

**Official websites use .gov**              A **.gov** website belongs to an official government organization in the United States.            

**Secure .gov websites use HTTPS**              A **lock** (  LockA locked padlock) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.            

https://www.nist.gov/itl/iad/mig/trecvid-2015-evaluation-surveillance-event-detection

            -  Topics  -  All Topics 
-  Advanced communications 
-  Artificial intelligence 
-  Bioscience 
-  Buildings and construction 
-  Chemistry 
-  Climate 
-  Cybersecurity 
-  Electronics 
-  Energy 
-  Environment 
-  Fire 
-  Forensic science 
-  Health 
-  Information technology 
-  Infrastructure 
-  Manufacturing 
-  Materials 
-  Mathematics and statistics 
-  Metrology 
-  Nanotechnology 
-  Neutron research 
-  Performance excellence 
-  Physics 
-  Public safety 
-  Resilience 
-  Standards 
-  Transportation 
 
            -  Publications 
            -  Labs & Major Programs  -  Laboratories    -  Communications Technology Laboratory 
  -  Engineering Laboratory 
  -  Information Technology Laboratory 
  -  Material Measurement Laboratory 
  -  Physical Measurement Laboratory 
-  User Facilities    -  NIST Center for Neutron Research 
  -  CNST NanoFab 
-  Research Test Beds 
-  Research Projects 
-  Tools & Instruments 
-  Major Programs    -  Baldrige Performance Excellence Program 
  -  CHIPS for America Initiative 
  -  Manufacturing Extension Partnership (MEP) 
  -  Office of Advanced Manufacturing 
  -  Special Programs Office 
  -  Technology Partnerships Office 
 
            -  Services & Resources  -  Standards and Measurements    -  Calibration Services 
  -  Laboratory Accreditation (NVLAP) 
  -  Quality System 
  -  Standard Reference Materials (SRMs) 
  -  Standards.gov 
  -  Time Services 
  -  Office of Weights and Measures 
-  Software 
-  Data    -  Chemistry WebBook 
  -  National Vulnerability Database 
  -  Physical Reference Data 
  -  Standard Reference Data (SRD) 
-  Storefront 
-  License & Patents 
-  Computer Security Resource Center (CSRC) 
-  NIST Research Library 
 
            -  News & Events  -  News 
-  Events 
-  Blogs 
-  Feature Stories 
-  Awards 
-  Video Gallery 
-  Image Gallery 
-  Media Contacts 
 
            -  About NIST  -  About Us    -  Leadership 
  -  Organization Structure 
  -  Budget & Planning 
-  Contact Us 
-  Visit 
-  Careers    -  Student programs 
-  Work with NIST 
-  History    -  NIST Digital Archives 
  -  NIST Museum 
  -  NIST and the Nobel 
-  Educational Resources 
 

Information Technology LaboratoryInformation Access Division

## Multimodal Information Group

-  Tools 
-  Past HLT Evaluation Projects 
-  Staff 

# TRECVID 2015 Evaluation for Surveillance Event Detection

## Share

Facebook

Linkedin

Twitter

Email

The Interactive Surveillance Event Detection (SED) evaluation is a track within the TRECVID Evaluation. The goal of the 2015 Interactive SED evaluation track is to support the development of interactive (i.e., human in the loop) technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The interactive aspect of SED was added in 2012. The main evaluation will be implemented using a 9 hour subset of the multi-camera airport surveillance domain evaluation data collected by the Home Office Scientific Development Branch (HOSDB). A new *Group Dynamic Subset* using only 2 hours of this video and limited to the Embrace, PeopleMeet and PeopleSplitUp events is introduced in 2015.

The 2015 SED evaluation is the 8th evaluation of event detection technologies preceded by the 2008, 2009, 2010, 2011, 2012, 2013 and 2014 evaluations. The interactive evaluation methodology is similar to previous evaluations, i.e., systems report the same information and are scored with the same metrics, however, unlike previous evaluations the data set size will be smaller and each search for an event by a searcher can take no more than 25 elapsed minutes (wall clock time), measured from the time the searcher is given the event to look for until the time the result set is considered final. The system may use an unlimited amount of computational resources prior to beginning the interactive search. 

We have a mailing list for announcements/discussion -- contact sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov) to subscribe.

### Evaluation Planning 

The following resources are available:
                  - Evaluation Plans : Read the  TRECVID SED 2015 Evaluation Plan-V01
                  - Event Definitions: 
                  An annotation guidelines document 
TRECVid09_Event_Annotation_Guidelines_v1.0.pdf
 is available. 
The following paper 
EventDet-ViPER-format-v4.pdf
 documents the XML Schema Definition (XSD) proposed for a ViPER XML Representation of Events. 
The following 
TrecVid08_XSD_v2.tar
 includes the XSD and a sample XML file for events. 

                  - Experiment Control Files: 
                  The following 
TRECVid08_ecf_v2.tar
 includes the XSD and a sample XML file for Experiment Control Files (ECFs). 

### Software

Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page. You need at least F4DE 3.2.1 (released March 24th 2015) to use the SED15 SubmissionChecker tool.

### Dry Run (DEV15)

The dry run period for the 2015 TRECVID Interactive Surveillance Event Detection Evaluation will run through June 2015. Dry run submissions will be accepted at any time during that period. The procedure for participating is as follows:
                  - Sign up as a TRECVID Participant as specified on the  TRECVID  web site. 
                  - Obtain the data sets by completing the licensing agreement as specified below. The dry run will use part of the development corpus. 
                  - Download the Experiment Control File (ECF) [updated for 2015]:  expt_2015_allED_DEV15_ENG_s-camera_NIST_1.xml 
                  - Run experimental system(s) on the video intervals specified in the ECF. 
                  - Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details) 
                  - Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV15SED-SubmissionChecker tool.
                  - Notify  sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov) . Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person. 
                  - Submissions are due 11:59PM Eastern Time (UTC-4), June 30th, 2015. 
                  - Results of the dry run will be returned by July 7th, 2014 (approx.).

### Formal Evaluation (EVAL15)

Submissions for the formal evaluation are due July 22rd, 2015 **(7/20 update: submission deadline is extended until Monday July 27th at 1600 UTC)**. The procedure for participating is as follows:
                  - [Recommended] Complete the dry run as specified above. 
                  - Download the Experiment Control Files (ECF): There is only one ECF file defined in the eval plan for the interactive event detection and the retrospective event detection:    expt_2015_allED_EVAL15_ENG_s-camera_NIST_1.xml 
                  - Run experimental system(s) on the video intervals specified in the ECF.
                  - Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details).
                  - Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV15SED-SubmissionChecker tool. 
                  - Notify  sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov.)  Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person. 
                  - Submissions are due 11:59AM Eastern Daylight Time (UTC-4), July  27nd , 2015.
                  - Results of the formal evaluation will be returned by August 5th, 2015 (approx.).

**Group Dynamic Subset (SUB15)**

Submissions for the Group Dynamic Subset are due July 22rd, 2015 **(7/20 update: submission deadline is extended until Monday July 27th at 1600 UTC)**. The procedure for participating is as follows:
                  - [Recommended] Complete the dry run as specified above.
                  - Download the Experiment Control Files (ECF): There is only one ECF file defined in the eval plan for the interactive event detection and the retrospective event detection:   expt_2015_allED_SUB15_ENG_s-camera_NIST_1.xml 
                  - Run experimental system(s) on the video intervals specified in the ECF.
                  - Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details).
                  - Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV15SED-SubmissionChecker tool. 
                  - Notify  sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov) . Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person. 
                  - Submissions are due 11:59AM Eastern Daylight Time (UTC-4), July  27nd , 2015.
                  - Results of the Group Dynamic Subset will be returned by August 5th, 2015 (approx.). 

### Data Resources

                  - The data resources provided for 2015 are the same as previous SED evaluations. 
                  - For more information, see:  Licensing/Participation Agreements 

Information technologyData and informaticsImage and signal processing

### HEADQUARTERS

301-975-2000

Webmaster | Contact Us | Our Other Offices

TwitterFacebookLinkedInInstagramYouTubeGiphyRSS FeedMailing List

Feedback

            -  Site Privacy 
            -  Accessibility 
            -  Privacy Program 
            -  Copyrights 
            -  Vulnerability Disclosure 
            -  No Fear Act Policy 
            -  FOIA 
            -  Environmental Policy 
            -  Scientific Integrity 
            -  Information Quality Standards 
            -  Commerce.gov 
            -  Science.gov 
            -  USA.gov 
            -  Vote.gov