U.S. flag
        An official website of the United States government
          Here’s how you know
        Here’s how you know**Official websites use .gov**
              
              A **.gov** website belongs to an official government organization in the United States.
        **Secure .gov websites use HTTPS**
              
              A **lock** (  A locked padlock
) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. https://www.nist.gov/itl/iad/mig/trecvid-2014-evaluation-surveillance-event-detection
  Search NIST
    Search
              Menu

      


Close
    Topics
        All Topics
                                                        Advanced communications
                                                        Artificial intelligence
                                                        Bioscience
                                                        Buildings and construction
                                                        Chemistry
                                                        Climate
                                                        Cybersecurity
                                                        Electronics
                                                        EnergyEnvironment
                                                        Fire
                                                        Forensic science
                                                        Health
                                                        Information technology
                                                        Infrastructure
                                                        Manufacturing
                                                        Materials
                                                        Mathematics and statisticsMetrology
                                                        Nanotechnology
                                                        Neutron research
                                                        Performance excellence
                                                        Physics
                                                        Public safety
                                                        Resilience
                                                        Standards
                                                        Transportation
            Publications
            Labs & Major Programs
        Laboratories
                                Communications Technology Laboratory
                                  Engineering Laboratory
                                  Information Technology Laboratory
                                  Material Measurement Laboratory
                                  Physical Measurement LaboratoryUser Facilities
                                NIST Center for Neutron Research
                                  CNST NanoFab
                                                        Research Test Beds
                                                        Research Projects
                                                        Tools & InstrumentsMajor Programs
                                Baldrige Performance Excellence Program
                                  CHIPS for America Initiative
                                  Manufacturing Extension Partnership (MEP)
                                  Office of Advanced Manufacturing
                                  Special Programs Office
                                  Technology Partnerships Office
            Services & Resources
        Standards and Measurements
                                Calibration Services
                                  Laboratory Accreditation (NVLAP)
                                  Quality System
                                  Standard Reference Materials (SRMs)
                                  Standards.gov
                                  Time Services
                                  Office of Weights and MeasuresSoftware
                                                        Data
                                Chemistry WebBook
                                  National Vulnerability Database
                                  Physical Reference Data
                                  Standard Reference Data (SRD)
                                                        Storefront
                                                        License & PatentsComputer Security Resource Center (CSRC)
                                                        NIST Research Library
            News & Events
        News
                                                        Events
                                                        Blogs
                                                        Feature Stories
                                                        AwardsVideo Gallery
                                                        Image Gallery
                                                        Media Contacts
            About NIST
        About Us
                                Leadership
                                  Organization Structure
                                  Budget & Planning
                                                        Contact Us
                                                        Visit
                                                        Careers
                                Student programsWork with NIST
                                                        History
                                NIST Digital Archives
                                  NIST Museum
                                  NIST and the Nobel
                                                        Educational Resources
 
 Information Technology Laboratory /                  Information Access Division
      Multimodal Information Group
    
      
  Tools
          Past HLT Evaluation Projects
          StaffTRECVID 2014 Evaluation for Surveillance Event Detection

Share
  Facebook
                      Linkedin
                      Twitter
                      Email

TRECVID 2014 Evaluation for Surveillance Event Detection

The Interactive Surveillance Event Detection (SED) evaluation is a track within the TRECVID Evaluation. The goal of the 2014 Interactive SED evaluation track is to support the development of interactive (i.e., human in the loop) technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The interactive aspect of SED was added in 2012. The evaluation will be implemented using an 11 hour subset of the multi-camera airport surveillance domain evaluation data collected by the Home Office Scientific Development Branch (HOSDB).

The 2014 SED evaluation is the 6th evaluation of event detection technologies preceded by the 2008, 2009, 2010, 2011, 2012 and 2013 evaluations. The interactive evaluation methodology is similar to previous evaluation, i.e., systems report the same information and are scored with the same metrics, however, unlike previous evaluations the data set size will be smaller and each search for an event by a searcher can take no more than 25 elapsed minutes (wall clock time), measured from the time the searcher is given the event to look for until the time the result set is considered final. The system may use an unlimited amount of computional resources prior to beginning the interactive search.

We have a mailing list for announcements/discussion -- contact martial.michel [at] nist.gov(martial[dot]michel[at]nist[dot]gov) to subscribe.

Evaluation Planning

The following resources are available:

**Evaluation Plans**: Read the TRECVID SED 2014 Evaluation Plan-V01
	Event Definitions:
		An annotation guidelines document TRECVid09_Event_Annotation_Guidelines_v1.0.pdf is available.
			The following paper EventDet-ViPER-format-v4.pdf documents the XML Schema Definition (XSD) proposed for a ViPER XML Representation of Events.
			The following TrecVid08_XSD_v2.tar includes the XSD and a sample XML file for events.
	Experiment Control Files:
		The following TRECVid08_ecf_v2.tar includes the XSD and a sample XML file for Experiment Control Files (ECFs).Software

Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page. You need at least F4DE 3.1.1 (released July 3rd 2014) to use the SED14 SubmissionChecker tool.

Dry Run

The dry run period for the 2014 TRECVID Interactive Surveillance Event Detection Evaluation will run until July 16th 2014. Dry run submissions will be accepted at any time during that period. The procedure for participating is as follows:

Sign up as a TRECVID Participant as specified on the TRECVID web site.
	Obtain the data sets by completing the licensing agreement as specifiec below. The dry run will use part of the development corpus.
	Download the Experiment Control File (ECF) [same as 2009, 2010, 2011, 2012 and 2013]: expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
	Run experimental system(s) on the video intervals specified in the ECF.
	Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details)
	Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV14SED-SubmissionChecker tool. Note: this has not yet been released.
	Notify martial.michel [at] nist.gov(martial[dot]michel[at]nist[dot]gov). Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person.
	Submissions are due 11:59PM Eastern Time (UTC-4), July 16th, 2014 (extended from June 30th)
	Results of the dry run will be returned by July 20th, 2014 (approx.).Formal Evaluation

Submissions for the formal evaluation are due July 23rd, 2014. The procedure for participating is as follows:

If possible: Complete the dry run as specified above.
	**[UPDATED]** Download the Experiment Control Files (ECF): There is only one ECF file defined in the eval plan for the interactive event detection and the retrospective event detection: expt_2014_allED_EVAL14_ENG_s-camera_NIST_1.xml(different from previous years' ECF files in that 9 of the 11 hours were not evaluated in the past). **Note:** the reference data for the Formal Evaluation is being bootstrapped and will evolve as part of the evaluation process.
	Run experimental system(s) on the video intervals specified in the ECF.
	Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details).
	Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV14SED-SubmissionChecker tool.
	Notify martial.michel [at] nist.gov(martial[dot]michel[at]nist[dot]gov). Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person.
	Submissions are due 11:59AM Eastern Daylight Time (UTC-4), July 23rd, 2014.
	Results of the formal evaluation will be returned by July 30th, 2014 (approx.).Data Resources

The data resources provided for 2014 are the same as 2011.
	For more information, see: Licensing/Participation Agreements
  Information technology, Data and informatics and Image and signal processing

    
    Created June 13, 2014, Updated April 2, 2019
 HEADQUARTERS
            100 Bureau Drive
              Gaithersburg, MD 20899301-975-2000
            Webmaster | Contact Us | Our Other Offices
        Twitter
            
Facebook
            
LinkedIn
            
Instagram
            
YouTube
            
Giphy
            
RSS Feed
            
Mailing List
                              How are we doing? Feedback

  
  Site Privacy
                      Accessibility
                      Privacy Program
                      Copyrights
                      Vulnerability Disclosure
                      No Fear Act Policy
                      FOIA
                      Environmental Policy
                      Scientific Integrity
                      Information Quality Standards
                      Commerce.gov
                      Science.gov
                      USA.gov
                      Vote.gov