U.S. flag
        An official website of the United States government
          Here’s how you know
        Here’s how you know**Official websites use .gov**
              
              A **.gov** website belongs to an official government organization in the United States.
        **Secure .gov websites use HTTPS**
              
              A **lock** (  A locked padlock
) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. https://www.nist.gov/itl/iad/mig/opensat
  Search NIST
    Search
              Menu

      


Close
    Topics
        All Topics
                                                        Advanced communications
                                                        Artificial intelligence
                                                        Bioscience
                                                        Buildings and construction
                                                        Chemistry
                                                        Climate
                                                        Cybersecurity
                                                        Electronics
                                                        EnergyEnvironment
                                                        Fire
                                                        Forensic science
                                                        Health
                                                        Information technology
                                                        Infrastructure
                                                        Manufacturing
                                                        Materials
                                                        Mathematics and statisticsMetrology
                                                        Nanotechnology
                                                        Neutron research
                                                        Performance excellence
                                                        Physics
                                                        Public safety
                                                        Resilience
                                                        Standards
                                                        Transportation
            Publications
            Labs & Major Programs
        Laboratories
                                Communications Technology Laboratory
                                  Engineering Laboratory
                                  Information Technology Laboratory
                                  Material Measurement Laboratory
                                  Physical Measurement LaboratoryUser Facilities
                                NIST Center for Neutron Research
                                  CNST NanoFab
                                                        Research Test Beds
                                                        Research Projects
                                                        Tools & InstrumentsMajor Programs
                                Baldrige Performance Excellence Program
                                  CHIPS for America Initiative
                                  Manufacturing Extension Partnership (MEP)
                                  Office of Advanced Manufacturing
                                  Special Programs Office
                                  Technology Partnerships Office
            Services & Resources
        Standards and Measurements
                                Calibration Services
                                  Laboratory Accreditation (NVLAP)
                                  Quality System
                                  Standard Reference Materials (SRMs)
                                  Standards.gov
                                  Time Services
                                  Office of Weights and MeasuresSoftware
                                                        Data
                                Chemistry WebBook
                                  National Vulnerability Database
                                  Physical Reference Data
                                  Standard Reference Data (SRD)
                                                        Storefront
                                                        License & PatentsComputer Security Resource Center (CSRC)
                                                        NIST Research Library
            News & Events
        News
                                                        Events
                                                        Blogs
                                                        Feature Stories
                                                        AwardsVideo Gallery
                                                        Image Gallery
                                                        Media Contacts
            About NIST
        About Us
                                Leadership
                                  Organization Structure
                                  Budget & Planning
                                                        Contact Us
                                                        Visit
                                                        Careers
                                Student programsWork with NIST
                                                        History
                                NIST Digital Archives
                                  NIST Museum
                                  NIST and the Nobel
                                                        Educational Resources
 
 Information Technology Laboratory /                  Information Access Division
      Multimodal Information Group
    
      
  Tools
          Past HLT Evaluation Projects
          StaffOpenSAT

Share
  Facebook
                      Linkedin
                      Twitter
                      Email

Open Speech Analytic Technologies (OpenSAT) Evaluation Series

**The OpenSAT20 reopened June 18th and the schedule has been updated.**

The next evaluation will be the OpenSAT20 Evaluation. OpenSAT20 will continue with the same tasks: Automatic Speech Recognition (ASR), Speech Activity Detection (SAD), and Keyword Search (KWS).

The data domain for the OpenSAT20 Evaluation will be simulated public safety communicationsspoken in English. The evaluation data will be extracted from unexposed portions of the SAFE-T corpus that wascollected bytheLinguisticDataConsortium (LDC)and initially made availablefor theOpenSAT19Evaluation. Theaudio recordings in the SAFE-T corpuscontainspeech potentially with increased vocal effort induced byfirst-responder type background noiseconditions andis expected to be challenging for systems to process with a high degree of accuracy.

NIST intends to continue with this public-safety speech corpus in the OpenSAT series tomeasureyear-to-year system performance progress.The NIST Speech Analytic Technologies evaluation series (OpenSAT) goal is to provide broad support for the advancement of speech analytic technologies by including multiple speech analytic tasks and multiple data domains. Developers can choose from one to all tasks and from one to all data domains.

OpenSAT20 will beorganized in similar manner toOpenSAT19except as follows:

Only one data domain (public safety communications) will be available (as opposed to three domains).
	The evaluation data package will consist ofunexposedeval data(denoted as Test set)plus OpenSAT19 eval data(denoted as Progress set).
	Live public leaderboards will display scores fortheProgress setfor eachtask, and participants will have the option to anonymize their team names on the public leaderboard.
	System descriptions will be available to all participants from the OpenSAT20 web site.
	For those who cannot physically attend the workshop, provisions will be made to attend remotely.Current 2020 Schedule - updated

OpenSAT20 registration available until July 31, 2020
Training, Development, and Evaluation data available until July 31  
Scores posted to leaderboards for the Progress set portion of the evaluation data until July 31
Last date to upload system output to NIST for scoring is July 31 
Scores for the Test set portion of the evaluation data made available after July 31 (if a system description is uploaded)
Virtual Workshop will be held on September 16-17, 2020. Click **HERE**to register. Last day to register is September 04

Go to the **OpenSAT**website for more information and to register.

OpenSAT20 Objectives:

for NIST to measure performance of the state-of-the-art speech analytics technologies in thepublic safety communications domain,
	to provide a forum for the speech analytics community to further test and develop multiple technologies in parallel with other developers using a common data set, and
	to enable opportunities for sharing and leveraging knowledge by bringing together developers whose primary focus may be the same analytic task or one of the other two tasks.**Download the** 2020 OpenSAT Evaluation Plan V1.6 (pdf). Updated July 1, 2020

Send email to opensat_poc [at] nist.gov(opensat_poc[at]nist[dot]gov) with request to be added to the mailing list, to receive updates, or to ask questions or leave comments.

**OpenSAT19 (schedule updated 2/06/19)**

OpenSAT19 Evaluation Plan (Updated 3/28/2019)
03/29/2018 - 06-14-2019     Development data release (updated dates)
06/17/2019 - 07/01/2019     Evaluation data release (updated date)
08/20/2019 - 08/21/2019     Post Evaluation Workshop

**Tasks**
Speech Activity Detection (SAD)
Automatic Speech Recognition (ASR)
Key Word Search (KWS)

**Data**
For SAD, ASR, KWS tasks     Low Resource Language - (Pashto language) from the IARPA Babel collection
For SAD, KWS tasks                Audio extracted from amateur online videos - from the Video Annotation for Speech Technologies (VAST) collection (English language) 
For SAD, ASR, KWS tasks     Simulated public safety communications - from the PSC collection (English language)

**OpenSAT Pilot 2017**

**Tasks**
Speech Activity Detection (SAD)
Automatic Speech Recognition (ASR)
Key Word Search (KWS)

**Data**
For SAD, ASR, KWS tasks     Low Resource Language - from the IARPA Babel collection (Pashto language)
For SAD task only                   Audio extracted from YouTube videos - from the Video Annotation for Speech Technologies (VAST) collection (Arabic, Mandarin and English languages) 
For SAD, ASR, KWS tasks     First responder/dispatcher operational recordings - from the June 18th 2007, Charleston, South Carolina, Sofa Super Store Fire (English language)

**Documentation**Open Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation PlanOpen Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation Report

    
    Created September 30, 2016, Updated August 28, 2020
 HEADQUARTERS
            100 Bureau Drive
              Gaithersburg, MD 20899301-975-2000
            Webmaster | Contact Us | Our Other Offices
        Twitter
            
Facebook
            
LinkedIn
            
Instagram
            
YouTube
            
Giphy
            
RSS Feed
            
Mailing List
                              How are we doing? Feedback

  
  Site Privacy
                      Accessibility
                      Privacy Program
                      Copyrights
                      Vulnerability Disclosure
                      No Fear Act Policy
                      FOIA
                      Environmental Policy
                      Scientific Integrity
                      Information Quality Standards
                      Commerce.gov
                      Science.gov
                      USA.gov
                      Vote.gov