U.S. flag
        An official website of the United States government
          Here’s how you know
        Here’s how you know**Official websites use .gov**
              
              A **.gov** website belongs to an official government organization in the United States.
        **Secure .gov websites use HTTPS**
              
              A **lock** (  A locked padlock
) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. https://www.nist.gov/itl/iad/mig/openasr-challenge
  Search NIST
    Search
              Menu

      


Close
    Topics
        All Topics
                                                        Advanced communications
                                                        Artificial intelligence
                                                        Bioscience
                                                        Buildings and construction
                                                        Chemistry
                                                        Climate
                                                        Cybersecurity
                                                        Electronics
                                                        EnergyEnvironment
                                                        Fire
                                                        Forensic science
                                                        Health
                                                        Information technology
                                                        Infrastructure
                                                        Manufacturing
                                                        Materials
                                                        Mathematics and statisticsMetrology
                                                        Nanotechnology
                                                        Neutron research
                                                        Performance excellence
                                                        Physics
                                                        Public safety
                                                        Resilience
                                                        Standards
                                                        Transportation
            Publications
            Labs & Major Programs
        Laboratories
                                Communications Technology Laboratory
                                  Engineering Laboratory
                                  Information Technology Laboratory
                                  Material Measurement Laboratory
                                  Physical Measurement LaboratoryUser Facilities
                                NIST Center for Neutron Research
                                  CNST NanoFab
                                                        Research Test Beds
                                                        Research Projects
                                                        Tools & InstrumentsMajor Programs
                                Baldrige Performance Excellence Program
                                  CHIPS for America Initiative
                                  Manufacturing Extension Partnership (MEP)
                                  Office of Advanced Manufacturing
                                  Special Programs Office
                                  Technology Partnerships Office
            Services & Resources
        Standards and Measurements
                                Calibration Services
                                  Laboratory Accreditation (NVLAP)
                                  Quality System
                                  Standard Reference Materials (SRMs)
                                  Standards.gov
                                  Time Services
                                  Office of Weights and MeasuresSoftware
                                                        Data
                                Chemistry WebBook
                                  National Vulnerability Database
                                  Physical Reference Data
                                  Standard Reference Data (SRD)
                                                        Storefront
                                                        License & PatentsComputer Security Resource Center (CSRC)
                                                        NIST Research Library
            News & Events
        News
                                                        Events
                                                        Blogs
                                                        Feature Stories
                                                        AwardsVideo Gallery
                                                        Image Gallery
                                                        Media Contacts
            About NIST
        About Us
                                Leadership
                                  Organization Structure
                                  Budget & Planning
                                                        Contact Us
                                                        Visit
                                                        Careers
                                Student programsWork with NIST
                                                        History
                                NIST Digital Archives
                                  NIST Museum
                                  NIST and the Nobel
                                                        Educational Resources
 
 Information Technology Laboratory /                  Information Access Division
      Multimodal Information Group
    
      
  Tools
          Past HLT Evaluation Projects
          StaffOpenASR Challenge

Share
  Facebook
                      Linkedin
                      Twitter
                      Email

The goal of the OpenASR (Open Automatic Speech Recognition) Challenge is to assess the state of the art of ASR technologies for low-resource languages.The OpenASR Challenge is an open challenge created out of theIARPA(Intelligence Advanced Research Projects Activity)MATERIAL(Machine Translation for English Retrieval of Information in Any Language) program that encompasses more tasks, including CLIR (cross-language information retrieval), domain classification, and summarization. Also see NIST'sMATERIALpage.For every year of MATERIAL, NIST supports a simplified, smaller scale evaluation open to all, focusing on a particular technology aspect of MATERIAL. CLIR technologies were the focus of the first open challenge in 2019,OpenCLIR. Since 2020, the focus has been on ASR. The capabilities tested in the open challenges are expected to ultimately support the MATERIAL task of effective triage and analysis of large volumes of text and audio content in a variety of less-studied languages.OpenASR-Related PublicationsOpenASR21: The Second Open Challenge for Automatic Speech Recognition of Low-Resource LanguagesOpenASR20: An Open Challenge for Automatic Speech Recognition of Conversational Telephone Speech in Low-Resource Languages**Contact**Please email openasr_poc [at] nist.gov(openasr_poc[at]nist[dot]gov) for any questions or comments regarding the OpenASR Challenge.OpenASR21 ChallengeThe second OpenASR Challenge associated with MATERIAL, OpenASR21, opened for registration August 9, 2021. The evaluation period in November 2021. OpenASR21 features ASR evaluation opportunities for 15 low-resource languages:All ten languages from OpenASR20**Five NEW languages for OpenASR21**For the languages from OpenASR20, the same evaluation datasets from 2020 will be used, consisting of conversational telephone speech (CTS) data. For the five new languages, the main evaluation dataset will also consist of CTS data. These datasets will be scored (where applicable) case-insensitively.**New** for OpenASR21 will be **case-sensitive scoring** for three of the new languages, as indicated below. Case-sensitive scoring will be performed for system output from separate evaluation datasets from a mix of genres for these languages, in order to assess low-resource ASR performance specifically on proper nouns.OpenASR21 languages:AmharicCantonese**New: Farsi****New: Georgian**GuaraniJavanese**New: Kazakh (including additional evaluation dataset for case-sensitive scoring)**Kurmanji KurdishMongolianPashtoSomali**New: Swahili (including additional evaluation dataset for case-sensitive scoring)****New: Tagalog (including additional evaluation dataset for case-sensitive scoring)**TamilVietnameseOpenASR21 will be implemented as a track of NIST’sOpenSAT(Open Speech Analytic Technologies) evaluation series, using theOpenSAT web serverfor registration, data access, submission, and scoring purposes.For more details, please refer to the OpenASR21 Challenge Evaluation Plan in the Documentation and Resources section below.Schedule**Milestone****Date**Evaluation plan releaseJuly 2021**Registration period****August 9 – October 15, 2021****Development period****August 9 – November 2, 2021 (potentially longer but excluding evaluation period)**- Build and Dev datasets releaseAugust 9, 2021- Scoring server accepts submissions for Dev datasetsAugust 30 – November 2, 2021 (potentially longer but excluding evaluation period)Registration closesOctober 15, 2021**Evaluation period****November 3 – 10, 2021**- Release of Eval datasetsNovember 3, 2021- Scoring server accepts submissionsNovember 4 – 10, 2021- System output due at NISTNovember 10, 2021System description due at NISTNovember 19, 2021RegistrationRegistration opened on August 9, 2021. Please register via the OpenSAT web server.Documentation and ResourcesOpenASR21 Challenge Evaluation Plan v1_3(August 31, 2021)IARPA Babel Program Data Specifications for PerformersIARPA MATERIAL Program Transcription ConventionsOpenASR toolkit (for file conversion, normalization, validation) v0.1.2Speech Recognition Scoring Toolkit SCTK (includes SCLITE, ASCLITE, tranfilt, hubscr, SLATreport and utf_filt scoring tools)ResultsThe OpenASR21 evaluation was conducted in November 2021. Please see the OpenASR21 Challenge Results page.Low-Resource ASR Development Special Session at INTERSPEECH 2022OpenASR21 participants, as well as others working in the low-resource ASR problem space, are strongly encouraged to submit their work to the preliminarily accepted Low-Resource ASR Development special session at INTERSPEECH 2022. Please see the Call for Papers. This special session welcomes contributions from anyone working in the low-resource ASR problem space.OpenASR20 ChallengeThe first OpenASR Challenge associated with MATERIAL, OpenASR20, was opened for registration in July 2020, with an evaluation period in November 2020. It featured ASR evaluation opportunities for these ten low-resource languages:AmharicCantoneseGuaraniJavaneseKurmanji KurdishMongolianPashtoSomaliTamilVietnameseIt was implemented as a track of NIST’sOpenSAT(Open Speech Analytic Technologies) evaluation series, using theOpenSAT web serverfor registration, data access, submission, and scoring purposes.The evaluation plan posted in the Documentation and Resources section below describes the OpenASR20 Challenge in detail.RegistrationRegistration for the OpenASR20 Challenge is now closed.Documentation and ResourcesOpenASR20 Challenge Evaluation Planv1_5 (October 8, 2020)IARPA Babel Program Data Specifications for PerformersOpenASR toolkit (for file conversion, normalization, validation) v0.1.1Speech Recognition Scoring Toolkit SCTK (includes SCLITE, ASCLITE, tranfilt, hubscr, SLATreport and utf_filt scoring tools)ResultsThe OpenASR20 evaluation was conducted in November 2020. Please see the OpenASR20 Challenge Results page.OpenASR20 and Low-Resource ASR Development Special Session at INTERSPEECH 2021OpenASR20 participants, as well as others working in the low-resource ASR problem space, were strongly encouraged to submit their work to an OpenASR special session at INTERSPEECH 2021. Please see the OpenASR20 and Low-Resource ASR Development Special Session at INTERSPEECH 2021 Call for Papers. This special session also welcomed contributions from others working in the low-resource ASR problem space who did not participate in OpenASR20.

    
    Created June 10, 2020, Updated February 15, 2024
 HEADQUARTERS
            100 Bureau Drive
              Gaithersburg, MD 20899301-975-2000
            Webmaster | Contact Us | Our Other Offices
        Twitter
            
Facebook
            
LinkedIn
            
Instagram
            
YouTube
            
Giphy
            
RSS Feed
            
Mailing List
                              How are we doing? Feedback

  
  Site Privacy
                      Accessibility
                      Privacy Program
                      Copyrights
                      Vulnerability Disclosure
                      No Fear Act Policy
                      FOIA
                      Environmental Policy
                      Scientific Integrity
                      Information Quality Standards
                      Commerce.gov
                      Science.gov
                      USA.gov
                      Vote.gov