Skip to main content    

An official website of the United States government

Here’s how you know

**Official websites use .gov**              A **.gov** website belongs to an official government organization in the United States.            

**Secure .gov websites use HTTPS**              A **lock** (  LockA locked padlock) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.            

https://www.nist.gov/itl/iad/mig/openasr20-challenge-results

            -  Topics  -  All Topics 
-  Advanced communications 
-  Artificial intelligence 
-  Bioscience 
-  Buildings and construction 
-  Chemistry 
-  Climate 
-  Cybersecurity 
-  Electronics 
-  Energy 
-  Environment 
-  Fire 
-  Forensic science 
-  Health 
-  Information technology 
-  Infrastructure 
-  Manufacturing 
-  Materials 
-  Mathematics and statistics 
-  Metrology 
-  Nanotechnology 
-  Neutron research 
-  Performance excellence 
-  Physics 
-  Public safety 
-  Resilience 
-  Standards 
-  Transportation 
 
            -  Publications 
            -  Labs & Major Programs  -  Laboratories    -  Communications Technology Laboratory 
  -  Engineering Laboratory 
  -  Information Technology Laboratory 
  -  Material Measurement Laboratory 
  -  Physical Measurement Laboratory 
-  User Facilities    -  NIST Center for Neutron Research 
  -  CNST NanoFab 
-  Research Test Beds 
-  Research Projects 
-  Tools & Instruments 
-  Major Programs    -  Baldrige Performance Excellence Program 
  -  CHIPS for America Initiative 
  -  Manufacturing Extension Partnership (MEP) 
  -  Office of Advanced Manufacturing 
  -  Special Programs Office 
  -  Technology Partnerships Office 
 
            -  Services & Resources  -  Standards and Measurements    -  Calibration Services 
  -  Laboratory Accreditation (NVLAP) 
  -  Quality System 
  -  Standard Reference Materials (SRMs) 
  -  Standards.gov 
  -  Time Services 
  -  Office of Weights and Measures 
-  Software 
-  Data    -  Chemistry WebBook 
  -  National Vulnerability Database 
  -  Physical Reference Data 
  -  Standard Reference Data (SRD) 
-  Storefront 
-  License & Patents 
-  Computer Security Resource Center (CSRC) 
-  NIST Research Library 
 
            -  News & Events  -  News 
-  Events 
-  Blogs 
-  Feature Stories 
-  Awards 
-  Video Gallery 
-  Image Gallery 
-  Media Contacts 
 
            -  About NIST  -  About Us    -  Leadership 
  -  Organization Structure 
  -  Budget & Planning 
-  Contact Us 
-  Visit 
-  Careers    -  Student programs 
-  Work with NIST 
-  History    -  NIST Digital Archives 
  -  NIST Museum 
  -  NIST and the Nobel 
-  Educational Resources 
 

# OpenASR20 Challenge Results

## Share

Facebook

Linkedin

Twitter

Email

The OpenASR (Open Automatic Speech Recognition) 2020 Challenge was the second open challenge associated with the IARPA MATERIAL program, after the OpenCLIR (Open Cross-Language Information Retrieval) 2019 Challenge. Capabilities tested in these open challenges are expected to ultimately support the MATERIAL task of effective triage and analysis of large volumes of text and audio content in a variety of less-studied languages. OpenASR20 was implemented as a track of NIST’s OpenSAT (Speech Analytic Technologies) evaluation series.

The goal of the OpenASR20 challenge was to assess the state of the art of automatic speech recognition (ASR) technologies for low-resource languages. ASR was performed on speech datasets, and written text output had to be produced.

Please refer to the OpenASR20 Challenge Evaluation Plan for a full description of the challenge and its rules and procedures.

## **Languages**

OpenASR20 was offered for the following ten low-resource languages, of which participants could attempt as many as they wished:
                  - Amharic (AMH)
                  - Cantonese (CAN)
                  - Guarani (GUA)
                  - Javanese (JAV)
                  - Kurmanji Kurdish (KUR)
                  - Mongolian (MON)
                  - Pashto (PAS)
                  - Somali (SOM)
                  - Tamil (TAM)
                  - Vietnamese (VIE)

## **Data**

The data for the challenge consisted of conversational telephone speech stemming from the IARPA Babel program, with the exception of Somali which stemmed from MATERIAL. More details regarding technical data details can be found in section 3 of the IARPA Babel Data Specifications for Performers. For each language, separate training, development, and evaluation datasets were provided.

## **Training Conditions**

The challenge offered two training conditions: 
                  - Constrained training (mandatory):                      - Speech data: Limited to a 10-hour subset designated for Constrained training in provided training dataset for the language in question
                    - Non-speech data: Any publicly available data
                  - Unconstrained training (optional): Any publicly available data

## **Metrics**

                  - Primary metric: Word Error Rate (WER)
                  - Additional metrics:                      - Character Error Rate (CER)
                    - Time and memory resources used (self-reported)

## **Schedule**

The most important milestones of the schedule of the challenge were as follows:
                  - Registration: August - October, 2020
                  - Development period: August 2020 - June 15, 2021
                  - Evaluation period: November 3-10, 2020

## **Participation**

28 teams from 12 countries registered to participate, out of which nine fully completed the challenge (i.e. submitted valid output for at least one language under the Constrained training condition). Table 1 lists the fully participating teams.

Table 1: OpenASR20 Participants

## **Results**

Table 2 lists the best WER result achieved by each team, ordered by language, training condition (Unconstrained submissions in *italics*), and WER score. Late submissions are marked as such and listed at the bottom of the table. Self-reported time and memory resources are not included in this overview of results.

Table 2: OpenASR20 Results. † = late submission.

## System Descriptions

As part of the evaluation submission, participants were required to include a paper to describe their systems. Participants were also encouraged to submit their work to be included in the OpenASR and Low-Resource ASR Development Special Session at INTERSPEECH 2021. The system descriptions with consent to be released publicly are provided below:
                  - Catskills OpenASR20 System Description
                  - NSYSU-MITLab OpenASR20 System Description
                  - TalTech OpenASR20 System Description
                  - THUEE OpenASR20 System Description
                  - TNT+MMT OpenASR20 System Description

## Disclaimer

NIST serves to coordinate the evaluations in order to support research and to help advance the state- of-the-art. NIST evaluations are not viewed as a competition, and such results reported by NIST are not to be construed, or represented, as endorsements of any participant’s system, or as official findings on the part of NIST or the U.S. Government.

## **Contact**

Please email openasr_poc [at] nist.gov (openasr_poc[at]nist[dot]gov) for any questions or comments regarding the OpenASR Challenge.

Information technology

### HEADQUARTERS

301-975-2000

Webmaster | Contact Us | Our Other Offices

TwitterFacebookLinkedInInstagramYouTubeGiphyRSS FeedMailing List

Feedback

            -  Site Privacy 
            -  Accessibility 
            -  Privacy Program 
            -  Copyrights 
            -  Vulnerability Disclosure 
            -  No Fear Act Policy 
            -  FOIA 
            -  Environmental Policy 
            -  Scientific Integrity 
            -  Information Quality Standards 
            -  Commerce.gov 
            -  Science.gov 
            -  USA.gov 
            -  Vote.gov