Skip to main content    

An official website of the United States government

Here’s how you know

**Official websites use .gov**              A **.gov** website belongs to an official government organization in the United States.            

**Secure .gov websites use HTTPS**              A **lock** (  LockA locked padlock) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.            

https://www.nist.gov/itl/iad/mig/openasr21-challenge-results

            -  Topics  -  All Topics 
-  Advanced communications 
-  Artificial intelligence 
-  Bioscience 
-  Buildings and construction 
-  Chemistry 
-  Climate 
-  Cybersecurity 
-  Electronics 
-  Energy 
-  Environment 
-  Fire 
-  Forensic science 
-  Health 
-  Information technology 
-  Infrastructure 
-  Manufacturing 
-  Materials 
-  Mathematics and statistics 
-  Metrology 
-  Nanotechnology 
-  Neutron research 
-  Performance excellence 
-  Physics 
-  Public safety 
-  Resilience 
-  Standards 
-  Transportation 
 
            -  Publications 
            -  Labs & Major Programs  -  Laboratories    -  Communications Technology Laboratory 
  -  Engineering Laboratory 
  -  Information Technology Laboratory 
  -  Material Measurement Laboratory 
  -  Physical Measurement Laboratory 
-  User Facilities    -  NIST Center for Neutron Research 
  -  CNST NanoFab 
-  Research Test Beds 
-  Research Projects 
-  Tools & Instruments 
-  Major Programs    -  Baldrige Performance Excellence Program 
  -  CHIPS for America Initiative 
  -  Manufacturing Extension Partnership (MEP) 
  -  Office of Advanced Manufacturing 
  -  Special Programs Office 
  -  Technology Partnerships Office 
 
            -  Services & Resources  -  Standards and Measurements    -  Calibration Services 
  -  Laboratory Accreditation (NVLAP) 
  -  Quality System 
  -  Standard Reference Materials (SRMs) 
  -  Standards.gov 
  -  Time Services 
  -  Office of Weights and Measures 
-  Software 
-  Data    -  Chemistry WebBook 
  -  National Vulnerability Database 
  -  Physical Reference Data 
  -  Standard Reference Data (SRD) 
-  Storefront 
-  License & Patents 
-  Computer Security Resource Center (CSRC) 
-  NIST Research Library 
 
            -  News & Events  -  News 
-  Events 
-  Blogs 
-  Feature Stories 
-  Awards 
-  Video Gallery 
-  Image Gallery 
-  Media Contacts 
 
            -  About NIST  -  About Us    -  Leadership 
  -  Organization Structure 
  -  Budget & Planning 
-  Contact Us 
-  Visit 
-  Careers    -  Student programs 
-  Work with NIST 
-  History    -  NIST Digital Archives 
  -  NIST Museum 
  -  NIST and the Nobel 
-  Educational Resources 
 

# OpenASR21 Challenge Results

## Share

Facebook

Linkedin

Twitter

Email

The OpenASR (Open Automatic Speech Recognition) 2021 Challenge was the third open challenge and the second OpenASR challenge associated with the IARPA MATERIAL program, after the OpenCLIR (Open Cross-Language Information Retrieval) 2019 Challenge and the OpenASR 2020 Challenge Capabilities tested in these open challenges are expected to ultimately support the MATERIAL task of effective triage and analysis of large volumes of text and audio content in a variety of less-studied languages. OpenASR21 was implemented as a track of NIST’s OpenSAT (Speech Analytic Technologies) evaluation series.

The goal of the OpenASR21 challenge was to assess the state of the art of automatic speech recognition (ASR) technologies for low-resource languages. ASR was performed on speech datasets, and written text output had to be produced.

Please refer to the OpenASR21 Challenge Evaluation Plan for a full description of the challenge and its rules and procedures.

## **Languages and Casing**

OpenASR21 was offered for 15 low-resource languages, of which participating teams could attempt as many as they wished. Ten of these languages were repeated from OpenASR20 with unchanged datasets; five languages were new. All languages were offered with a case-insensitive scoring dataset. Of the five new languages, three were offered with an additional case-sensitive scoring dataset. The case-sensitive scoring was to serve as a proxy for assessment of ASR performance on proper nouns. The new languages are marked as such below:
                  - Amharic (AMH)
                  - Cantonese (CAN)
                  - New: Georgian (GEO)
                  - New: Farsi (FAR)
                  - Guarani (GUA)
                  - Javanese (JAV)
                  - New: Kazakh (KAZ) (including additional case-sensitive scoring dataset)
                  - Kurmanji Kurdish (KUR)
                  - Mongolian (MON)
                  - Pashto (PAS)
                  - Somali (SOM)
                  - New: Swahili (SWA) (including additional case-sensitive scoring dataset)
                  - New: Tagalog (TAG) (including additional case-sensitive scoring dataset)
                  - Tamil (TAM)
                  - Vietnamese (VIE)

## **Data**

The data for the challenge consisted of speech data stemming from the IARPA Babel program, with the exception of Somali. which stemmed from the IARPA MATERIAL program. More details regarding technical data details can be found in section 3 of the IARPA Babel Data Specifications for Performers. For each language (and case scoring condition, where applicable), separate training, development, and evaluation datasets were provided.

For CIS, the datasets for most of the languages stemmed from the IARPA Babel program. The Somali and Farsi datasets stemmed from the IARPA MATERIAL program. For the case-sensitive scoring datasets, all data stemmed from the IARPA MATERIAL program.

The data consisted of conversational speech for the case-insensitive scoring datasets. For the case-sensitive scoring datasets, the data consisted of a mix of three genres: conversational speech, news broadcast, and topical broadcast.

## **Training Conditions**

The challenge offered three training conditions. The Constrained-plus training was new for OpenASR21.
                  - Constrained  training (mandatory):	                     - Speech data: Limited to a 10-hour subset designated for Constrained training in provided training dataset for the language in question
                    - Non-speech data: Any publicly available data
                  - New: Constrained-plus  training (optional):	                     - Same training data restrictions as Constrained Training, but additionally allowed publicly available and previously existing speech pretrained models as follows:		                       - Pretrained models created from unlabeled speech data in any language
                      - Pretrained models created from labeled speech data in any language except the language being processed
                  - Unconstrained  training (optional):	                     - Any publicly available data

## **Metrics**

                  - Primary metric: Word Error Rate (WER)
                  - Additional metrics:	                     - Character Error Rate (CER)
                    - Time and memory resources used (self-reported)

## **Schedule**

The most important milestones of the schedule of the challenge were as follows:
                  - Registration: August - October, 2021
                  - Evaluation period: November 3-10, 2021

## **Participation**

Table 1 lists the teams that submitted valid output, and for which languages they submitted valid output.

26 teams from 13 countries originally registered to participate. Fifteen teams submitted valid output for at least one language under the Constrained training condition.

The table marks the following issues in the pertinent table cells:
                  - The team only made late submissions (up to seven days after the official evaluation period ended) for the language in question.
                  - The team did not make a required Constrained training condition submission for the language in question.
                  - The team did not submit the required system description.

Table 1: OpenASR21 Participating teams and languages.

## **Results**

Table 2 lists the best WER result achieved by each team, ordered by language, scoring, training condition, and WER score. The CER score for the same submission is also provided.

The table marks the following issues in the Notes column:
                  - The submission was late (up to seven days after the official evaluation period ended). For some cases, this results in two best WER scores shown for a given team, language, and conditions - their best score, which was for a late submission, as well as their best on-time score. Late submissions are listed at the bottom of the table.
                  - The team did not make a required Constrained training condition submission for the language in question.
                  - The team did not submit the required system description.

Self-reported time and memory resources are not included in this overview of results.

Table 2: OpenASR21 Results. Submissions with best WER for each participating team by language, scoring condition, and training condition.

## System Descriptions

As part of the evaluation submission, teams were required to include a paper to describe their systems. Submitted system descriptions are provided below.
                  - Catskills OpenASR21 System Description
                  - CRIM OpenASR21 System Description
                  - I2R OpenASR21 System Description
                  - MMT OpenASR21 System Description: See TNT and MMT OpenASR21 System Description
                  - SRIB_ASR OpenASR21 System Description
                  - TalTech OpenASR21 System Description
                  - THUEE OpenASR21 System Description
                  - TNT and MMT OpenASR21 System Description
                  - ustc-nelslip OpenASR21 System Description
                  - zxy OpenASR21 System Description

Teams were also encouraged to submit their updated work to be included in a Low-Resource ASR Development Special Session at INTERSPEECH 2022.

## Disclaimer

NIST serves to coordinate the evaluations in order to support research and to help advance the state- of-the-art. NIST evaluations are not viewed as a competition, and such results reported by NIST are not to be construed, or represented, as endorsements of any team's system, or as official findings on the part of NIST or the U.S. Government.

## **Contact**

Please email openasr_poc [at] nist.gov (openasr_poc[at]nist[dot]gov) for any questions or comments regarding the OpenASR Challenge.

Information technology

### HEADQUARTERS

301-975-2000

Webmaster | Contact Us | Our Other Offices

TwitterFacebookLinkedInInstagramYouTubeGiphyRSS FeedMailing List

Feedback

            -  Site Privacy 
            -  Accessibility 
            -  Privacy Program 
            -  Copyrights 
            -  Vulnerability Disclosure 
            -  No Fear Act Policy 
            -  FOIA 
            -  Environmental Policy 
            -  Scientific Integrity 
            -  Information Quality Standards 
            -  Commerce.gov 
            -  Science.gov 
            -  USA.gov 
            -  Vote.gov