Skip to main content    

An official website of the United States government

Here’s how you know

**Official websites use .gov**              A **.gov** website belongs to an official government organization in the United States.            

**Secure .gov websites use HTTPS**              A **lock** (  LockA locked padlock) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.            

https://www.nist.gov/itl/iad/mig/openmt15-evaluation

            -  Topics  -  All Topics 
-  Advanced communications 
-  Artificial intelligence 
-  Bioscience 
-  Buildings and construction 
-  Chemistry 
-  Climate 
-  Cybersecurity 
-  Electronics 
-  Energy 
-  Environment 
-  Fire 
-  Forensic science 
-  Health 
-  Information technology 
-  Infrastructure 
-  Manufacturing 
-  Materials 
-  Mathematics and statistics 
-  Metrology 
-  Nanotechnology 
-  Neutron research 
-  Performance excellence 
-  Physics 
-  Public safety 
-  Resilience 
-  Standards 
-  Transportation 
 
            -  Publications 
            -  Labs & Major Programs  -  Laboratories    -  Communications Technology Laboratory 
  -  Engineering Laboratory 
  -  Information Technology Laboratory 
  -  Material Measurement Laboratory 
  -  Physical Measurement Laboratory 
-  User Facilities    -  NIST Center for Neutron Research 
  -  CNST NanoFab 
-  Research Test Beds 
-  Research Projects 
-  Tools & Instruments 
-  Major Programs    -  Baldrige Performance Excellence Program 
  -  CHIPS for America Initiative 
  -  Manufacturing Extension Partnership (MEP) 
  -  Office of Advanced Manufacturing 
  -  Special Programs Office 
  -  Technology Partnerships Office 
 
            -  Services & Resources  -  Standards and Measurements    -  Calibration Services 
  -  Laboratory Accreditation (NVLAP) 
  -  Quality System 
  -  Standard Reference Materials (SRMs) 
  -  Standards.gov 
  -  Time Services 
  -  Office of Weights and Measures 
-  Software 
-  Data    -  Chemistry WebBook 
  -  National Vulnerability Database 
  -  Physical Reference Data 
  -  Standard Reference Data (SRD) 
-  Storefront 
-  License & Patents 
-  Computer Security Resource Center (CSRC) 
-  NIST Research Library 
 
            -  News & Events  -  News 
-  Events 
-  Blogs 
-  Feature Stories 
-  Awards 
-  Video Gallery 
-  Image Gallery 
-  Media Contacts 
 
            -  About NIST  -  About Us    -  Leadership 
  -  Organization Structure 
  -  Budget & Planning 
-  Contact Us 
-  Visit 
-  Careers    -  Student programs 
-  Work with NIST 
-  History    -  NIST Digital Archives 
  -  NIST Museum 
  -  NIST and the Nobel 
-  Educational Resources 
 

Information Technology LaboratoryInformation Access Division

## Multimodal Information Group

-  Tools 
-  Past HLT Evaluation Projects 
-  Staff 

# OpenMT15 Evaluation

## Share

Facebook

Linkedin

Twitter

Email

The NIST Open Machine Translation 2015 Evaluation (OpenMT15) will be implemented according to the OpenMT15 evaluation plan. It will take place in February 2015, followed by a workshop in June 2015.

## Highlights

                  - Evaluation on informal data genres (SMS/Chat, Conversational Telephone Speech) for Arabic-to-English and Chinese-to-English
                  - Inclusion of audio input track
                  - Explore common MT measurement techniques on these informal data genres 

## Documentation

                  - Evaluation plan  (v0.9, updated January 28, 2015 - fixed inconsistency about filename extensions)
                  - Registration form
                  - LDC data license agreement  (v2.3, updated December 8, 2014 - Catalog IDs were added to the last remaining training corpora who previously did not have any catalog ID. Those who signed the license do not have to resign the agreement but should work with the LDC to make sure they have the needed corpora. This means look at the updated license, determine which of the new corpora you would like to have, and put in a request to the LDC for those corpora.)
                  - System description template 

## Schedule

                  - February 1, 2014: Initial evaluation plan available (will be updated periodically as more information is available)
                  - February 1 - December 31, 2014: Registration period (early registration highly encouraged)
                  - February - December 31, 2014: Training data available from LDC with incremental releases of new genres
                  - November 3, 2014 - January 31, 2015: Dry run period
                  - February 2 - 6, 2015: Main evaluation period for audio track                       - Input available for download on February 2 at 11:59am ET
                    - Output due at NIST February 6, 11:59am ET
                  - February 9 - 13, 2015: Main evaluation period for text track                       - Input available for download on February 9 at 11:59am ET
                    - Output due at NIST February 13, 11:59am ET
                  - February 13, 2015: System description due to NIST
                  - February 23 - June 8, 2015: Human assessment period
                  - June 17, 2015: Preliminary slides due to NIST
                  - June 25-26, 2015:  Workshop in the Washington DC area
                  - December 2015:  Official public release of results

## Dry Run Evaluation

All registered participants are encouraged to demonstrate evaluation readiness by participating in the dry run evaluation by showing that their submissions pass validation. The dry run evaluation will conclude on Jan 31, 2015. The dry run submissions are used to exercise the evaluation pipeline and scores from these submissions will not be used in any official report.

To participate in the dry run evaluation, registered participants need to sign up for a user account at https://openmteval.nist.gov. You must register (fill out the registration given above) before you can sign up for a user account.

## Contact

mt_poc [at] nist.gov (mt_poc[at]nist[dot]gov)

[ Main ]

Information technology

### HEADQUARTERS

301-975-2000

Webmaster | Contact Us | Our Other Offices

TwitterFacebookLinkedInInstagramYouTubeGiphyRSS FeedMailing List

Feedback

            -  Site Privacy 
            -  Accessibility 
            -  Privacy Program 
            -  Copyrights 
            -  Vulnerability Disclosure 
            -  No Fear Act Policy 
            -  FOIA 
            -  Environmental Policy 
            -  Scientific Integrity 
            -  Information Quality Standards 
            -  Commerce.gov 
            -  Science.gov 
            -  USA.gov 
            -  Vote.gov