U.S. flag
        An official website of the United States government
          Here’s how you know
        Here’s how you know**Official websites use .gov**
              
              A **.gov** website belongs to an official government organization in the United States.
        **Secure .gov websites use HTTPS**
              
              A **lock** (  A locked padlock
) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. https://www.nist.gov/itl/iad/mig/lorehlt-evaluations
  Search NIST
    Search
              Menu

      


Close
    Topics
        All Topics
                                                        Advanced communications
                                                        Artificial intelligence
                                                        Bioscience
                                                        Buildings and construction
                                                        Chemistry
                                                        Climate
                                                        Cybersecurity
                                                        Electronics
                                                        EnergyEnvironment
                                                        Fire
                                                        Forensic science
                                                        Health
                                                        Information technology
                                                        Infrastructure
                                                        Manufacturing
                                                        Materials
                                                        Mathematics and statisticsMetrology
                                                        Nanotechnology
                                                        Neutron research
                                                        Performance excellence
                                                        Physics
                                                        Public safety
                                                        Resilience
                                                        Standards
                                                        Transportation
            Publications
            Labs & Major Programs
        Laboratories
                                Communications Technology Laboratory
                                  Engineering Laboratory
                                  Information Technology Laboratory
                                  Material Measurement Laboratory
                                  Physical Measurement LaboratoryUser Facilities
                                NIST Center for Neutron Research
                                  CNST NanoFab
                                                        Research Test Beds
                                                        Research Projects
                                                        Tools & InstrumentsMajor Programs
                                Baldrige Performance Excellence Program
                                  CHIPS for America Initiative
                                  Manufacturing Extension Partnership (MEP)
                                  Office of Advanced Manufacturing
                                  Special Programs Office
                                  Technology Partnerships Office
            Services & Resources
        Standards and Measurements
                                Calibration Services
                                  Laboratory Accreditation (NVLAP)
                                  Quality System
                                  Standard Reference Materials (SRMs)
                                  Standards.gov
                                  Time Services
                                  Office of Weights and MeasuresSoftware
                                                        Data
                                Chemistry WebBook
                                  National Vulnerability Database
                                  Physical Reference Data
                                  Standard Reference Data (SRD)
                                                        Storefront
                                                        License & PatentsComputer Security Resource Center (CSRC)
                                                        NIST Research Library
            News & Events
        News
                                                        Events
                                                        Blogs
                                                        Feature Stories
                                                        AwardsVideo Gallery
                                                        Image Gallery
                                                        Media Contacts
            About NIST
        About Us
                                Leadership
                                  Organization Structure
                                  Budget & Planning
                                                        Contact Us
                                                        Visit
                                                        Careers
                                Student programsWork with NIST
                                                        History
                                NIST Digital Archives
                                  NIST Museum
                                  NIST and the Nobel
                                                        Educational Resources
 
 Information Technology Laboratory /                  Information Access Division
      Multimodal Information Group
    
      
  Tools
          Past HLT Evaluation Projects
          StaffLoReHLT Evaluations

Share
  Facebook
                      Linkedin
                      Twitter
                      Email

Low Resource Languages for Emergent Incidents (LORELEI) is a DARPA-sponsored program. The goal of the program is to dramatically advance the state of computational linguistics and human language technology to enable rapid, low-cost development of capabilities for low-resource languages.
	This web page serves as a portal for all information pertaining to the Low Resource (LoRe) HLT open evaluations of component technologies relevant to LORELEI. For general questions and comments, please e-mail lorehlt_poc [at] nist.gov(lorehlt_poc[at]nist[dot]gov).  For more information about a specific evaluation cycle, including evaluation documentation, see the year-specific sections below.  For general announcements and discussing regarding the LoReHLT evaluation series, you can subscribe to the LoReHLT mailing list hosted by NIST, lorehlt [at] nist.gov(lorehlt[at]nist[dot]gov), by sending e-mail to lorehlt_list-requeset [at] nist.gov(lorehlt_list-request[at]nist[dot]gov) with "subscribe" in the subject line.

LoReHLT19

Tasks

Machine translation (MT)
	Entity Discovery and Linking (EDL)
	Situation Frame (SF)
	Sentiment, Emotion, & Cognitive State (SEC) (Pilot)Highlights

Surprise language evaluation (two surprise languages)
	New SEC pilot evaluation
	Two evaluation checkpoints to gauge performance based on time and training resources givenDocumentation

LoReHLT 2019 Evaluation Plan v1.0, initial release July 19, 2019
	System description templateTools

SF:
		Situation Frame submission JSON schema V1LoReHLT18

Tasks

Machine translation (MT)
	Entity Discovery and Linking (EDL)
	Situation Frame (SF)Highlights

Surprise language evaluation (two surprise languages)
	Two training conditions: constrained (required) and unconstrained
	Two evaluation checkpoints to gauge performance based on time and training resources givenRegistration

Use this form to sign up to participate: https://goo.gl/forms/Xoj6mbLFU1V5BEJ93
	Non-LORELEI participants only: Complete and return the LDC LoReHLT18 data license agreement.Documentation

LoReHLT 2018 Evaluation Plan v 1.0.4(updated June 7, 2018)Tools

Situation Frame:
		Scoring software package (version 2.0.1): LoReHLT18_SF_Scorer_v0.8_20180515.zipPublications and System Descriptions

LoReHLT18 system descriptions (not peer-reviewed)LoReHLT17

Tasks

Machine translation (MT)
	Entity Discovery and Linking (EDL)
	Situation Frame (SF)Highlights

Surprise language evaluation (two surprise languages)
	Two training conditions: constrained (required) and unconstrained
	Three evaluation checkpoints to gauge performance based on time and training resources givenDocumentation

LoReHLT17 Evaluation Plan v 2.3.3 (updated July 27, 2017)
	System description template (one per team, describing all tasks participated in and submissions)Tools

Test data encryption package: OpenSSLTest-v2.tar.bz2 (md5)
	MT:
		BLEU and "NIST" score scoring script: mteval-v14c.pl
	EDL:
		Scoring script: https://github.com/wikilinks/neleval
	Situation Frame (Text):
		Scoring software package (version 2.0.1):  LoReHLT_Frame_Scorer_v2.0.1_20170626.tgz (md5)
			System output schema (version 1.1):  LoReHLT17_v1.1_system-output-schema.tgz (md5)
	Situation Frame (Speech) -- We would like to thank the Signal Analysis and Interpretation Laboratory (SAIL) at the University of Southern California for provided the scoring software for the Situation Frame Speech task:
		Scoring software package (version 5): https://github.com/usc-sail/lorelei-speech-evaluation
			System output schema (version 1.0): LoReHLT17_SF-Speech_v1.0_system-output-schema.tgz (md5)Publications and System Descriptions

LoReHLT17 system descriptions (not peer-reviewed)LoReHLT16

Tasks

Machine translation (MT)
	Situation Frame (SF)
	Named entity recognition (NER)Highlights

Surprise language evaluation
	Two training conditions: constrained (required) and unconstrained
	Three evaluation checkpoints to gauge performance based on time and training resources givenDocumentation

LoReHLT16 Evaluation Plan v2-8  (updated June 30, 2016)
	Description of evaluation incident language components and structure (LDC-authored; uploaded May 27, 2016)
	Description of development incident language components and structure (LDC-authored; uploaded May 27, 2016)
	System description template (one per team, describing all tasks participated in and submissions)
	For questions about the individual evaluation tasks, consult the LoReHLT16 Evaluations FAQ.
	LoReHLT16 workshop agenda with links to presentation slides, password-protectedTools

Test data encryption package: ftp://jaguar.ncsl.nist.gov/pub/OpenSSLTest-v2.tar.bz2
	Situation Frame scoring software package: LoReHLT_Frame_Scorer_v1.1.2_20160825.tgz (md5)
	Machine translation BLEU/NIST scoring script: mteval-v14.plPublications and System Descriptions

Machine Translation, Volume 32, Issue 1-2, June 2018, Special Issue: NLP in Low-Resource Languages
	LoReHLT16 system descriptions (not peer-reviewed)Disclaimer

Any mention of commercial products within NIST web pages is for information only; it does not imply recommendation or endorsement by NIST.

    
    Created November 19, 2015, Updated June 20, 2019
 HEADQUARTERS
            100 Bureau Drive
              Gaithersburg, MD 20899301-975-2000
            Webmaster | Contact Us | Our Other Offices
        Twitter
            
Facebook
            
LinkedIn
            
Instagram
            
YouTube
            
Giphy
            
RSS Feed
            
Mailing List
                              How are we doing? Feedback

  
  Site Privacy
                      Accessibility
                      Privacy Program
                      Copyrights
                      Vulnerability Disclosure
                      No Fear Act Policy
                      FOIA
                      Environmental Policy
                      Scientific Integrity
                      Information Quality Standards
                      Commerce.gov
                      Science.gov
                      USA.gov
                      Vote.gov