TREC-COVID: Building a 

Pandemic Information Retrieval 

Test Collection 

Dr. Ellen Voorhees 

1
Test Collections Measure Search Engine Effectiveness

0

10

20

0 10 20 30

Number Relevant Retrieved

Number Retrieved

R

R

1960’s

1980’s

Cyril Cleverdon

Test collection defines ideal retrieval so

can calculate system’s distance from it

∆

2

ideal retrieval

system output
Text REtrieval Conference (TREC)

Pioneered use of “pooling” for 

building large collections

Built > 150 test collections for dozens 

of search tasks

Hundreds of participant teams

world-wide

Premier venue for determining 

research methodology

Model for other efforts in IR and 

related fields

Series of community 

evaluations that build 

research infrastructure.

3
TREC is an annual benchmarking 

exercise that has become a de 

facto standard in Information 

Retrieval evaluation.

Stephen Robertson

Microsoft

SIGIR 2007

Establish research 

methodology

TREC has proven to be a valuable 

forum in which IBM Research has 

contributed to an improved 

understanding of search, while at 

the same time the insights 

obtained by participating in TREC 

have helped to improve IBM’s 

products and services.

Alan Marwick, et al.

IBM chapter of the TREC book

2005

Facilitate 

technology transfer

The TREC data revitalized 

research on information retrieval. 

Having a standard, widely 

available, and carefully 

constructed set of data laid the 

groundwork for further innovation 

in the field. The yearly TREC 

conference fostered collaboration, 

innovation, and a measured dose 

of competition (and bragging 

rights) that led to better 

information retrieval.

Hal Varian

Google Chief Economist

March 4, 2008

Improve the state 

of the art

This project [the TREC Legal 

track] can be expected to identify 

both cost effective and reliable 

search and information retrieval 

methodologies and best practice 

recommendations, which, if 

adhered to, certainly would 

support an argument that the party 

employing them performed a 

reasonable ESI search, whether 

for privilege review or other 

purposes.

Magistrate Judge Paul Grimm 

Victor Stanley v. Creative Pipe

Solidify a research 

community

In other words, for every $1 NIST 

and its partners invested in TREC, 

at least $3.35 to $5.07 in benefits 

accrued to IR researchers...These 

responses suggest that the 

benefits of TREC to both private 

and academic organizations go 

well beyond those quantified by 

this study’s economic benefits.

RTI International

Economic Impact Assessment of NIST’s 

TREC Program

December 2010

Amortize the costs 

of infrastructure

Community Evaluations

Credit: IEEE

Credit: Atomic Taco, CC BY-SA 2.0

Credit: MIT Press

Credit: 10th Mountain Films

4
Effective search helps clinical personnel get the 

evidence-based answers they need

In response to an OSTP call for action, 

TREC-COVID is building the infrastructure to 

improve search systems for future biomedical 

crises...

...while providing human-curated answer 

document sets for today’s questions.

Images by Alexandra 

Koch from Pixabay 5
TREC-COVID

Why is a new collection needed? 

Because the information landscape for clinical researchers during a crisis event 

is different from other scientific literature search.

Images by Gerd Altmann from Pixabay

Rapid evolution of topics of interest Large percentage of literature not 

peer reviewed and evolving

Literature in area explodes

6
7

TREC-COVID

Documents

CORD-19, an open-access collection of the scientific 

literature about COVID-19 and other corona viruses. 

New snap-shots released weekly. 

Questions

Inspired by search logs at medical libraries including 

NLM. New questions added in each round.

Relevance Judgments

Subset of retrieved documents annotated each 

round by individuals with biomedical expertise.
TREC-COVID

Round 2

May 4–May 13

May 1 release of CORD-19,

~60k articles

35 topics

51 teams, 136 submissions

~ 20k cumulative judgments

Round 1

April 15—April 23

April 10 release of CORD-19,

~47k articles

30 topics

56 teams, 143 submissions

~8.5k judgments

Two rounds completed; third in progress.

8

Round 3

May 26—June 3

May 19 release of CORD-19, 

~128k articles

40 topics

addition of a Kaggle task running in 

parallel
Topics, judgments, submissions, score reports all archived on public TREC-COVID website, 

http://ir.nist.gov/covidSubmit/

TREC-COVID (Interim) Results

9

Participant’s score report for one submission

Summary of effectiveness over all submissions 

Analysis of collection quality
10

TREC-COVID Future

Plan is for 5 rounds total

Conclude mid-August on current schedule.

Final collection

Cumulative set of 50 topics and and ~50k judgments.

Metadata will allow series of individual rounds to be 

recovered as needed.

Rich trove of data to study collectionbuilding under adverse conditions Image by Gerd Altmann from Pixabay