PAPER

TREC-COVID: Constructing a Pandemic
Information Retrieval Test Collection
Ellen Voorhees
National Institute of Standards and Technology
ellen.voorhees@nist.gov
Tasmeer Alam
National Institute of Standards and Technology
Steven Bedrick
Oregon Health and Science University

Dina Demner-Fushman
U.S. National Library of Medicine

William R. Hersh
Oregon Health and Science University

Kyle Lo
Allen Institute for AI

Kirk Roberts
University of Texas Health Science Center at Houston
Ian Soboroff
National Institute of Standards and Technology

Lucy Lu Wang
Allen Institute for AI

Abstract
TREC-COVID is a community evaluation designed to build a test collection that captures theinformation needs of biomedical researchers using the scientific literature during a pandemic.
One of the key characteristics of pandemic search is the accelerated rate of change: thetopics of interest evolve as the pandemic progresses and the scientific literature in the area
explodes. The COVID-19 pandemic provides an opportunity to capture this progression as
it happens. TREC-COVID, in creating a test collection around COVID-19 literature, isbuilding infrastructure to support new research and technologies in pandemic search.

1

Introduction

The COVID-19 pandemic has caused not only a public health crisis but also an information crisis.
The global Internet can spread incorrect or incomplete information faster than the spread of thevirus. Even the scientific community is challenged, as the platform for open science, especiallypreprints, allows scientific information to spread that has not been fully vetted. This results in a
ACM SIGIR Forum

1

Vol. 54 No. 1 June 2020

difficult tension between wanting to disseminate scientific information as quickly as possible whilewanting it to be vetted (i.e., peer-reviewed) as well.
To allow scientists, clinicians, and policy makers to avail themselves of the knowledge contained
within this new information landscape, the White House requested that the Allen Institute for
Artificial Intelligence and their collaborators aggregate a structured dataset of coronavirus research
for the global research community [Wang et al., 2020]. This COVID-19 Open Research Dataset, or
CORD-19, contains scholarly articles about COVID-19 and the coronavirus family of viruses. The
articles are drawn from peer-reviewed literature in PubMed Central1 as well as from archival sites
such as medRxiv2 and bioRxiv.3 Importantly, CORD-19 is not a static document collection, but
is regularly updated with new versions released once a week. The different versions of CORD-19
capture the growth and change in the COVID-19 literature over time.
The availability of CORD-19 provides the opportunity to create a test collection that captures
the information needs of biomedical researchers using the scientific literature during a pandemic,
and the TREC-COVID evaluation aims to do precisely that. The twin goals of TREC-COVID
are• to evaluate search algorithms and systems for helping scientists, clinicians, policy makers,
and others manage the existing and rapidly growing corpus of scientific literature related to
COVID-19, and• to discover methods that will assist with managing scientific information in future global
biomedical crises.
Based on the TREC4 framework, TREC-COVID is building a pandemic test collection througha series of rounds. It builds on a history of biomedical TREC tracks [Hersh and Voorhees, 2009,
Voorhees and Hersh, 2012, Roberts et al., 2015, 2017], and specifically on the Genomics, Clinical
Decision Support, and Precision Medicine tracks that focused on retrieval from the scientific literature. But as discussed in this article, TREC-COVID must also extend the evaluation proceduresof these tracks to realize the goal of capturing the pandemic information environment. These extensions include multi-round evaluation, mid-evaluation updates to the topic and document sets,
and tight deadlines for both participation and assessment.
As of this writing, the first round of TREC-COVID has just completed and the second roundhas begun. This paper provides a snapshot of TREC-COVID’s process and results at this point
in time.

2

Structure of TREC-COVID

Each round of TREC-COVID is structured as an independent community evaluation challengemuch like a typical TREC track. The systems’ task is a classic ad hoc search task using CORD-19
as the document set and a set of biomedical questions as the topics (i.e., statements of information need). Systems produce a ranked list of documents per topic where each list is ordered by1

https://www.ncbi.nlm.nih.gov/pmc/
https://www.medrxiv.org/
3
https://www.biorxiv.org/
4
http://trec.nist.gov
2

ACM SIGIR Forum2

Vol. 54 No. 1 June 2020

decreasing likelihood that the document matches the information need (this is a run). The set ofsubmitted runs are used to define a much smaller set of documents for human assessors to judge
for relevance. The documents, topics, and relevance judgments together form a test collectionthat can be used to evaluate the relative effectiveness of competing retrieval approaches.
TREC-COVID differs from a typical TREC track in two important ways. The most obvious
difference is the compressed time schedule. Because we want to foster research on systems thatare able to pivot quickly, deadlines are very short: roughly one week between when the test topics
become available and the run submission deadline, followed by ten days for relevance assessing
before the next round begins.
The second difference is that while each round is treated as an independent evaluation, thedocument and topic sets in each round are supersets of previous rounds’ sets. Relevance judgmentsfrom earlier rounds provide training data for later rounds (and also mean residual collection scoring
must be done in later rounds as described below). The cumulative document, topic, and relevancejudgment sets will reflect the changes observed in real use of search systems during the current
pandemic.
The remainder of this section describes the three components of the test collection in moredetail.

2.1

Documents

TREC-COVID uses the document set provided by CORD-19 [Wang et al., 2020]. CORD-19
consists of new publications and preprints on the subject of COVID-19, as well as relevant historicalresearch on coronaviruses, including SARS and MERS. The April 10, 2020 release of CORD-19,
which is used for the first round of TREC-COVID, includes 51K papers, with full text available
for 39K. As of May 1, 2020, CORD-19 consists of 60K papers, of which full text is available for
48K. Due to the ongoing nature of the pandemic, the CORD-19 corpus is constantly evolving andupdated regularly (approximately weekly) as more new or historical research becomes available.
Each TREC-COVID round uses the release of the dataset that is the most up-to-date as of thestart of the round.
Content in CORD-19 is derived from multiple sources, and harmonization logic is applied
to identify and remove duplicate entries. Each resulting entry in CORD-19 is given a unique
identifier, the CORD UID, which corresponds to the notion of a paper. Each paper can beassociated with multiple documents; for example, the main paper document plus supplementary
files, or potentially, multiple versions of a paper such as its preprint and camera-ready editions.
In TREC-COVID, provisions are made during assessment to adapt to this notion of a paper, byshowing all associated documents to assessors at the time of judgement.
Researchers have also made significant use of preprints during this epidemic to rapidly disseminate contributions. Preprints, much more than publications, are subject to iteration, andcontent can change dramatically between versions. Due to all versions inherently being part ofthe same paper, the identifiers for these preprints persist across CORD-19 corpus releases. In
cases where changes occur in the documents associated with the same paper (identified by UID)
between CORD-19 corpus releases, prior judgments on the original documents may no longer hold.
A judgment in TREC-COVID is associated with a particular round to indicate the version of the
document that received the judgment.
ACM SIGIR Forum

3

Vol. 54 No. 1 June 2020

CORD-19 presents a real life document set that evolves in response to new research. Though
there are difficulties in working with such a document set, it is an opportunity to study ways tojudge changing relevance in a realistic environment. Further challenges and proposed solutions
are detailed in Section 4.

2.2

Topics

The topics used in TREC-COVID have been written by its organizers with biomedical training,
inspired by consumer questions submitted to the National Library of Medicine, discussions by
medical influencers on social media, and suggestions solicited on Twitter via the #COVIDSearch
tag in late March 2020. They are representative of the high-level concerns related to the pandemic.
So far, the topics have not included detailed biological questions (e.g., RNA mutation rates,
structure of a certain protein), though in future rounds more of these topics may be added.
Each topic is composed of three fields:
1. query: a short keyword query
2. question: a more precise natural language question
3. narrative: a longer description that further elaborates on the question, often providing
specific types of information that would fall under the topic score
Note that while the question is a superset of the information in the query, the narrative is notintended to be a superset of the information in the question. Instead, it helps to further specify
user intent. For three examples of topics from Round 1, see Figure 1.
There was considerable discussion amongst the organizers prior to Round 1—and some discussion with participants after Round 1—relating to the terminology used to refer to the virus/disease.
Early real-world queries referred to the virus in informal terms (e.g., “Wuhan virus”, “Chineseflu”), then evolved to more formal terms such as “2019-nCoV ” and informally simply “coronavirus”, and finally the official use of “SARS-CoV-2 ” to refer to the virus and “COVID-19 ”
to refer to the condition it causes. This is a fundamental linguistic problem in dealing with an
emerging situation like a pandemic. A specific problem is that “coronavirus” refers to a class ofviruses, not just SARS-CoV-2. However, the user intent of the information need is clear: the user
is focused on COVID-19/SARS-CoV-2. A hybrid solution was taken to this problem in topic construction. The query field often uses less formal terms, such as “coronavirus”, while the narrative
field is more likely to use a formal term such as “COVID-19 ” or “SARS-CoV-2 ”. At least one of
the unambiguous terms is guaranteed to be used in at least one of the fields for the topic, ensuring
that—to a human, at least—it is clear that the topic is about the current pandemic. Of course,
some information about prior coronaviruses is still useful in the current pandemic (which is why
articles on these are included in CORD-19). The extent to which this background information isuseful depends on the topic, how much context is needed, and to some extent the user’s expectation about what may be available about a novel virus. Understanding user intent, then, is left asa challenge to the participant systems.
In future rounds, new topics will be added (5 per round are planned). These new topics willcapture some of the trends that emerge as the crisis unfolds (especially new information after
Round 1). More detailed, scientifically challenging topics will also be added to test the range ofcapabilities of systems.
ACM SIGIR Forum

4

Vol. 54 No. 1 June 2020

number:query:
question:narrative:

7
serological tests for coronavirus
are there serological tests that detect antibodies to coronavirus?
Looking for assays that measure immune response to COVID-19 that will help
determine past infection and subsequent possible immunity.

number:query:
question:

20
coronavirus and ACE inhibitors
are patients taking Angiotensin-converting enzyme inhibitors (ACE) at increased
risk for COVID-19?
Looking for information on interactions between coronavirus and angiotensin
converting enzyme 2 (ACE2) receptors, risk for patients taking these medications, and recommendations for these patients.

narrative:

number:query:
question:narrative:

29
coronavirus drug repurposing
which SARS-CoV-2 proteins-human proteins interactions indicate potential for
drug targets. Are there approved drugs that can be repurposed based on thisinformation?
Seeking information about protein-protein interactions for any of the SARSCoV-2 structural proteins that represent a promising therapeutic target, andthe drug molecules that may inhibit the virus and the host cell receptors atentry step.

Figure 1: Example topics from the Round 1 test set.

2.3

Relevance Judgments

As with the other TREC biomedical tasks, the annotators making relevance assessments for
TREC-COVID are individuals with clinical expertise. To date, we have been able to enlist thehelp of ten Oregon Health and Science University medical students whose clinical activities weredisplaced due to Covid-19 restrictions, as well as additional relevance assessment help from professional indexers from the National Library of Medicine.
The relevance assessors use the Web-based system illustrated in Figure 2. The assessment
system shows the assessor the topic and a list of documents to be judged. Assessors mark eachdocument in the list as either ‘Relevant’,‘Partially Relevant’, or ‘Not Relevant’. Each document
may have one or more forms, e.g., title/abstract, preprint, PDF from publisher, and/or entry into
PubMed Cental that are included under different tabs. The document list on the left side of thescreen also indicates whether the document has been judged.
As mentioned earlier, the compressed schedule of TREC-COVID means that the amount oftime available for performing relevance assessments is very short. Since computing which documents to judge is driven by the submitted runs, there are only two weeks of judging time perround. Assessments performed during the week that starts immediately after the run submission deadline of Round X runs are called X judgments. Assessments performed in the following week when participants are constructing their Round X+1 submissions are called X.5 judgACM SIGIR Forum

5

Vol. 54 No. 1 June 2020

Figure 2: Screenshot of the interface used for TREC-COVID document assessing.
ments. Both judgment sets X and X.5 are derived from Round X runs and use the version of
CORD-19 that runs in Round X searched. However, the relevance judgments used to evaluate
Round X runs, and which are publicly released before Round X+1 begins, consist of judgment
sets X-.5 and X. The standard format of a relevance judgments file (the so-called qrels file) is
topic-id iteration doc-id judgment where the iteration field is traditionally zero. TRECCOVID qrels files record the judgment round when the judgment was made in this field. Notethat a 0.5 judgment set exists, too. Because assessors were available before the first submission deadline, TREC-COVID organizers produced three runs based on the Anserini5 system andused depth-40 pools across these three runs using the Round 1 release of CORD-19 (April 10) tocreate the 0.5 set. Figure 3 illustrates the relationship between TREC-COVID rounds and thecorresponding judgment sets.
The pool of available assessors constrains the amount of judgments that can be made to about
150–200 per topic per week. To obtain this number of assessments, a topic may be judged bymore than one assessor. For Round 1, the maximum number of assessors for a topic was two.
With only about 300 documents judged per topic in a given round, the relevance judgments forthat round are incomplete to the point that run comparisons are likely unstable for many measures.
That is, if the unjudged documents in the runs were judged, the preferred order of the two runs
could change. While full trec eval output as produced by the current round’s qrels file is reported
for each run6 , Round 1 summaries therefore focused on three measures: Bpref, NDCG@10, andP@5. Bpref was designed for collections with incomplete judgments and is computed over only
5
6

https://github.com/castorini/anserini
All submitted runs and score reports are archived at http://ir.nist.gov/covidSubmit/Archive.html.

ACM SIGIR Forum

6

Vol. 54 No. 1 June 2020

Figure 3: Illustration of the process to create the Round X qrels. The Round X qrels file is theunion of the set of judgments made the week following the Round X submission deadline and theset of judgments made in the preceding week. For the special case of “Round 0”, organizers createdthree runs and pooled those to start the judgment process prior to receiving any participant runs.
the judged set. P@5 and NDCG@10 are each affected by only the most highly ranked documents,
which are the documents most likely to be judged. The computation of NDCG uses gain valuesof 1 for ‘Partially Relevant’ documents and 2 for ‘Relevant’ documents. Measures that use binary
judgments are computed using both types of relevant documents as the relevant set.
The Round X qrels file is posted to the TREC-COVID web site at the time the Round X+1
test set is released. Participants are free to use judgments from earlier rounds to construct runs
for later rounds. To properly account for the use of relevance feedback when scoring runs, eachround after the first will use residual collection evaluation [Salton and Buckley, 1997]. In residualcollection evaluation, any document that has already been judged for a topic (in any round) isremoved from the collection before scoring. This is implemented by removing all documents judged
for a topic in an earlier round from all submitted runs.
While residual collection evaluation is methodologically valid, it means that each round’s runs
can only be evaluated using a single round’s qrels, not the cumulative set of judgments for a topic.
Thus each round will most likely continue to be affected by incomplete judgments issues. Further,residual collection evaluation will make effectiveness scores appear to degrade from round to round

ACM SIGIR Forum

7

Vol. 54 No. 1 June 2020

Table 1: Counts of total numbers of judged documents and number of relevant documents per
topic. Percent relevant is the fraction of judged documents that are some form of relevant.
Topic
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Total
Judged
323
284
337
357
336
321
275
360
298
191
344
324
373
222
348

Partially
Relevant
45
21
66
32
35
80
2
46
25
35
67
76
97
24
45

Relevant
56
26
24
27
96
83
47
30
16
50
5
126
49
5
12

Percent
Relevant
0.313
0.165
0.267
0.165
0.390
0.508
0.178
0.211
0.138
0.445
0.209
0.623
0.391
0.131
0.164

Topic
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Total
Judged
340
243
267
301
247
319
259
256
249
308
312
300
180
218
199

Partially
Relevant
42
32
79
27
41
15
17
4
14
9
19
30
9
42
39

Relevant
11
45
32
16
25
70
30
22
19
62
106
44
29
58
16

Percent
Relevant
0.156
0.317
0.416
0.143
0.267
0.266
0.181
0.102
0.133
0.231
0.401
0.247
0.211
0.459
0.276

since the set of relevant documents in the collection is reduced in each round. The final pandemic
test collection that is the end product of TREC-COVID will contain the cumulative judgmentsfrom all rounds and can therefore be expected to be much more stable than the individual-roundcollections.

3

Round 1 Results

TREC-COVID Round 1 received 143 runs from 56 teams. Teams came from Asia, Australia,
Europe, and North America. One hundred of the runs are automatic runs (a run in which thesystem is given the topic file and produces ranked output with no human involvement at all) andthe remaining 43 runs are manual runs (everything else).
The number and diversity of the submitted runs meant that staying within the judgment
budget of 200 average documents per topic was challenging. Anticipating the possibility that wewould not be able to judge all runs, participants were asked to assign a judging priority (1–3) totheir runs at submission time. Depth-7 pools7 created from only the first priority run from eachteam amounted to almost 7000 documents to be judged across the 30 topics. Subtracting the
0.5 judgments from that set left a total of just over 6000 documents. These 6000 documents are
the 1.0 judgment set. The union of judgment sets 0.5 and 1.0, which constitute the Round 1 qrels,
contains 8,691 judgments, a mean of 289.7 judgments per topic with a range between 180–373 per
topic (see Table 1).
7

See summary papers such as [Voorhees, 2002] or TREC Overview papers for details on pooling.

ACM SIGIR Forum

8

Vol. 54 No. 1 June 2020

Figure 4: Number of documents in top 50 ranks that are judged over all 30 topics.
The maximum size of depth-7 pools across 56 runs (one run per team) is 11,760 (56 × 30 × 7).
The 7000 size obtained in Round 1 pooling is greater than half of that maximum, which is a highpercentage compared to typical TREC collections. By definition, this means there is comparatively
little overlap among top-ranked documents across runs. Many runs do contain other runs’ top
documents deeper in their ranked lists, however. Figure 4 shows a box-and-whiskers plot of thenumber of judged documents in the top 50 ranks per topic for all submitted runs. The runs areplotted on the x-axis and are ordered by decreasing median number of documents judged (theheavy black line). About half the runs have a median of approximately 25, so for those runs atleast half of the topics have at least half of the top-50 documents judged. The variance acrosstopics is large for most runs, and there is a set of runs to the far right in the graph that retrieved
practically no documents in common with other runs’ top documents down to depth 50.
Figure 4 shows statistics computed over judged documents, not relevant documents only. Historically, runs with extremely low overlap with other runs tend to be runs created with errors
where the assumption that unjudged documents are not relevant holds. We cannot know this tobe true for the runs on the far right of the graph without more judgments, however. Judgmentsfrom set 1.5, which are currently in process, may provide a partial answer.
Table 1 shows relevance counts for each topic. The table includes the total number of judged
documents, the number of documents judged ‘Partially Relevant’ and ‘Relevant’, and the percentage of judged documents that are (some form of) relevant per topic in the Round 1 qrels. The
total number relevant per topic ranges from 26–202. The percentage of judged that are relevant
is of interest as an indicator of the stability (or reusability) of the collection. Historically, whenmore than a third of the judged documents are relevant for a topic, it is highly likely that many
more relevant documents that have not yet been identified remain in the collection [Voorhees,
ACM SIGIR Forum

9

Vol. 54 No. 1 June 2020

Figure 5: Per-topic NDCG@10 scores aggregated over all Round 1 runs.

2018]. Having fewer than one third relevant is not a guarantee that the collection is stable, butmore than a third is strong evidence that it is not. For the Round 1 qrels, more than a quarter of
the topics (8/30) have relevant percentages greater than 0.33.
Figure 5 provides a view of how effective the participants’ systems were as a group. The figurecontains a box-and-whiskers plot of the NDCG@10 scores across all 143 submitted runs for each
topic. The median NDCG@10 score for the topic is plotted as the black bar inside the box; thetop and bottom of the box represent the third and first quartiles of the scores,respectively; thewhiskers extend out another 1.5 times the interquartile range; and outliers are plotted as points.
Taken together, the set of systems was effective at finding relevant documents for every topic.
Eight topics have maximum NDCG@10 scores of 1.0, meaning at least one system retrieved ten
fully relevant documents in the top 10 ranks. Nonetheless, the spread of scores across runs isgood for each topic suggesting that the test collection is not too easy and that it will be able todiscriminate between systems.

4

Challenges

The rapid organization of TREC-COVID and the shifting nature of knowledge of the underlying
COVID-19 disease create challenges for organizing a TREC-style evaluation. In this section welist some of the limitations and cautions for using the TREC-COVID test collection.

ACM SIGIR Forum

10

Vol. 54 No. 1 June 2020

The short time available for relevance judgments has had two main effects. First, it puts ahard limit on the number of judgments that can be obtained, which when coupled with many
participants results in quite shallow judgment pools. Shallow pools in turn lead to incomplete
judgment sets and relatively large uncertainty in systems’ evaluation scores. The measures forwhich the uncertainty is small, such as P@5, are not necessarily the best measures to understandthe target user task. Second, it requires using multiple assessors per topic. This can damage
the internal consistency of the judgments for a topic, since different assessors are likely to havedifferent notions of relevance.
Capturing the dynamic nature of COVID-19 literature is a primary goal of TREC-COVID, andusing a dynamic collection as the basis for a retrieval test collection presents its own challenges.
Changes come in different forms. The papers themselves can change over time as preprints are
updated, then published, or (potentially) retracted. Papers may have only the abstract available
at first, with the full text released sometime later. For example, of the 51K papers in the April
10 release of CORD-19 used in Round 1, 1041 papers had some change in the title, abstract, orfull text by the May 1 release. The vast majority of these changes were minor such as simple
text edits. But 284 papers that had either no abstract or full text in the April 10 release didhave those components in the May 1 release. Such changes to a paper can invalidate previously
made relevance judgments, so assessors re-judge changed papers for all previously-judged topics.
The relevance judgments in the final test collection will therefore not be a single judgment per
document per topic as is the norm, but rather a series of judgments where each entry in the series
is tied to a particular CORD-19 release through the round number.
An assessor’s notion of what it means to be relevant may also change as the science around
COVID-19 evolves. The very public nature of the disease means that the assessment process
cannot happen in a vacuum: assessors are aware of many of the latest medical developments,
and that may alter their perception of relevance. This impact is difficult to quantify empirically,
though we do have anecdotal evidence from assessors regarding how they may have judged prior
results differently based on what they now know. This sort of re-assessment of prior judgments is
not done. The judgments need to be viewed as occurring in the time frame of its associated the
CORD-19 release.

5

FutureAt the time of writing, TREC-COVID is entering Round 2 with five new topics, an updated
document collection and the original set of judges. The immediate plans include three moreevaluation rounds, which will bring us to Round 5 in July. By that time, we anticipate to better
understand the trends in the growth of the document collection, retrieval of residual relevant
documents, stability of the systems’ ranking, and the interest of research community in continuing
the evaluation. These factors will inform the organizers if additional rounds will be needed tobuild a permanent COVID-19 pandemic collection and resolve any remaining pandemic response
system-building and evaluation questions. The completion of the last round and the analysis of
the results will accomplish the first goal of TREC-COVID – an evaluation of the systems andapproaches that directly address information needs brought about by the COVID-19 pandemic.
The final collection built over the entire set of the evaluation rounds will support the second
goal of TREC-COVID: to discover approaches to managing scientific information and satisfy
ACM SIGIR Forum

11

Vol. 54 No. 1 June 2020

information needs of scientists during global outbreaks and similar health threats. The final
collection will consist of the clearly marked versions of the data, topics and relevance judgments
corresponding to each evaluation round. The cumulative relevance judgments in combination withthe final document collection will allow researchers to develop approaches in a traditional post
hoc use of TREC collections, probably with some caveats mentioned in section 4. Whereas using
the versions of the rounds as they evolve, the researchers can study approaches to addressing
information needs as they evolve during pandemics.

References
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide,
Kathryn Funk, Rodney Kinney, Ziyang Liu, William. Merrill, Paul Mooney, Dewey A. Murdick,
Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang,
Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, andSebastian Kohlmeier. Cord-19: The covid-19 open research dataset. ArXiv, abs/2004.10706,
2020.
William Hersh and Ellen Voorhees. TREC Genomics special issue overview. Information Retrieval,
12:1–15, 2009.
Ellen M. Voorhees and William Hersh. Overview of the TREC 2012 Medical Records track. In
The Twenty-First Text REtrieval Conference Proceedings, 2012.
Kirk Roberts, Matthew S. Simpson, Ellen Voorhees, and William Hersh. Overview of the TREC
2015 Clinical Decision Support Track. In The Twenty-Fourth Text REtrieval Conference Proceedings (TREC 2015). NIST Special Publication 500-319., 2015.
Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, William R. Hersh, Steven Bedrick,Alexandar Lazar, and Shubham Pant. Overview of the TREC 2017 Precision Medicine Track.
In The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). NIST Special Publication 500-324., 2017.
Gerard Salton and Chris Buckley. Improving retrieval performance by relevance feedback. Journalof the Association for Information Science and Technology, 41:355–364, 1997.
Ellen M. Voorhees. The philosophy of information retrieval evaluation. In Evaluation of CrossLanguage Information Retrieval Systems. Proceedings of CLEF 2001, number 2406 in Lecture
Notes in Computer Science, pages 355–370, 2002.
Ellen M. Voorhees. On building fair and reusable test collections using bandit techniques. In
Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, page 407–416, 2018. doi: 10.1145/3269206.3271766.

ACM SIGIR Forum

12

Vol. 54 No. 1 June 2020