TREC 2021 Fair Ranking Track
Participant Instructions
Michael D. Ekstrand
michaelekstrand@boisestate.edu
Amifa Raj
amifaraj@u.boisestate.edu

Graham McDonald
graham.mcdonald@glasgow.ac.uk

Isaac Johnson
isaac@wikimedia.org

Morten Warncke-Wang
mwang@wikimedia.org

May 5, 2021
These instructions may be updated; see https://fair-trec.github.io for the latest version.

1

Introduction

The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel
retrieval algorithms that can provide a fair exposure to a mixture of demographics or attributes, such asethnicity, that are represented by relevant documents in response to a search query. For example, particulardemographics or attributes can be represented by the documents’ topical content or authors.
The 2021 Fair Ranking Track adopts a resource allocation task. The task is focused on supporting
Wikipedia editors who are looking to improve the encyclopedia’s coverage of topics under the purview of a
WikiProject1 . WikiProject coordinators and/or Wikipedia editors search for Wikipedia documents that arein need of editing to improve the quality of the article.
The Fair Ranking track aims to ensure that documents that are about, or somehow represent, certain
protected characteristics receive a fair exposure to the Wikipedia editors, so that they have an equal opportunity of being well represented in Wikipedia. The under-representation of particular protected characteristics
in Wikipedia can result in systematic biases that can have a negative human, social, and economic impact,
particularly for disadvantaged or protected societal groups [3, 4].

2Task Definition

The Fair Ranking Track uses an ad hoc retrieval protocol. Participants will be provided with a corpus ofdocuments (a subset of the English language Wikipedia) and a set of queries. A query will be of the form ofa short list of search terms that represent a WikiProject. Each document in the corpus is relevant to zero
to many WikiProjects and associated with zero to many fairness categories.
There are two tasks in the 2021 Fair Ranking Track. In each of the tasks, for a given query, participants
are to produce document rankings that are:
1. Relevant to a particular WikiProject.
2. Provide a fair exposure to articles that are associated to particular protected attributes.
The tasks share a topic set, the corpus, the basic problem structure and the fairness objective. However, they differ in their target user persona, system output (static ranking vs. sequences of rankings) andevaluation metrics. The common problem setup is as follows:
1 https://en.wikipedia.org/wiki/WikiProject1• Queries are provided by the organizers and derived from the topics of existing or hypothetical WikiProjects.
• Documents are Wikipedia articles that may or may not be relevant to any particular WikiProjectthat is represented by a query.
• Rankings should be ranked lists of articles for editors to consider working on.
• Fairness of exposure should be achieved with respect to the geographic location of the articles (wewill provide geographic location annotations). For the evaluation topics, in addition to geographic
fairness, to the extent that biographical articles are relevant to the topic, the rankings should also befair with respect to an undisclosed demographic attribute of the people that the biographies cover.

2.1

Task 1: WikiProject Coordinators

The first task is focused on WikiProject coordinators as users of the search system; their goal is to search
for relevant articles and produce a ranked list of articles needing work that other editors can then consult
when looking for work to do.
Output: The output for this task is a single ranking per query, consisting of 1000 articles.
Evaluation will be a multi-objective assessment of rankings by the following two criteria:• Relevance to a WikiProject topic. We will provide relevance assessments for articles for the training
queries derived from existing Wikipedia data; evaluation query relevance will be assessed by NIST
assessors. Ranking relevance will be computed with nDCG, using binary relevance and logarithmic
decay.
• Fairness with respect to the exposure of different fairness categories in the articles returned in response
to a query.
We will use attention-weighted rank fairness [5] to measure the fairness of each ranked list. This compares
cumulative exposure  across groups with a population estimator p̂ reflecting the target distribution; thesystem is more fair if the cumulative group exposure is close to the target distribution. This yields thefollowing metric for a result list L:
AWRF(L) = ∆((L), p̂)(1)
To compare the exposure distributions with a metric with the same range and similar interpretation as
nDCG, we will use the one minus the Jenson-Shannon divergence:∆(P1 , P2 ) = 1 −M=

1(DKL (P1 |M ) + DKL (P2 |M ))21(P1 + P2 )
2(2)
(3)

The final metric will be the product of AWRF and nDCG:
Metric1 = nDCG(L) · AWRF(L)(4)

This has the effect of requiring a system to do well on both relevance and fairness simultaneously in orderto score well on the overall task.

2

2.2

Task 2: Wikipedia Editors

The second task is focused on individual Wikipedia editors looking for work associated with a project. The
conceptual model is that rather than maintaining a fixed work list as in Task 1, a WikiProject coordinatorwould create a saved search, and when an editor looks for work they re-run the search. This means thatdifferent editors may receive different rankings for the same query, and differences in these rankings may beleveraged for providing fairness.
Output: The output of this task is 100 rankings per query, each consisting of 50 articles.
Evaluation will be a multi-objective assessment of rankings by the following three criteria:• Relevance to a WikiProject topic. We will provide relevance assessments for articles for the training
queries derived from existing Wikipedia data; evaluation query relevance will be assessed by NIST
assessors. Ranking relevance will be computed with nDCG.
• Work needed on the article (articles needing more work preferred). We provide the output of anarticle quality assessment tool for each article in the corpus; for the purposes of this track, we assume
lower-quality articles need more work.
• Fairness with respect to the exposure of different fairness categories in the articles returned in response
to a query.
This task will use expected exposure to compare the exposure article subjects receive in result rankings
to the ideal (or target) exposure they would receive based on their relevance and work-needed [2]. Thisaddresses fundamental limits in the ability to provide fair exposure in a single ranking by examining theexposure over multiple rankings.
Given a query q, a ranking policy will provide a distribution πq over rankings, the set of all (truncated)
permutations of documents. We consider the 100 rankings to be samples from this distribution. Note that
this is how we interpret the queries, but it does not mean that a stochastic policy is how the system must beimplemented — other implementation designs are certainly possible. The objective is to provide comparable
exposure to documents of comparable relevance and work-needed; to operationalize this, we define an anideal policy τ .
These policies are then used with a browsing model η : L → Rn to compute the per-query system
exposure q = Eπq [η] and target exposure ∗q = Eτq [η] (Diaz et al. [1] define two ways of computing these; wetentatively plan to use the ERR-based formula). The final metric is the expected exposure loss, the squared
Euclidean distance between system and target expected exposure:
EELq = kq − ∗q k22∗∗ 2= kq k22 − 2Tq q + kq k2

(5)
(6)

This metric decomposes into two sub-metrics that can be used to assess fairness/relevance tradeoffs: theexpected exposure disparity EED = kk22 and the expected exposure relevance EER = 2T ∗ . Our primaryevaluation metric will be group-aggregated EEL, to measure group fairness; for comparison, we will alsomeasure individual fairness for documents or document authors, and report EED and EER to characterizefairness/relevance tradeoffs. Since topical relevance and work-needed are integrated into expected exposure
loss through the target policy, we will not be combining EEL with separate relevance metrics; systems will
be ranked on their EEL.

3Data

This section provides details of the format of the test collection, topics and ground truth.
3

3.1

Obtaining the Data

The data set is distributed via Globus, and can be obtained in two ways.
First, it can be obtained via Globus, from our repository at http://boi.st/TREC2021Globus. From thissite, you can log in using your institution’s Globus account or your own Google account, and synchronize it
to your local Globus install or download it with Globus Connect Personal2 . This method has robust support
for restarting downloads and dealing with intermittent connections.
Second, it can be downloaded directly via HTTP from:https://data.boisestate.edu/library/Ekstrand-2021/TRECFairRanking2021/.

3.2Corpus

The corpus consists of articles from English Wikipedia. We have removed all redirect articles, but have left
the wikitext (markup Wikipedia uses to describe formatting) intact. This is provided as a JSON file, with
one record per line, and compressed with gzip (trec corpus.json.gz).
Each record contains the following fields:
id The unique numeric Wikipedia article identifier.
title The article title.
url The article URL, to comply with Wikipedia licensing attribution requirements.
text The full article text.
The contents of this corpus are prepared in accordance with, and licensed under, the CC BY-SA 3.0
license3 . The raw Wikipedia dump files used to produce this corpus are available in the source directory;
this is primarily for archival purposes, because Wikipedia does not publish dumps indefinitely.

3.3

TopicsEach of the track’s training topics is based on a single Wikiproject. The topic is also GZIP-compressed
JSON lines (file trec topics.json.gz), with each record containing:
id A query identifier (int)
title The Wikiproject title (string)
keywords A collection of search keywords forming the query text (list of str)
scope A textual description of the project scope, from its project page (string)
homepage The URL for the Wikiproject. This is provided for attribution and not expected to be used byyour system as it will not be present in the evaluation data (string)
rel docs A list of the page IDs of relevant pages (list of int)
The keywords are the primary query text. The scope is there to provide some additional context andpotentially support techniques for refining system queries.
In addition to topical relevance, for Task 2: Wikipedia Editors (Section 2.2), participants will also
be expected to return relevant documents that need more editing work done more highly than relevant
documents that need less work done.
2 https://www.globus.org/globus-connect-personal
3 https://creativecommons.org/licenses/by-sa/3.0/

4

3.4

Metadata and Fairness Categories

For training data, participants will be provided with a geographical fairness ground truth. For the evaluation
data, submitted systems will be evaluated on how fair their rankings are to the geographical fairness category
and an undisclosed personal demographic attribute.
We also provide a simple Wikimedia quality score (a float between 0 and 1 where 0 is no content on thepage and 1 is high quality) for optimizing for work-needed in Task 2. Work-needed can be operationalized
as the reverse—i.e. 1 minus this quality score. The discretized quality scores will be used as work-needed
for final system evaluation.
This data is provided together in a metadata file (trec metadata.json.gz), in which each line is themetadata for one article represented as a JSON record with the following keys:
page id Unique page identifier (int)quality score Continuous measure of article quality with 0 representing low quality and 1 representing highquality (float in range [0, 1])
quality score disc Discrete quality score in which the quality score is mapped to six ordinal categories
from low to high: Stub, Start, C, B, GA, FA (string)
geographic locations Continents that are associated with the article topic. Zero or many of: Africa,Antarctica, Asia, Europe, Latin America and the Caribbean, Northern America, Oceania (list of string)

3.5

OutputFor Task 1, participants should output results in rank order in a tab-separated file with two columns:
id The query ID for the topic
page id ID for the recommended article
For Task 2, this file should have 3 columns, to account for repeated rankings per query:
id Query ID
rep number Repeat Number (1-100)
page id ID for the recommended article

4

Submission Instructions

TBA with eval data.

5Limitations

The data and metrics in this task address a few specific types of unfairness, and do so partially. This isfundamentally true of any fairness intervention, and does not in any way diminish the value of the effort —it is impossible for any data set, task definition, or metric to fully capture fairness in a universal way, andall data and analyses have limitations.
We will provide a fuller accounting of known limitations in the notebook paper, but some include:5• For each Wikipedia article, we ascertain which, if any, continents are relevant to the content. Thisis determined by directly looking up several community-maintained (Wikidata) structured data statements about the article. These properties are checked for the presence of countries, which are thenmapped to continents via the United Nation’s geoscheme.4 While this data must meet Wikidata’s
verifiability guidelines,5 it does suffer from varying levels of incompleteness. For example, only 73%
of people on Wikidata have a country of citizenship property.6 Furthermore, structured data is itself
limited—e.g., country of citizenship does not appropriately capture people who are considered statelessthough these people may have many strong ties to a country. It is not easy to evaluate whether thisdata is missing at random or biased against certain regions of the world. Care should be taken wheninterpreting the absence of associated continents in the data, though evaluations for TREC will bebased solely on the known values. Further details can be found in the code repository.7• Our proxy for work-needed is a coarse proxy. It is based on just a few simple features (page length,sections, images, and references) and does not reflect the nuances of the work needed to craft a topquality Wikipedia article. A fully-fledged system for supporting Wikiprojects would also include amore nuanced approach to understanding the work needed for each article and how to appropriately
allocate this work.
• The task is limited to topics for which Wikipedia already has articles. These tasks are not able tocounteract biases in the processes by which articles come to exist (or are deleted) — recommending
articles that should exist but don’t is an interesting area for future study.

References[1] F. Diaz, D. Metzler, and S. Amer-Yahia. Relevance and ranking in online dating systems. In Proc. SIGIR’10, 2010.
[2] F. Diaz, B. Mitra, M. D. Ekstrand, A. J. Biega, and B. Carterette. Evaluating stochastic rankings withexpected exposure. In Proc. CIKM ’20, 2020. URL https://arxiv.org/abs/2004.13157.
[3] D. Pedreshi, S. Ruggieri, and F. Turini. Discrimination-aware data mining. In Proceedings of the 14th
ACM SIGKDD international conference on Knowledge discovery and data mining, pages 560–568, 2008.
[4] M. Redi, M. Gerlach, I. Johnson, J. Morgan, and L. Zia. A taxonomy of knowledge gaps for wikimedia
projects (first draft). arXiv preprint arXiv:2008.12314, 2020.
[5] P. Sapiezynski, W. Zeng, R. E Robertson, A. Mislove, and C. Wilson. Quantifying the impact of user
attentionon fair group representation in ranked lists. In Companion Proceedings of The 2019 World WideWeb Conference, pages 553–562, 2019.

4 https://en.wikipedia.org/wiki/United_Nations_geoscheme
5 https://www.wikidata.org/wiki/Wikidata:Verifiability
6 https://humaniki.wmcloud.org/gender-by-country
7 https://github.com/geohci/wiki-region-groundtruth

6