TREC Fair Ranking Track
Participant Instructions
August 13, 2019
For 2019, we will be adopting an academic search task, where we have a corpus of academic article
abstracts and queries submitted to a production academic search engine. The central goal of the Fair
Ranking track is to provide fair exposure to different groups of authors (a group fairness framing). We
recognize that there may be multiple group definitions (e.g. based on demographics, stature, topic) and hopefor the systems to be robust to these. As such, participants are expected to develop systems to optimize forfairness and relevance for arbitrary group definitions. We will not reveal the exact group definitions until
after the evaluation runs are submitted.
The track is set up as a reranking task. The track organizers will provide a sequence of queries, eachaccompanied by a varying-size set of documents; the task is to rerank the documents to produce result lists
that are fair and relevant.

1

ProtocolFor our fair ranking evaluation, we will be providing participants with a sequence Q of queries accompanied
by unordered sets of documents to rank. The document sets are of varying size. For each request (query q
and set of documents Dq ), participants should provide a ranked list of the documents from Dq . The final
system output is a sequence of rankings. Algorithm 1 presents a pseudocode of the evaluation protocol.
The rankings produced in response to queries in the sequence should balance two goals: (1) be relevantto the consumers and (2) be fair to the producers.
Algorithm 1 Evaluation protocolΠ ← {}for q, Dq ∈ Q doπ ← System(q, Dq )Π ← Π + [π]end forreturn Π2Evaluation

Unlike previous TREC tracks, you will receive multiple copies of the same query text–although the query id
will be different–and you may submit different rankings for each instance of the query. At evaluation time,
we will measure amortized performance over rankings produced for each given query, as well as across all
rankings and queries (macro- and micro- amortization, respectively.)
Given a sequence of queries Q and associated system rankings, we will be evaluating systems accordingto fair exposure of authors (Section 2.1.1) and relevance of documents (Section 2.2).

1i<latexit sha1_base64="8FUkZETWdPjzGvcCGIw94QQA49I=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuK+ix6MVjBfuB7VKyabYNzWZDkhXK2n/hxYMiXv033vw3pu0etPXBwOO9GWbmBZIzbVz328mtrW9sbuW3Czu7e/sHxcOjlo4TRWiTxDxWnQBrypmgTcMMpx2pKI4CTtvB+Gbmtx+p0iwW92YiqR/hoWAhI9hY6UGW9VNPsj477xdLbsWdA60SLyMlyNDoF796g5gkERWGcKx113Ol8VOsDCOcTgu9RFOJyRgPaddSgSOq/XR+8RSdWWWAwljZEgbN1d8TKY60nkSB7YywGellbyb+53UTE175KRMyMVSQxaIw4cjEaPY+GjBFieETSzBRzN6KyAgrTIwNqWBD8JZfXiWtasWrVap3F6X6dRZHHk7gFMrgwSXU4RYa0AQCAp7hFd4c7bw4787HojXnZDPH8AfO5w8hu5CP</latexit>p(s|⇡i )(1p(s|⇡i ))<latexit sha1_base64="nDnQ6dNvD4fdfslyMy1PdlePjKE=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0VoF5akCrosunFZwT6gCWEynbRDZyZhZiLUWPwVNy4Ucet/uPNvnLZZaOuBC4dz7uXee8KEUaUd59taWl5ZXVsvbBQ3t7Z3du29/ZaKU4lJE8cslp0QKcKoIE1NNSOdRBLEQ0ba4fB64rfviVQ0Fnd6lBCfo76gEcVIGymwD70+4hyV3dOkrB69hAa0UgnsklN1poCLxM1JCeRoBPaX14txyonQmCGluq6TaD9DUlPMyLjopYokCA9Rn3QNFYgT5WfT68fwxCg9GMXSlNBwqv6eyBBXasRD08mRHqh5byL+53VTHV36GRVJqonAs0VRyqCO4SQK2KOSYM1GhiAsqbkV4gGSCGsTWNGE4M6/vEhatap7Vq3dnpfqV3kcBXAEjkEZuOAC1MENaIAmwOABPINX8GY9WS/Wu/Uxa12y8pkD8AfW5w/eh5Qy</latexit>s<latexit sha1_base64="8r6dWOvdyfgmPOHEanDSQrmgcis=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48t2FpoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgTj25n/8IRK81jem0mCfkSHkoecUWOlpu6XK27VnYOsEi8nFcjR6Je/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSrlW9i2qteVmp3+RxFOEETuEcPLiCOtxBA1rAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MH3/OM+w==</latexit><latexit sha1_base64="hA2BMjcr0btVk5qY1/HOr4xO94Q=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48t2FpoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgTj25n/8IRK81jem0mCfkSHkoecUWOlJu+XK27VnYOsEi8nFcjR6Je/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSrlW9i2qteVmp3+RxFOEETuEcPLiCOtxBA1rAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MH0MuM8Q==</latexit><latexit sha1_base64="4aZVlc3yUwfgZs3BmfSYU5mN3iY=">AAAB9XicbVBNSwMxEJ31s9avqkcvwSJUhLJbBT0WvXisYD+gXUs2zbah2WxIskpZ+z+8eFDEq//Fm//GtN2Dtj4YeLw3w8y8QHKmjet+O0vLK6tr67mN/ObW9s5uYW+/oeNEEVonMY9VK8CaciZo3TDDaUsqiqOA02YwvJ74zQeqNIvFnRlJ6ke4L1jICDZWupcl/dSRrJuyU2980i0U3bI7BVokXkaKkKHWLXx1ejFJIioM4VjrtudK46dYGUY4Hec7iaYSkyHu07alAkdU++n06jE6tkoPhbGyJQyaqr8nUhxpPYoC2xlhM9Dz3kT8z2snJrz0UyZkYqggs0VhwpGJ0SQC1GOKEsNHlmCimL0VkQFWmBgbVN6G4M2/vEgalbJ3Vq7cnherV1kcOTiEIyiBBxdQhRuoQR0IKHiGV3hzHp0X5935mLUuOdnMAfyB8/kDynuSCw==</latexit>p(s|⇡i+1 )

i+1<latexit sha1_base64="V6JzOiRykNnsEUNaKm3lz7+CrK0=">AAAB6nicbVBNS8NAEJ34WetX1aOXxSIIQkmqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+Oyura+sbm4Wt4vbO7t5+6eCwqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7dRvPaHSPJaPZpygH9GB5CFn1FjpgZ97vVLZrbgzkGXi5aQMOeq90le3H7M0QmmYoFp3PDcxfkaV4UzgpNhNNSaUjegAO5ZKGqH2s9mpE3JqlT4JY2VLGjJTf09kNNJ6HAW2M6JmqBe9qfif10lNeO1nXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtO0YbgLb68TJrVindRqd5flms3eRwFOIYTOAMPrqAGd1CHBjAYwDO8wpsjnBfn3fmYt644+cwR/IHz+QOnEY1h</latexit>(1<latexit sha1_base64="vpLKKegzKxYUGZcnDMdLRHs/VCc=">AAACAXicbVDLSgMxFM3UV62vUTeCm8EitIhlpgq6LLpxWcE+oDMMmTTThiaZkGSEMtaNv+LGhSJu/Qt3/o3pY6GtBy4czrmXe++JBCVKu+63lVtaXlldy68XNja3tnfs3b2mSlKJcAMlNJHtCCpMCccNTTTFbSExZBHFrWhwPfZb91gqkvA7PRQ4YLDHSUwQ1EYK7QO/BxmDJe9UlNSDL0iYkRNvVC6HdtGtuBM4i8SbkSKYoR7aX343QSnDXCMKlep4rtBBBqUmiOJRwU8VFhANYA93DOWQYRVkkw9GzrFRuk6cSFNcOxP190QGmVJDFplOBnVfzXtj8T+vk+r4MsgIF6nGHE0XxSl1dOKM43C6RGKk6dAQiCQxtzqoDyVE2oRWMCF48y8vkma14p1VqrfnxdrVLI48OARHoAQ8cAFq4AbUQQMg8AiewSt4s56sF+vd+pi25qzZzD74A+vzB5UKla4=</latexit>p(s|⇡i+1 ))Figure 1: Attention model.

2.1

Measuring Fairness

2.1.1

Measuring Author Exposure for a Single Ranking

In order to measure exposure, we adopt the browsing model underlying the Expected Reciprocal Rank metric[1]. Given a ranking π, the exposure of author a is,

ni−1X
Y
γ i−1eπa =(1 − p(s|πj )) I(πi ∈ Da )i=1

j=1n number of documents in ranking πDa documents including a as an author
πi document at position iγ continuation probability (fixed to 0 for the final position in the ranking)p(s|d) probability of stopping given user examined d
We present a graphical depiction of this model in Figure 1.
We will provide the value of the discounting factor γ, and will assume p(s|d) = f (rd ), where rd is therelevance of the document d and f is a monotonic transform of that relevance into a probability of being
satisfied.
We compute the amortized exposure for a as,X
ea =eπa(1)π∈Π

where Π is the sequence of all system rankings.
2.1.2

Measuring Author Relevance for a Single Ranking

The author relevance for a ranking π is defined as,raπ =Xp(s|d)(2)d∈Da

Notice that this metric is independent of the ranking but not the query. As with amortized exposure, wedefine amortized relevance as the sum over all rankings Π.

2

2.1.3

Measuring Group Fairness

Assume that each author is assigned to exactly one of |G| groups. Let Ag be the set of all authors in groupg. The group exposure and relevance metrics are defined as,
Pa∈Ag ea
PEg = P(3)g 0 ∈Ga∈Ag0 eaPa∈Ag raP
Rg = P(4)g 0 ∈Ga∈Ag0 raWe assume that groups should receive exposure proportional to relevance. We adopt the following
measure to quantify the deviation from this ideal exposure,∆g = Eg − Rg

(5)

Given this relevance-normalized measure of exposure, we can compute the fair exposure using the `2norm,sX∆2g(6)∆=g∈G

Because this metric does not capture some of the nuance of how over- and under-exposure is distributed, wewill adopt secondary metrics in our final analysis of results.

2.2Measuring Relevance

We will measure the quality of a ranking for the searchers as the expected utility, assuming the same attention
model as used for our fairness metric,
ni−1XYγ i−1uπ =(1 − p(s|πj )) p(s|πi )
(7)
i=1

j=1

P
1

We average all utilities of rankings, U = |Π|2.3

ππ∈Π u as our final relevance metric.

Trading Off Fairness and Relevance

Although a system could, in theory, achieve the optimal relevance and fairness, in practice, relevance will
degrade as fairness improves. As such, we will measure the trade-offs between fairness to producers andquality for consumers as an auxiliary metric.

3Data

3.1

Input

There are three main inputs available to you: the corpus of articles to search, the example group definition
file to help you develop and test your solution, and the queries.
3.1.1

Corpus

The corpus for this project is the Semantic Scholar (S2) Open Corpus from the Allen Institute for Artificial
Intelligence. It can be downloaded from http://api.semanticscholar.org/corpus/, and consists of 47
1GB data files. Each file is compressed JSON, where each line is a JSON object describing one paper. Thefollowing data are available for most papers:
3• S2 Paper ID• DOI• Title• Abstract• Authors (resolved to author IDs)
• Inbound and outbound citations (resolved to S2 paper IDs)
3.1.2

Example Group Definition Data

To help you get started, we have provided the file fair-TREC-sample-author-groups.csv containing group
ids for authors in the S2 corpus. This group definition will not be our final group definition (we will evaluate
using multiple various group definitions), but you can use it to start working on the task.
This CSV file contains two columns:
1. The author column has the S2 ID of the author.
2. The gid column has the author’s group identifier.
3.1.3

Queries

The training queries are in fair-TREC-training-sample.json; this is a JSON file where each line is a
JSON object (a dictionary) describing one query:
1. Query ID (’qid’)
2. Query string (’query’)3. Query frequency (’frequency’)
4. A list of documents (’documents’) with relevance information; this is a list of dictionaries with twokeys: ‘doc id’ and ‘relevance’
Runs will be submitted over query sequences: ordered sequences of queries that may contain duplicates.
We will provide the set of query sequences for official runs 1 month prior to the deadline; for training anddevelopment, use the provided script (query-sequence-generator.py) to generate query sequences fromthe training queries distributions.
A query sequence is a CSV file with the following fields:
1. <sequence id>.<query number in sequence>
2. <query id> (to look up in query file)
Please remember to treat each sequence in the file separately. You should not treat this as asingle sequence to operate on.
The evaluation query sequences will be accompanied by a query file that has no relevance informationand has all query frequencies set to 0, to enable you to only look up the queries and document sets to rerank.

4

3.2

OutputFor each query sequence, your submitted ranking will be a JSON file where each line is a JSON object (adictionary) containing your ranking results:• <sequence id>.<query number in sequence> (‘q num’)• <query id> (to look up in query file) (‘qid’)• An ordered list of document IDs (of the documents to be reranked for the query) (‘ranking’)

References[1] O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In
Proceedings of the 18th ACM conference on Information and knowledge management, CIKM ’09, pages
621–630, New York, NY, USA, 2009. ACM.

5