TREC 2020 Fair Ranking Track
Participant Instructions
August 19, 2020
For 2020, we will again be adopting an academic search task, where we have a corpus of academic
article abstracts and queries submitted to a production academic search engine. The central goal of theFair Ranking track is to provide fair exposure to different groups of authors (a group fairness framing). We
recognize that there may be multiple group definitions (e.g. based on demographics, stature, topic) and hopefor the systems to be robust to these. As such, participants are expected to develop systems to optimize forfairness and relevance for arbitrary group definitions. We will not reveal the exact group definitions until
after the evaluation runs are submitted.
The track is set up as two tasks, reranking and retrieval, with a shared evaluation.
Rerank runs will sort a query-dependent list of documents to simultaneously provide fairness and relevance.
Retrieval runs will return 100-item rankings from the corpus in response to a query string.
The track organizers will provide a sequence of queries, each accompanied by a varying-size set of documents. The same queries will be used for both tasks; participants are asked not to use the test queries’
rerank sets as a component of their retrieval model training.

1Protocol

For our fair ranking evaluation, we will be providing participants with a sequence Q of queries accompanied
by unordered sets of documents to rank. The document sets are of varying size. For each request (queryq and set of documents Dq ), participants should provide a ranked list of the documents from Dq . For theretrieval task, q is the set of all documents in our corpus and participants will be asked to return a fixed setof documents. The final system output is a sequence of rankings. Algorithm 1 presents a pseudocode of theevaluation protocol.
The rankings produced in response to queries in the sequence should balance two goals: (1) be relevantto the consumers and (2) be fair to the producers.
Algorithm 1 Evaluation protocol∀q, Dq ∈ Q, Πq ← {}for q, Dq ∈ Q doπ ← System(q, Dq )
Πq ← Πq ∪ {π}end forreturn {Πq }1i<latexit sha1_base64="8FUkZETWdPjzGvcCGIw94QQA49I=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuK+ix6MVjBfuB7VKyabYNzWZDkhXK2n/hxYMiXv033vw3pu0etPXBwOO9GWbmBZIzbVz328mtrW9sbuW3Czu7e/sHxcOjlo4TRWiTxDxWnQBrypmgTcMMpx2pKI4CTtvB+Gbmtx+p0iwW92YiqR/hoWAhI9hY6UGW9VNPsj477xdLbsWdA60SLyMlyNDoF796g5gkERWGcKx113Ol8VOsDCOcTgu9RFOJyRgPaddSgSOq/XR+8RSdWWWAwljZEgbN1d8TKY60nkSB7YywGellbyb+53UTE175KRMyMVSQxaIw4cjEaPY+GjBFieETSzBRzN6KyAgrTIwNqWBD8JZfXiWtasWrVap3F6X6dRZHHk7gFMrgwSXU4RYa0AQCAp7hFd4c7bw4787HojXnZDPH8AfO5w8hu5CP</latexit>p(s|⇡i )(1p(s|⇡i ))<latexit sha1_base64="nDnQ6dNvD4fdfslyMy1PdlePjKE=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0VoF5akCrosunFZwT6gCWEynbRDZyZhZiLUWPwVNy4Ucet/uPNvnLZZaOuBC4dz7uXee8KEUaUd59taWl5ZXVsvbBQ3t7Z3du29/ZaKU4lJE8cslp0QKcKoIE1NNSOdRBLEQ0ba4fB64rfviVQ0Fnd6lBCfo76gEcVIGymwD70+4hyV3dOkrB69hAa0UgnsklN1poCLxM1JCeRoBPaX14txyonQmCGluq6TaD9DUlPMyLjopYokCA9Rn3QNFYgT5WfT68fwxCg9GMXSlNBwqv6eyBBXasRD08mRHqh5byL+53VTHV36GRVJqonAs0VRyqCO4SQK2KOSYM1GhiAsqbkV4gGSCGsTWNGE4M6/vEhatap7Vq3dnpfqV3kcBXAEjkEZuOAC1MENaIAmwOABPINX8GY9WS/Wu/Uxa12y8pkD8AfW5w/eh5Qy</latexit>

s<latexit sha1_base64="8r6dWOvdyfgmPOHEanDSQrmgcis=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48t2FpoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgTj25n/8IRK81jem0mCfkSHkoecUWOlpu6XK27VnYOsEi8nFcjR6Je/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSrlW9i2qteVmp3+RxFOEETuEcPLiCOtxBA1rAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MH3/OM+w==</latexit><latexit sha1_base64="hA2BMjcr0btVk5qY1/HOr4xO94Q=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48t2FpoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgTj25n/8IRK81jem0mCfkSHkoecUWOlJu+XK27VnYOsEi8nFcjR6Je/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSrlW9i2qteVmp3+RxFOEETuEcPLiCOtxBA1rAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MH0MuM8Q==</latexit><latexit sha1_base64="4aZVlc3yUwfgZs3BmfSYU5mN3iY=">AAAB9XicbVBNSwMxEJ31s9avqkcvwSJUhLJbBT0WvXisYD+gXUs2zbah2WxIskpZ+z+8eFDEq//Fm//GtN2Dtj4YeLw3w8y8QHKmjet+O0vLK6tr67mN/ObW9s5uYW+/oeNEEVonMY9VK8CaciZo3TDDaUsqiqOA02YwvJ74zQeqNIvFnRlJ6ke4L1jICDZWupcl/dSRrJuyU2980i0U3bI7BVokXkaKkKHWLXx1ejFJIioM4VjrtudK46dYGUY4Hec7iaYSkyHu07alAkdU++n06jE6tkoPhbGyJQyaqr8nUhxpPYoC2xlhM9Dz3kT8z2snJrz0UyZkYqggs0VhwpGJ0SQC1GOKEsNHlmCimL0VkQFWmBgbVN6G4M2/vEgalbJ3Vq7cnherV1kcOTiEIyiBBxdQhRuoQR0IKHiGV3hzHp0X5935mLUuOdnMAfyB8/kDynuSCw==</latexit>p(s|⇡i+1 )

i+1<latexit sha1_base64="V6JzOiRykNnsEUNaKm3lz7+CrK0=">AAAB6nicbVBNS8NAEJ34WetX1aOXxSIIQkmqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+Oyura+sbm4Wt4vbO7t5+6eCwqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7dRvPaHSPJaPZpygH9GB5CFn1FjpgZ97vVLZrbgzkGXi5aQMOeq90le3H7M0QmmYoFp3PDcxfkaV4UzgpNhNNSaUjegAO5ZKGqH2s9mpE3JqlT4JY2VLGjJTf09kNNJ6HAW2M6JmqBe9qfif10lNeO1nXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtO0YbgLb68TJrVindRqd5flms3eRwFOIYTOAMPrqAGd1CHBjAYwDO8wpsjnBfn3fmYt644+cwR/IHz+QOnEY1h</latexit>(1<latexit sha1_base64="vpLKKegzKxYUGZcnDMdLRHs/VCc=">AAACAXicbVDLSgMxFM3UV62vUTeCm8EitIhlpgq6LLpxWcE+oDMMmTTThiaZkGSEMtaNv+LGhSJu/Qt3/o3pY6GtBy4czrmXe++JBCVKu+63lVtaXlldy68XNja3tnfs3b2mSlKJcAMlNJHtCCpMCccNTTTFbSExZBHFrWhwPfZb91gqkvA7PRQ4YLDHSUwQ1EYK7QO/BxmDJe9UlNSDL0iYkRNvVC6HdtGtuBM4i8SbkSKYoR7aX343QSnDXCMKlep4rtBBBqUmiOJRwU8VFhANYA93DOWQYRVkkw9GzrFRuk6cSFNcOxP190QGmVJDFplOBnVfzXtj8T+vk+r4MsgIF6nGHE0XxSl1dOKM43C6RGKk6dAQiCQxtzqoDyVE2oRWMCF48y8vkma14p1VqrfnxdrVLI48OARHoAQ8cAFq4AbUQQMg8AiewSt4s56sF+vd+pi25qzZzD74A+vzB5UKla4=</latexit>p(s|⇡i+1 ))Figure 1: Attention model.

2

Evaluation

Unlike previous TREC tracks, you will receive multiple copies or impressions of the same query text–although
the query id will be different–and you may submit different rankings for each query impression. At evaluation
time, we will measure expected exposure of groups over rankings produced for each given query [2].
Given a sequence of queries Q and associated system rankings, we will be evaluating systems according
to fair exposure of authors and relevance of documents.

2.1

Measuring Author Exposure for a Single QueryIn order to measure exposure, we adopt the browsing model underlying the Expected Reciprocal Rank metric[1]. Given a static ranking π in response to a query impression, the exposure of author a is,
ni−1XYγ i−1eπa =(1 − p(s|πj )) I(πi ∈ Da )
i=1

j=1

n number of documents in ranking πDa documents including a as an author
πi document at position iγ continuation probability (fixed to 0 for the final position in the ranking)p(s|d) probability of stopping given user examined dWe present a graphical depiction of this model in Figure 1. We will use a discounting factor γ = 0.5, andwill assume p(s|d) = f (rd ), where rd is the relevance of the document d and f is a monotonic transform ofthat relevance into a probability of being satisfied.
A unique query will have a number of impressions. In order to compute the expected exposure for a, weconsider the set of all rankings presented by the system for that query Πq ,Xea =eπa(1)π∈Πq

The target expected exposure for a query is derived from Equation 1 assuming a policy that randomizes
amongst all permutations whose relevance monotonically degrades with rank [2].

22.2

Measuring Group Exposure for a Single Query

Assume that each author is assigned to exactly one of |G| groups. Let Ag be the set of all authors in groupg. The group expected exposure is defined as,
Xea(2)
Eg =a∈Ag

We define the target group expected exposure Eg∗ as Equation 2 using the individual target expected exposure(Section 2.1).

2.3

Expected Exposure Metric

We will evaluate systems using the per-query difference in system group expected exposure and target groupexpected exposure,
 12
X
g∈G

(Eg − Eg∗ )2 (3)Per-query metrics will be averaged to compute the summary metric for the run.

3Data

3.1

Input

There are three main inputs available to you: the corpus of articles to search, the example group definition
file to help you develop and test your solution, and the queries.
3.1.1

Paper and Author Data

The paper and author metadata CSV files we will be providing with the query data provide summary
information for papers and their authors in the rerank set. There are three files:
paper metadata.csv contains basic paper information: ID, title, year, venue, and the number of citations.
author metadata.csv contains author information: ID, name, citation count, paper count, and H-index.
authors for papers.csv contains the author list for each paper: paper ID, author ID, and position.
These files do not contain abstracts. To create a usable text index, you will need the corpus in the next
section.
3.1.2

Corpus

The full corpus for this project is the Semantic Scholar (S2) Open Corpus from the Allen Institute for Artificial
Intelligence. It can be downloaded from http://api.semanticscholar.org/corpus/, and consists of 47
1GB data files. Each file is compressed JSON, where each line is a JSON object describing one paper. The
following data are available for most papers:• S2 Paper ID• DOI• Title3• Abstract• Authors (resolved to author IDs)
• Inbound and outbound citations (resolved to S2 paper IDs)
3.1.3

Example Group Definition Data

To help you get started, we have provided the file fair-TREC-sample-author-groups.csv containing groupids for authors in the S2 corpus. This group definition will not be our final group definition (we will evaluate
using multiple various group definitions), but you can use it to start working on the task.
This CSV file contains two columns:
1. The author column has the S2 ID of the author.
2. The gid column has the author’s group identifier.
3.1.4

Queries

The training queries are in fair-TREC-training-sample.json; this is a JSON file where each line is a
JSON object (a dictionary) describing one query:
1. Query ID (’qid’)
2. Query string (’query’)3. Query frequency (’frequency’)
4. A list of documents (’documents’) with relevance information; this is a list of dictionaries with twokeys: ‘doc id’ and ‘relevance’
Runs will be submitted over query sequences: ordered sequences of queries that may contain duplicates.
We will provide the set of query sequences for official runs 1 month prior to the deadline; for training anddevelopment, use the provided script (query-sequence-generator.py) to generate query sequences fromthe training queries distributions.
A query sequence is a CSV file with the following fields:
1. <sequence id>.<query number in sequence>
2. <query id> (to look up in query file)
Please remember to treat each sequence in the file separately. You should not treat this as asingle sequence to operate on.
The evaluation query sequences will be accompanied by a query file that has no relevance informationand has all query frequencies set to 0, to enable you to only look up the queries and document sets to re-rank.

3.2

OutputFor each query sequence, your submitted ranking will be a JSON file where each line is a JSON object (adictionary) containing your ranking results:• <sequence id>.<query number in sequence> (‘q num’)• <query id> (to look up in query file) (‘qid’)• An ordered list of document IDs (of the documents to be re-ranked for the query) (‘ranking’)

4References[1] O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In
Proceedings of the 18th ACM conference on Information and knowledge management, CIKM ’09, pages
621–630, New York, NY, USA, 2009. ACM.
[2] F. Diaz, B. Mitra, M. D. Ekstrand, A. J. Biega, and B. Carterette. Evaluating stochastic rankings withexpected exposure, 2020.

5